Sliced Coordinate Analysis for Effective Dimension Reduction and Nonlinear Extensions

Sliced inverse regression (SIR) is an important method for reducing the dimensionality of input variables. Its goal is to estimate the effective dimension reduction directions. In classification settings, SIR is closely related to Fisher discriminant analysis. Motivated by reproducing kernel theory, we propose a notion of nonlinear effective dimension reduction and develop a nonlinear extension of SIR called kernel SIR (KSIR). Both SIR and KSIR are based on principal component analysis. Alternatively, based on principal coordinate analysis, we propose the dual versions of SIR and KSIR, which we refer to as sliced coordinate analysis (SCA) and kernel sliced coordinate analysis (KSCA), respectively. In the classification setting, we also call them discriminant coordinate analysis and kernel discriminant coordinate analysis. The computational complexities of SIR and KSIR rely on the dimensionality of the input vector and the number of input vectors, respectively, while those of SCA and KSCA both rely on the number of slices in the output. Thus, SCA and KSCA are very efficient dimension reduction methods.     

Dimension estimation in sufficient dimension reduction: A unifying approach

Sufficient Dimension Reduction (SDR) in regression comprises the estimation of the dimension of the smallest (central) dimension reduction subspace and its basis elements. For SDR methods based on a kernel matrix, such as SIR and SAVE, the dimension estimation is equivalent to the estimation of the rank of a random matrix which is the sample based estimate of the kernel. A test for the rank of a random matrix amounts to testing how many of its eigen or singular values are equal to zero. We propose two tests based on the smallest eigen or singular values of the estimated matrix: an asymptotic weighted chi-square test and a Wald-type asymptotic chi-square test. We also provide an asymptotic chi-square test for assessing whether elements of the left singular vectors of the random matrix are zero. These methods together constitute a unified approach for all SDR methods based on a kernel matrix that covers estimation of the central subspace and its dimension, as well as assessment of variable contribution to the lower-dimensional predictor projections with variable selection, a special case. A small power simulation study shows that the proposed and existing tests, specific to each SDR method, perform similarly with respect to power and achievement of the nominal level. Also, the importance of the choice of the number of slices as a tuning parameter is further exhibited.     

纵向数据边际模型非参数光滑方法的比较

对于纵向数据边际模型的均值函数, 有很多非参数估计方法, 其中回归样条, 光滑样条, 似乎不相关(SUR)核估计等方法在工作协方差阵正确指定时具有最小的渐近方差. 回归样条的渐近偏差与工作协方差阵无关, 而SUR核估计和光滑样条估计的渐近偏差却依赖于工作协方差阵. 本文主要研究了回归样条, 光滑样条和SUR核估计的效率问题. 通过模拟比较发现回归样条估计的表现比较稳定, 在大多数情况下比光滑样条估计和SUR核估计的效率高.     

Dimension reduction in functional regression with categorical predictor

In the present paper, we consider dimension reduction methods for functional regression with a scalar response and the predictors including a random curve and a categorical random variable. To deal with the categorical random variable, we propose three potential dimension reduction methods: partial functional sliced inverse regression, marginal functional sliced inverse regression and conditional functional sliced inverse regression. Furthermore, we investigate the relationships among the three methods. In addition, a new modified BIC criterion for determining the dimension of the effective dimension reduction space is developed. Real and simulation data examples are then presented to show the effectiveness of the proposed methods.     

Efficient and fast spline-backfitted kernel smoothing of additive models

A great deal of effort has been devoted to the inference of additive model in the last decade. Among existing procedures, the kernel type are too costly to implement for high dimensions or large sample sizes, while the spline type provide no asymptotic distribution or uniform convergence. We propose a one step backfitting estimator of the component function in an additive regression model, using spline estimators in the first stage followed by kernel/local linear estimators. Under weak conditions, the proposed estimator’s pointwise distribution is asymptotically equivalent to an univariate kernel/local linear estimator, hence the dimension is effectively reduced to one at any point. This dimension reduction holds uniformly over an interval under assumptions of normal errors. Monte Carlo evidence supports the asymptotic results for dimensions ranging from low to very high, and sample sizes ranging from moderate to large. The proposed confidence band is applied to the Boston housing data for linearity diagnosis. Supported in part by NSF awards DMS 0405330, 0706518, BCS 0308420 and SES 0127722.     

Dimension reduction based on weighted variance estimate

In this paper, we propose a new estimate for dimension reduction, called the weighted variance estimate (WVE), which includes Sliced Average Variance Estimate (SAVE) as a special case. Bootstrap method is used to select the best estimate from the WVE and to estimate the structure dimension. And this selected best estimate usually performs better than the existing methods such as Sliced Inverse Regression (SIR), SAVE, etc. Many methods such as SIR, SAVE, etc. usually put the same weight on each observation to estimate central subspace (CS). By introducing a weight function, WVE puts different weights on different observations according to distance of observations from CS. The weight function makes WVE have very good performance in general and complicated situations, for example, the distribution of regressor deviating severely from elliptical distribution which is the base of many methods, such as SIR, etc. And compared with many existing methods, WVE is insensitive to the distribution of the regressor. The consistency of the WVE is established. Simulations to compare the performances of WVE with other existing methods confirm the advantage of WVE. This work was supported by National Natural Science Foundation of China (Grant No. 10771015)     

Local bases for refinable spaces

We provide a new representation of a refinable shift invariant space with a compactly supported generator, in terms of functions with a special property of homogeneity. In particular, these functions include all the homogeneous polynomials that are reproducible by the generator, which links this representation to the accuracy of the space. We completely characterize the class of homogeneous functions in the space and show that they can reproduce the generator. As a result we conclude that the homogeneous functions can be constructed from the vectors associated to the spectrum of the scale matrix (a finite square matrix with entries from the mask of the generator). Furthermore, we prove that the kernel of the transition operator has the same dimension as the kernel of this finite matrix. This relation provides an easy test for the linear independence of the integer translates of the generator. This could be potentially useful in applications to approximation theory, wavelet theory and sampling.

     

W_2~m空间中样条插值算子与最佳逼近算子的一致性

由线性微分算子确定的样条是连接多项式样条与希氏空间中抽象算子样条的重要环节,对微分算子样条的研究,既可从更高的观点揭示和概括多项式样条,又可启示我们去发现抽象算子样条的一些新的理论和应用． Ｇｒｅｅｎ函数是研究微分算子样条的重要工具 ［１］,但在微分算子插值样条的计算及将样条用于数值分析中,再生核方法起着更重要的作用．文献［２］［３］给出了与二阶线性微分算子插值样条有关的再生核解析表达式;由此得到了二阶微分算子插值样条与空间Ｗ_２～１［ａ,ｂ］中最佳插值逼近算子的一致性;而且还利用再生核给出了Ｈｉ…     

On kernel method for sliced average variance estimation

In this paper, we use the kernel method to estimate sliced average variance estimation (SAVE) and prove that this estimator is both asymptotically normal and root n consistent. We use this kernel estimator to provide more insight about the differences between slicing estimation and other sophisticated local smoothing methods. Finally, we suggest a Bayes information criterion (BIC) to estimate the dimensionality of SAVE. Examples and real data are presented for illustrating our method.     

Asymptotics for Kernel Estimation of Slicing Average Third-Moment Estimation

To estimate central dimension-reduction space in multivariate nonparametric rcgression, Sliced Inverse Regression （SIR）, Sliced Average Variance Estimation （SAVE） and Slicing Average Third-moment Estimation （SAT） have been developed, Since slicing estimation has very different asymptotic behavior for SIR, and SAVE, the relevant study has been madc case by case, when the kernel estimators of SIH and SAVE share similar asymptotic properties. In this paper, we also investigate kernel estimation of SAT. We. prove the asymptotic normality, and show that, compared with tile existing results, the kernel Slnoothing for SIR, SAVE and SAT has very similar asymptotic behavior,     

W_2~m空间中样条插值算子与最佳逼近算子的一致性

This paper discusses generalized interpolating splines which determined by n order linear differential operators, and the best operators of interpolating approximation in W_2~m spaces, The explicit constructive method for the reproducing kernel in W_2~m space is presented, and proves the uniformity of spline interpolating operators and the best operators of interpolating approximation W_2~m space by reproducing kernel. The explicit expression of approximation error on a bounded ball in W_2~m space, and error estimation of spline operator of approximation are obtained.     

Sufficient Dimension Folding for Regression Mean Function

In this article, we consider sufficient dimension folding for the regression mean function when predictors are matrix- or array-valued. We propose a new concept named central mean dimension folding subspace and its two local estimation methods: folded outer product of gradients estimation (folded-OPG) and folded minimum average variance estimation (folded-MAVE). We establish the asymptotic properties for folded-MAVE. A modified BIC criterion is used to determine the dimensions of the central mean dimension folding subspace. We evaluate the performances of the two local estimation methods by simulated examples and demonstrate the efficacy of folded-MAVE in finite samples. And in particular, we apply our methods to analyze a longitudinal study of primary biliary cirrhosis. Supplementary materials for this article are available online.     

On the Nyström and Column-Sampling Methods for the Approximate Principal Components Analysis of Large Datasets

In this article, we analyze approximate methods for undertaking a principal components analysis (PCA) on large datasets. PCA is a classical dimension reduction method that involves the projection of the data onto the subspace spanned by the leading eigenvectors of the covariance matrix. This projection can be used either for exploratory purposes or as an input for further analysis, for example, regression. If the data have billions of entries or more, the computational and storage requirements for saving and manipulating the design matrix in fast memory are prohibitive. Recently, the Nyström and column-sampling methods have appeared in the numerical linear algebra community for the randomized approximation of the singular value decomposition of large matrices. However, their utility for statistical applications remains unclear. We compare these approximations theoretically by bounding the distance between the induced subspaces and the desired, but computationally infeasible, PCA subspace. Additionally we show empirically, through simulations and a real data example involving a corpus of emails, the trade-off of approximation accuracy and computational complexity.     

Solving Fredholm integral equations by approximating kernels by spline quasi-interpolants

We study two methods for solving a univariate Fredholm integral equation of the second kind, based on (left and right) partial approximations of the kernel K by a discrete quartic spline quasi-interpolant. The principle of each method is to approximate the kernel with respect to one variable, the other remaining free. This leads to an approximation of K by a degenerate kernel. We give error estimates for smooth functions, and we show that the method based on the left (resp. right) approximation of the kernel has an approximation order O(h ⁵) (resp. O(h ⁶)). We also compare the obtained formulae with projection methods.     

样条型矩阵有理插值

The matrix valued rational interpolation is very useful in the partial realization problem and model reduction for all the linear system theory. Lagrange basic functions have been used in matrix valued rational interpolation. In this paper, according to the property of cardinal spline interpolation, we constructed a kind of spline type matrix valued rational interpolation, which based on cardinal spline. This spline type interpolation can avoid instability of high order polynomial interpolation and we obtained a useful formula.     

Multivariate approximation by PDE splines

This work deals with an approximation method for multivariate functions from data constituted by a given data point set and a partial differential equation (PDE). The solution of our problem is called a PDE spline. We establish a variational characterization of the PDE spline and a convergence result of it to the function which the data are obtained. We estimate the order of the approximation error and finally, we present an example to illustrate the fitting method.     

Nonlinear approximation using Gaussian kernels

It is well known that nonlinear approximation has an advantage over linear schemes in the sense that it provides comparable approximation rates to those of the linear schemes, but to a larger class of approximands. This was established for spline approximations and for wavelet approximations, and more recently by DeVore and Ron (in press) [2] for homogeneous radial basis function (surface spline) approximations. However, no such results are known for the Gaussian function, the preferred kernel in machine learning and several engineering problems. We introduce and analyze in this paper a new algorithm for approximating functions using translates of Gaussian functions with varying tension parameters. At heart it employs the strategy for nonlinear approximation of DeVore-Ron, but it selects kernels by a method that is not straightforward. The crux of the difficulty lies in the necessity to vary the tension parameter in the Gaussian function spatially according to local information about the approximand: error analysis of Gaussian approximation schemes with varying tension are, by and large, an elusive target for approximators. We show that our algorithm is suitably optimal in the sense that it provides approximation rates similar to other established nonlinear methodologies like spline and wavelet approximations. As expected and desired, the approximation rates can be as high as needed and are essentially saturated only by the smoothness of the approximand.     

Successive direction extraction for estimating the central subspace in a multiple-index regression

In this paper we propose a dimension reduction method for estimating the directions in a multiple-index regression based on information extraction. This extends the recent work of Yin and Cook [X. Yin, R.D. Cook, Direction estimation in single-index regression, Biometrika 92 (2005) 371-384] who introduced the method and used it to estimate the direction in a single-index regression. While a formal extension seems conceptually straightforward, there is a fundamentally new aspect of our extension: We are able to show that, under the assumption of elliptical predictors, the estimation of multiple-index regressions can be decomposed into successive single-index estimation problems. This significantly reduces the computational complexity, because the nonparametric procedure involves only a one-dimensional search at each stage. In addition, we developed a permutation test to assist in estimating the dimension of a multiple-index regression.     

Integral representation of slowly growing equidistant splines

In this paper we consider polynomial splines S(x) with equidistant nodes which may grow as O (|x|^s). We present an integral representation of such splines with a distribution kernel. This representation is related to the Fourier integral of slowly growing functions. The part of the Fourier exponentials herewith play the so called exponential splines by Schoenberg. The integral representation provides a flexible tool for dealing with the growing equidistant splines. First, it allows us to construct a rich library of splines possessing the property that translations of any such spline form a basis of corresponding spline space. It is shown that any such spline is associated with a dual spline whose translations form a biorthogonal basis. As examples we present solutions of the problems of projection of a growing function onto spline spaces and of spline interpolation of a growing function. We derive formulas for approximate evaluation of splines projecting a function onto the spline space and establish therewith exact estimations of the approximation errors.     

Spline methods in geodetic approximation problems

This paper is concerned with spline methods in a reproducing kernel Hilbert space consisting of functions defined and harmonic in the outer space of a regular surface (e.g. sphere, ellipsoid, telluroid, geoid, (regularized) earth's surface). Spline methods are used to solve interpolation and smoothing problems with respect to a (fundamental) system of linear functional giving information about earth's gravity field. Best approximations to linear functionals are discussed. The spline of interpolation is characterized as the spline of best approximation in the sense of an appropriate (energy) norm.