Dimension estimation in sufficient dimension reduction: A unifying approach

Sufficient Dimension Reduction (SDR) in regression comprises the estimation of the dimension of the smallest (central) dimension reduction subspace and its basis elements. For SDR methods based on a kernel matrix, such as SIR and SAVE, the dimension estimation is equivalent to the estimation of the rank of a random matrix which is the sample based estimate of the kernel. A test for the rank of a random matrix amounts to testing how many of its eigen or singular values are equal to zero. We propose two tests based on the smallest eigen or singular values of the estimated matrix: an asymptotic weighted chi-square test and a Wald-type asymptotic chi-square test. We also provide an asymptotic chi-square test for assessing whether elements of the left singular vectors of the random matrix are zero. These methods together constitute a unified approach for all SDR methods based on a kernel matrix that covers estimation of the central subspace and its dimension, as well as assessment of variable contribution to the lower-dimensional predictor projections with variable selection, a special case. A small power simulation study shows that the proposed and existing tests, specific to each SDR method, perform similarly with respect to power and achievement of the nominal level. Also, the importance of the choice of the number of slices as a tuning parameter is further exhibited.     

On a dimension reduction regression with covariate adjustment

In this paper, we consider a semiparametric modeling with multi-indices when neither the response nor the predictors can be directly observed and there are distortions from some multiplicative factors. In contrast to the existing methods in which the response distortion deteriorates estimation efficacy even for a simple linear model, the dimension reduction technique presented in this paper interestingly does not have to account for distortion of the response variable. The observed response can be used directly whether distortion is present or not. The resulting dimension reduction estimators are shown to be consistent and asymptotically normal. The results can be employed to test whether the central dimension reduction subspace has been estimated appropriately and whether the components in the basis directions in the space are significant. Thus, the method provides an alternative for determining the structural dimension of the subspace and for variable selection. A simulation study is carried out to assess the performance of the proposed method. The analysis of a real dataset demonstrates the potential usefulness of distortion removal.     

An adaptive estimation of MAVE

Minimum average variance estimation (MAVE, Xia et al. (2002) [29]) is an effective dimension reduction method. It requires no strong probabilistic assumptions on the predictors, and can consistently estimate the central mean subspace. It is applicable to a wide range of models, including time series. However, the least squares criterion used in MAVE will lose its efficiency when the error is not normally distributed. In this article, we propose an adaptive MAVE which can be adaptive to different error distributions. We show that the proposed estimate has the same convergence rate as the original MAVE. An EM algorithm is proposed to implement the new adaptive MAVE. Using both simulation studies and a real data analysis, we demonstrate the superior finite sample performance of the proposed approach over the existing least squares based MAVE when the error distribution is non-normal and the comparable performance when the error is normal.     

Structural test in regression on functional variables

Many papers deal with structural testing procedures in multivariate regression. More recently, various estimators have been proposed for regression models involving functional explanatory variables. Thanks to these new estimators, we propose a theoretical framework for structural testing procedures adapted to functional regression. The procedures introduced in this paper are innovative and make the link between former works on functional regression and others on structural testing procedures in multivariate regression. We prove asymptotic properties of the level and the power of our procedures under general assumptions that cover a large scope of possible applications: tests for no effect, linearity, dimension reduction, …     

Multivariate analysis of variance with fewer observations than the dimension

In this article, we consider the problem of testing a linear hypothesis in a multivariate linear regression model which includes the case of testing the equality of mean vectors of several multivariate normal populations with common covariance matrix Σ, the so-called multivariate analysis of variance or MANOVA problem. However, we have fewer observations than the dimension of the random vectors. Two tests are proposed and their asymptotic distributions under the hypothesis as well as under the alternatives are given under some mild conditions. A theoretical comparison of these powers is made.     

Estimating the error distribution in nonparametric multiple regression with applications to model testing

In this paper we consider the estimation of the error distribution in a heteroscedastic nonparametric regression model with multivariate covariates. As estimator we consider the empirical distribution function of residuals, which are obtained from multivariate local polynomial fits of the regression and variance functions, respectively. Weak convergence of the empirical residual process to a Gaussian process is proved. We also consider various applications for testing model assumptions in nonparametric multiple regression. The model tests obtained are able to detect local alternatives that converge to zero at an n^−1/2-rate, independent of the covariate dimension. We consider in detail a test for additivity of the regression function.     

Partial sum process to check regression models with multiple correlated response: With an application for testing a change-point in profile data

We consider regression models with multiple correlated responses for each design point. Under the null hypothesis, a linear regression is assumed. For the least-squares residuals of this linear regression, we establish the limit of the partial sums. This limit is a projection on a certain subspace of the reproducing Kernel Hilbert space of a multivariate Brownian motion. Based on this limit, we propose a significance test of Kolmogorov-Smirnov type to test the null hypothesis and show that this result can be used to study a change-point problem in the case of linear profile data (panel data). We compare our proposed method, which does not rely on any distributional assumptions, with the likelihood ratio test in a simulation study.     

Estimation, prediction and the Stein phenomenon under divergence loss

We consider two problems: (1) estimate a normal mean under a general divergence loss introduced in [S. Amari, Differential geometry of curved exponential families — curvatures and information loss, Ann. Statist. 10 (1982) 357-387] and [N. Cressie, T.R.C. Read, Multinomial goodness-of-fit tests, J. Roy. Statist. Soc. Ser. B. 46 (1984) 440-464] and (2) find a predictive density of a new observation drawn independently of observations sampled from a normal distribution with the same mean but possibly with a different variance under the same loss. The general divergence loss includes as special cases both the Kullback-Leibler and Bhattacharyya-Hellinger losses. The sample mean, which is a Bayes estimator of the population mean under this loss and the improper uniform prior, is shown to be minimax in any arbitrary dimension. A counterpart of this result for predictive density is also proved in any arbitrary dimension. The admissibility of these rules holds in one dimension, and we conjecture that the result is true in two dimensions as well. However, the general Baranchick [A.J. Baranchick, a family of minimax estimators of the mean of a multivariate normal distribution, Ann. Math. Statist. 41 (1970) 642-645] class of estimators, which includes the James-Stein estimator and the Strawderman [W.E. Strawderman, Proper Bayes minimax estimators of the multivariate normal mean, Ann. Math. Statist. 42 (1971) 385-388] class of estimators, dominates the sample mean in three or higher dimensions for the estimation problem. An analogous class of predictive densities is defined and any member of this class is shown to dominate the predictive density corresponding to a uniform prior in three or higher dimensions. For the prediction problem, in the special case of Kullback-Leibler loss, our results complement to a certain extent some of the recent important work of Komaki [F. Komaki, A shrinkage predictive distribution for multivariate normal observations, Biometrika 88 (2001) 859-864] and George, Liang and Xu [E.I. George, F. Liang, X. Xu, Improved minimax predictive densities under Kullbak-Leibler loss, Ann. Statist. 34 (2006) 78-92]. While our proposed approach produces a general class of predictive densities (not necessarily Bayes, but not excluding Bayes predictors) dominating the predictive density under a uniform prior. We show also that various modifications of the James-Stein estimator continue to dominate the sample mean, and by the duality of estimation and predictive density results which we will show, similar results continue to hold for the prediction problem as well.     

Robust dimension reduction based on canonical correlation

The canonical correlation (CANCOR) method for dimension reduction in a regression setting is based on the classical estimates of the first and second moments of the data, and therefore sensitive to outliers. In this paper, we study a weighted canonical correlation (WCANCOR) method, which captures a subspace of the central dimension reduction subspace, as well as its asymptotic properties. In the proposed WCANCOR method, each observation is weighted based on its Mahalanobis distance to the location of the predictor distribution. Robust estimates of the location and scatter, such as the minimum covariance determinant (MCD) estimator of Rousseeuw [P.J. Rousseeuw, Multivariate estimation with high breakdown point, Mathematical Statistics and Applications B (1985) 283-297], can be used to compute the Mahalanobis distance. To determine the number of significant dimensions in WCANCOR, a weighted permutation test is considered. A comparison of SIR, CANCOR and WCANCOR is also made through simulation studies to show the robustness of WCANCOR to outlying observations. As an example, the Boston housing data is analyzed using the proposed WCANCOR method.     

Minimax estimation for singular linear multivariate models with mixed uncertainty

The problem of minimax estimation is examined for the linear multivariate statistically indeterminate observation model with mixed uncertainty. The a priori information on the distributions of model parameters is formulated in terms of second-order moment characteristics. It is shown that in the regular case the minimax estimate is defined explicitly via the solution of the dual optimization problem. For singular models, the method of dual optimization is developed by means of using the Tikhonov regularization techniques. Several particular cases which are widely used in practice are also considered.     

A new test for multivariate normality

We propose a new class of rotation invariant and consistent goodness-of-fit tests for multivariate distributions based on Euclidean distance between sample elements. The proposed test applies to any multivariate distribution with finite second moments. In this article we apply the new method for testing multivariate normality when parameters are estimated. The resulting test is affine invariant and consistent against all fixed alternatives. A comparative Monte Carlo study suggests that our test is a powerful competitor to existing tests, and is very sensitive against heavy tailed alternatives.     

On the structure of the quadratic subspace in discriminant analysis

The concept of quadratic subspace is introduced as a helpful tool for dimension reduction in quadratic discriminant analysis (QDA). It is argued that an adequate representation of the quadratic subspace may lead to better methods for both data representation and classification. Several theoretical results describe the structure of the quadratic subspace, that is shown to contain some of the subspaces previously proposed in the literature for finding differences between the class means and covariances. A suitable assumption of orthogonality between location and dispersion subspaces allows us to derive a convenient reduced version of the full QDA rule. The behavior of these ideas in practice is illustrated with three real data examples.     

The pairwise beta distribution: A flexible parametric multivariate model for extremes

We present a new parametric model for the angular measure of a multivariate extreme value distribution. Unlike many parametric models that are limited to the bivariate case, the flexible model can describe the extremes of random vectors of dimension greater than two. The novel construction method relies on a geometric interpretation of the requirements of a valid angular measure. An advantage of this model is that its parameters directly affect the level of dependence between each pair of components of the random vector, and as such the parameters of the model are more interpretable than those of earlier parametric models for multivariate extremes. The model is applied to air quality data and simulated spatial data.     

Testing multivariate normality in incomplete data of small sample size

In longitudinal studies with small samples and incomplete data, multivariate normal-based models continue to be a powerful tool for analysis. This has included a broad scope of biomedical studies. Testing the assumption of multivariate normality (MVN) is critical. Although many methods are available for testing normality in complete data with large samples, a few deal with the testing in small samples. For example, Liang et al. (J. Statist. Planning and Inference 86 (2000) 129) propose a projection procedure for testing MVN for complete-data with small samples where the sample sizes may be close to the dimension. To our knowledge, no statistical methods for testing MVN in incomplete data with small samples are yet available. This article develops a test procedure in such a setting using multiple imputations and the projection test. To utilize the incomplete data structure in multiple imputation, we adopt a noniterative inverse Bayes formulae (IBF) sampling procedure instead of the iterative Gibbs sampling to generate iid samples. Simulations are performed for both complete and incomplete data when the sample size is less than the dimension. The method is illustrated with a real study on an anticancer drug.     

Multivariate nonparametric tests in a randomized complete block design

In this paper multivariate extensions of the Friedman and Page tests for the comparison of several treatments are introduced. Related unadjusted and adjusted treatment effect estimates for the multivariate response variable are also found and their properties discussed. The test statistics and estimates are analogous to the traditional univariate methods. In test constructions, the univariate ranks are replaced by multivariate spatial ranks (J. Nonparam. Statist. 5 (1995) 201). Asymptotic theory is developed to provide approximations for the limiting distributions of the test statistics and estimates. Limiting efficiencies of the tests and treatment effect estimates are found in the multivariate normal and t distribution cases. The tests are rotation invariant only, but affine invariant versions can be easily constructed. The theory is illustrated by an example.     

Asymptotic distributions of two “synthetic data” estimators for censored single-index models

The censored single-index model provides a flexible way for modelling the association between a response and a set of predictor variables when the response variable is randomly censored and the link function is unknown. It presents a technique for “dimension reduction” in semiparametric censored regression models and generalizes the existing accelerated failure time models for survival analysis. This paper proposes two methods for estimation of single-index models with randomly censored samples. We first transform the censored data into synthetic data or pseudo-responses unbiasedly, then obtain estimates of the index coefficients by the rOPG or rMAVE procedures of Xia (2006) [1]. Finally, we estimate the unknown nonparametric link function using techniques for univariate censored nonparametric regression. The estimators for the index coefficients are shown to be root-n consistent and asymptotically normal. In addition, the estimator for the unknown regression function is a local linear kernel regression estimator and can be estimated with the same efficiency as the parameters are known. Monte Carlo simulations are conducted to illustrate the proposed methodologies.     

Power of edge exclusion tests for graphical log-linear models

Asymptotic multivariate normal approximations to the joint distributions of edge exclusion test statistics for saturated graphical log-linear models, with all variables binary, are derived. Non-signed and signed square-root versions of the likelihood ratio, Wald and score test statistics are considered. Non-central chi-squared approximations are also considered for the non-signed versions of the test statistics. Simulation results are used to assess the quality of the proposed approximations. These approximations are used to estimate the overall power of edge exclusion tests. Power calculations are illustrated using data on university admissions.     

A note on the structure of the quadratic subspace in discriminant analysis

This paper explores some properties of the quadratic subspace, a tool for dimension reduction in discriminant analysis ( [Velilla, 2008] and [Velilla, 2010]). This linear manifold has a fairly complex structure, and it may sometimes include components with both mean and covariance separation properties. In this case, an assumption of orthogonality between the leading location directions and the bulk of the dispersion subspaces can help to find an adequate directional representation of it in practice. Two real data sets are analyzed.     

Stable Estimation in Dimension Reduction

We introduce stable estimation procedures for several aspects of a sufficient dimension-reduction matrix. We first propose a stable method for estimating structural dimension, which only selects the correct directions in the central subspace with no false positive selection. We then provide a Grassmann manifold sparse estimate for the central subspace. By using subsampling, we develop an ensemble method to obtain a stable nonsparse estimate for the central subspace. This ensemble idea is also used to stabilize the choice of the number of slices in sliced inverse methods. Theoretical results are established, and the efficacy of the proposed stable methods is demonstrated by simulation studies and the analysis of Hitters’ salary data. Supplementary materials for this article are available online.     

Estimation of a multivariate stochastic volatility density by kernel deconvolution

We consider a continuous time stochastic volatility model. The model contains a stationary volatility process. We aim to estimate the multivariate density of the finite-dimensional distributions of this process. We assume that we observe the process at discrete equidistant instants of time. The distance between two consecutive sampling times is assumed to tend to zero.A multivariate Fourier-type deconvolution kernel density estimator based on the logarithm of the squared processes is proposed to estimate the multivariate volatility density. An expansion of the bias and a bound on the variance are derived.