Sparse principal component analysis subject to prespecified cardinality of loadings

Most of the existing procedures for sparse principal component analysis (PCA) use a penalty function to obtain a sparse matrix of weights by which a data matrix is post-multiplied to produce PC scores. In this paper, we propose a new sparse PCA procedure which differs from the existing ones in two ways. First, the new procedure does not sparsify the weight matrix. Instead, the so-called loadings matrix is sparsified by which the score matrix is post-multiplied to approximate the data matrix. Second, the cardinality of the loading matrix i.e., the total number of nonzero loadings, is pre-specified to be an integer without using penalty functions. The procedure is called unpenalized sparse loading PCA (USLPCA). A desirable property of USLPCA is that the indices for the percentages of explained variances can be defined in the same form as in the standard PCA. We develop an alternate least squares algorithm for USLPCA which uses the fact that the PCA loss function can be decomposed as a sum of a term irrelevant to the loadings, and another one being easily minimized under cardinality constraints. A procedure is also presented for selecting the best cardinality using information criteria. The procedures are assessed in a simulation study and illustrated with real data examples.     

Estimation of multivariate normal covariance and precision matrices in a star-shape model with missing data

In this paper, we study the problem of estimating the covariance matrix Σ and the precision matrix Ω (the inverse of the covariance matrix) in a star-shape model with missing data. By considering a type of Cholesky decomposition of the precision matrix Ω=Ψ^′Ψ, where Ψ is a lower triangular matrix with positive diagonal elements, we get the MLEs of the covariance matrix and precision matrix and prove that both of them are biased. Based on the MLEs, unbiased estimators of the covariance matrix and precision matrix are obtained. A special group G, which is a subgroup of the group consisting all lower triangular matrices, is introduced. By choosing the left invariant Haar measure on G as a prior, we obtain the closed forms of the best equivariant estimates of Ω under any of the Stein loss, the entropy loss, and the symmetric loss. Consequently, the MLE of the precision matrix (covariance matrix) is inadmissible under any of the above three loss functions. Some simulation results are given for illustration.     

Consistent tuning parameter selection in high dimensional sparse linear regression

An exhaustive search as required for traditional variable selection methods is impractical in high dimensional statistical modeling. Thus, to conduct variable selection, various forms of penalized estimators with good statistical and computational properties, have been proposed during the past two decades. The attractive properties of these shrinkage and selection estimators, however, depend critically on the size of regularization which controls model complexity. In this paper, we consider the problem of consistent tuning parameter selection in high dimensional sparse linear regression where the dimension of the predictor vector is larger than the size of the sample. First, we propose a family of high dimensional Bayesian Information Criteria (HBIC), and then investigate the selection consistency, extending the results of the extended Bayesian Information Criterion (EBIC), in Chen and Chen (2008) to ultra-high dimensional situations. Second, we develop a two-step procedure, the SIS+AENET, to conduct variable selection in p>n situations. The consistency of tuning parameter selection is established under fairly mild technical conditions. Simulation studies are presented to confirm theoretical findings, and an empirical example is given to illustrate the use in the internet advertising data.     

Effective PCA for high-dimension, low-sample-size data with singular value decomposition of cross data matrix

In this paper, we propose a new methodology to deal with PCA in high-dimension, low-sample-size (HDLSS) data situations. We give an idea of estimating eigenvalues via singular values of a cross data matrix. We provide consistency properties of the eigenvalue estimation as well as its limiting distribution when the dimension d and the sample size n both grow to infinity in such a way that n is much lower than d. We apply the new methodology to estimating PC directions and PC scores in HDLSS data situations. We give an application of the findings in this paper to a mixture model to classify a dataset into two clusters. We demonstrate how the new methodology performs by using HDLSS data from a microarray study of prostate cancer.     

Effective PCA for high-dimension, low-sample-size data with noise reduction via geometric representations

In this article, we propose a new estimation methodology to deal with PCA for high-dimension, low-sample-size (HDLSS) data. We first show that HDLSS datasets have different geometric representations depending on whether a ρ-mixing-type dependency appears in variables or not. When the ρ-mixing-type dependency appears in variables, the HDLSS data converge to an n-dimensional surface of unit sphere with increasing dimension. We pay special attention to this phenomenon. We propose a method called the noise-reduction methodology to estimate eigenvalues of a HDLSS dataset. We show that the eigenvalue estimator holds consistency properties along with its limiting distribution in HDLSS context. We consider consistency properties of PC directions. We apply the noise-reduction methodology to estimating PC scores. We also give an application in the discriminant analysis for HDLSS datasets by using the inverse covariance matrix estimator induced by the noise-reduction methodology.     

Multivariate analysis of variance with fewer observations than the dimension

In this article, we consider the problem of testing a linear hypothesis in a multivariate linear regression model which includes the case of testing the equality of mean vectors of several multivariate normal populations with common covariance matrix Σ, the so-called multivariate analysis of variance or MANOVA problem. However, we have fewer observations than the dimension of the random vectors. Two tests are proposed and their asymptotic distributions under the hypothesis as well as under the alternatives are given under some mild conditions. A theoretical comparison of these powers is made.     

An augmented Lagrangian approach for sparse principal component analysis

Principal component analysis (PCA) is a widely used technique for data analysis and dimension reduction with numerous applications in science and engineering. However, the standard PCA suffers from the fact that the principal components (PCs) are usually linear combinations of all the original variables, and it is thus often difficult to interpret the PCs. To alleviate this drawback, various sparse PCA approaches were proposed in the literature (Cadima and Jolliffe in J Appl Stat 22:203–214, 1995; d’Aspremont et?al. in J Mach Learn Res 9:1269–1294, 2008; d’Aspremont et?al. SIAM Rev 49:434–448, 2007; Jolliffe in J Appl Stat 22:29–35, 1995; Journée et?al. in J Mach Learn Res 11:517–553, 2010; Jolliffe et?al. in J Comput Graph Stat 12:531–547, 2003; Moghaddam et?al. in Advances in neural information processing systems 18:915–922, MIT Press, Cambridge, 2006; Shen and Huang in J Multivar Anal 99(6):1015–1034, 2008; Zou et?al. in J Comput Graph Stat 15(2):265–286, 2006). Despite success in achieving sparsity, some important properties enjoyed by the standard PCA are lost in these methods such as uncorrelation of PCs and orthogonality of loading vectors. Also, the total explained variance that they attempt to maximize can be too optimistic. In this paper we propose a new formulation for sparse PCA, aiming at finding sparse and nearly uncorrelated PCs with orthogonal loading vectors while explaining as much of the total variance as possible. We also develop a novel augmented Lagrangian method for solving a class of nonsmooth constrained optimization problems, which is well suited for our formulation of sparse PCA. We show that it converges to a feasible point, and moreover under some regularity assumptions, it converges to a stationary point. Additionally, we propose two nonmonotone gradient methods for solving the augmented Lagrangian subproblems, and establish their global and local convergence. Finally, we compare our sparse PCA approach with several existing methods on synthetic (Zou et?al. in J Comput Graph Stat 15(2):265–286, 2006), Pitprops (Jeffers in Appl Stat 16:225–236, 1967), and gene expression data (Chin et?al in Cancer Cell 10:529C–541C, 2006), respectively. The computational results demonstrate that the sparse PCs produced by our approach substantially outperform those by other methods in terms of total explained variance, correlation of PCs, and orthogonality of loading vectors. Moreover, the experiments on random data show that our method is capable of solving large-scale problems within a reasonable amount of time.     

On testing for clusters using the sample covariance

This paper analyzes the problem of using the sample covariance matrix to detect the presence of clustering in p-variate data in the special case when the component covariance matrices are known up to a constant multiplier. For the case of testing one population against a mixture of two populations, tests are derived and shown to be optimal in a certain sense. Some of their distribution properties are derived exactly. Some remarks on the extensions of these tests to mixtures of k ≤ p populations are included. The paper is essentially a formal treatment (in a special case) of some well-known procedures. The methods used in deriving the distribution properties are applicable to a variety of other situations involving mixtures.     

Eigenvectors of a kurtosis matrix as interesting directions to reveal cluster structure

In this paper we study the properties of a kurtosis matrix and propose its eigenvectors as interesting directions to reveal the possible cluster structure of a data set. Under a mixture of elliptical distributions with proportional scatter matrix, it is shown that a subset of the eigenvectors of the fourth-order moment matrix corresponds to Fisher’s linear discriminant subspace. The eigenvectors of the estimated kurtosis matrix are consistent estimators of this subspace and its calculation is easy to implement and computationally efficient, which is particularly favourable when the ratio n/p is large.     

Some tests for the covariance matrix with fewer observations than the dimension under non-normality

This article analyzes whether some existing tests for the p×p covariance matrix Σ of the N independent identically distributed observation vectors work under non-normality. We focus on three hypotheses testing problems: (1) testing for sphericity, that is, the covariance matrix Σ is proportional to an identity matrix I_p; (2) the covariance matrix Σ is an identity matrix I_p; and (3) the covariance matrix is a diagonal matrix. It is shown that the tests proposed by Srivastava (2005) for the above three problems are robust under the non-normality assumption made in this article irrespective of whether N≤p or N≥p, but (N,p)→∞, and N/p may go to zero or infinity. Results are asymptotic and it may be noted that they may not hold for finite (N,p).     

Constructing priors based on model size for nondecomposable Gaussian graphical models: A simulation based approach

A method for constructing priors is proposed that allows the off-diagonal elements of the concentration matrix of Gaussian data to be zero. The priors have the property that the marginal prior distribution of the number of nonzero off-diagonal elements of the concentration matrix (referred to below as model size) can be specified flexibly. The priors have normalizing constants for each model size, rather than for each model, giving a tractable number of normalizing constants that need to be estimated. The article shows how to estimate the normalizing constants using Markov chain Monte Carlo simulation and supersedes the method of Wong et al. (2003) [24] because it is more accurate and more general. The method is applied to two examples. The first is a mixture of constrained Wisharts. The second is from Wong et al. (2003) [24] and decomposes the concentration matrix into a function of partial correlations and conditional variances using a mixture distribution on the matrix of partial correlations. The approach detects structural zeros in the concentration matrix and estimates the covariance matrix parsimoniously if the concentration matrix is sparse.     

Grouped Dirichlet distribution: A new tool for incomplete categorical data analysis

Motivated by the likelihood functions of several incomplete categorical data, this article introduces a new family of distributions, grouped Dirichlet distributions (GDD), which includes the classical Dirichlet distribution (DD) as a special case. First, we develop distribution theory for the GDD in its own right. Second, we use this expanded family as a new tool for statistical analysis of incomplete categorical data. Starting with a GDD with two partitions, we derive its stochastic representation that provides a simple procedure for simulation. Other properties such as mixed moments, mode, marginal and conditional distributions are also derived. The general GDD with more than two partitions is considered in a parallel manner. Three data sets from a case-control study, a leprosy survey, and a neurological study are used to illustrate how the GDD can be used as a new tool for analyzing incomplete categorical data. Our approach based on GDD has at least two advantages over the commonly used approach based on the DD in both frequentist and conjugate Bayesian inference: (a) in some cases, both the maximum likelihood and Bayes estimates have closed-form expressions in the new approach, but not so when they are based on the commonly-used approach; and (b) even if a closed-form solution is not available, the EM and data augmentation algorithms in the new approach converge much faster than in the commonly-used approach.     

Auto-associative models and generalized principal component analysis

In this paper, we propose auto-associative (AA) models to generalize Principal component analysis (PCA). AA models have been introduced in data analysis from a geometrical point of view. They are based on the approximation of the observations scatter-plot by a differentiable manifold. In this paper, they are interpreted as Projection pursuit models adapted to the auto-associative case. Their theoretical properties are established and are shown to extend the PCA ones. An iterative algorithm of construction is proposed and its principle is illustrated both on simulated and real data from image analysis.     

On the distribution of the ratio of the largest eigenvalue to the trace of a Wishart matrix

The ratio of the largest eigenvalue divided by the trace of a p×p random Wishart matrix with n degrees of freedom and an identity covariance matrix plays an important role in various hypothesis testing problems, both in statistics and in signal processing. In this paper we derive an approximate explicit expression for the distribution of this ratio, by considering the joint limit as both p,n→∞ with p/n→c. Our analysis reveals that even though asymptotically in this limit the ratio follows a Tracy-Widom (TW) distribution, one of the leading error terms depends on the second derivative of the TW distribution, and is non-negligible for practical values of p, in particular for determining tail probabilities. We thus propose to explicitly include this term in the approximate distribution for the ratio. We illustrate empirically using simulations that adding this term to the TW distribution yields a quite accurate expression to the empirical distribution of the ratio, even for small values of p,n.     

稳健稀疏主成分分析法及其实证研究

主成分分析是多元统计分析中一种非常经典的降维技术。然而,经典主成分分析却是对离群值非常敏感的,常因离群值的存在导致结果与实际不相符。另一方面,当主成分分析用于综合评价时,主成分的含义常因载荷间绝对值大小不分明而含糊不清,从而导致综合评价难以展开。本文通过使用稳健稀疏主成分分析法进行模拟实验和实证分析,结果表明:该方法不仅能很好地抵抗离群值的影响,而且还能准确地识别出离群样本。通过该方法得出的主成分的含义也较经典主成分分析和稳健主成分分析更加地明确和贴近实际。     

The restricted EM algorithm under inequality restrictions on the parameters

One of the most powerful algorithms for maximum likelihood estimation for many incomplete-data problems is the EM algorithm. The restricted EM algorithm for maximum likelihood estimation under linear restrictions on the parameters has been handled by Kim and Taylor (J. Amer. Statist. Assoc. 430 (1995) 708-716). This paper proposes an EM algorithm for maximum likelihood estimation under inequality restrictions A₀β?0, where β is the parameter vector in a linear model W=Xβ+ε and ε is an error variable distributed normally with mean zero and a known or unknown variance matrix Σ>0. Some convergence properties of the EM sequence are discussed. Furthermore, we consider the consistency of the restricted EM estimator and a related testing problem.     

Three-way analysis of imprecise data

Data are often affected by uncertainty. Uncertainty is usually referred to as randomness. Nonetheless, other sources of uncertainty may occur. In particular, the empirical information may also be affected by imprecision. Also in these cases it can be fruitful to analyze the underlying structure of the data. In this paper we address the problem of summarizing a sample of three-way imprecise data. In order to manage the different sources of uncertainty a twofold strategy is adopted. On the one hand, imprecise data are transformed into fuzzy sets by means of the so-called fuzzification process. The so-obtained fuzzy data are then analyzed by suitable generalizations of the Tucker3 and CANDECOMP/PARAFAC models, which are the two most popular three-way extensions of Principal Component Analysis. On the other hand, the statistical validity of the obtained underlying structure is evaluated by (nonparametric) bootstrapping. A simulation experiment is performed for assessing whether the use of fuzzy data is helpful in order to summarize three-way uncertain data. Finally, to show how our models work in practice, an application to real data is discussed.     

Asymptotic distributions of robust shape matrices and scales

It has been frequently observed in the literature that many multivariate statistical methods require the covariance or dispersion matrix Σ of an elliptical distribution only up to some scaling constant. If the topic of interest is not the scale but only the shape of the elliptical distribution, it is not meaningful to focus on the asymptotic distribution of an estimator for Σ or another matrix Γ∝Σ. In the present work, robust estimators for the shape matrix and the associated scale are investigated. Explicit expressions for their joint asymptotic distributions are derived. It turns out that if the joint asymptotic distribution is normal, the estimators presented are asymptotically independent for one and only one specific choice of the scale function. If it is non-normal (this holds for example if the estimators for the shape matrix and scale are based on the minimum volume ellipsoid estimator) only the scale function presented leads to asymptotically uncorrelated estimators. This is a generalization of a result obtained by Paindaveine [D. Paindaveine, A canonical definition of shape, Statistics and Probability Letters 78 (2008) 2240-2247] in the context of local asymptotic normality theory.     

Group-Wise Principal Component Analysis for Exploratory Data Analysis

In this article, we propose a new framework for matrix factorization based on principal component analysis (PCA) where sparsity is imposed. The structure to impose sparsity is defined in terms of groups of correlated variables found in correlation matrices or maps. The framework is based on three new contributions: an algorithm to identify the groups of variables in correlation maps, a visualization for the resulting groups, and a matrix factorization. Together with a method to compute correlation maps with minimum noise level, referred to as missing-data for exploratory data analysis (MEDA), these three contributions constitute a complete matrix factorization framework. Two real examples are used to illustrate the approach and compare it with PCA, sparse PCA, and structured sparse PCA. Supplementary materials for this article are available online.     

Methods for improvement in estimation of a normal mean matrix

This paper is concerned with the problem of estimating a matrix of means in multivariate normal distributions with an unknown covariance matrix under invariant quadratic loss. It is first shown that the modified Efron-Morris estimator is characterized as a certain empirical Bayes estimator. This estimator modifies the crude Efron-Morris estimator by adding a scalar shrinkage term. It is next shown that the idea of this modification provides a general method for improvement of estimators, which results in the further improvement on several minimax estimators. As a new method for improvement, an adaptive combination of the modified Stein and the James-Stein estimators is also proposed and is shown to be minimax. Through Monte Carlo studies of the risk behaviors, it is numerically shown that the proposed, combined estimator inherits the nice risk properties of both individual estimators and thus it has a very favorable risk behavior in a small sample case. Finally, the application to a two-way layout MANOVA model with interactions is discussed.