Optimal training sample allocation and asymptotic expansions for error rates in discriminant analysis

The sample-based rule obtained from Bayes classification rule by replacing the unknown parameters by ML estimates from a stratified training sample is used for the classification of a random observationX into one ofL populations. The asymptotic expansions in terms of the inverses of the training sample sizes for cross-validation, apparent and plug-in error rates are found. These are used to compare estimation methods of the error rate for a wide range of regular distributions as probability models for considered populations. The optimal training sample allocation minimizing the asymptotic expected error regret is found in the cases of widely applicable, positively skewed distributions (Rayleigh and Maxwell distributions). These probability models for populations are often met in ecology and biology. The results indicate that equal training sample sizes for each populations sometimes are not optimal, even when prior probabilities of populations are equal.     

An extension of Fisher's discriminant analysis for stochastic processes

In this paper we present a general notion of Fisher's linear discriminant analysis that extends the classical multivariate concept to situations that allow for function-valued random elements. The development uses a bijective mapping that connects a second order process to the reproducing kernel Hilbert space generated by its within class covariance kernel. This approach provides a seamless transition between Fisher's original development and infinite dimensional settings that lends itself well to computation via smoothing and regularization. Simulation results and real data examples are provided to illustrate the methodology.     

Convergence proof of matrix dynamics for online linear discriminant analysis

In this paper, we analyze matrix dynamics for online linear discriminant analysis (online LDA). Convergence of the dynamics have been studied for nonsingular cases; our main contribution is an analysis of singular cases, that is a key for efficient calculation without full-size square matrices. All fixed points of the dynamics are identified and their stability is examined.     

On nonparametric classification with missing covariates

General procedures are proposed for nonparametric classification in the presence of missing covariates. Both kernel-based imputation as well as Horvitz-Thompson-type inverse weighting approaches are employed to handle the presence of missing covariates. In the case of imputation, it is a certain regression function which is being imputed (and not the missing values). Using the theory of empirical processes, the performance of the resulting classifiers is assessed by obtaining exponential bounds on the deviations of their conditional errors from that of the Bayes classifier. These bounds, in conjunction with the Borel-Cantelli lemma, immediately provide various strong consistency results.     

On the consistency properties of linear and quadratic discriminant analyses

The limit behavior of the conditional probability of error of linear and quadratic discriminant analyses is studied under wide assumptions on the class conditional distributions. Results obtained may help to explain analytically the behavior in applications of linear and quadratic discrimination techniques.     

The efficiency of logistic regression compared to normal discriminant analysis under class-conditional classification noise

In many real world classification problems, class-conditional classification noise (CCC-Noise) frequently deteriorates the performance of a classifier that is naively built by ignoring it. In this paper, we investigate the impact of CCC-Noise on the quality of a popular generative classifier, normal discriminant analysis (NDA), and its corresponding discriminative classifier, logistic regression (LR). We consider the problem of two multivariate normal populations having a common covariance matrix. We compare the asymptotic distribution of the misclassification error rate of these two classifiers under CCC-Noise. We show that when the noise level is low, the asymptotic error rates of both procedures are only slightly affected. We also show that LR is less deteriorated by CCC-Noise compared to NDA. Under CCC-Noise contexts, the Mahalanobis distance between the populations plays a vital role in determining the relative performance of these two procedures. In particular, when this distance is small, LR tends to be more tolerable to CCC-Noise compared to NDA.     

High breakdown mixture discriminant analysis

Robust S-estimation is proposed for multivariate Gaussian mixture models generalizing the work of Hastie and Tibshirani (J. Roy. Statist. Soc. Ser. B 58 (1996) 155). In the case of Gaussian Mixture models, the unknown location and scale parameters are estimated by the EM algorithm. In the presence of outliers, the maximum likelihood estimators of the unknown parameters are affected, resulting in the misclassification of the observations. The robust S-estimators of the unknown parameters replace the non-robust estimators from M-step of the EM algorithm. The results were compared with the standard mixture discriminant analysis approach using the probability of misclassification criterion. This comparison showed a slight reduction in the average probability of misclassification using robust S-estimators as compared to the standard maximum likelihood estimators.     

Optimal discriminant functions for normal populations

A class of discriminant rules which includes Fisher’s linear discriminant function and the likelihood ratio criterion is defined. Using asymptotic expansions of the distributions of the discriminant functions in this class, we derive a formula for cut-off points which satisfy some conditions on misclassification probabilities, and derive the optimal rules for some criteria. Some numerical experiments are carried out to examine the performance of the optimal rules for finite numbers of samples.     

Results in statistical discriminant analysis: a review of the former Soviet Union literature

Much work in discriminant analysis and statistical pattern recognition has been performed in the former Soviet Union. However, most results derived by former Soviet Union researchers are unknown to statisticians and statistical pattern recognition researchers in the West. We attempt to give a succinct overview of important contributions by Soviet Block researchers to several topics in the discriminant analysis literature concerning the small training-sample size problem. We also include a partial review of corresponding work done in the West.     

A note on the structure of the quadratic subspace in discriminant analysis

This paper explores some properties of the quadratic subspace, a tool for dimension reduction in discriminant analysis ( [Velilla, 2008] and [Velilla, 2010]). This linear manifold has a fairly complex structure, and it may sometimes include components with both mean and covariance separation properties. In this case, an assumption of orthogonality between the leading location directions and the bulk of the dispersion subspaces can help to find an adequate directional representation of it in practice. Two real data sets are analyzed.     

Sparse principal component analysis via regularized low rank matrix approximation

Principal component analysis (PCA) is a widely used tool for data analysis and dimension reduction in applications throughout science and engineering. However, the principal components (PCs) can sometimes be difficult to interpret, because they are linear combinations of all the original variables. To facilitate interpretation, sparse PCA produces modified PCs with sparse loadings, i.e. loadings with very few non-zero elements. In this paper, we propose a new sparse PCA method, namely sparse PCA via regularized SVD (sPCA-rSVD). We use the connection of PCA with singular value decomposition (SVD) of the data matrix and extract the PCs through solving a low rank matrix approximation problem. Regularization penalties are introduced to the corresponding minimization problem to promote sparsity in PC loadings. An efficient iterative algorithm is proposed for computation. Two tuning parameter selection methods are discussed. Some theoretical results are established to justify the use of sPCA-rSVD when only the data covariance matrix is available. In addition, we give a modified definition of variance explained by the sparse PCs. The sPCA-rSVD provides a uniform treatment of both classical multivariate data and high-dimension-low-sample-size (HDLSS) data. Further understanding of sPCA-rSVD and some existing alternatives is gained through simulation studies and real data examples, which suggests that sPCA-rSVD provides competitive results.     

On the structure of the quadratic subspace in discriminant analysis

The concept of quadratic subspace is introduced as a helpful tool for dimension reduction in quadratic discriminant analysis (QDA). It is argued that an adequate representation of the quadratic subspace may lead to better methods for both data representation and classification. Several theoretical results describe the structure of the quadratic subspace, that is shown to contain some of the subspaces previously proposed in the literature for finding differences between the class means and covariances. A suitable assumption of orthogonality between location and dispersion subspaces allows us to derive a convenient reduced version of the full QDA rule. The behavior of these ideas in practice is illustrated with three real data examples.     

Fisher information for generalised linear mixed models

The Fisher information for the canonical link exponential family generalised linear mixed model is derived. The contribution from the fixed effects parameters is shown to have a particularly simple form.     

Some notes on extremal discriminant analysis

Classical discriminant analysis focusses on Gaussian and nonparametric models where in the second case the unknown densities are replaced by kernel densities based on the training sample. In the present article we assume that it suffices to base the classification on exceedances above higher thresholds, which can be interpreted as observations in a conditional framework. Therefore, the statistical modeling of truncated distributions is merely required. In this context, a nonparametric modeling is not adequate because the kernel method is inaccurate in the upper tail region. Yet one may deal with truncated parametric distributions like the Gaussian ones. Our primary aim is to replace truncated Gaussian distributions by appropriate generalized Pareto distributions and to explore properties and the relationship of discriminant functions in both models.     

Analysis of NMAR missing data without specifying missing-data mechanisms in a linear latent variate model

It is natural to assume that a missing-data mechanism depends on latent variables in the analysis of incomplete data in latent variate modeling because latent variables are error-free and represent key notions investigated by applied researchers. Unfortunately, the missing-data mechanism is then not missing at random (NMAR). In this article, a new estimation method is proposed, which leads to consistent and asymptotically normal estimators for all parameters in a linear latent variate model, where the missing mechanism depends on the latent variables and no concrete functional form for the missing-data mechanism is used in estimation. The method to be proposed is a type of multi-sample analysis with or without mean structures, and hence, it is easy to implement. Complete-case analysis is shown to produce consistent estimators for some important parameters in the model.     

On the performance of some statistics in discriminant analysis based on covariates

In this paper we study the behavior of three statistics suggested for testing the hypothesis, H₀ : μ₁ = μ₂, in the two sample case, in the presence of covariables. Power comparisons are made in the case when δ₂, the difference of the mean vectors in the covariates, is not equal to zero. This extends an earlier paper of the authors [Sanklya Ser. B35 51–78], where δ₂ was assumed to be equal to zero. The results reiterate those obtained in the above cited paper that for low observed values of D_q² one would use t₂ otherwise t₃ would be recommended. The statistic t₁ does not seem to be appropriate for testing this hypothesis.     

Portfolio value at risk based on independent component analysis

Risk management technology applied to high-dimensional portfolios needs simple and fast methods for calculation of value at risk (VaR). The multivariate normal framework provides a simple off-the-shelf methodology but lacks the heavy-tailed distributional properties that are observed in data. A principle component-based method (tied closely to the elliptical structure of the distribution) is therefore expected to be unsatisfactory. Here, we propose and analyze a technology that is based on independent component analysis (ICA). We study the proposed ICVaR methodology in an extensive simulation study and apply it to a high-dimensional portfolio situation. Our analysis yields very accurate VaRs.     

Multivariate analysis of variance with fewer observations than the dimension

In this article, we consider the problem of testing a linear hypothesis in a multivariate linear regression model which includes the case of testing the equality of mean vectors of several multivariate normal populations with common covariance matrix Σ, the so-called multivariate analysis of variance or MANOVA problem. However, we have fewer observations than the dimension of the random vectors. Two tests are proposed and their asymptotic distributions under the hypothesis as well as under the alternatives are given under some mild conditions. A theoretical comparison of these powers is made.     

Minimum variance estimators for misclassification probabilities in discriminant analysis

Let α(n₁, n₂) be the probability of classifying an observation from population Π₁ into population Π₂ using Fisher's linear discriminant function based on samples of size n₁ and n₂. A standard estimator of α, denoted by T₁, is the proportion of observations in the first sample misclassified by the discriminant function. A modification of T₁, denoted by T₂, is obtained by eliminating the observation being classified from the calculation of the discriminant function. The UMVU estimators, $\text{T}{}_1{}^1$ and $\text{T}{}_2{}^1$, of ET₁ = τ₁(n₁, n₂) and ET₂ = τ₂(n₁, n₂) = α(n₁ ? 1, n₂) are derived for the case when the populations have multivariate normal distributions with common dispersion matrix. It is shown that $\text{T}{}_1{}^1$ and $\text{T}{}_2{}^1$ are nonincreasing functions of D², the Mahalanobis sample distance. This result is used to derive the sampling distributions and moments of $\text{T}{}_1{}^1$ and $\text{T}{}_2{}^1$. It is also shown that α is a decreasing function of Δ² = (μ₁ ? μ₂)′Σ^?1(μ₁ ? μ₂). Hence, by truncating $\text{T}{}_1{}^1$ and $\text{T}{}_2{}^1$ (or any estimator) at the value of α for Σ = 0, new estimators are obtained which, for all samples, are as close or closer to α.     

Eigenvectors of a kurtosis matrix as interesting directions to reveal cluster structure

In this paper we study the properties of a kurtosis matrix and propose its eigenvectors as interesting directions to reveal the possible cluster structure of a data set. Under a mixture of elliptical distributions with proportional scatter matrix, it is shown that a subset of the eigenvectors of the fourth-order moment matrix corresponds to Fisher’s linear discriminant subspace. The eigenvectors of the estimated kurtosis matrix are consistent estimators of this subspace and its calculation is easy to implement and computationally efficient, which is particularly favourable when the ratio n/p is large.