Improved estimation of a multinormal precision matrix

Stein's technique is used to obtain improved estimators of the multinormal precision matrix under quadratic loss. The technique is to obtain a certain differential inequality involving the eigenvalues of the sample covariance matrix. Several improved estimators are obtained by solving the differential inequality.     

On improved estimation of normal precision matrix and discriminant coefficients

The problem of estimating the precision matrix of a multivariate normal distribution model is considered with respect to a quadratic loss function. A number of covariance estimators originally intended for a variety of loss functions are adapted so as to obtain alternative estimators of the precision matrix. It is shown that the alternative estimators have analytically smaller risks than the unbiased estimator of the precision matrix. Through numerical studies of risk values, it is shown that the new estimators have substantial reduction in risk. In addition, we consider the problem of the estimation of discriminant coefficients, which arises in linear discriminant analysis when Fisher's linear discriminant function is viewed as the posterior log-odds under the assumption that two classes differ in mean but have a common covariance matrix. The above method is also adapted for this problem in order to obtain improved estimators of the discriminant coefficients under the quadratic loss function. Furthermore, a numerical study is undertaken to compare the properties of a collection of alternatives to the “unbiased” estimator of the discriminant coefficients.     

Regularized bridge-type estimation with multiple penalties

Annals of the Institute of Statistical Mathematics - The aim of this paper is to introduce an adaptive penalized estimator for identifying the true reduced parametric model under the sparsity...     

Asymptotic theory of the adaptive Sparse Group Lasso

We study the asymptotic properties of a new version of the Sparse Group Lasso estimator (SGL), called adaptive SGL. This new version includes two distinct regularization parameters, one for the Lasso penalty and one for the Group Lasso penalty, and we consider the adaptive version of this regularization, where both penalties are weighted by preliminary random coefficients. The asymptotic properties are established in a general framework, where the data are dependent and the loss function is convex. We prove that this estimator satisfies the oracle property: the sparsity-based estimator recovers the true underlying sparse model and is asymptotically normally distributed. We also study its asymptotic properties in a double-asymptotic framework, where the number of parameters diverges with the sample size. We show by simulations and on real data that the adaptive SGL outperforms other oracle-like methods in terms of estimation precision and variable selection.

     

Matrix-free preconditioning using partial matrix estimation

We consider matrix-free solver environments where information about the underlying matrix is available only through matrix vector computations which do not have access to a fully assembled matrix. We introduce the notion of partial matrix estimation for constructing good algebraic preconditioners used in Krylov iterative methods in such matrix-free environments, and formulate three new graph coloring problems for partial matrix estimation. Numerical experiments utilizing one of these formulations demonstrate the viability of this approach. AMS subject classification (2000) 65F10, 65F50, 49M37, 90C06     

Subset selection for vector autoregressive processes via adaptive Lasso

Subset selection is a critical component of vector autoregressive (VAR) modeling. This paper proposes simple and hybrid subset selection procedures for VAR models via the adaptive Lasso. By a proper choice of tuning parameters, one can identify the correct subset and obtain the asymptotic normality of the nonzero parameters with probability tending to one. Simulation results show that for small samples, a particular hybrid procedure has the best performance in terms of prediction mean squared errors, estimation errors and subset selection accuracy under various settings. The proposed method is also applied to modeling the IS-LM data for illustration.     

Minimax estimators for a multinormal precision matrix

Let S_p×p ～ Wishart (Σ, k), Σ unknown, k > p + 1. Minimax estimators of Σ^?1 are given for L₁, an Empirical Bayes loss function; and L₂, a standard loss function (R_i ≡ E(L_i ∣ Σ), i = 1, 2). The estimators are $\text{Σ}\text{?}{}^{\mathrm{?1}}\text{=}\text{a}\text{S}{}^{\mathrm{?1}}\text{+}\text{br}\text{(S)I}{}_{\mathrm{p\times p}}$, a, b ≥ 0, r(·) a functional on $\text{R}{}^{\mathrm{<mtext>p(p+2)</mtext><mtext>2</mtext>}}$. Stein, Efron, and Morris studied the special cases $\text{Σ}{}_{\mathrm{<mtext>a</mtext>}}{}^{\mathrm{?1}}\text{=}\text{a}\text{S}{}^{\mathrm{?1}}\text{(E}\text{Σ}\text{?}{}_{\mathrm{<mtext>k?p?1</mtext>}}{}^{\mathrm{?1}}\text{= Σ}{}^{\mathrm{?1}}\text{)}$ and $\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{=}\text{a}\text{S}{}^{\mathrm{?1}}\text{+ (b/tr S)I}$, for certain, a, b. From their work $\text{R}{}_1\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{; S) ≤ R}{}_1\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_{\mathrm{<mtext>a</mtext>}}{}^{\mathrm{?1}}\text{; S) (?Σ)}$, a = k ? p ? 1, b = p² + p ? 2; whereas, we prove $\text{R}{}_2\text{(Σ}{}^{\mathrm{?1}}\text{Σ}\text{?}{}_{\mathrm{<mtext>a</mtext>}}{}^{\mathrm{?1}}\text{; S) ≤}\text{R}{}_2\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{; S) (?Σ)}$. The reversal is surprising because $\text{L}{}_1\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{; S) →}\text{L}{}_2\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{; S)}$ a.e. (for a particular L₂). Assume $\text{R}$ (compact) ? $\text{S}$, $\text{S}$ the set of p × p p.s.d. matrices. A “divergence theorem” on functions F_p×p : $\text{R}$ → $\text{S}$ implies identities for R_i, i = 1, 2. Then, conditions are given for $\text{R}{}_{\mathrm{i}}\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}^{\mathrm{?1}}\text{; S) ≤}\text{R}{}_{\mathrm{i}}\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_1{}^{\mathrm{?1}}\text{; S) ≤}\text{R}{}_{\mathrm{i}}\text{(Σ}{}^{\mathrm{?1}}\text{,}\text{Σ}\text{?}{}_{\mathrm{<mtext>a</mtext>}}{}^{\mathrm{?1}}\text{; S) (?Σ)}$, i = 1, 2. Most of our results concern estimators with r(S) = t(U)/tr(S), U = p ∣S∣^1/p/tr(S).     

Pointwise adaptive estimation of a multivariate function

In this paper, we address the problem of pointwise estimation in the Gaussian white noise model. We propose a new data-driven procedure that achieves (up to a multiplicative logarithmic term) the minimax rate of convergence over a scale of anisotropic Hölder spaces. Moreover we present a general criterion in order to define what should be an “optimal” estimation procedure and we prove that our procedure satisfies this criterion. The extra logarithmic term can thus be viewed as an unavoidable price to pay for adaptation.     

Improved estimation of a covariance matrix in an elliptically contoured matrix distribution

In this paper, the problem of estimating the covariance matrix of the elliptically contoured distribution (ECD) is considered. A new class of estimators which shrink the eigenvalues towards their arithmetic mean is proposed. It is shown that this new estimator dominates the unbiased estimator under the squared error loss function. Two special classes of ECD, namely, the multivariate-elliptical t distribution and the ε-contaminated normal distribution are considered. A simulation study is carried out and indicates that this new shrinkage estimator provides a substantial improvement in risk under most situations.     

Problem of precision of the method of a minimal pseudoinverse matrix

Translated from Matematicheskie Zametki, Vol. 49, No. 4, pp. 81–87, April, 1991.     

An adaptive estimation of MAVE

Minimum average variance estimation (MAVE, Xia et al. (2002) [29]) is an effective dimension reduction method. It requires no strong probabilistic assumptions on the predictors, and can consistently estimate the central mean subspace. It is applicable to a wide range of models, including time series. However, the least squares criterion used in MAVE will lose its efficiency when the error is not normally distributed. In this article, we propose an adaptive MAVE which can be adaptive to different error distributions. We show that the proposed estimate has the same convergence rate as the original MAVE. An EM algorithm is proposed to implement the new adaptive MAVE. Using both simulation studies and a real data analysis, we demonstrate the superior finite sample performance of the proposed approach over the existing least squares based MAVE when the error distribution is non-normal and the comparable performance when the error is normal.     

High dimensional covariance matrix estimation using multi-factor models from incomplete information

Covariance matrix plays an important role in risk management, asset pricing, and portfolio allocation. Covariance matrix estimation becomes challenging when the dimensionality is comparable or much larger than the sample size. A widely used approach for reducing dimensionality is based on multi-factor models. Although it has been well studied and quite successful in many applications, the quality of the estimated covariance matrix is often degraded due to a nontrivial amount of missing data in the factor matrix for both technical and cost reasons. Since the factor matrix is only approximately low rank or even has full rank, existing matrix completion algorithms are not applicable. We consider a new matrix completion paradigm using the factor models directly and apply the alternating direction method of multipliers for the recovery. Numerical experiments show that the nuclear-norm matrix completion approaches are not suitable but our proposed models and algorithms are promising.     

Sharp adaptive estimation of quadratic functionals

Estimation of a quadratic functional of a function observed in the Gaussian white noise model is considered. A data-dependent method for choosing the amount of smoothing is given. The method is based on comparing certain quadratic estimators with each other. It is shown that the method is asymptotically sharp or nearly sharp adaptive simultaneously for the “regular” and “irregular” region. We consider l_p bodies and construct bounds for the risk of the estimator which show that for p=4 the estimator is exactly optimal and for example when p ∈[3,100], then the upper bound is at most 1.055 times larger than the lower bound. We show the connection of the estimator to the theory of optimal recovery. The estimator is a calibration of an estimator which is nearly minimax optimal among quadratic estimators. Writing of this article was financed by Deutsche Forschungsgemeinschaft under project MA1026/6-2, CIES, France, and Jenny and AnttiWihuri Foundation.     

On adaptive wavelet estimation of a quadratic functional from a deconvolution problem

We observe a stochastic process where a convolution product of an unknown function and a known function is corrupted by Gaussian noise. We wish to estimate the squared \mathbbL²{\mathbb{L}^2} -norm of the unknown function from the observations. To reach this goal, we develop adaptive estimators based on wavelet and thresholding. We prove that they achieve (near) optimal rates of convergence under the mean squared error over a wide range of smoothness classes.     

A modified adaptive Lasso for identifying interactions in the Cox model with the heredity constraint

In many biomedical studies, identifying effects of covariate interactions on survival is a major goal. Important examples are treatment–subgroup interactions in clinical trials, and gene–gene or gene–environment interactions in genomic studies. A common problem when implementing a variable selection algorithm in such settings is the requirement that the model must satisfy the strong heredity constraint, wherein an interaction may be included in the model only if the interaction’s component variables are included as main effects. We propose a modified Lasso method for the Cox regression model that adaptively selects important single covariates and pairwise interactions while enforcing the strong heredity constraint. The proposed method is based on a modified log partial likelihood including two adaptively weighted penalties, one for main effects and one for interactions. A two-dimensional tuning parameter for the penalties is determined by generalized cross-validation. Asymptotic properties are established, including consistency and rate of convergence, and it is shown that the proposed selection procedure has oracle properties, given proper choice of regularization parameters. Simulations illustrate that the proposed method performs reliably across a range of different scenarios.     

Bayesian estimation of a covariance matrix with flexible prior specification

Bayesian analysis for a covariance structure has been in use for decades. The commonly adopted Bayesian setup involves the conjugate inverse Wishart prior specification for the covariance matrix. Here we depart from this approach and adopt a novel prior specification by considering a multivariate normal prior for the elements of the matrix logarithm of the covariance structure. This specification allows for a richer class of prior distributions for the covariance structure with respect to strength of beliefs in prior location hyperparameters and the added ability to model potential correlation amongst the covariance structure. We provide three computational methods for calculating the posterior moment of the covariance matrix. The moments of interest are calculated based upon computational results via Importance sampling, Laplacian approximation and Markov Chain Monte Carlo/Metropolis–Hastings techniques. As a particular application of the proposed technique we investigate educational test score data from the project talent data set.     

Spectral decomposition of a matrix using the generalized sign matrix

An algorithm for spectral decomposition is presented which does not require knowledge of eigenvalues and eigenvectors. A set of eigenprojectors are defined which covers the entire spectrum of a matrix, and special attention is given to the projection on the zero eigenvalue. Some useful applications are discussed in the paper.     

Methods for improvement in estimation of a normal mean matrix

This paper is concerned with the problem of estimating a matrix of means in multivariate normal distributions with an unknown covariance matrix under invariant quadratic loss. It is first shown that the modified Efron-Morris estimator is characterized as a certain empirical Bayes estimator. This estimator modifies the crude Efron-Morris estimator by adding a scalar shrinkage term. It is next shown that the idea of this modification provides a general method for improvement of estimators, which results in the further improvement on several minimax estimators. As a new method for improvement, an adaptive combination of the modified Stein and the James-Stein estimators is also proposed and is shown to be minimax. Through Monte Carlo studies of the risk behaviors, it is numerically shown that the proposed, combined estimator inherits the nice risk properties of both individual estimators and thus it has a very favorable risk behavior in a small sample case. Finally, the application to a two-way layout MANOVA model with interactions is discussed.     

Non-parametric adaptive estimation of the drift for a jump diffusion process

In this article, we consider a jump diffusion process _(Xt)t≥0${\left(X_t\right)}_{t\ge 0}$ observed at discrete times t=0,Δ,…,nΔ$t=0,\Delta ,\dots ,n\Delta$. The sampling interval Δ$\Delta$ tends to 0 and nΔ$n\Delta$ tends to infinity. We assume that _(Xt)t≥0${\left(X_t\right)}_{t\ge 0}$ is ergodic, strictly stationary and exponentially β$\beta$-mixing. We use a penalised least-square approach to compute two adaptive estimators of the drift function b$b$. We provide bounds for the risks of the two estimators.     

On estimation and adaptive estimation for locally asymptotically normal families

Summary Locally asymptotically minimax (LAM) estimates are constructed for locally asymptotically normal (LAN) families under very mild additional assumptions. Adaptive estimation is also considered and a sufficient condition is given for an estimate to be locally asymptotically minimax adaptive. Incidently, it is shown that a well known lower bound due to Hájek (1972) for the local asymptotic minimax risk is not sharp.Research partially supported by NSF grants no. MCS 78-02846 and MCS 77-03493-01