Improving Penalized Least Squares through Adaptive Selection of Penalty and Shrinkage

Estimation of the mean function in nonparametric regression is usefully separated into estimating the means at the observed factor levels—a one-way layout problem—and interpolation between the estimated means at adjacent factor levels. Candidate penalized least squares (PLS) estimators for the mean vector of a one-way layout are expressed as shrinkage estimators relative to an orthogonal regression basis determined by the penalty matrix. The shrinkage representation of PLS suggests a larger class of candidate monotone shrinkage (MS) estimators. Adaptive PLS and MS estimators choose the shrinkage vector and penalty matrix to minimize estimated risk. The actual risks of shrinkage-adaptive estimators depend strongly upon the economy of the penalty basis in representing the unknown mean vector. Local annihilators of polynomials, among them difference operators, generate penalty bases that are economical in a range of examples. Diagnostic techniques for adaptive PLS or MS estimators include basis-economy plots and estimates of loss or risk.     

A stochastic restricted ridge regression estimator

Groß [J. Groß, Restricted ridge estimation, Statistics & Probability Letters 65 (2003) 57–64] proposed a restricted ridge regression estimator when exact restrictions are assumed to hold. When there are stochastic linear restrictions on the parameter vector, we introduce a new estimator by combining ideas underlying the mixed and the ridge regression estimators under the assumption that the errors are not independent and identically distributed. Apart from [J. Groß, Restricted ridge estimation, Statistics & Probability Letters 65 (2003) 57–64], we call this new estimator as the stochastic restricted ridge regression (SRRR) estimator. The performance of the SRRR estimator over the mixed estimator in respect of the variance and the mean square error matrices is examined. We also illustrate our findings with a numerical example. The shrinkage generalized least squares (GLS) and the stochastic restricted shrinkage GLS estimators are proposed.     

正态-逆Wishart先验下多元线性模型中经验Bayes估计的优良性

在正态-逆Wishart先验下研究了多元线性模型中参数的经验Bayes估计及其优良性问题.当先验分布中含有未知参数时,构造了回归系数矩阵和误差方差矩阵的经验Bayes估计,并在Bayes均方误差(简称BMSE)准则和Bayes均方误差阵(简称BMSEM)准则下,证明了经验Bayes估计优于最小二乘估计.最后,进行了Monte Carlo模拟研究,进一步验证了理论结果.     

A multivariate ultrastructural errors-in-variables model with equation error

This paper deals with asymptotic results on a multivariate ultrastructural errors-in-variables regression model with equation errors. Sufficient conditions for attaining consistent estimators for model parameters are presented. Asymptotic distributions for the line regression estimators are derived. Applications to the elliptical class of distributions with two error assumptions are presented. The model generalizes previous results aimed at univariate scenarios.     

Generalized Bayes minimax estimation of the normal mean matrix with unknown covariance matrix

This paper addresses the problem of estimating the normal mean matrix in the case of unknown covariance matrix. This problem is solved by considering generalized Bayesian hierarchical models. The resulting generalized Bayes estimators with respect to an invariant quadratic loss function are shown to be matricial shrinkage equivariant estimators and the conditions for their minimaxity are given.     

Multivariate Locally Weighted Polynomial Fitting and Partial Derivative Estimation

Nonparametric regression estimator based on locally weighted least squares fitting has been studied by Fan and Ruppert and Wand. The latter paper also studies, in the univariate case, nonparametric derivative estimators given by a locally weighted polynomial fitting. Compared with traditional kernel estimators, these estimators are often of simpler form and possess some better properties. In this paper, we develop current work on locally weighted regression and generalize locally weighted polynomial fitting to the estimation of partial derivatives in a multivariate regression context. Specifically, for both the regression and partial derivative estimators we prove joint asymptotic normality and derive explicit asymptotic expansions for their conditional bias and conditional convariance matrix (given observations of predictor variables) in each of the two important cases of local linear fit and local quadratic fit.     

Comparisons among some estimators in misspecified linear models with multicollinearity

In this paper we deal with comparisons among several estimators available in situations of multicollinearity (e.g., the r-k class estimator proposed by Baye and Parker, the ordinary ridge regression (ORR) estimator, the principal components regression (PCR) estimator and also the ordinary least squares (OLS) estimator) for a misspecified linear model where misspecification is due to omission of some relevant explanatory variables. These comparisons are made in terms of the mean square error (mse) of the estimators of regression coefficients as well as of the predictor of the conditional mean of the dependent variable. It is found that under the same conditions as in the true model, the superiority of the r-k class estimator over the ORR, PCR and OLS estimators and those of the ORR and PCR estimators over the OLS estimator remain unchanged in the misspecified model. Only in the case of comparison between the ORR and PCR estimators, no definite conclusion regarding the mse dominance of one over the other in the misspecified model can be drawn.     

齐次线性约束回归模型参数的主成分压缩估计

当设计矩阵X复共线时,对齐次线性约束回归模型参数的约束最小二乘估计进行改进,提出参数的主成分压缩估计,并对新参数估计的性质进行了讨论,最后进行了数值模拟,验证了算法的参数估计优于约束最小二乘估计.     

Sequential Co-Sparse Factor Regression

In multivariate regression models, a sparse singular value decomposition of the regression component matrix is appealing for reducing dimensionality and facilitating interpretation. However, the recovery of such a decomposition remains very challenging, largely due to the simultaneous presence of orthogonality constraints and co-sparsity regularization. By delving into the underlying statistical data-generation mechanism, we reformulate the problem as a supervised co-sparse factor analysis, and develop an efficient computational procedure, named sequential factor extraction via co-sparse unit-rank estimation (SeCURE), that completely bypasses the orthogonality requirements. At each step, the problem reduces to a sparse multivariate regression with a unit-rank constraint. Nicely, each sequentially extracted sparse and unit-rank coefficient matrix automatically leads to co-sparsity in its pair of singular vectors. Each latent factor is thus a sparse linear combination of the predictors and may influence only a subset of responses. The proposed algorithm is guaranteed to converge, and it ensures efficient computation even with incomplete data and/or when enforcing exact orthogonality is desired. Our estimators enjoy the oracle properties asymptotically; a non-asymptotic error bound further reveals some interesting finite-sample behaviors of the estimators. The efficacy of SeCURE is demonstrated by simulation studies and two applications in genetics. Supplementary materials for this article are available online.     

Lower Rank Approximation of Matrices Based on Fast and Robust Alternating Regression

We introduce fast and robust algorithms for lower rank approximation to given matrices based on robust alternating regression. The alternating least squares regression, also called criss-cross regression, was used for lower rank approximation of matrices, but it lacks robustness against outliers in these matrices. We use robust regression estimators and address some of the complications arising from this approach. We find it helpful to use high breakdown estimators in the initial iterations, followed by M estimators with monotone score functions in later iterations towards convergence. In addition to robustness, the computational speed is another important consideration in the development of our proposed algorithm, because alternating robust regression can be computationally intensive for large matrices. Based on a mix of the least trimmed squares (LTS) and Huber's M estimators, we demonstrate that fast and robust lower rank approximations are possible for modestly large matrices.     

A comparison of asymptotic covariance matrices of three consistent estimators in the Poisson regression model with measurement errors

We consider a Poisson model, where the mean depends on certain covariates in a log-linear way with unknown regression parameters. Some or all of the covariates are measured with errors. The covariates as well as the measurement errors are both jointly normally distributed, and the error covariance matrix is supposed to be known. Three consistent estimators of the parameters—the corrected score, a structural, and the quasi-score estimators—are compared to each other with regard to their relative (asymptotic) efficiencies. The paper extends an earlier result for a scalar covariate.     

Estimation of covariance matrices in fixed and mixed effects linear models

The estimation of the covariance matrix or the multivariate components of variance is considered in the multivariate linear regression models with effects being fixed or random. In this paper, we propose a new method to show that usual unbiased estimators are improved on by the truncated estimators. The method is based on the Stein–Haff identity, namely the integration by parts in the Wishart distribution, and it allows us to handle the general types of scale-equivariant estimators as well as the general fixed or mixed effects linear models.     

IMPROVED ESTIMATES OF THE COVARIANCE MATRIX IN GENERAL LINEAR MIXED MODELS

In this article, the problem of estimating the covariance matrix in general linear mixed models is considered. Two new classes of estimators obtained by shrinking the eigenvalues towards the origin and the arithmetic mean, respectively, are proposed. It is shown that these new estimators dominate the unbiased estimator under the squared error loss function. Finally, some simulation results to compare the performance of the proposed estimators with that of the unbiased estimator are reported. The simulation results indicate that these new shrinkage estimators provide a substantial improvement in risk under most situations.     

具有共同效应的信度回归模型

在经典的Hachemeister(1975)信度回归模型中,各个风险被假定为相互独立的.本文假设风险之间存在由共同效应导致的风险相依,建立了共同效应的信度回归模型,得到未来索赔的信度预测与风险参数的信度估计.结论表明,在共同效应模型,信度估计仍然是个体索赔数据与聚合保费的加权和.     

A Class of Linear Biased Estimators of Regression ParameterMatrix in the Growth Curve Model

ＡＣｌａｓｓｏｆＬｉｎｅａｒＢｉａｓｅｄＥｓｔｉｍａｔｏｒｓｏｆＲｅｇｒｅｓｓｉｏｎＰａｒａｍｅｔｅｒＭａｔｒｉｘｉｎｔｈｅＧｒｏｗｔｈＣｕｒｖｅＭｏｄｅｌ￥ＧｕｉＱｉｎｇｍｉｎｇ（ＺｈｅｎｇｚｈｏｕＩｎｓｔｉｔｕｔｅｏｆＳｕｒｖｅｙｉｎｇａｎｄＭａ...     

多元协方差阵扰动模型岭估计的影响分析

本文研究了多元线性同归模型岭估计的影响分析问题.利用最小二乘估计方法,获得了多元协方差阵扰动模型与原模型参数阵之间的岭估计的一些关系式,给出了度量影响大小的基于岭估计的广义Cook距离.     

Asymptotically optimal estimation in the linear regression problem in the case of violation of some classical assumptions

We consider the problem of estimating the unknown parameters of linear regression in the case when the variances of observations depend on the unknown parameters of the model. A two-step method is suggested for constructing asymptotically linear estimators. Some general sufficient conditions for the asymptotic normality of the estimators are found, and an explicit form is established of the best asymptotically linear estimators. The behavior of the estimators is studied in detail in the case when the parameter of the regression model is one-dimensional.     

An overview on the shrinkage properties of partial least squares regression

The aim of this paper is twofold. In the first part, we recapitulate the main results regarding the shrinkage properties of partial least squares (PLS) regression. In particular, we give an alternative proof of the shape of the PLS shrinkage factors. It is well known that some of the factors are >1. We discuss in detail the effect of shrinkage factors for the mean squared error of linear estimators and argue that we cannot extend the results to PLS directly, as it is nonlinear. In the second part, we investigate the effect of shrinkage factors empirically. In particular, we point out that experiments on simulated and real world data show that bounding the absolute value of the PLS shrinkage factors by 1 seems to leads to a lower mean squared error.     

An Information-theoretic Approach to the Effective Usage of Auxiliary Information from Survey Data

In this paper, we propose an information-theoretic approach to the effective usage of auxiliary information from survey data, which is suitable for both simple and complex survey data. Our estimator under simple random sampling without replacement will be consistent and asymptotically normal. We show that the resulting estimates have smaller asymptotic variances than the usual estimates which do not use auxiliary information. For more complex survey designs, the resulting estimator is in essence asymptotically equivalent to a pseudo empirical likelihood estimator. Results of a limited simulation study show that the proposed estimators perform well among a number of competitors.     

Stochastic integrals of empirical-type processes with applications to censored regression

Motivated by the analysis of linear rank estimators and the Buckley-James nonparametric EM estimator in censored regression models, we study herein the asymptotic properties of stochastic integrals of certain two-parameter empirical processes. Applications of these results on empirical processes and their stochastic integrals to the asymptotic analysis of censored regression estimators are also given.