Improving Penalized Least Squares through Adaptive Selection of Penalty and Shrinkage

Estimation of the mean function in nonparametric regression is usefully separated into estimating the means at the observed factor levels—a one-way layout problem—and interpolation between the estimated means at adjacent factor levels. Candidate penalized least squares (PLS) estimators for the mean vector of a one-way layout are expressed as shrinkage estimators relative to an orthogonal regression basis determined by the penalty matrix. The shrinkage representation of PLS suggests a larger class of candidate monotone shrinkage (MS) estimators. Adaptive PLS and MS estimators choose the shrinkage vector and penalty matrix to minimize estimated risk. The actual risks of shrinkage-adaptive estimators depend strongly upon the economy of the penalty basis in representing the unknown mean vector. Local annihilators of polynomials, among them difference operators, generate penalty bases that are economical in a range of examples. Diagnostic techniques for adaptive PLS or MS estimators include basis-economy plots and estimates of loss or risk.     

Improved ridge regression estimators for the logistic regression model

The estimation of the regression parameters for the ill-conditioned logistic regression model is considered in this paper. We proposed five ridge regression (RR) estimators, namely, unrestricted RR, restricted ridge regression, preliminary test RR, shrinkage ridge regression and positive rule RR estimators for estimating the parameters $(\beta )$ when it is suspected that the parameter $\beta $ may belong to a linear subspace defined by $H\beta =h$ . Asymptotic properties of the estimators are studied with respect to quadratic risks. The performances of the proposed estimators are compared based on the quadratic bias and risk functions under both null and alternative hypotheses, which specify certain restrictions on the regression parameters. The conditions of superiority of the proposed estimators for departure and ridge parameters are given. Some graphical representations and efficiency analysis have been presented which support the findings of the paper.     

Linear plus quadratic (LPQ) quasiminimax estimation in the linear regression model

If the errors in the linear regression model are assumed to be independent with nonvanishing third and finite fourth moments, then it is possible to improve all linear estimators by so-called linear plus quadratic (LPQ) estimators. These consist of linear and quadratic terms in the endogeneous variable and depend on the unknown moments of the errors which, in general, have to be estimated from the data. In this paper, we will use LPQ estimators for quasiminimax estimation and some related problems.Support by Deutsche Forschungsgemeinschaft Grant No. Tr 253/1-2 is gratefully acknowledged.     

具有特殊协方差结构的 SURE 模型中参数估计的若干结果

本文讨论具有特殊协方差结构似乎不相关回归方程（SURE）模型中参数的估计问题．除非另有说明,损失函数将取为二次损失和矩阵损失．本文证明了回归系数的线性可估函数的最小二乘估计是极小极大的且在矩阵损失函数下是可容许的;还分别在仿射交换群和平移群下导出了存在回归系数的线性可估函数的一致最小风险同变（UMRE）估计的充要条件,并证明了在仿射交换和二次损失下不存在协方差阵和方差的UMRE估计．     

The Minimax Estimator of Stochastic Regression Coefficients and Parameters in the Class of All Estimators

In this paper, the authors address the problem of the minimax estimator of linear combinations of stochastic regression coefficients and parameters in the general normal linear model with random effects. Under a quadratic loss function, the minimax property of linear estimators is investigated. In the class of all estimators, the minimax estimator of estimable functions, which is unique with probability 1, is obtained under a multivariate normal distribution.     

Positive shrinkage, improved pretest and absolute penalty estimators in partially linear models

Shrinkage estimators of a partially linear regression parameter vector are constructed by shrinking estimators in the direction of the estimate which is appropriate when the regression parameters are restricted to a linear subspace. We investigate the asymptotic properties of positive Stein-type and improved pretest semiparametric estimators under quadratic loss. Under an asymptotic distributional quadratic risk criterion, their relative dominance picture is explored analytically. It is shown that positive Stein-type semiparametric estimators perform better than the usual Stein-type and least square semiparametric estimators and that an improved pretest semiparametric estimator is superior to the usual pretest semiparametric estimator. We also consider an absolute penalty type estimator for partially linear models and give a Monte Carlo simulation comparisons of positive shrinkage, improved pretest and the absolute penalty type estimators. The comparison shows that the shrinkage method performs better than the absolute penalty type estimation method when the dimension of the parameter space is much larger than that of the linear subspace.     

二次损失下随机回归系数和参数的线性Minimax估计

对带有随机效应的一般线性模型,本文提出了随机回归系数和参数线性组合的Minimax估计问题. 在二次损失下,研究了线性估计的极小极大性.关于适当的假设,得到了可估函数的唯一线性Mjnimax 估计.     

增长曲线模型中UMRE估计和UMRU估计的存在性

对于一般的增长曲线模型,在一般的矩阵损失和二次损失下,用统一的方法分别给出了回归系数矩阵的任一指定可估函数存在一致最小风险同变（ＵＭＲＥ）估计（分别在仿真变换群和转换变换群下）和一致最小风险无编（ＵＭＲＵ）估计的充要条件,以及所有可估函数恒存在ＵＭＲＥ估计和ＵＭＲＵ估计的允要条件。最后将结果应用于一些特殊模型。     

A subclass of bayes linear estimators that are minimax

In the linear regression model with ellipsoidal parameter constraints, the problem of estimating the unknown parameter vector is studied. A well-described subclass of Bayes linear estimators is proposed in the paper. It is shown that for each member of this subclass, a generalized quadratic risk function exists so that the estimator is minimax. Moreover, some of the proposed Bayes linear estimators are admissible with respect to all possible generalized quadratic risks. Also, a necessary and sufficient condition is given to ensure that the considered Bayes linear estimator improves the least squares estimator over the whole ellipsoid whatever generalized risk function is chosen.     

Confidence ellipsoids based on a general family of shrinkage estimators for a linear model with non-spherical disturbances

This paper considers a general family of Stein rule estimators for the coefficient vector of a linear regression model with nonspherical disturbances, and derives estimators for the Mean Squared Error (MSE) matrix, and risk under quadratic loss for this family of estimators. The confidence ellipsoids for the coefficient vector based on this family of estimators are proposed, and the performance of the confidence ellipsoids under the criterion of coverage probability and expected volumes is investigated. The results of a numerical simulation are presented to illustrate the theoretical findings, which could be applicable in the area of economic growth modeling.     

ASP fits to multi-way layouts

The balanced complete multi-way layout with ordinal or nominal factors is a fundamental data-type that arises in medical imaging, agricultural field trials, DNA microassays, and other settings where analysis of variance (ANOVA) is an established tool. ASP algorithms weigh competing biased fits in order to reduce risk through variance-bias tradeoff. The acronym ASP stands for Adaptive Shrinkage of Penalty bases. Motivating ASP is a penalized least squares criterion that associates a separate quadratic penalty term with each main effect and each interaction in the general ANOVA decomposition of means. The penalty terms express plausible conjecture about the mean function, respecting the difference between ordinal and nominal factors. Multiparametric asymptotics under a probability model and experiments on data elucidate how ASP dominates least squares, sometimes very substantially. ASP estimators for nominal factors recover Stein's superior shrinkage estimators for one- and two-way layouts. ASP estimators for ordinal factors bring out the merits of smoothed fits to multi-way layouts, a topic broached algorithmically in work by Tukey. This research was supported in part by National Science Foundation Grants DMS 0300806 and 0404547.     

Double k-Class Estimators in Regression Models with Non-spherical Disturbances

In this paper, we consider a family of feasible generalised double k-class estimators in a linear regression model with non-spherical disturbances. We derive the large sample asymptotic distribution of the proposed family of estimators and compare its performance with the feasible generalized least squares and Stein-rule estimators using the mean squared error matrix and risk under quadratic loss criteria. A Monte-Carlo experiment investigates the finite sample behaviour of the proposed family of estimators.     

Robust model selection for a semimartingale continuous time regression from discrete data

The paper considers the problem of estimating a periodic function in a continuous time regression model observed under a general semimartingale noise with an unknown distribution in the case when continuous observation cannot be provided and only discrete time measurements are available. Two specific types of noises are studied in detail: a non-Gaussian Ornstein–Uhlenbeck process and a time-varying linear combination of a Brownian motion and compound Poisson process. We develop new analytical tools to treat the adaptive estimation problems from discrete data. A lower bound for the frequency sampling, needed for the efficiency of the procedure constructed by discrete observations, has been found. Sharp non-asymptotic oracle inequalities for the robust quadratic risk have been derived. New convergence rates for the efficient procedures have been obtained. An example of the regression with a martingale noise exhibits that the minimax robust convergence rate may be both higher or lower as compared with the minimax rate for the “white noise” model. The results of Monte-Carlo simulations are given.     

Risk and Pitman closeness properties of feasible generalized double k-class estimators in linear regression models with non-spherical disturbances under balanced loss function

In this article, a family of feasible generalized double k-class estimator in a linear regression model with non-spherical disturbances is considered. The performance of this estimator is judged with feasible generalized least-squares and feasible generalized Stein-rule estimators under balanced loss function using the criteria of quadratic risk and general Pitman closeness. A Monte-Carlo study investigates the finite sample properties of several estimators arising from the family of feasible double k-class estimators.     

Linear regression estimators for multinormal distributions in optimization of stochastic programming problems

Several linear regression estimators are presented, which approximate the distribution function of the m-dimensional normal distribution, or the distribution function along a line. These regression estimators are quadratic functions, or simple functions of quadratic functions and can be applied in numerical problems, arising during optimization of stochastic programming problems. A root finding procedure is developed, that can be used to find the intersection of a line and the border of the feasible set. Directional derivatives and gradient of the normal distribution can be computed. Some numerical results are also presented.     

广义偏最小二乘(PLS)估计类

本文研究了一类含有偏最小二乘（partialleastsquaresPLS）估计的估计类．给出了PLS估计的一般代数形式；讨论了含有PLS估计的广义PPLS估计类的统计性质；给出了该估计类优于最小二乘估计的条件．     

Change-of-variance sensitivities in regression analysis

Summary The change-of-variance function is defined for estimators of regression coefficients. Both an unstandardized and a standardized form of the change-of-variance sensitivity are introduced, and their relation with the corresponding gross-error-sensitivities is investigated. The problems of optimal robustness lead to the Hampel-Krasker and the Krasker-Welsch estimators. At the same time, also the scale parameter has to be estimated robustly. By means of the change-of-variance sensitivity, optimal robust redescending scale estimators are constructed.     

Improved estimation in multiple linear regression models with measurement error and general constraint

In this paper, we define two restricted estimators for the regression parameters in a multiple linear regression model with measurement errors when prior information for the parameters is available. We then construct two sets of improved estimators which include the preliminary test estimator, the Stein-type estimator and the positive rule Stein type estimator for both slope and intercept, and examine their statistical properties such as the asymptotic distributional quadratic biases and the asymptotic distributional quadratic risks. We remove the distribution assumption on the error term, which was generally imposed in the literature, but provide a more general investigation of comparison of the quadratic risks for these estimators. Simulation studies illustrate the finite-sample performance of the proposed estimators, which are then used to analyze a dataset from the Nurses Health Study.     

Tree-based multivariate regression and density estimation with right-censored data

We propose a unified strategy for estimator construction, selection, and performance assessment in the presence of censoring. This approach is entirely driven by the choice of a loss function for the full (uncensored) data structure and can be stated in terms of the following three main steps. (1) First, define the parameter of interest as the minimizer of the expected loss, or risk, for a full data loss function chosen to represent the desired measure of performance. Map the full data loss function into an observed (censored) data loss function having the same expected value and leading to an efficient estimator of this risk. (2) Next, construct candidate estimators based on the loss function for the observed data. (3) Then, apply cross-validation to estimate risk based on the observed data loss function and to select an optimal estimator among the candidates. A number of common estimation procedures follow this approach in the full data situation, but depart from it when faced with the obstacle of evaluating the loss function for censored observations. Here, we argue that one can, and should, also adhere to this estimation road map in censored data situations.Tree-based methods, where the candidate estimators in Step 2 are generated by recursive binary partitioning of a suitably defined covariate space, provide a striking example of the chasm between estimation procedures for full data and censored data (e.g., regression trees as in CART for uncensored data and adaptations to censored data). Common approaches for regression trees bypass the risk estimation problem for censored outcomes by altering the node splitting and tree pruning criteria in manners that are specific to right-censored data. This article describes an application of our unified methodology to tree-based estimation with censored data. The approach encompasses univariate outcome prediction, multivariate outcome prediction, and density estimation, simply by defining a suitable loss function for each of these problems. The proposed method for tree-based estimation with censoring is evaluated using a simulation study and the analysis of CGH copy number and survival data from breast cancer patients.     

A wavelet-based spectral procedure for steady-state simulation analysis

We develop WASSP, a wavelet-based spectral method for steady-state simulation analysis. First WASSP determines a batch size and a warm-up period beyond which the computed batch means form an approximately stationary Gaussian process. Next WASSP computes the discrete wavelet transform of the bias-corrected log-smoothed-periodogram of the batch means, using a soft-thresholding scheme to denoise the estimated wavelet coefficients. Then taking the inverse discrete wavelet transform of the thresholded wavelet coefficients, WASSP computes estimators of the batch means log-spectrum and the steady-state variance parameter (i.e., the sum of covariances at all lags) for the original (unbatched) process. Finally by combining the latter estimator with the batch means grand average, WASSP provides a sequential procedure for constructing a confidence interval on the steady-state mean that satisfies user-specified requirements concerning absolute or relative precision as well as coverage probability. An experimental performance evaluation demonstrates WASSP’s effectiveness compared with other simulation analysis methods.