Shrinkage estimation and variable selection in multiple regression models with random coefficient autoregressive errors

In this paper, we consider improved estimation strategies for the parameter vector in multiple regression models with first-order random coefficient autoregressive errors (RCAR(1)). We propose a shrinkage estimation strategy and implement variable selection methods such as lasso and adaptive lasso strategies. The simulation results reveal that the shrinkage estimators perform better than both lasso and adaptive lasso when and only when there are many nuisance variables in the model.     

Hierarchically penalized additive hazards model with diverging number of parameters

In many applications,covariates can be naturally grouped.For example,for gene expression data analysis,genes belonging to the same pathway might be viewed as a group.This paper studies variable selection problem for censored survival data in the additive hazards model when covariates are grouped.A hierarchical regularization method is proposed to simultaneously estimate parameters and select important variables at both the group level and the within-group level.For the situations in which the number of parameters tends to∞as the sample size increases,we establish an oracle property and asymptotic normality property of the proposed estimators.Numerical results indicate that the hierarchically penalized method performs better than some existing methods such as lasso,smoothly clipped absolute deviation(SCAD)and adaptive lasso.     

��Ԫ�ع���ѡ���Ա�����һ�ּ򵥷���

在线性回归模型建模中, 回归自变量选择是一个受到广泛关注、文献众多, 具有很强的理论和实际意义的问题. 回归自变量选择子集的相合性是其中一个重要问题, 如果某种自变量选择方法选择的子集在样本量趋于无穷时是相合的, 而且预测均方误差较小, 则这种方法是可取的. 利用BIC准则可以挑选相合的自变量子集, 但是在自变量个数很多时计算量过大; 适应lasso方法具有较高计算效率, 也能找到相合的自变量子集; 本文提出一种更简单的自变量选择方法, 只需要计算两次普通线性回归: 第一次进行全集回归, 得到全集的回归系数估计, 然后利用这些回归系数估计挑选子集, 然后只要在挑选的自变量子集上再进行一次普通线性回归就得到了回归结果. 考虑如下的回归模型: 其中回归系数中非零分量下标的集合为, 设是本文方法选择的自变量子集下标集合, 是本文方法估计的回归系数(未选中的自变量对应的系数为零), 本文证明了, 在适当条件下, 其中表示的 分量下标在中的元素的组成的向量, 是误差方差, 是与 矩阵极限有关的矩阵和常数. 数值模拟结果表明本文方法具有很好的中小样本性质.     

Model selection via adaptive shrinkage with t priors

We discuss a model selection procedure, the adaptive ridge selector, derived from a hierarchical Bayes argument, which results in a simple and efficient fitting algorithm. The hierarchical model utilized resembles an un-replicated variance components model and leads to weighting of the covariates. We discuss the intuition behind this type estimator and investigate its behavior as a regularized least squares procedure. While related alternatives were recently exploited to simultaneously fit and select variablses/features in regression models (Tipping in J Mach Learn Res 1:211–244, 2001; Figueiredo in IEEE Trans Pattern Anal Mach Intell 25:1150–1159, 2003), the extension presented here shows considerable improvement in model selection accuracy in several important cases. We also compare this estimator’s model selection performance to those offered by the lasso and adaptive lasso solution paths. Under randomized experimentation, we show that a fixed choice of tuning parameter leads to results in terms of model selection accuracy which are superior to the entire solution paths of lasso and adaptive lasso when the underlying model is a sparse one. We provide a robust version of the algorithm which is suitable in cases where outliers may exist.     

测量误差模型的自适应LASSO变量选择方法研究

本文研究测量误差模型的自适应LASSO(least absolute shrinkage and selection operator)变量选择和系数估计问题.首先分别给出协变量有测量误差时的线性模型和部分线性模型自适应LASSO参数估计量,在一些正则条件下研究估计量的渐近性质,并且证明选择合适的调整参数,自适应LASSO参数估计量具有oracle性质.其次讨论估计的实现算法及惩罚参数和光滑参数的选择问题.最后通过模拟和一个实际数据分析研究了自适应LASSO变量选择方法的表现,结果表明,变量选择和参数估计效果良好.     

Variable selection of generalized regression models based on maximum rank correlation

In this paper, we investigate the variable selection problem of the generalized regression models. To estimate the regression parameter, a procedure combining the rank correlation method and the adaptive lasso technique is developed, which is proved to have oracle properties. A modified IMO （iterative marginal optimization） algorithm which directly aims to maximize the penalized rank correlation function is proposed. The effects of the estimating procedure are illustrated by simulation studies.     

Lag weighted lasso for time series model

The adaptive lasso can consistently identify the true model in regression model. However, the adaptive lasso cannot account for lag effects, which are essential for a time series model. Consequently, the adaptive lasso can not reflect certain properties of a time series model. To improve the forecast accuracy of a time series model, we propose a lag weighted lasso. The lag weighted lasso imposes different penalties on each coefficient based on weights that reflect not only the coefficients size but also the lag effects. Simulation studies and a real example show that the proposed method is superior to both the lasso and the adaptive lasso in forecast accuracy.     

面板数据分位数回归模型的参数估计与变量选择

本文研究了基于面板数据的分位数回归模型的变量选择问题.通过增加改进的自适应Lasso惩罚项,同时实现了固定效应面板数据的分位数回归和变量选择,得到了模型中参数的选择相合性和渐近正态性.随机模拟验证了该方法的有效性.推广了文献[14]的结论.     

ConvexLAR: An Extension of Least Angle Regression

The least angle regression (LAR) was proposed by Efron, Hastie, Johnstone and Tibshirani in the year 2004 for continuous model selection in linear regression. It is motivated by a geometric argument and tracks a path along which the predictors enter successively and the active predictors always maintain the same absolute correlation (angle) with the residual vector. Although it gains popularity quickly, its extensions seem rare compared to the penalty methods. In this expository article, we show that the powerful geometric idea of LAR can be generalized in a fruitful way. We propose a ConvexLAR algorithm that works for any convex loss function and naturally extends to group selection and data adaptive variable selection. After simple modification, it also yields new exact path algorithms for certain penalty methods such as a convex loss function with lasso or group lasso penalty. Variable selection in recurrent event and panel count data analysis, Ada-Boost, and Gaussian graphical model is reconsidered from the ConvexLAR angle. Supplementary materials for this article are available online.     

Bayesian lasso binary quantile regression

In this paper, a Bayesian hierarchical model for variable selection and estimation in the context of binary quantile regression is proposed. Existing approaches to variable selection in a binary classification context are sensitive to outliers, heteroskedasticity or other anomalies of the latent response. The method proposed in this study overcomes these problems in an attractive and straightforward way. A Laplace likelihood and Laplace priors for the regression parameters are proposed and estimated with Bayesian Markov Chain Monte Carlo. The resulting model is equivalent to the frequentist lasso procedure. A conceptional result is that by doing so, the binary regression model is moved from a Gaussian to a full Laplacian framework without sacrificing much computational efficiency. In addition, an efficient Gibbs sampler to estimate the model parameters is proposed that is superior to the Metropolis algorithm that is used in previous studies on Bayesian binary quantile regression. Both the simulation studies and the real data analysis indicate that the proposed method performs well in comparison to the other methods. Moreover, as the base model is binary quantile regression, a much more detailed insight in the effects of the covariates is provided by the approach. An implementation of the lasso procedure for binary quantile regression models is available in the R-package bayesQR.     

Efficient estimation and variable selection for infinite variance autoregressive models

In this paper, a self-weighted composite quantile regression estimation procedure is developed to estimate unknown parameter in an infinite variance autoregressive (IVAR) model. The proposed estimator is asymptotically normal and more efficient than a single quantile regression estimator. At the same time, the adaptive least absolute shrinkage and selection operator (LASSO) for variable selection are also suggested. We show that the adaptive LASSO based on the self-weighted composite quantile regression enjoys the oracle properties. Simulation studies and a real data example are conducted to examine the performance of the proposed approaches.     

Variable selection in a class of single-index models

In this paper we discuss variable selection in a class of single-index models in which we do not assume the error term as additive. Following the idea of sufficient dimension reduction, we first propose a unified method to recover the direction, then reformulate it under the least square framework. Differing from many other existing results associated with nonparametric smoothing methods for density function, the bandwidth selection in our proposed kernel function essentially has no impact on its root-n consistency or asymptotic normality. To select the important predictors, we suggest using the adaptive lasso method which is computationally efficient. Under some regularity conditions, the adaptive lasso method enjoys the oracle property in a general class of single-index models. In addition, the resulting estimation is shown to be asymptotically normal, which enables us to construct a confidence region for the estimated direction. The asymptotic results are augmented through comprehensive simulations, and illustrated by an analysis of air pollution data.     

生长曲线模型的变量选择

生长曲线模型是一个典型的多元线性模型, 在现代统计学上占有重要地位. 文章首先基于Potthoff-Roy变换后的生长曲线模型, 采用自适应LASSO为惩罚函数给出了参数矩阵的惩罚最小二乘估计, 实现了变量的选择. 其次, 基于局部渐近二次估计, 对生长曲线模型的惩罚最小二乘估计给出了统一的近似估计表达式. 接着, 讨论了经过Potthoff-Roy变换后模型的惩罚最小二乘估计, 证明了自适应LASSO具有Oracle性质. 最后对几种变量选择方法进行了数据模拟. 结果表明自适应LASSO效果比较好. 另外, 综合考虑, Potthoff-Roy变换优于拉直变换.     

Variable Selection for Partially Linear Models via Adaptive LASSO

Partially linear model is a class of commonly used semiparametric models, this paper focus on variable selection and parameter estimation for partially linear models via adaptive LASSO method. Firstly, based on profile least squares and adaptive LASSO method, the adaptive LASSO estimator for partially linear models are constructed, and the selections of penalty parameter and bandwidth are discussed. Under some regular conditions, the consistency and asymptotic normality for the estimator are investigated, and it is proved that the adaptive LASSO estimator has the oracle properties. The proposed method can be easily implemented. Finally a Monte Carlo simulation study is conducted to assess the finite sample performance of the proposed variable selection procedure, results show the adaptive LASSO estimator behaves well.     

Handling selection bias when choosing actions in retail credit applications

In many situations one needs to know which action one should take with a customer to yield the greatest response. Typically, estimates of the response functions of different actions will be based on the responses of customers previously assigned to each action. Often, however, the previous assignments will not have been random, so that estimates of the response functions will be biased. We examine the case of two possible actions. We look at the error arising from using the simple OLS estimate ignoring the selection bias, and also explore the possibility of using the Heckman model to allow for the sample selectivity. The performance of Heckman’s model is then compared with the simple OLS through simulation.     

Post selection shrinkage estimation for high‐dimensional data analysis

In high‐dimensional data settings where p  ? n , many penalized regularization approaches were studied for simultaneous variable selection and estimation. However, with the existence of covariates with weak effect, many existing variable selection methods, including Lasso and its generations, cannot distinguish covariates with weak and no contribution. Thus, prediction based on a subset model of selected covariates only can be inefficient. In this paper, we propose a post selection shrinkage estimation strategy to improve the prediction performance of a selected subset model. Such a post selection shrinkage estimator (PSE) is data adaptive and constructed by shrinking a post selection weighted ridge estimator in the direction of a selected candidate subset. Under an asymptotic distributional quadratic risk criterion, its prediction performance is explored analytically. We show that the proposed post selection PSE performs better than the post selection weighted ridge estimator. More importantly, it improves the prediction performance of any candidate subset model selected from most existing Lasso‐type variable selection methods significantly. The relative performance of the post selection PSE is demonstrated by both simulation studies and real‐data analysis. Copyright © 2016 John Wiley & Sons, Ltd.     

Adaptive Penalized Weighted Least Absolute Deviations Estimation for the Accelerated Failure Time Model

The accelerated failure time model always offers a valuable complement to the traditional Cox proportional hazards model due to its direct and meaningful interpretation. We propose a variable selection method in the context of the accelerated failure time model for survival data, which can simultaneously complete variable selection and parameter estimation. Meanwhile, the proposed method can deal with the potential outliers in survival times as well as heteroscedastic model errors, which are frequently encountered in practice. Specifically, utilizing the general nonconvex penalty, we propose the adaptive penalized weighted least absolute deviation estimator for the accelerated failure time model. Under some regularity conditions, we show that the proposed method yields consistent estimator and possesses the oracle property. In addition, we propose a new algorithm to compute the estimate in the high dimensional settings, and evaluate the practical utility of the proposed method through extensive simulation studies and two real examples.     

Non-Convex Global Minimization and False Discovery Rate Control for the TREX

The TREX is a recently introduced method for performing sparse high-dimensional regression. Despite its statistical promise as an alternative to the lasso, square-root lasso, and scaled lasso, the TREX is computationally challenging in that it requires solving a nonconvex optimization problem. This article shows a remarkable result: despite the nonconvexity of the TREX problem, there exists a polynomial-time algorithm that is guaranteed to find the global minimum. This result adds the TREX to a very short list of nonconvex optimization problems that can be globally optimized (principal components analysis being a famous example). After deriving and developing this new approach, we demonstrate that (i) the ability of the preexisting TREX heuristic to reach the global minimum is strongly dependent on the difficulty of the underlying statistical problem, (ii) the new polynomial-time algorithm for TREX permits a novel variable ranking and selection scheme, (iii) this scheme can be incorporated into a rule that controls the false discovery rate (FDR) of included features in the model. To achieve this last aim, we provide an extension of the results of Barber and Candes to establish that the knockoff filter framework can be applied to the TREX. This investigation thus provides both a rare case study of a heuristic for nonconvex optimization and a novel way of exploiting nonconvexity for statistical inference.     

Tweedie’s Compound Poisson Model With Grouped Elastic Net

Tweedie’s compound Poisson model is a popular method to model data with probability mass at zero and nonnegative, highly right-skewed distribution. Motivated by wide applications of the Tweedie model in various fields such as actuarial science, we investigate the grouped elastic net method for the Tweedie model in the context of the generalized linear model. To efficiently compute the estimation coefficients, we devise a two-layer algorithm that embeds the blockwise majorization descent method into an iteratively reweighted least square strategy. Integrated with the strong rule, the proposed algorithm is implemented in an easy-to-use R package HDtweedie, and is shown to compute the whole solution path very efficiently. Simulations are conducted to study the variable selection and model fitting performance of various lasso methods for the Tweedie model. The modeling applications in risk segmentation of insurance business are illustrated by analysis of an auto insurance claim dataset. Supplementary materials for this article are available online.     

Threshold variable selection of asymmetric stochastic volatility models

A threshold stochastic volatility (SV) model is used for capturing time-varying volatilities and nonlinearity. Two adaptive Markov chain Monte Carlo (MCMC) methods of model selection are designed for the selection of threshold variables for this family of SV models. The first method is the direct estimation which approximates the model posterior probabilities of competing models. Using parallel MCMC sampling to estimate these probabilities, the best threshold variable is selected with the highest posterior model probability. The second method is to use the deviance information criterion to compare among these competing models and select the best one. Simulation results lead us to conclude that for large samples the posterior model probability approximation method can give an accurate approximation of the posterior probability in Bayesian model selection. The method delivers a powerful and sharp model selection tool. An empirical study of five Asian stock markets provides strong support for the threshold variable which is formulated as a weighted average of important variables.