删失回归模型中的变量选择

在一个删失回归模型("Tobit"模型)中,我们常常要研究如何选择重要的预报变量.本文提出了基于信息理论准则的两种变量选择程序,并建立了它们的相合性.     

Gradient-based Regularization Parameter Selection for Problems With Nonsmooth Penalty Functions

In high-dimensional and/or nonparametric regression problems , regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically, the parameters are chosen to minimize the error on a separate validation set using a simple grid search or a gradient-free optimization method. It is more efficient to tune parameters if the gradient can be determined, but this is often difficult for problems with nonsmooth penalty functions. Here, we show that for many penalized regression problems, the validation loss is actually smooth almost-everywhere with respect to the penalty parameters. We can, therefore, apply a modified gradient descent algorithm to tune parameters. Through simulation studies on example regression problems, we find that increasing the number of penalty parameters and tuning them using our method can decrease the generalization error.     

Regularized estimation for preference disaggregation in multiple criteria decision making

Disaggregation methods have been extensively used in multiple criteria decision making to infer preferential information from reference examples, using linear programming techniques. This paper proposes simple extensions of existing formulations, based on the concept of regularization which has been introduced within the context of the statistical learning theory. The properties of the resulting new formulations are analyzed for both ranking and classification problems and experimental results are presented demonstrating the improved performance of the proposed formulations over the ones traditionally used in preference disaggregation analysis.     

Variable Selection via A Combination of the L0 and L1 Penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L₁ penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L₀ penalty is attractive for variable selection because it directly penalizes the number of non zero coefficients. However, the optimization involved is discontinuous and non convex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L₀ and L₁ penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L₀ and L₁ penalties in terms of variable selection while maintaining good prediction accuracy.     

Test-Based Variable Selection via Cross-Validation

Abstract

Test-based variable selection algorithms in regression often are based on sequential comparison of test statistics to cutoff values. A predetermined a level typically is used to determine the cutoffs based on an assumed probability distribution for the test statistic. For example, backward elimination or forward stepwise involve comparisons of test statistics to prespecified t or F cutoffs in Gaussian linear regression, while a likelihood ratio. Wald, or score statistic, is typically used with standard normal or chi square cutoffs in nonlinear settings. Although such algorithms enjoy widespread use, their statistical properties are not well understood, either theoretically or empirically. Two inherent problems with these methods are that (1) as in classical hypothesis testing, the value of α is arbitrary, while (2) unlike hypothesis testing, there is no simple analog of type I error rate corresponding to application of the entire algorithm to a data set. In this article we propose a new method, backward elimination via cross-validation (BECV), for test-based variable selection in regression. It is implemented by first finding the empirical p value α*, which minimizes a cross-validation estimate of squared prediction error, then selecting the model by running backward elimination on the entire data set using α* as the nominal p value for each test. We present results of an extensive computer simulation to evaluate BECV and compare its performance to standard backward elimination and forward stepwise selection.     

��λ���ع�ı�Ҷ˹����ѡ�񷽷�

??When the data has heavy tail feature or contains outliers, conventional variable selection methods based on penalized least squares or likelihood functions perform poorly. Based on Bayesian inference method, we study the Bayesian variable selection problem for median linear models. The Bayesian estimation method is proposed by using Bayesian model selection theory and Bayesian estimation method through selecting the Spike and Slab prior for regression coefficients, and the effective posterior Gibbs sampling procedure is also given. Extensive numerical simulations and Boston house price data analysis are used to illustrate the effectiveness of the proposed method.     

Bayesian Variable Selection for Median Regression

When the data has heavy tail feature or contains outliers, conventional variable selection methods based on penalized least squares or likelihood functions perform poorly. Based on Bayesian inference method, we study the Bayesian variable selection problem for median linear models. The Bayesian estimation method is proposed by using Bayesian model selection theory and Bayesian estimation method through selecting the Spike and Slab prior for regression coefficients, and the effective posterior Gibbs sampling procedure is also given. Extensive numerical simulations and Boston house price data analysis are used to illustrate the effectiveness of the proposed method.     

Automatic Feature Selection via Weighted Kernels and Regularization

Selecting important features in nonlinear kernel spaces is a difficult challenge in both classification and regression problems. This article proposes to achieve feature selection by optimizing a simple criterion: a feature-regularized loss function. Features within the kernel are weighted, and a lasso penalty is placed on these weights to encourage sparsity. This feature-regularized loss function is minimized by estimating the weights in conjunction with the coefficients of the original classification or regression problem, thereby automatically procuring a subset of important features. The algorithm, KerNel Iterative Feature Extraction (KNIFE), is applicable to a wide variety of kernels and high-dimensional kernel problems. In addition, a modification of KNIFE gives a computationally attractive method for graphically depicting nonlinear relationships between features by estimating their feature weights over a range of regularization parameters. The utility of KNIFE in selecting features through simulations and examples for both kernel regression and support vector machines is demonstrated. Feature path realizations also give graphical representations of important features and the nonlinear relationships among variables. Supplementary materials with computer code and an appendix on convergence analysis are available online.     

Variable Selection for Varying Coefficient Models Via Kernel Based Regularized Rank Regression

A robust and efficient shrinkage-type variable selection procedure for varying coefficient models is proposed, selection consistency and oracle properties are established. Furthermore, a BIC-type criterion is suggested for shrinkage parameter selection and theoretical property is discussed. Numerical studies and real data analysis also are included to illustrate the finite sample performance of our method.     

Adaptive Markov Chain Monte Carlo for Bayesian Variable Selection

We describe adaptive Markov chain Monte Carlo (MCMC) methods for sampling posterior distributions arising from Bayesian variable selection problems. Point-mass mixture priors are commonly used in Bayesian variable selection problems in regression. However, for generalized linear and nonlinear models where the conditional densities cannot be obtained directly, the resulting mixture posterior may be difficult to sample using standard MCMC methods due to multimodality. We introduce an adaptive MCMC scheme that automatically tunes the parameters of a family of mixture proposal distributions during simulation. The resulting chain adapts to sample efficiently from multimodal target distributions. For variable selection problems point-mass components are included in the mixture, and the associated weights adapt to approximate marginal posterior variable inclusion probabilities, while the remaining components approximate the posterior over nonzero values. The resulting sampler transitions efficiently between models, performing parameter estimation and variable selection simultaneously. Ergodicity and convergence are guaranteed by limiting the adaptation based on recent theoretical results. The algorithm is demonstrated on a logistic regression model, a sparse kernel regression, and a random field model from statistical biophysics; in each case the adaptive algorithm dramatically outperforms traditional MH algorithms. Supplementary materials for this article are available online.     

A Confidence Region Approach to Tuning for Variable Selection

We develop an approach to tuning of penalized regression variable selection methods by calculating the sparsest estimator contained in a confidence region of a specified level. Because confidence intervals/regions are generally understood, tuning penalized regression methods in this way is intuitive and more easily understood by scientists and practitioners. More importantly, our work shows that tuning to a fixed confidence level often performs better than tuning via the common methods based on Akaike information criterion (AIC), Bayesian information criterion (BIC), or cross-validation (CV) over a wide range of sample sizes and levels of sparsity. Additionally, we prove that by tuning with a sequence of confidence levels converging to one, asymptotic selection consistency is obtained, and with a simple two-stage procedure, an oracle property is achieved. The confidence-region-based tuning parameter is easily calculated using output from existing penalized regression computer packages. Our work also shows how to map any penalty parameter to a corresponding confidence coefficient. This mapping facilitates comparisons of tuning parameter selection methods such as AIC, BIC, and CV, and reveals that the resulting tuning parameters correspond to confidence levels that are extremely low, and can vary greatly across datasets. Supplemental materials for the article are available online.     

Bootstrapping Likelihood for Model Selection with Small Samples

Abstract

Akaike's information criterion (AIC), derived from asymptotics of the maximum likelihood estimator, is widely used in model selection. However, it has a finite-sample bias that produces overfitting in linear regression. To deal with this problem, Ishiguro, Sakamoto, and Kitagawa proposed a bootstrap-based extension to AIC which they called EIC. This article compares model-selection performance of AIC, EIC, a bootstrap-smoothed likelihood cross-validation (BCV) and its modification (632CV) in small-sample linear regression, logistic regression, and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting problem and that BCV may be better than EIC. Hence, the three methods based on bootstrapping the likelihood establish themselves as important alternatives to AIC in model selection with small samples.     

Variable Selection in Varying-Coefficient Models Using P-Splines

In this article, we consider nonparametric smoothing and variable selection in varying-coefficient models. Varying-coefficient models are commonly used for analyzing the time-dependent effects of covariates on responses measured repeatedly (such as longitudinal data). We present the P-spline estimator in this context and show its estimation consistency for a diverging number of knots (or B-spline basis functions). The combination of P-splines with nonnegative garrote (which is a variable selection method) leads to good estimation and variable selection. Moreover, we consider APSO (additive P-spline selection operator), which combines a P-spline penalty with a regularization penalty, and show its estimation and variable selection consistency. The methods are illustrated with a simulation study and real-data examples. The proofs of the theoretical results as well as one of the real-data examples are provided in the online supplementary materials.     

纵向数据下线性EV模型的变量选择

本文考虑了纵向数据线性EV模型的变量选择.基于二次推断函数方法和压缩方法的思想提出了一种新的偏差校正的变量选择方法.在选择适当的调整参数下,我们证明了所得到的估计量的相合性和渐近正态性.最后通过模拟研究验证了所提出的变量选择方法的有限样本性质.     

Universal Margin Factors and Selection Criteria in Variable Environment

For nonautonomous dynamical systems, the principle of inheriting local properties by global Poincaré maps is developed. Using this method, a selection criterion for systems of close competitors is found: to gain competitive advantage, it suffices to outproduce other populations with a margin. The margin factor in question remains uniformly bounded as the number of competitors in the community grows.     

Linear Model Selection Based on Risk Estimation

The problem of selecting one model from a family of linear models to describe a normally distributed observed data vector is considered. The notion of the model of given dimension nearest to the observation vector is introduced and methods of estimating the risk associated with such a nearest model are discussed. This leads to new model selection criteria one of which, called the "partial bootstrap", seems particularly promising. The methods are illustrated by specializing to the problem of estimating the non-zero components of a parameter vector on which noisy observations are available.     

失混合效应模型的分位回归及变量选择

纵向数据常常用正态混合效应模型进行分析.然而,违背正态性的假定往往会导致无效的推断.与传统的均值回归相比较,分位回归可以给出响应变量条件分布的完整刻画,对于非正态误差分布也可以给稳健的估计结果．本文主要考虑右删失响应下纵向混合效应模型的分位回归估计和变量选择问题．首先,逆删失概率加权方法被用来得到模型的参数估计．其次,结合逆删失概率加权和LASSO惩罚变量选择方法考虑了模型的变量选择问题.蒙特卡洛模拟显示所提方法要比直接删除删失数据的估计方法更具优势．最后,分析了一组艾滋病数据集来展示所提方法的实际应用效果．     

A Multiobjective Exploratory Procedure for Regression Model Selection

Variable selection is recognized as one of the most critical steps in statistical modeling. The problems encountered in engineering and social sciences are commonly characterized by over-abundance of explanatory variables, nonlinearities, and unknown interdependencies between the regressors. An added difficulty is that the analysts may have little or no prior knowledge on the relative importance of the variables. To provide a robust method for model selection, this article introduces the multiobjective genetic algorithm for variable selection (MOGA-VS) that provides the user with an optimal set of regression models for a given dataset. The algorithm considers the regression problem as a two objective task, and explores the Pareto-optimal (best subset) models by preferring those models over the other which have less number of regression coefficients and better goodness of fit. The model exploration can be performed based on in-sample or generalization error minimization. The model selection is proposed to be performed in two steps. First, we generate the frontier of Pareto-optimal regression models by eliminating the dominated models without any user intervention. Second, a decision-making process is executed which allows the user to choose the most preferred model using visualizations and simple metrics. The method has been evaluated on a recently published real dataset on Communities and Crime Within the United States.     

ɾʧ�ع�ģ����һ��LASSO�ͱ���ѡ��͹��Ʒ���

删失回归模型是一种很重要的模型,它在计量经济学中有着广泛的应用. 然而,它的变量选择问题在现今的参考文献中研究的比较少.本文提出了一个LASSO型变量选择和估计方法,称之为多样化惩罚$L_1$限制方法, 简称为DPLC. 另外,我们给出了非0回归系数估计的大样本渐近性质. 最后,大量的模拟研究表明了DPLC方法和一般的最优子集选择方法在变量选择和估计方面有着相同的能力.     

Variable Selection for Logistic Regression via Smooth LASSO and Spline LASSO

Considering a parameter estimation and variable selection problem in logistic regression, we propose Smooth LASSO and Spline LASSO. When the variables is continuous, using Smooth LASSO can select local constant coefficient in each group. However, in some case, the coefficient might be different and change smoothly. Using Spline Lasso to estimate parameter is more appropriate. In this article, we prove the reliability of the model by theory. Finally using coordinate descent algorithm to solve the model. Simulations show that the model works very effectively both in feature selection and prediction accuracy.