Variable Selection Diagnostics Measures for High-Dimensional Regression

Markov chain Monte Carlo (MCMC) is nowadays a standard approach to numerical computation of integrals of the posterior density π of the parameter vector η. Unfortunately, Bayesian inference using MCMC is computationally intractable when the posterior density π is expensive to evaluate. In many such problems, it is possible to identify a minimal subvector β of η responsible for the expensive computation in the evaluation of π. We propose two approaches, DOSKA and INDA, that approximate π by interpolation in ways that exploit this computational structure to mitigate the curse of dimensionality. DOSKA interpolates π directly while INDA interpolates π indirectly by interpolating functions, for example, a regression function, upon which π depends. Our primary contribution is derivation of a Gaussian processes interpolant that provably improves over some of the existing approaches by reducing the effective dimension of the interpolation problem from dim(η) to dim(β). This allows a dramatic reduction of the number of expensive evaluations necessary to construct an accurate approximation of π when dim(η) is high but dim(β) is low.

We illustrate the proposed approaches in a case study for a spatio-temporal linear model for air pollution data in the greater Boston area.

Supplemental materials include proofs, details, and software implementation of the proposed procedures.     

Variable Selection via A Combination of the L0 and L1 Penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L₁ penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L₀ penalty is attractive for variable selection because it directly penalizes the number of non zero coefficients. However, the optimization involved is discontinuous and non convex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L₀ and L₁ penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L₀ and L₁ penalties in terms of variable selection while maintaining good prediction accuracy.     

Variable selection and coefficient estimation via composite quantile regression with randomly censored data

Composite quantile regression with randomly censored data is studied. Moreover, adaptive LASSO methods for composite quantile regression with randomly censored data are proposed. The consistency, asymptotic normality and oracle property of the proposed estimators are established. The proposals are illustrated via simulation studies and the Australian AIDS dataset.     

Variable Selection for Logistic Regression via Smooth LASSO and Spline LASSO

Considering a parameter estimation and variable selection problem in logistic regression, we propose Smooth LASSO and Spline LASSO. When the variables is continuous, using Smooth LASSO can select local constant coefficient in each group. However, in some case, the coefficient might be different and change smoothly. Using Spline Lasso to estimate parameter is more appropriate. In this article, we prove the reliability of the model by theory. Finally using coordinate descent algorithm to solve the model. Simulations show that the model works very effectively both in feature selection and prediction accuracy.     

���ع�ģ�͵�Smooth LASSO��Spline LASSO����ѡ��

??Considering a parameter estimation and variable selection problem in logistic regression, we propose Smooth LASSO and Spline LASSO. When the variables is continuous, using Smooth LASSO can select local constant coefficient in each group. However, in some case, the coefficient might be different and change smoothly. Using Spline Lasso to estimate parameter is more appropriate. In this article, we prove the reliability of the model by theory. Finally using coordinate descent algorithm to solve the model. Simulations show that the model works very effectively both in feature selection and prediction accuracy.     

Variable Selection in Regression via Repeated Data Splitting

Abstract

A new algorithm—backward elimination via repeated data splitting (BERDS)—is proposed for variable selection in regression. Initially, the data are partitioned into two sets {E, V}, and an exhaustive backward elimination (BE) is performed in E. For each p value cutoff α used in BE, the corresponding fitted model from E is validated in V by computing the sum of squared deviations of observed from predicted values. This is repeated m times, and the α minimizing the sum of the m sums of squares is used as the cutoff in a final BE on the entire data set. BERDS is a modification of the algorithm BECV proposed by Thall, Simon, and Grier (1992). An extensive simulation study shows that, compared to BECV, BERDS has a smaller model error and higher probabilities of excluding noise variables, of selecting each of several uncorrelated true predictors, and of selecting exactly one of two or three highly correlated true predictors. BERDS is also superior to standard BE with cutoffs .05 or .10, and this superiority increases with the number of noise variables in the data and the degree of correlation among true predictors. An application is provided for illustration.     

Lasso变量选择的分布式算法

Lasso是机器学习中比较常用的一种变量选择方法,适用于具有稀疏性的回归问题.当样本量巨大或者海量的数据存储在不同的机器上时,分布式计算是减少计算时间提高效率的重要方式之一.本文在给出Lasso模型等价优化模型的基础上,将ADMM算法应用到此优化变量可分离的模型中,构造了一种适用于Lasso变量选择的分布式算法,证明了...     

淘汰式变权TOPSIS法

传统的TOPSIS法不能直接用于常见的淘汰选优的实际决策.提出淘汰式变权TOPSIS法,通过逐步淘汰明显较劣方案,调整符合决策人偏好的权重,可以更好地反映实际决策行为.实例分析表明该法是简单实用的.     

多种分布下选择后变量显著性分析及其在CEPS数据中的应用

在使用变量选择方法选出模型后,如何评价模型中变量系数的显著性是统计学重点关注的前沿问题之一.文章从适应性Lasso变量选择方法的选择结果出发,在考虑实践中误差分布多样性的前提下,基于选择事件构造了模型保留变量系数的条件检验统计量,并给出了该统计量的一致收敛性质的证明过程.模拟研究显示,在多种误差分布下所提方法均可进一步优化变量选择结果,有较强的实用价值.应用此方法对CEPS学生数据进行了实证分析,最终选取了学生认知能力等10个变量作为影响中学生成绩的主要因素,为相关研究提供了有益的参考.     

删失回归模型中的变量选择

在一个删失回归模型("Tobit"模型)中,我们常常要研究如何选择重要的预报变量.本文提出了基于信息理论准则的两种变量选择程序,并建立了它们的相合性.     

Bayesian Ising Graphical Model for Variable Selection

In this article, we propose a new Bayesian variable selection (BVS) approach via the graphical model and the Ising model, which we refer to as the “Bayesian Ising graphical model” (BIGM). The BIGM is developed by showing that the BVS problem based on the linear regression model can be considered as a complete graph and described by an Ising model with random interactions. There are several advantages of our BIGM: it is easy to (i) employ the single-site updating and cluster updating algorithm, both of which are suitable for problems with small sample sizes and a larger number of variables, (ii) extend this approach to nonparametric regression models, and (iii) incorporate graphical prior information. In our BIGM, the interactions are determined by the linear model coefficients, so we systematically study the performance of different scale normal mixture priors for the model coefficients by adopting the global-local shrinkage strategy. Our results indicate that the best prior for the model coefficients in terms of variable selection should place substantial weight on small, nonzero shrinkage. The methods are illustrated with simulated and real data. Supplementary materials for this article are available online.     

Criteria for selection of response variables and the asymptotic properties in a multivariate calibration

Summary Let a set ofp responsesy=(y ₁,...y _p )′ has a multivariate linear regression on a set ofq explanatory variablesx=(x ₁,...x _q )′. Our aim is to select the most informative subset of responses for making inferences about an unknownx from an observedy. Under normality ony, two selection methods, based on the asymptotic mean squared error and on the Akaike's information criterion, are proposed by Fujikoshi and Nishii (1986,Hiroshima Math. J.,16, 269–277). In this paper, under a mild condition we will derive the cross-validation criterion and obtain the asymptotic properties of the three procedures.     

��λ���ع�ı�Ҷ˹����ѡ�񷽷�

??When the data has heavy tail feature or contains outliers, conventional variable selection methods based on penalized least squares or likelihood functions perform poorly. Based on Bayesian inference method, we study the Bayesian variable selection problem for median linear models. The Bayesian estimation method is proposed by using Bayesian model selection theory and Bayesian estimation method through selecting the Spike and Slab prior for regression coefficients, and the effective posterior Gibbs sampling procedure is also given. Extensive numerical simulations and Boston house price data analysis are used to illustrate the effectiveness of the proposed method.     

Bayesian Variable Selection for Median Regression

When the data has heavy tail feature or contains outliers, conventional variable selection methods based on penalized least squares or likelihood functions perform poorly. Based on Bayesian inference method, we study the Bayesian variable selection problem for median linear models. The Bayesian estimation method is proposed by using Bayesian model selection theory and Bayesian estimation method through selecting the Spike and Slab prior for regression coefficients, and the effective posterior Gibbs sampling procedure is also given. Extensive numerical simulations and Boston house price data analysis are used to illustrate the effectiveness of the proposed method.     

Variable Selection in General Frailty Models Using Penalized H-Likelihood

Variable selection methods using a penalized likelihood have been widely studied in various statistical models. However, in semiparametric frailty models, these methods have been relatively less studied because the marginal likelihood function involves analytically intractable integrals, particularly when modeling multicomponent or correlated frailties. In this article, we propose a simple but unified procedure via a penalized h-likelihood (HL) for variable selection of fixed effects in a general class of semiparametric frailty models, in which random effects may be shared, nested, or correlated. We consider three penalty functions (least absolute shrinkage and selection operator [LASSO], smoothly clipped absolute deviation [SCAD], and HL) in our variable selection procedure. We show that the proposed method can be easily implemented via a slight modification to existing HL estimation approaches. Simulation studies also show that the procedure using the SCAD or HL penalty performs well. The usefulness of the new method is illustrated using three practical datasets too. Supplementary materials for the article are available online.     

A Note on the Optimality of Generalized Cross-validation Bandwidth Selection in Partially Linear Models with Kernel Smoothing Estimator

The issue of selection of bandwidth in kernel smoothing method is considered within the context of partially linear models, hi this paper, we study the asymptotic behavior of the bandwidth choice based on generalized cross-validation （CCV） approach and prove that this bandwidth choice is asymptotically optimal. Numerical simulation are also conducted to investigate the empirical performance of generalized cross-valldation.     

高维部分线性模型的变量选择和估计

考虑高维部分线性模型,提出了同时进行变量选择和估计兴趣参数的变量选择方法.将Dantzig变量选择应用到线性部分及非参数部分的各阶导数,从而获得参数和非参数部分的估计,且参数部分的估计具有稀疏性,证明了估计的非渐近理论界.最后,模拟研究了有限样本的性质.     

Spatial Lasso With Applications to GIS Model Selection

Geographic information systems (GIS) organize spatial data in multiple two-dimensional arrays called layers. In many applications, a response of interest is observed on a set of sites in the landscape, and it is of interest to build a regression model from the GIS layers to predict the response at unsampled sites. Model selection in this context then consists not only of selecting appropriate layers, but also of choosing appropriate neighborhoods within those layers. We formalize this problem as a linear model and propose the use of Lasso to simultaneously select variables, choose neighborhoods, and estimate parameters. Spatially dependent errors are accounted for using generalized least squares and spatial smoothness in selected coefficients is incorporated through use of a priori spatial covariance structure. This leads to a modification of the Lasso procedure, called spatial Lasso. The spatial Lasso can be implemented by a fast algorithm and it performs well in numerical examples, including an application to prediction of soil moisture. The methodology is also extended to generalized linear models. Supplemental materials including R computer code and data analyzed in this article are available online.     

Interaction Model and Model Selection for Function-on-Function Regression

Regression models with interaction effects have been widely used in multivariate analysis to improve model flexibility and prediction accuracy. In functional data analysis, however, due to the challenges of estimating three-dimensional coefficient functions, interaction effects have not been considered for function-on-function linear regression. In this article, we propose function-on-function regression models with interaction and quadratic effects. For a model with specified main and interaction effects, we propose an efficient estimation method that enjoys a minimum prediction error property and has good predictive performance in practice. Moreover, converting the estimation of three-dimensional coefficient functions of the interaction effects to the estimation of two- and one-dimensional functions separately, our method is computationally efficient. We also propose adaptive penalties to account for varying magnitudes and roughness levels of coefficient functions. In practice, the forms of the models are usually unspecified. We propose a stepwise procedure for model selection based on a predictive criterion. This method is implemented in our R package FRegSigComp. Supplemental materials are available online.     

Unbiased Recursive Partitioning: A Conditional Inference Framework

Recursive binary partitioning is a popular tool for regression analysis. Two fundamental problems of exhaustive search procedures usually applied to fit such models have been known for a long time: overfitting and a selection bias towards covariates with many possible splits or missing values. While pruning procedures are able to solve the overfitting problem, the variable selection bias still seriously affects the interpretability of tree-structured regression models. For some special cases unbiased procedures have been suggested, however lacking a common theoretical foundation. We propose a unified framework for recursive partitioning which embeds tree-structured regression models into a well defined theory of conditional inference procedures. Stopping criteria based on multiple test procedures are implemented and it is shown that the predictive performance of the resulting trees is as good as the performance of established exhaustive search procedures. It turns out that the partitions and therefore the models induced by both approaches are structurally different, confirming the need for an unbiased variable selection. Moreover, it is shown that the prediction accuracy of trees with early stopping is equivalent to the prediction accuracy of pruned trees with unbiased variable selection. The methodology presented here is applicable to all kinds of regression problems, including nominal, ordinal, numeric, censored as well as multivariate response variables and arbitrary measurement scales of the covariates. Data from studies on glaucoma classification, node positive breast cancer survival and mammography experience are re-analyzed.