Semiparametric and Additive Model Selection Using an Improved Akaike Information Criterion

Abstract

An improved AIC-based criterion is derived for model selection in general smoothing-based modeling, including semiparametric models and additive models. Examples are provided of applications to goodness-of-fit, smoothing parameter and variable selection in an additive model and semiparametric models, and variable selection in a model with a nonlinear function of linear terms.     

Testing Serial Correlation in Partially Linear Additive Models

Acta Mathematicae Applicatae Sinica, English Series - As an extension of partially linear models and additive models, partially linear additive model is useful in statistical modelling. This paper...     

Genetic algorithms for the selection of smoothing parameters in additive models

Summary  Additive models of the type y=f ₁(x ₁)+...+f _p(x _p)+ε where f _j , j=1,..,p, have unspecified functional form, are flexible statistical regression models which can be used to characterize nonlinear regression effects. One way of fitting additive models is the expansion in B-splines combined with penalization which prevents overfitting. The performance of this penalized B-spline (called P-spline) approach strongly depends on the choice of the amount of smoothing used for components f _j . In particular for higher dimensional settings this is a computationaly demanding task. In this paper we treat the problem of choosing the smoothing parameters for P-splines by genetic algorithms. In several simulation studies this approach is compared to various alternative methods of fitting additive models. In particular functions with different spatial variability are considered and the effect of constant respectively local adaptive smoothing parameters is evaluated.     

Shrinkage estimation for identification of linear components in additive models

In this short paper, we demonstrate that the popular penalized estimation method typically used for variable selection in parametric or semiparametric models can actually provide a way to identify linear components in additive models. Unlike most studies in the literature, we are NOT performing variable selection. Due to the difficulty in a priori deciding which predictors should enter the partially linear additive model as the linear components, such a method will prove useful in practice.     

Performance comparisons of greedy algorithms in compressed sensing

Compressed sensing has motivated the development of numerous sparse approximation algorithms designed to return a solution to an underdetermined system of linear equations where the solution has the fewest number of nonzeros possible, referred to as the sparsest solution. In the compressed sensing setting, greedy sparse approximation algorithms have been observed to be both able to recover the sparsest solution for similar problem sizes as other algorithms and to be computationally efficient; however, little theory is known for their average case behavior. We conduct a large‐scale empirical investigation into the behavior of three of the state of the art greedy algorithms: Normalized Iterative Hard Thresholding (NIHT), Hard Thresholding Pursuit (HTP), and CSMPSP. The investigation considers a variety of random classes of linear systems. The regions of the problem size in which each algorithm is able to reliably recover the sparsest solution is accurately determined, and throughout this region, additional performance characteristics are presented. Contrasting the recovery regions and the average computational time for each algorithm, we present algorithm selection maps, which indicate, for each problem size, which algorithm is able to reliably recover the sparsest vector in the least amount of time. Although no algorithm is observed to be uniformly superior, NIHT is observed to have an advantageous balance of large recovery region, absolute recovery time, and robustness of these properties to additive noise across a variety of problem classes. A principle difference between the NIHT and the more sophisticated HTP and CSMPSP is the balance of asymptotic convergence rate against computational cost prior to potential support set updates. The data suggest that NIHT is typically faster than HTP and CSMPSP because of greater flexibility in updating the support that limits unnecessary computation on incorrect support sets. The algorithm selection maps presented here are the first of their kind for compressed sensing. Copyright © 2014 John Wiley & Sons, Ltd.     

Consistent tuning parameter selection in high dimensional sparse linear regression

An exhaustive search as required for traditional variable selection methods is impractical in high dimensional statistical modeling. Thus, to conduct variable selection, various forms of penalized estimators with good statistical and computational properties, have been proposed during the past two decades. The attractive properties of these shrinkage and selection estimators, however, depend critically on the size of regularization which controls model complexity. In this paper, we consider the problem of consistent tuning parameter selection in high dimensional sparse linear regression where the dimension of the predictor vector is larger than the size of the sample. First, we propose a family of high dimensional Bayesian Information Criteria (HBIC), and then investigate the selection consistency, extending the results of the extended Bayesian Information Criterion (EBIC), in Chen and Chen (2008) to ultra-high dimensional situations. Second, we develop a two-step procedure, the SIS+AENET, to conduct variable selection in p>n situations. The consistency of tuning parameter selection is established under fairly mild technical conditions. Simulation studies are presented to confirm theoretical findings, and an empirical example is given to illustrate the use in the internet advertising data.     

Compactly supported shearlets are optimally sparse

Cartoon-like images, i.e., C² functions which are smooth apart from a C² discontinuity curve, have by now become a standard model for measuring sparse (nonlinear) approximation properties of directional representation systems. It was already shown that curvelets, contourlets, as well as shearlets do exhibit sparse approximations within this model, which are optimal up to a log-factor. However, all those results are only applicable to band-limited generators, whereas, in particular, spatially compactly supported generators are of uttermost importance for applications.In this paper, we present the first complete proof of optimally sparse approximations of cartoon-like images by using a particular class of directional representation systems, which indeed consists of compactly supported elements. This class will be chosen as a subset of (non-tight) shearlet frames with shearlet generators having compact support and satisfying some weak directional vanishing moment conditions.     

Discovering model structure for partially linear models

Partially linear models (PLMs) have been widely used in statistical modeling, where prior knowledge is often required on which variables have linear or nonlinear effects in the PLMs. In this paper, we propose a model-free structure selection method for the PLMs, which aims to discover the model structure in the PLMs through automatically identifying variables that have linear or nonlinear effects on the response. The proposed method is formulated in a framework of gradient learning, equipped with a flexible reproducing kernel Hilbert space. The resultant optimization task is solved by an efficient proximal gradient descent algorithm. More importantly, the asymptotic estimation and selection consistencies of the proposed method are established without specifying any explicit model assumption, which assure that the true model structure in the PLMs can be correctly identified with high probability. The effectiveness of the proposed method is also supported by a variety of simulated and real-life examples.

     

Variable selection for spatial semivarying coefficient models

Spatial semiparametric varying coefficient models are a useful extension of spatial linear model. Nevertheless, how to conduct variable selection for it has not been well investigated. In this paper, by basis spline approximation together with a general M-type loss function to treat mean, median, quantile and robust mean regressions in one setting, we propose a novel partially adaptive group \(L_{r} (r\ge 1)\) penalized M-type estimator, which can select variables and estimate coefficients simultaneously. Under mild conditions, the selection consistency and oracle property in estimation are established. The new method has several distinctive features: (1) it achieves robustness against outliers and heavy-tail distributions; (2) it is more flexible to accommodate heterogeneity and allows the set of relevant variables to vary across quantiles; (3) it can keep balance between efficiency and robustness. Simulation studies and real data analysis are included to illustrate our approach.     

Model Selection Using the Estimative and the Approximate p* Predictive Densities

Model selection procedures, based on a simple cross-validation technique and on suitable predictive densities, are taken into account. In particular, the selection criterion involving the estimative predictive density is recalled and a procedure based on the approximate p* predictive density is defined. This new model selection procedure, compared with some other well-known techniques on the basis of the squared prediction error, gives satisfactory results. Moreover, higher-order asymptotic expansions for the selection statistics based on the estimative and the approximate p* predictive densities are derived, whenever a natural exponential model is assumed. These approximations correspond to meaningful modifications of the Akaike's model selection statistic.     

Robust and efficient variable selection for semiparametric partially linear varying coefficient model based on modal regression

Semiparametric partially linear varying coefficient models (SPLVCM) are frequently used in statistical modeling. With high-dimensional covariates both in parametric and nonparametric part for SPLVCM, sparse modeling is often considered in practice. In this paper, we propose a new estimation and variable selection procedure based on modal regression, where the nonparametric functions are approximated by $B$ -spline basis. The outstanding merit of the proposed variable selection procedure is that it can achieve both robustness and efficiency by introducing an additional tuning parameter (i.e., bandwidth $h$ ). Its oracle property is also established for both the parametric and nonparametric part. Moreover, we give the data-driven bandwidth selection method and propose an EM-type algorithm for the proposed method. Monte Carlo simulation study and real data example are conducted to examine the finite sample performance of the proposed method. Both the simulation results and real data analysis confirm that the newly proposed method works very well.     

Test-Based Variable Selection via Cross-Validation

Abstract

Test-based variable selection algorithms in regression often are based on sequential comparison of test statistics to cutoff values. A predetermined a level typically is used to determine the cutoffs based on an assumed probability distribution for the test statistic. For example, backward elimination or forward stepwise involve comparisons of test statistics to prespecified t or F cutoffs in Gaussian linear regression, while a likelihood ratio. Wald, or score statistic, is typically used with standard normal or chi square cutoffs in nonlinear settings. Although such algorithms enjoy widespread use, their statistical properties are not well understood, either theoretically or empirically. Two inherent problems with these methods are that (1) as in classical hypothesis testing, the value of α is arbitrary, while (2) unlike hypothesis testing, there is no simple analog of type I error rate corresponding to application of the entire algorithm to a data set. In this article we propose a new method, backward elimination via cross-validation (BECV), for test-based variable selection in regression. It is implemented by first finding the empirical p value α*, which minimizes a cross-validation estimate of squared prediction error, then selecting the model by running backward elimination on the entire data set using α* as the nominal p value for each test. We present results of an extensive computer simulation to evaluate BECV and compare its performance to standard backward elimination and forward stepwise selection.     

A Framework for Unbiased Model Selection Based on Boosting

Variable and model selection are of major concern in many statistical applications, especially in high-dimensional regression models. Boosting is a convenient statistical method that combines model fitting with intrinsic model selection. We investigate the impact of base-learner specification on the performance of boosting as a model selection procedure. We show that variable selection may be biased if the covariates are of different nature. Important examples are models combining continuous and categorical covariates, especially if the number of categories is large. In this case, least squares base-learners offer increased flexibility for the categorical covariate and lead to a preference even if the categorical covariate is noninformative. Similar difficulties arise when comparing linear and nonlinear base-learners for a continuous covariate. The additional flexibility in the nonlinear base-learner again yields a preference of the more complex modeling alternative. We investigate these problems from a theoretical perspective and suggest a framework for bias correction based on a general class of penalized least squares base-learners. Making all base-learners comparable in terms of their degrees of freedom strongly reduces the selection bias observed in naive boosting specifications. The importance of unbiased model selection is demonstrated in simulations. Supplemental materials including an application to forest health models, additional simulation results, additional theorems, and proofs for the theorems are available online.     

First Nonlinear Syzygies of Ideals Associated to Graphs

Consider an ideal I ? K[x ₁,…, x _n ], with K an arbitrary field, generated by monomials of degree two. Assuming that I does not have a linear resolution, we determine the step s of the minimal graded free resolution of I where nonlinear syzygies first appear, we show that at this step of the resolution nonlinear syzygies are concentrated in degree s + 3, and we compute the corresponding graded Betti number β _{s, s+3}. The multidegrees of these nonlinear syzygies are also determined and the corresponding multigraded Betti numbers are shown to be all equal to 1.     

Efficient computer experiment-based optimization through variable selection

A computer experiment-based optimization approach employs design of experiments and statistical modeling to represent a complex objective function that can only be evaluated pointwise by running a computer model. In large-scale applications, the number of variables is huge, and direct use of computer experiments would require an exceedingly large experimental design and, consequently, significant computational effort. If a large portion of the variables have little impact on the objective, then there is a need to eliminate these before performing the complete set of computer experiments. This is a variable selection task. The ideal variable selection method for this task should handle unknown nonlinear structure, should be computationally fast, and would be conducted after a small number of computer experiment runs, likely fewer runs (N) than the number of variables (P). Conventional variable selection techniques are based on assumed linear model forms and cannot be applied in this “large P and small N” problem. In this paper, we present a framework that adds a variable selection step prior to computer experiment-based optimization, and we consider data mining methods, using principal components analysis and multiple testing based on false discovery rate, that are appropriate for our variable selection task. An airline fleet assignment case study is used to illustrate our approach.     

Nonlinear Preserver Problems on B（H）

Let H be a complex Hilbert space of dimension greater than 2, and B(H) denote the Banach algebra of all bounded linear operators on H. For A, B ∈ B(H), define the binary relation A ≤* B by A*A = A*B and AA* = AB*. Then (B(H), “≤*”) is a partially ordered set and the relation “≤*” is called the star order on B(H). Denote by Bs(H) the set of all self-adjoint operators in B(H). In this paper, we first characterize nonlinear continuous bijective maps on B _s (H) which preserve the star order in both directions. We characterize also additive maps (or linear maps) on B(H) (or nest algebras) which are multiplicative at some invertible operator.     

Model selection via adaptive shrinkage with t priors

We discuss a model selection procedure, the adaptive ridge selector, derived from a hierarchical Bayes argument, which results in a simple and efficient fitting algorithm. The hierarchical model utilized resembles an un-replicated variance components model and leads to weighting of the covariates. We discuss the intuition behind this type estimator and investigate its behavior as a regularized least squares procedure. While related alternatives were recently exploited to simultaneously fit and select variablses/features in regression models (Tipping in J Mach Learn Res 1:211–244, 2001; Figueiredo in IEEE Trans Pattern Anal Mach Intell 25:1150–1159, 2003), the extension presented here shows considerable improvement in model selection accuracy in several important cases. We also compare this estimator’s model selection performance to those offered by the lasso and adaptive lasso solution paths. Under randomized experimentation, we show that a fixed choice of tuning parameter leads to results in terms of model selection accuracy which are superior to the entire solution paths of lasso and adaptive lasso when the underlying model is a sparse one. We provide a robust version of the algorithm which is suitable in cases where outliers may exist.     

Robust and accurate inference for generalized linear models

In the framework of generalized linear models, the nonrobustness of classical estimators and tests for the parameters is a well known problem, and alternative methods have been proposed in the literature. These methods are robust and can cope with deviations from the assumed distribution. However, they are based on first order asymptotic theory, and their accuracy in moderate to small samples is still an open question. In this paper, we propose a test statistic which combines robustness and good accuracy for moderate to small sample sizes. We combine results from Cantoni and Ronchetti [E. Cantoni, E. Ronchetti, Robust inference for generalized linear models, Journal of the American Statistical Association 96 (2001) 1022–1030] and Robinson, Ronchetti and Young [J. Robinson, E. Ronchetti, G.A. Young, Saddlepoint approximations and tests based on multivariate M-estimators, The Annals of Statistics 31 (2003) 1154–1169] to obtain a robust test statistic for hypothesis testing and variable selection, which is asymptotically χ²-distributed as the three classical tests but with a relative error of order O(n⁻¹). This leads to reliable inference in the presence of small deviations from the assumed model distribution, and to accurate testing and variable selection, even in moderate to small samples.     

部分线性变系数模型Backfitting估计的渐近性质

作为部分线性模型与变系数模型的推广,部分线性变系数模型是一类应用广泛的数据分析模型.利用Backfitting方法拟合这类特殊的可加模型,可得到模型中常值系数估计量的精确解析表达式,该估计量被证明是n~(1/2)相合的.最后通过数值模拟考察了所提估计方法的有效性.     

Superconvergence of a stabilized finite element approximation for the Stokes equations using a local coarse mesh L2 projection

This article first recalls the results of a stabilized finite element method based on a local Gauss integration method for the stationary Stokes equations approximated by low equal‐order elements that do not satisfy the inf‐sup condition. Then, we derive general superconvergence results for this stabilized method by using a local coarse mesh L² projection. These supervergence results have three prominent features. First, they are based on a multiscale method defined for any quasi‐uniform mesh. Second, they are derived on the basis of a large sparse, symmetric positive‐definite system of linear equations for the solution of the stationary Stokes problem. Third, the finite elements used fail to satisfy the inf‐sup condition. This article combines the merits of the new stabilized method with that of the L² projection method. This projection method is of practical importance in scientific computation. Finally, a series of numerical experiments are presented to check the theoretical results obtained. © 2010 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 28: 115‐126, 2012