Simultaneous change point analysis and variable selection in a regression problem

In this paper, an information-based criterion is proposed for carrying out change point analysis and variable selection simultaneously in linear models with a possible change point. Under some weak conditions, this criterion is shown to be strongly consistent in the sense that with probability one, it chooses the smallest true model for large n. Its byproducts include strongly consistent estimates of the regression coefficients regardless if there is a change point. In case that there is a change point, its byproducts also include a strongly consistent estimate of the change point parameter. In addition, an algorithm is given which has significantly reduced the computation time needed by the proposed criterion for the same precision. Results from a simulation study are also presented.     

Conditional and unconditional methods for selecting variables in linear mixed models

In the problem of selecting the explanatory variables in the linear mixed model, we address the derivation of the (unconditional or marginal) Akaike information criterion (AIC) and the conditional AIC (cAIC). The covariance matrices of the random effects and the error terms include unknown parameters like variance components, and the selection procedures proposed in the literature are limited to the cases where the parameters are known or partly unknown. In this paper, AIC and cAIC are extended to the situation where the parameters are completely unknown and they are estimated by the general consistent estimators including the maximum likelihood (ML), the restricted maximum likelihood (REML) and other unbiased estimators. We derive, related to AIC and cAIC, the marginal and the conditional prediction error criteria which select superior models in light of minimizing the prediction errors relative to quadratic loss functions. Finally, numerical performances of the proposed selection procedures are investigated through simulation studies.     

Conditional information criteria for selecting variables in linear mixed models

In this paper, we consider the problem of selecting the variables of the fixed effects in the linear mixed models where the random effects are present and the observation vectors have been obtained from many clusters. As the variable selection procedure, here we use the Akaike Information Criterion, AIC. In the context of the mixed linear models, two kinds of AIC have been proposed: marginal AIC and conditional AIC. In this paper, we derive three versions of conditional AIC depending upon different estimators of the regression coefficients and the random effects. Through the simulation studies, it is shown that the proposed conditional AIC’s are superior to the marginal and conditional AIC’s proposed in the literature in the sense of selecting the true model. Finally, the results are extended to the case when the random effects in all the clusters are of the same dimension but have a common unknown covariance matrix.     

A strongly consistent information criterion for linear model selection based on M-estimation

In this paper, a linear model selection procedure based on M-estimation is proposed, which includes many classical model selection criteria as its special cases. It is shown that the proposed criterion is strongly consistent under certain mild conditions, for instance without assuming normality of the distribution of the random errors. The results from a simulation study are also presented. Received: 13 October 1997 / Revised version: 10 August 1998     

Corrected version of AIC for selecting multivariate normal linear regression models in a general nonnormal case

This paper deals with the bias reduction of Akaike information criterion (AIC) for selecting variables in multivariate normal linear regression models when the true distribution of observation is an unknown nonnormal distribution. We propose a corrected version of AIC which is partially constructed by the jackknife method and is adjusted to the exact unbiased estimator of the risk when the candidate model includes the true model. It is pointed out that the influence of nonnormality in the bias of our criterion is smaller than the ones in AIC and TIC. We verify that our criterion is better than the AIC, TIC and EIC by conducting numerical experiments.     

Nonparametric additive model-assisted estimation for survey data

An additive model-assisted nonparametric method is investigated to estimate the finite population totals of massive survey data with the aid of auxiliary information. A class of estimators is proposed to improve the precision of the well known Horvitz-Thompson estimators by combining the spline and local polynomial smoothing methods. These estimators are calibrated, asymptotically design-unbiased, consistent, normal and robust in the sense of asymptotically attaining the Godambe-Joshi lower bound to the anticipated variance. A consistent model selection procedure is further developed to select the significant auxiliary variables. The proposed method is sufficiently fast to analyze large survey data of high dimension within seconds. The performance of the proposed method is assessed empirically via simulation studies.     

Consistent variable selection in large panels when factors are observable

In this paper we develop an econometric method for consistent variable selection in the context of a linear factor model with observable factors for panels of large dimensions. The subset of factors that best fit the data is sequentially determined. Firstly, a partial R² rule is used to show the existence of an optimal ordering of the candidate variables. Secondly, We show that for a given order of the regressors, the number of factors can be consistently estimated using the Bayes information criterion. The Akaike will asymptotically lead to overfitting of the model. The theory is established under approximate factor structure which allows for limited cross-section and serial dependence in the idiosyncratic term. Simulations show that the proposed two-step selection technique has good finite sample properties. The likelihood of selecting the correct specification increases with the number of cross-sections both asymptotically and in small samples. Moreover, the proposed variable selection method is computationally attractive. For K potential candidate factors, the search requires only 2K regressions compared to 2^K for an exhaustive search.     

Fixed and Random Effects Selection by REML and Pathwise Coordinate Optimization

We propose a two-stage model selection procedure for the linear mixed-effects models. The procedure consists of two steps: First, penalized restricted log-likelihood is used to select the random effects, and this is done by adopting a Newton-type algorithm. Next, the penalized log-likelihood is used to select the fixed effects via pathwise coordinate optimization to improve the computation efficiency. We prove that our procedure has the oracle properties. Both simulation studies and a real data example are carried out to examine finite sample performance of the proposed fixed and random effects selection procedure. Supplementary materials including R code used in this article and proofs for the theorems are available online.     

Estimation in mixed effects model with errors in variables

In this paper, we consider a linear mixed-effects model with measurement errors in both fixed and random effects and find the moment of estimators for the parameters of interest. The strong consistency and asymptotic normality of the estimators are obtained under regularity conditions. Moreover, we obtain the strong consistent estimators of the asymptotic covariance matrices involved in the limiting theory. Simulations are reported for illustration.     

Parametric estimation of a bivariate stable Lévy process

We propose a parametric model for a bivariate stable Lévy process based on a Lévy copula as a dependence model. We estimate the parameters of the full bivariate model by maximum likelihood estimation. As an observation scheme we assume that we observe all jumps larger than some ε>0 and base our statistical analysis on the resulting compound Poisson process. We derive the Fisher information matrix and prove asymptotic normality of all estimates when the truncation point ε→0. A simulation study investigates the loss of efficiency because of the truncation.     

Direct variable selection for discrimination among several groups

We propose a criterion for variable selection in discriminant analysis. This criterion permits to arrange the variables in decreasing order of adequacy for discrimination, so that the variable selection problem reduces to that of the estimation of suitable permutation and dimensionality. Then, estimators for these parameters are proposed and the resulting method for selecting variables is shown to be consistent. In a simulation study, we compute proportions of correct classification after variable selection in order to gain understanding of the performance of our proposal and to compare it to existing methods.     

Generalized beta prior models on fraction defective in reliability test planning

In many reliability analyses, the probability of obtaining a defective unit in a production process should not be considered constant even though the process is stable and in control. Engineering experience or previous data of similar or related products may often be used in the proper selection of a prior model to describe the random fluctuations in the fraction defective. A generalized beta family of priors, several maximum entropy priors and other prior models are considered for this purpose. In order to determine the acceptability of a product based on the lifelengths of some test units, failure-censored reliability sampling plans for location-scale distributions using average producer and consumer risks are designed. Our procedure allows the practitioners to incorporate a restricted parameter space into the reliability analysis, and it is reasonably insensitive to small disturbances in the prior information. Impartial priors are used to reflect prior neutrality between the producer and the consumer when a consensus on the elicited prior model is required. Nonetheless, our approach also enables the producer and the consumer to assume their own prior distributions. The use of substantial prior information can, in many cases, significantly reduce the amount of testing required. However, the main advantage of utilizing a prior model for the fraction defective is not necessarily reduced sample size but improved assessment of the true sampling risks. An example involving shifted exponential lifetimes is considered to illustrate the results.     

Maximum likelihood estimator consistency for a ballistic random walk in a parametric random environment

We consider a one dimensional ballistic random walk evolving in an i.i.d. parametric random environment. We provide a maximum likelihood estimation procedure of the parameters based on a single observation of the path till the time it reaches a distant site, and prove that the estimator is consistent as the distant site tends to infinity. Our main tool consists in using the link between random walks and branching processes in random environments and explicitly characterising the limiting distribution of the process that arises. We also explore the numerical performance of our estimation procedure.     

Offline and online weighted least squares estimation of nonstationary power ARCH processes

This paper proposes two estimation methods based on a weighted least squares criterion for non-(strictly) stationary power ARCH models. The weights are the squared volatilities evaluated at a known value in the parameter space. The first method is adapted for fixed sample size data while the second one allows for online data available in real time. It will be shown that these methods provide consistent and asymptotically Gaussian estimates having asymptotic variance equal to that of the quasi-maximum likelihood estimate (QMLE) regardless of the value of the weighting parameter. Finite-sample performances of the proposed WLS estimates are shown via a simulation study for various sub-classes of power ARCH models.     

Parameter estimation under ambiguity and contamination with the spurious model

Recently, we proposed variants as a statistical model for treating ambiguity. If data are extracted from an object with a machine then it might not be able to give a unique safe answer due to ambiguity about the correct interpretation of the object. On the other hand, the machine is often able to produce a finite number of alternative feature sets (of the same object) that contain the desired one. We call these feature sets variants of the object. Data sets that contain variants may be analyzed by means of statistical methods and all chapters of multivariate analysis can be seen in the light of variants. In this communication, we focus on point estimation in the presence of variants and outliers. Besides robust parameter estimation, this task requires also selecting the regular objects and their valid feature sets (regular variants). We determine the mixed MAP-ML estimator for a model with spurious variants and outliers as well as estimators based on the integrated likelihood. We also prove asymptotic results which show that the estimators are nearly consistent.The problem of variant selection turns out to be computationally hard; therefore, we also design algorithms for efficient approximation. We finally demonstrate their efficacy with a simulated data set and a real data set from genetics.     

Best linear unbiased prediction for linear combinations in general mixed linear models

The general mixed linear model can be written as . In this paper, we mainly deal with two problems. Firstly, the problem of predicting a general linear combination of fixed effects and realized values of random effects in a general mixed linear model is considered and an explicit representation of the best linear unbiased predictor (BLUP) is derived. In addition, we apply the resulting conclusion to several special models and offer an alternative to characterization of BLUP. Secondly, we recall the notion of linear sufficiency and consider it as regards the BLUP problem and characterize it in several different ways. Further, we study the concepts of linear sufficiency, linear minimal sufficiency and linear completeness, and give relations among them. Finally, four concluding remarks are given.     

Optimal adaptive generalized Pólya urn design for multi-arm clinical trials

A class of optimal adaptive multi-arm clinical trial designs is proposed based on an extended generalized Pólya urn (GPU) model. The design is applicable to both the qualitative and quantitative responses and achieves, asymptotically, some pre-specified optimality criterion. Such criterion is specified by a functional of the response distributions and is implemented through the relationship between the design matrix and its first eigenvector. The asymptotic properties of the design are studied using the existing methods on GPU. Some examples for commonly used clinical designs are given as illustration.     

A test for the mean vector with fewer observations than the dimension

In this paper, we consider a test for the mean vector of independent and identically distributed multivariate normal random vectors where the dimension p is larger than or equal to the number of observations N. This test is invariant under scalar transformations of each component of the random vector. Theories and simulation results show that the proposed test is superior to other two tests available in the literature. Interest in such significance test for high-dimensional data is motivated by DNA microarrays. However, the methodology is valid for any application which involves high-dimensional data.     

On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding

We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu [K. Knight, W. Fu, Asymptotics for lasso-type estimators, Annals of Statistics 28 (2000) 1356–1378] and Fan and Li [J. Fan, R. Li, Variable selection via non-concave penalized likelihood and its oracle properties, Journal of the American Statistical Association 96 (2001) 1348–1360]. We show that the distributions are typically highly non-normal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than n^−1/2 in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators’ distribution function is also provided.     

Asymptotics for estimation and testing procedures under loss of identifiability

Statistical analyses commonly make use of models that suffer from loss of identifiability. In this paper, we address important issues related to the parameter estimation and hypothesis testing in models with loss of identifiability. That is, there are multiple parameter points corresponding to the same true model. We refer the set of these parameter points to as the set of true parameter values. We consider the case where the set of true parameter values is allowed to be very large or even infinite, some parameter values may lie on the boundary of the parameter space, and the data are not necessarily independently and identically distributed. Our results are applicable to a large class of estimators and their related testing statistics derived from optimizing an objective function such as a likelihood. We examine three specific examples: (i) a finite mixture logistic regression model; (ii) stationary ARMA processes; (iii) general quadratic approximation using Hellinger distance. The applications to these examples demonstrate the applicability of our results in a broad range of difficult statistical problems.