A multistage algorithm for best-subset model selection based on the Kullback–Leibler discrepancy

The selection of a best-subset regression model from a candidate family is a common problem that arises in many analyses. The Akaike information criterion (AIC) and the corrected AIC (\(\text {AIC}_c\)) are frequently used for this purpose. AIC and \(\text {AIC}_c\) are designed to estimate the expected Kullback–Leibler discrepancy. For best-subset selection, both AIC and \(\text {AIC}_c\) are negatively biased, and the use of either criterion will lead to the selection of overfitted models. To correct for this bias, we introduce an “improved” AIC variant, \(\text {AIC}_i\), which has a penalty term evaluated using Monte Carlo simulation. A multistage model selection procedure \(\text {AIC}_{\text {aps}}\), which utilizes \(\text {AIC}_i\), is proposed for best-subset selection. Simulation studies are compiled to compare the performances of the different model selection methods.     

Developing and Testing the Delay-Time Model

This paper further develops the basic delay-time model for inspections of repairable machinery described in a recent paper by Baker and Wang. In our earlier work, one of a set of simple models was selected and fitted to objective data by minimizing the Akaike Information Criterion (AIC). In the present work, several extensions to the basic model are derived, which relax earlier model assumptions, and are designed to cope with the complexities of real-world situations. These extensions include allowing the age of a machine to influence both the period u from replacement of a component to visibility of a defect, and the subsequent period h to failure; allowing an inspection to have a hazardous or beneficial effect on the lifetime of a component; and allowing several mechanisms that induce a correlation between the two hitherto independent periods u and h. The resulting models were fitted to objective data, and it was found that some model extensions improved the fit for particular components. In general, we conclude that a broad range of models should be explored. This is necessary both to capture the complexity of real data, and also to give confidence in the adequacy of simpler models.     

An Application of Adaptive Independent Chain Metropolis–Hastings Algorithms in Bayesian Hazard Rate Estimation

In this paper, we consider adaptive independent chain (AIC) Metropolis–Hastings algorithms as introduced in a special context in Gåsemyr et al. (2001) and developed theoretically in Gåsemyr (2003). The algorithms aim at producing samples from a specific target distribution , and are adaptive, non-Markovian versions of the Metropolis–Hastings independent chain. A certain parametric class of possible proposal distributions is fixed, and the parameters of the proposal distribution are updated periodically on the basis of the recent history of the chain, thereby obtaining proposals that get ever closer to . In the former paper a version of these algorithms was shown to be very efficient compared to standard simulation techniques when applied to Bayesian inference in reliability models with at most three dependent parameters. The aim of the present paper is to investigate the performance of the AIC algorithm when the number of dependent parameters and the complexity of the model increases. As a test case we consider a model treated in Arjas and Gasbarra (1996). The target distribution is the posterior distribution for the vector X=(X ₁,...,X _n ) of dependent parameters, representing a piecewise constant approximation to the hazard rate X(t), where t ₀ t t _n . Especially, for the case n=12 it turned out that some versions of the AIC were very efficient compared to standard simulation techniques and also to the algorithm applied in Arjas and Gasbarra (1996). This includes a version of the componentwise adaptive independent chain the basic idea of which was given in Gåsemyr (2003).     

Modeling of the response of elastic plastic materials treated as a mixture of hard and soft regions

Inelastic materials that form dislocation cells on being deformed are modeled as a constrained-mixture of plastically hard and soft regions by associating different natural states with these regions. The deformation gradient from the reference configuration to the natural configuration is identified as the plastic deformation tensor and the stress is measured from a changing set of natural configurations. Two sets of natural configurations are introduced: one for the hard phase and the other for the soft phase. The full elastic response of the body is determined by elastic responses from different natural configurations. The energy stored in the dislocation networks is explicitly accounted for in the Helmholtz potential. Within a specialized constitutive set up, the soft phase is assumed to be non-hardening while the hardening response of the hard phase is dependent upon the response of both the hard and soft phases. These special forms are used to model the response of the material that forms cellular structures when subjected to cyclic loading.     

Bootstrapping Log Likelihood and EIC, an Extension of AIC

Akaike (1973, 2nd International Symposium on Information Theory, 267-281,Akademiai Kiado, Budapest) proposed AIC as an estimate of the expected loglikelihood to evaluate the goodness of models fitted to a given set of data.The introduction of AIC has greatly widened the range of application ofstatistical methods. However, its limit lies in the point that it can beapplied only to the cases where the parameter estimation are performed bythe maximum likelihood method. The derivation of AIC is based on theassessment of the effect of data fluctuation through the asymptoticnormality of MLE. In this paper we propose a new information criterion EICwhich is constructed by employing the bootstrap method to simulate the datafluctuation. The new information criterion, EIC, is regarded as an extensionof AIC. The performance of EIC is demonstrated by some numerical examples.     

Sensitivity analysis of M-estimates

Bahadur representation of the difference of estimators of regression coefficients for the full data set and for the set from which one observation was deleted is given for the M-estimators which are generated by a continuous -function. The representation is invariant with respect to the scale of residuals and it indicates that the bound of the norm of the difference is proportional to the gross error sensitivity. Then for the -function which corresponds to the median it is shown that the difference of the estimates for the full data and for data without one observation, although being bounded in probability, can be much larger than indicated by the gross error sensitivity.     

Error Bounds for quasi-Monte Carlo integration with nets

We analyze the error introduced by approximately calculating the -dimensional Lebesgue measure of a Jordan-measurable subset of . We give an upper bound for the error of a method using a -net, which is a set with a very regular distribution behavior. When the subset of is defined by some function of bounded variation on , the error is estimated by means of the variation of the function and the discrepancy of the point set which is used. A sharper error bound is established when a -net is used. Finally a lower bound of the error is given, for a method using a -net. The special case of the 2-dimensional Hammersley point set is discussed.

     

Statistical forecasting for stochastic processes

The main purpose of this paper is to review the efficiency properties of least-squares predictors when the parameters are estimated. It is shown that the criterion of asymptotic best unbiased predictors for general stochastic models is a natural analogue of the minimum mean-square error criterion used traditionally in linear prediction for linear models. The results are applied to log-linear models and autoregressive processes. Both stationary and non-stationary processes are considered.This paper is based on a key note lecture given at the meeting of The Institute of Management Sciences and the Operations Research Society of America, held in Williamsburg, Virginia, January 7–9, 1985.     

SETAR model selection-A bootstrap approach

Summary  The aim of this paper is to propose new selection criteria for the orders of selfexciting threshold autoregressive (SETAR) models. These criteria use bootstrap methodology; they are based on a weighted mean of the apparent error rate in the sample and the average error rate obtained from bootstrap samples not containing the point being predicted. These new criteria are compared with the traditional ones based on the Akaike information criterion (AIC). A simulation study and an example on a real data set end the paper.     

Selection of smoothing parameters in<Emphasis Type="Italic">B</Emphasis>-spline nonparametric regression models using information criteria

We consider the use ofB-spline nonparametric regression models estimated by the maximum penalized likelihood method for extracting information from data with complex nonlinear structure. Crucial points inB-spline smoothing are the choices of a smoothing parameter and the number of basis functions, for which several selectors have been proposed based on cross-validation and Akaike information criterion known as AIC. It might be however noticed that AIC is a criterion for evaluating models estimated by the maximum likelihood method, and it was derived under the assumption that the ture distribution belongs to the specified parametric model. In this paper we derive information criteria for evaluatingB-spline nonparametric regression models estimated by the maximum penalized likelihood method in the context of generalized linear models under model misspecification. We use Monte Carlo experiments and real data examples to examine the properties of our criteria including various selectors proposed previously.     

Parallel structural optimization applied to bone remodeling on distributed memory machines

In this paper, we investigate parallel structural optimization methods on distributed memory MIMD machines. We have restricted ourselves to the case of minimizing a multivariate non-linear function subject to bounds on the independent variables, when the objective function is expensive to evaluate as compared to the linear algebra portion of the optimization. This is the case in structural applications, when a large three-dimensional finite element mesh is used to model the structure.This paper demonstrates how parallelism can be exploited during the function and gradient computation as well as the optimization iterations. For the finite element analysis, a torus wrap skyline solver is used. The reflective Newton method, which attempts to reduce the number of iterations at the expense of more linear algebra per iteration, is compared with the more conventional active set method. All code is developed for an Intel iPSC/860, but can be ported to other distributed memory machines.The methods developed are applied to problems in bone remodeling. In the area of biomechanics, optimization models can be used to predict changes in the distribution of material properties in bone due to the presence of an artificial implant. The model we have used minimizes a linear combination of the mass and strain energy in the entire domain subject to bounds on the densities in each finite element.Early results show that the reflective Newton method can outperform active set methods when few variables are active at the minimum.     

Estimation of spatial autoregressive models with measurement error for large data sets

Maximum likelihood (ML) estimation of spatial autoregressive models for large spatial data sets is well established by making use of the commonly sparse nature of the contiguity matrix on which spatial dependence is built. Adding a measurement error that naturally separates the spatial process from the measurement error process are not well established in the literature, however, and ML estimation of such models to large data sets is challenging. Recently a reduced rank approach was suggested which re-expresses and approximates such a model as a spatial random effects model (SRE) in order to achieve fast fitting of large data sets by fitting the corresponding SRE. In this paper we propose a fast and exact method to accomplish ML estimation and restricted ML estimation of complexity of \(O(n^{3/2})\) operations when the contiguity matrix is based on a local neighbourhood. The methods are illustrated using the well known data set on house prices in Lucas County in Ohio.     

The Log-Linear Birnbaum-Saunders Power Model

In this paper the sinh-power model is developed as a natural follow up to the log-linear Birnbaum-Saunders power model. The class of models resulting, incorporates the sinh-power-normal model, the ordinary sinh-normal model and the log-linear Birnbaum-Saunders model (Rieck and Nedelman, Technometrics 33:51–60, 1991). Maximum likelihood estimation is developed with the Hessian matrix used for standard error estimation. An application is reported for the data set on lung cancer studied in Kalbfleisch and Prentice (2002), where it is shown that the log-linear Birnbaum-Saunders power-normal model presents better fit than the log-linear Birnbaum-Saunders model. Another application is devoted to a fatigue data set previously analyzed in the literature. A nonlinear Birnbaum-Saunders power-normal model is fitted to the data set, with satisfactory performance.     

Model selection and prediction: Normal regression

This paper discusses the topic of model selection for finite-dimensional normal regression models. We compare model selection criteria according to prediction errors based upon prediction with refitting, and prediction without refitting. We provide a new lower bound for prediction without refitting, while a lower bound for prediction with refitting was given by Rissanen. Moreover, we specify a set of sufficient conditions for a model selection criterion to achieve these bounds. Then the achievability of the two bounds by the following selection rules are addressed: Rissanen's accumulated prediction error criterion (APE), his stochastic complexity criterion, AIC, BIC and the FPE criteria. In particular, we provide upper bounds on overfitting and underfitting probabilities needed for the achievability. Finally, we offer a brief discussion on the issue of finite-dimensional vs. infinite-dimensional model assumptions.Support from the National Science Foundation, grant DMS 8802378 and support from ARO, grant DAAL03-91-G-007 to B. Yu during the revision are gratefully acknowledged.     

Asymptotic normality and confidence intervals for inverse regression models with convolution-type operators

We consider inverse regression models with convolution-type operators which mediate convolution on (d≥1) and prove a pointwise central limit theorem for spectral regularisation estimators which can be applied to construct pointwise confidence regions. Here, we cope with the unknown bias of such estimators by undersmoothing. Moreover, we prove consistency of the residual bootstrap in this setting and demonstrate the feasibility of the bootstrap confidence bands at moderate sample sizes in a simulation study.     

The ε-algorithm for the identification of a transfer-function model: some applications

The aim of this paper is to apply an algorithm related to the rational approximation for the identification of the lag structure in a transfer-function model. In fact, we apply the -algorithm proposed by Berlinet [3–5] to determine the polynomial orders in univariate and multivariate ARMA models. Furthermore, it has been proposed by Berlinet [5], González and Cano [13, 14] and González et al. [15] for a transfer-function model with one input and multiple inputs, respectively.The main contribution in this paper concerns the study of the relative significance of the elements in the -algorithm table, in the same way as that given by Berlinet and Francq [7] for ARMA models, to confirm the pattern used to specify the model. Two examples will be considered, namely, the sales series M [8] and a simulated model [20].A comparison is also made between the results of the -algorithm and the corner method generally used in the econometric literature. Although the -algorithm requires a more advanced theory in Numerical Analysis, it can be applied in a more simple way than the corner method.     

Robust inference in linear mixed model with skew normal-symmetric error

Linear mixed effects models with general skew normal-symmetric (SNS) error are considered and several properties of the SNS distributions are obtained. Under the SNS settings, ANOVA-type estimates of variance components in the model are unbiased, the ANOVA-type F-tests are exact F-tests in SNS setting, and the exact confidence intervals for fixed effects are constructed. Also the power of ANOVA-type F-tests for components are free of the skewing function if the random effects normally distributed. For illustration of the main results, simulation studies on the robustness of the models are given by comparisons of multivariate skew-normal, multivariate skew normal-Laplace, multivariate skew normal-uniform, multivariate skew normal-symmetric, and multivariate normal distributed errors. A real example is provided for the illustration of the proposed method.     

Chapter 11 Simulation studies of efficiency,returns to scale and misspecification with nonlinear functions in DEA

Using statistically designed experiments, 12,500 observations are generated from a 4-pieced Cobb-Douglas function exhibiting increasing and decreasing returns to scale in its different pieces. Performances of DEA and frontier regressions represented by COLS (Corrected Ordinary Least Squares) are compared at sample sizes ofn=50, 100, 150 and 200. Statistical consistency is exhibited, with performances improving as sample sizes increase. Both DEA and COLS generally give good results at all sample sizes. In evaluating efficiency, DEA generally shows superior performance, with BCC models being best (except at corner points), followed by the CCR model and then by COLS, with log-linear regressions performing better than their translog counterparts at almost all sample sizes. Because of the need to consider locally varying behavior, only the CCR and translog models are used for returns to scale, with CCR being the better performer. An additional set of 7,500 observations were generated under conditions that made it possible to compare efficiency evaluations in the presence of collinearity and with model misspecification in the form of added and omitted variables. Results were similar to the larger experiment: the BCC model is the best performer. However, COLS exhibited surprisingly good performances — which suggests that COLS may have previously unidentified robustness properties — while the CCR model is the poorest performer when one of the variables used to generate the observations is omitted.     

On the shape–from–moments problem and recovering edges from noisy Radon data

We consider the problem of reconstructing a planar convex set from noisy observations of its moments. An estimation method based on pointwise recovering of the support function of the set is developed. We study intrinsic accuracy limitations in the shape–from–moments estimation problem by establishing a lower bound on the rate of convergence of the mean squared error. It is shown that the proposed estimator is near–optimal in the sense of the order. An application to tomographic reconstruction is discussed, and it is indicated how the proposed estimation method can be used for recovering edges from noisy Radon data.Mathematics Subject Classification (2000):62C20, 62G20, 94A12     

Improved Generalization via Tolerant Training

Theoretical and computational justification is given for improved generalization when the training set is learned with less accuracy. The model used for this investigation is a simple linear one. It is shown that learning a training set with a tolerance improves generalization, over zero-tolerance training, for any testing set satisfying a certain closeness condition to the training set. These results, obtained via a mathematical programming formulation, are placed in the context of some well-known machine learning results. Computational confirmation of improved generalization is given for linear systems (including nine of the twelve real-world data sets tested), as well as for nonlinear systems such as neural networks for which no theoretical results are available at present. In particular, the tolerant training method improves generalization on noisy, sparse, and overparameterized problems.