Adaptive penalized M-estimation with current status data

Current status data arises when a continuous response is reduced to an indicator of whether the response is greater or less than a random threshold value. In this article we consider adaptive penalized M-estimators (including the penalized least squares estimators and the penalized maximum likelihood estimators) for nonparametric and semiparametric models with current status data, under the assumption that the unknown nonparametric parameters belong to unknown Sobolev spaces. The Cox model is used as a representative of the semiparametric models. It is shown that the modified penalized M-estimators of the nonparametric parameters can achieve adaptive convergence rates, even when the degrees of smoothing are not known in advance. consistency, asymptotic normality and inference based on the weighted bootstrap for the estimators of the regression parameter in the Cox model are also established. A simulation study is conducted for the Cox model to evaluate the finite sample efficacy of the proposed approach and to compare it with the ordinary maximum likelihood estimator. It is demonstrated that the proposed method is computationally superior.We apply the proposed approach to the California Partner Study analysis.     

Testing for no effect via splines

A spline-based test statistic for a constant mean function is proposed based on the penalized residual sum-of-squares difference between the null model and a B-spline model in which the regression function is approximated with P-splines approach. When the number of knots is fixed, the limiting null distribution of the test statistic is shown to be the distribution of a linear combination of independent chi-squared random variables, each with one degree of freedom. A smoothing parameter is selected by setting a specified value equal to the expected value of the test statistic under the null hypothesis. Simulation experiments are conducted to study the proposed spline-based test statistic’s finite-sample properties.     

Penalized Likelihood-type Estimators for Generalized Nonparametric Regression

We consider the asymptotic analysis of penalized likelihood type estimators for generalized nonparametric regression problems in which the target parameter is a vector-valued function defined in terms of the conditional distribution of a response given a set of covariates. A variety of examples including ones related to generalized linear models and robust smoothing are covered by the theory. Linear approximations to the estimator are constructed using Taylor expansions in Hilbert spaces. An application which is treated is upper bounds on rates of convergence for the penalized likelihood-type estimators.     

Spline-based sieve maximum likelihood estimation in the partly linear model under monotonicity constraints

We study a spline-based likelihood method for the partly linear model with monotonicity constraints. We use monotone B-splines to approximate the monotone nonparametric function and apply the generalized Rosen algorithm to compute the estimators jointly. We show that the spline estimator of the nonparametric component achieves the possible optimal rate of convergence under the smooth assumption and that the estimator of the regression parameter is asymptotically normal and efficient. Moreover, a spline-based semiparametric likelihood ratio test is established to make inference of the regression parameter. Also an observed profile information method to consistently estimate the standard error of the spline estimator of the regression parameter is proposed. A simulation study is conducted to evaluate the finite sample performance of the proposed method. The method is illustrated by an air pollution study.     

Conditional and unconditional methods for selecting variables in linear mixed models

In the problem of selecting the explanatory variables in the linear mixed model, we address the derivation of the (unconditional or marginal) Akaike information criterion (AIC) and the conditional AIC (cAIC). The covariance matrices of the random effects and the error terms include unknown parameters like variance components, and the selection procedures proposed in the literature are limited to the cases where the parameters are known or partly unknown. In this paper, AIC and cAIC are extended to the situation where the parameters are completely unknown and they are estimated by the general consistent estimators including the maximum likelihood (ML), the restricted maximum likelihood (REML) and other unbiased estimators. We derive, related to AIC and cAIC, the marginal and the conditional prediction error criteria which select superior models in light of minimizing the prediction errors relative to quadratic loss functions. Finally, numerical performances of the proposed selection procedures are investigated through simulation studies.     

Bootstrapping Likelihood for Model Selection with Small Samples

Abstract

Akaike's information criterion (AIC), derived from asymptotics of the maximum likelihood estimator, is widely used in model selection. However, it has a finite-sample bias that produces overfitting in linear regression. To deal with this problem, Ishiguro, Sakamoto, and Kitagawa proposed a bootstrap-based extension to AIC which they called EIC. This article compares model-selection performance of AIC, EIC, a bootstrap-smoothed likelihood cross-validation (BCV) and its modification (632CV) in small-sample linear regression, logistic regression, and Cox regression. Simulation results show that EIC largely overcomes AIC's overfitting problem and that BCV may be better than EIC. Hence, the three methods based on bootstrapping the likelihood establish themselves as important alternatives to AIC in model selection with small samples.     

SICA for Cox’s proportional hazards model with a diverging number of parameters

The smooth integration of counting and absolute deviation （SICA） penalized variable selection procedure for high-dimensional linear regression models is proposed by Lv and Fan （2009）. In this article, we extend their idea to Cox＇s proportional hazards （PH） model by using a penalized log partial likelihood with the SICA penalty. The number of the regression coefficients is allowed to grow with the sample size. Based on an approximation to the inverse of the Hessian matrix, the proposed method can be easily carried out with the smoothing quasi-Newton （SQN） algorithm. Under appropriate sparsity conditions, we show that the resulting estimator of the regression coefficients possesses the oracle property. We perform an extensive simulation study to compare our approach with other methods and illustrate it on a well known PBC data for predicting survival from risk factors.     

Nonlinear logistic discrimination via regularized radial basis functions for classifying high-dimensional data

A flexible nonparametric method is proposed for classifying high- dimensional data with a complex structure. The proposed method can be regarded as an extended version of linear logistic discriminant procedures, in which the linear predictor is replaced by a radial-basis-expansion predictor. Radial basis functions with a hyperparameter are used to take the information on covariates and class labels into account; this was nearly impossible within the previously proposed hybrid learning framework. The penalized maximum likelihood estimation procedure is employed to obtain stable parameter estimates. A crucial issue in the model-construction process is the choice of a suitable model from candidates. This issue is examined from information-theoretic and Bayesian viewpoints and we employed Ando et al. (Japanese Journal of Applied Statistics, 31, 123–139, 2002)’s model evaluation criteria. The proposed method is available not only for the high-dimensional data but also for the variable selection problem. Real data analysis and Monte Carlo experiments show that our proposed method performs well in classifying future observations in practical situations. The simulation results also show that the use of the hyperparameter in the basis functions improves the prediction performance.     

Nonparametric Regression on a Graph

The ‘Signal plus Noise’ model for nonparametric regression can be extended to the case of observations taken at the vertices of a graph. This model includes many familiar regression problems. This article discusses the use of the edges of a graph to measure roughness in penalized regression. Distance between estimate and observation is measured at every vertex in the L₂ norm, and roughness is penalized on every edge in the L₁ norm. Thus the ideas of total variation penalization can be extended to a graph. The resulting minimization problem presents special computational challenges, so we describe a new and fast algorithm and demonstrate its use with examples.

The examples include image analysis, a simulation applicable to discrete spatial variation, and classification. In our examples, penalized regression improves upon kernel smoothing in terms of identifying local extreme values on planar graphs. In all examples we use fully automatic procedures for setting the smoothing parameters. Supplemental materials are available online.     

Iterative Approximation of Statistical Distributions and Relation to Information Geometry

The optimal control of stochastic processes through sensor estimation of probability density functions is given a geometric setting via information theory and the information metric. Information theory identifies the exponential distribution as the maximum entropy distribution if only the mean is known and the Γ distribution if also the mean logarithm is known. The surface representing Γ models has a natural Riemannian information metric. The exponential distributions form a one-dimensional subspace of the two-dimensional space of all Γ distributions, so we have an isometric embedding of the random model as a subspace of the Γ models. This geometry provides an appropriate structure on which to represent the dynamics of a process and algorithms to control it. This short paper presents a comparative study on the parameter estimation performance between the geodesic equation and the B-spline function approximations when they are used to optimize the parameters of the Γ family distributions. In this case, the B-spline functions are first used to approximate the Γ probability density function on a fixed length interval; then the coefficients of the approximation are related, through mean and variance calculations, to the two parameters (i.e. μ and β) in Γ distributions. A gradient based parameter tuning method has been used to produce the trajectories for (μ, β) when B-spline functions are used, and desired results have been obtained which are comparable to the trajectories obtained from the geodesic equation. This revised version was published online in August 2006 with corrections to the Cover Date.     

Penalized Nonparametric Scalar-on-Function Regression via Principal Coordinates

A number of classical approaches to nonparametric regression have recently been extended to the case of functional predictors. This article introduces a new method of this type, which extends intermediate-rank penalized smoothing to scalar-on-function regression. In the proposed method, which we call principal coordinate ridge regression, one regresses the response on leading principal coordinates defined by a relevant distance among the functional predictors, while applying a ridge penalty. Our publicly available implementation, based on generalized additive modeling software, allows for fast optimal tuning parameter selection and for extensions to multiple functional predictors, exponential family-valued responses, and mixed-effects models. In an application to signature verification data, principal coordinate ridge regression, with dynamic time warping distance used to define the principal coordinates, is shown to outperform a functional generalized linear model. Supplementary materials for this article are available online.     

Asymptotic theory of nonparametric regression estimates with censored data

For regression analysis, some useful information may have been lost when the responses are right censored. To estimate nonparametric functions, several estimates based on censored data have been proposed and their consistency and convergence rates have been studied in literature, but the optimal rates of global convergence have not been obtained yet. Because of the possible information loss, one may think that it is impossible for an estimate based on censored data to achieve the optimal rates of global convergence for nonparametric regression, which were established by Stone based on complete data. This paper constructs a regression spline estimate of a general nonparametric regression function based on right_censored response data, and proves, under some regularity conditions, that this estimate achieves the optimal rates of global convergence for nonparametric regression. Since the parameters for the nonparametric regression estimate have to be chosen based on a data driven criterion, we also obtain the asymptotic optimality of AIC, AICC, GCV, Cp and FPE criteria in the process of selecting the parameters.     

Estimation of Smooth Time-Varying Parameters in State Space Models

In this article, we propose a penalized likelihood method to estimate time-varying parameters in standard linear state space models. The time-varying parameter is modeled as a smoothing spline and then expressed as a state space model. The maximum likelihood method is used to estimate the smoothing parameter. The proposed method is assessed by a simulation study and applied to virological response data from an HIV-infected patient receiving antiretroviral treatment.     

M-Cross-Validation in Local Median Estimation

M-cross-validation criterion is proposed for selecting a smoothing parameter in a nonparametric median regression model in which a uniform weak convergency rate for the M-cross-validated local median estimate, and the upper and lower bounds of the smoothing parameter selected by the proposed criterion are established. The main contribution of this study shows a drastic difference from those encountered in the classical L2-, L1- cross-validation technique, which leads only to the consistency in the sense of the average. Obviously, our results are novel and nontrivial from the point of view of mathematics and statistics, which provides insight and possibility for practitioners substituting maximum deviation for average deviation to evaluate the performance of the data-driven technique.     

Nonparametric estimation of distributions with categorical and continuous data

In this paper we consider the problem of estimating an unknown joint distribution which is defined over mixed discrete and continuous variables. A nonparametric kernel approach is proposed with smoothing parameters obtained from the cross-validated minimization of the estimator's integrated squared error. We derive the rate of convergence of the cross-validated smoothing parameters to their ‘benchmark’ optimal values, and we also establish the asymptotic normality of the resulting nonparametric kernel density estimator. Monte Carlo simulations illustrate that the proposed estimator performs substantially better than the conventional nonparametric frequency estimator in a range of settings. The simulations also demonstrate that the proposed approach does not suffer from known limitations of the likelihood cross-validation method which breaks down with commonly used kernels when the continuous variables are drawn from fat-tailed distributions. An empirical application demonstrates that the proposed method can yield superior predictions relative to commonly used parametric models.     

‘Model selection for generalized linear models with factor‐augmented predictors’

This paper considers generalized linear models in a data‐rich environment in which a large number of potentially useful explanatory variables are available. In particular, it deals with the case that the sample size and the number of explanatory variables are of similar sizes. We adopt the idea that the relevant information of explanatory variables concerning the dependent variable can be represented by a small number of common factors and investigate the issue of selecting the number of common factors while taking into account the effect of estimated regressors. We develop an information criterion under model mis‐specification for both the distributional and structural assumptions and show that the proposed criterion is a natural extension of the Akaike information criterion (AIC). Simulations and empirical data analysis demonstrate that the proposed new criterion outperforms the AIC and Bayesian information criterion. Copyright © 2009 John Wiley & Sons, Ltd.     

对部分线性模型用惩罚最小二乘估计时最优光滑化的注记

部分线性模型也就是响应变量关于一个或者多个协变量是线性的, 但对于其他的协变量是非线性的关系\bd 对于部分线性模型中的参数和非参数部分的估计方法, 惩罚最小二乘估计是重要的估计方法之一\bd 对于这种估计方法, 广义交叉验证法提供了一种确定光滑参数的方法\bd 但是, 在部分线性模型中, 用广义交叉验证法确定光滑参数的最优性还没有被证明\bd 本文证明了利用惩罚最小二乘估计对于部分线性模型估计时, 用广义交叉验证法选择光滑参数的最优性\bd 通过模拟验证了本文中所提出的用广义交叉验证法选择光滑参数具有很好的效果, 同时, 本文在模拟部分比较了广义交叉验证和最小二乘交叉验证的优劣.     

Logspline Density Estimation for Censored Data

Abstract

Logspline density estimation is developed for data that may be right censored, left censored, or interval censored. A fully automatic method, which involves the maximum likelihood method and may involve stepwise knot deletion and either the Akaike information criterion (AIC) or Bayesian information criterion (BIC), is used to determine the estimate. In solving the maximum likelihood equations, the Newton–Raphson method is augmented by occasional searches in the direction of steepest ascent. Also, a user interface based on S is described for obtaining estimates of the density function, distribution function, and quantile function and for generating a random sample from the fitted distribution.     

Optimal tuning parameter estimation in maximum penalized likelihood method

In maximum penalized or regularized methods, it is important to select a tuning parameter appropriately. This paper proposes a direct plug-in method for tuning parameter selection. The tuning parameters selected using a generalized information criterion (Konishi and Kitagawa, Biometrika, 83, 875–890, 1996) and cross-validation (Stone, Journal of the Royal Statistical Society, Series B, 58, 267–288, 1974) are shown to be asymptotically equivalent to those selected using the proposed method, from the perspective of estimation of an optimal tuning parameter. Because of its directness, the proposed method is superior to the two selection methods mentioned above in terms of computational cost. Some numerical examples which contain the penalized spline generalized linear model regressions are provided.     

Optimal design for smoothing splines

In the common nonparametric regression model we consider the problem of constructing optimal designs, if the unknown curve is estimated by a smoothing spline. A special basis for the space of natural splines is introduced and the local minimax property for these splines is used to derive two optimality criteria for the construction of optimal designs. The first criterion determines the design for a most precise estimation of the coefficients in the spline representation and corresponds to D-optimality, while the second criterion is the G-optimality criterion and corresponds to an accurate prediction of the curve. Several properties of the optimal designs are derived. In general, D- and G-optimal designs are not equivalent. Optimal designs are determined numerically and compared with the uniform design.