Variable selection in the partially linear errors-in-variables models for longitudinal data

This paper proposes a new approach for variable selection in partially linear errors-in-variables (EV) models for longitudinal data by penalizing appropriate estimating functions. We apply the SCAD penalty to simultaneously select significant variables and estimate unknown parameters. The rate of convergence and the asymptotic normality of the resulting estimators are established. Furthermore, with proper choice of regularization parameters, we show that the proposed estimators perform as well as the oracle procedure. A new algorithm is proposed for solving penalized estimating equation. The asymptotic results are augmented by a simulation study.     

An Iterative Coordinate Descent Algorithm for High-Dimensional Nonconvex Penalized Quantile Regression

We propose and study a new iterative coordinate descent algorithm (QICD) for solving nonconvex penalized quantile regression in high dimension. By permitting different subsets of covariates to be relevant for modeling the response variable at different quantiles, nonconvex penalized quantile regression provides a flexible approach for modeling high-dimensional data with heterogeneity. Although its theory has been investigated recently, its computation remains highly challenging when p is large due to the nonsmoothness of the quantile loss function and the nonconvexity of the penalty function. Existing coordinate descent algorithms for penalized least-squares regression cannot be directly applied. We establish the convergence property of the proposed algorithm under some regularity conditions for a general class of nonconvex penalty functions including popular choices such as SCAD (smoothly clipped absolute deviation) and MCP (minimax concave penalty). Our Monte Carlo study confirms that QICD substantially improves the computational speed in the p ? n setting. We illustrate the application by analyzing a microarray dataset.     

SICA for Cox’s proportional hazards model with a diverging number of parameters

The smooth integration of counting and absolute deviation （SICA） penalized variable selection procedure for high-dimensional linear regression models is proposed by Lv and Fan （2009）. In this article, we extend their idea to Cox＇s proportional hazards （PH） model by using a penalized log partial likelihood with the SICA penalty. The number of the regression coefficients is allowed to grow with the sample size. Based on an approximation to the inverse of the Hessian matrix, the proposed method can be easily carried out with the smoothing quasi-Newton （SQN） algorithm. Under appropriate sparsity conditions, we show that the resulting estimator of the regression coefficients possesses the oracle property. We perform an extensive simulation study to compare our approach with other methods and illustrate it on a well known PBC data for predicting survival from risk factors.     

Selection of smoothing parameters in<Emphasis Type="Italic">B</Emphasis>-spline nonparametric regression models using information criteria

We consider the use ofB-spline nonparametric regression models estimated by the maximum penalized likelihood method for extracting information from data with complex nonlinear structure. Crucial points inB-spline smoothing are the choices of a smoothing parameter and the number of basis functions, for which several selectors have been proposed based on cross-validation and Akaike information criterion known as AIC. It might be however noticed that AIC is a criterion for evaluating models estimated by the maximum likelihood method, and it was derived under the assumption that the ture distribution belongs to the specified parametric model. In this paper we derive information criteria for evaluatingB-spline nonparametric regression models estimated by the maximum penalized likelihood method in the context of generalized linear models under model misspecification. We use Monte Carlo experiments and real data examples to examine the properties of our criteria including various selectors proposed previously.     

Detecting Changes in Slope With an L0 Penalty

While there are many approaches to detecting changes in mean for a univariate time series, the problem of detecting multiple changes in slope has comparatively been ignored. Part of the reason for this is that detecting changes in slope is much more challenging: simple binary segmentation procedures do not work for this problem, while existing dynamic programming methods that work for the change in mean problem cannot be used for detecting changes in slope. We present a novel dynamic programming approach, CPOP, for finding the “best” continuous piecewise linear fit to data under a criterion that measures fit to data using the residual sum of squares, but penalizes complexity based on an L₀ penalty on changes in slope. We prove that detecting changes in this manner can lead to consistent estimation of the number of changepoints, and show empirically that using an L₀ penalty is more reliable at estimating changepoint locations than using an L₁ penalty. Empirically CPOP has good computational properties, and can analyze a time series with 10,000 observations and 100 changes in a few minutes. Our method is used to analyze data on the motion of bacteria, and provides better and more parsimonious fits than two competing approaches. Supplementary material for this article is available online.     

An Estimation of Exact Penalty for Infinite-Dimensional Inequality-Constrained Minimization Problems

We use the penalty approach in order to study inequality-constrained minimization problems in infinite dimensional spaces. A penalty function is said to have the exact penalty property if there is a penalty coefficient for which a solution of an unconstrained penalized problem is a solution of the corresponding constrained problem. In this paper we consider a large class of inequality-constrained minimization problems for which a constraint is a mapping with values in a normed ordered space. For this class of problems we introduce a new type of penalty functions, establish the exact penalty property and obtain an estimation of the exact penalty. Using this exact penalty property we obtain necessary and sufficient optimality conditions for the constrained minimization problems.     

A Penalized Likelihood Approach to Rotation of Principal Components

A new paradigm for enhancing the interpretability of principal components through rotation is presented within the framework of penalized likelihood. The rotated components are computed as the maximizers of a Gaussian-based profile log-likelihood function plus a penalty term defined by a standard rotation criterion. This method enjoys a number of advantages over other methods for principal component rotation, notably (1) the rotation specifically targets ill-defined principal components, which may benefit the most from rotation, and (2) the connection with likelihood allows assessment of the fidelity of the rotated components to the data, thereby guiding the choice of penalty parameter. The method is illustrated with an application to a small functional dataset. Efficient computation of the penalized likelihood solution is possible using recently developed algorithms for optimization under orthogonality constraints.     

A uniform framework for the combination of penalties in generalized structured models

Penalized estimation has become an established tool for regularization and model selection in regression models. A variety of penalties with specific features are available and effective algorithms for specific penalties have been proposed. But not much is available to fit models with a combination of different penalties. When modeling the rent data of Munich as in our application, various types of predictors call for a combination of a Ridge, a group Lasso and a Lasso-type penalty within one model. We propose to approximate penalties that are (semi-)norms of scalar linear transformations of the coefficient vector in generalized structured models—such that penalties of various kinds can be combined in one model. The approach is very general such that the Lasso, the fused Lasso, the Ridge, the smoothly clipped absolute deviation penalty, the elastic net and many more penalties are embedded. The computation is based on conventional penalized iteratively re-weighted least squares algorithms and hence, easy to implement. New penalties can be incorporated quickly. The approach is extended to penalties with vector based arguments. There are several possibilities to choose the penalty parameter(s). A software implementation is available. Some illustrative examples show promising results.     

Simultaneous Credible Regions for Multiple Changepoint Locations

Within a Bayesian retrospective framework, we present a way of examining the distribution of changepoints through a novel set estimator. For a given level, α, we aim at smallest sets that cover all changepoints with a probability of at least 1 ? α. These so-called smallest simultaneous credible regions, computed for certain values of α, provide parsimonious representations of the possible changepoint locations. In addition, combining them for a range of different α’s enables very informative yet condensed visualizations. Therewith we allow for the evaluation of model choices and the analysis of changepoint data to an unprecedented degree. This approach exhibits superior sensitivity, specificity, and interpretability in comparison with highest density regions, marginal inclusion probabilities, and confidence intervals inferred by stepR. While their direct construction is usually intractable, asymptotically correct solutions can be derived from posterior samples. This leads to a novel NP-complete problem. Through reformulations into an Integer Linear Program we show empirically that a fast greedy heuristic computes virtually exact solutions. Supplementary material for this article is available online.     

Power Penalty Method for a Linear Complementarity Problem Arising from American Option Valuation

In this paper, we present a power penalty function approach to the linear complementarity problem arising from pricing American options. The problem is first reformulated as a variational inequality problem; the resulting variational inequality problem is then transformed into a nonlinear parabolic partial differential equation (PDE) by adding a power penalty term. It is shown that the solution to the penalized equation converges to that of the variational inequality problem with an arbitrary order. This arbitrary-order convergence rate allows us to achieve the required accuracy of the solution with a small penalty parameter. A numerical scheme for solving the penalized nonlinear PDE is also proposed. Numerical results are given to illustrate the theoretical findings and to show the effectiveness and usefulness of the method. This work was partially supported by a research grant from the University of Western Australia and the Research Grant Council of Hong Kong, Grants PolyU BQ475 and PolyU BQ493.     

Penalized likelihood regression for generalized linear models with non-quadratic penalties

One of the popular method for fitting a regression function is regularization: minimizing an objective function which enforces a roughness penalty in addition to coherence with the data. This is the case when formulating penalized likelihood regression for exponential families. Most of the smoothing methods employ quadratic penalties, leading to linear estimates, and are in general incapable of recovering discontinuities or other important attributes in the regression function. In contrast, non-linear estimates are generally more accurate. In this paper, we focus on non-parametric penalized likelihood regression methods using splines and a variety of non-quadratic penalties, pointing out common basic principles. We present an asymptotic analysis of convergence rates that justifies the approach. We report on a simulation study including comparisons between our method and some existing ones. We illustrate our approach with an application to Poisson non-parametric regression modeling of frequency counts of reported acquired immune deficiency syndrome (AIDS) cases in the UK.     

An iterative algorithm for sparse and constrained recovery with applications to divergence-free current reconstructions in magneto-encephalography

We propose an iterative algorithm for the minimization of a ? ₁-norm penalized least squares functional, under additional linear constraints. The algorithm is fully explicit: it uses only matrix multiplications with the three matrices present in the problem (in the linear constraint, in the data misfit part and in the penalty term of the functional). None of the three matrices must be invertible. Convergence is proven in a finite-dimensional setting. We apply the algorithm to a synthetic problem in magneto-encephalography where it is used for the reconstruction of divergence-free current densities subject to a sparsity promoting penalty on the wavelet coefficients of the current densities. We discuss the effects of imposing zero divergence and of imposing joint sparsity (of the vector components of the current density) on the current density reconstruction.     

Exact penalty property for a class of inequality-constrained minimization problems

We use the penalty approach in order to study constrained minimization problems. A penalty function is said to have the exact penalty property if there is a penalty coefficient for which a solution of an unconstrained penalized problem is a solution of the corresponding constrained problem. In this paper we establish the exact penalty property for a large class of inequality-constrained minimization problems.     

Existence of Exact Penalty for Constrained Optimization Problems in Metric Spaces

In this paper we use the penalty approach in order to study constrained minimization problems in a complete metric space with locally Lipschitzian mixed constraints. A penalty function is said to have the exact penalty property if there is a penalty coefficient for which a solution of an unconstrained penalized problem is a solution of the corresponding constrained problem. In this paper we establish sufficient conditions for the exact penalty property.      

A sufficient condition for exact penalty functions

In this paper, we use the penalty approach for constrained minimization problems in infinite dimensional Banach spaces. A penalty function is said to have the exact penalty property if there is a penalty coefficient for which a solution of an unconstrained penalized problem is a solution of the corresponding constrained problem. We establish a simple sufficient condition for exact penalty property for two large classes of constrained minimization problems.     

Penalty Method for Variational Inequalities

In this paper we deal with the existence theory for a problem and give the proof of the existence by a penalty argument. We shall treat the problem for a variational inequality by introducing the penalized differential equation and then taking the limits of the equation resulting from the penalized approximation. We also discuss the error estimate for the difference of the two solutions in an appropriate norm.     

Sufficient conditions for exact penalty in constrained optimization

In this paper we use the penalty approach in order to study constrained minimization problems. A penalty function is said to have the exact penalty property if there is a penalty coefficient for which a solution of an unconstrained penalized problem is a solution of the corresponding constrained problem. We discuss very simple sufficient conditions for the exact penalty property. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Dual Convergence for Penalty Algorithms in Convex Programming

Algorithms for convex programming, based on penalty methods, can be designed to have good primal convergence properties even without uniqueness of optimal solutions. Taking primal convergence for granted, in this paper we investigate the asymptotic behavior of an appropriate dual sequence obtained directly from primal iterates. First, under mild hypotheses, which include the standard Slater condition but neither strict complementarity nor second-order conditions, we show that this dual sequence is bounded and also, each cluster point belongs to the set of Karush–Kuhn–Tucker multipliers. Then we identify a general condition on the behavior of the generated primal objective values that ensures the full convergence of the dual sequence to a specific multiplier. This dual limit depends only on the particular penalty scheme used by the algorithm. Finally, we apply this approach to prove the first general dual convergence result of this kind for penalty-proximal algorithms in a nonlinear setting.     

Minimal Penalties for Gaussian Model Selection

This paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimization of a penalized least-squares type criterion within some general Gaussian framework including the classical ones. As compared to our previous paper on this topic (Birgé and Massart in J. Eur. Math. Soc. 3, 203–268 (2001)), more elaborate forms of the penalties are given which are shown to be, in some sense, optimal. We indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms, showing that the use of smaller penalties may lead to disastrous results. These lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown. We provide an illustration of the method for the problem of estimating a piecewise constant signal in Gaussian noise when neither the number, nor the location of the change points are known.     

Applying a Power Penalty Method to Numerically Pricing American Bond Options

In this paper, we aim to develop a numerical scheme to price American options on a zero-coupon bond based on a power penalty approach. This pricing problem is formulated as a variational inequality problem (VI) or a complementarity problem (CP). We apply a fitted finite volume discretization in space along with an implicit scheme in time, to the variational inequality problem, and obtain a discretized linear complementarity problem (LCP). We then develop a power penalty approach to solve the LCP by solving a system of nonlinear equations. The unique solvability and convergence of the penalized problem are established. Finally, we carry out numerical experiments to examine the convergence of the power penalty method and to testify the efficiency and effectiveness of our numerical scheme.