Ridge Fusion in Statistical Learning

This article proposes a penalized likelihood method to jointly estimate multiple precision matrices for use in quadratic discriminant analysis (QDA) and model-based clustering. We use a ridge penalty and a ridge fusion penalty to introduce shrinkage and promote similarity between precision matrix estimates. We use blockwise coordinate descent for optimization, and validation likelihood is used for tuning parameter selection. Our method is applied in QDA and semi-supervised model-based clustering.     

Variable Selection in General Frailty Models Using Penalized H-Likelihood

Variable selection methods using a penalized likelihood have been widely studied in various statistical models. However, in semiparametric frailty models, these methods have been relatively less studied because the marginal likelihood function involves analytically intractable integrals, particularly when modeling multicomponent or correlated frailties. In this article, we propose a simple but unified procedure via a penalized h-likelihood (HL) for variable selection of fixed effects in a general class of semiparametric frailty models, in which random effects may be shared, nested, or correlated. We consider three penalty functions (least absolute shrinkage and selection operator [LASSO], smoothly clipped absolute deviation [SCAD], and HL) in our variable selection procedure. We show that the proposed method can be easily implemented via a slight modification to existing HL estimation approaches. Simulation studies also show that the procedure using the SCAD or HL penalty performs well. The usefulness of the new method is illustrated using three practical datasets too. Supplementary materials for the article are available online.     

Penalized maximum likelihood estimation of a stochastic multivariate regression model

We study the large-sample properties of the penalized maximum likelihood estimator of a multivariate stochastic regression model with contemporaneously correlated data. The penalty is in terms of the square norm of some (vector) linear function of the regression coefficients. The model subsumes the so-called common transfer function model useful for extracting common signals in a panel of short time series. We show that, under mild regularity conditions, the penalized maximum likelihood estimator is consistent and asymptotically normal. The asymptotic bias of the regression coefficient estimator is also derived.     

Space alternating penalized Kullback proximal point algorithms for maximizing likelihood with nondifferentiable penalty

The EM algorithm is a widely used methodology for penalized likelihood estimation. Provable monotonicity and convergence are the hallmarks of the EM algorithm and these properties are well established for smooth likelihood and smooth penalty functions. However, many relaxed versions of variable selection penalties are not smooth. In this paper, we introduce a new class of space alternating penalized Kullback proximal extensions of the EM algorithm for nonsmooth likelihood inference. We show that the cluster points of the new method are stationary points even when they lie on the boundary of the parameter set. We illustrate the new class of algorithms for the problems of model selection for finite mixtures of regression and of sparse image reconstruction.     

Penalized likelihood regression for generalized linear models with non-quadratic penalties

One of the popular method for fitting a regression function is regularization: minimizing an objective function which enforces a roughness penalty in addition to coherence with the data. This is the case when formulating penalized likelihood regression for exponential families. Most of the smoothing methods employ quadratic penalties, leading to linear estimates, and are in general incapable of recovering discontinuities or other important attributes in the regression function. In contrast, non-linear estimates are generally more accurate. In this paper, we focus on non-parametric penalized likelihood regression methods using splines and a variety of non-quadratic penalties, pointing out common basic principles. We present an asymptotic analysis of convergence rates that justifies the approach. We report on a simulation study including comparisons between our method and some existing ones. We illustrate our approach with an application to Poisson non-parametric regression modeling of frequency counts of reported acquired immune deficiency syndrome (AIDS) cases in the UK.     

Model selection using modified AIC and BIC in joint modeling of paired functional data

A modified version of the Akaike information criterion and two modified versions of the Bayesian information criterion are proposed to select the number of principal components and to choose the penalty parameters of penalized splines in a joint model of paired functional data. Numerical results show that, compared with an existing procedure using the cross-validation, the procedure based on the information criteria is computationally much faster while giving a similar performance.     

Precision Matrix Estimation With ROPE

It is known that the accuracy of the maximum likelihood-based covariance and precision matrix estimates can be improved by penalized log-likelihood estimation. In this article, we propose a ridge-type operator for the precision matrix estimation, ROPE for short, to maximize a penalized likelihood function where the Frobenius norm is used as the penalty function. We show that there is an explicit closed form representation of a shrinkage estimator for the precision matrix when using a penalized log-likelihood, which is analogous to ridge regression in a regression context. The performance of the proposed method is illustrated by a simulation study and real data applications. Computer code used in the example analyses as well as other supplementary materials for this article are available online.     

Inference for multivariate normal mixtures

Multivariate normal mixtures provide a flexible model for high-dimensional data. They are widely used in statistical genetics, statistical finance, and other disciplines. Due to the unboundedness of the likelihood function, classical likelihood-based methods, which may have nice practical properties, are inconsistent. In this paper, we recommend a penalized likelihood method for estimating the mixing distribution. We show that the maximum penalized likelihood estimator is strongly consistent when the number of components has a known upper bound. We also explore a convenient EM-algorithm for computing the maximum penalized likelihood estimator. Extensive simulations are conducted to explore the effectiveness and the practical limitations of both the new method and the ratified maximum likelihood estimators. Guidelines are provided based on the simulation results.     

SICA for Cox’s proportional hazards model with a diverging number of parameters

The smooth integration of counting and absolute deviation （SICA） penalized variable selection procedure for high-dimensional linear regression models is proposed by Lv and Fan （2009）. In this article, we extend their idea to Cox＇s proportional hazards （PH） model by using a penalized log partial likelihood with the SICA penalty. The number of the regression coefficients is allowed to grow with the sample size. Based on an approximation to the inverse of the Hessian matrix, the proposed method can be easily carried out with the smoothing quasi-Newton （SQN） algorithm. Under appropriate sparsity conditions, we show that the resulting estimator of the regression coefficients possesses the oracle property. We perform an extensive simulation study to compare our approach with other methods and illustrate it on a well known PBC data for predicting survival from risk factors.     

求解广义纳什均衡问题的指数型惩罚函数方法

本文利用指数型惩罚函数部分地惩罚耦合约束,从而将广义纳什均衡问题(GNEP)的求解转化为求解一系列光滑的惩罚纳什均衡问题 (NEP)。我们证明了若光滑的惩罚NEP序列的解序列的聚点处EMFCQ成立,则此聚点是 GNEP的一个解。进一步,我们把惩罚 NEP的KKT条件转化为一个非光滑方程系统,然后应用带有 Armijo 线搜索的半光滑牛顿法来求解此系统。最后,数值结果表明我们的指数型惩罚函数方法是有效的。     

混合von Mises 模型的参数估计

有限混合von Mises模型在天文学、生物学、地理和医药等许多领域都有重要的应用．可是,不论样本量有多大,此模型的似然函数都是无界的．因此,参数的最大似然估计(MLE)是不相合的．我们发现,与混合正态模型一样,上述困难可以通过引入关于分布浓度参数的一个惩罚函数或对参数空间添加适当的约束来克服．在此文中,我们从理论上证明了这两种方法是可行的,相应的参数估计是强相合的,且是渐近有效的．我们还通过计算机模拟来探讨这些新方法在有限样本情况下的统计性质,并与现有的矩估计作了比较．结果发现,惩罚极大似然估计在均方误差方面表现最佳．最后我们还分析了一组实际数据,以进一步介绍新的估计方法．     

Automatic Response Category Combination in Multinomial Logistic Regression

We propose a penalized likelihood method that simultaneously fits the multinomial logistic regression model and combines subsets of the response categories. The penalty is nondifferentiable when pairs of columns in the optimization variable are equal. This encourages pairwise equality of these columns in the estimator, which corresponds to response category combination. We use an alternating direction method of multipliers algorithm to compute the estimator and we discuss the algorithm’s convergence. Prediction and model selection are also addressed. Supplemental materials for this article are available online.     

On penalty method for equilibrium problems in lexicographic order

In this paper, we consider lexicographic vector equilibrium problems. We propose a penalty function method for solving such problems. We show that every penalty trajectory of the penalized lexicographic equilibrium problem tends to the solution of the original problem. Using the regularized gap function to obtain an error bound result for such penalized problems is given.     

Penalized Nonparametric Scalar-on-Function Regression via Principal Coordinates

A number of classical approaches to nonparametric regression have recently been extended to the case of functional predictors. This article introduces a new method of this type, which extends intermediate-rank penalized smoothing to scalar-on-function regression. In the proposed method, which we call principal coordinate ridge regression, one regresses the response on leading principal coordinates defined by a relevant distance among the functional predictors, while applying a ridge penalty. Our publicly available implementation, based on generalized additive modeling software, allows for fast optimal tuning parameter selection and for extensions to multiple functional predictors, exponential family-valued responses, and mixed-effects models. In an application to signature verification data, principal coordinate ridge regression, with dynamic time warping distance used to define the principal coordinates, is shown to outperform a functional generalized linear model. Supplementary materials for this article are available online.     

A Lower Bound for the Penalty Parameter in the Exact Minimax Penalty Function Method for Solving Nondifferentiable Extremum Problems

In the paper, we consider the exact minimax penalty function method used for solving a general nondifferentiable extremum problem with both inequality and equality constraints. We analyze the relationship between an optimal solution in the given constrained extremum problem and a minimizer in its associated penalized optimization problem with the exact minimax penalty function under the assumption of convexity of the functions constituting the considered optimization problem (with the exception of those equality constraint functions for which the associated Lagrange multipliers are negative—these functions should be assumed to be concave). The lower bound of the penalty parameter is given such that, for every value of the penalty parameter above the threshold, the equivalence holds between the set of optimal solutions in the given extremum problem and the set of minimizers in its associated penalized optimization problem with the exact minimax penalty function.     

Exact penalty functions method for mathematical programming problems involving invex functions

In this paper, some new results on the exact penalty function method are presented. Simple optimality characterizations are given for the differentiable nonconvex optimization problems with both inequality and equality constraints via exact penalty function method. The equivalence between sets of optimal solutions in the original mathematical programming problem and its associated exact penalized optimization problem is established under suitable invexity assumption. Furthermore, the equivalence between a saddle point in the invex mathematical programming problem and an optimal point in its exact penalized optimization problem is also proved.     

A new exact exponential penalty function method and nonconvex mathematical programming

A new exact penalty function method, called the l₁ exact exponential penalty function method, is introduced. In this approach, the so-called the exponential penalized optimization problem with the l₁ exact exponential penalty function is associated with the original optimization problem with both inequality and equality constraints. The l₁ exact exponential penalty function method is used to solve nonconvex mathematical programming problems with r-invex functions (with respect to the same function η). The equivalence between sets of optimal solutions of the original mathematical programming problem and of its associated exponential penalized optimization problem is established under suitable r-invexity assumption. Also lower bounds on the penalty parameter are given, for which above these values, this result is true.     

Power Penalty Method for a Linear Complementarity Problem Arising from American Option Valuation

In this paper, we present a power penalty function approach to the linear complementarity problem arising from pricing American options. The problem is first reformulated as a variational inequality problem; the resulting variational inequality problem is then transformed into a nonlinear parabolic partial differential equation (PDE) by adding a power penalty term. It is shown that the solution to the penalized equation converges to that of the variational inequality problem with an arbitrary order. This arbitrary-order convergence rate allows us to achieve the required accuracy of the solution with a small penalty parameter. A numerical scheme for solving the penalized nonlinear PDE is also proposed. Numerical results are given to illustrate the theoretical findings and to show the effectiveness and usefulness of the method. This work was partially supported by a research grant from the University of Western Australia and the Research Grant Council of Hong Kong, Grants PolyU BQ475 and PolyU BQ493.     

The penalized profile sampler

The penalized profile sampler for semiparametric inference is an extension of the profile sampler method [B.L. Lee, M.R. Kosorok, J.P. Fine, The profile sampler, Journal of the American Statistical Association 100 (2005) 960-969] obtained by profiling a penalized log-likelihood. The idea is to base inference on the posterior distribution obtained by multiplying a profiled penalized log-likelihood by a prior for the parametric component, where the profiling and penalization are applied to the nuisance parameter. Because the prior is not applied to the full likelihood, the method is not strictly Bayesian. A benefit of this approximately Bayesian method is that it circumvents the need to put a prior on the possibly infinite-dimensional nuisance components of the model. We investigate the first and second order frequentist performance of the penalized profile sampler, and demonstrate that the accuracy of the procedure can be adjusted by the size of the assigned smoothing parameter. The theoretical validity of the procedure is illustrated for two examples: a partly linear model with normal error for current status data and a semiparametric logistic regression model. Simulation studies are used to verify the theoretical results.     

The Exactness Property of the Vector Exact l1 Penalty Function Method in Nondifferentiable Invex Multiobjective Programming

In this article, the vector exact l₁ penalty function method used for solving nonconvex nondifferentiable multiobjective programming problems is analyzed. In this method, the vector penalized optimization problem with the vector exact l₁ penalty function is defined. Conditions are given guaranteeing the equivalence of the sets of (weak) Pareto optimal solutions of the considered nondifferentiable multiobjective programming problem and of the associated vector penalized optimization problem with the vector exact l₁ penalty function. This equivalence is established for nondifferentiable invex vector optimization problems. Some examples of vector optimization problems are presented to illustrate the results established in the article.