Variable Selection via A Combination of the L0 and L1 Penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L₁ penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L₀ penalty is attractive for variable selection because it directly penalizes the number of non zero coefficients. However, the optimization involved is discontinuous and non convex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L₀ and L₁ penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L₀ and L₁ penalties in terms of variable selection while maintaining good prediction accuracy.     

The Exactness Property of the Vector Exact l1 Penalty Function Method in Nondifferentiable Invex Multiobjective Programming

In this article, the vector exact l₁ penalty function method used for solving nonconvex nondifferentiable multiobjective programming problems is analyzed. In this method, the vector penalized optimization problem with the vector exact l₁ penalty function is defined. Conditions are given guaranteeing the equivalence of the sets of (weak) Pareto optimal solutions of the considered nondifferentiable multiobjective programming problem and of the associated vector penalized optimization problem with the vector exact l₁ penalty function. This equivalence is established for nondifferentiable invex vector optimization problems. Some examples of vector optimization problems are presented to illustrate the results established in the article.     

Adaptive estimation in circular functional linear models

We consider the problem of estimating the slope parameter in circular functional linear regression, where scalar responses Y ₁, ..., Y _n are modeled in dependence of 1-periodic, second order stationary random functions X ₁, ...,X _n . We consider an orthogonal series estimator of the slope function β, by replacing the first m theoretical coefficients of its development in the trigonometric basis by adequate estimators. We propose a model selection procedure for m in a set of admissible values, by defining a contrast function minimized by our estimator and a theoretical penalty function; this first step assumes the degree of ill-posedness to be known. Then we generalize the procedure to a random set of admissible m’s and a random penalty function. The resulting estimator is completely data driven and reaches automatically what is known to be the optimal minimax rate of convergence, in terms of a general weighted L ²-risk. This means that we provide adaptive estimators of both β and its derivatives.     

SR-HARDI: Spatially Regularizing High Angular Resolution Diffusion Imaging

High angular resolution diffusion imaging (HARDI) has recently been of great interest in mapping the orientation of intravoxel crossing fibers, and such orientation information allows one to infer the connectivity patterns prevalent among different brain regions and possible changes in such connectivity over time for various neurodegenerative and neuropsychiatric diseases. The aim of this article is to propose a penalized multiscale adaptive regression model (PMARM) framework to spatially and adaptively infer the orientation distribution function (ODF) of water diffusion in regions with complex fiber configurations. In PMARM, we reformulate the HARDI imaging reconstruction as a weighted regularized least-square regression (WRLSR) problem. Similarity and distance weights are introduced to account for spatial smoothness of HARDI, while preserving the unknown discontinuities (e.g., edges between white matter and gray matter) of HARDI. The L₁ penalty function is introduced to ensure the sparse solutions of ODFs, while a scaled L₁ weighted estimator is calculated to correct the bias introduced by the L₁ penalty at each voxel. In PMARM, we integrate the multiscale adaptive regression models, the propagation-separation method, and Lasso (least absolute shrinkage and selection operator) to adaptively estimate ODFs across voxels. Experimental results indicate that PMARM can reduce the angle detection errors on fiber crossing area and provide more accurate reconstruction than standard voxel-wise methods. Supplementary materials for this article are available online.     

An exact penalty function algorithm for semi-infinite programmes

An algorithm for semi-inifinite programming using sequential quadratic programming techniques together with anL exact penalty function is presented, and global convergence is shown. An important feature of the convergence proof is that it does not require an implicit function theorem to be applicable to the semi-infinite constraints; a much weaker assumption concerning the finiteness of the number of global maximizers of each semi-infinite constraint is sufficient. In contrast to proofs based on an implicit function theorem, this result is also valid for a large class ofC ¹ problems.     

New formulations of the multiple sequence alignment problem

A well known formulation of the multiple sequence alignment (MSA) problem is the maximum weight trace (MWT), a 0–1 linear programming problem. In this paper, we propose a new integer quadratic programming formulation of the MSA. The number of constraints and variables in the problem are only of the order of kL ², where, k is the number of sequences and L is the total length of the sequences, that is, L = ?_i=1^kl_i{L= \sum_{i=1}^{k}l_{i}} , where l _i is the length of sequence i. Based on this formulation we introduce an equivalent linear constrained 0–1 quadratic programming problem. We also propose a 0–1 linear programming formulation of the MWT problem, with polynomially many constraints. Our formulation provides the first direct compact formulation that ensures that the critical circuit inequalities (which are exponentially many) are all met.     

Numerical experiments in semi-infinite programming

A quasi-Newton algorithm for semi-infinite programming using an L exact penalty function is described, and numerical results are presented. Comparisons with three Newton algorithms and one other quasi-Newton algorithm show that the algorithm is very promising in practice.     

A new exact exponential penalty function method and nonconvex mathematical programming

A new exact penalty function method, called the l₁ exact exponential penalty function method, is introduced. In this approach, the so-called the exponential penalized optimization problem with the l₁ exact exponential penalty function is associated with the original optimization problem with both inequality and equality constraints. The l₁ exact exponential penalty function method is used to solve nonconvex mathematical programming problems with r-invex functions (with respect to the same function η). The equivalence between sets of optimal solutions of the original mathematical programming problem and of its associated exponential penalized optimization problem is established under suitable r-invexity assumption. Also lower bounds on the penalty parameter are given, for which above these values, this result is true.     

Variational Analysis on Local Sharp Minima via Exact Penalization

In this paper we study local sharp minima of the nonlinear programming problem via exact penalization. Utilizing generalized differentiation tools in variational analysis such as subderivatives and regular subdifferentials, we obtain some primal and dual characterizations for a penalty function associated with the nonlinear programming problem to have a local sharp minimum. These general results are then applied to the ? _p penalty function with 0 ≤ p ≤ 1. In particular, we present primal and dual equivalent conditions in terms of the original data of the nonlinear programming problem, which guarantee that the ? _p penalty function has a local sharp minimum with a finite penalty parameter in the case of \(p\in (\frac {1}{2}, 1]\) and \(p=\frac {1}{2}\) respectively. By assuming the Guignard constraint qualification (resp. the generalized Guignard constraint qualification), we also show that a local sharp minimum of the nonlinear programming problem can be an exact local sharp minimum of the ? _p penalty function with p ∈ [0, 1] (resp. \(p\in [0, \frac {1}{2}]\)). Finally, we give some formulas for calculating the smallest penalty parameter for a penalty function to have a local sharp minimum.     

An exact lower order penalty function and its smoothing in nonlinear programming

In this article, we consider a lower order penalty function and its ε-smoothing for an inequality constrained nonlinear programming problem. It is shown that any strict local minimum satisfying the second-order sufficiency condition for the original problem is a strict local minimum of the lower order penalty function with any positive penalty parameter. By using an ε-smoothing approximation to the lower order penalty function, we get a modified smooth global exact penalty function under mild assumptions.     

The convergence of the best discrete linear Lp approximation as p → 1

It is well known that the best discrete linear L_p approximation converges to a special best Chebyshev approximation as p → ∞. In this paper it is shown that the corresponding result for the case p → 1 is also true. Furthermore, the special best L₁ approximation obtained as the limit is characterized as the unique solution of a nonlinear programming problem on the set of all L₁ solutions.     

A Path Algorithm for the Fused Lasso Signal Approximator

The Lasso is a very well-known penalized regression model, which adds an L₁ penalty with parameter λ₁ on the coefficients to the squared error loss function. The Fused Lasso extends this model by also putting an L₁ penalty with parameter λ₂ on the difference of neighboring coefficients, assuming there is a natural ordering. In this article, we develop a path algorithm for solving the Fused Lasso Signal Approximator that computes the solutions for all values of λ₁ and λ₂. We also present an approximate algorithm that has considerable speed advantages for a moderate trade-off in accuracy. In the Online Supplement for this article, we provide proofs and further details for the methods developed in the article.     

Multigrid methods for the symmetric interior penalty method on graded meshes

The symmetric interior penalty (SIP) method on graded meshes and its fast solution by multigrid methods are studied in this paper. We obtain quasi‐optimal error estimates in both the energy norm and the L₂ norm for the SIP method, and prove uniform convergence of the W‐cycle multigrid algorithm for the resulting discrete problem. The performance of these methods is illustrated by numerical results. Copyright © 2009 John Wiley & Sons, Ltd.     

Dual existence theorems for a special class of linear programming problems in reflexive Banach lattices

In this work the existence of dual optimal solutions for a special class of linear programming problems in a reflexive Banach space is investigated. Then these statements are applied to linear optimization problems with Noethebian operator-constraints. Finally, a maximal condition for an optimal control problem with Noehebiari operator: constraints its Derived in L _∞[0,T].

     

Exact L 2-norm plane separation

We consider the problem of separating two sets of points in an n-dimensional real space with a (hyper)plane that minimizes the sum of L _p -norm distances to the plane of points lying on the wrong side of it. Despite recent progress, practical techniques for the exact solution of cases other than the L ₁ and L _∞-norm were unavailable. We propose and implement a new approach, based on non-convex quadratic programming, for the exact solution of the L ₂-norm case. We solve in reasonable computing times artificial problems of up to 20000 points (in 6 dimensions) and 13 dimensions (with 2000 points). We also observe that, for difficult real-life instances from the UCI Repository, computation times are substantially reduced by incorporating heuristic results in the exact solution process. Finally, we compare the classification performance of the planes obtained for the L ₁, L ₂ and L _∞ formulations. It appears that, despite the fact that L ₂ formulation is computationally more expensive, it does not give significantly better results than the L ₁ and L _∞ formulations.     

A nonlinear programming algorithm based on non-coercive penalty functions

 We consider first the differentiable nonlinear programming problem and study the asymptotic behavior of methods based on a family of penalty functions that approximate asymptotically the usual exact penalty function. We associate two parameters to these functions: one is used to control the slope and the other controls the deviation from the exact penalty. We propose a method that does not change the slope for feasible iterates and show that for problems satisfying the Mangasarian-Fromovitz constraint qualification all iterates will remain feasible after a finite number of iterations. The same results are obtained for non-smooth convex problems under a Slater qualification condition. Received: September 2000 / Accepted: June 2002 Published online: March 21, 2003 Research partially supported by CAPES, Brazil Research partially supported by CNPq, Brazil, and CONICIT, Venezuela. Mathematics Subject Classification (1991): 20E28, 20G40, 20C20     

One Approach to synthesizing predicate circuits on the basis of generalized variables

The asymptotic behavior of the Shannon’s function L _B(n) is studied for complexity of n-variable predicate implementation with the use of predicate circuits over arbitrary complete basis B. A new definition of the reduced weight of the predicate is introduced, regarding it as a solution of a specific linear programming problem based on a system of predicate’s generalized variables. New, more exact upper elements for L _B(n) in a number of bases are acquired by the means of special decompositions of initial predicates using universal sets of predicates constructed for circuits consisting of bases elements with minimal reduced weight.     

Solving ℓ1 Regularization Problems With Piecewise Linear Losses

This article is concerned with the computational aspect of ?₁ regularization problems with a certain class of piecewise linear loss functions. The problem of computing the ?₁ regularization path for a piecewise linear loss can be formalized as a parametric linear programming problem. We propose an efficient implementation method of the parametric simplex algorithm for such a problem. We also conduct a simulation study to investigate the behavior of the number of “breakpoints” of the regularization path when both the number of observations and the number of explanatory variables vary. Our method is also applicable to the computation of the regularization path for a piecewise linear loss and the blockwise ?_∞ penalty. This article has supplementary material online.     

A comparative study of some reconstruction methods for linear inverse problems

In this note, we compare several reconstruction methods to solve a linear ill-conditioned problem, a finite Haussdorf moment problem. The methods are the usual L₂-regularization method, the linear programming method, and two maxentropic reconstruction methods. The scale seems to lean toward the maxentropic reconstructions methods.