Variable Selection via A Combination of the L0 and L1 Penalties

Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L₁ penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L₀ penalty is attractive for variable selection because it directly penalizes the number of non zero coefficients. However, the optimization involved is discontinuous and non convex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L₀ and L₁ penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L₀ and L₁ penalties in terms of variable selection while maintaining good prediction accuracy.     

A Path Algorithm for Constrained Estimation

Many least-square problems involve affine equality and inequality constraints. Although there are a variety of methods for solving such problems, most statisticians find constrained estimation challenging. The current article proposes a new path-following algorithm for quadratic programming that replaces hard constraints by what are called exact penalties. Similar penalties arise in l ₁ regularization in model selection. In the regularization setting, penalties encapsulate prior knowledge, and penalized parameter estimates represent a trade-off between the observed data and the prior knowledge. Classical penalty methods of optimization, such as the quadratic penalty method, solve a sequence of unconstrained problems that put greater and greater stress on meeting the constraints. In the limit as the penalty constant tends to ∞, one recovers the constrained solution. In the exact penalty method, squared penalties are replaced by absolute value penalties, and the solution is recovered for a finite value of the penalty constant. The exact path-following method starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, slides along, and exits from the various constraints. Path following in Lasso penalized regression, in contrast, starts with a large value of the penalty constant and works its way downward. In both settings, inspection of the entire solution path is revealing. Just as with the Lasso and generalized Lasso, it is possible to plot the effective degrees of freedom along the solution path. For a strictly convex quadratic program, the exact penalty algorithm can be framed entirely in terms of the sweep operator of regression analysis. A few well-chosen examples illustrate the mechanics and potential of path following. This article has supplementary materials available online.     

A uniform framework for the combination of penalties in generalized structured models

Penalized estimation has become an established tool for regularization and model selection in regression models. A variety of penalties with specific features are available and effective algorithms for specific penalties have been proposed. But not much is available to fit models with a combination of different penalties. When modeling the rent data of Munich as in our application, various types of predictors call for a combination of a Ridge, a group Lasso and a Lasso-type penalty within one model. We propose to approximate penalties that are (semi-)norms of scalar linear transformations of the coefficient vector in generalized structured models—such that penalties of various kinds can be combined in one model. The approach is very general such that the Lasso, the fused Lasso, the Ridge, the smoothly clipped absolute deviation penalty, the elastic net and many more penalties are embedded. The computation is based on conventional penalized iteratively re-weighted least squares algorithms and hence, easy to implement. New penalties can be incorporated quickly. The approach is extended to penalties with vector based arguments. There are several possibilities to choose the penalty parameter(s). A software implementation is available. Some illustrative examples show promising results.     

带惩罚的容错设施布局问题的近似算法

在带惩罚的容错设施布局问题中, 给定顾客集合、地址集合、以及每个顾客和各个地址之间的连接费用, 这里假设连接费用是可度量的. 每位顾客有各自的服务需求, 每个地址可以开设任意多个设施, 顾客可以被安排连接到某些地址的一些开设的设施上以满足其需求, 也可以被拒绝, 但这时要支付拒绝该顾客所带来的惩罚费用. 目标是确定哪些顾客的服务需求被拒绝并开设一些设施, 将未被拒绝的顾客连接到不同的开设设施上, 使得开设费用、连接费用和惩罚费用总和最小. 给出了带惩罚的容错设施布局问题的线性整数规划及其对偶规划, 进一步, 给出了基于其线性规划和对偶规划舍入的4-近似算法.     

Penalties,rewards, and inspection: provisions for quality in supply chain contracts

Rewards for better quality, penalties for poorer quality, and the type of inspection policy are among the most common quality-related provisions of supply chain contracts. In this paper, we examine the effect of rewards, penalities, and inspection policies on the behaviour of an expected cost minimizing supplier. We assume that the supplier selects a batch size and target quality level in order to meet a buyer's deterministic demand. We show that the reward and/or penalty that motivates a supplier to deliver the buyer's target quality depends upon the inspection policy. We also show that, when sampling inspection is used, penalties and rewards are substitutes for one another in motivating the supplier and that there exists a unique reward/penalty combination at which the buyer's expected cost of quality is zero.     

On-line scheduling of unit time jobs with rejection: minimizing the total completion time

We consider on-line scheduling of unit time jobs on a single machine with job-dependent penalties. The jobs arrive on-line (one by one) and can be either accepted and scheduled, or be rejected at the cost of a penalty. The objective is to minimize the total completion time of the accepted jobs plus the sum of the penalties of the rejected jobs.We give an on-line algorithm for this problem with competitive ratio . Moreover, we prove that there does not exist an on-line algorithm with competitive ratio better than 1.63784.     

A MODIFIED VARIABLE-PENALTY ALTERNATING DIRECTIONS METHOD FOR MONOTONE VARIATIONAL INEQUALITIES

Alternating directions method is one of the approaches for solving linearly constrained separate monotone variational inequalities. Experience on applications has shown that the number of iteration significantly depends on the penalty for the system of linearly constrained equations and therefore the method with variable penalties is advantageous in practice. In this paper, we extend the Kontogiorgis and Meyer method [12] by removing the monotonicity assumption on the variable penalty matrices. Moreover, we introduce a self-adaptive rule that leads the method to be more efficient and insensitive for various initial penalties. Numerical results for a class of Fermat-Weber problems show that the modified method and its self-adaptive technique are proper and necessary in practice.     

A general class of penalty/barrier path-following Newton methods for nonlinear programming

In the present article rather general penalty/barrier-methods (e.g. logarithmic barriers, SUMT, exponential penalties), which define a local continuously differentiable primal and dual path, are analyzed in case of strict local minima of nonlinear problems with inequality as well as equality constraints. In particular, the radius of convergence of Newton's method depending on the penalty/barrier-parameter is estimated. Unlike using self-concordance properties, the convergence bounds are derived by direct estimations of the solutions of the Newton equations. By means of the obtained results parameter selection rules are studied which guarantee the local convergence of the considered penalty/barrier-techniques with only a finite number of Newton steps at each parameter level. Numerical examples illustrate the practical behavior of the proposed class of methods.     

General primal-dual penalty/barrier path-following Newton methods for nonlinear programming

In the present article rather general penalty/barrier-methods are considered, that define a local continuously differentiable primal-dual path. The class of penalty/barrier terms includes most of the usual techniques like logarithmic barriers, SUMT, quadratic loss functions as well as exponential penalties, and the optimization problem which may contain inequality as well as equality constraints. The convergence of the corresponding general primal-dual path-following method is shown for local minima that satisfy strong second-order sufficiency conditions with linear independence constraint qualification (LICQ) and strict complementarity. A basic tool in the analysis of these methods is to estimate the radius of convergence of Newton's method depending on the penalty/barrier-parameter. Without using self-concordance properties convergence bounds are derived by direct estimations of the solutions of the Newton equations. Parameter selection rules are proposed which guarantee the local convergence of the considered penalty/barrier-techniques with only a finite number of Newton steps at each parameter level. Numerical examples illustrate the practical behavior of the proposed class of methods.     

Penalized likelihood regression for generalized linear models with non-quadratic penalties

One of the popular method for fitting a regression function is regularization: minimizing an objective function which enforces a roughness penalty in addition to coherence with the data. This is the case when formulating penalized likelihood regression for exponential families. Most of the smoothing methods employ quadratic penalties, leading to linear estimates, and are in general incapable of recovering discontinuities or other important attributes in the regression function. In contrast, non-linear estimates are generally more accurate. In this paper, we focus on non-parametric penalized likelihood regression methods using splines and a variety of non-quadratic penalties, pointing out common basic principles. We present an asymptotic analysis of convergence rates that justifies the approach. We report on a simulation study including comparisons between our method and some existing ones. We illustrate our approach with an application to Poisson non-parametric regression modeling of frequency counts of reported acquired immune deficiency syndrome (AIDS) cases in the UK.     

带惩罚的动态设施选址问题的近似算法

本文研究带惩罚的动态设施选址问题,在该问题中假设不同时段内设施的开放费用、用户的需求及连接费用可以不相同,而且允许用户的需求不被满足,但是要有惩罚.对此问题我们给出了第-个近似比为1.8526的原始对偶(组合)算法.     

Stochastic programming problems with generalized integrated chance constraints

If the constraints in an optimization problem are dependent on a random parameter, we would like to ensure that they are fulfilled with a high level of reliability. The most natural way is to employ chance constraints. However, the resulting problem is very hard to solve. We propose an alternative formulation of stochastic programs using penalty functions. The expectations of penalties can be left as constraints leading to generalized integrated chance constraints, or incorporated into the objective as a penalty term. We show that the penalty problems are asymptotically equivalent under quite mild conditions. We discuss applications of sample-approximation techniques to the problems with generalized integrated chance constraints and propose rates of convergence for the set of feasible solutions. We will direct our attention to the case when the set of feasible solutions is finite, which can appear in integer programming. The results are then extended to the bounded sets with continuous variables. Additional binary variables are necessary to solve sample-approximated chance-constrained problems, leading to a large mixed-integer non-linear program. On the other hand, the problems with penalties can be solved without adding binary variables; just continuous variables are necessary to model the penalties. The introduced approaches are applied to the blending problem leading to comparably reliable solutions.     

The penalty method for grid matching in mixed finite element methods

In this paper we prove the possibility of the use of the penalty method for grid matching in mixed finite element methods. We consider the Hermann-Johnson scheme for biharmonic equation. The main idea is to construct a perturbed problem with two parameters which play roles of penalties. The perturbed problem is built by the replacement of essential conditions on the interface in the mixed variational statement with natural conditions that contain parameters. The perturbed problem is discretized by the finite element method. We estimate the norm of the difference between a solution of the discrete perturbed problem and a solution of the initial problem; the obtained estimates depend on the step and the penalties. We give recommendations for the choice of penalties depending on the step.     

Scheduling a single machine to minimize earliness penalties subject to the SLK due-date determination method

This paper considers a single machine scheduling problem. There are n jobs to be processed on a single machine. The problem is to minimize total earliness penalties subject to no tardy jobs. The problem is NP-complete if the due-dates are arbitrary. We study the problem when the due-dates are determined by the equal slack (SLK) method. Two special cases of the problem are solved in polynomial time. The first one is the problem with equally weighted monotonous penalty objective function. The second one is the problem with weighted linear penalty objective function.     

An exact penalty function for semi-infinite programming

This paper introduces a global approach to the semi-infinite programming problem that is based upon a generalisation of the ℓ₁ exact penalty function. The advantages are that the ensuing penalty function is exact and the penalties include all violations. The merit function requires integrals for the penalties, which provides a consistent model for the algorithm. The discretization is a result of the approximate quadrature rather than an a priori aspect of the model. This research was partially supported by Natural Sciences and Engineering Research Council of Canada grants A-8639 and A-8442. This paper was typeset using software developed at Bell Laboratories and the University of California at Berkeley.     

Analysis of Current Penalty Schemes for Violations of Antitrust Laws

The main feature of the penalty schemes described in current sentencing guidelines is that the fine is based on the accumulated gains from cartel activities or price-fixing activities for the firm. The regulations suggest modeling the penalty as an increasing function of the accumulated illegal gains from price fixing to the firm, so that the history of the violation is taken into account. We incorporate these features of the penalty scheme into an optimal control model of a profit-maximizing firm under antitrust enforcement. To determine the effect of taking into account the history of the violation, we compare the outcome of this model with a model where the penalty is fixed. The analysis of the latter model implies that complete deterrence can be achieved only at the cost of shutting down the firm. The proportional scheme improves upon the fixed penalty, since it can ensure complete deterrence in the long run, even when penalties are moderate. Phase-diagram analysis shows that, the higher the probability and severity of punishment, the sooner cartel formation is blocked. Further, a sensitivity analysis is provided to show which strategies are most successful in reducing the degree of price fixing. It turns out that, when the penalties are already high, the antitrust policy aiming at a further increase in the severity of punishment is less efficient than the policy that increases the probability of punishment. The authors thank Eric van Damme, Thomas Fent, and an anonymous referee for stimulating discussions and valuable comments.     

Minimal Penalties for Gaussian Model Selection

This paper is mainly devoted to a precise analysis of what kind of penalties should be used in order to perform model selection via the minimization of a penalized least-squares type criterion within some general Gaussian framework including the classical ones. As compared to our previous paper on this topic (Birgé and Massart in J. Eur. Math. Soc. 3, 203–268 (2001)), more elaborate forms of the penalties are given which are shown to be, in some sense, optimal. We indeed provide more precise upper bounds for the risk of the penalized estimators and lower bounds for the penalty terms, showing that the use of smaller penalties may lead to disastrous results. These lower bounds may also be used to design a practical strategy that allows to estimate the penalty from the data when the amount of noise is unknown. We provide an illustration of the method for the problem of estimating a piecewise constant signal in Gaussian noise when neither the number, nor the location of the change points are known.     

带次模惩罚和随机需求的设施选址问题

考虑带次模惩罚和随机需求的设施选址问题,目的是开设设施集合的一个子集,把客户连接到开设的设施上并对没有连接的客户进行惩罚,使得开设费用、连接费用、库存费用、管理费用和惩罚费用之和达到最小. 根据该问题的特殊结构,给出原始对偶3-近似算法. 在算法的第一步,构造了一组对偶可行解;在第二步中构造了对应的一组原始整数可行解,这组原始整数可行解给出了最后开设的设施集合和被惩罚的客户集合. 最后,证明了算法在多项式时间内可以完成,并且算法所给的整数解不会超过最优解的3倍.     

高维线性回归模型中非凸惩罚岭回归

非凸惩罚函数包括SCAD惩罚和MCP惩罚, 这类惩罚函数具有无偏性、连续性和稀疏性等特点,岭回归方法能够很好的克服共线性问题. 本文将非凸惩罚函数和岭回归方法的优势结合起来(简记为 NPR),研究了自变量间存在高相关性问题时NPR估计的Oracle性质. 这里主要研究了参数个数$p_n$ 随样本量$n$ 呈指数阶增长的情况. 同时, 通过模拟研究和实例分析进一步验证了NPR 方法的表现.     

Job scheduling with forbidden setups and two objectives using genetic algorithms and penalties

One of the most important tasks in service and manufacturing systems is how to schedule arriving jobs such that some criteria will be satisfied. Up to now there have been defined a great variety of scheduling problems as well as corresponding models and solution approaches. Most models suffer from such more or less restrictive assumptions like single machine, unique processing times, zero set-up times or a single criterion. On the other hand some classical approaches like linear or dynamic programming are practicable for small-size problems only. Therefore over the past years we can observe an increasing application of heuristic search methods. But scheduling problems with multiple machines, forbidden setups and multiple objectives are scarcely considered. In our paper we apply a Genetic Algorithm to such a problem which was found at a continuous casting plant. Because of the forbidden setups the probability for a random generated schedule to be feasible is nearly zero. To resolve this problem we use three kinds of penalties, a global, a local and a combined approach. For performance investigations of these penalty types we applied our approaches to a real world test instance with 96 jobs, three machines and two objectives. We tested five different penalty levels with 51 independent runs to evaluate the impact of the penalties.