An Alternating Direction Method of Multipliers for MCP-penalized Regression with High-dimensional Data

The minimax concave penalty (MCP) has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection and parameter estimation. In this paper, we develop an efficient alternating direction method of multipliers (ADMM) with continuation algorithm for solving the MCP-penalized least squares problem in high dimensions. Under some mild conditions, we study the convergence properties and the Karush–Kuhn–Tucker (KKT) optimality conditions of the proposed method. A high-dimensional BIC is developed to select the optimal tuning parameters. Simulations and a real data example are presented to illustrate the efficiency and accuracy of the proposed method.     

A Newton Method for Linear Programming

A fast Newton method is proposed for solving linear programs with a very large (10⁶) number of constraints and a moderate (10²) number of variables. Such linear programs occur in data mining and machine learning. The proposed method is based on the apparently overlooked fact that the dual of an asymptotic exterior penalty formulation of a linear program provides an exact least 2-norm solution to the dual of the linear program for finite values of the penalty parameter but not for the primal linear program. Solving the dual problem for a finite value of the penalty parameter yields an exact least 2-norm solution to the dual, but not a primal solution unless the parameter approaches zero. However, the exact least 2-norm solution to the dual problem can be used to generate an accurate primal solution if mn and the primal solution is unique. Utilizing these facts, a fast globally convergent finitely terminating Newton method is proposed. A simple prototype of the method is given in eleven lines of MATLAB code. Encouraging computational results are presented such as the solution of a linear program with two million constraints that could not be solved by CPLEX 6.5 on the same machine.     

Multi-class support vector machine optimized by inter-cluster distance and self-adaptive deferential evolution

Support vector machine (SVM) is a popular tool for machine learning task. It has been successfully applied in many fields, but the parameter optimization for SVM is an ongoing research issue. In this paper, to tune the parameters of SVM, one form of inter-cluster distance in the feature space is calculated for all the SVM classifiers of multi-class problems. Inter-cluster distance in the feature space shows the degree the classes are separated. A larger inter-cluster distance value implies a pair of more separated classes. For each classifier, the optimal kernel parameter which results in the largest inter-cluster distance is found. Then, a new continuous search interval of kernel parameter which covers the optimal kernel parameter of each class pair is determined. Self-adaptive differential evolution algorithm is used to search the optimal parameter combination in the continuous intervals of kernel parameter and penalty parameter. At last, the proposed method is applied to several real word datasets as well as fault diagnosis for rolling element bearings. The results show that it is both effective and computationally efficient for parameter optimization of multi-class SVM.     

Exact penalty functions in nonlinear programming

It is shown that the existence of a strict local minimum satisfying the constraint qualification of [16] or McCormick's [12] second order sufficient optimality condition implies the existence of a class of exact local penalty functions (that is ones with a finite value of the penalty parameter) for a nonlinear programming problem. A lower bound to the penalty parameter is given by a norm of the optimal Lagrange multipliers which is dual to the norm used in the penalty function.Sponsored by the United States Army under Contract No. DAAG29-75-C-0024 and by the National Science Foundation under Grant No. MCS74-20584 A02.     

Global optimality conditions and optimization methods for quadratic assignment problems

In this paper some global optimality conditions for general quadratic {0, 1} programming problems with linear equality constraints are discussed and then some global optimality conditions for quadratic assignment problems (QAP) are presented. A local optimization method for (QAP) is derived according to the necessary global optimality conditions. A global optimization method for (QAP) is presented by combining the sufficient global optimality conditions, the local optimization method and some auxiliary functions. Some numerical examples are given to illustrate the efficiency of the given optimization methods.     

A mathematical analysis for a model arising from public goods games

We study the effects of altruistic behaviors in a public goods game model which describes the competition between the farmers and the exploiters. Corresponding to different parametric regions, we analyze in detail the stability of the equilibrium states and obtain attraction regions for stable equilibria. Then using the upper–lower solution method and monotone iterations, we further show that for a family of wave speeds, there exist traveling wave solutions connecting one of the unstable states to the stable state. This answers a conjecture made by Wakano in [J.Y. Wakano, A mathematical analysis on public goods games in the continuous space, Math. Biosci. 201 (2006) 72–89]. The results indicate that when the penalty for the altruistic behavior is small, the growth rate of the population determines its survival or extinction states in the long run. Furthermore, if the two populations have the same total growth rate, altruism in the competition leads to a wide range of co-existent states. Numerical simulations are also presented to illustrate the theoretical results.     

Production and replacement policies for a deteriorating manufacturing system under random demand and quality

This work investigates the production planning of an unreliable deteriorating manufacturing system under uncertainties. The effect of the deterioration phenomenon on the machine is mainly observed in its availability and the quality of the parts produced, with the rates of failure and defectives increasing with the age of the machine. The option to replace the machine should be considered to mitigate the effect of deterioration in order to ensure long-term satisfaction of demand. The objective of this paper is to find the production rate and the replacement policy that minimize the total discounted cost, which includes inventory, backlog, production, repair and replacement costs, over an infinite planning horizon. We formulate the stochastic control problem in the framework of a semi-Markov decision process to consider the machine's history. The integration of random demand and quality behaviour led us to propose a new modeling approach by developing optimality conditions in terms of a second-order approximation of Hamilton–Jacobi–Bellman (HJB) equations. Numerical methods are used to obtain the optimal control policies. Finally, a numerical example and a sensitivity analysis are presented in order to illustrate and confirm the structure of the optimal solution obtained.     

A Unified Approach to the Global Exactness of Penalty and Augmented Lagrangian Functions I: Parametric Exactness

In this two-part study, we develop a unified approach to the analysis of the global exactness of various penalty and augmented Lagrangian functions for constrained optimization problems in finite-dimensional spaces. This approach allows one to verify in a simple and straightforward manner whether a given penalty/augmented Lagrangian function is exact, i.e., whether the problem of unconstrained minimization of this function is equivalent (in some sense) to the original constrained problem, provided the penalty parameter is sufficiently large. Our approach is based on the so-called localization principle that reduces the study of global exactness to a local analysis of a chosen merit function near globally optimal solutions. In turn, such local analysis can be performed with the use of optimality conditions and constraint qualifications. In the first paper, we introduce the concept of global parametric exactness and derive the localization principle in the parametric form. With the use of this version of the localization principle, we recover existing simple, necessary, and sufficient conditions for the global exactness of linear penalty functions and for the existence of augmented Lagrange multipliers of Rockafellar–Wets’ augmented Lagrangian. We also present completely new necessary and sufficient conditions for the global exactness of general nonlinear penalty functions and for the global exactness of a continuously differentiable penalty function for nonlinear second-order cone programming problems. We briefly discuss how one can construct a continuously differentiable exact penalty function for nonlinear semidefinite programming problems as well.     

A new look at the Lagrange method for continuous-time stochastic optimization

We formulate a Lagrange method for continuous-time stochastic optimization in an appropriate normed space by using a proper stochastic process as the Lagrange multiplier.The obtained optimality conditions are applied to different types of problems.Some examples selected from control theory and economic theory are studied to test and illustrate the potential applications of the method.     

Exact penalty function algorithm with simple updating of the penalty parameter

A new globally convergent algorithm for minimizing an objective function subject to equality and inequality constraints is presented. The algorithm determines a search direction by solving a quadratic programming subproblem, which always has an optimal solution, and uses an exact penalty function to compute the steplength along this direction through an Armijo-type scheme. The special structure of the quadratic subproblem is exploited to construct a new and simple method for updating the penalty parameter. This method may increase or reduce the value of the penalty parameter depending on some easily performed tests. A new method for updating the Hessian of the Lagrangian is presented, and a Q-superlinear rate of convergence is established.This work was supported in part by the British Council and the Conselho Nacional de Desenvolvimento Cientifico & Tecnologico/CNPq, Rio de Janeiro, Brazil.The authors are very grateful to Mr. Lam Yeung for his invaluable assistance in computing the results and to a reviewer for constructive advice.     

Optimal multigrid preconditioned semi‐monotonic augmented Lagrangians applied to the Stokes problem

We propose an optimal computational complexity algorithm for the solution of quadratic programming problems with equality constraints arising from partial differential equations. The algorithm combines a variant of the semi‐monotonic augmented Lagrangian (SMALE) method with adaptive precision control and a multigrid preconditioning for the Hessian of the cost function and for the inner product on the space of Lagrange variables. The update rule for penalty parameter acts as preconditioning of constraints. The optimality of the algorithm is theoretically proven and confirmed by numerical experiments for the two‐dimensional Stokes problem. Copyright © 2007 John Wiley & Sons, Ltd.     

Inexact-Restoration Algorithm for Constrained Optimization1

We introduce a new model algorithm for solving nonlinear programming problems. No slack variables are introduced for dealing with inequality constraints. Each iteration of the method proceeds in two phases. In the first phase, feasibility of the current iterate is improved; in second phase, the objective function value is reduced in an approximate feasible set. The point that results from the second phase is compared with the current point using a nonsmooth merit function that combines feasibility and optimality. This merit function includes a penalty parameter that changes between consecutive iterations. A suitable updating procedure for this penalty parameter is included by means of which it can be increased or decreased along consecutive iterations. The conditions for feasibility improvement at the first phase and for optimality improvement at the second phase are mild, and large-scale implementation of the resulting method is possible. We prove that, under suitable conditions, which do not include regularity or existence of second derivatives, all the limit points of an infinite sequence generated by the algorithm are feasible, and that a suitable optimality measure can be made as small as desired. The algorithm is implemented and tested against the LANCELOT algorithm using a set of hard-spheres problems.     

A Power Penalty Approach to Numerical Solutions of Two-Asset American Options

This paper aims to develop a power penalty method for a linear parabolic variational inequality (Ⅵ) in two spatial dimensions governing the two-asset Ameri-can option valuation. This method yields a two-dimensional nonlinear parabolic PDE containing a power penalty term with penalty constant λ>1 and a power parameter k>0. We show that the nonlinear PDE is uniquely solvable and the solution of the PDE converges to that of the VI at the rate of order (λ<-k/2>). A fitted finite volume method is designed to solve the nonlinear PDE, and some numerical experiments are performed to illustrate the usefulness of this method.     

求解定常不可压Stokes方程的两层罚函数方法

借助于两套有限元网格空间提出了一种求解定常不可压Stokes方程的两层罚函数方法.该方法只需要求解粗网格空间上的Stokes方程和细网格空间上的两个易于求解的罚参数方程（离散后的线性方程组具有相同的对称正定系数矩阵）.收敛性分析表明粗网格空间相对于细网格空间可以选择很小，并且罚参数的选取只与粗网格步长和问题的正则性有关.因此罚参数不必选择很小仍能够得到最优解.最后通过数值算例验证了上述理论结果，并且数值对比可知两层罚函数方法对于求解定常不可压Stokes方程具有很好的效果.     

On the average cost optimality equation and the structure of optimal policies for partially observable Markov decision processes

We consider partially observable Markov decision processes with finite or countably infinite (core) state and observation spaces and finite action set. Following a standard approach, an equivalent completely observed problem is formulated, with the same finite action set but with anuncountable state space, namely the space of probability distributions on the original core state space. By developing a suitable theoretical framework, it is shown that some characteristics induced in the original problem due to the countability of the spaces involved are reflected onto the equivalent problem. Sufficient conditions are then derived for solutions to the average cost optimality equation to exist. We illustrate these results in the context of machine replacement problems. Structural properties for average cost optimal policies are obtained for a two state replacement problem; these are similar to results available for discount optimal policies. The set of assumptions used compares favorably to others currently available.This research was supported in part by the Advanced Technology Program of the State of Texas, in part by the Air Force Office of Scientific Research under Grant AFOSR-86-0029, in part by the National Science Foundation under Grant ECS-8617860, and in part by the Air Force Office of Scientific Research (AFSC) under Contract F49620-89-C-0044.     

Maintenance optimal control three-machine replacement model under technological breakthrough expectations

This paper addresses a finite-horizon profit maximization three-machine replacement problem. More precisely, a model is formulated allowing for preventive maintenance to slow down machine quality and profit reduction caused by obsolescence, to determine the timing of replacing an existing machine by another available machine with improved technology. This decision is considered under uncertainty regarding the introduction time of a machine with a not-yetachieved technology. Given an exponential probability distribution function of the introduction time, the optimality of a bang-bang nonincreasing preventive maintenance control is shown.Moreover, subproblems maximizing the expected discounted profit are analyzed. Closed-form solutions are provided to compare machines of different technologies and to derive an analytical sensitivity analysis concerned with many issues related to the problem. The results are not necessarily intuitive and simple. For example, different relationships between the planning horizon and the preventive maintenance switching time are presented for the three-machine problem versus the single-machine problem.The focus of this paper is on the formulation and the analytical analysis of the problem rather than on its computational aspects.     

Global Optimality Conditions and Optimization Methods for Quadratic Knapsack Problems

The quadratic knapsack problem (QKP) maximizes a quadratic objective function subject to a binary and linear capacity constraint. Due to its simple structure and challenging difficulty, it has been studied intensively during the last two decades. This paper first presents some global optimality conditions for (QKP), which include necessary conditions and sufficient conditions. Then a local optimization method for (QKP) is developed using the necessary global optimality condition. Finally a global optimization method for (QKP) is proposed based on the sufficient global optimality condition, the local optimization method and an auxiliary function. Several numerical examples are given to illustrate the efficiency of the presented optimization methods.     

求解多主多从线性斯坦伯格问题的新方法

在本文中,我们研究斯坦伯格问题,发展了罚函数法。     

Analysis of optimal preconditioners for CutFEM

In this article, we consider a class of unfitted finite element methods for scalar elliptic problems. These so-called CutFEM methods use standard finite element spaces on a fixed unfitted triangulation combined with the Nitsche technique and a ghost penalty stabilization. As a model problem we consider the application of such a method to the Poisson interface problem. We introduce and analyze a new class of preconditioners that is based on a subspace decomposition approach. The unfitted finite element space is split into two subspaces, where one subspace is the standard finite element space associated to the background mesh and the second subspace is spanned by all cut basis functions corresponding to nodes on the cut elements. We will show that this splitting is stable, uniformly in the discretization parameter and in the location of the interface in the triangulation. Based on this we introduce an efficient preconditioner that is uniformly spectrally equivalent to the stiffness matrix. Using a similar splitting, it is shown that the same preconditioning approach can also be applied to a fictitious domain CutFEM discretization of the Poisson equation. Results of numerical experiments are included that illustrate optimality of such preconditioners for the Poisson interface problem and the Poisson fictitious domain problem.     

A dual differentiable exact penalty function

A new penalty function is associated with an inequality constrained nonlinear programming problem via its dual. This penalty function is globally differentiable if the functions defining the original problem are twice globally differentiable. In addition, the penalty parameter remains finite. This approach reduces the original problem to a simple problem of maximizing a globally differentiable function on the product space of a Euclidean space and the nonnegative orthant of another Euclidean space. Many efficient algorithms exist for solving this problem. For the case of quadratic programming, the penalty function problem can be solved effectively by successive overrelaxation (SOR) methods which can handle huge problems while preserving sparsity features. Sponsored by the United States Army under Contract No. DAAG 29-80-C-0041. This material is based upon work supported by the National Science Foundation under Grants No. MCS-790166 and ENG-7903881.