On attraction of linearly constrained Lagrangian methods and of stabilized and quasi-Newton SQP methods to critical multipliers

It has been previously demonstrated that in the case when a Lagrange multiplier associated to a given solution is not unique, Newton iterations [e.g., those of sequential quadratic programming (SQP)] have a tendency to converge to special multipliers, called critical multipliers (when such critical multipliers exist). This fact is of importance because critical multipliers violate the second-order sufficient optimality conditions, and this was shown to be the reason for slow convergence typically observed for problems with degenerate constraints (convergence to noncritical multipliers results in superlinear rate despite degeneracy). Some theoretical and numerical validation of this phenomenon can be found in Izmailov and Solodov (Comput Optim Appl 42:231–264, 2009; Math Program 117:271–304, 2009). However, previous studies concerned the basic forms of Newton iterations. The question remained whether the attraction phenomenon still persists for relevant modifications, as well as in professional implementations. In this paper, we answer this question in the affirmative by presenting numerical results for the well known MINOS and SNOPT software packages applied to a collection of degenerate problems. We also extend previous theoretical considerations to the linearly constrained Lagrangian methods and to the quasi-Newton SQP, on which MINOS and SNOPT are based. Experiments also show that in the stabilized version of SQP the attraction phenomenon still exists but appears less persistent.     

Stabilized sequential quadratic programming for optimization and a stabilized Newton-type method for variational problems

The stabilized version of the sequential quadratic programming algorithm (sSQP) had been developed in order to achieve fast convergence despite possible degeneracy of constraints of optimization problems, when the Lagrange multipliers associated to a solution are not unique. Superlinear convergence of sSQP had been previously established under the strong second-order sufficient condition for optimality (without any constraint qualification assumptions). We prove a stronger superlinear convergence result than the above, assuming the usual second-order sufficient condition only. In addition, our analysis is carried out in the more general setting of variational problems, for which we introduce a natural extension of sSQP techniques. In the process, we also obtain a new error bound for Karush–Kuhn–Tucker systems for variational problems that holds under an appropriate second-order condition.     

On attraction of Newton-type iterates to multipliers violating second-order sufficiency conditions

Assuming that the primal part of the sequence generated by a Newton-type (e.g., SQP) method applied to an equality-constrained problem converges to a solution where the constraints are degenerate, we investigate whether the dual part of the sequence is attracted by those Lagrange multipliers which satisfy second-order sufficient condition (SOSC) for optimality, or by those multipliers which violate it. This question is relevant at least for two reasons: one is speed of convergence of standard methods; the other is applicability of some recently proposed approaches for handling degenerate constraints. We show that for the class of damped Newton methods, convergence of the dual sequence to multipliers satisfying SOSC is unlikely to occur. We support our findings by numerical experiments. We also suggest a simple auxiliary procedure for computing multiplier estimates, which does not have this undesirable property. Finally, some consequences for the case of mixed equality and inequality constraints are discussed.     

Strong calmness of perturbed KKT system for a class of conic programming with degenerate solutions

ABSTRACT

This paper is concerned with the strong calmness of the KKT solution mapping for a class of canonically perturbed conic programming, which plays a central role in achieving fast convergence under situations when the Lagrange multiplier associated to a solution of these conic optimization problems is not unique. We show that the strong calmness of the KKT solution mapping is equivalent to a local error bound for the solutions to the perturbed KKT system, and is also equivalent to the pseudo-isolated calmness of the stationary point mapping along with the calmness of the multiplier set mapping at the corresponding reference point. Sufficient conditions are also provided for the strong calmness by establishing the pseudo-isolated calmness of the stationary point mapping in terms of the noncriticality of the associated multiplier, and the calmness of the multiplier set mapping in terms of a relative interior condition for the multiplier set. These results cover and extend the existing ones in Hager and Gowda [Stability in the presence of degeneracy and error estimation. Math Program. 1999;85:181–192]; Izmailov and Solodov [Stabilized SQP revisited. Math Program. 2012;133:93–120] for nonlinear programming and in Cui et al. [On the asymptotic superlinear convergence of the augmented Lagrangian method for semidefinite programming with multiple solutions. 2016, arXiv: 1610.00875v1]; Zhang and Zhang [Critical multipliers in semidefinite programming. 2018, arXiv: 1801.02218v1] for semidefinite programming.     

Generalized convexity for non-regular optimization problems with conic constraints

In non-regular problems the classical optimality conditions are totally inapplicable. Meaningful results were obtained for problems with conic constraints by Izmailov and Solodov (SIAM J Control Optim 40(4):1280–1295, 2001). They are based on the so-called 2-regularity condition of the constraints at a feasible point. It is well known that generalized convexity notions play a very important role in optimization for establishing optimality conditions. In this paper we give the concept of Karush–Kuhn–Tucker point to rewrite the necessary optimality condition given in Izmailov and Solodov (SIAM J Control Optim 40(4):1280–1295, 2001) and the appropriate generalized convexity notions to show that the optimality condition is both necessary and sufficient to characterize optimal solutions set for non-regular problems with conic constraints. The results that exist in the literature up to now, even for the regular case, are particular instances of the ones presented here.     

On the attraction of Newton’s method to critical lagrange multipliers

The attraction of dual trajectories of Newton’s method for the Lagrange system to critical Lagrange multipliers is analyzed. This stable effect, which has been confirmed by numerical practice, leads to the Newton-Lagrange method losing its superlinear convergence when applied to problems with irregular constraints. At the same time, available theoretical results are of “negative” character; i.e., they show that convergence to a noncritical multiplier is not possible or unlikely. In the case of a purely quadratic problem with a single constraint, a “positive” result is proved for the first time demonstrating that the critical multipliers are attractors for the dual trajectories. Additionally, the influence exerted by the attraction to critical multipliers on the convergence rate of direct and dual trajectories is characterized.     

An Extension of Yuan’s Lemma and Its Applications in Optimization

We prove an extension of Yuan’s lemma to more than two matrices, as long as the set of matrices has rank at most 2. This is used to generalize the main result of Baccari and Trad (SIAM J Optim 15(2):394–408, 2005), where the classical necessary second-order optimality condition is proved, under the assumption that the set of Lagrange multipliers is a bounded line segment. We prove the result under the more general assumption that the Hessian of the Lagrangian, evaluated at the vertices of the Lagrange multiplier set, is a matrix set with at most rank 2. We apply the results to prove the classical second-order optimality condition to problems with quadratic constraints and without constant rank of the Jacobian matrix.     

The p-Factor-Lagrange Methods for Degenerate Nonlinear Programming

The paper presents a new approach to solving nonlinear programming (NLP) problems for which the strict complementarity condition (SCC), a constraint qualification (CQ), and a second-order sufficient condition (SOSC) for optimality are not necessarily satisfied at a solution. Our approach is based on the construction of p-regularity and on reformulating the inequality constraints as equalities. Namely, by introducing the slack variables, we get the equality constrained problem, for which the Lagrange optimality system is singular at the solution of the NLP problem in the case of the violation of the CQs, SCC and/or SOSC. To overcome the difficulty of singularity, we propose the p-factor method for solving the Lagrange system. The method has a superlinear rate of convergence under a mild assumption. We show that our assumption is always satisfied under a standard second-order sufficient condition (SOSC) for optimality. At the same time, we give examples of the problems where the SOSC does not hold, but our assumption is satisfied. Moreover, no estimation of the set of active constraints is required. The proposed approach can be applied to a variety of problems.     

Second-order conditions in stability analysis for state constrained optimal control

The paper presents an outline of the stability results, for state-constrained optimal control problems, recently obtained in Malanowski (Appl. Math. Optim. 55, 255–271, 2007), Malanowski (Optimization, to be published), Malanowski (SIAM J. Optim., to be published). The pricipal novelty of the results is a weakening of the second-order sufficient optimality conditions, under which the solutions and the Lagrange multipliers are locally Lipschitz continuous functions of the parameter. The conditions are weakened by taking into account strongly active state constraints.     

A special complementarity function revisited

ABSTRACT

Recently, a local framework of Newton-type methods for constrained systems of equations has been developed. Applied to the solution of Karush–Kuhn–Tucker (KKT) systems, the framework enables local quadratic convergence under conditions that allow nonisolated and degenerate KKT points. This result is based on a reformulation of the KKT conditions as a constrained piecewise smooth system of equations. It is an open question whether a comparable result can be achieved for other (not piecewise smooth) reformulations. It will be shown that this is possible if the KKT system is reformulated by means of the Fischer–Burmeister complementarity function under conditions that allow degenerate KKT points and nonisolated Lagrange multipliers. To this end, novel constrained Levenberg–Marquardt subproblems are introduced. They allow significantly longer steps for updating the multipliers. Based on this, a convergence rate of at least 1.5 is shown.     

Accelerating convergence of a globalized sequential quadratic programming method to critical Lagrange multipliers

This paper concerns the issue of asymptotic acceptance of the true Hessian and the full step by the sequential quadratic programming algorithm for equality-constrained optimization problems. In order to enforce global convergence, the algorithm is equipped with a standard Armijo linesearch procedure for a nonsmooth exact penalty function. The specificity of considerations here is that the standard assumptions for local superlinear convergence of the method may be violated. The analysis focuses on the case when there exist critical Lagrange multipliers, and does not require regularity assumptions on the constraints or satisfaction of second-order sufficient optimality conditions. The results provide a basis for application of known acceleration techniques, such as extrapolation, and allow the formulation of algorithms that can outperform the standard SQP with BFGS approximations of the Hessian on problems with degenerate constraints. This claim is confirmed by some numerical experiments.

     

无罚函数和滤子的一个新的QP-free方法

通过构造一个等价于原约束问题一阶KKT条件的非光滑方程组, 提出一类新的QP-free方法. 在迭代中采用了无罚函数和无滤子线搜索方法, 在此基础上, 通过牛顿-拟牛顿迭代得到满足KKT最优条件的解, 并证明该算法是可实现、具有全局收敛性. 另外, 在较弱条件下可以证明该方法具有超线性收敛性.     

Lagrange Multiplier Rules for Weak Approximate Pareto Solutions of Constrained Vector Optimization Problems in Hilbert Spaces

In the Hilbert space case, in terms of proximal normal cone and proximal coderivative, we establish a Lagrange multiplier rule for weak approximate Pareto solutions of constrained vector optimization problems. In this case, our Lagrange multiplier rule improves the main result on vector optimization in Zheng and Ng (SIAM J. Optim. 21: 886–911, 2011). We also introduce a notion of a fuzzy proximal Lagrange point and prove that each Pareto (or weak Pareto) solution is a fuzzy proximal Lagrange point.     

3-分片线性NCP函数的滤子QP-free算法

本文定义一个3-分片线性的NCP函数,并对非线性约束优化问题,提出了带有这分片NCP函数的QP-free非可行域算法.根据优化问题的一阶KKT条件,利用乘子和NCP函数,得到非光滑方程,本文给出一个非光滑方程的迭代算法.这算法包含原始-对偶变量,在局部意义下,可看成关于一阶KKT最优条件的的扰动拟牛顿迭代算法.在线性搜索时,这算法采用滤子方法.本文给出的算法是可实现的并具有全局收敛性,且在适当假设下具有超线性收敛性.     

新的滤子方法

本文定义了一种新的滤子方法,并提出了求解光滑不等式约束最优化问题的滤子QP-free非可行域方法.通过乘子和分片线性非线性互补函数,构造一个等价于原约束问题一阶KKT条件的非光滑方程组.在此基础上,通过牛顿-拟牛顿迭代得到满足KKT最优条件的解,在迭代中采用了滤子线搜索方法,证明了该算法是可实现,并具有全局收敛性.另外,在较弱条件下可以证明该方法具有超线性收敛性.     

无罚函数和滤子的QP-free非可行域方法

提出了求解光滑不等式约束最优化问题的无罚函数和无滤子QP-free非可行域方法. 通过乘子和非线性互补函数, 构造一个等价于原约束问题一阶KKT条件的非光滑方程组. 在此基础上, 通过牛顿-拟牛顿迭代得到满足KKT最优性条件的解, 在迭代中采用了无罚函数和无滤子线搜索方法, 并证明该算法是可实现,具有全局收敛性. 另外, 在较弱条件下可以证明该方法具有超线性收敛性.     

Quasi-Multiplier Rules for Quasidifferentiable Extremum Problems

The purpose of the present article is to contribute to clarify the role of the Lagrange multipliers within the theory of the first order necessary optimality conditions for nonsmooth constrained optimization, when the directional derivatives of functions involved in the extremum problems are not sublinear. This task is accomplished in the particular case of quasidifferentiable problems with side constraints. In such setting, making use of the image-space approach, it is possible to establish a generalized (nonlinear) separation result by means of which a new Lagrange principle is obtained. According to this principle, which seems to fit better quasidifferentiable extremum problems than the classic one, the concept of linear multiplier is to be replaced with that of quasi-multiplier, a sublinear and continuous functional whose existence can be guaranteed under mild assumptions, even when classic multipliers fail to exist. Such as extension allows to formulate in terms of Lagrange function the known optimality necessary condition for unconstrained quasidifferentiable optimization expressed in form of quasidifferential inclusion. Along with this, other multiplier rules are established.     

分片线性NCP函数滤子QP-free算法

本文定义了分片线性NCP函数,并对非线性约束优化问题,提出了带有这分片NCP函数的QP-free非可行域算法.利用优化问题的一阶KKT条件,乘子和NCP函数,得到对应的非光滑方程组.本文给出解这非光滑方程组算法,它包含原始-对偶变量,在局部意义下,可看成关扰动牛顿-拟牛顿迭代算法.在线性搜索时,这算法采用滤子方法.本文给出的算法是可实现的并具有全局收敛性,在适当假设下算法具有超线性收敛性.     

Erratum to: An abstract model for branching and its application to mixed integer programming

Stabilized sequential quadratic programming (sSQP) methods for nonlinear optimization generate a sequence of iterates with fast local convergence regardless of whether or not the active-constraint gradients are linearly dependent. This paper concerns the local convergence analysis of an sSQP method that uses a line search with a primal-dual augmented Lagrangian merit function to enforce global convergence. The method is provably well-defined and is based on solving a strictly convex quadratic programming subproblem at each iteration. It is shown that the method has superlinear local convergence under assumptions that are no stronger than those required by conventional stabilized SQP methods. The fast local convergence is obtained by allowing a small relaxation of the optimality conditions for the quadratic programming subproblem in the neighborhood of a solution. In the limit, the line search selects the unit step length, which implies that the method does not suffer from the Maratos effect. The analysis indicates that the method has the same strong first- and second-order global convergence properties that have been established for augmented Lagrangian methods, yet is able to transition seamlessly to sSQP with fast local convergence in the neighborhood of a solution. Numerical results on some degenerate problems are reported.     

A modified Solodov-Svaiter method for solving nonmonotone variational inequality problems

Numerical Algorithms - In a very interesting paper (SIAM J. Control Optim. 37(3): 765–776, 1999), Solodov and Svaiter introduced an effective projection algorithm with linesearch for finding...