Convergence Analysis of an Inexact Potential Reduction Method for Convex Quadratic Programming

We analyze the convergence of an infeasible inexact potential reduction method for quadratic programming problems. We show that the convergence of this method is achieved if the residual of the KKT system satisfies a bound related to the duality gap. This result suggests stopping criteria for inner iterations that can be used to adapt the accuracy of the computed direction to the quality of the potential reduction iterate in order to achieve computational efficiency. This research was partially supported by the Italian MIUR, Project FIRB—Large Scale Nonlinear Optimization # RBNE01WBBB and Project PRIN—Innovative Problems and Methods in Nonlinear Optimization # 2005017083.     

A special complementarity function revisited

ABSTRACT

Recently, a local framework of Newton-type methods for constrained systems of equations has been developed. Applied to the solution of Karush–Kuhn–Tucker (KKT) systems, the framework enables local quadratic convergence under conditions that allow nonisolated and degenerate KKT points. This result is based on a reformulation of the KKT conditions as a constrained piecewise smooth system of equations. It is an open question whether a comparable result can be achieved for other (not piecewise smooth) reformulations. It will be shown that this is possible if the KKT system is reformulated by means of the Fischer–Burmeister complementarity function under conditions that allow degenerate KKT points and nonisolated Lagrange multipliers. To this end, novel constrained Levenberg–Marquardt subproblems are introduced. They allow significantly longer steps for updating the multipliers. Based on this, a convergence rate of at least 1.5 is shown.     

On the iterative solution of KKT systems in potential reduction software for large-scale quadratic problems

Iterative solvers appear to be very promising in the development of efficient software, based on Interior Point methods, for large-scale nonlinear optimization problems. In this paper we focus on the use of preconditioned iterative techniques to solve the KKT system arising at each iteration of a Potential Reduction method for convex Quadratic Programming. We consider the augmented system approach and analyze the behaviour of the Constraint Preconditioner with the Conjugate Gradient algorithm. Comparisons with a direct solution of the augmented system and with MOSEK show the effectiveness of the iterative approach on large-scale sparse problems. Work partially supported by the Italian MIUR FIRB Project Large Scale Nonlinear Optimization, grant no. RBNE01WBBB.     

On PDE solution in transient optimization of gas networks

Operative planning in gas distribution networks leads to large-scale mixed-integer optimization problems involving a hyperbolic PDE defined on a graph. We consider the NLP obtained under prescribed combinatorial decisions—or as relaxation in a branch-and-bound framework, addressing in particular the KKT systems arising in primal–dual interior methods. We propose a custom solution algorithm using sparse projections locally in time, based on the KKT systems’ structural properties in space as induced by the discretized gas flow equations in combination with the underlying network topology. The numerical efficiency and accuracy of the algorithm are investigated, and detailed computational comparisons with a previously developed control space method and with the multifrontal solver MA27 are provided.     

Stabilized SQP revisited

The stabilized version of the sequential quadratic programming algorithm (sSQP) had been developed in order to achieve superlinear convergence in situations when the Lagrange multipliers associated to a solution are not unique. Within the framework of Fischer (Math Program 94:91–124, 2002), the key to local superlinear convergence of sSQP are the following two properties: upper Lipschitzian behavior of solutions of the Karush-Kuhn-Tucker (KKT) system under canonical perturbations and local solvability of sSQP subproblems with the associated primal-dual step being of the order of the distance from the current iterate to the solution set of the unperturbed KKT system. According to Fernández and Solodov (Math Program 125:47–73, 2010), both of these properties are ensured by the second-order sufficient optimality condition (SOSC) without any constraint qualification assumptions. In this paper, we state precise relationships between the upper Lipschitzian property of solutions of KKT systems, error bounds for KKT systems, the notion of critical Lagrange multipliers (a subclass of multipliers that violate SOSC in a very special way), the second-order necessary condition for optimality, and solvability of sSQP subproblems. Moreover, for the problem with equality constraints only, we prove superlinear convergence of sSQP under the assumption that the dual starting point is close to a noncritical multiplier. Since noncritical multipliers include all those satisfying SOSC but are not limited to them, we believe this gives the first superlinear convergence result for any Newtonian method for constrained optimization under assumptions that do not include any constraint qualifications and are weaker than SOSC. In the general case when inequality constraints are present, we show that such a relaxation of assumptions is not possible. We also consider applying sSQP to the problem where inequality constraints are reformulated into equalities using slack variables, and discuss the assumptions needed for convergence in this approach. We conclude with consequences for local regularization methods proposed in (Izmailov and Solodov SIAM J Optim 16:210–228, 2004; Wright SIAM J. Optim. 15:673–676, 2005). In particular, we show that these methods are still locally superlinearly convergent under the noncritical multiplier assumption, weaker than SOSC employed originally.     

An efficient Lagrangian smoothing heuristic for Max-Cut

Max-Cut is a famous NP-hard problem in combinatorial optimization. In this article, we propose a Lagrangian smoothing algorithm for Max-Cut, where the continuation subproblems are solved by the truncated Frank-Wolfe algorithm. We establish practical stopping criteria and prove that our algorithm finitely terminates at a KKT point, the distance between which and the neighbour optimal solution is also estimated. Additionally, we obtain a new sufficient optimality condition for Max-Cut. Numerical results indicate that our approach outperforms the existing smoothing algorithm in less time.     

Karush-Kuhn-Tucker systems: regularity conditions,error bounds and a class of Newton-type methods

 We consider optimality systems of Karush-Kuhn-Tucker (KKT) type, which arise, for example, as primal-dual conditions characterizing solutions of optimization problems or variational inequalities. In particular, we discuss error bounds and Newton-type methods for such systems. An exhaustive comparison of various regularity conditions which arise in this context is given. We obtain a new error bound under an assumption which we show to be strictly weaker than assumptions previously used for KKT systems, such as quasi-regularity or semistability (equivalently, the R ₀-property). Error bounds are useful, among other things, for identifying active constraints and developing efficient local algorithms. We propose a family of local Newton-type algorithms. This family contains some known active-set Newton methods, as well as some new methods. Regularity conditions required for local superlinear convergence compare favorably with convergence conditions of nonsmooth Newton methods and sequential quadratic programming methods. Received: December 10, 2001 / Accepted: July 28, 2002 Published online: February 14, 2003 Key words. KKT system – regularity – error bound – active constraints – Newton method Mathematics Subject Classification (1991): 90C30, 65K05     

Using Improved Directions of Negative Curvature for the Solution of Bound-Constrained Nonconvex Problems

In this work, we describe the efficient use of improved directions of negative curvature for the solution of bound-constrained nonconvex problems. We follow an interior-point framework, in which the key point is the inclusion of computational low-cost procedures to improve directions of negative curvature obtained from a factorisation of the KKT matrix. From a theoretical point of view, it is well known that these directions ensure convergence to second-order KKT points. As a novelty, we consider the convergence rate of the algorithm with exploitation of negative curvature information. Finally, we test the performance of our proposal on both CUTEr/st and simulated problems, showing empirically that the enhanced directions affect positively the practical performance of the procedure.     

A Simply Constrained Optimization Reformulation of KKT Systems Arising from Variational Inequalities

The Karush—Kuhn—Tucker (KKT) conditions can be regarded as optimality conditions for both variational inequalities and constrained optimization problems. In order to overcome some drawbacks of recently proposed reformulations of KKT systems, we propose casting KKT systems as a minimization problem with nonnegativity constraints on some of the variables. We prove that, under fairly mild assumptions, every stationary point of this constrained minimization problem is a solution of the KKT conditions. Based on this reformulation, a new algorithm for the solution of the KKT conditions is suggested and shown to have some strong global and local convergence properties. Accepted 10 December 1997     

Subsequent Convergence of Iterative Methods with Applications to Real-Time Model-Predictive Control

In performing online model-predictive control of dynamical systems, it is necessary to solve a sequence of optimization problems (typically quadratic programs) in real time so as to generate the best trajectory. Since only a low fixed number of iterations can be executed in real time, it is not possible to solve each quadratic program to optimality. However, numerical experiments show that, if we use information from the numerical solution of the previous quadratic program to construct a warm start for the current quadratic program, there is a time step after which the usual stopping criteria will be satisfied within the fixed number of iterations for all subsequent optimization problems. This phenomenon is called subsequent convergence and will be analyzed for families of nonlinear equations. Computational results are presented to illustrate the theory and associated computational artifacts.     

A Regularized Iterative Scheme for Solving Nonlinear Ill-Posed Problems

In this article, we consider a regularized iterative scheme for solving nonlinear ill-posed problems. The convergence analysis and error estimates are derived by choosing the regularization parameter according to both a priori and a posteriori methods. The iterative scheme is stopped using an a posteriori stopping rule, and we prove that the scheme converges to the solution of the well-known Lavrentiev scheme. The salient features of the proposed scheme are: (i) convergence and error estimate analysis require only weaker assumptions compared to standard assumptions followed in literature, and (ii) consideration of an adaptive a posteriori stopping rule and a parameter choice strategy that gives the same convergence rate as that of an a priori method without using the smallness assumption, the source condition. The above features are very useful from theory and application points of view. We also supply the numerical results to illustrate that the method is adaptable. Further, we compare the numerical result of the proposed method with the standard approach to demonstrate that our scheme is stable and achieves good computational output.     

Convergence Analysis of the Inexact Infeasible Interior-Point Method for Linear Optimization

We present the convergence analysis of the inexact infeasible path-following (IIPF) interior-point algorithm. In this algorithm, the preconditioned conjugate gradient method is used to solve the reduced KKT system (the augmented system). The augmented system is preconditioned by using a block triangular matrix. The KKT system is solved approximately. Therefore, it becomes necessary to study the convergence of the interior-point method for this specific inexact case. We present the convergence analysis of the inexact infeasible path-following (IIPF) algorithm, prove the global convergence of this method and provide complexity analysis. Communicated by Y. Zhang.     

基于KKT和量子遗传算法的风火电联合上网最优决策

研究了同时考虑节能减排效益和经济效益时,风火电联合上网的决策模型,并采用提出的KKT框架下的量子遗传算法进行模型的求解。综合考虑风电和火电的特点,建立经济效益函数和节能减排效益函数以及相关的约束条件,最终确立多目标决策模型。在KKT框架下将多目标函数转化为单目标,并利用量子遗传算法进行模型的求解。算例分析显示本文提出的KKT框架下的量子遗传算法在决策模型的求解时能够利用更少的CPU运行时间获得更优的决策结果,与其他常用的优化模型相比具有较高的优越性。     

Stopping criteria for iterations in finite element methods

Summary. This work extends the results of Arioli [1], [2] on stopping criteria for iterative solution methods for linear finite element problems to the case of nonsymmetric positive-definite problems. We show that the residual measured in the norm induced by the symmetric part of the inverse of the system matrix is relevant to convergence in a finite element context. We then use Krylov solvers to provide alternative ways of calculating or estimating this quantity and present numerical experiments which validate our criteria.Mathematics Subject Classification (2000): 65N30, 65F10, 65F35     

Inexact Two-Grid Methods for Eigenvalue Problems

We discuss the inexact two-grid methods for solving eigenvalue problems, including both partial differential and integral equations. Instead of solving the linear system exactly in both traditional two-grid and accelerated two-grid method, we point out that it is enough to apply an inexact solver to the fine grid problems, which will cut down the computational cost. Different stopping criteria for both methods are developed for keeping the optimality of the resulting solution. Numerical examples are provided to verify our theoretical analyses.     

Structured regularization for barrier NLP solvers

Barrier methods have led to several nonlinear programming (NLP) solvers (e.g. IPOPT, KNITRO, LOQO). However, certain regularity conditions are required for convergence of these methods. These conditions are violated for optimization models with dependent constraints, thus leading to method failure. These shortcomings can be identified by checking the inertia of the KKT matrix, and current solvers either add regularizing terms to correct the inertia of the KKT matrix or revert to more expensive trust region methods to solve the barrier problem. This study improves on these approaches with a new structured regularization strategy; within the Newton step it identifies an independent subset of equality constraints and removes the remaining constraints without modifying the KKT matrix structure. This approach leads to more accurate Newton steps and faster convergence, while maintaining global convergence properties. Implemented in IPOPT with linear solvers HSL_MA57, HSL_MA97 and MUMPS, we present numerical experiments on hundreds of examples from the CUTEr test set, modified for dependency. These results show an average reduction in iterations of more than 50 % over the current version of IPOPT. In addition, several nonlinear blending problems are solved with the proposed algorithm, and improvements over existing regularization strategies are further demonstrated.     

Quadratic Convergence of a Primal-Dual Interior Point Method for Degenerate Nonlinear Optimization Problems

Recently studies of numerical methods for degenerate nonlinear optimization problems have been attracted much attention. Several authors have discussed convergence properties without the linear independence constraint qualification and/or the strict complementarity condition. In this paper, we are concerned with quadratic convergence property of a primal-dual interior point method, in which Newton’s method is applied to the barrier KKT conditions. We assume that the second order sufficient condition and the linear independence of gradients of equality constraints hold at the solution, and that there exists a solution that satisfies the strict complementarity condition, and that multiplier iterates generated by our method for inequality constraints are uniformly bounded, which relaxes the linear independence constraint qualification. Uniform boundedness of multiplier iterates is satisfied if the Mangasarian-Fromovitz constraint qualification is assumed, for example. By using the stability theorem by Hager and Gowda (1999), and Wright (2001), the distance from the current point to the solution set is related to the residual of the KKT conditions.By controlling a barrier parameter and adopting a suitable line search procedure, we prove the quadratic convergence of the proposed algorithm.     

A general steady state distribution based stopping criteria for finite length genetic algorithms

We propose two general stopping criteria for finite length, simple genetic algorithms based on steady state distributions, and empirically investigate the impact of mutation rate, string length, crossover rate and population size on their convergence. Our first stopping criterion is based on the second largest eigenvalue of the genetic algorithm transition matrix, and the second stopping criterion is based on minorization conditions.     

Contraction behaviour of iteration–discretization based on gradient type projections

There is a wide range of iterative methods in infinite dimensional spaces to treat variational equations or variational inequalities. As a rule, computational handling of problems in infinite dimensional spaces requires some discretization. Any useful discretization of the original problem leads to families of problems over finite dimensional spaces. Thus, two infinite techniques, namely discretization and iteration are embedded into each other. In the present paper, the behaviour of truncated iterative methods is studied, where at each discretization level only a finite number of steps is performed. In our study no accuracy dependent a posteriori stopping criterion is used. From an algorithmic point of view, the considered methods are of iteration–discretization type. The major aim here is to provide the convergence analysis for the introduced abstract iteration–discretization methods. A special emphasis is given on algorithms for the treatment of variational inequalities with strongly monotone operators over fixed point sets of quasi-nonexpansive mappings.     

Local Convergence Behavior of Some Projection-Type Methods for Affine Variational Inequalities

In this paper, we study the local convergence behavior of four projection-type methods for the solution of the affine variational inequality (AVI) problem. It is shown that, if the sequence generated by one of the methods converges to a nondegenerate KKT point of the AVI problem, then after a finite number of iterations, some index sets in the dual variables at each iterative point coincide with the index set of the active constraints in the primal variables at the KKT point. As a consequence, we find that, after finitely many iterations, the four methods need not compute projections and their iterative equations are of reduced dimension.