A practical relative error criterion for augmented Lagrangians

This paper develops a new error criterion for the approximate minimization of augmented Lagrangian subproblems. This criterion is practical since it is readily testable given only a gradient (or subgradient) of the augmented Lagrangian. It is also “relative” in the sense of relative error criteria for proximal point algorithms: in particular, it uses a single relative tolerance parameter, rather than a summable parameter sequence. Our analysis first describes an abstract version of the criterion within Rockafellar’s general parametric convex duality framework, and proves a global convergence result for the resulting algorithm. Specializing this algorithm to a standard formulation of convex programming produces a version of the classical augmented Lagrangian method with a novel inexact solution condition for the subproblems. Finally, we present computational results drawn from the CUTE test set—including many nonconvex problems—indicating that the approach works well in practice.     

An inexact alternating direction method of multipliers with relative error criteria

In this paper, we study an inexact version of the alternating direction method of multipliers (ADMM) for solving two-block separable linearly constrained convex optimization problems. Specifically, the two subproblems in the classic ADMM are allowed to be solved inexactly by certain relative error criteria, in the sense that only two parameters are needed to control the inexactness. Related convergence analysis are established under the assumption that the solution set to the KKT system of the problem is not empty. Numerical results on solving a class of sparse signal recovery problems are also provided to demonstrate the efficiency of the proposed algorithm.     

Matroid optimisation problems with nested non-linear monomials in the objective function

We derive a new approximate version of the alternating direction method of multipliers (ADMM) which uses a relative error criterion. The new version is somewhat restrictive and allows only one of the two subproblems to be minimized approximately, but nevertheless covers commonly encountered special cases. The derivation exploits the long-established relationship between the ADMM and both the proximal point algorithm (PPA) and Douglas–Rachford (DR) splitting for maximal monotone operators, along with a relative-error of the PPA due to Solodov and Svaiter. In the course of analysis, we also derive a version of DR splitting in which one operator may be evaluated approximately using a relative error criterion. We computationally evaluate our method on several classes of test problems and find that it significantly outperforms several alternatives on one problem class.     

On inexact ADMMs with relative error criteria

In this paper, we develop two inexact alternating direction methods of multipliers (ADMMs) with relative error criteria for which only a few parameters are needed to control the error tolerance. In many practical applications, the numerical performance is often improved if a larger step-length is used. Hence in this paper we also consider to seek a larger step-length to update the Lagrangian multiplier for better numerical efficiency. Specifically, if we only allow one subproblem in the classic ADMM to be solved inexactly by a certain relative error criterion, then a larger step-length can be used to update the Lagrangian multiplier. Related convergence analysis of those proposed algorithms is also established under the assumption that the solution set to the KKT system of the problem is not empty. Numerical experiments on solving total variation (TV)-based image denosing and analysis sparse recovery problems are provided to demonstrate the effectiveness of the proposed methods and the advantage of taking a larger step-length.     

A class of nonlinear Lagrangians for nonconvex second order cone programming

This paper focuses on the study of a class of nonlinear Lagrangians for solving nonconvex second order cone programming problems. The nonlinear Lagrangians are generated by Löwner operators associated with convex real-valued functions. A set of conditions on the convex real-valued functions are proposed to guarantee the convergence of nonlinear Lagrangian algorithms. These conditions are satisfied by well-known nonlinear Lagrangians appeared in the literature. The convergence properties for the nonlinear Lagrange method are discussed when subproblems are assumed to be solved exactly and inexactly, respectively. The convergence theorems show that, under the second order sufficient conditions with sigma-term and the strict constraint nondegeneracy condition, the algorithm based on any of nonlinear Lagrangians in the class is locally convergent when the penalty parameter is less than a threshold and the error bound of solution is proportional to the penalty parameter. Compared to the analysis in nonlinear Lagrangian methods for nonlinear programming, we have to deal with the sigma term in the convergence analysis. Finally, we report numerical results by using modified Frisch’s function, modified Carroll’s function and the Log-Sigmoid function.     

Solving Continuous Min-Max Problems by an Iterative Entropic Regularization Method

We propose a method of outer approximations, with each approximate problem smoothed using entropic regularization, to solve continuous min-max problems. By using a well-known uniform error estimate for entropic regularization, convergence of the overall method is shown while allowing each smoothed problem to be solved inexactly. In the case of convex objective function and linear constraints, an interior-point algorithm is proposed to solve the smoothed problem inexactly. Numerical examples are presented to illustrate the behavior of the proposed method.     

Parallel Variable Distribution for Constrained Optimization

In the parallel variable distribution framework for solving optimization problems (PVD), the variables are distributed among parallel processors with each processor having the primary responsibility for updating its block of variables while allowing the remaining secondary variables to change in a restricted fashion along some easily computable directions. For constrained nonlinear programs convergence theory for PVD algorithms was previously available only for the case of convex feasible set. Additionally, one either had to assume that constraints are block-separable, or to use exact projected gradient directions for the change of secondary variables. In this paper, we propose two new variants of PVD for the constrained case. Without assuming convexity of constraints, but assuming block-separable structure, we show that PVD subproblems can be solved inexactly by solving their quadratic programming approximations. This extends PVD to nonconvex (separable) feasible sets, and provides a constructive practical way of solving the parallel subproblems. For inseparable constraints, but assuming convexity, we develop a PVD method based on suitable approximate projected gradient directions. The approximation criterion is based on a certain error bound result, and it is readily implementable. Using such approximate directions may be especially useful when the projection operation is computationally expensive.     

Linearized Alternating Direction Method of Multipliers for Separable Convex Optimization of Real Functions in Complex Domain

The alternating direction method of multipliers (ADMM) for separable convex optimization of real functions in complex variables has been proposed recently[21]. Furthermore, the convergence and $O(1/K)$ convergence rate of ADMM in complex domain have also been derived[22]. In this paper, a fast linearized ADMM in complex domain has been presented as the subproblems do not have closed solutions. First, some useful results in complex domain are developed by using the Wirtinger Calculus technique. Second, the convergence of the linearized ADMM in complex domain based on the VI is established. Third, an extended model of least absolute shrinkage and selectionator operator (LASSO) is solved by using linearized ADMM in complex domain. Finally, some numerical simulations are provided to show that linearized ADMM in complex domain has the rapid speed.     

An alternative framework to Lagrangian relaxation approach for job shop scheduling

A new Lagrangian relaxation (LR) approach is developed for job shop scheduling problems. In the approach, operation precedence constraints rather than machine capacity constraints are relaxed. The relaxed problem is decomposed into single or parallel machine scheduling subproblems. These subproblems, which are NP-complete in general, are approximately solved by using fast heuristic algorithms. The dual problem is solved by using a recently developed “surrogate subgradient method” that allows approximate optimization of the subproblems. Since the algorithms for subproblems do not depend on the time horizon of the scheduling problems and are very fast, our new LR approach is efficient, particularly for large problems with long time horizons. For these problems, the machine decomposition-based LR approach requires much less memory and computation time as compared to a part decomposition-based approach as demonstrated by numerical testing.     

Surrogate Gradient Algorithm for Lagrangian Relaxation

The subgradient method is used frequently to optimize dual functions in Lagrangian relaxation for separable integer programming problems. In the method, all subproblems must be solved optimally to obtain a subgradient direction. In this paper, the surrogate subgradient method is developed, where a proper direction can be obtained without solving optimally all the subproblems. In fact, only an approximate optimization of one subproblem is needed to get a proper surrogate subgradient direction, and the directions are smooth for problems of large size. The convergence of the algorithm is proved. Compared with methods that take effort to find better directions, this method can obtain good directions with much less effort and provides a new approach that is especially powerful for problems of very large size.     

A UNIFIED FRAMEWORK FOR SOME INEXACT PROXIMAL POINT ALGORITHMS*

We present a unified framework for the design and convergence analysis of a class of algorithms based on approximate solution of proximal point subproblems. Our development further enhances the constructive approximation approach of the recently proposed hybrid projection–proximal and extragradient–proximal methods. Specifically, we introduce an even more flexible error tolerance criterion, as well as provide a unified view of these two algorithms. Our general method possesses global convergence and local (super)linear rate of convergence under standard assumptions, while using a constructive approximation criterion suitable for a number of specific implementations. For example, we show that close to a regular solution of a monotone system of semismooth equations, two Newton iterations are sufficient to solve the proximal subproblem within the required error tolerance. Such systems of equations arise naturally when reformulating the nonlinear complementarity problem.

     

On a Chebyshev accelerated splitting iteration method with application to two‐by‐two block linear systems

The Chebyshev accelerated preconditioned modified Hermitian and skew‐Hermitian splitting (CAPMHSS) iteration method is presented for solving the linear systems of equations, which have two‐by‐two block coefficient matrices. We derive an iteration error bound to show that the new method is convergent as long as the eigenvalue bounds are not underestimated. Even when the spectral information is lacking, the CAPMHSS iteration method could be considered as an exponentially converging iterative scheme for certain choices of the method parameters. In this case, the convergence rate is independent of the parameters. Besides, the linear subsystems in each iteration can be solved inexactly, which leads to the inexact CAPMHSS iteration method. The iteration error bound of the inexact method is derived also. We discuss in detail the implementation of CAPMHSS for solving two models arising from the Galerkin finite‐element discretizations of distributed control problems and complex symmetric linear systems. The numerical results show the robustness and the efficiency of the new methods.     

A trust-region method with a conic model for unconstrained optimization

In this paper, we propose and analyze a new conic trust-region algorithm for solving the unconstrained optimization problems. A new strategy is proposed to construct the conic model and the relevant conic trust-region subproblems are solved by an approximate solution method. This approximate solution method is not only easy to implement but also preserves the strong convergence properties of the exact solution methods. Under reasonable conditions, the locally linear and superlinear convergence of the proposed algorithm is established. The numerical experiments show that this algorithm is both feasible and efficient. Copyright © 2008 John Wiley & Sons, Ltd.     

Multiphysics mixed finite element method with Nitsche's technique for Stokes-poroelasticity problem

In this article, we propose a multiphysics mixed finite element method with Nitsche's technique for Stokes-poroelasticity problem. Firstly, we reformulate the poroelasticity part of the original problem by introducing two pseudo-pressures to into a “fluid–fluid” coupled problem so that we can use the classical stable finite element pairs to deal with this problem conveniently. Then, we prove the existence and uniqueness of weak solution of the reformulated problem. And we use Nitsche's technique to approximate the coupling condition at the interface to propose a loosely-coupled time-stepping method to solve three subproblems at each time step–a Stokes problem, a generalized Stokes problem and a mixed diffusion problem. And the proposed method does not require any restriction on the choice of the discrete approximation spaces on each side of the interface provided that appropriate quadrature methods are adopted. Also, we give the stability analysis and error estimates of the loosely-coupled time-stepping method. Finally, we give the numerical tests to show that the proposed numerical method has a good stability and no “locking” phenomenon.     

On the convergence of inexact augmented Lagrangian methods for problems with convex constraints

We consider the augmented Lagrangian method (ALM) for constrained optimization problems in the presence of convex inequality and convex abstract constraints. We focus on the case where the Lagrangian sub-problems are solved up to approximate stationary points, with increasing accuracy. We analyze two different criteria of approximate stationarity for the sub-problems and we prove the global convergence to stationary points of ALM in both cases.     

On the Convergence of Formal Solutions of a System of Partial Differential Equations

We study a version of the classical problem on the convergence of formal solutions of systems of partial differential equations. A necessary and sufficient condition for the convergence of a given formal solution (found by any method) is proved. This convergence criterion applies to systems of partial differential equations (possibly, nonlinear) solved for the highest-order derivatives or, which is most important, “almost solved for the highest-order derivatives.”     

Inexact solution of NLP subproblems in MINLP

In the context of convex mixed integer nonlinear programming (MINLP), we investigate how the outer approximation method and the generalized Benders decomposition method are affected when the respective nonlinear programming (NLP) subproblems are solved inexactly. We show that the cuts in the corresponding master problems can be changed to incorporate the inexact residuals, still rendering equivalence and finiteness in the limit case. Some numerical results will be presented to illustrate the behavior of the methods under NLP subproblem inexactness.     

A primal-dual augmented Lagrangian

Nonlinearly constrained optimization problems can be solved by minimizing a sequence of simpler unconstrained or linearly constrained subproblems. In this paper, we consider the formulation of subproblems in which the objective function is a generalization of the Hestenes-Powell augmented Lagrangian function. The main feature of the generalized function is that it is minimized with respect to both the primal and the dual variables simultaneously. The benefits of this approach include: (i) the ability to control the quality of the dual variables during the solution of the subproblem; (ii) the availability of improved dual estimates on early termination of the subproblem; and (iii) the ability to regularize the subproblem by imposing explicit bounds on the dual variables. We propose two primal-dual variants of conventional primal methods: a primal-dual bound constrained Lagrangian (pdBCL) method and a primal-dual ℓ ₁ linearly constrained Lagrangian (pdℓ ₁LCL) method. Finally, a new sequential quadratic programming (pdSQP) method is proposed that uses the primal-dual augmented Lagrangian as a merit function.     

An Inexact Alternating Direction Method for Structured Variational Inequalities

Recently, the alternating direction method of multipliers has attracted great attention. For a class of variational inequalities (VIs), this method is efficient, when the subproblems can be solved exactly. However, the subproblems could be too difficult or impossible to be solved exactly in many practical applications. In this paper, we propose an inexact method for structured VIs based on the projection and contraction method. Instead of solving the subproblems exactly, we use the simple projection to get a predictor and correct it to approximate the subproblems’ real solutions. The convergence of the proposed method is proved under mild assumptions and its efficiency is also verified by some numerical experiments.     

A note on the convergence of ADMM for linearly constrained convex optimization problems

This note serves two purposes. Firstly, we construct a counterexample to show that the statement on the convergence of the alternating direction method of multipliers (ADMM) for solving linearly constrained convex optimization problems in a highly influential paper by Boyd et al. (Found Trends Mach Learn 3(1):1–122, 2011) can be false if no prior condition on the existence of solutions to all the subproblems involved is assumed to hold. Secondly, we present fairly mild conditions to guarantee the existence of solutions to all the subproblems of the ADMM and provide a rigorous convergence analysis on the ADMM with a computationally more attractive large step-length that can even exceed the practically much preferred golden ratio of \((1+\sqrt{5})/2\).