A forward–backward penalty scheme with inertial effects for monotone inclusions. Applications to convex bilevel programming

ABSTRACT

We investigate a forward–backward splitting algorithm of penalty type with inertial effects for finding the zeros of the sum of a maximally monotone operator and a cocoercive one and the convex normal cone to the set of zeroes of an another cocoercive operator. Weak ergodic convergence is obtained for the iterates, provided that a condition expressed via the Fitzpatrick function of the operator describing the underlying set of the normal cone is verified. Under strong monotonicity assumptions, strong convergence for the sequence of generated iterates is proved. As a particular instance we consider a convex bilevel minimization problem including the sum of a non-smooth and a smooth function in the upper level and another smooth function in the lower level. We show that in this context weak non-ergodic and strong convergence can be also achieved under inf-compactness assumptions for the involved functions.     

A First-Order Stochastic Primal-Dual Algorithm with Correction Step

In this article, we investigate the convergence properties of a stochastic primal-dual splitting algorithm for solving structured monotone inclusions involving the sum of a cocoercive operator and a composite monotone operator. The proposed method is the stochastic extension to monotone inclusions of a proximal method studied in the literature for saddle point problems. It consists in a forward step determined by the stochastic evaluation of the cocoercive operator, a backward step in the dual variables involving the resolvent of the monotone operator, and an additional forward step using the stochastic evaluation of the cocoercive operator introduced in the first step. We prove weak almost sure convergence of the iterates by showing that the primal-dual sequence generated by the method is stochastic quasi-Fejér-monotone with respect to the set of zeros of the considered primal and dual inclusions. Additional results on ergodic convergence in expectation are considered for the special case of saddle point models.     

An inertial forward-backward-forward primal-dual splitting algorithm for solving monotone inclusion problems

We introduce and investigate the convergence properties of an inertial forward-backward-forward splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitzian operator. By making use of the product space approach, we expand it to the solving of inclusion problems involving mixtures of linearly composed and parallel-sum type monotone operators. We obtain in this way an inertial forward-backward-forward primal-dual splitting algorithm having as main characteristic the fact that in the iterative scheme all operators are accessed separately either via forward or via backward evaluations. We present also the variational case when one is interested in the solving of a primal-dual pair of convex optimization problems with complexly structured objectives, which we also illustrate by numerical experiments in image processing.     

A stochastic inertial forward–backward splitting algorithm for multivariate monotone inclusions

We propose an inertial forward–backward splitting algorithm to compute a zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal–dual splitting methods for solving structured systems of composite monotone inclusions and prove their convergence. Our results extend to the stochastic and inertial setting various types of structured monotone inclusion problems and corresponding algorithmic solutions. Application to minimization problems is discussed.     

Iterative algorithms with the regularization for the constrained convex minimization problem and maximal monotone operators

In this paper, we prove a strong convergence theorem for finding a common element of the solution set of a constrained convex minimization problem and the set of solutions of a finite family of variational inclusion problems in Hilbert space. A strong convergence theorem for finding a common element of the solution set of a constrained convex minimization problem and the solution sets of a finite family of zero points of the maximal monotone operator problem in Hilbert space is also obtained. Using our main result, we have some additional results for various types of non-linear problems in Hilbert space.     

An Answer to S. Simons’ Question on the Maximal Monotonicity of the Sum of a Maximal Monotone Linear Operator and a Normal Cone Operator

The question whether or not the sum of two maximal monotone operators is maximal monotone under Rockafellar’s constraint qualification—that is, whether or not “the sum theorem” is true—is the most famous open problem in Monotone Operator Theory. In his 2008 monograph “From Hahn-Banach to Monotonicity”, Stephen Simons asked whether or not the sum theorem holds for the special case of a maximal monotone linear operator and a normal cone operator of a closed convex set provided that the interior of the set makes a nonempty intersection with the domain of the linear operator. In this note, we provide an affirmative answer to Simons’ question. In fact, we show that the sum theorem is true for a maximal monotone linear relation and a normal cone operator. The proof relies on Rockafellar’s formula for the Fenchel conjugate of the sum as well as some results featuring the Fitzpatrick function.      

Forward-Backward and Tseng’s Type Penalty Schemes for Monotone Inclusion Problems

We deal with monotone inclusion problems of the form 0 ∈ A x + D x + N _C (x) in real Hilbert spaces, where A is a maximally monotone operator, D a cocoercive operator and C the nonempty set of zeros of another cocoercive operator. We propose a forward-backward penalty algorithm for solving this problem which extends the one proposed by Attouch et al. (SIAM J. Optim. 21(4): 1251-1274, 2011). The condition which guarantees the weak ergodic convergence of the sequence of iterates generated by the proposed scheme is formulated by means of the Fitzpatrick function associated to the maximally monotone operator that describes the set C. In the second part we introduce a forward-backward-forward algorithm for monotone inclusion problems having the same structure, but this time by replacing the cocoercivity hypotheses with Lipschitz continuity conditions. The latter penalty type algorithm opens the gate to handle monotone inclusion problems with more complicated structures, for instance, involving compositions of maximally monotone operators with linear continuous ones.     

Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions

We provide two weakly convergent algorithms for finding a zero of the sum of a maximally monotone operator, a cocoercive operator, and the normal cone to a closed vector subspace of a real Hilbert space. The methods exploit the intrinsic structure of the problem by activating explicitly the cocoercive operator in the first step, and taking advantage of a vector space decomposition in the second step. The second step of the first method is a Douglas–Rachford iteration involving the maximally monotone operator and the normal cone. In the second method, it is a proximal step involving the partial inverse of the maximally monotone operator with respect to the vector subspace. Connections between the proposed methods and other methods in the literature are provided. Applications to monotone inclusions with finitely many maximally monotone operators and optimization problems are examined.     

Tight global linear convergence rate bounds for Douglas–Rachford splitting

Recently, several authors have shown local and global convergence rate results for Douglas–Rachford splitting under strong monotonicity, Lipschitz continuity, and cocoercivity assumptions. Most of these focus on the convex optimization setting. In the more general monotone inclusion setting, Lions and Mercier showed a linear convergence rate bound under the assumption that one of the two operators is strongly monotone and Lipschitz continuous. We show that this bound is not tight, meaning that no problem from the considered class converges exactly with that rate. In this paper, we present tight global linear convergence rate bounds for that class of problems. We also provide tight linear convergence rate bounds under the assumptions that one of the operators is strongly monotone and cocoercive, and that one of the operators is strongly monotone and the other is cocoercive. All our linear convergence results are obtained by proving the stronger property that the Douglas–Rachford operator is contractive.     

Strong convergence studied by a hybrid type method for monotone operators in a Banach space

In this paper, we study a strong convergence for monotone operators. We first introduce the hybrid type algorithm for monotone operators. Next, we obtain a strong convergence theorem (Theorem 3.3) for finding a zero point of an inverse-strongly monotone operator in a Banach space. Finally, we apply our convergence theorem to the problem of finding a minimizer of a convex function.     

Outer Approximation Method for Constrained Composite Fixed Point Problems Involving Lipschitz Pseudo Contractive Operators

We propose a method for solving constrained fixed point problems involving compositions of Lipschitz pseudo contractive and firmly nonexpansive operators in Hilbert spaces. Each iteration of the method uses separate evaluations of these operators and an outer approximation given by the projection onto a closed half-space containing the constraint set. Its convergence is established and applications to monotone inclusion splitting and constrained equilibrium problems are demonstrated.     

Non-convex sweeping processes involving maximal monotone operators

By using a regularization method, we study in this paper the global existence and uniqueness property of a new variant of non-convex sweeping processes involving maximal monotone operators. The system can be considered as a maximal monotone differential inclusion under a control term of normal cone type forcing the trajectory to be always contained in the desired moving set. When the set is fixed, one can show that the unique solution is right-differentiable everywhere and its right-derivative is right-continuous. Non-smooth Lyapunov pairs for this system are also analysed.     

Strong convergence theorem by hybrid method for equilibrium problems,variational inequality problems and maximal monotone operators

We introduce an iterative scheme for finding a common element of the solution set of the equilibrium problem, the solution set of the variational inequality problem for an inverse-strongly-monotone operators and the solution set of a maximal monotone operator in a 2-uniformly convex and uniformly smooth Banach space, and then we present strong convergence theorems which generalize the results of many others.     

A new iterative algorithm for the variational inequality problem over the fixed point set of a firmly nonexpansive mapping

Many problems to appear in signal processing have been formulated as the variational inequality problem over the fixed point set of a nonexpansive mapping. In particular, convex optimization problems over the fixed point set are discussed, and operators which are considered to the problems satisfy the monotonicity. Hence, the uniqueness of the solution of the problem is not always guaranteed. In this article, we present the variational inequality problem for a monotone, hemicontinuous operator over the fixed point set of a firmly nonexpansive mapping. The main aim of the article is to solve the proposed problem by using an iterative algorithm. To this goal, we present a new iterative algorithm for the proposed problem and its convergence analysis. Numerical examples for the proposed algorithm for convex optimization problems over the fixed point set are provided in the final section.     

To the theory of operator monotone and operator convex functions

We prove that a real function is operator monotone (operator convex) if the corresponding monotonicity (convexity) inequalities are valid for some normal state on the algebra of all bounded operators in an infinite-dimensional Hilbert space. We describe the class of convex operator functions with respect to a given von Neumann algebra in dependence of types of direct summands in this algebra. We prove that if a function from ℝ⁺ into ℝ⁺ is monotone with respect to a von Neumann algebra, then it is also operator monotone in the sense of the natural order on the set of positive self-adjoint operators affiliated with this algebra.     

On the superlinear convergence of the variable metric proximal point algorithm using Broyden and BFGS matrix secant updating

In previous work, the authors provided a foundation for the theory of variable metric proximal point algorithms in Hilbert space. In that work conditions are developed for global, linear, and super–linear convergence. This paper focuses attention on two matrix secant updating strategies for the finite dimensional case. These are the Broyden and BFGS updates. The BFGS update is considered for application in the symmetric case, e.g., convex programming applications, while the Broyden update can be applied to general monotone operators. Subject to the linear convergence of the iterates and a quadratic growth condition on the inverse of the operator at the solution, super–linear convergence of the iterates is established for both updates. These results are applied to show that the Chen–Fukushima variable metric proximal point algorithm is super–linearly convergent when implemented with the BFGS update. Received: September 12, 1996 / Accepted: January 7, 2000?Published online March 15, 2000     

Strong convergence of new iterative projection methods with regularization for solving monotone variational inequalities in Hilbert spaces

In this paper, we introduce two new numerical methods for solving a variational inequality problem involving a monotone and Lipschitz continuous operator in a Hilbert space. We describe how to incorporate a regularization term depending on a parameter in the projection method and then establish the strong convergence of the resulting iterative regularization projection methods. Unlike known hybrid methods, the strong convergence of the new methods comes from the regularization technique. The first method is designed to work in the case where the Lipschitz constant of cost operator is known, whereas the second one is more easily implemented without this requirement. The reason is because the second method has used a simple computable stepsize rule. The variable stepsizes are generated by the second method at each iteration and based on the previous iterates. These stepsizes are found with only one cheap computation without line-search procedure. Several numerical experiments are implemented to show the computational effectiveness of the new methods over existing methods.     

A Proximal Point Method in Nonreflexive Banach Spaces

We propose an inexact version of the proximal point method and study its properties in nonreflexive Banach spaces which are duals of separable Banach spaces, both for the problem of minimizing convex functions and of finding zeroes of maximal monotone operators. By using surjectivity results for enlargements of maximal monotone operators, we prove existence of the iterates in both cases. Then we recover most of the convergence properties known to hold in reflexive and smooth Banach spaces for the convex optimization problem. When dealing with zeroes of monotone operators, our convergence result requests that the regularization parameters go to zero, as is the case for standard (non-proximal) regularization schemes.     

Penalty and Barrier Methods for Convex Semidefinite Programming

In this paper we present penalty and barrier methods for solving general convex semidefinite programming problems. More precisely, the constraint set is described by a convex operator that takes its values in the cone of negative semidefinite symmetric matrices. This class of methods is an extension of penalty and barrier methods for convex optimization to this setting. We provide implementable stopping rules and prove the convergence of the primal and dual paths obtained by these methods under minimal assumptions. The two parameters approach for penalty methods is also extended. As for usual convex programming, we prove that after a finite number of steps all iterates will be feasible.     

A proximal point method for the sum of maximal monotone operators

In this paper, we concentrate on the maximal inclusion problem of locating the zeros of the sum of maximal monotone operators in the framework of proximal point method. Such problems arise widely in several applied mathematical fields such as signal and image processing. We define two new maximal monotone operators and characterize the solutions of the considered problem via the zeros of the new operators. The maximal monotonicity and resolvent of both of the defined operators are proved and calculated, respectively. The traditional proximal point algorithm can be therefore applied to the considered maximal inclusion problem, and the convergence is ensured. Furthermore, by exploring the relationship between the proposed method and the generalized forward‐backward splitting algorithm, we point out that this algorithm is essentially the proximal point algorithm when the operator corresponding to the forward step is the zero operator. Copyright © 2013 John Wiley & Sons, Ltd.