Solving the split equality problem without prior knowledge of operator norms

The split equality problem has extraordinary utility and broad applicability in many areas of applied mathematics. Recently, Moudafi proposed an alternating CQ algorithm and its relaxed variant to solve it. However, to employ Moudafi’s algorithms, one needs to know a priori norm (or at least an estimate of the norm) of the bounded linear operators (matrices in the finite-dimensional framework). To estimate the norm of an operator is very difficult, but not an impossible task. It is the purpose of this paper to introduce a projection algorithm with a way of selecting the stepsizes such that the implementation of the algorithm does not need any priori information about the operator norms. We also practise this way of selecting stepsizes for variants of the projection algorithm, including a relaxed projection algorithm where the two closed convex sets are both level sets of convex functions, and a viscosity algorithm. Both weak and strong convergence are investigated.     

Outer-approximation algorithms for nonsmooth convex MINLP problems

In this work, we combine outer-approximation (OA) and bundle method algorithms for dealing with mixed-integer non-linear programming (MINLP) problems with nonsmooth convex objective and constraint functions. As the convergence analysis of OA methods relies strongly on the differentiability of the involved functions, OA algorithms may fail to solve general nonsmooth convex MINLP problems. In order to obtain OA algorithms that are convergent regardless the structure of the convex functions, we solve the underlying OA’s non-linear subproblems by a specialized bundle method that provides necessary information to cut off previously visited (non-optimal) integer points. This property is crucial for proving (finite) convergence of OA algorithms. We illustrate the numerical performance of the given proposal on a class of hybrid robust and chance-constrained problems that involve a random variable with finite support.     

Optimal subgradient algorithms for large-scale convex optimization in simple domains

This paper describes two optimal subgradient algorithms for solving structured large-scale convex constrained optimization. More specifically, the first algorithm is optimal for smooth problems with Lipschitz continuous gradients and for Lipschitz continuous nonsmooth problems, and the second algorithm is optimal for Lipschitz continuous nonsmooth problems. In addition, we consider two classes of problems: (i) a convex objective with a simple closed convex domain, where the orthogonal projection onto this feasible domain is efficiently available; and (ii) a convex objective with a simple convex functional constraint. If we equip our algorithms with an appropriate prox-function, then the associated subproblem can be solved either in a closed form or by a simple iterative scheme, which is especially important for large-scale problems. We report numerical results for some applications to show the efficiency of the proposed schemes.     

Another version of the proximal point algorithm in a Banach space

In this paper, we study some non-traditional schemes of proximal point algorithm for nonsmooth convex functionals in a Banach space. The proximal approximations to their minimal points and/or their minimal values are considered separately for unconstrained and constrained minimization problems on convex closed sets. For the latter we use proximal point algorithms with the metric projection operators and first establish the estimates of the convergence rate with respect to functionals. We also investigate the perturbed projection proximal point algorithms and prove their stability. Some results concerning the classical proximal point method for minimization problems in a Banach space is also presented in this paper.     

Convergence of the gradient projection method and Newton’s method as applied to optimization problems constrained by intersection of a spherical surface and a convex closed set

The gradient projection method and Newton’s method are generalized to the case of nonconvex constraint sets representing the set-theoretic intersection of a spherical surface with a convex closed set. Necessary extremum conditions are examined, and the convergence of the methods is analyzed.     

A majorization–minimization algorithm for split feasibility problems

The classical multi-set split feasibility problem seeks a point in the intersection of finitely many closed convex domain constraints, whose image under a linear mapping also lies in the intersection of finitely many closed convex range constraints. Split feasibility generalizes important inverse problems including convex feasibility, linear complementarity, and regression with constraint sets. When a feasible point does not exist, solution methods that proceed by minimizing a proximity function can be used to obtain optimal approximate solutions to the problem. We present an extension of the proximity function approach that generalizes the linear split feasibility problem to allow for non-linear mappings. Our algorithm is based on the principle of majorization–minimization, is amenable to quasi-Newton acceleration, and comes complete with convergence guarantees under mild assumptions. Furthermore, we show that the Euclidean norm appearing in the proximity function of the non-linear split feasibility problem can be replaced by arbitrary Bregman divergences. We explore several examples illustrating the merits of non-linear formulations over the linear case, with a focus on optimization for intensity-modulated radiation therapy.     

Proximity Function Minimization Using Multiple Bregman Projections, with Applications to Split Feasibility and Kullback–Leibler Distance Minimization

Problems in signal detection and image recovery can sometimes be formulated as a convex feasibility problem (CFP) of finding a vector in the intersection of a finite family of closed convex sets. Algorithms for this purpose typically employ orthogonal or generalized projections onto the individual convex sets. The simultaneous multiprojection algorithm of Censor and Elfving for solving the CFP, in which different generalized projections may be used at the same time, has been shown to converge for the case of nonempty intersection; still open is the question of its convergence when the intersection of the closed convex sets is empty.Motivated by the geometric alternating minimization approach of Csiszár and Tusnády and the product space formulation of Pierra, we derive a new simultaneous multiprojection algorithm that employs generalized projections of Bregman to solve the convex feasibility problem or, in the inconsistent case, to minimize a proximity function that measures the average distance from a point to all convex sets. We assume that the Bregman distances involved are jointly convex, so that the proximity function itself is convex. When the intersection of the convex sets is empty, but the closure of the proximity function has a unique global minimizer, the sequence of iterates converges to this unique minimizer. Special cases of this algorithm include the Expectation Maximization Maximum Likelihood (EMML) method in emission tomography and a new convergence result for an algorithm that solves the split feasibility problem.     

A splitting algorithm for a class of bilevel equilibrium problems involving nonexpansive mappings

We propose splitting, parallel algorithms for solving strongly equilibrium problems over the intersection of a finite number of closed convex sets given as the fixed-point sets of nonexpansive mappings in real Hilbert spaces. The algorithm is a combination between the gradient method and the Mann-Krasnosel’skii iterative scheme, where the projection can be computed onto each set separately rather than onto their intersection. Strong convergence is proved. Some special cases involving bilevel equilibrium problems with inverse strongly monotone variational inequality, monotone equilibrium constraints and maximal monotone inclusions are discussed. An illustrative example involving a system of integral equations is presented.     

Minibatch stochastic subgradient-based projection algorithms for feasibility problems with convex inequalities

In this paper we consider convex feasibility problems where the feasible set is given as the intersection of a collection of closed convex sets. We assume that each set is specified algebraically as a convex inequality, where the associated convex function is general (possibly non-differentiable). For finding a point satisfying all the convex inequalities we design and analyze random projection algorithms using special subgradient iterations and extrapolated stepsizes. Moreover, the iterate updates are performed based on parallel random observations of several constraint components. For these minibatch stochastic subgradient-based projection methods we prove sublinear convergence results and, under some linear regularity condition for the functional constraints, we prove linear convergence rates. We also derive sufficient conditions under which these rates depend explicitly on the minibatch size. To the best of our knowledge, this work is the first deriving conditions that show theoretically when minibatch stochastic subgradient-based projection updates have a better complexity than their single-sample variants when parallel computing is used to implement the minibatch. Numerical results also show a better performance of our minibatch scheme over its non-minibatch counterpart.

     

A relaxed self-adaptive CQ algorithm for the multiple-sets split feasibility problem

The multiple-sets split feasibility problem (MSFP) is to find a point belongs to the intersection of a family of closed convex sets in one space, such that its image under a linear transformation belongs to the intersection of another family of closed convex sets in the image space. Many iterative methods can be employed to solve the MSFP. Jinling Zhao et al. proposed a modification for the CQ algorithm and a relaxation scheme for this modification to solve the MSFP. The strong convergence of these algorithms are guaranteed in finite-dimensional Hilbert spaces. Recently López et al. proposed a relaxed CQ algorithm for solving split feasibility problem, this algorithm can be implemented easily since it computes projections onto half-spaces and has no need to know a priori the norm of the bounded linear operator. However, this algorithm has only weak convergence in the setting of infinite-dimensional Hilbert spaces. In this paper, we introduce a new relaxed self-adaptive CQ algorithm for solving the MSFP where closed convex sets are level sets of some convex functions such that the strong convergence is guaranteed in the framework of infinite-dimensional Hilbert spaces. Our result extends and improves the corresponding results.     

A sequential iteration algorithm with non-monotoneous behaviour in the method of projections onto convex sets

The method of projections onto convex sets to find a point in the intersection of a finite number of closed convex sets in a Euclidean space, may lead to slow convergence of the constructed sequence when that sequence enters some narrow “corridor” between two or more convex sets. A way to leave such corridor consists in taking a big step at different moments during the iteration, because in that way the monotoneous behaviour that is responsible for the slow convergence may be interrupted. In this paper we present a technique that may introduce interruption of the monotony for a sequential algorithm, but that at the same time guarantees convergence of the constructed sequence to a point in the intersection of the sets. We compare experimentally the behaviour concerning the speed of convergence of the new algorithm with that of an existing monotoneous algorithm.     

Long-Step Interior-Point Algorithms for a Class of Variational Inequalities with Monotone Operators

This paper describes two interior-point algorithms for solving a class of monotone variational inequalities defined over the intersection of an affine set and a closed convex set. The first algorithm is a long-step path-following method, and the second is an extension of the first, incorporating weights in the gradient of the barrier function. Global convergence of the algorithms is proven under the assumptions of monotonicity and differentiability of the operator.     

Finding projections onto the intersection of convex sets in Hilbert spaces. II

We present a parallel iterative algorithm to find the shortest distance projection of a given point onto the intersection of a finite number of closed convex sets in a real Hilbert space; the number of sets used at each iteration step, corresponding to the number of available processors, may be smaller than the total number of sets. The relaxation coefficient at each iteraction step is determined by a geometric condition in an associated Hilbert space, while for the weights mild conditions are given to assure norm convergence of the resulting sequence. These mild conditions leave enough flexibility to determine the weights more specifically in order to improve the speed of convergence.     

A successive projection method

It is of both theoretical and practical interest to find the projection of a point to the intersection of a finite number of closed convex sets by a sequence of projections to the individual sets successively. In this paper we study such a method and analyze its convergence properties. A main feature of the method is its capability to decompose the projection problem into several small ones. For some structured sparse problems these small subproblems can be solved independently and the presented method has a potential use in parallel computation.This material is based on work supported in part by the National Science Foundation under Grant DMS-8602419 and by the Center for Supercomputing Research and Development, University of Illinois.     

Finding the projection of a point onto the intersection of convex sets via projections onto half-spaces

We present a modification of Dykstra's algorithm which allows us to avoid projections onto general convex sets. Instead, we calculate projections onto either a half-space or onto the intersection of two half-spaces. Convergence of the algorithm is established and special choices of the half-spaces are proposed.The option to project onto half-spaces instead of general convex sets makes the algorithm more practical. The fact that the half-spaces are quite general enables us to apply the algorithm in a variety of cases and to generalize a number of known projection algorithms.The problem of projecting a point onto the intersection of closed convex sets receives considerable attention in many areas of mathematics and physics as well as in other fields of science and engineering such as image reconstruction from projections.In this work we propose a new class of algorithms which allow projection onto certain super half-spaces, i.e., half-spaces which contain the convex sets. Each one of the algorithms that we present gives the user freedom to choose the specific super half-space from a family of such half-spaces. Since projecting a point onto a half-space is an easy task to perform, the new algorithms may be more useful in practical situations in which the construction of the super half-spaces themselves is not too difficult.     

Random algorithms for convex minimization problems

This paper deals with iterative gradient and subgradient methods with random feasibility steps for solving constrained convex minimization problems, where the constraint set is specified as the intersection of possibly infinitely many constraint sets. Each constraint set is assumed to be given as a level set of a convex but not necessarily differentiable function. The proposed algorithms are applicable to the situation where the whole constraint set of the problem is not known in advance, but it is rather learned in time through observations. Also, the algorithms are of interest for constrained optimization problems where the constraints are known but the number of constraints is either large or not finite. We analyze the proposed algorithm for the case when the objective function is differentiable with Lipschitz gradients and the case when the objective function is not necessarily differentiable. The behavior of the algorithm is investigated both for diminishing and non-diminishing stepsize values. The almost sure convergence to an optimal solution is established for diminishing stepsize. For non-diminishing stepsize, the error bounds are established for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected sub-optimality of the function values along the weighted averages.     

Adaptive Projected Subgradient Method for Asymptotic Minimization of Sequence of Nonnegative Convex Functions

Abstract

This paper presents an algorithm, named adaptive projected subgradient method that can minimize asymptotically a certain sequence of nonnegative convex functions over a closed convex set in a real Hilbert space. The proposed algorithm is a natural extension of the Polyak's subgradient algorithm, for nonsmooth convex optimization problem with a fixed target value, to the case where the convex objective itself keeps changing in the whole process. The main theorem, showing the strong convergence of the algorithm as well as the asymptotic optimality of the sequence generated by the algorithm, can serve as a unified guiding principle of a wide range of set theoretic adaptive filtering schemes for nonstationary random processes. These include not only the existing adaptive filtering techniques; e.g., NLMS, Projected NLMS, Constrained NLMS, APA, and Adaptive parallel outer projection algorithm etc., but also new techniques; e.g., Adaptive parallel min-max projection algorithm, and their embedded constraint versions. Numerical examples show that the proposed techniques are well-suited for robust adaptive signal processing problems.     

Computational acceleration of projection algorithms for the linear best approximation problem

This is an experimental computational account of projection algorithms for the linear best approximation problem. We focus on the sequential and simultaneous versions of Dykstra’s algorithm and the Halpern-Lions-Wittmann-Bauschke algorithm for the best approximation problem from a point to the intersection of closed convex sets in the Euclidean space. These algorithms employ different iterative approaches to reach the same goal but no mathematical connection has yet been found between their algorithmic schemes. We compare these algorithms on linear best approximation test problems that we generate so that the solution will be known a priori and enable us to assess the relative computational merits of these algorithms. For the simultaneous versions we present a new component-averaging variant that substantially accelerates their initial behavior for sparse systems.     

Distance majorization and its applications

The problem of minimizing a continuously differentiable convex function over an intersection of closed convex sets is ubiquitous in applied mathematics. It is particularly interesting when it is easy to project onto each separate set, but nontrivial to project onto their intersection. Algorithms based on Newton’s method such as the interior point method are viable for small to medium-scale problems. However, modern applications in statistics, engineering, and machine learning are posing problems with potentially tens of thousands of parameters or more. We revisit this convex programming problem and propose an algorithm that scales well with dimensionality. Our proposal is an instance of a sequential unconstrained minimization technique and revolves around three ideas: the majorization-minimization principle, the classical penalty method for constrained optimization, and quasi-Newton acceleration of fixed-point algorithms. The performance of our distance majorization algorithms is illustrated in several applications.     

Outer approximation methods for solving variational inequalities in Hilbert space

In this paper, we study variational inequalities in a real Hilbert space, which are governed by a strongly monotone and Lipschitz continuous operator F over a closed and convex set C. We assume that the set C can be outerly approximated by the fixed point sets of a sequence of certain quasi-nonexpansive operators called cutters. We propose an iterative method, the main idea of which is to project at each step onto a particular half-space constructed using the input data. Our approach is based on a method presented by Fukushima in 1986, which has recently been extended by several authors. In the present paper, we establish strong convergence in Hilbert space. We emphasize that to the best of our knowledge, Fukushima’s method has so far been considered only in the Euclidean setting with different conditions on F. We provide several examples for the case where C is the common fixed point set of a finite number of cutters with numerical illustrations of our theoretical results.