On the acceleration of the double smoothing technique for unconstrained convex optimization problems

In this article, we investigate the possibilities of accelerating the double smoothing (DS) technique when solving unconstrained nondifferentiable convex optimization problems. This approach relies on the regularization in two steps of the Fenchel dual problem associated with the problem to be solved into an optimization problem having a differentiable strongly convex objective function with Lipschitz continuous gradient. The doubly regularized dual problem is then solved via a fast gradient method. The aim of this article is to show how the properties of the functions in the objective of the primal problem influence the implementation of the DS approach and its rate of convergence. The theoretical results are applied to linear inverse problems by making use of different regularization functionals.     

An Infinite-Dimensional Semismooth Newton Method for Elasto-Plastic Contact Problems

A Fenchel dualization scheme for the one-step time-discretized elasto-plastic contact problem with kinematic or isotropic hardening is considered. The associated path is induced by a combined Moreau-Yosida / Tichonov regularization of the dual problem. The sequence of solutions to the regularized problems is shown to converge strongly to the solution of the original problem. This property relies on the density of the intersection of certain convex sets. The corresponding conditions are worked out and customary regularization approaches are shown to be valid in this context. It is also argued that without higher regularity assumptions on the data the resulting problems possess Newton differentiable optimality systems in infinite dimensions [2]. Consequently, each regularized subsystem can be solved mesh-independently at a local superlinear rate of convergence [6]. Numerically the problems are solved using conforming finite elements. (© 2013 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Regularization of quasi-variational inequalities

An ill-posed quasi-variational inequality with contaminated data can be stabilized by employing the elliptic regularization. Under suitable conditions, a sequence of bounded regularized solutions converges strongly to a solution of the original quasi-variational inequality. Moreover, the conditions that ensure the boundedness of regularized solutions, become sufficient solvability conditions. It turns out that the regularization theory is quite strong for quasi-variational inequalities with set-valued monotone maps but restrictive for generalized pseudo-monotone maps. The results are quite general and are applicable to ill-posed variational inequalities, hemi-variational inequalities, inverse problems, and split feasibility problem, among others.     

Convex regularization of local volatility models from option prices: Convergence analysis and rates

We study a convex regularization of the local volatility surface identification problem for the Black-Scholes partial differential equation from prices of European call options. This is a highly nonlinear ill-posed problem which in practice is subject to different noise levels associated to bid-ask spreads and sampling errors. We analyze, in appropriate function spaces, different properties of the parameter-to-solution map that assigns to a given volatility surface the corresponding option prices. Using such properties, we show stability and convergence of the regularized solutions in terms of the Bregman distance with respect to a class of convex regularization functionals when the noise level goes to zero.We improve convergence rates available in the literature for the volatility identification problem. Furthermore, in the present context, we relate convex regularization with the notion of exponential families in Statistics. Finally, we connect convex regularization functionals with convex risk measures through Fenchel conjugation. We do this by showing that if the source condition for the regularization functional is satisfied, then convex risk measures can be constructed.     

Saddle point methods,and alogorithms,for non-symmetric linear equations ∗

This paper describes methods for solving non-singular, non-symmetric linear equations whose symmetric part is positive definite. First, the solutions are characterized as saddle points of a convex-concave function. The associated primal and dual variational principles provide quadratic, strictly convex, functions whose minima are the solutions of the original equation and which generalize the energy function for symmetric problems.

Direct iterative methods for finding the saddle point are then developed and analyzed. A globally convergent algorithm for finding the saddle points is described. We show that requiring conjugacy of successive search directions with respect to the symmetric part of the equation is a poor strategy.     

Multi-kernel regularized classifiers

A family of classification algorithms generated from Tikhonov regularization schemes are considered. They involve multi-kernel spaces and general convex loss functions. Our main purpose is to provide satisfactory estimates for the excess misclassification error of these multi-kernel regularized classifiers when the loss functions achieve the zero value. The error analysis consists of two parts: regularization error and sample error. Allowing multi-kernels in the algorithm improves the regularization error and approximation error, which is one advantage of the multi-kernel setting. For a general loss function, we show how to bound the regularization error by the approximation in some weighted L^q$L^q$ spaces. For the sample error, we use a projection operator. The projection in connection with the decay of the regularization error enables us to improve convergence rates in the literature even for the one-kernel schemes and special loss functions: least-square loss and hinge loss for support vector machine soft margin classifiers. Existence of the optimization problem for the regularization scheme associated with multi-kernels is verified when the kernel functions are continuous with respect to the index set. Concrete examples, including Gaussian kernels with flexible variances and probability distributions with some noise conditions, are used to illustrate the general theory.     

A primal–dual regularized interior-point method for convex quadratic programs

Interior-point methods in augmented form for linear and convex quadratic programming require the solution of a sequence of symmetric indefinite linear systems which are used to derive search directions. Safeguards are typically required in order to handle free variables or rank-deficient Jacobians. We propose a consistent framework and accompanying theoretical justification for regularizing these linear systems. Our approach can be interpreted as a simultaneous proximal-point regularization of the primal and dual problems. The regularization is termedexact to emphasize that, although the problems are regularized, the algorithm recovers a solution of the original problem, for appropriate values of the regularization parameters.     

Implicit solution function of P0 and Z matrix linear complementarity constraints

Using the least element solution of the P₀ and Z matrix linear complementarity problem (LCP), we define an implicit solution function for linear complementarity constraints (LCC). We show that the sequence of solution functions defined by the unique solution of the regularized LCP is monotonically increasing and converges to the implicit solution function as the regularization parameter goes down to zero. Moreover, each component of the implicit solution function is convex. We find that the solution set of the irreducible P₀ and Z matrix LCP can be represented by the least element solution and a Perron?CFrobenius eigenvector. These results are applied to convex reformulation of mathematical programs with P₀ and Z matrix LCC. Preliminary numerical results show the effectiveness and the efficiency of the reformulation.     

Subdifferential of the supremum function: moving back and forth between continuous and non-continuous settings

In this paper we establish general formulas for the subdifferential of the pointwise supremum of convex functions, which cover and unify both the compact continuous and the non-compact non-continuous settings. From the non-continuous to the continuous setting, we proceed by a compactification-based approach which leads us to problems having compact index sets and upper semi-continuously indexed mappings, giving rise to new characterizations of the subdifferential of the supremum by means of upper semicontinuous regularized functions and an enlarged compact index set. In the opposite sense, we rewrite the subdifferential of these new regularized functions by using the original data, also leading us to new results on the subdifferential of the supremum. We give two applications in the last section, the first one concerning the nonconvex Fenchel duality, and the second one establishing Fritz-John and KKT conditions in convex semi-infinite programming.

     

Nonlinear separation approach for the augmented Lagrangian in nonlinear semidefinite programming

This paper aims at showing that the class of augmented Lagrangian functions for nonlinear semidefinite programming problems can be derived, as a particular case, from a nonlinear separation scheme in the image space associated with the given problem. By means of the image space analysis, a global saddle point condition for the augmented Lagrangian function is investigated. It is shown that the existence of a saddle point is equivalent to a regular nonlinear separation of two suitable subsets of the image space. Without requiring the strict complementarity, it is proved that, under second order sufficiency conditions, the augmented Lagrangian function admits a local saddle point. The existence of global saddle points is then obtained under additional assumptions that do not require the compactness of the feasible set. Motivated by the result on global saddle points, we propose two modified primal-dual methods based on the augmented Lagrangian using different strategies and prove their convergence to a global solution and the optimal value of the original problem without requiring the boundedness condition of the multiplier sequence.     

Regularization of the Cauchy problem for the Laplace equation

We suggest a method for regularizing the solution of the Cauchy problem for the Laplace equation by introducing the biharmonic operator with a small parameter. We show that if there exists a solution of the original problem, then the difference between the spectral expansions of solutions of the original and regularized equations tends to zero in the space of square integrable functions as the regularization parameter tends to zero. If the original solution belongs to a Sobolev class, then we use results of Il’in’s spectral theory to derive an estimate for the rate of the convergence of the regularized solution to the exact solution.     

Gap Function Approach to the Generalized Nash Equilibrium Problem

We consider an optimization reformulation approach for the generalized Nash equilibrium problem (GNEP) that uses the regularized gap function of a quasi-variational inequality (QVI). The regularized gap function for QVI is in general not differentiable, but only directionally differentiable. Moreover, a simple condition has yet to be established, under which any stationary point of the regularized gap function solves the QVI. We tackle these issues for the GNEP in which the shared constraints are given by linear equalities, while the individual constraints are given by convex inequalities. First, we formulate the minimization problem involving the regularized gap function and show the equivalence to GNEP. Next, we establish the differentiability of the regularized gap function and show that any stationary point of the minimization problem solves the original GNEP under some suitable assumptions. Then, by using a barrier technique, we propose an algorithm that sequentially solves minimization problems obtained from GNEPs with the shared equality constraints only. Further, we discuss the case of shared inequality constraints and present an algorithm that utilizes the transformation of the inequality constraints to equality constraints by means of slack variables. We present some results of numerical experiments to illustrate the proposed approach.     

Epi-convergence Properties of Smoothing by Infimal Convolution

This paper concerns smoothing by infimal convolution for two large classes of functions: convex, proper and lower semicontinous as well as for (the nonconvex class of) convex-composite functions. The smooth approximations are constructed so that they epi-converge (to the underlying nonsmooth function) and fulfill a desirable property with respect to graph convergence of the gradient mappings to the subdifferential of the original function under reasonable assumptions. The close connection between epi-convergence of the smoothing functions and coercivity properties of the smoothing kernel is established.     

Global Uniqueness and Solvability of Tensor Variational Inequalities

We consider the extragradient method to minimize the sum of two functions, the first one being smooth and the second being convex. Under the Kurdyka–?ojasiewicz assumption, we prove that the sequence produced by the extragradient method converges to a critical point of the problem and has finite length. The analysis is extended to the case when both functions are convex. We provide, in this case, a sublinear convergence rate, as for gradient-based methods. Furthermore, we show that the recent small-prox complexity result can be applied to this method. Considering the extragradient method is an occasion to describe an exact line search scheme for proximal decomposition methods. We provide details for the implementation of this scheme for the one-norm regularized least squares problem and demonstrate numerical results which suggest that combining nonaccelerated methods with exact line search can be a competitive choice.     

Moreau–Yosida regularization of Lagrangian-dual functions for a class of convex optimization problems

In this paper, we consider the Lagrangian dual problem of a class of convex optimization problems, which originates from multi-stage stochastic convex nonlinear programs. We study the Moreau–Yosida regularization of the Lagrangian-dual function and prove that the regularized function η is piecewise C ², in addition to the known smoothness property. This property is then used to investigate the semismoothness of the gradient mapping of the regularized function. Finally, we show that the Clarke generalized Jacobian of the gradient mapping is BD-regular under some conditions.      

Interior proximal methods and central paths for convex second-order cone programming

We make a unified analysis of interior proximal methods of solving convex second-order cone programming problems. These methods use a proximal distance with respect to second-order cones which can be produced with an appropriate closed proper univariate function in three ways. Under some mild conditions, the sequence generated is bounded with each limit point being a solution, and global rates of convergence estimates are obtained in terms of objective values. A class of regularized proximal distances is also constructed which can guarantee the global convergence of the sequence to an optimal solution. These results are illustrated with some examples. In addition, we also study the central paths associated with these distance-like functions, and for the linear SOCP we discuss their relations with the sequence generated by the interior proximal methods. From this, we obtain improved convergence results for the sequence for the interior proximal methods using a proximal distance continuous at the boundary of second-order cones.     

On generalized equilibrium points

Generalizations of the usual definition of saddle point and equilibrium point are introduced in this paper. The existence of these points is shown to be related to a class of functions that we call perturbed convex functions. First and second order conditions regarding the existence of these points are also proved.     

Regularized state-constrained boundary optimal control of the Navier-Stokes equations

The numerical solution of the Dirichlet boundary optimal control problem of the Navier-Stokes equations in presence of pointwise state constraints is investigated. Two different regularization techniques are considered. First, a Moreau-Yosida regularization of the problem is studied. Optimality conditions are derived and the convergence of the regularized solutions towards the original one is proved. A source representation of the control combined with a Lavrentiev type regularization strategy is also presented. The analysis concerning optimality conditions and convergence of the regularized solutions is carried out. In the last part of the paper numerical experiments are presented. For the numerical solution of each regularized problem a semi-smooth Newton method is applied.     

Lagrange multipliers, (exact) regularization and error bounds for monotone variational inequalities

We examine two central regularization strategies for monotone variational inequalities, the first a direct regularization of the operative monotone mapping, and the second via regularization of the associated dual gap function. A key link in the relationship between the solution sets to these various regularized problems is the idea of exact regularization, which, in turn, is fundamentally associated with the existence of Lagrange multipliers for the regularized variational inequality. A regularization is said to be exact if a solution to the regularized problem is a solution to the unregularized problem for all parameters beyond a certain value. The Lagrange multipliers corresponding to a particular regularization of a variational inequality, on the other hand, are defined via the dual gap function. Our analysis suggests various conceptual, iteratively regularized numerical schemes, for which we provide error bounds, and hence stopping criteria, under the additional assumption that the solution set to the unregularized problem is what we call weakly sharp of order greater than one.