Multi-step nonlinear conjugate gradient methods for unconstrained minimization

Conjugate gradient methods are appealing for large scale nonlinear optimization problems, because they avoid the storage of matrices. Recently, seeking fast convergence of these methods, Dai and Liao (Appl. Math. Optim. 43:87–101, 2001) proposed a conjugate gradient method based on the secant condition of quasi-Newton methods, and later Yabe and Takano (Comput. Optim. Appl. 28:203–225, 2004) proposed another conjugate gradient method based on the modified secant condition. In this paper, we make use of a multi-step secant condition given by Ford and Moghrabi (Optim. Methods Softw. 2:357–370, 1993; J. Comput. Appl. Math. 50:305–323, 1994) and propose two new conjugate gradient methods based on this condition. The methods are shown to be globally convergent under certain assumptions. Numerical results are reported.     

Variational discretization of Lavrentiev-regularized state constrained elliptic optimal control problems

In the present work, we apply a variational discretization proposed by the first author in (Comput. Optim. Appl. 30:45–61, 2005) to Lavrentiev-regularized state constrained elliptic control problems. We extend the results of (Comput. Optim. Appl. 33:187–208, 2006) and prove weak convergence of the adjoint states and multipliers of the regularized problems to their counterparts of the original problem. Further, we prove error estimates for finite element discretizations of the regularized problem and investigate the overall error imposed by the finite element discretization of the regularized problem compared to the continuous solution of the original problem. Finally we present numerical results which confirm our analytical findings.     

Global convergence of a modified Hestenes-Stiefel nonlinear conjugate gradient method with Armijo line search

In this article, based on the modified secant equation, we propose a modified Hestenes-Stiefel (HS) conjugate gradient method which has similar form as the CG-DESCENT method proposed by Hager and Zhang (SIAM J Optim 16:170–192, 2005). The presented method can generate sufficient descent directions without any line search. Under some mild conditions, we show that it is globally convergent with Armijo line search. Moreover, the R-linear convergence rate of the modified HS method is established. Preliminary numerical results show that the proposed method is promising, and competitive with the well-known CG-DESCENT method.     

A priori error estimates for space–time finite element discretization of semilinear parabolic optimal control problems

In this paper, a priori error estimates for space–time finite element discretizations of optimal control problems governed by semilinear parabolic PDEs and subject to pointwise control constraints are derived. We extend the approach from Meidner and Vexler (SIAM Control Optim 47(3):1150–1177, 2008; SIAM Control Optim 47(3):1301–1329, 2008) where linear-quadratic problems have been considered, discretizing the state equation by usual conforming finite elements in space and a discontinuous Galerkin method in time. Error estimates for controls discretized by piecewise constant functions in time and cellwise constant functions in space are derived in detail and we explain how error estimate for further discretization approaches, e.g., cellwise linear discretization in space, the postprocessing approach from Meyer and R?sch (SIAM J Control Optim 43:970–985, 2004), and the variationally discrete approach from Hinze (J Comput Optim Appl 30:45–63, 2005) can be obtained. In addition, we derive an estimate for a setting with finitely many time-dependent controls.     

Error estimates for abstract linear–quadratic optimal control problems using proper orthogonal decomposition

In this paper we investigate POD discretizations of abstract linear–quadratic optimal control problems with control constraints. We apply the discrete technique developed by Hinze (Comput. Optim. Appl. 30:45–61, 2005) and prove error estimates for the corresponding discrete controls, where we combine error estimates for the state and the adjoint system from Kunisch and Volkwein (Numer. Math. 90:117–148, 2001; SIAM J. Numer. Anal. 40:492–515, 2002). Finally, we present numerical examples that illustrate the theoretical results.     

Feasible Method for Generalized Semi-Infinite Programming

In this paper, we analyze the outer approximation property of the algorithm for generalized semi-infinite programming from Stein and Still (SIAM J. Control Optim. 42:769–788, 2003). A simple bound on the regularization error is found and used to formulate a feasible numerical method for generalized semi-infinite programming with convex lower-level problems. That is, all iterates of the numerical method are feasible points of the original optimization problem. The new method has the same computational cost as the original algorithm from Stein and Still (SIAM J. Control Optim. 42:769–788, 2003). We also discuss the merits of this approach for the adaptive convexification algorithm, a feasible point method for standard semi-infinite programming from Floudas and Stein (SIAM J. Optim. 18:1187–1208, 2007).     

Global convergence of some modified PRP nonlinear conjugate gradient methods

Recently, similar to Hager and Zhang (SIAM J Optim 16:170–192, 2005), Yu (Nonlinear self-scaling conjugate gradient methods for large-scale optimization problems. Thesis of Doctors Degree, Sun Yat-Sen University, 2007) and Yuan (Optim Lett 3:11–21, 2009) proposed modified PRP conjugate gradient methods which generate sufficient descent directions without any line searches. In order to obtain the global convergence of their algorithms, they need the assumption that the stepsize is bounded away from zero. In this paper, we take a little modification to these methods such that the modified methods retain sufficient descent property. Without requirement of the positive lower bound of the stepsize, we prove that the proposed methods are globally convergent. Some numerical results are also reported.     

Truncated regularized Newton method for convex minimizations

Recently, Li et al. (Comput. Optim. Appl. 26:131–147, 2004) proposed a regularized Newton method for convex minimization problems. The method retains local quadratic convergence property without requirement of the singularity of the Hessian. In this paper, we develop a truncated regularized Newton method and show its global convergence. We also establish a local quadratic convergence theorem for the truncated method under the same conditions as those in Li et al. (Comput. Optim. Appl. 26:131–147, 2004). At last, we test the proposed method through numerical experiments and compare its performance with the regularized Newton method. The results show that the truncated method outperforms the regularized Newton method. The work was supported by the 973 project granted 2004CB719402 and the NSF project of China granted 10471036.     

Decomposition-based Method for Sparse Semidefinite Relaxations of Polynomial Optimization Problems

We consider polynomial optimization problems pervaded by a sparsity pattern. It has been shown in Lasserre (SIAM J. Optim. 17(3):822–843, 2006) and Waki et al. (SIAM J. Optim. 17(1):218–248, 2006) that the optimal solution of a polynomial programming problem with structured sparsity can be computed by solving a series of semidefinite relaxations that possess the same kind of sparsity. We aim at solving the former relaxations with a decomposition-based method, which partitions the relaxations according to their sparsity pattern. The decomposition-based method that we propose is an extension to semidefinite programming of the Benders decomposition for linear programs (Benders, Comput. Manag. Sci. 2(1):3–19, 2005).     

Further Results on the Robust Regulation of a One-Dimensional Dam-River System

In this paper, we consider a one-dimensional dam-river system studied by Chentouf and Wang (SIAM J. Control Optim. 47: 2275–2302, 2008). Then, using the frequency multiplier method, we provide a simple and alternative proof of stabilization and regulation results obtained in the work cited above. Moreover, we relax the conditions on the feedback gains involved in the feedback law and give a partial answer to the open problem left by the authors Chentouf and Wang (J. Optim. Theory Appl. 134: 223–239, 2007 and SIAM J. Control Optim. 47: 2275–2302, 2008) concerning the tuning of the gains.     

A block coordinate gradient descent method for regularized convex separable optimization and covariance selection

We consider a class of unconstrained nonsmooth convex optimization problems, in which the objective function is the sum of a convex smooth function on an open subset of matrices and a separable convex function on a set of matrices. This problem includes the covariance selection problem that can be expressed as an ℓ ₁-penalized maximum likelihood estimation problem. In this paper, we propose a block coordinate gradient descent method (abbreviated as BCGD) for solving this class of nonsmooth separable problems with the coordinate block chosen by a Gauss-Seidel rule. The method is simple, highly parallelizable, and suited for large-scale problems. We establish global convergence and, under a local Lipschizian error bound assumption, linear rate of convergence for this method. For the covariance selection problem, the method can terminate in O(n³/e){O(n^3/\epsilon)} iterations with an e{\epsilon}-optimal solution. We compare the performance of the BCGD method with the first-order methods proposed by Lu (SIAM J Optim 19:1807–1827, 2009; SIAM J Matrix Anal Appl 31:2000–2016, 2010) for solving the covariance selection problem on randomly generated instances. Our numerical experience suggests that the BCGD method can be efficient for large-scale covariance selection problems with constraints.     

On local convergence of sequential quadratically-constrained quadratic-programming type methods,with an extension to variational problems

We consider the class of quadratically-constrained quadratic-programming methods in the framework extended from optimization to more general variational problems. Previously, in the optimization case, Anitescu (SIAM J. Optim. 12, 949–978, 2002) showed superlinear convergence of the primal sequence under the Mangasarian-Fromovitz constraint qualification and the quadratic growth condition. Quadratic convergence of the primal-dual sequence was established by Fukushima, Luo and Tseng (SIAM J. Optim. 13, 1098–1119, 2003) under the assumption of convexity, the Slater constraint qualification, and a strong second-order sufficient condition. We obtain a new local convergence result, which complements the above (it is neither stronger nor weaker): we prove primal-dual quadratic convergence under the linear independence constraint qualification, strict complementarity, and a second-order sufficiency condition. Additionally, our results apply to variational problems beyond the optimization case. Finally, we provide a necessary and sufficient condition for superlinear convergence of the primal sequence under a Dennis-Moré type condition. Research of the second author is partially supported by CNPq Grants 300734/95-6 and 471780/2003-0, by PRONEX–Optimization, and by FAPERJ.     

The convergence rate of a three-term HS method with restart strategy for unconstrained optimization problems

Although the Hesteness and Stiefel (HS) method is a well-known method, if an inexact line search is used, researches about its convergence rate are very rare. Recently, Zhang, Zhou and Li [Some descent three-term conjugate gradient methods and their global convergence, Optim. Method Softw. 22 (2007), pp. 697–711] proposed a three-term Hestenes–Stiefel method for unconstrained optimization problems. In this article, we investigate the convergence rate of this method. We show that the three-term HS method with the Wolfe line search will be n-step superlinearly and even quadratically convergent if some restart technique is used under reasonable conditions. Some numerical results are also reported to verify the theoretical results. Moreover, it is more efficient than the previous ones.     

Quasiconvex Minimization on a Locally Finite Union of Convex Sets

Extending the approach initiated in Aussel and Hadjisavvas (SIAM J. Optim. 16:358–367, 2005) and Aussel and Ye (Optimization 55:433–457, 2006), we obtain the existence of a local minimizer of a quasiconvex function on the locally finite union of closed convex subsets of a Banach space. We apply the existence result to some difficult nonconvex optimization problems such as the disjunctive programming problem and the bilevel programming problem. Dedicated to Jean-Pierre Crouzeix on the occasion of his 65th birthday. The authors thank two anonymous referees for careful reading of the paper and helpful suggestions. The research of the second author was partially supported by NSERC/Canada.     

A Smoothing Newton Method with Fischer-Burmeister Function for Second-Order Cone Complementarity Problems

The second-order cone complementarity problem (SOCCP) is an important class of problems containing a lot of optimization problems. The SOCCP can be transformed into a system of nonsmooth equations. To solve this nonsmooth system, smoothing techniques are often used. Fukushima, Luo and Tseng (SIAM J. Optim. 12:436–460, 2001) studied concrete theories and properties of smoothing functions for the SOCCP. Recently, a practical computational method using the smoothed natural residual function to solve the SOCCP was given by Chen, Sun and Sun (Comput. Optim. Appl. 25:39–56, 2003). In the present paper, we propose an algorithm to solve the SOCCP by using the smoothed Fischer-Burmeister function. Some preliminary numerical results are given.     

Improved Full-Newton Step <Emphasis Type="Italic">O</Emphasis>(<Emphasis Type="Italic">nL</Emphasis>) Infeasible Interior-Point Method for Linear Optimization

We present several improvements of the full-Newton step infeasible interior-point method for linear optimization introduced by Roos (SIAM J. Optim. 16(4):1110–1136, 2006). Each main step of the method consists of a feasibility step and several centering steps. We use a more natural feasibility step, which targets the μ ⁺-center of the next pair of perturbed problems. As for the centering steps, we apply a sharper quadratic convergence result, which leads to a slightly wider neighborhood for the feasibility steps. Moreover, the analysis is much simplified and the iteration bound is slightly better.     

On the convergence of Newton-type methods under mild differentiability conditions

We introduce the new idea of recurrent functions to provide a new semilocal convergence analysis for Newton-type methods, under mild differentiability conditions. It turns out that our sufficient convergence conditions are weaker, and the error bounds are tighter than in earlier studies in some interesting cases (Chen, Ann Inst Stat Math 42:387–401, 1990; Chen, Numer Funct Anal Optim 10:37–48, 1989; Cianciaruso, Numer Funct Anal Optim 24:713–723, 2003; Cianciaruso, Nonlinear Funct Anal Appl 2009; Dennis 1971; Deuflhard 2004; Deuflhard, SIAM J Numer Anal 16:1–10, 1979; Gutiérrez, J Comput Appl Math 79:131–145, 1997; Hernández, J Optim Theory Appl 109:631–648, 2001; Hernández, J Comput Appl Math 115:245–254, 2000; Huang, J Comput Appl Math 47:211–217, 1993; Kantorovich 1982; Miel, Numer Math 33:391–396, 1979; Miel, Math Comput 34:185–202, 1980; Moret, Computing 33:65–73, 1984; Potra, Libertas Mathematica 5:71–84, 1985; Rheinboldt, SIAM J Numer Anal 5:42–63, 1968; Yamamoto, Numer Math 51: 545–557, 1987; Zabrejko, Numer Funct Anal Optim 9:671–684, 1987; Zinc̆ko 1963). Applications and numerical examples, involving a nonlinear integral equation of Chandrasekhar-type, and a differential equation are also provided in this study.     

Proximal-Like Algorithm Using the Quasi D-Function for Convex Second-Order Cone Programming

In this paper, we present a measure of distance in a second-order cone based on a class of continuously differentiable strictly convex functions on ℝ₊₊. Since the distance function has some favorable properties similar to those of the D-function (Censor and Zenios in J. Optim. Theory Appl. 73:451–464 [1992]), we refer to it as a quasi D-function. Then, a proximal-like algorithm using the quasi D-function is proposed and applied to the second-cone programming problem, which is to minimize a closed proper convex function with general second-order cone constraints. Like the proximal point algorithm using the D-function (Censor and Zenios in J. Optim. Theory Appl. 73:451–464 [1992]; Chen and Teboulle in SIAM J. Optim. 3:538–543 [1993]), under some mild assumptions we establish the global convergence of the algorithm expressed in terms of function values; we show that the sequence generated by the proposed algorithm is bounded and that every accumulation point is a solution to the considered problem. Research of Shaohua Pan was partially supported by the Doctoral Starting-up Foundation (B13B6050640) of GuangDong Province. Jein-Shan Chen is a member of the Mathematics Division, National Center for Theoretical Sciences, Taipei Office. The author’s work was partially supported by National Science Council of Taiwan.     

On the Surrogate Gradient Algorithm for Lagrangian Relaxation

When applied to large-scale separable optimization problems, the recently developed surrogate subgradient method for Lagrangian relaxation (Zhao et al.: J. Optim. Theory Appl. 100, 699–712, 1999) does not need to solve optimally all the subproblems to update the multipliers, as the traditional subgradient method requires. Based on it, the penalty surrogate subgradient algorithm was further developed to address the homogenous solution issue (Guan et al.: J. Optim. Theory Appl. 113, 65–82, 2002; Zhai et al.: IEEE Trans. Power Syst. 17, 1250–1257, 2002). There were flaws in the proofs of Zhao et al., Guan et al., and Zhai et al.: for problems with inequality constraints, projection is necessary to keep the multipliers nonnegative; however, the effects of projection were not properly considered. This note corrects the flaw, completes the proofs, and asserts the correctness of the methods. This work is supported by the NSFC Grant Nos. 60274011, 60574067, the NCET program (No. NCET-04-0094) of China. The third author was supported in part by US National Science Foundation under Grants ECS-0323685 and DMI-0423607.     

On Convergence Results for Weak Efficiency in Vector Optimization Problems with Equilibrium Constraints

In this note, we prove that the convergence results for vector optimization problems with equilibrium constraints presented in Wu and Cheng (J. Optim. Theory Appl. 125, 453–472, 2005) are not correct. Actually, we show that results of this type cannot be established at all. This is due to the possible lack, even under nice assumptions, of lower convergence of the solution map for equilibrium problems, already deeply investigated in Loridan and Morgan (Optimization 20, 819–836, 1989) and Lignola and Morgan (J. Optim. Theory Appl. 93, 575–596, 1997).