Using approximate secant equations in limited memory methods for multilevel unconstrained optimization

The properties of multilevel optimization problems defined on a hierarchy of discretization grids can be used to define approximate secant equations, which describe the second-order behavior of the objective function. Following earlier work by Gratton and Toint (2009) we introduce a quasi-Newton method (with a linesearch) and a nonlinear conjugate gradient method that both take advantage of this new second-order information. We then present numerical experiments with these methods and formulate recommendations for their practical use.     

Recognizing underlying sparsity in optimization

Exploiting sparsity is essential to improve the efficiency of solving large optimization problems. We present a method for recognizing the underlying sparsity structure of a nonlinear partially separable problem, and show how the sparsity of the Hessian matrices of the problem’s functions can be improved by performing a nonsingular linear transformation in the space corresponding to the vector of variables. A combinatorial optimization problem is then formulated to increase the number of zeros of the Hessian matrices in the resulting transformed space, and a heuristic greedy algorithm is applied to this formulation. The resulting method can thus be viewed as a preprocessor for converting a problem with hidden sparsity into one in which sparsity is explicit. When it is combined with the sparse semidefinite programming relaxation by Waki et al. for polynomial optimization problems, the proposed method is shown to extend the performance and applicability of this relaxation technique. Preliminary numerical results are presented to illustrate this claim. S. Kim’s research was supported by Kosef R01-2005-000-10271-0. M. Kojima’s research was supported by Grant-in-Aid for Scientific Research on Priority Areas 16016234.     

Componentwise fast convergence in the solution of full-rank systems of nonlinear equations

The asymptotic convergence of parameterized variants of Newton’s method for the solution of nonlinear systems of equations is considered. The original system is perturbed by a term involving the variables and a scalar parameter which is driven to zero as the iteration proceeds. The exact local solutions to the perturbed systems then form a differentiable path leading to a solution of the original system, the scalar parameter determining the progress along the path. A path-following algorithm, which involves an inner iteration in which the perturbed systems are approximately solved, is outlined. It is shown that asymptotically, a single linear system is solved per update of the scalar parameter. It turns out that a componentwise Q-superlinear rate may be attained, both in the direct error and in the residuals, under standard assumptions, and that this rate may be made arbitrarily close to quadratic. Numerical experiments illustrate the results and we discuss the relationships that this method shares with interior methods in constrained optimization. Received: September 8, 2000 / Accepted: September 17, 2001?Published online February 14, 2002     

Forcing sparsity by projecting with respect to a non-diagonally weighted frobenius norm

This note deals with the computational problem of determining the projection of a given symmetric matrix onto the subspace of symmetric matrices that have a fixed sparsity pattern. This projection is performed with respect to a weighted Frobenius norm involving a metric that is not diagonal. It is shown that the solution to this question is computationally feasible when the metric appearing in the norm is a low rank modification to the identity. Also, generalization to perturbations of higher rank is shown to be increasingly costly in terms of computation.     

Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould and Toint (Part I, Math Program, doi:, 2009), generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser, Deuflhard and Erdmann (Optim Methods Softw 22(3):413–431, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most O(e^-3/2){\mathcal{O}(\epsilon^{-3/2})} iterations, or equivalently, function- and gradient-evaluations, to drive the norm of the gradient of the objective below the desired accuracy e{\epsilon}, and O(e^-3){\mathcal{O}(\epsilon^{-3})} iterations, to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved for Algorithm 3.3 of Nesterov and Polyak which minimizes the cubic model globally on each iteration. Our approach is more general in that it allows the cubic model to be solved only approximately and may employ approximate Hessians.     

A primal-dual trust-region algorithm for non-convex nonlinear programming

A new primal-dual algorithm is proposed for the minimization of non-convex objective functions subject to general inequality and linear equality constraints. The method uses a primal-dual trust-region model to ensure descent on a suitable merit function. Convergence is proved to second-order critical points from arbitrary starting points. Numerical results are presented for general quadratic programs. Received: July 19, 1999 / Accepted: February 1, 2000?Published online March 15, 2000     

On the accelerating property of an algorithm for function minimization without calculating derivatives

This paper studies the speed of convergence of a general algorithm for function minimization without calculating derivatives. This algorithm contains Powell's 1964 algorithm as well as Zangwill's second modification of this procedure. The main results are Theorems 3.1 and 4.1 which show that, if the algorithm behaves well, then asymptotically almost conjugate directions are built; therefore, the algorithm has an every-iteration superlinear speed of convergence. The paper hinges on ideas of McCormick and Ritter and Powell.The authors wish to thank the Namur Department of Mathematics, especially its optimization group, for many discussions and encouragements. The authors also thank the reviewer for many helpful suggestions.     

On the use of an inverse shortest paths algorithm for recovering linearly correlated costs

This paper considers the inverse shortest paths problem where arc costs are subject to correlation constraints. The motivation for this research arises from applications in traffic modelling and seismic tomography. A new method is proposed for solving this class of problems. It is constructed as a generalization of the algorithm presented in Burton and Toint (Mathematical Programming 53, 1992) for uncorrelated inverse shortest paths. Preliminary numerical experience with the new method is presented and discussed.     

Global convergence of the partitioned BFGS algorithm for convex partially separable optimization

Global convergence is proved for a partitioned BFGS algorithm, when applied on a partially separable problem with a convex decomposition. This case convers a known practical optimization method for large dimensional unconstrained problems. Inexact solution of the linear system defining the search direction and variants of the steplength rule are also shown to be acceptable without affecting the global convergence properties.