Two classes of multisecant methods for nonlinear acceleration

Many applications in science and engineering lead to models that require solving large‐scale fixed point problems, or equivalently, systems of nonlinear equations. Several successful techniques for handling such problems are based on quasi‐Newton methods that implicitly update the approximate Jacobian or inverse Jacobian to satisfy a certain secant condition. We present two classes of multisecant methods which allow to take into account a variable number of secant equations at each iteration. The first is the Broyden‐like class, of which Broyden's family is a subclass, and Anderson mixing is a particular member. The second class is that of the nonlinear Eirola–Nevanlinna‐type methods. This work was motivated by a problem in electronic structure calculations, whereby a fixed point iteration, known as the self‐consistent field (SCF) iteration, is accelerated by various strategies termed ‘mixing’. Copyright © 2008 John Wiley & Sons, Ltd.     

On compact vector formats in the solution of the chemical master equation with backward differentiation

A stochastic chemical system with multiple types of molecules interacting through reaction channels can be modeled as a continuous‐time Markov chain with a countably infinite multidimensional state space. Starting from an initial probability distribution, the time evolution of the probability distribution associated with this continuous‐time Markov chain is described by a system of ordinary differential equations, known as the chemical master equation (CME). This paper shows how one can solve the CME using backward differentiation. In doing this, a novel approach to truncate the state space at each time step using a prediction vector is proposed. The infinitesimal generator matrix associated with the truncated state space is represented compactly, and exactly, using a sum of Kronecker products of matrices associated with molecules. This exact representation is already compact and does not require a low‐rank approximation in the hierarchical Tucker decomposition (HTD) format. During transient analysis, compact solution vectors in HTD format are employed with the exact, compact, and truncated generated matrices in Kronecker form, and the linear systems are solved with the Jacobi method using fixed or adaptive rank control strategies on the compact vectors. Results of simulation on benchmark models are compared with those of the proposed solver and another version, which works with compact vectors and highly accurate low‐rank approximations of the truncated generator matrices in quantized tensor train format and solves the linear systems with the density matrix renormalization group method. Results indicate that there is a reason to solve the CME numerically, and adaptive rank control strategies on compact vectors in HTD format improve time and memory requirements significantly.     

Optimally conditioned scaled ABS algorithms for linear systems

Using a strict bound of Spedicato to the condition number of bordered positive-definite matrices, we show that the scaling parameter in the ABS class for linear systems can always be chosen so that the bound of a certain update matrix is globally minimized. Moreover, if the scaling parameter is so chosen at every iteration, then the condition number itself is globally minimized. The resulting class of optimally conditioned algorithms contains as a special case the class of optimally stable algorithms in the sense of Broyden.This work was done in the framework of research supported by MPI, Rome, Italy, 60% Program.     

Computational experience with known variable metric updates

Variable metric methods from the Broyden family are well known and commonly used for unconstrained minimization. These methods have good theoretical and practical convergence properties which depend on a selection of free parameters. We demonstrate, using extensive computational experiments, the influence of both the Biggs stabilization parameter and Oren scaling parameter on 12 individual variable metric updates, two of which are new. This paper focuses on a class of variable metric updates belonging to the so-called preconvex part of the Broyden family. These methods outperform the more familiar BFGS method. We also experimentally demonstrate the efficiency of the controlled scaling strategy for problems of sufficient size and sparsity.     

On the superlinear convergence of the variable metric proximal point algorithm using Broyden and BFGS matrix secant updating

In previous work, the authors provided a foundation for the theory of variable metric proximal point algorithms in Hilbert space. In that work conditions are developed for global, linear, and super–linear convergence. This paper focuses attention on two matrix secant updating strategies for the finite dimensional case. These are the Broyden and BFGS updates. The BFGS update is considered for application in the symmetric case, e.g., convex programming applications, while the Broyden update can be applied to general monotone operators. Subject to the linear convergence of the iterates and a quadratic growth condition on the inverse of the operator at the solution, super–linear convergence of the iterates is established for both updates. These results are applied to show that the Chen–Fukushima variable metric proximal point algorithm is super–linearly convergent when implemented with the BFGS update. Received: September 12, 1996 / Accepted: January 7, 2000?Published online March 15, 2000     

Matrix conditioning and nonlinear optimization

In a series of recent papers, Oren, Oren and Luenberger, Oren and Spedicato, and Spedicato have developed the self-scaling variable metric algorithms. These algorithms alter Broyden's single parameter family of approximations to the inverse Hessian to a double parameter family. Conditions are given on the new parameter to minimize a bound on the condition number of the approximated inverse Hessian while insuring improved step-wise convergence.Davidon has devised an update which also minimizes the bound on the condition number while remaining in the Broyden single parameter family.This paper derives initial scalings for the approximate inverse Hessian which makes members of the Broyden class self-scaling. The Davidon, BFGS, and Oren—Spedicato updates are tested for computational efficiency and stability on numerous test functions, with the results indicating strong superiority computationally for the Davidon and BFGS update over the self-scaling update, except on a special class of functions, the homogeneous functions.     

Representations of quasi-Newton matrices and their use in limited memory methods

We derive compact representations of BFGS and symmetric rank-one matrices for optimization. These representations allow us to efficiently implement limited memory methods for large constrained optimization problems. In particular, we discuss how to compute projections of limited memory matrices onto subspaces. We also present a compact representation of the matrices generated by Broyden's update for solving systems of nonlinear equations.These authors were supported by the Air Force Office of Scientific Research under Grant AFOSR-90-0109, the Army Research Office under Grant DAAL03-91-0151 and the National Science Foundation under Grants CCR-8920519 and CCR-9101795.This author was supported by the U.S. Department of Energy, under Grant DE-FG02-87ER25047-A001, and by National Science Foundation Grants CCR-9101359 and ASC-9213149.     

When is a matrix unitary or Hermitian plus low rank?

Hermitian and unitary matrices are two representatives of the class of normal matrices whose full eigenvalue decomposition can be stably computed in quadratic computing complexity once the matrix has been reduced, for instance, to tridiagonal or Hessenberg form. Recently, fast and reliable eigensolvers dealing with low‐rank perturbations of unitary and Hermitian matrices have been proposed. These structured eigenvalue problems appear naturally when computing roots, via confederate linearizations, of polynomials expressed in, for example, the monomial or Chebyshev basis. Often, however, it is not known beforehand whether or not a matrix can be written as the sum of a Hermitian or unitary matrix plus a low‐rank perturbation. In this paper, we give necessary and sufficient conditions characterizing the class of Hermitian or unitary plus low‐rank matrices. The number of singular values deviating from 1 determines the rank of a perturbation to bring a matrix to unitary form. A similar condition holds for Hermitian matrices; the eigenvalues of the skew‐Hermitian part differing from 0 dictate the rank of the perturbation. We prove that these relations are linked via the Cayley transform. Then, based on these conditions, we identify the closest Hermitian or unitary plus rank k matrix to a given matrix A, in Frobenius and spectral norm, and give a formula for their distance from A. Finally, we present a practical iteration to detect the low‐rank perturbation. Numerical tests prove that this straightforward algorithm is effective.     

A preconditioned low‐rank CG method for parameter‐dependent Lyapunov matrix equations

This paper is concerned with the numerical solution of symmetric large‐scale Lyapunov equations with low‐rank right‐hand sides and coefficient matrices depending on a parameter. Specifically, we consider the situation when the parameter dependence is sufficiently smooth, and the aim is to compute solutions for many different parameter samples. On the basis of existing results for Lyapunov equations and parameter‐dependent linear systems, we prove that the tensor containing all solution samples typically allows for an excellent low multilinear rank approximation. Stacking all sampled equations into one huge linear system, this fact can be exploited by combining the preconditioned CG method with low‐rank truncation. Our approach is flexible enough to allow for a variety of preconditioners based, for example, on the sign function iteration or the alternating direction implicit method. Copyright © 2013 John Wiley & Sons, Ltd.     

Adjoint-based quasi-Newton methods for partially separable problems

In this paper the concepts of partitioned quasi-Newton methods are applied to adjoint Broyden updates. Consequently a corresponding partitioned adjoint Broyden update is presented and local convergence results are given. Numerical results compare the partitioned adjoint Broyden update methods to the corresponding unpartitioned quasi-Newton method and to Newton's method for nonlinear equations. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Symmetric minimum-norm updates for use in gibbs free energy calculations

This paper examines a type of symmetric quasi-Newton update for use in nonlinear optimization algorithms. The updates presented here impose additional properties on the Hessian approximations that do not result if the usual quasi-Newton updating schemes are applied to certain Gibbs free energy minimization problems. The updates derived in this paper are symmetric matrices that satisfy a given matrix equation and are least squares solutions to the secant equation. A general representation for this class of updates is given. The update in this class that has the minimum weighted Frobenius norm is also presented. This work was done at Sandia National Laboratories and supported by the US Dept. of Energy under contract no. DE-AC04-76DP00789.     

Optimal parameters in the HSS‐like methods for saddle‐point problems

For the Hermitian and skew‐Hermitian splitting iteration method and its accelerated variant for solving the large sparse saddle‐point problems, we compute their quasi‐optimal iteration parameters and the corresponding quasi‐optimal convergence factors for the more practical but more difficult case that the (1, 1)‐block of the saddle‐point matrix is not algebraically equivalent to the identity matrix. In addition, the algebraic behaviors and the clustering properties of the eigenvalues of the preconditioned matrices with respect to these two iterations are investigated in detail, and the formulas for computing good iteration parameters are given under certain principle for optimizing the distribution of the eigenvalues. Copyright © 2008 John Wiley & Sons, Ltd.     

Coloring update methods

For linear update methods (such as SOR), a coloring method is introduced for which the multicolor iteration matrix has the same spectrum as the original iteration matrix. It applies to general linear systems, not necessarily arising from PDEs. When the iteration matrices are nonsingular, it is shown that they are similar to each other.     

On the effect of different parameterizations of the flow rule on the efficiency of variational constitutive updates

The numerical efficiency of so-called variational constitutive updates for finite strain plasticity theory is analyzed. These updates compute the unknowns such as the plastic strains by minimizing an appropriate functional. Within the present paper, different parameterizations of the flow rule are utilized within the variational constitutive update scheme. It is shown that comparing to the return-mapping algorithm, the variational updates require significantly less iteration steps and thus, is numerically highly efficient. (© 2010 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Family of optimally conditioned quasi-Newton updates for unconstrained optimization

In this note, a general optimal conditioning problem for updates which satisfy the quasi-Newton equation is solved. The new solution is a family of updates which contains other known optimally conditioned updates but also includes new formulas of increased rank. A new factorization formula for the Broyden family and some preliminary numerical results are also given.     

Radial basis collocation method and quasi‐Newton iteration for nonlinear elliptic problems

This work presents a radial basis collocation method combined with the quasi‐Newton iteration method for solving semilinear elliptic partial differential equations. The main result in this study is that there exists an exponential convergence rate in the radial basis collocation discretization and a superlinear convergence rate in the quasi‐Newton iteration of the nonlinear partial differential equations. In this work, the numerical error associated with the employed quadrature rule is considered. It is shown that the errors in Sobolev norms for linear elliptic partial differential equations using radial basis collocation method are bounded by the truncation error of the RBF. The combined errors due to radial basis approximation, quadrature rules, and quasi‐Newton and Newton iterations are also presented. This result can be extended to finite element or finite difference method combined with any iteration methods discussed in this work. The numerical example demonstrates a good agreement between numerical results and analytical predictions. The numerical results also show that although the convergence rate of order 1.62 of the quasi‐Newton iteration scheme is slightly slower than rate of order 2 in the Newton iteration scheme, the former is more stable and less sensitive to the initial guess. © 2007 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2008     

MODIFIED NEWTON＇S ALGORITHM FOR COMPUTING THE GROUP INVERSES OF SINGULAR TOEPLITZ MATRICES

Newton＇s iteration is modified for the computation of the group inverses of singular Toeplitz matrices. At each iteration, the iteration matrix is approximated by a matrix with a low displacement rank. Because of the displacement structure of the iteration matrix, the matrix-vector multiplication involved in Newton＇s iteration can be done efficiently. We show that the convergence of the modified Newton iteration is still very fast. Numerical results are presented to demonstrate the fast convergence of the proposed method.     

On inexact Newton methods based on doubling iteration scheme for non‐symmetric algebraic Riccati equations

Newton iteration method can be used to find the minimal non‐negative solution of a certain class of non‐symmetric algebraic Riccati equations. However, a serious bottleneck exists in efficiency and storage for the implementation of the Newton iteration method, which comes from the use of some direct methods in exactly solving the involved Sylvester equations. In this paper, instead of direct methods, we apply a fast doubling iteration scheme to inexactly solve the Sylvester equations. Hence, a class of inexact Newton iteration methods that uses the Newton iteration method as the outer iteration and the doubling iteration scheme as the inner iteration is obtained. The corresponding procedure is precisely described and two practical methods of monotone convergence are algorithmically presented. In addition, the convergence property of these new methods is studied and numerical results are given to show their feasibility and effectiveness for solving the non‐symmetric algebraic Riccati equations. Copyright © 2010 John Wiley & Sons, Ltd.     

Broyden method for inverse non-symmetric Sturm-Liouville problems

In this paper, we propose a derivative-free method for recovering symmetric and non-symmetric potential functions of inverse Sturm-Liouville problems from the knowledge of eigenvalues. A class of boundary value methods obtained as an extension of Numerov’s method is the major tool for approximating the eigenvalues in each Broyden iteration step. Numerical examples demonstrate that the method is able to reduce the number of iteration steps, in particular for non-symmetric potentials, without accuracy loss.