A numerical evaluation of some collinear scaling algorithms for unconstrained

Collinear scaling algorithms for unconstrained minimization were first proposed by Davidon (1977,80) so that they may incorporate more information about the problem than is possible with quasi–Newton algorithms. Sorensen (1980,82), and Ariyawansa (1983,90) have derived collinear scaling algorithms as natural extensions of quasi–Newton algorithms. In this paper we describe the results of a comprehensive numerical evaluation of four members in the classes of collinear scaling algorithms derived by Sorensen (1980,82) and Ariyawansa (1983,90), relative to the quasi–Newton algorithms they extend.     

A class of collinear scaling algorithms for bound-constrained optimization: Convergence theorems

A family of algorithms for approximate solution of the bound-constrained minimization problem was introduced in [K.A. Ariyawansa, W.L. Tabor, A class of collinear scaling algorithms for bound-constrained optimization: Derivation and computational results, Technical Report 2003-1, Department of Mathematics, Washington State University, Pullman, WA, 2003, submitted for publication. Available at http://www.math.wsu.edu/math/TRS/2003-1.pdf]. These algorithms employ the standard barrier method, with the inner iteration based on trust region methods. Local models are conic functions rather than the usual quadratic functions, and are required to match first and second derivatives of the barrier function at the current iterate. The various members of the family are distinguished by the choice of a vector-valued parameter, which is the zero vector in the degenerate case that quadratic local models are used. This paper presents a convergence analysis of the family of algorithms presented in [K.A. Ariyawansa, W.L. Tabor, A class of collinear scaling algorithms for bound-constrained optimization: Derivation and computational results, Technical Report 2003-1, Department of Mathematics, Washington State University, Pullman, WA, 2003, submitted for publication. Available at http://www.math.wsu.edu/math/TRS/2003-1.pdf]. Specifically, convergence properties similar to those of barrier methods using quadratic local models are established.     

Local convergence of collinear scaling algorithms related to direct least‐change secant update methods for minimization

Collinear scaling algorithms related to direct fixed-scale and rescaled least-change secant update methods for unconstrained minimization are derived. Theorems on local and q-superlinear convergence of these algorithms are presented. These results are extensions of those of Dennis and Walker [14] for direct least-change secant update methods. This revised version was published online in August 2006 with corrections to the Cover Date.     

On affine scaling algorithms for nonconvex quadratic programming

We investigate the use of interior algorithms, especially the affine-scaling algorithm, to solve nonconvex — indefinite or negative definite — quadratic programming (QP) problems. Although the nonconvex QP with a polytope constraint is a hard problem, we show that the problem with an ellipsoidal constraint is easy. When the hard QP is solved by successively solving the easy QP, the sequence of points monotonically converge to a feasible point satisfying both the first and the second order optimality conditions.Research supported in part by NSF Grant DDM-8922636 and the College Summer Grant, College of Business Administration, The University of Iowa.     

Second order algorithms for the posynomial geometric programming dual,part I: Analysis

In this paper we analyse algorithms for the geometric dual of posynomial programming problems, that make explicit use of second order information. Out of two possible approaches to the problem, it is shown that one is almost always superior. Interestingly enough, it is the second, inferior approach that has dominated the geometric programming literature.This work was partially supported by the National Research Council of Canada, Grant A3552 and National Science Foundation Grant ENG78-21615.Earlier versions of this paper were presented at the Optimization Days Conference in Montreal (May 1976) and the TIMS meeting in Athens (July 1977).     

Comparative analysis of affine scaling algorithms based on simplifying assumptions

We analyze several affine potential reduction algorithms for linear programming based on simplifying assumptions. We show that, under a strong probabilistic assumption regarding the distribution of the data in an iteration, the decrease in the primal potential function will be with high probability, compared to the guaranteed(1). ( 2n is a parameter in the potential function andn is the number of variables.) Under the same assumption, we further show that the objective reduction rate of Dikin's affine scaling algorithm is with high probability, compared to no guaranteed convergence rate.Research supported in part by NSF Grant DDM-8922636.     

Optimal conditioning of self-scaling variable Metric algorithms

Variable Metric Methods are Newton—Raphson-like algorithms for unconstrained minimization in which the inverse Hessian is replaced by an approximation, inferred from previous gradients and updated at each iteration. During the past decade various approaches have been used to derive general classes of such algorithms having the common properties of being Conjugate Directions methods and having quadratic termination. Observed differences in actual performance of such methods motivated recent attempts to identify variable metric algorithms having additional properties that may be significant in practical situations (e.g. nonquadratic functions, inaccurate linesearch, etc.). The SSVM algorithms, introduced by this first author, are such methods that among their other properties, they automatically compensate for poor scaling of the objective function. This paper presents some new theoretical results identifying a subclass of SSVM algorithms that have the additional property of minimizing a sharp bound on the condition number of the inverse Hessian approximation at each iteration. Reducing this condition number is important for decreasing the roundoff error. The theoretical properties of this subclass are explored and two of its special cases are tested numerically in comparison with other SSVM algorithms.This work has been done while this author was a visiting fellow at the Engineering Economic System Department, Stanford University.     

Numerical comparison of several variable-metric algorithms

The paper compares the numerical performances of the LDL decomposition of the BFGS variable-metric algorithm, the Dennis-Mei dogleg algorithm on the BFGS update, and Davidon's projections with the BFGS update with the straight BFGS update on a number of standard test problems. Numerical results indicate that the standard BFGS algorithm is superior to all of the more complex strategies.This research was supported by the National Science Foundation under Research Grant No. MCS77-07327.     

On the global optimization properties of finite-difference local descent algorithms

The paper is devoted to the convergence properties of finite-difference local descent algorithms in global optimization problems with a special -convex structure. It is assumed that the objective function can be closely approximated by some smooth convex function. Stability properties of the perturbed gradient descent and coordinate descent methods are investigated. Basing on this results some global optimization properties of finite-difference local descent algorithms, in particular, coordinate descent method, are discovered. These properties are not inherent in methods using exact gradients.The paper was presented at the II. IIASA-Workshop on Global Optimization, Sopron (Hungary), December 9–14, 1990.     

Two new unconstrained optimization algorithms which use function and gradient values

Two new methods for unconstrained optimization are presented. Both methods employ a hybrid direction strategy which is a modification of Powell's 1970 dogleg strategy. They also employ a projection technique introduced by Davidon in his 1975 algorithm which uses projection images of x and g in updating the approximate Hessian. The first method uses Davidon's optimally conditioned update formula, while the second uses only the BFGS update. Both methods performed well without Powell's special iterations and singularity safeguards, and the numerical results are very promising.This research was supported by the National Science Foundation under Grant No. GJ-40903.     

A CLASS OF COLLINEAR SCALING ALGORITHMS FOR UNCONSTRAINED OPTIMIZATON

A Class of Collinear Scaling Algorithms for Unconstrained Optimization. An appealing approach to the solution of nonlinear optimization problems based on conic models of the objective function has been in troduced by Davidon (1980). It leads to a broad class of algorithms which can be considered to generalize the existing quasi-Newton methods. One particular member of this class has been deeply discussed by Sorensen (1980), who has proved some interesting theoretical properties. In this paper, we generalize Sorensen's technique to Spedicato three-parameter family of variable-metric updates. Furthermore, we point out that the collinear scaling three- parameter family is essentially equivalent to the Spedicato three-parameter family. In addition, numerical expriments have been carried out to compare some colliner scaling algorithms with a straightforward implementation of the BFGS quasi-Newton method.     

The quality of the Diophantine approximations found by the Jacobi-Perron algorithm and related algorithms

Given a vector of real numbers=(₁,... _d ) ^d , the Jacobi-Perron algorithm and related algorithms, such as Brun's algorithm and Selmer's algorithm, produce a sequence of (d+1)×(d+1) convergent matrices {C⁽ⁿ⁾():n1} whose rows provide Diophantine approximations to . Such algorithms are specified by two mapsT:[0, 1] ^d [0, 1] ^d and A:[0,1] ^d GL(d+1,), which compute convergent matrices C⁽ⁿ⁾())...A(T())A(). The quality of the Diophantine approximations these algorithms find can be measured in two ways. The best approximation exponent is the upper bound of those values of for which there is some row of the convergent matrices such that for infinitely many values ofn that row of C⁽ⁿ⁾() has . The uniform approximation exponent is the upper bound of those values of such that for all sufficiently large values ofn and all rows of C⁽ⁿ⁾⁽⁾ one has . The paper applies Oseledec's multiplicative ergodic theorem to show that for a large class of such algorithms and take constant values and on a set of Lebesgue measure one. It establishes the formula where are the two largest Lyapunov exponents attached by Oseledec's multiplicative ergodic theorem to the skew-product (T, A,d), whered is aT-invariant measure, absolutely continuous with respect to Lebesgue measure. We conjecture that holds for a large class of such algorithms. These results apply to thed-dimensional Jacobi-Perron algorithm and Selmer's algorithm. We show that; experimental evidence of Baldwin (1992) indicates (nonrigorously) that. We conjecture that holds for alld2.     

Global convergence of the affine scaling methods for degenerate linear programming problems

In this paper we show the global convergence of the affine scaling methods without assuming any condition on degeneracy. The behavior of the method near degenerate faces is analyzed in detail on the basis of the equivalence between the affine scaling methods for homogeneous LP problems and Karmarkar's method. It is shown that the step-size 1/8, where the displacement vector is normalized with respect to the distance in the scaled space, is sufficient to guarantee the global convergence of the affine scaling methods.This paper was presented at the International Symposium Interior Point Methods for Linear Programming: Theory and Practice, held on January 18–19, 1990, at the Europa Hotel, Scheveningen, the Netherlands.     

Subordination des résolvantes

Résumé Soit (V ₎₀ une résolvante définie sur un espace mesurable telle que le noyau initial est borné; on trouve une condition nécéssaire et suffisante pour qu'un noyau borné U possède une résolvante (U ₎₀ telle que U V pour tout 0. On donne plusieurs applications de ce résultat.     

On the updating scheme in a class of collinear scaling algorithms for sparse minimization

Sorensen (Ref. 1) has proposed a class of algorithms for sparse unconstrained minimization where the sparsity pattern of the Cholesky factors of the Hessian is known. His updates at each iteration depend on the choice of a vector, and in Ref. 1 the question of choosing this vector is essentially left open. In this note, we propose a variational problem whose solution may be used to choose this vector. The major part of the computation of a solution to this variational problem is similar to the computation of a trust-region step in unconstrained minimization. Therefore, well-developed techniques available for the latter problem can be used to compute this vector and to perform the updating.This research was supported by NSF Grant DMS-8414460 and by DOE Grant DE-FG06-85ER25007, awarded to Washington State University, and by the Applied Mathematical Sciences Subprogram of the US Department of Energy under Contract W-31-109-Eng-38 while the first author was visiting the Mathematics and Computer Science Division of Argonne National Laboratory.     

On the summability of Fourier series with the method of lacunary arithmetic means

. f- ,S _n (f)— . {n _k }, n _k+1/n _k >1+ck ^– ,— , 0<1/2, f ₀, .     

Simplicial decomposition in nonlinear programming algorithms

Simplicial decomposition is a special version of the Dantzig—Wolfe decomposition principle, based on Carathéodory's theorem. The associated class of algorithms has the following features and advantages: The master and the subprogram are constructed without dual variables; the methods remain therefore well-defined for non-concave objective functions, and pseudo-concavity suffices for convergence to global maxima. The subprogram produces affinely independent sets of feasible generator points defining simplices, which the master program keeps minimal by dropping redundant generator points and finding maximizers in the relative interiors of the resulting subsimplices. The use of parallel subspaces allows the direct application of any unrestricted optimization method in the master program; thus the best unconstrained procedure for any type of objective function can be used to find constrained maximizers for it.The paper presents the theory for this class of algorithms, the APL-code of a demonstration method and some computational experience with Colville's test problems.I am grateful to Philip Wolfe for encouraging me to write this paper, and I am indebted to him and a referee for helpful comments.Research was partially supported by a grant of the University of Alberta.     

Family of Projected Descent Methods for Optimization Problems with Simple Bounds

This paper presents a family of projected descent direction algorithms with inexact line search for solving large-scale minimization problems subject to simple bounds on the decision variables. The global convergence of algorithms in this family is ensured by conditions on the descent directions and line search. Whenever a sequence constructed by an algorithm in this family enters a sufficiently small neighborhood of a local minimizer satisfying standard second-order sufficiency conditions, it gets trapped and converges to this local minimizer. Furthermore, in this case, the active constraint set at is identified in a finite number of iterations. This fact is used to ensure that the rate of convergence to a local minimizer, satisfying standard second-order sufficiency conditions, depends only on the behavior of the algorithm in the unconstrained subspace. As a particular example, we present projected versions of the modified Polak–Ribière conjugate gradient method and the limited-memory BFGS quasi-Newton method that retain the convergence properties associated with those algorithms applied to unconstrained problems.     

Universal Margin Factors and Selection Criteria in Variable Environment

For nonautonomous dynamical systems, the principle of inheriting local properties by global Poincaré maps is developed. Using this method, a selection criterion for systems of close competitors is found: to gain competitive advantage, it suffices to outproduce other populations with a margin. The margin factor in question remains uniformly bounded as the number of competitors in the community grows.