New least-square algorithms

New algorithms are presented for approximating the minimum of the sum of squares ofM real and differentiable functions over anN-dimensional space. These algorithms update estimates for the location of a minimum after each one of the functions and its first derivatives are evaluated, in contrast with other least-square algorithms which evaluate allM functions and their derivatives at one point before using any of this information to make an update. These new algorithms give estimates which fluctuate about a minimum rather than converging to it. For many least-square problems, they give an adequate approximation for the solution more quickly than do other algorithms.It is a pleasure to thank J. Chesick of Haverford College for suggesting and implementing the application of this algorithm to x-ray crystallography.     

Computational experience with bank-one positive definite quasi-newton algorithms

Rank-one positive definite Quasi-Newton algorithms for unconstrained minimization of the type described by Kleinmichel and Spedicato are numerically evaluated versus the BFS algorithm. Results show that, while on certain functions some rank-one methods perform better, overall the BFS still comes first, its superiority being more evident for larger dimensional problems.     

Computational experience with known variable metric updates

Variable metric methods from the Broyden family are well known and commonly used for unconstrained minimization. These methods have good theoretical and practical convergence properties which depend on a selection of free parameters. We demonstrate, using extensive computational experiments, the influence of both the Biggs stabilization parameter and Oren scaling parameter on 12 individual variable metric updates, two of which are new. This paper focuses on a class of variable metric updates belonging to the so-called preconvex part of the Broyden family. These methods outperform the more familiar BFGS method. We also experimentally demonstrate the efficiency of the controlled scaling strategy for problems of sufficient size and sparsity.     

On quasi-Newton and pseudo-Newton algorithms

This paper surveys some of the existing approaches to quasi-Newton methods and introduces a new way for constructing inverse Hessian approximations for such algorithms. This new approach is based on restricting Newton's method to subspaces over which the inverse Hessian is assumed to be known, while expanding this subspace using gradient information. It is shown that this approach can lead to some well-known formulas for updating the inverse Hessian approximation. Deriving such updates through this approach provides new understanding of these formulas and their relation to the pseudo-Newton-Raphson algorithm.     

Numerical experience with a polyhedral-norm CDT trust-region algorithm

In this paper, we study a modification of the Celis-Dennis-Tapia trust-region subproblem, which is obtained by replacing thel ₂-norm with a polyhedral norm. The polyhedral norm Celis-Dennis-Tapia (CDT) subproblem can be solved using a standard quadratic programming code.We include computational results which compare the performance of the polyhedral-norm CDT trust-region algorithm with the performance of existing codes. The numerical results validate the effectiveness of the approach. These results show that there is not much loss of robustness or speed and suggest that the polyhedral-norm CDT algorithm may be a viable alternative. The topic merits further investigation.The first author was supported in part by the REDI foundation and State of Texas Award, Contract 1059 as Visiting Member of the Center for Research on Parallel Computation, Rice University, Houston, Texas, He thanks Rice University for the congenial scientific atmosphere provided. The second author was supported in part by the National Science Foundation, Cooperative Agreement CCR-88-09615, Air Force Office of Scientific Research Grant 89-0363, and Department of Energy Contract DEFG05-86-ER25017.     

A method for determining the equilibrium states of dynamic systems

A heuristic method is presented for determining the equilibrium states of motion of dynamic systems, in particular, spacecraft. The method can also be applied to the solution of sets of linear or nonlinear algebraic equations. A positive-semidefinite functional is formed to convert the problem to that of finding those minimum points where the functional vanishes. The process is initiated within a selecteddomain of interest by random search; convergence to a minimum is obtained by a modified Davidon's deflected gradient technique. To render this approach feasible in the presence of constraints, the functional is modified to include penalty terms which cause the functional to approach infinity at the constraint boundaries. Close approximations to solutions near the constraint boundaries are found by applying Carroll's approach in successively reducing the weighting factors of the penalty terms. After finding a minimum, the local domain around this point is eliminated by adding to the functional an interior constraint term, representing the surface under a hypersphere centered at the minimum point. The domain of consideration now becomes the subdomain formed by subtracting the space contained within this hypersphere from the previous domain of interest. Minima are now sought within the remaining space, as before.This paper is derived from research performed by the author while employed by TRW Systems Group, Redondo Beach, California.The author acknowledges the helpful suggestions of Dr. G. Bekey, University of Southern California, and those of Mr. E. A. Quast, Dr. M. P. Scher, and Dr. R. J. Wiley, Dynamics Department, TRW Systems Group, Redondo Beach, California.     

New variable-metric algorithms for nondifferentiable optimization problems

This paper deals with new variable-metric algorithms for nonsmooth optimization problems, the so-called adaptive algorithms. The essence of these algorithms is that there are two simultaneously working gradient algorithms: the first is in the main space and the second is in the space of the matrices that modify the main variables. The convergence of these algorithms is proved for different cases. The results of numerical experiments are also given.     

Computational experience using an edge search algorithm for linear reverse convex programs

This paper presents computational experience with a rather straight forward implementation of an edge search algorithm for obtaining the globally optimal solution for linear programs with an additional reverse convex constraint. The paper's purpose is to provide a collection of problems, with known optimal solutions, and performance information for an edge search implementation so that researchers may have some benchmarks with which to compare new methods for reverse convex programs or concave minimization problems. There appears to be nothing in the literature that provides computational experience with a basic edge search procedure. The edge search implementation uses a depth first strategy. As such, this paper's implementation of the edge search algorithm is a modification of Hillestad's algorithm [11]. A variety of test problems is generated by using a modification of the method of Sung and Rosen [20], as well as a new method that is presented in this paper. Test problems presented may be obtained at ftp://newton.ee.ucla.edu/nonconvex/pub/.     

A parallel variable-metric dynamic programming algorithm

This paper presents a potentially parallel iterative algorithm for the solution of the unconstrainedN-stage decision problem of dynamic programming. The basis of the algorithm is the use of variable-metric minimization techniques to develop a quadratic approximation to the cost function at each stage. The algorithm is applied to various problems, and comparisons with other algorithms are made.This research forms part of the author's PhD program, and is supported by the Department of Scientific and Industrial Research of the New Zealand Government. The author is indebted to Dr. B. A. Murtagh, PhD supervisor, for his encouragement and support during the preparation of this paper.     

Numerical experiments with variations of the Gauss-Newton algorithm for nonlinear least squares

In this paper, the classical Gauss-Newton method for the unconstrained least squares problem is modified by introducing a quasi-Newton approximation to the second-order term of the Hessian. Various quasi-Newton formulas are considered, and numerical experiments show that most of them are more efficient on large residual problems than the Gauss-Newton method and a general purpose minimization algorithm based upon the BFGS formula. A particular quasi-Newton formula is shown numerically to be superior. Further improvements are obtained by using a line search that exploits the special form of the function.     

Computational experience with an interior point algorithm on the satisfiability problem

We apply the zero-one integer programming algorithm described in Karmarkar [12] and Karmarkar, Resende and Ramakrishnan [13] to solve randomly generated instances of the satisfiability problem (SAT). The interior point algorithm is briefly reviewed and shown to be easily adapted to solve large instances of SAT. Hundreds of instances of SAT (having from 100 to 1000 variables and 100 to 32,000 clauses) are randomly generated and solved. For comparison, we attempt to solve the problems via linear programming relaxation with MINOS.     

A simplex algorithm for piecewise-linear programming III: Computational analysis and applications

The first two parts of this paper have developed a simplex algorithm for minimizing convex separable piecewise-linear functions subject to linear constraints. This concluding part argues that a direct piecewiselinear simplex implementation has inherent advantages over an indirect approach that relies on transformation to a linear program. The advantages are shown to be implicit in relationships between the linear and piecewise-linear algorithms, and to be independent of many details of implementation. Two sets of computational results serve to illustarate these arguments; the piecewise-linear simplex algorithm is observed to run 2–6 times faster than a comparable linear algorithm, not including any additional expense that might be incurred in setting up the equivalent linear program. Further support for the practical value of a good piecewise-linear programming algorithm is provided by a survey of many varied applications.This research has been supported in part by the National Science Foundation under grant DMS-8217261.     

New trajectory-following polynomial-time algorithm for linear programming problems

A new interior point method for the solution of the linear programming problem is presented. It is shown that the method admits a polynomial time bound. The method is based on the use of the trajectory of the problem, which makes it conceptually very simple. It has the advantage above related methods that it requires no problem transformation (either affine or projective) and that the feasible region may be unbounded. More importantly, the method generates at each stage solutions of both the primal and the dual problem. This implies that, contrary to the simplex method, the quality of the present solution is known at each stage. The paper also contains a practical (i.e., deepstep) version of the algorithm.The author is indebted to J. Bisschop, P. C. J. M. Geven, J. H. Van Lint, J. Ponstein, and J. P. Vial for their remarks on an earlier version of this paper.     

A Trust-Region Method for Nonlinear Bilevel Programming: Algorithm and Computational Experience

We consider the approximation of nonlinear bilevel mathematical programs by solvable programs of the same type, i.e., bilevel programs involving linear approximations of the upper-level objective and all constraint-defining functions, as well as a quadratic approximation of the lower-level objective. We describe the main features of the algorithm and the resulting software. Numerical experiments tend to confirm the promising behavior of the method.     

A feasible conjugate-direction method to solve linearly constrained minimization problems

An iterative procedure is presented which uses conjugate directions to minimize a nonlinear function subject to linear inequality constraints. The method (i) converges to a stationary point assuming only first-order differentiability, (ii) has ann-q step superlinear or quadratic rate of convergence with stronger assumptions (n is the number of variables,q is the number of constraints which are binding at the optimum), (iii) requires the computation of only the objective function and its first derivatives, and (iv) is experimentally competitive with well-known methods.For helpful suggestions, the author is much indebted to C. R. Glassey and K. Ritter.This research has been partially supported by the National Research Council of Canada under Grants Nos. A8189 and C1234.     

A cutting-plane algorithm with linear and geometric rates of convergence

This paper presents a cutting-plane algorithm for nonlinear programming which, under suitable conditions, exhibits a linear or geometric global rate of convergence. Other known rates of convergence for cutting-plane algorithms are no better than arithmetic for problems not satisfying a Haar condition. The feature responsible for this improved rate of convergence is the addition at each iteration of a new cut for each constraint, rather than adding only one new cut corresponding to the most violated constraint as is typically the case. Certain cuts can be dropped at each iteration, and there is a uniform upper bound on the number of old cuts retained. Geometric convergence is maintained if the subproblems at each iteration are approximated, rather than solved exactly, so the algorithm is implementable. The algorithm is flexible with respect to the point used to generate new cuts.The author is grateful to W. Oettli for bringing to his attention the linearly convergent cutting-plane algorithm of Ref. 15 and to the referee for a comment that stimulated an extension of the convergence rate results from an earlier version where _k depended on certain parameters of the problem.     

A variable-metric variant of the Karmarkar algorithm for linear programming

The most time-consuming part of the Karmarkar algorithm for linear programming is the projection of a vector onto the nullspace of a matrix that changes at each iteration. We present a variant of the Karmarkar algorithm that uses standard variable-metric techniques in an innovative way to approximate this projection. In limited tests, this modification greatly reduces the number of matrix factorizations needed for the solution of linear programming problems. Research sponsored by DOE DE-AS05-82ER13016, ARO DAAG-29-83-K-0035, AFOSR 85-0243. Research sponsored by ARO DAAG-29-83-K-0035, AFOSR 85-0243, Shell Development Company.     

关于非线性规划的同伦算法与罚函数法

本文揭示了关于非线性规划问题的同伦算法与外点罚函数法的关系，并讨论了有关同伦算法的收敛条件，给出了一些典型的检验问题的计算结果以表明利用结构的分段线性同伦算法的有效性。     

Square-root variable-metric methods for minimization

Variable-metric methods are presented which do not need an accurate one-dimensional search and eliminate roundoff error problems which can occur in updating the metric for large-dimension systems. The methods are based on updating the square root of the metric, so that a positive-definite metric always results. The disadvantage of intentionally relaxing the accuracy of the one-dimensional search is that the number of iterations (and hence, gradient evaluations) increases. For problems involving a large number of variables, the square-root method is presented in a triangular form to reduce the amount of computation. Also, for usual optimization problems, the square-root procedure can be carried out entirely in terms of the metric, eliminating storage and computer time associated with computations of the square root of the metric.     

A sparse sequential quadratic programming algorithm

Described here is the structure and theory for a sequential quadratic programming algorithm for solving sparse nonlinear optimization problems. Also provided are the details of a computer implementation of the algorithm along with test results. The algorithm maintains a sparse approximation to the Cholesky factor of the Hessian of the Lagrangian. The solution to the quadratic program generated at each step is obtained by solving a dual quadratic program using a projected conjugate gradient algorithm. An updating procedure is employed that does not destroy sparsity.