Parallel quasi-Newton methods for unconstrained optimization

We discuss methods for solving the unconstrained optimization problem on parallel computers, when the number of variables is sufficiently small that quasi-Newton methods can be used. We concentrate mainly, but not exclusively, on problems where function evaluation is expensive. First we discuss ways to parallelize both the function evaluation costs and the linear algebra calculations in the standard sequential secant method, the BFGS method. Then we discuss new methods that are appropriate when there are enough processors to evaluate the function, gradient, and part but not all of the Hessian at each iteration. We develop new algorithms that utilize this information and analyze their convergence properties. We present computational experiments showing that they are superior to parallelization either the BFGS methods or Newton's method under our assumptions on the number of processors and cost of function evaluation. Finally we discuss ways to effectively utilize the gradient values at unsuccessful trial points that are available in our parallel methods and also in some sequential software packages.Research supported by AFOSR grant AFOSR-85-0251, ARO contract DAAG 29-84-K-0140, NSF grants DCR-8403483 and CCR-8702403, and NSF cooperative agreement DCR-8420944.     

Asynchronous iterations with flexible communication: contracting operators

The concept of flexible communication permits one to model efficient asynchronous iterations on parallel computers. This concept is particularly useful in two practical situations. Firstly, when communications are requested while a processor has completed the current update only partly, and secondly, in the context of inner/outer iterations, when processors are also allowed to make use of intermediate results obtained during the inner iteration in other processors.In the general case of nonlinear or linear fixed point problems, we give a global convergence results for asynchronous iterations with flexible communication whereby the iteration operators satisfy certain contraction hypotheses. In this manner we extend to a contraction context previous results obtained for monotone operators with respect to a partial ordering.     

Spectral Scaling BFGS Method

In this paper, we scale the quasiNewton equation and propose a spectral scaling BFGS method. The method has a good selfcorrecting property and can improve the behavior of the BFGS method. Compared with the standard BFGS method, the single-step convergence rate of the spectral scaling BFGS method will not be inferior to that of the steepest descent method when minimizing an n-dimensional quadratic function. In addition, when the method with exact line search is applied to minimize an n-dimensional strictly convex function, it terminates within n steps. Under appropriate conditions, we show that the spectral scaling BFGS method with Wolfe line search is globally and R-linear convergent for uniformly convex optimization problems. The reported numerical results show that the spectral scaling BFGS method outperforms the standard BFGS method.     

Convergence rate analysis of an asynchronous space decomposition method for convex minimization

We analyze the convergence rate of an asynchronous space decomposition method for constrained convex minimization in a reflexive Banach space. This method includes as special cases parallel domain decomposition methods and multigrid methods for solving elliptic partial differential equations. In particular, the method generalizes the additive Schwarz domain decomposition methods to allow for asynchronous updates. It also generalizes the BPX multigrid method to allow for use as solvers instead of as preconditioners, possibly with asynchronous updates, and is applicable to nonlinear problems. Applications to an overlapping domain decomposition for obstacle problems are also studied. The method of this work is also closely related to relaxation methods for nonlinear network flow. Accordingly, we specialize our convergence rate results to the above methods. The asynchronous method is implementable in a multiprocessor system, allowing for communication and computation delays among the processors.

     

A globally convergent BFGS method for nonlinear monotone equations without any merit functions

Since 1965, there has been significant progress in the theoretical study on quasi-Newton methods for solving nonlinear equations, especially in the local convergence analysis. However, the study on global convergence of quasi-Newton methods is relatively fewer, especially for the BFGS method. To ensure global convergence, some merit function such as the squared norm merit function is typically used. In this paper, we propose an algorithm for solving nonlinear monotone equations, which combines the BFGS method and the hyperplane projection method. We also prove that the proposed BFGS method converges globally if the equation is monotone and Lipschitz continuous without differentiability requirement on the equation, which makes it possible to solve some nonsmooth equations. An attractive property of the proposed method is that its global convergence is independent of any merit function.We also report some numerical results to show efficiency of the proposed method.

     

任意长—维DFT的MIMD并行算法

本文提出了共享与分布式存储计算机上任意长—维ＤＦＴ的ＭＩＭＤ并行算法，若Ｎ＝Ｏ（ｐ，ｑ），则算法需要次算术运算。其中，Ｐ与Ｎ可为任意自然数，分别表示处理机台数与ＤＦＴ长度．本文算法具有很高的并行效率．     

An adaptive scaled BFGS method for unconstrained optimization

A new adaptive scaled Broyden-Fletcher-Goldfarb-Shanno (BFGS) method for unconstrained optimization is presented. The third term in the standard BFGS update formula is scaled in order to reduce the large eigenvalues of the approximation to the Hessian of the minimizing function. Under the inexact Wolfe line search conditions, the global convergence of the adaptive scaled BFGS method is proved in very general conditions without assuming the convexity of the minimizing function. Using 80 unconstrained optimization test functions with a medium number of variables, the preliminary numerical experiments show that this variant of the scaled BFGS method is more efficient than the standard BFGS update or than some other scaled BFGS methods.     

A combined class of self-scaling and modified quasi-Newton methods

Techniques for obtaining safely positive definite Hessian approximations with self-scaling and modified quasi-Newton updates are combined to obtain ??better?? curvature approximations in line search methods for unconstrained optimization. It is shown that this class of methods, like the BFGS method, has the global and superlinear convergence for convex functions. Numerical experiments with this class, using the well-known quasi-Newton BFGS, DFP and a modified SR1 updates, are presented to illustrate some advantages of the new techniques. These experiments show that the performance of several combined methods are substantially better than that of the standard BFGS method. Similar improvements are also obtained if the simple sufficient function reduction condition on the steplength is used instead of the strong Wolfe conditions.     

On the conjugacy classes in maximal unipotent subgroups of simple algebraic groups

Let G be a simple algebraic group over the algebraically closed field k of characteristic p ≥ 0. Assume p is zero or good for G. Let B be a Borel subgroup of G; we write U for the unipotent radical of B and u for the Lie algebra of U. Using relative Springer isomorphisms} we analyze the adjoint orbits of U in u. In particular, we show that an adjoint orbit of U in u contains a unique so-called minimal representative. In case p > 0, assume G is defined and split over the finite field of p elements F_p. Let q be a power of p and let G(q) be the finite group of F_q-rational points of G. Let F be the Frobenius morphism such that G(q) = G^F. Assume B is F-stable, so that U is also F-stable and U(q) is a Sylow p-subgroup of G(q). We show that the conjugacy classes of U(q) are in correspondence with the F-stable adjoint orbits of U in u. This allows us to deduce results about the conjugacy classes of U(q).     

强单调对称非线性方程组的BFGS算法

提出一种求解强单调非线性方程组的BFGS算法,该算法的一个明显优点是Bκ的条件数比Li-Fukushima^[3]提出的GNBFGS中Bκ的条件数小得多。且该算法是一种无需计算导数的下降算法。在一定的条件下,证明了算法的全局收敛性和超线性收敛性。最后进行数值试验,结果表明,本文算法具有较好的数值结果。而且验证了本文所提出的算法中Bκ的条件数要比GNBFGS算法的条件数小得多。     

一类非线性代数方程组的并行算法——适用于MIMD系统

为连续对角映射.而A=(a_(ij)∈L(R~n)是单调矩阵,B∈L(R~n)为非负矩阵,b∈R~n为已知向量. 方程组(1.1)具有丰富的实际背景,许多非线性微分方程的求解问题,经过有限元或差分离散,均可归纳为(1.1)的求解.特别,如[7],[10]以及[11]讨论的弱非线性椭圆方程和Stefan问题等,均可作为(1.1)的特例.     

Asynchronous and corrected-asynchronous finite difference solutions of PDEs on MIMD multiprocessors

A major problem in achieving significant speed-up on parallel machines is the overhead involved with synchronizing the concurrent processes. Removing the synchronization constraint has the potential of speeding up the computation, while maintaining greater computation flexibility (e.g. differences in processors speed; differences in the data input to processors). We construct asynchronous (AS) finite difference schemes for the solution of PDEs by removing the synchronization constraint. We analyze the numerical properties of these schemes. Based on the analysis, we develop corrected-asynchronous (CA) finite difference schemes which are specifically constructed for an asynchronous processing. We present asynchronous (AS) and corrected-asynchronous (CA) finite difference schemes for the multi-dimensional heat equation. Although our discussion concentrates on the Euler scheme it should serve only as a sample, as it can be extended to other schemes and other PDEs.These schemes are implemented on the shared-memory multi-userSequent Balance machine. Numerical results for one and two dimensional problems are presented. It is shown experimentally that synchronization penalty can be about 50% of run time: in most cases, the asynchronous scheme runs twice as fast as the parallel synchronous scheme. In general, the efficiency of the parallel schemes increases with processor load, with the time-level, and with the problem dimension. The efficiency of the AS may reach 90% and over, but it provides accurate results only for steady-state values. The CA, on the other hand, is less efficient but provides more accurate results for intermediate (non steady-state) values. The results show the potential of developing asynchronous finite deference schemes for steady-state as well as non steadystate problems.This research was partially supported by a grant from The Basic Research Foundation administrated by The Israel Academy of Sciences and Humanities.A reduced version of the paper was presented at the 4th SIAM Conference on Parallel Processing for Scientific Computing, Dec. 11–13, 1989, Chicago, USA.The work by this author was supported by research grant 337 of the Israeli National Council for Research and Development in the years 1990–1991.This research was supported by the National Aeronautics and Space Administration under NASA Contract No. NASI-18107 while the author was in residence at the Institute for Computer Applications in Sciences and Engineering (ICASE), NASA Langley Research Center, Hampton, VA 23665, USA.     

Quasi-Newton methods for multiobjective optimization problems

This work is an attempt to develop multiobjective versions of some well-known single objective quasi-Newton methods, including BFGS, self-scaling BFGS (SS-BFGS), and the Huang BFGS (H-BFGS). A comprehensive and comparative study of these methods is presented in this paper. The Armijo line search is used for the implementation of these methods. The numerical results show that the Armijo rule does not work the same way for the multiobjective case as for the single objective case, because, in this case, it imposes a large computational effort and significantly decreases the speed of convergence in contrast to the single objective case. Hence, we consider two cases of all multi-objective versions of quasi-Newton methods: in the presence of the Armijo line search and in the absence of any line search. Moreover, the convergence of these methods without using any line search under some mild conditions is shown. Also, by introducing a multiobjective subproblem for finding the quasi-Newton multiobjective search direction, a simple representation of the Karush–Kuhn–Tucker conditions is derived. The H-BFGS quasi-Newton multiobjective optimization method provides a higher-order accuracy in approximating the second order curvature of the problem functions than the BFGS and SS-BFGS methods. Thus, this method has some benefits compared to the other methods as shown in the numerical results. All mentioned methods proposed in this paper are evaluated and compared with each other in different aspects. To do so, some well-known test problems and performance assessment criteria are employed. Moreover, these methods are compared with each other with regard to the expended CPU time, the number of iterations, and the number of function evaluations.     

On the convergence of asynchronous parallel algorithm for large-scale linearly constrained minimization problem

As a synchronization parallel framework, the parallel variable transformation (PVT) algorithm is effective to solve unconstrained optimization problems. In this paper, based on the idea that a constrained optimization problem is equivalent to a differentiable unconstrained optimization problem by introducing the Fischer Function, we propose an asynchronous PVT algorithm for solving large-scale linearly constrained convex minimization problems. This new algorithm can terminate when some processor satisfies terminal condition without waiting for other processors. Meanwhile, it can enhances practical efficiency for large-scale optimization problem. Global convergence of the new algorithm is established under suitable assumptions. And in particular, the linear rate of convergence does not depend on the number of processors.     

A regularized limited memory BFGS method for nonconvex unconstrained minimization

The limited memory BFGS method (L-BFGS) is an adaptation of the BFGS method for large-scale unconstrained optimization. However, The L-BFGS method need not converge for nonconvex objective functions and it is inefficient on highly ill-conditioned problems. In this paper, we proposed a regularization strategy on the L-BFGS method, where the used regularization parameter may play a compensation role in some sense when the condition number of Hessian approximation tends to become ill-conditioned. Then we proposed a regularized L-BFGS method and established its global convergence even when the objective function is nonconvex. Numerical results show that the proposed method is efficient.     

同时求多项式全部零点的一族快速并行迭代和区间迭代(Ⅱ)

本文对[1]所提出的一族同时求多项式全部零点的并行迭代兼区间迭代加以进一步的发展。首先,作为纯粹的并行迭代法,我们在§2把每步并行迭代扩展为q个并行子步,这样得到的并行迭代法对只有单零点的多项式的全部零点的收敛是q(p 1)阶的。值得注意的是,在这里阶的提高大大超过了每步计算代价的增加,例如,当q=2时,每步     

Efficient sampling of random permutations

We show how to uniformly distribute data at random (not to be confounded with permutation routing) in two settings that are able to deal with massive data: coarse grained parallelism and external memory. In contrast to previously known work for parallel setups, our method is able to fulfill the three criteria of uniformity, work-optimality and balance among the processors simultaneously. To guarantee the uniformity we investigate the matrix of communication requests between the processors. We show that its distribution is a generalization of the multivariate hypergeometric distribution and we give algorithms to sample it efficiently in the two settings.     

Optimization of Molecular Similarity Index with Applications to Biomolecules

Molecular similarity index measures the similarity between two molecules. Computing the optimal similarity index is a hard global optimization problem. Since the objective function value is very hard to compute and its gradient vector is usually not available, previous research has been based on non-gradient algorithms such as random search and the simplex method. In a recent paper, McMahon and King introduced a Gaussian approximation so that both the function value and the gradient vector can be computed analytically. They then proposed a steepest descent algorithm for computing the optimal similarity index of small molecules. In this paper, we consider a similar problem. Instead of computing atom-based derivatives, we directly compute the derivatives with respect to the six free variables describing the relative positions of the two molecules.. We show that both the function value and gradient vector can be computed analytically and apply the more advanced BFGS method in addition to the steepest descent algorithm. The algorithms are applied to compute the similarities among the 20 amino acids and biomolecules like proteins. Our computational results show that our algorithm can achieve more accuracy than previous methods and has a 6-fold speedup over the steepest descent method.     

The Global Convergence of Self-Scaling BFGS Algorithm with Nonmonotone Line Search for Unconstrained Nonconvex Optimization Problems

The self-scaling quasi-Newton method solves an unconstrained optimization problem by scaling the Hessian approximation matrix before it is updated at each iteration to avoid the possible large eigenvalues in the Hessian approximation matrices of the objective function. It has been proved in the literature that this method has the global and superlinear convergence when the objective function is convex （or even uniformly convex）. We propose to solve unconstrained nonconvex optimization problems by a self-scaling BFGS algorithm with nonmonotone linear search. Nonmonotone line search has been recognized in numerical practices as a competitive approach for solving large-scale nonlinear problems. We consider two different nonmonotone line search forms and study the global convergence of these nonmonotone self-scale BFGS algorithms. We prove that, under some weaker condition than that in the literature, both forms of the self-scaling BFGS algorithm are globally convergent for unconstrained nonconvex optimization problems.     

Parallel Asynchronous Global Search and the Nested Optimization Scheme

In this paper, a parallel asynchronous information algorithm for solving multidimensional Lipschitz global optimization problems, where times for evaluating the objective function can be different from point to point, is proposed. This method uses the nested optimization scheme and a new parallel asynchronous global optimization method for solving core univariate subproblems generated by the nested scheme. The properties of the scheme related to parallel computations are investigated. Global convergence conditions for the new method and theoretical conditions of speed up, which can be reached by using asynchronous parallelization in comparison with the pure sequential case, are established. Numerical experiments comparing sequential, synchronous, and asynchronous algorithms are also reported.