Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations

In this paper, we review briefly some methods for minimizing a functionF(x), which proceed by follwoing the solution curve of a system of ordinary differential equations. Such methods have often been thought to be unacceptably expensive; but we show, by means of extensive numerical tests, using a variety of algorithms, that the ODE approach can in fact be implemented in such a way as to be more than competitive with currently available conventional techniques.This work was supported by a SERC research studentship for the first author. Both authors are indebted to Dr. J. J. McKeown and Dr. K. D. Patel of SCICON Ltd, the collaborating establishment, for their advice and encouragement.     

New ODE Methods for Equality Constrained Optimization (2) — Algorithms

As a continuation of [1], this paper considers implementation of ODE approaches. A modified Hamming's algorithm for integration of (ECP)-equation is suggested to obtain a local solution. In addition to the main algorithm, three supporting algorithms are also described: two are for evaluation of the right-hand side of (ECP)-equation, which may be especially suitable for certain kinds of (ECP)-equation when applied to large scale problems; the third one, with a convergence theorem, is for computing an initial feasible point. Our numerical results obtained by executing these algorithms on an example of (ECP)-equation given in [1] on five test problems indicate their remarkable superiority of performance to Tanabe's ODE version that is recently claimed to be much better than some well-known SQP techniques.     

A type of efficient feasible SQP algorithms for inequality constrained optimization

In this paper, motivated by Zhu et al. methods [Z.B. Zhu, K.C. Zhang, J.B. Jian, An improved SQP algorithm for inequality constrained optimization, Math. Meth. Oper. Res. 58 (2003) 271-282; Zhibin Zhu, Jinbao Jian, An efficient feasible SQP algorithm for inequality constrained optimization, Nonlinear Anal. Real World Appl. 10(2) (2009) 1220-1228], we propose a type of efficient feasible SQP algorithms to solve nonlinear inequality constrained optimization problems. By solving only one QP subproblem with a subset of the constraints estimated as active, a class of revised feasible descent directions are generated per single iteration. These methods are implementable and globally convergent. We also prove that the algorithms have superlinear convergence rate under some mild conditions.     

Parametric Sensitivity Analysis of Perturbed PDE Optimal Control Problems with State and Control Constraints

We study parametric optimal control problems governed by a system of time-dependent partial differential equations (PDE) and subject to additional control and state constraints. An approach is presented to compute the optimal control functions and the so-called sensitivity differentials of the optimal solution with respect to perturbations. This information plays an important role in the analysis of optimal solutions as well as in real-time optimal control.The method of lines is used to transform the perturbed PDE system into a large system of ordinary differential equations. A subsequent discretization then transcribes parametric ODE optimal control problems into perturbed nonlinear programming problems (NLP), which can be solved efficiently by SQP methods.Second-order sufficient conditions can be checked numerically and we propose to apply an NLP-based approach for the robust computation of the sensitivity differentials of the optimal solutions with respect to the perturbation parameters. The numerical method is illustrated by the optimal control and sensitivity analysis of the Burgers equation.Communicated by H. J. Pesch     

Dynamical Systems and Adaptive Timestepping in ODE Solvers

Initial value problems for ODEs are often solved numerically using adaptive timestepping algorithms. These algorithms are controlled by a user-defined tolerance which bounds from above the estimated error committed at each step. We formulate a large class of such algorithms as discrete dynamical systems which are discontinuous and of higher dimension than the underlying ODE. By assuming sufficiently strong finite-time convergence results on some neighbourhood of an attractor of the ODE we prove existence and upper semicontinuity results for a nearby numerical attractor as the tolerance tends to zero.This assumption of sufficiently strong finite-time convergence results is then examined for adaptive algorithms that use a pair of explicit Runge-Kutta methods of different order to estimate the one-step error. For arbitrary Runge-Kutta pairs the necessary finite-time convergence results fail to hold on a set of points in the phase space that includes all the equilibria of the ODE. Therefore, in general, the asymptotic convergence results cannot be applied to attractors containing equilibria. However, for a particular class of Runge-Kutta pairs, the finite-time convergence results can be strengthened to include neighbourhoods of equilibrium points for which the Jacobian is invertible.     

关于不等式约束的信赖域算法

对于具有不等式约束的非线性优化问题，本文给出一个依赖域算法，由于算法中依赖区域约束采用向量的∞范数约束的形式，从而使子问题变二次规划，同时使算法变得更实用。在通常假设条件下，证明了算法的整体收敛性和超线性收敛性。     

Sequential Quadratic Programming Methods for Large-Scale Problems

Sequential quadratic (SQP) programming methodsare the method of choice when solving small or medium-sized problems. Sincethey are complex methods they are difficult (but not impossible) to adapt tosolve large-scale problems. We start by discussing the difficulties that needto be addressed and then describe some general ideas that may be used toresolve these difficulties. A number of SQP codes have been written to solve specific applications and there is a general purposed SQP code called SNOPT,which is intended for general applications of a particular type. These aredescribed briefly together with the ideas on which they are based. Finally wediscuss new work on developing SQP methods using explicit second derivatives.     

An efficient time-step-based self-adaptive algorithm for predictor-corrector methods of Runge-Kutta type

Finding an efficient implementation variant for the numerical solution of problems from computational science and engineering involves many implementation decisions that are strongly influenced by the specific hardware architecture. The complexity of these architectures makes it difficult to find the best implementation variant by manual tuning. For numerical solution methods from linear algebra, auto-tuning techniques based on a global search engine as they are used for ATLAS or FFTW can be used successfully. These techniques generate different implementation variants at installation time and select one of these implementation variants either at installation time or at runtime, before the computation starts. For some numerical methods, auto-tuning at installation time cannot be applied directly, since the best implementation variant may strongly depend on the specific numerical problem to be solved. An example is solution methods for initial value problems (IVPs) of ordinary differential equations (ODEs), where the coupling structure of the ODE system to be solved has a large influence on the efficient use of the memory hierarchy of the hardware architecture. In this context, it is important to use auto-tuning techniques at runtime, which is possible because of the time-stepping nature of ODE solvers.In this article, we present a sequential self-adaptive ODE solver that selects the best implementation variant from a candidate pool at runtime during the first time steps, i.e., the auto-tuning phase already contributes to the progress of the computation. The implementation variants differ in the loop structure and the data structures used to realize the numerical algorithm, a predictor-corrector (PC) iteration scheme with Runge-Kutta (RK) corrector considered here as an example. For those implementation variants in the candidate pool that use loop tiling to exploit the memory hierarchy of a given hardware platform we investigate the selection of tile sizes. The self-adaptive ODE solver combines empirical search with a model-based approach in order to reduce the search space of possible tile sizes. Runtime experiments demonstrate the efficiency of the self-adaptive solver for different IVPs across a range of problem sizes and on different hardware architectures.     

SEQUENTIAL SYSTEMS OF LINEAR EQUATIONS ALGORITHM FOR NONLINEAR OPTIMIZATION PROBLEMS-INEQUALITY CONSTRAINED PROBLEMS

AbstractIn this paper, a new superlinearly convergent algorithm of sequential systems of linear equations (SSLE) for nonlinear optimization problems with inequality constraints is proposed. Since the new algorithm only needs to solve several systems of linear equations having a same coefficient matrix per iteration, the computation amount of the algorithm is much less than that of the existing SQP algorithms per iteration. Moreover, for the SQP type algorithms, there exist so-called inconsistent problems, i.e., quadratic programming subproblems of the SQP algorithms may not have a solution at some iterations, but this phenomenon will not occur with the SSLE algorithms because the related systems of linear equations always have solutions. Some numerical results are reported.     

An efficient time-step-based self-adaptive algorithm for predictor–corrector methods of Runge–Kutta type

Finding an efficient implementation variant for the numerical solution of problems from computational science and engineering involves many implementation decisions that are strongly influenced by the specific hardware architecture. The complexity of these architectures makes it difficult to find the best implementation variant by manual tuning. For numerical solution methods from linear algebra, auto-tuning techniques based on a global search engine as they are used for ATLAS or FFTW can be used successfully. These techniques generate different implementation variants at installation time and select one of these implementation variants either at installation time or at runtime, before the computation starts. For some numerical methods, auto-tuning at installation time cannot be applied directly, since the best implementation variant may strongly depend on the specific numerical problem to be solved. An example is solution methods for initial value problems (IVPs) of ordinary differential equations (ODEs), where the coupling structure of the ODE system to be solved has a large influence on the efficient use of the memory hierarchy of the hardware architecture. In this context, it is important to use auto-tuning techniques at runtime, which is possible because of the time-stepping nature of ODE solvers.In this article, we present a sequential self-adaptive ODE solver that selects the best implementation variant from a candidate pool at runtime during the first time steps, i.e., the auto-tuning phase already contributes to the progress of the computation. The implementation variants differ in the loop structure and the data structures used to realize the numerical algorithm, a predictor–corrector (PC) iteration scheme with Runge–Kutta (RK) corrector considered here as an example. For those implementation variants in the candidate pool that use loop tiling to exploit the memory hierarchy of a given hardware platform we investigate the selection of tile sizes. The self-adaptive ODE solver combines empirical search with a model-based approach in order to reduce the search space of possible tile sizes. Runtime experiments demonstrate the efficiency of the self-adaptive solver for different IVPs across a range of problem sizes and on different hardware architectures.     

Stochastic ultimate load analysis:models and solution methods 1

The stochastic ultimate load analysis model used in the safety analysis of engineering structures can be treated as a special case of chance-constrained problems (CCP) which minimize a stochastic cost function subject to some probabilistic constraints. Some special cases (such as a deterministic cost function with probabilistic constraints or deterministic constraints with a random cost function) for ultimate load analysis have airady been investigated by various researchers. In this paper, a generai probabilistic approach to stochastic ultimate load analysis is given. In doing so, some approximation techniques are needed due to the fact that the problems at hand are too complicated to evaluate precisely. We propose two extensions of the SQP method in which the variables appear in the algorithms inexactly. These algorithms are shown to be globally convergent for all models and locally superlinearly convergent for some special cases     

Nonlinear and Geometric Programming – Current Status

Great strides have been made in nonlinear programming (NLP) in the last 5 years. In smooth NLP, there are now several reliable and efficient codes capable of solving large problems. Most of these implement GRG or SQP methods, and new software using interior point algorithms is under development. NLP software is now much easier to use, as it is interfaced with many modeling systems, including MSC/NASTRAN, and ANSYS for structural problems, GAMS and AMPL for general optimization, Matlab and Mathcad for general mathematical problems, and the widely used Microsoft Excel spreadsheet. For mixed integer problems, branch and bound and outer approximation codes are now available and are coupled to some of the above modeling systems, while search methods like Tabu Search and Genetic algorithms permit combinatorial, nonsmooth, and nonconvex problems to be attacked.     

An algorithm of sequential systems of linear equations for nonlinear optimization problems with arbitrary initial point

For current sequential quadratic programming (SQP) type algorithms, there exist two problems: (i) in order to obtain a search direction, one must solve one or more quadratic programming subproblems per iteration, and the computation amount of this algorithm is very large. So they are not suitable for the large-scale problems; (ii) the SQP algorithms require that the related quadratic programming subproblems be solvable per iteration, but it is difficult to be satisfied. By using ε-active set procedure with a special penalty function as the merit function, a new algorithm of sequential systems of linear equations for general nonlinear optimization problems with arbitrary initial point is presented. This new algorithm only needs to solve three systems of linear equations having the same coefficient matrix per iteration, and has global convergence and local superlinear convergence. To some extent, the new algorithm can overcome the shortcomings of the SQP algorithms mentioned above. Project partly supported by the National Natural Science Foundation of China and Tianyuan Foundation of China.     

Subspace-stabilized sequential quadratic programming

The stabilized sequential quadratic programming (SQP) method has nice local convergence properties: it possesses local superlinear convergence under very mild assumptions not including any constraint qualifications. However, any attempts to globalize convergence of this method indispensably face some principal difficulties concerned with intrinsic deficiencies of the steps produced by it when relatively far from solutions; specifically, it has a tendency to produce long sequences of short steps before entering the region where its superlinear convergence shows up. In this paper, we propose a modification of the stabilized SQP method, possessing better “semi-local” behavior, and hence, more suitable for the development of practical realizations. The key features of the new method are identification of the so-called degeneracy subspace and dual stabilization along this subspace only; thus the name “subspace-stabilized SQP”. We consider two versions of this method, their local convergence properties, as well as a practical procedure for approximation of the degeneracy subspace. Even though we do not consider here any specific algorithms with theoretically justified global convergence properties, subspace-stabilized SQP can be a relevant substitute for the stabilized SQP in such algorithms using the latter at the “local phase”. Some numerical results demonstrate that stabilization along the degeneracy subspace is indeed crucially important for success of dual stabilization methods.     

A Trust Region SQP Algorithm for Equality Constrained Parameter Estimation with Simple Parameter Bounds

We describe a new algorithm for a class of parameter estimation problems, which are either unconstrained or have only equality constraints and bounds on parameters. Due to the presence of unobservable variables, parameter estimation problems may have non-unique solutions for these variables. These can also lead to singular or ill-conditioned Hessians and this may be responsible for slow or non-convergence of nonlinear programming (NLP) algorithms used to solve these problems. For this reason, we need an algorithm that leads to strong descent and converges to a stationary point. Our algorithm is based on Successive Quadratic Programming (SQP) and constrains the SQP steps in a trust region for global convergence. We consider the second-order information in three ways: quasi-Newton updates, Gauss-Newton approximation, and exact second derivatives, and we compare their performance. Finally, we provide results of tests of our algorithm on various problems from the CUTE and COPS sets.     

A new constraint test-case generator and the importance of hybrid optimizers

In the past decade, significant progress has been made in understanding problem complexity of discrete constraint problems. In contrast, little similar work has been done for constraint problems in the continuous domain. In this paper, we study the complexity of typical methods for non-linear constraint problems and present hybrid solvers with improved performance. To facilitate the empirical study, we propose a new test-case generator for generating non-linear constraint satisfaction problems (CSPs) and constrained optimization problems (COPs). The optimization methods tested include a sequential quadratic programming (SQP) method, a penalty method with a fixed penalty function, a penalty method with a sequence of penalty functions, and an augmented Lagrangian method. For hybrid solvers, we focus on the form that combines two or more optimization methods in sequence. In the experiments, we apply these methods to solve a series of continuous constraint problems with increasing constraint-to-variable ratios. The test problems include artificial benchmark problems from the test-case generator and problems derived from controlling a hyper-redundant modular manipulator. We obtain novel results on complexity phase transition phenomena of the various methods. Specifically, for constraint satisfaction problems, the SQP method is the best on weakly constrained problems, whereas the augmented Lagrangian method is the best on highly constrained ones. Although the static penalty method performs poorly by itself, by combining it with the SQP method, we show a hybrid solver that is significantly better than any of the individual methods on problems with moderate to large constraint-to-variable ratios. For constrained optimization problems, the hybrid solver obtains much better solutions than SQP, while spending comparable amount of time. In addition, the hybrid solver is flexible and can achieve good results on time-bounded applications by setting parameters according to the time limits.     

Global convergence on an active set SQP for inequality constrained optimization

Sequential quadratic programming (SQP) has been one of the most important methods for solving nonlinearly constrained optimization problems. In this paper, we present and study an active set SQP algorithm for inequality constrained optimization. The active set technique is introduced which results in the size reduction of quadratic programming (QP) subproblems. The algorithm is proved to be globally convergent. Thus, the results show that the global convergence of SQP is still guaranteed by deleting some “redundant” constraints.     

Fourth-Order Symmetric DIRK Methods for Periodic Stiff Problems

New symmetric DIRK methods specially adapted to the numerical integration of first-order stiff ODE systems with periodic solutions are obtained. Our interest is focused on the dispersion (phase errors) of the dominant components in the numerical oscillations when these methods are applied to the homogeneous linear test model. Based on this homogeneous test model we derive the dispersion conditions for symmetric DIRK methods as well as symmetric stability functions with real poles and maximal dispersion order. Two new fourth-order symmetric methods with four and five stages are obtained. One of the methods is fourth-order dispersive whereas the other method is symplectic and sixth-order dispersive. These methods have been applied to a number of test problems (linear as well as nonlinear) and some numerical results are presented to show their efficiency when they are compared with the symplectic DIRK method derived by Sanz-Serna and Abia (SIAM J. Numer. Anal. 28 (1991) 1081–1096).     

Four-stage symplectic and P-stable SDIRKN methods with dispersion of high order

New SDIRKN methods specially adapted to the numerical integration of second-order stiff ODE systems with periodic solutions are obtained. Our interest is focused on the dispersion (phase errors) of the dominant components in the numerical oscillations when these methods are applied to the homogeneous linear test model. Based on this homogeneous test model we derive the dispersion and P-stability conditions for SDIRKN methods which are assumed to be zero dissipative. Two four-stage symplectic and P-stable methods with algebraic order 4 and high order of dispersion are obtained. One of the methods is symmetric and sixth-order dispersive whereas the other method is nonsymmetric and eighth-order dispersive. These methods have been applied to a number of test problems (linear as well as nonlinear) and some numerical results are presented to show their efficiency when they are compared with other methods derived by Sharp et al. [IMA J. Numer. Anal. 10 (1990) 489–504].