Convergent stepsizes for constrained optimization algorithms

A fundamental problem in constrained nonlinear optimization algorithms is the design of a satisfactory stepsize strategy which converges to unity. In this paper, we discuss stepsize strategies for Newton or quasi-Newton algorithms which require the solution of quadratic optimization subproblems. Five stepsize strategies are considered for three different subproblems, and the conditions under which the stepsizes will converge to unity are established. It is shown that these conditions depend critically on the convergence of the Hessian approximations used in the algorithms. The stepsize strategies are constructed using basic principles from which the conditions to unit stepsizes follow. Numerical results are discussed in an Appendix.Paper presented to the XI Symposium on Mathematical Programming, Bonn, Germany, 1982.This work was completed while the author was visiting the European University in Florence where, in particular, Professors Fitoussi and Velupillai provided the opportunity for its completion. The author is grateful to Dr. L. C. W. Dixon for his helpful comments and criticisms on numerous versions of the paper, and to R. G. Becker for programming the algorithms in Section 3 and for helpful discussions concerning these algorithms.     

Adaptive stepsize based on control theory for stochastic differential equations

The numerical solution of stochastic differential equations (SDEs) has been focussed recently on the development of numerical methods with good stability and order properties. These numerical implementations have been made with fixed stepsize, but there are many situations when a fixed stepsize is not appropriate. In the numerical solution of ordinary differential equations, much work has been carried out on developing robust implementation techniques using variable stepsize. It has been necessary, in the deterministic case, to consider the “best” choice for an initial stepsize, as well as developing effective strategies for stepsize control—the same, of course, must be carried out in the stochastic case.

In this paper, proportional integral (PI) control is applied to a variable stepsize implementation of an embedded pair of stochastic Runge–Kutta methods used to obtain numerical solutions of nonstiff SDEs. For stiff SDEs, the embedded pair of the balanced Milstein and balanced implicit method is implemented in variable stepsize mode using a predictive controller for the stepsize change. The extension of these stepsize controllers from a digital filter theory point of view via PI with derivative (PID) control will also be implemented. The implementations show the improvement in efficiency that can be attained when using these control theory approaches compared with the regular stepsize change strategy.     

Modified Two-Point Stepsize Gradient Methods for Unconstrained Optimization

For unconstrained optimization, the two-point stepsize gradient method is preferable over the classical steepest descent method both in theory and in real computations. In this paper we interpret the choice for the stepsize in the two-point stepsize gradient method from the angle of interpolation and propose two modified two-point stepsize gradient methods. The modified methods are globally convergent under some mild assumptions on the objective function. Numerical results are reported, which suggest that improvements have been achieved.     

Stepsize analysis for descent methods

The convergence rates of descent methods with different stepsize rules are compared. Among the stepsize rules considered are: constant stepsize, exact minimization along a line, Goldstein-Armijo rules, and stepsize equal to that which yields the minimum of certain interpolatory polynomials. One of the major results shown is that the rate of convergence of descent methods with the Goldstein-Armijo stepsize rules can be made as close as desired to the rate of convergence of methods that require exact minimization along a line. Also, a descent algorithm that combines a Goldstein-Armijo stepsize rule with a secant-type step is presented. It is shown that this algorithm has a convergence rate equal to the convergence of descent methods that require exact minimization along a line and that, eventually (i.e., near the minimum), it does not require a search to determine an acceptable stepsize.     

The Incremental Gauss-Newton Algorithm with Adaptive Stepsize Rule

In this paper, we consider the Extended Kalman Filter (EKF) for solving nonlinear least squares problems. EKF is an incremental iterative method based on Gauss-Newton method that has nice convergence properties. Although EKF has the global convergence property under some conditions, the convergence rate is only sublinear under the same conditions. One of the reasons why EKF shows slow convergence is the lack of explicit stepsize. In the paper, we propose a stepsize rule for EKF and establish global convergence of the algorithm under the boundedness of the generated sequence and appropriate assumptions on the objective function. A notable feature of the stepsize rule is that the stepsize is kept greater than or equal to 1 at each iteration, and increases at a linear rate of k under an additional condition. Therefore, we can expect that the proposed method converges faster than the original EKF. We report some numerical results, which demonstrate that the proposed method is promising.     

Error propagation of general linear methods for ordinary differential equations

We discuss error propagation for general linear methods for ordinary differential equations up to terms of order p+2, where p is the order of the method. These results are then applied to the estimation of local discretization errors for methods of order p and for the adjacent order p+1. The results of numerical experiments confirm the reliability of these estimates. This research has applications in the design of robust stepsize and order changing strategies for algorithms based on general linear methods.     

伪谱的边界曲线及其跟踪算法的步长控制

1 引言 1990年L.N.Trefethen为度量矩阵的非正规性引入了矩阵的伪谱,对于任意的ε≥0,矩阵A∈C~(n×n)的ε-伪谱定义为 ∧_ε(A)={z:z∈σ(A △A),||△A||≤ε},其中σ(·)表示矩阵的特征值集合(谱集合),||·||表示矩阵的2-范数。利用预解式(zI—A)~(-1),     

梯度法简述

梯度法是一类求解优化问题的一阶方法。梯度法形式简单、计算开销小,在大规模问题的求解中得到了广泛应用。系统地介绍了光滑无约束问题梯度法的迭代格式、理论框架。梯度法中最重要的参数是步长,步长的选取直接决定了梯度法的收敛性质与收敛速度。从线搜索框架、近似技巧、随机技巧和交替重复步长四方面介绍了梯度步长的构造思想及相应梯度法的收敛性结果,还对非光滑及约束问题的梯度法、梯度法加速技巧和随机梯度法等扩展方向做了简要介绍。     

Adaptive Two-Point Stepsize Gradient Algorithm

Combined with the nonmonotone line search, the two-point stepsize gradient method has successfully been applied for large-scale unconstrained optimization. However, the numerical performances of the algorithm heavily depend on M, one of the parameters in the nonmonotone line search, even for ill-conditioned problems. This paper proposes an adaptive nonmonotone line search. The two-point stepsize gradient method is shown to be globally convergent with this adaptive nonmonotone line search. Numerical results show that the adaptive nonmonotone line search is specially suitable for the two-point stepsize gradient method.     

An efficient gradient method with approximate optimal stepsize for the strictly convex quadratic minimization problem

In this paper, a new type of stepsize, approximate optimal stepsize, for gradient method is introduced to interpret the Barzilai–Borwein (BB) method, and an efficient gradient method with an approximate optimal stepsize for the strictly convex quadratic minimization problem is presented. Based on a multi-step quasi-Newton condition, we construct a new quadratic approximation model to generate an approximate optimal stepsize. We then use the two well-known BB stepsizes to truncate it for improving numerical effects and treat the resulted approximate optimal stepsize as the new stepsize for gradient method. We establish the global convergence and R-linear convergence of the proposed method. Numerical results show that the proposed method outperforms some well-known gradient methods.     

Nordsieck Methods with an Off-Step Point

Hybrid methods, incorporating one or more off-step points, are difficult to implement in a variable stepsize situation using the standard representation of input and output data in each step. However, instead of representing this data in terms of solution values and derivative values at a sequence of step points, it is possible to reformulate the method so that it operates on a Nordsieck vector. This has the consequence of reducing stepsize adjustments to nothing more than rescaling the components of the Nordsieck vector. This paper shows how to derive methods in both formulations and considers some implementation details. It is also possible to derive a new type of hybrid method using the Norsieck representation as the starting point and this is also discussed in the paper. The new method is found to have comparable accuracy for corresponding work expended as for standard methods.     

Analysis of trigonometric implicit Runge–Kutta methods

Using generalized collocation techniques based on fitting functions that are trigonometric (rather than algebraic as in classical integrators), we develop a new class of multistage, one-step, variable stepsize, and variable coefficients implicit Runge–Kutta methods to solve oscillatory ODE problems. The coefficients of the methods are functions of the frequency and the stepsize. We refer to this class as trigonometric implicit Runge–Kutta (TIRK) methods. They integrate an equation exactly if its solution is a trigonometric polynomial with a known frequency. We characterize the order and A-stability of the methods and establish results similar to that of classical algebraic collocation RK methods.     

A NOTE ON THE GRADIENT PROJECTION METHOD WITH EXACT STEPSIZE RULE

In this paper, we give some convergence results on the gradient projection method with exact stepsize rule for solving the minimization problem with convex constraints. Especially, we show that if the objective function is convex and its gradient is Lipschitz continuous, then the whole sequence of iterations produced by this method with bounded exact stepsizes converges to a solution of the concerned problem.     

在一个新步长规则下梯度投影算法的全局收敛性

考虑约束最优化问题：minx∈Ωf(x)其中:f:R^n→R是连续可微函数，Ω是一闭凸集。本文研究了解决此问题的梯度投影方法，在步长的选取时采用了一种新的策略，在较弱的条件下，证明了梯度投影响方法的全局收敛性。     

On the bounds of approximations of holomorphic semigroups

The upper bounds of variable stepsize approximations of holomorphic semigroups are derived.     

Experiments with a New Fifth Order Method

Almost Runge–Kutta methods were introduced to obtain many of the advantages of Runge–Kutta methods without their disadvantages. We consider the construction of fourth order methods of this type with a special choice of the free parameters to ensure that, at least for constant stepsize, order 5 behaviour is achieved. It is shown how this can be extended to variable stepsize.     

Equilibrium states for predictor-corrector methods

A theory is developed that explains the stepsize patterns observed when standard predictor-corrector methods with variable stepsize strategy are used to solve stiff or mildly stiff problems. In some cases an algorithmic steady state occurs with smooth almost constant stepsizes; at other times an oscillating stepsize pattern of stepsizes is observed with the possibility of frequent rejected steps.     

A Strong Approximation Theorem for Stochastic Recursive Algorithms

The constant stepsize analog of Gelfand–Mitter type discrete-time stochastic recursive algorithms is shown to track an associated stochastic differential equation in the strong sense, i.e., with respect to an appropriate divergence measure.     

Modified Goldstein–Levitin–Polyak Projection Method for Asymmetric Strongly Monotone Variational Inequalities

In this paper, we present a modified Goldstein–Levitin–Polyak projection method for asymmetric strongly monotone variational inequality problems. A practical and robust stepsize choice strategy, termed self-adaptive procedure, is developed. The global convergence of the resulting algorithm is established under the same conditions used in the original projection method. Numerical results and comparison with some existing projection-type methods are given to illustrate the efficiency of the proposed method.     

带有固定步长的非单调自适应信赖域算法

提出了求解无约束优化问题带有固定步长的非单调自适应信赖域算法.信赖域半径的修正采用自适应技术,算法在试探步不被接受时,采用固定步长寻找下一迭代点.并在适当的条件下,证明算法具有全局收敛性和超线性收敛性.初步的数值试验表明算法对高维问题具有较好的效果.