A new approach to constrained function optimization

A new approach to the constrained function optimization problem is presented. It is shown that the ordinary Lagrange multiplier method and the penalty function method may be generalized and combined, and the new concept ofmultiplier function is introduced. The problem may then be converted into an unconstrained well-conditioned optimization problem. Methods for numerical solution are discussed, and new algorithms are derived.The author wishes to express his gratitude to Professor K. J. Åström for his encouragement and assistance and to Professor P. Falb for valuable suggested improvements. This work was supported by the Swedish Board for Technical Development, Contract No. 70-337/U270.     

An optimization‐based domain decomposition method for numerical simulation of the incompressible Navier‐Stokes flows

This article is concerned about an optimization‐based domain decomposition method for numerical simulation of the incompressible Navier‐Stokes flows. Using the method, an classical domain decomposition problem is transformed into a constrained minimization problem for which the objective functional is chosen to measure the jump in the dependent variables across the common interfaces between subdomains. The Lagrange multiplier rule is used to transform the constrained optimization problem into an unconstrained one and that rule is applied to derive an optimality system from which optimal solutions may be obtained. The optimality system is also derived using “sensitivity” derivatives instead of the Lagrange multiplier rule. We consider a gradient‐type approach to the solution of domain decomposition problem. The results of some numerical experiments are presented to demonstrate the feasibility and applicability of the algorithm developed in this article. © 2009 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2011     

k-均值问题的差分隐私算法综述

$k$-均值问题是机器学习和组合优化领域十分重要的问题。它是经典的NP-难问题, 被广泛的应用于数据挖掘、企业生产决策、图像处理、生物医疗科技等领域。随着时代的发展, 人们越来越注重于个人的隐私保护:在决策通常由人工智能算法做出的情况下, 如何保证尽可能多地从数据中挖掘更多信息,同时不泄露个人隐私。近十年来不断有专家学者研究探索带隐私保护的$k$-均值问题, 得到了许多具有理论指导意义和实际应用价值的结果, 本文主要介绍关于$k$-均值问题的差分隐私算法供读者参考。     

对等式约束非线性规划问题的Hestenes-Powell增广拉格朗日函数的进一步研究

本文对用无约束极小化方法求解等式约束非线性规划问题的Hestenes-Powell 增广拉格朗日函数作了进一步研究．在适当的条件下,我们建立了Hestenes-Powell增广拉格朗日函数在原问题变量空间上的无约束极小与原约束问题的解之间的关系,并且也给出了Hestenes-Powell增广拉格朗日函数在原问题变量和乘子变量的积空间上的无约束极小与原约束问题的解之间的一个关系．因此,从理论的观点来看,原约束问题的解和对应的拉格朗日乘子值不仅可以用众所周知的乘子法求得,而且可以通过对Hestenes-Powell 增广拉格朗日函数在原问题变量和乘子变量的积空间上执行一个单一的无约束极小化来获得．     

On the use of the energy norm in trust-region and adaptive cubic regularization subproblems

We propose a multi-time scale quasi-Newton based smoothed functional (QN-SF) algorithm for stochastic optimization both with and without inequality constraints. The algorithm combines the smoothed functional (SF) scheme for estimating the gradient with the quasi-Newton method to solve the optimization problem. Newton algorithms typically update the Hessian at each instant and subsequently (a) project them to the space of positive definite and symmetric matrices, and (b) invert the projected Hessian. The latter operation is computationally expensive. In order to save computational effort, we propose in this paper a quasi-Newton SF (QN-SF) algorithm based on the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update rule. In Bhatnagar (ACM TModel Comput S. 18(1): 27–62, 2007), a Jacobi variant of Newton SF (JN-SF) was proposed and implemented to save computational effort. We compare our QN-SF algorithm with gradient SF (G-SF) and JN-SF algorithms on two different problems – first on a simple stochastic function minimization problem and the other on a problem of optimal routing in a queueing network. We observe from the experiments that the QN-SF algorithm performs significantly better than both G-SF and JN-SF algorithms on both the problem settings. Next we extend the QN-SF algorithm to the case of constrained optimization. In this case too, the QN-SF algorithm performs much better than the JN-SF algorithm. Finally we present the proof of convergence for the QN-SF algorithm in both unconstrained and constrained settings.     

Generalized Lagrange multiplier technique for nonlinear programming

Our aim here is to present numerical methods for solving a general nonlinear programming problem. These methods are based on transformation of a given constrained minimization problem into an unconstrained maximin problem. This transformation is done by using a generalized Lagrange multiplier technique. Such an approach permits us to use Newton's and gradient methods for nonlinear programming. Convergence proofs are provided, and some numerical results are given.     

Lagrange Multiplier Rules for Weak Approximate Pareto Solutions of Constrained Vector Optimization Problems in Hilbert Spaces

In the Hilbert space case, in terms of proximal normal cone and proximal coderivative, we establish a Lagrange multiplier rule for weak approximate Pareto solutions of constrained vector optimization problems. In this case, our Lagrange multiplier rule improves the main result on vector optimization in Zheng and Ng (SIAM J. Optim. 21: 886–911, 2011). We also introduce a notion of a fuzzy proximal Lagrange point and prove that each Pareto (or weak Pareto) solution is a fuzzy proximal Lagrange point.     

Solving quadratically constrained least squares using black box solvers

We present algorithms for solving quadratically constrained linear least squares problems that do not necessarily require expensive dense matrix factorizations. Instead, only black box solvers for certain related unconstrained least squares problems, as well as the solution of two related linear systems involving the coefficient matrixA and the constraint matrixB, are required. Special structures in the problem can thus be exploited in these solvers, and iterative as well as direct solvers can be used. Our approach is to solve for the Lagrange multiplier as the root of an implicitly-defined secular equation. We use both a linear and a rational (Hebden) local model and a Newton and secant method. We also derive a formula for estimating the Lagrange multiplier which depends on the amount the unconstrained solution violates the constraint and an estimate of the smallest generalized singular value ofA andB. The Lagrange multiplier estimate can be used as a good initial guess for solving the secular equation. We also show conditions under which this estimate is guaranteed to be an acceptable solution without further refinement. Numerical results comparing the different algorithms are presented.Research supported by SRI International and by the National Science Foundation under grants DMS-87-14612 and ASC 9003002.     

Optimization of unconstrained functions with sparse Hessian matrices—Quasi-Newton methods

Newton-type methods and quasi-Newton methods have proven to be very successful in solving dense unconstrained optimization problems. Recently there has been considerable interest in extending these methods to solving large problems when the Hessian matrix has a known a priori sparsity pattern. This paper treats sparse quasi-Newton methods in a uniform fashion and shows the effect of loss of positive-definiteness in generating updates. These sparse quasi-Newton methods coupled with a modified Cholesky factorization to take into account the loss of positive-definiteness when solving the linear systems associated with these methods were tested on a large set of problems. The overall conclusions are that these methods perform poorly in general—the Hessian matrix becomes indefinite even close to the solution and superlinear convergence is not observed in practice. Research for this paper was performed at the Department of Operations Research, Stanford, CA 94305. The research was partially supported by the Department of Energy Contract AM03-76SF00326. PA# DE-AT03-76ER72018: Office of Naval Research Contract N00014-75-C-0267; National Science Foundation Grants MCS-7681259, MCS-7926009 and ECS-8012974.     

A gradient projection-multiplier method for nonlinear programming

This paper describes a gradient projection-multiplier method for solving the general nonlinear programming problem. The algorithm poses a sequence of unconstrained optimization problems which are solved using a new projection-like formula to define the search directions. The unconstrained minimization of the augmented objective function determines points where the gradient of the Lagrangian function is zero. Points satisfying the constraints are located by applying an unconstrained algorithm to a penalty function. New estimates of the Lagrange multipliers and basis constraints are made at points satisfying either a Lagrangian condition or a constraint satisfaction condition. The penalty weight is increased only to prevent cycling. The numerical effectiveness of the algorithm is demonstrated on a set of test problems.The author gratefully acknowledges the helpful suggestions of W. H. Ailor, J. L. Searcy, and D. A. Schermerhorn during the preparation of this paper. The author would also like to thank D. M. Himmelblau for supplying a number of interesting test problems.     

A lagrange multiplier rule in Hilbert space

A representation theorem in Hilbert space is used to obtain a Lagrange multiplier rule for a large class of linear problems, which includes the linear optimal control problem. Several examples are presented.The preparation of this paper was sponsored in part by the U. S. Army Research Office, Grant No. DA-31-124-ARO(D)-355. This paper combines results arrived at independently by the first two authors and by the third. The authors wish to thank Professor M. R. Hestenes for many helpful suggestions.     

A Switching Algorithm Based on Modified Quasi-Newton Equation

In this paper, a switching method for unconstrained minimization is proposed. The method is based on the modified BFGS method and the modified SR1 method. The eigenvalues and condition numbers of both the modified updates are evaluated and used in the switching rule. When the condition number of the modified SR1 update is superior to the modified BFGS update, the step in the proposed quasi-Newton method is the modified SR1 step. Otherwise the step is the modified BFGS step. The efficiency of the proposed method is tested by numerical experiments on small, medium and large scale optimization. The numerical results are reported and analyzed to show the superiority of the proposed method.     

Solving Quadratically Constrained Least Squares Problems Using a Differential-Geometric Approach

A quadratically constrained linear least squares problem is usually solved using a Lagrange multiplier for the constraint and then solving iteratively a nonlinear secular equation for the optimal Lagrange multiplier. It is well-known that, due to the closeness to a pole for the secular equation, standard methods for solving the secular equation can be slow, and sometimes it is not easy to select a good starting value for the iteration. The problem can be reformulated as that of minimizing the residual of the least squares problem on the unit sphere. Using a differential-geometric approach we formulate Newton's method on the sphere, and thereby avoid the difficulties associated with the Lagrange multiplier formulation. This Newton method on the sphere can be implemented efficiently, and since it is easy to find a good starting value for the iteration, and the convergence is often quite fast, it has a clear advantage over the Lagrange multiplier method. A numerical example is given.     

A primal-dual Newton-type algorithm for geometric programs with equality constraints

An augmented Lagrangian algorithm is used to find local solutions of geometric programming problems with equality constraints (GPE). The algorithm is Newton's method for unconstrained minimization. The complexity of the algorithm is defined by the number of multiplications and divisions. By analyzing the algorithm we obtain results about the influence of each parameter in the GPE problem on the complexity of an iteration. An attempt is made to estimate the number of iterations needed for convergence. In practice, certain hypotheses are tested, such as the influence of the penalty coefficient update formula, the distance of the starting point from the optimum, and the stopping criterion. For these tests, a random problem generator was constructed, many problems were run, and the results were analyzed by statistical methods.The authors are grateful to Dr. J. Moré, Argonne National Laboratory for his valuable comments.This research was partially funded by the Fund for the Advancement of Research at the Technion and by the Applied Mathematical Sciences Research Program (KC-04-02), Office of Energy Research, US Department of Energy, Contract No. W-31-109-Eng-38.     

A class of superlinearly convergent projection algorithms with relaxed stepsizes

A quasi-Newton extension of the Goldstein-Levitin-Polyak (GLP) projected gradient algorithm for constrained optimization is considered. Essentially, this extension projects an unconstrained descent step on to the feasible region. The determination of the stepsize is divided into two stages. The first is a stepsize sequence, chosen from the range [1,2], converging to unity. This determines the size of the unconstrained step. The second is a stepsize chosen from the range [0,1] according to a stepsize strategy and determines the length of the projected step. Two such strategies are considered. The first bounds the objective function decrease by a conventional linear functional, whereas the second uses a quadratic functional as a bound.The introduction of the unconstrained step provides the option of taking steps that are larger than unity. It is shown that unit steplengths and subsequently superlinear convergence rates are attained if the projection of the quasi-Newton Hessian approximation approaches the projection of the Hessian at the solution. Thus, the requirement in the GLP algorithm for a positive definite Hessian at the solution is relaxed. This allows the use of strictly positive definite Hessian approximations, thereby simplifying the quadratic subproblem involved, even if the Hessian at the solution is not strictly positive definite.This research was funded by a Science and Engineering Research Council Advanced Fellowship. The author is also grateful to an anonymous referee for numerous constructive criticisms and comments.     

On vector variational-like inequalities and set-valued optimization problems

In this paper, some solution relationships between set-valued optimization problems and vector variational-like inequalities are established under generalized invexities. In addition, a generalized Lagrange multiplier rule for a constrained set-valued optimization problem is obtained under C-preinvexity.     

The lagrange multiplier set and the generalized gradient set of the marginal function of a differentiable program in a Banach space

We prove that, under the usual constraint qualification and a stability assumption, the generalized gradient set of the marginal function of a differentiable program in a Banach space contains the Lagrange multiplier set. From there, we deduce a sufficient condition in order that, in finite-dimensional spaces, the Lagrange multiplier set be equal to the generalized gradient set of the marginal function.The author wishes to thank J. B. Hiriart-Urruty for many helpful suggestions during the preparation of this paper. He also wishes to express his appreciation to the referees for their many valuable comments.     

Nonzero lagrange multipliers

Solutions of constrained minimization problems give rise to Lagrange multiplier rules. In this paper, we show that a simple condition on a specific constraint implies that the associated coefficient in the Lagrange multiplier rule is not zero. We conclude with an example which shows that such knowledge increases the information available about the solution of a problem of minimal curvature.This work supported in part by NSF Grant No. MCS-75-05581-A01.     

A SELF-ADAPTIVE ALGORITHM FOR NONLINEAR LEAST SQUARES WITH LINEAR CONSTRAINTS

An algorithm for solving nonlinear least squares problems with general linear inequality constraints is described.At each step,the problem is reduced to an unconstrained linear least squares problem in a subs pace defined by the active constraints,which is solved using the quasi-Newton method.The major update formula is similar to the one given by Dennis,Gay and Welsch (1981).In this paper,we state the detailed implement of the algorithm,such as the choice of active set,the solution of subproblem and the avoidance of zigzagging.We also prove the globally convergent property of the algorithm.     

A general multiplier rule

A general multiplier rule which is an extension of the multiplier rule given by Hestenes is proved. This multiplier rule is applied to obtain the necessary conditions given by Neustadt for a solution to a canonical optimization problem which includes many optimal control problems as special cases.This research is part of a dissertation submitted in partial satisfaction of the requirements for the Ph.D. degree in mathematics at the University of California, Los Angeles. The author would like to express his appreciation to Dr. M. R. Hestenes for his guidance.