KKT Conditions for Rank-Deficient Nonlinear Least-Square Problems with Rank-Deficient Nonlinear Constraints

In nonlinear least-square problems with nonlinear constraints, the function , where f ₂ is a nonlinear vector function, is to be minimized subject to the nonlinear constraints f ₁(x)=0. This problem is ill-posed if the first-order KKT conditions do not define a locally unique solution. We show that the problem is ill-posed if either the Jacobian of f ₁ or the Jacobian of J is rank-deficient (i.e., not of full rank) in a neighborhood of a solution satisfying the first-order KKT conditions. Either of these ill-posed cases makes it impossible to use a standard Gauss–Newton method. Therefore, we formulate a constrained least-norm problem that can be used when either of these ill-posed cases occur. By using the constant-rank theorem, we derive the necessary and sufficient conditions for a local minimum of this minimum-norm problem. The results given here are crucial for deriving methods solving the rank-deficient problem.     

Indirect boundary integral formulation for the solution of shear‐thinning flow inside single rotor mixers

An efficient indirect boundary integral formulation for the evaluation of inelastic non‐Newtonian shear‐thinning flows at low Reynolds number is presented in this article. The formulation is based on the solution of a homogeneous Stokes flow field and the use of a particular solution for the nonlinear non‐Newtonian terms that yields the complete solution to the problem. Matrix multiplications are reduced in comparison to other means of handling nonlinear terms in boundary integral formulations such as the dual reciprocity method. The iterative solution of the nonlinear system of equations has been performed with a modified Newton‐Raphson method obtaining accurate results for values of the power law index as low as 0.4 without domain partitioning. Geometries such as Couette flow and a typical industrial polymer mixer have been analyzed with the proposed method obtaining good results with a reduction in computational cost compared with other equivalent formulations. © 2010 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 27:1610–1627, 2011     

Splitting methods for second‐order initial value problems

We consider implicit integration methods for the solution of stiff initial value problems for second-order differential equations of the special form y' = f(y). In implicit methods, we are faced with the problem of solving systems of implicit relations. This paper focuses on the construction and analysis of iterative solution methods which are effective in cases where the Jacobian of the right‐hand side of the differential equation can be split into a sum of matrices with a simple structure. These iterative methods consist of the modified Newton method and an iterative linear solver to deal with the linear Newton systems. The linear solver is based on the approximate factorization of the system matrix associated with the linear Newton systems. A number of convergence results are derived for the linear solver in the case where the Jacobian matrix can be split into commuting matrices. Such problems often arise in the spatial discretization of time‐dependent partial differential equations. Furthermore, the stability matrix and the order of accuracy of the integration process are derived in the case of a finite number of iterations. This revised version was published online in August 2006 with corrections to the Cover Date.     

A smoothing SQP method for nonlinear programs with?stability constraints arising from power systems

This paper investigates a new class of optimization problems arising from power systems, known as nonlinear programs with stability constraints (NPSC), which is an extension of ordinary nonlinear programs. Since the stability constraint is described generally by eigenvalues or norm of Jacobian matrices of systems, this results in the semismooth property of NPSC problems. The optimal conditions of both NPSC and its smoothing problem are studied. A smoothing SQP algorithm is proposed for solving such optimization problem. The global convergence of algorithm is established. A numerical example from optimal power flow (OPF) is done. The computational results show efficiency of the new model and algorithm.     

Riemannian Newton-type methods for joint diagonalization on the Stiefel manifold with application to independent component analysis

The joint approximate diagonalization of non-commuting symmetric matrices is an important process in independent component analysis. This problem can be formulated as an optimization problem on the Stiefel manifold that can be solved using Riemannian optimization techniques. Among the available optimization techniques, this study utilizes the Riemannian Newton’s method for the joint diagonalization problem on the Stiefel manifold, which has quadratic convergence. In particular, the resultant Newton’s equation can be effectively solved by means of the Kronecker product and the vec and veck operators, which reduce the dimension of the equation to that of the Stiefel manifold. Numerical experiments are performed to show that the proposed method improves the accuracy of the approximate solution to this problem. The proposed method is also applied to independent component analysis for the image separation problem. The proposed Newton method further leads to a novel and fast Riemannian trust-region Newton method for the joint diagonalization problem.     

A family of constrained pressure residual preconditioners for parallel reservoir simulations

Large‐scale reservoir simulations are extremely time‐consuming because of the solution of large‐scale linear systems arising from the Newton or Newton–Raphson iterations. The problem becomes even worse when highly heterogeneous geological models are employed. This paper introduces a family of multi‐stage preconditioners for parallel black oil simulations, which are based on the famous constrained pressure residual preconditioner. Numerical experiments demonstrate that our preconditioners are robust, efficient, and scalable. Copyright © 2015 John Wiley & Sons, Ltd.     

Semilocal and global convergence of the Newton‐HSS method for systems of nonlinear equations

Newton‐HSS methods, which are variants of inexact Newton methods different from the Newton–Krylov methods, have been shown to be competitive methods for solving large sparse systems of nonlinear equations with positive‐definite Jacobian matrices (J. Comp. Math. 2010; 28 :235–260). In that paper, only local convergence was proved. In this paper, we prove a Kantorovich‐type semilocal convergence. Then we introduce Newton‐HSS methods with a backtracking strategy and analyse their global convergence. Finally, these globally convergent Newton‐HSS methods are shown to work well on several typical examples using different forcing terms to stop the inner iterations. Copyright © 2010 John Wiley & Sons, Ltd.     

Investigating the Newton–Raphson basins of attraction in the restricted three-body problem with modified Newtonian gravity

The planar circular restricted three-body problem with modified Newtonian gravity is used in order to determine the Newton–Raphson basins of attraction associated with the equilibrium points. The evolution of the position of the five Lagrange points is monitored when the value of the power p of the gravitational potential of the second primary varies in predefined intervals. The regions on the configuration (x, y) plane occupied by the basins of attraction are revealed using the multivariate version of the Newton–Raphson iterative scheme. The correlations between the basins of convergence of the equilibrium points and the corresponding number of iterations needed for obtaining the desired accuracy are also illustrated. We conduct a thorough and systematic numerical investigation by demonstrating how the dynamical quantity p influences the shape as well as the geometry of the basins of attractions. Our results strongly suggest that the power p is indeed a very influential parameter in both cases of weaker or stronger Newtonian gravity.     

An improved approximate Newton method for implicit Runge-Kutta formulas

Implicit Runge-Kutta (IRK) methods (such as the s-stage Radau IIA method with s=3,5, or 7) for solving stiff ordinary differential equation systems have excellent stability properties and high solution accuracy orders, but their high computing costs in solving their nonlinear stage equations have seriously limited their applications to large scale problems. To reduce such a cost, several approximate Newton algorithms were developed, including a commonly used one called the simplified Newton method. In this paper, a new approximate Jacobian matrix and two new test rules for controlling the updating of approximate Jacobian matrices are proposed, yielding an improved approximate Newton method. Theoretical and numerical analysis show that the improved approximate Newton method can significantly improve the convergence and performance of the simplified Newton method.     

An interior penalty method for a finite-dimensional linear complementarity problem in financial engineering

In this work we study an interior penalty method for a finite-dimensional large-scale linear complementarity problem (LCP) arising often from the discretization of stochastic optimal problems in financial engineering. In this approach, we approximate the LCP by a nonlinear algebraic equation containing a penalty term linked to the logarithmic barrier function for constrained optimization problems. We show that the penalty equation has a solution and establish a convergence theory for the approximate solutions. A smooth Newton method is proposed for solving the penalty equation and properties of the Jacobian matrix in the Newton method have been investigated. Numerical experimental results using three non-trivial test examples are presented to demonstrate the rates of convergence, efficiency and usefulness of the method for solving practical problems.     

On the convergence of Newton iterations to non-stationary points

We study conditions under which line search Newton methods for nonlinear systems of equations and optimization fail due to the presence of singular non-stationary points. These points are not solutions of the problem and are characterized by the fact that Jacobian or Hessian matrices are singular. It is shown that, for systems of nonlinear equations, the interaction between the Newton direction and the merit function can prevent the iterates from escaping such non-stationary points. The unconstrained minimization problem is also studied, and conditions under which false convergence cannot occur are presented. Several examples illustrating failure of Newton iterations for constrained optimization are also presented. The paper also shows that a class of line search feasible interior methods cannot exhibit convergence to non-stationary points. This author was supported by Air Force Office of Scientific Research grant F49620-00-1-0162, Army Research Office Grant DAAG55-98-1-0176, and National Science Foundation grant INT-9726199.This author was supported by Department of Energy grant DE-FG02-87ER25047-A004.This author was supported by National Science Foundation grant CCR-9987818 and Department of Energy grant DE-FG02-87ER25047-A004.     

Interior point models for power system stability problems

In this paper, a detailed analysis of the use of optimization techniques in the study of voltage stability problems, leading to the incorporation of voltage stability criteria in traditional Optimal Power Flow (OPF) formulations is presented. Optimal power flow problems are highly nonlinear programming problems that are used to find the optimal control settings in electrical power systems. The relationship between the Lagrangian Multipliers of the OPF problem and the classification of the maximum loading point level of the system is given. Finally, the paper presents a sequential OPF technique to enhance voltage stability using reactive power and voltage rescheduling with no increase in real (active) generation cost.     

Solvability of Newton equations in smoothing-type algorithms for the SOCCP

In this paper, we first investigate the invertibility of a class of matrices. Based on the obtained results, we then discuss the solvability of Newton equations appearing in the smoothing-type algorithm for solving the second-order cone complementarity problem (SOCCP). A condition ensuring the solvability of such a system of Newton equations is given. In addition, our results also show that the assumption that the Jacobian matrix of the function involved in the SOCCP is a P₀-matrix is not enough for ensuring the solvability of such a system of Newton equations, which is different from the one of smoothing-type algorithms for solving many traditional optimization problems in ℜⁿ.     

Matrix-free monolithic homotopy continuation with application to computational aerodynamics

A matrix-free monolithic homotopy continuation algorithm is developed which allows for approximate numerical solutions to nonlinear systems of equations without the need to solve a linear system, thereby avoiding the formation of any Jacobian or preconditioner matrices. The algorithm can converge from an arbitrary starting guess, under suitable conditions, and can give a sufficiently accurate approximation to the converged solution such that a rapid locally convergent method such as Newton’s method will converge successfully. Several forms of the algorithm are presented, as are augmentations to the algorithms which can lead to improved efficiency or stability. The method is validated and the stability and efficiency are investigated numerically based on a computational aerodynamics flow solver.     

A smoothing Newton's method for the construction of a damped vibrating system from noisy test eigendata

In this paper we consider an inverse problem for a damped vibration system from the noisy measured eigendata, where the mass, damping, and stiffness matrices are all symmetric positive‐definite matrices with the mass matrix being diagonal and the damping and stiffness matrices being tridiagonal. To take into consideration the noise in the data, the problem is formulated as a convex optimization problem involving quadratic constraints on the unknown mass, damping, and stiffness parameters. Then we propose a smoothing Newton‐type algorithm for the optimization problem, which improves a pre‐existing estimate of a solution to the inverse problem. We show that the proposed method converges both globally and quadratically. Numerical examples are also given to demonstrate the efficiency of our method. Copyright © 2008 John Wiley & Sons, Ltd.     

Greedy Gauss-Newton algorithms for finding sparse solutions to nonlinear underdetermined systems of equations

We consider the problem of finding sparse solutions to a system of underdetermined non-linear system of equations. The methods are based on a Gauss–Newton approach with line search where the search direction is found by solving a linearized problem using only a subset of the columns in the Jacobian. The choice of columns in the Jacobian is made through a greedy approach looking at either maximum descent or an approach corresponding to orthogonal matching for linear problems. The methods are shown to be convergent and efficient and outperform the l1 approach on the test problems presented.     

On the final steps of Newton and higher order methods

The Newton method is one of the most used methods for solving nonlinear system of equations when the Jacobian matrix is nonsingular. The method converges to a solution with Q-order two for initial points sufficiently close to the solution. The method of Halley and the method of Chebyshev are among methods that have local and cubic rate of convergence. Combining these methods with a backtracking and curvilinear strategy for unconstrained optimization problems these methods have been shown to be globally convergent. The backtracking forces a strict decrease of the function of the unconstrained optimization problem. It is shown that no damping of the step in the backtracking routine is needed close to a strict local minimizer and the global method behaves as a local method. The local behavior for the unconstrained optimization problem is investigated by considering problems with two unknowns and it is shown that there are no significant differences in the region where the global method turn into a local method for second and third order methods. Further, the final steps to reach a predefined tolerance are investigated. It is shown that the region where the higher order methods terminate in one or two iteration is significantly larger than the corresponding region for Newton’s method.     

Nonsingularity in matrix conic optimization induced by spectral norm via a smoothing metric projector

Matrix conic optimization induced by spectral norm (MOSN) has found important applications in many fields. This paper focus on the optimality conditions and perturbation analysis of the MOSN problem. The Karush–Kuhn–Tucker (KKT) conditions of the MOSN problem can be reformulated as a nonsmooth system via the metric projector over the cone. We show in this paper, the nonsingularity of the Clarke’s generalized Jacobian of the smoothing KKT system constructed by a smoothing metric projector, the strong regularity and the strong second-order sufficient condition under constraint nondegeneracy are all equivalent. Moreover, this nonsingularity is used in several globally convergent smoothing Newton methods.     

Nonparametric Estimation of Hazard Rate Under the Constraint of Monotonicity

This article shows how to smoothly “monotonize” standard kernel estimators of hazard rate, using bootstrap weights. Our method takes a variety of forms, depending on choice of kernel estimator and on the distance function used to define a certain constrained optimization problem. We confine attention to a particularly simple kernel approach and explore a range of distance functions. It is straightforward to reduce “quadratic” inequality constraints to “linear” equality constraints, and so our method may be implemented using little more than conventional Newton–Raphson iteration. Thus, the necessary computational techniques are very familiar to statisticians. We show both numerically and theoretically that monotonicity, in either direction, can generally be imposed on a kernel hazard rate estimator regardless of the monotonicity or otherwise of the true hazard rate. The case of censored data is easily accommodated. Our methods have straightforward extension to the problem of testing for monotonicity of hazard rate, where the distance function plays the role of a test statistic.