A stable approach to Newton's method for general mathematical programming problems inR n

The usual approach to Newton's method for mathematical programming problems with equality constraints leads to the solution of linear systems ofn +m equations inn +m unknowns, wheren is the dimension of the space andm is the number of constraints. Moreover, these linear systems are never positive definite. It is our feeling that this approach is somewhat artificial, since in the unconstrained case the linear systems are very often positive definite. With this in mind, we present an alternate Newton-like approach for the constrained problem in which all the linear systems are of order less than or equal ton. Furthermore, when the Hessian of the Lagrangian at the solution is positive definite (a situation frequently occurring), all our systems will be positive definite. Hence, in all cases, our Newton-like method offers greater numerical stability. We demonstrate that the convergence properties of this Newton-like method are superior to those of the standard approach to Newton's method. The operation count for the new method using Gaussian elimination is of the same order as the operation count for the standard method. However, if the Hessian of the Lagrangian at the solution is positive definite and we use Cholesky decomposition, then the order of the operation count for the new method is half that for the standard approach to Newton's method. This theory is generalized to problems with both equality and inequality constraints.     

Local convergence in Fermat's problem

The general Fermat problem is to find the minimum of the weighted sum of distances fromm destination points in Euclideann-space. Kuhn recently proved that a classical iterative algorithm converges to the unique minimizing point , for any choice of the initial point except for a denumerable set. In this note, it is shown that although convergence is global, the rapidity of convergence depends strongly upon whether or not  is a destination. If  is not a destination, then locally convergence is always linear with upper and lower asymptotic convergence boundsλ andλ′ (λ ≥ 1/2, whenn=2). If  is a destination, then convergence can be either linear, quadratic or sublinear. Three numerical examples which illustrate the different possibilities are given and comparisons are made with the use of Steffensen's scheme to accelerate convergence.     

A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training

Support vector machines (SVMs) training may be posed as a large quadratic program (QP) with bound constraints and a single linear equality constraint. We propose a (block) coordinate gradient descent method for solving this problem and, more generally, linearly constrained smooth optimization. Our method is closely related to decomposition methods currently popular for SVM training. We establish global convergence and, under a local error bound assumption (which is satisfied by the SVM QP), linear rate of convergence for our method when the coordinate block is chosen by a Gauss-Southwell-type rule to ensure sufficient descent. We show that, for the SVM QP with n variables, this rule can be implemented in O(n) operations using Rockafellar’s notion of conformal realization. Thus, for SVM training, our method requires only O(n) operations per iteration and, in contrast to existing decomposition methods, achieves linear convergence without additional assumptions. We report our numerical experience with the method on some large SVM QP arising from two-class data classification. Our experience suggests that the method can be efficient for SVM training with nonlinear kernel.     

An interior proximal method in vector optimization

This paper studies the vector optimization problem of finding weakly efficient points for maps from Rⁿ to R^m, with respect to the partial order induced by a closed, convex, and pointed cone C⊂R^m, with nonempty interior. We develop for this problem an extension of the proximal point method for scalar-valued convex optimization problem with a modified convergence sensing condition that allows us to construct an interior proximal method for solving VOP on nonpolyhedral set.     

Global linear convergence of a path-following algorithm for some monotone variational inequality problems

We consider a primal-scaling path-following algorithm for solving a certain class of monotone variational inequality problems. Included in this class are the convex separable programs considered by Monteiro and Adler and the monotone linear complementarity problem. This algorithm can start from any interior solution and attain a global linear rate of convergence with a convergence ratio of 1 ?c/√m, wherem denotes the dimension of the problem andc is a certain constant. One can also introduce a line search strategy to accelerate the convergence of this algorithm.     

Probability of unique integer solution to a system of linear equations

We consider a system of m linear equations in n variables Ax = d and give necessary and sufficient conditions for the existence of a unique solution to the system that is integer: x ∈ {−1, 1}ⁿ. We achieve this by reformulating the problem as a linear program and deriving necessary and sufficient conditions for the integer solution to be the unique primal optimal solution. We show that as long as m is larger than n/2, then the linear programming reformulation succeeds for most instances, but if m is less than n/2, the reformulation fails on most instances. We also demonstrate that these predictions match the empirical performance of the linear programming formulation to very high accuracy.     

Bound reduction using pairs of linear inequalities

We describe a procedure to reduce variable bounds in mixed integer nonlinear programming (MINLP) as well as mixed integer linear programming (MILP) problems. The procedure works by combining pairs of inequalities of a linear programming (LP) relaxation of the problem. This bound reduction procedure extends the feasibility based bound reduction technique on linear functions, used in MINLP and MILP. However, it can also be seen as a special case of optimality based bound reduction, a method to infer variable bounds from an LP relaxation of the problem. For an LP relaxation with m constraints and n variables, there are O(m ²) pairs of constraints, and a naïve implementation of our bound reduction scheme has complexity O(n ³) for each pair. Therefore, its overall complexity O(m ² n ³) can be prohibitive for relatively large problems. We have developed a more efficient procedure that has complexity O(m ² n ²), and embedded it in two Open-Source solvers: one for MINLP and one for MILP. We provide computational results which substantiate the usefulness of this bound reduction technique for several instances.     

An application of homotopy to solving linear programs

Consider a linear program inm inequality constraints andn nonnegative variables. An application of homotopy to the problem gives an algorithm similar to Dantzig's self-dual method. Howeve, the homotopy approach allows one to recognize several previously undescribed and potentially interesting properties. For example, the algorithm can be initiated in such a way as to produce a path which is primal-dual feasible. Moreover, one can theoretically identify an orthant with the property that if one initiates the algorithm at any point in that orthant then, after a ‘phase I’ requiring at most min{m, n} pivots, convergence is obtained in one step.     

On the connection between weak and strong convergencies in Cm

In this work wome connections are pursued between weak and strong convergence in the spaces C^m (m-times continuously differentiable functions on Rⁿ). Let f_n, f?C^{m + 1}, where n = 1, 2,…, and m is a nonnegative integer. Suppose that the sequence {f_n} converges to f relative to the weak topology of C^{m + 1}. It is shown that this implies the convergence of {f_n} to f with respect to the strong topology of C^m. Several corollaries to this theorem are established; among them is a sufficient condition for uniform convergence. A stronger result is shown to exist when the sequence constitutes an output sequence of a linear weakly continuous operator.     

Solving Quadratic Equations via PhaseLift When There Are About as Many Equations as Unknowns

This note shows that we can recover any complex vector \(\boldsymbol {x}_{0} \in \mathbb {C}^{n}\) exactly from on the order of n quadratic equations of the form |〈a _i ,x ₀〉|²=b _i , i=1,…,m, by using a semidefinite program known as PhaseLift. This improves upon earlier bounds in Candès et al. (Commun. Pure Appl. Math. 66:1241–1274, 2013), which required the number of equations to be at least on the order of nlogn. Further, we show that exact recovery holds for all input vectors simultaneously, and also demonstrate optimal recovery results from noisy quadratic measurements; these results are much sharper than previously known results.     

Approximate methods for convex minimization problems with series–parallel structure

Consider a problem of minimizing a separable, strictly convex, monotone and differentiable function on a convex polyhedron generated by a system of m linear inequalities. The problem has a series–parallel structure, with the variables divided serially into n disjoint subsets, whose elements are considered in parallel. This special structure is exploited in two algorithms proposed here for the approximate solution of the problem. The first algorithm solves at most min{m, ν − n + 1} subproblems; each subproblem has exactly one equality constraint and at most n variables. The second algorithm solves a dynamically generated sequence of subproblems; each subproblem has at most ν − n + 1 equality constraints, where ν is the total number of variables. To solve these subproblems both algorithms use the authors’ Projected Newton Bracketing method for linearly constrained convex minimization, in conjunction with the steepest descent method. We report the results of numerical experiments for both algorithms.     

Construction of highly nonlinear resilient S-boxes with given degree

We provide two new construction methods for nonlinear resilient S-boxes with given degree. The first method is based on the use of linear error correcting codes together with highly nonlinear S-boxes. Given a [u, m, t + 1] linear code where u = n?d?1, d > m, we show that it is possible to construct (n, m, t, d) resilient S-boxes which have currently best known nonlinearity. Our second construction provides highly nonlinear (n, m, t, d) resilient S-boxes which do not have linear structure, then an improved version of this construction is given.     

Block-Coordinate Gradient Descent Method for?Linearly Constrained Nonsmooth Separable Optimization

We consider the problem of minimizing the weighted sum of a smooth function f and a convex function P of n real variables subject to m linear equality constraints. We propose a block-coordinate gradient descent method for solving this problem, with the coordinate block chosen by a Gauss-Southwell-q rule based on sufficient predicted descent. We establish global convergence to first-order stationarity for this method and, under a local error bound assumption, linear rate of convergence. If f is convex with Lipschitz continuous gradient, then the method terminates in O(n ²/ε) iterations with an ε-optimal solution. If P is separable, then the Gauss-Southwell-q rule is implementable in O(n) operations when m=1 and in O(n ²) operations when m>1. In the special case of support vector machines training, for which f is convex quadratic, P is separable, and m=1, this complexity bound is comparable to the best known bound for decomposition methods. If f is convex, then, by gradually reducing the weight on P to zero, the method can be adapted to solve the bilevel problem of minimizing P over the set of minima of f+δ _X , where X denotes the closure of the feasible set. This has application in the least 1-norm solution of maximum-likelihood estimation. This research was supported by the National Science Foundation, Grant No. DMS-0511283.     

Recursive self preconditioning method based on Schur complement for Toeplitz matrices

In this paper, we propose to solve the Toeplitz linear systems T _n x?=?b by a recursive-based method. The method is based on repeatedly dividing the original problem into two subproblems that involve the solution of systems containing the Schur complement of the leading principal submatrix of the previous level. The idea is to solve the linear systems S _m y?=?d, where S _m is the Schur complement of T _2m (the principal submatrix of T _n ), by using a self preconditioned iterative methods. The preconditioners, which are the approximate inverses of S _m , are constructed based on famous Gohberg–Semencul formula. All occurring matrices are represented by proper generating vectors of their displacement rank characterization. We show that, for well conditioned problems, the proposed method is efficient and robust. For ill-conditioned problems, by using some iterative refinement method, the new method would be efficient and robust. Numerical experiments are presented to show the effectiveness of our new method.     

A multi-objective tabu search for a single-machine scheduling problem with sequence-dependent setup times

An m-objective tabu search algorithm for sequencing of n jobs on a single machine with sequence-dependent setup times is proposed. The algorithm produces a solution set that is reflective of the objectives’ weights and close to the best observed values of the objectives. We also formulate a mixed integer linear program to obtain the optimal solution of a three-objective problem. Numerical examples are used to study the behavior of the proposed m-objective tabu search algorithm and compare its solutions with those of the mixed integer linear program.     

A fast Fourier-collocation method for second boundary integral equations

In this paper we develop a fast collocation method for second boundary integral equations by the trigonometric polynomials. We propose a convenient way to compress the dense matrix representation of a compact integral operator with a smooth kernel under the Fourier basis and the corresponding collocation functionals. The compression leads to a sparse matrix with only O(nlog²n) number of nonzero entries, where 2n+1 denotes the order of the matrix. Thus we develop a fast Fourier-collocation method. We prove that the fast Fourier-collocation method gives the optimal convergence order up to a logarithmic factor. Moreover, we design a fast scheme for solving the corresponding truncated linear system. We establish that this algorithm preserves the quasi-optimal convergence of the approximate solution with requiring a number of O(nlog³n) multiplications.     

Multiple point evaluation on combined tensor product supports

We consider the multiple point evaluation problem for an n-dimensional space of functions [???1,1[ ^d ?? spanned by d-variate basis functions that are the restrictions of simple (say linear) functions to tensor product domains. For arbitrary evaluation points this task is faced in the context of (semi-)Lagrangian schemes using adaptive sparse tensor approximation spaces for boundary value problems in moderately high dimensions. We devise a fast algorithm for performing m?≥?n point evaluations of a function in this space with computational cost O(mlog ^d n). We resort to nested segment tree data structures built in a preprocessing stage with an asymptotic effort of O(nlog ^d???1 n).     

A Legendre-Galerkin method for solving general Volterra functional integral equations

We propose in this paper a fully discrete Legendre-Galerkin method for solving general Volterra functional integral equations. The focus of this paper is the stability analysis of this method. Based on this stability result, we prove that the approximation equation has a unique solution, and then show that the Legendre-Galerkin method gives the optimal convergence order \(\mathcal {O}(n^{-m})\), where m denotes the degree of the regularity of the exact solution and n+1 denotes the dimensional number of the approximation space. Moreover, we establish that the spectral condition constant of the coefficient matrix relative to the corresponding linear system is uniformly bounded for sufficiently large n. Finally, we use numerical examples to confirm the theoretical prediction.