Greedy expansions in convex optimization

This paper is a follow-up to the author’s previous paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there the three most popular greedy algorithms in nonlinear approximation in Banach spaces-Weak Chebyshev Greedy Algorithm, Weak Greedy Algorithm with Free Relaxation, and Weak Relaxed Greedy Algorithm-for solving convex optimization problems. We continue to study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. In this paper we concentrate on greedy algorithms that provide expansions, which means that the approximant at the mth iteration is equal to the sum of the approximant from the previous, (m ? 1)th, iteration and one element from the dictionary with an appropriate coefficient. The problem of greedy expansions of elements of a Banach space is well studied in nonlinear approximation theory. At first glance the setting of a problem of expansion of a given element and the setting of the problem of expansion in an optimization problem are very different. However, it turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy expansions technique, can be adjusted for finding a sparse solution of an optimization problem given by an expansion with respect to a given dictionary.     

Truncated Trust Region Methods Based on Preconditioned Iterative Subalgorithms for Large Sparse Systems of Nonlinear Equations

This paper is devoted to globally convergent methods for solving large sparse systems of nonlinear equations with an inexact approximation of the Jacobian matrix. These methods include difference versions of the Newton method and various quasi-Newton methods. We propose a class of trust region methods together with a proof of their global convergence and describe an implementable globally convergent algorithm which can be used as a realization of these methods. Considerable attention is concentrated on the application of conjugate gradient-type iterative methods to the solution of linear subproblems. We prove that both the GMRES and the smoothed COS well-preconditioned methods can be used for the construction of globally convergent trust region methods. The efficiency of our algorithm is demonstrated computationally by using a large collection of sparse test problems.     

Performance comparisons of greedy algorithms in compressed sensing

Compressed sensing has motivated the development of numerous sparse approximation algorithms designed to return a solution to an underdetermined system of linear equations where the solution has the fewest number of nonzeros possible, referred to as the sparsest solution. In the compressed sensing setting, greedy sparse approximation algorithms have been observed to be both able to recover the sparsest solution for similar problem sizes as other algorithms and to be computationally efficient; however, little theory is known for their average case behavior. We conduct a large‐scale empirical investigation into the behavior of three of the state of the art greedy algorithms: Normalized Iterative Hard Thresholding (NIHT), Hard Thresholding Pursuit (HTP), and CSMPSP. The investigation considers a variety of random classes of linear systems. The regions of the problem size in which each algorithm is able to reliably recover the sparsest solution is accurately determined, and throughout this region, additional performance characteristics are presented. Contrasting the recovery regions and the average computational time for each algorithm, we present algorithm selection maps, which indicate, for each problem size, which algorithm is able to reliably recover the sparsest vector in the least amount of time. Although no algorithm is observed to be uniformly superior, NIHT is observed to have an advantageous balance of large recovery region, absolute recovery time, and robustness of these properties to additive noise across a variety of problem classes. A principle difference between the NIHT and the more sophisticated HTP and CSMPSP is the balance of asymptotic convergence rate against computational cost prior to potential support set updates. The data suggest that NIHT is typically faster than HTP and CSMPSP because of greater flexibility in updating the support that limits unnecessary computation on incorrect support sets. The algorithm selection maps presented here are the first of their kind for compressed sensing. Copyright © 2014 John Wiley & Sons, Ltd.     

Tensor methods for large sparse systems of nonlinear equations

This paper introduces tensor methods for solving large sparse systems of nonlinear equations. Tensor methods for nonlinear equations were developed in the context of solving small to medium-sized dense problems. They base each iteration on a quadratic model of the nonlinear equations, where the second-order term is selected so that the model requires no more derivative or function information per iteration than standard linear model-based methods, and hardly more storage or arithmetic operations per iteration. Computational experiments on small to medium-sized problems have shown tensor methods to be considerably more efficient than standard Newton-based methods, with a particularly large advantage on singular problems. This paper considers the extension of this approach to solve large sparse problems. The key issue considered is how to make efficient use of sparsity in forming and solving the tensor model problem at each iteration. Accomplishing this turns out to require an entirely new way of solving the tensor model that successfully exploits the sparsity of the Jacobian, whether the Jacobian is nonsingular or singular. We develop such an approach and, based upon it, an efficient tensor method for solving large sparse systems of nonlinear equations. Test results indicate that this tensor method is significantly more efficient and robust than an efficient sparse Newton-based method, in terms of iterations, function evaluations, and execution time. © 1998 The Mathematical Programming Society, Inc. Published by Elsevier Science B.V.Work supported by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Computational and Technology Research, US Department of Energy, under Contract W-31-109-Eng-38, by the National Aerospace Agency under Purchase Order L25935D, and by the National Science Foundation, through the Center for Research on Parallel Computation, under Cooperative Agreement No. CCR-9120008.Research supported by AFOSR Grants No. AFOSR-90-0109 and F49620-94-1-0101, ARO Grants No. DAAL03-91-G-0151 and DAAH04-94-G-0228, and NSF Grant No. CCR-9101795.     

Analytical solution of a near-equidiffusional flame problem

The near-equidiffusional flame problem for determining the effectof acoustic disturbances having a time scale comparable withthe diffusion time is reduced to a set of nonlinear Volterraequations for a pair of functions related to the burning rateand the flame temperature perturbation. Further analysis yieldsa small-time approximation of general validity and an explicitsolution for arbitrary small-amplitude disturbances. A numericalmethod for solving the Volterra-equation problem is described.Numerical examples are exhibited to compare the behaviour ofthe linear approximation to the solution of the full model.Slowly decaying oscillatory behaviour for flames with Lewisnumber near the right stability boundary is predicted by thelinear approximation and observed in numerical examples forthe nonlinear model.     

A Rayleigh–Ritz preconditioner for the iterative solution to large scale nonlinear problems

The approximation to the solution of large sparse symmetric linear problems arising from nonlinear systems of equations is considered. We are focusing herein on reusing information from previous processes while solving a succession of linear problems with a Conjugate Gradient algorithm. We present a new Rayleigh–Ritz preconditioner that is based on the Krylov subspaces and superconvergence properties, and consists of a suitable reuse of a given set of Ritz vectors. The relevance and the mathematical foundations of the current approach are detailed and the construction of the preconditioner is presented either for the unconstrained or the constrained problems. A corresponding practical preconditioner for iterative domain decomposition methods applied to nonlinear elasticity is addressed, and numerical validation is performed on a poorly-conditioned large-scale practical problem. This revised version was published online in June 2006 with corrections to the Cover Date.     

Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equations

We address finding the semi-global solutions to optimal feedback control and the Hamilton–Jacobi–Bellman (HJB) equation. Using the solution of an HJB equation, a feedback optimal control law can be implemented in real-time with minimum computational load. However, except for systems with two or three state variables, using traditional techniques for numerically finding a semi-global solution to an HJB equation for general nonlinear systems is infeasible due to the curse of dimensionality. Here we present a new computational method for finding feedback optimal control and solving HJB equations which is able to mitigate the curse of dimensionality. We do not discretize the HJB equation directly, instead we introduce a sparse grid in the state space and use the Pontryagin’s maximum principle to derive a set of necessary conditions in the form of a boundary value problem, also known as the characteristic equations, for each grid point. Using this approach, the method is spatially causality free, which enjoys the advantage of perfect parallelism on a sparse grid. Compared with dense grids, a sparse grid has a significantly reduced size which is feasible for systems with relatively high dimensions, such as the 6-D system shown in the examples. Once the solution obtained at each grid point, high-order accurate polynomial interpolation is used to approximate the feedback control at arbitrary points. We prove an upper bound for the approximation error and approximate it numerically. This sparse grid characteristics method is demonstrated with three examples of rigid body attitude control using momentum wheels.     

Approximate factoring of the inverse

Computation of approximate factors for the inverse constitutes an algebraic approach to preconditioning large and sparse linear systems. In this paper, the aim is to combine standard preconditioning ideas with sparse approximate inverse approximation, to have dense approximate inverse approximations (implicitly). For optimality, the approximate factoring problem is associated with a minimization problem involving two matrix subspaces. This task can be converted into an eigenvalue problem for a Hermitian positive semidefinite operator whose smallest eigenpairs are of interest. Because of storage and complexity constraints, the power method appears to be the only admissible algorithm for devising sparse–sparse iterations. The subtle issue of choosing the matrix subspaces is addressed. Numerical experiments are presented.     

Conjugate Gradient Method for Rank Deficient Saddle Point Problems

We propose an alternative iterative method to solve rank deficient problems arising in many real applications such as the finite element approximation to the Stokes equation and computational genetics. Our main contribution is to transform the rank deficient problem into a smaller full rank problem, with structure as sparse as possible. The new system improves the condition number greatly. Numerical experiments suggest that the new iterative method works very well for large sparse rank deficient saddle point problems.     

Approximation for Navier-Stokes equations around a rotating obstacle

This work presents an approximation method for Navier-Stokes equations around a rotating obstacle. The detail of this method is that the exterior domain is truncated into a bounded domain and a new exterior domain by introducing a large ball. The approximation problem is composed of the nonlinear problem in the bounded domain and the linear problem in the new exterior domain. We derive the approximation error between the solutions of Navier-Stokes equations and the approximation problem.     

An approach to the Gummel map by vector extrapolation methods

The numerical approximation of nonlinear partial differential equations requires the computation of large nonlinear systems, that are typically solved by iterative schemes. At each step of the iterative process, a large and sparse linear system has to be solved, and the amount of time elapsed per step grows with the dimensions of the problem. As a consequence, the convergence rate may become very slow, requiring massive cpu-time to compute the solution. In all such cases, it is important to improve the rate of convergence of the iterative scheme. This can be achieved, for instance, by vector extrapolation methods. In this work, we apply some vector extrapolation methods to the electronic device simulation to improve the rate of convergence of the family of Gummel decoupling algorithms. Furthermore, a different approach to the topological ε-algorithm is proposed and preliminary results are presented.     

Application of the largest Lyapunov exponent and non-linear fractal extrapolation algorithm to short-term load forecasting

Precise short-term load forecasting (STLF) plays a key role in unit commitment, maintenance and economic dispatch problems. Employing a subjective and arbitrary predictive step size is one of the most important factors causing the low forecasting accuracy. To solve this problem, the largest Lyapunov exponent is adopted to estimate the maximal predictive step size so that the step size in the forecasting is no more than this maximal one. In addition, in this paper a seldom used forecasting model, which is based on the non-linear fractal extrapolation (NLFE) algorithm, is considered to develop the accuracy of predictions. The suitability and superiority of the two solutions are illustrated through an application to real load forecasting using New South Wales electricity load data from the Australian National Electricity Market. Meanwhile, three forecasting models: the gray model, the seasonal autoregressive integrated moving average approach and the support vector machine method, which received high approval in STLF, are selected to compare with the NLFE algorithm. Comparison results also show that the NLFE model is outstanding, effective, practical and feasible.     

A piecewise linearization framework for retail shelf space management models

Managing shelf space is critical for retailers to attract customers and optimize profits. This article develops a shelf-space allocation optimization model that explicitly incorporates essential in-store costs and considers space- and cross-elasticities. A piecewise linearization technique is used to approximate the complicated nonlinear space-allocation model. The approximation reformulates the non-convex optimization problem into a linear mixed integer programming (MIP) problem. The MIP solution not only generates near-optimal solutions for large scale optimization problems, but also provides an error bound to evaluate the solution quality. Consequently, the proposed approach can solve single category-shelf space management problems with as many products as are typically encountered in practice and with more complicated cost and profit structures than currently possible by existing methods. Numerical experiments show the competitive accuracy of the proposed method compared with the mixed integer nonlinear programming shelf-space model. Several extensions of the main model are discussed to illustrate the flexibility of the proposed methodology.     

Static computation of a laminated nonuniform inclined shell in a temperature field

The finite difference method is used to obtain a solution of a nonlinear static problem for a laminated inclined rectangular shell in a plane acted on by a force load and a temperature field. The approximating system of nonlinear equations is obtained using an approximation of the equation of variations or systems of differential equations.Translated fromMatematicheskie Metody i Fiziko-Mekhanicheskie Polya, Issue 27, 1988, pp. 86–89.     

Corrected sequential linear programming for sparse minimax optimization

We present a new algorithm for nonlinear minimax optimization which is well suited for large and sparse problems. The method is based on trust regions and sequential linear programming. On each iteration a linear minimax problem is solved for a basic step. If necessary, this is followed by the determination of a minimum norm corrective step based on a first-order Taylor approximation. No Hessian information needs to be stored. Global convergence is proved. This new method has been extensively tested and compared with other methods, including two well known codes for nonlinear programming. The numerical tests indicate that in many cases the new method can find the solution in just as few iterations as methods based on approximate second-order information. The tests also show that for some problems the corrective steps give much faster convergence than for similar methods which do not employ such steps.Research supported partly by The Nordic Council of Ministers, The Icelandic Science Council, The University of Iceland Research Fund and The Danish Science Research Council.     

Deep Learning for Real-Time Crime Forecasting and Its Ternarization

Real-time crime forecasting is important. However, accurate prediction of when and where the next crime will happen is difficult. No known physical model provides a reasonable approximation to such a complex system. Historical crime data are sparse in both space and time and the signal of interests is weak. In this work, the authors first present a proper representation of crime data. The authors then adapt the spatial temporal residual network on the well represented data to predict the distribution of crime in Los Angeles at the scale of hours in neighborhood-sized parcels. These experiments as well as comparisons with several existing approaches to prediction demonstrate the superiority of the proposed model in terms of accuracy. Finally, the authors present a ternarization technique to address the resource consumption issue for its deployment in real world. This work is an extension of our short conference proceeding paper [Wang, B., Zhang, D., Zhang, D. H., et al., Deep learning for real time Crime forecasting, 2017, arXiv: 1707.03340].     

The Reduced Basis Method for an Elastic Buckling Problem

In this work, we apply the Reduced Basis (RB) Method to the field of nonlinear elasticity. In this first stage of research, we analyze a buckling problem for a compressed 2D column: Here, the trivial linear solution is computed for an arbitrary load; the critical load, marking the transition to nonlinearity, is then identified through an eigenvalue problem. The linear problem satisfies the Lax-Milgram conditions, allowing the implementation of both a Successive Constraint Method for an inexpensive lower bound of the coercivity constant and of a rigorous and efficient a posteriori error estimator for the RB approximation. Even though only a non-rigorous estimator is available for the buckling problem, the actual RB approximation of the output is more than satisfactory, and the gain in computational efficiency significant. (© 2013 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Electric load forecasting by support vector model

Accurately electric load forecasting has become the most important management goal, however, electric load often presents nonlinear data patterns. Therefore, a rigid forecasting approach with strong general nonlinear mapping capabilities is essential. Support vector regression (SVR) applies the structural risk minimization principle to minimize an upper bound of the generalization errors, rather than minimizing the training errors which are used by ANNs. The purpose of this paper is to present a SVR model with immune algorithm (IA) to forecast the electric loads, IA is applied to the parameter determine of SVR model. The empirical results indicate that the SVR model with IA (SVRIA) results in better forecasting performance than the other methods, namely SVMG, regression model, and ANN model.     

Computing Aviation Sparing Policies: Solving a Large Nonlinear Integer Program

Deployed US Navy aircraft carriers must stock a large number of spare parts to support the various types of aircraft embarked on the ship. The sparing policy determines the spares that will be stocked on the ship to keep the embarked aircraft ready to fly. Given a fleet of ten or more aircraft carriers and a cost of approximately 50 million dollars per carrier plus the cost of spares maintained in warehouses in the United States, the sparing problem constitutes a significant portion of the Navy’s resources. The objective of this work is to find a minimum-cost sparing policy that meets the readiness requirements of the embarked aircraft. This is a very large, nonlinear, integer optimization problem. The cost function is piecewise linear and convex while the constraint mapping is highly nonlinear. The distinguishing characteristics of this problem from an optimization viewpoint are that a large number of decision variables are required to be integer and that the nonlinear constraint functions are essentially “black box” functions; that is, they are very difficult (and expensive) to evaluate and their derivatives are not available. Moreover, they are not convex. Integer programming problems with a large number of variables are difficult to solve in general and most successful approaches to solving nonlinear integer problems have involved linear approximation and relaxation techniques that, because of the complexity of the constraint functions, are inappropriate for attacking this problem. We instead employ a pattern search method to each iteration of an interior point-type algorithm to solve the relaxed version of the problem. From the solution found by the pattern search on each interior point iteration, we begin another pattern search on the integer lattice to find a good integer solution. The best integer solution found across all interations is returned as the optimal solution. The pattern searches are distributed across a local area network of non-dedicated, heterogeneous computers in an office environment, thus, drastically reducing the time required to find the solution.