On the augmented system approach to sparse least-squares problems

Summary We study the augmented system approach for the solution of sparse linear least-squares problems. It is well known that this method has better numerical properties than the method based on the normal equations. We use recent work by Arioli et al. (1988) to introduce error bounds and estimates for the components of the solution of the augmented system. In particular, we find that, using iterative refinement, we obtain a very robust algorithm and our estimates of the error are accurate and cheap to compute. The final error and all our error estimates are much better than the classical or Skeel's error analysis (1979) indicates. Moreover, we prove that our error estimates are independent of the row scaling of the augmented system and we analyze the influence of the Björck scaling (1967) on these estimates. We illustrate this with runs both on large-scale practical problems and contrived examples, comparing the numerical behaviour of the augmented systems approach with a code using the normal equations. These experiments show that while the augmented system approach with iterative refinement can sometimes be less efficient than the normal equations approach, it is comparable or better when the least-squares matrix has a full row, and is, in any case, much more stable and robust.This author was visiting Harwell and was funded by a grant from the Italian National Council of Research (CNR), Istituto di Elaborazione dell'Informazione-CNR, via S. Maria 46, I-56100 Pisa, ItalyThis author was visiting Harwell from Faculty of Mathematics and Computer Science of the University of Amsterdam     

Null Space Algorithm and Spanning Trees in Solving Darcy's Equation

A Null Space algorithm is considered to solve the augmented system produced by the mixed finite element approximation of Darcy's Law. The method is based on the combination of a LU factorization technique for sparse matrices with an iterative Krylov solver. The computational efficiency of the method relies on the use of spanning trees to compute the LU factorization without fill-in and on a suitable stopping criterion for the iterative solver. We experimentally investigate its performance on a realistic set of selected application problems.This revised version was published online in October 2005 with corrections to the Cover Date.     

The augmented system variant of IPMs in two-stage stochastic linear programming computation

The application of interior point methods (IPM) to solve the deterministic equivalent of two-stage stochastic linear programming problems is a known and natural idea. Experiments have proved that among the interior point methods, the augmented system approach gives the best performance on these problems. However, most of their implementations encounter numerical difficulties in certain cases, which can result in loss of efficiency. We present a new approach for the decomposition of the augmented system, which ‘automatically’ exploits the special behavior of the problems. We show that the suggested approach can be implemented in a fast and numerically robust way by solving a number of large-scale two-stage stochastic linear programming problems. The comparison of our solver with fo1aug, which is considered as a state-of-the-art augmented system implementation of interior point methods, is also given.     

Preconditioning and iterative solution of symmetric indefinite linear systems arising from interior point methods for linear programming

We propose to compute the search direction at each interior-point iteration for a linear program via a reduced augmented system that typically has a much smaller dimension than the original augmented system. This reduced system is potentially less susceptible to the ill-conditioning effect of the elements in the (1,1) block of the augmented matrix. A preconditioner is then designed by approximating the block structure of the inverse of the transformed matrix to further improve the spectral properties of the transformed system. The resulting preconditioned system is likely to become better conditioned toward the end of the interior-point algorithm. Capitalizing on the special spectral properties of the transformed matrix, we further proposed a two-phase iterative algorithm that starts by solving the normal equations with PCG in each IPM iteration, and then switches to solve the preconditioned reduced augmented system with symmetric quasi-minimal residual (SQMR) method when it is advantageous to do so. The experimental results have demonstrated that our proposed method is competitive with direct methods in solving large-scale LP problems and a set of highly degenerate LP problems. Research supported in parts by NUS Research Grant R146-000-076-112 and SMA IUP Research Grant.     

A special Hermitian and skew-Hermitian splitting method for image restoration

In this paper, we consider an ill-posed image restoration problem with a noise contaminated observation, and a known convolution kernel. A special Hermitian and skew-Hermitian splitting (HSS) iterative method is established for solving the linear systems from image restoration. Our approach is based on an augmented system formulation. The convergence and operation cost of the special HSS iterative method for image restoration problems are discussed. The optimal parameter minimizing the spectral radius of the iteration matrix is derived. We present a detailed algorithm for image restoration problems. Numerical examples are given to demonstrate the performance of the presented method. Finally, the SOR acceleration scheme for the special HSS iterative method is discussed.     

On Two Iterative Least-Squares Finite Element Schemes for the Incompressible Navier–Stokes Problem

This paper is mainly devoted to a comparative study of two iterative least-squares finite element schemes for solving the stationary incompressible Navier–Stokes equations with velocity boundary condition. Introducing vorticity as an additional unknown variable, we recast the Navier–Stokes problem into a first-order quasilinear velocity–vorticity–pressure system. Two Picard-type iterative least-squares finite element schemes are proposed to approximate the solution to the nonlinear first-order problem. In each iteration, we adopt the usual L ² least-squares scheme or a weighted L ² least-squares scheme to solve the corresponding Oseen problem and provide error estimates. We concentrate on two-dimensional model problems using continuous piecewise polynomial finite elements on uniform meshes for both iterative least-squares schemes. Numerical evidences show that the iterative L ² least-squares scheme is somewhat suitable for low Reynolds number flow problems, whereas for flows with relatively higher Reynolds numbers the iterative weighted L ² least-squares scheme seems to be better than the iterative L ² least-squares scheme. Numerical simulations of the two-dimensional driven cavity flow are presented to demonstrate the effectiveness of the iterative least-squares finite element approach.     

Computational approach for composite materials with coupled constitutive laws

A numerical scheme based on fast Fourier transform is presented to compute the effective response and the local fields within a heterogeneous material which exhibits a coupled constitutive law. It consists in the iterative resolution of periodic coupled Lippmann–Schwinger equations. This approach is illustrated in the case of electroelastic composite materials. By using an augmented Lagrangian formulation, a simple iterative scheme relying on the uncoupled Green operators for the elastic and electrostatics problems is proposed. This computational framework, which allows to consider composite materials with an infinite contrast on the local properties, is assessed in the case of porous and fiber-reinforced piezoelectric materials.     

A Cheaper Way to Compute Generalized Cross-Validation as a Stopping Rule for Linear Stationary Iterative Methods

We apply generalized cross-validation (GCV) as a stopping rule for general linear stationary iterative methods for solving very large-scale, ill-conditioned problems. We present a new general formula for the influence operator for these methods and, using this formula and a Monte Carlo approach, we show how to compute the GCV function at a cheaper cost. Then we apply our approach to a well known iterative method (ART) with simulated data in positron emission tomography (PET).     

改进的预处理共轭斜量法及其在工程有限元分析中的应用

本文就预处理共轭斜量法(PCCG法)给出了两个具有理论和实际意义的定理,它们分别讨论了迭代解的定性性质和迭代矩阵的构造原则.作者提出了新的非M-矩阵的不完全LU分解技术和迭代矩阵的构造方法.用此改进的PCCG法,对病态问题和大型三维有限元问题进行了计算并与其他方法作了对比,分析了PCCG法在求解病态方程组时的反常现象.计算结果表明本文建议的方法是求解大型有限元方程组和病态方程组的一种十分有效的方法.     

Augmented Lagrangian Algorithms Based on the Spectral Projected Gradient Method for Solving Nonlinear Programming Problems

The spectral projected gradient method SPG is an algorithm for large-scale bound-constrained optimization introduced recently by Birgin, Martínez, and Raydan. It is based on the Raydan unconstrained generalization of the Barzilai-Borwein method for quadratics. The SPG algorithm turned out to be surprisingly effective for solving many large-scale minimization problems with box constraints. Therefore, it is natural to test its perfomance for solving the sub-problems that appear in nonlinear programming methods based on augmented Lagrangians. In this work, augmented Lagrangian methods which use SPG as the underlying convex-constraint solver are introduced (ALSPG) and the methods are tested in two sets of problems. First, a meaningful subset of large-scale nonlinearly constrained problems of the CUTE collection is solved and compared with the perfomance of LANCELOT. Second, a family of location problems in the minimax formulation is solved against the package FFSQP.     

Computational approach for composite materials with coupled constitutive laws

A numerical scheme based on fast Fourier transform is presented to compute the effective response and the local fields within a heterogeneous material which exhibits a coupled constitutive law. It consists in the iterative resolution of periodic coupled Lippmann–Schwinger equations. This approach is illustrated in the case of electroelastic composite materials. By using an augmented Lagrangian formulation, a simple iterative scheme relying on the uncoupled Green operators for the elastic and electrostatics problems is proposed. This computational framework, which allows to consider composite materials with an infinite contrast on the local properties, is assessed in the case of porous and fiber-reinforced piezoelectric materials.     

The solution of large-scale least-squares problems on supercomputers

We discuss methods for solving medium to large-scale sparse least-squares problems on supercomputers, illustrating our remarks by experiments on the CRAY-2 supercomputer at Harwell. The method we are primarily concerned with is an augmented system approach which has the merit of both robustness and accuracy, in addition to a kernel operation that is just the solution of a symmetric indefinite system. We consider extensions to handle weighted and constrained problems, and include experiments on systems similar to those arising in the Karmarkar algorithm for linear programming. We indicate how recent improvements to the kernel software could greatly improve the performance of the least-squares code.This paper is based on an invited talk by the author at a Workshop on Supercomputers and Large-Scale Optimization held at the Minnesota Supercomputing Center on 16th to 18th May, 1988.     

Integrating memetic search into the BioHEL evolutionary learning system for large-scale datasets

Local search methods are widely used to improve the performance of evolutionary computation algorithms in all kinds of domains. Employing advanced and efficient exploration mechanisms becomes crucial in complex and very large (in terms of search space) problems, such as when employing evolutionary algorithms to large-scale data mining tasks. Recently, the GAssist Pittsburgh evolutionary learning system was extended with memetic operators for discrete representations that use information from the supervised learning process to heuristically edit classification rules and rule sets. In this paper we first adapt some of these operators to BioHEL, a different evolutionary learning system applying the iterative learning approach, and afterwards propose versions of these operators designed for continuous attributes and for dealing with noise. The performance of all these operators and their combination is extensively evaluated on a broad range of synthetic large-scale datasets to identify the settings that present the best balance between efficiency and accuracy. Finally, the identified best configurations are compared with other classes of machine learning methods on both synthetic and real-world large-scale datasets and show very competent performance.     

QP-Free,Truncated Hybrid Methods for Large-Scale Nonlinear Constrained Optimization

1.IntroductionInthispaperweconsiderthefollowingnonlinearprogrammingproblemminimizef(x)subjecttogj(x)2o,jEJ={1,...,m}.(1'1)Extensionstoproblemincludingalsoequalityconstraintswillbepossible.Thefunctionf:W-Rlandgj:Rn-R',jEJaretwicecontinuouslydifferentiable.Inpaxticular,weapplyQP-free(withoutquadraticprogrammingsubproblems),truncatedhybridmethodsforsolvingthelarge-scaJenonlinearprogrammingproblems,inwhichthenumberofvariablesandthenumberofconstraiotsin(1.1)aregreat.Wediscussthecase,wheresecon…     

Using a hybrid preconditioner for solving large-scale linear systems arising from interior point methods

We devise a hybrid approach for solving linear systems arising from interior point methods applied to linear programming problems. These systems are solved by preconditioned conjugate gradient method that works in two phases. During phase I it uses a kind of incomplete Cholesky preconditioner such that fill-in can be controlled in terms of available memory. As the optimal solution of the problem is approached, the linear systems becomes highly ill-conditioned and the method changes to phase II. In this phase a preconditioner based on the LU factorization is found to work better near a solution of the LP problem. The numerical experiments reveal that the iterative hybrid approach works better than Cholesky factorization on some classes of large-scale problems.     

Outer Trust-Region Method for Constrained Optimization

Given an algorithm A for solving some mathematical problem based on the iterative solution of simpler subproblems, an outer trust-region (OTR) modification of A is the result of adding a trust-region constraint to each subproblem. The trust-region size is adaptively updated according to the behavior of crucial variables. The new subproblems should not be more complex than the original ones, and the convergence properties of the OTR algorithm should be the same as those of Algorithm A. In the present work, the OTR approach is exploited in connection with the “greediness phenomenon” of nonlinear programming. Convergence results for an OTR version of an augmented Lagrangian method for nonconvex constrained optimization are proved, and numerical experiments are presented.     

A stable primal–dual approach for linear programming under nondegeneracy assumptions

This paper studies a primal–dual interior/exterior-point path-following approach for linear programming that is motivated on using an iterative solver rather than a direct solver for the search direction. We begin with the usual perturbed primal–dual optimality equations. Under nondegeneracy assumptions, this nonlinear system is well-posed, i.e. it has a nonsingular Jacobian at optimality and is not necessarily ill-conditioned as the iterates approach optimality. Assuming that a basis matrix (easily factorizable and well-conditioned) can be found, we apply a simple preprocessing step to eliminate both the primal and dual feasibility equations. This results in a single bilinear equation that maintains the well-posedness property. Sparsity is maintained. We then apply either a direct solution method or an iterative solver (within an inexact Newton framework) to solve this equation. Since the linearization is well posed, we use affine scaling and do not maintain nonnegativity once we are close enough to the optimum, i.e. we apply a change to a pure Newton step technique. In addition, we correctly identify some of the primal and dual variables that converge to 0 and delete them (purify step). We test our method with random nondegenerate problems and problems from the Netlib set, and we compare it with the standard Normal Equations NEQ approach. We use a heuristic to find the basis matrix. We show that our method is efficient for large, well-conditioned problems. It is slower than NEQ on ill-conditioned problems, but it yields higher accuracy solutions.     

Sparse approximate inverse preconditioning for dense linear systems arising in computational electromagnetics

We investigate the use of sparse approximate inverse preconditioners for the iterative solution of linear systems with dense complex coefficient matrices arising in industrial electromagnetic problems. An approximate inverse is computed via a Frobenius norm approach with a prescribed nonzero pattern. Some strategies for determining the nonzero pattern of an approximate inverse are described. The results of numerical experiments suggest that sparse approximate inverse preconditioning is a viable approach for the solution of large-scale dense linear systems on parallel computers. This revised version was published online in August 2006 with corrections to the Cover Date.     

Large-scale Tikhonov regularization via reduction by orthogonal projection

This paper presents a new approach to computing an approximate solution of Tikhonov-regularized large-scale ill-posed least-squares problems with a general regularization matrix. The iterative method applies a sequence of projections onto generalized Krylov subspaces. A suitable value of the regularization parameter is determined by the discrepancy principle.     

Solvability for p-Laplacian generalized fractional coupled systems with two-sided memory effects

This paper determines the solvability of multipoint boundary value problems for p-Laplacian generalized fractional differential systems with Riesz–Caputo derivative, which exhibits two-sided nonlocal memory effects. An equivalent integral form for the generalized fractional differential system is deduced by transformation. First, we obtain the existence of solutions on the basis of the upper–lower solutions method, in which an explicit iterative approach for approximating the solution is established. Second, we deal with a special case of our fractional differential system; in order to obtain novel results, an abstract sum-type operator equation A(x,x)+Bx+e=x on ordered Banach space is discussed. Without requiring the existence of upper–lower solutions or compactness conditions, we get several unique results of solutions for this operator equation, which provide new inspiration for the study of boundary value problems. Then, we apply these abstract results to get the uniqueness of solutions for our differential system.