Bounds for the least squares distance using scaled total least squares

Summary. The standard approaches to solving overdetermined linear systems construct minimal corrections to the data to make the corrected system compatible. In ordinary least squares (LS) the correction is restricted to the right hand side c, while in scaled total least squares (STLS) [14,12] corrections to both c and B are allowed, and their relative sizes are determined by a real positive parameter . As , the STLS solution approaches the LS solution. Our paper [12] analyzed fundamentals of the STLS problem. This paper presents a theoretical analysis of the relationship between the sizes of the LS and STLS corrections (called the LS and STLS distances) in terms of . We give new upper and lower bounds on the LS distance in terms of the STLS distance, compare these to existing bounds, and examine the tightness of the new bounds. This work can be applied to the analysis of iterative methods which minimize the residual norm, and the generalized minimum residual method (GMRES) [15] is used here to illustrate our theory. Received July 20, 2000 / Revised version received February 28, 2001 / Published online July 25, 2001     

An algorithm for linear least squares problems with equality and nonnegativity constraints

We present a new algorithm for solving a linear least squares problem with linear constraints. These are equality constraint equations and nonnegativity constraints on selected variables. This problem, while appearing to be quite special, is the core problem arising in the solution of the general linearly constrained linear least squares problem. The reduction process of the general problem to the core problem can be done in many ways. We discuss three such techniques.The method employed for solving the core problem is based on combining the equality constraints with differentially weighted least squares equations to form an augmented least squares system. This weighted least squares system, which is equivalent to a penalty function method, is solved with nonnegativity constraints on selected variables.Three types of examples are presented that illustrate applications of the algorithm. The first is rank deficient, constrained least squares curve fitting. The second is concerned with solving linear systems of algebraic equations with Hilbert matrices and bounds on the variables. The third illustrates a constrained curve fitting problem with inconsistent inequality constraints.     

A stationary iterative pseudoinverse algorithm

Iterative methods applied to the normal equationsA ^T Ax=A ^T b are sometimes used for solving large sparse linear least squares problems. However, when the matrix is rank-deficient many methods, although convergent, fail to produce the unique solution of minimal Euclidean norm. Examples of such methods are the Jacobi and SOR methods as well as the preconditioned conjugate gradient algorithm. We analyze here an iterative scheme that overcomes this difficulty for the case of stationary iterative methods. The scheme combines two stationary iterative methods. The first method produces any least squares solution whereas the second produces the minimum norm solution to a consistent system. This work was supported by the Swedish Research Council for Engineering Sciences, TFR.     

Semiconvergence analysis of the randomized row iterative method and its extended variants

The row iterative method is popular in solving the large‐scale ill‐posed problems due to its simplicity and efficiency. In this work we consider the randomized row iterative (RRI) method to tackle this issue. First, we present the semiconvergence analysis of RRI method for the overdetermined and inconsistent system, and derive upper bounds for the noise error propagation in the iteration vectors. To achieve a least squares solution, we then propose an extended version of the RRI (ERRI) method, which in fact can converge in expectation to the solution of the overdetermined or underdetermined, consistent or inconsistent systems. Finally, some numerical examples are given to demonstrate the convergence behaviors of the RRI and ERRI methods for these types of linear system.     

AOR iterative methods for rank deficient least squares problems

In this paper we study the semiconvergence of accelerated overrelaxation (AOR) iterative methods for the least squares solution of minimal norm of rank deficient linear systems. Necessary and sufficient conditions for the semiconvergence of the AOR and JOR iterative methods are given. The optimum parameters and the associated convergence factor are derived.     

不定最小二乘问题的改进的不完全双曲Gram-Schmidt预处理算法

应用改进的不完全双曲Gram-Schmidt(IHMGS)方法预处理不定最小二乘问题的共轭梯度法(CGILS)、正交分解法(ILSQR)与广义的最小剩余法(GMRES)等迭代算法来求解大型稀疏的不定最小二乘问题.数值实验表明,IHMGS预处理方法可有效提高相应算法的迭代速度,且当矩阵的条件数比较大时,效果更加显著.     

Iterative refinement enhances the stability ofQR factorization methods for solving linear equations

Iterative refinement is a well-known technique for improving the quality of an approximate solution to a linear system. In the traditional usage residuals are computed in extended precision, but more recent work has shown that fixed precision is sufficient to yield benefits for stability. We extend existing results to show that fixed precision iterative refinement renders anarbitrary linear equations solver backward stable in a strong, componentwise sense, under suitable assumptions. Two particular applications involving theQR factorization are discussed in detail: solution of square linear systems and solution of least squares problems. In the former case we show that one step of iterative refinement suffices to produce a small componentwise relative backward error. Our results are weaker for the least squares problem, but again we find that iterative refinement improves a componentwise measure of backward stability. In particular, iterative refinement mitigates the effect of poor row scaling of the coefficient matrix, and so provides an alternative to the use of row interchanges in the HouseholderQR factorization. A further application of the results is described to fast methods for solving Vandermonde-like systems.     

Minimal residual methods for large scale Lyapunov equations

Projection methods have emerged as competitive techniques for solving large scale matrix Lyapunov equations. We explore the numerical solution of this class of linear matrix equations when a Minimal Residual (MR) condition is used during the projection step. We derive both a new direct method, and a preconditioned operator-oriented iterative solver based on CGLS, for solving the projected reduced least squares problem. Numerical experiments with benchmark problems show the effectiveness of an MR approach over a Galerkin procedure using the same approximation space.     

Minimization of functionals on the solution of a large-scale discrete ill-posed problem

In this work we study the minimization of a linear functional defined on a set of approximate solutions of a discrete ill-posed problem. The primary application of interest is the computation of confidence intervals for components of the solution of such a problem. We exploit the technique introduced by Eldén in 1990, utilizing a parametric programming reformulation involving the solution of a sequence of quadratically constrained least squares problems. Our iterative method, which uses the connection between Lanczos bidiagonalization and Gauss-type quadrature rules to bound certain matrix functionals, is well-suited for large-scale problems, and offers a significant reduction in matrix-vector product evaluations relative to available methods.     

Error estimates for the regularization of least squares problems

The a posteriori estimate of the errors in the numerical solution of ill-conditioned linear systems with contaminated data is a complicated problem. Several estimates of the norm of the error have been recently introduced and analyzed, under the assumption that the matrix is square and nonsingular. In this paper we study the same problem in the case of a rectangular and, in general, rank-deficient matrix. As a result, a class of error estimates previously introduced by the authors (Brezinski et al., Numer Algorithms, in press, 2008) are extended to the least squares solution of consistent and inconsistent linear systems. Their application to various direct and iterative regularization methods are also discussed, and the numerical effectiveness of these error estimates is pointed out by the results of an extensive experimentation.     

A condition analysis of the weighted linear least squares problem using dual norms

In this paper, based on the theory of adjoint operators and dual norms, we define condition numbers for a linear solution function of the weighted linear least squares problem. The explicit expressions of the normwise and componentwise condition numbers derived in this paper can be computed at low cost when the dimension of the linear function is low due to dual operator theory. Moreover, we use the augmented system to perform a componentwise perturbation analysis of the solution and residual of the weighted linear least squares problems. We also propose two efficient condition number estimators. Our numerical experiments demonstrate that our condition numbers give accurate perturbation bounds and can reveal the conditioning of individual components of the solution. Our condition number estimators are accurate as well as efficient.     

矩阵方程AXA^T＋BYB^T=C的双对称最小二乘解及其最佳逼近

基于共轭梯度法的思想,通过特殊的变形,建立了一类求矩阵方程AXA^T＋BYB^T=C的双对称最小二乘解的迭代算法．对任意的初始双对称矩阵．在没有舍人误差的情况下,经过有限步迭代得到它的双对称最小二乘解;在选取特殊的初始双对称矩阵时,能得到它的的极小范数双对称最小二乘解．另外,给定任意矩阵,利用此方法可得到它的最佳逼近双对称解,数值例子表明,这种方法是有效的．     

Convex programming with single separable constraint and bounded variables

In this paper a minimization problem with convex objective function subject to a separable convex inequality constraint “≤” and bounded variables (box constraints) is considered. We propose an iterative algorithm for solving this problem based on line search and convergence of this algorithm is proved. At each iteration, a separable convex programming problem with the same constraint set is solved using Karush-Kuhn-Tucker conditions. Convex minimization problems subject to linear equality/ linear inequality “≥” constraint and bounds on the variables are also considered. Numerical illustration is included in support of theory.     

The Equality Constrained Indefinite Least Squares Problem: Theory and Algorithms

We present theory and algorithms for the equality constrained indefinite least squares problem, which requires minimization of an indefinite quadratic form subject to a linear equality constraint. A generalized hyperbolic QR factorization is introduced and used in the derivation of perturbation bounds and to construct a numerical method. An alternative method is obtained by employing a generalized QR factorization in combination with a Cholesky factorization. Rounding error analysis is given to show that both methods have satisfactory numerical stability properties and numerical experiments are given for illustration. This work builds on recent work on the unconstrained indefinite least squares problem by Chandrasekaran, Gu, and Sayed and by the present authors.     

Krylov subspace methods with deflation and balancing preconditioners for least squares problems

For solving least squares problems, the CGLS method is a typical method in the point of view of iterative methods. When the least squares problems are ill-conditioned, the convergence behavior of the CGLS method will present a deteriorated result. We expect to select other iterative Krylov subspace methods to overcome the disadvantage of CGLS. Here the GMRES method is a suitable algorithm for the reason that it is derived from the minimal residual norm approach, which coincides with least squares problems. Ken Hayami proposed BAGMRES for solving least squares problems in [\emph{GMRES Methods for Least Squares Problems, SIAM J. Matrix Anal. Appl., 31(2010)}, pp.2400-2430]. The deflation and balancing preconditioners can optimize the convergence rate through modulating spectral distribution. Hence, in this paper we utilize preconditioned iterative Krylov subspace methods with deflation and balancing preconditioners in order to solve ill-conditioned least squares problems. Numerical experiments show that the methods proposed in this paper are better than the CGLS method.     

Weighted least squares solutions to general coupled Sylvester matrix equations

This paper is concerned with weighted least squares solutions to general coupled Sylvester matrix equations. Gradient based iterative algorithms are proposed to solve this problem. This type of iterative algorithm includes a wide class of iterative algorithms, and two special cases of them are studied in detail in this paper. Necessary and sufficient conditions guaranteeing the convergence of the proposed algorithms are presented. Sufficient conditions that are easy to compute are also given. The optimal step sizes such that the convergence rates of the algorithms, which are properly defined in this paper, are maximized and established. Several special cases of the weighted least squares problem, such as a least squares solution to the coupled Sylvester matrix equations problem, solutions to the general coupled Sylvester matrix equations problem, and a weighted least squares solution to the linear matrix equation problem are simultaneously solved. Several numerical examples are given to illustrate the effectiveness of the proposed algorithms.     

Improved componentwise verified error bounds for least squares problems and underdetermined linear systems

Recently Miyajima presented algorithms to compute componentwise verified error bounds for the solution of full-rank least squares problems and underdetermined linear systems. In this paper we derive simpler and improved componentwise error bounds which are based on equalities for the error of a given approximate solution. Equalities are not improvable, and the expressions are formulated in a way that direct evaluation yields componentwise and rigorous estimates of good quality. The computed bounds are correct in a mathematical sense covering all sources of errors, in particular rounding errors. Numerical results show a gain in accuracy compared to previous results.     

Condition Number Bounds for Problems with Integer Coefficients

An explicit a priori bound for the condition number associated to each of the following problems is given: general linear equation solving, least squares, nonsymmetric eigenvalue problems, solving univariate polynomials, and solving systems of multivariate polynomials. It is assumed that the input has integer coefficients and is not on the degeneracy locus of the respective problem (i.e., the condition number is finite). Our bounds are stated in terms of the dimension and of the bit-size of the input.  In the same setting, bounds are given for the speed of convergence of the following iterative algorithms: QR iteration without shift for the symmetric eigenvalue problem and Graeffe iteration for univariate polynomials.     

High-dimensional variable screening and bias in subsequent inference,with an empirical comparison

We review variable selection and variable screening in high-dimensional linear models. Thereby, a major focus is an empirical comparison of various estimation methods with respect to true and false positive selection rates based on 128 different sparse scenarios from semi-real data (real data covariables but synthetic regression coefficients and noise). Furthermore, we present some theoretical bounds for the bias in subsequent least squares estimation, using the selected variables from the first stage, which have direct implications for construction of p-values for regression coefficients.