A NUMERICALLY STABLE BLOCK MODIFIED GRAM-SCHMIDT ALGORITHM FOR SOLVING STIFF WEIGHTED LEAST SQUARES PROBLEMS

Recently, Wei in proved that perturbed stiff weighted pseudoinverses and stiff weighted least squares problems are stable, if and only if the original and perturbed coefficient matrices A and A^- satisfy several row rank preservation conditions. According to these conditions, in this paper we show that in general, ordinary modified Gram-Schmidt with column pivoting is not numerically stable for solving the stiff weighted least squares problem. We then propose a row block modified Gram-Schmidt algorithm with column pivoting, and show that with appropriately chosen tolerance, this algorithm can correctly determine the numerical ranks of these row partitioned sub-matrices, and the computed QR factor R^- contains small roundoff error which is row stable. Several numerical experiments are also provided to compare the results of the ordinary Modified Gram-Schmidt algorithm with column pivoting and the row block Modified Gram-Schmidt algorithm with column pivoting.     

An algorithm for linear least squares problems with equality and nonnegativity constraints

We present a new algorithm for solving a linear least squares problem with linear constraints. These are equality constraint equations and nonnegativity constraints on selected variables. This problem, while appearing to be quite special, is the core problem arising in the solution of the general linearly constrained linear least squares problem. The reduction process of the general problem to the core problem can be done in many ways. We discuss three such techniques.The method employed for solving the core problem is based on combining the equality constraints with differentially weighted least squares equations to form an augmented least squares system. This weighted least squares system, which is equivalent to a penalty function method, is solved with nonnegativity constraints on selected variables.Three types of examples are presented that illustrate applications of the algorithm. The first is rank deficient, constrained least squares curve fitting. The second is concerned with solving linear systems of algebraic equations with Hilbert matrices and bounds on the variables. The third illustrates a constrained curve fitting problem with inconsistent inequality constraints.     

Stability analysis of the G-algorithm and a note on its application to sparse least squares problems

The G-algorithm was proposed by Bareiss [1] as a method for solving the weighted linear least squares problem. It is a square root free algorithm similar to the fast Givens method except that it triangularizes a rectangular matrix a column at a time instead of one element at a time.In this paper an error analysis of the G-algorithm is presented which shows that it is as stable as any of the standard orthogonal decomposition methods for solving least squares problems. The algorithm is shown to be a competitive method for sparse least squares problems.A pivoting strategy is given for heavily weighted problems similar to that in [14] for the Householder-Golub algorithm. The strategy is prohibitively expensive, but it is not necessary for most of the least squares problems that arise in practice.The research was supported by the National Science Foundation under contract no. MCS-8201065 and by the Office of Naval Research under contract no. N0014-80-0517.     

等式约束不定最小二乘问题的双曲MGS消去算法

众所周知,加权法是解等式约束不定最小二乘问题的方法之一.通过探讨极限意义下,双曲MGS算法解对应加权问题的本质,得到一类消去算法.实验表明,该算法以和文献中现有的GHQR算法达到一样的精度,但实际计算量只需要GHQR算法的一半.     

A weighted pseudoinverse,generalized singular values,and constrained least squares problems

The weighted pseudoinverse providing the minimum semi-norm solution of the weighted linear least squares problem is studied. It is shown that it has properties analogous to those of the Moore-Penrose pseudoinverse. The relation between the weighted pseudoinverse and generalized singular values is explained. The weighted pseudoinverse theory is used to analyse least squares problems with linear and quadratic constraints. A numerical algorithm for the computation of the weighted pseudoinverse is briefly described.This work was supported in part by the Swedish Institute for Applied Mathematics.     

A factorization method for the solution of constrained linear least squares problems allowing subsequent data changes

Summary In this paper we describe how to use Gram-Schmidt factorizations of Daniel et al. [1] to realize the method of [8] for solving linearly constrained linear least squares problems. The main advantage of using these factorizations is that it is relatively easy to take data changes into account, if necessary.The algorithm is compared numerically with two other codes, one of them published by Lawson and Hanson [3]. Further computational tests show the efficiency of the proposed methods, if a few data of the original problem are changed subsequently.This paper was sponsored by Deutsche Forschungsgemeinschaft, Bonn-Bad Godesberg     

一类线性约束矩阵不等式及其最小二乘问题

本文主要考虑一类线性矩阵不等式及其最小二乘问题,它等价于相应的矩阵不等式最小非负偏差问题.之前相关文献提出了求解该类最小非负偏差问题的迭代方法,但该方法在每步迭代过程中需要精确求解一个约束最小二乘子问题,因此对规模较大的问题,整个迭代过程需要耗费巨大的计算量.为了提高计算效率,本文在现有算法的基础上,提出了一类修正迭代方法.该方法在每步迭代过程中利用有限步的矩阵型LSQR方法求解一个低维矩阵Krylov子空间上的约束最小二乘子问题,降低了整个迭代所需的计算量.进一步运用投影定理以及相关的矩阵分析方法证明了该修正算法的收敛性,最后通过数值例子验证了本文的理论结果以及算法的有效性.     

Round off error analysis for Gram-Schmidt method and solution of linear least squares problems

Round off error analysis for the classical Gram-Schmidt orthogonalization method with re-orthogonalization is presented. The effect of the round-off error on the orthogonality of the derived vectors and also on the solution of the linear least squares problems when solved by the Gram-Schmidt algorithm are given. Numerical results compared favorably with the results of other methods. The classical case when no re-orthogonalization takes place is also discussed.     

A stable method for solving certain constrained least squares problems

This paper presents a feasible descent algorithm for solving certain constrained least squares problems. These problems are specially structured quadratic programming problems with positive semidefinite Hessian matrices that are allowed to be singular. The algorithm generates a finite sequence of subproblems that are solved using the numerically stable technique of orthogonal factorization with reorthogonalization and Given's transformation updating.This material is based upon work supported by the National Science Foundation under Grant No. MCS 78-06716 and by the International Institute for Applied Systems Analysis.     

Perturbation theory for generalized and constrained linear least squares

The perturbation analysis of weighted and constrained rank‐deficient linear least squares is difficult without the use of the augmented system of equations. In this paper a general form of the augmented system is used to get simple perturbation identities and perturbation bounds for the general linear least squares problem both for the full‐rank and rank‐deficient problem. Perturbation identities for the rank‐deficient weighted and constrained case are found as a special case. Interesting perturbation bounds and condition numbers are derived that may be useful when considering the stability of a solution of the rank‐deficient general least squares problem. Copyright © 2000 John Wiley & Sons, Ltd.     

A parallel multisplitting solution of the least squares problem

The linear least squares problem, min_x∥Ax − b∥₂, is solved by applying a multisplitting (MS) strategy in which the system matrix is decomposed by columns into p blocks. The b and x vectors are partitioned consistently with the matrix decomposition. The global least squares problem is then replaced by a sequence of local least squares problems which can be solved in parallel by MS. In MS the solutions to the local problems are recombined using weighting matrices to pick out the appropriate components of each subproblem solution. A new two-stage algorithm which optimizes the global update each iteration is also given. For this algorithm the updates are obtained by finding the optimal update with respect to the weights of the recombination. For the least squares problem presented, the global update optimization can also be formulated as a least squares problem of dimension p. Theoretical results are presented which prove the convergence of the iterations. Numerical results which detail the iteration behavior relative to subproblem size, convergence criteria and recombination techniques are given. The two-stage MS strategy is shown to be effective for near-separable problems. © 1998 John Wiley & Sons, Ltd.     

Algebraic relations between the total least squares and least squares problems with more than one solution

Summary This paper completes our previous discussion on the total least squares (TLS) and the least squares (LS) problems for the linear systemAX=B which may contain more than one solution [12, 13], generalizes the work of Golub and Van Loan [1,2], Van Huffel [8], Van Huffel and Vandewalle [11]. The TLS problem is extended to the more general case. The sets of the solutions and the squared residuals for the TLS and LS problems are compared. The concept of the weighted squares residuals is extended and the difference between the TLS and the LS approaches is derived. The connection between the approximate subspaces and the perturbation theories are studied.It is proved that under moderate conditions, all the corresponding quantities for the solution sets of the TLS and the modified LS problems are close to each other, while the quantities for the solution set of the LS problem are close to the corresponding ones of a subset of that of the TLS problem.This work was financially supported by the Education Committee, People's Republic of China     

A Newton Acceleration of the Weiszfeld Algorithm for Minimizing the Sum of Euclidean Distances

The Weiszfeld algorithm for continuous location problems can be considered as an iteratively reweighted least squares method. It generally exhibits linear convergence. In this paper, a Newton algorithm with similar simplicity is proposed to solve a continuous multifacility location problem with the Euclidean distance measure. Similar to the Weiszfeld algorithm, the main computation can be solving a weighted least squares problem at each iteration. A Cholesky factorization of a symmetric positive definite band matrix, typically with a small band width (e.g., a band width of two for a Euclidean location problem on a plane) is performed. This new algorithm can be regarded as a Newton acceleration to the Weiszfeld algorithm with fast global and local convergence. The simplicity and efficiency of the proposed algorithm makes it particularly suitable for large-scale Euclidean location problems and parallel implementation. Computational experience suggests that the proposed algorithm often performs well in the absence of the linear independence or strict complementarity assumption. In addition, the proposed algorithm is proven to be globally convergent under similar assumptions for the Weiszfeld algorithm. Although local convergence analysis is still under investigation, computation results suggest that it is typically superlinearly convergent.     

Perturbation Identities for Regularized Tikhonov Inverses and Weighted Pseudoinverses

We consider the perturbation analysis of two important problems for solving ill-conditioned or rank-deficient linear least squares problems. The Tikhonov regularized problem is a linear least squares problem with a regularization term balancing the size of the residual against the size of the weighted solution. The weight matrix can be a non-square matrix (usually with fewer rows than columns). The minimum-norm problem is the minimization of the size of the weighted solutions given by the set of solutions to the, possibly rank-deficient, linear least squares problem.It is well known that the solution of the Tikhonov problem tends to the minimum-norm solution as the regularization parameter of the Tikhonov problem tends to zero. Using this fact and the generalized singular value decomposition enable us to make a perturbation analysis of the minimum-norm problem with perturbation results for the Tikhonov problem. From the analysis we attain perturbation identities for Tikhonov inverses and weighted pseudoinverses.     

A penalty-free method with line search for nonlinear equality constrained optimization

A new line search method is introduced for solving nonlinear equality constrained optimization problems. It does not use any penalty function or a filter. At each iteration, the trial step is determined such that either the value of the objective function or the measure of the constraint violation is sufficiently reduced. Under usual assumptions, it is shown that every limit point of the sequence of iterates generated by the algorithm is feasible, and there exists at least one limit point that is a stationary point for the problem. A simple modification of the algorithm by introducing second order correction steps is presented. It is shown that the modified method does not suffer from the Maratos’ effect, so that it converges superlinearly. The preliminary numerical results are reported.     

A numerically stable least squares solution to the quadratic programming problem

The strictly convex quadratic programming problem is transformed to the least distance problem — finding the solution of minimum norm to the system of linear inequalities. This problem is equivalent to the linear least squares problem on the positive orthant. It is solved using orthogonal transformations, which are memorized as products. Like in the revised simplex method, an auxiliary matrix is used for computations. Compared to the modified-simplex type methods, the presented dual algorithm QPLS requires less storage and solves ill-conditioned problems more precisely. The algorithm is illustrated by some difficult problems.      

Robust regression for mixed Poisson–Gaussian model

This paper focuses on efficient computational approaches to compute approximate solutions of a linear inverse problem that is contaminated with mixed Poisson–Gaussian noise, and when there are additional outliers in the measured data. The Poisson–Gaussian noise leads to a weighted minimization problem, with solution-dependent weights. To address outliers, the standard least squares fit-to-data metric is replaced by the Talwar robust regression function. Convexity, regularization parameter selection schemes, and incorporation of non-negative constraints are investigated. A projected Newton algorithm is used to solve the resulting constrained optimization problem, and a preconditioner is proposed to accelerate conjugate gradient Hessian solves. Numerical experiments on problems from image deblurring illustrate the effectiveness of the methods.     

On growth factors of the modified Gram–Schmidt algorithm

Growth factors play a central role in studying the stability properties and roundoff estimates of matrix factorizations; therefore, they have attracted many numerical analysts to study upper bounds of these growth factors. In this article, we derive several upper bounds of row‐wise growth factors of the modified Gram–Schmidt (MGS) algorithm to solve the least squares (LS) problem and the weighted LS problem. We also extend the analysis to the MGS‐like algorithm to solve the constrained LS problem. Copyright © 2008 John Wiley & Sons, Ltd.     

An algorithm for solving the indefinite least squares problem with equality constraints

An algorithm for computing the solution of indefinite least squares problems and of indefinite least squares problems with equality constrained is presented. Such problems arise when solving total least squares problems and in H ^∞-smoothing. The proposed algorithm relies only on stable orthogonal transformations reducing recursively the associated augmented matrix to proper block anti-triangular form. Some numerical results are reported showing the properties of the algorithm.     

Condition Number Theorems in Linear-Quadratic Optimization

The condition number of a given mathematical problem is often related to the reciprocal of its distance from ill-conditioning. Such a property is proved here in the infinite-dimensional setting for linear-quadratic convex optimization of two types: linearly constrained convex quadratic problems, and minimum norm least squares solutions. A uniform version of such theorem is obtained in both cases for suitably equi-bounded classes of optimization problems. An application to the conditioning of a Ritz method is presented. For least squares problems it is shown that the semi-Fredholm property of the operators involved determines the validity of a condition number theorem.