Perturbation analysis and condition numbers of scaled total least squares problems

The standard approaches to solving an overdetermined linear system Ax ≈ b find minimal corrections to the vector b and/or the matrix A such that the corrected system is consistent, such as the least squares (LS), the data least squares (DLS) and the total least squares (TLS). The scaled total least squares (STLS) method unifies the LS, DLS and TLS methods. The classical normwise condition numbers for the LS problem have been widely studied. However, there are no such similar results for the TLS and the STLS problems. In this paper, we first present a perturbation analysis of the STLS problem, which is a generalization of the TLS problem, and give a normwise condition number for the STLS problem. Different from normwise condition numbers, which measure the sizes of both input perturbations and output errors using some norms, componentwise condition numbers take into account the relation of each data component, and possible data sparsity. Then in this paper we give explicit expressions for the estimates of the mixed and componentwise condition numbers for the STLS problem. Since the TLS problem is a special case of the STLS problem, the condition numbers for the TLS problem follow immediately from our STLS results. All the discussions in this paper are under the Golub-Van Loan condition for the existence and uniqueness of the STLS solution. Yimin Wei is supported by the National Natural Science Foundation of China under grant 10871051, Shanghai Science & Technology Committee under grant 08DZ2271900 and Shanghai Education Committee under grant 08SG01. Sanzheng Qiao is partially supported by Shanghai Key Laboratory of Contemporary Applied Mathematics of Fudan University during his visiting.     

Bounds for the least squares distance using scaled total least squares

Summary. The standard approaches to solving overdetermined linear systems construct minimal corrections to the data to make the corrected system compatible. In ordinary least squares (LS) the correction is restricted to the right hand side c, while in scaled total least squares (STLS) [14,12] corrections to both c and B are allowed, and their relative sizes are determined by a real positive parameter . As , the STLS solution approaches the LS solution. Our paper [12] analyzed fundamentals of the STLS problem. This paper presents a theoretical analysis of the relationship between the sizes of the LS and STLS corrections (called the LS and STLS distances) in terms of . We give new upper and lower bounds on the LS distance in terms of the STLS distance, compare these to existing bounds, and examine the tightness of the new bounds. This work can be applied to the analysis of iterative methods which minimize the residual norm, and the generalized minimum residual method (GMRES) [15] is used here to illustrate our theory. Received July 20, 2000 / Revised version received February 28, 2001 / Published online July 25, 2001     

Backward perturbation analysis for scaled total least‐squares problems

The scaled total least‐squares (STLS) method unifies the ordinary least‐squares (OLS), the total least‐squares (TLS), and the data least‐squares (DLS) methods. In this paper we perform a backward perturbation analysis of the STLS problem. This also unifies the backward perturbation analyses of the OLS, TLS and DLS problems. We derive an expression for an extended minimal backward error of the STLS problem. This is an asymptotically tight lower bound on the true minimal backward error. If the given approximate solution is close enough to the true STLS solution (as is the goal in practice), then the extended minimal backward error is in fact the minimal backward error. Since the extended minimal backward error is expensive to compute directly, we present a lower bound on it as well as an asymptotic estimate for it, both of which can be computed or estimated more efficiently. Our numerical examples suggest that the lower bound gives good order of magnitude approximations, while the asymptotic estimate is an excellent estimate. We show how to use our results to easily obtain the corresponding results for the OLS and DLS problems in the literature. Copyright © 2009 John Wiley & Sons, Ltd.     

Scaled total least squares fundamentals

Summary. The standard approaches to solving overdetermined linear systems construct minimal corrections to the vector c and/or the matrix B such that the corrected system is compatible. In ordinary least squares (LS) the correction is restricted to c, while in data least squares (DLS) it is restricted to B. In scaled total least squares (STLS) [22], corrections to both c and B are allowed, and their relative sizes depend on a real positive parameter . STLS unifies several formulations since it becomes total least squares (TLS) when , and in the limit corresponds to LS when , and DLS when . This paper analyzes a particularly useful formulation of the STLS problem. The analysis is based on a new assumption that guarantees existence and uniqueness of meaningful STLS solutions for all parameters . It makes the whole STLS theory consistent. Our theory reveals the necessary and sufficient condition for preserving the smallest singular value of a matrix while appending (or deleting) a column. This condition represents a basic matrix theory result for updating the singular value decomposition, as well as the rank-one modification of the Hermitian eigenproblem. The paper allows complex data, and the equivalences in the limit of STLS with DLS and LS are proven for such data. It is shown how any linear system can be reduced to a minimally dimensioned core system satisfying our assumption. Consequently, our theory and algorithms can be applied to fully general systems. The basics of practical algorithms for both the STLS and DLS problems are indicated for either dense or large sparse systems. Our assumption and its consequences are compared with earlier approaches. Received June 2, 1999 / Revised version received July 3, 2000 / Published online July 25, 2001     

Zur Kondition des linearen Ausgleichsproblems mit linearen Gleichungen als Nebenbedingungen

Summary F.L. Bauer has treated in several papers [1, 3, 4] the condition related to the solution of linear equations and to the algebraic eigenvalue problem. We study the condition for the linear least squares problem with linear equality constraints (problem LSE). A perturbation theory of problem LSE is presented and three condition numbers are defined. Problem LSE includes the linear least squares problem (problem LS). There are examples with identical solution of problem LSE and of problem LS. Sometimes the condition of problem LSE is better and sometimes the condition of problem LS is better. Several numerical tests illustrate the theory.

Herrn Prof. Dr. Dr. F.L. Bauer zum 60. Geburtstag gewidmet     

The structured sensitivity of Vandermonde-like systems

Summary We consider a general class of structured matrices that includes (possibly confluent) Vandermonde and Vandermonde-like matrices. Here the entries in the matrix depend nonlinearly upon a vector of parameters. We define, condition numbers that measure the componentwise sensitivity of the associated primal and dual solutions to small componentwise perturbations in the parameters and in the right-hand side. Convenient expressions are derived for the infinity norm based condition numbers, and order-of-magnitude estimates are given for condition numbers defined in terms of a general vector norm. We then discuss the computation of the corresponding backward errors. After linearising the constraints, we derive an exact expression for the infinity norm dual backward error and show that the corresponding primal backward error is given by the minimum infinity-norm solution of an underdetermined linear system. Exact componentwise condition numbers are also derived for matrix inversion and the least squares problem, and the linearised least squares backward error is characterised.     

Analysis of the Structured Total Least Squares Problem for Hankel/Toeplitz Matrices

The Structured Total Least Squares (STLS) problem is a natural extension of the Total Least Squares (TLS) approach when structured matrices are involved and a similarly structured rank deficient approximation of that matrix is desired. In many of those cases the STLS approach yields a Maximum Likelihood (ML) estimate as opposed to, e.g., TLS.In this paper we analyze the STLS problem for Hankel matrices (the theory can be extended in a straightforward way to Toeplitz matrices, block Hankel and block Toeplitz matrices). Using a particular parametrisation of rank-deficient Hankel matrices, we show that this STLS problem suffers from multiple local minima, the properties of which depend on the parameters of the new parametrisation. The latter observation makes initial estimates an important issue in STLS problems and a new initialization method is proposed. The new initialization method is applied to a speech compression example and the results confirm the improved performance compared to other previously proposed initialization methods.     

Perturbation analysis of singular subspaces and deflating subspaces

Summary. Perturbation expansions for singular subspaces of a matrix and for deflating subspaces of a regular matrix pair are derived by using a technique previously described by the author. The perturbation expansions are then used to derive Fr\'echet derivatives, condition numbers, and th-order perturbation bounds for the subspaces. Vaccaro's result on second-order perturbation expansions for a special class of singular subspaces can be obtained from a general result of this paper. Besides, new perturbation bounds for singular subspaces and deflating subspaces are derived by applying a general theorem on solution of a system of nonlinear equations. The results of this paper reveal an important fact: Each singular subspace and each deflating subspace have individual perturbation bounds and individual condition numbers. Received July 26, 1994     

Iterative refinement for constrained and weighted linear least squares

We present an algorithm for mixed precision iterative refinement on the constrained and weighted linear least squares problem, the CWLSQ problem. The approximate solution is obtained by solving the CWLSQ problem with the weightedQR factorization [6]. With backward errors for the weightedQR decomposition together with perturbation bounds for the CWLSQ problem we analyze the convergence behaviour of the iterative refinement procedure.In the unweighted case the initial convergence rate of the error of the iteratively refined solution is determined essentially by the condition number. For the CWLSQ problem the initial convergence behaviour is more complicated. The analysis shows that the initial convergence is dependent both on the condition of the problem related to the solution,x, and the vector =Wr, whereW is the weight matrix andr is the residual.We test our algorithm on two examples where the solution is known and the condition number of the problem can be varied. The computational test confirms the theoretical results and verifies that mixed precision iterative refinement, using the system matrix and the weightedQR decomposition, is an effective way of improving an approximate solution to the CWLSQ problem.     

On condition numbers and algorithms for determining a rigid body movement

Using a set of landmarks to represent a rigid body, a rotation of the body can be determined in least-squares sense as the solution of an orthogonal Procrustes problem. We discuss some geometrical properties of the condition number for the problem of determining the orthogonal matrix representing the rotation. It is shown that the condition number critically depends on the configuration of the landmarks. The problem is also reformulated as an unconstrained nonlinear least-squares problem and the condition number is related to the geometry of such problems. In the common 3-D case, the movement can be represented by using a screw axis. Also the condition numbers for the problem of determining the screw axis representation are shown to closely depend on the configuration of the landmarks. The condition numbers are finally used to show that the used algorithms are stable.     

Perturbation analysis of generalized saddle point systems

We discuss the perturbation analysis of generalized saddle point systems in this paper. We give the nonlinear perturbation bounds, then derive the condition numbers, and analyze the sensitivity of the computed solutions.     

A matrix perturbation method for computing the steady-state probability distributions of probabilistic Boolean networks with gene perturbations

Modeling genetic regulatory interactions is an important issue in systems biology. Probabilistic Boolean networks (PBNs) have been proved to be a useful tool for the task. The steady-state probability distribution of a PBN gives important information about the captured genetic network. The computation of the steady-state probability distribution involves the construction of the transition probability matrix of the PBN. The size of the transition probability matrix is 2ⁿ×2ⁿ where n is the number of genes. Although given the number of genes and the perturbation probability in a perturbed PBN, the perturbation matrix is the same for different PBNs, the storage requirement for this matrix is huge if the number of genes is large. Thus an important issue is developing computational methods from the perturbation point of view. In this paper, we analyze and estimate the steady-state probability distribution of a PBN with gene perturbations. We first analyze the perturbation matrix. We then give a perturbation matrix analysis for the captured PBN problem and propose a method for computing the steady-state probability distribution. An approximation method with error analysis is then given for further reducing the computational complexity. Numerical experiments are given to demonstrate the efficiency of the proposed methods.     

Estimation of optimal backward perturbation bounds for the linear least squares problem

In this paper a method of estimating the optimal backward perturbation bound for the linear least squares problem is presented. In contrast with the optimal bound, which requires a singular value decomposition, this method is better suited for practical use on large problems since it requiresO(mn) operations. The method presented involves the computation of a strict lower bound for the spectral norm and a strict upper bound for the Frobenius norm which gives a gap in which the optimal bounds for the spectral and the Frobenius norm must be. Numerical tests are performed showing that this method produces an efficient estimate of the optimal backward perturbation bound.     

Fast algorithm for solving the Hankel/Toeplitz Structured Total Least Squares problem

The Structured Total Least Squares (STLS) problem is a natural extension of the Total Least Squares (TLS) problem when constraints on the matrix structure need to be imposed. Similar to the ordinary TLS approach, the STLS approach can be used to determine the parameter vector of a linear model, given some noisy measurements. In many signal processing applications, the imposition of this matrix structure constraint is necessary for obtaining Maximum Likelihood (ML) estimates of the parameter vector. In this paper we consider the Toeplitz (Hankel) STLS problem (i.e., an STLS problem in which the Toeplitz (Hankel) structure needs to be preserved). A fast implementation of an algorithm for solving this frequently occurring STLS problem is proposed. The increased efficiency is obtained by exploiting the low displacement rank of the involved matrices and the sparsity of the associated generators. The fast implementation is compared to two other implementations of algorithms for solving the Toeplitz (Hankel) STLS problem. The comparison is carried out on a recently proposed speech compression scheme. The numerical results confirm the high efficiency of the newly proposed fast implementation: the straightforward implementations have a complexity of O((m+n)³) and O(m³) whereas the proposed implementation has a complexity of O(mn+n²). This revised version was published online in June 2006 with corrections to the Cover Date.     

A contribution to the theory of condition

Summary In this paper the use of the condition number of a problem, as defined by Rice in 1966, is discussed. For the eigenvalue, eigenvector, and linear least squares problems either condition numbers according to various norms are determined or lower and upper bounds for them are derived.     

Isospectral flow method for nonnegative inverse eigenvalue problem with prescribed structure

The nonnegative inverse eigenvalue problem is that given a family of complex numbers λ={λ₁,…,λ_n}, find a nonnegative matrix of order n with spectrum λ. This problem is difficult and remains unsolved partially. In this paper, we focus on its generalization that the reconstructed nonnegative matrices should have some prescribed entries. It is easy to see that this new problem will come back to the common nonnegative inverse eigenvalue problem if there is no constraint of the locations of entries. A numerical isospectral flow method which is developed by hybridizing the optimization theory and steepest descent method is used to study the reconstruction. Moreover, an error estimate of the numerical iteration for ordinary differential equations on the matrix manifold is presented. After that, a numerical method for the nonnegative symmetric inverse eigenvalue problem with prescribed entries and its error estimate are considered. Finally, the approaches are verified by the numerical test results.     

On the condition number of the antireflective transform

Deconvolution problems with a finite observation window require appropriate models of the unknown signal in order to guarantee uniqueness of the solution. For this purpose it has recently been suggested to impose some kind of antireflectivity of the signal. With this constraint, the deconvolution problem can be solved with an appropriate modification of the fast sine transform, provided that the convolution kernel is symmetric. The corresponding transformation is called the antireflective transform. In this work we determine the condition number of the antireflective transform to first order, and use this to show that the so-called reblurring variant of Tikhonov regularization for deconvolution problems is a regularization method. Moreover, we establish upper bounds for the regularization error of the reblurring strategy that hold uniformly with respect to the size n of the algebraic system, even though the condition number of the antireflective transform grows with n. We briefly sketch how our results extend to higher space dimensions.     

Perturbation analysis of generalized singular subspaces

This paper, as a continuation of the paper [20] in Numerische Mathematik, studies the subspaces associated with the generalized singular value decomposition. Second order perturbation expansions, Fréchet derivatives and condition numbers, and perturbation bounds for the subspaces are derived. Received January 26, 1996 / Revised version received May 14, 1997     

On backward errors of structured polynomial eigenproblems solved by structure preserving linearizations

We derive explicit computable expressions of structured backward errors of approximate eigenelements of structured matrix polynomials including symmetric, skew-symmetric, Hermitian, skew-Hermitian, even and odd polynomials. We determine minimal structured perturbations for which approximate eigenelements are exact eigenelements of the perturbed polynomials. We also analyze structured pseudospectra of a structured matrix polynomial and establish a partial equality between unstructured and structured pseudospectra. Finally, we analyze the effect of structure preserving linearizations of structured matrix polynomials on the structured backward errors of approximate eigenelements and show that structure preserving linearizations which minimize structured condition numbers of eigenvalues also minimize the structured backward errors of approximate eigenelements.     

Structured mixed and componentwise condition numbers of some structured matrices

Structured matrices, such as Cauchy, Vandermonde, Toeplitz, Hankel, and circulant matrices, are considered in this paper. We apply a Kronecker product-based technique to deduce the structured mixed and componentwise condition numbers for the matrix inversion and for the corresponding linear systems.