Solving shortest length least-squares problems via dynamic programming

If the matrixA is not of full rank, there may be many solutions to the problem of minimizing Ax–b overx. Among such vectorsx, the unique one for which x is minimum is of importance in applications. This vector may be represented asx=A ⁺ b. In this paper, the functional equation technique of dynamic programming is used to find the shortest solution to the least-squares problem in a sequential fashion. The algorithm is illustrated with an example.Our debt to the late Professor Richard Bellman is clear, and we wish to thank Professor Harriet Kagiwada for many stimulating conversations concerning least-squares problems over a long period of years.     

KKT Conditions for Rank-Deficient Nonlinear Least-Square Problems with Rank-Deficient Nonlinear Constraints

In nonlinear least-square problems with nonlinear constraints, the function , where f ₂ is a nonlinear vector function, is to be minimized subject to the nonlinear constraints f ₁(x)=0. This problem is ill-posed if the first-order KKT conditions do not define a locally unique solution. We show that the problem is ill-posed if either the Jacobian of f ₁ or the Jacobian of J is rank-deficient (i.e., not of full rank) in a neighborhood of a solution satisfying the first-order KKT conditions. Either of these ill-posed cases makes it impossible to use a standard Gauss–Newton method. Therefore, we formulate a constrained least-norm problem that can be used when either of these ill-posed cases occur. By using the constant-rank theorem, we derive the necessary and sufficient conditions for a local minimum of this minimum-norm problem. The results given here are crucial for deriving methods solving the rank-deficient problem.     

Approximating minimum norm solutions of rank-deficient least squares problems

In this paper we consider the solution of linear least squares problems min_x∥Ax - b∥²₂ where the matrix A ∈ R ^{m × n} is rank deficient. Put p = min{m, n}, let σ_i, i = 1, 2,…, p, denote the singular values of A, and let u_i and v_i denote the corresponding left and right singular vectors. Then the minimum norm solution of the least squares problem has the form x* = ∫^r_{i = 1}(u^T_ib/σ_i)v_i, where r ≤ p is the rank of A. The Riley–Golub iteration, x_{k + 1} = arg min_x{∥Ax - b∥²₂ + λ∥x − x_k∥²₂} converges to the minimum norm solution if x₀ is chosen equal to zero. The iteration is implemented so that it takes advantage of a bidiagonal decomposition of A. Thus modified, the iteration requires only O(p) flops (floating point operations). A further gain of using the bidiagonalization of A is that both the singular values σ_i and the scalar products u^T_ib can be computed at marginal extra cost. Moreover, we determine the regularization parameter, λ, and the number of iterations, k, in a way that minimizes the difference x* − x_k with respect to a certain norm. Explicit rules are derived for calculating these parameters. One advantage of our approach is that the numerical rank can be easily determined by using the singular values. Furthermore, by the iterative procedure, x* is approximated without computing the singular vectors of A. This gives a fast and reliable method for approximating minimum norm solutions of well-conditioned rank-deficient least squares problems. Numerical experiments illustrate the viability of our ideas, and demonstrate that the new method gives more accurate approximations than an approach based on a QR decomposition with column pivoting. © 1998 John Wiley & Sons, Ltd.     

Perturbation Identities for Regularized Tikhonov Inverses and Weighted Pseudoinverses

We consider the perturbation analysis of two important problems for solving ill-conditioned or rank-deficient linear least squares problems. The Tikhonov regularized problem is a linear least squares problem with a regularization term balancing the size of the residual against the size of the weighted solution. The weight matrix can be a non-square matrix (usually with fewer rows than columns). The minimum-norm problem is the minimization of the size of the weighted solutions given by the set of solutions to the, possibly rank-deficient, linear least squares problem.It is well known that the solution of the Tikhonov problem tends to the minimum-norm solution as the regularization parameter of the Tikhonov problem tends to zero. Using this fact and the generalized singular value decomposition enable us to make a perturbation analysis of the minimum-norm problem with perturbation results for the Tikhonov problem. From the analysis we attain perturbation identities for Tikhonov inverses and weighted pseudoinverses.