A modified Gram–Schmidt algorithm with iterative orthogonalization and column pivoting

Iterative orthogonalization is aimed to ensure small deviation from orthogonality in the Gram–Schmidt process. Former applications of this technique are restricted to classical Gram–Schmidt (CGS) and column-oriented modified Gram–Schmidt (MGS). The major aim of this paper is to explain how iterative orthogonalization is incorporated into row-oriented MGS. The interest that we have in a row-oriented iterative MGS comes from the observation that this method is capable of performing column pivoting. The use of column pivoting delays the deteriorating effects of rounding errors and helps to handle rank-deficient least-squares problems.

A second modification proposed in this paper considers the use of Gram–Schmidt QR factorization for solving linear least-squares problems. The standard solution method is based on one orthogonalization of the r.h.s. vector b against the columns of Q. The outcome of this process is the residual vector, r^*, and the solution vector, x^*. The modified scheme is a natural extension of the standard solution method that allows it to apply iterative orthogonalization. This feature ensures accurate computation of small residuals and helps in cases when Q has some deviation from orthogonality.     

An accurate parallel block Gram–Schmidt algorithm without reorthogonalization

The modified Gram–Schmidt (MGS) orthogonalization process—used for example in the Arnoldi algorithm—often constitutes the bottleneck that limits parallel efficiencies. Indeed, a number of communications, proportional to the square of the problem size, are required to compute the dot‐products. A block formulation is attractive but it suffers from potential numerical instability. In this paper, we address this issue and propose a simple procedure that allows the use of a block Gram—Schmidt algorithm while guaranteeing a numerical accuracy close to that of MGS. The main idea is to determine the size of the blocks dynamically. The main advantages of this dynamic procedure are two‐fold: first, high performance matrix–vector multiplications can be used to decrease the execution time. Next, in a parallel environment, the number of communications is reduced. Performance comparisons with the alternative Iterated CGS also show an improvement for a moderate number of processors. Copyright © 2000 John Wiley & Sons, Ltd.     

Accuracy of Gram–Schmidt orthogonalization and Householder transformation for the solution of linear least squares problems

Accuracy of a Gram–Schmidt algorithm for the solution of linear least squares equations is compared with accuracy of least squares subroutines in three highly respected mathematical packages that use Householder transformations. Results from the four programs for 13 test problems were evaluated at 16 digit precision on four different desktop computers using four different compilers. Singular values obtained from the different programs are compared and the effect of pivoting to improve the accuracy is discussed. Solution vectors from the program using the Gram–Schmidt algorithm were generally more accurate or comparable to solution vectors from the programs using the Householder transformations. © 1997 John Wiley & Sons, Ltd.     

A generalization of Gram–Schmidt orthogonalization generating all Parseval frames

Given an arbitrary finite sequence of vectors in a finite-dimensional Hilbert space, we describe an algorithm, which computes a Parseval frame for the subspace generated by the input vectors while preserving redundancy exactly. We further investigate several of its properties. Finally, we apply the algorithm to several numerical examples.      

On growth factors of the modified Gram–Schmidt algorithm

Growth factors play a central role in studying the stability properties and roundoff estimates of matrix factorizations; therefore, they have attracted many numerical analysts to study upper bounds of these growth factors. In this article, we derive several upper bounds of row‐wise growth factors of the modified Gram–Schmidt (MGS) algorithm to solve the least squares (LS) problem and the weighted LS problem. We also extend the analysis to the MGS‐like algorithm to solve the constrained LS problem. Copyright © 2008 John Wiley & Sons, Ltd.     

Computing projections via Householder transformations and Gram–Schmidt orthogonalizations

Let x * denote the solution of a linear least‐squares problem of the form where A is a full rank m × n matrix, m > n. Let r *= b ‐ A x * denote the corresponding residual vector. In most problems one is satisfied with accurate computation of x *. Yet in some applications, such as affine scaling methods, one is also interested in accurate computation of the unit residual vector r */∥ r *∥₂. The difficulties arise when ∥ r *∥₂ is much smaller than ∥ b ∥₂. Let x? and r? denote the computed values of x * and r *, respectively. Let εdenote the machine precision in our computations, and assume that r? is computed from the equality r? = b ‐A x? . Then, no matter how accurate x? is, the unit residual vector û = r? /∥ r? ∥₂ contains an error vector whose size is likely to exceed ε∥ b ∥₂/∥ r* ∥₂. That is, the smaller ∥ r* ∥₂ the larger the error. Thus although the computed unit residual should satisfy A^T û = 0 , in practice the size of ∥A^T û ∥₂ is about ε∥A∥₂∥ b ∥₂/∥ r* ∥₂. The methods discussed in this paper compute a residual vector, r? , for which ∥A^T r? ∥₂ is not much larger than ε∥A∥₂∥ r? ∥₂. Numerical experiments illustrate the difficulties in computing small residuals and the usefulness of the proposed safeguards. Copyright © 2004 John Wiley & Sons, Ltd.     

Least‐squares mixed finite element methods for the RLW equations

A least‐squares mixed finite element (LSMFE) schemes are formulated to solve the 1D regularized long wave (RLW) equations and the convergence is discussed. The L² error estimates of LSMFE methods for RLW equations under the standard regularity assumption on the finite element partition are given.© 2007 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2008     

An algorithm for least squares analysis of spectroscopic data

This paper describes a variant of the Gauss-Newton-Hartley algorithm for nonlinear least squares, in which aQR implementation is used to solve the linear least squares problem. We follow Grey's idea of updating variables at intermediate stages of the orthogonalization. This technique, applied in partitions identified with known or suspected spectral lines, appears to be especially suited to the analysis of spectroscopic data. We suggest that this algorithm is an attractive candidate for the optimization role in Ekenberg's interactive computer graphics curve fitting program.     

On the subdomain‐Galerkin/least squares method for 2‐ and 3‐D mixed elliptic problems with reaction terms

In this article we apply the subdomain‐Galerkin/least squares method, which is first proposed by Chang and Gunzburger for first‐order elliptic systems without reaction terms in the plane, to solve second‐order non‐selfadjoint elliptic problems in two‐ and three‐dimensional bounded domains with triangular or tetrahedral regular triangulations. This method can be viewed as a combination of a direct cell vertex finite volume discretization step and an algebraic least‐squares minimization step in which the pressure is approximated by piecewise linear elements and the flux by the lowest order Raviart‐Thomas space. This combined approach has the advantages of both finite volume and least‐squares methods. Among other things, the combined method is not subject to the Ladyzhenskaya‐Babus?ka‐Brezzi condition, and the resulting linear system is symmetric and positive definite. An optimal error estimate in the H¹(Ω) × H(div; Ω) norm is derived. An equivalent residual‐type a posteriori error estimator is also given. © 2002 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 18: 738–751, 2002; Published online in Wiley InterScience (www.interscience.wiley.com); DOI 10.1002/num.10030.     

Least‐squares spectral method for velocity‐vorticity‐pressure form of the Stokes equations

The aim of this article is to present and analyze first‐order system least‐squares spectral method for the Stokes equations in two‐dimensional spaces. The Stokes equations are transformed into a first‐order system of equations by introducing vorticity as a new variable. The least‐squares functional is then defined by summing up the ‐ and ‐norms of the residual equations. The ‐norm in the least‐squares functional is replaced by suitable operator. Continuous and discrete homogeneous least‐squares functionals are shown to be equivalent to ‐norm of velocity and ‐norm of vorticity and pressure for spectral Galerkin and pseudospectral method. The spectral convergence of the proposed methods are given and the theory is validated by numerical experiment. Mass conservation is also briefly investigated. © 2015 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 32: 661–680, 2016     

Dual‐ and triple‐mode matrix approximation and regression modelling

We propose a dual‐ and triple‐mode least squares for matrix approximation. This technique applied to the singular value decomposition produces the classical solution with a new interpretation. Applied to regression modelling, this approach corresponds to a regularized objective and yields a new solution with properties of a ridge regression. The results for regression are robust and suggest a convenient tool for the analysis and interpretation of the model coefficients. Numerical results are given for a marketing research data set. Copyright © 2003 John Wiley & Sons, Ltd.     

Split least‐squares finite element methods for non‐Fickian flow in porous media

In this article, we introduce two least‐squares finite element procedures for parabolic integro‐differential equations arising in the modeling of non‐Fickian flow in porous media. By selecting the least‐squares functional properly the presented procedure can be split into two independent subprocedures, one subprocedure is for the primitive unknown and the other is for the flux. The optimal order convergence analysis is established. Numerical examples are given to show the efficiency of the introduced schemes. © 2012 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2013     

New a priori analysis of first‐order system least‐squares finite element methods for parabolic problems

We provide new insights into the a priori theory for a time‐stepping scheme based on least‐squares finite element methods for parabolic first‐order systems. The elliptic part of the problem is of general reaction‐convection‐diffusion type. The new ingredient in the analysis is an elliptic projection operator defined via a nonsymmetric bilinear form, although the main bilinear form corresponding to the least‐squares functional is symmetric. This new operator allows to prove optimal error estimates in the natural norm associated to the problem and, under additional regularity assumptions, in the L² norm. Numerical experiments are presented which confirm our theoretical findings.     

Least‐squares method for the Oseen equation

This article studies the least‐squares finite element method for the linearized, stationary Navier–Stokes equation based on the stress‐velocity‐pressure formulation in d dimensions (d = 2 or 3). The least‐squares functional is simply defined as the sum of the squares of the L² norm of the residuals. It is shown that the homogeneous least‐squares functional is elliptic and continuous in the norm. This immediately implies that the a priori error estimate of the conforming least‐squares finite element approximation is optimal in the energy norm. The L² norm error estimate for the velocity is also established through a refined duality argument. Moreover, when the right‐hand side f belongs only to , we derive an a priori error bound in a weaker norm, that is, the norm. © 2016 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 32: 1289–1303, 2016     

An analysis for the compressible Stokes equations by first‐order system of least‐squares finite element method

This article applies the first‐order system least‐squares (fosls) finite element method developed by Cai, Manteuffel and McCormick to the compressible Stokes equations. By introducing a new dependent velocity flux variable, we recast the compressible Stokes equations as a first‐order system. Then it is shown that the ellipticity and continuity hold for the least‐squares functionals employing the mixture of H^?1 and L², so that the fosls finite element methods yield best approximations for the velocity flux and velocity. © 2001 John Wiley & Sons, Inc. Numer Methods Partial Differential Eq 17:689–699, 2001     

Least‐squares mixed finite element methods for the incompressible Navier‐Stokes equations

Least‐squares mixed finite element schemes are formulated to solve the evolutionary Navier‐Stokes equations and the convergence is analyzed. We recast the Navier‐Stokes equations as a first‐order system by introducing a vorticity flux variable, and show that a least‐squares principle based on L² norms applied to this system yields optimal discretization error estimates. © 2002 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 18: 441–453, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/num.10015     

Backward perturbation analysis for scaled total least‐squares problems

The scaled total least‐squares (STLS) method unifies the ordinary least‐squares (OLS), the total least‐squares (TLS), and the data least‐squares (DLS) methods. In this paper we perform a backward perturbation analysis of the STLS problem. This also unifies the backward perturbation analyses of the OLS, TLS and DLS problems. We derive an expression for an extended minimal backward error of the STLS problem. This is an asymptotically tight lower bound on the true minimal backward error. If the given approximate solution is close enough to the true STLS solution (as is the goal in practice), then the extended minimal backward error is in fact the minimal backward error. Since the extended minimal backward error is expensive to compute directly, we present a lower bound on it as well as an asymptotic estimate for it, both of which can be computed or estimated more efficiently. Our numerical examples suggest that the lower bound gives good order of magnitude approximations, while the asymptotic estimate is an excellent estimate. We show how to use our results to easily obtain the corresponding results for the OLS and DLS problems in the literature. Copyright © 2009 John Wiley & Sons, Ltd.     

First‐Order system least squares for generalized‐Newtonian coupled Stokes‐Darcy flow

The coupled problem for a generalized Newtonian Stokes flow in one domain and a generalized Newtonian Darcy flow in a porous medium is studied in this work. Both flows are treated as a first‐order system in a stress‐velocity formulation for the Stokes problem and a volumetric flux‐hydraulic potential formulation for the Darcy problem. The coupling along an interface is done using the well‐known Beavers–Joseph–Saffman interface condition. A least squares finite element method is used for the numerical approximation of the solution. It is shown that under some assumptions on the viscosity the error is bounded from above and below by the least squares functional. An adaptive refinement strategy is examined in several numerical examples where boundary singularities are present. Due to the nonlinearity of the problem a Gauss–Newton method is used to iteratively solve the problem. It is shown that the linear variational problems arising in the Gauss–Newton method are well posed. © 2014 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 31: 1150–1173, 2015     

Optimal uniform error estimates for moving least‐squares collocation with application to option pricing under jump‐diffusion processes

In this study, we derive optimal uniform error bounds for moving least‐squares (MLS) mesh‐free point collocation (also called finite point method) when applied to solve second‐order elliptic partial integro‐differential equations (PIDEs). In the special case of elliptic partial differential equations (PDEs), we show that our estimate improves the results of Cheng and Cheng (Appl. Numer. Math. 58 (2008), no. 6, 884–898) both in terms of the used error norm (here the uniform norm and there the discrete vector norm) and the obtained order of convergence. We then present optimal convergence rate estimates for second‐order elliptic PIDEs. We proceed by some numerical experiments dealing with elliptic PDEs that confirm the obtained theoretical results. The article concludes with numerical approximation of the linear parabolic PIDE arising from European option pricing problem under Merton's and Kou's jump‐diffusion models. The presented computational results (including the computation of option Greeks) and comparisons with other competing approaches suggest that the MLS collocation scheme is an efficient and reliable numerical method to solve elliptic and parabolic PIDEs arising from applied areas such as financial engineering.