On the convergence of the partial least squares path modeling algorithm

This paper adds to an important aspect of Partial Least Squares (PLS) path modeling, namely the convergence of the iterative PLS path modeling algorithm. Whilst conventional wisdom says that PLS always converges in practice, there is no formal proof for path models with more than two blocks of manifest variables. This paper presents six cases of non-convergence of the PLS path modeling algorithm. These cases were estimated using Mode A combined with the factorial scheme or the path weighting scheme, which are two popular options of the algorithm. As a conclusion, efforts to come to a proof of convergence under these schemes can be abandoned, and users of PLS should triangulate their estimation results.     

Local convergence of a secant type method for solving least squares problems

Local convergence of a secant type iterative method for approximating a solution of nonlinear least squares problems is investigated in this paper. The radius of convergence is determined as well as usable error estimates. Numerical examples are also provided.     

On the consistency of a least squares identification procedure in linear evolution systems

Conditions for the convergence of parameter estimates to the true value applicable in continuous time linear stochastic evolution systems are presented. A special case, continuous time linear stochastic systems with delays, is also considered. A persistent excitation property is proved by control theory methods     

Gamma-convergence and the least squares method

We study the asymptotic behaviour of the sequence of first order equations _tu––a_h(y)_xu=0 by means of the related energies

Convergence conditions of the least squares method

Least squares estimation of parameters is considered. Sufficient conditions of strong consistency are obtained for a regression model in the presence of random errors.Kiev University. Translated from Vychislitel'naya i Prikladnaya Matematika, No. 75, pp. 38–43, 1991.     

On the linear convergence of the alternating direction method of multipliers

We analyze the convergence rate of the alternating direction method of multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex separable functions subject to linear constraints. Previous analysis of the ADMM typically assumes that the objective function is the sum of only two convex functions defined on two separable blocks of variables even though the algorithm works well in numerical experiments for three or more blocks. Moreover, there has been no rate of convergence analysis for the ADMM without strong convexity in the objective function. In this paper we establish the global R-linear convergence of the ADMM for minimizing the sum of any number of convex separable functions, assuming that a certain error bound condition holds true and the dual stepsize is sufficiently small. Such an error bound condition is satisfied for example when the feasible set is a compact polyhedron and the objective function consists of a smooth strictly convex function composed with a linear mapping, and a nonsmooth \(\ell _1\) regularizer. This result implies the linear convergence of the ADMM for contemporary applications such as LASSO without assuming strong convexity of the objective function.     

200 years of least squares method

---

200 years of least squares method

---

     

Minimax estimation and the least squares method

This paper is devoted to the problem of minimax estimation of parameters in linear regression models with uncertain second order statistics. The solution to the problem is shown to be the least squares estimator corresponding to the least favourable matrix of the second moments. This allows us to construct a new algorithm for minimax estimation closely connected with the least squares method. As an example, we consider the problem of polynomial regression introduced by A. N. Kolmogorov     

On polynomial approximation in the uniform norm by the discrete least squares method

The discrete least squares method is convenient for computing polynomial approximations to functions. We investigate the possibility of using this method to obtain polynomial approximants good in the uniform norm, and find that for a given set ofm nodes, the degreen of the approximating polynomial should be selected so that there is a subset ofn+1 nodes which are close ton+1 Fejér points for the curve. Numerical examples are presented.Sponsored by the United States Army under Contract No. DAAG29-80-C-0041.     

Exponential convergence in a Galerkin least squares hp-FEM for Stokes flow

A stabilized hp-finite element method (FEM) of Galerkin leastsquares (GLS) type is analysed for the Stokes equations in polygonaldomains. Contrary to the standard Galerkin FEM, the GLSFEM admitsthe implementationally attractive equal-order interpolationin the velocity and the pressure. In conjunction with geometricallyrefined meshes and linearly increasing approximation ordersit is shown that the hp-GLSFEM leads to exponential rates ofconvergence for solutions exhibiting singularities near corners.To obtain this result a novel hp-interpolation result is provedthat allows the approximation of pressure functions in certainweighted Sobolev spaces in a conforming way and at exponentialrates of convergence on geometric meshes. Received 6 June 1999. Accepted 14 March 2000.     

Development of the collocations and least squares method

We propose and implement new, more general versions of the method of collocations and least squares (the CLS method) and, for a system of linear algebraic equations, an orthogonal method for accelerating the convergence of an iterative solution. The use of the latter method and the proper choice of values of control parameters, based on the results of investigating the dependence of the properties of the CLS method on these parameters, as well as some other improvements of the CLS method suggested in this paper, allow one to solve numerically problems for Navier-Stokes equations in a reasonable time using a single-processor computer even for grids as large as 1280 × 1280. In this case, the total number of unknown variables is ～ 25 · 10⁶. The numerical results for the problem of the lid-driven cavity flow of a viscous fluid are in good agreement with known results of other authors, including those obtained by means of schemes of higher approximation order with a small artificial viscosity. This and some other facts prove that the new versions of the CLS method make it possible to obtain an approximate solution with high accuracy.     

Sparse linear problems of the least squares method

One gives a survey of the direct method of solving large sparse diverse overdetermined linear systems of full column rank in the least squares sense. The survey covers practically all investigations on this topic, published in the seventies and in the beginning of the eighties.Translated from Itogi Nauki i Tekhniki, Seriya Matematicheskii Analiz, Vol. 23, pp. 219–285, 1985.     

The accuracy of the method of least squares

A generalization of the method of least squares is considered. Asymptotic properties of the estimates obtained with the help of the method in the case of stationary noise are studied. Bibliography:1 title. Translated fromZapiski Nauchnykh Seminarov POMI, Vol. 228, 1996, pp. 294–299     

On the approximation to solutions of operator equations by the least squares method

We consider the equation Au = f, where A is a linear operator with compact inverse A ^–1 in a separable Hilbert space . For the approximate solution u _n of this equation by the least squares method in a coordinate system {e _k } _k that is an orthonormal basis of eigenvectors of a self-adjoint operator B similar to A ( (B) = (A)), we give a priori estimates for the asymptotic behavior of the expressions r _n = u _n – u and R _n = Au _n – f as n . A relationship between the order of smallness of these expressions and the degree of smoothness of u with respect to the operator B is established.__________Translated from Funktsional nyi Analiz i Ego Prilozheniya, Vol. 39, No. 1, pp. 85–90, 2005Original Russian Text Copyright © by M. L. GorbachukSupported by CRDF and Ukrainian Government Joint Grant UM1-2567-OD03.Translated by V. M. Volosov     

Interpolation of functions by the least squares method

We study the norm of a best quadratic trigonometric approximation operator on a finite uniform grid. Dedicated to the memory of my scientific adviser, Professor S. B. Stechkin Translated fromMatematicheskie Zametki, Vol. 66, No. 3, pp. 372–379, September, 1999.     

On the weighting method for least squares problems with linear equality constraints

The weighting method for solving a least squares problem with linear equality constraints multiplies the constraints by a large number and appends them to the top of the least squares problem, which is then solved by standard techniques. In this paper we give a new analysis of the method, based on the QR decomposition, that exhibits many features of the algorithm. In particular it suggests a natural criterion for chosing the weighting factor. This work was supported in part by the National Science Foundation under grant CCR 95503126.