Linear least squares problems with data over incomplete grids

The paper addresses bivariate surface fitting problems, where data points lie on the vertices of a rectangular grid. Efficient and stable algorithms can be found in the literature to solve such problems. If data values are missing at some grid points, there exists a computational method for finding a least squares spline by fixing appropriate values for the missing data. We extended this technique to arbitrary least squares problems as well as to linear least squares problems with linear equality constraints. Numerical examples are given to show the effectiveness of the technique presented. AMS subject classification (2000)  65D05, 65D07, 65D10, 65F05, 65F20     

Implicitly-weighted total least squares

In a total least squares (TLS) problem, we estimate an optimal set of model parameters X, so that (A-ΔA)X=B-ΔB, where A is the model matrix, B is the observed data, and ΔA and ΔB are corresponding corrections. When B is a single vector, Rao (1997) and Paige and Strakoš (2002) suggested formulating standard least squares problems, for which ΔA=0, and data least squares problems, for which ΔB=0, as weighted and scaled TLS problems. In this work we define an implicitly-weighted TLS formulation (ITLS) that reparameterizes these formulations to make computation easier. We derive asymptotic properties of the estimates as the number of rows in the problem approaches infinity, handling the rank-deficient case as well. We discuss the role of the ratio between the variances of errors in A and B in choosing an appropriate parameter in ITLS. We also propose methods for computing the family of solutions efficiently and for choosing the appropriate solution if the ratio of variances is unknown. We provide experimental results on the usefulness of the ITLS family of solutions.     

Perturbation theory for generalized and constrained linear least squares

The perturbation analysis of weighted and constrained rank‐deficient linear least squares is difficult without the use of the augmented system of equations. In this paper a general form of the augmented system is used to get simple perturbation identities and perturbation bounds for the general linear least squares problem both for the full‐rank and rank‐deficient problem. Perturbation identities for the rank‐deficient weighted and constrained case are found as a special case. Interesting perturbation bounds and condition numbers are derived that may be useful when considering the stability of a solution of the rank‐deficient general least squares problem. Copyright © 2000 John Wiley & Sons, Ltd.     

Bivariate least squares approximation with linear constraints

In this article linear least squares problems with linear equality constraints are considered, where the data points lie on the vertices of a rectangular grid. A fast and efficient computational method for the case when the linear equality constraints can be formulated in a tensor product form is presented. Using the solution of several univariate approximation problems the solution of the bivariate approximation problem can be derived easily. AMS subject classification (2000)  65D05, 65D07, 65D10, 65F05, 65F20     

Deep partial least squares for instrumental variable regression

In this paper, we propose deep partial least squares for the estimation of high-dimensional nonlinear instrumental variable regression. As a precursor to a flexible deep neural network architecture, our methodology uses partial least squares for dimension reduction and feature selection from the set of instruments and covariates. A central theoretical result, due to Brillinger (2012) Selected Works of Daving Brillinger. 589-606, shows that the feature selection provided by partial least squares is consistent and the weights are estimated up to a proportionality constant. We illustrate our methodology with synthetic datasets with a sparse and correlated network structure and draw applications to the effect of childbearing on the mother's labor supply based on classic data of Chernozhukov et al. Ann Rev Econ. (2015b):649–688. The results on synthetic data as well as applications show that the deep partial least squares method significantly outperforms other related methods. Finally, we conclude with directions for future research.     

Parallel variable distribution for total least squares

A novel parallel method for determining an approximate total least squares (TLS) solution is introduced. Based on domain decomposition, the global TLS problem is partitioned into several dependent TLS subproblems. A convergent algorithm using the parallel variable distribution technique (SIAM J. Optim. 1994; 4 :815–832) is presented. Numerical results support the development and analysis of the algorithms. Copyright © 2005 John Wiley & Sons, Ltd.     

Matrix stretching for sparse least squares problems

For linear least squares problems min _x ‖Ax − b‖₂, where A is sparse except for a few dense rows, a straightforward application of Cholesky or QR factorization will lead to catastrophic fill in the factor R. We consider handling such problems by a matrix stretching technique, where the dense rows are split into several more sparse rows. We develop both a recursive binary splitting algorithm and a more general splitting method. We show that for both schemes the stretched problem has the same set of solutions as the original least squares problem. Further, the condition number of the stretched problem differs from that of the original by only a modest factor, and hence the approach is numerically stable. Experimental results from applying the recursive binary scheme to a set of modified matrices from the Harwell‐Boeing collection are given. We conclude that when A has a small number of dense rows relative to its dimension, there is a significant gain in sparsity of the factor R. A crude estimate of the optimal number of splits is obtained by analysing a simple model problem. Copyright © 2000 John Wiley & Sons, Ltd.     

Circle fitting by linear and nonlinear least squares

The problem of determining the circle of best fit to a set of points in the plane (or the obvious generalization ton-dimensions) is easily formulated as a nonlinear total least-squares problem which may be solved using a Gauss-Newton minimization algorithm. This straight-forward approach is shown to be inefficient and extremely sensitive to the presence of outliers. An alternative formulation allows the problem to be reduced to a linear least squares problem which is trivially solved. The recommended approach is shown to have the added advantage of being much less sensitive to outliers than the nonlinear least squares approach.This work was completed while the author was visiting the Numerical Optimisation Centre, Hatfield Polytechnic and benefitted from the encouragement and helpful suggestions of Dr. M. C. Bartholomew-Biggs and Professor L. C. W. Dixon.     

Approximation to the distribution of the least squares estimators in two dimensional cosine models by randomly weighted bootstrap

Recently, Kundu and Gupta （Metrika, 48：83 C 97, 1998） established the asymptotic normality of the least squares estimators in the two dimensional cosine model. In this paper, we give the approximation to the general least squares estimators by using random weights which is called the Bayesian bootstrap or the random weighting method by Rubin （Annals of Statistics, 9：130 C 134, 1981） and Zheng （Acta Math. Appl. Sinica （in Chinese）, 10（2）： 247 C 253, 1987）. A simulation study shows that this approximation works very well.     

纵向数据下部分线性模型的渐近性质

纵向数据是在实际应用中很常见的一种数据类型,在解决实际问题时建立纵向数据模型,进行统计分析很实用。本文研究一类重要的纵向数据下部分线性回归模型,所分析的纵向数据是随机观测而得到的,根据纵向数据的特性构造模型中未知参数分量和未知函数的估计量,进而研究了估计量的渐近性质,通过实例分析,证实了该方法的有效性和可操作性,有很好的使用价值。     

Minimax estimation and the least squares method

This paper is devoted to the problem of minimax estimation of parameters in linear regression models with uncertain second order statistics. The solution to the problem is shown to be the least squares estimator corresponding to the least favourable matrix of the second moments. This allows us to construct a new algorithm for minimax estimation closely connected with the least squares method. As an example, we consider the problem of polynomial regression introduced by A. N. Kolmogorov     

基于纵向数据的半参数变系数部分线性回归模型

纵向数据是数理统计研究中的复杂数据类型之一0,在生物、医学和经济学中具有广泛的应用．在实际中经常需要对纵向数据进行统计分析和建模．文章讨论了纵向数据下的半参数变系数部分线性回归模型,这里的纵向数据的在纵向观察在时间上可以是不均等的,也可看成是按某一随机过程来发生．所研究的半参数变系数模型包括了许多半参数模型,比如部分线性模型和变系数模型等．利用计数过程理论和局部线性回归方法,对于纵向数据下半参数变系数进行了统计推断,给出了参数分量和非参数分量的profile最小二乘估计,研究了这些估计的渐近性质,获得这些估计的相合性和渐近正态性．     

Algorithms for separable nonlinear least squares with application to modelling time-resolved spectra

The multiexponential analysis problem of fitting kinetic models to time-resolved spectra is often solved using gradient-based algorithms that treat the spectral parameters as conditionally linear. We make a comparison of the two most-applied such algorithms, alternating least squares and variable projection. A numerical study examines computational efficiency and linear approximation standard error estimates. A new derivation of the Fisher information matrix under the full Golub-Pereyra gradient allows a numerical comparison of parameter precision under variable projection variants. Under the criteria of efficiency, quality of standard error estimates and parameter precision, we conclude that the Kaufman variable projection technique performs well, while techniques based on alternating least squares have significant disadvantages for application in the problem domain.     

A Berry-Esseen bound for least squares error variance estimators of regression parameters

We construct a precise Berry-Esseen bound for the least squares error variance estimators of regression parameters. Our bound depends explicitly on the sequence of design variables and is of the order O(N ^−1/2) if this sequence is “regular” enough. Supported by the Lithuanian State Science and Studies Foundation. Vilnius University, Naugarduko 24; Institute of Mathematics and Informatics, Akademijos 4, 2600 Vilnius, Lithuania. Published in Lietuvos Matematikos Rinkinys, Vol. 39, No. 1, pp. 1–8, January–March, 1999.     

删失数据下线性回归模型的参数估计

主要叙述在数据观测不完全的情况下,采用最小二乘法对线性回归模型回归系数的估计及估计量的渐进性质,并给出数据模拟．     

The condition numbers for weighted Moore-Penrose inverse and weighted linear least squares problem

Condition numbers play an important role in numerical analysis. Classical normise condition numbers are used to measure the size of both input perturbations and output errors. In this paper, we study the weighted normwise relative condition numbers for the weighted Moore-Penrose inverse and the weighted linear least-squares (WLS) problems in the case of the full-column rank matrix. The bounds or formulas for the weighted condition numbers are presented. The obtained results can be viewed as extensions of the earlier works studied by others.     

A weighted least squares method for scattered data fitting

In this paper, we present a weighted least squares method to fit scattered data with noise. Existence and uniqueness of a solution are proved and an error bound is derived. The numerical experiments illustrate that our weighted least squares method has better performance than the traditional least squares method in case of noisy data.     

A dynamic meshless method for the least squares problem with some noisy subdomains

The interpolation method by radial basis functions is used widely for solving scattered data approximation. However, sometimes it makes more sense to approximate the solution by least squares fit. This is especially true when the data are contaminated with noise. A meshfree method namely, meshless dynamic weighted least squares (MDWLS) method, is presented in this paper to solve least squares problems with noise. The MDWLS method by Gaussian radial basis function is proposed to fit scattered data with some noisy areas in the problem’s domain. Existence and uniqueness of a solution is proved. This method has one parameter which can adjusts the accuracy according to the size of noises. Another advantage of the developed method is that it can be applied to problems with nonregular geometrical domains. The new approach is applied for some problems in two dimensions and the obtained results confirm the accuracy and efficiency of the proposed method. The numerical experiments illustrate that our MDWLS method has better performance than the traditional least squares method in case of noisy data.     

泛最小二乘法的改进及其容许性

考虑线性回归模型,当设计阵呈病态或秩亏时,我们用泛最小二乘法给出参数的估计,并证明其容许性;然后针对泛最小二乘估计对最小二乘估计过度压缩的缺点加以改进,使之更合理,有效.