ON THE ACCURACY OF THE LEAST SQUARES AND THE TOTAL LEAST SQUARES METHODS

Consider solving an overdetermined system of linear algebraic equations by both the least squares method (LS) and the total least squares method (TLS). Extensive published computational evidence shows that when the original system is consistent. one often obtains more accurate solutions by using the TLS method rather than the LS method. These numerical observations contrast with existing analytic perturbation theories for the LS and TLS methods which show that the upper bounds for the LS solution are always smaller than the corresponding upper bounds for the TLS solutions. In this paper we derive a new upper bound for the TLS solution and indicate when the TLS method can be more accurate than the LS method.Many applied problems in signal processing lead to overdetermined systems of linear equations where the matrix and right hand side are determined by the experimental observations (usually in the form of a lime series). It often happens that as the number of columns of the matrix becomes larger, the ra     

A Hybrid Method for Nonlinear Least Squares Problems

A negative curvature method is applied to nonlinear least squares problems with indefinite Hessian approximation matrices. With the special structure of the method, a new switch is proposed to form a hybrid method. Numerical experiments show that this method is feasible and effective for zero-residual, small-residual and large-residual problems.     

A QR DECOMPOSITION BASED SOLVER FOR THE LEAST SQUARES PROBLEMS FROM THE MINIMAL RESIDUAL METHOD FOR THE SYLVESTER EQUATION

Based on the generalized minimal residual （GMRES） principle, Hu and Reichel proposed a minimal residual algorithm for the Sylvester equation. The algorithm requires the solution of a structured least squares problem. They form the normal equations of the least squares problem and then solve it by a direct solver, so it is susceptible to instability. In this paper, by exploiting the special structure of the least squares problem and working on the problem directly, a numerically stable QR decomposition based algorithm is presented for the problem. The new algorithm is more stable than the normal equations algorithm of Hu and Reichel. Numerical experiments are reported to confirm the superior stability of the new algorithm.     

一般Gauss—Markov模型下最小二乘估计是最佳估计的充要条件

In this paper, we discuss the decomposition of the space μ(X : V) and the invariance with respect to the choice of a generalized inverse of matrix X in the general Gauss-Markov model. In Theorem 1, we give necessary and sufficient conditions for the least squares estimator P_xy = BLUE(Xβ) under the general Gauss-Markov model M = {y,Xβ,σ²V}. In Theorem 2, we prove that P_xy= BLUE(Xβ) under model M and invariant with respect to the choice of a generalized inverse of matrix X are equivalent.     

Relationship Between the Stiffly Weighted Pseudoinverse and Multi-Level Constrained Rseudoinverse

It is known that for a given matrix A of rank r, and a set D of positive diagonal matrices, supw∈D‖（W^1/2A) W^1/2‖ = (miniσ (A^(i))^-1, in which (A^(i) is a submatrix of A formed with r = (rank(A)) rows of A, such that (A^(i) has full row rank r. In many practical applications this value is too large to be used. In this paper we consider the case that both A and W(∈D) are fixed with W severely stiff. We show that in this case the weighted pseudoinverse (W^1/2‖A) W^1/2‖ is close to a multilevel constrained weighted pseudoinverse therefore ‖(W^1/2A) W^1/‖2 is uniformly bounded.We also prove that in this case the solution set the stiffly weighted least squares problem is close to that of corresponding multi-level constrained least squares problem.     

A SUCCESSIVE LEAST SQUARES METHOD FOR STRUCTURED TOTAL LEAST SQUARES

A new method for Total Least Squares (TLS) problems is presented. It differs from previous approaches and is based on the solution of successive Least Squares problems.The method is quite suitable for Structured TLS (STLS) problems. We study mostly the case of Toeplitz matrices in this paper. The numerical tests illustrate that the method converges to the solution fast for Toeplitz STLS problems. Since the method is designed for general TLS problems, other structured problems can be treated similarly.     

Iterative Weighted Semiparametric Least Squares Estimation in Repeated Measurement Partially Linear Regression Models

Consider a repeated measurement partially linear regression model with an unknown vector parameter β, an unknown function g(.), and unknown heteroscedastic error variances. In order to improve the semiparametric generalized least squares estimator (SGLSE) of β, we propose an iterative weighted semiparametric least squares estimator (IWSLSE) and show that it improves upon the SGLSE in terms of asymptotic covariance matrix. An adaptive procedure is given to determine the number of iterations. We also show that when the number of replicates is less than or equal to two, the IWSLSE can not improve upon the SGLSE. These results are generalizations of those in [2] to the case of semiparametric regressions.     

Local linear regression for data with AR errors

In many statistical applications, data are collected over time, and they are likely correlated. In this paper, we investigate how to incorporate the correlation information into the local linear regression. Under the assumption that the error process is an auto-regressive process, a new estimation procedure is proposed for the nonparametric regression by using local linear regression method and the profile least squares techniques. We further propose the SCAD penalized profile least squares method to determine the order of auto-regressive process. Extensive Monte Carlo simulation studies are conducted to examine the finite sample performance of the proposed procedure, and to compare the performance of the proposed procedures with the existing one. From our empirical studies, the newly proposed procedures can dramatically improve the accuracy of naive local linear regression with working-independent error structure. We illustrate the proposed methodology by an analysis of real data set.     

ON THE SEPARABLE NONLINEAR LEAST SQUARES PROBLEMS

Separable nonlinear least squares problems are a special class of nonlinear least squares problems, where the objective functions are linear and nonlinear on different parts of variables. Such problems have broad applications in practice. Most existing algorithms for this kind of problems are derived from the variable projection method proposed by Golub and Pereyra, which utilizes the separability under a separate framework. However, the methods based on variable projection strategy would be invalid if there exist some constraints to the variables, as the real problems always do, even if the constraint is simply the ball constraint. We present a new algorithm which is based on a special approximation to the Hessian by noticing the fact that certain terms of the Hessian can be derived from the gradient. Our method maintains all the advantages of variable projection based methods, and moreover it can be combined with trust region methods easily and can be applied to general constrained separable nonlinear problems. Convergence analysis of our method is presented and numerical results are also reported.     

Generalized profile LSE in varying-coefficient partially linear models with measurement errors

This paper is concerned with the estimating problem of a semiparametric varying-coefficient partially linear errors-in-variables model Yi=Xτiβ+Zτiα(Ui)+εi , Wi=Xi+ξi,i=1, ··· , n. Due to measurement errors, the usual profile least square estimator of the parametric component, local polynomial estimator of the nonparametric component and profile least squares based estimator of the error variance are biased and inconsistent. By taking the measurement errors into account we propose a generalized profile least squares estimator for the parametric component and show it is consistent and asymptotically normal. Correspondingly, the consistent estimation of the nonparametric component and error variance are proposed as well. These results may be used to make asymptotically valid statistical inferences. Some simulation studies are conducted to illustrate the finite sample performance of these proposed estimations.     

TLS问题和LS问题解加权残量的比较

总体最小二乘(TLS)问题和最小二乘(LS)问题解残量的比较已有多篇文献予以探讨.本文对TLS问题和LS问题解的加权残量进行了比较. 导出了TLS解、改进的LS解及普通LS解加权残量之间的误差界. 从而进一步完善了已有的相关结果.     

关于TLS和LS解的扰动分析

1．引言本文采用卜］的记号．最小二乘（LS）和总体最小二乘（TLS）是科学计算中的两种重要方法．尤是TLS，近来已有多篇论文讨论［1－6，8－16］．奇异值分解（SVD）和CS分解是研究TLS和LS的重要工具．令ACm，BCm,C=(A,B)，A和C的SVD分别为（1．1）（1．2）其中P51为某个正整数，U，U，V，V均为西矩阵，UI，UI，VI，VI为上述矩阵的前P列，z1一山。g（。1，…，内），】2＝di。g（内十l，…，。小】1＝dl。g（61；…，站，】2二diag（4＋1；…，dk），。l三··2。120和dl三…三d。20分别为C和A的奇异值，Z＝mhfm．n十以…     

关于TLS问题

1.引 言考虑观测线性系统AX=B,（1.1a）其中A∈Cm×n,B∈Cm×d（本文通篇假设m≥n d）,分别是精确但不可观测的A0∈Cm×n,B0∈Cm×d的近似,即精确线性系统是A0X=B0.（1.1b）Golub和Van Loan于1980年提出的总体最小二乘问题（以下简称TLS问题）就是求解线性系统AX=B（1.2）     

关于广义线性最小二乘问题的扰动分析

1问题 在应用统计中，常用的参数估计方法之一是广义线性最小二乘min(Cx-y)~TW~+(Cx-y).(1.1)其中C为m×n矩阵，W为m×m对称半正定矩阵，上标+代表Moore-Penrose广义逆Paige~[1]注意到：从统计观点看，W一般未必可逆，且通常具有对称满秩分解W=BB~T，因而，把问题改述为下述形式更合适     

关于TLS的可解性及扰动分析

尽管有关总体最小二乘问题的研究工作是大量的，然而ＴＬＳ可解的充分必要条件一直没有得到。本文首先给出完整的可解性分析，然后建立了ＴＬＳ的扰动上界。     

Algebraic relations between the total least squares and least squares problems with more than one solution

Summary This paper completes our previous discussion on the total least squares (TLS) and the least squares (LS) problems for the linear systemAX=B which may contain more than one solution [12, 13], generalizes the work of Golub and Van Loan [1,2], Van Huffel [8], Van Huffel and Vandewalle [11]. The TLS problem is extended to the more general case. The sets of the solutions and the squared residuals for the TLS and LS problems are compared. The concept of the weighted squares residuals is extended and the difference between the TLS and the LS approaches is derived. The connection between the approximate subspaces and the perturbation theories are studied.It is proved that under moderate conditions, all the corresponding quantities for the solution sets of the TLS and the modified LS problems are close to each other, while the quantities for the solution set of the LS problem are close to the corresponding ones of a subset of that of the TLS problem.This work was financially supported by the Education Committee, People's Republic of China     

加权总体最小二乘问题的分析

总体最小二乘问题由Golub和Van Loan首先进行数学的分析,随后人们对于总体最小二乘问题的算法、解的各种形式、总体最小二乘解和最小二乘解的关系、总体最小二乘解的扰动理论以及数值试验作了大量的研究工作。近来,[10]中给出了总体最小二乘问题(TLS)较一般地讨论。另一方面,Golub和Van Loan研究了总体最小二乘问题的特殊均加权形式。本文试图在[10,11]的基础上讨论最一般的总体最小二     

Perturbation analysis for mixed least squares–total least squares problems

In many linear parameter estimation problems, one can use the mixed least squares–total least squares (MTLS) approach to solve them. This paper is devoted to the perturbation analysis of the MTLS problem. Firstly, we present the normwise, mixed, and componentwise condition numbers of the MTLS problem, and find that the normwise, mixed, and componentwise condition numbers of the TLS problem and the LS problem are unified in the ones of the MTLS problem. In the analysis of the first‐order perturbation, we first provide an upper bound based on the normwise condition number. In order to overcome the problems encountered in calculating the normwise condition number, we give an upper bound for computing more effectively for the MTLS problem. As two estimation techniques for solving the linear parameter estimation problems, interesting connections between their solutions, their residuals for the MTLS problem, and the LS problem are compared. Finally, some numerical experiments are performed to illustrate our results.