Detection of outliers in weighted least squares regression

In multiple linear regression model, we have presupposed assumptions (independence, normality, variance homogeneity and so on) on error term. When case weights are given because of variance heterogeneity, we can estimate efficiently regression parameter using weighted least squares estimator. Unfortunately, this estimator is sensitive to outliers like ordinary least squares estimator. Thus, in this paper, we proposed some statistics for detection of outliers in weighted least squares regression.     

Second-order nonlinear least squares estimation

The ordinary least squares estimation is based on minimization of the squared distance of the response variable to its conditional mean given the predictor variable. We extend this method by including in the criterion function the distance of the squared response variable to its second conditional moment. It is shown that this “second-order” least squares estimator is asymptotically more efficient than the ordinary least squares estimator if the third moment of the random error is nonzero, and both estimators have the same asymptotic covariance matrix if the error distribution is symmetric. Simulation studies show that the variance reduction of the new estimator can be as high as 50% for sample sizes lower than 100. As a by-product, the joint asymptotic covariance matrix of the ordinary least squares estimators for the regression parameter and for the random error variance is also derived, which is only available in the literature for very special cases, e.g. that random error has a normal distribution. The results apply to both linear and nonlinear regression models, where the random error distributions are not necessarily known.     

Bayesian bandwidth estimation for a semi-functional partial linear regression model with unknown error density

In the context of semi-functional partial linear regression model, we study the problem of error density estimation. The unknown error density is approximated by a mixture of Gaussian densities with means being the individual residuals, and variance a constant parameter. This mixture error density has a form of a kernel density estimator of residuals, where the regression function, consisting of parametric and nonparametric components, is estimated by the ordinary least squares and functional Nadaraya–Watson estimators. The estimation accuracy of the ordinary least squares and functional Nadaraya–Watson estimators jointly depends on the same bandwidth parameter. A Bayesian approach is proposed to simultaneously estimate the bandwidths in the kernel-form error density and in the regression function. Under the kernel-form error density, we derive a kernel likelihood and posterior for the bandwidth parameters. For estimating the regression function and error density, a series of simulation studies show that the Bayesian approach yields better accuracy than the benchmark functional cross validation. Illustrated by a spectroscopy data set, we found that the Bayesian approach gives better point forecast accuracy of the regression function than the functional cross validation, and it is capable of producing prediction intervals nonparametrically.     

r-k Class estimation in regression model with concomitant variables

We treat with the r-k class estimation in a regression model, which includes the ordinary least squares estimator, the ordinary ridge regression estimator and the principal component regression estimator as special cases of the r-k class estimator. Many papers compared total mean square error of these estimators. Sarkar (1989, Ann. Inst. Statist. Math., 41, 717–724) asserts that the results of this comparison are still valid in a misspecified linear model. We point out some confusions of Sarkar and show additional conditions under which his assertion holds.     

James-Stein estimators for time series regression models

The least squares (LS) estimator seems the natural estimator of the coefficients of a Gaussian linear regression model. However, if the dimension of the vector of coefficients is greater than 2 and the residuals are independent and identically distributed, this conventional estimator is not admissible. James and Stein [Estimation with quadratic loss, Proceedings of the Fourth Berkely Symposium vol. 1, 1961, pp. 361-379] proposed a shrinkage estimator (James-Stein estimator) which improves the least squares estimator with respect to the mean squares error loss function. In this paper, we investigate the mean squares error of the James-Stein (JS) estimator for the regression coefficients when the residuals are generated from a Gaussian stationary process. Then, sufficient conditions for the JS to improve the LS are given. It is important to know the influence of the dependence on the JS. Also numerical studies illuminate some interesting features of the improvement. The results have potential applications to economics, engineering, and natural sciences.     

Comparisons among some estimators in misspecified linear models with multicollinearity

In this paper we deal with comparisons among several estimators available in situations of multicollinearity (e.g., the r-k class estimator proposed by Baye and Parker, the ordinary ridge regression (ORR) estimator, the principal components regression (PCR) estimator and also the ordinary least squares (OLS) estimator) for a misspecified linear model where misspecification is due to omission of some relevant explanatory variables. These comparisons are made in terms of the mean square error (mse) of the estimators of regression coefficients as well as of the predictor of the conditional mean of the dependent variable. It is found that under the same conditions as in the true model, the superiority of the r-k class estimator over the ORR, PCR and OLS estimators and those of the ORR and PCR estimators over the OLS estimator remain unchanged in the misspecified model. Only in the case of comparison between the ORR and PCR estimators, no definite conclusion regarding the mse dominance of one over the other in the misspecified model can be drawn.     

回归系数的广义岭型主相关估计及其优良性

In this paper, we propose a new biased estimator of the regression parameters, the generalized ridge and principal correlation estimator. We present its some properties and prove that it is superior to LSE （least squares estimator）, principal correlation estimator, ridge and principal correlation estimator under MSE （mean squares error） and PMC （Pitman closeness） criterion, respectively.     

PLS regression: A directional signal-to-noise ratio approach

We present a new approach to univariate partial least squares regression (PLSR) based on directional signal-to-noise ratios (SNRs). We show how PLSR, unlike principal components regression, takes into account the actual value and not only the variance of the ordinary least squares (OLS) estimator. We find an orthogonal sequence of directions associated with decreasing SNR. Then, we state partial least squares estimators as least squares estimators constrained to be null on the last directions. We also give another procedure that shows how PLSR rebuilds the OLS estimator iteratively by seeking at each step the direction with the largest difference of signals over the noise. The latter approach does not involve any arbitrary scale or orthogonality constraints.     

包含多余回归变量的错误指定模型的随机约束Liu估计

对由于包含多余回归自变量而导致的错误指定线性回归模型,本文导出了回归系数的最小二乘估计,普通混合估计以及随机约束Liu估计,并在均方误差矩阵准则下对这三个估计的优良性进行了比较,给出了随机约束Liu估计优于最小二乘估计和普通混合估计的充要条件.此外,对它们所对应的经典预测值的优良性也进行了讨论.     

受约束回归模型参数的统一几乎无偏估计

考虑实际回归问题中存在更多受约束条件的情况,提出了带约束的统一几乎无偏估计类,统一了常见的具有线性约束的回归模型的几乎无偏估计,进一步的研究给出了在均方误差和均方误差矩阵意义下,带约束的统一几乎无偏估计优于一般带约束的最小二乘估计的充分条件和椭球范围.     

Sequences of bias-adjusted covariance matrix estimators under heteroskedasticity of unknown form

The linear regression model is commonly used by practitioners to model the relationship between the variable of interest and a set of explanatory variables. The assumption that all error variances are the same, known as homoskedasticity, is oftentimes violated when cross sectional data are used. Consistent standard errors for the ordinary least squares estimators of the regression parameters can be computed following the approach proposed by White (Econometrica 48:817–838, 1980). Such standard errors, however, are considerably biased in samples of typical sizes. An improved covariance matrix estimator was proposed by Qian and Wang (J Stat Comput Simul 70:161–174, 2001). In this paper, we improve upon the Qian–Wang estimator by defining a sequence of bias-adjusted estimators with increasing accuracy. The numerical results show that the Qian–Wang estimator is typically much less biased than the estimator proposed by Halbert White and that our correction to the former can be quite effective in small samples. Finally, we show that the Qian–Wang estimator can be generalized into a broad class of heteroskedasticity-consistent covariance matrix estimators, and our results can be easily extended to such a class of estimators.     

加权平衡损失函数下回归系数的最佳线性无偏估计

这篇文章我们研究了回归系数的最佳线性无偏估计. 在加权平衡损失函数下, 我们得到了回归系数的最佳线性无偏估计. 同时提出了度量最佳线性无偏估计和最小二乘估计的相对效率. 并且我们给出了它们的上下界.     

The existence theorem in general ridge regression

Hoerl and Kennard (1970) state that, like the ordinary ridge estimator, the general ridge estimator is also better than the least squares estimator relative to a mean square error. The proof of this result is given in this note.     

A Generalization of Rao's Covariance Structure with Applications to Several Linear Models

This paper presents a generalization of Rao's covariance structure. In a general linear regression model, we classify the error covariance structure into several categories and investigate the efficiency of the ordinary least squares estimator (OLSE) relative to the Gauss–Markov estimator (GME). The classification criterion considered here is the rank of the covariance matrix of the difference between the OLSE and the GME. Hence our classification includes Rao's covariance structure. The results are applied to models with special structures: a general multivariate analysis of variance model, a seemingly unrelated regression model, and a serial correlation model.     

Local Weighted Composite Quantile Estimating for Varying Coefficient Models

A generalization of classical linear models is varying coefficient models, which offer a flexible approach to modeling nonlinearity between covariates. A method of local weighted composite quantile regression is suggested to estimate the coefficient functions. The local Bahadur representation of the local estimator is derived and the asymptotic normality of the resulting estimator is established. Comparing to the local least squares estimator, the asymptotic relative efficiency is examined for the local weighted composite quantile estimator. Both theoretical analysis and numerical simulations reveal that the local weighted composite quantile estimator can obtain more efficient than the local least squares estimator for various non-normal errors. In the normal error case, the local weighted composite quantile estimator is almost as efficient as the local least squares estimator. Monte Carlo results are consistent with our theoretical findings. An empirical application demonstrates the potential of the proposed method.     

An implicit least squares algorithm for nonlinear rational model parameter estimation

In this study a new insight into least squares regression is identified and immediately applied to estimating the parameters of nonlinear rational models. From the beginning the ordinary explicit expression for linear in the parameters model is expanded into an implicit expression. Then a generic algorithm in terms of least squares error is developed for the model parameter estimation. It has been proved that a nonlinear rational model can be expressed as an implicit linear in the parameters model, therefore, the developed algorithm can be comfortably revised for estimating the parameters of the rational models. The major advancement of the generic algorithm is its conciseness and efficiency in dealing with the parameter estimation problems associated with nonlinear in the parameters models. Further, the algorithm can be used to deal with those regression terms which are subject to noise. The algorithm is reduced to an ordinary least square algorithm in the case of linear or linear in the parameters models. Three simulated examples plus a realistic case study are used to test and illustrate the performance of the algorithm.     

Superiority of empirical Bayes estimation of error variance in linear model

In this paper, the Bayes estimator of the error variance is derived in a linear regression model, and the parametric empirical Bayes estimator (PEBE) is constructed. The superiority of the PEBE over the least squares estimator (LSE) is investigated under the mean square error (MSE) criterion. Finally, some simulation results for the PEBE are obtained.     

Permutation tests using least distance estimator in the multivariate regression model

This paper proposes two permutation tests based on the least distance estimator in a multivariate regression model. One is a type of t test statistic using the bootstrap method, and the other is a type of F test statistic using the sum of distances between observed and predicted values under the full and reduced models. We conducted a simulation study to compare the power of the proposed permutation tests with that of the parametric tests based on the least squares estimator for three types of hypotheses in several error distributions. The results indicate that the power of the proposed permutation tests is greater than that of the parametric tests when the error distribution is skewed like the Wishart distribution, has a heavy tail like the Cauchy distribution, or has outliers.     

带有结构变化的线性模型中参数估计的一些结果

本文在一些纯量损失和矩阵损失下研究带有结构变化的正态线性模型中参数的估计问题．分别给出 了存在回归系数的一致最小风险无偏（ＵＭＲＵ）估计和一致最小风险同变（ＵＭＲＥ）估计的充要条件, 证明了不存在误差方差在仿射变换群下的ＵＭＲＥ估计．导出了回归系数的最小二乘估计的可容许性 和极小极大性．