Iteratively reweighted least squares: A comparison of several single step algorithms for linear models

Various algorithms for the computation of linear M-estimates for convex as well as for nonconvex criterion functions are considered. In the convex case a scaling parameters is estimated together with the parameter vectorx as a solution of some nonlinear minimization problem. Several single step algorithms for the solution of this minimization problem are investigated. On the base of known procedures new methods are developed, which use locally majorizing approximations of the objective function, exact line search and efficient updating techniques. These algorithms also turn out to be efficient for the computation of M-estimates which are defined by nonconvex criterion functions and a constant scaling parameter. The numerical efficiency of the different procedures is compared by means of 30 simulated test problems.Part of this work was performed while the author was in residence at the Institut für Angewandte Mathematik, Universität Hamburg, Bundesstr. 55, D(West)-2000 Hamburg 13, Germany.     

Finite alogorithms for robust linear regression

In this paper Hubert's M-estimator for robust linear regression is analyzed. Newton type methods for solution of the problem are defined and analyzed, and finite convergence is proved. Numerical experiments with a large number of test problems demonstrate efficiency and indicate that this kind of approach may be useful also in solving thel ₁ problem.     

On Newton's method for Huber's robust M-estimation problems in linear regression

The Newton method of Madsen and Nielsen (1990) for computing Huber's robust M-estimate in linear regression is considered. The original method was proved to converge finitely for full rank problems under some additional restrictions on the choice of the search direction and the step length in some degenerate cases. It was later observed that these requirements can be relaxed in a practical implementation while preserving the effectiveness and even improving the efficiency of the method. In the present paper these enhancements to the original algorthm are studied and the finite termination property of the algorithm is proved without any assumptions on the M-estimation problems. Research supported by NATO Collaborative Research Grant CRG-94-0609.     

Efficiency of a Liu-type estimator in semiparametric regression models

In this paper we consider the semiparametric regression model, y=Xβ+f+ε. Recently, Hu [11] proposed ridge regression estimator in a semiparametric regression model. We introduce a Liu-type (combined ridge-Stein) estimator (LTE) in a semiparametric regression model. Firstly, Liu-type estimators of both β and f are attained without a restrained design matrix. Secondly, the LTE estimator of β is compared with the two-step estimator in terms of the mean square error. We describe the almost unbiased Liu-type estimator in semiparametric regression models. The almost unbiased Liu-type estimator is compared with the Liu-type estimator in terms of the mean squared error matrix. A numerical example is provided to show the performance of the estimators.     

Best linear unbiased prediction for linear combinations in general mixed linear models

The general mixed linear model can be written as . In this paper, we mainly deal with two problems. Firstly, the problem of predicting a general linear combination of fixed effects and realized values of random effects in a general mixed linear model is considered and an explicit representation of the best linear unbiased predictor (BLUP) is derived. In addition, we apply the resulting conclusion to several special models and offer an alternative to characterization of BLUP. Secondly, we recall the notion of linear sufficiency and consider it as regards the BLUP problem and characterize it in several different ways. Further, we study the concepts of linear sufficiency, linear minimal sufficiency and linear completeness, and give relations among them. Finally, four concluding remarks are given.     

A new algorithm for special Vandermonde systems

In this paper a new algorithm for solving special Vandermonde systems is presented, useful when then points defining the matrix are thek th roots ofm complex numbers (n=km); if they are real and positive andn=2m, the usual case of real points symmetrically ranged around zero is obtained. The algorithm is based on an inverse matrix formulation by means of the Kronecker product and is particularly suitable for parallel implementation. Its computational complexity is analysed and compared both in the sequential and parallel formulation.     

Bayes estimation in linear models: A coordinate-free approach

The unified theory of Bayes estimation in linear models is presented, using a coordinate-free approach. The results are applied to the problem of linear and quadratic estimation in linear regression model.     

Conditional information criteria for selecting variables in linear mixed models

In this paper, we consider the problem of selecting the variables of the fixed effects in the linear mixed models where the random effects are present and the observation vectors have been obtained from many clusters. As the variable selection procedure, here we use the Akaike Information Criterion, AIC. In the context of the mixed linear models, two kinds of AIC have been proposed: marginal AIC and conditional AIC. In this paper, we derive three versions of conditional AIC depending upon different estimators of the regression coefficients and the random effects. Through the simulation studies, it is shown that the proposed conditional AIC’s are superior to the marginal and conditional AIC’s proposed in the literature in the sense of selecting the true model. Finally, the results are extended to the case when the random effects in all the clusters are of the same dimension but have a common unknown covariance matrix.     

Minimax least squares and quasiminimax estimation in linear models with missing values

We consider the problem of estimating the parameter vector in the linear model when observations on the independent variables are partially missing or incorrect. New estimators are developed, which systematically combine prior information with the incomplete data. We compare these methods with the alternative strategy of deleting incomplete observations.Support by Deutsche Forschungsgemeinschaft, Grant No. 284/1-2 is gratefully acknowledged.     

Linear M-estimation with bounded variables

A subproblem in the trust region algorithm for non-linear M-estimation by Ekblom and Madsen is to find the restricted step. It is found by calculating the M-estimator of the linearized model, subject to anL ₂-norm bound on the variables. In this paper it is shown that this subproblem can be solved by applying Hebden-iterations to the minimizer of the Lagrangian function. The new method is compared with an Augmented Lagrange implementation.     

Theory and inference for regression models with missing responses and covariates

In this paper, we carry out an in-depth theoretical investigation for inference with missing response and covariate data for general regression models. We assume that the missing data are missing at random (MAR) or missing completely at random (MCAR) throughout. Previous theoretical investigations in the literature have focused only on missing covariates or missing responses, but not both. Here, we consider theoretical properties of the estimates under three different estimation settings: complete case (CC) analysis, a complete response (CR) analysis that involves an analysis of those subjects with only completely observed responses, and the all case (AC) analysis, which is an analysis based on all of the cases. Under each scenario, we derive general expressions for the likelihood and devise estimation schemes based on the EM algorithm. We carry out a theoretical investigation of the three estimation methods in the normal linear model and analytically characterize the loss of information for each method, as well as derive and compare the asymptotic variances for each method assuming the missing data are MAR or MCAR. In addition, a theoretical investigation of bias for the CC method is also carried out. A simulation study and real dataset are given to illustrate the methodology.     

Estimation in generalized linear models for functional data via penalized likelihood

We analyze in a regression setting the link between a scalar response and a functional predictor by means of a Functional Generalized Linear Model. We first give a theoretical framework and then discuss identifiability of the model. The functional coefficient of the model is estimated via penalized likelihood with spline approximation. The L² rate of convergence of this estimator is given under smoothness assumption on the functional coefficient. Heuristic arguments show how these rates may be improved for some particular frameworks.     

A family of estimators for multivariate kurtosis in a nonnormal linear regression model

In this paper, we propose a new estimator for a kurtosis in a multivariate nonnormal linear regression model. Usually, an estimator is constructed from an arithmetic mean of the second power of the squared sample Mahalanobis distances between observations and their estimated values. The estimator gives an underestimation and has a large bias, even if the sample size is not small. We replace this squared distance with a transformed squared norm of the Studentized residual using a monotonic increasing function. Our proposed estimator is defined by an arithmetic mean of the second power of these squared transformed squared norms with a correction term and a tuning parameter. The correction term adjusts our estimator to an unbiased estimator under normality, and the tuning parameter controls the sizes of the squared norms of the residuals. The family of our estimators includes estimators based on ordinary least squares and predicted residuals. We verify that the bias of our new estimator is smaller than usual by constructing numerical experiments.     

Thresholding projection estimators in functional linear models

We consider the problem of estimating the regression function in functional linear regression models by proposing a new type of projection estimators which combine dimension reduction and thresholding. The introduction of a threshold rule allows us to get consistency under broad assumptions as well as minimax rates of convergence under additional regularity hypotheses. We also consider the particular case of Sobolev spaces generated by the trigonometric basis which permits us to get easily mean squared error of prediction as well as estimators of the derivatives of the regression function. We prove that these estimators are minimax and rates of convergence are given for some particular cases.     

Empirical likelihood inference in partially linear single-index models for longitudinal data

The empirical likelihood method is especially useful for constructing confidence intervals or regions of parameters of interest. Yet, the technique cannot be directly applied to partially linear single-index models for longitudinal data due to the within-subject correlation. In this paper, a bias-corrected block empirical likelihood (BCBEL) method is suggested to study the models by accounting for the within-subject correlation. BCBEL shares some desired features: unlike any normal approximation based method for confidence region, the estimation of parameters with the iterative algorithm is avoided and a consistent estimator of the asymptotic covariance matrix is not needed. Because of bias correction, the BCBEL ratio is asymptotically chi-squared, and hence it can be directly used to construct confidence regions of the parameters without any extra Monte Carlo approximation that is needed when bias correction is not applied. The proposed method can naturally be applied to deal with pure single-index models and partially linear models for longitudinal data. Some simulation studies are carried out and an example in epidemiology is given for illustration.     

Risk and Pitman closeness properties of feasible generalized double k-class estimators in linear regression models with non-spherical disturbances under balanced loss function

In this article, a family of feasible generalized double k-class estimator in a linear regression model with non-spherical disturbances is considered. The performance of this estimator is judged with feasible generalized least-squares and feasible generalized Stein-rule estimators under balanced loss function using the criteria of quadratic risk and general Pitman closeness. A Monte-Carlo study investigates the finite sample properties of several estimators arising from the family of feasible double k-class estimators.     

A decomposition of a linear model

In this paper we consider a general linear model in a continuous time. We propose a decomposition of the process which helps us to understand the structure of the model. Moreover, the sufficiency of the BLUE estimator of the expectation of the process can be characterized in terms of the Gaussian character of a component of the decomposition.     

Spectral methods in linear minimax estimation

We consider the minimax-linear estimator in a linear regression model with circular constraints. Two necessary and sufficient conditions for the optimality of an estimator, the socalled left spectral equation and the right spectral equation (Girko spectral equation), are derived. For the special case of a simple maximal eigenvalue and a single eigenspace explicit estimation formulas are derived. These formulas also show some of the shortcomings of the minimax-linear estimator (MILE). Finally, the relation with Bayesian analysis and the Hoffmann-Läuter estimator is outlined.     

Some equalities for estimations of variance components in a general linear model and its restricted and transformed models

For the unknown positive parameter σ² in a general linear model , the two commonly used estimations are the simple estimator (SE) and the minimum norm quadratic unbiased estimator (MINQUE). In this paper, we derive necessary and sufficient conditions for the equivalence of the SEs and MINQUEs of the variance component σ² in the original model ?, the restricted model , the transformed model , and the misspecified model .