Estimating a reduced rank regression model for non-normal variables

A method using third order moments for estimating the regression coefficients as well as the latent state scores of the reduced-rank regression model when the latent variable(s) are non-normally distributed is presented in this paper. It is shown that the factor analysis type indeterminacy of the regression coefficient matrices is eliminated. A real life example of the proposed method is presented. Differences of this solution with the reduced-rank regression eigen solution are discussed.     

Robust model selection in regression

A robust version of Akaike's model selection procedure for regression models is introduced and its relationship with robust testing procedures is discussed.     

Dimensionality reduction and volume minimization—generalization of the determinant minimization criterion for reduced rank regression problems

In this article we propose a generalization of the determinant minimization criterion. The problem of minimizing the determinant of a matrix expression has implicit assumptions that the objective matrix is always nonsingular. In case of singular objective matrix the determinant would be zero and the minimization problem would be meaningless. To be able to handle all possible cases we generalize the determinant criterion to rank reduction and volume minimization of the objective matrix. The generalized minimization criterion is used to solve the following ordinary reduced rank regression problem:

min_rank(X)=kdet(B-XA)(B-XA)^T,     

Robust regression function estimation

A robust estimator of the regression function is proposed combining kernel methods as introduced for density estimation and robust location estimation techniques. Weak and strong consistency and asymptotic normality are shown under mild conditions on the kernel sequence. The asymptotic variance is a product from a factor depending only on the kernel and a factor similar to the asymptotic variance in robust estimation of location. The estimation is minimax robust in the sense of Huber (1964). Robust estimation of a location parameter. Ann. Math. Statist.33 73–101.     

Robust nonparametric regression estimation

In this paper we define a robust conditional location functional without requiring any moment condition. We apply the nonparametric proposals considered by C. Stone (Ann. Statist. 5 (1977), 595–645) to this functional equation in order to obtain strongly consistent, robust nonparametric estimates of the regression function. We give some examples by using nearest neighbor weights or weights based on kernel methods under no assumptions whatsoever on the probability measure of the vector (X,Y). We also derive strong convergence rates and the asymptotic distribution of the proposed estimates.     

Nonlinear Schwarz iterations with reduced rank extrapolation

Extrapolation methods can be a very effective technique used for accelerating the convergence of vector sequences. In this paper, these methods are used to accelerate the convergence of Schwarz iterative methods for nonlinear problems. A new implementation of the reduced-rank-extrapolation (RRE) method is introduced. Some convergence analysis is presented, and it is shown numerically that certain extrapolation methods can indeed be very effective in accelerating the convergence of Schwarz methods.     

On the weighted multivariate Wilcoxon rank regression estimate

Zhou (2010) introduced a multivariate Wilcoxon regression estimate which possesses some nice properties: computational ease, asymptotic normality and high efficiency. However, it is sensitive to the leverage points. To circumvent this problem, we propose a weighted multivariate Wilcoxon regression estimate. Under some regularity conditions, the asymptotic normality is established. We further study the robustness of the proposed estimate through the influence function. By properly choosing the weight functions, our results show that the corresponding estimate can have bounded influence function on both response and covariates.     

Kolmogorov-Smirnov two-sample test based on regression rank scores

We derive the two-sample Kolmogorov-Smirnov type test when a nuisance linear regression is present. The test is based on regression rank scores and provides a natural extension of the classical Kolmogorov-Smirnov test. Its asymptotic distributions under the hypothesis and the local alternatives coincide with those of the classical test. (Supplement to the special issue of Appl. Math. 53 (2008), No. 3) This work was supported by the Czech Science Foundation under Grant No. 201/05/H007 and by Research Project LC06024.     

Asymptotically most powerful rank tests for regression parameters in manova

Summary Srivastava [5] proposed a class of rank score tests for testing the hypothesis that β₁=⋯β _p =0 in the linear regression modely _i =β₁ x _1i +β₂ x _2i +⋯+β _p +x _pi +ɛ _i under weaker conditions than Hájek [2]. In this paper, under the same weak conditions, a class of rank score tests is proposed for testing β₁=⋯β _q =0 in the multivariate linear regression modely _i =β₁ x _1i +β₂ x _2i +⋯+β _p +x _pi +ɛ _i ,q≦p, where β _i ’s arek-vectors. The limiting distribution of the test statistic is shown to be central χ _qk ² underH and non-central χ _qk ² under a sequence of alternatives tending to the hypothesis at a suitable rate. Research supported by Canada Council and National Research Council of Canada.     

On estimation in some reduced rank extended growth curve models

The general multivariate analysis of variance model has been extensively studied in the statistical literature and successfully applied in many different fields for analyzing longitudinal data. In this article, we consider the extension of this model having two sets of regressors constituting a growth curve portion and a multivariate analysis of variance portion, respectively. Nowadays, the data collected in empirical studies have relatively complex structures though often demanding a parsimonious modeling. This can be achieved for example through imposing rank constraints on the regression coefficient matrices. The reduced rank regression structure also provides a theoretical interpretation in terms of latent variables. We derive likelihood based estimators for the mean parameters and covariance matrix in this type of models. A numerical example is provided to illustrate the obtained results.     

Robust estimation for nonparametric generalized regression

This paper focuses on nonparametric regression estimation for the parameters of a discrete or continuous distribution, such as the Poisson or Gamma distributions, when anomalous data are present. The proposal is a natural extension of robust methods developed in the setting of parametric generalized linear models. Robust estimators bounding either large values of the deviance or of the Pearson residuals are introduced and their asymptotic behaviour is derived. Through a Monte Carlo study, for the Poisson and Gamma distributions, the finite properties of the proposed procedures are investigated and their performance is compared with that of the classical ones. A resistant cross-validation method to choose the smoothing parameter is also considered.     

Robust algorithms for multiphase regression models

This paper proposes a robust procedure for solving multiphase regression problems that is efficient enough to deal with data contaminated by atypical observations due to measurement errors or those drawn from heavy-tailed distributions. Incorporating the expectation and maximization algorithm with the M-estimation technique, we simultaneously derive robust estimates of the change-points and regression parameters, yet as the proposed method is still not resistant to high leverage outliers we further suggest a modified version by first moderately trimming those outliers and then implementing the new procedure for the trimmed data. This study sets up two robust algorithms using the Huber loss function and Tukey's biweight function to respectively replace the least squares criterion in the normality-based expectation and maximization algorithm, illustrating the effectiveness and superiority of the proposed algorithms through extensive simulations and sensitivity analyses. Experimental results show the ability of the proposed method to withstand outliers and heavy-tailed distributions. Moreover, as resistance to high leverage outliers is particularly important due to their devastating effect on fitting a regression model to data, various real-world applications show the practicability of this approach.     

Minimal polynomial and reduced rank extrapolation methods are related

Minimal Polynomial Extrapolation (MPE) and Reduced Rank Extrapolation (RRE) are two polynomial methods used for accelerating the convergence of sequences of vectors {x _m }. They are applied successfully in conjunction with fixed-point iterative schemes in the solution of large and sparse systems of linear and nonlinear equations in different disciplines of science and engineering. Both methods produce approximations s _k to the limit or antilimit of {x _m } that are of the form \(\boldsymbol {s}_{k}={\sum }^{k}_{i=0}\gamma _{i}\boldsymbol {x}_{i}\) with \({\sum }^{k}_{i=0}\gamma _{i}=1\), for some scalars γ _i . The way the two methods are derived suggests that they might, somehow, be related to each other; this has not been explored so far, however. In this work, we tackle this issue and show that the vectors \(\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}\) and \(\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}\) produced by the two methods are related in more than one way, and independently of the way the x _m are generated. One of our results states that RRE stagnates, in the sense that \(\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}=\boldsymbol {s}_{k-1}^{\textit {{\tiny {RRE}}}}\), if and only if \(\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}\) does not exist. Another result states that, when \(\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}\) exists, there holds

$$\mu_{k}\boldsymbol{s}_{k}^{\textit{{\tiny{RRE}}}}=\mu_{k-1}\boldsymbol{s}_{k-1}^{\textit{{\tiny{RRE}}}}+ \nu_{k}\boldsymbol{s}_{k}^{\textit{{\tiny{MPE}}}}\quad \text{with}\quad \mu_{k}=\mu_{k-1}+\nu_{k}, $$

for some positive scalars μ _k , μ _k?1, and ν _k that depend only on \(\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}\), \(\boldsymbol {s}_{k-1}^{\textit {{\tiny {RRE}}}}\), and \(\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}\), respectively. Our results are valid when MPE and RRE are defined in any weighted inner product and the norm induced by it. They also contain as special cases the known results pertaining to the connection between the method of Arnoldi and the method of generalized minimal residuals, two important Krylov subspace methods for solving nonsingular linear systems.

     

Robust weighted orthogonal regression in the errors-in-variables model

This paper focuses on robust estimation in the structural errors-in-variables (EV) model. A new class of robust estimators, called weighted orthogonal regression estimators, is introduced. Robust estimators of the parameters of the EV model are simply derived from robust estimators of multivariate location and scatter such as the M-estimators, the S-estimators and the MCD estimator. The influence functions of the proposed estimators are calculated and shown to be bounded. Moreover, we derive the asymptotic distributions of the estimators and illustrate the results on simulated examples and on a real-data set.     

Robust errors-in-variables linear regression via Laplace distribution

Robust estimation procedures for linear and mixture linear errors-in-variables regression models are proposed based on the relationship between the least absolute deviation criterion and maximum likelihood estimation in a Laplace distribution. The finite sample performance of the proposed procedures is evaluated by simulation studies.     

Robust kernel ridge regression based on M-estimation

Ridge regression (RR) and kernel ridge regression (KRR) are important tools to avoid the effects of multicollinearity. However, the predictions of RR and KRR become inappropriate for use in regression models when data are contaminated by outliers. In this paper, we propose an algorithm to obtain a nonlinear robust prediction without specifying a nonlinear model in advance. We combine M-estimation and kernel ridge regression to obtain the nonlinear prediction. Then, we compare the proposed method with some other methods.     

On a class of rank order tests for the identity of two multiple regression surfaces

Summary That the central theorem is valid for (forward) martingales is a result with a long history, beginning with Lévy [6], the most refined and recent results being due to Billingsley [3].Not altogether surprisingly, an analogous result holds for backwards martingales, and the proof, which parallels closely that of Billingsley, occupies Section 1. Examples are given in Section 2.     

Robust model selection for a semimartingale continuous time regression from discrete data

The paper considers the problem of estimating a periodic function in a continuous time regression model observed under a general semimartingale noise with an unknown distribution in the case when continuous observation cannot be provided and only discrete time measurements are available. Two specific types of noises are studied in detail: a non-Gaussian Ornstein–Uhlenbeck process and a time-varying linear combination of a Brownian motion and compound Poisson process. We develop new analytical tools to treat the adaptive estimation problems from discrete data. A lower bound for the frequency sampling, needed for the efficiency of the procedure constructed by discrete observations, has been found. Sharp non-asymptotic oracle inequalities for the robust quadratic risk have been derived. New convergence rates for the efficient procedures have been obtained. An example of the regression with a martingale noise exhibits that the minimax robust convergence rate may be both higher or lower as compared with the minimax rate for the “white noise” model. The results of Monte-Carlo simulations are given.     

Privacy preserving linear regression modeling of distributed databases

Statistical analysis is one of the important tools in data mining field. Little work has been conducted to investigate how statistical analysis could be performed when dataset are distributed among a number of data owners. Due to confidentiality or other proprietary reasons, data owners are reluctant to share data with others, while they wish to perform statistical analysis cooperatively. We address the important tradeoff between privacy and global statistical analysis such as linear regression, and present a privacy preserving linear regression model based on fully homomorphic encryption scheme.