Who's afraid of reduced-rank parameterizations of multivariate models? Theory and example

Reduced-rank restrictions can add useful parsimony to coefficient matrices of multivariate models, but their use is limited by the daunting complexity of the methods and their theory. The present work takes the easy road, focusing on unifying themes and simplified methods. For Gaussian and non-Gaussian (GLM, GAM, mixed normal, etc.) multivariate models, the present work gives a unified, explicit theory for the general asymptotic (normal) distribution of maximum likelihood estimators (MLE). MLE can be complex and computationally hard, but we show a strong asymptotic equivalence between MLE and a relatively simple minimum (Mahalanobis) distance estimator. The latter method yields particularly simple tests of rank, and we describe its asymptotic behavior in detail. We also examine the method's performance in simulation and via analytical and empirical examples.     

Estimators of scale parameters in linear regression

This note discusses the asymptotic distribution of two scale and location invariant estimators of two scale parameters in the multiple linear regression model. Both of these estimators need an initial estimator of the regression parameter vector. The asymptotic distribution of one of these estimators does not depend on this initial estimator. Both of these estimators are useful in the computation of scale and translation invariant adaptive estimators and M-estimators of the regression parameter vector.     

Statistical inference in a panel data semiparametric regression model with serially correlated errors

We consider a panel data semiparametric partially linear regression model with an unknown vector β of regression coefficients, an unknown nonparametric function g(·) for nonlinear component, and unobservable serially correlated errors. The correlated errors are modeled by a vector autoregressive process which involves a constant intraclass correlation. Applying the pilot estimators of β and g(·), we construct estimators of the autoregressive coefficients, the intraclass correlation and the error variance, and investigate their asymptotic properties. Fitting the error structure results in a new semiparametric two-step estimator of β, which is shown to be asymptotically more efficient than the usual semiparametric least squares estimator in terms of asymptotic covariance matrix. Asymptotic normality of this new estimator is established, and a consistent estimator of its asymptotic covariance matrix is presented. Furthermore, a corresponding estimator of g(·) is also provided. These results can be used to make asymptotically efficient statistical inference. Some simulation studies are conducted to illustrate the finite sample performances of these proposed estimators.     

MOMENT ESTIMATION FOR MULTIVARIATE EXTREME VALUE DISTRIBUTION

Moment estimation for multivariate extreme value distribution is described in this paper. Asymptotic covariance matrix of the estimators is given. The relative efficiencies of moment estimators as compared with the maximum likelihood and the stepwise estimators are computed. We show that when there is strong dependence between the variates, the generalized variance of moment estimators is much lower than the stepwise estimators. It becomes more obvious when the dimension increases.     

Moderate deviations of generalized method of moments and empirical likelihood estimators

This paper studies moderate deviation behaviors of the generalized method of moments and generalized empirical likelihood estimators for generalized estimating equations, where the number of equations can be larger than the number of unknown parameters. We consider two cases for the data generating probability measure: the model assumption and local contaminations or deviations from the model assumption. For both cases, we characterize the first-order terms of the moderate deviation error probabilities of these estimators. Our moderate deviation analysis complements the existing literature of the local asymptotic analysis and misspecification analysis for estimating equations, and is useful to evaluate power and robust properties of statistical tests for estimating equations which typically involve some estimators for nuisance parameters.     

Asymptotic properties of conditional maximum likelihood estimator in a certain exponential model

The conditional maximum likelihood estimator is suggested as an alternative to the maximum likelihood estimator and is favorable for an estimator of a dispersion parameter in the normal distribution, the inverse-Gaussian distribution, and so on. However, it is not clear whether the conditional maximum likelihood estimator is asymptotically efficient in general. Consider the case where it is asymptotically efficient and its asymptotic covariance depends only on an objective parameter in an exponential model. This remand implies that the exponential model possesses a certain parallel foliation. In this situation, this paper investigates asymptotic properties of the conditional maximum likelihood estimator and compares the conditional maximum likelihood estimator with the maximum likelihood estimator. We see that the bias of the former is more robust than that of the latter and that two estimators are very close, especially in the sense of bias-corrected version. The mean Pythagorean relation is also discussed.     

Asymptotics of Bayesian median loss estimation

We establish the consistency, asymptotic normality, and efficiency for estimators derived by minimizing the median of a loss function in a Bayesian context. We contrast this procedure with the behavior of two Frequentist procedures, the least median of squares (LMS) and the least trimmed squares (LTS) estimators, in regression problems. The LMS estimator is the Frequentist version of our estimator, and the LTS estimator approaches a median-based estimator as the trimming approaches 50% on each side. We argue that the Bayesian median-based method is a good tradeoff between the two Frequentist estimators.     

On weighting of bivariate margins in pairwise likelihood

Composite and pairwise likelihood methods have recently been increasingly used. For clustered data with varying cluster sizes, we study asymptotic relative efficiencies for various weighted pairwise likelihoods, with weight being a function of cluster size. For longitudinal data, we also study weighted pairwise likelihoods with weights that can depend on lag. Good choice of weights are needed to avoid the undesirable behavior of estimators with low efficiency. Some analytic results are obtained using the multivariate normal distribution. For clustered data, a practically good choice of weight is obtained after study of relative efficiencies for an exchangeable multivariate normal model; they are different from weights that had previously been suggested. For longitudinal data, there are advantages to only include bivariate margins of adjacent or nearly adjacent pairs in the weighted pairwise likelihood.     

Asymptotic expansion of Bayes estimators for small diffusions

Summary Using the Malliavin calculus we derived asymptotic expansion of the distributions of the Bayes estimators for small diffusions. The second order efficiency of the Bayes estimator is proved.     

Parameter estimation under ambiguity and contamination with the spurious model

Recently, we proposed variants as a statistical model for treating ambiguity. If data are extracted from an object with a machine then it might not be able to give a unique safe answer due to ambiguity about the correct interpretation of the object. On the other hand, the machine is often able to produce a finite number of alternative feature sets (of the same object) that contain the desired one. We call these feature sets variants of the object. Data sets that contain variants may be analyzed by means of statistical methods and all chapters of multivariate analysis can be seen in the light of variants. In this communication, we focus on point estimation in the presence of variants and outliers. Besides robust parameter estimation, this task requires also selecting the regular objects and their valid feature sets (regular variants). We determine the mixed MAP-ML estimator for a model with spurious variants and outliers as well as estimators based on the integrated likelihood. We also prove asymptotic results which show that the estimators are nearly consistent.The problem of variant selection turns out to be computationally hard; therefore, we also design algorithms for efficient approximation. We finally demonstrate their efficacy with a simulated data set and a real data set from genetics.     

Strong consistency of least-squares estimation in linear regression models with vague concepts

Linear regression models with vague concepts extend the classical single equation linear regression models by admitting observations in form of fuzzy subsets instead of real numbers. They have recently been introduced [cf. Krätschmer, Induktive statistik auf basis unscharfer meßkonzepte am beispiel linearer regressionsmodelle, Unpublished Habilitation Monograph, Faculty of Law and Economics of the University of Saarland, Saarbrücken, 2001] to improve the empirical meaningfulness of the relationship between the involved items by a more sensitive attention to the problems of data measurement, in particular the fundamental problem of adequacy. The parameters of such models are still real numbers, and a method of estimation can be applied which extends directly the ordinary least-squares method. This paper deals with some first asymptotic properties of estimators obtained by the method. Firstly, strong consistency will be shown, and secondly, the convergence rate will be investigated. The later result will be the starting point for a future study which will calculate the limit distributions of the estimators.     

A robust and efficient adaptive reweighted estimator of multivariate location and scatter

This article proposes a reweighted estimator of multivariate location and scatter, with weights adaptively computed from the data. Its breakdown point and asymptotic behavior under elliptical distributions are established. This adaptive estimator is able to attain simultaneously the maximum possible breakdown point for affine equivariant estimators and full asymptotic efficiency at the multivariate normal distribution. For the special case of hard-rejection weights and the MCD as initial estimator, it is shown to be more efficient than its non-adaptive counterpart for a broad range of heavy-tailed elliptical distributions. A Monte Carlo study shows that the adaptive estimator is as robust as its non-adaptive relative for several types of bias-inducing contaminations, while it is remarkably more efficient under normality for sample sizes as small as 200.     

Asymptotic properties of computationally efficient alternative estimators for a class of multivariate normal models

Parameters of Gaussian multivariate models are often estimated using the maximum likelihood approach. In spite of its merits, this methodology is not practical when the sample size is very large, as, for example, in the case of massive georeferenced data sets. In this paper, we study the asymptotic properties of the estimators that minimize three alternatives to the likelihood function, designed to increase the computational efficiency. This is achieved by applying the information sandwich technique to expansions of the pseudo-likelihood functions as quadratic forms of independent normal random variables. Theoretical calculations are given for a first-order autoregressive time series and then extended to a two-dimensional autoregressive process on a lattice. We compare the efficiency of the three estimators to that of the maximum likelihood estimator as well as among themselves, using numerical calculations of the theoretical results and simulations.     

Robust weighted orthogonal regression in the errors-in-variables model

This paper focuses on robust estimation in the structural errors-in-variables (EV) model. A new class of robust estimators, called weighted orthogonal regression estimators, is introduced. Robust estimators of the parameters of the EV model are simply derived from robust estimators of multivariate location and scatter such as the M-estimators, the S-estimators and the MCD estimator. The influence functions of the proposed estimators are calculated and shown to be bounded. Moreover, we derive the asymptotic distributions of the estimators and illustrate the results on simulated examples and on a real-data set.     

Semiparametric analysis of longitudinal zero-inflated count data

In this article, we consider a semiparametric zero-inflated Poisson mixed model that postulates a possible nonlinear relationship between the natural logarithm of the mean of the counts and a particular covariate in the longitudinal studies. A penalized log-likelihood function is proposed and Monte Carlo expectation-maximization algorithm is used to derive the estimates. Under some mild conditions, we establish the consistency and asymptotic normality of the resulting estimators. Simulation studies are carried out to investigate the finite sample performance of the proposed method. For illustration purposes, the method is applied to a data set from a pharmaceutical company where the variable of interest is the number of episodes of side effects after the patient has taken the treatments.     

On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding

We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu [K. Knight, W. Fu, Asymptotics for lasso-type estimators, Annals of Statistics 28 (2000) 1356–1378] and Fan and Li [J. Fan, R. Li, Variable selection via non-concave penalized likelihood and its oracle properties, Journal of the American Statistical Association 96 (2001) 1348–1360]. We show that the distributions are typically highly non-normal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than n^−1/2 in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators’ distribution function is also provided.     

Adaptive estimation of an ergodic diffusion process based on sampled data

We consider adaptive maximum likelihood type estimation of both drift and diffusion coefficient parameters for an ergodic diffusion process based on discrete observations. Two kinds of adaptive maximum likelihood type estimators are proposed and asymptotic properties of the adaptive estimators, including convergence of moments, are obtained.     

Asymptotic properties of particle filter-based maximum likelihood estimators for state space models

We study the asymptotic performance of approximate maximum likelihood estimators for state space models obtained via sequential Monte Carlo methods. The state space of the latent Markov chain and the parameter space are assumed to be compact. The approximate estimates are computed by, firstly, running possibly dependent particle filters on a fixed grid in the parameter space, yielding a pointwise approximation of the log-likelihood function. Secondly, extensions of this approximation to the whole parameter space are formed by means of piecewise constant functions or B-spline interpolation, and approximate maximum likelihood estimates are obtained through maximization of the resulting functions. In this setting we formulate criteria for how to increase the number of particles and the resolution of the grid in order to produce estimates that are consistent and asymptotically normal.     

Dual divergence estimators and tests: Robustness results

The class of dual ?-divergence estimators (introduced in Broniatowski and Keziou (2009) [5]) is explored with respect to robustness through the influence function approach. For scale and location models, this class is investigated in terms of robustness and asymptotic relative efficiency. Some hypothesis tests based on dual divergence criteria are proposed and their robustness properties are studied. The empirical performances of these estimators and tests are illustrated by Monte Carlo simulation for both non-contaminated and contaminated data.     

Statistical inference of minimum BD estimators and classifiers for varying-dimensional models

Stochastic modeling for large-scale datasets usually involves a varying-dimensional model space. This paper investigates the asymptotic properties, when the number of parameters grows with the available sample size, of the minimum- estimators and classifiers under a broad and important class of Bregman divergence (), which encompasses nearly all of the commonly used loss functions in the regression analysis, classification procedures and machine learning literature. Unlike the maximum likelihood estimators which require the joint likelihood of observations, the minimum-BD estimators are useful for a range of models where the joint likelihood is unavailable or incomplete. Statistical inference tools developed for the class of large dimensional minimum- estimators and related classifiers are evaluated via simulation studies, and are illustrated by analysis of a real dataset.