Asymptotic properties of computationally efficient alternative estimators for a class of multivariate normal models

Parameters of Gaussian multivariate models are often estimated using the maximum likelihood approach. In spite of its merits, this methodology is not practical when the sample size is very large, as, for example, in the case of massive georeferenced data sets. In this paper, we study the asymptotic properties of the estimators that minimize three alternatives to the likelihood function, designed to increase the computational efficiency. This is achieved by applying the information sandwich technique to expansions of the pseudo-likelihood functions as quadratic forms of independent normal random variables. Theoretical calculations are given for a first-order autoregressive time series and then extended to a two-dimensional autoregressive process on a lattice. We compare the efficiency of the three estimators to that of the maximum likelihood estimator as well as among themselves, using numerical calculations of the theoretical results and simulations.     

The asymptotic distribution of the likelihood ratio for autoregressive time series with a regression trend

It is shown that the likelihood ratio of an autoregressive time series of finite order with a regression trend is asymptotically normal. This result is used to derive the power of a test for positive correlation of the residuals under local autoregressive alternatives. The test is based on the Durbin-Watson statistics.     

Log-linear Poisson autoregression

We consider a log-linear model for time series of counts. This type of model provides a framework where both negative and positive association can be taken into account. In addition time dependent covariates are accommodated in a straightforward way. We study its probabilistic properties and maximum likelihood estimation. It is shown that a perturbed version of the process is geometrically ergodic, and, under some conditions, it approaches the non-perturbed version. In addition, it is proved that the maximum likelihood estimator of the vector of unknown parameters is asymptotically normal with a covariance matrix that can be consistently estimated. The results are based on minimal assumptions and can be extended to the case of log-linear regression with continuous exogenous variables. The theory is applied to aggregated financial transaction time series. In particular, we discover positive association between the number of transactions and the volatility process of a certain stock.     

Linear relations in time series models. II

In this paper an asymptotic theory is developed for a new time series model which was introduced in a previous paper [5]. An algorithm for computing estimates of the parameters of this time series model is given, and it is shown that these estimators are asymptotically efficient in the sense that they have the same asymptotic distribution as the maximum likelihood estimators.     

Maximum likelihood estimation for all-pass time series models

An autoregressive-moving average model in which all roots of the autoregressive polynomial are reciprocals of roots of the moving average polynomial and vice versa is called an all-pass time series model. All-pass models generate uncorrelated (white noise) time series, but these series are not independent in the non-Gaussian case. An approximate likelihood for a causal all-pass model is given and used to establish asymptotic normality for maximum likelihood estimators under general conditions. Behavior of the estimators for finite samples is studied via simulation. A two-step procedure using all-pass models to identify and estimate noninvertible autoregressive-moving average models is developed and used in the deconvolution of a simulated water gun seismogram.     

Innovations algorithm asymptotics for periodically stationary time series with heavy tails

The innovations algorithm can be used to obtain parameter estimates for periodically stationary time series models. In this paper we compute the asymptotic distribution for these estimates in the case where the underlying noise sequence has infinite fourth moment but finite second moment. In this case, the sample covariances on which the innovations algorithm are based are known to be asymptotically stable. The asymptotic results developed here are useful to determine which model parameters are significant. In the process, we also compute the asymptotic distributions of least squares estimates of parameters in an autoregressive model.     

Relative efficiencies of estimates using patterned covariances or correlations in the multivariate normal estimation problem

Summary The relative efficiency of maximum likelihood estimates is studied when taking advantage of underlying linear patterns in the covariances or correlations when estimating covariance matrices. We compare the variances of estimates of the covariance matrix obtained under two nested patterns with the assumption that the more restricted pattern is the true state. Formulas for the asymptotic variances are given which are exact for linear covariance patterns when explicit maximum likelihood estimates exist. Several specific examples are given using complete symmetry, circular symmetry and general covariance patterns as well as an example involving a covariance matrix with a linear pattern in the correlations.     

Second-order risk comparison of SLSE with GLSE and MLE in a regression with serial correlation

First, the second-order bias of the estimator of the autoregressive parameter based on the ordinary least squares residuals in a linear model with serial correlation is given. Second, the second-order expansion of the risk matrix of a generalized least squares estimator with the above estimated parameter is obtained. This expansion is the same as that based on a suitable estimator of the autoregressive parameter independent of the sample. Third, it is shown that the risk matrix of the generalized least squares estimator is asymptotically equivalent to that of the maximum likelihood estimator up to the second order. Last, a sufficient condition is given for the term due to the estimation of the autoregressive parameter in this expansion to vanish under Grenander's condition for the explanatory variates.     

Generalized Levinson-Durbin sequences, binomial coefficients and autoregressive estimation

For a discrete time second-order stationary process, the Levinson-Durbin recursion is used to determine the coefficients of the best linear predictor of the observation at time k+1, given k previous observations, best in the sense of minimizing the mean square error. The coefficients determined by the recursion define a Levinson-Durbin sequence. We also define a generalized Levinson-Durbin sequence and note that binomial coefficients form a special case of a generalized Levinson-Durbin sequence. All generalized Levinson-Durbin sequences are shown to obey summation formulas which generalize formulas satisfied by binomial coefficients. Levinson-Durbin sequences arise in the construction of several autoregressive model coefficient estimators. The least squares autoregressive estimator does not give rise to a Levinson-Durbin sequence, but least squares fixed point processes, which yield least squares estimates of the coefficients unbiased to order 1/T, where T is the sample length, can be combined to construct a Levinson-Durbin sequence. By contrast, analogous fixed point processes arising from the Yule-Walker estimator do not combine to construct a Levinson-Durbin sequence, although the Yule-Walker estimator itself does determine a Levinson-Durbin sequence. The least squares and Yule-Walker fixed point processes are further studied when the mean of the process is a polynomial time trend that is estimated by least squares.     

Linear mixed models and penalized least squares

Linear mixed-effects models are an important class of statistical models that are used directly in many fields of applications and also are used as iterative steps in fitting other types of mixed-effects models, such as generalized linear mixed models. The parameters in these models are typically estimated by maximum likelihood or restricted maximum likelihood. In general, there is no closed-form solution for these estimates and they must be determined by iterative algorithms such as EM iterations or general nonlinear optimization. Many of the intermediate calculations for such iterations have been expressed as generalized least squares problems. We show that an alternative representation as a penalized least squares problem has many advantageous computational properties including the ability to evaluate explicitly a profiled log-likelihood or log-restricted likelihood, the gradient and Hessian of this profiled objective, and an ECME update to refine this objective.     

On same-realization prediction in an infinite-order autoregressive process

Let observations come from an infinite-order autoregressive (AR) process. For predicting the future of the observed time series (referred to as the same-realization prediction), we use the least-squares predictor obtained by fitting a finite-order AR model. We also allow the order to become infinite as the number of observations does in order to obtain a better approximation. Moment bounds for the inverse sample covariance matrix with an increasing dimension are established under various conditions. We then apply these results to obtain an asymptotic expression for the mean-squared prediction error of the least-squares predictor in same-realization and increasing-order settings. The second-order term of this expression is the sum of two terms which measure both the goodness of fit and model complexity. It forms the foundation for a companion paper by Ing and Wei (Order selection for same-realization predictions in autoregressive processes, Technical report C-00-09, Institute of Statistical Science, Academia Sinica, Taipei, Taiwan, ROC, 2000) which provides the first theoretical verification that AIC is asymptotically efficient for same-realization predictions. Finally, some comparisons between the least-squares predictor and the ridge regression predictor are also given.     

Inference in canonical correlation analysis

The asymptotic behavior, for large sample size, is given for the distribution of the canonical correlation coefficients. The result is used to examine the Bartlett-Lawley test that the residual population canonical correlation coefficients are zero. A marginal likelihood function for the population coefficients is obtained and the maximum marginal likelihood estimates are shown to provide a bias correction.     

A note on maximum likelihood estimation for the complex-valued first-order autoregressive process

The maximum likelihood estimation of the parameters of a complex-valued zero-mean normal stationary first-order autoregressive process is investigated. It is shown that the likelihood function corresponding to independent replicated series is uniquely maximized at a point in the interior of the parameter space. A closed-form expression is given for the estimator.     

The multiple hybrid bootstrap — Resampling multivariate linear processes

The paper reconsiders the autoregressive aided periodogram bootstrap (AAPB) which has been suggested in Kreiss and Paparoditis (2003) [18]. Their idea was to combine a time domain parametric and a frequency domain nonparametric bootstrap to mimic not only a part but as much as possible the complete covariance structure of the underlying time series. We extend the AAPB in two directions. Our procedure explicitly leads to bootstrap observations in the time domain and it is applicable to multivariate linear processes, but agrees exactly with the AAPB in the univariate case, when applied to functionals of the periodogram. The asymptotic theory developed shows validity of the multiple hybrid bootstrap procedure for the sample mean, kernel spectral density estimates and, with less generality, for autocovariances.     

Likelihood ratio tests for goodness-of-fit of a nonlinear regression model

We propose likelihood and restricted likelihood ratio tests for goodness-of-fit of nonlinear regression. The first-order Taylor approximation around the MLE of the regression parameters is used to approximate the null hypothesis and the alternative is modeled nonparametrically using penalized splines. The exact finite sample distribution of the test statistics is obtained for the linear model approximation and can be easily simulated. We recommend using the restricted likelihood instead of the likelihood ratio test because restricted maximum-likelihood estimates are not as severely biased as the maximum-likelihood estimates in the penalized splines framework.     

Parametric estimation of a bivariate stable Lévy process

We propose a parametric model for a bivariate stable Lévy process based on a Lévy copula as a dependence model. We estimate the parameters of the full bivariate model by maximum likelihood estimation. As an observation scheme we assume that we observe all jumps larger than some ε>0 and base our statistical analysis on the resulting compound Poisson process. We derive the Fisher information matrix and prove asymptotic normality of all estimates when the truncation point ε→0. A simulation study investigates the loss of efficiency because of the truncation.     

Offline and online weighted least squares estimation of nonstationary power ARCH processes

This paper proposes two estimation methods based on a weighted least squares criterion for non-(strictly) stationary power ARCH models. The weights are the squared volatilities evaluated at a known value in the parameter space. The first method is adapted for fixed sample size data while the second one allows for online data available in real time. It will be shown that these methods provide consistent and asymptotically Gaussian estimates having asymptotic variance equal to that of the quasi-maximum likelihood estimate (QMLE) regardless of the value of the weighting parameter. Finite-sample performances of the proposed WLS estimates are shown via a simulation study for various sub-classes of power ARCH models.     

Quasi-maximum likelihood estimation in GARCH processes when some coefficients are equal to zero

The asymptotic distribution of the quasi-maximum likelihood (QML) estimator is established for generalized autoregressive conditional heteroskedastic (GARCH) processes, when the true parameter may have zero coefficients. This asymptotic distribution is the projection of a normal vector distribution onto a convex cone. The results are derived under mild conditions. For an important subclass of models, no moment condition is imposed on the GARCH process. The main practical implication of these results concerns the estimation of overidentified GARCH models.     

A matricial extension of the Helson-Szegö theorem and its application in multivariate prediction

It is shown that the positivity of the angle between the past and future of a multivariate stationary process is sufficient for the existence of a mean-convergent autoregressive series representation of its linear predictor. A large class of multivariate processes whose past and future are at positive angle is characterized, thus providing a matricial extension of the Helson-Szegö theorem.     

Wavelet methods for continuous-time prediction using Hilbert-valued autoregressive processes

We consider the prediction problem of a continuous-time stochastic process on an entire time-interval in terms of its recent past. The approach we adopt is based on the notion of autoregressive Hilbert processes that represent a generalization of the classical autoregressive processes to random variables with values in a Hilbert space. A careful analysis reveals, in particular, that this approach is related to the theory of function estimation in linear ill-posed inverse problems. In the deterministic literature, such problems are usually solved by suitable regularization techniques. We describe some recent approaches from the deterministic literature that can be adapted to obtain fast and feasible predictions. For large sample sizes, however, these approaches are not computationally efficient.With this in mind, we propose three linear wavelet methods to efficiently address the aforementioned prediction problem. We present regularization techniques for the sample paths of the stochastic process and obtain consistency results of the resulting prediction estimators. We illustrate the performance of the proposed methods in finite sample situations by means of a real-life data example which concerns with the prediction of the entire annual cycle of climatological El Niño-Southern Oscillation time series 1 year ahead. We also compare the resulting predictions with those obtained by other methods available in the literature, in particular with a smoothing spline interpolation method and with a SARIMA model.