Efficient and robust estimation for autoregressive regression models using shape mixtures of skewt normal distribution

Multiple linear regression model based on normally distributed and uncorrelated errors is a popular statistical tool with application in various fields. But these assumptions of normality and no serial correlation are hardly met in real life. Hence, this study considers the linear regression time series model for series with outliers and autocorrelated errors. These autocorrelated errors are represented by a covariance-stationary autoregressive process where the independent innovations are driven by shape mixture of skew-t normal distribution. The shape mixture of skew-t normal distribution is a flexible extension of the skew-t normal with an additional shape parameter that controls skewness and kurtosis. With this error model, stochastic modeling of multiple outliers is possible with an adaptive robust maximum likelihood estimation of all the parameters. An Expectation Conditional Maximization Either algorithm is developed to carryout the maximum likelihood estimation. We derive asymptotic standard errors of the estimators through an information-based approximation. The performance of the estimation procedure developed is evaluated through Monte Carlo simulations and real life data analysis.

     

Robust statistical modeling using the Birnbaum‐Saunders‐t distribution applied to insurance

In this paper, we carry out robust modeling and influence diagnostics in Birnbaum‐Saunders (BS) regression models. Specifically, we present some aspects related to BS and log‐BS distributions and their generalizations from the Student‐t distribution, and develop BS‐t regression models, including maximum likelihood estimation based on the EM algorithm and diagnostic tools. In addition, we apply the obtained results to real data from insurance, which shows the uses of the proposed model. Copyright © 2011 John Wiley & Sons, Ltd.     

The restricted EM algorithm under inequality restrictions on the parameters

One of the most powerful algorithms for maximum likelihood estimation for many incomplete-data problems is the EM algorithm. The restricted EM algorithm for maximum likelihood estimation under linear restrictions on the parameters has been handled by Kim and Taylor (J. Amer. Statist. Assoc. 430 (1995) 708-716). This paper proposes an EM algorithm for maximum likelihood estimation under inequality restrictions A₀β?0, where β is the parameter vector in a linear model W=Xβ+ε and ε is an error variable distributed normally with mean zero and a known or unknown variance matrix Σ>0. Some convergence properties of the EM sequence are discussed. Furthermore, we consider the consistency of the restricted EM estimator and a related testing problem.     

On the mean square error of maximum likelihood estimates of the distribution density of sufficient statistics of the multivariate normal distribution

The extensive use of maximum likelihood estimates underscores the importance of the problem of statistical estimation of their errors. These estimates are of utmost importance in cases where the family of normal distributions and the families related to the normal distributions are considered [1, 2, 4]. The mean square errors of the maximum likelihood estimates of the normal density were investigated in the author's paper [3]. The mean square errors of statistical estimates of some families of densities related to the normal distributions were considered in the papers [4–6]. In the present paper, we obtain an asymptotic expansion of the mean square error of the maximum likelihood estimates of the densities of the joint distribution of sufficient statistics of the family of multivariate normal distributions. The results obtained allow us to construct the mean square errors of the maximum likelihood estimates for the chi-square density and Wishart's density. Translated fromStatisticheskie Metody Otsenivaniya i Proverki Gipotez, pp. 4–11, Perm. 1990.     

Maximum likelihood estimates and likelihood ratio criteria for location and scale parameters of the multivariatel 1-norm symmetric distributions

In this paper we introduce three families of multivariate and matrixl ₁-norm symmetric distributions with location and scale parameters and discuss their maximum likelihood estimates and likelihood ratio criteria. It is shown that under certain condition sthey have the same form as those for independent exponential variates.Projects supported by the science Fund of the Chinese Academy of Sciences.     

Automatic Smoothing With Wavelets for a Wide Class of Distributions

Wavelet-based denoising techniques are well suited to estimate spatially inhomogeneous signals. Waveshrink (Donoho and Johnstone) assumes independent Gaussian errors and equispaced sampling of the signal. Various articles have relaxed some of these assumptions, but a systematic generalization to distributions such as Poisson, binomial, or Bernoulli is missing. We consider a unifying l₁-penalized likelihood approach to regularize the maximum likelihood estimation by adding an l₁ penalty of the wavelet coefficients. Our approach works for all types of wavelets and for a range of noise distributions. We develop both an algorithm to solve the estimation problem and rules to select the smoothing parameter automatically. In particular, using results from Poisson processes, we give an explicit formula for the universal smoothing parameter to denoise Poisson measurements. Simulations show that the procedure is an improvement over other methods. An astronomy example is given.     

On the Consistency of Prony's Method and Related Algorithms

Abstract

Modifications of Prony's classical technique for estimating rate constants in exponential fitting problems have many contemporary applications. In this article the consistency of Prony's method and of related algorithms based on maximum likelihood is discussed as the number of observations n → ∞ by considering the simplest possible models for fitting sums of exponentials to observed data. Two sampling regimes are relevant, corresponding to transient problems and problems of frequency estimation, each of which is associated with rather different kinds of behavior. The general pattern is that the stronger results are obtained for the frequency estimation problem. However, the algorithms considered are all scaling dependent and consistency is not automatic. A new feature that emerges is the importance of an appropriate choice of scale in order to ensure consistency of the estimates in certain cases. The tentative conclusion is that algorithms referred to as Objective function Reweighting Algorithms (ORA's) are superior to their exact maximum likelihood counterparts, referred to as Gradient condition Reweighting Algorithms (GRA's), especially in the frequency estimation problem. This conclusion does not extend to fitting other families of functions such as rational functions.     

Fast Algorithms for Nonparametric Curve Estimation

Abstract

Naive implementations of local polynomial fits and kernel estimators require almost O(n ²) operations. In this article two fast O(n) algorithms for nonparametric local polynomial fitting are presented. They are based on updating normal equations. Numerical stability is guaranteed by controlling ill-conditioned situations for small bandwidths and data-tuned restarting of the updating procedure. Restarting at every output point results in a moderately fast but highly stable O(n ^7/5) algorithm. Applicability of algorithms is evaluated for estimation of regression curves and their derivatives. The idea is also applied to kernel estimators of regression curves and densities.     

MM Algorithms for Some Discrete Multivariate Distributions

The MM (minorization–maximization) principle is a versatile tool for constructing optimization algorithms. Every EM algorithm is an MM algorithm but not vice versa. This article derives MM algorithms for maximum likelihood estimation with discrete multivariate distributions such as the Dirichlet-multinomial and Connor–Mosimann distributions, the Neerchal–Morel distribution, the negative-multinomial distribution, certain distributions on partitions, and zero-truncated and zero-inflated distributions. These MM algorithms increase the likelihood at each iteration and reliably converge to the maximum from well-chosen initial values. Because they involve no matrix inversion, the algorithms are especially pertinent to high-dimensional problems. To illustrate the performance of the MM algorithms, we compare them to Newton’s method on data used to classify handwritten digits.     

Adaptive Estimation of FCG Using Nonlinear State-Space Models

Abstract

In this paper, an efficient adaptive nonlinear algorithm for estimation and identification, the so-called adaptive Lainiotis filter (ALF), is applied to the problem of fatigue crack growth (FCG) estimation, identification, and prediction of the final crack (failure). A suitable nonlinear state-space FCG model is introduced for both ALF and extended Kalman filter (EKF). Both algorithms are tested in order to compare their efficiency. Through extensive analysis and simulation, it is demonstrated that the ALF has superior performance both in FCG estimation, as well as in predicting the remaining lifetime to failure. Furthermore, it is shown that the ALF is faster and easier to implement in a parallel/distributed processing mode, and much more robust than the classic EKF.     

Asymptotic Distribution of Estimated Affinity between Multiparameter Exponential Families

Let F ₁,...,F _J be the distributions of J independent multiparameter exponential families, and _J (F ₁,...,F _J ) denote the affinity between F ₁,...,F _J . We consider the problem of estimating _J on the basis of independent random samples from these distributions. Subject to some mild regularity conditions, we derive the asymptotic distribution of the maximum likelihood estimator of _J . Applications to hypothesis testing and discriminant analysis are discussed, and an example is provided.     

Robust estimation ofk-component univariate normal mixtures

The estimating equations derived from minimising aL ₂ distance between the empirical distribution function and the parametric distribution representing a mixture ofk normal distributions with possibly different means and/or different dispersion parameters are given explicitly. The equations are of theM estimator form in which the function is smooth, bounded and has bounded partial derivatives. As a consequence it is shown that there is a solution of the equations which is robust. In particular there exists a weakly continuous, Fréchet differentiable root and hence there is a consistent root of the equations which is asymptotically normal. These estimating equations offer a robust alternative to the maximum likelihood equations, which are known to yield nonrobust estimators.     

Adaptive estimates for autoregressive processes

Let {X _t :t=0, ±1, ±2, ...} be a stationaryrth order autoregressive process whose generating disturbances are independent identically distributed random variables with marginal distribution functionF. Adaptive estimates for the parameters of {X _t } are constructed from the observed portion of a sample path. The asymptotic efficiency of these estimates relative to the least squares estimates is greater than or equal to one for all regularF. The nature of the adaptive estimates encourages stable behavior for moderate sample sizes. A similar approach can be taken to estimation problems in the general linear model. This research was partially supported by National Science Foundation Grant GP-31091X. American Mathematical Society 1970 subject classification. Primary 62N10; Secondary 62G35. Key words and phrases: autoregressive process, adaptive estimates, robust estimates.     

Likelihood Maximization and Moment Matching in Low SNR Gaussian Mixture Models

We derive an asymptotic expansion for the log-likelihood of Gaussian mixture models (GMMs) with equal covariance matrices in the low signal-to-noise regime. The expansion reveals an intimate connection between two types of algorithms for parameter estimation: the method of moments and likelihood optimizing algorithms such as Expectation-Maximization (EM). We show that likelihood optimization in the low SNR regime reduces to a sequence of least squares optimization problems that match the moments of the estimate to the ground truth moments one by one. This connection is a stepping stone towards the analysis of EM and maximum likelihood estimation in a wide range of models. A motivating application for the study of low SNR mixture models is cryo-electron microscopy data, which can be modeled as a GMM with algebraic constraints imposed on the mixture centers. We discuss the application of our expansion to algebraically constrained GMMs, among other example models of interest. © 2022 The Authors. Communications on Pure and Applied Mathematics published by Wiley Periodicals LLC.     

Gaussian model selection

Our purpose in this paper is to provide a general approach to model selection via penalization for Gaussian regression and to develop our point of view about this subject. The advantage and importance of model selection come from the fact that it provides a suitable approach to many different types of problems, starting from model selection per se (among a family of parametric models, which one is more suitable for the data at hand), which includes for instance variable selection in regression models, to nonparametric estimation, for which it provides a very powerful tool that allows adaptation under quite general circumstances. Our approach to model selection also provides a natural connection between the parametric and nonparametric points of view and copes naturally with the fact that a model is not necessarily true. The method is based on the penalization of a least squares criterion which can be viewed as a generalization of Mallows’C _p . A large part of our efforts will be put on choosing properly the list of models and the penalty function for various estimation problems like classical variable selection or adaptive estimation for various types of l _p -bodies. Received February 1, 1999 / final version received January 10, 2001?Published online April 3, 2001     

An Application of Adaptive Independent Chain Metropolis–Hastings Algorithms in Bayesian Hazard Rate Estimation

In this paper, we consider adaptive independent chain (AIC) Metropolis–Hastings algorithms as introduced in a special context in Gåsemyr et al. (2001) and developed theoretically in Gåsemyr (2003). The algorithms aim at producing samples from a specific target distribution , and are adaptive, non-Markovian versions of the Metropolis–Hastings independent chain. A certain parametric class of possible proposal distributions is fixed, and the parameters of the proposal distribution are updated periodically on the basis of the recent history of the chain, thereby obtaining proposals that get ever closer to . In the former paper a version of these algorithms was shown to be very efficient compared to standard simulation techniques when applied to Bayesian inference in reliability models with at most three dependent parameters. The aim of the present paper is to investigate the performance of the AIC algorithm when the number of dependent parameters and the complexity of the model increases. As a test case we consider a model treated in Arjas and Gasbarra (1996). The target distribution is the posterior distribution for the vector X=(X ₁,...,X _n ) of dependent parameters, representing a piecewise constant approximation to the hazard rate X(t), where t ₀ t t _n . Especially, for the case n=12 it turned out that some versions of the AIC were very efficient compared to standard simulation techniques and also to the algorithm applied in Arjas and Gasbarra (1996). This includes a version of the componentwise adaptive independent chain the basic idea of which was given in Gåsemyr (2003).     

Space alternating penalized Kullback proximal point algorithms for maximizing likelihood with nondifferentiable penalty

The EM algorithm is a widely used methodology for penalized likelihood estimation. Provable monotonicity and convergence are the hallmarks of the EM algorithm and these properties are well established for smooth likelihood and smooth penalty functions. However, many relaxed versions of variable selection penalties are not smooth. In this paper, we introduce a new class of space alternating penalized Kullback proximal extensions of the EM algorithm for nonsmooth likelihood inference. We show that the cluster points of the new method are stationary points even when they lie on the boundary of the parameter set. We illustrate the new class of algorithms for the problems of model selection for finite mixtures of regression and of sparse image reconstruction.     

A fuzzy adaptive network approach to parameter estimation in cases where independent variables come from an exponential distribution

In a regression analysis, it is assumed that the observations come from a single class in a data cluster and the simple functional relationship between the dependent and independent variables can be expressed using the general model; Y=f(X)+ε. However; a data cluster may consist of a combination of observations that have different distributions that are derived from different clusters. When faced with issues of estimating a regression model for fuzzy inputs that have been derived from different distributions, this regression model has been termed the ‘switching regression model’ and it is expressed with . Here l_i indicates the class number of each independent variable and p is indicative of the number of independent variables [J.R. Jang, ANFIS: Adaptive-network-based fuzzy inference system, IEEE Transaction on Systems, Man and Cybernetics 23 (3) (1993) 665–685; M. Michel, Fuzzy clustering and switching regression models using ambiguity and distance rejects, Fuzzy Sets and Systems 122 (2001) 363–399; E.Q. Richard, A new approach to estimating switching regressions, Journal of the American Statistical Association 67 (338) (1972) 306–310].In this study, adaptive networks have been used to construct a model that has been formed by gathering obtained models. There are methods that suggest the class numbers of independent variables heuristically. Alternatively, in defining the optimal class number of independent variables, the use of suggested validity criterion for fuzzy clustering has been aimed. In the case that independent variables have an exponential distribution, an algorithm has been suggested for defining the unknown parameter of the switching regression model and for obtaining the estimated values after obtaining an optimal membership function, which is suitable for exponential distribution.