Method of Moments Using Monte Carlo Simulation

Abstract

We present a computational approach to the method of moments using Monte Carlo simulation. Simple algebraic identities are used so that all computations can be performed directly using simulation draws and computation of the derivative of the log-likelihood. We present a simple implementation using the Newton-Raphson algorithm with the understanding that other optimization methods may be used in more complicated problems. The method can be applied to families of distributions with unknown normalizing constants and can be extended to least squares fitting in the case that the number of moments observed exceeds the number of parameters in the model. The method can be further generalized to allow “moments” that are any function of data and parameters, including as a special case maximum likelihood for models with unknown normalizing constants or missing data. In addition to being used for estimation, our method may be useful for setting the parameters of a Bayes prior distribution by specifying moments of a distribution using prior information. We present two examples—specification of a multivariate prior distribution in a constrained-parameter family and estimation of parameters in an image model. The former example, used for an application in pharmacokinetics, motivated this work. This work is similar to Ruppert's method in stochastic approximation, combines Monte Carlo simulation and the Newton-Raphson algorithm as in Penttinen, uses computational ideas and importance sampling identities of Gelfand and Carlin, Geyer, and Geyer and Thompson developed for Monte Carlo maximum likelihood, and has some similarities to the maximum likelihood methods of Wei and Tanner.     

Iterative Plug-In Algorithms for SEMIFAR Models—Definition,Convergence, and Asymptotic Properties

This article proposes data-driven algorithms for fitting SEMIFAR models. The algorithms combine the data-driven estimation of the nonparametric trend and maximum likelihood estimation of the parameters. Convergence and asymptotic properties of the proposed algorithms are investigated. A large simulation study illustrates the practical performance of the methods.     

The restricted EM algorithm under inequality restrictions on the parameters

One of the most powerful algorithms for maximum likelihood estimation for many incomplete-data problems is the EM algorithm. The restricted EM algorithm for maximum likelihood estimation under linear restrictions on the parameters has been handled by Kim and Taylor (J. Amer. Statist. Assoc. 430 (1995) 708-716). This paper proposes an EM algorithm for maximum likelihood estimation under inequality restrictions A₀β?0, where β is the parameter vector in a linear model W=Xβ+ε and ε is an error variable distributed normally with mean zero and a known or unknown variance matrix Σ>0. Some convergence properties of the EM sequence are discussed. Furthermore, we consider the consistency of the restricted EM estimator and a related testing problem.     

New estimating equation approaches with application in lifetime data analysis

Estimating equation approaches have been widely used in statistics inference. Important examples of estimating equations are the likelihood equations. Since its introduction by Sir R. A. Fisher almost a century ago, maximum likelihood estimation (MLE) is still the most popular estimation method used for fitting probability distribution to data, including fitting lifetime distributions with censored data. However, MLE may produce substantial bias and even fail to obtain valid confidence intervals when data size is not large enough or there is censoring data. In this paper, based on nonlinear combinations of order statistics, we propose new estimation equation approaches for a class of probability distributions, which are particularly effective for skewed distributions with small sample sizes and censored data. The proposed approaches may possess a number of attractive properties such as consistency, sufficiency and uniqueness. Asymptotic normality of these new estimators is derived. The construction of new estimation equations and their numerical performance under different censored schemes are detailed via Weibull distribution and generalized exponential distribution.     

MM Algorithms for Some Discrete Multivariate Distributions

The MM (minorization–maximization) principle is a versatile tool for constructing optimization algorithms. Every EM algorithm is an MM algorithm but not vice versa. This article derives MM algorithms for maximum likelihood estimation with discrete multivariate distributions such as the Dirichlet-multinomial and Connor–Mosimann distributions, the Neerchal–Morel distribution, the negative-multinomial distribution, certain distributions on partitions, and zero-truncated and zero-inflated distributions. These MM algorithms increase the likelihood at each iteration and reliably converge to the maximum from well-chosen initial values. Because they involve no matrix inversion, the algorithms are especially pertinent to high-dimensional problems. To illustrate the performance of the MM algorithms, we compare them to Newton’s method on data used to classify handwritten digits.     

Computing Chernoff's Distribution

A distribution that arises in problems of estimation of monotone functions is that of the location of the maximum of two-sided Brownian motion minus a parabola. Using results from the first author's earlier work, we present algorithms and programs for computation of this distribution and its quantiles. We also present some comparisons with earlier computations and simulations.     

Memetic Algorithms: The Polynomial Local Search Complexity Theory Perspective

In previous work (Krasnogor, . In: Studies on the Theory and Design Space of Memetic Algorithms. Ph.D. thesis, University of the West of England, Bristol, UK, 2002; Krasnogor and Smith, IEEE Trans Evol Algorithms 9(6):474–488, 2005) we develop a syntax-only classification of evolutionary algorithms, in particular so-called memetic algorithms (MAs). When “syntactic sugar” is added to our model, we are able to investigate the polynomial local search (PLS) complexity of memetic algorithms. In this paper we show the PLS-completeness of whole classes of problems that occur when memetic algorithms are applied to the travelling salesman problem using a range of mutation, crossover and local search operators. Our PLS-completeness results shed light on the worst case behaviour that can be expected of a memetic algorithm under these circumstances. Moreover, we point out in this paper that memetic algorithms for graph partitioning and maximum network flow (both with important practical applications) also give rise to PLS-complete problems.

---

Electronic Supplementary Material The online version of this article (doi:) contains supplementary material, which is available to authorized users.      

Improvements of the Maximum Pseudo-Likelihood Estimators in Various Spatial Statistical Models

Abstract

Maximum pseudo-likelihood estimation has hitherto been viewed as a practical but flawed alternative to maximum likelihood estimation, necessary because the maximum likelihood estimator is too hard to compute, but flawed because of its inefficiency when the spatial interactions are strong. We demonstrate that a single Newton-Raphson step starting from the maximum pseudo-likelihood estimator produces an estimator which is close to the maximum likelihood estimator in terms of its actual value, attained likelihood, and efficiency, even in the presence of strong interactions. This hybrid technique greatly increases the practical applicability of pseudo-likelihood-based estimation. Additionally, in the case of the spatial point processes, we propose a proper maximum pseudo-likelihood estimator which is different from the conventional one. The proper maximum pseudo-likelihood estimator clearly shows better performance than the conventional one does when the spatial interactions are strong.     

Normal/Independent Distributions and Their Applications in Robust Regression

Abstract

Maximum likelihood estimation with nonnormal error distributions provides one method of robust regression. Certain families of normal/independent distributions are particularly attractive for adaptive, robust regression. This article reviews the properties of normal/independent distributions and presents several new results. A major virtue of these distributions is that they lend themselves to EM algorithms for maximum likelihood estimation. EM algorithms are discussed for least L_p regression and for adaptive, robust regression based on the t, slash, and contaminated normal families. Four concrete examples illustrate the performance of the different methods on real data.     

Calculation of Posterior Bounds Given Convex Sets of Prior Probability Measures and Likelihood Functions

Abstract

This article presents alternatives and improvements to Lavine's algorithm, currently the most popular method for calculation of posterior expectation bounds induced by sets of probability measures. First, methods from probabilistic logic and Walley's and White-Snow's algorithms are reviewed and compared to Lavine's algorithm. Second, the calculation of posterior bounds is reduced to a fractional programming problem. From the unifying perspective of fractional programming, Lavine's algorithm is derived from Dinkelbach's algorithm, and the White-Snow algorithm is shown to be similar to the Charnes-Cooper transformation. From this analysis, a novel algorithm for expectation bounds is derived. This algorithm provides a complete solution for the calculation of expectation bounds from priors and likelihood functions specified as convex sets of measures. This novel algorithm is then extended to handle the situation where several independent identically distributed measurements are available. Examples are analyzed through a software package that performs robust inferences and that is publicly available.     

Maximum likelihood estimation in a partially observed stratified regression model with censored data

The stratified proportional intensity model generalizes Cox’s proportional intensity model by allowing different groups of the population under study to have distinct baseline intensity functions. In this article, we consider the problem of estimation in this model when the variable indicating the stratum is unobserved for some individuals in the studied sample. In this setting, we construct nonparametric maximum likelihood estimators for the parameters of the stratified model and we establish their consistency and asymptotic normality. Consistent estimators for the limiting variances are also obtained.     

Prony's Method for Multivariate Signals

The problem of recovering translates and corresponding amplitudes of sparse sums of Gaussians out of sampling values as well as reconstructing sparse sums of exponentials are nonlinear inverse problems that can be solved for example by Prony's method. Here, we want to demonstrate a new extension to multivariate input data. (© 2015 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Empirical Likelihood Confidence Bands in Density Estimation

Abstract

Empirical likelihood methods are developed for constructing confidence bands in problems of nonparametric density estimation. These techniques have an advantage over more conventional methods in that the shape of the bands is determined solely by the data. We show how to construct an empirical likelihood functional, rather than a function, and contour it to produce the confidence bands. Analogs of Wilks's theorem are established in this infinite-parameter setting and may be used to select the appropriate contour. An alternative calibration, based on the bootstrap, is also suggested. Large-sample theory is developed to show that the bands have asymptotically correct coverage, and a numerical example is presented to demonstrate the technique. Comparisons are made with the use of bootstrap replications to choose both the shape and size of the bands.     

Likelihood Maximization and Moment Matching in Low SNR Gaussian Mixture Models

We derive an asymptotic expansion for the log-likelihood of Gaussian mixture models (GMMs) with equal covariance matrices in the low signal-to-noise regime. The expansion reveals an intimate connection between two types of algorithms for parameter estimation: the method of moments and likelihood optimizing algorithms such as Expectation-Maximization (EM). We show that likelihood optimization in the low SNR regime reduces to a sequence of least squares optimization problems that match the moments of the estimate to the ground truth moments one by one. This connection is a stepping stone towards the analysis of EM and maximum likelihood estimation in a wide range of models. A motivating application for the study of low SNR mixture models is cryo-electron microscopy data, which can be modeled as a GMM with algebraic constraints imposed on the mixture centers. We discuss the application of our expansion to algebraically constrained GMMs, among other example models of interest. © 2022 The Authors. Communications on Pure and Applied Mathematics published by Wiley Periodicals LLC.     

Likelihood Inferences for High-Dimensional Factor Analysis of Time Series With Applications in Finance

This article investigates likelihood inferences for high-dimensional factor analysis of time series data. We develop a matrix decomposition technique to obtain expressions of the likelihood functions and its derivatives. With such expressions, the traditional delta method that relies heavily on score function and Hessian matrix can be extended to high-dimensional cases. We establish asymptotic theories, including consistency and asymptotic normality. Moreover, fast computational algorithms are developed for estimation. Applications to high-dimensional stock price data and portfolio analysis are discussed. The technical proofs of the asymptotic results and the computer codes are available online.     

Likelihood adaptively modified penalties

A new family of penalty functions, ie, adaptive to likelihood, is introduced for model selection in general regression models. It arises naturally through assuming certain types of prior distribution on the regression parameters. To study the stability properties of the penalized maximum‐likelihood estimator, 2 types of asymptotic stability are defined. Theoretical properties, including the parameter estimation consistency, model selection consistency, and asymptotic stability, are established under suitable regularity conditions. An efficient coordinate‐descent algorithm is proposed. Simulation results and real data analysis show that the proposed approach has competitive performance in comparison with the existing methods.     

Linear mixed models and penalized least squares

Linear mixed-effects models are an important class of statistical models that are used directly in many fields of applications and also are used as iterative steps in fitting other types of mixed-effects models, such as generalized linear mixed models. The parameters in these models are typically estimated by maximum likelihood or restricted maximum likelihood. In general, there is no closed-form solution for these estimates and they must be determined by iterative algorithms such as EM iterations or general nonlinear optimization. Many of the intermediate calculations for such iterations have been expressed as generalized least squares problems. We show that an alternative representation as a penalized least squares problem has many advantageous computational properties including the ability to evaluate explicitly a profiled log-likelihood or log-restricted likelihood, the gradient and Hessian of this profiled objective, and an ECME update to refine this objective.     

一类零膨胀广义线性模型极大似然估计的相合性与渐近正态性

讨论了响应变量为单参数指数族且在零点处膨胀的广义线性模型的大样本性质,对其参数进行了极大似然估计,给出了一些正则条件.基于所提出的正则条件,证明了模型参数极大似然估计的相合性与渐近正态性.     

Heuristic Search Algorithms for the Minimum Volume Ellipsoid

Abstract

A method of robust estimation of multivariate location and shape that has attracted a lot of attention recently is Rousseeuw's minimum volume ellipsoid estimator (MVE). This estimator has a high breakdown point but is difficult to compute successfully. In this article, we apply methods of heuristic search to this problem, including simulated annealing, genetic algorithms, and tabu search, and compare the results to the undirected random search algorithm that is often cited. Heuristic search provides several effective algorithms that are far more computationally efficient than random search. Furthermore, random search, as currently implemented, is shown to be ineffective for larger problems.     

Computing Estimates in the Proportional Odds Model

The semiparametric proportional odds model for survival data is useful when mortality rates of different groups converge over time. However, fitting the model by maximum likelihood proves computationally cumbersome for large datasets because the number of parameters exceeds the number of uncensored observations. We present here an alternative to the standard Newton-Raphson method of maximum likelihood estimation. Our algorithm, an example of a minorization-maximization (MM) algorithm, is guaranteed to converge to the maximum likelihood estimate whenever it exists. For large problems, both the algorithm and its quasi-Newton accelerated counterpart outperform Newton-Raphson by more than two orders of magnitude.