Estimation of parameters in latent class models using fuzzy clustering algorithms

A mixture approach to clustering is an important technique in cluster analysis. A mixture of multivariate multinomial distributions is usually used to analyze categorical data with latent class model. The parameter estimation is an important step for a mixture distribution. Described here are four approaches to estimating the parameters of a mixture of multivariate multinomial distributions. The first approach is an extended maximum likelihood (ML) method. The second approach is based on the well-known expectation maximization (EM) algorithm. The third approach is the classification maximum likelihood (CML) algorithm. In this paper, we propose a new approach using the so-called fuzzy class model and then create the fuzzy classification maximum likelihood (FCML) approach for categorical data. The accuracy, robustness and effectiveness of these four types of algorithms for estimating the parameters of multivariate binomial mixtures are compared using real empirical data and samples drawn from the multivariate binomial mixtures of two classes. The results show that the proposed FCML algorithm presents better accuracy, robustness and effectiveness. Overall, the FCML algorithm has the superiority over the ML, EM and CML algorithms. Thus, we recommend FCML as another good tool for estimating the parameters of mixture multivariate multinomial models.     

Efficient Bayesian Inference for Generalized Bradley–Terry Models

The Bradley–Terry model is a popular approach to describe probabilities of the possible outcomes when elements of a set are repeatedly compared with one another in pairs. It has found many applications including animal behavior, chess ranking, and multiclass classification. Numerous extensions of the basic model have also been proposed in the literature including models with ties, multiple comparisons, group comparisons, and random graphs. From a computational point of view, Hunter has proposed efficient iterative minorization-maximization (MM) algorithms to perform maximum likelihood estimation for these generalized Bradley–Terry models whereas Bayesian inference is typically performed using Markov chain Monte Carlo algorithms based on tailored Metropolis–Hastings proposals. We show here that these MM algorithms can be reinterpreted as special instances of expectation-maximization algorithms associated with suitable sets of latent variables and propose some original extensions. These latent variables allow us to derive simple Gibbs samplers for Bayesian inference. We demonstrate experimentally the efficiency of these algorithms on a variety of applications.     

Multivariate approximations to portfolio return distribution

This article proposes a three-step procedure to estimate portfolio return distributions under the multivariate Gram–Charlier (MGC) distribution. The method combines quasi maximum likelihood (QML) estimation for conditional means and variances and the method of moments (MM) estimation for the rest of the density parameters, including the correlation coefficients. The procedure involves consistent estimates even under density misspecification and solves the so-called ‘curse of dimensionality’ of multivariate modelling. Furthermore, the use of a MGC distribution represents a flexible and general approximation to the true distribution of portfolio returns and accounts for all its empirical regularities. An application of such procedure is performed for a portfolio composed of three European indices as an illustration. The MM estimation of the MGC (MGC-MM) is compared with the traditional maximum likelihood of both the MGC and multivariate Student’s t (benchmark) densities. A simulation on Value-at-Risk (VaR) performance for an equally weighted portfolio at 1 and 5 % confidence indicates that the MGC-MM method provides reasonable approximations to the true empirical VaR. Therefore, the procedure seems to be a useful tool for risk managers and practitioners.     

MM Algorithms for Variance Components Models

Variance components estimation and mixed model analysis are central themes in statistics with applications in numerous scientific disciplines. Despite the best efforts of generations of statisticians and numerical analysts, maximum likelihood estimation (MLE) and restricted MLE of variance component models remain numerically challenging. Building on the minorization–maximization (MM) principle, this article presents a novel iterative algorithm for variance components estimation. Our MM algorithm is trivial to implement and competitive on large data problems. The algorithm readily extends to more complicated problems such as linear mixed models, multivariate response models possibly with missing data, maximum a posteriori estimation, and penalized estimation. We establish the global convergence of the MM algorithm to a Karush–Kuhn–Tucker point and demonstrate, both numerically and theoretically, that it converges faster than the classical EM algorithm when the number of variance components is greater than two and all covariance matrices are positive definite. Supplementary materials for this article are available online.     

Mixture decomposition via the simulated annealing algorithm

This paper presents the problem of the evaluation of the maximum likelihood estimator, when the likelihood function has multiple maxima, using the stochastic algorithm called ‘simulated annealing’. Analysis of the particular case of the decomposition of a mixture of five univariate normal distributions shows the properties of this methodology with respect to the E—M algorithm. The results are compared considering some distance measures between the estimated distribution functions and the true one.     

Maximum likelihood estimation in the multi-path change-point problem

Maximum likelihood estimators of the parameters of the distributions before and after the change and the distribution of the time to change in the multi-path change-point problem are derived and shown to be consistent. The maximization of the likelihood can be carried out by using either the EM algorithm or results from mixture distributions. In fact, these two approaches give equivalent algorithms. Simulations to evaluate the performance of the maximum likelihood estimators under practical conditions, and two examples using data on highway fatalities in the United States, and on the health effects of urea formaldehyde foam insulation, are also provided.This work was supported in part by the Natural Science and Engineering Council of Canada, and the Fonds pour la Formation de chercheurs et l'aide à la Recherche Gouvernment du Québec.Lawrence Joseph is also a member of the Department of Epidemiology and Biostatistics of McGill University.     

Using Spherical–Radial Quadrature to Fit Generalized Linear Mixed Effects Models

Although generalized linear mixed effects models have received much attention in the statistical literature, there is still no computationally efficient algorithm for computing maximum likelihood estimates for such models when there are a moderate number of random effects. Existing algorithms are either computationally intensive or they compute estimates from an approximate likelihood. Here we propose an algorithm—the spherical–radial algorithm—that is computationally efficient and computes maximum likelihood estimates. Although we concentrate on two-level, generalized linear mixed effects models, the same algorithm can be applied to many other models as well, including nonlinear mixed effects models and frailty models. The computational difficulty for estimation in these models is in integrating the joint distribution of the data and the random effects to obtain the marginal distribution of the data. Our algorithm uses a multidimensional quadrature rule developed in earlier literature to integrate the joint density. This article discusses how this rule may be combined with an optimization algorithm to efficiently compute maximum likelihood estimates. Because of stratification and other aspects of the quadrature rule, the resulting integral estimator has significantly less variance than can be obtained through simple Monte Carlo integration. Computational efficiency is achieved, in part, because relatively few evaluations of the joint density may be required in the numerical integration.     

Reliability inference of stress–strength model for the truncated proportional hazard rate distribution under progressively Type-II censored samples

This paper considers the reliability inference for the truncated proportional hazard rate stress–strength model based on progressively Type-II censoring scheme. When the stress and strength variables follow the truncated proportional hazard rate distributions, the maximum likelihood estimation and the pivotal quantity estimation of stress–strength reliability are derived. Based on the percentile bootstrap sampling technique, the 95% confidence interval of stress–strength reliability is obtained, as well as the related coverage percentage. Moreover, based on the Fisher Z transformation and the modified generalized pivotal quantity, the 95% modified generalized confidence interval for the stress–strength reliability is obtained. The performance of the proposed method is evaluated by the Monte Carlo simulation. The numerical results show that the pivotal quantity estimators performs better than the maximum likelihood estimators. At last, two real datasets are analyzed by the proposed methodology for illustrative purpose. The results of real example analysis show that our model can be applied to the practical problem, the truncated proportional hazard rate distribution can fit the failure data better than other distributions, and the algorithms in this paper are suitable to handle the small sample data.     

On maximization of the likelihood for the generalized gamma distribution

We explore computational aspects of likelihood maximization for the generalized gamma (GG) distribution. We formulate a version of the score equations such that the equations involved are individually uniquely solvable. We observe that the resulting algorithm is well-behaved and competitive with the application of standard optimisation procedures. We also show that a somewhat neglected alternative existing approach to solving the score equations is good too, at least in the basic, three-parameter case. Most importantly, we argue that, in practice far from being problematic as a number of authors have suggested, the GG distribution is actually particularly amenable to maximum likelihood estimation, by the standards of general three- or more-parameter distributions. We do not, however, make any theoretical advances on questions of convergence of algorithms or uniqueness of roots.     

Computing Estimates in the Proportional Odds Model

The semiparametric proportional odds model for survival data is useful when mortality rates of different groups converge over time. However, fitting the model by maximum likelihood proves computationally cumbersome for large datasets because the number of parameters exceeds the number of uncensored observations. We present here an alternative to the standard Newton-Raphson method of maximum likelihood estimation. Our algorithm, an example of a minorization-maximization (MM) algorithm, is guaranteed to converge to the maximum likelihood estimate whenever it exists. For large problems, both the algorithm and its quasi-Newton accelerated counterpart outperform Newton-Raphson by more than two orders of magnitude.     

Fitting phase–type scale mixtures to heavy–tailed data and distributions

We consider the fitting of heavy tailed data and distributions with a special attention to distributions with a non–standard shape in the “body” of the distribution. To this end we consider a dense class of heavy tailed distributions introduced in Bladt et al. (Scand. Actuar. J., 573–591 2015), employing an EM algorithm for the maximum likelihood estimation of its parameters. We present methods for fitting to observed data, histograms, censored data, as well as to theoretical distributions. Numerical examples are provided with simulated data and a benchmark reinsurance dataset. Empirical examples show that the methods will in most cases adequately fit both body and tail simultaneously.     

Parameter estimation in geometric process with Weibull distribution

We consider geometric process (GP) when the distribution of the first occurrence time of an event is assumed to be Weibull. Explicit estimators of the parameters in GP are derived by using the method of modified maximum likelihood (MML) proposed by Tiku [24]. Asymptotic distributions and consistency properties of these estimators are obtained. We show that our estimators are more efficient than the widely used modified moment (MM) estimators via Monte Carlo simulation study. Further, two real life examples are given at the end of the paper.     

Normal/Independent Distributions and Their Applications in Robust Regression

Abstract

Maximum likelihood estimation with nonnormal error distributions provides one method of robust regression. Certain families of normal/independent distributions are particularly attractive for adaptive, robust regression. This article reviews the properties of normal/independent distributions and presents several new results. A major virtue of these distributions is that they lend themselves to EM algorithms for maximum likelihood estimation. EM algorithms are discussed for least L_p regression and for adaptive, robust regression based on the t, slash, and contaminated normal families. Four concrete examples illustrate the performance of the different methods on real data.     

Bayesian and likelihood inference from equally weighted mixtures

Equally weighted mixture models are recommended for situations where it is required to draw precise finite sample inferences requiring population parameters, but where the population distribution is not constrained to belong to a simple parametric family. They lead to an alternative procedure to the Laird-DerSimonian maximum likelihood algorithm for unequally weighted mixture models. Their primary purpose lies in the facilitation of exact Bayesian computations via importance sampling. Under very general sampling and prior specifications, exact Bayesian computations can be based upon an application of importance sampling, referred to as Permutable Bayesian Marginalization (PBM). An importance function based upon a truncated multivariatet-distribution is proposed, which refers to a generalization of the maximum likelihood procedure. The estimation of discrete distributions, by binomial mixtures, and inference for survivor distributions, via mixtures of exponential or Weibull distributions, are considered. Equally weighted mixture models are also shown to lead to an alternative Gibbs sampling methodology to the Lavine-West approach.     

Balanced random interval arithmetic in market model estimation

The possibility of estimating bounds for the econometric likelihood function using balanced random interval arithmetic is experimentally investigated. The experiments on the likelihood function with data from housing starts have proved the assumption that distributions of centres and radii of evaluated balanced random intervals are normal. Balanced random interval arithmetic can therefore be used to estimate bounds for this function and global optimization algorithms based on this arithmetic are applicable to optimize it. The interval branch and bound algorithms with bounds calculated using standard and balanced random interval arithmetic were used to optimize the likelihood function. Results of the experiments show that when reliability is essential the algorithm with standard interval arithmetic should be used, but when speed of optimization is more important, the algorithm with balanced random interval arithmetic should be used which in this case finishes faster and provides good, although not always optimal, values.     

The EM algorithm for ML Estimators under nonlinear inequalities restrictions on the parameters

One of the most powerful algorithms for obtaining maximum likelihood estimates for many incomplete-data problems is the EM algorithm. However, when the parameters satisfy a set of nonlinear restrictions, It is difficult to apply the EM algorithm directly. In this paper,we propose an asymptotic maximum likelihood estimation procedure under a set of nonlinear inequalities restrictions on the parameters, in which the EM algorithm can be used. Essentially this kind of estimation problem is a stochastic optimization problem in the M-step. We make use of methods in stochastic optimization to overcome the difficulty caused by nonlinearity in the given constraints.     

Heteroscedastic replicated measurement error models under asymmetric heavy-tailed distributions

We propose a heteroscedastic replicated measurement error model based on the class of scale mixtures of skew-normal distributions, which allows the variances of measurement errors to vary across subjects. We develop EM algorithms to calculate maximum likelihood estimates for the model with or without equation error. An empirical Bayes approach is applied to estimate the true covariate and predict the response. Simulation studies show that the proposed models can provide reliable results and the inference is not unduly affected by outliers and distribution misspecification. The method has also been used to analyze a real data of plant root decomposition.     

An MM Algorithm for Multicategory Vertex Discriminant Analysis

This article introduces a new method of supervised learning based on linear discrimination among the vertices of a regular simplex in Euclidean space. Each vertex represents a different category. Discrimination is phrased as a regression problem involving ?-insensitive residuals and a quadratic penalty on the coefficients of the linear predictors. The objective function can by minimized by a primal MM (majorization–minimization) algorithm that (a) relies on quadratic majorization and iteratively re-weighted least squares, (b) is simpler to program than algorithms that pass to the dual of the original optimization problem, and (c) can be accelerated by step doubling. Limited comparisons on real and simulated data suggest that the MM algorithm is competitive in statistical accuracy and computational speed with the best currently available algorithms for discriminant analysis.     

A stochastic expectation-maximization algorithm for the analysis of system lifetime data with known signature

Statistical estimation of the model parameters of component lifetime distribution based on system lifetime data with known system structure is discussed here. We propose the use of stochastic expectation-maximization (SEM) algorithm for obtaining the maximum likelihood estimates of model parameters based on complete and censored system lifetimes. Different ways of implementing the SEM algorithm are also studied. We have shown that the proposed methods are feasible and are easy to implement for various families of component lifetime distributions. The proposed methodologies are then illustrated with two popular lifetime models—the Weibull and Birnbaum-Saunders distributions. Monte Carlo simulation is then used to compare the performance of the proposed methods with the corresponding estimation by direct maximization. Finally, two illustrative examples are presented along with some concluding remarks.     

On some properties of the “apparent length” distribution and a compound distribution

This paper extends and supplements some of the properties of the apparent length distribution introduced by Baker (J. Appl. Statist. 27(1) (2000) 2–21). Recurrence relations for moments and formulae for moments derived via difference and differential operators will be given. We also establish maximum likelihood estimator and Bayesian estimators of a parameter of this distribution. Compound distributions based on the apparent length distribution are also discussed.