Using Spherical–Radial Quadrature to Fit Generalized Linear Mixed Effects Models

Although generalized linear mixed effects models have received much attention in the statistical literature, there is still no computationally efficient algorithm for computing maximum likelihood estimates for such models when there are a moderate number of random effects. Existing algorithms are either computationally intensive or they compute estimates from an approximate likelihood. Here we propose an algorithm—the spherical–radial algorithm—that is computationally efficient and computes maximum likelihood estimates. Although we concentrate on two-level, generalized linear mixed effects models, the same algorithm can be applied to many other models as well, including nonlinear mixed effects models and frailty models. The computational difficulty for estimation in these models is in integrating the joint distribution of the data and the random effects to obtain the marginal distribution of the data. Our algorithm uses a multidimensional quadrature rule developed in earlier literature to integrate the joint density. This article discusses how this rule may be combined with an optimization algorithm to efficiently compute maximum likelihood estimates. Because of stratification and other aspects of the quadrature rule, the resulting integral estimator has significantly less variance than can be obtained through simple Monte Carlo integration. Computational efficiency is achieved, in part, because relatively few evaluations of the joint density may be required in the numerical integration.     

Maximum likelihood least squares identification for systems with autoregressive moving average noise

Maximum likelihood methods are important for system modeling and parameter estimation. This paper derives a recursive maximum likelihood least squares identification algorithm for systems with autoregressive moving average noises, based on the maximum likelihood principle. In this derivation, we prove that the maximum of the likelihood function is equivalent to minimizing the least squares cost function. The proposed algorithm is different from the corresponding generalized extended least squares algorithm. The simulation test shows that the proposed algorithm has a higher estimation accuracy than the recursive generalized extended least squares algorithm.     

Efficiency of Monte Carlo EM and Simulated Maximum Likelihood in Two-Stage Hierarchical Models

Likelihood estimation in hierarchical models is often complicated by the fact that the likelihood function involves an analytically intractable integral. Numerical approximation to this integral is an option but it is generally not recommended when the integral dimension is high. An alternative approach is based on the ideas of Monte Carlo integration, which approximates the intractable integral by an empirical average based on simulations. This article investigates the efficiency of two Monte Carlo estimation methods, the Monte Carlo EM (MCEM) algorithm and simulated maximum likelihood (SML). We derive the asymptotic Monte Carlo errors of both methods and show that, even under the optimal SML importance sampling distribution, the efficiency of SML decreases rapidly (relative to that of MCEM) as the missing information about the unknown parameter increases. We illustrate our results in a simple mixed model example and perform a simulation study which shows that, compared to MCEM, SML can be extremely inefficient in practical applications.     

Generalized least squares and maximum likelihood estimations of multivariate polychoric correlations

This paper investigates the generalized least squares estimation and the maximum likelihood estimation of the parameters in a multivariate polychoric correlations model, based on data from a multidimensional contingency table. Asymptotic properties of the estimators are discussed. An iterative procedure based on the Gauss-Newton algorithm is implemented to produce the generalized least squares estimates and the standard errors estimates. It is shown that via an iteratively reweighted method, the algorithm produces the maximum likelihood estimates as well. Numerical results on the finite sample behaviors of the methods are reported.     

Bayesian Estimation of Discrete Multivariate Latent Structure Models With Structural Zeros

In multivariate categorical data, models based on conditional independence assumptions, such as latent class models, offer efficient estimation of complex dependencies. However, Bayesian versions of latent structure models for categorical data typically do not appropriately handle impossible combinations of variables, also known as structural zeros. Allowing nonzero probability for impossible combinations results in inaccurate estimates of joint and conditional probabilities, even for feasible combinations. We present an approach for estimating posterior distributions in Bayesian latent structure models with potentially many structural zeros. The basic idea is to treat the observed data as a truncated sample from an augmented dataset, thereby allowing us to exploit the conditional independence assumptions for computational expediency. As part of the approach, we develop an algorithm for collapsing a large set of structural zero combinations into a much smaller set of disjoint marginal conditions, which speeds up computation. We apply the approach to sample from a semiparametric version of the latent class model with structural zeros in the context of a key issue faced by national statistical agencies seeking to disseminate confidential data to the public: estimating the number of records in a sample that are unique in the population on a set of publicly available categorical variables. The latent class model offers remarkably accurate estimates of population uniqueness, even in the presence of a large number of structural zeros.     

Estimation of parameters in latent class models using fuzzy clustering algorithms

A mixture approach to clustering is an important technique in cluster analysis. A mixture of multivariate multinomial distributions is usually used to analyze categorical data with latent class model. The parameter estimation is an important step for a mixture distribution. Described here are four approaches to estimating the parameters of a mixture of multivariate multinomial distributions. The first approach is an extended maximum likelihood (ML) method. The second approach is based on the well-known expectation maximization (EM) algorithm. The third approach is the classification maximum likelihood (CML) algorithm. In this paper, we propose a new approach using the so-called fuzzy class model and then create the fuzzy classification maximum likelihood (FCML) approach for categorical data. The accuracy, robustness and effectiveness of these four types of algorithms for estimating the parameters of multivariate binomial mixtures are compared using real empirical data and samples drawn from the multivariate binomial mixtures of two classes. The results show that the proposed FCML algorithm presents better accuracy, robustness and effectiveness. Overall, the FCML algorithm has the superiority over the ML, EM and CML algorithms. Thus, we recommend FCML as another good tool for estimating the parameters of mixture multivariate multinomial models.     

Implementations of the Monte Carlo EM Algorithm

The Monte Carlo EM (MCEM) algorithm is a modification of the EM algorithm where the expectation in the E-step is computed numerically through Monte Carlo simulations. The most exible and generally applicable approach to obtaining a Monte Carlo sample in each iteration of an MCEM algorithm is through Markov chain Monte Carlo (MCMC) routines such as the Gibbs and Metropolis–Hastings samplers. Although MCMC estimation presents a tractable solution to problems where the E-step is not available in closed form, two issues arise when implementing this MCEM routine: (1) how do we minimize the computational cost in obtaining an MCMC sample? and (2) how do we choose the Monte Carlo sample size? We address the first question through an application of importance sampling whereby samples drawn during previous EM iterations are recycled rather than running an MCMC sampler each MCEM iteration. The second question is addressed through an application of regenerative simulation. We obtain approximate independent and identical samples by subsampling the generated MCMC sample during different renewal periods. Standard central limit theorems may thus be used to gauge Monte Carlo error. In particular, we apply an automated rule for increasing the Monte Carlo sample size when the Monte Carlo error overwhelms the EM estimate at any given iteration. We illustrate our MCEM algorithm through analyses of two datasets fit by generalized linear mixed models. As a part of these applications, we demonstrate the improvement in computational cost and efficiency of our routine over alternative MCEM strategies.     

Incorporating scatter search and threshold accepting in finding maximum likelihood estimates for the multinomial probit model

This paper presents a procedure that incorporates scatter search and threshold accepting to find the maximum likelihood estimates for the multinomial probit (MNP) model. Scatter search, widely used in optimization-related studies, is a type of evolutionary algorithm that uses a small set of solutions as the selection pool for mating and generating new solutions to search for a globally optimal solution. Threshold accepting is applied to the scatter search to improve computational efficiency while maintaining the same level of solution quality. A set of numerical experiments, based on synthetic data sets with known model specifications and error structures, were conducted to test the effectiveness and efficiency of the proposed framework. The results indicated that the proposed procedure enhanced performance in terms of likelihood function value and computational efficiency for MNP model estimation as compared to the original scatter search framework.     

Initial classification of joint data in EM estimation of latent class joint model

The latent class mixture-of-experts joint model is one of the important methods for jointly modelling longitudinal and recurrent events data when the underlying population is heterogeneous and there are nonnormally distributed outcomes. The maximum likelihood estimates of parameters in latent class joint model are generally obtained by the EM algorithm. The joint distances between subjects and initial classification of subjects under study are essential to finding good starting values of the EM algorithm through formulas. In this article, separate distances and joint distances of longitudinal markers and recurrent events are proposed for classification purposes, and performance of the initial classifications based on the proposed distances and random classification are compared in a simulation study and demonstrated in an example.     

A modified candecomp algorithm for fitting the latent class model: Implementation and evaluation

In this paper an implementation is discussed of a modified CANDECOMP algorithm for fitting Lazarsfeld's latent class model. The CANDECOMP algorithm is modified such that the resulting parameter estimates are non-negative and ‘best asymptotically normal’. In order to achieve this, the modified CANDECOMP algorithm minimizes a weighted least squares function instead of an unweighted least squares function as the traditional CANDECOMP algorithm does. To evaluate the new procedure, the modified CANDECOMP procedure with different weighting schemes is compared on five published data sets with the widely-used iterative proportional fitting procedure for obtaining maximum likelihood estimates of the parameters in the latent class model. It is found that, with appropriate weights, the modified CANDECOMP algorithm yields solutions that are nearly identical with those obtained by means of the maximum likelihood procedure. While the modified CANDECOMP algorithm tends to be computationally more intensive than the maximum likelihood method, it is very flexible in that it easily allows one to try out different weighting schemes.     

On maximization of the likelihood for the generalized gamma distribution

We explore computational aspects of likelihood maximization for the generalized gamma (GG) distribution. We formulate a version of the score equations such that the equations involved are individually uniquely solvable. We observe that the resulting algorithm is well-behaved and competitive with the application of standard optimisation procedures. We also show that a somewhat neglected alternative existing approach to solving the score equations is good too, at least in the basic, three-parameter case. Most importantly, we argue that, in practice far from being problematic as a number of authors have suggested, the GG distribution is actually particularly amenable to maximum likelihood estimation, by the standards of general three- or more-parameter distributions. We do not, however, make any theoretical advances on questions of convergence of algorithms or uniqueness of roots.     

Nonparametric likelihood based estimation for a multivariate Lipschitz density

We consider a problem of nonparametric density estimation under shape restrictions. We deal with the case where the density belongs to a class of Lipschitz functions. Devroye [L. Devroye, A Course in Density Estimation, in: Progress in Probability and Statistics, vol. 14, Birkhäuser Boston Inc., Boston, MA, 1987] considered these classes of estimates as tailor-made estimates, in contrast in some way to universally consistent estimates. In our framework we get the existence and uniqueness of the maximum likelihood estimate as well as strong consistency. This NPMLE can be easily characterized but it is not easy to compute. Some simpler approximations are also considered.     

Clustering bivariate mixed-type data via the cluster-weighted model

The cluster-weighted model (CWM) is a mixture model with random covariates that allows for flexible clustering/classification and distribution estimation of a random vector composed of a response variable and a set of covariates. Within this class of models, the generalized linear exponential CWM is here introduced especially for modeling bivariate data of mixed-type. Its natural counterpart in the family of latent class models is also defined. Maximum likelihood parameter estimates are derived using the expectation-maximization algorithm and some computational issues are detailed. Through Monte Carlo experiments, the classification performance of the proposed model is compared with other mixture-based approaches, consistency of the estimators of the regression coefficients is evaluated, and several likelihood-based information criteria are compared for selecting the number of mixture components. An application to real data is also finally considered.     

Particle-based online estimation of tangent filters with application to parameter estimation in nonlinear state-space models

This paper presents a novel algorithm for efficient online estimation of the filter derivatives in general hidden Markov models. The algorithm, which has a linear computational complexity and very limited memory requirements, is furnished with a number of convergence results, including a central limit theorem with an asymptotic variance that can be shown to be uniformly bounded in time. Using the proposed filter derivative estimator, we design a recursive maximum likelihood algorithm updating the parameters according the gradient of the one-step predictor log-likelihood. The efficiency of this online parameter estimation scheme is illustrated in a simulation study.

     

On maximum likelihood estimation of a Pareto mixture

In this paper we deal with maximum likelihood estimation (MLE) of the parameters of a Pareto mixture. Standard MLE procedures are difficult to apply in this setup, because the distributions of the observations do not have common support. We study the properties of the estimators under different hypotheses; in particular, we show that, when all the parameters are unknown, the estimators can be found maximizing the profile likelihood function. Then we turn to the computational aspects of the problem, and develop three alternative procedures: an EM-type algorithm, a Simulated Annealing and an algorithm based on Cross-Entropy minimization. The work is motivated by an application in the operational risk measurement field: we fit a Pareto mixture to operational losses recorded by a bank in two different business lines. Under the assumption that each population follows a Pareto distribution, the appropriate model is a mixture of Pareto distributions where all the parameters have to be estimated.     

Parameter estimation and model selection for mixtures of truncated exponentials

Bayesian networks with mixtures of truncated exponentials (MTEs) support efficient inference algorithms and provide a flexible way of modeling hybrid domains (domains containing both discrete and continuous variables). On the other hand, estimating an MTE from data has turned out to be a difficult task, and most prevalent learning methods treat parameter estimation as a regression problem. The drawback of this approach is that by not directly attempting to find the parameter estimates that maximize the likelihood, there is no principled way of performing subsequent model selection using those parameter estimates. In this paper we describe an estimation method that directly aims at learning the parameters of an MTE potential following a maximum likelihood approach. Empirical results demonstrate that the proposed method yields significantly better likelihood results than existing regression-based methods. We also show how model selection, which in the case of univariate MTEs amounts to partitioning the domain and selecting the number of exponential terms, can be performed using the BIC score.     

Marshall-Olkin�������ֲ��ı�Ҷ˹����

??Kundu and Gupta proposed to use the importance sampling method to compute the Bayesian estimation of the unknown parameters of the Marshall-Olkin bivariate Weibull distribution. However, we find that the performance of the importance sampling method becomes worse as the sample size gets larger. In this paper, we introduce latent variables to simplify the likelihood function, and use MCMC algorithm to estimate the unknown parameters. Numerical simulations are carried out to assess the performance of the proposed method by comparing with the maximum likelihood estimation, and we find that the Bayesian estimates perform better even for the case of small sample size. A real data is also analyzed for illustrative purpose.     

Monte Carlo maximum likelihood circle fitting using circular density functions

Finding the “best-fitting” circle to describe a set of points in two dimensions is discussed in terms of maximum likelihood estimation. Several combinations of distributions are proposed to describe the stochastic nature of points in the plane, as the points are considered to have a common, typically unknown center, a random radius, and random angular orientation. A Monte Carlo search algorithm over part of the parameter space is suggested for finding the maximum likelihood parameter estimates. Examples are presented, and comparisons are drawn between circles fit by this proposed method, least squares, and other maximum likelihood methods found in the literature.     

Maximum likelihood estimation for multivariate skew normal mixture models

This paper provides a flexible mixture modeling framework using the multivariate skew normal distribution. A feasible EM algorithm is developed for finding the maximum likelihood estimates of parameters in this context. A general information-based method for obtaining the asymptotic covariance matrix of the maximum likelihood estimators is also presented. The proposed methodology is illustrated with a real example and results are also compared with those obtained from fitting normal mixtures.     

A Non-Iterative Alternative to Ordinal Log-Linear Models

Log-linear modeling is a popular statistical tool for analysing a contingency table. This presentation focuses on an alternative approach to modeling ordinal categorical data. The technique, based on orthogonal polynomials, provides a much simpler method of model fitting than the conventional approach of maximum likelihood estimation, as it does not require iterative calculations nor the fitting and refitting to search for the best model. Another advantage is that quadratic and higher order effects can readily be included, in contrast to conventional log-linear models which incorporate linear terms only.

The focus of the discussion is the application of the new parameter estimation technique to multi-way contingency tables with at least one ordered variable. This will also be done by considering singly and doubly ordered two-way contingency tables. It will be shown by example that the resulting parameter estimates are numerically similar to corresponding maximum likelihood estimates for ordinal log-linear models.