Identifying Mixtures of Mixtures Using Bayesian Estimation

The use of a finite mixture of normal distributions in model-based clustering allows us to capture non-Gaussian data clusters. However, identifying the clusters from the normal components is challenging and in general either achieved by imposing constraints on the model or by using post-processing procedures. Within the Bayesian framework, we propose a different approach based on sparse finite mixtures to achieve identifiability. We specify a hierarchical prior, where the hyperparameters are carefully selected such that they are reflective of the cluster structure aimed at. In addition, this prior allows us to estimate the model using standard MCMC sampling methods. In combination with a post-processing approach which resolves the label switching issue and results in an identified model, our approach allows us to simultaneously (1) determine the number of clusters, (2) flexibly approximate the cluster distributions in a semiparametric way using finite mixtures of normals and (3) identify cluster-specific parameters and classify observations. The proposed approach is illustrated in two simulation studies and on benchmark datasets. Supplementary materials for this article are available online.     

混合狄雷克利分布和高维表的贝叶斯估计

本文讨论多项分布情况下的高维列联表使用混合狄雷克利分布为先验分布时，贝叶斯估计的表达，以及独立性条件的表述．将文献［４］和［５］的结论推广到高维列联表中．     

Reference priors for exponential families with simple quadratic variance function

Reference analysis is one of the most successful general methods to derive noninformative prior distributions. In practice, however, reference priors are often difficult to obtain. Recently developed theory for conditionally reducible natural exponential families identifies an attractive reparameterization which allows one, among other things, to construct an enriched conjugate prior. In this paper, under the assumption that the variance function is simple quadratic, the order-invariant group reference prior for the above parameter is found. Furthermore, group reference priors for the mean- and natural parameter of the families are obtained. A brief discussion of the frequentist coverage properties is also presented. The theory is illustrated for the multinomial and negative-multinomial family. Posterior computations are especially straightforward due to the fact that the resulting reference distributions belong to the corresponding enriched conjugate family. A substantive application of the theory relates to the construction of reference priors for the Bayesian analysis of two-way contingency tables with respect to two alternative parameterizations.     

Approximation of Laws of Multinomial Parameters by Mixtures of Dirichlet Distributions with Applications to Bayesian Inference

Within the framework of Bayesian inference, when observations are exchangeable and take values in a finite space X, a prior P is approximated (in the Prokhorov metric) with any precision by explicitly constructed mixtures of Dirichlet distributions. Likewise, the posteriors are approximated with some precision by the posteriors of these mixtures of Dirichlet distributions. Approximations in the uniform metric for distribution functions are also given. These results are applied to obtain a method for eliciting prior beliefs and to approximate both the predictive distribution (in the variational metric) and the posterior distribution function of d (in the Lévy metric), when is a random probability having distribution P.     

Frequentistic approximations to Bayesian prevision of exchangeable random elements

Posterior and predictive distributions for m future trials, given the first n elements of an infinite exchangeable sequence ${\tilde{\xi }}_1,{\tilde{\xi }}_2,\dots$, are considered in a nonparametric Bayesian setting. The former distribution is compared to the unit mass at the empirical distribution ${\tilde{\mathfrak{e}}}_n:=\frac{1}{n}{\sum }_{i=1}^n{\delta }_{{\tilde{\xi }}_i}$ of the n past observations, while the latter is compared to the m-fold product ${\tilde{\mathfrak{e}}}_n^m$. Comparisons are made by means of distinguished probability distances inducing topologies that are equivalent to (or finer than) the topology of weak convergence of probability measures. After stating almost sure convergence to zero of these distances as n goes to infinity, the paper focuses on the analysis of the rate of approach to zero, so providing a quantitative evaluation of the approximation of posterior and predictive distributions through their frequentistic counterparts ${\delta }_{{\tilde{\mathfrak{e}}}_n}$ and ${\tilde{\mathfrak{e}}}_n^m$, respectively. Characteristic features of the present work, with respect to more common literature on Bayesian consistency, are: first, comparisons are made between entities which depend on the n past observation only; second, the approximations are studied under the actual (exchangeable) law of the ${\tilde{\xi }}_n$'s, and not under hypothetical product laws ${\mathfrak{p}}_0^{\infty }$, as ${\mathfrak{p}}_0$ varies among the admissible determinations of a random probability measure.     

Posterior Simulation of Normalized Random Measure Mixtures

This article describes posterior simulation methods for mixture models whose mixing distribution has a Normalized Random Measure prior. The methods use slice sampling ideas and introduce no truncation error. The approach can be easily applied to both homogeneous and nonhomogeneous Normalized Random Measures and allows the updating of the parameters of the random measure. The methods are illustrated on data examples using both Dirichlet and Normalized Generalized Gamma process priors. In particular, the methods are shown to be computationally competitive with previously developed samplers for Dirichlet process mixture models. Matlab code to implement these methods is available as supplemental material.     

Merging experts’ opinions: A Bayesian hierarchical model with mixture of prior distributions

In this paper, a general approach is proposed to address a full Bayesian analysis for the class of quadratic natural exponential families in the presence of several expert sources of prior information. By expressing the opinion of each expert as a conjugate prior distribution, a mixture model is used by the decision maker to arrive at a consensus of the sources. A hyperprior distribution on the mixing parameters is considered and a procedure based on the expected Kullback–Leibler divergence is proposed to analytically calculate the hyperparameter values. Next, the experts’ prior beliefs are calibrated with respect to the combined posterior belief over the quantity of interest by using expected Kullback–Leibler divergences, which are estimated with a computationally low-cost method. Finally, it is remarkable that the proposed approach can be easily applied in practice, as it is shown with an application.     

Recursive formulas for discrete distributions

We apply power series techniques for differential equations on probability generating functions to derive recursive formulas for discrete compound distributions. Such formulas are computationally effective and useful in risk theory.     

Existence of Bayesian Estimates for the Polychotomous Quantal Response Models

This paper investigates the existence of Bayesian estimates for polychotomous quantal response models using a uniform improper prior distribution on the regression parameters. Necessary and sufficient conditions for the propriety of the posterior distribution with a general link function are established. In addition, the sufficient conditions for the existence of the posterior moments and the posterior moment generating function are obtained. It is also found that the propriety guarantees the existence of the maximum likelihood estimate.     

Functional Compatibility,Markov Chains,and Gibbs Sampling with Improper Posteriors

Abstract

The members of a set of conditional probability density functions are called compatible if there exists a joint probability density function that generates them. We generalize this concept by calling the conditionals functionally compatible if there exists a non-negative function that behaves like a joint density as far as generating the conditionals according to the probability calculus, but whose integral over the whole space is not necessarily finite. A necessary and sufficient condition for functional compatibility is given that provides a method of calculating this function, if it exists. A Markov transition function is then constructed using a set of functionally compatible conditional densities and it is shown, using the compatibility results, that the associated Markov chain is positive recurrent if and only if the conditionals are compatible. A Gibbs Markov chain, constructed via “Gibbs conditionals” from a hierarchical model with an improper posterior, is a special case. Therefore, the results of this article can be used to evaluate the consequences of applying the Gibbs sampler when the posterior's impropriety is unknown to the user. Our results cannot, however, be used to detect improper posteriors. Monte Carlo approximations based on Gibbs chains are shown to have undesirable limiting behavior when the posterior is improper. The results are applied to a Bayesian hierarchical one-way random effects model with an improper posterior distribution. The model is simple, but also quite similar to some models with improper posteriors that have been used in conjunction with the Gibbs sampler in the literature.     

A Bayesian algorithm for global optimization

A crucial step in global optimization algorithms based on random sampling in the search domain is decision about the achievement of a prescribed accuracy. In order to overcome the difficulties related to such a decision, the Bayesian Nonparametric Approach has been introduced. The aim of this paper is to show the effectiveness of the approach when an ad hoc clustering technique is used for obtaining promising starting points for a local search algorithm. Several test problems are considered.     

Propriety of posterior distribution for dichotomous quantal response models

In this article, we investigate the property of posterior distribution for dichotomous quantal response models using a uniform prior distribution on the regression parameters. Sufficient and necessary conditions for the propriety of the posterior distribution with a general link function are established. In addition, the sufficient conditions for the existence of the posterior moments and the posterior moment generating function are also obtained. Finally, the relationship between the propriety of posterior distribution and the existence of the maximum likelihood estimate is examined.

     

Bayesian Nonparametric Analysis for a Generalized Dirichlet Process Prior

This paper considers a generalization of the Dirichlet process which is obtained by suitably normalizing superposed independent gamma processes having increasing integer-valued scale parameter. A comprehensive treatment of this random probability measure is provided. We prove results concerning its finite-dimensional distributions, moments, predictive distributions and the distribution of its mean. Most expressions are given in terms of multiple hypergeometric functions, thus highlighting the interplay between Bayesian Nonparametrics and special functions. Finally, a suitable simulation algorithm is applied in order to compute quantities of statistical interest.     

MM Algorithms for Some Discrete Multivariate Distributions

The MM (minorization–maximization) principle is a versatile tool for constructing optimization algorithms. Every EM algorithm is an MM algorithm but not vice versa. This article derives MM algorithms for maximum likelihood estimation with discrete multivariate distributions such as the Dirichlet-multinomial and Connor–Mosimann distributions, the Neerchal–Morel distribution, the negative-multinomial distribution, certain distributions on partitions, and zero-truncated and zero-inflated distributions. These MM algorithms increase the likelihood at each iteration and reliably converge to the maximum from well-chosen initial values. Because they involve no matrix inversion, the algorithms are especially pertinent to high-dimensional problems. To illustrate the performance of the MM algorithms, we compare them to Newton’s method on data used to classify handwritten digits.     

A Computational Approach for Full Nonparametric Bayesian Inference Under Dirichlet Process Mixture Models

Widely used parametric generalized linear models are, unfortunately, a somewhat limited class of specifications. Nonparametric aspects are often introduced to enrich this class, resulting in semiparametric models. Focusing on single or k-sample problems, many classical nonparametric approaches are limited to hypothesis testing. Those that allow estimation are limited to certain functionals of the underlying distributions. Moreover, the associated inference often relies upon asymptotics when nonparametric specifications are often most appealing for smaller sample sizes. Bayesian nonparametric approaches avoid asymptotics but have, to date, been limited in the range of inference. Working with Dirichlet process priors, we overcome the limitations of existing simulation-based model fitting approaches which yield inference that is confined to posterior moments of linear functionals of the population distribution. This article provides a computational approach to obtain the entire posterior distribution for more general functionals. We illustrate with three applications: investigation of extreme value distributions associated with a single population, comparison of medians in a k-sample problem, and comparison of survival times from different populations under fairly heavy censoring.     

On perturbations of strongly admissible prior distributions

Consider a parametric statistical model, P(dx|θ), and an improper prior distribution, ν(dθ), that together yield a (proper) formal posterior distribution, Q(dθ|x). The prior is called strongly admissible if the generalized Bayes estimator of every bounded function of θ is admissible under squared error loss. Eaton [M.L. Eaton, A statistical diptych: Admissible inferences-recurrence of symmetric Markov chains, Annals of Statistics 20 (1992) 1147–1179] used the Blyth–Stein Lemma to develop a sufficient condition, call it , for strong admissibility of ν. Our main result says that, under mild regularity conditions, if ν satisfies and g(θ) is a bounded, non-negative function, then the perturbed prior distribution g(θ)ν(dθ) also satisfies and is therefore strongly admissible. Our proof has three basic components: (i) Eaton's [M.L. Eaton, A statistical diptych: Admissible inferences-recurrence of symmetric Markov chains, Annals of Statistics 20 (1992) 1147–1179] result that the condition is equivalent to the local recurrence of the Markov chain whose transition function is R(dθ|η)=∫Q(dθ|x)P(dx|η); (ii) a new result for general state space Markov chains giving conditions under which local recurrence is equivalent to recurrence; and (iii) a new generalization of Hobert and Robert's [J.P. Hobert, C.P. Robert, Eaton's Markov chain, its conjugate partner and -admissibility, Annals of Statistics 27 (1999) 361–373] result that says Eaton's Markov chain is recurrent if and only if the chain with transition function is recurrent. One important application of our results involves the construction of strongly admissible prior distributions for estimation problems with restricted parameter spaces.     

Computation of reference Bayesian inference for variance components in longitudinal studies

Generalized linear mixed models (GLMMs) have been applied widely in the analysis of longitudinal data. This model confers two important advantages, namely, the flexibility to include random effects and the ability to make inference about complex covariances. In practice, however, the inference of variance components can be a difficult task due to the complexity of the model itself and the dimensionality of the covariance matrix of random effects. Here we first discuss for GLMMs the relation between Bayesian posterior estimates and penalized quasi-likelihood (PQL) estimates, based on the generalization of Harville’s result for general linear models. Next, we perform fully Bayesian analyses for the random covariance matrix using three different reference priors, two with Jeffreys’ priors derived from approximate likelihoods and one with the approximate uniform shrinkage prior. Computations are carried out via the combination of asymptotic approximations and Markov chain Monte Carlo methods. Under the criterion of the squared Euclidean norm, we compare the performances of Bayesian estimates of variance components with that of PQL estimates when the responses are non-normal, and with that of the restricted maximum likelihood (REML) estimates when data are assumed normal. Three applications and simulations of binary, normal, and count responses with multiple random effects and of small sample sizes are illustrated. The analyses examine the differences in estimation performance when the covariance structure is complex, and demonstrate the equivalence between PQL and the posterior modes when the former can be derived. The results also show that the Bayesian approach, particularly under the approximate Jeffreys’ priors, outperforms other procedures.     

Wavelet modeling of priors on triangles

Parameters in statistical problems often live in a geometry of certain shape. For example, count probabilities in a multinomial distribution belong to a simplex. For these problems, Bayesian analysis needs to model priors satisfying certain constraints imposed by the geometry. This paper investigates modeling of priors on triangles by use of wavelets constructed specifically for triangles. Theoretical analysis and numerical simulations show that our modeling is flexible and is superior to the commonly used Dirichlet prior.     

Generalized Moment Theory and Bayesian Robustness Analysis for Hierarchical Mixture Models

In applications of Bayesian analysis one problem that arises is the evaluation of the sensitivity, or robustness, of the adopted inferential procedure with respect to the components of the formulated statistical model. In particular, it is of interest to study robustness with respect to the prior, when this latter cannot be uniquely elicitated, but a whole class Γ of probability measures, agreeing with the available information, can be identified. In this situation, the analysis of robustness consists of finding the extrema of posterior functionals under Γ. In this paper, we provide a theoretical framework for the treatment of a global robustness problem in the context of hierarchical mixture modeling, where the mixing distribution is a random probability whose law belongs to a generalized moment class Γ. Under suitable conditions on the functions describing the problem, the solution of this latter coincides with the solution of a linear semi-infinite programming problem.