Nonparametric Bayesian Assessment of the Order of Dependence for Binary Sequences

This article discusses inference on the order of dependence in binary sequences. The proposed approach is based on the notion of partial exchangeability of order k. A partially exchangeable binary sequence of order k can be represented as a mixture of Markov chains. The mixture is with respect to the unknown transition probability matrix θ. We use this defining property to construct a semiparametric model for binary sequences by assuming a nonparametric prior on the transition matrix θ. This enables us to consider inference on the order of dependence without constraint to a particular parametric model. Implementing posterior simulation in the proposed model is complicated by the fact that the dimension of θ changes with the order of dependence k. We discuss appropriate posterior simulation schemes based on a pseudo prior approach. We extend the model to include covariates by considering an alternative parameterization as an autologistic regression which allows for a straightforward introduction of covariates. The regression on covariates raises the additional inference problem of variable selection. We discuss appropriate posterior simulation schemes, focusing on inference about the order of dependence. We discuss and develop the model with covariates only to the extent needed for such inference.     

Bayesian Nonparametric Modeling for Multivariate Ordinal Regression

Univariate or multivariate ordinal responses are often assumed to arise from a latent continuous parametric distribution, with covariate effects that enter linearly. We introduce a Bayesian nonparametric modeling approach for univariate and multivariate ordinal regression, which is based on mixture modeling for the joint distribution of latent responses and covariates. The modeling framework enables highly flexible inference for ordinal regression relationships, avoiding assumptions of linearity or additivity in the covariate effects. In standard parametric ordinal regression models, computational challenges arise from identifiability constraints and estimation of parameters requiring nonstandard inferential techniques. A key feature of the nonparametric model is that it achieves inferential flexibility, while avoiding these difficulties. In particular, we establish full support of the nonparametric mixture model under fixed cut-off points that relate through discretization the latent continuous responses with the ordinal responses. The practical utility of the modeling approach is illustrated through application to two datasets from econometrics, an example involving regression relationships for ozone concentration, and a multirater agreement problem. Supplementary materials with technical details on theoretical results and on computation are available online.     

Efficient Bayesian Inference for Multivariate Probit Models With Sparse Inverse Correlation Matrices

We propose a Bayesian approach for inference in the multivariate probit model, taking into account the association structure between binary observations. We model the association through the correlation matrix of the latent Gaussian variables. Conditional independence is imposed by setting some off-diagonal elements of the inverse correlation matrix to zero and this sparsity structure is modeled using a decomposable graphical model. We propose an efficient Markov chain Monte Carlo algorithm relying on a parameter expansion scheme to sample from the resulting posterior distribution. This algorithm updates the correlation matrix within a simple Gibbs sampling framework and allows us to infer the correlation structure from the data, generalizing methods used for inference in decomposable Gaussian graphical models to multivariate binary observations. We demonstrate the performance of this model and of the Markov chain Monte Carlo algorithm on simulated and real datasets. This article has online supplementary materials.     

High-dimensional generation of Bernoulli random vectors

The objective of this paper is to explore different modeling strategies to generate high-dimensional Bernoulli vectors. We discuss the multivariate Bernoulli (MB) distribution, probe its properties and examine three models for generating random vectors. A latent multivariate normal model whose bivariate distributions are approximated with Plackett distributions with univariate normal distributions is presented. A conditional mean model is examined where the conditional probability of success depends on previous history of successes. A mixture of beta distributions is also presented that expresses the probability of the MB vector as a product of correlated binary random variables. Each method has a domain of effectiveness. The latent model offers unpatterned correlation structures while the conditional mean and the mixture model provide computational feasibility for high-dimensional generation of MB vectors.     

变量的有限混合模型对有序数据的Bayes聚类分析

本文基于隐变量的有限混合模型, 提出了一种用于有序数据的Bayes聚类方法\bd 我们采用EM算法获得模型参数的估计, 用BIC准则确定类数, 用类似于Bayes判别的方法对各观测分类\bd 模拟研究结果表明, 本文提出的方法有较好的聚类效果, 对于中等规模的数据集, 计算量是可以接受的.     

On the Bayesian Mixture Model and Identifiability

This article is concerned with Bayesian mixture models and identifiability issues. There are two sources of unidentifiability: the well-known likelihood invariance under label switching and the perhaps less well-known parameter identifiability problem. When using latent allocation variables determined by the mixture model, these sources of unidentifiability create arbitrary labeling that renders estimation of the model very difficult. We endeavor to tackle these problems by proposing a prior distribution on the allocations, which provides an explicit interpretation for the labeling by removing gaps with high probability. We propose a Markov chain Monte Carlo (MCMC) estimation method and present supporting illustrations.     

Bayesian Inference with Wavelets: Density Estimation

Abstract

We propose a prior probability model in the wavelet coefficient space. The proposed model implements wavelet coefficient thresholding by full posterior inference in a coherent probability model. We introduce a prior probability model with mixture priors for the wavelet coefficients. The prior includes a positive prior probability mass at zero which leads to a posteriori thresholding and generally to a posteriori shrinkage on the coefficients. We discuss an efficient posterior simulation scheme to implement inference in the proposed model. The discussion is focused on the density estimation problem. However, the introduced prior probability model on the wavelet coefficient space and the Markov chain Monte Carlo scheme are general.     

Bayesian Inference for the RC(m) Association Model

Describing the structure in a two-way contingency table in terms of an RC(m) association model, we are concerned with the computation of posterior distributions of the model parameters using prior distributions which take into account the nonlinear restrictions of the model. We are further involved with the determination of the order of association m, based on Bayesian arguments. Using projection methods, a prior distribution over the parameters of the simpler RC(m) model is induced from a prior of the parameters of the saturated model. The fit of the assumed RC(m) model is evaluated using the posterior distribution of its distance from the full model. Our methods are illustrated with a popular dataset.     

Sampling Correlation Matrices in Bayesian Models With Correlated Latent Variables

Hierarchical model specifications using latent variables are frequently used to reflect correlation structure in data. Motivated by the structure of a Bayesian multivariate probit model, we demonstrate a parameter-extended Metropolis-Hastings algorithm for sampling from the posterior distribution of a correlation matrix. Our sampling algorithms lead directly to two readily interpretable families of prior distributions for a correlation matrix. The methodology is illustrated through a simulation study and through an application with repeated binary outcomes on individuals from a study of a suicide prevention intervention.     

The Shadow Prior

In this article we consider posterior simulation in models with constrained parameter or sampling spaces. Constraints on the support of sampling and prior distributions give rise to a normalization constant in the complete conditional posterior distribution for the (hyper-) parameters of the respective distribution, complicating posterior simulation.

To mitigate the problem of evaluating normalization constants, we propose a computational approach based on model augmentation. We include an additional level in the probability model to separate the (hyper-) parameter from the constrained probability model, and we refer to this additional level in the probability model as a shadow prior. This approach can significantly reduce the overall computational burden if the original (hyper-) prior includes a complicated structure, but a simple form is chosen for the shadow prior, for example, if the original prior includes a mixture model or multivariate distribution, and the shadow prior defines a set of shadow parameters that are iid given the (hyper-) parameters. Although introducing the shadow prior changes the posterior inference on the original parameters, we argue that by appropriate choices of the shadow prior, the change is minimal and posterior simulation in the augmented probability model provides a meaningful approximation to the desired inference. Data used in this article are available online.     

Joint Modeling of Multiple Network Views

Latent space models (LSM) for network data rely on the basic assumption that each node of the network has an unknown position in a D-dimensional Euclidean latent space: generally the smaller the distance between two nodes in the latent space, the greater their probability of being connected. In this article, we propose a variational inference approach to estimate the intractable posterior of the LSM. In many cases, different network views on the same set of nodes are available. It can therefore be useful to build a model able to jointly summarize the information given by all the network views. For this purpose, we introduce the latent space joint model (LSJM) that merges the information given by multiple network views assuming that the probability of a node being connected with other nodes in each network view is explained by a unique latent variable. This model is demonstrated on the analysis of two datasets: an excerpt of 50 girls from “Teenage Friends and Lifestyle Study” data at three time points and the Saccharomyces cerevisiae genetic and physical protein–protein interactions. Supplementary materials for this article are available online.     

Copula-Type Estimators for Flexible Multivariate Density Modeling Using Mixtures

Copulas are popular as models for multivariate dependence because they allow the marginal densities and the joint dependence to be modeled separately. However, they usually require that the transformation from uniform marginals to the marginals of the joint dependence structure is known. This can only be done for a restricted set of copulas, for example, a normal copula. Our article introduces copula-type estimators for flexible multivariate density estimation which also allow the marginal densities to be modeled separately from the joint dependence, as in copula modeling, but overcomes the lack of flexibility of most popular copula estimators. An iterative scheme is proposed for estimating copula-type estimators and its usefulness is demonstrated through simulation and real examples. The joint dependence is modeled by mixture of normals and mixture of normal factor analyzer models, and mixture of t and mixture of t-factor analyzer models. We develop efficient variational Bayes algorithms for fitting these in which model selection is performed automatically. Based on these mixture models, we construct four classes of copula-type densities which are far more flexible than current popular copula densities, and outperform them in a simulated dataset and several real datasets. Supplementary material for this article is available online.     

Covariance Decompositions for Accurate Computation in Bayesian Scale-Usage Models

Analyses of multivariate ordinal probit models typically use data augmentation to link the observed (discrete) data to latent (continuous) data via a censoring mechanism defined by a collection of “cutpoints.” Most standard models, for which effective Markov chain Monte Carlo (MCMC) sampling algorithms have been developed, use a separate (and independent) set of cutpoints for each element of the multivariate response. Motivated by the analysis of ratings data, we describe a particular class of multivariate ordinal probit models where it is desirable to use a common set of cutpoints. While this approach is attractive from a data-analytic perspective, we show that the existing efficient MCMC algorithms can no longer be accurately applied. Moreover, we show that attempts to implement these algorithms by numerically approximating required multivariate normal integrals over high-dimensional rectangular regions can result in severely degraded estimates of the posterior distribution. We propose a new data augmentation that is based on a covariance decomposition and that admits a simple and accurate MCMC algorithm. Our data augmentation requires only that univariate normal integrals be evaluated, which can be done quickly and with high accuracy. We provide theoretical results that suggest optimal decompositions within this class of data augmentations, and, based on the theory, recommend default decompositions that we demonstrate work well in practice. This article has supplementary material online.     

Model-based two-way clustering of second-level units in ordinal multilevel latent Markov models

In this paper, an ordinal multilevel latent Markov model based on separate random effects is proposed. In detail, two distinct second-level discrete effects are considered in the model, one affecting the initial probability vector and the other affecting the transition probability matrix of the first-level ordinal latent Markov process. To model these separate effects, we consider a bi-dimensional mixture specification that allows to avoid unverifiable assumptions on the random effect distribution and to derive a two-way clustering of second-level units. Starting from a general model where the two random effects are dependent, we also obtain the independence model as a special case. The proposal is applied to data on the physical health status of a sample of elderly residents grouped into nursing homes. A simulation study assessing the performance of the proposal is also included.

     

Inference for the Number of Topics in the Latent Dirichlet Allocation Model via Bayesian Mixture Modeling

In latent Dirichlet allocation, the number of topics, T, is a hyperparameter of the model that must be specified before one can fit the model. The need to specify T in advance is restrictive. One way of dealing with this problem is to put a prior on T, but unfortunately the distribution on the latent variables of the model is then a mixture of distributions on spaces of different dimensions, and estimating this mixture distribution by Markov chain Monte Carlo is very difficult. We present a variant of the Metropolis–Hastings algorithm that can be used to estimate this mixture distribution, and in particular the posterior distribution of the number of topics. We evaluate our methodology on synthetic data and compare it with procedures that are currently used in the machine learning literature. We also give an illustration on two collections of articles from Wikipedia. Supplemental materials for this article are available online.     

Non-parametric clustering over user features and latent behavioral functions with dual-view mixture models

We present a dual-view mixture model to cluster users based on their features and latent behavioral functions. Every component of the mixture model represents a probability density over a feature view for observed user attributes and a behavior view for latent behavioral functions that are indirectly observed through user actions or behaviors. Our task is to infer the groups of users as well as their latent behavioral functions. We also propose a non-parametric version based on a Dirichlet Process to automatically infer the number of clusters. We test the properties and performance of the model on a synthetic dataset that represents the participation of users in the threads of an online forum. Experiments show that dual-view models outperform single-view ones when one of the views lacks information.     

Interval estimates for posterior probabilities in a multivariate normal classification model

This paper is devoted to the asymptotic distribution of estimators for the posterior probability that a p-dimensional observation vector originates from one of k normal distributions with identical covariance matrices. The estimators are based on training samples for the k distributions involved. Observation vector and prior probabilities are regarded as given constants. The validity of various estimators and approximate confidence intervals is investigated by simulation experiments.     

A latent variables approach for clustering mixed binary and continuous variables within a Gaussian mixture model

For clustering objects, we often collect not only continuous variables, but binary attributes as well. This paper proposes a model-based clustering approach with mixed binary and continuous variables where each binary attribute is generated by a latent continuous variable that is dichotomized with a suitable threshold value, and where the scores of the latent variables are estimated from the binary data. In economics, such variables are called utility functions and the assumption is that the binary attributes (the presence or the absence of a public service or utility) are determined by low and high values of these functions. In genetics, the latent response is interpreted as the ??liability?? to develop a qualitative trait or phenotype. The estimated scores of the latent variables, together with the observed continuous ones, allow to use a multivariate Gaussian mixture model for clustering, instead of using a mixture of discrete and continuous distributions. After describing the method, this paper presents the results of both simulated and real-case data and compares the performances of the multivariate Gaussian mixture model and of a mixture of joint multivariate and multinomial distributions. Results show that the former model outperforms the mixture model for variables with different scales, both in terms of classification error rate and reproduction of the clusters means.     

Graphical Models for Ordinal Data

This article considers a graphical model for ordinal variables, where it is assumed that the data are generated by discretizing the marginal distributions of a latent multivariate Gaussian distribution. The relationships between these ordinal variables are then described by the underlying Gaussian graphical model and can be inferred by estimating the corresponding concentration matrix. Direct estimation of the model is computationally expensive, but an approximate EM-like algorithm is developed to provide an accurate estimate of the parameters at a fraction of the computational cost. Numerical evidence based on simulation studies shows the strong performance of the algorithm, which is also illustrated on datasets on movie ratings and an educational survey.     

Bayesian Estimation of Discrete Multivariate Latent Structure Models With Structural Zeros

In multivariate categorical data, models based on conditional independence assumptions, such as latent class models, offer efficient estimation of complex dependencies. However, Bayesian versions of latent structure models for categorical data typically do not appropriately handle impossible combinations of variables, also known as structural zeros. Allowing nonzero probability for impossible combinations results in inaccurate estimates of joint and conditional probabilities, even for feasible combinations. We present an approach for estimating posterior distributions in Bayesian latent structure models with potentially many structural zeros. The basic idea is to treat the observed data as a truncated sample from an augmented dataset, thereby allowing us to exploit the conditional independence assumptions for computational expediency. As part of the approach, we develop an algorithm for collapsing a large set of structural zero combinations into a much smaller set of disjoint marginal conditions, which speeds up computation. We apply the approach to sample from a semiparametric version of the latent class model with structural zeros in the context of a key issue faced by national statistical agencies seeking to disseminate confidential data to the public: estimating the number of records in a sample that are unique in the population on a set of publicly available categorical variables. The latent class model offers remarkably accurate estimates of population uniqueness, even in the presence of a large number of structural zeros.