Approximate Predetermined Convergence Properties of the Gibbs Sampler

This article aims to provide a method for approximately predetermining convergence properties of the Gibbs sampler. This is to be done by first finding an approximate rate of convergence for a normal approximation of the target distribution. The rates of convergence for different implementation strategies of the Gibbs sampler are compared to find the best one. In general, the limiting convergence properties of the Gibbs sampler on a sequence of target distributions (approaching a limit) are not the same as the convergence properties of the Gibbs sampler on the limiting target distribution. Theoretical results are given in this article to justify that under conditions, the convergence properties of the Gibbs sampler can be approximated as well. A number of practical examples are given for illustration.     

Convergence rate of gibbs sampler and its application

Based on the convergence rate defined by the Pearson-χ~2 distance,this pa- per discusses properties of different Gibbs sampling schemes.Under a set of regularity conditions,it is proved in this paper that the rate of convergence on systematic scan Gibbs samplers is the norm of a forward operator.We also discuss that the collapsed Gibbs sam- pler has a faster convergence rate than the systematic scan Gibbs sampler as proposed by Liu et al.Based on the definition of convergence rate of the Pearson-χ~2 distance, this paper proved this result quantitatively.According to Theorem 2,we also proved that the convergence rate defined with the spectral radius of matrix by Robert and Shau is equivalent to the corresponding radius of the forward operator.     

Metropolis-Hastings Within Partially Collapsed Gibbs Samplers

The partially collapsed Gibbs (PCG) sampler offers a new strategy for improving the convergence of a Gibbs sampler. PCG achieves faster convergence by reducing the conditioning in some of the draws of its parent Gibbs sampler. Although this can significantly improve convergence, care must be taken to ensure that the stationary distribution is preserved. The conditional distributions sampled in a PCG sampler may be incompatible and permuting their order may upset the stationary distribution of the chain. Extra care must be taken when Metropolis-Hastings (MH) updates are used in some or all of the updates. Reducing the conditioning in an MH within Gibbs sampler can change the stationary distribution, even when the PCG sampler would work perfectly if MH were not used. In fact, a number of samplers of this sort that have been advocated in the literature do not actually have the target stationary distributions. In this article, we illustrate the challenges that may arise when using MH within a PCG sampler and develop a general strategy for using such updates while maintaining the desired stationary distribution. Theoretical arguments provide guidance when choosing between different MH within PCG sampling schemes. Finally, we illustrate the MH within PCG sampler and its computational advantage using several examples from our applied work.     

Convergence rates of the blocked Gibbs sampler with random scan in the Wasserstein metric

ABSTRACT

This paper establishes explicit estimates of convergence rates for the blocked Gibbs sampler with random scan under the Dobrushin conditions. The estimates of convergence in the Wasserstein metric are obtained by taking purely analytic approaches.     

Multifractal analysis and the variance of Gibbs measures

The multifractal decomposition of Gibbs measures for a conformaliterated function system is well known. We look at a finer decompositionwhich also takes into account the rate of convergence. Thisis motivated by the work of Olsen in the self-similar case.Our study of this finer decomposition involves investigationof the variance of Gibbs measures. This is a problem of independentinterest.     

Convergence rates of symmetric scan Gibbs sampler

We consider the symmetric scan Gibbs sampler, and give some explicit estimates of convergence rates on the Wasserstein distance for this Markov chain Monte Carlo under the Dobrushin uniqueness condition.     

From metropolis to diffusions: Gibbs states and optimal scaling

This paper investigates the behaviour of the random walk Metropolis algorithm in high-dimensional problems. Here we concentrate on the case where the components in the target density is a spatially homogeneous Gibbs distribution with finite range. The performance of the algorithm is strongly linked to the presence or absence of phase transition for the Gibbs distribution; the convergence time being approximately linear in dimension for problems where phase transition is not present. Related to this, there is an optimal way to scale the variance of the proposal distribution in order to maximise the speed of convergence of the algorithm. This turns out to involve scaling the variance of the proposal as the reciprocal of dimension (at least in the phase transition-free case). Moreover, the actual optimal scaling can be characterised in terms of the overall acceptance rate of the algorithm, the maximising value being 0.234, the value as predicted by studies on simpler classes of target density. The results are proved in the framework of a weak convergence result, which shows that the algorithm actually behaves like an infinite-dimensional diffusion process in high dimensions.     

ASIS�㷨�Ƿ�Ӧ�ù㷺����?

本文将辅助--充分交织策略, 即Yu和Meng (2011)中提到的ASIS算法, 应用于Gibbs抽样算法中以提高两个方差参数的收敛性. 我们通过对潜在规模缩减因子(PSRF)、轨迹图及后验估计 比较了ASIS算法与普通Gibbs抽样算法的性能, 其中一个参数的收敛性有了很大的提高, 但另一个参数没有很明显的提高. 然而, 由于ASIS算法相与普通的Gibbs抽样算法相比极大地减少了为达到收敛所 需要的循环次数, 整体的抽样性能得到了极大的提高.     

On the convergence rate of Fourier series

The rate of convergence of a Fourier‐series representation of a given function depends on the nature of the function and of its derivatives. This is shown by using the graphical outputs of a desk computer for different cases. For full‐range series, the effects of continuity and discontinuity of the function and its first derivatives are shown first. The advantage of half‐range formulae due to the free choice of function in the second half‐range are demonstrated next, along with the importance of choice of sine and cosine series according to the function being represented. Finally, a method is given of modifying the given function in a simple manner whereby a dramatic increase in the rate of convergence of the Fourier series is obtained. Examples of the Gibbs overshoot and its elimination are included.     

Primal-dual interior-point method for thermodynamic gas-particle partitioning

A mathematical model for the computation of the phase equilibrium and gas-particle partitioning in atmospheric organic aerosols is presented. The thermodynamic equilibrium is determined by the global minimum of the Gibbs free energy under equality and inequality constraints for a system that involves one gas phase and many liquid phases. A primal-dual interior-point algorithm is presented for the efficient solution of the phase equilibrium problem and the determination of the active constraints. The first order optimality conditions are solved with a Newton iteration. Sequential quadratic programming techniques are incorporated to decouple the different scales of the problem. Decomposition methods that control the inertia of the matrices arising in the resolution of the Newton system are proposed. A least-squares initialization of the algorithm is proposed to favor the convergence to a global minimum of the Gibbs free energy. Numerical results show the efficiency of the approach for the prediction of gas-liquid-liquid equilibrium for atmospheric organic aerosol particles.     

On Polymer Expansions for Equilibrium Systems of Oscillators with Ternary Interaction

For Gibbs lattice systems characterized by a measurable space at sites of a d-dimensional hypercubic lattice and potential energy with pair complex potential, we formulate conditions that guarantee the convergence of polymer (cluster) expansions. We establish that the Gibbs correlation functions and reduced density matrices of classical and quantum systems of linear oscillators with ternary interaction can be expressed in terms of correlation functions of these systems.     

Limits of discrete distributions and Gibbs measures on random graphs

Building upon the theory of graph limits and the Aldous–Hoover representation and inspired by Panchenko’s work on asymptotic Gibbs measures [Annals of Probability 2013], we construct continuous embeddings of discrete probability distributions. We show that the theory of graph limits induces a meaningful notion of convergence and derive a corresponding version of the Szemerédi regularity lemma. Moreover, complementing recent work Bapst et al. (2015), we apply these results to Gibbs measures induced by sparse random factor graphs and verify the “replica symmetric solution” predicted in the physics literature under the assumption of non-reconstruction.     

Optimizing random scan Gibbs samplers

The Gibbs sampler is a popular Markov chain Monte Carlo routine for generating random variates from distributions otherwise difficult to sample. A number of implementations are available for running a Gibbs sampler varying in the order through which the full conditional distributions used by the Gibbs sampler are cycled or visited. A common, and in fact the original, implementation is the random scan strategy, whereby the full conditional distributions are updated in a randomly selected order each iteration. In this paper, we introduce a random scan Gibbs sampler which adaptively updates the selection probabilities or “learns” from all previous random variates generated during the Gibbs sampling. In the process, we outline a number of variations on the random scan Gibbs sampler which allows the practitioner many choices for setting the selection probabilities and prove convergence of the induced (Markov) chain to the stationary distribution of interest. Though we emphasize flexibility in user choice and specification of these random scan algorithms, we present a minimax random scan which determines the selection probabilities through decision theoretic considerations on the precision of estimators of interest. We illustrate and apply the results presented by using the adaptive random scan Gibbs sampler developed to sample from multivariate Gaussian target distributions, to automate samplers for posterior simulation under Dirichlet process mixture models, and to fit mixtures of distributions.     

Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

Gibbs Phenomenon in Wavelet Analysis

We prove Dirichlet-type pointwise convergence theorems for the wavelet transforms and series of discontinuous functions and we examine the Gibbs ripples close to the jump location. Examples are given of wavelets without ripples, and an example (the Mexican hat) shows that the Gibbs ripple in continuous wavelet analysis can be 3.54% instead of 8.9% of the Fourier case. For the discrete case we show that there exist two Meyer type wavelets the first one has maximum ripple 3.58% and the second 9.8%. Moreover we describe several examples and methods for estimating Gibbs ripples both in continuous and discrete cases. Finally we discuss how a wavelet transform generates a summability method for the Fourier case.     

Gibbs Sampling Will Fail in Outlier Problems with Strong Masking

Abstract

This article discusses the convergence of the Gibbs sampling algorithm when it is applied to the problem of outlier detection in regression models. Given any vector of initial conditions, theoretically, the algorithm converges to the true posterior distribution. However, the speed of convergence may slow down in a high-dimensional parameter space where the parameters are highly correlated. We show that the effect of the leverage in regression models makes very difficult the convergence of the Gibbs sampling algorithm in sets of data with strong masking. The problem is illustrated with examples.     

Linex损失及PA样本下单边截断型分布族参数函数的EB估计

在Linex损失函数下,运用同分布PA样本密度函数的核估计方法,构造了一类单边截断型分布族参数函数的EB估计,并建立了它的收敛速度.在一定条件下,证明了这个收敛速度可充分接近1.     

Digital total variation filtering as postprocessing for Chebyshev pseudospectral methods for conservation laws

Digital total variation filtering is analyzed as a fast, robust, post-processing method for accelerating the convergence of pseudospectral approximations that have been contaminated by Gibbs oscillations. The method, which originated in image processing, can be combined with spectral filters to quickly post-process large data sets with sharp resolution of discontinuities and with exponential accuracy away from the discontinuities.     

渐近循环马氏链的收敛速度

引入了渐近循环马氏链的概念,它是循环马氏链概念的推广.首先研究了在强遍历的条件下,可列循环马氏链的收敛速度,作为主要结论给出了当满足不同条件时可列渐近循环马氏链的C-强遍历性,一致C-强遍历性和一致C-强遍历的收敛速度     

Performance of the Gibbs,Hit-and-Run,and Metropolis Samplers

Abstract

We consider the performance of three Monte Carlo Markov-chain samplers—the Gibbs sampler, which cycles through coordinate directions; the Hit-and-Run (H&R) sampler, which randomly moves in any direction; and the Metropolis sampler, which moves with a probability that is a ratio of likelihoods. We obtain several analytical results. We provide a sufficient condition of the geometric convergence on a bounded region S for the H&R sampler. For a general region S, we review the Schervish and Carlin sufficient geometric convergence condition for the Gibbs sampler. We show that for a multivariate normal distribution this Gibbs sufficient condition holds and for a bivariate normal distribution the Gibbs marginal sample paths are each an AR(1) process, and we obtain the standard errors of sample means and sample variances, which we later use to verify empirical Monte Carlo results. We empirically compare the Gibbs and H&R samplers on bivariate normal examples. For zero correlation, the Gibbs sampler provides independent data, resulting in better performance than H&R. As the absolute value of the correlation increases, H&R performance improves, with H&R substantially better for correlations above .9. We also suggest and study methods for choosing the number of replications, for estimating the standard error of point estimators and for reducing point-estimator variance. We suggest using a single long run instead of using multiple iid separate runs. We suggest using overlapping batch statistics (obs) to get the standard errors of estimates; additional empirical results show that obs is accurate. Finally, we review the geometric convergence of the Metropolis algorithm and develop a Metropolisized H&R sampler. This sampler works well for high-dimensional and complicated integrands or Bayesian posterior densities.