Approximate Predetermined Convergence Properties of the Gibbs Sampler

This article aims to provide a method for approximately predetermining convergence properties of the Gibbs sampler. This is to be done by first finding an approximate rate of convergence for a normal approximation of the target distribution. The rates of convergence for different implementation strategies of the Gibbs sampler are compared to find the best one. In general, the limiting convergence properties of the Gibbs sampler on a sequence of target distributions (approaching a limit) are not the same as the convergence properties of the Gibbs sampler on the limiting target distribution. Theoretical results are given in this article to justify that under conditions, the convergence properties of the Gibbs sampler can be approximated as well. A number of practical examples are given for illustration.     

Convergence rates of the blocked Gibbs sampler with random scan in the Wasserstein metric

ABSTRACT

This paper establishes explicit estimates of convergence rates for the blocked Gibbs sampler with random scan under the Dobrushin conditions. The estimates of convergence in the Wasserstein metric are obtained by taking purely analytic approaches.     

Convergence rates of symmetric scan Gibbs sampler

We consider the symmetric scan Gibbs sampler, and give some explicit estimates of convergence rates on the Wasserstein distance for this Markov chain Monte Carlo under the Dobrushin uniqueness condition.     

Approximate source conditions in Tikhonov–Phillips regularization and consequences for inverse problems with multiplication operators

The object of this paper is threefold. First, we investigate in a Hilbert space setting the utility of approximate source conditions in the method of Tikhonov–Phillips regularization for linear ill‐posed operator equations. We introduce distance functions measuring the violation of canonical source conditions and derive convergence rates for regularized solutions based on those functions. Moreover, such distance functions are verified for simple multiplication operators in L²(0, 1). The second aim of this paper is to emphasize that multiplication operators play some interesting role in inverse problem theory. In this context, we give examples of non‐linear inverse problems in natural sciences and stochastic finance that can be written as non‐linear operator equations in L²(0, 1), for which the forward operator is a composition of a linear integration operator and a non‐linear superposition operator. The Fréchet derivative of such a forward operator is a composition of a compact integration and a non‐compact multiplication operator. If the multiplier function defining the multiplication operator has zeros, then for the linearization an additional ill‐posedness factor arises. By considering the structure of canonical source conditions for the linearized problem it could be expected that different decay rates of multiplier functions near a zero, for example the decay as a power or as an exponential function, would lead to completely different ill‐posedness situations. As third we apply the results on approximate source conditions to such composite linear problems in L²(0, 1) and indicate that only integrals of multiplier functions and not the specific character of the decay of multiplier functions in a neighbourhood of a zero determine the convergence behaviour of regularized solutions. Copyright © 2005 John Wiley & Sons, Ltd.     

Optimizing random scan Gibbs samplers

The Gibbs sampler is a popular Markov chain Monte Carlo routine for generating random variates from distributions otherwise difficult to sample. A number of implementations are available for running a Gibbs sampler varying in the order through which the full conditional distributions used by the Gibbs sampler are cycled or visited. A common, and in fact the original, implementation is the random scan strategy, whereby the full conditional distributions are updated in a randomly selected order each iteration. In this paper, we introduce a random scan Gibbs sampler which adaptively updates the selection probabilities or “learns” from all previous random variates generated during the Gibbs sampling. In the process, we outline a number of variations on the random scan Gibbs sampler which allows the practitioner many choices for setting the selection probabilities and prove convergence of the induced (Markov) chain to the stationary distribution of interest. Though we emphasize flexibility in user choice and specification of these random scan algorithms, we present a minimax random scan which determines the selection probabilities through decision theoretic considerations on the precision of estimators of interest. We illustrate and apply the results presented by using the adaptive random scan Gibbs sampler developed to sample from multivariate Gaussian target distributions, to automate samplers for posterior simulation under Dirichlet process mixture models, and to fit mixtures of distributions.     

Performance of the Gibbs,Hit-and-Run,and Metropolis Samplers

Abstract

We consider the performance of three Monte Carlo Markov-chain samplers—the Gibbs sampler, which cycles through coordinate directions; the Hit-and-Run (H&R) sampler, which randomly moves in any direction; and the Metropolis sampler, which moves with a probability that is a ratio of likelihoods. We obtain several analytical results. We provide a sufficient condition of the geometric convergence on a bounded region S for the H&R sampler. For a general region S, we review the Schervish and Carlin sufficient geometric convergence condition for the Gibbs sampler. We show that for a multivariate normal distribution this Gibbs sufficient condition holds and for a bivariate normal distribution the Gibbs marginal sample paths are each an AR(1) process, and we obtain the standard errors of sample means and sample variances, which we later use to verify empirical Monte Carlo results. We empirically compare the Gibbs and H&R samplers on bivariate normal examples. For zero correlation, the Gibbs sampler provides independent data, resulting in better performance than H&R. As the absolute value of the correlation increases, H&R performance improves, with H&R substantially better for correlations above .9. We also suggest and study methods for choosing the number of replications, for estimating the standard error of point estimators and for reducing point-estimator variance. We suggest using a single long run instead of using multiple iid separate runs. We suggest using overlapping batch statistics (obs) to get the standard errors of estimates; additional empirical results show that obs is accurate. Finally, we review the geometric convergence of the Metropolis algorithm and develop a Metropolisized H&R sampler. This sampler works well for high-dimensional and complicated integrands or Bayesian posterior densities.     

Range Inclusions and Approximate Source Conditions with General Benchmark Functions

The paper is devoted to the analysis of ill-posed operator equations Ax = y with injective linear operator A and solution x ₀ in a Hilbert space setting. We present some new ideas and results for finding convergence rates in Tikhonov regularization based on the concept of approximate source conditions by means of using distance functions with a general benchmark. For the case of compact operator A and benchmark functions of power-type, we can show that there is a one-to-one correspondence between the maximal power-type decay rate of the distance function and the best possible Hölder exponent for the noise-free convergence rate in Tikhonov regularization. As is well-known, this exponent coincides with the supremum of exponents in power-type source conditions. The main theorem of this paper is devoted to the impact of range inclusions under the smoothness assumption that x ₀ is in the range of some positive self-adjoint operator G. It generalizes a convergence rate result proven for compact G in Hofmann and Yamamoto (Inverse Problems 2005; 21:805–820) to the case of general operators G with nonclosed range.     

Divergence L 2-Coercivity Inequalities

This paper derives sharp L ²-coercivity inequalities for the divergence operator on bounded Lipschitz regions in ? ⁿ . They hold for fields in H(div,Ω) that are orthogonal to N(div). The optimal constants in the inequality are defined by a variational principle and are identified as the least eigenvalue of a nonstandard boundary value problem for a linear biharmonic type operator. The dependence of the optimal constant under dilations of the region is described and a generalization that involves weighted surface integrals is also proved. When n = 2, this also yields a similar coercivity result for the curl operator.     

Positive reynolds operators and generating derivations

It is shown that the spectrum of a positive Reynolds operator on C₀(X) is contained in the disc centered at 1/2 with radius 1/2. Moreover, every positive Reynolds operator T with dense range is injective. In this case, the operator D = 1 — T^?1 is a densely defined derivation, which generates a one — parameter semigroup of algebra homomorphisms. This semigroup yields an integral representation of T. Along the way, it is proved that a densely defined closed derivation D generates a semigroup if, and only if, R(1, D) exists and is a positive operator.     

Implementing random scan Gibbs samplers

Summary  The Gibbs sampler, being a popular routine amongst Markov chain Monte Carlo sampling methodologies, has revolutionized the application of Monte Carlo methods in statistical computing practice. The performance of the Gibbs sampler relies heavily on the choice of sweep strategy, that is, the means by which the components or blocks of the random vector X of interest are visited and updated. We develop an automated, adaptive algorithm for implementing the optimal sweep strategy as the Gibbs sampler traverses the sample space. The decision rules through which this strategy is chosen are based on convergence properties of the induced chain and precision of statistical inferences drawn from the generated Monte Carlo samples. As part of the development, we analytically derive closed form expressions for the decision criteria of interest and present computationally feasible implementations of the adaptive random scan Gibbs sampler via a Gaussian approximation to the target distribution. We illustrate the results and algorithms presented by using the adaptive random scan Gibbs sampler developed to sample multivariate Gaussian target distributions, and screening test and image data. Research by RL and ZY supported in part by a US National Science Foundation FRG grant 0139948 and a grant from Lawrence Livermore National Laboratory, Livermore, California, USA.     

On the Convergence Complexity of Gibbs Samplers for a Family of Simple Bayesian Random Effects Models

The emergence of big data has led to so-called convergence complexity analysis, which is the study of how Markov chain Monte Carlo (MCMC) algorithms behave as the sample size, n, and/or the number of parameters, p, in the underlying data set increase. This type of analysis is often quite challenging, in part because existing results for fixed n and p are simply not sharp enough to yield good asymptotic results. One of the first convergence complexity results for an MCMC algorithm on a continuous state space is due to Yang and Rosenthal (2019), who established a mixing time result for a Gibbs sampler (for a simple Bayesian random effects model) that was introduced and studied by Rosenthal (Stat Comput 6:269–275, 1996). The asymptotic behavior of the spectral gap of this Gibbs sampler is, however, still unknown. We use a recently developed simulation technique (Qin et al. Electron J Stat 13:1790–1812, 2019) to provide substantial numerical evidence that the gap is bounded away from 0 as n → ∞. We also establish a pair of rigorous convergence complexity results for two different Gibbs samplers associated with a generalization of the random effects model considered by Rosenthal (Stat Comput 6:269–275, 1996). Our results show that, under a strong growth condition, the spectral gaps of these Gibbs samplers converge to 1 as the sample size increases.

     

Efficient algorithms for generating truncated multivariate normal distributions

Sampling from a truncated multivariate normal distribution (TMVND) constitutes the core computational module in fitting many statistical and econometric models. We propose two efficient methods, an iterative data augmentation (DA) algorithm and a non-iterative inverse Bayes formulae (IBF) sampler, to simulate TMVND and generalize them to multivariate normal distributions with linear inequality constraints. By creating a Bayesian incomplete-data structure, the posterior step of the DA algorithm directly generates random vector draws as opposed to single element draws, resulting obvious computational advantage and easy coding with common statistical software packages such as S-PLUS, MATLAB and GAUSS. Furthermore, the DA provides a ready structure for implementing a fast EM algorithm to identify the mode of TMVND, which has many potential applications in statistical inference of constrained parameter problems. In addition, utilizing this mode as an intermediate result, the IBF sampling provides a novel alternative to Gibbs sampling and eliminates problems with convergence and possible slow convergence due to the high correlation between components of a TMVND. The DA algorithm is applied to a linear regression model with constrained parameters and is illustrated with a published data set. Numerical comparisons show that the proposed DA algorithm and IBF sampler are more efficient than the Gibbs sampler and the accept-reject algorithm.     

Dynamical Systems Method of gradient type for solving nonlinear equations with monotone operators

A version of the Dynamical Systems Method (DSM) of gradient type for solving equation F(u)=f where F:H→H is a monotone Fréchet differentiable operator in a Hilbert space H is studied in this paper. A discrepancy principle is proposed and the convergence to the minimal-norm solution is justified. Based on the DSM an iterative scheme is formulated and the convergence of this scheme to the minimal-norm solution is proved.     

Estimating a density and its derivatives via the minimum distance method

Summary This paper deals with minimum distance (MD) estimators and minimum penalized distance (MPD) estimators which are based on the L _p distance. Rates of strong consistency of MPD density estimators are established within the family of density functions which have a bounded m-th derivative. For the case p=2, it is also proved that the MPD density estimator achieves the optimum rate of decrease of the mean integrated square error and the L ₁ error. Estimation of derivatives of the density is considered as well.In a class parametrized by entire functions, it is proved that the rate of convergence of the MD density estimator (and its derivatives) to the unknown density (its derivatives) is of order in expected L ₁ and L ₂ distances. In the same class of distributions, MD estimators of unknown density and its derivatives are proved to achieve an extraordinary rate (log log n/n)^1/2 of strong consistency.     

Rate of convergence of the mean curvature flow

We study the flow M_t of a smooth, strictly convex hypersurface by its mean curvature in ?^{n + 1}. The surface remains smooth and convex, shrinking monotonically until it disappears at a critical time T and point x^* (which is due to Huisken). This is equivalent to saying that the corresponding rescaled mean curvature flow converges to a sphere Sⁿ of radius √n. In this paper we will study the rate of exponential convergence of a rescaled flow. We will present here a method that tells us that the rate of the exponential decay is at least 2/n. We can define the “arrival time” u of a smooth, strictly convex, n‐dimensional hypersurface as it moves with normal velocity equal to its mean curvature via u(x) = t if x ∈ M_t for x ∈ Int(M₀). Huisken proved that, for n ≥ 2, u(x) is C² near x^*. The case n = 1 has been treated by Kohn and Serfaty [11]; they proved C³‐regularity of u. As a consequence of the obtained rate of convergence of the mean curvature flow, we prove that u is not necessarily C³ near x^* for n ≥ 2. We also show that the obtained rate of convergence 2/n, which arises from linearizing a mean curvature flow, is the optimal one, at least for n ≥ 2. © 2007 Wiley Periodicals, Inc.     

Operator theory in the Hardy space over the bidisk (II)

This paper is a continuation of a project of developing a systematic operator theory inH ²(D ²). A large part of it is devoted to a study ofevaluation operator which is a very useful tool in the theory. A number of elementary properties of the evaluation operator are exhibited, and these properties are used to derive results in other topics such as interpretation of characteristic opertor function inH ²(D ²), spectral equivalence, compactness, compressions of shift operators, etc., Even though some results reflect the two variable nature ofH ²(D ²), the goal of this paper is to manifest a close tie between the operator theory inH ²(D ²) and classical single operator theory. The unilateral shift of a finite multiplicity and the Bergman shift will be used as examples to illustrate some of the results.Research in this paper is partially supported by a grant from the national science foundation DMS 9970932.     

The Pseudo-differential Operator hμ,a on Hankel Invariant Spaces

Using Hankel transform the symbol 'a' is defined and the pseudo-differential operator (p.d.o.) h_μ,a associated with the Bessel operator d ²/dx ² + (1 ? 4μ ²)/4x ² in terms of this symbol is defined. It is shown that the operator h_μ,a is a continuous linear map of a Hankel invariant space into itself. A special pseudo-differential operator called the Hankel potential is defined and some of its properties are investigated.     

A Sharp Jackson Inequality in Lp(ℝd) with Dunkl Weight

A sharp Jackson inequality in the space L_p(?^d), 1 ≤ p < 2, with Dunkl weight is proved. The best approximation is realized by entire functions of exponential spherical type. The modulus of continuity is defined by means of a generalized shift operator bounded on L_p, which was constructed earlier by the authors. In the case of the unit weight, this operator coincides with the mean-value operator on the sphere.

     

Weyl Pseudo-Differential Operator and Wigner Transform on the Poincaré Disk

The purpose of this paper is to investigate some relations between the kernel of a Weyl pseudo-differential operator and the Wigner transform on Poincaré disk defined in our previous paper [11]. The composition formula for the class of the operators defined in [11] has not been proved yet. However, some properties and relations, which are analogous to the Euclidean case, between the Weyl pseudo-differential operator and the Wigner transform have been investigated in [11]. In the present paper, an asymptotic formula for the Wigner transform of the kernel of a Weyl pseudo-differential operator as 0 is given. We also introduce a space of functions on the cotangent bundle T ^* D whose definition is based on the notion of the Schwartz space on the Poincaré disk. For an S ¹-invariant symbol in that space, we obtain a formula to reproduce the symbol from the kernel of the Weyl pseudo-differential operator.     

On the injectivity of the Pompeiu transform for integral ball means

A uniqueness theorem is proved for functions defined in \mathbb Rⁿ,   n 3 2 {{\mathbb R}^n}, \; {n \geq 2} , with vanishing integrals over the balls of fixed radius and a given majorant of growth. The problem of unimprovability of this theorem is analyzed.