On geometric and algebraic transience for discrete-time Markov chains

General characterizations of ergodic Markov chains have been developed in considerable detail. In this paper, we study the transience for discrete-time Markov chains on general state spaces, including the geometric transience and algebraic transience. Criteria are presented through bounding the modified moment of the first return time and establishing the appropriate drift condition. Moreover, we apply the criteria to the random walk on the half line and the skip-free chain on nonnegative integers.     

An invariance principle for non-symmetric Markov processes and reflecting diffusions in random domains

Summary We study an invariance principle for additive functionals of nonsymmetric Markov processes with singular mean forward velocities. We generalize results of Kipnis and Varadhan [KV] and De Masi et al. [De] in two directions: Markov processes are non-symmetric, and mean forward velocities are distributions. We study continuous time Markov processes. We use our result to homogenize non-symmetric reflecting diffusions in random domains.     

Discretisation of stochastic control problems for continuous time dynamics with delay

As a main step in the numerical solution of control problems in continuous time, the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space. A new feature in this context is to allow for delay in the dynamics. The existence of an optimal strategy with respect to the cost functional can be guaranteed in the class of relaxed controls. Weak convergence of the approximating extended Markov chains to the original process together with convergence of the associated optimal strategies is established.     

Interaction information and Markov chain

The problem of multivariate information analysis is considered. First, the interaction information in each dimension is defined analogously according to McGill [4] and then applied to Markov chains. The property of interaction information zero deeply relates to a certain class of weakly dependent random variables. For homogeneous, recurrent Markov chains with m states, m ≥ n ≥3, the zero criterion of n-dimensional interaction information is achieved only by (n ? 2)-dependent Markov chains, which are generated by some nilpotent matrices. Further for Gaussian Markov chains, it gives the decomposition rule of the variables into mutually correlated subchains.     

New perturbation bounds for denumerable Markov chains

This paper is devoted to perturbation analysis of denumerable Markov chains. Bounds are provided for the deviation between the stationary distribution of the perturbed and nominal chain, where the bounds are given by the weighted supremum norm. In addition, bounds for the perturbed stationary probabilities are established. Furthermore, bounds on the norm of the asymptotic decomposition of the perturbed stationary distribution are provided, where the bounds are expressed in terms of the norm of the ergodicity coefficient, or the norm of a special residual matrix. Refinements of our bounds for Doeblin Markov chains are considered as well. Our results are illustrated with a number of examples.     

A classification of invariant distributions and convergence of imprecise Markov chains

We analyse the structure of imprecise Markov chains and study their convergence by means of accessibility relations. We first identify the sets of states, so-called minimal permanent classes, that are the minimal sets capable of containing and preserving the whole probability mass of the chain. These classes generalise the essential classes known from the classical theory. We then define a class of extremal imprecise invariant distributions and show that they are uniquely determined by the values of the upper probability on minimal permanent classes. Moreover, we give conditions for unique convergence to these extremal invariant distributions.     

Imprecise Markov chains with absorption

We consider convergence of Markov chains with uncertain parameters, known as imprecise Markov chains, which contain an absorbing state. We prove that under conditioning on non-absorption the imprecise conditional probabilities converge independently of the initial imprecise probability distribution if some regularity conditions are assumed. This is a generalisation of a known result from the classical theory of Markov chains by Darroch and Seneta [6].     

Fluctuations of interacting Markov chain Monte Carlo methods

We present a multivariate central limit theorem for a general class of interacting Markov chain Monte Carlo algorithms used to solve nonlinear measure-valued equations. These algorithms generate stochastic processes which belong to the class of nonlinear Markov chains interacting with their empirical occupation measures. We develop an original theoretical analysis based on resolvent operators and semigroup techniques to analyze the fluctuations of their occupation measures around their limiting values.     

Uniform ergodicities and perturbation bounds of Markov chains on base norm spaces

It is known that Dobrushin's ergodicity coefficient is one of the effective tools in the investigations of limiting behavior of Markov processes. Several interesting properties of the ergodicity coefficient of a positive mapping defined on base norm spaces have been studied. In this paper, we consider uniformly mean ergodic and asymptotically stable Markov operators on such spaces. In terms of the ergodicity coefficient, we establish uniform mean ergodicity criterion. Moreover, we develop the perturbation theory for uniformly asymptotically stable Markov chains on base norm spaces. In particularly, main results open new perspectives in the perturbation theory for quantum Markov processes defined on von Neumann algebras.     

Subgeometric rates of convergence of f-ergodic strong Markov processes

We provide a condition in terms of a supermartingale property for a functional of the Markov process, which implies (a) f$f$-ergodicity of strong Markov processes at a subgeometric rate, and (b) a moderate deviation principle for an integral (bounded) functional. An equivalent condition in terms of a drift inequality on the extended generator is also given. Results related to (f,r)$(f,r)$-regularity of the process, of some skeleton chains and of the resolvent chain are also derived. Applications to specific processes are considered, including elliptic stochastic differential equations, Langevin diffusions, hypoelliptic stochastic damping Hamiltonian systems and storage models.     

Coupling in the theory of interacting systems

In this paper we consider the construction of couplings for Markovian evolutions on a state space of the formE , with (measurable) and a countable group (^d for example). The evolutions we focus on are mainly systems of linearly interacting diffusions, withE compact. We explain and state properties of such couplings and show how they are used to obtain information on the behaviour of the evolution in finite time and as time tends to infinity. An important property of a coupling is to be a successful coupling. The latter concept is introduced here in the context of interacting systems, which is different from the classical concept for Markov chains or processes with state space ^d. The analysis of the question when a coupling is successful depends heavily on the structure of the interaction term and is investigated in detail. We formulate some open problems and conjectures.The paper puts in perspective the coupling statements appearing in the proofs of various results and is largely based on the works of Cox and Greven, Fleischmann and Greven, Dawson and Greven, Greven, and Cox, Greven and Shiga.     

The law of iterated logarithm for additive functionals and martingale additive functionals of Harris recurrent Markov processes

We use Nummelin splitting in continuous time in order to prove laws of iterated logarithm for additive functionals of a Harris recurrent Markov process, with deterministic or random renormalization.     

Conditional law of risk processes given that ruin occurs

A risk process that can be Markovised is conditioned on ruin. We prove that the process remains a Markov process. If the risk process is a PDMP, it is shown that the conditioned process remains a PDMP. For many examples the asymptotics of the parameters in both the light-tailed case and the heavy-tailed case are discussed.     

Cutoffs for product chains

We consider products of ergodic Markov chains and discuss their cutoffs in total variation. Our framework is general in that rates to pick up coordinates are not necessary equal, and different coordinates may correspond to distinct chains. We give necessary and sufficient conditions for cutoffs of product chains in terms of those of coordinate chains under certain conditions. A comparison of mixing times between the product chain and its coordinate chains is made in detail as well. Examples are given to show that neither cutoffs for product chains nor for coordinate chains imply others in general.     

Lower bound technique in the theory of a stochastic differential equation

We establish a criterion for the existence of an invariant measure for Markov processes acting on measures defined on an arbitrary complete separable metric space. This criterion is applied to time-homogeneous Markov processes associated with a nonlinear heat equation driven by an impulsive noise.     

Study of dependence for some stochastic processes: Symbolic Markov copulae

We study dependence between components of multivariate (nice Feller) Markov processes: what conditions need to be satisfied by a multivariate Markov process so that its components are Markovian with respect to the filtration of the entire process and such that they follow prescribed laws? To answer this question, we introduce a symbolic approach, which is rooted in the concept of pseudo-differential operator (PDO). We investigate connections between dependence, in the sense described above, and the PDOs. In particular, we study the problem of constructing a multivariate nice Feller process with given marginal laws in terms of symbols of the related PDOs. This approach leads to relatively simple conditions, which provide solutions to this problem.     

Blackwell Optimality in the Class of Markov Policies for Continuous-Time Controlled Markov Chains

This paper deals with Blackwell optimality for continuous-time controlled Markov chains with compact Borel action space, and possibly unbounded reward (or cost) rates and unbounded transition rates. We prove the existence of a deterministic stationary policy which is Blackwell optimal in the class of all admissible (nonstationary) Markov policies, thus extending previous results that analyzed Blackwell optimality in the class of stationary policies. We compare our assumptions to the corresponding ones for discrete-time Markov controlled processes.     

Fluctuations of empirical means at low temperature for finite Markov chains with rare transitions in the general case

Summary. The integrated autocovariance and autocorrelation time are essential tools to understand the dynamical behavior of a Markov chain. We study here these two objects for Markov chains with rare transitions with no reversibility assumption. We give upper bounds for the autocovariance and the integrated autocorrelation time, as well as exponential equivalents at low temperature. We also link their slowest modes with the underline energy landscape under mild assumptions. Our proofs will be based on large deviation estimates coming from the theory of Wentzell and Freidlin and others [4, 3, 12], and on coupling arguments (see [6] for a review on the coupling method). Received 5 August 1996 / In revised form: 6 August 1997     

Markov solutions for the 3D stochastic Navier–Stokes equations with state dependent noise

We construct a Markov family of solutions for the 3D Navier-Stokes equations perturbed by a non degenerate noise. We improve the result of [3] in two directions. We see that in fact not only a transition semigroup but a Markov family of solutions can be constructed. Moreover, we consider a state dependant noise. Another feature of this work is that we greatly simplify the proofs of [3]. Dedicated to Giuseppe Da Prato on the occasion of his 70th birthday     

Some stochastic properties of “semi-magic” and “magic” Markov chains

Gustafson and Styan (Gustafson and Styan, Superstochastic matrices and Magic Markov chains, Linear Algebra Appl. 430 (2009) 2705-2715) examined the mathematical properties of superstochastic matrices, the transition matrices of “magic” Markov chains formed from scaled “magic squares”. This paper explores the main stochastic properties of such chains as well as “semi-magic” chains (with doubly-stochastic transition matrices). Stationary distribution, generalized inverses of Markovian kernels, mean first passage times, variances of the first passage times and expected times to mixing are considered. Some general results are developed, some observations from the chains generated by MATLAB are discussed, some conjectures are presented and some special cases, involving three and four states, are explored in detail.