The Limit Behaviour of Imprecise Continuous-Time Markov Chains

We study the limit behaviour of a nonlinear differential equation whose solution is a superadditive generalisation of a stochastic matrix, prove convergence, and provide necessary and sufficient conditions for ergodicity. In the linear case, the solution of our differential equation is equal to the matrix exponential of an intensity matrix and can then be interpreted as the transition operator of a homogeneous continuous-time Markov chain. Similarly, in the generalised nonlinear case that we consider, the solution can be interpreted as the lower transition operator of a specific set of non-homogeneous continuous-time Markov chains, called an imprecise continuous-time Markov chain. In this context, our convergence result shows that for a fixed initial state, an imprecise continuous-time Markov chain always converges to a limiting distribution, and our ergodicity result provides a necessary and sufficient condition for this limiting distribution to be independent of the initial state.     

Inverse problems for ergodicity of Markov chains

For continuous-time Markov chains, we provide criteria for non-ergodicity, non-algebraic ergodicity, non-exponential ergodicity, and non-strong ergodicity. For discrete-time Markov chains, criteria for non-ergodicity, non-algebraic ergodicity, and non-strong ergodicity are given. Our criteria are in terms of the existence of solutions to inequalities involving the Q-matrix (or transition matrix P in time-discrete case) of the chain. Meanwhile, these practical criteria are applied to some examples, including a special class of single birth processes and several multi-dimensional models.     

Integral-type functionals of first hitting times for continuous-time Markov chains

We investigate integral-type functionals of the first hitting times for continuous-time Markov chains. Recursive formulas and drift conditions for calculating or bounding integral-type functionals are obtained. The connection between the subexponential integral-type functionals and the subexponential ergodicity is established. Moreover, these results are applied to the birth-death processes. Polynomial integral-type functionals and polynomial ergodicity are studied, and a sufficient criterion for a central limit theorem is also presented.     

Subgeometric ergodicity for continuous-time Markov chains

In this paper, subgeometric ergodicity is investigated for continuous-time Markov chains. Several equivalent conditions, based on the first hitting time or the drift function, are derived as the main theorem. In its corollaries, practical drift criteria are given for ?-ergodicity and computable bounds on subgeometric convergence rates are obtained for stochastically monotone Markov chains. These results are illustrated by examples.     

Average stochastic games for continuous-time jump processes

We consider nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion. The state and action spaces are Borel spaces and reward rates are unbounded. We introduce an approximating sequence of stochastic game models with extended state space, for which the uniform exponential ergodicity is obtained. Moreover, we prove the existence of a stationary almost Markov Nash equilibrium by introducing auxiliary static game models. Finally, a cash flow model is employed to illustrate the results.     

On the weak ergodicity of nonhomogeneous continuous-time Markov chains

The conditions of weak ergodicity for nonhomogeneous continuous-time Markov chains are obtained in terms of intensities. The birth and death process on a finite state space is studied as a special case. Supported by the Russian Foundation for Fundamental Research (grant No. 96-01-01369). Proceedings of the Seminar on Stability Problems for Stochastic Models, Hajdúszoboszló, Hungary, 1997, Part III.     

On categorical time series models with covariates

We study the problem of stationarity and ergodicity for autoregressive multinomial logistic time series models which possibly include a latent process and are defined by a GARCH-type recursive equation. We improve considerably upon the existing conditions about stationarity and ergodicity of those models. Proofs are based on theory developed for chains with complete connections. A useful coupling technique is employed for studying ergodicity of infinite order finite-state stochastic processes which generalize finite-state Markov chains. Furthermore, for the case of finite order Markov chains, we discuss ergodicity properties of a model which includes strongly exogenous but not necessarily bounded covariates.     

Ergodic properties of some piecewise-deterministic Markov process with application to gene expression modelling

We investigate a piecewise-deterministic Markov process with a Polish state space, whose deterministic behaviour between random jumps is governed by a finite number of semiflows. We provide tractable conditions ensuring a form of exponential ergodicity and the strong law of large numbers for the chain given by the post-jump locations. Further, we establish a one-to-one correspondence between invariant measures of the chain and those of the continuous-time process. These results enable us to derive the strong law of large numbers for the latter. The studied dynamical system is inspired by certain models of gene expression, which are also discussed here.     

非齐次马尔科夫链遍历性的一些结果

本文利用两个非齐次马尔科夫链的转移矩阵列的比较，讨论了两个链启遍历性的关系，得到一个非齐次马尔科夫链是强遍历的一些充分条件．本文还分析了非齐次马尔科夫链的一致强、弱遍历性的关系，得到一个非齐次马尔科夫链是一致强遍历的一些充分条件．     

Blackwell Optimality in the Class of Markov Policies for Continuous-Time Controlled Markov Chains

This paper deals with Blackwell optimality for continuous-time controlled Markov chains with compact Borel action space, and possibly unbounded reward (or cost) rates and unbounded transition rates. We prove the existence of a deterministic stationary policy which is Blackwell optimal in the class of all admissible (nonstationary) Markov policies, thus extending previous results that analyzed Blackwell optimality in the class of stationary policies. We compare our assumptions to the corresponding ones for discrete-time Markov controlled processes.     

On ergodicity conditions in a polling model with Markov modulated input and state-dependent routing

A polling system with switchover times and state-dependent server routing is studied. Input flows are modulated by a random external environment. Input flows are ordinary Poisson flows in each state of the environment, with intensities determined by the environment state. Service and switchover durations have exponential laws of probability distribution. A continuous-time Markov chain is introduced to describe the dynamics of the server, the sizes of the queues and the states of the environment. By means of the iterative-dominating method a sufficient condition for ergodicity of the system is obtained for the continuous-time Markov chain. This condition also ensures the existence of a stationary probability distribution of the embedded Markov chain at instants of jumps. The customers sojourn cost during the period of unloading the stable queueing system is chosen as a performance metric. Numerical study in case of two input flows and a class of priority and threshold routing algorithms is conducted. It is demonstrated that in case of light inputs a priority routing rule doesn’t seem to be quasi-optimal.     

Ergodic Control of a Singularly Perturbed Markov Process in Discrete Time with General State and Compact Action Spaces

Ergodic control of singularly perturbed Markov chains with general state and compact action spaces is considered. A new method is given for characterization of the limit of invariant measures, for perturbed chains, when the perturbation parameter goes to zero. It is also demonstrated that the limit control principle is satisfied under natural ergodicity assumptions about controlled Markov chains. These assumptions allow for the presence of transient states, a situation that has not been considered in the literature before in the context of control of singularly perturbed Markov processes with long-run-average cost functionals. Accepted 3 December 1996     

Risk-sensitive stopping problems for continuous-time Markov chains

In this paper we consider stopping problems for continuous-time Markov chains under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. More precisely our aim is to maximize the certainty equivalent of the stopping reward minus cost over the time horizon. We derive optimality equations for the value functions and prove the existence of optimal stopping times. The exponential utility is treated as a special case. In contrast to risk-neutral stopping problems it may be optimal to stop between jumps of the Markov chain. We briefly discuss the influence of the risk sensitivity on the optimal stopping time and consider a special house selling problem as an example.     

Estimations pour les chaînes de Markov réversibles

We give Gaussian lower and upper bounds for reversible Markov chains on a graph under two geometric assumptions (volume regularity and Poincaré inequality). This is first proved for continuous-time Markov chains via a parabolic Harnack inequality. Then, the estimates for the discrete-time Markov chains are derived by comparison.     

几类遍历性的显式判别准则

关于一维扩散过程或生灭过程的几类遍历性,新近很快地找到了显式判别准则,详见表5．1和5．2,其一是生灭过程的指数遍历性判准,曾是马链的一个长期未解问题,有趣的是,此论题与多个数学分支有关,我们详细介绍后一问题的想法,也建议了若干进一步研究课题。     

渐近循环马氏链的收敛速度

引入了渐近循环马氏链的概念,它是循环马氏链概念的推广.首先研究了在强遍历的条件下,可列循环马氏链的收敛速度,作为主要结论给出了当满足不同条件时可列渐近循环马氏链的C-强遍历性,一致C-强遍历性和一致C-强遍历的收敛速度     

Entrywise relative perturbation bounds for exponentials of essentially non-negative matrices

A real square matrix is said to be essentially non-negative if all of its off-diagonal entries are non-negative. We establish entrywise relative perturbation bounds for the exponential of an essentially non-negative matrix. Our bounds are sharp and contain a condition number that is intrinsic to the exponential function. As an application, we study sensitivity of continuous-time Markov chains. J. Xue was supported by the National Science Foundation of China under grant number 10571031, the Program for New Century Excellent Talents in Universities of China and Shanghai Pujiang Program. Q. Ye was supported in part by NSF under Grant DMS-0411502.     

Uniform ergodicities and perturbation bounds of Markov chains on base norm spaces

It is known that Dobrushin's ergodicity coefficient is one of the effective tools in the investigations of limiting behavior of Markov processes. Several interesting properties of the ergodicity coefficient of a positive mapping defined on base norm spaces have been studied. In this paper, we consider uniformly mean ergodic and asymptotically stable Markov operators on such spaces. In terms of the ergodicity coefficient, we establish uniform mean ergodicity criterion. Moreover, we develop the perturbation theory for uniformly asymptotically stable Markov chains on base norm spaces. In particularly, main results open new perspectives in the perturbation theory for quantum Markov processes defined on von Neumann algebras.     

Markov branching processes with immigration-migration and resurrection

We consider a modified Markov branching process incorporating with both state-independent immigration-migration and resurrection. The effect of state-independent immigration-migration is firstly in- vestigated in detail. The explicit expressions for the extinction probabilities and mean extinction times are presented. The ergodicity and stability properties of the process incorporating with resurrection structure are then investigated. The conditions for recurrence, ergodicity and exponential ergodicity are...     

Determination of the Spectral Index of Ergodicity of a Birth-and-Death Process

We obtain a new explicit relation for the calculation of the spectral index of ergodicity of a birth-and-death process with continuous time. The calculation of the index is reduced to the solution of an optimization problem of nonlinear programming that contains the infinitesimal matrix of the process. As an example, we use the proposed method for finding the exact values of the indices of exponential ergodicity for certain Markov queuing systems.