Markov transition probabilities

Summary A continuous-parameter Markov process on a general state space has transition function P _t (x,E). The theory of regenerative phenomena is applied to the question: what functions of t can arise in this way? Particular attention is paid to processes of purely discontinuous type, to which are extended known results for processes with a countable state space.     

On the optimality equation for average cost Markov control processes with Feller transition probabilities

We consider Markov control processes with Borel state space and Feller transition probabilities, satisfying some generalized geometric ergodicity conditions. We provide a new theorem on the existence of a solution to the average cost optimality equation.     

On the Kolmogorov inequality for fractional derivatives on the half-line

В РАБОтЕ пРИВЕДЕНы НЕ ОБхОДИМыЕ И ДОстАтОЧ НыЕ УслОВИь сУЩЕстВОВАНИь НЕРАВ ЕНстВА НА пОлУпРьМОИ R₊=[0, ∞): $$\left\| {(D^\alpha x)( \cdot )} \right\|_{C(R_ + )} \leqq K\left\| {x( \cdot )} \right\|_{L_2 (R_ + )}^{v_1 } \left\| {(D^n x)( \cdot )} \right\|_{L_2 (R_ + )}^{v_2 } ,$$ гДЕ А-пРОИжВОльНОЕ ВЕ ЩЕстВЕННОЕ ЧИслО,n≧1 — цЕлОЕ Иv _i>0,i=1,2. ДРОБНАь пРОИжВОД НАьD ^α пОНИМАЕтсь В сМыслЕ г. ВЕИль. ВыЧИслЕНА НАИ лУЧшАь (т.Е. НАИМЕНьшАь Иж ВОжМ ОжНых) кОНстАНтАк=к(п, А) В ЁтО М НЕРАВЕНстВЕ И ВыпИс АНА ЁкстРЕМАльНАь ФУНкц Иь, НА кОтОРОИ НЕРАВЕНстВО пРЕВРАЩАЕтсь В РАВЕН стВО.     

Finite approximation for finite-horizon continuous-time Markov decision processes

In this paper we study the continuous-time Markov decision processes with a denumerable state space, a Borel action space, and unbounded transition and cost rates. The optimality criterion to be considered is the finite-horizon expected total cost criterion. Under the suitable conditions, we propose a finite approximation for the approximate computations of an optimal policy and the value function, and obtain the corresponding error estimations. Furthermore, our main results are illustrated with a controlled birth and death system.     

Bias optimality for multichain continuous-time Markov decision processes

This paper deals with the bias optimality of multichain models for finite continuous-time Markov decision processes. Based on new performance difference formulas developed here, we prove the convergence of a so-called bias-optimal policy iteration algorithm, which can be used to obtain bias-optimal policies in a finite number of iterations.     

连续时间Markov决策过程的均值-方差优化问题

本文考虑连续时间Markov决策过程折扣模型的均值-方差优化问题.假设状态空间和行动空间均为Polish空间,转移率和报酬率函数均无界.本文的优化目标是在折扣最优平稳策略类里,选取相应方差最小的策略.本文致力于寻找Polish空间下Markov决策过程均值-方差最优策略存在的条件.利用首次进入分解方法,本文证明均值-方差优化问题可以转化为"等价"的期望折扣优化问题,进而得到关于均值-方差优化问题的"最优方程"和均值-方差最优策略的存在性以及它相应的特征.最后,本文给出若干例子说明折扣最优策略的不唯一性和均值-方差最优策略的存在性.     

Martingale approximations for continuous-time and discrete-time stationary Markov processes

We show that the method of Kipnis and Varadhan [Comm. Math. Phys. 104 (1986) 1–19] to construct a Martingale approximation to an additive functional of a stationary ergodic Markov process via the resolvent is universal in the sense that a martingale approximation exists if and only if the resolvent representation converges. A sufficient condition for the existence of a martingale approximation is also given. As examples we discuss moving average processes and processes with normal generator.     

Bounds and approximations for continuous-time Markovian transition probabilities and large systems

We propose new bounds and approximations for the transition probabilities of a continuous-time Markov process with finite but large state-space. The bounding and approximating procedures have been exposed in another paper [S. Mercier, Numerical bounds for semi-Markovian quantities and applications to reliability, in revision for Methodology and Computing in Applied Probability] in the more general context of a continuous-time semi-Markov process with countable state-space. Such procedures are here specialized to the Markovian finite case, leading to much simpler algorithms. The aim of this paper is to test such algorithms versus other algorithms from the literature near from ours, such as forward Euler approximation, external uniformization and a finite volume method from [C. Cocozza-Thivent, R. Eymard, Approximation of the marginal distributions of a semi-Markov process using a finite volume scheme, ESAIM: M2AN 38(5) (2004) 853–875].     

Lyapunov coupled equations for continuous-time infinite Markov jump linear systems

This paper deals with Lyapunov equations for continuous-time Markov jump linear systems (MJLS). Out of the bent which wends most of the literature on MJLS, we focus here on the case in which the Markov chain has a countably infinite state space. It is shown that the infinite MJLS is stochastically stabilizable (SS) if and only if the associated countably infinite coupled Lyapunov equations have a unique norm bounded strictly positive solution. It is worth mentioning here that this result do not hold for mean square stabilizability (MSS), since SS and MSS are no longer equivalent in our set up (see, e.g., [J. Baczynski, Optimal control for continuous time LQ-problems with infinite Markov jump parameters, Ph.D. Thesis, Federal University of Rio de Janeiro, UFRJ/COPPE, 2000]). To some extent, a decomplexification technique and tools from operator theory in Banach space and, in particular, from semigroup theory are the very technical underpinning of the paper.     

Average optimality inequality for continuous-time Markov decision processes in Polish spaces

In this paper, we study the average optimality for continuous-time controlled jump Markov processes in general state and action spaces. The criterion to be minimized is the average expected costs. Both the transition rates and the cost rates are allowed to be unbounded. We propose another set of conditions under which we first establish one average optimality inequality by using the well-known “vanishing discounting factor approach”. Then, when the cost (or reward) rates are nonnegative (or nonpositive), from the average optimality inequality we prove the existence of an average optimal stationary policy in all randomized history dependent policies by using the Dynkin formula and the Tauberian theorem. Finally, when the cost (or reward) rates have neither upper nor lower bounds, we also prove the existence of an average optimal policy in all (deterministic) stationary policies by constructing a “new” cost (or reward) rate. Research partially supported by the Natural Science Foundation of China (Grant No: 10626021) and the Natural Science Foundation of Guangdong Province (Grant No: 06300957).     

Average sample-path optimality for continuous-time Markov decision processes in Polish spaces

In this paper we study the average sample-path cost(ASPC) problem for continuous-time Markov decision processes in Polish spaces.To the best of our knowledge,this paper is a first attempt to study the ASPC criterion on continuous-time MDPs with Polish state and action spaces.The corresponding transition rates are allowed to be unbounded,and the cost rates may have neither upper nor lower bounds.Under some mild hypotheses,we prove the existence of ε(ε≥ 0)-ASPC optimal stationary policies based on two differe...     

New sufficient conditions for average optimality in continuous-time Markov decision processes

This paper is devoted to studying continuous-time Markov decision processes with general state and action spaces, under the long-run expected average reward criterion. The transition rates of the underlying continuous-time Markov processes are allowed to be unbounded, and the reward rates may have neither upper nor lower bounds. We provide new sufficient conditions for the existence of average optimal policies. Moreover, such sufficient conditions are imposed on the controlled process’ primitive data and thus they are directly verifiable. Finally, we apply our results to two new examples.     

Variance minimization for continuous-time Markov decision processes: two approaches

This paper studies the limit average variance criterion for continuous-time Markov decision processes in Polish spaces. Based on two approaches, this paper proves not only the existence of solutions to the variance minimization optimality equation and the existence of a variance minimal policy that is canonical, but also the existence of solutions to the two variance minimization optimality inequalities and the existence of a variance minimal policy which may not be canonical. An example is given to illustrate all of our conditions.