Some new results on transition probability

In this paper, we study the basic properties of stationary transition probability of Markov processes on a general measurable space （E, δ）, such as the continuity, maximum probability, zero point, positive probability set,standardization, and obtain a series of important results such as Continuity Theorem, Representation Theorem, Levy Theorem and so on. These results are very useful for us to study stationary tri-point transition probability on a general measurable space （E, δ）. Our main tools such as Egoroff＇s Theorem, Vitali-Hahn-Saks＇s Theorem and the theory of atomic set and well- posedness of measure are also very interesting and fashionable.     

On the adaptive control of a class of partially observed Markov decision processes

This paper is concerned with the adaptive control problem, over the infinite horizon, for partially observable Markov decision processes whose transition functions are parameterized by an unknown vector. We treat finite models and impose relatively mild assumptions on the transition function. Provided that a sequence of parameter estimates converging in probability to the true parameter value is available, we show that the certainty equivalence adaptive policy is optimal in the long-run average sense.     

Goodness-of-fit test for homogeneous Markov processes

We give chi-squared goodness-of-fit tests for homogeneous Markov processes with unknown transition intensities or with transition intensities of known form depending on a finite-dimensional parameter.     

A relation between two‐parameter and one‐parameter transition functions for Markov processes

In this paper, we consider a relation between two‐parameter and one‐parameter transition functions for Markov processes and obtain a very useful result to be regarded as relation theorem. Copyright © 2012 John Wiley & Sons, Ltd.     

Equivalence of Lyapunov stability criteria in a class of Markov decision processes

We are concerned with Markov decision processes with countable state space and discrete-time parameter. The main structural restriction on the model is the following: under the action of any stationary policy the state space is acommunicating class. In this context, we prove the equivalence of ten stability/ergodicity conditions on the transition law of the model, which imply the existence of average optimal stationary policies for an arbitrary continuous and bounded reward function; these conditions include the Lyapunov function condition (LFC) introduced by A. Hordijk. As a consequence of our results, the LFC is proved to be equivalent to the following: under the action of any stationary policy the corresponding Markov chain has a unique invariant distribution which depends continuously on the stationary policy being used. A weak form of the latter condition was used by one of the authors to establish the existence of optimal stationary policies using an approach based on renewal theory.This research was supported in part by the Third World Academy of Sciences (TWAS) under Grant TWAS RG MP 898-152.     

Filtering,Smoothing and M-ary Detection with Discrete Time Poisson Observations

Abstract

In this article, we solve a class of estimation problems, namely, filtering smoothing and detection for a discrete time dynamical system with integer-valued observations. The observation processes we consider are Poisson random variables observed at discrete times. Here, the distribution parameter for each Poisson observation is determined by the state of a Markov chain. By appealing to a duality between forward (in time) filter and its corresponding backward processes, we compute dynamics satisfied by the unnormalized form of the smoother probability. These dynamics can be applied to construct algorithms typically referred to as fixed point smoothers, fixed lag smoothers, and fixed interval smoothers. M-ary detection filters are computed for two scenarios: one for the standard model parameter detection problem and the other for a jump Markov system.     

Stochastic model of phase transition and metastability

The evolution of a system with phase transition is simulated by a Markov process whose transition probabilities depend on a parameter. The change of the stationary distribution of the Markov process with a change of this parameter is interpreted as a phase transition of the system from one thermodynamic equilibrium state to another. Calculations and computer experiments are performed for condensation of a vapor. The sample paths of the corresponding Markov process have parts where the radius of condensed drops is approximately constant. These parts are interpreted as metastable states. Two metastable states occur, initial (gaseous steam) and intermediate (fog). The probability distributions of the drop radii in the metastable states are estimated. Translated from Teoreticheskaya i Matematicheskaya Fizika, Vol. 123, No. 1, pp. 94–106, April, 2000.     

Ergodic and adaptive control of hidden Markov models

A partially observed stochastic system is described by a discrete time pair of Markov processes. The observed state process has a transition probability that is controlled and depends on a hidden Markov process that also can be controlled. The hidden Markov process is completely observed in a closed set, which in particular can be the empty set and only observed through the other process in the complement of this closed set. An ergodic control problem is solved by a vanishing discount approach. In the case when the transition operators for the observed state process and the hidden Markov process depend on a parameter and the closed set, where the hidden Markov process is completely observed, is nonempty and recurrent an adaptive control is constructed based on this family of estimates that is almost optimal.     

Column continuous transition functions

A column continuous transition function is by definition a standard transition function P(t) whose every column is continuous for t?0 in the norm topology of bounded sequence space l_∞. We will prove that it has a stable q-matrix and that there exists a one-to-one relationship between column continuous transition functions and increasing integrated semigroups on l_∞. Using the theory of integrated semigroups, we give some necessary and sufficient conditions under which the minimal q-function is column continuous, in terms of its generator (of the Markov semigroup) as well as its q-matrix. Furthermore, we will construct all column continuous Q-functions for a conservative, single-exit and column bounded q-matrix Q. As applications, we find that many interesting continuous-time Markov chains (CTMCs), say Feller-Reuter-Riley processes, monotone processes, birth-death processes and branching processes, etc., have column continuity.     

Robust Markov perfect equilibria

In this paper we study a Markov decision model with quasi-hyperbolic discounting and transition probability function depending on an unknown parameter. Assuming that the set of parameters is finite, the sets of states and actions are Borel and the transition probabilities satisfy some additivity conditions and are atomless, we prove the existence of a non-randomised robust Markov perfect equilibrium.     

Discounted continuous-time Markov decision processes with unbounded rates and randomized history-dependent policies: the dynamic programming approach

This paper deals with a continuous-time Markov decision process in Borel state and action spaces and with unbounded transition rates. Under history-dependent policies, the controlled process may not be Markov. The main contribution is that for such non-Markov processes we establish the Dynkin formula, which plays important roles in establishing optimality results for continuous-time Markov decision processes. We further illustrate this by showing, for a discounted continuous-time Markov decision process, the existence of a deterministic stationary optimal policy (out of the class of history-dependent policies) and characterizing the value function through the Bellman equation.     

The minimal solution of a Markov renewal equation

Given a killed Markov process, one can use a procedure of Ikedaet al. to revive the process at the killing times. The revived process is again a Markov process and its transition function is the minimal solution of a Markov renewal equation. In this paper we will calculate such solutions for a class of revived processes.     

可数马尔科夫过程的强马氏性

§1 引 理 对非齐次马尔科夫过程转移概率的分析性质及非齐次可数马科夫过程样本函数的性质,人们已做了较为系统的研究。本文讨论的是非齐次可数马尔科夫过程(以下简称为马氏链)的强马氏性问题,这是马氏链基本理论的一个重要组成部分。若一个右标准马氏链可分、Borel可测且右下半连续,则称其为右正则马氏链(详见定义2.1)。本文首先指出:任何一个右标准马氏链都有右正则修正;继而,通过考察推移过程的性质,证明了:任何右正则马氏链均具有强马氏性。从而在右标准马氏链情形,本文将〔6〕第二章§4§6中过程右     

Applications of Feller–Reuter–Riley Transition Functions

A Feller–Reuter–Riley function is a Markov transition function whose corresponding semigroup maps the set of the real-valued continuous functions vanishing at infinity into itself. The aim of this paper is to investigate applications of such functions in the dual problem, Markov branching processes, and the Williams-matrix. The remarkable property of a Feller–Reuter–Riley function is that it is a Feller minimal transition function with a stable q-matrix. By using this property we are able to prove that, in the theory of branching processes, the branching property is equivalent to the requirement that the corresponding transition function satisfies the Kolmogorov forward equations associated with a stable q-matrix. It follows that the probabilistic definition and the analytic definition for Markov branching processes are actually equivalent. Also, by using this property, together with the Resolvent Decomposition Theorem, a simple analytical proof of the Williams' existence theorem with respect to the Williams-matrix is obtained. The close link between the dual problem and the Feller–Reuter–Riley transition functions is revealed. It enables us to prove that a dual transition function must satisfy the Kolmogorov forward equations. A necessary and sufficient condition for a dual transition function satisfying the Kolmogorov backward equations is also provided.     

On the link between infinite horizon control and quasi-stationary distributions

We study infinite horizon control of continuous-time non-linear branching processes with almost sure extinction for general (positive or negative) discount. Our main goal is to study the link between infinite horizon control of these processes and an optimization problem involving their quasi-stationary distributions and the corresponding extinction rates. More precisely, we obtain an equivalent of the value function when the discount parameter is close to the threshold where the value function becomes infinite, and we characterize the optimal Markov control in this limit. To achieve this, we present a new proof of the dynamic programming principle based upon a pseudo-Markov property for controlled jump processes. We also prove the convergence to a unique quasi-stationary distribution of non-linear branching processes controlled by a Markov control conditioned on non-extinction.     

Sulľapplicazione della funzione ellitticap u alla teoria dei poligoni di poncelet

Abstract  In this paper we study strongly continuous positive semigroups on particular classes of weighted continuous function space on a locally compact Hausdorff space X having a countable base. In particular we characterize those positive semigroups which are the transition semigroups of suitable Markov processes. Some applications are also discussed. Keywords Positive semigroup, Markov transition function, Markov process, Weighted continuous function space, Degenerate second order differential operator Mathematics Subject Classification (2000) 47D06, 47D07, 60J60     

Regenerative simulation of TES processes

Regenerative simulation has become a familiar and established tool for simulation-based estimation. However, many applications (e.g., traffic in high-speed communications networks) call for autocorrelated stochastic models to which traditional regenerative theory is not directly applicable. Consequently, extensions of regenerative simulation to dependent time series is increasingly gaining in theoretical and practical interest, with Markov chains constituting an important case. Fortunately, a regenerative structure can be identified in Harris-recurrent Markov chains with minor modification, and this structure can be exploited for standard regenerative estimation. In this paper we focus on a versatile class of Harris-recurrent Markov chains, called TES (Transform-Expand-Sample). TES processes can generate a variety of sample paths with arbitrary marginal distributions, and autocorrelation functions with a variety of functional forms (monotone, oscillating and alternating). A practical advantage of TES processes is that they can simultaneously capture the first and second order statistics of empirical sample paths (raw field measurements). Specifically, the TES modeling methodology can simultaneously match the empirical marginal distribution (histogram), as well as approximate the empirical autocorrelation function. We explicitly identify regenerative structures in TES processes and proceed to address efficiency and accuracy issues of prospective simulations. To show the efficacy of our approach, we report on a TES/M/1 case study. In this study, we used the likelihood ratio method to calculate the mean waiting time performance as a function of the regenerative structure and the intrinsic TES parameter controlling burstiness (degree of autocorrelation) in the arrival process. The score function method was used to estimate the corresponding sensitivity (gradient) with respect to the service rate. Finally, we demonstrated the importance of the particular regenerative structure selected in regard to the estimation efficiency and accuracy induced by the regeneration cycle length.     

A note on chaotic and predictable representations for Itô–Markov additive processes

In this article, we provide predictable and chaotic representations for Itô–Markov additive processes X. Such a process is governed by a finite-state continuous time Markov chain J which allows one to modify the parameters of the Itô-jump process (in so-called regime switching manner). In addition, the transition of J triggers the jump of X distributed depending on the states of J just prior to the transition. This family of processes includes Markov modulated Itô–Lévy processes and Markov additive processes. The derived chaotic representation of a square-integrable random variable is given as a sum of stochastic integrals with respect to some explicitly constructed orthogonal martingales. We identify the predictable representation of a square-integrable martingale as a sum of stochastic integrals of predictable processes with respect to Brownian motion and power-jumps martingales related to all the jumps appearing in the model. This result generalizes the seminal result of Jacod–Yor and is of importance in financial mathematics. The derived representation then allows one to enlarge the incomplete market by a series of power-jump assets and to price all market-derivatives.     

Perturbed Markov Chains with Damping Component

The paper is devoted to studies of regularly and singularly perturbed Markov chains with damping component. In such models, a matrix of transition probabilities is regularised by adding a special damping matrix multiplied by a small damping (perturbation) parameter ε. We perform a detailed perturbation analysis for such Markov chains, particularly, give effective upper bounds for the rate of approximation for stationary distributions of unperturbed Markov chains by stationary distributions of perturbed Markov chains with regularised matrices of transition probabilities, asymptotic expansions for approximating stationary distributions with respect to damping parameter, explicit coupling type upper bounds for the rate of convergence in ergodic theorems for n-step transition probabilities, as well as ergodic theorems in triangular array mode.