非时齐部分可观察Markov决策规划的最优策略问题

本文讨论了一类非时齐部分可观察Markov决策模型．在不改变状态空间可列性的条件下，把该模型转化为[5]中的一般化折扣模型，从而解决了其最优策略问题，并且得到了该模型的有限阶段逼近算法，其中该算法涉及的状态是可列的．     

Transformation of partially observable Markov decision processes into piecewise linear ones

This paper considers how partially observable Markov decision processes may be transformed into piecewise linear ones, which have many advantages in that they are easily represented in a computer. Also we specify how to find the products of simple partitions on which cost functions are piecewise linear.     

Time aggregated Markov decision processes via standard dynamic programming

This note addresses the time aggregation approach to ergodic finite state Markov decision processes with uncontrollable states. We propose the use of the time aggregation approach as an intermediate step toward constructing a transformed MDP whose state space is comprised solely of the controllable states. The proposed approach simplifies the iterative search for the optimal solution by eliminating the need to define an equivalent parametric function, and results in a problem that can be solved by simpler, standard MDP algorithms.     

基于动态规划的高阶隐马氏模型推广的Viterbi算法

首先通过Hadar等价变换方法将高阶隐马氏模型转换为与之等价的一阶向量值隐马氏模型,然后利用动态规划原理建立了一阶向量值隐马氏模型的Viterbi算法,最后通过高阶隐马氏模型和一阶向量值隐马氏模型之间的等价关系建立了高阶隐马氏模型基于动态规划推广的Viterbi算法．研究结果在一定程度上推广了几乎所有隐马氏模型文献中所涉及到的解码问题的Viterbi算法,从而进一步丰富和发展了高阶隐马氏模型的算法理论．     

Countably many simultaneous random changes of time scales

Simultaneous changes of time scales of the components of a vector Markov process are defined and developed. Measurability properties, Dynkin's lemma, and the strong Markov property are established for the transformed process.     

非齐次可列马尔可夫过程的轨道性质

§1 状态分类 定义1.1 设I是非负整数集,P={P_(ij)(s,t)|i,j∈I,α≤s≤t≤b}是转移函数矩阵。称P对i在t右标准,若limp_(ii)(t,t+h)=1;称P对i在t左标准,若limP_(ii)(t-h,t)=1.若P对i在t同时为右标准的和左标准的,则称P对i在t标准。若P对i在每个t标准,则称P对i标准。P对i右标准或左标准与此类似。若P对每个i标准,则称P标准。P右标准或左标准与此类似(参看[5]、[6])。     

The Geometric Markov Renewal Processes with Application to Finance

We introduce the geometric Markov renewal processes as a model for a security market and study this processes in a series scheme. We consider its approximations in the form of averaged, merged and double averaged geometric Markov renewal processes. Weak convergence analysis and rates of convergence of ergodic geometric Markov renewal processes are presented. Martingale properties, infinitesimal operators of geometric Markov renewal processes are presented and a Markov renewal equation for expectation is derived. As an application, we consider the case of two ergodic classes. Moreover, we consider a generalized binomial model for a security market induced by a position dependent random map as a special case of a geometric Markov renewal process.     

非时齐马氏过程的耦合基本定理

本文将耦合方法应用于非时齐马氏过程,推广了时齐情形的耦合基本定理,为后续研究非时齐马氏过程的耦合提供了理论基础.     

Markov cubature rules for polynomial processes

We study discretizations of polynomial processes using finite state Markov processes satisfying suitable moment matching conditions. The states of these Markov processes together with their transition probabilities can be interpreted as Markov cubature rules. The polynomial property allows us to study such rules using algebraic techniques. Markov cubature rules aid the tractability of path-dependent tasks such as American option pricing in models where the underlying factors are polynomial processes.     

Recent Results on Recurrent Solutions and Limit Distributions of SDEs

The limit distribution for homogeneous Markov processes is studied extensively and well understood, but it is not the case for inhomogeneous Markov processes. In this paper, we review some recent results on inhomogeneous Markov processes generated by non-autonomous stochastic (partial) differential equations (SDE in short). Under some suitable conditions, we show that the distribution of recurrent solutions of SDEs constitutes the limit distribution of the corresponding inhomogeneous Markov processes.     

Entropy Maximization for Markov and Semi-Markov Processes

The literature about maximum of entropy for Markov processes deals mainly with discrete-time Markov chains. Very few papers dealing with continuous-time jump Markov processes exist and none dealing with semi-Markov processes. It is the aim of this paper to contribute to fill this lack. We recall the basics concerning entropy for Markov and semi-Markov processes and we study several problems to give an overview of the possible directions of use of maximum entropy in connection with these processes. Numeric illustrations are presented, in particular in application to reliability.     

A mean–variance optimization problem for discounted Markov decision processes

In this paper, we consider a mean–variance optimization problem for Markov decision processes (MDPs) over the set of (deterministic stationary) policies. Different from the usual formulation in MDPs, we aim to obtain the mean–variance optimal policy that minimizes the variance over a set of all policies with a given expected reward. For continuous-time MDPs with the discounted criterion and finite-state and action spaces, we prove that the mean–variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and Markov properties. Then, we show that a mean–variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations. We also address related issues such as a mutual fund theorem and illustrate our results with an example.     

工程管理的马尔可夫骨架过程建模

在以前,一个工程项目的进程的管理问题在各工段的施工时间为相互独立且都服从负指数分布的假设下通过一个网络图化为对一个马尔可夫链的控制和优化的研究.本文在一般情况下(放弃对各工程段的时间服从负指数分布的假设)把工程项目进程的管理问题化为对一个马尔可夫骨架过程的控制和优化问题.     

非时齐马氏过程的随机单调性

本文研究了非时齐马氏过程的随机单调性问题.利用时齐的马氏过程随机单调性的相关证明方法,加以改进,获得了非时齐马氏过程随机单调性的显式判定方法,并进一步将这一充分性条件推广为等价条件.     

信息熵的收敛性与收敛速度

本讨论了连续时间，状态空间有限的马氏过程的信息熵的收敛性质，给出了可测转移矩阵的极限形式，得出具有可测转移矩阵的马氏过程的信息熵在时间趋于无穷大时存在且有限，对于具有强遍历转移函数的马氏过程的信息熵按多项式一致收敛。     

Isomorphism theorems for Markov chains

The isomorphism theorem of Dynkin is definitely an important tool to investigate the problems raised in terms of local times of Markov processes. This theorem concerns continuous time Markov processes. We give here an equivalent version for Markov chains.     

Birthing Markov processes at random rates

Summary This paper studies processes constructed by birthing the trajectories of a given Markov process along time according to random probabilities. Getoor has considered the case where the random probabilities are determined by comultiplicative functionals and proved for right processes that the post-birth process has the Markov property. Here randomizations of comultiplicative functionals are described which give rise to conditionally Markov processes. The main argument is developed for general Markov processes and the transition probabilities of the new process, including those from the pre-birth state, are explicited.     

Trajectory composition of Poisson time changes and Markov counting systems

Changing time of simple continuous-time Markov counting processes by independent unit-rate Poisson processes results in Markov counting processes for which we provide closed-form transition rates via composition of trajectories and with which we construct novel, simpler infinitesimally over-dispersed processes.     

Drift transforms and Green function estimates for discontinuous processes

In this paper, we consider Girsanov transforms of pure jump type for discontinuous Markov processes. We show that, under some quite natural conditions, the Green functions of the Girsanov transformed process are comparable to those of the original process. As an application of the general results, the drift transform of symmetric stable processes is studied in detail. In particular, we show that the relativistic α-stable process in a bounded C^1,1-smooth open set D can be obtained from symmetric α-stable process in D through a combination of a pure jump Girsanov transform and a Feynman-Kac transform. From this, we deduce that the Green functions for these two processes in D are comparable.     

TCP and iso-stationary transformations

We consider piecewise-deterministic Markov processes that occur as scaling limits of discrete-time Markov chains that describe the Transmission Control Protocol (TCP). The class of processes allows for general increase and decrease profiles. Our key observation is that stationary results for the general class follow directly from the stationary results for the idealized TCP process. The latter is a Markov process that increases linearly and experiences downward jumps at times governed by a Poisson process. To establish this connection, we apply space–time transformations that preserve the properties of the class of Markov processes.