连续时间马氏决策过程的基本假设

在讨论连续时间马氏决策过程(CTMDP)的文献中，对于转移速率qij(t)，常见的假设是：(a)qij(t)是一致有界的^[1-4]；(b)qij(t)关于t是连续的，或者是几乎处处连续的^[4-8]。在本文中，我们放弃了对qij(t)的上述假设(a)和(b)，给出了一些较弱的基本假设以取代(a)，(b)，使CTMDP的研究范围得以扩大。     

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.     

New Average Optimality Conditions for Semi-Markov Decision Processes in Borel Spaces

This paper deals with semi-Markov decision processes under the average expected criterion. The state and action spaces are Borel spaces, and the cost/reward function is allowed to be unbounded from above and from below. We give another set of conditions, under which the existence of an optimal (deterministic) stationary policy is proven by a new technique of two average optimality inequalities. Our conditions are slightly weaker than those in the existing literature, and some new sufficient conditions for the verifications of our assumptions are imposed on the primitive data of the model. Finally, we illustrate our results with three examples.     

Reliability Measures of Semi-Markov Systems with General State Space

The aim of this paper is to present a systematic modeling of reliability and related measures: availability, maintainability, failure rate, rate of occurrence of failures, mean times, etc., known in the literature under the term dependability. This model includes the continuous and discrete time semi-Markov processes with general state space. This is one of the most general models in reliability theory since it includes as particular cases the Markov and renewal processes.     

The Discounted Method and Equivalence of Average Criteria for Risk-Sensitive Markov Decision Processes on Borel Spaces

This note concerns discrete-time controlled Markov chains with Borel state and action spaces. Given a nonnegative cost function, the performance of a control policy is measured by the superior limit risk-sensitive average criterion associated with a constant and positive risk sensitivity coefficient. Within such a framework, the discounted approach is used (a) to establish the existence of solutions for the corresponding optimality inequality, and (b) to show that, under mild conditions on the cost function, the optimal value functions corresponding to the superior and inferior limit average criteria coincide on a certain subset of the state space. The approach of the paper relies on standard dynamic programming ideas and on a simple analytical derivation of a Tauberian relation.     

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.     

基于状态空间模型与改进HSMM相结合的装备故障诊断方法研究

提出了一种将状态空间模型与隐半马尔科夫模型(HSMM)相结合的装备故障诊断方法,将能有效利用直至前一时刻状态监测数据的HSMM预测概率模型与能有效利用直至当前状态监测数据的状态空间模型进行了有机结合.针对已有的HSMM研究中状态持续时间均服从指数分布,小样本下求解状态持续时间分布难的缺陷,以及将每一监测时刻的状态剩余持续时间与状态的持续时间完全等同起来造成诊断误差的不足,对HSMM进行了两点改进:一是将状态持续时间的概率分布连续化、参数化,并将其先验分布设定为威布尔分布;二是基于状态开始时间识别提出了状态剩余持续时间的概念.对基于改进HSMM的预测概率进行了推导,给出了基于直至当前状态监测数据下故障诊断判据的计算公式和应用这一新方法进行故障诊断的步骤.案例研究表明方法是合理有效的.     

A Counterexample on Sample-Path Optimality in Stable Markov Decision Chains with the Average Reward Criterion

This note deals with Markov decision chains evolving on a denumerable state space. Under standard continuity-compactness requirements, an explicit example is provided to show that, with respect to a strong sample-path average reward criterion, the Lyapunov function condition does not ensure the existence of an optimal stationary policy.     

一般状态空间马氏链随机泛函的矩

研究了一般状态空间马氏链随机泛函的矩,利用最小非负解理论,得到了随机泛函的矩是相应方程的最小非负解,作为应用,证明了随机泛函的矩与漂移条件等价.     

Queuing Systems with Semi-Markov Flow in Average and Diffusion Approximation Schemes

We study asymptotic average and diffusion approximation schemes for semi-Markov queuing systems by a random evolution approach and using compensating operator of the corresponding extended Markov renewal process. These results generalize Markov and renewal flow queuing systems.      

Optimal Stopping Time on Semi-Markov Processes with Finite Horizon

Journal of Optimization Theory and Applications - In this paper, we consider the optimal stopping problems on semi-Markov processes (sMPs) with finite horizon and aim to establish the existence and...     

The Central Limit Theorem for Markov Chains with General State Space

We consider a Markov chain with general state space and an embedded Markov chain sampled at the times of successive returns to a subsetA₀ of the state space.We assume that the latter chain is uniformly ergodic but the originalMarkov chain need not possess this property.We develop amodification of the spectralmethod and utilize it in proving the central limit theorem for theMarkov chain under consideration.     

A System of Poisson Equations for a Nonconstant Varadhan Functional on a Finite State Space

Given a discrete-time Markov chain with finite state space and a stationary transition matrix, a system of "local" Poisson equations characterizing the (exponential) Varadhan's functional J(·) is given. The main results, which are derived for an arbitrary transition structure so that J(·) may be nonconstant, are as follows: (i) Any solution to the local Poisson equations immediately renders Varadhan's functional, and (ii) a solution of the system always exist. The proof of this latter result is constructive and suggests a method to solve the local Poisson equations.     

Borel状态空间非平稳MDP的平均方差准则

本文考虑具有 Ｂｏｒｅｌ状态空间和行动空间非平稳 ＭＤＰ的平均方差准则．首先,在遍历条件下,利用最优方程,证明了关于平均期望目标最优马氏策略的存在性．然后,通过构造新的模型,利用马氏过程的理论,进一步证明了在关于平均期望目标是最优的一类马氏策略中,存在一个马氏策略使得平均方差达到最小．作为本文的特例还得到了 Ｄｙｎｋｉｎ Ｅ． Ｂ．和 Ｙｕｓｈｋｅｖｉｃｈ Ａ． Ａ．及 Ｋｕｒａｎｏ Ｍ．等中的主要结果．