Decision diagrams for the approximate analysis of Markov models

Decision diagrams of various types can be used to encode the exact state space and transition rate matrix of large Markov models. However, the exact solution of such models still requires to store at least one real vector with one entry per reachable state, a formidable limitation to the practical use of these encodings. Thus, we discuss automatic techniques for the approximate computation of performance measures when the Markov model can be compactly encoded but not exactly solved. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

向量值马氏决策规划的线性加权解法

类似［１－４］,折扣向量值马氏决策规划（ＤＶＭＤＰ）描述为： ｛Ｓ,（Ａ（ｉ）,ｉ∈Ｓ）,ｑ,ｒ,Ｖβ｝,（１）其中Ｓ为可数状态集,Ａ（ｉ）是有限决策集,ｑ（ｊ｜ｉ,ａ）是转移概率,ｒ＝ｒ（ｉ,ｊ,ａ）＝（ｒ１（ｉ,ｊ,ａ）,ｒ２（ｉ,ｊ,ａ）,…,ｒｐ（ｉ,ｊ,ａ））是状态ｉ处使用决策ａ于下一步转移到状态ｊ时所获的ｐ维报酬向量,ｒ（ｉ,ｊ,ａ）对ｉ,ｊ,ａ一致有界,Ｖβ是β折扣目标． 以Π表一般策略集,若　π　∈Π有 Ｖβ（π）≤ Ｖβ（π*）,则称π* 为ＤＶＭＤＰ的“强有效策略”,Ｖβ（π*）为“…     

Second order bounds for Markov Decision Processes

The paper describes a bound on the optimal value of a Markov Decision Process using the iterates of the value iteration algorithm. Previous bounds have depended upon the values of the last two iterations, but here we consider a bound which depends on the last three iterates. We show that in a maximisation problem, this leads to a better lower bound on the optimal value.     

An exact solution for a cavity model

Résumé On considère l'écoulement plan, permanent et irrotationnel d'un jet gazeux aux grandes vitesses subsoniques. En appliquant le procédé deFalkovich, concernant la méthode hodographique deChaplygin, on obtient la solution exacte pour le modèle de Roshko situé dans un canal aux parois parallèles (Figure). On détermine l'expression exacte du coefficient de résistance et on donne quelques relations entre les différents paramètres de la configuration.     

On the Generation of Markov Decision Processes

Comparisons of the performance of solution algorithms for Markov decision processes rely heavily on problem generators to provide sizeable sets of test problems. Existing generation techniques allow little control over the properties of the test problems and often result in problems which are not typical of real-world examples. This paper identifies the properties of Markov decision processes which affect the performance of solution algorithms, and also describes a new problem generation technique which allows all of these properties to be controlled.     

The Transformation Method for Continuous-Time Markov Decision Processes

In this paper, we show that a discounted continuous-time Markov decision process in Borel spaces with randomized history-dependent policies, arbitrarily unbounded transition rates and a non-negative reward rate is equivalent to a discrete-time Markov decision process. Based on a completely new proof, which does not involve Kolmogorov??s forward equation, it is shown that the value function for both models is given by the minimal non-negative solution to the same Bellman equation. A verifiable necessary and sufficient condition for the finiteness of this value function is given, which induces a new condition for the non-explosion of the underlying controlled process.     

离散事件系统静态稳定性的马氏决策过程方法

本文用马氏决策过程方法来讨论离散事件系统（DES）的静态稳定性问题,包括强吸引域和弱吸引域的计算,同时讨论了弱吸引域的稳定控制器的计算问题,所用方法不要求给定谓词的∑－（或∑u-）不变性和对环路的判别。     

Phase diagrams of multicomponent lattice models

We consider q-component models on both the integer lattice ℤ² and a second-order Cayley tree and study phase diagrams of these models. __________ Translated from Teoreticheskaya i Matematicheskaya Fizika, Vol. 149, No. 2, pp. 244–251, November, 2006.     

The exact solution of the general stochastic rumour

A characterization is given of the complete time-dependent evolution of a general stochastic rumour which includes the two Daley-Kendall models and the Maki-Thompson model as special cases.     

Leroux's method for general hidden Markov models

The method introduced by Leroux [Maximum likelihood estimation for hidden Markov models, Stochastic Process Appl. 40 (1992) 127–143] to study the exact likelihood of hidden Markov models is extended to the case where the state variable evolves in an open interval of the real line. Under rather minimal assumptions, we obtain the convergence of the normalized log-likelihood function to a limit that we identify at the true value of the parameter. The method is illustrated in full details on the Kalman filter model.     

Nonsingular dynamical systems,Bratteli diagrams and Markov odometers

We prove that any ergodic non-singular transformation is orbit equivalent to a Markov odometer which is uniquely ergodic.     

Posterior consistency for partially observed Markov models

We establish the posterior consistency for parametric, partially observed, fully dominated Markov models. The prior is assumed to assign positive probability to all neighborhoods of the true parameter, for a distance induced by the expected Kullback–Leibler divergence between the parametric family members’ Markov transition densities. This assumption is easily checked in general. In addition, we show that the posterior consistency is implied by the consistency of the maximum likelihood estimator. The result is extended to possibly improper priors and non-stationary observations. Finally, we check our assumptions on a linear Gaussian model and a well-known stochastic volatility model.     

Adirectional Markov models for 2D systems

Several deterministic Markovian properties are introduced inan acausal context. We analyse the relationship between theseproperties and investigate to which kind of models they correspondin the scalar case. We find that, in this case, the nearest-neighbourmodel is the natural model describing such systems. In the generalmultivariable case, we need higher-order models even in theID case.