Correction to: Using Semi-Markov Chains to Solve Semi-Markov Processes

Methodology and Computing in Applied Probability - The original version of this article contained mistakes, and the author would like to correct them.     

Semi-Markov failure rates processes

Usually, a reliability function is defined by a failure rate which is a real function taking the non-negative real values. In this paper the failure rate is assumed to be a stochastic process with non-negative and right continuous trajectories. The reliability function is defined as an expectation of a function of that random process. Particularly, the failure rate defined by the semi-Markov processes is considered here. The theorems dealing with the renewal equations for the conditional reliability functions with a semi-Markov process as a failure rate are presented in this paper. A system of that kind of equations for the discrete state space semi-Markov process is applied for calculating the reliability function for the 3-states semi-Markov random walk. Using the introduced system of renewal equations for the countable state space, the reliability function for the Furry-Yule failure rate process is obtained.     

Semi-Markov and reward fields

We introduce semi-Markov fields and provide formulations for the basic terms in the semi-Markov theory. In particular we define and consider a class of associated reward fields. Then we present a formula for the expected reward at any multidimensional time epoch. The formula is indeed new even for the classical semi-Markov processes. It gives the expected cumulative reward for fairly large classes of reward functions; in particular, it provides the formulas for the expected cumulative reward given in Masuda and Sumitau (1991), Soltani (1996) and Soltani and Khorshidian (1998).     

Reward Algorithms for Semi-Markov Processes

New algorithms for computing power moments of hitting times and accumulated rewards of hitting type for semi-Markov processes are developed. The algorithms are based on special techniques of sequential phase space reduction and recurrence relations connecting moments of rewards. Applications are discussed as well as possible generalizations of presented results and examples.     

Approximation of Noncooperative Semi-Markov Games

An approximation of a general V-ergodic semi-Markov game with Borel state space by discrete-state space strongly-ergodic games is studied. The standard expected ratio-average criterion as well as the expected time-average criterion are considered. New theorems on the existence of ∊-equilibria are given.Communicated by D. A. CarlsonThe authors thank an anonymous referee for constructive comments. This work is supported by MEiN Grant 1P03A 01030.     

Semi-Markov models for human-resource modelling

For human-resource modelling, models of wastage—such asthe lognormal model—ignore the variations in leaving ratesbetween grades, while models of the grade, structuresuch asthe Markov model—ignore duration dependence inthe ratesof movement to other grades or leaving. The semi-Markov approachallows us to combine a flexible means of modelling durationdependentstay in a grade with a transition matrix of movements betweengrades. We may therefore describe the whole system and predictits future development. We discuss the use of semi-Markov models for human resourcemanagement and provide a means of estimating the parameters.The use of the model to assess promotion patterns is also considered.     

连续时间马氏决策过程的基本假设

在讨论连续时间马氏决策过程(CTMDP)的文献中，对于转移速率qij(t)，常见的假设是：(a)qij(t)是一致有界的^[1-4]；(b)qij(t)关于t是连续的，或者是几乎处处连续的^[4-8]。在本文中，我们放弃了对qij(t)的上述假设(a)和(b)，给出了一些较弱的基本假设以取代(a)，(b)，使CTMDP的研究范围得以扩大。     

Semi-Markov Modelling for Multi-State Systems

In this work we focus on multi state systems that we model by means of semi-Markov processes. The sojourn times are seen to be independent not identically distributed random variables and assumed to belong to a general class of distributions that includes several popular reliability distributions like the exponential, Weibull, and Pareto. We obtain maximum likelihood estimators of the parameters of interest and investigate their asymptotic properties. Plug-in type estimators are furnished for various quantities related to the system under study.     

Semi-Markov processes and α-invariant distributions

A semi-Markov process is easily made Markov by adding some auxiliary random variables. This paper discusses the I-type quasi-stationary distributions of such “extended” processes, and the α-invariant distributions for the corresponding Markov transition probabilities; and we show that there is an intimate relation between the two. The results have relevance in the study of the time to “absorption” or “death” of semi-Markov processes. The particular case of a terminating renewal process is studied as an example.     

Entropy Maximization for Markov and Semi-Markov Processes

The literature about maximum of entropy for Markov processes deals mainly with discrete-time Markov chains. Very few papers dealing with continuous-time jump Markov processes exist and none dealing with semi-Markov processes. It is the aim of this paper to contribute to fill this lack. We recall the basics concerning entropy for Markov and semi-Markov processes and we study several problems to give an overview of the possible directions of use of maximum entropy in connection with these processes. Numeric illustrations are presented, in particular in application to reliability.     

Portfolio Optimization in a Semi-Markov Modulated Market

We address a portfolio optimization problem in a semi-Markov modulated market. We study both the terminal expected utility optimization on finite time horizon and the risk-sensitive portfolio optimization on finite and infinite time horizon. We obtain optimal portfolios in relevant cases. A numerical procedure is also developed to compute the optimal expected terminal utility for finite horizon problem. This work was supported in part by a DST project: SR/S4/MS: 379/06; also supported in part by a grant from UGC via DSA-SAP Phase IV, and in part by a CSIR Fellowship.     

半马尔科夫下电场可修系统可靠性分析

众所周知,在马尔科夫模型下,在每个状态下的逗留时间服从指数分布,这经常太受限制,尤其是在电场可修系统中,不太适合操作数据要求.因此,提出了利用半马尔科夫过程去研究电场可修系统,而且,利用了马尔科夫更新理论和概率分析方法讨论了电场系统的稳态可用度.最后,通过一个数值例子陈述了获得的结果.     

Partially Observable Semi-Markov Games with Discounted Payoff

Abstract

We study partially observable semi-Markov game with discounted payoff on a Borel state space. We study both zero sum and nonzero sum games. We establish saddle point equilibrium and Nash equilibrium for zero sum and nonzero sum cases, respectively.     

Estimating Hidden Semi-Markov Chains From Discrete Sequences

This article addresses the estimation of hidden semi-Markov chains from nonstationary discrete sequences. Hidden semi-Markov chains are particularly useful to model the succession of homogeneous zones or segments along sequences. A discrete hidden semi-Markov chain is composed of a nonobservable state process, which is a semi-Markov chain, and a discrete output process. Hidden semi-Markov chains generalize hidden Markov chains and enable the modeling of various durational structures. From an algorithmic point of view, a new forward-backward algorithm is proposed whose complexity is similar to that of the Viterbi algorithm in terms of sequence length (quadratic in the worst case in time and linear in space). This opens the way to the maximum likelihood estimation of hidden semi-Markov chains from long sequences. This statistical modeling approach is illustrated by the analysis of branching and flowering patterns in plants.     

非负费用折扣半马氏决策过程

本文考虑可数状态非负费用的折扣半马氏决策过程.首先在给定半马氏决策核和策略下构造一个连续时间半马氏决策过程,然后用最小非负解方法证明值函数满足最优方程和存在ε-最优平稳策略,并进一步给出最优策略的存在性条件及其一些性质.最后,给出了值迭代算法和一个数值算例.     

Constrained Semi-Markov decision processes with average rewards

This paper deals with constrained average reward Semi-Markov Decision Processes (SMDPs) with finite state and action sets. We consider two average reward criteria. The first criterion is time-average rewards, which equal the lower limits of the expected average rewards per unit time, as the horizon tends to infinity. The second criterion is ratio-average rewards, which equal the lower limits of the ratios of the expected total rewards during the firstn steps to the expected total duration of thesen steps asn . For both criteria, we prove the existence of optimal mixed stationary policies for constrained problems when the constraints are of the same nature as the objective functions. For unichain problems, we show the existence of randomized stationary policies which are optimal for both criteria. However, optimal mixed stationary policies may be different for each of these critria even for unichain problems. We provide linear programming algorithms for the computation of optimal policies.     

向量值半Markov决策规划的线性加权解法

对向量值半Markov决策规划给出了线性加权解法 .通过该方法还容易地证明了向量值半Markov决策规划存在平稳最优策略的结论 ,并给出了强最优策略存在与否的另一个判别法 .