半马氏环境连续时间马氏决策过程：平均准则

本文讨论半马氏环境连续时间马氏决策过程中的平均准则．首先讨论了半马氏报酬过程中的逼近问题，进而讨论平均目标函数逼近问题。     

Quasi-Stationary Asymptotics for Perturbed Semi-Markov Processes in Discrete Time

We consider a discrete time semi-Markov process where the characteristics defining the process depend on a small perturbation parameter. It is assumed that the state space consists of one finite communicating class of states and, in addition, one absorbing state. Our main object of interest is the asymptotic behavior of the joint probabilities of the position of the semi-Markov process and the event of non-absorption as time tends to infinity and the perturbation parameter tends to zero. The main result gives exponential expansions of these probabilities together with a recursive algorithm for computing the coefficients in the expansions. An application to perturbed epidemic SIS models is discussed.     

Optimal Stopping Time on Semi-Markov Processes with Finite Horizon

Journal of Optimization Theory and Applications - In this paper, we consider the optimal stopping problems on semi-Markov processes (sMPs) with finite horizon and aim to establish the existence and...     

Reward Algorithms for Semi-Markov Processes

New algorithms for computing power moments of hitting times and accumulated rewards of hitting type for semi-Markov processes are developed. The algorithms are based on special techniques of sequential phase space reduction and recurrence relations connecting moments of rewards. Applications are discussed as well as possible generalizations of presented results and examples.     

Numerical Solution of non-Homogeneous Semi-Markov Processes in Transient Case*

In this article a numerical solution for the evolution equation of a continuous time non-homogeneous semi-Markov process (NHSMP) is obtained using a quadrature method. The paper, after a short introduction to continuous time NHSMP, presents the numerical solution of the process evolution equation with a general quadrature method. Furthermore, the paper gives results that justify this approach, proving that the numerical solution tends to the evolution equation of the continuous time NHSMP. Moreover, the formulae related to some specific quadrature methods are given and a method for obtaining the discrete time NHSMP by applying a very particular quadrature formula for the discretization is shown. In this way the relation between the continuous and discrete time NHSMP is proved. Then, the problem of obtaining the continuous time NHSMP from the discrete one is considered. This problem is solved showing that the discrete process converges in law to the continuous one if the discretized time interval tends to zero. In addition, the discrete time NHSMP in matrix form is presented, and the fact that the solution to this process always exists is proved. Finally, an algorithm for solving the discrete time NHSMP is given. To illustrate the use of this algorithm for a discrete NHSMP, an example in the area of finance is presented.     

An Algorithmic Approach to Discrete Time Non-homogeneous Backward Semi-Markov Reward Processes with an Application to Disability Insurance

In this paper semi-Markov reward models are presented. Higher moments of the reward process is presented for the first time applied to in time non-homogeneous semi-Markov insurance problems. Also an example is presented based on real disability data. Different algorithmic approaches to solve the problem is described. This work is partly supported by the Knowledge Foundation and Sparbankens Stiftelse Nya. The authors would like to thank the anonymous referee.     

A Non-Homogeneous Continuous Time Semi-Markov Model for the Study of Accumulated Claim Process

The accumulated claim process is the summed total of all claims starting from time t. The semi-Markov environment, at authors’ opinion, is able to follow the evolution of this process. In the paper a continuous time non-homogeneous semi-Markov model with a denumerable set of states will be used to follow the stochastic evolution of the accumulated claim process.     

Entropy Maximization for Markov and Semi-Markov Processes

The literature about maximum of entropy for Markov processes deals mainly with discrete-time Markov chains. Very few papers dealing with continuous-time jump Markov processes exist and none dealing with semi-Markov processes. It is the aim of this paper to contribute to fill this lack. We recall the basics concerning entropy for Markov and semi-Markov processes and we study several problems to give an overview of the possible directions of use of maximum entropy in connection with these processes. Numeric illustrations are presented, in particular in application to reliability.     

Parametric and Non Homogeneous Semi-Markov Process for HIV Control

In AIDS control, physicians have a growing need to use pragmatically useful and interpretable tools in their daily medical taking care of patients. Semi-Markov process seems to be well adapted to model the evolution of HIV-1 infected patients. In this study, we introduce and define a non homogeneous semi-Markov (NHSM) model in continuous time. Then the problem of finding the equations that describe the biological evolution of patient is studied and the interval transition probabilities are computed. A parametric approach is used and the maximum likelihood estimators of the process are given. A Monte Carlo algorithm is presented for realizing non homogeneous semi-Markov trajectories. As results, interval transition probabilities are computed for distinct times and follow-up has an impact on the evolution of patients.      

Estimation of Local Smoothness Coefficients for Continuous Time Processes

We consider processes that satisfied a local Hölder condition with coefficient ₀. According to the sampling times of observations given by i _n with i=0,...,n–1, we study two general classes of estimators for ₀. Their almost sure rates of convergence depend on asymptotic independence of the observed processes, on _n and eventually on an extra parameter ₀. Since this last parameter is in general unknown, we construct a family of preliminary estimators for ₀ with their rates of almost sure convergence. Finally we present some numerical simulations in order to compare the behaviour of our various estimators.     

Central Limit Theorem for Nonlinear Semi-Markov Reward Processes

Abstract

In this work, we obtain a central limit theorem for reward processes defined on a finite state space semi-Markov process, when reward functions assumed to have general forms and are not of constant rates. Martingale theory is the main tool which have been used for establishing the convergence of scaled and shifted reward process to a zero mean Brownian motion. The striking point in this article is considering general forms for the reward functions which are realistic in applications. The conditions needed for these results are existence of variances for sojourn times in each state and second order integrability of reward functions with respect to sojourn times distributions.     

Remarks on Estimation Problem for Stationary Processes in Continuous Time

The problem of estimating of the law (in the space of the paths) and the common marginal distribution for a strictly stationary ergodic process X is discussed. We show, in particular, that:(1) The empirical measure

Discrete and Continuous Time Extremes of Gaussian Processes

Joint distribution of maximums of a Gaussian stationary process in continuous time and in uniform grid on the real axis is studied. When the grid is sufficiently sparse, maxima are asymptotically independent. When the grid is sufficiently tight, the maximums asymptotically coincide. In the boundary case which we call Pickands grid, the limit distribution is non-degenerate. It calculated in terms of a Pickands type constant.AMS 2000 Subject Classification. Primary—60G70, Secondary—60G15*Partially supported by the Scientific foundation of the Netherlands, RFFI grant 0401-00700 and grant DFG 436 RUS 113/722.     

Variational Approach for Learning Markov Processes from Time Series Data

Journal of Nonlinear Science - Inference, prediction, and control of complex dynamical systems from time series is important in many areas, including financial markets, power grid management,...     

Estimation for Discrete-time Semi-Markov Reward Processes: Analysis and Inference

In this paper, the moments of a class of reward processes defined on a discrete-time semi-Markov process and the asymptotic behaviors of the corresponding empirical estimators have been investigated. Some known results concerning the asymptotic distribution and properties of semi-Markov kernel have been obtained by a different approach. By using the empirical estimator of the semi-Markov kernel and the mentioned approach, the estimators for the moments of the reward process have been introduced and their asymptotic properties have been established. As a consequence of the strong consistency and asymptotic normality, the confidence intervals have also been obtained. A numerical example illustrates the results.     

New Average Optimality Conditions for Semi-Markov Decision Processes in Borel Spaces

This paper deals with semi-Markov decision processes under the average expected criterion. The state and action spaces are Borel spaces, and the cost/reward function is allowed to be unbounded from above and from below. We give another set of conditions, under which the existence of an optimal (deterministic) stationary policy is proven by a new technique of two average optimality inequalities. Our conditions are slightly weaker than those in the existing literature, and some new sufficient conditions for the verifications of our assumptions are imposed on the primitive data of the model. Finally, we illustrate our results with three examples.