A computational procedure for the asymptotic analysis of homogeneous semi-Markov processes

A new algorithm for classifying the states of a homogeneous Markov chain having finitely many states is presented, which enables the investigation of the asymptotic behavior of semi-Markov processes in which the Markov chain is embedded. An application of the algorithm to a social security problem is also presented.     

Optimal stopping time on discounted semi-Markov processes

This paper attempts to study the optimal stopping time for semi- Markov processes (SMPs) under the discount optimization criteria with unbounded cost rates. In our work, we introduce an explicit construction of the equivalent semi-Markov decision processes (SMDPs). The equivalence is embodied in the expected discounted cost functions of SMPs and SMDPs, that is, every stopping time of SMPs can induce a policy of SMDPs such that the value functions are equal, and vice versa. The existence of the optimal stopping time of SMPs is proved by this equivalence relation. Next, we give the optimality equation of the value function and develop an effective iterative algorithm for computing it. Moreover, we show that the optimal and ε-optimal stopping time can be characterized by the hitting time of the special sets. Finally, to illustrate the validity of our results, an example of a maintenance system is presented in the end.     

On mean reward variance in semi-Markov processes

As an extension of the discrete-time case, this note investigates the variance of the total cumulative reward for the embedded Markov chain of semi-Markov processes. Under the assumption that the chain is aperiodic and contains a single class of recurrent states recursive formulae for the variance are obtained which show that the variance growth rate is asymptotically linear in time. Expressions are provided to compute this growth rate.     

Stability of stochastic jump-parameter semi-Markov linear systems of differential equations

The asymptotic stability of stochastic Itô-type jump-parameter semi-Markov systems of linear differential equations is examined. A system of integral matrix equations is derived which has the property that the existence of a positive definite solution of the system implies the asymptotic stability of the stochastic semi-Markov system. Finally, an illustrative example is presented.     

Rank tests for a class of semi-Markov models with censored matched pairs

Semi-Markov models are used in a study of censored, matched pairs. A partial likelihood function is obtained in the presence of univariate censoring for a class of semi-Markov models. A test statistics is derived for testing treatment effects of matched-pairs based on this partial likelihood function. Some directions for further generalization are also outlined.     

The robustness of the semi-Markov and delay time single-component inspection models to the Markov assumption

The context of planned preventive maintenance lends itself readilyto probabilistic modelling. Indeed, many of the published theoreticalmodels to be found in the literature adopt a Markov approach,where states are usually ‘operating’, ‘operatingat one of several levels of deterioration’, and ‘failed’.However, most of these models assume the required Markovianproperty and do not address the issue of testing the assumption,or the related task of estimating parameters. It is possiblethat data are inadequate to test the assumption, or that theMarkov property is believed to be not strictly valid, but acceptableas an approximation. In this paper we consider within a specificinspection–maintenance context the robustness of a Markov-basedmodel when the Markov assumption is not valid. This is achievedby comparing the output of an exact delay time model of an inspection–maintenanceproblem with that of a semi-Markov approximation. The importanceof establishing the vadility of the Markov property in the modellingapplication is highlighted. If the plant behaviour is seen tobe nearly Markov, in the case considered the semi-Markov modelgives a good approximation to the exact model. Conversley ifthe Markov assumption is not a good approximation, the semi-Markovmodel can lead to inappropriate advice.     

Finite horizon semi-Markov decision processes with application to maintenance systems

This paper investigates finite horizon semi-Markov decision processes with denumerable states. The optimality is over the class of all randomized history-dependent policies which include states and also planning horizons, and the cost rate function is assumed to be bounded below. Under suitable conditions, we show that the value function is a minimum nonnegative solution to the optimality equation and there exists an optimal policy. Moreover, we develop an effective algorithm for computing optimal policies, derive some properties of optimal policies, and in addition, illustrate our main results with a maintenance system.     

Continuity of mean recurrence times in denumerable semi-Markov processes

Summary For a family of semi-Markov processes where the transition matrices for the embedded Markov chains and the mean sojourn times depend continuously on a parameter, we give equivalent as well as sufficient conditions for the continuity of the mean recurrence times. The results will be used in a subsequent paper on average costs in a dynamic programming model.This work was supported by the Deutsche Forschungsgemeinschaft, Sonderforschungsbereich 72 at the University of Bonn     

A semi-Markov model of disease recurrence in insured dogs

We use a semi-Markov model to analyse the stochastic dynamics of disease occurrence of dogs insured in Canada from 1990 to 1999, and the probability pattern of death from illness. After statistically justifying the use of a stochastic model, we demonstrate that a stationary first-order semi-Markov process is appropriate for analysing the available data set. The probability transition function is estimated and its stationarity is tested statistically. Homogeneity of the semi-Markov model with respect to important covariates (such as geographic location, insurance plan, breed and age) is also statistically examined. We conclude with discussions and implications of our results in veterinary contents. Copyright © 2007 John Wiley & Sons, Ltd.     

Constrained semi-Markov decision processes with ratio and time expected average criteria in Polish spaces

This paper deals with the ratio and time expected average criteria for constrained semi-Markov decision processes (SMDPs). The state and action spaces are Polish spaces, the rewards and costs are unbounded from above and from below, and the mean holding times are allowed to be unbounded from above. First, under general conditions we prove the existence of constrained-optimal policies for the ratio expected average criterion by developing a technique of occupation measures including the mean holding times for SMDPs, which are the generalizations of those for the standard discrete-time and continuous-time MDPs. Then, we give suitable conditions under which we establish the equivalence of the two average criteria by the optional sampling theorem, and thus we show the existence of constrained-optimal policies for the time expected average criterion. Finally, we illustrate the application of our main results with a controlled linear system, for which an exact optimal policy is obtained.     

A note on the Drazin inverse of an anti-triangular matrix

In this paper we give formulae for the generalized Drazin inverse M^d of an anti-triangular matrix M under some conditions. Moreover, some particular cases of these results are also considered.     

Nonparametric Estimation for Semi-Markov Processes Based on its Hazard Rate Functions

The problem of estimating the Markov renewal matrix and the semi-Markov transition matrix based on a history of a finite semi-Markov process censored at time T (fixed) is addressed for the first time. Their asymptotic properties are studied. We begin by the definition of the transition rate of this process and propose a maximum likelihood estimator for the hazard rate functions and then we show that this estimator is uniformly strongly consistent and converges weakly to a normal random variable. We construct a new estimator for an absolute continous semi-Markov kernel and give detailed derivation of uniform strong consistency and weak convergence of this estimator as the censored time tends to infinity. This revised version was published online in June 2006 with corrections to the Cover Date.     

The generalized variance of a stationary autoregressive process

For a stationary autoregressive process of order p and disturbance variance σ² it is shown that the determinant of the covariance of T (≥p) consecutive random variables of the process is (σ²)^T Π_i,j=1^p (1 − w_iw_j)⁻¹, where w₁, …, w_p are the roots of the associated polynomial equation.     

Optimal threshold probability and expectation in semi-Markov decision processes

We consider undiscounted semi-Markov decision process with a target set and our main concern is a problem minimizing threshold probability. We formulate the problem as an infinite horizon case with a recurrent class. We show that an optimal value function is a unique solution to an optimality equation and there exists a stationary optimal policy. Also several value iteration methods and a policy improvement method are given in our model. Furthermore, we investigate a relationship between threshold probabilities and expectations for total rewards.     

Cholesky decomposition of a positive semidefinite matrix with known kernel

The Cholesky decomposition of a symmetric positive semidefinite matrix A is a useful tool for solving the related consistent system of linear equations or evaluating the action of a generalized inverse, especially when A is relatively large and sparse. To use the Cholesky decomposition effectively, it is necessary to identify reliably the positions of zero rows or columns of the factors and to choose these positions so that the nonsingular submatrix of A of the maximal rank is reasonably conditioned. The point of this note is to show how to exploit information about the kernel of A to accomplish both tasks. The results are illustrated by numerical experiments.     

The group inverse of a product

In this paper, we characterize the existence and give an expression of the group inverse of a product of two regular elements by means of a ring unit.     

Self-adjusting stochastic processes

Let A_t(i, j) be the transition matrix at time t of a process with n states. Such a process may be called self-adjusting if the occurrence of the transition from state h to state k at time t results in a change in the hth row such that A_t+1(h, k) ? A_t(h, k). If the self-adjustment (due to transition h → kx) is A_{t + 1}(h, j) = λA_t(h, j) + (1 ? λ)δ_jk (0 < λ < 1), then with probability 1 the process is eventually periodic. If A₀(i, j) < 1 for all i, j and if the self-adjustment satisfies A_{t + 1}(h, k) = ?(A_t(h, k)) with ?(x) twice differentiable and increasing, x < ?(x) < 1 for 0 ? x < 1,?(1) = ?′(1) = 1, then, with probability 1, lim A_t does not exist.     

Multivariate aging properties of epoch times of nonhomogeneous processes

The purpose of this paper is to give conditions on the parameters of nonhomogeneous Poisson and nonhomogeneous pure birth processes, under which the corresponding random vector of the first n epoch times has some multivariate stochastic properties. These results provide an inside to understand the effect of the time over the occurrence of events in such processes. Some applications of these results are given.