Markov modulated Bernoulli process

We consider a Bernoulli process where the success probability changes with respect to a Markov chain. Such a model represents an interesting application of stochastic processes where the parameters are not constants; rather, they are stochastic processes themselves due to their dependence on a randomly changing environment. The model operates in a random environment depicted by a Markov chain so that the probability of success at each trial depends on the state of the environment. We will concentrate, in particular, on applications in reliability theory to motivate our model. The analysis will focus on transient as well as long-term behaviour of various processes involved.     

Bayesian Nonparametric Estimation for Reinforced Markov Renewal Processes

Starting from the definitions and the properties of reinforced renewal processes and reinforced Markov renewal processes, we characterize, via exchangeability and de Finetti’s representation theorem, a prior that consists of a family of Dirichlet distributions on the space of Markov transition matrices and beta-Stacy processes on distribution functions. Then, we show that this family is conjugate and give some estimate results.

---

     

Bernoulli thinning of a Markov renewal process

A Bernoulli thinning of a Markov renewal process is investigated. The properties of the thinned process are considered and are related to the properties of the original process. The parameters, moments and equilibrium of the thinned process are determined in terms of the parameters defining the underlying Markov renewal process. Results are illustrated by examples. © 1998 John Wiley & Sons, Ltd.     

隐马尔可夫因子分析模型的半参数贝叶斯分析

因子模型在刻画潜在因素(因子)与观测变量间的影响关系并进而解释多元观测指标(变量)间的相关性方面具有重要作用.在实际应用中,观测数据往往呈现出时序变异多峰,偏态等特性.将经典的因子分析延伸到带有时齐隐马尔可夫模型的动力因子模型,并建立了半参数贝叶斯分析程序.分块GIBBS抽样器用以后验抽样.经验结果展示所建立的统计程序是有效的.     

Bayesian Analysis of a Two-State Markov Modulated Poisson Process

Abstract

We postulate observations from a Poisson process whose rate parameter modulates between two values determined by an unobserved Markov chain. The theory switches from continuous to discrete time by considering the intervals between observations as a sequence of dependent random variables. A result from hidden Markov models allows us to sample from the posterior distribution of the model parameters given the observed event times using a Gibbs sampler with only two steps per iteration.     

Lumpability and Commutativity of Markov Processes

Abstract

We introduce the concepts of lumpability and commutativity of a continuous time discrete state space Markov process, and provide a necessary and sufficient condition for a lumpable Markov process to be commutative. Under suitable conditions we recover some of the basic quantities of the original Markov process from the jump chain of the lumped Markov process.     

Computational Bayesian Analysis of Hidden Markov Models

Abstract

Versions of the Gibbs Sampler are derived for the analysis of data from hidden Markov chains and hidden Markov random fields. The principal new development is to use the pseudolikelihood function associated with the underlying Markov process in place of the likelihood, which is intractable in the case of a Markov random field, in the simulation step for the parameters in the Markov process. Theoretical aspects are discussed and a numerical study is reported.     

Zero Expectile Processes and Bayesian Spatial Regression

We introduce new classes of stationary spatial processes with asymmetric, sub-Gaussian marginal distributions using the idea of expectiles. We derive theoretical properties of the proposed processes. Moreover, we use the proposed spatial processes to formulate a spatial regression model for point-referenced data where the spatially correlated errors have skewed marginal distribution. We introduce a Bayesian computational procedure for model fitting and inference for this class of spatial regression models. We compare the performance of the proposed method with the traditional Gaussian process-based spatial regression through simulation studies and by applying it to a dataset on air pollution in California.     

Generation of Bernoulli processes

We consider five different algorithms for generating Bernoulli processes and discuss their efficiency. Simulation results are presented as well.     

Entropy Maximization for Markov and Semi-Markov Processes

The literature about maximum of entropy for Markov processes deals mainly with discrete-time Markov chains. Very few papers dealing with continuous-time jump Markov processes exist and none dealing with semi-Markov processes. It is the aim of this paper to contribute to fill this lack. We recall the basics concerning entropy for Markov and semi-Markov processes and we study several problems to give an overview of the possible directions of use of maximum entropy in connection with these processes. Numeric illustrations are presented, in particular in application to reliability.     

Uniqueness and Extinction of Weighted Markov Branching Processes

This paper focuses on discussing some basic properties of the weighted Markov branching process which is a natural generalisation of the ordinary Markov branching process. The regularity and uniqueness criteria, which are very easy to verify, are firstly established. Some important characteristics regarding the hitting times of such structure are obtained. In particular, the closed forms for the mean extinction time and conditional mean extinction time are presented. The explosion behaviour of the process is investigated and then the mean explosion time is derived. The mean global holding time and the mean total survival time are also obtained. AMS 2000 Subject Classification Primary 60 J27; Secondary 60 J80     

Continuity of the Value of Competitive Markov Decision Processes

We provide a bound for the variation of the function that assigns to every competitive Markov decision process and every discount factor its discounted value. This bound implies that the undiscounted value of a competitive Markov decision process is continuous in the relative interior of the space of transition rules.     

Bayesian Monte Carlo estimation for profile hidden Markov models

Hidden Markov models are used as tools for pattern recognition in a number of areas, ranging from speech processing to biological sequence analysis. Profile hidden Markov models represent a class of so-called “left–right” models that have an architecture that is specifically relevant to classification of proteins into structural families based on their amino acid sequences. Standard learning methods for such models employ a variety of heuristics applied to the expectation-maximization implementation of the maximum likelihood estimation procedure in order to find the global maximum of the likelihood function. Here, we compare maximum likelihood estimation to fully Bayesian estimation of parameters for profile hidden Markov models with a small number of parameters. We find that, relative to maximum likelihood methods, Bayesian methods assign higher scores to data sequences that are distantly related to the pattern consensus, show better performance in classifying these sequences correctly, and continue to perform robustly with regard to misspecification of the number of model parameters. Though our study is limited in scope, we expect our results to remain relevant for models with a large number of parameters and other types of left–right hidden Markov models.     

Directed Markov Point Processes as Limits of Partially Ordered Markov Models

In this paper, we consider spatial point processes and investigate members of a subclass of the Markov point processes, termed the directed Markov point processes (DMPPs), whose joint distribution can be written in closed form and, as a consequence, its parameters can be estimated directly. Furthermore, we show how the DMPPs can be simulated rapidly using a one-pass algorithm. A subclass of Markov random fields on a finite lattice, called partially ordered Markov models (POMMs), has analogous structure to that of DMPPs. In this paper, we show that DMPPs are the limits of auto-Poisson and auto-logistic POMMs. These and other results reveal a close link between inference and simulation for DMPPs and POMMs.     

A Markov Property for Set-Indexed Processes

We consider a type of Markov property for set-indexed processes which is satisfied by all processes with independent increments and which allows us to introduce a transition system theory leading to the construction of the process. A set-indexed generator is defined such that it completely characterizes the distribution of the process.     

对称马氏过程的Girsanov变换

本文讨论一类对称马氏过程的Girsanov变换,这类Girsanov变换是由该对称马氏过程所联系的狄氏型定义域中的函数来确定的．我们证明了对称马氏过程经变换后还是对称的马氏过程,并且给出经变换后的马氏过程所联系的狄氏型．这些结果将前人的相应结论从有界函数推广到更有应用意义的一类无界函数之上．     

两参数随机过程马氏性间的关系

本文讨论两参数随机过程的马氏性问题,发现了两参数过程X的单点马氏性与X_t~1、X_s~2的马氏性间的关系,两参数中心高斯过程Y的*-马氏性与Y_t~1、Y_s~2的马氏性间的关系.     

A Markov modulated Poisson model for software reliability

In this paper, we consider a latent Markov process governing the intensity rate of a Poisson process model for software failures. The latent process enables us to infer performance of the debugging operations over time and allows us to deal with the imperfect debugging scenario. We develop the Bayesian inference for the model and also introduce a method to infer the unknown dimension of the Markov process. We illustrate the implementation of our model and the Bayesian approach by using actual software failure data.     

Partially Observable Markov Decision Processes incorporating epistemic uncertainties

The use of Markov Decision Processes for Inspection Maintenance and Rehabilitation of civil engineering structures relies on the use of several transition matrices related to the stochastic degradation process, maintenance actions and imperfect inspections. Point estimators for these matrices are usually used and they are evaluated using statistical inference methods and/or expert evaluation methods. Thus, considerable epistemic uncertainty often veils the true values of these matrices. Our contribution through this paper is threefold. First, we present a methodology for incorporating epistemic uncertainties in dynamic programming algorithms used to solve finite horizon Markov Decision Processes (which may be partially observable). Second, we propose a methodology based on the use of Dirichlet distributions which answers, in our sense, much of the controversy found in the literature about estimating Markov transition matrices. Third, we show how the complexity resulting from the use of Monte-Carlo simulations for the transition matrices can be greatly overcome in the framework of dynamic programming. The proposed model is applied to concrete bridge under degradation, in order to provide the optimal strategy for inspection and maintenance. The influence of epistemic uncertainties on the optimal solution is underlined through sensitivity analysis regarding the input data.     

On Queues with Markov Modulated Service Rates

In this paper, we consider a queue whose service speed changes according to an external environment that is governed by a Markov process. It is possible that the server changes its service speed many times while serving a customer. We derive first and second moments of the service time of customers in system using first step analysis to obtain an insight on the service process. In fact, we obtain an intriguing result in that the moments of service time actually depend on the arrival process! We also show that the mean service rate is not the reciprocal of the mean service time. Further, since it is not possible to obtain a closed form expression for the queue length distribution, we use matrix geometric methods to compute performance measures such as average queue length and waiting time. We apply the method of large deviations to obtain tail distributions of the workload in the queue using the concept of effective bandwidth. We present two applications in computer systems: (1) Web server with multi-class requests and (2) CPU with multiple processes. We illustrate the analysis and various methods discussed with the help of numerical examples for the above two applications. AMS subject classification: 90B22, 68M20