Partially Observable Markov Decision Processes incorporating epistemic uncertainties

The use of Markov Decision Processes for Inspection Maintenance and Rehabilitation of civil engineering structures relies on the use of several transition matrices related to the stochastic degradation process, maintenance actions and imperfect inspections. Point estimators for these matrices are usually used and they are evaluated using statistical inference methods and/or expert evaluation methods. Thus, considerable epistemic uncertainty often veils the true values of these matrices. Our contribution through this paper is threefold. First, we present a methodology for incorporating epistemic uncertainties in dynamic programming algorithms used to solve finite horizon Markov Decision Processes (which may be partially observable). Second, we propose a methodology based on the use of Dirichlet distributions which answers, in our sense, much of the controversy found in the literature about estimating Markov transition matrices. Third, we show how the complexity resulting from the use of Monte-Carlo simulations for the transition matrices can be greatly overcome in the framework of dynamic programming. The proposed model is applied to concrete bridge under degradation, in order to provide the optimal strategy for inspection and maintenance. The influence of epistemic uncertainties on the optimal solution is underlined through sensitivity analysis regarding the input data.     

半序度量空间中公共二元随机重合点定理及其应用

在完备可分的半序度量空间中,引入了随机映射对(F,G)关于g随机半序弱增以及(F,G)随机半序弱增的定义,研究了在满足一定非线性压缩条件下的随机映射列F_k:Ω×X×X→X,k=1,2…,g:Ω×X→X和h:Ω×X→X的公共二元随机重合点与公共二元随机不动点问题,所得结果推广了已有文献中的一些不动点定理.     

Averaging in Markov Models with Fast Markov Switches and Applications to Queueing Models

An approximation of Markov type queueing models with fast Markov switches by Markov models with averaged transition rates is studied. First, an averaging principle for two-component Markov process (x _n (t), _n (t)) is proved in the following form: if a component x _n () has fast switches, then under some asymptotic mixing conditions the component _n () weakly converges in Skorokhod space to a Markov process with transition rates averaged by some stationary measures constructed by x _n (). The convergence of a stationary distribution of (x _n (), _n ()) is studied as well. The approximation of state-dependent queueing systems of the type M _M,Q /M _M,Q /m/N with fast Markov switches is considered.     

Computational Bayesian Analysis of Hidden Markov Models

Abstract

Versions of the Gibbs Sampler are derived for the analysis of data from hidden Markov chains and hidden Markov random fields. The principal new development is to use the pseudolikelihood function associated with the underlying Markov process in place of the likelihood, which is intractable in the case of a Markov random field, in the simulation step for the parameters in the Markov process. Theoretical aspects are discussed and a numerical study is reported.     

Conditionally Specified Space-Time Models for Multivariate Processes

This article proposes a class of conditionally specified models for the analysis of multivariate space-time processes. Such models are useful in situations where there is sparse spatial coverage of one of the processes and much more dense coverage of the other process(es). The dependence structure across processes and over space, and time is completely specified through a neighborhood structure. These models are applicable to both point and block sources; for example, multiple pollutant monitors (point sources) or several county-level exposures (block sources). We introduce several computational tricks that are integral for model fitting, give some simple sufficient and necessary conditions for the space-time covariance matrix to be positive definite, and implement a Gibbs sampler, using Hybrid MC steps, to sample from the posterior distribution of the parameters. Model fit is assessed via the DIC. Predictive accuracy, over both time and space, is assessed both relatively and absolutely via mean squared prediction error and coverage probabilities. As an illustration of these models, we fit them to particulate matter and ozone data collected in the Los Angeles, CA, area in 1995 over a three-month period. In these data, the spatial coverage of particulate matter was sparse relative to that of ozone.     

Classification of Mixtures of Spatial Point Processes via Partial Bayes Factors

Motivated by the problem of minefield detection, we investigate the problem of classifying mixtures of spatial point processes. In particular we are interested in testing the hypothesis that a given dataset was generated by a Poisson process versus a mixture of a Poisson process and a hard-core Strauss process. We propose testing this hypothesis by comparing the evidence for each model by using partial Bayes factors. We use the term partial Bayes factor to describe a Bayes factor, a ratio of integrated likelihoods, based on only part of the available information, namely that information contained in a small number of functionals of the data. We applied our method to both real and simulated data, and considering the difficulty of classifying these point patterns by eye, our approach overall produced good results.     

Cluster Algorithms for Spatial Point Processes with Symmetric and Shift Invariant Interactions

Abstract

In this article, Swendsen–Wang–Wolff algorithms are extended to simulate spatial point processes with symmetric and stationary interactions. Convergence of these algorithms is considered. Some further generalizations of the algorithms are discussed. The ideas presented in this article can also be useful in handling some large and complicated systems.     

Lumpability and Commutativity of Markov Processes

Abstract

We introduce the concepts of lumpability and commutativity of a continuous time discrete state space Markov process, and provide a necessary and sufficient condition for a lumpable Markov process to be commutative. Under suitable conditions we recover some of the basic quantities of the original Markov process from the jump chain of the lumped Markov process.     

An Interior Point Heuristic for the Hamiltonian Cycle Problem via Markov Decision Processes

We consider the Hamiltonian cycle problem embedded in a singularly perturbed Markov decision process (MDP). More specifically, we consider the HCP as an optimization problem over the space of long-run state-action frequencies induced by the MDP's stationary policies. We show that Hamiltonian cycles (if any) correspond to the global minima of a suitably constructed indefinite quadratic programming problem over the frequency space. We show that the above indefinite quadratic can be approximated by quadratic functions that are `nearly convex' and as such suitable for the application of logarithmic barrier methods. We develop an interior-point type algorithm that involves an arc elimination heuristic that appears to perform rather well in moderate size graphs. The approach has the potential for further improvements.     

Uniqueness and Extinction of Weighted Markov Branching Processes

This paper focuses on discussing some basic properties of the weighted Markov branching process which is a natural generalisation of the ordinary Markov branching process. The regularity and uniqueness criteria, which are very easy to verify, are firstly established. Some important characteristics regarding the hitting times of such structure are obtained. In particular, the closed forms for the mean extinction time and conditional mean extinction time are presented. The explosion behaviour of the process is investigated and then the mean explosion time is derived. The mean global holding time and the mean total survival time are also obtained. AMS 2000 Subject Classification Primary 60 J27; Secondary 60 J80     

On Dynkin's Markov property of random fields associated with symmetric processes

Let p(t, x, y) be a symmetric transition density with respect to a σ-finite measure m on (E, $\text{E}$), g(x,y)=∫p(t,x,y)dt, and $\text{M}\text{={σ-finite measures μ?0:∫g(x,y)μ(}\text{d}\text{x)μ(}\text{d}\text{y)<∞}}$. There exists a Gaussian random field $\text{Φ={?}{}_{\mathrm{\mu }}\text{:μ?}\text{M}\text{}}$ with mean 0 and covariance $\text{E}\text{?}{}_{\mathrm{\mu }}\text{?}{}_{\mathrm{\nu }}\text{=∫g(x,y)μ(}\text{d}\text{x)ν(}\text{d}\text{y)}$. Letting $\text{F}\text{(B)=σ{?}{}_{\mathrm{\mu }}\text{:μ(B}{}^{\mathrm{c}}\text{)=0}}$ we consider necessary and sufficient conditions for the Markov property (MP) on sets B, C: $\text{F}$(B), $\text{F}$(C) c.i. given $\text{F}$(B ∩ C). Of crucial importance is the following, proved by Dynkin: $\text{E}\text{{?}{}_{\mathrm{\mu }}\text{∣}\text{F}\text{(B)}=?}{}_{\mathrm{\mu B}}$, where μ_B is the hitting distribution of the process corresponding to p, m with initial law μ. Another important fact is that ?_μ=?_ν iff μ, ν have the same potential. Putting these together with an additional transience assumption, we present a potential theoretic proof of the following necessary and sufficient condition for (MP) on sets B, C: For every x?E, T_B∩C=T_B+T_C∮ θ_TB=T_C+T_B∮θ_TC a.s. P^x where, for D ? $\text{E}$, T_D is the hitting time of D for the process associated with p, m. This implies a necessary condition proved by Dynkin in a recent preprint for the case where B∪C=E and B, C are finely closed.     

Estimation in Stationary Markov Renewal Processes,with Application to Earthquake Forecasting in Turkey

Consider a process in which different events occur, with random inter-occurrence times. In Markov renewal processes as well as in semi-Markov processes, the sequence of events is a Markov chain and the waiting distributions depend only on the types of the last and the next event. Suppose that the state-space is finite and that the process started far in the past, achieving stationary. Weibull distributions are proposed for the waiting times and their parameters are estimated jointly with the transition probabilities through maximum likelihood, when one or several realizations of the process are observed over finite windows. The model is illustrated with data of earthquakes of three types of severity that occurred in Turkey during the 20th century.AMS 2000 Subject Classification: 60K20     

两参数随机过程马氏性间的关系

本文讨论两参数随机过程的马氏性问题,发现了两参数过程X的单点马氏性与X_t~1、X_s~2的马氏性间的关系,两参数中心高斯过程Y的*-马氏性与Y_t~1、Y_s~2的马氏性间的关系.     

半序赋范空间及增算子的不动点定理

在赋范线性空间E中定义了由E上连续性泛函确定的半序,并由半序引出E上的锥,讨论了半序和锥的若干性质,最后证明了几个单调增算子的不动点定理。     

Spatial Regression Models for Extremes

Meteorological data are often recorded at a number of spatial locations. This gives rise to the possibility of pooling data through a spatial model to overcome some of the limitations imposed on an extreme value analysis by a lack of information. In this paper we develop a spatial model for extremes based on a standard representation for site-wise extremal behavior, combined with a spatial latent process for parameter variation over the region. A smooth, but possibly non-linear, spatial structure is an intrinsic feature of the model, and difficulties in computation are solved using Markov chain Monte Carlo inference. A simulation study is carried out to illustrate the potential gain in efficiency achieved by the spatial model. Finally, the model is applied to data generated from a climatological model in order to characterize the hurricane climate of the Gulf and Atlantic coasts of the United States.     

马尔可夫骨架过程的极限分布

本文运用基本更新定理和Smith关键更新定理等理论和方法,对马尔可夫骨架过程的极限分布进行深入研究,得到主要结果如下:去掉了原有结果中要求的绝对连续的条件,给出了马尔可夫骨架过程极限分布存在的充分条件;得到了马尔可夫骨架过程极限分布的具体公式,并证明了该极限分布为概率分布.     

随机环境中两性分枝过程的马氏性和对偶性

证明了独立同分布环境中的两性分枝过程是时奇的马氏链,给出了过程灭绝一爆炸这一对偶性的一个新的证明。在随机环境情形下,证明了一类单调函数的存在性。     

On Entire Moments of Self-Similar Markov Processes

We consider a risk-based asset allocation problem in a Markov, regime-switching, pure jump model. With a convex risk measure of the terminal wealth of an investor as a proxy for risk, we formulate the risk-based asset allocation problem as a zero-sum, two-person, stochastic differential game between the investor and the market. The HJB dynamic programming approach is used to discuss the game problem. A semi-analytical solution of the game problem is obtained in a particular case.     

Continuity of the Value of Competitive Markov Decision Processes

We provide a bound for the variation of the function that assigns to every competitive Markov decision process and every discount factor its discounted value. This bound implies that the undiscounted value of a competitive Markov decision process is continuous in the relative interior of the space of transition rules.