Risk-sensitive stopping problems for continuous-time Markov chains

In this paper we consider stopping problems for continuous-time Markov chains under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. More precisely our aim is to maximize the certainty equivalent of the stopping reward minus cost over the time horizon. We derive optimality equations for the value functions and prove the existence of optimal stopping times. The exponential utility is treated as a special case. In contrast to risk-neutral stopping problems it may be optimal to stop between jumps of the Markov chain. We briefly discuss the influence of the risk sensitivity on the optimal stopping time and consider a special house selling problem as an example.     

Occupation times for two-state Markov chains

For a two-state Markov chain explicit results are derived for the distribution of the number of visits to state j during the time-interval (1,n], given that the initial state (at time 0) was i. The proof is based on combinatorial results of partition theory.     

Stopping times for quantum Markov chains

In the paper we introduce stopping times for quantum Markov states. We study algebras and maps corresponding to stopping times, give a condition of strong Markov property and give classification of projections for the property of accessibility. Our main result is a new recurrence criterium in terms of stopping times (Theorem 1 and Corollary 2). As an application of the criterium we study how, in Section 6, the quantum Markov chain associated with the one-dimensional Heisenberg (usually non-Markovian) process, obtained from this quantum Markov chain by restriction to a diagonal subalgebra, is such that all its states are recurrent. We were not able to obtain this result from the known recurrence criteria of classical probability.Supported by GNAFA-CNR, Bando n. 211.01.25.     

Transition probabilities for Markov chains having fuzzy states

We consider a Markov Chain in which the states are fuzzy subsets defined on some finite state space. Building on the relationship between set-valued Markov chains to the Dempster-Shafer combination rule, we construct a procedure for finding transition probabilities from one fuzzy state to another. This construction involves Dempster-Shafer type mass functions having fuzzy focal elements. It also involves a measure of the degree to which two fuzzy sets are equal. We also show how to find approximate transition probabilities from a fuzzy state to a crisp state in the original state space     

Future independent times and Markov chains

Summary A random timeT is a future independent time for a Markov chain (X _n ) ₀ ifT is independent of (X _T+n ) _n ^/ =0 and if (X _T+n ) _n ^/ =0 is a Markov chain with initial distribution and the same transition probabilities as (X _n ) ₀ . This concept is used (with the conditional stationary measure) to give a new and short proof of the basic limit theorem of Markov chains, improving somewhat the result in the null-recurrent case.This work was supported by the Swedish Natural Science Research Council and done while the author was visiting the Department of Statistics, Stanford University     

Discrete time Markov chains with interval probabilities

The parameters of Markov chain models are often not known precisely. Instead of ignoring this problem, a better way to cope with it is to incorporate the imprecision into the models. This has become possible with the development of models of imprecise probabilities, such as the interval probability model. In this paper we discuss some modelling approaches which range from simple probability intervals to the general interval probability models and further to the models allowing completely general convex sets of probabilities. The basic idea is that precisely known initial distributions and transition matrices are replaced by imprecise ones, which effectively means that sets of possible candidates are considered. Consequently, sets of possible results are obtained and represented using similar imprecise probability models.We first set up the model and then show how to perform calculations of the distributions corresponding to the consecutive steps of a Markov chain. We present several approaches to such calculations and compare them with respect to the accuracy of the results. Next we consider a generalisation of the concept of regularity and study the convergence of regular imprecise Markov chains. We also give some numerical examples to compare different approaches to calculations of the sets of probabilities.     

Sum rules for hitting times of Markov chains

Using matrix algebra we obtain a general equation for the sum, normalized with suitable constants, of all the expected hitting times in an ergodic Markov chain. This equation yields as corollaries, among others, Broder and Karlin’s formula, Foster’s nth formula and an expression of the Kirchhoff index in terms of the eigenvalues of the Laplacian.     

Discounted Markov games; successive approximation and stopping times

This paper presents a number of successive approximation algorithms for the repeated two-person zero-sum game called Markov game using the criterion of total expected discounted rewards. AsWessels [1977] did for Markov decision processes stopping times are introduced in order to simplify the proofs. It is shown that each algorithm provides upper and lower bounds for the value of the game and nearly optimal stationary strategies for both players.     

Mean arrival times of sets for Markov chains

Translated from Ukrainskii Matematicheskii Zhurnal, Vol. 41, No. 9, pp. 1290–1292, September, 1989.     

Birth times,death times and time substitutions in Markov chains

Summary Given a Markov chain (X _n ) _n0, random times are studied which are birth times or death times in the sense that the post- and pre- processes are independent given the present (X _–1, X ) at time and the conditional post- process (birth times) or the conditional pre- process (death times) is again Markovian. The main result for birth times characterizes all time substitutions through homogeneous random sets with the property that all points in the set are birth times. The main result for death times is the dual of this and appears as the birth time theorem with the direction of time reversed.Part of this work was done while the author was visiting the Department of Mathematics, University of California at San DiegoThe support of The Danish Natural Science Research Council is gratefully acknowledged     

Couplings of markov chains by randomized stopping times

Summary We consider a Markov chain on (E, ) generated by a Markov kernel P. We study the question, when we can find for two initial distributions and two randomized stopping times T of (X _n ) _nN and S of ( X _n ) _nN , such that the distribution of X _T equals the one of X _S and T, S are both finite.The answer is given in terms of -, h with h bounded harmonic, or in terms of .For stopping times T, S for two chains ( X _n ) _nN ,( X _n ) _nN we consider measures , on (E, ) defined as follows: (A)=expected number of visits of ( X _n ) toA before T, (A)=expected number of visits of ( X _n ) toA before S.We show that we can construct T, S such that and are mutually singular and ( _v X _T )=( X _S . We relate and to the positive and negative part of certain solutions of the Poisson equation (I-P)(·)=-.     

Waiting time distributions of runs in higher order Markov chains

We consider a {0,1}-valuedm-th order stationary Markov chain. We study the occurrences of runs where two 1’s are separated byat most/exactly/at least k 0’s under the overlapping enumeration scheme wherek≥0 and occurrences of scans (at leastk ₁ successes in a window of length at mostk, 1≤k ₁≤k) under both non-overlapping and overlapping enumeration schemes. We derive the generating function of first two types of runs. Under the conditions, (1) strong tendency towards success and (2) strong tendency towards reversing the state, we establish the convergence of waiting times of ther-th occurrence of runs and scans to Poisson type distributions. We establish the central limit theorem and law of the iterated logarithm for the number of runs and scans up to timen.     

Bounds on regeneration times and convergence rates for Markov chains

In many applications of Markov chains, and especially in Markov chain Monte Carlo algorithms, the rate of convergence of the chain is of critical importance. Most techniques to establish such rates require bounds on the distribution of the random regeneration time T that can be constructed, via splitting techniques, at times of return to a “small set” C satisfying a minorisation condition P(x,·)(·), xC. Typically, however, it is much easier to get bounds on the time τ_C of return to the small set itself, usually based on a geometric drift function , where . We develop a new relationship between T and τ_C, and this gives a bound on the tail of T, based on ,λ and b, which is a strict improvement on existing results. When evaluating rates of convergence we see that our bound usually gives considerable numerical improvement on previous expressions.     

Markov chains with almost exponential hitting times

If a Markov chain converges rapidly to stationarity, then the time until the first hit on a rarely-visited set of states is approximately exponentially distributed; moreover an explicit bound for the error in this approximation can be given. This complements results of Keilson.     

Stopped Markov chains with stationary occupation times

Summary. Let E be a finite set equipped with a group G of bijective transformations and suppose that X is an irreducible Markov chain on E that is equivariant under the action of G. In particular, if E=G with the corresponding transformations being left or right multiplication, then X is a random walk on G. We show that when X is started at a fixed point there is a stopping time U such that the distribution of the random vector of pre-U occupation times is invariant under the action of G. When G acts transitively (that is, E is a homogeneous space), any non-zero, finite expectation stopping time with this property can occur no earlier than the time S of the first return to the starting point after all states have been visited. We obtain an expression for the joint Laplace transform of the pre-S occupation times for an arbitrary finite chain and show that even for random walk on the group of integers mod r the pre-S occupation times do not generally have a group invariant distribution. This appears to contrast with the Brownian analog, as there is considerable support for the conjecture that the field of local times for Brownian motion on the circle prior to the counterpart of S is stationary under circular shifts. Received: 6 December 1995 / In revised form: 11 June 1997     

Waiting times for clumps of patterns and for structured motifs in random sequences

This paper provides exact probability results for waiting times associated with occurrences of two types of motifs in a random sequence. First, we provide an explicit expression for the probability generating function of the interarrival time between two clumps of a pattern. It allows, in particular, to measure the quality of the Poisson approximation which is currently used for evaluation of the distribution of the number of clumps of a pattern. Second, we provide explicit expressions for the probability generating functions of both the waiting time until the first occurrence, and the interarrival time between consecutive occurrences, of a structured motif. Distributional results for structured motifs are of interest in genome analysis because such motifs are promoter candidates. As an application, we determine significant structured motifs in a data set of DNA regulatory sequences.