Identities for stopped markov chains and their applications

In this paper we obtain identities for some stopped Markov chains. These identities give a unified approach to many problems in optimal stopping of a Markovian sequence, extinction probability of a Markovian branching process and martingale theory.     

Limiting distributions of functionals of Markov chains

Let \s{X_n, n ? 0\s} and \s{Y_n, n ? 0\s} be two stochastic processes such that Y_n depends on X_n in a stationary manner, i.e. P(Y_n ? A\vbX_n) does not depend on n. Sufficient conditions are derived for Y_n to have a limiting distribution. If X_n is a Markov chain with stationary transition probabilities and Y_n = f(X_n,..., X_n+k) then Y_n depends on X_n is a stationary way. Two situations are considered: (i) \s{X_n, n ? 0\s} has a limiting distribution (ii) \s{X_n, n ? 0\s} does not have a limiting distribution and exits every finite set with probability 1. Several examples are considered including that of a non-homogeneous Poisson process with periodic rate function where we obtain the limiting distribution of the interevent times.     

On Markov chains induced by partitioned transition probability matrices

Let S be a denumerable state space and let P be a transition probability matrix on S. If a denumerable set M of nonnegative matrices is such that the sum of the matrices is equal to P, then we call M a partition of P.     

Isomorphism theorems for Markov chains

The isomorphism theorem of Dynkin is definitely an important tool to investigate the problems raised in terms of local times of Markov processes. This theorem concerns continuous time Markov processes. We give here an equivalent version for Markov chains.     

树指标马氏链的等价定义

国内外关于树指标随机过程的研究已经取得了一定的成果.Benjamini和Peres首先给出了树指标马氏链的定义.Berger和叶中行研究了齐次树图上平稳随机场熵率的存在性.杨卫国与刘文研究了树上马氏场的强大数定律与渐近均分性.杨卫国又研究了一般树指标马氏链的强大数定律.为了以后更有效的研究树指标随机过程的一系列相关问题,本文在分析研究前人成果的基础上,给出了树指标马氏链的等价定义,并用数学归纳法证明了其等价性.     

A fluctuation theory for Markov chains

A fluctuation theory for Markov chains on an ordered countable state space is developed, using ladder processes. These are shown to be Markov renewal processes. Results are given for the joint distribution of the extremum (maximum or minimum) and the first time the extremum is achieved. Also a new classification of the states of a Markov chain is suggested. Two examples are given.     

Regret bounds for Narendra-Shapiro bandit algorithms

Narendra-Shapiro (NS) algorithms are bandit-type algorithms developed in the 1960s. NS-algorithms have been deeply studied in infinite horizon but little non-asymptotic results exist for this type of bandit algorithms. In this paper, we focus on a non-asymptotic study of the regret and address the following question: are Narendra-Shapiro bandit algorithms competitive from this point of view? In our main result, we obtain some uniform explicit bounds for the regret of (over)-penalized-NS algorithms. We also extend to the multi-armed case some convergence properties of penalized-NS algorithms towards a stationary Piecewise Deterministic Markov Process (PDMP). Finally, we establish some new sharp mixing bounds for these processes.     

Backward stochastic Volterra integral equations on Markov chains

This paper studies Backward Stochastic Volterra Integral Equations (BSVIEs) driven by finite state, continuous time Markov chains. First, the existence and uniqueness of the solutions to two types of BSVIEs are established. Second, some scalar and vector comparison theorems are given. Finally, the applications of BSVIEs to a linear-quadratic optimal control problem and time-inconsistent coherent risk measures are presented.     

Statistics and probability theory in renewable energy: Teaching and research

In this paper, the key-role and utility of statistics and probability theory in the field of renewable energy are emphasized and illustrated via specific examples. It is demonstrated that renewable energy is a very suitable field to effectively teach and implement many statistical and probabilistic concepts and techniques. From a research point of view, statistical and probabilistic methods have been successfully employed in evaluating renewable energy systems. These methods will continue to be of core interest for the renewable energy sector in the future, as new and more complex renewable energy systems are developed and installed. In this context, some future research directions in relation to the evaluation of renewable energy systems are also presented.     

Estimation of spectral gap for Markov chains

The study of the convergent rate (spectral gap) in theL ²-sense is motivated from several different fields: probability, statistics, mathematical physics, computer science and so on and it is now an active research topic. Based on a new approach (the coupling technique) introduced in [7] for the estimate of the convergent rate and as a continuation of [4], [5], [7–9], [23] and [24], this paper studies the estimate of the rate for time-continuous Markov chains. Two variational formulas for the rate are presented here for the first time for birth-death processes. For diffusions, similar results are presented in an accompany paper [10]. The new formulas enable us to recover or improve the main known results. The connection between the sharp estimate and the corresponding eigenfunction is explored and illustrated by various examples. A previous result on optimal Markovian couplings^[4] is also extended in the paper.Research supported in part by NSFC, Qin Shi Sci & Tech. Foundation and the State Education Commission of China.     

On the representation of finite multiple Markov chains by weighted circuits

In this paper circuit chains of superior order are defined as multiple Markov chains for which transition probabilities are expressed in terms of the weights of a finite class of circuits in a finite set, in connection with kinetic properties along the circuits. Conversely, it is proved that if we join any finite doubly infinite strictly stationary Markov chain of order r for which transitions hold cyclically with a second chain with the same transitions for the inverse time-sense, then they may be represented as circuit chains of order r.     

Nonhomogeneous markov chains with desired properties - the nearest markovian model

A new class of operators performing an optimization (optimization operators or, simply, optimators) which generate transition matrices with required properties such as ergodicity, recurrence etc., is considered and their fundamental features are described. Some criteria for comparing such operators by taking into account their strenght are given and sufficient conditions for both weak and strong ergodicity are derived. The nearest Markovian model with respect to a given set of observed probability vectors is then defined as a sequence of transition matrices satisfying certain constraints that express our prior knowledge about the system. Finally, sufficient conditions for the existence of such a model are given and the related algorithm is illustrated by an example.     

Aggregation of Markov processes: Axiomatization

We give two simple axioms that characterize a simple functional form for aggregation of column stochastic matrices (i.e., Markov processes). Several additional observations are made about such aggregation, including the special case in which the aggregated process is Markovian relative to the original one.     

Stochastic modelling of reservoir operations

The nature of hydrologic parameters in reservoir management models is uncertain. In mathematical programming models the uncertainties are dealt with either indirectly (sensitivity analysis of a deterministic model) or directly by applying a chance-constrained type of formulation or some of the stochastic programming techniques (LP and DP based models). Various approaches are reviewed in the paper. Moran's theory of storage is an alternative stochastic modelling approach to mathematical programming techniques. The basis of the approach and its application is presented. Reliability programming is a stochastic technique based on the chance-constrained approach, where the reliabilities of the chance constraints are considered as extra decision variables in the model. The problem of random event treatment in the reservoir management model formulation using reliability programming is addressed in this paper.     

Exact formula for sensitivity analysis of Markov chains

Given a family of Markov chains whose transition matrices depend on a parameter vector, we given an exact formula for the gradient of the equilibrium distribution with respect to that parameter even in the case of multiple ergodic classes and transient states. This formula generalizes previous results in the ergodic case.     

Local limit theorems for transition densities of Markov chains converging to diffusions

We consider triangular arrays of Markov chains that converge weakly to a diffusion process. Local limit theorems for transition densities are proved. Received: 28 August 1998 / Revised version: 6 September 1999 / Published online: 14 June 2000     

Regenerative structure of Markov chains simulated via common random numbers

A standard strategy in simulation, for comparing two stochastic systems, is to use a common sequence of random numbers to drive both systems. Since regenerative output analysis of the steady-state of a system requires that the process be regenerative, it is of interest to derive conditions under which the method of common random numbers yields a regenerative process. It is shown here that if the stochastic systems are positive recurrent Markov chains with countable state space, then the coupled system is necessarily regenerative; in fact, we allow couplings more general than those induced by common random numbers. An example is given which shows that the regenerative property can fail to hold in general state space, even if the individual systems are regenerative.     

The effect of multiple optima on the simple GA run-time complexity

Genetic algorithms are stochastic search algorithms that have been applied to optimization problems. In this paper we analyze the run-time complexity of a genetic algorithm when we are interested in one of a set of distinguished solutions. One such case occurs when multiple optima exist. We define the worst case scenario and derive a probabilistic worst case bound on the number of iterations required to find one of these multiple solutions of interest.