Markov processes on Riesz spaces

Measure-free discrete time stochastic processes in Riesz spaces were formulated and studied by Kuo, Labuschagne and Watson. Aspects relating martingales, stopping times, convergence of these processes as well as various decomposition were considered. Here we formulate and study Markov processes in a measure-free Riesz space setting.     

Markov decision processes with multidimensional action spaces

We study controlled Markov processes where multiple decisions need to be made for each state. We present conditions on the cost structure and the state transition mechanism of the process under which optimal decisions are restricted to a subset of the decision space. As a result, the numerical computation of the optimal policy may be significantly expedited.     

Measure-valued Markov processes and stochastic flows on abstract spaces

We consider measure-valued processes with constant mass in Hilbert space. The stochastic flow which carries the mass satisfies a stochastic differential equation with coefficients depending on the mass distribution. This mass distribution can be considered as the conditional distribution of the solution of a certain SDE. In contrast to the filtration equation, in our case the random measure cannot diffuse: a single particle cannot break up or turn into clouds. The Markov structure of the measure-valued processes obtained is studied and a comparison with Fleming–Viot processes is presented.     

Construction of Markov processes and associated multiplicative functionals from given harmonic measures

Summary LetE be a noncompact locally compact second countable Hausdorff space. We consider the question when, given a family of finite nonzero measures onE that behave like harmonic measures associated with all relatively compact open sets inE (i.e. that satisfy a certain consistency condition), one can construct a Markov process onE and a multiplicative functional with values in [0, ) such that the hitting distributions of the process inflated by the multiplicative functional yield the given harmonic measures. We achieve this construction under weak continuity and local transience conditions on these measures that are natural in the theory of Markov processes, and a mild growth restriction on them. In particular, if the spaceE equipped with the measures satisfies the conditions of a harmonic space, such a Markov process and associated multiplicative functional exist. The result extends in a new direction the work of many authors, in probability and in axiomatic potential theory, on constructing Markov processes from given hitting distributions (i.e. from harmonic measures that have total mass no more than 1).     

Coupling and harmonic functions in the case of continuous time Markov processes

Consider two transient Markov processes (X^v_t)_tεR·, (X^μ_t)_tεR· with the same transition semigroup and initial distributions v and μ. The probability spaces supporting the processes each are also assumed to support an exponentially distributed random variable independent of the process.

We show that there exist (randomized) stopping times S for (X^v_t), T for (X^μ_t) with common final distribution, L(X^v_S|S < ∞) = L(X^μ_T|T < ∞), and the property that for t < S, resp. t < T, the processes move in disjoint portions of the state space. For such a coupling (S, T) it is shown

where denotes the bounded harmonic functions of the Markov transition semigroup. Extensions, consequences and applications of this result are discussed.     

Green's and Dirichlet spaces associated with fine Markov processes

This is the second paper in a series devoted to Green's and Dirichlet spaces. In the first paper, we have investigated Green's space $\text{K}$ and the Dirichlet space $\text{H}$ associated with a symmetric Markov transition function p_t(x, B). Now we assume that p is a transition function of a fine Markov process X and we prove that: (a) the space $\text{H}$ can be built from functions which are right continuous along almost all paths; (b) the positive cone $\text{K}$⁺ in $\text{K}$ can be identified with a cone M of measures on the state space; (c) the positive cone $\text{H}$⁺ in $\text{H}$ can be interpreted as the cone of Green's potentials of measures μ?M. To every measurable set B in the state space E there correspond a subspace $\text{K}$(B) of $\text{K}$ and a subspace $\text{H}$(B) of $\text{H}$. The orthogonal projections of $\text{K}$ onto $\text{K}$ and of $\text{H}$ onto $\text{H}$(B) can be expressed in terms of the hitting probabilities of B by the Markov process X. As the main tool, we use additive functionals of X corresponding to measures μ?M.     

Traces of harmonic functions,capacities, and traces of symmetric Markov processes

We derive explicit isomorphism formulas between weighted Dirichlet integrals for harmonic functions and boundary Dirichlet forms. Applications yield results on traces of Markov processes and convergence quasieverywhere of harmonic functions.     

On harmonic and asymptotically harmonic homogeneous spaces

We classify noncompact homogeneous spaces which are Einstein and asymptotically harmonic. This completes the classification of Riemannian harmonic spaces in the homogeneous case: Any simply connected homogeneous harmonic space is flat, or rank-one symmetric, or a nonsymmetric Damek–Ricci space. Independently, Y. Nikolayevsky has obtained the latter classification under the additional assumption of nonpositive sectional curvatures [N2]. Supported in part by DFG priority program “Global Differential Geometry” (SPP 1154). Received: September 2004; Revision: June 2005; Accepted: September 2005     

Average optimality inequality for continuous-time Markov decision processes in Polish spaces

In this paper, we study the average optimality for continuous-time controlled jump Markov processes in general state and action spaces. The criterion to be minimized is the average expected costs. Both the transition rates and the cost rates are allowed to be unbounded. We propose another set of conditions under which we first establish one average optimality inequality by using the well-known “vanishing discounting factor approach”. Then, when the cost (or reward) rates are nonnegative (or nonpositive), from the average optimality inequality we prove the existence of an average optimal stationary policy in all randomized history dependent policies by using the Dynkin formula and the Tauberian theorem. Finally, when the cost (or reward) rates have neither upper nor lower bounds, we also prove the existence of an average optimal policy in all (deterministic) stationary policies by constructing a “new” cost (or reward) rate. Research partially supported by the Natural Science Foundation of China (Grant No: 10626021) and the Natural Science Foundation of Guangdong Province (Grant No: 06300957).     

Average sample-path optimality for continuous-time Markov decision processes in Polish spaces

In this paper we study the average sample-path cost(ASPC) problem for continuous-time Markov decision processes in Polish spaces.To the best of our knowledge,this paper is a first attempt to study the ASPC criterion on continuous-time MDPs with Polish state and action spaces.The corresponding transition rates are allowed to be unbounded,and the cost rates may have neither upper nor lower bounds.Under some mild hypotheses,we prove the existence of ε(ε≥ 0)-ASPC optimal stationary policies based on two differe...     

Constrained Markov control processes in Borel spaces: the discounted case

We consider constrained discounted-cost Markov control processes in Borel spaces, with unbounded costs. Conditions are given for the constrained problem to be solvable, and also equivalent to an equality-constrained (EC) linear program. In addition, it is shown that there is no duality gap between EC and its dual program EC^*, and that, under additional assumptions, also EC^* is solvable, so that in fact the strong duality condition holds. Finally, a Farkas-like theorem is included, which gives necessary and sufficient conditions for the primal program EC to be consistent.     

Value iteration in average cost Markov control processes on Borel spaces

This paper deals with discrete-time Markov control processes withBorel state and control spaces, with possiblyunbounded costs andnoncompact control constraint sets, and the average cost criterion. Conditions are given for the convergence of the value iteration algorithm to the optimal average cost, and for a sequence of finite-horizon optimal policies to have an accumulation point which is average cost optimal.This research was partially supported by the Consejo Nacional de Ciencia y Tecnología (CONACyT) under grant 1332-E9206.     

Sobolev spaces and harmonic maps between singular spaces

Recently Korevaar and Schoen developed a Sobolev theory for maps from smooth (at least ) manifolds into general metric spaces by proving that the weak limit of appropriate average difference quotients is well behaved. Here we extend this theory to functions defined over Lipschitz manifold. As an application we then prove an existence theorem for harmonic maps from Lipschitz manifolds to NPC metric spaces. Received December 6, 1996 / Accepted March 4, 1997     

Geodesic transformations and harmonic spaces

We prove that a Riemannian manifold is harmonic if and only if there exists a divergence-preserving geodesic transformation with respect to each point which is not volume-preserving.