Continuous time markov decision processes with interventions

We study Markov jump decision processes with both continuously and instantaneouslyacting decisions and with deterministic drift between jumps. Such decision processes were recentlyintroduced and studied from discrete time approximations point of view by Van der Duyn Schouten.Weobtain necessary and sufficient optimality conditions for these decision processes in terms of equations and inequalities of quasi-variational type. By means of the latter we find simple necessaryand sufficient conditions for the existence of stationary optimal policies in such processes with finite state and action spaces, both in the discounted and average per unit time reward cases.     

Controlled jump processes

Finite and infinite planning horizon Markov decision problems are formulated for a class of jump processes with general state and action spaces and controls which are measurable functions on the time axis taking values in an appropriate metrizable vector space. For the finite horizon problem, the maximum expected reward is the unique solution, which exists, of a certain differential equation and is a strongly continuous function in the space of upper semi-continuous functions. A necessary and sufficient condition is provided for an admissible control to be optimal, and a sufficient condition is provided for the existence of a measurable optimal policy. For the infinite horizon problem, the maximum expected total reward is the fixed point of a certain operator on the space of upper semi-continuous functions. A stationary policy is optimal over all measurable policies in the transient and discounted cases as well as, with certain added conditions, in the positive and negative cases.     

Continuous time Markovian decision processes average return criterion

Continuous time Markovian decision models with countable state space are investigated. The existence of an optimal stationary policy is established for the expected average return criterion function. It is shown that the expected average return can be expressed as an expected discounted return of a related Markovian decision process. A policy iteration method is given which converges to an optimal deterministic policy, the policy so obtained is shown optimal over all Markov policies.     

Decomposable jump decision processes

A large class of continuous parameter jump decision processes is considered. Pontryagin's Maximum Principle is used to derive a necessary condition for optimality. An optimal strategy may frequently be obtained explicitly.     

Continuous time shock markov decision processes with discounted criterion

This paper presents a new concept of Markov decision processes: continuous time shock Markov decision processes, which model Markovian controlled systems sequentially shocked by its environment. Between two adjacent shocks, the system can be modeled by continuous time Markov decision processes. But according to each shock, the system's parameters are changed and an instantaneous state transition occurs. After presenting the model, we prove that the optimality equation, which consists of countable equations, has a unique solution in some function space Ω     

Gradient estimate for Ornstein-Uhlenbeck jump processes

By using absolutely continuous lower bounds of the Lévy measure, explicit gradient estimates are derived for the semigroup of the corresponding Lévy process with a linear drift. A derivative formula is presented for the conditional distribution of the process at time t under the condition that the process jumps before t. Finally, by using bounded perturbations of the Lévy measure, the resulting gradient estimates are extended to linear SDEs driven by Lévy-type processes.     

Periodic solutions of hybrid jump diffusion processes

We investigate periodic solutions of regime-switching jump diffusions. We first show the well-posedness of solutions to stochastic differential equations corresponding to the hybrid system. Then, we derive the strong Feller property and irreducibility of the associated time-inhomogeneous semigroups. Finally, we establish the existence and uniqueness of periodic solutions. Concrete examples are presented to illustrate the results.     

A thinning result for pure jump processes

A standard thinning procedure for point processes is extended to processes of pure jump type in which each jump is retained with probability p or deleted with probability 1 ? p, independently of everything else.Two theorems are proved, the first gives a sufficient condition for the existence of thinned pure jump processes, the second concerns the convergence of such processes to pure jump processes whose increments are generated by a Cox process. Some generalizations are discussed.     

State estimation for partially observed jump processes

In this paper partially observed jump processes are considered and optimal filtering equations are given for the conditional expectation of a functional on the past of the process.Rudemo [6] derived filtering equations for a partially observed jump Markov process. Snyder [3] gives equations for the conditional characteristic function of a jump process. Segall et al. [2] discuss filtering for processes with counting observations. Their work carries over to processes with counting observations the martingale methods that Fujisaki et al. [1] had used to derive nonlinear filtering equations for processes governed by Ito equations. Many further references to filtering for processes with discrete state measurements are given in the references cited.The objective of this paper is to show that by making use of the concept of a representation of a functional the idea of Rudemo's proof of [6, pp. 595–599] can be carried over to jump processes. The author feels that this is a very interesting proof because of its simplicity. It involves only calculations with conditional expectations and the rule for differentiation of a quotient.     

Average stochastic games for continuous-time jump processes

We consider nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion. The state and action spaces are Borel spaces and reward rates are unbounded. We introduce an approximating sequence of stochastic game models with extended state space, for which the uniform exponential ergodicity is obtained. Moreover, we prove the existence of a stationary almost Markov Nash equilibrium by introducing auxiliary static game models. Finally, a cash flow model is employed to illustrate the results.     

The viability property of controlled jump diffusion processes

In this paper, we first give a comparison theorem of viscosity solution to some nonlinear second order integrodifferential equation. And then using the comparison theorem, we obtain a necessary and sufficient condition for the viability property of some controlled jump diffusion processes which can keep the solution within a constraint K.     

Spectral gap for jump processes by decomposition method

By using a decomposition method, we give a criterion for the spectral gap of the reversible general jump process. This criterion enables us to obtain the lower bound for the spectral gap via Lyapunov drift condition. Some examples are presented to illustrate the results.      

Uniqueness in law for pure jump Markov processes

Let A be the operator defined on C ² functions by

Analysis of jump processes with nondegenerate jumping kernels

We prove regularity estimates for functions which are harmonic with respect to certain jump processes. The aim of this article is to extend the method of Bass–Levin (2002) [3] and Bogdan–Sztonyk (2005) [6] to more general processes. Furthermore, we establish a new version of the Harnack inequality that implies regularity estimates for corresponding harmonic functions.