Average stochastic games for continuous-time jump processes

We consider nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion. The state and action spaces are Borel spaces and reward rates are unbounded. We introduce an approximating sequence of stochastic game models with extended state space, for which the uniform exponential ergodicity is obtained. Moreover, we prove the existence of a stationary almost Markov Nash equilibrium by introducing auxiliary static game models. Finally, a cash flow model is employed to illustrate the results.     

Risk-sensitive finite-horizon piecewise deterministic Markov decision processes

This paper deals with risk-sensitive piecewise deterministic Markov decision processes, where the expected exponential utility of a finite-horizon reward is to be maximized. Both the transition rates and reward functions are allowed to be unbounded. Feynman–Kac’s formula is developed in our setup, using which along with an approximation technique, we establish the associated Hamilton–Jacobi–Bellman equation and the existence of risk-sensitive optimal policies under suitable conditions.     

On categorical time series models with covariates

We study the problem of stationarity and ergodicity for autoregressive multinomial logistic time series models which possibly include a latent process and are defined by a GARCH-type recursive equation. We improve considerably upon the existing conditions about stationarity and ergodicity of those models. Proofs are based on theory developed for chains with complete connections. A useful coupling technique is employed for studying ergodicity of infinite order finite-state stochastic processes which generalize finite-state Markov chains. Furthermore, for the case of finite order Markov chains, we discuss ergodicity properties of a model which includes strongly exogenous but not necessarily bounded covariates.     

Finite-horizon piecewise deterministic Markov decision processes with unbounded transition rates

This paper is concerned with the problem of minimizing the expected finite-horizon cost for piecewise deterministic Markov decision processes. The transition rates may be unbounded, and the cost functions are allowed to be unbounded from above and from below. The optimality is over the general history-dependent policies, where the control is continuously acting in time. The infinitesimal approach is employed to establish the associated Hamilton-Jacobi-Bellman equation, via which the existence of optimal policies is proved. An example is provided to verify all the assumptions proposed.     

A useful technique for piecewise deterministic Markov decision processes

This note presents a technique that is useful for the study of piecewise deterministic Markov decision processes (PDMDPs) with general policies and unbounded transition intensities. This technique produces an auxiliary PDMDP from the original one. The auxiliary PDMDP possesses certain desired properties, which may not be possessed by the original PDMDP. We apply this technique to risk-sensitive PDMDPs with total cost criteria, and comment on its connection with the uniformization technique.     

Blackwell Optimality in the Class of Markov Policies for Continuous-Time Controlled Markov Chains

This paper deals with Blackwell optimality for continuous-time controlled Markov chains with compact Borel action space, and possibly unbounded reward (or cost) rates and unbounded transition rates. We prove the existence of a deterministic stationary policy which is Blackwell optimal in the class of all admissible (nonstationary) Markov policies, thus extending previous results that analyzed Blackwell optimality in the class of stationary policies. We compare our assumptions to the corresponding ones for discrete-time Markov controlled processes.     

Geometric partitioning and robust ad-hoc network design

We consider bounds for the price of a European-style call option under regime switching. Stochastic semidefinite programming models are developed that incorporate a lattice generated by a finite-state Markov chain regime-switching model as a representation of scenarios (uncertainty) to compute bounds. The optimal first-stage bound value is equivalent to a Value at Risk quantity, and the optimal solution can be obtained via simple sorting. The upper (lower) bounds from the stochastic model are bounded below (above) by the corresponding deterministic bounds and are always less conservative than their robust optimization (min-max) counterparts. In addition, penalty parameters in the model allow controllability in the degree to which the regime switching dynamics are incorporated into the bounds. We demonstrate the value of the stochastic solution (bound) and computational experiments using the S&P 500 index are performed that illustrate the advantages of the stochastic programming approach over the deterministic strategy.     

Markov: A methodology for the solution of infinite time horizon markov decision processes

Algorithms are described for determining optimal policies for finite state, finite action, infinite discrete time horizon Markov decision processes. Both value-improvement and policy-improvement techniques are used in the algorithms. Computing procedures are also described. The algorithms are appropriate for processes that are either finite or infinite, deterministic or stochastic, discounted or undiscounted, in any meaningful combination of these features. Computing procedures are described in terms of initial data processing, bound improvements, process reduction, and testing and solution. Application of the methodology is illustrated with an example involving natural resource management. Management implications of certain hypothesized relationships between mallard survival and harvest rates are addressed by applying the optimality procedures to mallard population models.     

一类分支过程的收敛性

随机稳定性是各种随机模型中的至关重要的问题,随机稳定性中的关键问题是找出过程遍历,指数遍历和强遍历的准则．该文对一类重要的分支过程给出了过程指数遍历及强遍历的条件．在证明中主要应用了几种不同的比较方法,从该文的结果可以看出,这种方法是有效的,因而在其它情形中也是非常有意义的．而且所得结果的概率意义也是十分清楚的．     

Discounted continuous-time Markov decision processes with unbounded rates and randomized history-dependent policies: the dynamic programming approach

This paper deals with a continuous-time Markov decision process in Borel state and action spaces and with unbounded transition rates. Under history-dependent policies, the controlled process may not be Markov. The main contribution is that for such non-Markov processes we establish the Dynkin formula, which plays important roles in establishing optimality results for continuous-time Markov decision processes. We further illustrate this by showing, for a discounted continuous-time Markov decision process, the existence of a deterministic stationary optimal policy (out of the class of history-dependent policies) and characterizing the value function through the Bellman equation.     

Inverse problems for ergodicity of Markov chains

For continuous-time Markov chains, we provide criteria for non-ergodicity, non-algebraic ergodicity, non-exponential ergodicity, and non-strong ergodicity. For discrete-time Markov chains, criteria for non-ergodicity, non-algebraic ergodicity, and non-strong ergodicity are given. Our criteria are in terms of the existence of solutions to inequalities involving the Q-matrix (or transition matrix P in time-discrete case) of the chain. Meanwhile, these practical criteria are applied to some examples, including a special class of single birth processes and several multi-dimensional models.     

Large deviations for stochastic partial differential equations driven by a Poisson random measure

Stochastic partial differential equations driven by Poisson random measures (PRMs) have been proposed as models for many different physical systems, where they are viewed as a refinement of a corresponding noiseless partial differential equation (PDE). A systematic framework for the study of probabilities of deviations of the stochastic PDE from the deterministic PDE is through the theory of large deviations. The goal of this work is to develop the large deviation theory for small Poisson noise perturbations of a general class of deterministic infinite dimensional models. Although the analogous questions for finite dimensional systems have been well studied, there are currently no general results in the infinite dimensional setting. This is in part due to the fact that in this setting solutions may have little spatial regularity, and thus classical approximation methods for large deviation analysis become intractable. The approach taken here, which is based on a variational representation for nonnegative functionals of general PRMs, reduces the proof of the large deviation principle to establishing basic qualitative properties for controlled analogues of the underlying stochastic system. As an illustration of the general theory, we consider a particular system that models the spread of a pollutant in a waterway.     

Stochastic volatility Gaussian Heath-Jarrow-Morton models

This paper extends the class of deterministic volatility Heath-Jarrow-Morton models to a Markov chain stochastic volatility framework allowing for jump discontinuities and a variety of deformations of the term structure of forward rate volatilities. Analytical solutions for the dynamics of the volatility term structure are obtained. Semimartingale decompositions of the interest rates under a spot and forward martingale measures are identified. Stochastic volatility versions of the continuous time Ho-Lee and Hull-White extended Vasicek models are obtained. Introducing a regime shift in volatility that is an exponential function of time to maturity leads to a Vasicek dynamics with regime switching coefficients of the short rate.     

Du discret au continu pour des modèles de réseaux aléatoires d'atomes

We study in this Note the continuum (macroscopic) limit for some atomistic models for crystals. The purpose is to derive densities of mechanical energies from microscopic models. In contrast to the setting of a previous study, where the microscopic structure was assumed to be periodic, it is modelled here by a stochastic lattice, which enjoys some stationarity and ergodicity properties, following notions previously introduced elsewhere. To cite this article: X. Blanc et al., C. R. Acad. Sci. Paris, Ser. I 342 (2006).     

Normal approximations for discrete-time occupancy processes

We study normal approximations for a class of discrete-time occupancy processes, namely, Markov chains with transition kernels of product Bernoulli form. This class encompasses numerous models which appear in the complex networks literature, including stochastic patch occupancy models in ecology, network models in epidemiology, and a variety of dynamic random graph models. Bounds on the rate of convergence for a central limit theorem are obtained using Stein’s method and moment inequalities on the deviation from an analogous deterministic model. As a consequence, our work also implies a uniform law of large numbers for a subclass of these processes.     

M/G/∞ with batch arrivals

Let p_∞(n) be the distribution of the number N(∞) in the system at ergodicity for systems with an infinite number of servers, batch arrivals with general batch size distribution and general holding times. This distribution is of importance to a variety of studies in congestion theory, inventory and storage systems. To obtain this distribution, a more general problem is addressed. In this problem, each epoch of a Poisson process gives rise to an independent stochastic function on the lattice of integers, which may be viewed as stochastic impulse response. A continuum analogue to the lattice process is also provided.     

Blackwell optimality in the class of stationary policies in Markov decision chains with a Borel state space and unbounded rewards

This paper is the first part of a study of Blackwell optimal policies in Markov decision chains with a Borel state space and unbounded rewards. We prove here the existence of deterministic stationary policies which are Blackwell optimal in the class of all, in general randomized, stationary policies. We establish also a lexicographical policy improvement algorithm leading to Blackwell optimal policies and the relation between such policies and the Blackwell optimality equation. Our technique is a combination of the weighted norms approach developed in Dekker and Hordijk (1988) for countable models with unbounded rewards and of the weak-strong topology approach used in Yushkevich (1997a) for Borel models with bounded rewards.     

连续时间马尔可夫决策过程的折扣模型

本文考虑的是转移速率族任意且费用率函数可能无界的连续时间马尔可夫决策过程的折扣模型．放弃了传统的要求相应于每个策略的　Ｑ　－过程唯一等条件,而首次考虑相应每个策略的　Ｑ　－过程不一定唯一,　转移速率族也不一定保守,　费用率函数可能无界,　且允许行动空间非空任意的情形．　本文首次用"α－折扣费用最优不等式"更新了传统的α－折扣费用最优方程,并用"最优不等式"和新的方法,不仅证明了传统的主要结果即最优平稳策略的存在性,　而且还进一步探讨了（　∈＞０　　）－最优平稳策略,具有单调性质的最优平稳策略,　以及（∈≥０）　－最优决策过程的存在性,　得到了一些有意义的新结果．　最后,　提供了一个迁移率受控的生灭系统例子,　它满足本文的所有条件,　而传统的假设（见文献［１－１４］）均不成立．     

报酬函数及转移速率族均非一致有界的连续时间折扣马氏决策规划

本文首次在报酬函数及转移速率族均非一致有界的条件下，对可数状态空间，可地动集的连续时间折扣马氏决策规划进行研究，文中引入一类新的无界报酬函数，在一类新的马氏策略中，讨论了最优策略的存在性及春结构，除证明了在有界报酬和一致有界转移速率族下成立的主要结果外，本文还得到一些重要结论。     

Continuous-time constrained stochastic games with average criteria

In this paper, we consider the continuous-time nonzero-sum stochastic games under the constrained average criteria. The state space is denumerable and the action space of each player is a general Polish space. The transition rates, reward and cost functions are allowed to be unbounded. The main hypotheses in this paper include the standard drift conditions, continuity-compactness condition and some ergodicity assumptions. By applying the vanishing discount method, we obtain the existence of stationary constrained average Nash equilibria.