Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.     

Markov Decision Processes on Borel Spaces with Total Cost and Random Horizon

This paper deals with Markov Decision Processes (MDPs) on Borel spaces with possibly unbounded costs. The criterion to be optimized is the expected total cost with a random horizon of infinite support. In this paper, it is observed that this performance criterion is equivalent to the expected total discounted cost with an infinite horizon and a varying-time discount factor. Then, the optimal value function and the optimal policy are characterized through some suitable versions of the Dynamic Programming Equation. Moreover, it is proved that the optimal value function of the optimal control problem with a random horizon can be bounded from above by the optimal value function of a discounted optimal control problem with a fixed discount factor. In this case, the discount factor is defined in an adequate way by the parameters introduced for the study of the optimal control problem with a random horizon. To illustrate the theory developed, a version of the Linear-Quadratic model with a random horizon and a Logarithm Consumption-Investment model are presented.     

Large deviations and related problems for absorbing Markov chains

In this paper, large deviations and their connections with several other fundamental topics are investigated for absorbing Markov chains. A variational representation for the Dirichlet principal eigenvalues is given by the large deviation approach. Kingman’s decay parameters and mean ratio quasi-stationary distributions of the chains are also characterized by the large deviation rate function. As an application of these results, we interpret the “stationarity” of mean ratio quasi-stationary distributions via a concrete example. An application to quasi-ergodicity is also discussed.     

Backward stochastic differential equations associated to jump Markov processes and applications

In this paper we study backward stochastic differential equations (BSDEs) driven by the compensated random measure associated to a given pure jump Markov process X$X$ on a general state space K$K$. We apply these results to prove well-posedness of a class of nonlinear parabolic differential equations on K$K$, that generalize the Kolmogorov equation of X$X$. Finally we formulate and solve optimal control problems for Markov jump processes, relating the value function and the optimal control law to an appropriate BSDE that also allows to construct probabilistically the unique solution to the Hamilton–Jacobi–Bellman equation and to identify it with the value function.     

Explosion,implosion, and moments of passage times for continuous-time Markov chains: A semimartingale approach

We establish general theorems quantifying the notion of recurrence–through an estimation of the moments of passage times–for irreducible continuous-time Markov chains on countably infinite state spaces. Sharp conditions of occurrence of the phenomenon of explosion are also obtained. A new phenomenon of implosion is introduced and sharp conditions for its occurrence are proven. The general results are illustrated by treating models having a difficult behaviour even in discrete time.     

Optimal advertising and pricing in a class of general new-product adoption models

In [21], Sethi et al. introduced a particular new-product adoption model. They determine optimal advertising and pricing policies of an associated deterministic infinite horizon discounted control problem. Their analysis is based on the fact that the corresponding Hamilton–Jacobi–Bellman (HJB) equation is an ordinary non-linear differential equation which has an analytical solution. In this paper, generalizations of their model are considered. We take arbitrary adoption and saturation effects into account, and solve finite and infinite horizon discounted variations of associated control problems. If the horizon is finite, the HJB-equation is a 1st order non-linear partial differential equation with specific boundary conditions. For a fairly general class of models we show that these partial differential equations have analytical solutions. Explicit formulas of the value function and the optimal policies are derived. The controlled Bass model with isoelastic demand is a special example of the class of controlled adoption models to be examined and will be analyzed in some detail.     

Infinite horizon stopping problems with (nearly) total reward criteria

We study an infinite horizon optimal stopping Markov problem which is either undiscounted (total reward) or with a general Markovian discount rate. Using ergodic properties of the underlying Markov process, we establish the feasibility of the stopping problem and prove the existence of optimal and ε$\epsilon$-optimal stopping times. We show the continuity of the value function and its variational characterisation (in the viscosity sense) under different sets of assumptions satisfied by large classes of diffusion and jump–diffusion processes. In the case of a general discounted problem we relax a classical assumption that the discount rate is uniformly separated from zero.     

Markov snakes and superprocesses

Summary We suggest the name Markov snakes for a class of path-valued Markov processes introduced recently by J.-F. Le Gall in connection with the theory of branching measure-valued processes. Le Gall applied this class to investigate path properties of superdiffusions and to approach probabilistically partial differential equations involving a nonlinear operator v–v ². We establish an isomorphism theorem which allows to translate results on continuous superprocesses into the language of Markov snakes and vice versa. By using this theorem, we get limit theorems for discrete Markov snakes.Partially supported by National Science Foundation Grant DMS-9301315 and by The US Army Research Office through the Mathematical Sciences Institute at Cornell University     

Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.     

A pathwise approach to the extinction of branching processes with countably many types

We consider the extinction events of Galton–Watson processes with countably infinitely many types. In particular, we construct truncated and augmented Galton–Watson processes with finite but increasing sets of types. A pathwise approach is then used to show that, under some sufficient conditions, the corresponding sequence of extinction probability vectors converges to the global extinction probability vector of the Galton–Watson process with countably infinitely many types. Besides giving rise to a family of new iterative methods for computing the global extinction probability vector, our approach paves the way to new global extinction criteria for branching processes with countably infinitely many types.     

Large time asymptotic problems for optimal stochastic control with superlinear cost

The paper is concerned with stochastic control problems of finite time horizon whose running cost function is of superlinear growth with respect to the control variable. We prove that, as the time horizon tends to infinity, the value function converges to a function of variable separation type which is characterized by an ergodic stochastic control problem. Asymptotic problems of this type arise in utility maximization problems in mathematical finance. From the PDE viewpoint, our results concern the large time behavior of solutions to semilinear parabolic equations with superlinear nonlinearity in gradients.     

On the adaptive control of a class of partially observed Markov decision processes

This paper is concerned with the adaptive control problem, over the infinite horizon, for partially observable Markov decision processes whose transition functions are parameterized by an unknown vector. We treat finite models and impose relatively mild assumptions on the transition function. Provided that a sequence of parameter estimates converging in probability to the true parameter value is available, we show that the certainty equivalence adaptive policy is optimal in the long-run average sense.     

Drift transforms and Green function estimates for discontinuous processes

In this paper, we consider Girsanov transforms of pure jump type for discontinuous Markov processes. We show that, under some quite natural conditions, the Green functions of the Girsanov transformed process are comparable to those of the original process. As an application of the general results, the drift transform of symmetric stable processes is studied in detail. In particular, we show that the relativistic α-stable process in a bounded C^1,1-smooth open set D can be obtained from symmetric α-stable process in D through a combination of a pure jump Girsanov transform and a Feynman-Kac transform. From this, we deduce that the Green functions for these two processes in D are comparable.     

Optimal stopping for partially observed piecewise-deterministic Markov processes

This paper deals with the optimal stopping problem under partial observation for piecewise-deterministic Markov processes. We first obtain a recursive formulation of the optimal filter process and derive the dynamic programming equation of the partially observed optimal stopping problem. Then, we propose a numerical method, based on the quantization of the discrete-time filter process and the inter-jump times, to approximate the value function and to compute an ?$?$-optimal stopping time. We prove the convergence of the algorithms and bound the rates of convergence.     

Extinction properties of multi-type continuous-state branching processes

Recently in Barczy et al. (2015), the notion of a multi-type continuous-state branching process (with immigration) having $d$-types was introduced as a solution to an $d$-dimensional vector-valued SDE. Preceding that, work on affine processes, originally motivated by mathematical finance, in Duffie et al. (2003) also showed the existence of such processes. See also more recent contributions in this direction due to Gabrielli and Teichmann (2014) and Caballero and Pérez Garmendia (2017). Older work on multi-type continuous-state branching processes is more sparse but includes Watanabe (1969) and Ma (2013), where only two types are considered. In this paper we take a completely different approach and consider multi-type continuous-state branching process, now allowing for up to a countable infinity of types, defined instead as a super Markov chain with both local and non-local branching mechanisms. In the spirit of Engländer and Kypriano (2004) we explore their extinction properties and pose a number of open problems.     

Reflected BSDEs and robust optimal stopping for dynamic risk measures with jumps

We study the optimal stopping problem for dynamic risk measures represented by Backward Stochastic Differential Equations (BSDEs) with jumps and its relation with reflected BSDEs (RBSDEs). The financial position is given by an RCLL adapted process. We first state some properties of RBSDEs with jumps when the obstacle process is RCLL only. We then prove that the value function of the optimal stopping problem is characterized as the solution of an RBSDE. The existence of optimal stopping times is obtained when the obstacle is left-upper semi-continuous along stopping times. Finally, we investigate robust optimal stopping problems related to the case with model ambiguity and their links with mixed control/optimal stopping game problems. We prove that, under some hypothesis, the value function is equal to the solution of an RBSDE. We then study the existence of saddle points when the obstacle is left-upper semi-continuous along stopping times.     

Mirror and synchronous couplings of geometric Brownian motions

The paper studies the question of whether the classical mirror and synchronous couplings of two Brownian motions minimise and maximise, respectively, the coupling time of the corresponding geometric Brownian motions. We establish a characterisation of the optimality of the two couplings over any finite time horizon and show that, unlike in the case of Brownian motion, the optimality fails in general even if the geometric Brownian motions are martingales. On the other hand, we prove that in the cases of the ergodic average and the infinite time horizon criteria, the mirror coupling and the synchronous coupling are always optimal for general (possibly non-martingale) geometric Brownian motions. We show that the two couplings are efficient if and only if they are optimal over a finite time horizon and give a conjectural answer for the efficient couplings when they are suboptimal.     

Optimal stopping with irregular reward functions

We consider optimal stopping problems with finite horizon for one-dimensional diffusions. We assume that the reward function is bounded and Borel-measurable, and we prove that the value function is continuous and can be characterized as the unique solution of a variational inequality in the sense of distributions.     

带灾难的非线性马尔可夫分枝过程

本文在文献[6]的基础上，集中考虑一类带灾难的非线性马尔可夫分枝过程的基本问题-唯一性，正则性和灭绝性。文章首先给出其Q-过程唯一性的证明，然后得出该畔程的正则性与[3]非线性马尔币夫分枝过程一样，最后，我们给出该Q-过程以概1l灭绝的充要条件是Q-过程正则。     

Non-randomized policies for constrained Markov decision processes

This paper addresses constrained Markov decision processes, with expected discounted total cost criteria, which are controlled by non-randomized policies. A dynamic programming approach is used to construct optimal policies. The convergence of the series of finite horizon value functions to the infinite horizon value function is also shown. A simple example illustrating an application is presented.