Risk-sensitive average continuous-time Markov decision processes with unbounded rates

Abstract

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.     

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.     

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.     

Risk-Sensitive Control of Pure Jump Process on Countable Space with Near Monotone Cost

In this article, we study risk-sensitive control problem with controlled continuous time pure jump process on a countable space as state dynamics. We prove multiplicative dynamic programming principle, elliptic and parabolic Harnack’s inequalities. Using the multiplicative dynamic programing principle and the Harnack’s inequalities, we prove the existence and a characterization of optimal risk-sensitive control under the near monotone condition.     

Risk sensitive control of pure jump processes on a general state space

We study stochastic control problem for pure jump processes on a general state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we prove the existence and Hamilton–Jacobi–Bellman characterization of optimal α-discounted control for bounded cost function. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show the existence of the optimal risk-sensitive ergodic control.     

Mean-Field Pontryagin Maximum Principle

This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.     

A RISK-SENSITIVE STOCHASTIC MAXIMUM PRINCIPLE FOR OPTIMAL CONTROL OF JUMP DIFFUSIONS AND ITS APPLICATIONS

A stochastic maximum principle for the risk-sensitive optimal control problem of jump diffusion processes with an exponential-of-integral cost functional is derived assuming that the value function is smooth, where the diffusion and jump term may both depend on the control. The form of the maximum principle is similar to its risk-neutral counterpart. But the adjoint equations and the maximum condition heavily depend on the risk-sensitive parameter. As applications, a linear-quadratic risk-sensitive control problem is solved by using the maximum principle derived and explicit optimal control is obtained.     

A note on negative dynamic programming for risk-sensitive control

Negative dynamic programming for risk-sensitive control is studied. Under some compactness and semicontinuity assumptions the following results are proved: the convergence of the value iteration algorithm to the optimal expected total reward, the Borel measurability or upper semicontinuity of the optimal value functions, and the existence of an optimal stationary policy.     

The maximum principles for partially observed risk-sensitive optimal controls of Markov regime-switching jump-diffusion system

ABSTRACT

This paper studies partially observed risk-sensitive optimal control problems with correlated noises between the system and the observation. It is assumed that the state process is governed by a continuous-time Markov regime-switching jump-diffusion process and the cost functional is of an exponential-of-integral type. By virtue of a classical spike variational approach, we obtain two general maximum principles for the aforementioned problems. Moreover, under certain convexity assumptions on both the control domain and the Hamiltonian, we give a sufficient condition for the optimality. For illustration, a linear-quadratic risk-sensitive control problem is proposed and solved using the main results. As a natural deduction, a fully observed risk-sensitive maximum principle is also obtained and applied to study a risk-sensitive portfolio optimization problem. Closed-form expressions for both the optimal portfolio and the corresponding optimal cost functional are obtained.     

Optimality equations and inequalities in a class of risk-sensitive average cost Markov decision chains

This note concerns controlled Markov chains on a denumerable sate space. The performance of a control policy is measured by the risk-sensitive average criterion, and it is assumed that (a) the simultaneous Doeblin condition holds, and (b) the system is communicating under the action of each stationary policy. If the cost function is bounded below, it is established that the optimal average cost is characterized by an optimality inequality, and it is to shown that, even for bounded costs, such an inequality may be strict at every state. Also, for a nonnegative cost function with compact support, the existence an uniqueness of bounded solutions of the optimality equation is proved, and an example is provided to show that such a conclusion generally fails when the cost is negative at some state.     

一类半鞅状态的平稳型脉冲随机控制

本文提出了一类新的随机控制模型,这类模型不但在费用结构上推广了此前的平稳型脉冲随机控制,而且首次将一类半鞅引入脉冲控制模型的状态结构从而推广了相应的状态过程.通过对一类相当复杂的变分方程问题的研究并利用其有关结论,我们证明了新模型最佳控制的存在性并刻划出其结构.     

Dynamic asset management with risk-sensitive criterion and non-negative factor constraints: a differential game approach

In this paper, we study continuous time portfolio optimization problem where individual securities are directly affected by economic factors. We consider the risk-sensitive criterion function as is familiar in the robust control literature. This is the natural setting for studying the infinite horizon case of the control problem arising in portfolio optimization. Our result extends earlier works by imposing explicitly the non-negativity constraint on the economic factors. This is achieved by using reflected diffusions. The risk-sensitive control problem with reflected diffusion is then converted into a stochastic differential game. The lower value of this game leads immediately to the desired optimal strategy. Also we prove the existence of unique strong solution to reflected diffusions with bounded measurable drift coefficient which is the first result of its kind for higher dimensional reflected diffusions.     

General Maximum Principles for Partially Observed Risk-Sensitive Optimal Control Problems and Applications to Finance

This paper is concerned with partially observed risk-sensitive optimal control problems. Combining Girsanov’s theorem with a standard spike variational technique, we obtain some general maximum principles for the aforementioned problems. One of the distinctive differences between our results and the standard risk-neutral case is that the adjoint equations and variational inequalities strongly depend on a risk-sensitive parameter γ. Two examples are given to illustrate the applications of the theoretical results obtained in this paper. As a natural deduction, a general maximum principle is also obtained for a fully observed risk-sensitive case. At last, this result is applied to study a risk-sensitive optimal portfolio problem. An explicit optimal investment strategy and a cost functional are obtained. A numerical simulation result shows the influence of a risk-sensitive parameter on an optimal investment proportion; this coincides with its economic meaning and theoretical results. This work was partially supported by the National Natural Science Foundation (10671112), the National Basic Research Program of China (973 Program, No. 2007CB814904), the Natural Science Foundation of Shandong Province (Z2006A01) and the Doctoral Fund of the Education Ministry of China.     

Locally Optimal Risk-Sensitive Controllers for Strict-Feedback Nonlinear Systems1,2

For a class of risk-sensitive nonlinear stochastic control problems with dynamics in strict-feedback form, we obtain through a constructive derivation state-feedback controllers which (i) are locally optimal, (ii) are globally inverse optimal, and (iii) lead to closed-loop system trajectories that are bounded in probability. The first feature implies that a linearized version of these controllers solve a linear exponential-quadratic Gaussian (LEQG) problem, and the second feature says that there exists an appropriate cost function according to which these controllers are optimal.     

Remarks on Risk-Sensitive Control Problems

The main purpose of this paper is to investigate the asymptotic behavior of the discounted risk-sensitive control problem for periodic diffusion processes when the discount factor $\alpha$ goes to zero. If $u_\alpha(\theta,x)$ denotes the optimal cost function, $\theta$ being the risk factor, then it is shown that $\lim_{\alpha\to 0}\alpha u_\alpha(\theta,x)=\xi(\theta)$ where $\xi(\theta)$ is the average on $]0,\theta[$ of the optimal cost of the (usual) infinite horizon risk-sensitive control problem.     

The Discounted Method and Equivalence of Average Criteria for Risk-Sensitive Markov Decision Processes on Borel Spaces

This note concerns discrete-time controlled Markov chains with Borel state and action spaces. Given a nonnegative cost function, the performance of a control policy is measured by the superior limit risk-sensitive average criterion associated with a constant and positive risk sensitivity coefficient. Within such a framework, the discounted approach is used (a) to establish the existence of solutions for the corresponding optimality inequality, and (b) to show that, under mild conditions on the cost function, the optimal value functions corresponding to the superior and inferior limit average criteria coincide on a certain subset of the state space. The approach of the paper relies on standard dynamic programming ideas and on a simple analytical derivation of a Tauberian relation.     

Risk-Sensitive Dynamic Asset Management

This paper develops a continuous time portfolio optimization model where the mean returns of individual securities or asset categories are explicitly affected by underlying economic factors such as dividend yields, a firm's return on equity, interest rates, and unemployment rates. In particular, the factors are Gaussian processes, and the drift coefficients for the securities are affine functions of these factors. We employ methods of risk-sensitive control theory, thereby using an infinite horizon objective that is natural and features the long run expected growth rate, the asymptotic variance, and a single risk-aversion parameter. Even with constraints on the admissible trading strategies, it is shown that the optimal trading strategy has a simple characterization in terms of the factor levels. For particular factor levels, the optimal trading positions can be obtained as the solution of a quadratic program. The optimal objective value, as a function of the risk-aversion parameter, is shown to be the solution of a partial differential equation. A simple asset allocation example, featuring a Vasicek-type interest rate which affects a stock index and also serves as a second investment opportunity, provides some additional insight about the risk-sensitive criterion in the context of dynamic asset management. Accepted 10 December 1997     

Successive approximations in partially observable controlled Markov chains with risk-sensitive average criterion

Partially observable Markov decision chains with finite state, action and signal spaces are considered. The performance index is the risk-sensitive average criterion and, under conditions concerning reachability between the unobservable states and observability of the signals, it is shown that the value iteration algorithm can be implemented to approximate the optimal average cost, to determine a stationary policy whose performance index is arbitrarily close to the optimal one, and to establish the existence of solutions to the optimality equation. The results rely on an appropriate extension of the well-known Schweitzer's transformation.     

跳扩散模型下基金平衡管理的最优脉冲控制

在基金市值波动服从跳扩散过程, 基金持有的罚金成本为当前基金水平的二次函数及存在交易费的假设下研究了无穷时域的基金平衡管理的最小成本模型. 利用随机最优脉冲控制的拟变分不等式理论建立了判定定理,得到了最优脉冲控制策略的存在性, 同时通过构造方法给出了解的数学结构形式.