Risk-Sensitive Control and an Abstract Collatz–Wielandt Formula

The ‘value’ of infinite horizon risk-sensitive control is the principal eigenvalue of a certain positive operator. For the case of compact domain, Chang has built upon a nonlinear version of the Krein–Rutman theorem to give a ‘min–max’ characterization of this eigenvalue which may be viewed as a generalization of the classical Collatz–Wielandt formula for the Perron–Frobenius eigenvalue of a nonnegative irreducible matrix. We apply this formula to the Nisio semigroup associated with risk-sensitive control and derive a variational characterization of the optimal risk-sensitive cost. For the linear, i.e., uncontrolled case, this is seen to reduce to the celebrated Donsker–Varadhan formula for principal eigenvalue of a second-order elliptic operator.     

A note on negative dynamic programming for risk-sensitive control

Negative dynamic programming for risk-sensitive control is studied. Under some compactness and semicontinuity assumptions the following results are proved: the convergence of the value iteration algorithm to the optimal expected total reward, the Borel measurability or upper semicontinuity of the optimal value functions, and the existence of an optimal stationary policy.     

Risk-sensitive stopping problems for continuous-time Markov chains

In this paper we consider stopping problems for continuous-time Markov chains under a general risk-sensitive optimization criterion for problems with finite and infinite time horizon. More precisely our aim is to maximize the certainty equivalent of the stopping reward minus cost over the time horizon. We derive optimality equations for the value functions and prove the existence of optimal stopping times. The exponential utility is treated as a special case. In contrast to risk-neutral stopping problems it may be optimal to stop between jumps of the Markov chain. We briefly discuss the influence of the risk sensitivity on the optimal stopping time and consider a special house selling problem as an example.     

A RISK-SENSITIVE STOCHASTIC MAXIMUM PRINCIPLE FOR OPTIMAL CONTROL OF JUMP DIFFUSIONS AND ITS APPLICATIONS

A stochastic maximum principle for the risk-sensitive optimal control problem of jump diffusion processes with an exponential-of-integral cost functional is derived assuming that the value function is smooth, where the diffusion and jump term may both depend on the control. The form of the maximum principle is similar to its risk-neutral counterpart. But the adjoint equations and the maximum condition heavily depend on the risk-sensitive parameter. As applications, a linear-quadratic risk-sensitive control problem is solved by using the maximum principle derived and explicit optimal control is obtained.     

Risk-sensitive average continuous-time Markov decision processes with unbounded rates

Abstract

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.     

The maximum principles for partially observed risk-sensitive optimal controls of Markov regime-switching jump-diffusion system

ABSTRACT

This paper studies partially observed risk-sensitive optimal control problems with correlated noises between the system and the observation. It is assumed that the state process is governed by a continuous-time Markov regime-switching jump-diffusion process and the cost functional is of an exponential-of-integral type. By virtue of a classical spike variational approach, we obtain two general maximum principles for the aforementioned problems. Moreover, under certain convexity assumptions on both the control domain and the Hamiltonian, we give a sufficient condition for the optimality. For illustration, a linear-quadratic risk-sensitive control problem is proposed and solved using the main results. As a natural deduction, a fully observed risk-sensitive maximum principle is also obtained and applied to study a risk-sensitive portfolio optimization problem. Closed-form expressions for both the optimal portfolio and the corresponding optimal cost functional are obtained.     

General Maximum Principles for Partially Observed Risk-Sensitive Optimal Control Problems and Applications to Finance

This paper is concerned with partially observed risk-sensitive optimal control problems. Combining Girsanov’s theorem with a standard spike variational technique, we obtain some general maximum principles for the aforementioned problems. One of the distinctive differences between our results and the standard risk-neutral case is that the adjoint equations and variational inequalities strongly depend on a risk-sensitive parameter γ. Two examples are given to illustrate the applications of the theoretical results obtained in this paper. As a natural deduction, a general maximum principle is also obtained for a fully observed risk-sensitive case. At last, this result is applied to study a risk-sensitive optimal portfolio problem. An explicit optimal investment strategy and a cost functional are obtained. A numerical simulation result shows the influence of a risk-sensitive parameter on an optimal investment proportion; this coincides with its economic meaning and theoretical results. This work was partially supported by the National Natural Science Foundation (10671112), the National Basic Research Program of China (973 Program, No. 2007CB814904), the Natural Science Foundation of Shandong Province (Z2006A01) and the Doctoral Fund of the Education Ministry of China.     

Existence of Risk-Sensitive Optimal Stationary Policies for Controlled Markov Processes

In this paper we are concerned with the existence of optimal stationary policies for infinite-horizon risk-sensitive Markov control processes with denumerable state space, unbounded cost function, and long-run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk-sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy. Accepted 1 October 1997     

Zero-sum games for continuous-time Markov jump processes with risk-sensitive finite-horizon cost criterion

In this paper we study the zero-sum games for continuous-time Markov jump processes under the risk-sensitive finite-horizon cost criterion. The state space is a Borel space and the transition rates are allowed to be unbounded. Under the suitable conditions, we use a new value iteration approach to establish the existence of a solution to the risk-sensitive finite-horizon optimality equations of the players, obtain the existence of the value of the game and show the existence of saddle-point equilibria.     

Infinite horizon risk-sensitive control of diffusions without any blanket stability assumptions

We consider the infinite horizon risk-sensitive problem for nondegenerate diffusions with a compact action space, and controlled through the drift. We only impose a structural assumption on the running cost function, namely near-monotonicity, and show that there always exists a solution to the risk-sensitive Hamilton–Jacobi–Bellman (HJB) equation, and that any minimizer in the Hamiltonian is optimal in the class of stationary Markov controls. Under the additional hypothesis that the coefficients of the diffusion are bounded, and satisfy a condition that limits (even though it still allows) transient behavior, we show that any minimizer in the Hamiltonian is optimal in the class of all admissible controls. In addition, we present a sufficient condition, under which the solution of the HJB is unique (up to a multiplicative constant), and establish the usual verification result. We also present some new results concerning the multiplicative Poisson equation for elliptic operators in ${\mathbb{R}}^d$.     

Maximum Principle for Risk-Sensitive Stochastic Optimal Control Problem and Applications to Finance

This article is concerned with a risk-sensitive stochastic optimal control problem motivated by a kind of optimal portfolio choice problem in the financial market. The maximum principle for this kind of problem is obtained, which is similar in form to its risk-neutral counterpart. But the adjoint equations and maximum condition heavily depend on the risk-sensitive parameter. This result is used to solve a kind of optimal portfolio choice problem and the optimal portfolio choice strategy is obtained. Computational results and figures explicitly illustrate the optimal solution and the sensitivity to the volatility rate parameter.     

Risk sensitive control of pure jump processes on a general state space

We study stochastic control problem for pure jump processes on a general state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we prove the existence and Hamilton–Jacobi–Bellman characterization of optimal α-discounted control for bounded cost function. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show the existence of the optimal risk-sensitive ergodic control.     

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.     

Explicit Solution for a Network Control Problem in the Large Deviation Regime

We consider optimal control of a stochastic network, where service is controlled to prevent buffer overflow. We use a risk-sensitive escape time criterion, which in comparison to the ordinary escape time criteria heavily penalizes exits which occur on short time intervals. A limit as the buffer sizes tend to infinity is considered. In [2] we showed that, for a large class of networks, the limit of the normalized cost agrees with the value function of a differential game. In this game, one player controls the service discipline (who to serve and whether to serve), and the other player chooses arrival and service rates in the network. The game's value is characterized in [2] as the unique solution to a Hamilton–Jacobi–Bellman Partial Differential Equation (PDE). In the current paper we apply this general theory to the important case of a network of queues in tandem. Our main results are: (i) the construction of an explicit solution to the corresponding PDE, and (ii) drawing out the implications for optimal risk-sensitive and robust regulation of the network. In particular, the following general principle can be extracted. To avoid buffer overflow there is a natural competition between two tendencies. One may choose to serve a particular queue, since that will help prevent its own buffer from overflowing, or one may prefer to stop service, with the goal of preventing overflow of buffers further down the line. The solution to the PDE indicates the optimal choice between these two, specifying the parts of the state space where each queue must be served (so as not to lose optimality), and where it can idle. Referring to those queues which must be served as bottlenecks, one can use the solution to the PDE to explicitly calculate the bottleneck queues as a function of the system's state, in terms of a simple set of equations.     

Risk-Sensitive Control of Pure Jump Process on Countable Space with Near Monotone Cost

In this article, we study risk-sensitive control problem with controlled continuous time pure jump process on a countable space as state dynamics. We prove multiplicative dynamic programming principle, elliptic and parabolic Harnack’s inequalities. Using the multiplicative dynamic programing principle and the Harnack’s inequalities, we prove the existence and a characterization of optimal risk-sensitive control under the near monotone condition.     

Robust reward–risk ratio optimization with application in allocation of generation asset

In this article, we study reward–risk ratio models under partially known message of random variables, which is called robust (worst-case) performance ratio problem. Based on the positive homogenous and concave/convex measures of reward and risk, respectively, the new robust ratio model is reduced equivalently to convex optimization problems with a min–max optimization framework. Under some specially partial distribution situation, the convex optimization problem is converted into simple framework involving the expectation reward measure and conditional value-at-risk measure. Compared with the existing reward–risk portfolio research, the proposed ratio model has two characteristics. First, the addressed problem combines with two different aspects. One is to consider an incomplete information case in real-life uncertainty. The other is to focus on the performance ratio optimization problem, which can realize the best balance between the reward and risk. Second, the complicated optimization model is transferred into a simple convex optimization problem by the optimal dual theorem. This indeed improves the usability of models. The generation asset allocation in power systems is presented to validate the new models.     

Discrete–time nonlinear filtering with marked point process observations: ii. risk–sensitive filters

We discuss in this article the risk–sensitive filtering problem of estimating a nonlinear signal process, with nonadditive non–Gaussian noise, via a marked point process observation. This extends to the risk sensitive case all the risk–neutral results studied in Dufour and Kannan [2].By going into a change of measure, we derive the unnormalized conditional density of the signal conditioned on the observation history. We also derive the unnormalized prediction density. Using these, we present two separate expressions for the optimal estimate of the signal. A similar analysis of the smoothing density of the signal is also studied under both the risk–sensitive and risk–neutral cases. We specialize the above optimal estimation to the linear signal dynamics and marked point process observation under some Gaussian assumptions. We obtain a Kalman type risk-sensitive filter. Due to the special nature of the observation process, the conditional mean and covariance estimates directly depend now on the point process     

Explicit Solution to a Certain Non-ELQG Risk-sensitive Stochastic Control Problem

A risk-sensitive stochastic control problem with finite/infinite horizon is studied with a 1-dimensional controlled process defined by a linear SDE with a linear control-term in the drift. In the criterion function, a non-linear/quadratic term is introduced by using the solution to a Riccati differential equation, and hence, the problem is not ELQG (Exponential Linear Quadratic Gaussian) in general. For the problem, optimal value and control are calculated in explicit forms and the set of admissible risk-sensitive parameters is given in a concrete form. As applications, two types of large deviations control problems, i.e., maximizing an upside large deviations probability and minimizing a downside large deviations probability, are mentioned.