Deterministic Time-inconsistent Optimal Control Problems - an Essentially Cooperative Approach

A general deterministic time-inconsistent optimal control problem is formulated for ordinary differential equations. To find a time-consistent equilibrium value function and the corresponding time-consistent equilibrium control, a non-cooperative N-person differential game (but essentially cooperative in some sense) is introduced. Under certain conditions, it is proved that the open-loop Nash equilibrium value function of the N -person differential game converges to a time-consistent equilibrium value function of the original problem, which is the value function of a time-consistent optimal control problem. Moreover, it is proved that any optimal control of the time-consistent limit problem is a time-consistent equilibrium control of the original problem.     

Risk sensitive control of pure jump processes on a general state space

We study stochastic control problem for pure jump processes on a general state space with risk sensitive discounted and ergodic cost criteria. For the discounted cost criterion we prove the existence and Hamilton–Jacobi–Bellman characterization of optimal α-discounted control for bounded cost function. For the ergodic cost criterion we assume a Lyapunov type stability assumption and a small cost condition. Under these assumptions we show the existence of the optimal risk-sensitive ergodic control.     

Optimal impulsive control of piecewise deterministic Markov processes

In this paper, we study the infinite-horizon expected discounted continuous-time optimal control problem for Piecewise Deterministic Markov Processes with both impulsive and gradual (also called continuous) controls. The set of admissible control strategies is supposed to be formed by policies possibly randomized and depending on the past-history of the process. We assume that the gradual control acts on the jump intensity and on the transition measure, but not on the flow. The so-called Hamilton–Jacobi–Bellman (HJB) equation associated to this optimization problem is analyzed. We provide sufficient conditions for the existence of a solution to the HJB equation and show that the solution is in fact unique and coincides with the value function of the control problem. Moreover, the existence of an optimal control strategy is proven having the property to be stationary and non-randomized.     

Risk-sensitive average continuous-time Markov decision processes with unbounded rates

Abstract

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.     

Markov Decision Processes with Variance Minimization: A New Condition and Approach

Abstract

This article deals with the limiting average variance criterion for discrete-time Markov decision processes in Borel spaces. The costs may have neither upper nor lower bounds. We propose another set of conditions under which we prove the existence of a variance minimal policy in the class of average expected cost optimal stationary policies. Our conditions are weaker than those in the previous literature. Moreover, some sufficient conditions for the existence of a variance minimal policy are imposed on the primitive data of the model. In particular, the stochastic monotonicity condition in this paper has been first used to study the limiting average variance criterion. Also, the optimality inequality approach provided here is different from the “optimality equation approach” widely used in the previous literature. Finally, we use a controlled queueing system to illustrate our results.     

A Primal-Dual Active Set Method for Bilaterally Control Constrained Optimal Control of the Navier–Stokes Equations

Abstract

This paper is devoted to the analysis and numerical solution of distributed optimal control of the Navier–Stokes equations in presence of bilateral pointwise control constraints. The analysis of the problem involves the proof of existence of an optimal solution, as well as the presentation of necessary and sufficient conditions for optimality. For the numerical solution of the problem we apply a primal-dual active set strategy and show global and local convergence properties of the method. Finally, some numerical experiments, which illustrate the performance of the method, are discussed.     

Turnpike properties for a class of piecewise deterministic systems arising in manufacturing flow control

This paper deals with a general class of piecewise deterministic control systems that encompasses FMS flow control models. One uses the Markov renewal decision process formalism to characterize optimal policies via a discrete event dynamic programming approach. A family of control problems with a random stopping time is associated with these optimality conditions. These problems can be reformulated as infinite horizon deterministic control problems. It is then shown how the so-calledturnpike property should hold for these deterministic control problems under classical convexity assumptions. These turnpikes have the same generic properties as the attractors obtained via a problem specific approach in FMS flow control models and production planning and are calledhedging points in this literature.This research has been supported by NSERC-Canada, Grants No. A4952 by FCAR-Québec, Grant No. 88EQ3528, Actions Structurantes, MESS-Québec, Grant No. 6.1/7.4(28), and FNRS-Switzerland.     

Another Set of Conditions for Strong n (n = −1, 0) Discount Optimality in Markov Decision Processes

Abstract

In this paper we study discrete-time Markov decision processes with average expected costs (AEC) and discount-sensitive criteria in Borel state and action spaces. The costs may have neither upper nor lower bounds. We propose another set of conditions on the system's primitive data, and under which we prove (1) AEC optimality and strong ? 1-discount optimality are equivalent; (2) a condition equivalent to strong 0-discount optimal stationary policies; and (3) the existence of strong n (n = ?1, 0)-discount optimal stationary policies. Our conditions are weaker than those in the previous literature. In particular, the “stochastic monotonicity condition” in this paper has been first used to study strong n (n = ?1, 0)-discount optimality. Moreover, we provide a new approach to prove the existence of strong 0-discount optimal stationary policies. It should be noted that our way is slightly different from those in the previous literature. Finally, we apply our results to an inventory system and a controlled queueing system.     

Another set of conditions for Markov decision processes with average sample-path costs

This paper deals with discrete-time Markov decision processes with average sample-path costs (ASPC) in Borel spaces. The costs may have neither upper nor lower bounds. We propose new conditions for the existence of ε-ASPC-optimal (deterministic) stationary policies in the class of all randomized history-dependent policies. Our conditions are weaker than those in the previous literature. Moreover, some sufficient conditions for the existence of ASPC optimal stationary policies are imposed on the primitive data of the model. In particular, the stochastic monotonicity condition in this paper has first been used to study the ASPC criterion. Also, the approach provided here is slightly different from the “optimality equation approach” widely used in the previous literature. On the other hand, under mild assumptions we show that average expected cost optimality and ASPC-optimality are equivalent. Finally, we use a controlled queueing system to illustrate our results.     

Average optimality inequality for continuous-time Markov decision processes in Polish spaces

In this paper, we study the average optimality for continuous-time controlled jump Markov processes in general state and action spaces. The criterion to be minimized is the average expected costs. Both the transition rates and the cost rates are allowed to be unbounded. We propose another set of conditions under which we first establish one average optimality inequality by using the well-known “vanishing discounting factor approach”. Then, when the cost (or reward) rates are nonnegative (or nonpositive), from the average optimality inequality we prove the existence of an average optimal stationary policy in all randomized history dependent policies by using the Dynkin formula and the Tauberian theorem. Finally, when the cost (or reward) rates have neither upper nor lower bounds, we also prove the existence of an average optimal policy in all (deterministic) stationary policies by constructing a “new” cost (or reward) rate. Research partially supported by the Natural Science Foundation of China (Grant No: 10626021) and the Natural Science Foundation of Guangdong Province (Grant No: 06300957).     

Optimal Control Problem for Cancer Invasion Reaction–Diffusion System

Abstract

An optimal control problem constrained by a reaction–diffusion mathematical model which incorporates the cancer invasion and its treatment is considered. The state equations consisting of three unknown variables namely tumor cell density, normal cell density, and drug concentration. The main goal of the considered optimal control problem is to minimize the density of cancer cells and decreasing the side effects of treatment. Moreover, existence of a weak solution of brain tumor reaction–diffusion system and the corresponding adjoint system of optimal control problem is also investigated. Further, existence of minimizer for the optimal control problem is established and also the first-order optimality conditions are derived.     

Feedback control systems governed by evolution equations

ABSTRACT

The goal of this paper is to provide systematic approaches to study the feedback control systems governed by evolution equations in separable reflexive Banach spaces. We firstly give some existence results of mild solutions for the equations by applying the Banach's fixed point theorem and the Leray–Schauder alternative fixed point theorem with Lipschitz conditions and some types of boundedness conditions. Next, by using the Filippove theorem and the Cesari property, a new set of sufficient assumptions are formulated to guarantee the existence of feasible pairs for the feedback control systems. Some existence results for an optimal control problem are given. Finally, we apply our main result to obtain a controllability result for semilinear evolution equations and existence results for a class of differential variational inequalities and Clarke's subdifferential inclusions.     

Existence of Optimal Controls for Semilinear Parabolic Equations without Cesari-Type Conditions

Abstract. Optimal control problems governed by semilinear parabolic partial differential equations are considered. No Cesari-type conditions are assumed. By proving the existence theorem and the Pontryagin maximum principle of optimal ``state-control" pairs for the corresponding relaxed problems, an existence theorem of optimal pairs for the original problem is established.     

Optimal models for the first arrival time distribution function in continuous time — With a special case

Optimal Models for first arrival time ( _H ) and first arrival target total return (W _H ) distribution functions on MDP in continuous time are presented. Asymptotic expansions of _H andW _H are derived and expressed in simple, explicit forms, and some of their properties are discussed. Two methods to find an optimal policy for distribution function of _H are given. Several necessary and sufficient conditions for the existence of the optimal policy are obtained. This result leads to that the scope of finding the optimal policy is greatly reduced. A special case is also discussed and some deep results are given.     

Deterministic and Stochastic Control of Navier—Stokes Equation with Linear, Monotone, and Hyperviscosities

This paper deals with the optimal control of space—time statistical behavior of turbulent fields. We provide a unified treatment of optimal control problems for the deterministic and stochastic Navier—Stokes equation with linear and nonlinear constitutive relations. Tonelli type ordinary controls as well as Young type chattering controls are analyzed. For the deterministic case with monotone viscosity we use the Minty—Browder technique to prove the existence of optimal controls. For the stochastic case with monotone viscosity, we combine the Minty—Browder technique with the martingale problem formulation of Stroock and Varadhan to establish existence of optimal controls. The deterministic models given in this paper also cover some simple eddy viscosity type turbulence closure models. Accepted 7 June 1999     

Optimal control problem for cancer invasion parabolic system with nonlinear diffusion

ABSTRACT

In this paper, we study a distributed optimal control problem of a coupled nonlinear system of reaction–diffusion equations. The system consists of three partial differential equations to represent cancer cell density, matrix-degrading enzymes concentration and oxygen concentration, and an ordinary differential equation to describe the extracellular matrix concentration. Our aim is to minimize the growth of cancer cells by controlling the production of matrix-degrading enzymes. First, we prove the existence and uniqueness of solutions of the direct problem. Then, we prove the existence of an optimal control. Finally, we derive the first-order optimality conditions and prove the existence of weak solutions of the adjoint problem.     

Optimal Trade Execution Under Stochastic Volatility and Liquidity

Abstract

We study the problem of optimally liquidating a financial position in a discrete-time model with stochastic volatility and liquidity. We consider the three cases where the objective is to minimize the expectation, an expected exponential or a mean-variance criterion of the implementation cost. In the first case, the optimal solution can be fully characterized by a forward-backward system of stochastic equations depending on conditional expectations of future liquidity. In the other two cases, we derive Bellman equations from which the optimal solutions can be obtained numerically by discretizing the control space. In all three cases, we compute optimal strategies for different simulated realizations of prices, volatility and liquidity and compare the outcomes to the ones produced by the deterministic strategies of Bertsimas and Lo (1998; Optimal control of execution costs. Journal of Financial Markets, 1, 1–50) and Almgren and Chriss (2001; Optimal execution of portfolio transactions. Journal of Risk, 3, 5–33).     

HARA frontiers of optimal portfolios in stochastic markets

In this paper, we consider the optimal portfolio selection problem in continuous-time settings where the investor maximizes the expected utility of the terminal wealth in a stochastic market. The utility function has the structure of the HARA family and the market states change according to a Markov process. The states of the market describe the prevailing economic, financial, social and other conditions that affect the deterministic and probabilistic parameters of the model. This includes the distributions of the random asset returns as well as the utility function. We analyzed Black–Scholes type continuous-time models where the market parameters are driven by Markov processes. The Markov process that affects the state of the market is independent of the underlying Brownian motion that drives the stock prices. The problem of maximizing the expected utility of the terminal wealth is investigated and solved by stochastic optimal control methods for exponential, logarithmic and power utility functions. We found explicit solutions for optimal policy and the associated value functions. We also constructed the optimal wealth process explicitly and discussed some of its properties. In particular, it is shown that the optimal policy provides linear frontiers.