Dynamic programming in stochastic control of systems with delay

We consider optimal control problems for systems described by stochastic differential equations with delay (SDDE). We prove a version of Bellman's principle of optimality (the dynamic programming principle) for a general class of such problems. That the class in general means that both the dynamics and the cost depends on the past in a general way. As an application, we study systems where the value function depends on the past only through some weighted average. For such systems we obtain a Hamilton-Jacobi-Bellman partial differential equation that the value function must solve if it is smooth enough. The weak uniqueness of the SDDEs we consider is our main tool in proving the result. Notions of strong and weak uniqueness for SDDEs are introduced, and we prove that strong uniqueness implies weak uniqueness, just as for ordinary stochastic differential equations.     

Solvability and Optimal Controls of a Fractional Impulsive Stochastic Partial Integro-Differential Equation with State-Dependent Delay

In this paper, a new class of fractional impulsive stochastic partial integro-differential control systems with state-dependent delay and their optimal controls in a Hilbert space is studied. We firstly prove an existence result of mild solutions for the control systems by using stochastic analysis, analytic \(\alpha \)-resolvent operator, fractional powers of closed operators and suitable fixed point theorems. Then we derive the existence conditions of optimal pairs to the fractional impulsive stochastic control systems. Finally, an example is given to illustrate the effectiveness of our main results.     

Some Solvable Stochastic Control Problems With Delay

We consider optimal harvesting of systems described by stochastic differential equations with delay. We focus on those situations where the value function of the harvesting problem depends on the initial path of the process in a simple way, namely through its value at 0 and through some weighted averages

A verification theorem of variational inequality type is proved. This is applied to solve explicitly some classes of optimal harvesting delay problems     

A numerical scheme to solve nonlinear bsdes with lipschitz and non-lipschitz coefficients

In this paper, we attempt to present a new numerical approach to solve non-linear backward stochastic differential equations. First, we present some definitions and theorems to obtain the conditions, from which we can approximate the non-linear term of the backward stochastic differential equation (BSDE) and we get a continuous piecewise linear BSDE correspond with the original BSDE. We use the relationship between backward stochastic differential equations and stochastic controls by interpreting BSDEs as some stochastic optimal control problems, to solve the approximated BSDE and we prove that the approximated solution converges to the exact solution of the original non-linear BSDE in two different cases.     

An Indefinite Stochastic Linear Quadratic Optimal Control Problem with Delay and Related Forward–Backward Stochastic Differential Equations

In this paper, we will study an indefinite stochastic linear quadratic optimal control problem, where the controlled system is described by a stochastic differential equation with delay. By introducing the relaxed compensator as a novel method, we obtain the well-posedness of this linear quadratic problem for indefinite case. And then, we discuss the uniqueness and existence of the solutions for a kind of anticipated forward–backward stochastic differential delayed equations. Based on this, we derive the solvability of the corresponding stochastic Hamiltonian systems, and give the explicit representation of the optimal control for the linear quadratic problem with delay in an open-loop form. The theoretical results are validated as well on the control problems of engineering and economics under indefinite condition.     

带跳的倒向重随机系统的最大值原理及其应用

本文研究带跳的倒向重随机系统的随机控制问题的最优性条件。在控制域为凸且控制变量进入所有系数条件下,分别以局部形式和全局形式给出必要性最优条件和充分性最优条件。把上述最大值原理应用于重随机线性二次最优控制问题,得到唯一的最优控制,并且给出应用的例子。     

Classical and singular stochastic control for the optimal dividend policy when there is regime switching

Motivated by economic and empirical arguments, we consider a company whose cash surplus is affected by macroeconomic conditions. Specifically, we model the cash surplus as a Brownian motion with drift and volatility modulated by an observable continuous-time Markov chain that represents the regime of the economy. The objective of the management is to select the dividend policy that maximizes the expected total discounted dividend payments to be received by the shareholders. We study two different cases: bounded dividend rates and unbounded dividend rates. These cases generate, respectively, problems of classical stochastic control with regime switching and singular stochastic control with regime switching. We solve these problems, and obtain the first analytical solutions for the optimal dividend policy in the presence of business cycles. We prove that the optimal dividend policy depends strongly on macroeconomic conditions.     

Optimization of Nonlinear Systems of Stochastic Difference Equations

We present new results concerning the synthesis of optimal control for systems of difference equations that depend on a semi-Markov or Markov stochastic process. We obtain necessary conditions for the optimality of solutions that generalize known conditions for the optimality of deterministic systems of control. These necessary optimality conditions are obtained in the form convenient for the synthesis of optimal control. On the basis of Lyapunov stochastic functions, we obtain matrix difference equations of the Riccati type, the integration of which enables one to synthesize an optimal control. The results obtained generalize results obtained earlier for deterministic systems of difference equations.     

The Filtering Equations of Forward-Backward Stochastic Systems with Random Jumps and Applications to Partial Information Stochastic Optimal Control

In this article, we consider a filtering problem for forward-backward stochastic systems that are driven by Brownian motions and Poisson processes. This kind of filtering problem arises from the study of partially observable stochastic linear-quadratic control problems. Combining forward-backward stochastic differential equation theory with certain classical filtering techniques, the desired filtering equation is established. To illustrate the filtering theory, the theoretical result is applied to solve a partially observable linear-quadratic control problem, where an explicit observable optimal control is determined by the optimal filtering estimation.     

Time Discretisation and Rate of Convergence for the Optimal Control of Continuous-Time Stochastic Systems with Delay

We study a semi-discretisation scheme for stochastic optimal control problems whose dynamics are given by controlled stochastic delay (or functional) differential equations with bounded memory. Performance is measured in terms of expected costs. By discretising time in two steps, we construct a sequence of approximating finite-dimensional Markovian optimal control problems in discrete time. The corresponding value functions converge to the value function of the original problem, and we derive an upper bound on the discretisation error or, equivalently, a worst-case estimate for the rate of convergence.     

Mean-field stochastic control with elephant memory in finite and infinite time horizon

ABSTRACT

Our purpose of this paper is to study stochastic control problems for systems driven by mean-field stochastic differential equations with elephant memory, in the sense that the system (like the elephants) never forgets its history. We study both the finite horizon case and the infinite time horizon case.

• In the finite horizon case, results about existence and uniqueness of solutions of such a system are given. Moreover, we prove sufficient as well as necessary stochastic maximum principles for the optimal control of such systems. We apply our results to solve a mean-field linear quadratic control problem.

• For infinite horizon, we derive sufficient and necessary maximum principles.

  As an illustration, we solve an optimal consumption problem from a cash flow modelled by an elephant memory mean-field system.

     

Dynamic convex duality in constrained utility maximization

ABSTRACT

In this paper, we study a constrained utility maximization problem following the convex duality approach. After formulating the primal and dual problems, we construct the necessary and sufficient conditions for both the primal and dual problems in terms of forward and backward stochastic differential equations (FBSDEs) plus some additional conditions. Such formulation then allows us to explicitly characterize the primal optimal control as a function of the adjoint process coming from the dual FBSDEs in a dynamic fashion and vice versa. We also find that the optimal wealth process coincides with the adjoint process of the dual problem and vice versa. Finally we solve three constrained utility maximization problems, which contrasts the simplicity of the duality approach we propose and the technical complexity of solving the primal problem directly.     

Forward–Backward Stochastic Differential Games and Stochastic Control under Model Uncertainty

We study optimal stochastic control problems with jumps under model uncertainty. We rewrite such problems as stochastic differential games of forward–backward stochastic differential equations. We prove general stochastic maximum principles for such games, both in the zero-sum case (finding conditions for saddle points) and for the nonzero sum games (finding conditions for Nash equilibria). We then apply these results to study robust optimal portfolio-consumption problems with penalty. We establish a connection between market viability under model uncertainty and equivalent martingale measures. In the case with entropic penalty, we prove a general reduction theorem, stating that a optimal portfolio-consumption problem under model uncertainty can be reduced to a classical portfolio-consumption problem under model certainty, with a change in the utility function, and we relate this to risk sensitive control. In particular, this result shows that model uncertainty increases the Arrow–Pratt risk aversion index.     

On Stochastic Stabilization of Discrete‐Time Markovian Jump Systems with Delay in State

Abstract

In this paper, we investigate the stochastic stabilization problem for a class of linear discrete time‐delay systems with Markovian jump parameters. The jump parameters considered here is modeled by a discrete‐time Markov chain. Our attention is focused on the design of linear state feedback memoryless controller such that stochastic stability of the resulting closed‐loop system is guaranteed when the system under consideration is either with or without parameter uncertainties. Sufficient conditions are proposed to solve the above problems, which are in terms of a set of solutions of coupled matrix inequalities.     

Deterministic Global Optimization in Nonlinear Optimal Control Problems

The accurate solution of optimal control problems is crucial in many areas of engineering and applied science. For systems which are described by a nonlinear set of differential-algebraic equations, these problems have been shown to often contain multiple local minima. Methods exist which attempt to determine the global solution of these formulations. These algorithms are stochastic in nature and can still get trapped in local minima. There is currently no deterministic method which can solve, to global optimality, the nonlinear optimal control problem. In this paper a deterministic global optimization approach based on a branch and bound framework is introduced to address the nonlinear optimal control problem to global optimality. Only mild conditions on the differentiability of the dynamic system are required. The implementa-tion of the approach is discussed and computational studies are presented for four control problems which exhibit multiple local minima.     

STOCHASTIC MODEL OF LAKE SYSTEM INVASION AND ITS OPTIMAL CONTROL: NEURODYNAMIC PROGRAMMING AS A SOLUTION METHOD

Abstract We develop a stochastic model for the process of spread of an aquatic invader in a lake system. The invader is transported along with recreation boats, and treatment of boats allows one to control the invader spread. Optimization of invasion and control costs leads to a dynamic programming problem. However, standard stochastic dynamic programming (SDP) algorithms allow us to solve the problem for at most 13–14 lakes, which is significantly less than the size of systems arising in applications. We have developed a new technique, which is based upon the ideas of reinforcement learning and neurodynamic programming. This approach allows us to obtain approximate but reasonable control policy for essentially greater lake systems. The resulting numerical technique is comparatively easy to implement, and it can be applied to other spatially extended optimal control problems as well.     

Optimality of monotonic policies for two-action Markovian decision processes,with applications to control of queues with delayed information

We consider a discrete-time Markov decision process with a partially ordered state space and two feasible control actions in each state. Our goal is to find general conditions, which are satisfied in a broad class of applications to control of queues, under which an optimal control policy is monotonic. An advantage of our approach is that it easily extends to problems with both information and action delays, which are common in applications to high-speed communication networks, among others. The transition probabilities are stochastically monotone and the one-stage reward submodular. We further assume that transitions from different states are coupled, in the sense that the state after a transition is distributed as a deterministic function of the current state and two random variables, one of which is controllable and the other uncontrollable. Finally, we make a monotonicity assumption about the sample-path effect of a pairwise switch of the actions in consecutive stages. Using induction on the horizon length, we demonstrate that optimal policies for the finite- and infinite-horizon discounted problems are monotonic. We apply these results to a single queueing facility with control of arrivals and/or services, under very general conditions. In this case, our results imply that an optimal control policy has threshold form. Finally, we show how monotonicity of an optimal policy extends in a natural way to problems with information and/or action delay, including delays of more than one time unit. Specifically, we show that, if a problem without delay satisfies our sufficient conditions for monotonicity of an optimal policy, then the same problem with information and/or action delay also has monotonic (e.g., threshold) optimal policies.     

Maximum Principle of Optimal Control of the Primitive Equations of the Ocean with Two Point Boundary State Constraint

We consider a class of stochastic impulse control problems of general stochastic processes i.e. not necessarily Markovian. Under fairly general conditions we establish existence of an optimal impulse control. We also prove existence of combined optimal stochastic and impulse control of a fairly general class of diffusions with random coefficients. Unlike, in the Markovian framework, we cannot apply quasi-variational inequalities techniques. We rather derive the main results using techniques involving reflected BSDEs and the Snell envelope.     

Quadratic control for linear periodic systems

We propose a new quadratic control problem for linear periodic systems which can be finite or infinite dimensional. We consider both deterministic and stochastic cases. It is a generalization of average cost criterion, which is usually considered for time-invariant systems. We give sufficient conditions for the existence of periodic solutions.Under stabilizability and detectability conditions we show that the optimal control is given by a periodic feedback which involves the periodic solution of a Riccati equation. The optimal closed-loop system has a unique periodic solution which is globally exponentially asymptotically stable. In the stochastic case we also consider the quadratic problem under partial observation. Under two sets of stabilizability and detectability conditions we obtain the separation principle. The filter equation is not periodic, but we show that it can be effectively replaced by a periodic filter. The theory is illustrated by simple examples.This work was done while this author was a visiting professor at the Scuola Normale Superiore, Pisa.     

Stochastic Optimal Control for the Stochastic Heat Equation with Exponentially Growing Coefficients and with Control and Noise on a Subdomain

Abstract

We consider stochastic optimal control problems in Banach spaces, related to nonlinear controlled equations with dissipative non linearities: on the nonlinear term we do not impose any growth condition. The problems are treated via the backward stochastic differential equations approach, that allows also to solve in mild sense Hamilton Jacobi Bellman equations in Banach spaces. We apply the results to controlled stochastic heat equation, in space dimension 1, with control and noise acting on a subdomain.