A Deterministic Approach To Stochastic Optimal Control With Application To Anticipative Control

Using the decomposition of solution of SDE, we consider the stochastic optimal control problem with anticipative controls as a family of deterministic control problems parametrized by the paths of the driving Wiener process and of a newly introduced Lagrange multiplier stochastic process (nonanticipativity equality constraint). It is shown that the value function of these problems is the unique global solution of a robust equation (random partial differential equation) associated to a linear backward Hamilton-Jacobi-Bellman stochastic partial differential equation (HJB SPDE). This appears as limiting SPDE for a sequence of random HJB PDE's when linear interpolation approximation of the Wiener process is used. Our approach extends the Wong-Zakai type results [20] from SDE to the stochastic dynamic programming equation by showing how this arises as average of the limit of a sequence of deterministic dynamic programming equations. The stochastic characteristics method of Kunita [13] is used to represent the value function. By choosing the Lagrange multiplier equal to its nonanticipative constraint value the usual stochastic (nonanticipative) optimal control and optimal cost are recovered. This suggests a method for solving the anticipative control problems by almost sure deterministic optimal control. We obtain a PDE for the “cost of perfect information” the difference between the cost function of the nonanticipative control problem and the cost of the anticipative problem which satisfies a nonlinear backward HJB SPDE. Poisson bracket conditions are found ensuring this has a global solution. The cost of perfect information is shown to be zero when a Lagrangian submanifold is invariant for the stochastic characteristics. The LQG problem and a nonlinear anticipative control problem are considered as examples in this framework     

Optimal stopping of switching diffusions with state dependent switching rates

This paper is concerned with a continuous-time and infinite-horizon optimal stopping problem in switching diffusion models. In contrast to the assumption commonly made in the literature that the regime-switching is modeled by an independent Markov chain, we consider in this paper the case of state-dependent regime-switching. The Hamilton–Jacobi–Bellman (HJB) equation associated with the optimal stopping problem is given by a system of coupled variational inequalities. By means of the dynamic programming (DP) principle, we prove that the value function is the unique viscosity solution of the HJB system. As an interesting application in mathematical finance, we examine the problem of pricing perpetual American put options with state-dependent regime-switching. A numerical procedure is developed based on the DP approach and an efficient discrete tree approximation of the continuous asset price process. Numerical results are reported.     

Optimal impulsive control of piecewise deterministic Markov processes

In this paper, we study the infinite-horizon expected discounted continuous-time optimal control problem for Piecewise Deterministic Markov Processes with both impulsive and gradual (also called continuous) controls. The set of admissible control strategies is supposed to be formed by policies possibly randomized and depending on the past-history of the process. We assume that the gradual control acts on the jump intensity and on the transition measure, but not on the flow. The so-called Hamilton–Jacobi–Bellman (HJB) equation associated to this optimization problem is analyzed. We provide sufficient conditions for the existence of a solution to the HJB equation and show that the solution is in fact unique and coincides with the value function of the control problem. Moreover, the existence of an optimal control strategy is proven having the property to be stationary and non-randomized.     

An approximate-analytical solution for the Hamilton–Jacobi–Bellman equation via homotopy perturbation method

In this paper, we give an analytical-approximate solution for the Hamilton–Jacobi–Bellman (HJB) equation arising in optimal control problems using He’s polynomials based on homotopy perturbation method (HPM). Applying the HPM with He’s polynomials, solution procedure becomes easier, simpler and more straightforward. The comparison of the HPM results with the exact solution, Modal series and measure theoretical method are made. Some illustrative examples are given to demonstrate the simplicity and efficiency of the proposed method.     

Application of variational iteration method for Hamilton–Jacobi–Bellman equations

In this paper, we use the variational iteration method (VIM) for optimal control problems. First, optimal control problems are transferred to Hamilton–Jacobi–Bellman (HJB) equation as a nonlinear first order hyperbolic partial differential equation. Then, the basic VIM is applied to construct a nonlinear optimal feedback control law. By this method, the control and state variables can be approximated as a function of time. Also, the numerical value of the performance index is obtained readily. In view of the convergence of the method, some illustrative examples are presented to show the efficiency and reliability of the presented method.     

The Stochastic Control Model for Use Conversion of Land

In this paper, the authors investigate the optimal conversion rate at which land use is irreversibly converted from biodiversity conservation to agricultural production. This problem is formulated as a stochastic control model, then transformed into a HJB equation involving free boundary. Since the state equation has singularity, it is difficult to directly derive the boundary value condition for the HJB equation. They provide a new method to overcome the difficulty via constructing another auxi...     

Approximate Solutions to the Time-Invariant Hamilton–Jacobi–Bellman Equation

In this paper, we develop a new method to approximate the solution to the Hamilton–Jacobi–Bellman (HJB) equation which arises in optimal control when the plant is modeled by nonlinear dynamics. The approximation is comprised of two steps. First, successive approximation is used to reduce the HJB equation to a sequence of linear partial differential equations. These equations are then approximated via the Galerkin spectral method. The resulting algorithm has several important advantages over previously reported methods. Namely, the resulting control is in feedback form and its associated region of attraction is well defined. In addition, all computations are performed off-line and the control can be made arbitrarily close to optimal. Accordingly, this paper presents a new tool for designing nonlinear control systems that adhere to a prescribed integral performance criterion.     

Numerical Solution of Hamilton-Jacobi-Bellman Equations by an Upwind Finite Volume Method

In this paper we present a finite volume method for solving Hamilton-Jacobi-Bellman(HJB) equations governing a class of optimal feedback control problems. This method is based on a finite volume discretization in state space coupled with an upwind finite difference technique, and on an implicit backward Euler finite differencing in time, which is absolutely stable. It is shown that the system matrix of the resulting discrete equation is an M-matrix. To show the effectiveness of this approach, numerical experiments on test problems with up to three states and two control variables were performed. The numerical results show that the method yields accurate approximate solutions to both the control and the state variables.     

带利率和随机观测时间的布朗运动模型中最优分红策略

本文研究了带利率和随机观测时间的布朗运动模型中的最优分红问题.利用随机控制理论,获得了最优值函数相应的HJB方程,表明最优分红策略是障碍策略,并给出了最优值函数的显式表达式,推广了文献[19]的结果.     

On the value function for optimal control problems with infinite horizon

We consider optimal control problems of infinite horizon type, whose control laws are given by L¹_loc‐functions and whose objective function has the meaning of a discounted utility. Our main objective is the verification of the fact that the value function is a viscosity solution of the Hamilton‐Jacobi‐Bellman (HJB) equation in this framework. The usual final condition for the HJB‐equation in the finite horizon case (V (T, x) = 0 or V (T, x) = g(x)) has to be substituted by a decay condition at the infinity. Following the dynamic programming approach, we obtain Bellman's optimality principle and the dynamic programming equation (see (3)). We also prove a regularity result (local Lipschitz continuity) for the value function.     

Initialization of the Shooting Method via the Hamilton-Jacobi-Bellman Approach

The aim of this paper is to investigate from the numerical point of view the coupling of the Hamilton-Jacobi-Bellman (HJB) equation and the Pontryagin minimum principle (PMP) to solve some control problems. A rough approximation of the value function computed by the HJB method is used to obtain an initial guess for the PMP method. The advantage of our approach over other initialization techniques (such as continuation or direct methods) is to provide an initial guess close to the global minimum. Numerical tests involving multiple minima, discontinuous control, singular arcs and state constraints are considered.     

On homotopy analysis method applied to linear optimal control problems

In this paper, the homotopy analysis method (HAM) is employed to solve the linear optimal control problems (OCPs), which have a quadratic performance index. The study examines the application of the homotopy analysis method in obtaining the solution of equations that have previously been obtained using the Pontryagin’s maximum principle (PMP). The HAM approach is also applied in obtaining the solution of the matrix Riccati equation. Numerical results are presented for several test examples involving scalar and 2nd-order systems to demonstrate the applicability and efficiency of the method.     

Verification by Stochastic Perron's Method in stochastic exit time control problems

We apply the Stochastic Perron Method, created by Bayraktar and Sîrbu, to a stochastic exit time control problem. Our main assumption is the validity of the Strong Comparison Result for the related Hamilton–Jacobi–Bellman (HJB) equation. Without relying on Bellman's optimality principle we prove that inside the domain the value function is continuous and coincides with a viscosity solution of the Dirichlet boundary value problem for the HJB equation.     

Optimal control of stochastic functional differential equations with a bounded memory

This paper treats a finite time horizon optimal control problem in which the controlled state dynamics are governed by a general system of stochastic functional differential equations with a bounded memory. An infinite dimensional Hamilton–Jacobi–Bellman (HJB) equation is derived using a Bellman-type dynamic programming principle. It is shown that the value function is the unique viscosity solution of the HJB equation.     

Uniqueness of solution of production control problem in a manufacturing system with degenerate demand

This paper studies the production inventory problem of minimizing the expected discounted present value of production cost control in a manufacturing system with degenerate stochastic demand. We establish the existence of a unique solution of the Hamilton-Jacobi-Bellman (HJB) equation associated with this problem. The optimal control is given by a solution to the corresponding HJB equation.     

Robust Optimal Investment Problem with Delay under Heston’s Model

This paper considers a robust optimal portfolio problem under Heston model in which the risky asset price is related to the historical performance. The finance market includes a riskless asset and a risky asset whose price is controlled by a stochastic delay equation. The objective is to choose the investment strategy to maximize the minimal expected utility of terminal wealth. By employing dynamic programming principle and Hamilton-Jacobin-Bellman (HJB) equation, we obtain the specific expression of the optimal control and the explicit solution of the corresponding HJB equation. Besides, a verification theorem is provided to ensure the value function is indeed the solution of the HJB equation. Finally, we use numerical examples to illustrate the relationship between the optimal strategy and parameters.

     

Optimal advertising and pricing in a class of general new-product adoption models

In [21], Sethi et al. introduced a particular new-product adoption model. They determine optimal advertising and pricing policies of an associated deterministic infinite horizon discounted control problem. Their analysis is based on the fact that the corresponding Hamilton–Jacobi–Bellman (HJB) equation is an ordinary non-linear differential equation which has an analytical solution. In this paper, generalizations of their model are considered. We take arbitrary adoption and saturation effects into account, and solve finite and infinite horizon discounted variations of associated control problems. If the horizon is finite, the HJB-equation is a 1st order non-linear partial differential equation with specific boundary conditions. For a fairly general class of models we show that these partial differential equations have analytical solutions. Explicit formulas of the value function and the optimal policies are derived. The controlled Bass model with isoelastic demand is a special example of the class of controlled adoption models to be examined and will be analyzed in some detail.     

Time-consistent reinsurance–investment strategy for a mean–variance insurer under stochastic interest rate model and inflation risk

In this paper, we consider the time-consistent reinsurance–investment strategy under the mean–variance criterion for an insurer whose surplus process is described by a Brownian motion with drift. The insurer can transfer part of the risk to a reinsurer via proportional reinsurance or acquire new business. Moreover, stochastic interest rate and inflation risks are taken into account. To reduce the two kinds of risks, not only a risk-free asset and a risky asset, but also a zero-coupon bond and Treasury Inflation Protected Securities (TIPS) are available to invest in for the insurer. Applying stochastic control theory, we provide and prove a verification theorem and establish the corresponding extended Hamilton–Jacobi–Bellman (HJB) equation. By solving the extended HJB equation, we derive the time-consistent reinsurance–investment strategy as well as the corresponding value function for the mean–variance problem, explicitly. Furthermore, we formulate a precommitment mean–variance problem and obtain the corresponding time-inconsistent strategy to compare with the time-consistent strategy. Finally, numerical simulations are presented to illustrate the effects of model parameters on the time-consistent strategy.     

Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equations

We address finding the semi-global solutions to optimal feedback control and the Hamilton–Jacobi–Bellman (HJB) equation. Using the solution of an HJB equation, a feedback optimal control law can be implemented in real-time with minimum computational load. However, except for systems with two or three state variables, using traditional techniques for numerically finding a semi-global solution to an HJB equation for general nonlinear systems is infeasible due to the curse of dimensionality. Here we present a new computational method for finding feedback optimal control and solving HJB equations which is able to mitigate the curse of dimensionality. We do not discretize the HJB equation directly, instead we introduce a sparse grid in the state space and use the Pontryagin’s maximum principle to derive a set of necessary conditions in the form of a boundary value problem, also known as the characteristic equations, for each grid point. Using this approach, the method is spatially causality free, which enjoys the advantage of perfect parallelism on a sparse grid. Compared with dense grids, a sparse grid has a significantly reduced size which is feasible for systems with relatively high dimensions, such as the 6-D system shown in the examples. Once the solution obtained at each grid point, high-order accurate polynomial interpolation is used to approximate the feedback control at arbitrary points. We prove an upper bound for the approximation error and approximate it numerically. This sparse grid characteristics method is demonstrated with three examples of rigid body attitude control using momentum wheels.     

Optimal investment and reinsurance for an insurer under Markov-modulated financial market

This study examines optimal investment and reinsurance policies for an insurer with the classical surplus process. It assumes that the financial market is driven by a drifted Brownian motion with coefficients modulated by an external Markov process specified by the solution to a stochastic differential equation. The goal of the insurer is to maximize the expected terminal utility. This paper derives the Hamilton–Jacobi–Bellman (HJB) equation associated with the control problem using a dynamic programming method. When the insurer admits an exponential utility function, we prove that there exists a unique and smooth solution to the HJB equation. We derive the explicit optimal investment policy by solving the HJB equation. We can also find that the optimal reinsurance policy optimizes a deterministic function. We also obtain the upper bound for ruin probability in finite time for the insurer when the insurer adopts optimal policies.