Approximation of optimal feedback control: a dynamic programming approach

We consider the general continuous time finite-dimensional deterministic system under a finite horizon cost functional. Our aim is to calculate approximate solutions to the optimal feedback control. First we apply the dynamic programming principle to obtain the evolutive Hamilton–Jacobi–Bellman (HJB) equation satisfied by the value function of the optimal control problem. We then propose two schemes to solve the equation numerically. One is in terms of the time difference approximation and the other the time-space approximation. For each scheme, we prove that (a) the algorithm is convergent, that is, the solution of the discrete scheme converges to the viscosity solution of the HJB equation, and (b) the optimal control of the discrete system determined by the corresponding dynamic programming is a minimizing sequence of the optimal feedback control of the continuous counterpart. An example is presented for the time-space algorithm; the results illustrate that the scheme is effective.     

Penalty Approach to the HJB Equation Arising in European Stock Option Pricing with Proportional Transaction Costs

We present a novel penalty approach to the Hamilton-Jacobi-Bellman (HJB) equation arising from the valuation of European options with proportional transaction costs. We first approximate the HJB equation by a quasilinear 2nd-order partial differential equation containing two linear penalty terms with penalty parameters λ ₁ and λ ₂ respectively. Then, we show that there exists a unique viscosity solution to the penalized equation. Finally, we prove that, when both λ ₁ and λ ₂ approach infinity, the viscosity solution to the penalized equation converges to that of the corresponding original HJB equation.     

Optimal Consumption in a Growth Model with the CES Production Function

Abstract

This article studies the optimization problem of maximizing the expected discounted present value of lifetime utility of consumption in the framework of one-sector neoclassical growth model with the Constant Elasticity of Substitution (CES) production function. We establish the existence of a classical solution of the Hamilton–Jacobi–Bellman (HJB) equation associated with this problem by the technique of viscosity solutions under the strict concavity of the utility function, and hence derive an optimal consumption from the optimality conditions in the HJB equation.     

A new algorithm for finding numerical solutions of optimal feedback control

Bing Sun Department of Mathematics, Beijing Institute of Technology, Beijing 100081, People's Republic of China and School of Computational and Applied Mathematics, University of the Witwatersrand, Wits 2050, Johannesburg, South Africa Email: bzguo{at}iss.ac.cn Received on March 15, 2007; Revision received October 17, 2007. A new algorithm for finding numerical solutions of optimal feedbackcontrol based on dynamic programming is developed. The algorithmis based on two observations: (1) the value function of theoptimal control problem considered is the viscosity solutionof the associated Hamilton–Jacobi–Bellman (HJB)equation and (2) the appearance of the gradient of the valuefunction in the HJB equation is in the form of directional derivative.The algorithm proposes a discretization method for seeking optimalcontrol–trajectory pairs based on a finite-differencescheme in time through solving the HJB equation and state equation.We apply the algorithm to a simple optimal control problem,which can be solved analytically. The consistence of the numericalsolution obtained to its analytical counterpart indicates theeffectiveness of the algorithm.     

Verification by Stochastic Perron's Method in stochastic exit time control problems

We apply the Stochastic Perron Method, created by Bayraktar and Sîrbu, to a stochastic exit time control problem. Our main assumption is the validity of the Strong Comparison Result for the related Hamilton–Jacobi–Bellman (HJB) equation. Without relying on Bellman's optimality principle we prove that inside the domain the value function is continuous and coincides with a viscosity solution of the Dirichlet boundary value problem for the HJB equation.     

On the value function for optimal control problems with infinite horizon

We consider optimal control problems of infinite horizon type, whose control laws are given by L¹_loc‐functions and whose objective function has the meaning of a discounted utility. Our main objective is the verification of the fact that the value function is a viscosity solution of the Hamilton‐Jacobi‐Bellman (HJB) equation in this framework. The usual final condition for the HJB‐equation in the finite horizon case (V (T, x) = 0 or V (T, x) = g(x)) has to be substituted by a decay condition at the infinity. Following the dynamic programming approach, we obtain Bellman's optimality principle and the dynamic programming equation (see (3)). We also prove a regularity result (local Lipschitz continuity) for the value function.     

Dynamic programming and Lagrange multipliers for active relaxation of resources in nonlinear non-equilibrium systems

In power production problems maximum power and minimum entropy production and inherently connected by the Gouy–Stodola law. In this paper various mathematical tools are applied in dynamic optimization of power-maximizing paths, with special attention paid to nonlinear systems. Maximum power and/or minimum entropy production are governed by Hamilton–Jacobi–Bellman (HJB) equations which describe the value function of the problem and associated controls. Yet, in many cases optimal relaxation curve is non-exponential, governing HJB equations do not admit classical solutions and one has to work with viscosity solutions. Systems with nonlinear kinetics (e.g. radiation engines) are particularly difficult, thus, discrete counterparts of continuous HJB equations and numerical approaches are recommended. Discrete algorithms of dynamic programming (DP), which lead to power limits and associated availabilities, are effective. We consider convergence of discrete algorithms to viscosity solutions of HJB equations, discrete approximations, and the role of Lagrange multiplier λ associated with the duration constraint. In analytical discrete schemes, the Legendre transformation is a significant tool leading to original work function. We also describe numerical algorithms of dynamic programming and consider dimensionality reduction in these algorithms. Indications showing the method potential for other systems, in particular chemical energy systems, are given.     

Optimal investment–consumption problem: Post-retirement with minimum guarantee

We study the optimal investment–consumption problem for a member of defined contribution plan during the decumulation phase. For a fixed annuitization time, to achieve higher final annuity, we consider a variable consumption rate. Moreover, to have a minimum guarantee for the final annuity, a safety level for the wealth process is considered. To solve the stochastic optimal control problem via dynamic programming, we obtain a Hamilton–Jacobi–Bellman (HJB) equation on a bounded domain. The existence and uniqueness of classical solutions are proved through the dual transformation. We apply the finite difference method to find numerical approximations of the solution of the HJB equation. Finally, the simulation results for the optimal investment–consumption strategies, optimal wealth process and the final annuity for different admissible ranges of consumption are given. Furthermore, by taking into account the market present value of the cash flows before and after the annuitization, we compare the outcomes of different scenarios.     

Optimal portfolio and consumption selection with default risk

We investigate an optimal portfolio and consumption choice problem with a defaultable security. Under the goal of maximizing the expected discounted utility of the average past consumption, a dynamic programming principle is applied to derive a pair of second-order parabolic Hamilton-Jacobi-Bellman (HJB) equations with gradient constraints. We explore these HJB equations by a viscosity solution approach and characterize the post-default and pre-default value functions as a unique pair of constrained viscosity solutions to the HJB equations.     

A <Emphasis Type="Italic">d</Emphasis>-person Differential Game with State Space Constraints

We consider a network of d companies (insurance companies, for example) operating under a treaty to diversify risk. Internal and external borrowing are allowed to avert ruin of any member of the network. The amount borrowed to prevent ruin is viewed upon as control. Repayment of these loans entails a control cost in addition to the usual costs. Each company tries to minimize its repayment liability. This leads to a d -person differential game with state space constraints. If the companies are also in possible competition a Nash equilibrium is sought. Otherwise a utopian equilibrium is more appropriate. The corresponding systems of HJB equations and boundary conditions are derived. In the case of Nash equilibrium, the Hamiltonian can be discontinuous; there are d interlinked control problems with state constraints; each value function is a constrained viscosity solution to the appropriate discontinuous HJB equation. Uniqueness does not hold in general in this case. In the case of utopian equilibrium, each value function turns out to be the unique constrained viscosity solution to the appropriate HJB equation. Connection with Skorokhod problem is briefly discussed.     

Portfolio Optimization with Stochastic Volatilities and Constraints: An Application in High Dimension

In this paper we are interested in an investment problem with stochastic volatilities and portfolio constraints on amounts. We model the risky assets by jump diffusion processes and we consider an exponential utility function. The objective is to maximize the expected utility from the investor terminal wealth. The value function is known to be a viscosity solution of an integro-differential Hamilton-Jacobi-Bellman (HJB in short) equation which could not be solved when the risky assets number exceeds three. Thanks to an exponential transformation, we reduce the nonlinearity of the HJB equation to a semilinear equation. We prove the existence of a smooth solution to the latter equation and we state a verification theorem which relates this solution to the value function. We present an example that shows the importance of this reduction for numerical study of the optimal portfolio. We then compute the optimal strategy of investment by solving the associated optimization problem.     

Optimal control of stochastic functional differential equations with a bounded memory

This paper treats a finite time horizon optimal control problem in which the controlled state dynamics are governed by a general system of stochastic functional differential equations with a bounded memory. An infinite dimensional Hamilton–Jacobi–Bellman (HJB) equation is derived using a Bellman-type dynamic programming principle. It is shown that the value function is the unique viscosity solution of the HJB equation.     

The Stochastic Control Model for Use Conversion of Land

In this paper, the authors investigate the optimal conversion rate at which land use is irreversibly converted from biodiversity conservation to agricultural production. This problem is formulated as a stochastic control model, then transformed into a HJB equation involving free boundary. Since the state equation has singularity, it is difficult to directly derive the boundary value condition for the HJB equation. They provide a new method to overcome the difficulty via constructing another auxiliary stochastic control problem,and impose a proper boundary value condition. Moreover, they establish the existence and uniqueness of the viscosity solution of the HJB equation. Finally, they propose a stable numerical method for the HJB equation involving free boundary, and show some numerical results.     

Second order Hamilton–Jacobi–Bellman equations with an unbounded operator

This work is devoted to the study of a class of Hamilton–Jacobi–Bellman equations associated to an optimal control problem where the state equation is a stochastic differential inclusion with a maximal monotone operator. We show that the value function minimizing a Bolza-type cost functional is a viscosity solution of the HJB equation. The proof is based on the perturbation of the initial problem by approximating the unbounded operator. Finally, by providing a comparison principle we are able to show that the solution of the equation is unique.     

OPTIMAL DIVIDEND-PENALTY STRATEGIES FOR INSURANCE RISK MODELS WITH SURPLUS-DEPENDENT PREMIUMS

This paper concerns an optimal dividend-penalty problem for the risk models with surplus-dependent premiums. The objective is to maximize the difference of the expected cumulative discounted dividend payments received until the moment of ruin and a discounted penalty payment taken at the moment of ruin. Since the value function may be not smooth enough to be the classical solution of the HJB equation, the viscosity solution is involved.The optimal value function can be characterized as the smallest viscosity supersolution of the HJB equation and the optimal dividend-penalty strategy has a band structure. Finally,some numerical examples with gamma distribution for the claims are analyzed.     

Regularity properties in a state-constrained expected utility maximization problem

We consider a stochastic optimal control problem in a market model with temporary and permanent price impact, which is related to an expected utility maximization problem under finite fuel constraint. We establish the initial condition fulfilled by the corresponding value function and show its first regularity property. Moreover, we can prove the existence and uniqueness of an optimal strategy under rather mild model assumptions. This will then allow us to derive further regularity properties of the corresponding value function, in particular its continuity and partial differentiability. As a consequence of the continuity of the value function, we will prove a dynamic programming principle without appealing to the classical measurable selection arguments. This permits us to establish a tight relation between our value function and a nonlinear parabolic degenerated Hamilton–Jacobi–Bellman (HJB) equation with singularity. To conclude, we show a comparison principle, which allows us to characterize our value function as the unique viscosity solution of the HJB equation.     

Randomized filtering and Bellman equation in Wasserstein space for partial observation control problem

We study a stochastic optimal control problem for a partially observed diffusion. By using the control randomization method in Bandini et al. (2018), we prove a corresponding randomized dynamic programming principle (DPP) for the value function, which is obtained from a flow property of an associated filter process. This DPP is the key step towards our main result: a characterization of the value function of the partial observation control problem as the unique viscosity solution to the corresponding dynamic programming Hamilton–Jacobi–Bellman (HJB) equation. The latter is formulated as a new, fully non linear partial differential equation on the Wasserstein space of probability measures. An important feature of our approach is that it does not require any non-degeneracy condition on the diffusion coefficient, and no condition is imposed to guarantee existence of a density for the filter process solution to the controlled Zakai equation. Finally, we give an explicit solution to our HJB equation in the case of a partially observed non Gaussian linear–quadratic model.     

Numerical solution of a minimax ergodic optimal control problem

In this work we consider an L^∞ minimax ergodic optimal control problem with cumulative cost. We approximate the cost function as a limit of evolutions problems. We present the associated Hamilton-Jacobi-Bellman equation and we prove that it has a unique solution in the viscosity sense. As this HJB equation is consistent with a numerical procedure, we use this discretization to obtain a procedure for the primitive problem. For the numerical solution of the ergodic version we need a perturbation of the instantaneous cost function. We give an appropriate selection of the discretization and penalization parameters to obtain discrete solutions that converge to the optimal cost. We present numerical results. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

VISCOSITY SOLUTIONS OF HJB EQUATIONS ARISING FROM THE VALUATION OF EUROPEAN PASSPORT OPTIONS

The passport option is a call option on the balance of a trading account. The option holder retains the gain from trading, while the issuer is liable for the net loss. In this article, the mathematical foundation for pricing the European passport option is established. The pricing equation which is a fully nonlinear equation is derived using the dynamic programming principle. The comparison principle, uniqueness and convexity preserving of the viscosity solutions of related H J13 equation are proved. A relationship between the passport and lookback options is discussed.     

Parabolicity of second order differrential equations in hilbert space

We study conditions entailing that the differnetial equation u″+Bu′+Au=0 in a Hilbert space generates a corresponding equation in a larger extrapolation space which is parabolic. Subsequently we prove some relations between solutions of the original equation and the extrapolated equation. An example of applications to initial boundary value problems for partial differential equations is given. Research partially supported by Ministero della Pubblica Istruzione, Italy (Fondi 40%), Università di Bologna (Fondi 60%), and the GNAFA of CNR.