Optimal control of piecewise deterministic markov process

The trajectories of piecewise deterministic Markov processes are solutions of an ordinary (vector)differential equation with possible random jumps between the different integral curves. Both continuous deterministic motion and the random jumps of the processes are controlled in order to minimize the expected value of a performance functional consisting of continuous, jump and terminal costs. A limiting form of the Hamilton-Jacobi-Bellman partial differential equation is shown to be a necessary and sufficient optimality condition. The existence of an optimal strategy is proved and acharacterization of the value function as supremum of smooth subsolutions is also given. The approach consists of imbedding the original control problem tightly in a convex mathematical programming problem on the space of measures and then solving the latter by dualit     

Optimal trajectories of infinite-horizon deterministic control systems

We study the infinite-horizon deterministic control problem of minimizing ₀ ^T L(z, ) dt, T, whereL(z, ·) is convex in for fixedz but not necessarily jointly convex in (z, ). We prove the existence of a solution to the infinite-horizon Bellman equation and use it to define a differential inclusion, which reduces in certain cases to an ordinary differential equation. We discuss cases where solutions of this differential inclusion (equation) provide optimal solutions (in the overtaking optimality sense) to the optimization problem.A quantity of special interest is the minimal long-run average-cost growth rate. We compute it explicitly and show that it is equal to min _x L(x, 0) in the following two cases: one is the scalar casen = 1 and the other is' when the integrand is in a separated form      

Existence of optimal stationary policies in deterministic optimal control

This paper considers deterministic discrete-time optimal control problems over an infinite horizon involving a stationary system and a nonpositive cost per stage. Various results are provided relating to existence of an ?-optimal stationary policy, and existence of an optimal stationary policy assuming an optimal policy exists.     

Optimal impulsive control of piecewise deterministic Markov processes

In this paper, we study the infinite-horizon expected discounted continuous-time optimal control problem for Piecewise Deterministic Markov Processes with both impulsive and gradual (also called continuous) controls. The set of admissible control strategies is supposed to be formed by policies possibly randomized and depending on the past-history of the process. We assume that the gradual control acts on the jump intensity and on the transition measure, but not on the flow. The so-called Hamilton–Jacobi–Bellman (HJB) equation associated to this optimization problem is analyzed. We provide sufficient conditions for the existence of a solution to the HJB equation and show that the solution is in fact unique and coincides with the value function of the control problem. Moreover, the existence of an optimal control strategy is proven having the property to be stationary and non-randomized.     

Invariance of stochastic control systems with deterministic arguments

We prove that a closed set K of a finite-dimensional space is invariant under the stochastic control system

dX=b(X,v(t))dt+σ(X,v(t))dW(t),v(t)∈U,     

On the maximum principle for deterministic impulse control problems

This paper is devoted to a simple and direct proof of a version of the Blaquiere's maximum principle for deterministic impulse control problems.     

Impulse and continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of impulse and continuous control on the jump rate and post jump location parameters of piecewise-deterministic Markov processes (PDP's). In a companion paper we studied the optimal stopping with continuous control problem of PDP's assuming only absolutely continuity along trajectories hypothesis on the final cost function. In this paper we apply these results to obtain optimality equations for the impulse and continuous control problem of PDP's in terms of a set of quasi-variational inequalities as well as on the first jump time operator of the process. No continuity or differential assumptions on the whole state space, neither stability assumptions on the parameters of the problem are required. It is shown that if the post intervention operator satisfies some locally lipschitz continuity along trajectories properties then so will the value function of the impulse and continuous control problem.     

Optimal stopping with continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of optimal stopping and continuous control on some local parameters of a piecewise-deterministic Markov processes (PDP's). Optimality equations are obtained in terms of a set of variational inequalities as well as on the first jump time operator of the PDP. It is shown that if the final cost function is absolutely continuous along trajectories then so is the value function of the optimal stopping problem with continuous control. These results unify and generalize previous ones in the current literature.     

Approximate solutions of the bellman equation of deterministic control theory

We consider an infinite horizon discounted optimal control problem and its time discretized approximation, and study the rate of convergence of the approximate solutions to the value function of the original problem. In particular we prove the rate is of order 1 as the discretization step tends to zero, provided a semiconcavity assumption is satisfied. We also characterize the limit of the optimal controls for the approximate problems within the framework of the theory of relaxed controls.This work was done while the authors were visiting members of The Department of Mathematics of The University of Maryland at College Park.     

Optimal control of continuous-time deterministic systems with incomplete discrete feedback

Sufficient conditions for optimality are obtained for controls that depend on the current time and values of known functions of the state vector at finite points of the time interval. The equations for finding the required control laws are derived. An example is given for which an exact solution of the problem can be obtained.The author is grateful to Professor R. W. Rishel for his support during the preparation of the paper.     

Maximum principle,dynamic programming,and their connection in deterministic control

Two major tools for studying optimally controlled systems are Pontryagin's maximum principle and Bellman's dynamic programming, which involve the adjoint function, the Hamiltonian function, and the value function. The relationships among these functions are investigated in this work, in the case of deterministic, finite-dimensional systems, by employing the notions of superdifferential and subdifferential introduced by Crandall and Lions. Our results are essentially non-smooth versions of the classical ones. The connection between the maximum principle and the Hamilton-Jacobi-Bellman equation (in the viscosity sense) is thereby explained by virtue of the above relationship.This research was supported by the Natural Science Fund of China.This paper was written while the author visited Keio University, Japan. The author is indebted to Professors H. Tanaka and M. Nisio for their helpful suggestions and discussions. Thanks are also due to Professor X. J. Li for his comments and criticism.     

A deterministic resource scheduling model in epidemic control: A case study

The resources available to tackle an epidemic infection are usually limited, while the time and effort required to control it are increasing functions of the starting time of the containment effort. The problem of scheduling limited available resources, when there are several areas where the population is infected, is considered. A deterministic model, appropriate for large populations, where random interactions can be averaged out, is used for the epidemic’s rate of spread. The problem is tackled using the concept of deteriorating jobs, i.e. the model represents increasing loss rate as more susceptibles become infected, and increasing time and effort needed for the epidemic’s containment. A case study for a proposed application of the model in the case of the mass vaccination against A(H1N1)_v influenza in the Attica region, Greece and a comparative study of the model’s performance vs. the applied random practice are presented.     

Hamilton-Jacobi-Bellman inequality for the average control of piecewise deterministic Markov processes

ABSTRACT

The main goal of this paper is to study the infinite-horizon long run average continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. We provide conditions for the existence of a solution to an integro-differential optimality inequality, the so called Hamilton-Jacobi-Bellman (HJB) equation, and for the existence of a deterministic stationary optimal policy. These results are obtained by using the so-called vanishing discount approach, under some continuity and compactness assumptions on the parameters of the problem, as well as some non-explosive conditions for the process.     

Solution of a class of stochastic linear-convex control problems using deterministic equivalents

In this paper, we identify a new class of stochastic linearconvex optimal control problems, whose solution can be obtained by solving appropriate equivalent deterministic optimal control problems. The term linear-convex is meant to imply that the dynamics is linear and the cost function is convex in the state variables, linear in the control variables, and separable. Moreover, some of the coefficients in the dynamics are allowed to be random and the expectations of the control variables are allowed to be constrained. For any stochastic linear-convex problem, the equivalent deterministic problem is obtained. Furthermore, it is shown that the optimal feedback policy of the stochastic problem is affine in its current state, where the affine transformation depends explicitly on the optimal solution of the equivalent deterministic problem in a simple way. The result is illustrated by its application to a simple stochastic inventory control problem.This research was supported in part by NSERC Grant A4617, by SSHRC Grant 410-83-0888, and by an INRIA Post-Doctoral Fellowship.     

On the dynamics of a deterministic and stochastic model for mosquito control

In this work, we analyze a system of nonlinear difference equations describing community intervention in mosquito control. More specifically, we extend the model given in [M. Predescu, R. Levins, T. Awerbuch, Analysis of a nonlinear system for community intervention in mosquito control, Discrete Contin. Dyn. Syst. Ser. B 6 (3) (2006) 605–622] to allow for consciousness to be created in an ongoing way by educational efforts that are independent of the presence of mosquito breeding sites. In order to quantify the effect of random external events, such as weather or public concerns, we consider a stochastic version of the model. Numerical simulations show that the stochastic model is consistent with the deterministic one.     

Turnpike properties for a class of piecewise deterministic systems arising in manufacturing flow control

This paper deals with a general class of piecewise deterministic control systems that encompasses FMS flow control models. One uses the Markov renewal decision process formalism to characterize optimal policies via a discrete event dynamic programming approach. A family of control problems with a random stopping time is associated with these optimality conditions. These problems can be reformulated as infinite horizon deterministic control problems. It is then shown how the so-calledturnpike property should hold for these deterministic control problems under classical convexity assumptions. These turnpikes have the same generic properties as the attractors obtained via a problem specific approach in FMS flow control models and production planning and are calledhedging points in this literature.This research has been supported by NSERC-Canada, Grants No. A4952 by FCAR-Québec, Grant No. 88EQ3528, Actions Structurantes, MESS-Québec, Grant No. 6.1/7.4(28), and FNRS-Switzerland.     

Necessary and sufficient optimality conditions for control of piecewise deterministic markov processes

Piecewise deterministic Markov processes (PDPs) are continuous time homogeneous Markov processes whose trajectories are solutions of ordinary differential equations with random jumps between the different integral curves. Both continuous deterministic motion and the random jumps of the processes are controlled in order to minimize the expected value of a performance criterion involving discounted running and boundary costs. Under fairly general assumptions, we will show that there exists an optimal control, that the value function is Lipschitz continuous and that a generalized Bellman-Hamilton-Jacobi (BHJ) equation involving the Clarke generalized gradient is a necessary and sufficient optimality condition for the problem.     

A numerical approach to the infinite horizon problem of deterministic control theory

We are concerned with the Hamilton-Jacobi equation related to the infinite horizon problem of deterministic control theory. Approximate solutions are constructed by means of a discretization in time as well as in the state variable and we prove that their rate of convergence to the viscosity solution is of order 1, provided a semiconcavity assumption is satisfied. A computational algorithm, originally due to R. Gonzales and E. Rofman, is adapted and reformulated for the problem at hand in order to obtain an error estimate for the numerical approximate solutions.This work has been partially supported by CNR-GNAFA.     

Feedback control variables to restrain the Babesiosis disease

In this paper, we complete the study of the dynamics of a recognized continuous‐time model for the Babesiosis disease. The local and global asymptotic stability of the endemic state are established theoretically and experimentally. In addition, to restrain the disease in the original model when the endemic state exists, we propose and study the continuous model with feedback controls. The global stability of the boundary‐equilibrium point of this model is analyzed by means of rigorous mathematical methods. As an important consequence of this result, we propose a strategy to select feedback control variables in order to restrain the disease in the original model. This strategy allows us to make the disease vanish completely. In other words, the feedback controls are specially effective for restraining disease in the model. The validity of the established theoretical result is supported by a set of numerical simulations.     

On the convergence of the dual variables in optimal control problems

The coefficients of the plant are perturbed in an abstract, linear, convex problem on a Hilbert space. Necessary and sufficient conditions are obtained about the perturbations for the continuous dependence of the adjoint variables on the coefficients of the plant. An application is given to the optimal control of ordinary differential equations.This work was supported by CNR-CNAFA, Rome, Italy.