Controlled jump processes

Finite and infinite planning horizon Markov decision problems are formulated for a class of jump processes with general state and action spaces and controls which are measurable functions on the time axis taking values in an appropriate metrizable vector space. For the finite horizon problem, the maximum expected reward is the unique solution, which exists, of a certain differential equation and is a strongly continuous function in the space of upper semi-continuous functions. A necessary and sufficient condition is provided for an admissible control to be optimal, and a sufficient condition is provided for the existence of a measurable optimal policy. For the infinite horizon problem, the maximum expected total reward is the fixed point of a certain operator on the space of upper semi-continuous functions. A stationary policy is optimal over all measurable policies in the transient and discounted cases as well as, with certain added conditions, in the positive and negative cases.     

Quadratic integral games and causal synthesis

The game problem for an input-output system governed by a Volterra integral equation with respect to a quadratic performance functional is an untouched open problem. In this paper, it is studied by a new approach called projection causality. The main result is the causal synthesis which provides a causal feedback implementation of the optimal strategies in the saddle point sense. The linear feedback operator is determined by the solution of a Fredholm integral operator equation, which is independent of data functions and control functions. Two application examples are included. The first one is quadratic differential games of a linear system with arbitrary finite delays in the state variable and control variables. The second is the standard linear-quadratic differential games, for which it is proved that the causal synthesis can be reduced to a known result where the feedback operator is determined by the solution of a differential Riccati operator equation.

     

Impulse and continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of impulse and continuous control on the jump rate and post jump location parameters of piecewise-deterministic Markov processes (PDP's). In a companion paper we studied the optimal stopping with continuous control problem of PDP's assuming only absolutely continuity along trajectories hypothesis on the final cost function. In this paper we apply these results to obtain optimality equations for the impulse and continuous control problem of PDP's in terms of a set of quasi-variational inequalities as well as on the first jump time operator of the process. No continuity or differential assumptions on the whole state space, neither stability assumptions on the parameters of the problem are required. It is shown that if the post intervention operator satisfies some locally lipschitz continuity along trajectories properties then so will the value function of the impulse and continuous control problem.     

Algebraic Riccati equations with non-smoothing observation arising in hyperbolic and Euler-Bernoulli boundary control problems

Summary This paper considers the optimal quadratic cost problem (regulator problem) for a class of abstract differential equations with unbounded operators which, under the same unified framework, model in particular «concrete» boundary control problems for partial differential equations defined on a bounded open domain of any dimension, including: second order hyperbolic scalar equations with control in the Dirichlet or in the Neumann boundary conditions; first order hyperbolic systems with boundary control; and Euler-Bernoulli (plate) equations with (for instance) control(s) in the Dirichlet and/or Neumann boundary conditions. The observation operator in the quadratic cost functional is assumed to be non-smoothing (in particular, it may be the identity operator), a case which introduces technical difficulties due to the low regularity of the solutions. The paper studies existence and uniqueness of the resulting algebraic (operator) Riccati equation, as well as the relationship between exact controllability and the property that the Riccati operator be an isomorphism, a distinctive feature of the dynamics in question (emphatically not true for, say, parabolic boundary control problems). This isomorphism allows one to introduce a «dual» Riccati equation, corresponding to a «dual» optimal control problem. Properties between the original and the «dual» problem are also investigated.Research partially supported by the National Science Foundation under Grant NSF-DMS-8301668 and by the Air Force Office of Scientific Research under Grant AFOSR-84-0365.     

Optimal control with impulsive component for systems described by implicit parabolic operator differential equations

We study the problem of optimal control with impulsive component for systems described by abstract Sobolev-type differential equations with unbounded operator coefficients in Hilbert spaces. The operator coefficient of the time derivative may be noninvertible. The main assumption is a restriction imposed on the resolvent of the characteristic operator pencil in a certain right half plane. Applications to Sobolevtype partial differential equations are discussed.     

Optimization of Mayer Problem with Sturm–Liouville-Type Differential Inclusions

The present paper studies a new class of problems of optimal control theory with Sturm–Liouville-type differential inclusions involving second-order linear self-adjoint differential operators. Our main goal is to derive the optimality conditions of Mayer problem for differential inclusions with initial point constraints. By using the discretization method guaranteeing transition to continuous problem, the discrete and discrete-approximation inclusions are investigated. Necessary and sufficient conditions, containing both the Euler–Lagrange and Hamiltonian-type inclusions and “transversality” conditions are derived. The idea for obtaining optimality conditions of Mayer problem is based on applying locally adjoint mappings. This approach provides several important equivalence results concerning locally adjoint mappings to Sturm–Liouville-type set-valued mappings. The result strengthens and generalizes to the problem with a second-order non-self-adjoint differential operator; a suitable choice of coefficients then transforms this operator to the desired Sturm–Liouville-type problem. In particular, if a positive-valued, scalar function specific to Sturm–Liouville differential inclusions is identically equal to one, we have immediately the optimality conditions for the second-order discrete and differential inclusions. Furthermore, practical applications of these results are demonstrated by optimization of some “linear” optimal control problems for which the Weierstrass–Pontryagin maximum condition is obtained.     

Existence of solutions of two generalized riccati operator equations and their representation

The solutions of two generalized Riccati operator equations are discussed in terms of two critical parameter values, which are related to the application of optimal control under unknown disturbances. Explicit formulas for calculating these two critical parameters as well as the closedform solutions of these two generalized Riccati operator equations are given. The connection between these two parameters and a zero-sum differential game is also investigated.     

Optimal control of random evolutions

We consider a stochastic control problem for a random evolution. We study the Bellman equation of the problem and we prove the existence of an optimal stochastic control which is Markovian. This problem enables us to approximate the general problem of the optimal control of solutions of stochastic differential equations.     

Dynamic programming for some optimal control problems with hysteresis

We study an infinite horizon optimal control problem for a system with two state variables. One of them has the evolution governed by a controlled ordinary differential equation and the other one is related to the latter by a hysteresis relation, represented here by either a play operator or a Prandtl-Ishlinskii operator. By dynamic programming, we derive the corresponding (discontinuous) first order Hamilton-Jacobi equation, which in the first case is of finite dimension and in the second case is of infinite dimension. In both cases we prove that the value function is the only bounded uniformly continuous viscosity solution of the equation.     

Numreical solutoin of a class of parabolic partial differential equation arising in optimal control problems with uncertainty

In this paper the optimal control of uncertain parabolic systems of partial differential equations is investigated. In order to search for controllers that are insensitive to uncertainties in these systems, an iterative optimization procedure is proposed. This procedure involves the solution of a set of operator valued parabolic partial differential equations. The existence and uniqueness of solutions to these operator equations is proved, and a stable numerical algorithm to approximate the uncertain optimal control problem is proposed. The viability of the proposed algorithm is demonstrated by applying it to the control of parabolic systems having two different types of uncertainty.     

Optimal stopping with continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of optimal stopping and continuous control on some local parameters of a piecewise-deterministic Markov processes (PDP's). Optimality equations are obtained in terms of a set of variational inequalities as well as on the first jump time operator of the PDP. It is shown that if the final cost function is absolutely continuous along trajectories then so is the value function of the optimal stopping problem with continuous control. These results unify and generalize previous ones in the current literature.     

Banach空间中一类周期非线性受控系统最优控制的存在性

研究一类强非线性发展方程的周期解及相应的最优控制问题的存在性，首先，证明了Banach空间中一类包含非线性单调算子和非线性非单调扰动的强非线性发展方程周期解的存在性；其次，给出了保证相应的Lagrange最优控制的充分条件；最后，举例说明理论结果在拟线笥抛物方程周期问题及相应的最优控制问题中的应用。     

On the compactness method in general ergodic impulsive control of markov processes

In this paper the impulsive control of Feller Markov processes on compact state space with long run average cost criterion is studied. Under the assumption of compactness of the resolvent operator the optimal strategies corresponding to general cost for impulses are constructed. Also the case of purely jump Markov processes is considered     

A Maximum Principle for Stochastic Control with Partial Information

Abstract

We study the problem of optimal control of a jump diffusion, that is, a process which is the solution of a stochastic differential equation driven by Lévy processes. It is required that the control process is adapted to a given subfiltration of the filtration generated by the underlying Lévy processes. We prove two maximum principles (one sufficient and one necessary) for this type of partial information control. The results are applied to a partial information mean-variance portfolio selection problem in finance.     

Optimal damping control and nonlinear parabolic systemes

An optimal control problem for a parabolic equation when the control parameter is the zero order coefficient of the differential operator is considered. An optimality system is derived. Under a certain sign condition, the problem is solved completely, by proving uniqueness and providing a constructive existence proof for the nonlinear parabolic optimality system.     

A convergence result for a sequence of distributed-parameter optimal control problems

In this paper, we consider a sequence of abstract optimal control problems by allowing the cost integrand, the partial differential operator, and the control constraint set all to vary simultaneously. Using the notions of -convergence of functions,G-convergence of operators, and Kuratowski-Mosco convergence of sets, we show that the values of the approximating problems converge to that of the limit problem. Also we show that a convergent sequence of optimal pairs for the approximating problems has a limit which is optimal for the limit problem. A concrete example of parabolic optimal control problems is worked out in detail.This research was supported by NSF Grant No. DMS-88-02688.     

Target-level criterion in Markov decision processes

The Markov decision process is studied under the maximization of the probability that total discounted rewards exceed a target level. We focus on and study the dynamic programing equations of the model. We give various properties of the optimal return operator and, for the infinite planning-horizon model, we characterize the optimal value function as a maximal fixed point of the previous operator. Various turnpike results relating the finite and infinite-horizon models are also given.     

Predictive control of uncertain nonlinear parabolic PDE systems using a Galerkin/neural-network-based model

In this paper, a model predictive control (MPC) scheme for a class of parabolic partial differential equation (PDE) systems with unknown nonlinearities, arising in the context of transport-reaction processes, is proposed. A spatial operator of a parabolic PDE system is characterized by a spectrum that can be partitioned into a finite slow and an infinite fast complement. In this view, first, Galerkin method is used to derive a set of finite dimensional slow ordinary differential equation (ODE) system that captures the dominant dynamics of the initial PDE system. Then, a Multilayer Neural Network (MNN) is employed to parameterize the unknown nonlinearities in the resulting finite dimensional ODE model. Finally, a Galerkin/neural-network-based ODE model is used to predict future states in the MPC algorithm. The proposed controller is applied to stabilize an unstable steady-state of the temperature profile of a catalytic rod subject to input and state constraints.     

Optimality of monotonic policies for two-action Markovian decision processes,with applications to control of queues with delayed information

We consider a discrete-time Markov decision process with a partially ordered state space and two feasible control actions in each state. Our goal is to find general conditions, which are satisfied in a broad class of applications to control of queues, under which an optimal control policy is monotonic. An advantage of our approach is that it easily extends to problems with both information and action delays, which are common in applications to high-speed communication networks, among others. The transition probabilities are stochastically monotone and the one-stage reward submodular. We further assume that transitions from different states are coupled, in the sense that the state after a transition is distributed as a deterministic function of the current state and two random variables, one of which is controllable and the other uncontrollable. Finally, we make a monotonicity assumption about the sample-path effect of a pairwise switch of the actions in consecutive stages. Using induction on the horizon length, we demonstrate that optimal policies for the finite- and infinite-horizon discounted problems are monotonic. We apply these results to a single queueing facility with control of arrivals and/or services, under very general conditions. In this case, our results imply that an optimal control policy has threshold form. Finally, we show how monotonicity of an optimal policy extends in a natural way to problems with information and/or action delay, including delays of more than one time unit. Specifically, we show that, if a problem without delay satisfies our sufficient conditions for monotonicity of an optimal policy, then the same problem with information and/or action delay also has monotonic (e.g., threshold) optimal policies.     

Optimal Gevrey classes for the existence of solution operators for linear partial differential operators in three variables

For a constant coefficient linear partial differential operator acting on all infinitely differentiable functions or ω-ultradifferentiable functions of Beurling type on euclidean 3-space, the existence of a continuous linear solution operator is investigated. It is shown that there is an optimal weight ω in the sense that a solution operator exists for a weight σ if and only if ω=O(σ), provided that such an operator exists for at least one weight. Furthermore, the optimal class is either a Gevrey class of rational exponent or the class of all infinitely differentiable functions.