A general sufficiency theorem for minimax control

This paper considers optimal control problems where there is uncertainty in the differential equations describing the system. A minimax optimality criterion is used, and sufficient conditions for a control to be a minimax control are presented. These conditions are more general than those given in Refs. 1 and 2.This research was supported by AFOSR under Grant No. 76-2923.     

Optimal control problems with retardations and restricted phase coordinates

In this paper, we study necessary conditions for an optimal control problem both with and without restricted phase coordinates. The systems are specified by differential-difference equations with their allowed right-hand sides being a quasiconvex family. The multiplier rule developed by Neustadt is used, and maximum principles in integral form as well as in pointwise form are obtained.This research was supported by the Air Force Office of Scientific Research under Contract No. AF-AFOSR-1029-67A and was prepared by the author in partial fulfillment of the requirements for the Ph.D. degree. The author wishes to express his sincere gratitude to his academic advisor Dr. Lucien W. Neustadt for his suggestion of this problem, invaluable consultation, and support throughout this research.     

Switching control of piecewise-deterministic processes

A finite collection of piecewise-deterministic processes are controlled in order to minimize the expected value of a performance functional with continuous operating cost and discrete switching control costs. The solution of the associated dynamic programming equation is obtained by an iterative approximation using optimal stopping time problems.This research was supported in part by NSF Grant No. DMS-8508651 and by University of Tennessee Science Alliance Research Incentive Award.     

More results on the convergence of iterative methods for the symmetric linear complementarity problem

In an earlier paper, the author has given some necessary and sufficient conditions for the convergence of iterative methods for solving the linear complementarity problem. These conditions may be viewed as global in the sense that they apply to the methods regardless of the constant vector in the linear complementarity problem. More precisely, the conditions characterize a certain class of matrices for which the iterative methods will converge, in a certain sense, to a solution of the linear complementarity problem for all constant vectors. In this paper, we improve on our previous results and establish necessary and sufficient conditions for the convergence of iterative methods for solving each individual linear complementarity problem with a fixed constant vector. Unlike the earlier paper, our present analysis applies only to the symmetric linear complementarity problem. Various applications to a strictly convex quadratic program are also given.The author gratefully acknowledges several stimulating conversations with Professor O. Mangasarian on the subject of this paper. He is also grateful to a referee, who has suggested Lemma 2.2 and the present (stronger) version of Theorem 2.1 as well as several other constructive comments.This research was based on work supported by the National Science Foundation under Grant No. ECS-81-14571, sponsored by the United States Army under Contract No. DAAG29-80-C-0041, and was completed while the author was visiting the Mathematics Research Center at the University of Wisconsin, Madison, Wisconsin.     

Controlled jump processes

Finite and infinite planning horizon Markov decision problems are formulated for a class of jump processes with general state and action spaces and controls which are measurable functions on the time axis taking values in an appropriate metrizable vector space. For the finite horizon problem, the maximum expected reward is the unique solution, which exists, of a certain differential equation and is a strongly continuous function in the space of upper semi-continuous functions. A necessary and sufficient condition is provided for an admissible control to be optimal, and a sufficient condition is provided for the existence of a measurable optimal policy. For the infinite horizon problem, the maximum expected total reward is the fixed point of a certain operator on the space of upper semi-continuous functions. A stationary policy is optimal over all measurable policies in the transient and discounted cases as well as, with certain added conditions, in the positive and negative cases.     

Optimal processes in smooth-convex minimization problems

The paper elaborates a general method for studying smooth-convex conditional minimization problems that allows one to obtain necessary conditions for solutions of these problems in the case where the image of the mapping corresponding to the constraints of the problem considered can be of infinite codimension. On the basis of the elaborated method, the author proves necessary optimality conditions in the form of an analog of the Pontryagin maximum principle in various classes of quasilinear optimal control problems with mixed constraints; moreover, the author succeeds in preserving a unified approach to obtaining necessary optimality conditions for control systems without delays, as well as for systems with incommensurable delays in state coordinates and control parameters. The obtained necessary optimality conditions are of a constructive character, which allows one to construct optimal processes in practical problems (from biology, economics, social sciences, electric technology, metallurgy, etc.), in which it is necessary to take into account an interrelation between the control parameters and the state coordinates of the control object considered. The result referring to systems with aftereffect allows one to successfully study many-branch product processes, in particular, processes with constraints of the “bottle-neck” type, which were considered by R. Bellman, and also those modern problems of flight dynamics, space navigation, building, etc. in which, along with mixed constraints, it is necessary to take into account the delay effect. The author suggests a general scheme for studying optimal process with free right endpoint based on the application of the obtained necessary optimality conditions, which allows one to find optimal processes in those control systems in which no singular cases arise. The author gives an effective procedure for studying the singular case (the procedure for calculating a singular control in quasilinear systems with mixed constraints. Using the obtained necessary optimality conditions, the author constructs optimal processes in concrete control systems. __________ Translated from Sovremennaya Matematika i Ee Prilozheniya (Contemporary Mathematics and Its Applications), Vol. 42, Optimal Control, 2006.     

Recent results on conditions for the existence of average optimal stationary policies

This paper concerns countable state space Markov decision processes endowed with a (long-run expected)average reward criterion. For these models we summarize and, in some cases,extend some recent results on sufficient conditions to establish the existence of optimal stationary policies. The topics considered are the following: (i) the new assumptions introduced by Sennott in [20–23], (ii)necessary and sufficient conditions for the existence of a bounded solution to the optimality equation, and (iii) equivalence of average optimality criteria. Some problems are posed.This research was partially supported by the Third World Academy of Sciences (TWAS) under Grant No. TWAS RG MP 898-152.     

On heat kernel estimates and parabolic Harnack inequality for jump processes on metric measure spaces

In this paper, we discuss necessary and sufficient conditions on jumping kernels for a class of jump-type Markov processes on metric measure spaces to have scale-invariant finite range parabolic Harnack inequality.     

Necessary conditions for optimality of singular controls in systems governed by partial differential equations

In this paper, we present a method to obtain necessary conditions for optimality of singular controls in systems governed by partial differential equations (distributed-parameter systems). The method is based on the one developed earlier by the author for singular control problems described by ordinary differential equations. As applications, we consider conditions for optimality of singular controls in a Darboux-Goursat system and in control systems that describe chemical processes.This research was supported in part by the National Science Foundation under Grant No. NSF-MCS-80-02337 at the University of Michigan.The author wishes to express his deep gratitude to Professor L. Cesari for his valuable guidance and constant encouragement during the preparation of this paper.     

The Lagrange-Newton method for state constrained optimal control problems

Local convergence of the Lagrange-Newton method for optimization problems with two-norm discrepancy in abstract Banach spaces is investigated. Based on stability analysis of optimization problems with two-norm discrepancy, sufficient conditions for local superlinear convergence are derived. The abstract results are applied to optimal control problems for nonlinear ordinary differential equations subject to control and state constraints.This research was completed while the second author was a visitor at the University of Bayreuth, Germany, supported by grant No. CIPA3510CT920789 from the Commission of the European Communities.     

条件平均场随机微分方程的最优控制问题

作者研究了一个条件平均场随机微分方程的最优控制问题.这种方程和某些部分信息下的随机最优控制问题有关,并且可以看做是平均场随机微分方程的推广.作者以庞特里雅金最大值原理的形式给出最优控制满足的必要和充分条件.此外,文中给出一个线性二次最优控制问题来说明理论结果的应用.     

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.     

An impulse control of a geometric Brownian motion with quadratic costs

We examine an optimal impulse control problem of a stochastic system whose state follows a geometric Brownian motion. We suppose that, when an agent intervenes in the system, it requires costs consisting of a quadratic form of the system state. Besides the intervention costs, running costs are continuously incurred to the system, and they are also of a quadratic form. Our objective is to find an optimal impulse control of minimizing the expected total discounted sum of the intervention costs and running costs incurred over the infinite time horizon. In order to solve this problem, we formulate it as a stochastic impulse control problem, which is approached via quasi-variational inequalities (QVI). Under a suitable set of sufficient conditions on the given problem parameters, we prove the existence of an optimal impulse control such that, whenever the system state reaches a certain level, the agent intervenes in the system. Consequently it instantaneously reduces to another level.     

On the existence of an optimal solution of the epsilon variational problem

It has been shown that known sufficient conditions for existence of optimal control imply existence of an optimal solution of the nondynamic epsilon variational problem as formulated by Balakrishnan.This work forms a part of the doctoral dissertation submitted by the author in early 1969 at UCLA and was supported in part by the Air Force Office of Scientific Research under Grant No. 69–1646. The author expresses his gratitude to Professor A. V. Balakrishnan for many useful discussions in the preparation of this work.     

Hierarchical controls in stochastic manufacturing systems with convex costs

This paper presents an extension of earlier research on heirarchical control of stochastic manufacturing systems with linear production costs. A new method is introduced to construct asymptotically optimal open-loop and feedback controls for manufacturing systems in which the rates of machine breakdown and repair are much larger than the rate of fluctuation in demand and rate of discounting of cost. This new approach allows us to carry out an asymptotic analysis on manufacturing systems with convex inventory/backlog and production costs as well as obtain error bound estimates for constructed open loop controls. Under appropriate conditions, an asymptotically optimal Lipschitz feedback control law is obtained.This work was partly supported by the NSERC Grant A4619, URIF, General Motors of Canada, and Manufacturing Research Corporation of Ontario.     

Optimal control of diffusion processes with reflection

In this paper, we consider the problem of optimally controlling a diffusion process on a closed bounded region ofR ⁿ with reflection at the boundary. Employing methods similar to Fleming (Ref. 1), we present a constructive proof that there exists an optimal Markov control that is measurable or lower semicontinuous. We prove further that the expected cost function corresponding to the optimal control is the unique solution of the quasilinear parabolic differential equation of dynamic programming with Neumann boundary conditions and that there exists a diffusion process (in the sense of Stroock and Varadhan) corresponding to the optimal control.This work was partially supported by the National Science Foundation, Grant No. GK-18339, by the Office of Naval Research, Grant No. NR-042-264, and by the National Research Council of Canada, Grant No. A3609.The author would like to thank S. R. Pliska, J. Pisa, and N. Trudinger for helpful suggestions. He is especially grateful to Professor A. F. Veinott, Jr., for help and advice in the preparation of the doctoral dissertation, on which part of this paper is based. Finally, he wishes to thank one of the referees for the careful reading and constructive comments on an earlier version of this paper.     

Bias Optimality versus Strong 0-Discount Optimality in Markov Control Processes with Unbounded Costs

This paper deals with expected average cost (EAC) and discount-sensitive criteria for discrete-time Markov control processes on Borel spaces, with possibly unbounded costs. Conditions are given under which (a) EAC optimality and strong –1-discount optimality are equivalent; (b) strong 0-discount optimality implies bias optimality; and, conversely, under an additional hypothesis, (c) bias optimality implies strong 0-discount optimality. Thus, in particular, as the class of bias optimal policies is nonempty, (c) gives the existence of a strong 0-discount optimal policy, whereas from (b) and (c) we get conditions for bias optimality and strong 0-discount optimality to be equivalent. A detailed example illustrates our results.     

Measure-valued processes in the control of partially-observable stochastic systems

This paper is concerned with the optimal control of continuous-time Markov processes. The admissible control laws are based on white-noise corrupted observations of a function on the state processes. A separated control problem is introduced, whose states are probability measures on the original state space. The original and separated control problems are related via the nonlinear filter equation. The existence of a minimum for the separated problem is established. Under more restrictive assumptions it is shown that the minimum expected cost for the separated problem equals the infimum of expected costs for the original problem with partially observed states.This research was supported in part by the Air Force Office of Scientific Research under AF-AFOSR 77-3063, and in part by the National Science Foundation MCS76-07261.Paper presented at the Workshop on Stochastic Processes in Infinite Dimensional Spaces and Random Fields, UCLA April, 1979.     

Optimal switching among a finite number of Markov processes

This paper investigates the problem of the optimal switching among a finite number of Markov processes, generalizing some of the author's earlier results for controlled one-dimensional diffusion. Under rather general conditions, it is shown that the optimal discounted cost function is the unique solution of a functional equation. Under more restrictive assumptions, this function is shown to be the unique solution of some quasi-variational inequalities. These assumptions are verified for a large class of control problems. For controlled Markov chains and controlled one-dimensional diffusion, the existence of a stationary optimal policy is established. Finally, a policy iteration method is developed to calculate an optimal stationary policy, if one exists.This research was sponsored by the Air Force Office of Scientific Research (AFSC), United States Air Force, under Contract No. F-49620-79-C-0165.The author would like to thank the referee for bringing Refs. 7, 8, and 9 to his attention.     

On the optimality of a full-service policy for a queueing system with discounted costs

We provide weak sufficient conditions for a full-service policy to be optimal in a queueing control problem in which the service rate is a dynamic decision variable. In our model there are service costs and holding costs and the objective is to minimize the expected total discounted cost over an infinite horizon. We begin with a semi-Markov decision model for a single-server queue with exponentially distributed inter-arrival and service times. Then we present a general model with weak probabilistic assumptions and demonstrate that the full-service policy minimizes both finite-horizon and infinite-horizon total discounted cost on each sample path.