A survey of Markov decision models for control of networks of queues

We review models for the optimal control of networks of queues. Our main emphasis is on models based on Markov decision theory and the characterization of the structure of optimal control policies.This research was partially supported by the National Science Foundation under Grant No. DDM-8719825. The Government has certain rights in this material. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The research was also partially supported by the C.I.E.S. (France), while the author was on leave at INRIA, Sophia-Antipolis, 1991–92.     

Second-order optimality principle for singular optimal control problems

Recently published results of Gift (Ref. 1) are concerned with the necessary conditions for singular optimal control problems (in the sense of Pontryagin's minimum principle). However, those results are incorrect. An illustrative counterexample is given here.This paper was written while the author studied at Fudan University. The author wishes to thank Professor X. J. Li for his guidance and encouragement. The research was partially supported by NSF of China and the Chinese State Education Commission NSF.     

Penalty approximation of a Sobolev optimal control problem with state constraints

In this paper, we study the approximation by the penalty method of a control problem governed by a pseudo-parabolic equation with a noncoercive control functional and with control and state constraints. The existence of solutions to the penalized problems is established. In addition, the convergence of the penalized problems to the solution, the Lagrange multipliers, and the minimum value of the original problem is studied. The results apply to Sobolev and parabolic equations as well.This work was partially supported by the National Science Foundation, Grant No. MCS-79-02037. The author would like to thank Professor A. B. Schwarzkopf for his helpful comments on this paper.     

Optimal guaranteed control of linear systems under disturbances

This paper is devoted to the problem of the minimax control of a dynamical system with quadratic performance functional under external disturbances and geometric control constraints. The optimal guaranteed control strategy is obtained in explicit form.Translated fromMatematicheskie Zametki, Vol. 60, No. 2, pp. 198–205, August, 1996.This research was supported by the Russian Foundation for Basic Research under grant No. 95-01-000771a.     

A unified framework for linear control problems with state-variable inequality constraints

This paper briefly reviews the literature on necessary optimality conditions for optimal control problems with state-variable inequality constraints. Then, it attempts to unify the treatment of linear optimal control problems with state-variable inequality constraints in the framework of continuous linear programming. The duality theory in this framework makes it possible to relate the adjoint variables arising in different formulations of a problem; these relationships are illustrated by the use of a simple example. This framework also allows more general problems and admits a simplex-like algorithm to solve these problems.This research was partially supported by Grant No. A4619 from the National Research Council of Canada to the first author. The first author also acknowledges the support provided by the Brookhaven National Laboratory, where he conducted his research.     

Inverse programming and the linear vector maximization problem

This paper presents a new concept of efficient solution for the linear vector maximization problem. Briefly, these solutions are efficient with respect to the constraints, in addition to being efficient with respect to the multiple objectives. The duality theory of linear vector maximization is developed in terms of this solution concept and then is used to formulate the problem as a linear program.This research has been partially supported by grants from the Canada Council and the National Research Council of Canada.     

Small-time local controllability and continuity of the optimal time function for linear systems

For a finite-dimensional linear system, in which the control is restricted to belong to a completely arbitrary setJ, we give a simple necessary and sufficient condition for small-time local controllability from a pointp. The condition is equivalent to a characterization of the property that the Bellman function for the corresponding minimum-time optimal control problem is continuous atp.This work was partially supported by the National Science Foundation, Grant No. MCS-78-02442.     

Interior point methods for optimal control of discrete time systems

We show that recently developed interior point methods for quadratic programming and linear complementarity problems can be put to use in solving discrete-time optimal control problems, with general pointwise constraints on states and controls. We describe interior point algorithms for a discrete-time linear-quadratic regulator problem with mixed state/control constraints and show how they can be efficiently-incorporated into an inexact sequential quadratic programming algorithm for nonlinear problems. The key to the efficiency of the interior-point method is the narrow-banded structure of the coefficient matrix which is factorized at each iteration.This research was supported by the Applied Mathematical Sciences Subprogram of the Office of Energy Research, US Department of Energy, under Contract W-31-109-Eng-38.     

Optimal control systems with state variable jump discontinuities

Consideration is given to continuous-time, parameter-dependent optimal control problems with state-variable jump discontinuities atN variable interior times. A maximum principle involving known costate jump conditions is stated and is proved by transforming the problem into a standard Mayer control problem. An illustrative example for fisheries management is included.This work was partially supported by a grant from Control Data. The authors are grateful to Professor T. L. Vincent for drawing their attention to Refs. 4–6 listed below.     

General Maximum Principles for Partially Observed Risk-Sensitive Optimal Control Problems and Applications to Finance

This paper is concerned with partially observed risk-sensitive optimal control problems. Combining Girsanov’s theorem with a standard spike variational technique, we obtain some general maximum principles for the aforementioned problems. One of the distinctive differences between our results and the standard risk-neutral case is that the adjoint equations and variational inequalities strongly depend on a risk-sensitive parameter γ. Two examples are given to illustrate the applications of the theoretical results obtained in this paper. As a natural deduction, a general maximum principle is also obtained for a fully observed risk-sensitive case. At last, this result is applied to study a risk-sensitive optimal portfolio problem. An explicit optimal investment strategy and a cost functional are obtained. A numerical simulation result shows the influence of a risk-sensitive parameter on an optimal investment proportion; this coincides with its economic meaning and theoretical results. This work was partially supported by the National Natural Science Foundation (10671112), the National Basic Research Program of China (973 Program, No. 2007CB814904), the Natural Science Foundation of Shandong Province (Z2006A01) and the Doctoral Fund of the Education Ministry of China.     

Optimal control with a cost on changing control

In this paper, we consider a class of optimal control problems in which the cost functional is the sum of the terminal cost, the integral cost, and the full variation of control. The term involving the full variation of control is to measure the changes on the control action. A computational method based on the control parametrization technique is developed for solving this class of optimal control problems. This computational method is supported by a convergence analysis. For illustration, two numerical examples are solved using the proposed method.This project was partially supported by an Australian Research Grant.This paper is dedicated to Professor L. Cesari on the occasion of his 80th birthday.     

On the algebraic structure of functional matrices of special form

Algebraic properties of functional matrices arising in the constuction of graded Padé approximations are established. This construction plays an important role in the theory of transcendental numbers. Translated fromMatematicheskie Zametki, Vol. 60, No. 6, pp. 851–860, December, 1996. This research was partially supported by the Russian Foundation for Basic Research under grant No. 94-01-00739.     

A maximum principle for optimal control problem of fully coupled forward-backward stochastic systems with partial information

The paper is concerned with a stochastic optimal control problem in which the controlled system is described by a fully coupled nonlinear forward-backward stochastic differential equation driven by a Brownian motion. It is required that all admissible control processes are adapted to a given subfiltration of the filtration generated by the underlying Brownian motion. For this type of partial information control, one sufficient (a verification theorem) and one necessary conditions of optimality are proved. The control domain need to be convex and the forward diffusion coefficient of the system can contain the control variable. This work was partially supported by Basic Research Program of China (Grant No. 2007CB814904), National Natural Science Foundation of China (Grant No. 10325101) and Natural Science Foundation of Zhejiang Province (Grant No. Y605478, Y606667)     

Adaptive boundary control of stochastic linear distributed parameter systems described by analytic semigroups

A stochastic adaptive control problem is formulated and solved for some unknown linear, stochastic distributed parameter systems that are described by analytic semigroups. The control occurs on the boundary. The highest-order operator is assumed to be known but the lower-order operators contain unknown parameters. Furthermore, the linear operators of the state and the control on the boundary contain unknown parameters. The noise in the system is a cylindrical white Gaussian noise. The performance measure is an ergodic, quadratic cost functional. For the identification of the unknown parameters a diminishing excitation is used that has no effect on the ergodic cost functional but ensures sufficient excitation for strong consistency. The adaptive control is the certainty equivalence control for the ergodic, quadratic cost functional with switchings to the zero control.This research was partially supported by NSF Grants ECS-9102714, ECS-9113029, and DMS-9305936.     

Epi-convergent discretization of the generalized Bolza problem in dynamic optimization

The paper is devoted to well-posed discrete approximations of the so-called generalized Bolza problem of minimizing variational functionals defined via extended-real-valued functions. This problem covers more conventional Bolza-type problems in the calculus of variations and optimal control of differential inclusions as well of parameterized differential equations. Our main goal is find efficient conditions ensuring an appropriate epi-convergence of discrete approximations, which plays a significant role in both the qualitative theory and numerical algorithms of optimization and optimal control. The paper seems to be the first attempt to study epi-convergent discretizations of the generalized Bolza problem; it establishes several rather general results in this direction. Research of B. S. Mordukhovich was partially supported by the USA National Science Foundation under grants DMS-0304989 and DMS-0603846 and by the Australian Research Council under grant DP-0451168. Research of T. Pennanen was supported by the Finnish Academy of Sciences under contract No. 3385.     

Uniformly overtaking and weakly overtaking optimal solutions in infinite-horizon optimal control: When optimal solutions are agreeable

In this paper, we investigate the relationship between two classes of optimality which have arisen in the study of dynamic optimization problems defined on an infinite-time domain. We utilize an optimal control framework to discuss our results. In particular, we establish relationships between limiting objective functional type optimality concepts, commonly known as overtaking optimality and weakly overtaking optimality, and the finite-horizon solution concepts of decision-horizon optimality and agreeable plans. Our results show that both classes of optimality are implied by corresponding uniform limiting objective functional type optimality concepts, referred to here as uniformly overtaking optimality and uniformly weakly overtaking optimality. This observation permits us to extract sufficient conditions for optimality from known sufficient conditions for overtaking and weakly overtaking optimality by strengthening their hypotheses. These results take the form of a strengthened maximum principle. Examples are given to show that the hypotheses of these results can be realized.This research was supported by the National Science Foundation, Grant No. DMS-87-00706, and by the Southern Illinois University at Carbondale, Summer Research Fellowship Program.     

Inexact primal-dual interior point iteration for linear programs in function spaces

Motivated by a simple optimal control problem with state constraints, we consider an inexact implementation of the primal-dual interior point algorithm of Zhang, Tapia, and Dennis. We show how the control problem can be formulated as a linear program in an infinite dimensional space in two different ways and prove convergence results.The research of this author was supported by an Overseas Research Scholarship of the Ministry of Education, Science and Culture of Japan.The research of this author was supported by National Science Foundation grants #DMS-9024622 and #DMS-9321938, North Atlantic Treaty Organization grant #CRG 920067, and an allocation of computing resources from the North Carolina Supercomputing Program.The research of this author was supported by North Atlantic Treaty Organization grant #CRG 920067.     

A feasible directions algorithm for time-lag optimal control problems with control and terminal inequality constraints

A computational algorithm for a class of time-lag optimal control problems involving control and terminal inequality constraints is presented. The convergence properties of the algorithm is also investigated. To test the algorithm, an example is solved.This work was partially supported by the Australian Research Grant Committee.     

A generalization of a minimax theorem of Fan via a theorem of the alternative

A minimization problem for a functional on a convex subsetC of a normed linear space is considered. Under certain hypotheses, optimality in a certain subset ofC implies the validity of first-order necessary optimality conditions for the problem inC. The result is applied to a problem in optimal periodic control of neutral functional differential equations.This work was partially supported by a grant from Deutsche Forschungsgemeinschaft and by AFOSR under Grant No. AFOSR-84-0398.     

Stochastic optimal linear feedback control systems using available measurements

The stochastic optimal control of linear systems with time-varying and partially observable parameters is synthesized under noisy measurements and a quadratic performance criterion. The structure of the regulator is given, and the optimal solution is reduced to a two-point boundary-value problem. Comments on the numerical solution by appropriate integration schemes is included.