Dynamic programming and principles of optimality

A sequential decision model is developed in the context of which three principles of optimality are defined. Each of the principles is shown to be valid for a wide class of stochastic sequential decision problems. The relationship between the principles and the functional equations of dynamic programming is investigated and it is shown that the validity of each of them guarantees the optimality of the dynamic programming solutions. As no monotonicity assumption is made regarding the reward functions, the results presented in this paper resolve certain questions raised in the literature as to the relation among the principles of optimality and the optimality of the dynamic programming solutions.     

On a class of dynamic programming problems whose optimal controls and states are independent of the future

When applying dynamic programming for optimal decision making one usually needs considerable knowledge about the future. This knowledge, e.g. about future functions and parameters, necessary to determine optimal control policies, however, is often not available and thus precludes the application of dynamic programming.In the present paper it is shown that for a certain class of dynamic programming problems the optimal control policy is independent of the future. To illustrate the results an application in inventory control is given and further applications in the theories of economic growth and corporate finance are listed in the references.     

First-order optimization algorithms via inertial systems with Hessian driven damping

For controlled discrete-time stochastic processes we introduce a new class of dynamic risk measures, which we call process-based. Their main feature is that they measure risk of processes that are functions of the history of a base process. We introduce a new concept of conditional stochastic time consistency and we derive the structure of process-based risk measures enjoying this property. We show that they can be equivalently represented by a collection of static law-invariant risk measures on the space of functions of the state of the base process. We apply this result to controlled Markov processes and we derive dynamic programming equations. We also derive dynamic programming equations for multistage stochastic programming with decision-dependent distributions.

     

Fast value iteration: an application of Legendre-Fenchel duality to a class of deterministic dynamic programming problems in discrete time*

ABSTRACT

We propose an algorithm, which we call ‘Fast Value Iteration’ (FVI), to compute the value function of a deterministic infinite-horizon dynamic programming problem in discrete time. FVI is an efficient algorithm applicable to a class of multidimensional dynamic programming problems with concave return (or convex cost) functions and linear constraints. In this algorithm, a sequence of functions is generated starting from the zero function by repeatedly applying a simple algebraic rule involving the Legendre-Fenchel transform of the return function. The resulting sequence is guaranteed to converge, and the Legendre-Fenchel transform of the limiting function coincides with the value function.     

The dynamic programming method in the generalized traveling salesman problem

A procedure of the dynamic programming (DP) for the discrete-continuous problem of a route optimization is considered. It is possible to consider this procedure as a dynamic method of optimization of the towns choice in the well-known traveling salesman problem. In the considered version of DP, elements of a dynamic optimization are used. Two variants of the function of the aggregations of losses are investigated:

• 1.(1) the additive functions;
• 2.(2) the function characterizing the aggregation of losses in the bottle-neck problem.

     

A shortest path algorithm in robotics and its implementation on the FPS T-20 hypercube

A broad class of problems involving the optimal control of robot arms can be formulated as dynamic programming problems whose structure is particularly attractive for parallel processing. For certain simple cost functions the dynamic programming formulation reduces to determining the shortest path through a network. This algorithm has been implemented on a Floating Point Systems' T-20 hypercube computer. An analysis of the performance of the algorithm provides several important insights into the interplay between problem size and the number of processors in a parallel computer. The results also underscore the potential for parallel computers in real-time control applications.This work was supported in part by the Office of Naval Research, Contract N 00014-86-K-0693.     

A class of nondifferentiable continuous programming problems

Optimality conditions, duality and converse duality results are obtained for a class of continuous programming problems with a nondifferentiable term in the integrand of the objective function. The proofs are based on a Fritz John theorem for constrained optimization in abstract spaces. The results generalize various well-known results in variational problems with differentiable functions, and also give a dynamic analogue of certain nondifferentiable programming problems.     

Comparative studies on dynamic programming and integer programming approaches for concave cost production/inventory control problems

This paper is concerned with classical concave cost multi-echelon production/inventory control problems studied by W. Zangwill and others. It is well known that the problem with m production steps and n time periods can be solved by a dynamic programming algorithm in O(n ⁴ m) steps, which is considered as the fastest algorithm for solving this class of problems. In this paper, we will show that an alternative 0–1 integer programming approach can solve the same problem much faster particularly when n is large and the number of 0–1 integer variables is relatively few. This class of problems include, among others problem with set-up cost function and piecewise linear cost function with fewer linear pieces. The new approach can solve problems with mixed concave/convex cost functions, which cannot be solved by dynamic programming algorithms.     

On synthesizing impulse controls and the theory of fast controls

We describe the theory of feedback control in the class of impulse-type inputs which allow higher derivatives of delta functions. We provide solutions based on Hamiltonian techniques in the dynamic programming form. Further we describe physically realizable approximations of the “ideal” impulse-type solutions by bounded functions which may also serve as “fast” feedback controls that solve the target control problem in arbitrarily small time.     

A dynamic programming approach for solving single-source uncapacitated concave minimum cost network flow problems

In this paper, we describe a dynamic programming approach to solve optimally the single-source uncapacitated minimum cost network flow problem with general concave costs. This class of problems is known to be NP-Hard and there is a scarcity of methods to solve them in their full generality. The algorithms previously developed critically depend on the type of cost functions considered and on the number of nonlinear arc costs. Here, a new dynamic programming approach that does not depend on any of these factors is proposed. Computational experiments were performed using randomly generated problems. The computational results reported for small and medium size problems indicate the effectiveness of the proposed approach.     

Subgradients of value functions in parametric dynamic programming

We study in this paper the first-order behavior of value functions in parametric dynamic programming with linear constraints and nonconvex cost functions. By establishing an abstract result on the Fréchet subdifferential of value functions of parametric mathematical programming problems, some new formulas on the Fréchet subdifferential of value functions in parametric dynamic programming are obtained.     

Logarithmic transformations for discrete-time,finite-horizon stochastic control problems

Using logarithmic transformations, we construct discrete-time stochastic control problems where the optimal value function (cost-to-go) belongs to a same parametrized class of functions that remains invariant under the dynamic programming operator. This extends a well-known property of the classical LQG problems, where the optimal value function is a quadratic. Some related questions are also discussed.     

Dynamic programming and minimum risk paths

This paper addresses the problem of computing minimum risk paths by taking as objective the expected accident cost. The computation is based on a dynamic programming formulation which can be considered an extension of usual dynamic programming models: path costs are recursively computed via functions which are assumed to be monotonic. A large part of the paper is devoted to analyze in detail this formulation and provide some new results. Based on the dynamic programming model a linear programming model is also presented to compute minimum risk paths. This formulation turns out to be useful in solving a biobjective version of the problem, in which also expected travel length is taken into consideration. This leads to define nondominated mixed strategies. Finally it is shown how to extend the basic updating device of dynamic programming in order to enumerate all nondominated paths.     

Parametric simplex algorithms for solving a special class of nonconvex minimization problems

It is shown that parametric linear programming algorithms work efficiently for a class of nonconvex quadratic programming problems called generalized linear multiplicative programming problems, whose objective function is the sum of a linear function and a product of two linear functions. Also, it is shown that the global minimum of the sum of the two linear fractional functions over a polytope can be obtained by a similar algorithm. Our numerical experiments reveal that these problems can be solved in much the same computational time as that of solving associated linear programs. Furthermore, we will show that the same approach can be extended to a more general class of nonconvex quadratic programming problems.     

Viscosity Solutions of HJB Equations with Unbounded Data and Characteristic Points

We study a class of infinite horizon and exit-time control problems for nonlinear systems with unbounded data using the dynamic programming approach. We prove local optimality principles for viscosity super- and subsolutions of degenerate Hamilton–Jacobi equations in a very general setting. We apply these results to characterize the (possibly multiple) discontinuous solutions of Dirichlet and free boundary value problems as suitable value functions for the above-mentioned control problems.     

Viscosity Solutions of HJB Equations with Unbounded Data and Characteristic Points

We study a class of infinite horizon and exit-time control problems for nonlinear systems with unbounded data using the dynamic programming approach. We prove local optimality principles for viscosity super- and subsolutions of degenerate Hamilton–Jacobi equations in a very general setting. We apply these results to characterize the (possibly multiple) discontinuous solutions of Dirichlet and free boundary value problems as suitable value functions for the above-mentioned control problems.     

DC Programming: Overview

Mathematical programming problems dealing with functions, each of which can be represented as a difference of two convex functions, are called DC programming problems. The purpose of this overview is to discuss main theoretical results, some applications, and solution methods for this interesting and important class of programming problems. Some modifications and new results on the optimality conditions and development of algorithms are also presented.     

Intensional dynamic programming. A Rosetta stone for structured dynamic programming

We present intensional dynamic programming (IDP), a generic framework for structured dynamic programming over atomic, propositional and relational representations of states and actions. We first develop set-based dynamic programming and show its equivalence with classical dynamic programming. We then show how to describe state sets intensionally using any form of structured knowledge representation and obtain a generic algorithm that can optimally solve large, even infinite, MDPs without explicit state space enumeration. We derive two new Bellman backup operators and algorithms. In order to support the view of IDP as a Rosetta stone for structured dynamic programming, we review many existing techniques that employ either propositional or relational knowledge representation frameworks.     

Generalized Envelope Theorems: Applications to Dynamic Programming

We show in this paper that the class of Lipschitz functions provides a suitable framework for the generalization of classical envelope theorems for a broad class of constrained programs relevant to economic models, in which nonconvexities play a key role, and where the primitives may not be continuously differentiable. We give sufficient conditions for the value function of a Lipschitz program to inherit the Lipschitz property and obtain bounds for its upper and lower directional Dini derivatives. With strengthened assumptions we derive sufficient conditions for the directional differentiability, Clarke regularity, and differentiability of the value function, thus obtaining a collection of generalized envelope theorems encompassing many existing results in the literature. Some of our findings are then applied to decision models with discrete choices, to dynamic programming with and without concavity, to the problem of existence and characterization of Markov equilibrium in dynamic economies with nonconvexities, and to show the existence of monotone controls in constrained lattice programming problems.     

一类二层凸规划的对偶二层规划

本文讨论上层目标函数以下层子系统目标函数的最优值作为反馈的一类二层凸规划的对偶规划问题 ,在构成函数满足凸连续可微等条件的假设下 ,建立了二层凸规划的 Lagrange对偶二层规划 ,并证明了基本对偶定理 .