A stochastic programming approach for planning horizons of infinite horizon capacity planning problems

Planning horizon is a key issue in production planning. Different from previous approaches based on Markov Decision Processes, we study the planning horizon of capacity planning problems within the framework of stochastic programming. We first consider an infinite horizon stochastic capacity planning model involving a single resource, linear cost structure, and discrete distributions for general stochastic cost and demand data (non-Markovian and non-stationary). We give sufficient conditions for the existence of an optimal solution. Furthermore, we study the monotonicity property of the finite horizon approximation of the original problem. We show that, the optimal objective value and solution of the finite horizon approximation problem will converge to the optimal objective value and solution of the infinite horizon problem, when the time horizon goes to infinity. These convergence results, together with the integrality of decision variables, imply the existence of a planning horizon. We also develop a useful formula to calculate an upper bound on the planning horizon. Then by decomposition, we show the existence of a planning horizon for a class of very general stochastic capacity planning problems, which have complicated decision structure.     

Contractivity of Bellman operator in risk averse dynamic programming with infinite horizon

The paper deals with a risk averse dynamic programming problem with infinite horizon. First, the required assumptions are formulated to have the problem well defined. Then the Bellman equation is derived, which may be also seen as a standalone reinforcement learning problem. The fact that the Bellman operator is contraction is proved, guaranteeing convergence of various solution algorithms used for dynamic programming as well as reinforcement learning problems, which we demonstrate on the value iteration and the policy iteration algorithms.     

Pontryagin Type Maximum Principle for budget-constrained infinite horizon optimal control problems with linear dynamics

In this paper a class of infinite horizon optimal control problems with an isoperimetrical constraint, also interpreted as a budget constraint, is considered. Herein a linear both in the state and in the control dynamic is allowed. The problem setting includes a weighted Sobolev space as the state space. For this class of problems, we establish the necessary optimality conditions in form of a Pontryagin Type Maximum Principle including a transversality condition. The proved theoretical result is applied to a linear–quadratic regulator problem.     

Infinite horizon optimal control problems with multiple thermostatic hybrid dynamics

We study an optimal control problem for a hybrid system exhibiting several internal switching variables whose discrete evolutions are governed by some delayed thermostatic laws. By the dynamic programming technique we prove that the value function is the unique viscosity solution of a system of several Hamilton-Jacobi equations, suitably coupled. The method involves a contraction principle and some suitably adapted results for exit-time problems with discontinuous exit cost.     

Conditions implying the vanishing of the Hamiltonian at infinity in optimal control problems

In an infinite horizon optimal control problem the Hamiltonian vanishes at infinity when the differential equation is autonomous and the integrand in the criterion satisfies some weak integrability conditions. A generalization of Michel’s result (in Econometrica 50:975–985, 1982) is obtained.     

Different interpretations of the improper integral objective in an infinite horizon control problem

We provide an example of a convex infinite horizon problem with a linear objective functional where the different interpretations of the improper integral in either Lebesgue or Riemann sense lead to different but finite optimal values.     

Infinite horizon optimal policy for an inventory system with two types of product sharing common hardware platforms

We consider a periodic review inventory system and present its optimal policy in the infinite horizon setting. The optimal inventory policy that maximizes the infinite horizon expected discounted profit for the model is analytically obtained by relating to the finite horizon setting using results from variational analysis. Results are provided that elucidate the operations of the inventory system in the long run.     

Infinite horizon multiobjective optimal control problems in the discrete time case

This article studies multiobjective optimal control problems in the discrete time framework and in the infinite horizon case. The functions appearing in the problems satisfy smoothness conditions. This article generalizes to the multiobjective case results obtained for single-objective optimal control problems in that framework. The dynamics are governed by difference equations or difference inequations. Necessary conditions of Pareto optimality are presented, namely Pontryagin maximum principles in the weak form and in the strong form. Sufficient conditions are also provided. Other notions of Pareto optimality are defined when the infinite series do not necessarily converge.     

Value function,relaxation, and transversality conditions in infinite horizon optimal control

We investigate the value function $V:{\mathbb{R}}_{+}\times {\mathbb{R}}^n\to {\mathbb{R}}_{+}\cup \{+\infty \}$ of the infinite horizon problem in optimal control for a general—not necessarily discounted—running cost and provide sufficient conditions for its lower semicontinuity, continuity, and local Lipschitz regularity. Then we use the continuity of $V(t,?)$ to prove a relaxation theorem and to write the first order necessary optimality conditions in the form of a, possibly abnormal, maximum principle whose transversality condition uses limiting/horizontal supergradients of $V(0,?)$ at the initial point. When $V(0,?)$ is merely lower semicontinuous, then for a dense subset of initial conditions we obtain a normal maximum principle augmented by sensitivity relations involving the Fréchet subdifferentials of $V(t,?)$. Finally, when V is locally Lipschitz, we prove a normal maximum principle together with sensitivity relations involving generalized gradients of V for arbitrary initial conditions. Such relations simplify drastically the investigation of the limiting behavior at infinity of the adjoint state.     

Impulsive optimal control with finite or infinite time horizon

We consider a dynamical system subjected to feedback optimal control in such a way that the evolution of the state exhibits both sudden jumps and continuous changes. Previously obtained necessary conditions (Ref. 1) for such impulsive optimal feedback controls are generalized to admit the case of infinite time horizon; this generalization permits application to a wider class of problems. The results are illustrated by application to a version of the innkeeper's problem.Dedicated to G. Leitmann     

Necessary conditions for stochastic optimal control problems in infinite dimensions

The purpose of this paper is to establish the first and second order necessary conditions for stochastic optimal controls in infinite dimensions. The control system is governed by a stochastic evolution equation, in which both drift and diffusion terms may contain the control variable and the set of controls is allowed to be nonconvex. Only one adjoint equation is introduced to derive the first order necessary optimality condition either by means of the classical variational analysis approach or, under an additional assumption, by using differential calculus of set-valued maps. More importantly, in order to avoid the essential difficulty with the well-posedness of higher order adjoint equations, using again the classical variational analysis approach, only the first and the second order adjoint equations are needed to formulate the second order necessary optimality condition, in which the solutions to the second order adjoint equation are understood in the sense of the relaxed transposition.     

Lower closure and existence theorems for optimal control problems with infinite horizon

Lower closure theorems are proved for optimal control problems governed by ordinary differential equations for which the interval of definition may be unbounded. One theorem assumes that Cesari's property (Q) holds. Two theorems are proved which do not require property (Q), but assume either a generalized Lipschitz condition or a bound on the controls in an appropriateL _p-space. An example shows that these hypotheses can hold without property (Q) holding.     

Risk-averse stochastic optimal control: An efficiently computable statistical upper bound

In this paper, we discuss an application of the Stochastic Dual Dynamic Programming (SDDP) type algorithm to nested risk-averse formulations of Stochastic Optimal Control (SOC) problems. We propose a construction of a statistical upper bound for the optimal value of risk-averse SOC problems. This outlines an approach to a solution of a long standing problem in that area of research. The bound holds for a large class of convex and monotone conditional risk mappings. Finally, we show the validity of the statistical upper bound to solve a real-life stochastic hydro-thermal planning problem.     

Algorithms for solving discrete optimal control problems with infinite time horizon and determining minimal mean cost cycles in a directed graph as decision support tool

Time-discrete systems with a finite set of states are considered. Discrete optimal control problems with infinite time horizon for such systems are formulated. We introduce a certain graph-theoretic structure to model the transitions of the dynamical system. Algorithms for finding the optimal stationary control parameters are presented. Furthermore, we determine the optimal mean cost cycles. This approach can be used as a decision support strategy within such a class of problems; especially so-called multilayered decision problems which occur within environmental emission trading procedures can be modelled by such an approach.     

On the link between infinite horizon control and quasi-stationary distributions

We study infinite horizon control of continuous-time non-linear branching processes with almost sure extinction for general (positive or negative) discount. Our main goal is to study the link between infinite horizon control of these processes and an optimization problem involving their quasi-stationary distributions and the corresponding extinction rates. More precisely, we obtain an equivalent of the value function when the discount parameter is close to the threshold where the value function becomes infinite, and we characterize the optimal Markov control in this limit. To achieve this, we present a new proof of the dynamic programming principle based upon a pseudo-Markov property for controlled jump processes. We also prove the convergence to a unique quasi-stationary distribution of non-linear branching processes controlled by a Markov control conditioned on non-extinction.     

Asymptotically optimal production policies in dynamic stochastic jobshops with limited buffers

We consider a production planning problem for a jobshop with unreliable machines producing a number of products. There are upper and lower bounds on intermediate parts and an upper bound on finished parts. The machine capacities are modelled as finite state Markov chains. The objective is to choose the rate of production so as to minimize the total discounted cost of inventory and production. Finding an optimal control policy for this problem is difficult. Instead, we derive an asymptotic approximation by letting the rates of change of the machine states approach infinity. The asymptotic analysis leads to a limiting problem in which the stochastic machine capacities are replaced by their equilibrium mean capacities. The value function for the original problem is shown to converge to the value function of the limiting problem. The convergence rate of the value function together with the error estimate for the constructed asymptotic optimal production policies are established.     

An optimal solution for the stochastic version of the Wagner–Whitin dynamic lot-size model

We present an algorithm for determining the optimal solution over the entire planning horizon for the dynamic lot-size model where demand is stochastic and non-stationary. The optimal solution to the deterministic problem is the well-known Wagner–Whitin algorithm. The present work contributes principally to knowledge building and provides a tool for researchers. One potentially useful contribution to practice is the solution to an important special case, where demand follows normal distributions. Other contributions to practice will likely flow from the development of improved heuristics and the improved basis to evaluate heuristic performance.     

离散无穷时期不完全资产市场随机平衡

Abstract. This paper examines the existence of general equilibrium in a discrete time economywith the infinite horizon incomplete markets. There is a single good at each node in the eventtree. The existence of general equilibrium for the infinite horizon economy is proved by takinglimit of equilibria in truncated economies in which trade stops at a sequence of dates.     

Dynamic programming in stochastic control of systems with delay

We consider optimal control problems for systems described by stochastic differential equations with delay (SDDE). We prove a version of Bellman's principle of optimality (the dynamic programming principle) for a general class of such problems. That the class in general means that both the dynamics and the cost depends on the past in a general way. As an application, we study systems where the value function depends on the past only through some weighted average. For such systems we obtain a Hamilton-Jacobi-Bellman partial differential equation that the value function must solve if it is smooth enough. The weak uniqueness of the SDDEs we consider is our main tool in proving the result. Notions of strong and weak uniqueness for SDDEs are introduced, and we prove that strong uniqueness implies weak uniqueness, just as for ordinary stochastic differential equations.