Risk-averse dynamic programming for Markov decision processes

We introduce the concept of a Markov risk measure and we use it to formulate risk-averse control problems for two Markov decision models: a finite horizon model and a discounted infinite horizon model. For both models we derive risk-averse dynamic programming equations and a value iteration method. For the infinite horizon problem we develop a risk-averse policy iteration method and we prove its convergence. We also propose a version of the Newton method to solve a nonsmooth equation arising in the policy iteration method and we prove its global convergence. Finally, we discuss relations to min–max Markov decision models.     

Analysis of supply contracts with quantity flexibility

This paper explores a class of supply contracts under which a buyer receives discounts for committing to purchases in advance. The further in advance the commitment is made, the larger the discount. As time rolls forward, the buyer can increase the order quantities for future periods of the rolling horizon based on updated demand forecast information and inventory status. However, the buyer pays a higher per-unit cost for the incremental units. Such contracts are used by automobile and contract manufacturers, and are quite common in fuel oil and natural gas delivery markets. We develop a finite-horizon dynamic programming model to characterize the structure of the optimal replenishment strategy for the buyer. We present heuristic approaches to calculate the order volume in each period of the rolling horizon. Finally, we numerically evaluate the heuristic approaches and draw some managerial insights based on the findings.     

Short-term electricity procurement: A rolling horizon stochastic programming approach

This paper addresses the problem faced by a large electricity consumer in determining the optimal procurement plan over a short-term time horizon. The inherent complexity of the problem, due to its dynamic and stochastic nature, is dealt by means of the stochastic programming modeling framework. In particular, a two-stage problem is formulated with the aim of establishing the optimal amount of electricity to be purchased through bilateral contracts and in the Day-Ahead Electricity Market. Recourse actions are used to hedge against uncertainty related to future electricity prices and consumer’s needs. The optimal plan is defined so to minimize the overall cost and to control risk, which is measured in the form of violation of budget constraints. The stochastic model is dynamically solved in a rolling horizon fashion by iteratively considering more and more recent information and a planning horizon of decreasing length. Extensive numerical experiments have been carried out to assess the performance of the proposed dynamic decision approach. The results collected considering a real test case are very encouraging and provide evidence of the superiority of the approach also in comparison with other alternative procurement strategies.     

Advertising Budgeting,Wearout and Copy Replacement

In this paper we offer a formulation for the "Copy Life Cycle" phenomenon. In particular, we introduce simultaneously three decision variables, the optimal advertising expenditures during a given planning horizon, the optimal times for copy replacements and the optimal investments in the new copies. The analysis and the solution of the problem is carried out by a mixed optimization technique based on dynamic programming and optimal control theory.     

Sufficient Stochastic Maximum Principle for the Optimal Control of Semi-Markov Modulated Jump-Diffusion with Application to Financial Optimization

The finite state semi-Markov process is a generalization over the Markov chain in which the sojourn time distribution is any general distribution. In this article, we provide a sufficient stochastic maximum principle for the optimal control of a semi-Markov modulated jump-diffusion process in which the drift, diffusion, and the jump kernel of the jump-diffusion process is modulated by a semi-Markov process. We also connect the sufficient stochastic maximum principle with the dynamic programming equation. We apply our results to finite horizon risk-sensitive control portfolio optimization problem and to a quadratic loss minimization problem.     

Optimal Pricing and Production Planning for Multi-product Multi-period Systems with Backorders

In this paper, we develop models for production planning with coordinated dynamic pricing. The application that motivated this research is manufacturing pricing, where the products are non-perishable assets and can be stored to fulfill the future demands. We assume that the firm does not change the price list very frequently. However, the developed model and its solution strategy have the capability to handle the general case of manufacturing systems with frequent time-varying price lists. We consider a multi-product capacitated setting and introduce a demand-based model, where the demand is a function of the price. The key parts of the model are that the planning horizon is discrete-time multi-period, and backorders are allowed. As a result of this, the problem becomes a nonlinear programming problem with the nonlinearities in both the objective function and some constraints. We develop an algorithm which computes the optimal production and pricing policy on a finite time horizon. We illustrate the application of the algorithm through a detailed numerical example.     

Duality for nonlinear multiple-criteria optimization problems

In this paper, we develop a duality theory for nonlinear multiple-criteria optimization problems. The theory associates to efficient points a matrix, rather than a vector, of dual variables. We introduce a saddle-point dual problem, study stability concepts and Kuhn-Tucker conditions, and provide an economic interpretation of the dual matrix. The results are compared to the classical approach of deriving duality, by applying nonlinear programming duality theory to a problem obtained by conveniently weighting the criteria. Possible directions for future research are discussed.This work was performed under Grant No. MCS-77-24654, National Science Foundation.The author is grateful to Professors S. C. Graves and T. L. Magnanti, and two anonymous referees for helpful comments on an earlier version of this paper.     

A multi-parametric programming approach for constrained dynamic programming problems

In this work, we present a new algorithm for solving complex multi-stage optimization problems involving hard constraints and uncertainties, based on dynamic and multi-parametric programming techniques. Each echelon of the dynamic programming procedure, typically employed in the context of multi-stage optimization models, is interpreted as a multi-parametric optimization problem, with the present states and future decision variables being the parameters, while the present decisions the corresponding optimization variables. This reformulation significantly reduces the dimension of the original problem, essentially to a set of lower dimensional multi-parametric programs, which are sequentially solved. Furthermore, the use of sensitivity analysis circumvents non-convexities that naturally arise in constrained dynamic programming problems. The potential application of the proposed novel framework to robust constrained optimal control is highlighted.     

Horizon and stages in applications of stochastic programming in finance

To solve a decision problem under uncertainty via stochastic programming means to choose or to build a suitable stochastic programming model taking into account the nature of the real-life problem, character of input data, availability of software and computer technology. In applications of multistage stochastic programs additional rather complicated modeling issues come to the fore. They concern the choice of the horizon, stages, methods for generating scenario trees, etc. We shall discuss briefly the ways of selecting horizon and stages in financial applications. In our numerical studies, we focus on alternative choices of stages and their impact on optimal first-stage solutions of bond portfolio optimization problems. AMS Subject classification 90C15 . 92B28     

A convex analytic approach to Markov decision processes

Summary This paper develops a new framework for the study of Markov decision processes in which the control problem is viewed as an optimization problem on the set of canonically induced measures on the trajectory space of the joint state and control process. This set is shown to be compact convex. One then associates with each of the usual cost criteria (infinite horizon discounted cost, finite horizon, control up to an exit time) a naturally defined occupation measure such that the cost is an integral of some function with respect to this measure. These measures are shown to form a compact convex set whose extreme points are characterized. Classical results about existence of optimal strategies are recovered from this and several applications to multicriteria and constrained optimization problems are briefly indicated.Research supported by NSF Grant CDR-85-00108     

A stochastic programming approach for planning horizons of infinite horizon capacity planning problems

Planning horizon is a key issue in production planning. Different from previous approaches based on Markov Decision Processes, we study the planning horizon of capacity planning problems within the framework of stochastic programming. We first consider an infinite horizon stochastic capacity planning model involving a single resource, linear cost structure, and discrete distributions for general stochastic cost and demand data (non-Markovian and non-stationary). We give sufficient conditions for the existence of an optimal solution. Furthermore, we study the monotonicity property of the finite horizon approximation of the original problem. We show that, the optimal objective value and solution of the finite horizon approximation problem will converge to the optimal objective value and solution of the infinite horizon problem, when the time horizon goes to infinity. These convergence results, together with the integrality of decision variables, imply the existence of a planning horizon. We also develop a useful formula to calculate an upper bound on the planning horizon. Then by decomposition, we show the existence of a planning horizon for a class of very general stochastic capacity planning problems, which have complicated decision structure.     

Multi-period portfolio optimization for asset–liability management with bankrupt control

In this paper, we investigate a multi-period portfolio optimization problem for asset–liability management of an investor who intends to control the probability of bankruptcy before reaching the end of an investment horizon. We formulate the problem as a generalized mean–variance model that incorporates bankrupt control over intermediate periods. Based on the Lagrangian multiplier method, the embedding technique, the dynamic programming approach and the Lagrangian duality theory, we propose a method to solve the model. A numerical example is given to demonstrate our method and show the impact of bankrupt control and market parameters on the optimal portfolio strategy.     

A rolling horizon approach for disruption management of railway rolling stock

This paper deals with real-time disruption management of rolling stock in passenger railway transportation. We describe a generic framework for dealing with disruptions of railway rolling stock schedules. The framework is presented as an online combinatorial decision problem, where the uncertainty of a disruption is modeled by a sequence of information updates. To decompose the problem and to reduce the computation time, we propose a rolling horizon approach: rolling stock decisions are only considered if they are within a certain time horizon from the time of rescheduling. The schedules are then revised as time progresses and new information becomes available. We extend an existing model for rolling stock scheduling to the specific requirements of the real-time situation, and we apply it in the rolling horizon framework. We perform computational tests on instances constructed from real-life cases of Netherlands Railways (NS), the main operator of passenger trains in the Netherlands. We explore the consequences of different settings of the approach for the trade-off between solution quality and computation time.     

Multi-period lot-sizing with supplier selection: Structural results,complexity and algorithms

We consider a multi-period lot-sizing problem with multiple products and multiple suppliers. Demand is deterministic and time-varying. The objective is to determine order quantities to minimize the total cost over a finite planning horizon. This problem is strongly NP-hard. For a special case, we extend the classical zero-inventory-ordering principle and solve it by dynamic programming. Based on this new extension, we also develop a heuristic algorithm for the general problem and computationally show that it works well.     

International portfolio management with affine policies

While dynamic decision making has traditionally been represented as scenario trees, these may become severely intractable and difficult to compute with an increasing number of time periods. We present an alternative tractable approach to multiperiod international portfolio optimization based on an affine dependence between the decision variables and the past returns. Because local asset and currency returns are modeled separately, the original model is non-linear and non-convex. With the aid of robust optimization techniques, however, we develop a tractable semidefinite programming formulation of our model, where the uncertain returns are contained in an ellipsoidal uncertainty set. We add to our formulation the minimization of the worst case value-at-risk and show the close relationship with robust optimization. Numerical results demonstrate the potential gains from considering a dynamic multiperiod setting relative to a single stage approach.     

带有限反馈下层的二层规划问题的部分合作模型

对下层最优反馈为离散有限多个的二层规划问题的部分合作模型进行探讨. 当下层的合作程度依赖于上层的决策变量时, 给出一个确定合作系数函数的一般方法, 进而得到一个新的部分合作模型. 在适当地假设下, 可保证所给的部分合作模型一定可以找到比悲观解要好的解, 并结合新的部分合作模型对原不适定问题进行分析, 得到了一些有益的结论. 最后以实际算例说明了所给部分合作模型的可行性.     

Turnpike properties for a class of piecewise deterministic systems arising in manufacturing flow control

This paper deals with a general class of piecewise deterministic control systems that encompasses FMS flow control models. One uses the Markov renewal decision process formalism to characterize optimal policies via a discrete event dynamic programming approach. A family of control problems with a random stopping time is associated with these optimality conditions. These problems can be reformulated as infinite horizon deterministic control problems. It is then shown how the so-calledturnpike property should hold for these deterministic control problems under classical convexity assumptions. These turnpikes have the same generic properties as the attractors obtained via a problem specific approach in FMS flow control models and production planning and are calledhedging points in this literature.This research has been supported by NSERC-Canada, Grants No. A4952 by FCAR-Québec, Grant No. 88EQ3528, Actions Structurantes, MESS-Québec, Grant No. 6.1/7.4(28), and FNRS-Switzerland.     

Joint replacement in an operational planning phase

We consider the problem of combining replacements of multiple components in an operational planning phase. Within an infinite or finite time horizon, decisions concerning replacement of components are made at discrete time epochs. The optimal solution of this problem is limited to only a small number of components. We present a heuristic rolling horizon approach that decomposes the problem; at each decision epoch an initial plan is made that addresses components separately, and subsequently a deviation from this plan is allowed to enable joint replacement. This approach provides insight into why certain actions are taken. The time needed to determine an action at a certain epoch is only quadratic in the number of components. After dealing with harmonisation and horizon effects, our approach yields average costs less than 1% above the minimum value.     

Simplex-inspired algorithms for solving a class of convex programming problems

We consider a class of convex programming problems whose objective function is given as a linear function plus a convex function whose arguments are linear functions of the decision variables and whose feasible region is a polytope. We show that there exists an optimal solution to this class of problems on a face of the constraint polytope of dimension not more than the number of arguments of the convex function. Based on this result, we develop a method to solve this problem that is inspired by the simplex method for linear programming. It is shown that this method terminates in a finite number of iterations in the special case that the convex function has only a single argument. We then use this insight to develop a second algorithm that solves the problem in a finite number of iterations for an arbitrary number of arguments in the convex function. A computational study illustrates the efficiency of the algorithm and suggests that the average-case performance of these algorithms is a polynomial of low order in the number of decision variables. The work of T. C. Sharkey was supported by a National Science Foundation Graduate Research Fellowship. The work of H. E. Romeijn was supported by the National Science Foundation under Grant No. DMI-0355533.     

Optimal myopic policies and index policies for stochastic scheduling problems

Stochastic scheduling problems are considered by using discounted dynamic programming. Both, maximizing pure rewards and minimizing linear holding costs are treated in one common Markov decision problem. A sufficient condition for the optimality of the myopic policy for finite and infinite horizon is given. For the infinite horizon case we show the optimality of an index policy and give a sufficient condition for the index policy to be myopic. Moreover, the relation between the two sufficient conditions is discussed.