Optimal control of a dam under seasonal electricity prices

This study concerns the optimal control of a hydroelectric dam under seasonal electricity prices: high in winter, low in summer. The goal is to maximize the expected discounted infinite-horizon return through a policy that determines the amount of electricity to be produced in each time period, depending on the water level at the beginning of the period under consideration. The prices are assumed to be deterministic, and the flows into the reservoir are seasonal, stochastic, but independent from one period to another. The electric power generated is proportional to the amount of water flow through the turbines. There exist seepage and evaporation losses.It is shown that in the simplest price structure, the optimal policy is entirely determined by a single critical water level in each period of time, at which one starts producing. An example shows that the discretization of the reservoir levels can destroy this property. A method is proposed to avoid this difficulty. Another way of defining a policy is through goal-levels. This approach is shown to give higher returns than the standard approach.     

Optimal myopic policies and index policies for stochastic scheduling problems

Stochastic scheduling problems are considered by using discounted dynamic programming. Both, maximizing pure rewards and minimizing linear holding costs are treated in one common Markov decision problem. A sufficient condition for the optimality of the myopic policy for finite and infinite horizon is given. For the infinite horizon case we show the optimality of an index policy and give a sufficient condition for the index policy to be myopic. Moreover, the relation between the two sufficient conditions is discussed.     

Optimal replacement policies for two-unit machines with increasing running costs 1

A machine consists of two stochastically failing units. Failure of either of the units causes a failure of the machine and the failed unit has to be replaced immediately. Associated with the units are running costs which increase with the age of the unit because of increasing maintenance costs, decreasing output, etc.A preventive replacement policy is proposed under which, at failure points, we also replace the second unit if its age exceeds a predetermined control limit. It is proved that, for two identical units with exponential life-time distributions and linear running costs, this policy is optimal and the optimal control limit is calculated. In an additional model we take into consideration the length of time it takes to replace one unit or both units.The method of solution is a variation of dynamic semi-Markov programming. Analytical results are obtained and the influence of the various parameters on them is investigated. Finally, we study the saving due to our policy in comparison with a policy in which only failed units are replaced.     

Optimal greedy policies for stochastic control models

We introduce the notion of a greedy policy for general stochastic control models. Sufficient conditions for the optimality of the greedy policy for finite and infinite horizon are given. Moreover, we derive error bounds if the greedy policy is not optimal. The main results are illustrated by Bayesian information models, discounted Bayesian search problems, stochastic scheduling problems, single-server queueing networks and deterministic dynamic programs.     

Optimal ordering policies for periodic-review systems with a refined intra-cycle time scale

Chiang [C. Chiang, Optimal ordering policies for periodic-review systems with replenishment cycles, European Journal of Operational Research 170 (2006) 44–56] recently proposed a dynamic programming model for periodic-review systems in which a replenishment cycle consists of a number of small periods (each of identical but arbitrary length) and holding and shortage costs are charged based on the ending inventory of small periods. The current paper presents an alternative (and concise) dynamic programming model. Moreover, we allow the possibility of a positive fixed cost of ordering. The optimal policy is of the familiar (s, S) type because of the convexity of the one-cycle cost function. As in the periodic-review inventory literature, we extend this result to the lost-sales periodic problem with zero lead-time. Computation shows that the long-run average cost is rather insensitive to the choice of the period length. In addition, we show how the proposed model is modified to handle the backorder problem where shortage is charged on a per-unit basis irrespective of its duration. Finally, we also investigate the lost-sales problem with positive lead-time, and provide some computational results.     

Optimal feedback control policies for stochastic,distributed-parameter systems

This paper considers optimal feedback control policies for a class of discrete stochastic distributed-parameter systems. The class under consideration has the property that the random variable in the dynamic systems depends only on the time and possesses the Markovian property with stationary transition probabilities. A necessary condition for optimality of a feedback control policy, which has form similar to the Hamiltonian form in the deterministic case, is derived via a dynamic programming approach.     

Optimal intervention policies for a multidimensional simple epidemic process

We consider a deterministic simple epidemic process in which the susceptibles are exposed to n+1 diseases. It is assumed that one disease is relatively harmless while the others cause serious symptoms. Policies for introducing infection by the harmless disease are considered and, under a suitable cost structure, the optimal policy that minimises the future cost for every initial state is found. For the corresponding stochastic model, the optimal policy is found by implementing a suitable dynamic programming algorithm, and is compared numerically with the optimal policy for the deterministic model.     

Optimal economic growth with foreign trade and a learning industry

Ryder (Ref. 1) has developed a simple two-sector macroeconomic model incorporatinglearning by doing as a determinant of the growth of productivity-enhancing experience and its effect on foreign trade. In this paper, optimal foreign trade and capital allocation policies are determined, in the context of the above model, for ranges of the international trade price not considered by Ryder. An extension of Ryder's model to include a dual trade price structure is briefly discussed. A specific numerical example is used to ascertain the configuration of the various features occurring in the extremal fields at different price levels.     

Optimal control policy for a standing order inventory system

In this paper, we consider a standing order inventory system in which an order of fixed size arrives in each period. Since demand is stochastic, such a system must allow for procurement of extra units in the case of an emergency and sell-offs of excess inventory. Assuming the average-cost criterion, Rosenshine and Obee (Operations Research 24 (1976) 1143–1155) first studied such a system and devised a 4-parameter inventory control policy that is not generally optimal. The current paper uses dynamic programming to determine the optimal control policy for a standing order system, which consists of only two operational parameters: the dispose-down-to level and order-up-to level. Either the average-cost or discounted-cost criterion can be assumed in the proposed model. Also, both the backlogged and lost-sales problems are investigated in this paper. By using a convergence theorem, we stop the dynamic programming computation and obtain the two optimal parameters.     

Optimal cost and policy for a Markovian replacement problem

We consider the computation of the optimal cost and policy associated with a two-dimensional Markov replacement problem with partial observations, for two special cases of observation quality. Relying on structural results available for the optimal policy associated with these two particular models, we show that, in both cases, the infinitehorizon, optimal discounted cost function is piecewise linear, and provide formulas for computing the cost and the policy. Several examples illustrate the usefulness of the results.This research was supported by the Air Force Office of Scientific Research Grant AFOSR-86-0029, by the National Science Foundation Grant ECS-86-17860, by the Advanced Technology Program of the State of Texas, and by the Air Force Office of Scientific Research (AFSC) Contract F49620-89-C-0044.     

Optimal choice of dividend barriers for a risk process with stochastic return on investments

We consider a risk process with stochastic return on investments and we are interested in expected present value of all dividends paid until ruin occurs when the company uses a simple barrier strategy, i.e. when it pays dividends whenever its surplus reaches a level b. It is shown that given the barrier b, this expected value can be found by solving a boundary value problem for an integro-differential equation. The solution is then found in two special cases; when return on investments is constant and the surplus generating process is compound Poisson with exponentially distributed claims, and also when both return on investments as well as the surplus generating process are Brownian motions with drift. Also in this latter case we are able to find the optimal barrier b^*, i.e. the barrier that gives the highest expected present value of dividends. Parallell with this we treat the problem of finding the Laplace transform of the distribution of the time to ruin when a barrier strategy is employed, noting that the probability of eventual ruin is 1 in this case. The paper ends with a short discussion of the same problems when a time dependent barrier is employed.     

Optimal control of a facility with periodic interrupted demand

The system investigated consists of a stochastic periodic stream of raw material, a continuous processing operation with controllable deterministic service rates, and a storage facility. The arrival stream is periodically interrupted and divided into alternating on-off intervals of fixed length. The processing facility is allowed to operate during the off-interval. Superimposed on this system is a cost structure composed of processing and holding costs. Such operations may be found in manufacturing as well as service systems (for example, dry cleaners, machine shops, repair and maintenance shops, printers, information processing centers, etc). A service rate control rule that minimizes the infinite-horizon discounted expected total cost is found. Existence and uniqueness of long-term optimal cost and policy functions is shown. Since the optimal policy cannot be expressed explicitly, an approximate solution was obtained. An error bound on the optimal cost associated with this solution is exhibited. The approximate solution is characterized by a service rate control rule that is a linear function of the level of inventory at the start of each on-interval and a piecewise linear function of inventory at the start of each off-interval. The optimal discounted expected total cost is quadratic in the inventory level at the start of each interval. Computational results indicate relative cost errors in the order of 2–3 percent.This research was performed at the Sanitary Engineering Research Laboratory and Operations Research Center of the University of California, Berkeley. It was made possible by US Public Health Research Grant UI-00547 from the Environmental Control Administration-Bureau of Solid Waste Management and by National Science Foundation Grant GK-1684.The author thanks Professor C. R. Glassey for not only suggesting this research, but for his constant encouragement and suggestions throughout its duration. He also thanks Professors W. S. Jewell and P. H. McGauhey whose comments on the draft were very helpful.     

Control of a production-inventory system with returns under imperfect advance return information

We consider a production-inventory system with product returns that are announced in advance by the customers. Demands and announcements of returns occur according to independent Poisson processes. An announced return is either actually returned or cancelled after a random return lead time. We consider both lost sale and backorder situations. Using a Markov decision formulation, the optimal production policy, with respect to the discounted cost over an infinite horizon, is characterized for situations with and without advance return information. We give insights in the potential value of this information. Also some attention is paid to combining advance return and advance demand information. Further applications of the model as well as topics for further research are indicated.     

Optimal regulation of the service rate for a queue with finite waiting space

The optimal control for the service rate of a single-server queue with limited waiting space is derived. Costs are associated with providing the service and with waiting. A comparison with the traditional steady-state result is made.     

Optimal policy for a dynamic,non-stationary,stochastic inventory problem with capacity commitment

This paper studies a single-product, dynamic, non-stationary, stochastic inventory problem with capacity commitment, in which a buyer purchases a fixed capacity from a supplier at the beginning of a planning horizon and the buyer’s total cumulative order quantity over the planning horizon is constrained with the capacity. The objective of the buyer is to choose the capacity at the beginning of the planning horizon and the order quantity in each period to minimize the expected total cost over the planning horizon. We characterize the structure of the minimum sum of the expected ordering, storage and shortage costs in a period and thereafter and the optimal ordering policy for a given capacity. Based on the structure, we identify conditions under which a myopic ordering policy is optimal and derive an equation for the optimal capacity commitment. We then use the optimal capacity and the myopic ordering policy to evaluate the effect of the various parameters on the minimum expected total cost over the planning horizon.     

Optimal immunization rules for an epidemic with recovery

The classical Kermack-McKendrick model for the spread of an epidemic through a closed population has recently been extended by Billard to allow for the recovery and possible reinfection of infective cases. In this paper, we study the optimal control of such an epidemic through immunization of susceptibles when costs are proportional to the area under the infectives trajectory plus the total number of immunizations. When the immunization rate is bounded, optimal controls are of bang-bang type and are characterized by switching curves in the epidemic state space. Explicit expressions for these curves are obtained in the case of deterministic dynamics. When the epidemic is described by a Markov chain, numerical solutions for the switching curve are easy to obtain by dynamic programming, and useful analytic approximations to them are described. The results include those for the so-called general epidemic in which no recovery is allowed.The author is grateful to the referees for their detailed and constructive criticism of an earlier version of this paper.     

Optimal policies for a delay time model with postponed replacement

We develop a delay time model (DTM) to determine the optimal maintenance policy under a novel assumption: postponed replacement. Delay time is defined as the time lapse from the occurrence of a defect up until failure. Inspections can be performed to monitor the system state at non-negligible cost. Most works in the literature assume that instantaneous replacement is enforced as soon as a defect is detected at an inspection. In contrast, we relax this assumption and allow replacement to be postponed for an additional time period. The key motivation is to achieve better utilization of the system’s useful life, and reduce replacement costs by providing a sufficient time window to prepare maintenance resources. We model the preventive replacement cost as a non-increasing function of the postponement interval. We then derive the optimal policy under the modified assumption for a system with exponentially distributed defect arrival time, both for a deterministic delay time and for a more general random delay time. For the settings with a deterministic delay time, we also establish an upper bound on the cost savings that can be attained. A numerical case study is presented to benchmark the benefits of our modified assumption against conventional instantaneous replacement discussed in the literature.     

Optimal admission policies for a finite queue with bursty arrivals

We consider a service system with a single server, a finite waiting room and two classes of customers with deterministic service time. Primary jobs arrive at random and are admitted whenever there is room in the system. At the beginning of each period, secondary jobs can be admitted from an infinite pool. A revenue is earned upon admission of each job, with the primary jobs bringing a higher contribution than the secondary jobs, the objective being to maximize the total discounted revenue over an infinite horizon. We model the system as a discrete time Markov decision process and show that a monotone admission policy is optimal when the number of primary arrivals has a fixed distribution. Moreover, when the primary arrival distribution varies with time according to a finite state Markov chain, we show that the optimal policy is conditionally monotone and that the numerical computation of an optimal policy, in this case, is substantially more difficult than in the case of stationary arrivals.This research was supported in part by the National Science Foundation, under grant ECS-8803061, while the author was at the University of Arizona.     

Heston随机波动率市场中带VaR约束的最优投资策略

本文研究了Heston随机波动率市场下, 基于VaR约束下的动态最优投资组合问题。
假设Heston随机波动率市场由一个无风险资产和一个风险资产构成,投资者的目标为最大化其终端的期望效用。与此同时, 投资者将动态地评估其待选的投资组合的VaR风险,并将其控制在一个可接受的范围之内。本文在合理的假设下,使用动态规划的方法,来求解该问题的最优投资策略。在特定的参数范围内,利用数值方法计算出近似的最优投资策略和相应值函数, 并对结果进行了分析。     

Optimal policies for inventory systems with two types of product sharing common hardware platforms: Single period and finite horizon

In this paper, we consider an inventory system whose products share a common hardware platform but are differentiated by two types of software. Choice of different software results in different installation cost and different selling price of the whole product. Product with different software also faces different customer demand. We investigate the optimal proportion of an order to be installed with software 1 or 2, that maximizes expected profit in the single and multiple period scenarios, respectively. The optimal policy is analytically obtained and proved to be an order-up-to policy in each scenario. Our investigation reveals that whether to replenish, and how much to replenish each product depend not only on its own initial inventory level, and system parameters, but also the initial inventory level of the other product. We perform numerical experiments using the optimal policies we have derived in the paper.