Optimal myopic policy for a stochastic inventory problem with fixed and proportional backorder costs

In this paper, we consider a single product, periodic review, stochastic demand inventory problem where backorders are allowed and penalized via fixed and proportional backorder costs simultaneously. Fixed backorder cost associates a one-shot penalty with stockout situations whereas proportional backorder cost corresponds to a penalty for each demanded but yet waiting to be satisfied item. We discuss the optimality of a myopic base-stock policy for the infinite horizon case. Critical number of the infinite horizon myopic policy, i.e., the base-stock level, is denoted by S. If the initial inventory is below S then the optimal policy is myopic in general, i.e., regardless of the values of model parameters and demand density. Otherwise, the sufficient condition for a myopic optimum requires some restrictions on demand density or parameter values. However, this sufficient condition is not very restrictive, in the sense that it holds immediately for Erlang demand density family. We also show that the value of S can be computed easily for the case of Erlang demand. This special case is important since most real-life demand densities with coefficient of variation not exceeding unity can well be represented by an Erlang density. Thus, the myopic policy may be considered as an approximate solution, if the exact policy is intractable. Finally, we comment on a generalization of this study for the case of phase-type demands, and identify some related research problems which utilize the results presented here.     

Optimal greedy policies for stochastic control models

We introduce the notion of a greedy policy for general stochastic control models. Sufficient conditions for the optimality of the greedy policy for finite and infinite horizon are given. Moreover, we derive error bounds if the greedy policy is not optimal. The main results are illustrated by Bayesian information models, discounted Bayesian search problems, stochastic scheduling problems, single-server queueing networks and deterministic dynamic programs.     

Optimal policy for minimizing risk models in Markov decision processes

We consider the minimizing risk problems in discounted Markov decisions processes with countable state space and bounded general rewards. We characterize optimal values for finite and infinite horizon cases and give two sufficient conditions for the existence of an optimal policy in an infinite horizon case. These conditions are closely connected with Lemma 3 in White (1993), which is not correct as Wu and Lin (1999) point out. We obtain a condition for the lemma to be true, under which we show that there is an optimal policy. Under another condition we show that an optimal value is a unique solution to some optimality equation and there is an optimal policy on a transient set.     

Value Functions and Transversality Conditions for Infinite-Horizon Optimal Control Problems

This paper investigates a relationship between the maximum principle with an infinite horizon and dynamic programming and sheds new light upon the role of the transversality condition at infinity as necessary and sufficient conditions for optimality with or without convexity assumptions. We first derive the nonsmooth maximum principle and the adjoint inclusion for the value function as necessary conditions for optimality. We then present sufficiency theorems that are consistent with the strengthened maximum principle, employing the adjoint inequalities for the Hamiltonian and the value function. Synthesizing these results, necessary and sufficient conditions for optimality are provided for the convex case. In particular, the role of the transversality conditions at infinity is clarified.     

The infinite horizon non-stationary stochastic inventory problem: Near myopic policies and weak ergodicity

In this paper we consider a periodic review dynamic inventory problem with non-stationary demands. The purpose of this paper is to show that near myopic policies are sufficiently close to optimal decisions for the infinite horizon inventory problem. In order to show this we pay attention to the fact that inventory processes with base-stock policies are weakly ergodic, and we discuss how the weak ergodicity is related to near myopic policies. Then we derive the error bounds of near myopic policies for the optimal decisions and evaluate them with a number of numerical experiments.     

Control and scheduling in a two-station queueing network: Optimal policies and heuristics

Consider a two-station queueing network with two types of jobs: type 1 jobs visit station 1 only, while type 2 jobs visit both stations in sequence. Each station has a single server. Arrival and service processes are modeled as counting processes with controllable stochastic intensities. The problem is to control the arrival and service processes, and in particular to schedule the server in station 1 among the two job types, in order to minimize a discounted cost function over an infinite time horizon. Using a stochastic intensity control approach, we establish the optimality of a specific stationary policy, and show that its value function satisfies certain properties, which lead to a switching-curve structure. We further classify the problem into six parametric cases. Based on the structural properties of the stationary policy, we establish the optimality of some simple priority rules for three of the six cases, and develop heuristic policies for the other three cases.     

Non-randomized policies for constrained Markov decision processes

This paper addresses constrained Markov decision processes, with expected discounted total cost criteria, which are controlled by non-randomized policies. A dynamic programming approach is used to construct optimal policies. The convergence of the series of finite horizon value functions to the infinite horizon value function is also shown. A simple example illustrating an application is presented.     

Controlled jump processes

Finite and infinite planning horizon Markov decision problems are formulated for a class of jump processes with general state and action spaces and controls which are measurable functions on the time axis taking values in an appropriate metrizable vector space. For the finite horizon problem, the maximum expected reward is the unique solution, which exists, of a certain differential equation and is a strongly continuous function in the space of upper semi-continuous functions. A necessary and sufficient condition is provided for an admissible control to be optimal, and a sufficient condition is provided for the existence of a measurable optimal policy. For the infinite horizon problem, the maximum expected total reward is the fixed point of a certain operator on the space of upper semi-continuous functions. A stationary policy is optimal over all measurable policies in the transient and discounted cases as well as, with certain added conditions, in the positive and negative cases.     

A new strong optimality criterion for nonstationary Markov decision processes

This paper deals with a new optimality criterion consisting of the usual three average criteria and the canonical triplet (totally so-called strong average-canonical optimality criterion) and introduces the concept of a strong average-canonical policy for nonstationary Markov decision processes, which is an extension of the canonical policies of Herna′ndez-Lerma and Lasserre [16] (pages: 77) for the stationary Markov controlled processes. For the case of possibly non-uniformly bounded rewards and denumerable state space, we first construct, under some conditions, a solution to the optimality equations (OEs), and then prove that the Markov policies obtained from the OEs are not only optimal for the three average criteria but also optimal for all finite horizon criteria with a sequence of additional functions as their terminal rewards (i.e. strong average-canonical optimal). Also, some properties of optimal policies and optimal average value convergence are discussed. Moreover, the error bound in average reward between a rolling horizon policy and a strong average-canonical optimal policy is provided, and then a rolling horizon algorithm for computing strong average ε(>0)-optimal Markov policies is given.     

Necessity of limiting co-state arcs in Bolza-type infinite horizon problem

We investigate necessary conditions of optimality for the Bolza-type infinite horizon problem with free right end. The optimality is understood in the sense of weakly uniformly overtaking optimal control. No previous knowledge in the asymptotic behaviour of trajectories or adjoint variables is necessary. Following Seierstad’s idea, we obtain the necessary boundary condition at infinity in the form of a transversality condition for the maximum principle. Those transversality conditions may be expressed in the integral form through an Aseev–Kryazhimskii-type formulae for co-state arcs. The connection between these formulae and limiting gradients of pay-off function at infinity is identified; several conditions under which it is possible to explicitly specify the co-state arc through those Aseev–Kryazhimskii-type formulae are found. For infinite horizon problem of Bolza type, an example is given to clarify the use of the Aseev–Kryazhimskii formula as an explicit expression of the co-state arc.     

Optimality of (s, S) policies for jump inventory models

This paper is concerned with the optimality of (s, S) policies for a single-item inventory control problem which minimizes the total expected cost over an infinite planning horizon and where the demand is driven by a piecewise deterministic process. Our approach is based on the theory of quasi-variational inequality.     

Convex infinite horizon programs

We establish conditions under which a sequence of finite horizon convex programs monotonically increases in value to the value of the infinite program; a subsequence of optimal solutions converges to the optimal solution of the infinite problem. If the conditions we impose fail, then (roughtly) the optimal value of the infinite horizon problem is an improper convex function. Under more restrictive conditions we establish the necessary and sufficient conditions for optimality. This constructive procedure gives us a way to solve the infinite (long range) problem by solving a finite (short range) problem. It appears to work well in practice.     

Convexity and characterization of optimal policies in a dynamic routing problem

An infinite horizon, expected average cost, dynamic routing problem is formulated for a simple failure-prone queueing system, modelled as a continuous time, continuous state controlled stochastic process. We prove that the optimal average cost is independent of the initial state and that the cost-to-go functions of dynamic programming are convex. These results, together with a set of optimality conditions, lead to the conclusion that optimal policies are switching policies, characterized by a set of switching curves (or regions), each curve corresponding to a particular state of the nodes (servers).     

Existence of optimal stationary policies in deterministic optimal control

This paper considers deterministic discrete-time optimal control problems over an infinite horizon involving a stationary system and a nonpositive cost per stage. Various results are provided relating to existence of an ?-optimal stationary policy, and existence of an optimal stationary policy assuming an optimal policy exists.     

Optimal scheduling policies in time sharing service systems

We consider optimal scheduling problems in a TSSS (Time Sharing Service System), i.e., a tandem queueing network consisting of multiple service stations, all of which are served by a single server. In each station, a customer can receive service time up to the prescribed station dependent upper bound, but he must proceed to the next station in order to receive further service. After the total amount of the received services reaches his service requirement, he departs from the network. The optimal policy for this system minimizes the long-run average expected waiting cost per unit of time over the infinite planning horizon. It is first shown that, if the distribution of customer's service requirement is DMRL (Decreasing Mean Residual Life), the policy of giving the highest priority to the customer with the most attained service time is optimal under a set of some appropriate conditions. This implies that any policy without interruptions and preemptions of services is optimal. If the service requirement is DFR (Decreasing Failure Rate), on the other hand, it is shown that the policy of giving the highest priority to the customer with the least attained service time, i.e., the so-called LAST (Least Attained Service Time first) is optimal under another set of some appropriate conditions. These results can be generalized to the case in which there exist multiple classes of customers, but each class satisfies one of the above sets of conditions.     

Denumerable controlled Markov chains with strong average optimality criterion: Bounded & unbounded costs

This paper studies discrete-time nonlinear controlled stochastic systems, modeled by controlled Markov chains (CMC) with denumerable state space and compact action space, and with an infinite planning horizon. Recently, there has been a renewed interest in CMC with a long-run, expected average cost (AC) optimality criterion. A classical approach to study average optimality consists in formulating the AC case as a limit of the discounted cost (DC) case, as the discount factor increases to 1, i.e., as the discounting effectvanishes. This approach has been rekindled in recent years, with the introduction by Sennott and others of conditions under which AC optimal stationary policies are shown to exist. However, AC optimality is a rather underselective criterion, which completely neglects the finite-time evolution of the controlled process. Our main interest in this paper is to study the relation between the notions of AC optimality andstrong average cost (SAC) optimality. The latter criterion is introduced to asses the performance of a policy over long but finite horizons, as well as in the long-run average sense. We show that for bounded one-stage cost functions, Sennott's conditions are sufficient to guarantee thatevery AC optimal policy is also SAC optimal. On the other hand, a detailed counterexample is given that shows that the latter result does not extend to the case of unbounded cost functions. In this counterexample, Sennott's conditions are verified and a policy is exhibited that is both average and Blackwell optimal and satisfies the average cost inequality.     

Nonturnpike Optimal Solutions and Their Approximations in Infinite Horizon

Recently, the authors have proposed a new necessary and sufficient condition for turnpike optimality in calculus of variations with singular Euler equation. The method is based on a characterization of the value function and generalizes the well known method based on the Green theorem. Furthermore, it allows the optimality of a competition between several turnpikes to be characterized. For a class of such problems not enjoying the turnpike property, we give an explicit formula for the value function and show how to characterize the optimal solution as the limiting solution of a family of perturbed problems satisfying the turnpike property. The considered problems are scalar with infinite horizon.     

Optimal threshold probability and expectation in semi-Markov decision processes

We consider undiscounted semi-Markov decision process with a target set and our main concern is a problem minimizing threshold probability. We formulate the problem as an infinite horizon case with a recurrent class. We show that an optimal value function is a unique solution to an optimality equation and there exists a stationary optimal policy. Also several value iteration methods and a policy improvement method are given in our model. Furthermore, we investigate a relationship between threshold probabilities and expectations for total rewards.     

Optimal policies for inventory systems with discretionary sales, random yield and lost sales

We determine replenishment and sales decisions jointly for an inventory system with random demand, lost sales and random yield. Demands in consecutive periods are independent random variables and their distributions are known. We incorporate discretionary sales, when inventory may be set aside to satisfy future demand even if some present demand may be lost. Our objective is to minimize the total discounted cost over the problem horizon by choosing an optimal replenishment and discretionary sales policy. We obtain the structure of the optimal replenishment and discretionary sales policy and show that the optimal policy for finite horizon problem converges to that of the infinite horizon problem. Moreover, we compare the optimal policy under random yield with that under certain yield, and show that the optimal order quantity （sales quantity） under random yield is more （less） than that under certain yield.     

Stochastic Multiproduct Inventory Models with Limited Storage

This paper studies multiproduct inventory models with stochastic demands and a warehousing constraint. Finite horizon as well as stationary and nonstationary discounted-cost infinite-horizon problems are addressed. Existence of optimal feedback policies is established under fairly general assumptions. Furthermore, the structure of the optimal policies is analyzed when the ordering cost is linear and the inventory/backlog cost is convex. The optimal policies generalize the base-stock policies in the single-product case. Finally, in the stationary infinite-horizon case, a myopic policy is proved to be optimal if the product demands are independent and the cost functions are separable.