Monotonicity properties of a class of stochastic inventory systems

We consider inventory systems which are governed by an (r,q) or (r,nq) policy. We derive general conditions for monotonicity of the three optimal policy parameters, i.e., the optimal reorder level, order quantity and order-up-to level, as well as the optimal cost value, as a function of the various model primitives, be it cost parameters or complete cost rate functions or characteristics of the demand and leadtime processes. These results are obtained as corollaries from a few general theorems, with separate treatment given to the case where the policy parameters are continuous variables and that where they need to be restricted to integer values. The results are applied both to standard inventory models and to those with general shelf age and delay dependent inventory costs.     

Contraction mappings underlying undiscounted Markov decision problems

This paper is concerned with the properties of the value-iteration operator0 which arises in undiscounted Markov decision problems. We give both necessary and sufficient conditions for this operator to reduce to a contraction operator, in which case it is easy to show that the value-iteration method exhibits a uniform geometric convergence rate. As necessary conditions we obtain a number of important characterizations of the chain and periodicity structures of the problem, and as sufficient conditions, we give a general “scrambling-type” recurrency condition, which encompasses a number of important special cases. Next, we show that a data transformation turns every unichained undiscounted Markov Renewal Program into an equivalent undiscounted Markov decision problem, in which the value-iteration operator is contracting, because it satisfies this “scrambling-type” condition. We exploit this contraction property in order to obtain lower and upper bounds as well as variational characterizations for the fixed point of the optimality equation and a test for eliminating suboptimal actions.     

Nonstationary Markov decision problems with converging parameters

This paper considers the solution of Markov decision problems whose parameters can be obtained only via approximating schemes, or where it is computationally preferable to approximate the parameters, rather than employing exact algorithms for their computation.Various models are presented in which this situation occurs. Furthermore, it is shown that a modified value-iteration method may be employed, both for the discounted version and for the undiscounted version of the model, in order to solve the optimality equation and to find optimal policies. In both cases, the convergence rate is determined.As a side result, we characterize the asymptotic behavior of backward products of a geometrically convergent sequence of Markov matrices.     

Denumerable state semi-Markov decision processes with unbounded costs,average cost criterion

This paper establishes a rather complete optimality theory for the average cost semi-Markov decision model with a denumerable state space, compact metric action sets and unbounded one-step costs for the case where the underlying Markov chains have a single ergotic set. Under a condition which, roughly speaking, requires the existence of a finite set such that the supremum over all stationary policies of the expected time and the total expected absolute cost incurred until the first return to this set are finite for any starting state, we shall verify the existence of a finite solution to the average costs optimality equation and the existence of an average cost optimal stationary policy.     

Approximating queue size and waiting time distributions in general polling systems

Polling system models are extensively used to model a large variety of computer and communication networks as well as production and service systems in which multiple customer classes or a number of distinct items compete for the capacity of a common server or production facility. In this paper we describe an efficient approximation method for the steady state distributions of the queue sizes and waiting times. This method is highly accurate as demonstrated by an extensive numerical study. In addition, it is highly adaptable to a variety of arrival patterns and switching protocols, including exhaustive and gated regimes, simple cyclical systems as well as general polling tables. For a system withN stations, one finds the firstK probability density function values of the steady state queue size in any given station inO(max(N, K ²) time only. When executed on an IBM system RS/6000, we have observed an average CPU time of less than 1 second for systems with as many as 50 stations over a large variety of parameter settings.     

Foolproof convergence in multichain Policy Iteration

An example for undiscounted multichain Markov Renewal Programming shows that policies may exist such that the Policy Iteration Algorithm (PIA) can converge to these policies for some (but not all) choices of the additive constants in the relative values, and as a consequence that the PIA may cycle if the relative values are improperly determined. A class of rules for choosing the additive constants is given sufficient to guarantee the convergence of the PIA, as well as necessary and sufficient conditions for a policy to have the property that the PIA can converge to it for any relative value vector. Finally we give some properties of the policies that exhibit this foolproof convergence.     

Solution techniques for some allocation problems

This paper presents methods for solving allocation problems that can be stated as convex knapsack problems with generalized upper bounds. Such bounds may express upper limits on the total amount allocated to each of several subsets of activities. In addition our model arises as a subproblem in more complex mathematical programs. We therefore emphasize efficient procedures to recover optimality when minor changes in the parameters occur from one problem instance to the next. These considerations lead us to propose novel data structures for such problems. Also, we introduce an approximation method to solve certain equations, which arise during the procedures.     

On the functional equations in undiscounted and sensitive discounted stochastic games

This paper considers two-person zero-sum sequential games with finite state and action spaces. We consider the pair of functional equations (f.e.) that arises in the undiscounted infinite stage model, and show that a certain class of successive approximation schemes is guaranteed to converge to a solution pair whenever an equilibrium policy with respect to the average return per unit time criterion (AEP) exists. Existence of the latter thus implies the existence of a solution to this pair of f.e. whereas the converse implication is shown only to hold under special circumstances.In addition to this pair of f.e., a complete sequence of f.e. has to be considered when analyzing more sensitive optimality criteria that make further selections within the class of AEPs. A number of characterizations and interdependences between the existence of solutions to the f.e. and existence of stationary sensitive optimal equilibrium policies are obtained.

---

Zusammenfassung Die Arbeit behandelt sequentielle Zweipersonen-Nullsummenspiele mit endlichem Zustands- und endlichem Aktionenraum. Es wird das Paar von Funktionalgleichungen für das unendlich-stufige Modell ohne Diskontierung betrachtet und gezeigt, daß eine gewisse Klasse von sukzessiven Approximationen gegen ein Lösungspaar konvergiert, wenn eine Gleichgewichtspolitik für den Fall existiert, daß als Kriterium die durchschnittliche Auszahlung pro Zeiteinheit gewählt wird. Werden empfindlichere Optimalitätskriterien betrachtet, so muß zusätzlich zu dem obigen Funktionalgleichungspaar eine ganze Folge von Funktionalgleichungen untersucht werden. Weiter werden Resultate über die Existenz von Lösungen der Funktionalgleichungen und die damit zusammenhängende Existenz stationärer optimaler Gleichgewichtspolitiken hergeleitet.