Asymptotically optimal production policies in dynamic stochastic jobshops with limited buffers

We consider a production planning problem for a jobshop with unreliable machines producing a number of products. There are upper and lower bounds on intermediate parts and an upper bound on finished parts. The machine capacities are modelled as finite state Markov chains. The objective is to choose the rate of production so as to minimize the total discounted cost of inventory and production. Finding an optimal control policy for this problem is difficult. Instead, we derive an asymptotic approximation by letting the rates of change of the machine states approach infinity. The asymptotic analysis leads to a limiting problem in which the stochastic machine capacities are replaced by their equilibrium mean capacities. The value function for the original problem is shown to converge to the value function of the limiting problem. The convergence rate of the value function together with the error estimate for the constructed asymptotic optimal production policies are established.     

Hierarchical Production Control in Dynamic Stochastic Jobshops with Long-Run Average Cost

We consider a production planning problem for a dynamic jobshop producing a number of products and subject to breakdown and repair of machines. The machine capacities are assumed to be finite-state Markov chains. As the rates of change of the machine states approach infinity, an asymptotic analysis of this stochastic manufacturing systems is given. The analysis results in a limiting problem in which the stochastic machine availability is replaced by its equilibrium mean availability. The long-run average cost for the original problem is shown to converge to the long-run average cost of the limiting problem. The convergence rate of the long-run average cost for the original problem to that of the limiting problem together with an error estimate for the constructed asymptotic optimal control is established.     

Near-Optimal Controls of Discrete-Time Dynamic Systems Driven by Singularly-Perturbed Markov Chains

This work is devoted to near-optimal controls of large-scale discrete-time nonlinear dynamic systems driven by Markov chains; the underlying problem is to minimize an expected cost function. Our main goal is to reduce the complexity of the underlying systems. To achieve this goal, discrete-time control models under singularly-perturbed Markov chains are introduced. Using a relaxed control representation, our effort is devoted to finding near-optimal controls. Lumping the states in each irreducible class into a single state gives rise to a limit system. Applying near-optimal controls of the limit system to the original system, near-optimal controls of the original system are derived.     

An averaging theorem for two-point boundary value problems with applications to optimal control

An asymptotic result is obtained for a two-point boundary value problem for a vector system of nonlinear ordinary differential equations involving “fast” and “slow” inputs. The asymptotically limiting system is obtained by an averaging procedure. Using this result, an approximate analysis of the original system may be carried out by considering two lower-order systems each involving only one time scale. It is shown that some optimal control problems for systems with multiple time scales may be analyzed by this method.     

Hierarchical production policies in stochastic two-machine flowshops with finite buffers

This paper concerns production planning in manufacturing systems with two unreliable machines in tandem. The problem is formulated as a stochastic control problem in which the objective is to minimize the expected total cost of production, inventories, and backlogs. Since the sizes of the internal and external buffers are finite, the problem is one with state constraints. As the optimal solutions to this problem are extremely difficult to obtain due to the uncertainty in machine capacities as well as the presence of state constraints, a deterministic limting problem in which the stochastic machine capacities are replaced by their mean capacities is considered instead. The weak Lipschitz property of the value functions for the original and limiting problems is introduced and proved; a constraint domain approximation approach is developed to show that the value function of the original problem converges to that of the limiting problem as the rate of change in machine states approaches infinity. Asymptotic optimal production policies for the orginal problem are constructed explicity from the near-optimal policies of the limiting problem, and the error estimate for the policies constructed is obtained. Algorithms for constructing these policies are presented.This work was partly supported by CUHK Direct Grant 220500660, RGC Earmarked Grant CUHK 249/94E, and RGC Earmarked Grant CUHK 489/95E.     

Ergodic Control of Partially Degenerate Diffusions in a Compact Domain

The problem of ergodic control of a reflecting diffusion in a compact domain is analysed under the condition of partial degeneracy, i.e. when its transition kernel after some time is absolutely continuous with respect to the Lebesgue measure on a part of the state space. Existence of a value function and a “martingale dynamic programming principle” are established by mapping the problem to a discrete time control problem. Implications for existence of optimal controls are derived.     

Asymptotic solution of a singularly perturbed optimal control problem related to the reconstruction of a defective curve

The introduction of “cheap” controls for minimizing the simplest energy functional in an optimal control problem related to the reconstruction of a defective curve necessitates solving a singularly perturbed variational problem with fixed time and fixed ends. The construction of a uniform zero asymptotic approximation to the optimal control in the latter problem permits one to conclude that the optimal trajectories in the original optimal control problem combine a uniform motion in the interior of the time interval with rapid transition layers at the boundaries of the control interval.     

Controlled Markov Chains with Weak and Strong Interactions: Asymptotic Optimality and Applications to Manufacturing

This paper deals with the asymptotic optimality of a stochastic dynamic system driven by a singularly perturbed Markov chain with finite state space. The states of the Markov chain belong to several groups such that transitions among the states within each group occur much more frequently than transitions among the states in different groups. Aggregating the states of the Markov chain leads to a limit control problem, which is obtained by replacing the states in each group by the corresponding average distribution. The limit control problem is simpler to solve as compared with the original one. A nearly-optimal solution for the original problem is constructed by using the optimal solution to the limit problem. To demonstrate, the suggested approach of asymptotic optimal control is applied to examples of manufacturing systems of production planning.     

Discretisation of stochastic control problems for continuous time dynamics with delay

As a main step in the numerical solution of control problems in continuous time, the controlled process is approximated by sequences of controlled Markov chains, thus discretising time and space. A new feature in this context is to allow for delay in the dynamics. The existence of an optimal strategy with respect to the cost functional can be guaranteed in the class of relaxed controls. Weak convergence of the approximating extended Markov chains to the original process together with convergence of the associated optimal strategies is established.     

Lower Bounds on the Generalized Central Moments of the Optimal Alignments Score of Random Sequences

We present a general approach to the problem of determining tight asymptotic lower bounds for generalized central moments of the optimal alignment score of two independent sequences of i.i.d. random variables. At first, these are obtained under a main assumption for which sufficient conditions are provided. When the main assumption fails, we nevertheless develop a “uniform approximation” method leading to asymptotic lower bounds. Our general results are then applied to the length of the longest common subsequences of binary strings, in which case asymptotic lower bounds are obtained for the moments and the exponential moments of the optimal score. As a by-product, a local upper bound on the rate function associated with the length of the longest common subsequences of two binary strings is also obtained.     

document

This work develops asymptotically optimal controls for discrete-time singularly perturbed Markov decision processes (MDPs) having weak and strong interactions. The focus is on finite-state-space-MDP problems. The state space of the underlying Markov chain can be decomposed into a number of recurrent classes or a number of recurrent classes and a group of transient states. Using a hierarchical control approach, continuous-time limit problems that are much simpler to handle than the original ones are derived. Based on the optimal solutions for the limit problems, nearly optimal decisions for the original problems are obtained. The asymptotic optimality of such controls is proved and the rate of convergence is provided. Infinite horizon problems are considered; both discounted costs and long-run average costs are examined.     

Stochastic recursive inclusions with non-additive iterate-dependent Markov noise

In this paper we study the asymptotic behaviour of stochastic approximation schemes with set-valued drift function and non-additive iterate-dependent Markov noise. We show that a linearly interpolated trajectory of such a recursion is an asymptotic pseudotrajectory for the flow of a limiting differential inclusion obtained by averaging the set-valued drift function of the recursion w.r.t. the stationary distributions of the Markov noise. The limit set theorem by Benaim is then used to characterize the limit sets of the recursion in terms of the dynamics of the limiting differential inclusion. We then state two variants of the Markov noise assumption under which the analysis of the recursion is similar to the one presented in this paper. Scenarios where our recursion naturally appears are presented as applications. These include controlled stochastic approximation, subgradient descent, approximate drift problem and analysis of discontinuous dynamics all in the presence of non-additive iterate-dependent Markov noise.     

Asymptotic variance rate of the output of a transfer line with no buffer storage and cycle-dependent failures

In this study the variability properties of the output of transfer lines are investigated. The asymptotic variance rate of the output of an N-station synchronous transfer line with no interstation buffers and cycle-dependent failures is analytically determined. Unlike the other studies, the analytical method presented in this study yields a closed-form expression for the asymptotic variance rate of the output. The method is based on a general result derived for irreducible recurrent Markov chains. Namely, the limiting variance of the number of visits to a state of an irreducible recurrent Markov chain is obtained from the n-step transition probability function. Thus, the same method can be used in other applications where the limiting variance of the number of visits to a state of an irreducible recurrent Markov chain is of interest. Numerical results show that the asymptotic variance rate of the output does not monotonically increase as the number of stations in the transfer line increases. The asymptotic variance rate of the output may first increase and then decrease depending on the station parameters. This property of the production rate is investigated through numerical experiments and the results are presented.     

Identification of Regeneration Times in MCMC Simulation,With Application to Adaptive Schemes

Regeneration is a useful tool in Markov chain Monte Carlo simulation because it can be used to side-step the burn-in problem and to construct better estimates of the variance of parameter estimates themselves. It also provides a simple way to introduce adaptive behavior into a Markov chain, and to use parallel processors to build a single chain. Regeneration is often difficult to take advantage of because, for most chains, no recurrent proper atom exists, and it is not always easy to use Nummelin's splitting method to identify regeneration times. This article describes a constructive method for generating a Markov chain with a specified target distribution and identifying regeneration times. As a special case of the method, an algorithm which can be “wrapped” around an existing Markov transition kernel is given. In addition, a specific rule for adapting the transition kernel at regeneration times is introduced, which gradually replaces the original transition kernel with an independence-sampling Metropolis-Hastings kernel using a mixture normal approximation to the target density as its proposal density. Computational gains for the regenerative adaptive algorithm are demonstrated in examples.     

Asymptotic expansions of backward equations for two-time-scale Markov chains in continuous time

This work develops asymptotic expansions for solutions of systems of backward equations of time- inhomogeneous Maxkov chains in continuous time. Owing to the rapid progress in technology and the increasing complexity in modeling, the underlying Maxkov chains often have large state spaces, which make the computa- tional tasks ihfeasible. To reduce the complexity, two-time-scale formulations are used. By introducing a small parameter ε〉 0 and using suitable decomposition and aggregation procedures, it is formulated as a singular perturbation problem. Both Markov chains having recurrent states only and Maxkov chains including also tran- sient states are treated. Under certain weak irreducibility and smoothness conditions of the generators, the desired asymptotic expansions axe constructed. Then error bounds are obtained.     

The birth of the giant component

Limiting distributions are derived for the sparse connected components that are present when a random graph on n vertices has approximately 1/2n edges. In particular, we show that such a graph consists entirely of trees, unicyclic components, and bicyclic components with probability approaching √2/3 cosh √5/18 ≈ 0.9325 as n→∞. The limiting probability that it is consists of trees, unicyclic components, and at most one another component is approximately 0.9957; the limiting probability that it is planar lies between 0.987 and 0.9998. When a random graph evolves and the number of edges passes 1/2n, its components grow in cyclic complexity according to an interesting Markov process whose asymptotic structure is derived. The probability that there never is more than a single component with more edges than vertices, throughout the veolution, approaches 5 π/18 ≈ 0.8727. A “uniform” model of random graphs, which allows self-loops and multiple edges, is shown to lead to formulas that are substanitially simpler than the analogous formulas for the classical random graphs of Erdõs and Rényi. The notions of “excess” and “deficiency,” which are significant characteristics of the generating function as well as of the graphs themselves, lead to a mathematically attractive structural theory for the uniform model. A general approach to the study of stopping configurations makes it possible to sharpen previously obtained estimates in a uniform manner and often to obtain closed forms for the constants of interest. Empirical results are presented to complement the analysis, indicating the typical behavior when n is near 2oooO. © 1993 John Wiley & Sons, Inc.     

On the convergence of 2-dimensional markov chains in quadrants with boundary reflection

The aim of this paper is to prove a limit theorem on the weak convergence of a family of rescaled Markov chains in a quadrant with boundary reflection. The limiting process is specified in terms of solutions of a certain submartingale problem in the style used by Varadhan and Williams. The obtained result is then applied to the problem of approximating an arbitrary Brownian motion with oblique reflection in a wedge by a family of Markov chains.     

Singularly Perturbed Markov Chains with Two Small Parameters: A Matched Asymptotic Expansion

This work is concerned with asymptotic properties of solutions to forward equations for singularly perturbed Markov chains with two small parameters. It is motivated by the model of a cost-minimizing firm involving production planning and capacity expansion and a two-level hierarchical decomposition. Our effort focuses on obtaining asymptotic expansions of the solutions to the forward equation. Different from previous work on singularly perturbed Markov chains, the inner expansion terms are constructed by solving certain partial differential equations. The methods of undetermined coefficients are used. The error bound is obtained.     

Optimal control of piecewise deterministic markov process

The trajectories of piecewise deterministic Markov processes are solutions of an ordinary (vector)differential equation with possible random jumps between the different integral curves. Both continuous deterministic motion and the random jumps of the processes are controlled in order to minimize the expected value of a performance functional consisting of continuous, jump and terminal costs. A limiting form of the Hamilton-Jacobi-Bellman partial differential equation is shown to be a necessary and sufficient optimality condition. The existence of an optimal strategy is proved and acharacterization of the value function as supremum of smooth subsolutions is also given. The approach consists of imbedding the original control problem tightly in a convex mathematical programming problem on the space of measures and then solving the latter by dualit     

Weak lumpability in the k-SAT problem

In this note, we discuss a Markov chain formulation of the k-SAT problem and the properties of the resulting transition matrix. The motivation behind this work is to relate the phase transition in the k-SAT problem to the phenomenon of “cut-off” in Markov chains. We use the idea of weak-lumpability to reduce the dimension of our transition matrix to manageable proportions.