A new method for parameter sensitivity estimation in structural reliability analysis

For the parameter sensitivity estimation with implicit limit state functions in the time-invariant reliability analysis, the common Monte Carlo simulation based approach involves multiple trials for each parameter being varied, which will increase associated computational cost and the cost may become inevitably high especially when many random variables are involved. Another effective approach for this problem is featured as constructing the equivalent limit state function (usually called response surface) and performing the estimation in FORM/SORM. However, as the equivalent limit state function is polynomial in the traditional response surface method, it is not a good approximation especially for some highly non-linear limit state functions. To solve the above two problems, a new method, support vector regression based response surface method, is therefore presented in this paper. The support vector regression algorithm is employed to construct the equivalent limit state function and FORM/SORM is used in the parameter sensitivity estimation, and then two illustrative examples are given. It is shown that the computational cost of the sensitivity estimation can be greatly reduced and the accuracy can be retained, and results of the sensitivity estimation obtained by the proposed method are in satisfactory agreement with those computed by the conventional Monte Carlo methods.     

Risk-sensitive average continuous-time Markov decision processes with unbounded rates

Abstract

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.     

A new approach for solving repair limit problems

The usual approach to finding optimal repair limits on failure of a component is to use a finite state approximation Markov Decision Process (MDP). In this paper an alternative approach is introduced. Assuming a stochastically increasing repair cost, the optimum solution is shown to satisfy a certain two-point boundary condition, first order differential equation. An asymptotic formula for the optimal repair limit function is derived. Numerical solutions are obtained for some Weibull and Special Erlang distributed time to failure distributions. The structural form of the repair limit function results in a solution procedure which is several orders of magnitude faster than is achievable using previous methods.     

Denumerable controlled Markov chains with strong average optimality criterion: Bounded & unbounded costs

This paper studies discrete-time nonlinear controlled stochastic systems, modeled by controlled Markov chains (CMC) with denumerable state space and compact action space, and with an infinite planning horizon. Recently, there has been a renewed interest in CMC with a long-run, expected average cost (AC) optimality criterion. A classical approach to study average optimality consists in formulating the AC case as a limit of the discounted cost (DC) case, as the discount factor increases to 1, i.e., as the discounting effectvanishes. This approach has been rekindled in recent years, with the introduction by Sennott and others of conditions under which AC optimal stationary policies are shown to exist. However, AC optimality is a rather underselective criterion, which completely neglects the finite-time evolution of the controlled process. Our main interest in this paper is to study the relation between the notions of AC optimality andstrong average cost (SAC) optimality. The latter criterion is introduced to asses the performance of a policy over long but finite horizons, as well as in the long-run average sense. We show that for bounded one-stage cost functions, Sennott's conditions are sufficient to guarantee thatevery AC optimal policy is also SAC optimal. On the other hand, a detailed counterexample is given that shows that the latter result does not extend to the case of unbounded cost functions. In this counterexample, Sennott's conditions are verified and a policy is exhibited that is both average and Blackwell optimal and satisfies the average cost inequality.     

Partially observed semi-Markov zero-sum games with average payoff

We study a zero-sum partially observed semi-Markov game with average payoff on a countable state space. Under certain ergodicity conditions we show that a saddle point equilibrium exists. We achieve this by solving the corresponding average cost optimality equation using a span contraction method. The average value is shown to be the unique zero of a Lipschitz continuous function. A value iteration scheme is developed to compute the value.     

Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.     

Optimal Production Planning in a Stochastic Manufacturing System with Long-Run Average Cost

This paper is concerned with the optimal production planning in a dynamic stochastic manufacturing system consisting of a single machine that is failure prone and facing a constant demand. The objective is to choose the rate of production over time in order to minimize the long-run average cost of production and surplus. The analysis proceeds with a study of the corresponding problem with a discounted cost. It is shown using the vanishing discount approach that the Hamilton–Jacobi–Bellman equation for the average cost problem has a solution giving rise to the minimal average cost and the so-called potential function. The result helps in establishing a verification theorem. Finally, the optimal control policy is specified in terms of the potential function.     

Constrained markov decision processes with compact state and action spaces: the average case

Constrained Markov decision processes with compact state and action spaces are studied under long-run average reward or cost criteria. By introducing a corresponding Lagrange function, a saddle-point theorem is given, by which the existence of a constrained optimal pair of initial state distribution and policy is shown. Also, under the hypothesis of Doeblin, the functional characterization of a constrained optimal policy is obtained     

Optimality equations and inequalities in a class of risk-sensitive average cost Markov decision chains

This note concerns controlled Markov chains on a denumerable sate space. The performance of a control policy is measured by the risk-sensitive average criterion, and it is assumed that (a) the simultaneous Doeblin condition holds, and (b) the system is communicating under the action of each stationary policy. If the cost function is bounded below, it is established that the optimal average cost is characterized by an optimality inequality, and it is to shown that, even for bounded costs, such an inequality may be strict at every state. Also, for a nonnegative cost function with compact support, the existence an uniqueness of bounded solutions of the optimality equation is proved, and an example is provided to show that such a conclusion generally fails when the cost is negative at some state.     

Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria

This work is concerned with controlled Markov chains with finite state and action spaces. It is assumed that the decision maker has an arbitrary but constant risk sensitivity coefficient, and that the performance of a control policy is measured by the long-run average cost criterion. Within this framework, the existence of solutions of the corresponding risk-sensitive optimality equation for arbitrary cost function is characterized in terms of communication properties of the transition law.     

Existence of Risk-Sensitive Optimal Stationary Policies for Controlled Markov Processes

In this paper we are concerned with the existence of optimal stationary policies for infinite-horizon risk-sensitive Markov control processes with denumerable state space, unbounded cost function, and long-run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk-sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy. Accepted 1 October 1997     

Convexity and characterization of optimal policies in a dynamic routing problem

An infinite horizon, expected average cost, dynamic routing problem is formulated for a simple failure-prone queueing system, modelled as a continuous time, continuous state controlled stochastic process. We prove that the optimal average cost is independent of the initial state and that the cost-to-go functions of dynamic programming are convex. These results, together with a set of optimality conditions, lead to the conclusion that optimal policies are switching policies, characterized by a set of switching curves (or regions), each curve corresponding to a particular state of the nodes (servers).     

GUARANTEED COST CONTROL OF CONTINUOUS-TIME UNCERTAIN SINGULAR SYSTEM WITH BOTH STATE AND INPUT DELAYS

The guaranteed cost control problem for a continuous-time uncertain singular system with state and control delays, and a given quadratic cost function is studied in this paper. Sufficient conditions for the existence of the guaranteed cost controller are derived based on the linear inequality (LMI) approach. A parameterized characterization of the guaranteed cost laws is given in terms of the feasible solutions to a certain LMI, and the cost function of guaranteed cost controller exists an upper bound.     

Controlled Markov Chains with Weak and Strong Interactions: Asymptotic Optimality and Applications to Manufacturing

This paper deals with the asymptotic optimality of a stochastic dynamic system driven by a singularly perturbed Markov chain with finite state space. The states of the Markov chain belong to several groups such that transitions among the states within each group occur much more frequently than transitions among the states in different groups. Aggregating the states of the Markov chain leads to a limit control problem, which is obtained by replacing the states in each group by the corresponding average distribution. The limit control problem is simpler to solve as compared with the original one. A nearly-optimal solution for the original problem is constructed by using the optimal solution to the limit problem. To demonstrate, the suggested approach of asymptotic optimal control is applied to examples of manufacturing systems of production planning.     

Optimal control for parallel queues with a single batch server

We consider a queueing system with multiple Poisson arrival queues and a single batch server that has infinite capacity and a fixed service time. The problem is to allocate the server at each moment to minimize the long-run average waiting cost. We propose a Cost-Arrival Weighted (CAW) policy for this problem based on the structure of the optimal policy of a corresponding fluid model. We show that this simple policy enjoys a superior performance by numerical experiments.     

Intuitionistic fuzzy optimization technique for Pareto optimal solution of manufacturing inventory models with shortages

This paper discusses a manufacturing inventory model with shortages where carrying cost, shortage cost, setup cost and demand quantity are considered as fuzzy numbers. The fuzzy parameters are transformed into corresponding interval numbers and then the interval objective function has been transformed into a classical multi-objective EPQ (economic production quantity) problem. To minimize the interval objective function, the order relation that represents the decision maker’s preference between interval objective functions has been defined by the right limit, left limit, center and half width of an interval. Finally, the transformed problem has been solved by intuitionistic fuzzy programming technique. The proposed method is illustrated with a numerical example and Pareto optimality test has been applied as well.     

Exact formulation of stochastic EMQ model for an unreliable production system

The purpose of this paper is to present an exact formulation of stochastic EMQ model for an unreliable production system under a general framework in which the time to machine failure, corrective (emergency) and preventive (regular) repair times are assumed to be random variables. For exact financial implications of the lot-sizing decisions, the EMQ model is formulated based on the net present value (NPV) approach. Then, by taking limitation on the discount rate, the traditional long-run average cost model is obtained. The criteria for the existence and uniqueness of the optimal production time in both the models are derived under general failure and specific repair time distributions. Numerical examples are devoted to find the optimal production policies of the developed models and examine the sensitivity of the parameters involved. Computational results show that the optimal decision based on the NPV approach is superior to that based on the long-run average cost approach, though the performance level strongly depends on the pertinent failure and repair time distributions.     

Asymptotic properties of constrained Markov Decision Processes

We present in this paper several asymptotic properties of constrained Markov Decision Processes (MDPs) with a countable state space. We treat both the discounted and the expected average cost, with unbounded cost. We are interested in (1) the convergence of finite horizon MDPs to the infinite horizon MDP, (2) convergence of MDPs with a truncated state space to the problem with infinite state space, (3) convergence of MDPs as the discount factor goes to a limit. In all these cases we establish the convergence of optimal values and policies. Moreover, based on the optimal policy for the limiting problem, we construct policies which are almost optimal for the other (approximating) problems. Based on the convergence of MDPs with a truncated state space to the problem with infinite state space, we show that an optimal stationary policy exists such that the number of randomisations it uses is less or equal to the number of constraints plus one. We finally apply the results to a dynamic scheduling problem.This work was partially supported by the Chateaubriand fellowship from the French embassy in Israel and by the European Grant BRA-QMIPS of CEC DG XIII     

Temporal Difference-Based Policy Iteration for Optimal Control of Stochastic Systems

In this paper, a unified policy iteration approach is presented for the optimal control problem of stochastic system with discounted average cost and continuous state space. The approach consists of temporal difference learning-based potential function approximation algorithms and performance difference formula-based policy improvement. The approximation algorithms are derived by solving the Poisson equation-based fixed-point equation, which can be viewed as continuous versions of least squares policy evaluation algorithm and least squares temporal difference algorithm. The simulations are provided to illustrate the effectiveness of the approach.     

Controlled one dimensional diffusions with switching costs—average cost criterion

This paper deals with a one-dimensional controlled diffusion process on a compact interval with reflecting boundaries. The set of available actions is finite and the action can be changed only at countably many stopping times. The cost structure includes both a continuous movement cost rate depending on the state and the action, and a switching cost when the action is changed. The policies are evaluated with respect to the average cost criterion. The problem is solved by looking at, for each stationary policy, an embedded stochastic process corresponding to the state intervals visited in the sequence of switching times. The communicating classes of this process are classified into closed and transient groups and a method of calculating the average cost for the closed and transient classes is given. Also given are conditions to guarantee the optimality of a stationary policy. A Brownian motion control problem with quadratic cost is worked out in detail and the form of an optimal policy is established.