Dynamic event-triggered control for discrete-time nonlinear Markov jump systems using policy iteration-based adaptive dynamic programming

This paper investigates a dynamic event-triggered optimal control problem of discrete-time (DT) nonlinear Markov jump systems (MJSs) via exploring policy iteration (PI) adaptive dynamic programming (ADP) algorithms. The performance index function (PIF) defined in each subsystem is updated by utilizing an online PI algorithm, and the corresponding control policy is derived via solving the optimal PIF. Then, we adopt neural network (NN) techniques, including an actor network and a critic network, to estimate the iterative PIF and control policy. Moreover, the designed dynamic event-triggered mechanism (DETM) is employed to avoid wasting additional resources when the estimated iterative control policy is updated. Finally, based on the Lyapunov difference method, it is proved that the system stability and the convergence of all signals can be guaranteed under the developed control scheme. A simulation example for DT nonlinear MJSs with two system modes is presented to demonstrate the feasibility of the control design scheme.     

货币政策多目标协调和控制研究

货币政策多目标协调发展是经济平稳运行的重要保证。基于经济控制论的理论和方法,建立货币政策多目标系统仿真模型,通过检验货币政策多目标系统的能观测性、能控性和稳定性等控制特性,验证了我国货币政策多目标的有效性,并且发现货币政策最终目标之间存在可协调性,为研究货币政策多目标协调发展的策略提供了理论基础。运用Simulink建立货币政策多目标协调发展控制模型,得出要保证货币政策最终目标的协调发展需要进一步降准降息的结论,为当前货币政策实施的科学性提供了理论支撑。     

Economic off-line quality control strategy with two types of inspection errors

A commonly used quality control method is to inspect products to identify their quality and to perform the related disposition of acceptance, salvage or rejection based on the findings. While the issue of finding the most economical inspection/disposition policy has been studied for a batch of units produced from an unreliable system, previous studies assumed the inspections to be perfect. In this study, we further extend the inspection/disposition model to consider two types of inspection errors in order to facilitate the adaptation of this economic inspection/disposition model to real world applications. We first describe an inspection/disposition policy for the two types of inspection errors and then obtain the related mathematical formulae. An algorithm is presented for determining the optimal inspection/disposition policy. Finally, numerical examples are given to illustrate the effect of inspection errors on the optimal inspection/disposition policy under the following three quality control policies: cost minimizing, zero-defects and perfect information policy.     

Optimal replacement policies for two-unit machines with increasing running costs 1

A machine consists of two stochastically failing units. Failure of either of the units causes a failure of the machine and the failed unit has to be replaced immediately. Associated with the units are running costs which increase with the age of the unit because of increasing maintenance costs, decreasing output, etc.A preventive replacement policy is proposed under which, at failure points, we also replace the second unit if its age exceeds a predetermined control limit. It is proved that, for two identical units with exponential life-time distributions and linear running costs, this policy is optimal and the optimal control limit is calculated. In an additional model we take into consideration the length of time it takes to replace one unit or both units.The method of solution is a variation of dynamic semi-Markov programming. Analytical results are obtained and the influence of the various parameters on them is investigated. Finally, we study the saving due to our policy in comparison with a policy in which only failed units are replaced.     

Comparison among three pull control policies: kanban, base stock, and generalized kanban

This paper is concerned with maketostock pull control policies. A classical policy is the kanban policy. Another policy, very easy to implement, is the base stock policy. These two policies contain one design parameter per stage. A general control policy, known as the generalized kanban policy, can also be used to implement the pull mechanism. The generalized kanban policy includes, as special cases, the kanban and the base stock policies. This policy uses two parameters for each stage of the production system. The aim of this paper is to provide qualitative and quantitative comparisons of these three policies. The results of our study will help to choose the policy to implement in order to control a production system. We give practical rules. We also show that if there is no delay in filling orders, all three policies have similar costs. However, for the systems studied, we show that, if there is a delay in filling orders, generalized kanban systems and base stock systems yield close to optimal costs, which are lower than costs of kanban systems for the same service quality.     

Scheduling networks of queues: Heavy traffic analysis of a simple open network

We consider a queueing network with two single-server stations and two types of customers. Customers of type A require service only at station 1 and customers of type B require service first at station 1 and then at station 2. Each server has a different general service time distribution, and each customer type has a different general interarrival time distribution. The problem is to find a dynamic sequencing policy at station 1 that minimizes the long-run average expected number of customers in the system.The scheduling problem is approximated by a dynamic control problem involving Brownian motion. A reformulation of this control problem is solved, and the solution is interpreted in terms of the queueing system in order to obtain an effective sequencing policy. Also, a pathwise lower bound (for any sequencing policy) is obtained for the total number of customers in the network. We show via simulation that the relative difference between the performance of the proposed policy and the pathwise lower bound becomes small as the load on the network is increased toward the heavy traffic limit.     

A maxmin policy for bond management

A simple immunization problem is formulated as a maxmin optimal control problem and analytically solved by means of dynamic programming. The optimal control law, namely the immunization policy, turns out to be quite different from any duration-based immunization policy. Moreover, it is seemingly able to discriminate between favourable and unfavourable changes in the yield curve.     

Impact of RFID technology on inventory control policy

RFID application can improve operation performance in a supply chain by reducing or eliminating inventory misplacement and shrinkage. In this paper, we present a periodic review inventory model to investigate and characterize the multiperiod inventory control policies in both non-RFID and RFID cases when the firm encounters misplacement and shrinkage. The optimal inventory control policy is proved to be a two-control limit policy. The control limits in both the non-RFID case and the RFID case are analyzed and examined, while considering the impact of shrinkage and misplacement on inventory policies. A critical inventory level is determined to identify the relationship of higher inventory level control limits between the RFID case and the non-RFID case. An intensive numerical study with sensitivity analysis of selling price, misplacement rate, shrinkage rate, inventory recovery rate, and tag price is conducted. We find that when RFID technology is adopted, the inventory control policy in the RFID case is much more stable than that of the non-RFID case, as the misplaced inventory can be recovered perfectly and instantly for sale and the inventory shrinkage can be reduced by RFID technology. In addition, one of our intriguing findings is that when the shrinkage rate is below a threshold value which is independent of parameters, RFID application has no effect on inventory control policy if the misplaced inventory can be recovered in a timely manner by physical audit, which has not been revealed in previous studies.     

Continuous time control of the arrival process in an m/g/1 queue

The problem of continuously controlling the arrival process in an M/G/1 queue is studied. The control is exercised by keeping the facility open or closed for potential arrivals, and is based on the residual workload process. The reward structure includes a reward rate R when the server is busy, and a holding cost rate cx when the residual workload is x. The economic criterion used is long run average return. A control limit policy is shown to be optimal. An iterative method for calculating this control limit policy is suggested.     

Effects of system parameters on the optimal policy structure in a class of queueing control problems

This paper studies a class of queueing control problems involving commonly used control mechanisms such as admission control and pricing. It is well established that in a number of these problems, there is an optimal policy that can be described by a few parameters. From a design point of view, it is useful to understand how such an optimal policy varies with changes in system parameters. We present a general framework to investigate the policy implications of the changes in system parameters by using event-based dynamic programming. In this framework, the control model is represented by a number of common operators, and the effect of system parameters on the structured optimal policy is analyzed for each individual operator. Whenever a queueing control problem can be modeled by these operators, the effects of system parameters on the optimal policy follow from this analysis.      

Sequencing and loading of products on a flowline

One of the main objectives of balancing a flowline is to achieve maximum throughput with minimum WIP. To that end, various methods have been proposed to control the loading of products (flow of material) on a flowline. However, the sequence in which the products are released is an issue that has rarely been addressed in conjunction with the material flow control mechanisms. In this paper, we address the problem of determining the optimal cyclic sequence in which to load different products on a flowline in order to achieve optimum throughput at minimum WIP. A lower bound on the WIP level is developed which is then used to determine the desired WIP level and the sequence in which to load the products so as to achieve optimum throughput. A new product release policy is also developed. Comparative results of the new product release policy and the sequencing procedure with other product release policies and sequencing heuristics are presented and they indicate superiority of the proposed methods.     

Fluid analysis of an input control problem

A two-station network with controllable inputs and sequencing control, proposed by Wein (Oper. Res. 38:1065–1078, 1990), is analyzed. A control is sought to minimize holding cost subject to a throughput constraint. In a Lagrangian formulation, input vanishes in the fluid limit. Several alternative fluid models, including workload formulations, are analyzed to develop a heuristic policy for the stochastic network. Both the fluid heuristic and Wein’s diffusion solution are compared with the optimal policy by solving the dynamic program. Examples with up to six customer classes, using Poisson arrival and service processes, are presented. The fluid heuristic does well at sequencing control but the diffusion gives additional, and better, information on input control. The fluid analysis, in particular whether the fluid priorities are greedy, aids in determining whether the fluid heuristic contains useful information.     

Managing production with flexible capacity deployment for serial multi-stage manufacturing systems

This paper presents a model for serial multi-stage manufacturing systems facing variability from two sources. One source is demand uncertainty; the other is manufacturing uncertainty associated with all manufacturing stages. A production control policy based on the planned lead time and the manufacturing capacity requirement is developed. It is shown that this production control policy has the effect of reducing the variance of production output for all manufacturing stages. Some specific analyses are provided to illustrate the production control policy developed. The model developed provides a vehicle for examining the interrelationships among the production output, the planned lead time and the actual manufacturing flow time. The risk-pooling value over both demand randomness and manufacturing uncertainty, which is achieved through consolidating some manufacturing capacity and deploying flexible capacity among the manufacturing stages, is analyzed. This risk-pooling value can be realized in the form of either reduced manufacturing flow time or increased effective capacity to meet more demand. It is shown that the risk-pooling value increases as the planned lead time decreases.     

Approximate receding horizon approach for Markov decision processes: average reward case

We consider an approximation scheme for solving Markov decision processes (MDPs) with countable state space, finite action space, and bounded rewards that uses an approximate solution of a fixed finite-horizon sub-MDP of a given infinite-horizon MDP to create a stationary policy, which we call “approximate receding horizon control.” We first analyze the performance of the approximate receding horizon control for infinite-horizon average reward under an ergodicity assumption, which also generalizes the result obtained by White (J. Oper. Res. Soc. 33 (1982) 253-259). We then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub-MDP. The first control policy is based on a finite-horizon approximation of Howard's policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies. Along the study, we also provide a simple alternative proof on the policy improvement for countable state space. We finally discuss practical implementations of these schemes via simulation.     

Existence of closed-loop policies for constrained discrete-time linear systems with bounded disturbances

We consider discrete-time linear systems with constraints on both control and state variables and bounded disturbances. We exhibit a closed-form expression of the necessary and sufficient conditions for existence of closed-loop policy. These conditions are linear constraints on the initial state and the bounds on the disturbances, control variables and state variables. A simple illustrative example is presented.     

Simple expressions for finding recovery system inventory control parameter values

In recent years considerable effort has been devoted to the development of inventory control models for joint manufacturing and remanufacturing. Optimality of control policies is analyzed and algorithms for the determination of parameter values have been developed. However, there is still a lack of formulae or algorithms that allow for an easy computation of optimal or near optimal policy parameter values. This paper addresses the problem of computing the produce-up-to level S and the remanufacture-up-to level M in a periodic review inventory control model. We provide simple formulae for the policy parameter values, which can easily be implemented within spreadsheet applications. The approach is to derive news-vendor-type formulae that are based on underage and overage cost considerations. We propose different formulae depending on whether lead times for production and remanufacturing are identical or not. A numerical study shows that the obtained solutions provide relatively small cost deviations compared to the optimal solution within the investigated class of inventory control policies.     

The superiority of “optimal control” over simulation in policy analysis

The simulation approach to policy analysis usually concentrates on policy multipliers as a measure of the thrust of economic policy. However, this measure is inadequate for one branch of economic policy, namely, fiscal policy. The reason is that the effectiveness of fiscal policy depends, via the government budget constraint, on the method of finance. It is argued in this paper that for this very reason the conventional way of calculating simulation-based dynamic multipliers introduces a bias towards the no-crowding-out thesis. This bias arises even in models of monetarist persuasion. Furthermore, it is shown that this bias can be removed by utilizing multipliers based on optimal control. We illustrate this proposition by providing numerical results using a large-scale U.K. econometric model of international monetarist persuasion (the London Business School model, LBS). Section 1 builds up a framework through which policy optimization can be compared and evaluated to policy simulations. In Section 2 we derive and compare policy multipliers obtained through policy simulations and optimal control. Section 3 provides a numerical example with the findings being summarized in Section 4.     

交互视角下的客户关系管理时变决策模型研究

从企业和客户交互视角出发，提出关系承诺是客户关系持续的本质，它的驱动因素为信任、认知价值传递、关系管理努力和机会主义倾向。基于此，论文利用动态优化控制理论，研究了客户关系持续管理决策问题，建立了一种在动态环境中客户关系持续管理的时变模型，讨论了模型建立的依据，给出了决策模型的最优结果以及在客户关系管理实践中的意义。     

基于产品质量控制的设备维修优化模型

根据产品质量和生产该产品的设备退化状态之间的相关性,设计了周期性设备检测与产品质量控制相结合的设备维修策略。该策略是在对设备进行周期性检测的基础上,利用控制图进行产品质量异常波动的检测,结合对设备退化状态的检测选择设备应采取的维修活动。根据这一设备维修策略,利用更新过程理论和统计过程控制方法,构建了基于产品质量控制的设备维修优化模型,并用遗传算法对其进行求解。通过实例仿真验证了该模型的可行性与有效性。     

Structural properties of the optimal resource allocation policy for single-queue systems

This paper studies structural properties of the optimal resource allocation policy for single-queue systems. Jobs arrive at a service facility and are sent one by one to a pool of computing resources for parallel processing. The facility poses a constraint on the maximum expected sojourn time of a job. A central decision maker allocates the servers dynamically to the facility. We consider two models: a limited resource allocation model, where the allocation of resources can only be changed at the start of a new service, and a fully flexible allocation model, where the allocation of resources can also change during a service period. In these two models, the objective is to minimize the average utilization costs whilst satisfying the time constraint. To this end, we cast these optimization problems as Markov decision problems and derive structural properties of the relative value function. We show via dynamic programming that (1) the optimal allocation policy has a work-conservation property, and (2) the optimal number of servers follows a step function with as extreme policy the bang-bang control policy. Moreover, (3) we provide conditions under which the bang-bang control policy takes place. These properties give a full characterization of the optimal policy, which are illustrated by numerical experiments.