The Metamodel in Simulation Analysis: Can It Be Trusted?

The use of a mathematical metamodel such as a regression model, constructed from simulation data and used to aid in the analysis of the simulated system, has been studied in recent years. For practitioners, the vast benefits of establishing a functional relationship among the variables in an unfamiliar and complex simulated system may be largely overshadowed by the concern that the metamodel, being a strongly data-based technique, may be valid only for the one particular set of simulation-generated data that went into it, which is to say not valid at all. Based on a study of 30 simulation experiments using three different simulation models, the authors conclude that the simulation metamodel is a reliable and valid technique to use in post-simulation analysis, and is probably just as good as the simulation model on which it is based.     

Solving a multi-objective simulation model using a hybrid response surface method and lexicographical goal programming approach—a case study on integrated circuit ink-marking machines

In an integrated circuit (IC) packaging plant, the ink-marking machine has a significantly higher throughput than the other processing machines. When periodic demand surges result in backlog orders or in lost customers, there is a need to increase system throughput. To resolve this problem, the purchase of a new machine often results in excess capacity in addition to added operation and acquisition costs. Therefore, the productivity improvement effort has priority over the machine purchase decision. This paper seeks to optimize both throughput and cycle time performance for IC ink-marking machines. While throughput increase is the primary objective, there is an acceptable cycle time limit for a feasible solution. It is a multi-objective problem. The proposed solution methodology constructed a simulation metamodel for the ink-marking operation by using a fractional factorial experimental design and regression analysis. It is then solved by a hybrid response surface method and lexicographical goal programming approach. Solution results illustrated a successful application.     

A comparison of the performance of artificial intelligence techniques for optimizing the number of kanbans

This paper discusses the use of modern heuristic techniques coupled with a simulation model of a Just in Time system to find the optimum number of kanbans while minimizing cost. Three simulation search heuristic procedures based on Genetic Algorithms, Simulated Annealing, and Tabu Search are developed and compared both with respect to the best results achieved by each algorithm in a limited time span and their speed of convergence to the results. In addition, a Neural Network metamodel is developed and compared with the heuristic procedures according to the best results. The results indicate that Tabu Search performs better than the other heuristics and Neural Network metamodel in terms of computational effort.     

A fuzzy theoretic approach to simulation metamodeling

In this paper, we explore the potential application of fuzzy linear regression in developing simulation metamodels. It should be noted that the basic construct for simulation metamodels involves uncertainties and ambiguities that may be better addressed through fuzzy linear regression application. The solution techniques employed by fuzzy linear regression are very familiar, and the generation of fuzzy outputs may offer a wide range of solution space to the decision maker, thereby reducing the risk of making an incorrect economic decision. A numerical example is presented to show how a possibility distribution is used to capture the vagueness in a dependent variable for a regression metamodel.     

贝叶斯网络仿真元模型及其在装备战场损伤仿真中的应用研究

结合装备战场损伤仿真系统,研究了贝叶斯网络仿真元模型的构建方法.从条件概率角度描述了仿真模型输入参数与输出参数之间的映射关系,研究了构建贝叶斯网络仿真元模型的可行性,分析了贝叶斯网络仿真元模型的优点;研究了贝叶斯网络仿真元模型构建过程中的关键问题,包括:元模型参数的确定、原始模型参数向贝叶斯网络节点的转化、联结强度的计算、衍生元模型的构建;针对不完全信息条件下装备战场损伤快速定位问题,研究了基于K2算法的贝叶斯网络仿真元模型构建方法;构建了某型高炮的战场损伤贝叶斯网络仿真元模型.     

A more accurate second-order polynomial metamodel using a pseudo-random number assignment strategy

The response surface metamodel is a useful sequential methodology for approximating the relationship between the input variables and the output response in computer simulation. Several strategies have been proposed to increase the accuracy of the estimation of the metamodel. In the current paper, we introduce an effective pseudo-random number (PRN) assignment strategy with Box-Behnken design to construct a more accurate second-order polynomial metamodel to estimate the network reliability of a complex system. The results obtained from the simulation approach show that the reduction in maximum absolute relative error between the response surface approximation and the actual reliability function is 35.63% after the PRN assignment strategy is applied.     

Generalization of simulation results practicality of statistical methods

The major purpose of this paper is to evaluate the practical use of statistical techniques in both the generalization or analysis of simulation results, and the design of simulation experiments. This problem is investigated with the help of a real-life system, namely the container terminus of ECT in Rotterdam. This system is modeled by a simulation program. The relationship between the simulation response and its input variables is modeled by a linear regression model: metamodel or auxiliary model. The paper summarizes regression analysis including generalized least squares which might be used for simulation responses with non-constant variances. The validity of the postulated regression metamodel is tested statistically: F- and t-statistics. The selection of the situations to be simulated, is done through experimental design methodology, permitting both quantitative and qualitative factors. The statistical techniques apply not only to simulation but also to real-life experiments.     

Robustness of Kriging when interpolating in random simulation with heterogeneous variances: Some experiments

This paper investigates the use of Kriging in random simulation when the simulation output variances are not constant. Kriging gives a response surface or metamodel that can be used for interpolation. Because Ordinary Kriging assumes constant variances, this paper also applies Detrended Kriging to estimate a non-constant signal function, and then standardizes the residual noise through the heterogeneous variances estimated from replicated simulation runs. Numerical examples, however, suggest that Ordinary Kriging is a robust interpolation method.     

Optimum control laws in a non-autonomous dynamic model of a gas field

A model of gas field development described as a nonlinear optimum control problem with an infinite planning horizon is considered. The Pontryagin maximum principle is used to solve it. The theorem on sufficient optimumity conditions in terms of constructions of the Pontryagin maximum principles is used to substantiate the optimumity of the extremal solution. A procedure for constructing the optimum solution by dynamic programming is described and is of some methodological interest. The obtained optimum solution is used to construct the Bellman function. Reference is made to a work containing an economic interpretation of the problem.     

Using subsystem linear regression metamodels in stochastic simulation

This article explores the use of metamodels as simulation building blocks. The metamodel replaces a part of the simulation model with a mathematical function that mimics the input–output behavior of that part, with respect to some measure of interest to the designer. The integration of metamodels as components of the simulation model simplifies the model and reduces the simulation time. Such use of the metamodels also gives the designer a better understanding of the behavior of those parts of the model, making the simulation model as a whole more intelligible. The metamodel-based simulation model building process is examined, step by step, and the designer options are explored. This process includes the identification of the metamodel candidates and the construction of the metamodels themselves. The assessment of the proposed approach includes the evaluation of the integration effort of the metamodel into the metamodel-based simulation model, and the accuracy of the output data when compared to the original system.     

Construction and validation of distribution-based regression simulation metamodels

Metamodels are used as analysis tools for solving optimization problems. A metamodel is a simplification of the simulation model, representing the system's input–output relationship through a mathematical function with customized parameters. The proposed approach uses confidence intervals as an acceptance procedure, and as a predictive validation procedure when new points are employed. To improve the knowledge about the system, the response is depicted by modelling both the mean and variance functions of a normal distribution along the experimental region. Such metamodels are specially useful when the variance of the output varies significantly. These metamodels may be used for minimizing product quality loss and improving production robustness. The development of such metamodels is illustrated with two examples.     

Functionality defense through diversity: a design framework to multitier systems

Diversification is one of the most effective approaches to defend multitier systems against attacks, failure, and accidents. However, designing such a system with effective diversification is a challenging task because of stochastic user and attacker behaviors, combinatorial-explosive solution space, and multiple conflicting design objectives. In this study, we present a systematic framework for exploring the solution space, and consequently help the designer select a satisfactory system solution. A simulation model is employed to evaluate design solutions, and an artificial neural network is trained to approximate the behavior of the system based on simulation output. Guided by a trained neural network, a multi-objective evolutionary algorithm (MOEA) is proposed to search the solution space and identify potentially good solutions. Our MOEA incorporates the concept of Herbert Simon??s satisficing. It uses the decision maker??s aspiration levels for system performance metrics as its search direction to identity potentially good solutions. Such solutions are then evaluated via simulation. The newly-obtained simulation results are used to refine the neural network. The exploration process stops when the result converges or a satisfactory solution is found. We demonstrate and validate our framework using a design case of a three-tier web system.     

Simulation metamodeling with dynamic Bayesian networks

This paper presents a novel approach to simulation metamodeling using dynamic Bayesian networks (DBNs) in the context of discrete event simulation. A DBN is a probabilistic model that represents the joint distribution of a sequence of random variables and enables the efficient calculation of their marginal and conditional distributions. In this paper, the construction of a DBN based on simulation data and its utilization in simulation analyses are presented. The DBN metamodel allows the study of the time evolution of simulation by tracking the probability distribution of the simulation state over the duration of the simulation. This feature is unprecedented among existing simulation metamodels. The DBN metamodel also enables effective what-if analysis which reveals the conditional evolution of the simulation. In such an analysis, the simulation state at a given time is fixed and the probability distributions representing the state at other time instants are updated. Simulation parameters can be included in the DBN metamodel as external random variables. Then, the DBN offers a way to study the effects of parameter values and their uncertainty on the evolution of the simulation. The accuracy of the analyses allowed by DBNs is studied by constructing appropriate confidence intervals. These analyses could be conducted based on raw simulation data but the use of DBNs reduces the duration of repetitive analyses and is expedited by available Bayesian network software. The construction and analysis capabilities of DBN metamodels are illustrated with two example simulation studies.     

A Metamodel Specification for a Tomato Processing Plant

This paper reports on the specification of a tomato processing plant cost-function using the Schruben-Cogliano (S-C) experimental procedure for response surface identification. The method is applied to a simulation model of a tomato processing plant in order to generate a cost-function metamodel that can be used to derive conditional factor-demand equations. The results suggest that the S-C methodology provides a means of deriving conditional factor-demand equations which reduces cost-function specification error.     

Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations

Kriging metamodels (also called Gaussian process or spatial correlation models) approximate the Input/Output functions implied by the underlying simulation models. Such metamodels serve sensitivity analysis, especially for computationally expensive simulations. In practice, simulation analysts often know that this Input/Output function is monotonic. To obtain a Kriging metamodel that preserves this characteristic, this article uses distribution-free bootstrapping assuming each input combination is simulated several times to obtain more reliable averaged outputs. Nevertheless, these averages still show sampling variation, so the Kriging metamodel does not need to be an exact interpolator; bootstrapping gives a noninterpolating Kriging metamodel. Bootstrapping may use standard Kriging software. The method is illustrated through the popular M/M/1 model with either the mean or the 90% quantile as output; these outputs are monotonic functions of the traffic rate. The empirical results demonstrate that monotonicity-preserving bootstrapped Kriging gives higher probability of covering the true outputs, without lengthening the confidence interval.     

Discovering metamodels’ quality-of-fit for simulation via graphical techniques

Metamodels are used in many disciplines to replace simulation models of complex multivariate systems. To discover metamodels ‘quality-of-fit’ for simulation, simple information returned by average-based statistics, such as root-mean-square error RMSE, are often used. The sample of points used in determining these averages is restricted in size, especially for simulation models of complex multivariate systems. Obviously, decisions made based on average values can be misleading when the sample size is not adequate, and contributions made by each individual data point in such samples need to be examined. This paper presents methods that can be used to discover metamodels quality-of-fit graphically by means of two-dimensional plots. Three plot types are presented; these are the so-called circle plots, marksman plots, and ordinal plots. Such plots can be used to facilitate visual inspection of the effect on metamodel accuracy of each individual point in the data sample used for metamodel validation. The proposed methods can be used to complement quantitative validation statistics; in particular, for situations where there is not enough validation data or the validation data is too expensive to generate.     

A new method for reliability analysis of structures with mixed random and convex variables

This paper proposes a method combining projection-outline-based active learning strategy with Kriging metamodel for reliability analysis of structures with mixed random and convex variables. In this method, it is determined that the approximation accuracy of projection outlines on the limit-state surface is crucial for estimation of failure probability instead of the whole limit-state surface. To efficiently improve the approximation accuracy of projection outlines, a new projection-outline-based active learning strategy is developed to sequentially obtain update points located around the projection outlines. Taking into account the influence of metamodel uncertainty on the estimation of failure probability, a quantification function of metamodel uncertainty is developed and introduced in the stopping condition of Kriging metamodel update. Finally, Monte Carlo simulation is employed to calculate the failure probability based on the refined Kriging metamodel. Four examples including the Burro Creek Bridge and a piezoelectric energy harvester are tested to validate the performance of the proposed method. Results indicate that the proposed method is accurate and efficient for reliability analysis of structures with mixed random and convex variables.     

Support vector regression based metamodeling for seismic reliability analysis of structures

The present study deals with support vector regression-based metamodeling approach for efficient seismic reliability analysis of structure. Various metamodeling approaches e.g. response surface method, Kriging interpolation, artificial neural network, etc. are usually adopted to overcome computational challenge of simulation based seismic reliability analysis. However, the approximation capability of such empirical risk minimization principal-based metamodeling approach is largely affected by number of training samples. The support vector regression based on the principle of structural risk minimization has revealed improved response approximation ability using small sample learning. The approach is explored here for improved estimate of seismic reliability of structure in the framework of Monte Carlo Simulation technique. The parameters necessary to construct the metamodel are obtained by a simple effective search algorithm by solving an optimization sub-problem to minimize the mean square error obtained by cross-validation method. The simulation technique is readily applied by random selection of metamodel to implicitly consider record to record variations of earthquake. Without additional computational burden, the approach avoids a prior distribution assumption about approximated structural response unlike commonly used dual response surface method. The effectiveness of the proposed approach compared to the usual polynomial response surface and neural network based metamodels is numerically demonstrated.     

Validation of regression metamodels in simulation: Bootstrap approach

Simulation experiments are often analyzed through a linear regression model of their input/output data. Such an analysis yields a metamodel or response surface for the underlying simulation model. This metamodel can be validated through various statistics; this article studies (1) the coefficient of determination (R-square) for generalized least squares, and (2) a lack-of-fit F-statistic originally formulated by Rao [Biometrika 46 (1959) 49], who assumed multivariate normality. To derive the distributions of these two validation statistics, this paper shows how to apply bootstrapping—without assuming normality. To illustrate the performance of these bootstrapped validation statistics, the paper uses Monte Carlo experiments with simple models. For these models (i) R-square is a conservative statistic (rejecting a valid metamodel relatively rarely), so its power is low; (ii) Rao’s original statistic may reject a valid metamodel too often; (iii) bootstrapping Rao’s statistic gives only slightly conservative results, so its power is relatively high.     

Integer Programming to Minimize Labour Costs

The main purpose of this paper is to demonstrate a real-world application of pure integer programming to find the optimum solution to a labour cost problem. The length of a daily working shift is defined as an integer variable and several shift strategies are analysed to determine the optimum length and shift combinations that satisfy a predicted demand at minimum cost. The state-space model has been used to predict the stochastic behaviour of monthly demands for beer and soft drink. Savings of about 7% of the annual sales have been obtained as a result of implementing the integer programming approach. A numerical example shows that the solution obtained by rounding off the continuous optimal solution does not match with the integer optimal solution. It was also noted that if a rounded-off solution is feasible, then it provides an initial integer solution for the branch-and-bound algorithm that may reduce the computational time.