Non-probabilistic Bayesian update method for model validation

Model validation is the principal strategy to evaluate the accuracy and reliability of computational simulations. A systematic model validation procedure including uncertainty quantification, model update and prediction is described based on a non-probabilistic interval model. The crucial technical challenge in model validation is limited data, thus the non-probabilistic interval model is adopted to describe uncertain parameters. To establish the model update formula, the concepts of the interval escape rate and interval coverage rate are first described. Then, not only can the possibility of failure be estimated but also the credibility of the possibility of failure based on the proposed model validation method. The data in the validation experiment are used to update the credibility of each interval model, while the data from the accreditation experiment are used to conduct a final check of the validated models. To demonstrate that the proposed method can be applied to model validation problems successfully, a validation benchmark, the static frame challenge problem, is implemented. In addition, a practical aviation structure engineering validation problem is described. The results of these two validation problems show the feasibility and effectiveness of the proposed model validation method. The theoretical framework proposed in this paper is also suitable for model validation of computational simulations in other research fields.     

Orders of Validation in Mathematical Modelling

Recent work on validation has demonstrated the paucity of validation in operational research projects. What little validation is done usually takes place when the modeller builds a model directly for a decision-maker. This paper begins by reviewing and exploring the processes of model-building and usage with different levels of separation between the model-builder and the ultimate user of the model outputs. The different types of validation that these levels of separation imply are then discussed, with emphasis on validation within linear programming. A checklist of validation questions is suggested. New modelling tools coming onto the market affect the levels at which mathematical modelling takes place and alter the ways in which this modelling should best be carried out in business.     

The Validation of Expert Systems — Contrasts with Traditional Methods

Recent research has demonstrated that little validation is carried out during and after the building of operational research models for decision support. In a subject area that has been of great interest for more than a decade, this reflects the fact that validation is not yet firmly embedded into the methodology of mathematical modelling that underpins conventional decision support systems. With the interest now being shown in expert systems, it is important to devise an appropriate methodology of validation for such systems. This paper contributes to this development by describing the types of validation that should be performed with decision support systems, and compares and contrasts the features found in the validation of mathematical models with those required of validation of expert systems.     

Methods for the external validation of contiuous system simulation models:a review

External validation is an integral part of the process of simulation model development. Many methods of external validation of continuous system simulation models have been proposed and, inevitably, the approach selected in any given application is highly dependent upon the purpose of the model and the associated accuracy requirements. Although the process of external validation has subjective aspects, formalisation of methods in some safety-critical fields of application has provided a basis for the development of objective measures of model performance. Such measures can be applied in other application areas and allow the whole process of assessing model credibility to be put on a firmer foundation. This paper reviews available methods, discusses practical issues that can arise in the application of these validation techniques and gives a brief account of some validation problems in selected application areas.     

Validation of decision-aiding spreadsheets: the influence of contingency factors

This paper describes an investigation into the influence of contingency factors on the validation of spreadsheet-based decision support systems (DSS), and develops a methodological framework for validation that takes account of the effect of contingency factors. The research extends and confirms previous research by one of the authors who identified relevant contingency factors. The perceived influence of these contingency factors on validation effort was investigated by an empirical study, and the results of this study were then used to derive contingency-related guidance for validation. This was evaluated by a sample of practitioners and academics with experience in the development of spreadsheet-based DSS, and was then incorporated into a methodological framework for validation. The framework is designed to enable managers, decision-makers, project leaders and other non-OR specialists to identify the extent of validation effort likely to be required to ensure that the spreadsheet models that they build are valid. The research has implications for the validation of other types of OR models.     

Global minima of least square cross validation for a symmetric polynomial kernel with finite support

The least squares cross validated bandwidth is the minimizer of the cross validation function for choosing the smooth parameter of a kernel density estimator. It is a completely automatic method, but it requires inordinate amounts of computational time. We present a convenient formula for calculation of the cross validation function when the kernel function is a symmetric polynomial with finite support. Also we suggest an algorithm for finding global minima of the cross validation function.     

A systematic comparison of buildingEXODUS predictions with experimental data from the Stapelfeldt trials and the Milburn House evacuation

In this paper, the buildingEXODUS evacuation model is described and discussed and attempts at qualitative and quantitative model validation are presented. The data sets used for validation are the Stapelfeldt and Milburn House evacuation data. As part of the validation exercise, the sensitivity of the buildingEXODUS predictions to a range of variables is examined, including occupant drive, occupant location, exit flow capacity, exit size, occupant response times and geometry definition. An important consideration that has been highlighted by this work is that any validation exercise must be scrutinised to identify both the results generated and the considerations and assumptions on which they are based. During the course of the validation exercise, both data sets were found to be less than ideal for the purpose of validating complex evacuation models. However, the buildingEXODUS evacuation model was found to be able to produce reasonable qualitative and quantitative agreement with the experimental data.     

A Trajectory for Validating Computational Emulation Models of Organizations

Validation of complex simulation models is a challenging problem in computational organization theory research. In this paper, we describe a validation strategy suitable for emulation simulation systems, and show how a comprehensive validation consists of a sequence of steps that evaluate different aspects of a computational organizational simulation model. We demonstrate how this strategy can be applied to the evaluation of the Virtual Team Alliance (VTA), an emulation simulation system designed to guide managers charged with organizational change. VTA required a &201C;trajectory&201D; of successive validation experiments, before managers where willing to follow the recommendations of VTA. Ultimately, we believe this validation approach can be applied to a wide range of different simulation systems, and will make identification of the strengths and weaknesses of organizational simulations easier.     

Modelling and decision support in physiology and medicine: a methodological framework with lllustration

This paper presents a methodological framework for modelling that has found wide application in the complex domains of physiology and medicine. The processes of model validation are centrally embedded within this framework. The parallelism between modelling per se and the development of model-based decision support systems is then considered, showing that it is possible to devise a unified methodological framework which encompasses the requirements both of model validation and decision support system evaluation. The applicability of the framework is demonstrated in the validation of a mathematical model of blood glucose dynamics; and in the development and evaluation of decision support systems such as those which are aimed at addressing the problem of advising the insulin-dependent diabetic patient on the adjustment of insulin dosage     

Nonparametric regression function estimation with surrogate data and validation sampling

This paper develops estimation approaches for nonparametric regression analysis with surrogate data and validation sampling when response variables are measured with errors. Without assuming any error model structure between the true responses and the surrogate variables, a regression calibration kernel regression estimate is defined with the help of validation data. The proposed estimator is proved to be asymptotically normal and the convergence rate is also derived. A simulation study is conducted to compare the proposed estimators with the standard Nadaraya-Watson estimators with the true observations in the validation data set and the complete observations, respectively. The Nadaraya-Watson estimator with the complete observations can serve as a gold standard, even though it is practically unachievable because of the measurement errors.     

Model validation in operations research

Numerous articles have appeared in the literature expressing different degrees of concern with the methodology of OR in general and with the validation of OR models in particular. Suggestions have been formulated to remove some of the shortcomings of the methodology as currently practised and to introduce modifications in the approach because of the changing nature of the problems tackled. Advances in modeling capabilities and solution techniques have also had considerable impact on the way validation is perceived. Large scale computer-based mathematical models and especially simulation models have brought new dimensions to the notion of validation. Terms like ‘confidence’, ‘credibility and reliability’, ‘model assessment and evaluation’, ‘usefulness and usability of the model’ have become rather common. This paper is an attempt, through an interpretation of the literature, to put model validation and related issues in a framework that may be of use both to model-builders and to decision-makers.     

New Approach to Model Validation and Fault Diagnosis

The paper presents a new approach to model validation and fault diagnosis problems for a class of uncertain systems in which the uncertainty is described by an integral quadratic constraint. The new approach is developed by applying methods from linear quadratic optimal control theory. This leads to a method for model validation and fault diagnosis which is based around a robust Kalman filter type structure.     

The Verification of Dynamic Simulation Models

This paper is concerned with the use of ‘loop analysis’ in the evaluation and validation of dynamic simulation models. The part that loop analysis plays in the overall validation process is described, and it is shown, by reference to the analysis of a complex model, how the model's behaviour may be investigated. The analysis proceeds by identifying some of the elements of system behaviour such as damping, phase-shifts and gain/delay factors, and, in the process, highlights those feedback loops that are pseudo-positive and/or unconformable. In addition to the validation aspect of the use of loop analysis, there is a valuable contribution towards the knowledge and insight of the model-builder, enabling him better to identify areas of the system which may benefit from re-design.     

Proof construction and in-process validation – Validation activities of undergraduates in constructing mathematical proofs

Proof construction and proof validation are two situations of high importance in mathematical teaching and research. While both situations have usually been studied separately, the current study focuses on possible intersections. Based on research on acceptance criteria and validation strategies, 11 undergraduates’ proof construction processes are investigated in terms of the effective and non-effective validation activities that occur. By conducting a qualitative content analysis with subsequent type construction, we identified six different validation activities, namely reviewing, rating, correcting errors, reassuring, expressing doubts, and improving. Although some of these activities tend to be associated with successful or unsuccessful proving processes, their effectiveness depends primarily on their specific implementation. For example, reviewing is effective when accompanied by a knowledge-generating approach and based on structure- or meaning-oriented criteria to provide deeper understanding. Thus, the results suggest that difficulties in proof construction could be partly attributed to inadequate validation strategies or their poor implementation.     

A practical exercise in simulation model validation

The validation of continuous system simulation models is a matter of great practical importance. Unfortunately, however, it often receives very little attention in university-level courses on modelling and simulation. The paper considers some possible reasons for the lack of emphasis given to model validation issues in education and describes an exercise which has been designed to introduce students to some practical aspects of internal verification and external validation of nonlinear dynamic models. The work involves a laboratory-scale system based on two inter-connected tanks of liquid and also demonstrates some important limitations of a widely-used nonlinear model. Features which make the chosen system particularly suitable for this teaching application include the relatively simple physical nature of the system and the fact that all the key variables of the model are accessible for measurement. Students are exposed to a range of practical issues, such as the selection of the sampling rate for data collection and the design of experiments to provide data sets appropriate for external validation purposes. Dealing successfully with such questions and carrying out validation tests exposes students to concepts of model credibility and the whole process of model development and application. They are also challenged in terms of their practical abilities to use computer simulation techniques and in their understanding of the mathematical model and the physics of the real system. It is argued that an exercise of this type can have very important educational benefits.     

A survey of contingency factors affecting the validation of end-user spreadsheet-based decision support systems

Decision support systems have become widespread in recent years. However, most of these systems are not carefully validated and this has led in some cases to serious errors. One of the principal reasons for the lack of validity is the lack of an operationalised validation methodology. A simplified validity framework has already been developed by the authors as the first part of a 3-phase approach towards devising a practical validation methodology. This paper describes the second phase, reporting as it does on a survey that has been undertaken to unearth the contingency factors that influence how the process of validation, employing the validity framework, will be carried out for a particular kind of decision support system—one involving a spreadsheet model.     

Cross-validation using the t statistic

One application area of regression analysis is simulation where the regression model may explain the relationship between the simulation model's inputs and outputs.However, whether or not the regression model is used in a simulation context, its validity can be tested by comparing the model's forecast to one or more new observations not used in the estimation of the model's parameters. The familiar Student or t statistic is proposed for this comparison, combined with a Bonferroni approach accounting for the presence of multiple, dependent validation observations.A ‘trick’ is used to obtain as many validation observations as possible. This trick is also known as cross-validation.Several Monte Carlo experiments are performed to study the α and β errors of the proposed validation procedure. The experimental results suggest that the procedure is worthwhile.     

The impact of contingency factors on validation of problem structuring methods

Although much has been written about validating Decision support systems andother ‘hard’ OR models, less has been written about validating‘soft’ OR models. This article seeks to determine which contingencyfactors known to influence DSS validation are also important factors influencingproblem structuring methods (PSM) validation. In addition, after consultationwith PSM experts other contingency factors influencing PSM validation areproposed. Evidence from these PSM experts concerning the levels of influence ofthese factors is used to support proposals for a contingency approach to PSMvalidation.