A survey of contingency factors affecting the validation of end-user spreadsheet-based decision support systems

Decision support systems have become widespread in recent years. However, most of these systems are not carefully validated and this has led in some cases to serious errors. One of the principal reasons for the lack of validity is the lack of an operationalised validation methodology. A simplified validity framework has already been developed by the authors as the first part of a 3-phase approach towards devising a practical validation methodology. This paper describes the second phase, reporting as it does on a survey that has been undertaken to unearth the contingency factors that influence how the process of validation, employing the validity framework, will be carried out for a particular kind of decision support system—one involving a spreadsheet model.     

Modeling centralized organization of organizational change

Organizations change with the dynamics of the world. To enable organizations to change, certain structures and capabilities are needed. As all processes, a change process has an organization of its own. In this paper it is shown how within a formal organization modeling approach also organizational change processes can be modeled. A generic organization model (covering both organization structure and behavior) for organizational change is presented and formally evaluated for a case study. This model takes into account different phases in a change process considered in Organization Theory literature, such as unfreezing, movement and refreezing. Moreover, at the level of individuals, the internal beliefs and their changes are incorporated in the model. In addition, an internal mental model for (reflective) reasoning about expected role behavior is included in the organization model.     

A Trajectory for Validating Computational Emulation Models of Organizations

Validation of complex simulation models is a challenging problem in computational organization theory research. In this paper, we describe a validation strategy suitable for emulation simulation systems, and show how a comprehensive validation consists of a sequence of steps that evaluate different aspects of a computational organizational simulation model. We demonstrate how this strategy can be applied to the evaluation of the Virtual Team Alliance (VTA), an emulation simulation system designed to guide managers charged with organizational change. VTA required a &201C;trajectory&201D; of successive validation experiments, before managers where willing to follow the recommendations of VTA. Ultimately, we believe this validation approach can be applied to a wide range of different simulation systems, and will make identification of the strengths and weaknesses of organizational simulations easier.     

The Verification of Dynamic Simulation Models

This paper is concerned with the use of ‘loop analysis’ in the evaluation and validation of dynamic simulation models. The part that loop analysis plays in the overall validation process is described, and it is shown, by reference to the analysis of a complex model, how the model's behaviour may be investigated. The analysis proceeds by identifying some of the elements of system behaviour such as damping, phase-shifts and gain/delay factors, and, in the process, highlights those feedback loops that are pseudo-positive and/or unconformable. In addition to the validation aspect of the use of loop analysis, there is a valuable contribution towards the knowledge and insight of the model-builder, enabling him better to identify areas of the system which may benefit from re-design.     

Methods for the external validation of contiuous system simulation models:a review

External validation is an integral part of the process of simulation model development. Many methods of external validation of continuous system simulation models have been proposed and, inevitably, the approach selected in any given application is highly dependent upon the purpose of the model and the associated accuracy requirements. Although the process of external validation has subjective aspects, formalisation of methods in some safety-critical fields of application has provided a basis for the development of objective measures of model performance. Such measures can be applied in other application areas and allow the whole process of assessing model credibility to be put on a firmer foundation. This paper reviews available methods, discusses practical issues that can arise in the application of these validation techniques and gives a brief account of some validation problems in selected application areas.     

Orders of Validation in Mathematical Modelling

Recent work on validation has demonstrated the paucity of validation in operational research projects. What little validation is done usually takes place when the modeller builds a model directly for a decision-maker. This paper begins by reviewing and exploring the processes of model-building and usage with different levels of separation between the model-builder and the ultimate user of the model outputs. The different types of validation that these levels of separation imply are then discussed, with emphasis on validation within linear programming. A checklist of validation questions is suggested. New modelling tools coming onto the market affect the levels at which mathematical modelling takes place and alter the ways in which this modelling should best be carried out in business.     

Aligning simulation models: A case study and results

This paper develops the concepts and methods of a process we will call “alignment of computational models” or “docking” for short. Alignment is needed to determine whether two models can produce the same results, which in turn is the basis for critical experiments and for tests of whether one model can subsume another. We illustrate our concepts and methods using as a target a model of cultural transmission built by Axelrod. For comparison we use the Sugarscape model developed by Epstein and Axtell. The two models differ in many ways and, to date, have been employed with quite different aims. The Axelrod model has been used principally for intensive experimentation with parameter variation, and includes only one mechanism. In contrast, the Sugarscape model has been used primarily to generate rich “artificial histories”, scenarios that display stylized facts of interest, such as cultural differentiation driven by many different mechansims including resource availability, migration, trade, and combat. The Sugarscape model was modified so as to reproduce the results of the Axelrod cultural model. Among the questions we address are: what does it mean for two models to be equivalent, how can different standards of equivalence be statistically evaluated, and how do subtle differences in model design affect the results? After attaining a “docking” of the two models, the richer set of mechanisms of the Sugarscape model is used to provide two experiments in sensitivity analysis for the cultural rule of Axelrod's model. Our generally positive experience in this enterprise has suggested that it could be beneficial if alignment and equivalence testing were more widely practiced among computational modelers.     

Aggregation and the design of models for medium-term planning of production

A major issue in the design of an aggregate multi-period model for medium-term planning of production is the determination of the aggregation level of the variables. In the process of model design, one has to examine if and how the outcomes of the model are effected by applying different modes and levels of aggregation. This is especially important with regard to those outcomes which will be used for medium-term decisions and for laying out short-term and detailed plans. Also dependent on the aggregation level is the operational convenience of the model for decision support. In this paper it will be shown that simulation experiments with a preliminary model for medium-term planning can yield valuable information to aid this design problem.     

Why modelling and model use matter

When OR/MS analysts develop a model, how are they intending this model to be used? There are many different ways in which OR/MS models may be classified and one important categorisation is the intended use of the model. Some models are intended for routine use on a frequent basis, with little or no human intervention. Others form part of human decision process and provide support to that process. Considering model validation, data requirements, added value and possible pitfalls leads to a theory of model use based on four categories: decision automation, routine decision support, investigation and improvement, and generating insights for debate. A pilot investigation in an OR/MS group demonstrates that this categorisation could provide the basis for empirical research into a theory of model use in operational research. A theory of model use would be of value to academics, who could prioritise their work, and to practitioners, who could place their own work in a broader landscape.     

A practical exercise in simulation model validation

The validation of continuous system simulation models is a matter of great practical importance. Unfortunately, however, it often receives very little attention in university-level courses on modelling and simulation. The paper considers some possible reasons for the lack of emphasis given to model validation issues in education and describes an exercise which has been designed to introduce students to some practical aspects of internal verification and external validation of nonlinear dynamic models. The work involves a laboratory-scale system based on two inter-connected tanks of liquid and also demonstrates some important limitations of a widely-used nonlinear model. Features which make the chosen system particularly suitable for this teaching application include the relatively simple physical nature of the system and the fact that all the key variables of the model are accessible for measurement. Students are exposed to a range of practical issues, such as the selection of the sampling rate for data collection and the design of experiments to provide data sets appropriate for external validation purposes. Dealing successfully with such questions and carrying out validation tests exposes students to concepts of model credibility and the whole process of model development and application. They are also challenged in terms of their practical abilities to use computer simulation techniques and in their understanding of the mathematical model and the physics of the real system. It is argued that an exercise of this type can have very important educational benefits.     

Model-based parameter optimization for arc welding process simulation

This paper presents a model-based parameter optimization for simulating a metal-inert gas welding process. The computational model used in this study is based on computational fluid dynamics methods and implemented using the finite volume approach on a 3D computational domain. The wire electrode, the arc plasma and the workpiece are treated as a self-consistent system. Important welding parameters, including arc current, wire feed rate, workpiece thickness, welding speed and geometry, as well as the metal alloy types used for the wire and workpiece, were implemented as adjustable parameters. By tuning these parameters, the performance of the arc welding can be predicted, and different settings can be compared to optimize welding performance.A benchmarking study of the arc model against experimental measurements is presented to demonstrate the model's capabilities in the prediction of the weld pool changes and thermal dynamics involved in the welding process. Two numerical case studies are presented to demonstrate the use of the model-based optimization to quantify welding pool variations with the change in welding parameters. The first case study is the determination of the optimal arc current and welding speed settings for different workpiece thicknesses. The optimization process shows that the predictions are not only in agreement with established experimental welding experience on the direct relationship between workpiece thickness and arc current, but more importantly quantify this relationship for a given workpiece thickness. The second case study focuses on the welding parameters optimization for different metal alloys. The comparison suggests that the welding parameters suitable for some aluminium alloys are less likely to be successful in welding magnesium alloys. A further model validation of Mg alloy AZ31 welding shows an agreement with experimental measurements. The work presented shows the potential of model-based parameter optimization to assist process engineers in the practical improvement of the welding process.     

Response of moist paperboard during rapid compression and heating

The response of moist paperboard exposed to extensive compression and heating in short periods of time is investigated. A generic framework describing this response, in a thermodynamically consistent manner, has been derived previously. The present paper provides explicit formats of the necessary constitutive relations specific to moist paperboard exposed to extensive compression and heating in short periods of time. The transient transports of mass, momentum and energy, as well as specific interaction terms are considered for orthotropic paperboard. The elasto-plastic response is taken into account in a large strain setting. The exchange of mass between the water bound to the fibers and the water vapor during the sealing is also considered. Simulations of an idealized sealing of two sheets of paperboard are performed and the predicted distributions of temperature, vapor pressure, out-of-plane stress and Forchheimer number are studied. The discussion related to the results from the simulations provides a deeper insight to how the different transport processes will affect the paperboard and how these are coupled. The closed system of equations, including the explicit formats of constitutive relations, provided in this paper makes it possible to set up suitable experiments for validation of the model.     

A systematic comparison of buildingEXODUS predictions with experimental data from the Stapelfeldt trials and the Milburn House evacuation

In this paper, the buildingEXODUS evacuation model is described and discussed and attempts at qualitative and quantitative model validation are presented. The data sets used for validation are the Stapelfeldt and Milburn House evacuation data. As part of the validation exercise, the sensitivity of the buildingEXODUS predictions to a range of variables is examined, including occupant drive, occupant location, exit flow capacity, exit size, occupant response times and geometry definition. An important consideration that has been highlighted by this work is that any validation exercise must be scrutinised to identify both the results generated and the considerations and assumptions on which they are based. During the course of the validation exercise, both data sets were found to be less than ideal for the purpose of validating complex evacuation models. However, the buildingEXODUS evacuation model was found to be able to produce reasonable qualitative and quantitative agreement with the experimental data.     

Modelling a multi-objective allocation problem in a government sponsored entrepreneur development programme

Typically the vendor allocation problem for a commercial company requires the buyer to allocate business between the selected vendor suppliers to meet its needs at minimum cost, taking account of its quality and delivery performance requirements. However in a government sponsored entrepreneur development programme, the allocation of business cannot only be made dependent on criteria that reflect the current ability of the vendor nor solely on criteria fulfilling the current needs of the buyer. It must also consider how it may aid the vendors to improve their business performance in the future and also on how it affects the development of that indigenous industry sector. Furthermore, conditions change from one year to the next so the problem is dynamic over time.This paper describes how a combination of Goal Programming model (GP) with pre-emptive priority ranking of goal constraints with a linear programming model facilitates allocation of businesses to entrepreneurs. In particular, the subject used in this paper is a Malaysia Government sponsored entrepreneur development programme for furniture maker, which administered by a commercial company. An innovative aspect of the work is the use of GP as a modelling rather than a solution technique. It is used to convert the conflicting multi-objectives that were expressed only in general policy terms to operational terms on which business could be allocated. The paper shows how it can be used to try to understand, and then formally model how managers use their judgement and experience in a complex multi-criteria situation. The paper goes on to show that using the formal model leads to more consistency in decision-making and an improvement in the achievement of the objectives. This is important as several different managers are having to make independent decisions on subsets of the vendors.     

Goodness-of-fit indices for partial least squares path modeling

This paper discusses a recent development in partial least squares (PLS) path modeling, namely goodness-of-fit indices. In order to illustrate the behavior of the goodness-of-fit index (GoF) and the relative goodness-of-fit index (GoF_rel), we estimate PLS path models with simulated data, and contrast their values with fit indices commonly used in covariance-based structural equation modeling. The simulation shows that the GoF and the GoF_rel are not suitable for model validation. However, the GoF can be useful to assess how well a PLS path model can explain different sets of data.     

How persuaded are you? A typology of responses

Several recent studies have suggested that there are two different ways in which a person can proceed when assessing the persuasiveness of a mathematical argument: by evaluating whether it is personally convincing, or by evaluating whether it is publicly acceptable. In this paper, using Toulmin's (1958) argumentation scheme, we produce a more detailed theoretical classification of the ways in which participants can interpret a request to assess the persuasiveness of an argument. We suggest that there are (at least) five ways in which such a question can be interpreted. The classification is illustrated with data from a study that asked undergraduate students and research-active mathematicians to rate how persuasive they found a given argument. We conclude by arguing that researchers interested in mathematical conviction and proof validation need to be aware of the different ways in which participants can interpret questions about the persuasiveness of arguments, and that they must carefully control for these variations during their studies.     

Capital Investment — An Optimal Control Perspective

Many managers appear to have a mental model of how investment decisions should be carried out. This paper attempts to identify some of the characteristics of such a mental model by constructing an optimal control model of the process of authorizing an investment project. The key activities in the model are design, support generation and authorization. It is assumed that the effort that can be expended on each is limited. The objective function is the net present value of the cash flows associated with the project. The solution to the model is readily interpreted qualitatively. For viable projects, four different patterns of decision-making are found, each of which is optimum under appropriate circumstances. It is argued that in many actual investment decisions it should be possible for managers to approximate reasonably closely the optimal behaviour, and that therefore the optimal control model may shed light on managers' mental model.     

Modelling and decision support in physiology and medicine: a methodological framework with lllustration

This paper presents a methodological framework for modelling that has found wide application in the complex domains of physiology and medicine. The processes of model validation are centrally embedded within this framework. The parallelism between modelling per se and the development of model-based decision support systems is then considered, showing that it is possible to devise a unified methodological framework which encompasses the requirements both of model validation and decision support system evaluation. The applicability of the framework is demonstrated in the validation of a mathematical model of blood glucose dynamics; and in the development and evaluation of decision support systems such as those which are aimed at addressing the problem of advising the insulin-dependent diabetic patient on the adjustment of insulin dosage     

Classical model validation for control design purposes

Model Validation is at the heart of the System Identification process. Recently, much renewed interest has been expressed in so called "identification for control". This means that the design variables associated with the identification process are tailored to achieve models that are well suited for control design purposes. A separate, but closely related issue is to devise validation tests that give information about the model's quality and suitability for control design. This paper shows and discusses how a basic and classical residual test gives such information.     

The Paucity of Model Validation in Operational Research Projects

Validation as a concept is not well defined in the methodology of model building, although it is generally considered to be an important part of the modelling process. This paper considers the amount and type of validation carried out in a series of projects, and establishes contrasts between the expected and observed modes of validation.