Evaluating alternative system configurations using simulation: A nonparametric approach

The real utility of simulation lies in comparing different alternatives that might represent competing system designs. Conventional statistical techniques are not directly applicable to the analysis of simulation output data in the evaluation of competing alternatives since the usual assumptions of normality and common variance are difficult to justify in simulation experiments. This paper revisits a known nonparametric test whose application has recently become feasible due to considerable increases in computing power:randomization tests assess the significance of the observed value of the test statistic by evaluating different permutations of the data. The procedure only requires invariance of the data under all permutations.