The evaluation of probabilistic classification methods : Part 2. Comparison of SIMCA,ALLOC, CLASSY and LDA

The performance of four methods for supervised probabilistic classification (LDA, SIMCA, ALLOC and CLASSY) on three types of data sets is evaluated by means of a simulation study. The methods are also applied to some practical data sets (Iris and four data sets for wines). The evaluation criterion used for discriminatory ability is the CBS (complemented Brier score) because it has some advantages over other measures. The danger of applying resubstitution evaluation for method comparison is demonstrated, but leave-one-out evaluation is shown to perform satisfactorily. Horn's method for selecting the number of principal components in SIMCA and CLASSY models is shown to be superior to the average-eigenvalue criterion. It is concluded that CLASSY is a robust method, but that in practice all the methods investigated perform about equally well on average.