Expert consensus in solo voice performance evaluation

Experts were interviewed to identify criteria for evaluation of vocal performance. A scale was then constructed and inter- and intrajudge reliability assessed. Experts listened to 19 different performances, plus 6 presented a second time. Interjudge reliability for one judge was modest, but increased dramatically as the size of the judge panel increased. The most reliable items were overall score and intonation accuracy. Diction was less reliable than other items. Intrajudge reliability was higher for overall score than for any other item. A factor analysis on the test items yielded factors labelled intrinsic quality, execution, and diction. Another factor analysis, using the experts as variables, revealed two underlying evaluative dimensions. It was found that 13 experts were primarily influenced by execution, and that 8 were mainly affected by intrinsic quality. Interjudge and intrajudge reliabilities of these two groups differed.