Expanded understanding of student conceptions of engineers: Validation of the modified draw-an-engineer test (mDAET) scoring rubric

With this research, we sought to expand the pool of engineering education assessment instruments. We drew upon earlier efforts to define an expanded protocol for prompting children's drawings of engineers; and evaluate the reliability of a new scoring rubric for the modified Draw-an-Engineer Test (mDAET). An earlier paper introduced our ground theory approach in gathering theoretical propositions derived from children's drawings, explanations of engineers (n = 940) to develop the mDAET Scoring Rubric. Here we evaluate the reliability of the mDAET, a component of the larger mDAET rubric validation process. In the current study, eight participants scored twelve sets (including anchor sets) of drawings (n = 36) and, scores across drawings were evaluated by percentage match with the known-score and interrater reliability. Kappa values for exact matches across eight raters met the threshold for “substantial” agreement and matches that were exact or adjacent indicated “almost perfect” agreement. Notably, 92% of the possible matches among common anchor items were either exact or adjacent matches. These results indicate adequate inter-rater reliability evidence of the mDAET scoring rubric and demonstrate the feasibility of largescale administration that may no longer require individual child interviews.