Modification and validation of the mixed-format Engineering Concept Assessment for middle school students using many-facet Rasch measurement

Integrating engineering into the K-12 science curriculum continues to be a focus in national reform efforts in science education. Although there is an increasing interest in research in and practice of integrating engineering in K-12 science education, to date only a few studies have focused on the development of an assessment tool to measure students’ understanding of engineering design. Most of the existing measures focus only on knowledge and understanding of engineering design concepts using multiple-choice items with the exception of the mixed-format Engineering Concept Assessment (ECA). Also, advanced measurement models are lacking application in the testing of such mixed-format assessments in science education. This study applied many-faceted Rasch measurement to the modified ECA for eighth-grade (ECA/M8) and a newly constructed rubric applied by five judges across 497 eighth-grade students’ responses after experiencing an integrated learning unit on the engineering design process. The results supported the fit of the items and rubric rating scales to the Rasch specifications. Recommendations are made for item wording, and further reliability and validity testing of the ECA/M8, and use of the ECA/M8 in science education and research.