Examining Human and Automated Ratings of Elementary Students' Writing Quality: A Multivariate Generalizability Theory Application. (December 2022)

• Record Type: Journal Article
• Authors: Chen, Dandan; Hebert, Michael; Wilson, Joshua
• Is Part Of: American educational research journal. Volume 59:Number 6(2022)
• Page Start: 1122
• Page End: 1156
• Publication Date: 2022-12
• Subjects: elementary grades -- generalizability theory -- writing assessment; Education -- United States -- Periodicals; Education -- Research -- United States -- Periodicals; 370.97305
• DOI: 10.3102/00028312221106773
• Languages: English
• ISSNs: 0002-8312

Abstract

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students ( n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.