Accuracy of Automated Written Expression Curriculum-Based Measurement Scoring. (December 2021)

• Record Type: Journal Article
• Authors: Mercer, Sterett H.; Cannon, Joanna E.; Squires, Bonita; Guo, Yue; Pinco, Ella
• Is Part Of: Canadian journal of school psychology. Volume 36:Number 4(2021)
• Page Start: 304
• Page End: 317
• Publication Date: 2021-12
• Subjects: written expression -- curriculum-based measurement -- automated text evaluation -- screening -- progress monitoring; Educational psychology -- Periodicals; Psychology -- Canada -- Periodicals; 370.15
• DOI: 10.1177/0829573520987753
• Languages: English
• ISSNs: 0829-5735

Abstract

We examined the extent to which automated written expression curriculum-based measurement (aWE-CBM) can be accurately used to computer score student writing samples for screening and progress monitoring. Students ( n = 174) with learning difficulties in Grades 1 to 12 who received 1:1 academic tutoring through a community-based organization completed narrative writing samples in the fall and spring across two academic years. The samples were evaluated using four automated and hand-calculated WE-CBM scoring metrics. Results indicated automated and hand-calculated scores were highly correlated at all four timepoints for counts of total words written ( r s = 1.00), words spelled correctly ( r s = .99–1.00), correct word sequences (CWS; r s = .96–.97), and correct minus incorrect word sequences (CIWS; r s = .86–.92). For CWS and CIWS, however, automated scores systematically overestimated hand-calculated scores, with an unacceptable amount of error for CIWS for some types of decisions. These findings provide preliminary evidence that aWE-CBM can be used to efficiently score narrative writing samples, potentially improving the feasibility of implementing multi-tiered systems of support in which the written expression skills of large numbers of students are screened and monitored.