Classification accuracy and efficiency of writing screening using automated essay scoring. (October 2020)

• Record Type: Journal Article
• Authors: Wilson, Joshua; Rodrigues, Jessica
• Is Part Of: Journal of school psychology. Volume 82(2020)
• Page Start: 123
• Page End: 140
• Publication Date: 2020-10
• Subjects: Writing -- Screening -- Automated essay scoring -- At risk -- Writing assessment; School psychologists -- Periodicals; School psychology -- Periodicals; Psychology -- Periodicals; School Health Services -- Periodicals; Psychologues scolaires -- Périodiques; Child psychology -- Periodicals; Psychologie scolaire -- Périodiques; Onderwijspsychologie; School psychologists; School psychology; Periodicals; 155.405
• DOI: 10.1016/j.jsp.2020.08.008
• Languages: English
• ISSNs: 0022-4405

Abstract

Abstract: The present study leveraged advances in automated essay scoring (AES) technology to explore a proof of concept for a writing screener using the Project Essay Grade (PEG) program. First, the study investigated the extent to which an AES-scored multi-prompt writing screener accurately classified students as at risk of failing a Common Core-aligned English language arts state test. Second, the study explored whether a similar level of classification accuracy could be achieved with a more efficient form of the AES-screener with fewer writing prompts. Third, the classification accuracy of the AES-scored screeners was compared to that of screeners scored for word count. Students in Grades 3–5 ( n = 185, 167, and 187, respectively) composed six essays in response to multiple writing-prompt screeners on six different randomly assigned topics, consisting of two essays in each of three different genres (narrative, informative, and persuasive). Receiver operating characteristic (ROC) curve analysis was used to assess classification accuracy and to identify multiple cut scores with associated sensitivity and specificity values, and positive and negative posttest probabilities. Results indicated that the AES-scored multi-prompt screener and screeners with fewer prompts yield acceptable classification accuracy, are efficient, and are more accurate than screeners scored for word count. Overall, results illustrate the viability of writing screening using AES.