Evaluating the Advisory Flags and Machine Scoring Difficulty in the e‐rater® Automated Scoring Engine. Issue 2 (18th August 2016)

• Record Type: Journal Article
• Authors: Zhang, Mo; Chen, Jing; Ruan, Chunyi
• Is Part Of: ETS research report series. Issue 2(2016:Dec.)
• Page Start: 1
• Page End: 14
• Publication Date: 2016-08-18
• Subjects: Automated essay scoring -- unusual response -- advisory flag -- machine scoring difficulty -- e‐rater; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12116
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: Successful detection of unusual responses is critical for using machine scoring in the assessment context. This study evaluated the utility of approaches to detecting unusual responses in automated essay scoring. Two research questions were pursued. One question concerned the performance of various prescreening advisory flags, and the other related to the degree of machine scoring difficulty and whether the size of the human–machine discrepancy could be predicted. The results suggested that some advisory flags operated more consistently across measures and tasks in detecting responses that the machine was likely to score differently from human raters than did other flags, and relatively little scoring difficulty was found for three of the four tasks examined in this study, with the relationship between machine and human scores being reasonably strong. Limitations and future studies are also discussed. Abstract : Report Number: ETS RR‐16–30