Evaluating Comparative Judgment as an Approach to Essay Scoring. Issue 3 (2nd July 2016)

• Record Type: Journal Article
• Authors: Steedle, Jeffrey T.; Ferrara, Steve
• Is Part Of: Applied measurement in education. Volume 29:Issue 3(2016)
• Page Start: 211
• Page End: 223
• Publication Date: 2016-07-02
• Subjects: Educational tests and measurements -- United States -- Periodicals; Educational tests and measurements -- Periodicals; Education -- Standards -- Periodicals; 371.26
• DOI: 10.1080/08957347.2016.1171769
• Languages: English
• ISSNs: 0895-7347

Abstract

ABSTRACT: As an alternative to rubric scoring, comparative judgment generates essay scores by aggregating decisions about the relative quality of the essays. Comparative judgment eliminates certain scorer biases and potentially reduces training requirements, thereby allowing a large number of judges, including teachers, to participate in essay evaluation. The purpose of this study was to assess the validity, labor costs, and efficiency of comparative judgments as a potential substitute for rubric scoring. An analysis of two essay prompts revealed that comparative judgment measures were comparable to rubric scores at a level similar to that expected of two professional scorers. The comparative judgment measures correlated slightly higher than rubric scores with a multiple-choice writing test. Score reliability exceeding .80 was achieved with approximately nine judgments per response. The average judgment time was 94 seconds, which compared favorably to 119 seconds per rubric score. Practical challenges to future implementation are discussed.