Applying Cognitive Theory to the Human Essay Rating Process. Issue 3 (2nd July 2020)

• Record Type: Journal Article
• Authors: Finn, Bridgid; Arslan, Burcu; Walsh, Matthew
• Is Part Of: Applied measurement in education. Volume 33:Issue 3(2020)
• Page Start: 223
• Page End: 233
• Publication Date: 2020-07-02
• Subjects: Educational tests and measurements -- United States -- Periodicals; Educational tests and measurements -- Periodicals; Education -- Standards -- Periodicals; 371.26
• DOI: 10.1080/08957347.2020.1750405
• Languages: English
• ISSNs: 0895-7347

Abstract

ABSTRACT: To score an essay response, raters draw on previously trained skills and knowledge about the underlying rubric and score criterion. Cognitive processes such as remembering, forgetting, and skill decay likely influence rater performance. To investigate how forgetting influences scoring, we evaluated raters' scoring accuracy on TOEFL and GRE essays. We used binomial linear mixed effect models to evaluate how the effect of various predictors such as time spent scoring each response and days between scoring sessions relate to scoring accuracy. Results suggest that for both nonoperational (i.e., calibration samples completed prior to a scoring session) and operational scoring (i.e., validity samples interspersed among actual student responses), the number of days in a scoring gap negatively affects performance. The findings, as well as other results from the models are discussed in the context of cognitive influences on knowledge and skill retention.