Comparing the Effectiveness of Self‐Paced and Collaborative Frame‐of‐Reference Training on Rater Accuracy in a Large‐Scale Writing Assessment. Issue 3 (September 2015)

• Record Type: Journal Article
• Authors: Raczynski, Kevin R.; Cohen, Allan S.; Engelhard, George; Lu, Zhenqiu
• Is Part Of: Journal of educational measurement. Volume 52:Issue 3(2015)
• Page Start: 301
• Page End: 318
• Publication Date: 2015-09
• Subjects: Educational tests and measurements -- Periodicals; 371.2605
• DOI: 10.1111/jedm.12079
• Languages: English
• ISSNs: 0022-0655

Abstract

<abstract abstract-type="main"> <title> <x xml:space="preserve">Abstract</x> </title> <p>There is a large body of research on the effectiveness of rater training methods in the industrial and organizational psychology literature. Less has been reported in the measurement literature on large‐scale writing assessments. This study compared the effectiveness of two widely used rater training methods—self‐paced and collaborative frame‐of‐reference training—in the context of a large‐scale writing assessment. Sixty‐six raters were randomly assigned to the training methods. After training, all raters scored the same 50 representative essays prescored by a group of expert raters. A series of generalized linear mixed models were then fitted to the rating data. Results suggested that the self‐paced method was equivalent in effectiveness to the more time‐intensive and expensive collaborative method. Implications for large‐scale writing assessments and suggestions for further research are discussed.</p> </abstract>