A study on the impact of fatigue on human raters when scoring speaking responses. (October 2014)

• Record Type: Journal Article
• Authors: Ling, Guangming; Mollaun, Pamela; Xi, Xiaoming
• Is Part Of: Language testing. Volume 31:Number 4(2014:Oct.)
• Page Start: 479
• Page End: 499
• Publication Date: 2014-10
• Subjects: fatigue effects -- rating accuracy -- rating consistency -- session length -- shift length -- speaking responses; Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 407.6
• DOI: 10.1177/0265532214530699
• Languages: English
• ISSNs: 0265-5322

Abstract

The scoring of constructed responses may introduce construct-irrelevant factors to a test score and affect its validity and fairness. Fatigue is one of the factors that could negatively affect human performance in general, yet little is known about its effects on a human rater's scoring quality on constructed responses. In this study, we compared the scoring quality of 72 raters under four shift conditions differing on the shift length (total scoring time in a day) and session length (time continuously spent on a task). About 14, 000 audio responses to four TOEFL iBT speaking tasks were scored, including 5446 validity responses that have pre-assigned "true" scores used to measure scoring accuracy. Our results suggest that the overall scoring accuracy is high for the TOEFL iBT Speaking Test, but varying levels of rating accuracy and consistency exist across shift conditions. The raters working the shorter shifts or shorter sessions on average maintain greater rating productivity, accuracy, and consistency than those working longer shifts or sessions do. The raters working the 6-hour shift with three 2-hour sessions outperform those under other shift conditions in both rating accuracy and consistency.