Monitoring the performance of human and automated scores for spoken responses. (January 2018)

• Record Type: Journal Article
• Authors: Wang, Zhen; Zechner, Klaus; Sun, Yu
• Is Part Of: Language testing. Volume 35:Number 1(2018)
• Page Start: 101
• Page End: 120
• Publication Date: 2018-01
• Subjects: Automated speech scoring -- language assessment -- score monitoring -- Shewhart control chart -- human and machine scoring -- reliability; Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 407.6
• DOI: 10.1177/0265532216679451
• Languages: English
• ISSNs: 0265-5322

Abstract

As automated scoring systems for spoken responses are increasingly used in language assessments, testing organizations need to analyze their performance, as compared to human raters, across several dimensions, for example, on individual items or based on subgroups of test takers. In addition, there is a need in testing organizations to establish rigorous procedures for monitoring the performance of both human and automated scoring processes during operational administrations. This paper provides an overview of the automated speech scoring system SpeechRater SM and how to use charts and evaluation statistics to monitor and evaluate automated scores and human rater scores of spoken constructed responses.