Working with sparse data in rated language tests: Generalizability theory applications. (April 2017)

• Record Type: Journal Article
• Authors: Lin, Chih-Kai
• Is Part Of: Language testing. Volume 34:Number 2(2017)
• Page Start: 271
• Page End: 289
• Publication Date: 2017-04
• Subjects: Generalizability theory -- Monte Carlo simulation -- performance-based assessment -- reliability -- sparse-rated data; Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 407.6
• DOI: 10.1177/0265532216638890
• Languages: English
• ISSNs: 0265-5322

Abstract

Sparse-rated data are common in operational performance-based language tests, as an inevitable result of assigning examinee responses to a fraction of available raters. The current study investigates the precision of two generalizability-theory methods (i.e., the rating method and the subdividing method) specifically designed to accommodate the technical complexity involved in estimating score reliability from sparse-rated data. Examining the estimation precision of reliability is of great importance because the utility of any performance-based language test depends on its reliability. Results suggest that when some raters are expected to have greater score variability than other raters (e.g., a mixture of novice and experienced raters being deployed in a rating session), the sub-dividing method is recommended as it yields more precise reliability estimates. When all raters are expected to exhibit similar variability in their scoring, both the rating and sub-dividing methods are equally precise in estimating score reliability, and the rating method is recommended for operational use, as it is easier to implement in practice. Informed by these methodological results, the current study also demonstrates a step-by-step analysis for investigating the score reliability from sparse-rated data taken from a large-scale English speaking proficiency test. Implications for operational performance-based language tests are discussed.