Investigating Score Dependability in English/Chinese Interpreter Certification Performance Testing: A Generalizability Theory Approach. Issue 3 (2nd July 2016)

• Record Type: Journal Article
• Authors: Han, Chao
• Is Part Of: Language assessment quarterly. Volume 13:Issue 3(2016)
• Page Start: 186
• Page End: 201
• Publication Date: 2016-07-02
• Subjects: Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 418.0076
• DOI: 10.1080/15434303.2016.1211132
• Languages: English
• ISSNs: 1543-4303

Abstract

ABSTRACT: As a property of test scores, reliability/dependability constitutes an important psychometric consideration, and it underpins the validity of measurement results. A review of interpreter certification performance tests (ICPTs) reveals that (a) although reliability/dependability checking has been recognized as an important concern, its theoretical importance overshadows the operational efforts to measure score reliability/dependability; and (b) although multiple sources of measurement error can contribute variability to total score variance, rater effects have been regarded as the only source of error and modeled through classical test theory in the form of interrater reliability coefficients. Therefore, the study was initiated to investigate score dependability for a rater-mediated assessment of English-to-Chinese simultaneous interpreting, using generalizability theory. The results show the following: (a) the information completeness (InfoCom) ratings were more dependable than the ratings of fluency of delivery (FluDel) and target-language quality (TLQual); (b) the addition of tasks was more effective in raising dependability for InfoCom than the use of extra raters, but the effect was reversed for FluDel and TLQual; and (c) three different weighting schemes produced small variations in the composite-score dependability, but the InfoCom rating scale accounted for the largest proportions of the composite universe-score variances. These results are discussed for English/Chinese ICPTs.