Reporting practices of rater reliability in interpreting research: A mixed-methods review of 14 journals (2004–2014). (7th December 2016)

• Record Type: Journal Article
• Authors: Han, Chao
• Is Part Of: Journal of research design and statistics in linguistics and communication science. Volume 3:Number 1(2016)
• Page Start: 49
• Page End: 75
• Publication Date: 2016-12-07
• Subjects: consensus -- mixed-methods -- interpreting studies -- consistency -- rater reliability; Linguistics -- Research -- Periodicals; Linguistics -- Statistical methods -- Periodicals; Mathematical linguistics -- Periodicals; 410.72
• DOI: 10.1558/jrds.29622
• Languages: English
• ISSNs: 2052-417X

Abstract

Abstract : The issue addressed in this study is the reporting practices of rater reliability in interpreting research (IR), given that the use of raters as a method of measurement is a commonplace in IR, and that little is known about to what extent and how rater reliability estimates (RREs) have been reported. Drawing upon 447 articles from 14 translation and interpreting journals (2004--2014), this mixed-methods study attempts to gain quantitative and qualitative insights into the reporting practices. Data analysis reveals that: 1) almost 90% of the articles that needed to report RREs failed to do so; 2) potential problems emerged from those articles that reported RREs: lack of distinction between rater consensus and consistency, underreporting, misinterpretation and misuse of RREs, and lack of justification for the use of rater-generated measurements for subsequent data analysis. These findings highlight an urgent need for increased author awareness of reporting appropriate RREs in IR.