Raters' measurement of test-task authentic engagement in L2 oral-performance assessment: An exploration of scale development. (June 2020)

• Record Type: Journal Article
• Authors: Burton, J. Dylan
• Is Part Of: System. Volume 90(2020)
• Publication Date: 2020-06
• Subjects: Language testing -- Authenticity -- Response processes -- Speaking assessment -- Rater cognition; Language and languages -- Study and teaching -- Periodicals; Langage et langues -- Étude et enseignement -- Périodiques; Electronic journals; 407
• DOI: 10.1016/j.system.2020.102233
• Languages: English
• ISSNs: 0346-251X

Abstract

Abstract: In second language (L2) performance testing, an underlying assumption is that test takers engage with the test task and respond in authentic, spontaneous L2 speech. By doing so they tap into cognitive and psychological processes that resemble those used in the real world. Raters may be sensitive to whether test takers are authentically engaged; however, no empirical studies have looked at this aspect of rater judgement, perhaps because there is no concrete way to measure it. In this paper, I report on the development and validation of an instrument, with nine semantic differentials, to measure test takers' authentic test-task engagement. Fifty-eight raters used it to rate ten L2 English speech samples categorized into three levels of authenticity. The raters were consistent and classified the samples into three significantly different levels of authentic engagement, as expected. Principal components analysis revealed two underlying instrument components: authenticity and proficiency. Relevance (off-/on-topic) unexpectedly did not correlate strongly with authenticity. The scale may be useful for helping applied linguists understand raters' perceptions of authentic test-task engagement. Based on the results, rater training programs may need to clarify more what raters should do when they come across what they perceive as inauthentic engagement.