Evaluating the Stability of Test Score Means for the TOEIC® Speaking and Writing Tests. Issue 1 (11th October 2017)

• Record Type: Journal Article
• Authors: Qu, Yanxuan; Huo, Yan; Chan, Eric; Shotts, Matthew
• Is Part Of: ETS research report series. Issue 1(2017)
• Page Start: 1
• Page End: 8
• Publication Date: 2017-10-11
• Subjects: Harmonic regression -- ANOVA -- seasonality -- scale stability -- quality control; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12180
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract : For educational tests, it is critical to maintain consistency of score scales and to understand the sources of variation in score means over time. This practice helps to ensure that interpretations about test takers' abilities are comparable from one administration (or one form) to another. This study examines the consistency of reported scores for the TOEIC ® Speaking and Writing tests using statistical procedures. Specifically, the stability of the TOEIC Speaking score means from 431 forms administered in a 3‐year period was evaluated using harmonic regression, and the stability of TOEIC Writing score means from 66 forms administered in a 3‐year period was evaluated using analysis of variance. Results indicated that the fluctuations in the TOEIC Speaking or Writing score means mainly reflect changes in test takers' overall English speaking or writing ability levels instead of score inaccuracies. For both speaking and writing test scores, a large proportion of the variation in score means can be explained by seasonality (the rise or fall of score means associated with specific times of the year) and test takers' demographic information, which have been shown to be related to test‐taker ability. As a result, this finding provides evidence for the consistency of the TOEIC Speaking and Writing score scales across forms. Abstract : Report Number: ETS RR‐17–50