Evaluating the Stability of Test Score Means for the TOEIC® Speaking and Writing Tests. Issue 1 (11th October 2017) - Qu, Yanxuan; Huo, Yan; Chan, Eric; Shotts, Matthew
Abstract : For educational tests, it is critical to maintain consistency of score scales and to understand the sources of variation in score means over time. This practice helps to ensure that interpretations about test takers' abilities are comparable from one administration (or one form) to another. This study examines the consistency of reported scores for the TOEIC ® Speaking and Writing tests using statistical procedures. Specifically, the stability of the TOEIC Speaking score means from 431 forms administered in a 3‐year period was evaluated using harmonic regression, and the stability of TOEIC Writing score means from 66 forms administered in a 3‐year period was evaluated using analysis of variance. Results indicated that the fluctuations in the TOEIC Speaking or Writing score means mainly reflect changes in test takers' overall English speaking or writing ability levels instead of score inaccuracies. For both speaking and writing test scores, a large proportion of the variation in score means can be explained by seasonality (the rise or fall of score means associated with specific times of the year) and test takers' demographic information, which have been shown to be related to test‐taker ability. As a result, this finding provides evidence for the consistency of the TOEIC Speaking and Writing score scales across forms. Abstract : Report Number: ETS RR‐17–50