A Response to Holster and Lake Regarding Guessing and the Rasch Model. Issue 1 (2nd January 2017)

• Record Type: Journal Article
• Authors: Stewart, Jeffrey; McLean, Stuart; Kramer, Brandon
• Is Part Of: Language assessment quarterly. Volume 14:Issue 1(2017)
• Page Start: 69
• Page End: 74
• Publication Date: 2017-01-02
• Subjects: Language and languages -- Ability testing -- Periodicals; Language and languages -- Examinations -- Periodicals; 418.0076
• DOI: 10.1080/15434303.2016.1262377
• Languages: English
• ISSNs: 1543-4303

Abstract

ABSTRACT: Stewart questioned vocabulary size estimation methods proposed by Beglar and Nation for the Vocabulary Size Test, further arguing Rasch mean square (MSQ) fit statistics cannot determine the proportion of random guesses contained in the average learner's raw score, because the average value will be near 1 by design. He illustrated this by demonstrating this is true even of entirely random data. Holster and Lake appear to misinterpret this as a claim that Rasch analyses cannot distinguish random data from real responses. To test this, they compare real data to random and note that, predictably, the statistic easily distinguishes the two and that reliability for random data is near zero. However, while certainly true, this fact is not relevant to Stewart's argument that multiple-choice options inflate the test's size estimates and that MSQ fit statistics cannot be used to detect this. We further illustrate this by showing real data retains average MSQ values near 1, even when unknown items skipped by learners are imputed with random guesses. Furthermore, the imputed data do not exhibit "problematic guessing" under Holster & Lake's own criteria, despite size inflation under Beglar and Nation's suggested scoring. We conclude by discussing uses of the 3PL model.