Effectiveness of Item Response Theory (IRT) Proficiency Estimation Methods Under Adaptive Multistage Testing. Issue 1 (19th May 2015)

• Record Type: Journal Article
• Authors: Kim, Sooyeon; Moses, Tim; Yoo, Hanwook Henry
• Is Part Of: ETS research report series. Issue 1(2015:Jun.)
• Page Start: 1
• Page End: 19
• Publication Date: 2015-05-19
• Subjects: Multistage testing -- proficiency estimator -- item response theory -- routing accuracy; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12057
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract : The purpose of this inquiry was to investigate the effectiveness of item response theory (IRT) proficiency estimators in terms of estimation bias and error under multistage testing (MST). We chose a 2‐stage MST design in which 1 adaptation to the examinees' ability levels takes place. It includes 4 modules (1 at Stage 1, 3 at Stage 2) and 3 paths (low, middle, and high). When creating 2‐stage MST panels (i.e., forms), we manipulated 2 assembly conditions in each module, such as difficulty level and module length, to see if any interaction existed between IRT estimation methods and MST panel designs. For each panel, we compared the accuracy of examinees' proficiency levels derived from 7 IRT proficiency estimators. We found that the choice of Bayesian (prior) and non‐Bayesian (no prior) estimators was of more practical significance than the choice of number‐correct versus item‐pattern scoring. For the extreme proficiency levels, the decrease in standard error compensated for the increase in bias in the Bayesian estimates, resulting in smaller total error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for the extreme proficiency level examinees. The impact of misrouting at Stage 1 was minimal under the MST design used in this study. Abstract : Report Number: ETS RR–15–11