A Simulation‐Based Method for Finding the Optimal Number of Options for Multiple‐Choice Items on a Test. Issue 1 (14th June 2018) - Guo, Hongwen; Zu, Jiyun; Kyllonen, Patrick
Abstract: For a multiple‐choice test under development or redesign, it is important to choose the optimal number of options per item so that the test possesses the desired psychometric properties. On the basis of available data for a multiple‐choice assessment with 8 options, we evaluated the effects of changing the number of options on test properties (difficulty, reliability, and score comparability) using simulation. Using 2 criteria (low frequency and poor discrimination) to remove nonfunctioning options and 2 schemes (random and educated guessing) to model hypothetical response behavior for the removed options, we found that decreasing the number of options (from 8) created an easier test form but that a test form with reduced options could be more reliable if low‐discriminating options were removed and an educated guessing strategy were assumed. We present a rationale for the optimal number of options for this test being approximately 5, which would result in a shorter test while preserving its psychometric quality. Simulation methods discussed in this report could be applied to any test to compare the effects of changing the number of options. Abstract : Report Number: ETS RR–18‐22