A Simulation‐Based Method for Finding the Optimal Number of Options for Multiple‐Choice Items on a Test. Issue 1 (14th June 2018)

• Record Type: Journal Article
• Authors: Guo, Hongwen; Zu, Jiyun; Kyllonen, Patrick
• Is Part Of: ETS research report series. Issue 1(2018)
• Page Start: 1
• Page End: 17
• Publication Date: 2018-06-14
• Subjects: Test reliability -- nonfunctioning options -- item discrimination; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12209
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: For a multiple‐choice test under development or redesign, it is important to choose the optimal number of options per item so that the test possesses the desired psychometric properties. On the basis of available data for a multiple‐choice assessment with 8 options, we evaluated the effects of changing the number of options on test properties (difficulty, reliability, and score comparability) using simulation. Using 2 criteria (low frequency and poor discrimination) to remove nonfunctioning options and 2 schemes (random and educated guessing) to model hypothetical response behavior for the removed options, we found that decreasing the number of options (from 8) created an easier test form but that a test form with reduced options could be more reliable if low‐discriminating options were removed and an educated guessing strategy were assumed. We present a rationale for the optimal number of options for this test being approximately 5, which would result in a shorter test while preserving its psychometric quality. Simulation methods discussed in this report could be applied to any test to compare the effects of changing the number of options. Abstract : Report Number: ETS RR–18‐22