A Natural-Language-Processing-Based Procedure for Generating Distractors for Multiple-Choice Questions. Issue 4 (December 2022)

• Record Type: Journal Article
• Authors: Baldwin, Peter; Mee, Janet; Yaneva, Victoria; Paniagua, Miguel; D'Angelo, Jean; Swygert, Kimberly; Clauser, Brian E.
• Is Part Of: Evaluation & the health professions. Volume 45:Issue 4(2022)
• Page Start: 327
• Page End: 340
• Publication Date: 2022-12
• Subjects: automatic item generation -- natural language processing -- item writing -- large-scale testing -- test development; Medical care -- Evaluation -- Periodicals; Medicine -- Ability testing -- Periodicals; 362.105
• DOI: 10.1177/01632787211046981
• Languages: English
• ISSNs: 0163-2787

Abstract

One of the most challenging aspects of writing multiple-choice test questions is identifying plausible incorrect response options—i.e., distractors. To help with this task, a procedure is introduced that can mine existing item banks for potential distractors by considering the similarities between a new item's stem and answer and the stems and response options for items in the bank. This approach uses natural language processing to measure similarity and requires a substantial pool of items for constructing the generating model. The procedure is demonstrated with data from the United States Medical Licensing Examination (USMLE®). For about half the items in the study, at least one of the top three system-produced candidates matched a human-produced distractor exactly; and for about one quarter of the items, two of the top three candidates matched human-produced distractors. A study was conducted in which a sample of system-produced candidates were shown to 10 experienced item writers. Overall, participants thought about 81% of the candidates were on topic and 56% would help human item writers with the task of writing distractors.