Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools. Issue 1 (1st January 2023)

• Record Type: Journal Article
• Authors: Appelhaus, Stefan; Werner, Susanne; Grosse, Pascal; Kämmer, Juliane E.
• Is Part Of: Medical education online. Volume 28:Issue 1(2023)
• Publication Date: 2023-01-01
• Subjects: Assessment -- formative assessment -- multiple-choice questions -- item reuse -- disclosure -- feedback -- progress testing; Medical education -- Periodicals; Education, Medical -- Periodicals; 610.71
• DOI: 10.1080/10872981.2022.2143298
• Languages: English
• ISSNs: 1087-2981

Abstract

ABSTRACT: Background: Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive. Methods: We retrospectively analysed difficulty and discrimination coefficients of 10, 148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests. Results: The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp 2 = 0.011. On average, reused, disclosed items were significantly easier ( M = 0.83) than reused, not disclosed items ( M = 0.71) and entirely new items ( M = 0.66), P < .001, ηp 2 = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp 2 = 0.002. Conclusions: Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.