Crowdsourcing for assessment items to support adaptive learning. (3rd August 2018)

• Record Type: Journal Article
• Authors: Tackett, Sean; Raymond, Mark; Desai, Rishi; Haist, Steven A.; Morales, Amy; Gaglani, Shiv; Clyman, Stephen G.
• Is Part Of: Medical teacher. Volume 40:Number 8(2018)
• Page Start: 838
• Page End: 841
• Publication Date: 2018-08-03
• Subjects: Medical education -- Periodicals; 610.711
• DOI: 10.1080/0142159X.2018.1490704
• Languages: English
• ISSNs: 0142-159X

Abstract

Abstract: Purpose: Adaptive learning requires frequent and valid assessments for learners to track progress against their goals. This study determined if multiple-choice questions (MCQs) "crowdsourced" from medical learners could meet the standards of many large-scale testing programs. Methods: Users of a medical education app (Osmosis.org, Baltimore, MD) volunteered to submit case-based MCQs. Eleven volunteers were selected to submit MCQs targeted to second year medical students. Two hundred MCQs were subjected to duplicate review by a panel of internal medicine faculty who rated each item for relevance, content accuracy, and quality of response option explanations. A sample of 121 items was pretested on clinical subject exams completed by a national sample of U.S. medical students. Results: Seventy-eight percent of the 200 MCQs met faculty reviewer standards based on relevance, accuracy, and quality of explanations. Of the 121 pretested MCQs, 50% met acceptable statistical criteria. The most common reasons for exclusion were that the item was too easy or had a low discrimination index. Conclusions: Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. Similar models may be adopted by students and educators to augment item pools that support adaptive learning.