Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. (25th May 2017)

• Record Type: Journal Article
• Authors: Wallace, Byron C; Noel-Storr, Anna; Marshall, Iain J; Cohen, Aaron M; Smalheiser, Neil R; Thomas, James
• Is Part Of: Journal of the American Medical Informatics Association. Volume 24:Number 6(2017:Nov.)
• Page Start: 1165
• Page End: 1168
• Publication Date: 2017-05-25
• Subjects: machine learning -- evidence-based medicine -- crowdsourcing -- human computation -- natural language processing; Medical informatics -- Periodicals; Information Services -- Periodicals; Medical Informatics -- Periodicals; Médecine -- Informatique -- Périodiques; Informatica; Geneeskunde; Informatique médicale; Computer network resources; Electronic journals; 610.285
• DOI: 10.1093/jamia/ocx053
• Languages: English
• ISSNs: 1067-5027

Abstract

Abstract: Objectives: Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML. Methods: We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise. Results: Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%–99% recall) with substantially less effort (we observed a reduction of around 60%–80%) than relying on manual screening alone. Conclusions: Hybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.