Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews. Issue 1 (30th March 2022)

• Record Type: Journal Article
• Authors: Schneider, Jodi; Hoang, Linh; Kansara, Yogeshwar; Cohen, Aaron M; Smalheiser, Neil R
• Is Part Of: JAMIA open. Volume 5:Issue 1(2022)
• Publication Date: 2022-03-30
• Subjects: RCT Tagger -- systematic review automation -- randomized controlled trials -- information retrieval; Medical informatics -- Periodicals; 610.285
• DOI: 10.1093/jamiaopen/ooac015
• Languages: English
• ISSNs: 2574-2531

Abstract

Abstract: Objectives: To produce a systematic review (SR), reviewers typically screen thousands of titles and abstracts of articles manually to find a small number which are read in full text to find relevant articles included in the final SR. Here, we evaluate a proposed automated probabilistic publication type screening strategy applied to the randomized controlled trial (RCT) articles (i.e., those which present clinical outcome results of RCT studies) included in a corpus of previously published Cochrane reviews. Materials and Methods: We selected a random subset of 558 published Cochrane reviews that specified RCT study only inclusion criteria, containing 7113 included articles which could be matched to PubMed identifiers. These were processed by our automated RCT Tagger tool to estimate the probability that each article reports clinical outcomes of a RCT. Results: Removing articles with low predictive scores P < 0.01 eliminated 288 included articles, of which only 22 were actually typical RCT articles, and only 18 were actually typical RCT articles that MEDLINE indexed as such. Based on our sample set, this screening strategy led to fewer than 0.05 relevant RCT articles being missed on average per Cochrane SR. Discussion: This scenario, based on real SRs, demonstrates that automated tagging can identify RCT articles accurately while maintaining very high recall. However, we also found that even SRs whose inclusion criteria are restricted to RCT studies include not only clinical outcome articles per se, but a variety of ancillary article types as well. Conclusions: This encourages further studies learning how best to incorporate automated tagging of additional publication types into SR triage workflows. Lay Summary: In medical research, treatments are compared using randomized controlled trials (RCTs). To identify safe and effective treatments, systematic reviews are carried out which identify and analyze articles that present the results from multiple trials on the same question. We have previously created a tool called RCT Tagger, which identifies RCT articles automatically. The present paper verifies that the RCT Tagger would have identified nearly all relevant articles when applied to previously published systematic reviews. This suggests that people writing systematic reviews should consider automated publication type tagging as part of their screening for relevant articles.