Applying machine classifiers to update searches: Analysis from two case studies. (25th November 2021)

• Record Type: Journal Article
• Authors: Stansfield, Claire; Stokes, Gillian; Thomas, James
• Is Part Of: Research synthesis methods. Volume 13:Number 1(2022)
• Page Start: 121
• Page End: 133
• Publication Date: 2021-11-25
• Subjects: information retrieval -- supervised machine learning -- systematic reviews as topic -- update search; Research -- Methodology -- Periodicals; Research -- Statistical methods -- Periodicals; 507.2
• DOI: 10.1002/jrsm.1537
• Languages: English
• ISSNs: 1759-2879

Abstract

Abstract: Manual screening of citation records could be reduced by using machine classifiers to remove records of very low relevance. This seems particularly feasible for update searches, where a machine classifier can be trained from past screening decisions. However, feasibility is unclear for broad topics. We evaluate the performance and implementation of machine classifiers for update searches of public health research using two case studies. The first study evaluates the impact of using different sets of training data on classifier performance, comparing recall and screening reduction with a manual screening 'gold standard'. The second study uses screening decisions from a review to train a classifier that is applied to rank the update search results. A stopping threshold was applied in the absence of a gold standard. Time spent screening titles and abstracts of different relevancy‐ranked records was measured. Results: Study one: Classifier performance varies according to the training data used; all custom‐built classifiers had a recall above 93% at the same threshold, achieving screening reductions between 41% and 74%. Study two: applying a classifier provided a solution for tackling a large volume of search results from the update search, and screening volume was reduced by 61%. A tentative estimate indicates over 25 h screening time was saved. In conclusion, custom‐built machine classifiers are feasible for reducing screening workload from update searches across a range of public health interventions, with some limitation on recall. Key considerations include selecting a training dataset, agreeing stopping thresholds and processes to ensure smooth workflows.