Impact of hybrid supervision approaches on the performance of artificial intelligence for the classification of chest radiographs. (May 2020)

• Record Type: Journal Article
• Authors: Ellis, Ryan; Ellestad, Erik; Elicker, Brett; Hope, Michael D.; Tosun, Duygu
• Is Part Of: Computers in biology and medicine. Volume 120(2020)
• Publication Date: 2020-05
• Subjects: Chest radiographs -- Outpatient -- Hybrid supervision -- Two-view radiographs -- Attention mining -- Weak supervision -- Extra supervision; Medicine -- Data processing -- Periodicals; Biology -- Data processing -- Periodicals; 610.285
• DOI: 10.1016/j.compbiomed.2020.103699
• Languages: English
• ISSNs: 0010-4825

Abstract

Abstract: Purpose: To evaluate the impact of different supervision regimens on the training of artificial intelligence (AI) in the classification of chest radiographs as normal or abnormal in a moderately sized cohort of individuals more likely to be outpatients. Materials and methods: In a retrospective study, 7000 consecutive two-view chest radiographs obtained from 2012 to 2015 were labeled as normal or abnormal based on clinical reports. A convolutional neural network (CNN) was trained on this dataset and then evaluated with an unseen subset of 500 radiographs. Five different training approaches were tested: (1) weak supervision and four hybrid approaches combining weak supervision and extra supervision with annotation in (2) an unbalanced set of normal and abnormal cases, (3) a set of only abnormal cases, (4) a set of only normal cases, and (5) a balanced set of normal and abnormal cases. Standard binary classification metrics were assessed. Results: The weakly supervised model achieved an accuracy of 82%, but yielded 75 false negative cases, at a sensitivity of 70.0% and a negative predictive value (NPV) of 75.5%. Extra supervision increased NPV at the expense of the false positive rate and overall accuracy. Extra supervision with training using a balance of abnormal and normal radiographs resulted in the greatest increase in NPV (87.2%), improved sensitivity (92.8%), and reduced the number of false negatives by more than fourfold (18 compared to 75 cases). Conclusion: Extra supervision using a balance of annotated normal and abnormal cases applied to a weakly supervised model can minimize the number of false negative cases when classifying two-view chest radiographs. Further refinement of such hybrid training approaches for AI is warranted to refine models for practical clinical applications. Highlights: The cohort studied is unique in that only two view chest radiographs are included. A relatively high percentage of normal radiographs (40%) of more likely to be outpatients. Training regimen significantly influences the performance of CNNs in normal/abnormal classification of chest radiographs. Hybrid supervision schemes minimize false negative cases in normal/abnormal classification of chest radiographs.