Clinically relevant pretraining is all you need. (21st June 2021)

• Record Type: Journal Article
• Authors: Bear Don't Walk IV, Oliver J; Sun, Tony; Perotte, Adler; Elhadad, Noémie
• Is Part Of: Journal of the American Medical Informatics Association. Volume 28:Number 9(2021)
• Page Start: 1970
• Page End: 1976
• Publication Date: 2021-06-21
• Subjects: deep learning -- natural language processing -- transfer learning -- social determinants of health -- international classification of disease; Medical informatics -- Periodicals; Information Services -- Periodicals; Medical Informatics -- Periodicals; Médecine -- Informatique -- Périodiques; Informatica; Geneeskunde; Informatique médicale; Computer network resources; Electronic journals; 610.285
• DOI: 10.1093/jamia/ocab086
• Languages: English
• ISSNs: 1067-5027

Abstract

Abstract: Clinical notes present a wealth of information for applications in the clinical domain, but heterogeneity across clinical institutions and settings presents challenges for their processing. The clinical natural language processing field has made strides in overcoming domain heterogeneity, while pretrained deep learning models present opportunities to transfer knowledge from one task to another. Pretrained models have performed well when transferred to new tasks; however, it is not well understood if these models generalize across differences in institutions and settings within the clinical domain. We explore if institution or setting specific pretraining is necessary for pretrained models to perform well when transferred to new tasks. We find no significant performance difference between models pretrained across institutions and settings, indicating that clinically pretrained models transfer well across such boundaries. Given a clinically pretrained model, clinical natural language processing researchers may forgo the time-consuming pretraining step without a significant performance drop.