Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care and institutions. Issue 4 (30th December 2022)

• Record Type: Journal Article
• Authors: Chapman, Alec B; Peterson, Kelly S; Rutter, Elizabeth; Nevers, Mckenna; Zhang, Mingyuan; Ying, Jian; Jones, Makoto; Classen, David; Jones, Barbara
• Is Part Of: JAMIA open. Volume 5:Issue 4(2022)
• Publication Date: 2022-12-30
• Subjects: natural language processing -- pneumonia -- diagnostic errors -- quality indicators -- health care -- electronic health records; Medical informatics -- Periodicals; 610.285
• DOI: 10.1093/jamiaopen/ooac114
• Languages: English
• ISSNs: 2574-2531

Abstract

Abstract: Objective: To evaluate the feasibility, accuracy, and interoperability of a natural language processing (NLP) system that extracts diagnostic assertions of pneumonia in different clinical notes and institutions. Materials and Methods: A rule-based NLP system was designed to identify assertions of pneumonia in 3 types of clinical notes from electronic health records (EHRs): emergency department notes, radiology reports, and discharge summaries. The lexicon and classification logic were tailored for each note type. The system was first developed and evaluated using annotated notes from the Department of Veterans Affairs (VA). Interoperability was assessed using data from the University of Utah (UU). Results: The NLP system was comprised of 782 rules and achieved moderate-to-high performance in all 3 note types in VA (precision/recall/f1: emergency = 88.1/86.0/87.1; radiology = 71.4/96.2/82.0; discharge = 88.3/93.0/90.1). When applied to UU data, performance was maintained in emergency and radiology but decreased in discharge summaries (emergency = 84.7/94.3/89.3; radiology = 79.7/100.0/87.9; discharge = 65.5/92.7/76.8). Customization with 34 additional rules increased performance for all note types (emergency = 89.3/94.3/91.7; radiology = 87.0/100.0/93.1; discharge = 75.0/95.1/83.4). Conclusion: NLP can be used to accurately identify the diagnosis of pneumonia across different clinical settings and institutions. A limited amount of customization to account for differences in lexicon, clinical definition of pneumonia, and EHR structure can achieve high accuracy without substantial modification. Lay Summary: Electronic health records (EHRs) contain valuable data for studying patient diagnoses and quality of care. Much of this information is stored in free-text narratives and must be extracted using natural language processing (NLP). NLP systems are often designed for particular datasets in individual institutions, leading to limited reusability and generalizability of these tools. In this work, we developed and evaluated an NLP system for extracting assertions of pneumonia across clinical settings. We first developed and evaluated the system with 3 types of clinical texts from the Department of Veterans Affairs (VA): emergency department narratives, chest imaging reports, and discharge summaries. To assess interoperability, we then evaluated the system in the UU. The NLP system performed well in all 3 clinical settings in the VA. When testing in UU, performance decreased slightly due to differences in EHR structure but improved after a small amount of customization. These results support that the system can extract diagnoses of pneumonia across different clinical settings and healthcare institutions. Future work will apply this tool to create generalizable measures of diagnostic excellence in pneumonia.