Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. (26th December 2018)

• Record Type: Journal Article
• Authors: Aaron, Skye; McEvoy, Dustin S; Ray, Soumi; Hickman, Thu-Trang T; Wright, Adam
• Is Part Of: Journal of the American Medical Informatics Association. Volume 26:Number 1(2019)
• Page Start: 37
• Page End: 43
• Publication Date: 2018-12-26
• Subjects: clinical decision support -- electronic health records -- alerts -- override reasons -- sentiment analysis; Medical informatics -- Periodicals; Information Services -- Periodicals; Medical Informatics -- Periodicals; Médecine -- Informatique -- Périodiques; Informatica; Geneeskunde; Informatique médicale; Computer network resources; Electronic journals; 610.285
• DOI: 10.1093/jamia/ocy139
• Languages: English
• ISSNs: 1067-5027

Abstract

Abstract: Background: Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective: Investigate whether user override comments can be used to discover malfunctions. Methods: We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: "broken, " "not broken, but could be improved, " and "not broken." We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth. Results: Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738. Discussion: Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts. Conclusion: Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.