OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases. (27th October 2020)

• Record Type: Journal Article
• Authors: Kumar, Andre; Aikens, Rachael C; Hom, Jason; Shieh, Lisa; Chiang, Jonathan; Morales, David; Saini, Divya; Musen, Mark; Baiocchi, Michael; Altman, Russ; Goldstein, Mary K; Asch, Steven; Chen, Jonathan H
• Is Part Of: Journal of the American Medical Informatics Association. Volume 27:Number 12(2020)
• Page Start: 1850
• Page End: 1859
• Publication Date: 2020-10-27
• Subjects: informatics -- clinical care -- clinical decision support -- recommender systems -- human computer interaction -- usability testing -- collaborative filtering -- order sets -- electronic medical records -- clinical provider order entry; Medical informatics -- Periodicals; Information Services -- Periodicals; Medical Informatics -- Periodicals; Médecine -- Informatique -- Périodiques; Informatica; Geneeskunde; Informatique médicale; Computer network resources; Electronic journals; 610.285
• DOI: 10.1093/jamia/ocaa190
• Languages: English
• ISSNs: 1067-5027

Abstract

Abstract: Objective: To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases. Materials and Methods: 43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses. Results: Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows. Discussion: User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments. Conclusions: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.