"Think aloud" and "Near live" usability testing of two complex clinical decision support tools. (October 2017)

• Record Type: Journal Article
• Authors: Richardson, Safiya; Mishuris, Rebecca; O'Connell, Alexander; Feldstein, David; Hess, Rachel; Smith, Paul; McCullagh, Lauren; McGinn, Thomas; Mann, Devin
• Is Part Of: International journal of medical informatics. Volume 106(2017)
• Page Start: 1
• Page End: 8
• Publication Date: 2017-10
• Subjects: CDS clinical decision support -- EHR electronic health record -- GAS group A Streptococcus -- SUS System Usability Scale; Usability -- Usability testing -- User experience -- Clinical decision support -- Health informatics -- Provider adoption -- Workflow; Medical informatics -- Periodicals; Information science -- Periodicals; Computers -- Periodicals; Medical technology -- Periodicals; Medical Informatics -- Periodicals; Technology, Medical -- Periodicals; Computers; Information science; Medical informatics; Medical technology; Electronic journals; Periodicals; Electronic journals; 610.285
• DOI: 10.1016/j.ijmedinf.2017.06.003
• Languages: English
• ISSNs: 1386-5056

Abstract

Highlights: "Think Aloud" and "Near Live" usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. "Near Live" testing was helpful for eliciting key barriers to provider workflow and adoption. During "Near Live" testing participants were more critical of the tools' usefulness. Usability testing helps decision support reach its potential to improve health outcomes. Abstract: Objectives: Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud". Methods: This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods. Results: "Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit. Conclusions: These complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.