U"AI" Testing: User Interface and Usability Testing of a Chest X-ray AI Tool in a Simulated Real-World Workflow. Issue 2 (May 2023)

• Record Type: Journal Article
• Authors: Lam Shin Cheung, Jeffrey; Ali, Amna; Abdalla, Mohamed; Fine, Benjamin
• Is Part Of: Canadian Association of Radiologists journal. Volume 74:Issue 2(2023)
• Page Start: 314
• Page End: 325
• Publication Date: 2023-05
• Subjects: artificial intelligence -- x-ray -- usability -- efficiency -- radiology; Radiology, Medical -- Periodicals; Radiology, Medical -- Canada -- Periodicals; 616.0757
• DOI: 10.1177/08465371221131200
• Languages: English
• ISSNs: 0846-5371

Abstract

Purpose: To observe interactions of practicing radiologists with a chest x-ray AI tool and evaluate its usability and impact on workflow efficiency.Methods: Using a simulated clinical workflow and remote multi-monitor screensharing, we prospectively assessed the interactions of 10 staff radiologists (5–33 years of experience) with a PACS-embedded, regulatory-approved chest x-ray AI tool. Qualitatively, we collected feedback using a think-aloud method and post-testing semi-structured interview; transcript themes were categorized by: (1) AI tool features, (2) deployment considerations, and (3) broad human-AI interactions. Quantitatively, we used time-stamped video recordings to compare reporting and decision-making efficiency with and without AI assistance.Results: For AI tool features, radiologists appreciated the simple binary classification (normal vs abnormal) and found the heatmap essential to understand what the AI considered abnormal; users were uncertain of how to interpret confidence values. Regarding deployment considerations, radiologists thought the tool would be especially helpful for identifying subtle diagnoses; opinions were mixed on whether the tool impacted perceived efficiency, accuracy, and confidence. Considering general human-AI interactions, radiologists shared concerns about automation bias especially when relying on an automated triage function. Regarding decision-making and workflow efficiency, participants began dictating 5 seconds later (42% increase, P = .02) and took 14 seconds longer to complete cases (33% increase, P = .09) with AI assistance.Conclusions: Radiologist usability testing provided insights into effective AI tool features, deployment considerations, and human-AI interactions that can guide successful AI deployment. Early AI adoption may increase radiologists' decision-making and total reporting time but improves with experience.