Who needs explanation and when? Juggling explainable AI and user epistemic uncertainty. Issue 165 (September 2022)

• Record Type: Journal Article
• Authors: Jiang, Jinglu; Kahai, Surinder; Yang, Ming
• Is Part Of: International journal of human-computer studies. Issue 165(2022)
• Publication Date: 2022-09
• Subjects: AI explainability -- AI advice acceptance -- Medical AI -- Human-AI interaction -- Experiment; Human-machine systems -- Periodicals; Systems engineering -- Periodicals; Human engineering -- Periodicals; Human engineering; Human-machine systems; Systems engineering; Periodicals; Electronic journals; 004.019
• DOI: 10.1016/j.ijhcs.2022.102839
• Languages: English
• ISSNs: 1071-5819

Abstract

Highlights: AI explainability (XAI) plays opposing roles on human decision-making. Users' epistemic uncertainty matters when understanding XAI's impacts. As users' uncertainty increases, only providing prediction rationale is beneficial. Providing alternative advice may hinder users' advice acceptance. Showing prediction confidence scores may also hinder users' advice acceptance. Abstract: In recent years, AI explainability (XAI) has received wide attention. Although XAI is expected to play a positive role in decision-making and advice acceptance, various opposing effects have also been found. The opposing effects of XAI highlight the critical role of context, especially human factors, in understanding XAI's impacts. This study investigates the effects of providing three types of post-hoc explanations (alternative advice, prediction confidence scores, and prediction rationale) on two context-specific user decision-making outcomes (AI advice acceptance and advice adoption). Our field experiment results show that users' epistemic uncertainty matters when understanding XAI's impacts. As users' epistemic uncertainty increases, only providing prediction rationale is beneficial, whereas providing alternative advice and showing prediction confidence scores may hinder users' advice acceptance. Our study contributes to the emerging literature on the human aspects of XAI by clarifying XAI and showing that XAI may not always be desirable. It also contributes by highlighting the importance of considering user profiles when predicting XAI's impacts, designing XAI, and providing professional services with AI.