ExpMRC: explainability evaluation for machine reading comprehension. Issue 4 (April 2022)

• Record Type: Journal Article
• Authors: Cui, Yiming; Liu, Ting; Che, Wanxiang; Chen, Zhigang; Wang, Shijin
• Is Part Of: Heliyon. Volume 8:Issue 4(2022)
• Publication Date: 2022-04
• Subjects: Machine reading comprehension -- Explainable artificial intelligence -- Natural language processing; Research -- Periodicals; Medical sciences -- Periodicals; Natural history -- Periodicals; Social sciences -- Periodicals; Earth sciences -- Periodicals; Physical sciences -- Periodicals; 507.2
• DOI: 10.1016/j.heliyon.2022.e09290
• Languages: English
• ISSNs: 2405-8440

Abstract

Abstract: Achieving human-level performance on some Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the textual explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE +, and C 3, with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art PLMs to build baseline systems and adopt various unsupervised approaches to extract both answer and evidence spans without human-annotated evidence spans. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources (data and baselines) are available through https://github.com/ymcui/expmrc . Abstract : Machine reading comprehension; Explainable artificial intelligence; Natural language processing