A survey of methods for revealing and overcoming weaknesses of data-driven Natural Language Understanding. (22nd January 2023)

• Record Type: Journal Article
• Authors: Schlegel, Viktor; Nenadic, Goran; Batista-Navarro, Riza
• Is Part Of: Natural language engineering. Volume 29:Number 1(2023)
• Page Start: 1
• Page End: 31
• Publication Date: 2023-01-22
• Subjects: Natural Language Understanding -- Deep learning -- Machine reading comprehension -- Textual entailment -- Dataset artefacts; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324922000171
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Recent years have seen a growing number of publications that analyse Natural Language Understanding (NLU) datasets for superficial cues, whether they undermine the complexity of the tasks underlying those datasets and how they impact those models that are optimised and evaluated on this data. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets and the methods proposed to reveal and alleviate those weaknesses for the English language. We summarise and discuss the findings and conclude with a set of recommendations for possible future research directions. We hope that it will be a useful resource for researchers who propose new datasets to assess the suitability and quality of their data to evaluate various phenomena of interest, as well as those who propose novel NLU approaches, to further understand the implications of their improvements with respect to their model's acquired capabilities.