Evaluating factual accuracy in complex data-to-text. (May 2023)

• Record Type: Journal Article
• Authors: Thomson, Craig; Reiter, Ehud; Sundararajan, Barkavi
• Is Part Of: Computer speech & language. Volume 80(2023)
• Publication Date: 2023-05
• Subjects: Natural Language Generation -- Complex data-to-text -- Evaluation -- Annotation -- Factual accuracy -- Neural data-to-text; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2023.101482
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract: It is essential that data-to-text Natural Language Generation (NLG) systems produce texts which are factually accurate. We examine accuracy issues in the task of generating summaries of basketball games, including what accuracy means in this context, how accuracy errors can be detected by human annotators, as well as the types of accuracy mistakes made by both neural NLG systems and human authors. We also look at the effectiveness of automatic metrics in measuring factual accuracy. Highlights: Factual accuracy problems limit the usefulness of neural solutions for complex data-to-text. Existing evaluation methods miss many of these errors, such as hallucination. We propose and evaluate a gold standard protocol for detecting factual errors in generated text. We show how this gold standard can be used to measure the efficacy of other methods. We also explore the common types of error in both human-authored and neural data-to-text systems.