Evaluating factual accuracy in complex data-to-text. (May 2023) - Thomson, Craig; Reiter, Ehud; Sundararajan, Barkavi
Abstract: It is essential that data-to-text Natural Language Generation (NLG) systems produce texts which are factually accurate. We examine accuracy issues in the task of generating summaries of basketball games, including what accuracy means in this context, how accuracy errors can be detected by human annotators, as well as the types of accuracy mistakes made by both neural NLG systems and human authors. We also look at the effectiveness of automatic metrics in measuring factual accuracy. Highlights: Factual accuracy problems limit the usefulness of neural solutions for complex data-to-text. Existing evaluation methods miss many of these errors, such as hallucination. We propose and evaluate a gold standard protocol for detecting factual errors in generated text. We show how this gold standard can be used to measure the efficacy of other methods. We also explore the common types of error in both human-authored and neural data-to-text systems.