Human evaluation of automatically generated text: Current trends and best practice guidelines. (May 2021)

• Record Type: Journal Article
• Authors: van der Lee, Chris; Gatt, Albert; van Miltenburg, Emiel; Krahmer, Emiel
• Is Part Of: Computer speech & language. Volume 67(2021)
• Publication Date: 2021-05
• Subjects: Natural Language Generation -- Human evaluation -- Recommendations -- Literature review -- Open science -- Ethics; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101151
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: The current paper provides an overview of human evaluation practices in NLG. The current paper gives an overview of the steps necessary to undertake a human evaluation study. Building on findings from NLG, but also statistics and the behavioral sciences, the current paper provides a set of recommendations and best practices for human evaluation in NLG. Abstract: Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how (mostly intrinsic) human evaluation is currently conducted and presents a set of best practices, grounded in the literature. These best practices are also linked to the stages that researchers go through when conducting an evaluation research (planning stage; execution and release stage), and the specific steps in these stages. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.