Automatic evaluation of end-to-end dialog systems with adequacy-fluency metrics. (May 2019)

• Record Type: Journal Article
• Authors: D'Haro, Luis Fernando; Banchs, Rafael E.; Hori, Chiori; Li, Haizhou
• Is Part Of: Computer speech & language. Volume 55(2019)
• Page Start: 200
• Page End: 215
• Publication Date: 2019-05
• Subjects: Automatic evaluation metrics -- dialog systems -- DSTC -- adequacy and fluency; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2018.12.004
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Automatic metric for evaluating natural language generated sentences for dialog systems. Integration of adequacy and fluency information to jointly evaluate semantic and syntactic information. Higher correlation (up to 12.8%) with human evaluations than the best current objective metrics at a system and sentence level. Abstract: End-to-end dialog systems are gaining interest due to the recent advances of deep neural networks and the availability of large human–human dialog corpora. However, in spite of being of fundamental importance to systematically improve the performance of this kind of systems, automatic evaluation of the generated dialog utterances is still an unsolved problem. Indeed, most of the proposed objective metrics shown low correlation with human evaluations. In this paper, we evaluate a two-dimensional evaluation metric that is designed to operate at sentence level, which considers the syntactic and semantic information carried along the answers generated by an end-to-end dialog system with respect to a set of references. The proposed metric, when applied to outputs generated by the systems participating in track 2 of the DSTC-6 challenge, shows a higher correlation with human evaluations (up to 12.8% relative improvement at the system level) than the best of the alternative state-of-the-art automatic metrics currently available.