Overview of the seventh Dialog System Technology Challenge: DSTC7. (July 2020)

• Record Type: Journal Article
• Authors: D'Haro, Luis Fernando; Yoshino, Koichiro; Hori, Chiori; Marks, Tim K.; Polymenakos, Lazaros; Kummerfeld, Jonathan K.; Galley, Michel; Gao, Xiang
• Is Part Of: Computer speech & language. Volume 62(2020)
• Publication Date: 2020-07
• Subjects: Dialog System Technology Challenge -- end-to-end dialog systems -- Sentence Selection -- Natural Language Generation -- Audio Visual Scene-Aware Dialog; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101068
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: DSTC7: Dialog Challenge to build more robust and accurate end-to-end dialog systems. Track 1, Sentence selection for multiple domains, including variations where there are a large number of candidate options, and where the candidate set has zero, one, or multiple correct options. Track 2, Beyond Chitchat: Generation of informational responses grounded in external knowledge. Track 3, Audio visual scene-aware dialog systems to allow dynamic conversations about objects and events around users. Abstract: This paper provides detailed information about the seventh Dialog System Technology Challenge (DSTC7) and its three tracks aimed to explore the problem of building robust and accurate end-to-end dialog systems. In more detail, DSTC7 focuses on developing and exploring end-to-end technologies for the following three pragmatic challenges: (1) sentence selection for multiple domains, (2) generation of informational responses grounded in external knowledge, and (3) audio visual scene-aware dialog to allow conversations with users about objects and events around them. This paper summarizes the overall setup and results of DSTC7, including detailed descriptions of the different tracks, provided datasets and annotations, overview of the submitted systems and their final results. For Track 1, LSTM-based models performed best across both datasets, allowing teams to effectively handle task variants where no correct answer was present or when multiple paraphrases were included. For Track 2, RNN-based architectures augmented to incorporate facts by using two types of encoders: a dialog encoder and a fact encoder plus using attention mechanisms and a pointer-generator approach provided the best results. Finally, for Track 3, the best model used Hierarchical Attention mechanisms to combine the text and vision information obtaining a 22% better result than the baseline LSTM system for the human rating score. More than 220 participants were registered and about 40 teams participated in the final challenge. 32 scientific papers reporting the systems submitted to DSTC7, and 3 general technical papers for dialog technologies, were presented during the one-day wrap-up workshop at AAAI-19. During the workshop, we reviewed the state-of-the-art systems, shared novel approaches to the DSTC7 tasks, and discussed the future directions for the challenge (DSTC8).