Learning Multi-Level Information for Dialogue Response Selection by Highway Recurrent Transformer. (September 2020)

• Record Type: Journal Article
• Authors: Chiang, Ting-Rui; Huang, Chao-Wei; Su, Shang-Yu; Chen, Yun-Nung
• Is Part Of: Computer speech & language. Volume 63(2020)
• Publication Date: 2020-09
• Subjects: Response selection -- Transformer -- Attention mechanism -- Dialogue -- DSTC; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2020.101073
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: A new variant of attention mechanisms focuses on modeling cross-sentence attention. A novel model integrates highway attention in Transformer for modeling dialogues. Our model is capable of modeling complex dialogue-level information. The results on two response selection datasets show consistent performance. Graphical Abstract: Image, graphical abstract Abstract: With increasing research interests in dialogue modeling, there is an emerging branch that formulates this task as next sentence selection, where given the partial dialogue context, the goal is to determine the most probable next sentence. To model natural language information, recurrent models have been applied to sequence modeling and shown promising results in various NLP tasks (Sutskever et al., 2014 ). Recently, the Transformer (Vaswani et al., 2017) has advanced modeling semantics for natural language sentences via attention, achieving improvement for sequence modeling. However, the Transformer focuses on modeling the intra-sentence attention but ignores inter-sentence information. In terms of dialogue modeling, the cross-sentence information is salient to understand dialogue content, so that the response selection can be better determined. Therefore, this paper proposes a novel attention mechanism based on multi-head attention, called highway attention, in order to allow the model to pass information through multiple sentences, and then builds a recurrent model based on the Transformer and the proposed highway attention. We call this model Highway Recurrent Transformer . This model focuses on not only intra-sentence dependency, but also inter-sentence dependency in the structure of dialogues. Experiments on the response selection task of the seventh Dialog System Technology Challenge (DSTC7) demonstrate that the proposed Highway Recurrent Transformer is capable of modeling both utterance-level and dialogue-level information for achieving better performance than the original Transformer in the single positive response scenario.