A short text conversation generation model combining BERT and context attention mechanism. (13th October 2020)

• Record Type: Journal Article
• Authors: Zhao, Huan; Lu, Jian; Cao, Jie
• Is Part Of: International journal of computational science and engineering. Volume 23:Number 2(2020)
• Page Start: 136
• Page End: 144
• Publication Date: 2020-10-13
• Subjects: Seq2Seq -- short text conversation generation -- BERT -- attention mechanism -- fusion unit; Computer science -- Mathematics -- Periodicals; Computer simulation -- Mathematical aspects -- Periodicals; Computational intelligence -- Periodicals; 004.015105
• Languages: English
• ISSNs: 1742-7185

Abstract

The standard Seq2Seq neural network model tends to generate general and safe responses (e.g., I don't know) regardless of the input in the field of short text conversation generation. To address this problem, we propose a novel model that combines the standard Seq2Seq model with the BERT module (a pre-trained model) to improve the quality of responses. Specifically, the encoder of the model is divided into two parts: one is the standard Seq2Seq which generates a context attention vector; the other is the improved BERT module which encodes the input sentence into a semantic vector. Then through a fusion unit, the vectors generated by the two parts are fused to generate a new attention vector. Finally, the new attention vector is transmitted to the decoder. In particular, we describe two ways to acquire a new attention vector in the fusion unit. Empirical results from automatic and human evaluations demonstrate that our model improves the quality and diversity of the responses significantly.