Uncertainty-aware non-autoregressive neural machine translation. (March 2023)

• Record Type: Journal Article
• Authors: Liu, Chuanming; Yu, Jingqi
• Is Part Of: Computer speech & language. Volume 78(2023)
• Publication Date: 2023-03
• Subjects: Bayesian deep learning -- Non-autoregressive -- Machine translation -- Active learning -- Monte Carlo dropout; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2022.101444
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract: Most existing non-autoregressive neural machine translation (NAT) models generally employ the posterior probability to indicate the model confidence during training, which seems to lag behind the novel uncertainty estimations (UEs) methods successfully deployed in other natural language processing (NLP) tasks. Previous research has practically ignored the large-scale exploration of UE methods in the NAT problem. In this paper, we propose a strategy based on Active Learning employed to investigate whether these sophisticated uncertainty-aware methods are more effective in the NAT problem. Besides, we provide an in-depth analysis of the impact of different widely employed UE methods and propose several tailored ones. In the end, we incorporate these exceptional ones into the practical one-pass GLAT model to obtain enhanced performance. Experimental results demonstrate that sophisticated uncertainty-aware UE methods with the two-step training paradigm are potentially superior to represent the model confidence in facilitating token-level decision-making compared to the posterior probability in NAT to a certain extent.