A Cooperative Lightweight Translation Algorithm Combined with Sparse-ReLU. (28th May 2022)

• Record Type: Journal Article
• Authors: Xu, Xintao; Liu, Yi; Chen, Gang; Ye, Junbin; Li, Zhigang; Lu, Huaxiang
• Other Names: Fu Hanliang Academic Editor.
• Is Part Of: Computational intelligence and neuroscience. Volume 2022(2022)
• Publication Date: 2022-05-28
• Subjects: Neurosciences -- Data processing -- Periodicals; Computational intelligence -- Periodicals; Computational neuroscience -- Periodicals; 612.80285
• DOI: 10.1155/2022/4398839
• Languages: English
• ISSNs: 1687-5265

Abstract

Abstract : In the field of natural language processing (NLP), machine translation algorithm based on Transformer is challenging to deploy on hardware due to a large number of parameters and low parametric sparsity of the network weights. Meanwhile, the accuracy of lightweight machine translation networks also needs to be improved. To solve this problem, we first design a new activation function, Sparse-ReLU, to improve the parametric sparsity of weights and feature maps, which facilitates hardware deployment. Secondly, we design a novel cooperative processing scheme with CNN and Transformer and use Sparse-ReLU to improve the accuracy of the translation algorithm. Experimental results show that our method, which combines Transformer and CNN with the Sparse-ReLU, achieves a 2.32% BLEU improvement in prediction accuracy and reduces the number of parameters of the model by 23%, and the sparsity of the inference model increases by more than 50%.