Recurrent neural network from adder's perspective: Carry-lookahead RNN. (December 2021)

• Record Type: Journal Article
• Authors: Jiang, Haowei; Qin, Feiwei; Cao, Jin; Peng, Yong; Shao, Yanli
• Is Part Of: Neural networks. Volume 144(2021)
• Page Start: 297
• Page End: 306
• Publication Date: 2021-12
• Subjects: Deep learning -- Carry-lookahead -- Parallel computation -- Sequence modeling; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.08.032
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel. Then, we design the method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is proposed. CL-RNN takes advantages in parallelism and flexible receptive field. Through a comprehensive set of tests, we verify that CL-RNN can perform better than existing typical RNNs in sequence modeling tasks which are specially designed for RNNs. Code and models are available at: https://github.com/WinnieJiangHW/Carry-lookahead_RNN .