A high speed reconfigurable architecture for softmax and GELU in vision transformer. Issue 5 (14th March 2023)

• Record Type: Journal Article
• Authors: Li, Tianyang; Zhang, Fan; Xie, Guangwei; Fan, Xitian; Gao, Yanzhao; Sun, Mingqian
• Is Part Of: Electronics letters. Volume 59:Issue 5(2023)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2023-03-14
• Subjects: circuits and systems -- digital circuits -- field programmable gate arrays -- VLSI; Electronics -- Periodicals; 621.381
• DOI: 10.1049/ell2.12751
• Languages: English
• ISSNs: 0013-5194

Abstract

Abstract: Transformers have been widely used in various computer vision applications. Compared to traditional convolutional neural networks (CNNs), transformer's inference includes plenty of non‐linear operations, such as softmax and Gaussian error linear units (GELU). As the scale of transformers grows, an efficient hardware implementation of these operations is significant. However, the current works of computer vision neural network accelerators focus on CNN and less attention is paid to transformer. In addition, most current FPGA‐based softmax or GELU accelerators are not designed for vision transformer (ViT). To solve this problem, this work proposes a high speed reconfigurable accelerator. The architecture can support both softmax and GELU functions in ViT by reconfiguring the data path. This architecture on Xilinx XCVU37P is implemented through mathematical transformation and hardware optimization design, and achieve the performance of 102.4 Giga bits per second (Gbps) at 200 MHz. Experimental results show that the architecture achieves a very small accuracy loss in the ViT's inference by using fixed‐point 16‐bit quantization. Compared with existing accelerators, the design has greater throughput and area efficiency. Abstract : This work proposes a high speed reconfigurable accelerator, which can support both softmax and GELU functions in ViT by reconfiguring the data path. Experimental results show that the design can achieve a very small accuracy loss in ViT's inference. Compared with the existing designs, this design has higher throughput and area efficiency and is more suitable for large‐scale neural networks.