A high speed reconfigurable architecture for softmax and GELU in vision transformer. Issue 5 (14th March 2023)
- Record Type:
- Journal Article
- Title:
- A high speed reconfigurable architecture for softmax and GELU in vision transformer. Issue 5 (14th March 2023)
- Main Title:
- A high speed reconfigurable architecture for softmax and GELU in vision transformer
- Authors:
- Li, Tianyang
Zhang, Fan
Xie, Guangwei
Fan, Xitian
Gao, Yanzhao
Sun, Mingqian - Abstract:
- Abstract: Transformers have been widely used in various computer vision applications. Compared to traditional convolutional neural networks (CNNs), transformer's inference includes plenty of non‐linear operations, such as softmax and Gaussian error linear units (GELU). As the scale of transformers grows, an efficient hardware implementation of these operations is significant. However, the current works of computer vision neural network accelerators focus on CNN and less attention is paid to transformer. In addition, most current FPGA‐based softmax or GELU accelerators are not designed for vision transformer (ViT). To solve this problem, this work proposes a high speed reconfigurable accelerator. The architecture can support both softmax and GELU functions in ViT by reconfiguring the data path. This architecture on Xilinx XCVU37P is implemented through mathematical transformation and hardware optimization design, and achieve the performance of 102.4 Giga bits per second (Gbps) at 200 MHz. Experimental results show that the architecture achieves a very small accuracy loss in the ViT's inference by using fixed‐point 16‐bit quantization. Compared with existing accelerators, the design has greater throughput and area efficiency. Abstract : This work proposes a high speed reconfigurable accelerator, which can support both softmax and GELU functions in ViT by reconfiguring the data path. Experimental results show that the design can achieve a very small accuracy loss in ViT'sAbstract: Transformers have been widely used in various computer vision applications. Compared to traditional convolutional neural networks (CNNs), transformer's inference includes plenty of non‐linear operations, such as softmax and Gaussian error linear units (GELU). As the scale of transformers grows, an efficient hardware implementation of these operations is significant. However, the current works of computer vision neural network accelerators focus on CNN and less attention is paid to transformer. In addition, most current FPGA‐based softmax or GELU accelerators are not designed for vision transformer (ViT). To solve this problem, this work proposes a high speed reconfigurable accelerator. The architecture can support both softmax and GELU functions in ViT by reconfiguring the data path. This architecture on Xilinx XCVU37P is implemented through mathematical transformation and hardware optimization design, and achieve the performance of 102.4 Giga bits per second (Gbps) at 200 MHz. Experimental results show that the architecture achieves a very small accuracy loss in the ViT's inference by using fixed‐point 16‐bit quantization. Compared with existing accelerators, the design has greater throughput and area efficiency. Abstract : This work proposes a high speed reconfigurable accelerator, which can support both softmax and GELU functions in ViT by reconfiguring the data path. Experimental results show that the design can achieve a very small accuracy loss in ViT's inference. Compared with the existing designs, this design has higher throughput and area efficiency and is more suitable for large‐scale neural networks. … (more)
- Is Part Of:
- Electronics letters. Volume 59:Issue 5(2023)
- Journal:
- Electronics letters
- Issue:
- Volume 59:Issue 5(2023)
- Issue Display:
- Volume 59, Issue 5 (2023)
- Year:
- 2023
- Volume:
- 59
- Issue:
- 5
- Issue Sort Value:
- 2023-0059-0005-0000
- Page Start:
- n/a
- Page End:
- n/a
- Publication Date:
- 2023-03-14
- Subjects:
- circuits and systems -- digital circuits -- field programmable gate arrays -- VLSI
Electronics -- Periodicals
621.381 - Journal URLs:
- http://digital-library.theiet.org/content/journals/el ↗
http://estar.bl.uk/cgi-bin/sciserv.pl?collection=journals&journal=00135194 ↗
https://ietresearch.onlinelibrary.wiley.com/loi/1350911x ↗
http://www.theiet.org/ ↗ - DOI:
- 10.1049/ell2.12751 ↗
- Languages:
- English
- ISSNs:
- 0013-5194
- Deposit Type:
- Legaldeposit
- View Content:
- Available online (eLD content is only available in our Reading Rooms) ↗
- Physical Locations:
- British Library DSC - 3705.060000
British Library DSC - BLDSS-3PM
British Library HMNTS - ELD Digital store - Ingest File:
- 26633.xml