A novel structured sparse fully connected layer in convolutional neural networks. (31st January 2021)

• Record Type: Journal Article
• Authors: Matsumura, Naoki; Ito, Yasuaki; Nakano, Koji; Kasagi, Akihiko; Tabaru, Tsuguchika
• Other Names: Bordin Jacir Luiz guestEditor.; Ito Yasuaki guestEditor.; Wakrime Abderrahim Ait guestEditor.; Sellami Mohamed guestEditor.; Halima Riadh Ben guestEditor.
• Is Part Of: Concurrency and computation. Volume 35:Number 11(2023)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-31
• Subjects: convolutional neural network -- GPU -- model compression; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6213
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Convolutional Neural Networks (CNNs) are one of the factors supporting the rapid development of artificial intelligent techniques. However, as the ability of the network increases, the size of the network becomes larger. Thus far, several works related to reduction of the network size have been tackled. In many cases, these approaches produce an unstructured network which prevents efficient parallel computation. To avoid this problem, we propose a novel structured sparse fully connected layer (FCL) in the CNNs. The aim of our proposed approach is reduction of the number of network parameters in the FCLs which occupy a large part of network parameters. Unlike the general FCLs used in the popular CNNs such as VGG‐16, the proposed approach reduces the connection between the last convolutional layer and the first FCL. In addition, we show an implementation for the proposed sparse FCLs on the GPU using cuBLAS. As a result for ILSVRC‐2012 dataset, the proposed approach achieves a 21.3 times compression with 0.68% top‐1 accuracy and 0.31% top‐5 accuracy decreases for VGG‐16. The implementation of the proposed FCLs achieves speed‐up factor 14.97 and 16.67 for forward and backward propagation compared to that for the noncompressed FCLs, respectively.