Weight initialization based‐rectified linear unit activation function to improve the performance of a convolutional neural network model. (10th December 2020)

• Record Type: Journal Article
• Authors: Olimov, Bekhzod; Karshiev, Sanjar; Jang, Eungyeong; Din, Sadia; Paul, Anand; Kim, Jeonghong
• Other Names: Jeon Gwanggil guestEditor.; Chehri Abdellah guestEditor.; Lu Huimin guestEditor.; Guna Jože guestEditor.
• Is Part Of: Concurrency and computation. Volume 33:Number 22(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-12-10
• Subjects: activation function -- convolutional neural networks -- deep learning -- ReLU -- weight initialization; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6143
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Convolutional Neural Networks (CNNs) have made a great impact on attaining state‐of‐the‐art results in image task classification. Weight initialization is one of the fundamental steps in formulating a CNN model. It determines the failure or success of the CNN model. In this paper, we conduct a research based on the mathematical background of different weight initialization strategies to determine the one with better performance. To have smooth training, we expect the activation of each layer of the CNN model follow the standard normal distribution with mean 0 and SD 1. It prevents gradients from vanishing and leads to more smooth training. However, it was obtained that even with the appropriate weight initialization technique, a regular Rectified Linear Unit (ReLU) activation function increases the activation mean value. In this paper, we address this issue by proposing weight initialization based (WIB)‐ReLU activation function. The proposed method resulted in more smooth training. Moreover, the experiments showed that WIB‐ReLU outperforms ReLU, Leaky ReLU, parametric ReLU, and exponential linear unit activation functions and results in up to 20% decrease in loss value and 5% increase in accuracy score on both Fashion‐MNIST and CIFAR‐10 databases.