Ensemble of convolutional neural networks trained with different activation functions. (15th March 2021)

• Record Type: Journal Article
• Authors: Maguolo, Gianluca; Nanni, Loris; Ghidoni, Stefano
• Is Part Of: Expert systems with applications. Volume 166(2021)
• Publication Date: 2021-03-15
• Subjects: Neural networks -- Ensemble -- Activation functions -- MeLU; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2020.114048
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: Ensemble of convolutional neural networks. New activation function is proposed. Activations functions are compared and combined. Rectified Linear Units and different variants are compared. Abstract: Activation functions play a vital role in the training of Convolutional Neural Networks. For this reason, developing efficient and well-performing functions is a crucial problem in the deep learning community. The idea of these approaches is to allow a reliable parameter learning, avoiding vanishing gradient problems. The goal of this work is to propose an ensemble of Convolutional Neural Networks trained using several different activation functions. Moreover, a novel activation function is here proposed for the first time. Our aim is to improve the performance of Convolutional Neural Networks in small/medium sized biomedical datasets. Our results clearly show that the proposed ensemble outperforms Convolutional Neural Networks trained with a standard ReLU as activation function. The proposed ensemble outperforms with a p-value of 0.01 each tested stand-alone activation function; for reliable performance comparison we tested our approach on more than 10 datasets, using two well-known Convolutional Neural Networks: Vgg16 and ResNet50.