Stable improved softmax using constant normalisation. Issue 23 (1st November 2017)

• Record Type: Journal Article
• Authors: Lim, S.; Lee, D.
• Is Part Of: Electronics letters. Volume 53:Issue 23(2017)
• Page Start: 1504
• Page End: 1506
• Publication Date: 2017-11-01
• Subjects: learning (artificial intelligence) -- computational complexity -- transfer functions; constant normalisation -- improved softmax method -- rectified linear unit based functions -- deep learning architectures -- activation functions -- hidden layers -- computation complexity; Electronics -- Periodicals; 621.381
• DOI: 10.1049/el.2017.3394
• Languages: English
• ISSNs: 0013-5194

Abstract

Abstract : In deep learning architectures, rectified linear unit based functions are widely used as activation functions of hidden layers, and the softmax is used for the output layers. Two critical problems of the softmax are introduced, and an improved softmax method to resolve the problems is proposed. The proposed method minimises instability of the softmax while reducing its losses. Moreover, this method is straightforward so its computation complexity is low, but it is substantially reasonable and operates robustly. Therefore, the proposed method can replace the softmax functions.