A Simplified Natural Gradient Learning Algorithm. (24th July 2011)

• Record Type: Journal Article
• Authors: Bastian, Michael R.; Gunther, Jacob H.; Moon, Todd K.
• Other Names: Chakrabartty Shantanu Academic Editor.
• Is Part Of: Advances in artificial neural systems. (2011)
• Publication Date: 2011-07-24
• Subjects: Neural networks (Computer science) -- Periodicals; Neural networks (Computer science); Periodicals; Electronic journals; 006.32
• DOI: 10.1155/2011/407497
• Languages: English
• ISSNs: 1687-7594

Abstract

Abstract : Adaptive natural gradient learning avoids singularities in the parameter space of multilayer perceptrons. However, it requires a larger number of additional parameters than ordinary backpropagation in the form of the Fisher information matrix. This paper describes a new approach to natural gradient learning that uses a smaller Fisher information matrix. It also uses a prior distribution on the neural network parameters and an annealed learning rate. While this new approach is computationally simpler, its performance is comparable to that of adaptive natural gradient learning.