Estimation theory and Neural Networks revisited: REKF and RSVSF as optimization techniques for Deep-Learning. (December 2018)

• Record Type: Journal Article
• Authors: Ismail, Mahmoud; Attari, Mina; Habibi, Saeid; Ziada, Samir
• Is Part Of: Neural networks. Volume 108(2018)
• Page Start: 509
• Page End: 526
• Publication Date: 2018-12
• Subjects: Kalman filter -- Smooth variable structure filter -- Deep-Learning -- Neural Networks -- REKF -- RSVSF; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2018.09.012
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Deep-Learning has become a leading strategy for artificial intelligence and is being applied in many fields due to its excellent performance that has surpassed human cognitive abilities in a number of classification and control problems (Ciregan, Meier, & Schmidhuber, 2012; Mnih et al., 2015). However, the training process of Deep-Learning is usually slow and requires high-performance computing, capable of handling large datasets. The optimization of the training method can improve the learning rate of the Deep-Learning networks and result in a higher performance while using the same number of training epochs (cycles). This paper considers the use of estimation theory for training of large neural networks and in particular Deep-Learning networks. Two estimation strategies namely the Extended Kalman Filter (EKF) and the Smooth Variable Structure Filter (SVSF) have been revised (subsequently referred to as RSVSF and REKF) and used for network training. They are applied to several benchmark datasets and comparatively evaluated. Highlights: Two Deep-Learning training algorithms are introduced, called REKF and RSVSF. REKF and RSVSF allow Deep-Learning networks to learn fast. REKF and RSVSF use less memory than current adaptive methods.