A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence training. (November 2021)

• Record Type: Journal Article
• Authors: Haider, Adnan; Zhang, Chao; Kreyssig, Florian L.; Woodland, Philip C.
• Is Part Of: Neural networks. Volume 143(2021)
• Page Start: 537
• Page End: 549
• Publication Date: 2021-11
• Subjects: Second-order optimisation -- Hessian-free -- Natural gradient -- Conjugate gradient -- Discriminative sequence training; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2021.05.011
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF). A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used ( e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates. Highlights: Large batch optimisation: NGHF. Combines natural gradient (NG) & Hessian-free (HF). Faster convergence of each update estimated via improved conjugate gradient. Applied NG, HF & NGHF to discriminative sequence training for speech recognition. NG, HF, and NGHF require orders of magnitude fewer parameter updates than Adam & SGD. MPE training with NGHF achieves lower word error rates than with NG, HF, Adam & SGD.