An information fusion framework with multi-channel feature concatenation and multi-perspective system combination for the deep-learning-based robust recognition of microphone array speech. (November 2017)

• Record Type: Journal Article
• Authors: Tu, Yan-Hui; Du, Jun; Wang, Qing; Bao, Xiao; Dai, Li-Rong; Lee, Chin-Hui
• Is Part Of: Computer speech & language. Volume 46(2017)
• Page Start: 517
• Page End: 534
• Publication Date: 2017-11
• Subjects: CHiME challenge -- Deep learning -- Information fusion -- Microphone array -- Robust speech recognition; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2016.12.004
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: The early fusion by using multiple beamformings and feature concatenation. The late fusion of subnets from multiple perspectives. A simplified and effective MVDR beamforming approach. Building the best one-pass single DNN system among all submissions to CHiME-3. Abstract: We present an information fusion approach to the robust recognition of multi-microphone speech. It is based on a deep learning framework with a large deep neural network (DNN) consisting of subnets designed from different perspectives. Multiple knowledge sources are then reasonably integrated via an early fusion of normalized noisy features with multiple beamforming techniques, enhanced speech features, speaker-related features, and other auxiliary features concatenated as the input to each subnet to compensate for imperfect front-end processing. Furthermore, a late fusion strategy is utilized to leverage the complementary natures of the different subnets by combining the outputs of all subnets to produce a single output set. Testing on the CHiME-3 task of recognizing microphone array speech, we demonstrate in our empirical study that the different information sources complement each other and that both early and late fusions provide significant performance gains, with an overall word error rate of 10.55% when combining 12 systems. Furthermore, by utilizing an improved technique for beamforming and a powerful recurrent neural network (RNN)-based language model for rescoring, a WER of 9.08% can be achieved for the best single DNN system with one-pass decoding among all of the systems submitted to the CHiME-3 challenge.