A survey on the application of recurrent neural networks to statistical language modeling. (March 2015)

• Record Type: Journal Article
• Authors: De Mulder, Wim; Bethard, Steven; Moens, Marie-Francine
• Is Part Of: Computer speech & language. Volume 30(2015)
• Page Start: 61
• Page End: 98
• Publication Date: 2015-03
• Subjects: 68T05 -- 68T10 -- 68T50; Recurrent neural networks -- Natural language processing -- Language modeling -- Speech recognition -- Machine translation; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2014.09.005
• Languages: English
• ISSNs: 0885-2308

Abstract

Abstract : Highlights: We explain in detail the different steps in computing a language model based on a recurrent neural network. We survey the applications and findings based on the current literature. We survey the methods for reducing computational complexity. Abstract: In this paper, we present a survey on the application of recurrent neural networks to the task of statistical language modeling. Although it has been shown that these models obtain good performance on this task, often superior to other state-of-the-art techniques, they suffer from some important drawbacks, including a very long training time and limitations on the number of context words that can be taken into account in practice. Recent extensions to recurrent neural network models have been developed in an attempt to address these drawbacks. This paper gives an overview of the most important extensions. Each technique is described and its performance on statistical language modeling, as described in the existing literature, is discussed. Our structured overview makes it possible to detect the most promising techniques in the field of recurrent neural networks, applied to language modeling, but it also highlights the techniques for which further research is required.