An empirical study of statistical language models: n-gram language models vs. neural network language models. (2018)

• Record Type: Journal Article
• Authors: Mezzoudj, Freha; Benyettou, Abdelkader
• Is Part Of: International journal of innovative computing and applications. Volume 9:Number 4(2018)
• Page Start: 189
• Page End: 202
• Publication Date: 2018
• Subjects: language models -- n-grams -- Kneser-Ney smoothing -- modified Kneser-Ney smoothing -- Good-Turing smoothing -- interpolation -- back-off -- feed-forward neural networks -- continuous space language models -- CSLM -- recurrent neural networks -- RNN -- speech recognition -- machine translation; Evolutionary computation -- Periodicals; Neural networks (Computer science) -- Periodicals; Genetic programming (Computer science) -- Periodicals; Biologically-inspired computing -- Periodicals; Swarm intelligence -- Periodicals; Quantum computers -- Periodicals; 006.3
• Languages: English
• ISSNs: 1751-648X

Abstract

Statistical language models are an important module in many areas of successful applications such as speech recognition and machine translation. And n-gram models are basically the state-of-the-art. However, due to sparsity of data, the modelled language cannot be completely represented in the n-gram language model. In fact, if new words appear in the recognition or translation steps, we need to provide a smoothing method to distribute the model probabilities over the unknown values. Recently, neural networks were used to model language based on the idea of projecting words onto a continuous space and performing the probability estimation in this space. In this experimental work, we compare the behaviour of the most popular smoothing methods with statistical n-gram language models and neural network language models in different situations and with different parameters. The language models are trained on two corpora of French and English texts. Good empirical results are obtained by the recurrent neural network language models.