Constructing a Natural Language Inference dataset using generative neural networks. (November 2017)

• Record Type: Journal Article
• Authors: Starc, Janez; Mladenić, Dunja
• Is Part Of: Computer speech & language. Volume 46(2017)
• Page Start: 94
• Page End: 112
• Publication Date: 2017-11
• Subjects: Natural Language Inference -- Natural language generation -- Machine learning -- Dataset construction -- Generative neural network -- Recurrent neural network; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.04.009
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: New procedure for generating Natural Language Inference text hypotheses. New encoder-less text generative neural network. New evaluation metric—accuracy of the classifier trained on the generated dataset. The best model's accuracy is only 2.7% less than the accuracy on original dataset. Comparison of different metrics to explain what makes a good dataset. Abstract: Natural Language Inference is an important task for Natural Language Understanding. It is concerned with classifying the logical relation between two sentences. In this paper, we propose several text generative neural networks for generating text hypothesis, which allows construction of new Natural Language Inference datasets. To evaluate the models, we propose a new metric—the accuracy of the classifier trained on the generated dataset. The accuracy obtained by our best generative model is only 2.7% lower than the accuracy of the classifier trained on the original, human crafted dataset. Furthermore, the best generated dataset combined with the original dataset achieves the highest accuracy. The best model learns a mapping embedding for each training example. By comparing various metrics we show that datasets that obtain higher ROUGE or METEOR scores do not necessarily yield higher classification accuracies. We also provide analysis of what are the characteristics of a good dataset including the distinguishability of the generated datasets from the original one.