Domain adaptation using neural network joint model. (September 2017)

• Record Type: Journal Article
• Authors: Joty, Shafiq; Durrani, Nadir; Sajjad, Hassan; Abdelali, Ahmed
• Is Part Of: Computer speech & language. Volume 45(2017)
• Page Start: 161
• Page End: 179
• Publication Date: 2017-09
• Subjects: Machine translation -- Domain adaptation -- Neural network joint model -- Distributed representation of texts -- Noise contrastive estimation; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2016.12.006
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Two sets of novel extensions of NNJM model are proposed. The NDAM models that regularizes the loss function with respect to in-domain model, give an improvement of up to +0.4 BLEU points. The NFM models that fuse in- and out-domain NNJM models give an improvement of up to +0.9 BLEU points. The NFM models also beat state-of-the-art phrase-table adaptation methods. The gains obtained from NNJM and phrase-table adaptation were found to be additive. Abstract: We explore neural joint models for the task of domain adaptation in machine translation in two ways: ( i ) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); ( ii ) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data. We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.