Neural versus phrase-based MT quality: An in-depth analysis on English–German and English–French. (May 2018)

• Record Type: Journal Article
• Authors: Bentivogli, Luisa; Bisazza, Arianna; Cettolo, Mauro; Federico, Marcello
• Is Part Of: Computer speech & language. Volume 49(2018)
• Page Start: 52
• Page End: 70
• Publication Date: 2018-05
• Subjects: Machine translation (MT) -- Neural MT -- Phrase-based MT -- Evaluation; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2017.11.004
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: Evaluation through high quality post-edits performed by professional translators. Focus on English–German and English–French language directions. Neural MT makes impressively less lexical, morphology, and word order errors. Neural MT best models reordering of verbs (En-De) and nouns (En-Fr). Neural MT makes remarkably more errors in the translation of proper nouns. Abstract: Within the field of statistical machine translation, the neural approach (NMT) is currently pushing ahead the state of the art performance traditionally achieved by phrase-based approaches (PBMT), and is rapidly becoming the dominant technology in machine translation. Indeed, in the last IWSLT and WMT evaluation campaigns on machine translation, NMT outperformed well established state-of-the-art PBMT systems on many different language pairs. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based statistical machine translation outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. In this analysis, we focus on two language directions with different characteristics: English–German, known to be particularly hard because of morphology and syntactic differences, and English–French, where PBMT systems typically reach outstanding quality and thus represent a strong competitor for NMT. Our analysis provides useful insights on what linguistic phenomena are best modelled by neural models – such as the reordering of verbs and nouns – while pointing out other aspects that remain to be improved – like the correct translation of proper nouns.