A comprehensive understanding of popular machine translation evaluation metrics. (6th October 2022)

• Record Type: Journal Article
• Authors: Islam, Md. Adnanul; Mukta, Md. Saddam Hossain
• Is Part Of: International journal of computational science and engineering. Volume 25:Number 5(2022)
• Page Start: 467
• Page End: 478
• Publication Date: 2022-10-06
• Subjects: evaluation metrics -- translation performance -- bi-lingual evaluation understudy -- BLEU -- METEOR -- translation edit rate -- TER -- machine translation; Computer science -- Mathematics -- Periodicals; Computer simulation -- Mathematical aspects -- Periodicals; Computational intelligence -- Periodicals; 004.015105
• Languages: English
• ISSNs: 1742-7185

Abstract

Machine translation is one of the pioneer applications of natural language processing and artificial intelligence. Automatic evaluation of the translation performance of the machine translators is one of the most challenging tasks, as manual evaluation of large volumes of document translations is infeasible in practice. Thus, to facilitate the evaluation of translation performance automatically, several metrics have been introduced and utilised widely. Although these translation performance evaluation metrics cannot match the efficiency level of human evaluation, they are popularly employed in automatic evaluation of translation quality of texts across multifarious application domains. This article discusses three such widely used evaluation metrics - BLEU, METEOR, and TER, with relevant details by demonstrating step by step calculations. The main novelty of this article lies in the consideration of several example translations to present and clarify the calculation process of these three of the most popular evaluation metrics for measuring the performance or quality of machine translation. Moreover, the article presents a comparative analysis among these three metrics using two different datasets to reveal their similarities and distinctions in terms of behaviour.