Extracting parallel fragments from comparable documents using a generative model. (January 2019) - Bakhshaei, Somayeh; Safabakhsh, Reza; Khadivi, Shahram
Highlights: Study of comparable corpora(CC) for extracting parallel information. Generative model for extracting parallel fragments of CC without need of initial seed. Enhancing a SMT system using extracted parallel fragments from CC. Abstract: Although parallel corpora are essential language resources for many natural language processing tasks, they are rare or even not available for many language pairs. Instead, comparable corpora are widely available and contain parallel fragments of information that can be used in applications like statistical machine translation systems. In this research, we propose a generative latent Dirichlet allocation based model for extracting parallel fragments from comparable documents without using any initial parallel data or bilingual lexicon. The experimental results show significant improvement if the extracted fragments generated by the proposed method are used for augmenting an existing parallel corpus in an statistical machine translation system. According to the human judgment, the accuracy of the proposed method for an English-Persian task is about 59.7%. Also, the out of vocabulary error rate for the same task is reduced by 28%.