An Algorithmic Scheme for Statistical Thesaurus Construction in a Morphologically Rich Language. Issue 6 (12th May 2019)

• Record Type: Journal Article
• Authors: Liebeskind, Chaya; Dagan, Ido; Schler, Jonathan
• Is Part Of: Applied artificial intelligence. Volume 33:Issue 6(2019)
• Page Start: 483
• Page End: 496
• Publication Date: 2019-05-12
• Subjects: Artificial intelligence -- Periodicals; 006.3
• DOI: 10.1080/08839514.2019.1583447
• Languages: English
• ISSNs: 0883-9514

Abstract

ABSTRACT: Corpus-based automatic thesaurus construction uses linguistic methods, such as Part-of-Speech taggers and parsers, which often perform poorly on MRLs. Therefore, in this paper, we focused on the complex task of adapting corpus-based thesaurus construction methods for MRLs. We investigated two statistical approaches for thesaurus construction; a) a first-order co-occurrence-based approach and b) a second-order distributional-based approach. We explored alternative levels of morphological term representations complemented by grouping the morphological variants. We then introduced and adopted a generic algorithmic scheme for thesaurus construction in MRLs for both first-order and second-order approaches. Our scheme investigated alternative representation levels and offered alternative configurations. We demonstrated the empirical benefits of our methodology for a diachronic Hebrew thesaurus construction. We used morphological analysis tools, defined and applied a new annotation scheme, and demonstrated its optimal configuration, which outperforms the baseline for both first and second order corpus-based thesaurus construction approaches.