Benchmarks and goals. (September 2020)

• Record Type: Journal Article
• Authors: Church, Kenneth Ward
• Is Part Of: Natural language engineering. Volume 26:Part 5(2020)
• Page Start: 579
• Page End: 592
• Publication Date: 2020-09
• Subjects: Benchmarks, -- Knowledge graph completion, -- Bilingual lexicon induction, -- MUSE, -- WordNet; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324920000418
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Benchmarks can be a useful step toward the goals of the field (when the benchmark is on the critical path), as demonstrated by the GLUE benchmark, and deep nets such as BERT and ERNIE. The case for other benchmarks such as MUSE and WN18RR is less well established. Hopefully, these benchmarks are on a critical path toward progress on bilingual lexicon induction (BLI) and knowledge graph completion (KGC). Many KGC algorithms have been proposed such as Trans[DEHRM], but it remains to be seen how this work improves WordNet coverage. Given how much work is based on these benchmarks, the literature should have more to say than it does about the connection between benchmarks and goals. Is optimizing P@10 on WN18RR likely to produce more complete knowledge graphs? Is MUSE likely to improve Machine Translation?