An analysis of the semantic annotation task on the linked data cloud. (30th September 2019)

• Record Type: Journal Article
• Authors: Gagnon, Michel; Zouaq, Amal; Aranha, Francisco; Ensan, Faezeh; Jean-Louis, Ludovic
• Is Part Of: International journal of metadata, semantics and ontologies. Volume 13:Number 4(2019)
• Page Start: 317
• Page End: 329
• Publication Date: 2019-09-30
• Subjects: semantic annotation -- linked data cloud -- performance evaluation; Metadata -- Periodicals; Semantic Web -- Periodicals; Ontologies (Information retrieval) -- Periodicals; Data structures (Computer science) -- Periodicals; Information theory -- Periodicals; 005.74
• Languages: English
• ISSNs: 1744-2621

Abstract

Semantic annotation, the process of identifying key phrases in texts and linking them to concepts in a knowledge base, is an important basis for semantic information retrieval and the semantic web uptake. Despite the emergence of semantic annotation systems, very few comparative studies have been published on their performance. In this paper, we provide an evaluation of the performance of existing systems over three tasks: full semantic annotation, named entity recognition, and keyword detection. More specifically, the spotting capability (recognition of relevant surface forms in text) is evaluated for all three tasks, whereas the disambiguation (correctly associating an entity from Wikipedia or DBpedia to the spotted surface forms) is evaluated only for the first two tasks. We use logistic regression to identify significant performance differences. Although some of the annotators are specifically targeted at some task (NE, SA, KW), our results show that they do not necessarily obtain the best performance on those tasks. In fact, systems identified as full semantic annotators beat all other systems on all data sets. We also show that there is still much room for improvement for the identification of the most relevant entities described in a text.