Multi-domain evaluation framework for named entity recognition tools. (May 2017)

• Record Type: Journal Article
• Authors: Abdallah, Zahraa S.; Carman, Mark; Haffari, Gholamreza
• Is Part Of: Computer speech & language. Volume 43(2017)
• Page Start: 34
• Page End: 55
• Publication Date: 2017-05
• Subjects: Named entity recognition -- Multi-domain evaluation -- Qualitative data analysis -- Benchmark evaluation; Speech processing systems -- Periodicals; Automatic speech recognition -- Periodicals; Computers -- Periodicals; Linguistics -- Periodicals; Speech-Language Pathology -- Periodicals; Traitement automatique de la parole -- Périodiques; Reconnaissance automatique de la parole -- Périodiques; Automatic speech recognition; Speech processing systems; Electronic journals; Periodicals; 006.454
• DOI: 10.1016/j.csl.2016.10.003
• Languages: English
• ISSNs: 0885-2308

Abstract

Highlights: A flexible and extensible framework for integrating tools & domains. Assessing NER tools robustness across various domains. Evaluating a wide scale of tools including both commercial & non-commercial. Comprehensive analysis of NER tools from various perspectives. Abstract: Extracting structured information from unstructured text is important for the qualitative data analysis. Leveraging NLP techniques for qualitative data analysis will effectively accelerate the annotation process, allow for large-scale analysis and provide more insights into the text to improve the performance. The first step for gaining insights from the text is Named Entity Recognition (NER). A significant challenge that directly impacts the performance of the NER process is the domain diversity in qualitative data. The represented text varies according to its domain in many aspects including taxonomies, length, formality and format. In this paper we discuss and analyse the performance of state-of-the-art tools across domains to elaborate their robustness and reliability. In order to do that, we developed a standard, expandable and flexible framework to analyse and test tools performance using corpora representing text across various domains. We performed extensive analysis and comparison of tools across various domains and from various perspectives. The resulting comparison and analysis are of significant importance for providing a holistic illustration of the state-of-the-art tools.