CharaParser+EQ: Performance evaluation without gold standard. Issue 1 (24th February 2016)

• Record Type: Journal Article
• Authors: Cui, Hong; Dahdul, Wasila; Dececchi, Alexander T.; Ibrahim, Nizar; Mabee, Paula; Balhoff, James P.; Gopalakrishnan, Hariharan
• Is Part Of: Proceedings of the Association for Information Science and Technology. Volume 52:Issue 1(2015)
• Page Start: 1
• Page End: 10
• Publication Date: 2016-02-24
• Subjects: Phenotype character curation -- EQ statements -- Natural Language Processing -- curation inconsistency -- ontology search; Information science -- Congresses; Information technology -- Congresses; Information science; Information technology; Conference papers and proceedings; 020
• DOI: 10.1002/pra2.2015.145052010020
• Languages: English
• ISSNs: 2373-9231

Abstract

ABSTRACT: To make phenotypic characters of organisms widely useful for computerized biology research, biocurators manually convert character descriptions to a structured format, for example the Entity‐Quality (EQ) format. The manual approach is time consuming and affected by inter‐curator variations. In this paper we report a software application, CharaParser+EQ, to our knowledge the first software that produces EQ statements from textual character descriptions. We report a recent experiment that evaluates the performance of the software against three experienced biocurators. While the software is still far from being able to compete with biocurators on this highly intellectual task, the results show (1) CharaParser+EQ's performance (precision and recall) is greatly improved compared to a previous version, (2) the completeness of the ontologies used in the process has significant impact both on the software's EQ generation performance and on the agreement among curators, and (3) unlimited access to external knowledge (published papers, books) by curators has no significant impact on inter‐curator agreements. A detailed error analysis that compares machine and curator generated EQs is included.