Benchmarking short text semantic similarity. (5th April 2010)

• Record Type: Journal Article
• Authors: O'Shea, James; Bandar, Zuhair; Crockett, Keeley; McLean, David
• Is Part Of: International journal of intelligent information and database systems. Volume 4:Number 2(2010)
• Page Start: 103
• Page End: 120
• Publication Date: 2010-04-05
• Subjects: short texts -- short text similarity -- sentence similarity -- semantic similarity -- benchmarking -- datasets -- gold standard -- information retrieval -- dialogue management -- question answering -- intelligent information systems -- intelligent database systems -- semantic distance -- semantics; Database management -- Computer programs -- Periodicals; Information retrieval -- Computer programs -- Periodicals; Information storage and retrieval systems -- Computer programs -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Intelligent agents (Computer software) -- Periodicals; 006.33
• Languages: English
• ISSNs: 1751-5858

Abstract

Short text semantic similarity measurement is a new and rapidly growing field of research. 'Short texts' are typically sentence length but are not required to be grammatically correct. There is great potential for applying these measures in fields such as information retrieval, dialogue management and question answering. A dataset of 65 sentence pairs, with similarity ratings, produced in 2006 has become adopted as a de facto gold standard benchmark. This paper discusses the adoption of the 2006 dataset, lays down a number of criteria that can be used to determine whether a dataset should be awarded a 'gold standard' accolade and illustrates its use as a benchmark. Procedures for the generation of further gold standard datasets in this field are recommended.