Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems. Issue 4 (20th July 2015)

• Record Type: Journal Article
• Authors: Ravana, Sri Devi; TAHERI, MASUMEH SADAT; Rajagopal, Prabha
• Is Part Of: Aslib journal of information management. Volume 67:Issue 4(2015:Aug.)
• Page Start: 408
• Page End: 421
• Publication Date: 2015-07-20
• Subjects: Information science -- Periodicals; Library science -- Periodicals; 020.5
• DOI: 10.1108/AJIM-12-2014-0171
• Languages: English
• ISSNs: 2050-3806

Abstract

<abstract> <title> <x content-type="archive" xml:space="preserve">Abstract</x> </title> <sec> <title content-type="abstract-heading">Purpose</title> <p> – The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries. </p> </sec> <sec> <title content-type="abstract-heading">Design/methodology/approach</title> <p> – Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document's weight, which play the role of the mean average precision (MAP) score of the systems as a significance test's statics. The experiments were conducted using the TREC 9 Web track collection. </p> </sec> <sec> <title content-type="abstract-heading">Findings</title> <p> – The <italic>p</italic>-values generated through the two types of significance tests, namely the Student's <italic>t</italic>-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores. </p> </sec> <sec> <title content-type="abstract-heading">Originality/value</title> <p> – Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.</p> </sec> </abstract>