A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments. Issue 3 (8th May 2017)

• Record Type: Journal Article
• Authors: Behnert, Christiane; Lewandowski, Dirk
• Is Part Of: Journal of documentation. Volume 73:Issue 3(2017)
• Page Start: 509
• Page End: 527
• Publication Date: 2017-05-08
• Subjects: Evaluation -- Information retrieval -- Library systems -- Digital libraries -- Information systems -- Online catalogues -- Discovery systems -- Relevance assessments -- Search results; Documentation -- Periodicals; Library science -- Periodicals; 025
• DOI: 10.1108/JD-08-2016-0099
• Languages: English
• ISSNs: 0022-0418

Abstract

Abstract : Purpose: The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources. Design/methodology/approach: The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials. Findings: The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context. Practical implications: The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context. Originality/value: Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors' knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.