A multi-objective approach to determining the usefulness of papers in academic search. (18th May 2021)

• Record Type: Journal Article
• Authors: Khalid, Shah; Wu, Shengli; Zhang, Fang
• Is Part Of: Data technologies and applications. Volume 55:Number 5(2021)
• Page Start: 734
• Page End: 748
• Publication Date: 2021-05-18
• Subjects: Academic search -- Citation analysis -- Usefulness -- Relevance -- Impact -- Publication age -- Evaluation metrics; Information science -- Periodicals; Electronic information resources -- Periodicals; Knowledge management -- Periodicals; 020.5
• DOI: 10.1108/DTA-05-2020-0104
• Languages: English
• ISSNs: 2514-9288

Abstract

Abstract : Purpose: How to provide the most useful papers for searchers is a key issue for academic search engines. A lot of research has been carried out to address this problem. However, when evaluating the effectiveness of an academic search engine, most of the previous investigations assume that the only concern of the user is the relevancy of the paper to the query. The authors believe that the usefulness of a paper is determined not only by its relevance to the query but also by other aspects including its publication age and impact in the research community. This is vital, especially when a large number of papers are relevant to the query. Design/methodology/approach: This paper proposes a group of metrics to measure the usefulness of a ranked list of papers. When defining these metrics, three factors, including relevance, publication age and impact, are considered at the same time. To accommodate this, the authors propose a framework to rank papers by a combination of their relevance, publication age and impact scores. Findings: The framework is evaluated with the ACL (Association for Computational Linguistics Anthology Network) dataset. It demonstrates that the proposed ranking algorithm is effective for improving usefulness when two or three aspects of academic papers are considered at the same time, while the relevance of the retrieved papers is slightly down compared with the relevance-only retrieval. Originality/value: To the best of the authors' knowledge, the proposed multi-objective academic search framework is the first of its kind that is proposed and evaluated with a group of new evaluation metrics.