Offline recommender system evaluation: Challenges and new directions. Issue 2 (16th June 2022)

• Record Type: Journal Article
• Authors: Castells, Pablo; Moffat, Alistair
• Is Part Of: AI magazine. Volume 43:Issue 2 (2022)
• Page Start: 225
• Page End: 238
• Publication Date: 2022-06-16
• Subjects: Artificial intelligence -- Periodicals; Artificial intelligence -- Computer programs -- Periodicals; System design -- Periodicals; Artificial intelligence -- Technological innovations -- Periodicals; Artificial intelligence -- Industrial applications -- Periodicals; Artificial intelligence -- Educational applications -- Periodicals; Intelligence artificielle -- Périodiques; Intelligence artificielle -- Logiciels -- Périodiques; Conception de systèmes -- Périodiques; Intelligence artificielle -- Innovations -- Périodiques; Intelligence artificielle -- Applications industrielles -- Périodiques; Intelligence artificielle -- Applications en éducation -- Périodiques; Artificial intelligence; Artificial intelligence -- Computer programs; Artificial intelligence -- Educational applications; Artificial intelligence -- Industrial applications; System design; Kunstmatige intelligentie; Electronic journals; Periodicals; Periodicals; 006.305
• DOI: 10.1002/aaai.12051
• Languages: English
• ISSNs: 0738-4602

Abstract

Abstract: Offline evaluation is an essential complement to online experiments in the selection, improvement, tuning, and deployment of recommender systems. Offline methodologies for recommender system evaluation evolved from experimental practice in Machine Learning (ML) and Information Retrieval (IR). However, evaluating recommendations involves particularities that pose challenges to the assumptions upon which the ML and IR methodologies were developed. We recap and reflect on the development and current status of recommender system evaluation, providing an updated perspective. With a focus on offline evaluation, we review the adaptation of IR principles, procedures and metrics, and the implications of those techniques when applied to recommender systems. At the same time, we identify the singularities of recommendation that require different responses, or involve specific new needs. In addition, we provide an overview of important choices in the configuration of experiments that require particular care and understanding; discuss broader perspectives of evaluation such as recommendation value beyond accuracy; and survey open challenges such as experimental biases, and the cyclic dimension of recommendation.