Latent ranking analysis using pairwise comparisons in crowdsourcing platforms. (April 2017)

• Record Type: Journal Article
• Authors: Kim, Younghoon; Kim, Wooyeol; Shim, Kyuseok
• Is Part Of: Information systems. Volume 65(2017)
• Page Start: 7
• Page End: 21
• Publication Date: 2017-04
• Subjects: Learning to rank -- Pairwise comparison -- Active learning -- Crowdsourcing; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2016.10.002
• Languages: English
• ISSNs: 0306-4379

Abstract

Abstract: Ranking items is an essential problem in recommendation systems. Since comparing two items is the simplest type of queries in order to measure the relevance of items, the problem of aggregating pairwise comparisons to obtain a global ranking has been widely studied. Furthermore, ranking with pairwise comparisons has recently received a lot of attention in crowdsourcing systems where binary comparative queries can be used effectively to make assessments faster for precise rankings. In order to learn a ranking based on a training set of queries and their labels obtained from annotators, machine learning algorithms are generally used to find the appropriate ranking model which describes the data set the best. In this paper, we propose a probabilistic model for learning multiple latent rankings by using pairwise comparisons. Our novel model can capture multiple hidden rankings underlying the pairwise comparisons. Based on the model, we develop an efficient inference algorithm to learn multiple latent rankings as well as an effective inference algorithm for active learning to update the model parameters in crowdsourcing systems whenever new pairwise comparisons are supplied. The performance study with synthetic and real-life data sets confirms the effectiveness of our model and inference algorithms. Abstract : Highlights: We study how to learn multiple latent rankings from pairwise comparisons. We propose a novel probabilistic model to capture the process of pairwise comparison. We develop an efficient inference algorithm to learn multiple latent rankings. We also investigate active learning problem considering crowdsourcing platforms. Experiments with synthetic and real life data show the effectiveness of our algorithms.