Framework and Literature Analysis for Crowdsourcing's Answer Aggregation. Issue 1 (2nd January 2020)

• Record Type: Journal Article
• Authors: Moayedikia, Alireza; Yeoh, William; Ong, Kok-Leong; Boo, Yee-Ling
• Is Part Of: Journal of computer information systems. Volume 60:Issue 1(2020)
• Page Start: 49
• Page End: 60
• Publication Date: 2020-01-02
• Subjects: Crowdsourcing -- answer aggregation -- systematic literature analysis -- classification framework; Electronic data processing -- Study and teaching -- Periodicals; 658.4038011
• DOI: 10.1080/08874417.2017.1394171
• Languages: English
• ISSNs: 0887-4417

Abstract

ABSTRACT: This paper presents a classification framework and a systematic analysis of literature on answer aggregation techniques for the most popular and important type of crowdsourcing, i.e., micro-task crowdsourcing. In doing so, we analyzed research articles since 2006 and developed four classification taxonomies. First, we provided a classification framework based on the algorithmic characteristics of answer aggregation techniques. Second, we outlined the statistical and probabilistic foundations used by different types of algorithms and micro-tasks. Third, we provided a matrix catalog of the data characteristics for which an answer aggregation algorithm is designed. Fourth, a matrix catalog of the commonly used evaluation metrics for each type of micro-task was presented. This paper represents the first systematic literature analysis and classification of the answer aggregation techniques for micro-task crowdsourcing.