Better Crowdcoding: Strategies for Promoting Accuracy in Crowdsourced Content Analysis. Issue 2 (3rd April 2021)

• Record Type: Journal Article
• Authors: Budak, Ceren; Garrett, R. Kelly; Sude, Daniel
• Is Part Of: Communication methods and measures. Volume 15:Issue 2(2021)
• Page Start: 141
• Page End: 155
• Publication Date: 2021-04-03
• Subjects: Communication -- Methodology -- Periodicals; Communication -- Research -- Periodicals; Communication -- Study and teaching -- Periodicals; 302.2072
• DOI: 10.1080/19312458.2021.1895977
• Languages: English
• ISSNs: 1931-2458

Abstract

ABSTRACT: In this work, we evaluate different instruction strategies to improve the quality of crowdcoding for the concept of civility. We test the effectiveness of training, codebooks, and their combination through 2 × 2 experiments conducted on two different populations – students and Amazon Mechanical Turk workers. In addition, we perform simulations to evaluate the trade-off between cost and performance associated with different instructional strategies and the number of human coders. We find that training improves crowdcoding quality, while codebooks do not. We further show that relying on several human coders and applying majority rule to their assessments significantly improves performance.