Teaching citizen scientists to categorize glitches using machine learning guided training. (April 2020)

• Record Type: Journal Article
• Authors: Jackson, Corey; Østerlund, Carsten; Crowston, Kevin; Harandi, Mahboobeh; Allen, Sarah; Bahaadini, Sara; Coughlin, Scotty; Kalogera, Vicky; Katsaggelos, Aggelos; Larson, Shane; Rohani, Neda; Smith, Joshua; Trouille, Laura; Zevin, Michael
• Is Part Of: Computers in human behavior. Volume 105(2020)
• Publication Date: 2020-04
• Subjects: Citizen science -- Experiment -- Training -- Online communities -- Zooniverse -- User studies -- Scaffolding -- Learning; Interactive computer systems -- Periodicals; Man-machine systems -- Periodicals; 004.019
• DOI: 10.1016/j.chb.2019.106198
• Languages: English
• ISSNs: 0747-5632

Abstract

Abstract: Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90% vs. 54%), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers' motivation. Highlights: Peer production communities must provide training to teach individuals how to become high-performing contributors. Scaffolded training is advantageous; however, selection of training materials and continuous assessment is a challenge. A scaffolded machine learning guided training (MLGT) program that teaches volunteers to classify glitches is evaluated. The MLGT was evaluated in an online field experiment, and results show the MLGT increases performance and retention.