Theoretical guarantee for crowdsourcing learning with unsure option. (May 2023)

• Record Type: Journal Article
• Authors: Pan, Yigong; Tang, Ke; Sun, Guangzhong
• Is Part Of: Pattern recognition. Volume 137(2023)
• Publication Date: 2023-05
• Subjects: Machine learning -- Crowdsourcing learning -- Labeling; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109316
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The upper bound of minimally sufficient number of crowd labels required for learning a probably approximately correct (PAC) classification model with and without the unsure option, are given respectively. A condition under which providing (or not to provide) unsure option for crowdsourcing learning is derived. The first two theoretical results are extended to guide non-identical label options to different workers, i.e., provide different label options (with or without unsure option) to different workers. Several useful applications of theoretical results are presented. Abstract: Crowdsourcing learning, in which labels are collected from multiple workers through crowdsourcing platforms, has attracted much attention during the past decade. This learning paradigm would reduce the labeling cost since crowdsourcing workers may be non-expert and hence less costly. On the other hand, crowdsourcing learning algorithms also suffer from being misled by incorrect labels introduced by imperfect workers. To control such risks, recently, it has been suggested to provide workers an additional unsure option during the labeling process. Although the benefits of the unsure option have been empirically demonstrated, theoretical analysis is still limited. In this article, a theoretical analysis of crowdsourcing learning with the unsure option is presented. Specifically, an upper bound of minimally sufficient number of crowd labels required for learning a probably approximately correct (PAC) classification model with and without the unsure option are given respectively. Next, a condition under which providing (or not providing) an unsure option to workers is derived. Then, the theoretical results are extended to guide non-identical label options (with or without unsure options) to different workers. Last, several useful applications are proposed based on theoretical results.