The many dimensions of truthfulness: Crowdsourcing misinformation assessments on a multidimensional scale. Issue 6 (November 2021)

• Record Type: Journal Article
• Authors: Soprano, Michael; Roitero, Kevin; La Barbera, David; Ceolin, Davide; Spina, Damiano; Mizzaro, Stefano; Demartini, Gianluca
• Is Part Of: Information processing & management. Volume 58:Issue 6(2021)
• Publication Date: 2021-11
• Subjects: Truthfulness -- Crowdsourcing -- Misinformation -- Explainability; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2021.102710
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased ("cherrypicked"), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker's Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness. Highlights: We crowdsource truthfulness judgments using seven dimensions rather than just one. Judgments over the seven dimensions of truthfulness are sound and reliable. The seven dimensions are independent, not redundant, and measure different aspects. Multidimensional truthfulness may inform more explainable and transparent systems.