Evaluating the generalisability of neural rumour verification models. Issue 1 (January 2023)

• Record Type: Journal Article
• Authors: Kochkina, Elena; Hossain, Tamanna; Logan, Robert L.; Arana-Catania, Miguel; Procter, Rob; Zubiaga, Arkaitz; Singh, Sameer; He, Yulan; Liakata, Maria
• Is Part Of: Information processing & management. Volume 60:Issue 1(2023)
• Publication Date: 2023-01
• Subjects: Rumour verification -- Generalisability -- Rumour dataset -- Deep learning; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2022.103116
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: Research on automated social media rumour verification, the task of identifying the veracity of questionable information circulating on social media, has yielded neural models achieving high performance, with accuracy scores that often exceed 90%. However, none of these studies focus on the real-world generalisability of the proposed approaches, that is whether the models perform well on datasets other than those on which they were initially trained and tested. In this work we aim to fill this gap by assessing the generalisability of top performing neural rumour verification models covering a range of different architectures from the perspectives of both topic and temporal robustness. For a more complete evaluation of generalisability, we collect and release COVID-RV, a novel dataset of Twitter conversations revolving around COVID-19 rumours. Unlike other existing COVID-19 datasets, our COVID-RV contains conversations around rumours that follow the format of prominent rumour verification benchmarks, while being different from them in terms of topic and time scale, thus allowing better assessment of the temporal robustness of the models. We evaluate model performance on COVID-RV and three popular rumour verification datasets to understand limitations and advantages of different model architectures, training datasets and evaluation scenarios. We find a dramatic drop in performance when testing models on a different dataset from that used for training. Further, we evaluate the ability of models to generalise in a few-shot learning setup, as well as when word embeddings are updated with the vocabulary of a new, unseen rumour. Drawing upon our experiments we discuss challenges and make recommendations for future research directions in addressing this important problem. Highlights: We test the generalisability of rumour verification models across several datasets. We release a novel dataset of Twitter conversations discussing COVID-19 rumours. We examine various factors affecting performance drop. We show the benefits of using few-shot learning and COVID-19 informed embeddings.