Automated monitoring of online news accuracy with change classification models. Issue 6 (November 2022)

• Record Type: Journal Article
• Authors: Timmerman, Yoram; Bronselaer, Antoon
• Is Part Of: Information processing & management. Volume 59:Issue 6(2022)
• Publication Date: 2022-11
• Subjects: News accuracy -- Liquid news -- Error corrections -- Supervised machine learning; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2022.103105
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: In the past decade, news consumption has shifted from printed news media to online alternatives. Although these come with advantages, online news poses challenges as well. Notable here is the increased competition between online newspapers and other online news providers to attract readers. Hereby, speed is often favored over quality. As a consequence, the need for new tools to monitor online news accuracy has grown. In this work, a fundamentally new and automated procedure for the monitoring of online news accuracy is proposed. The approach relies on the fact that online news articles are often updated after initial publication, thereby also correcting errors. Automated observation of the changes being made to online articles and detection of the errors that are corrected may offer useful insights concerning news accuracy. The potential of the presented automated error correction detection model is illustrated by building supervised classification models for the detection of objective, subjective and linguistic errors in online news updates respectively. The models are built using a large news update data set being collected during two consecutive years for six different Flemish online newspapers. A subset of 21, 129 changes is then annotated using a combination of automated and human annotation via an online annotation platform. Finally, manually crafted features and text embeddings obtained by four different language models (TF-IDF, word2vec, BERTje and SBERT) are fed to three supervised machine learning algorithms (logistic regression, support vector machines and decision trees) and performance of the obtained models is subsequently evaluated. Results indicate that small differences in performance exist between the different learning algorithms and language models. Using the best-performing models, F 2 -scores of 0.45, 0.25 and 0.80 are obtained for the classification of objective, subjective and linguistic errors respectively. Highlights: Online news accuracy can be monitored by exploiting information in news updates. A dataset consisting of 21, 129 changes to online news is gathered and annotated. Automated classification models are built for error detection in online news. Performance of three learning algorithms and four language models is evaluated. Good predictive capability is obtained for objective and linguistic error detection.