The value of human data annotation for machine learning based anomaly detection in environmental systems. (1st November 2021)

• Record Type: Journal Article
• Authors: Russo, Stefania; Besmer, Michael D.; Blumensaat, Frank; Bouffard, Damien; Disch, Andy; Hammes, Frederik; Hess, Angelika; Lürig, Moritz; Matthews, Blake; Minaudo, Camille; Morgenroth, Eberhard; Tran-Khac, Viet; Villez, Kris
• Is Part Of: Water research. Volume 206(2021)
• Publication Date: 2021-11-01
• Subjects: Machine learning -- Anomaly detection -- Environmental systems -- Labels; Water -- Pollution -- Research -- Periodicals; 363.7394
• DOI: 10.1016/j.watres.2021.117695
• Languages: English
• ISSNs: 0043-1354

Abstract

Highlights: Supervised and unsupervised models are evaluated on environmental data. Anomaly detection performance and labelling efforts are taken into account. Access to expert-based data annotation is critical. The performance of a particular algorithm depends on the application domain. A preference for a setup cannot be based on anomaly detection performance alone. Abstract: Anomaly detection is the process of identifying unexpected data samples in datasets. Automated anomaly detection is either performed using supervised machine learning models, which require a labelled dataset for their calibration, or unsupervised models, which do not require labels. While academic research has produced a vast array of tools and machine learning models for automated anomaly detection, the research community focused on environmental systems still lacks a comparative analysis that is simultaneously comprehensive, objective, and systematic. This knowledge gap is addressed for the first time in this study, where 15 different supervised and unsupervised anomaly detection models are evaluated on 5 different environmental datasets from engineered and natural aquatic systems. To this end, anomaly detection performance, labelling efforts, as well as the impact of model and algorithm tuning are taken into account. As a result, our analysis reveals the relative strengths and weaknesses of the different approaches in an objective manner without bias for any particular paradigm in machine learning. Most importantly, our results show that expert-based data annotation is extremely valuable for anomaly detection based on machine learning.