Toward automating post processing of aquatic sensor data. (May 2022)

• Record Type: Journal Article
• Authors: Jones, Amber Spackman; Jones, Tanner Lex; Horsburgh, Jeffery S.
• Is Part Of: Environmental modelling & software. Volume 151(2022)
• Publication Date: 2022-05
• Subjects: Aquatic sensors -- Quality control -- Anomaly detection -- Python -- Data management -- Software and data availability; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2022.105364
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: Sensors measuring environmental phenomena at high frequency commonly report anomalies related to fouling, sensor drift and calibration, and datalogging and transmission issues. Suitability of data for analyses and decision making often depends on manual review and adjustment of data. Machine learning techniques have potential to automate identification and correction of anomalies, streamlining the quality control process. We explored approaches for automating anomaly detection and correction of aquatic sensor data for implementation in a Python package (pyhydroqc). We applied both classical and deep learning time series regression models that estimate values, identify anomalies based on dynamic thresholds, and offer correction estimates. Techniques were developed and performance assessed using data reviewed, corrected, and labeled by technicians in an aquatic monitoring use case. Auto-Regressive Integrated Moving Average (ARIMA) consistently performed best, and aggregating results from multiple models improved detection. pyhydroqc includes custom functions and a workflow for anomaly detection and correction. Highlights: pyhydroqc is a Python package for quality control of environmental sensor data. pyhydroqc includes functions for detection and correction of time series anomalies. Functionality was tested on aquatic sensor data from the Logan River Observatory. Time series models identified anomalies that were compared to technician labels. pyhydroqc can streamline quality control, reducing the need for manual data review.