Anomaly Detection in High-Dimensional Data. Issue 2 (3rd June 2021)

• Record Type: Journal Article
• Authors: Talagala, Priyanga Dilini; Hyndman, Rob J.; Smith-Miles, Kate
• Is Part Of: Journal of computational and graphical statistics. Volume 30:Issue 2(2021)
• Page Start: 360
• Page End: 374
• Publication Date: 2021-06-03
• Subjects: Extreme value theory -- High-dimensional data -- Nearest neighbor searching -- Temporal data -- Unsupervised outlier detection; Mathematical statistics -- Data processing -- Periodicals; Mathematical statistics -- Graphic methods -- Periodicals; 519.50285
• DOI: 10.1080/10618600.2020.1807997
• Languages: English
• ISSNs: 1061-8600

Abstract

Abstract: The HDoutliers algorithm is a powerful unsupervised algorithm for detecting anomalies in high-dimensional data, with a strong theoretical foundation. However, it suffers from some limitations that significantly hinder its performance level, under certain circumstances. In this article, we propose an algorithm that addresses these limitations. We define an anomaly as an observation where its k -nearest neighbor distance with the maximum gap is significantly different from what we would expect if the distribution of k -nearest neighbors with the maximum gap is in the maximum domain of attraction of the Gumbel distribution. An approach based on extreme value theory is used for the anomalous threshold calculation. Using various synthetic and real datasets, we demonstrate the wide applicability and usefulness of our algorithm, which we call the stray algorithm. We also demonstrate how this algorithm can assist in detecting anomalies present in other data structures using feature engineering. We show the situations where the stray algorithm outperforms the HDoutliers algorithm both in accuracy and computational time. This framework is implemented in the open source R package stray. Supplementary materials for this article are available online.