A procedure for anomaly detection and analysis. (January 2023)

• Record Type: Journal Article
• Authors: Koren, Oded; Koren, Michal; Peretz, Or
• Is Part Of: Engineering applications of artificial intelligence. Volume 117:Part A(2023)
• Publication Date: 2023-01
• Subjects: Anomaly detection -- AutoML -- Isolation forest -- Local outlier factor -- SVM; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2022.105503
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: Anomaly detection is often used to identify and remove outliers in datasets. However, detecting and analyzing the pattern of outliers can contribute to future business decisions or increase the accuracy of a learning algorithm. Selecting the applicable outlier detection method for a dataset requires human intervention and analysis due to the challenge of choosing an efficient technique suitable for all types of attributes. This work presents a procedure for anomaly detection and analysis. The procedure is feature-wise (i.e., processes each feature independently), uses T different anomaly detection techniques (for T > 1 ), and estimates the best technique using predefined thresholds. It is a generic method that does not depend on the model type and can be applied to supervised and unsupervised learning. In addition, this method does not impute or remove the outliers, as they should be adapted according to the dataset or business requirements. The significant advantage of this method is the ability to use different techniques to detect anomalies since it is applied per feature and not per record, as in traditional anomaly detection methods. Furthermore, the method uses a new measure, Noise Ratio (NR), which describes the level of agreement between our method's result and traditional anomaly detection techniques. The results showed that all the compared techniques identified non-anomalous features with consistent results between the various algorithms. In the proposed method, NR found up to 20% of the non-anomalous values marked as outliers and improved up to 10% in finding outliers in datasets compared to traditional anomaly detection algorithms.