Time-series anomaly detection with stacked Transformer representations and 1D convolutional network. (April 2023)

• Record Type: Journal Article
• Authors: Kim, Jina; Kang, Hyeongwon; Kang, Pilsung
• Is Part Of: Engineering applications of artificial intelligence. Volume 120(2023)
• Publication Date: 2023-04
• Subjects: Time series anomaly detection -- Transformer -- Convolution Neural Network; Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2023.105964
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: Time-series anomaly detection is a task of detecting data that do not follow normal data distribution among continuously collected data. It is used for system maintenance in various industries; hence, studies on time-series anomaly detection are being carried out actively. Most of the methodologies are based on Long Short-Term Memory (LSTM) and Convolution Neural Network (CNN) to model the temporal structure of time-series data. In this study, we propose an unsupervised prediction-based time-series anomaly detection methodology using Transformer, which shows superior performance to LSTM and CNN in learning dynamic patterns of sequential data through a self-attention mechanism. The prediction model consists of an encoder comprising multiple Transformer encoder layers and a decoder that includes a 1D convolution layer. The output representation of each Transformer layer is accumulated in the encoder to obtain a representation with multi-level, rich information. The decoder fuses this representation through a 1d convolution operation. Consequently, the model can perform predictions considering both the global trend and local variability of the input time-series. The anomaly score is defined as the difference between the predicted and the actual value at the corresponding timestamp, assuming that the trained model produces the predictions that follow the normal data distribution. Finally, the data with an anomaly score above the threshold is detected as an anomaly. Experiments on the benchmark datasets show that the proposed method has performance superior to those of the baselines. Highlights: A prediction-based unsupervised time-series anomaly detection method using Transformer is proposed. The proposed method can detect anomalies, considering both the global trend and the local variability of the given time-series. The proposed method achieves superior performance compared to the baselines. The proposed method is more robust to the data characteristics.