Spatiotemporal consistency-enhanced network for video anomaly detection. (January 2022)

• Record Type: Journal Article
• Authors: Hao, Yi; Li, Jie; Wang, Nannan; Wang, Xiaoyu; Gao, Xinbo
• Is Part Of: Pattern recognition. Volume 121(2022)
• Publication Date: 2022-01
• Subjects: Anomaly detection -- Unsupervised learning -- Spatiotemporal consistency; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108232
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a spatiotemporal consistency-enhanced network (STCEN) to highlight the disturbances of abnormal data from both spatial and temporal aspects. A 3D CNN-based discriminator is designed to measure the spatiotemporal consistency between the generated future frame and its former frames. A well-designed 3D-2D U-shape structure is introduced to extract motion and appearance fusion features from the input video clip, which focuses on extracting spatiotemporal high-level features and generating a rational future frame. Moreover, a resampling module is used to enlarge the score gaps between normal and abnormal contents for the video anomaly detection task. Extensive experiments on three datasets demonstrate the potential of our model with competitive performance compared with state-of-the-art approaches. We also provide discussions for these datasets, which could be useful for future works. Abstract: Video anomaly detection aims to detect abnormal segments in a video sequence, which is a key problem in video surveillance. Based on deep prediction methods, we propose a spatiotemporal consistency-enhanced network to generate spatiotemporal consistency predictions. A 3D CNN-based encoder and 2D CNN-based decoder constitute the main part of our model. A resampling strategy is applied to the latent space vector when the model is trained by the normal data, yet this can cause the model to perform poorly if the data include abnormal data. Moreover, we combine an input clip with a generated frame into a reformed video clip, which is then fed into a discriminator that is constructed by the 3D CNN to evaluate the consistency of the input clip. Owing to the adversarial training between the generator and discriminator, the spatiotemporal consistency of the generated results is enhanced. During the testing stage, the abnormal data generates a different appearance and motion changes, which affect the ability of our model to predict spatiotemporal consistency in future images. Then, the prediction quality gap between normal and anomalous contents is used to infer whether anomalies occur. Extensive experiments confirm that the proposed method achieves state-of-the-art performance on three benchmark datasets, including ShanghaiTech, CUHK Avenue, and UCSD Ped2.