Multi‐frame based adversarial learning approach for video surveillance. (February 2022)

• Record Type: Journal Article
• Authors: Patil, Prashant W.; Dudhane, Akshay; Chaudhary, Sachin; Murala, Subrahmanyam
• Is Part Of: Pattern recognition. Volume 122(2022)
• Publication Date: 2022-02
• Subjects: Temporal sampling -- Multi-scale adversarial learning -- Foreground-background segmentation and video surveillance; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108350
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Recent advancements in transportation systems, artificial intelligence and surveillance cameras have shown that the extracting moving or foreground objects in video i.e. Foreground-background Segmentation (FBS), plays an important role in many automated video processing applications. The temporal encoding mechanism-based generative adversarial learning framework for the foreground-background segmentation task is proposed with multi-scale inception and residual connection based dense module. Unlike traditional training-testing of the network, we have analysed the learning of network in different ways like cross-data, disjoint and global training-testing for FBS. To the best of our knowledge, this is the first end-to-end recurrent generative adversarial learning framework with multi-scale inception and residual connection based dense module for FBS. The effectiveness (qualitatively and quantitatively) of the proposed approach is examined with different training-testing techniques (cross-data, disjoint and global training-testing) on three different benchmark video databases, namely Grayscale-Thermal Foreground Detection (GTFD) [6], Densely Annotated VIdeo Segmentation (DAVIS)-2016, ChangeDetection.net (CDnet)-2014) for FBS task. Abstract: Foreground-background segmentation (FBS) is one of the prime tasks for automated video-based applications like traffic analysis and surveillance. The different practical scenarios like weather degraded videos, irregular moving objects, dynamic background, etc., make FBS a challenging task. The existing FBS algorithms mainly depend on one of the three different factors, namely (1) complicated training process, (2) additionally trained modules for other applications, or (3) neglect the inter-frame spatio-temporal structural dependencies. In this paper, a novel multi-frame-based adversarial learning network is proposed with multi-scale inception and residual module for FBS. As, FBS is a temporal enlightenment-based problem, a temporal encoding mechanism with decreasing variable intervals is proposed for the input frame selection. The proposed network comprises multi-scale inception and residual connection-based dense modules to learn prominent features of the foreground object(s). Also, feedback of the estimated foreground map of previous frame is utilized to exhibit more temporal consistency. Learning of the network is concentrated in different ways like cross-data, disjoint, and global training-testing for FBS. The qualitative and quantitative experimental analysis of the proposed approach is done on three benchmark datasets for FBS. Experimental analysis on three benchmark datasets proves the significance of the proposed approach as compared to state-of-the-art FBS approaches.