A multimodal deep fusion graph framework to detect social distancing violations and FCGs in pandemic surveillance. (August 2021)

• Record Type: Journal Article
• Authors: Varghese, Elizabeth B.; Thampi, Sabu M.
• Is Part Of: Engineering applications of artificial intelligence. Volume 103(2021)
• Publication Date: 2021-08
• Subjects: Pandemic surveillance -- Multimodal data fusion -- Social distancing -- Free-standing conversation group (FCG) -- Convolutional Variational Autoencoder (CVAE); Engineering -- Data processing -- Periodicals; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Ingénierie -- Informatique -- Périodiques; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; Artificial intelligence; Engineering -- Data processing; Expert systems (Computer science); Periodicals; 620.00285
• DOI: 10.1016/j.engappai.2021.104305
• Languages: English
• ISSNs: 0952-1976

Abstract

Abstract: In pandemic surveillance, ensuring social distance has emerged as a challenging issue due to the lack of proper therapeutic agents, and this envisages the need for automated social distance monitoring to avoid the formation of social gatherings and free-standing conversation groups (FCGs). The robustness sought in detecting these groups cannot be achieved when there are illumination variation and occlusion among subjects by solely relying on video data from distributed cameras. In this paper, we propose a deep learning framework for integrating data from multiple sensor modalities taking into account the spatial properties necessary to manage illumination variation and occlusion of video data. From the fused data, social distance compliance violations are notified by the presence of social groups as graphs detected using a pre-trained deep framework and connected components in graph theory. A cost function is devised for social group graph clustering to identify FCGs by using the socio-psychological theory of Friends-formation. Experiment analysis on four benchmark datasets shows that the proposed approach excels at detecting social distance violations and FCGs, and succeeds in analyzing the potential risk of pandemic spread in an area by the calculation of violation scores and rate of violation. Highlights: Proposed convolutional variational autoencoder based multimodal fusion to detect social distancing violations. Detected violations from the fusion of multimodal data by identifying social groups. Addressed the problem of occlusion and illumination variation through multimodal fusion. Defined a cost function for social group graph clustering to detect F-formations.