SGM-Net: Skeleton-guided multimodal network for action recognition. (August 2020)

• Record Type: Journal Article
• Authors: Li, Jianan; Xie, Xuemei; Pan, Qingzhe; Cao, Yuhan; Zhao, Zhifu; Shi, Guangming
• Is Part Of: Pattern recognition. Volume 104(2020:Aug.)
• Publication Date: 2020-08
• Subjects: Action recognition -- multi-modality -- skeleton-guided; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107356
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose skeleton-guided multimodal network (SGM-Net). The guided block in SGM-Net is proposed to obtain action-related object feature. Two schemes of correlation operation in guided block are investigated. The experimental results demonstrate that our method achieves state-of-the-art performance on NTU and Sub-JHMDB datasets. Abstract: Single-modality human action recognition on RGB or skeleton has been extensively studied. Each of these two modalities has its own advantages as well as limitations, because they depict action from different perspectives. The feature of different modalities can complement each other for describing actions. Therefore, it is meaningful to fuse these two modalities using their complementarity for action recognition. However, existing multimodal methods fail to fully exploit the complementarity of RGB and skeleton modalities. In this paper, we propose a Skeleton-Guided Multimodal Network (SGM-Net) for human action recognition. The proposed method takes full use of the complementarity of these two modalities at semantic feature level. From the technical perspective, we introduce a guided block, the key component of SGM-Net. It enables skeleton feature to guide on RGB feature, so that the important RGB information strongly related to the action is enhanced. Moreover, in the guided block, two schemes of correlation operation are explored. We perform a series of ablation experiments to verify the effectiveness of the guided block. The experimental results show that our approach achieves state-of-the-art performance over the existing methods on NTU and Sub-JHMDB datasets.