Global spatio-temporal synergistic topology learning for skeleton-based action recognition. (August 2023)

• Record Type: Journal Article
• Authors: Dai, Meng; Sun, Zhonghua; Wang, Tianyi; Feng, Jinchao; Jia, Kebin
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Action recognition -- Spatio-temporal synergistic -- Skeleton -- Topology learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109540
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a global spatio-temporal synergistic topology learning network, which can learn spatio-temporal topology from the joints in synergistic way. Spatio-temporal synergistic features of the joints are captured effectively. We propose a global temporal features learning network that can capture the global temporal features of joints, thus avoiding the performance degradation caused by the loss of temporal information. In addition, we add temporal and channel subsampling, which can effectively reduce the number of parameters while maintaining the network's ability. We perform comparative experiments on three challenging datasets: NTU RGB + D, NTU RGB + D 120, and NW-UCLA. Compared with previous networks, our method achieves excellent performance with fewer parameters, which shows the effectiveness of our method. Abstract: Compared to RGB video-based action recognition, skeleton-based action recognition algorithm has attracted much more attention due to being more lightweight, better generalization and robustness. The extraction of temporal and spatial features is a crucial factor for skeleton-based action recognition. However, existing feature extraction methods suffer from two limitations: (1) the isolated extraction of temporal and spatial feature cannot capture temporal feature connections among non-adjacent joints and (2) convolution-limited perceptual fields cannot capture global temporal features of joints effectively. In this work, we propose a global spatio-temporal synergistic feature learning module (GSTL), which generates global spatio-temporal synergistic topology of joints by spatio-temporal feature fusion. By further combining the GSTL with a temporal modeling unit, we develop a powerful global spatio-temporal synergistic topology learning network (GSTLN), and it achieves competitive performance with fewer parameters on three challenge datasets: NTU RGB + D, NTU RGB + D 120, and NW-UCLA.