Relation-mining self-attention network for skeleton-based human action recognition. (July 2023)

• Record Type: Journal Article
• Authors: Gedamu, Kumie; Ji, Yanli; Gao, LingLing; Yang, Yang; Shen, Heng Tao
• Is Part Of: Pattern recognition. Volume 139(2023)
• Publication Date: 2023-07
• Subjects: Action recognition -- Relation-mining self-attention -- Pairwise self-attention -- Unary self-attention -- Position attention; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109455
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A Relation-mining Self-Attention Network (RSA-Net) is proposed for skeleton-based human action recognition that effectively capture discriminative intra-frame and inter-frame action features. The pairwise self-attention module captures the relative relationship between every two body joints. The unary self-attention module is designed to learn a general correlation feature among one key body joint over all other query joints. The position embedding attention computes the correlation between action semantics and position information independently with separate projection matrices that removes noisy correlations over heterogeneous embedding. Abstract: Modeling spatiotemporal global dependencies and dynamics of body joints are crucial to recognizing actions from 3D skeleton sequences. We propose a Relation-mining Self-Attention Network (RSA-Net) for skeleton-based human action recognition. The proposed RSA-Net is motivated by two important observations: (1) body joint relationships can be modeled independently as pairwise and unary to reduce the difficulty of action feature learning. (2) Computing action semantics and position information independently removes noisy correlations over heterogeneous embedding. The proposed RSA-Net contains pairwise self-attention, unary self-attention, and position embedding attention modules. The pairwise self-attention captures the relationship between every two body joints. The unary self-attention learns a general correlation features among one key joint over all other query joints. The position embedding attention module computes the correlation between action semantics and position information independently with separate projection matrices. Extensive evaluations are performed in the NTU-60, NTU-120, and UESTC datasets with CS, CV, CSet, and A-view evaluation benchmarks. The proposed RSA-Net outperforms existing transformer-based approaches and comparable results with state-of-the-art graph ConvNet methods. The source code is available in Github 1 .