Spatio-temporal hard attention learning for skeleton-based activity recognition. (July 2023)

• Record Type: Journal Article
• Authors: Nikpour, Bahareh; Armanfard, Narges
• Is Part Of: Pattern recognition. Volume 139(2023)
• Publication Date: 2023-07
• Subjects: Temporal attention -- Spatial attention -- Spatio-temporal attention -- Activity recognition -- Skeleton data -- Deep reinforcement learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109428
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The novel problem of finding spatio-temporal hard attention in skeleton videos using deep learning for human activity recognition is discovered. A temporal agent and a special agent are proposed for respectively detecting informative frames and relevant joints in frames. The two agents are trained using deep reinforcement learning. A framework for cooperatively training the temporal and spatial agents is proposed. The effectiveness of the proposed method in improving recognition performance and the training phase run time is demonstrated using three widely used bench-mark activity recognition datasets. Graphical abstract: Abstract: The use of skeleton data for activity recognition has become prevalent due to its advantages over RGB data. A skeleton video includes frames showing two- or three-dimensional coordinates of human body joints. For recognizing an activity, not all the video frames are informative, and only a few key frames can well represent an activity. Moreover, not all joints participate in every activity; i.e., the key joints may vary across frames and activities. In this paper, we propose a novel framework for finding temporal and spatial attentions in a cooperative manner for activity recognition. The proposed method, which is called STH-DRL, consists of a temporal agent and a spatial agent. The temporal agent is responsible for finding the key frames, i.e., temporal hard attention finding, and the spatial agent attempts to find the key joints, i.e., spatial hard attention finding. We formulate the search problems as Markov decision processes and train both agents through interacting with each other using deep reinforcement learning. Experimental results on three widely used activity recognition benchmark datasets demonstrate the effectiveness of our proposed method.