Weakly-supervised temporal attention 3D network for human action recognition. (November 2021)

• Record Type: Journal Article
• Authors: Kim, Jonghyun; Li, Gen; Yun, Inyong; Jung, Cheolkon; Kim, Joongkyu
• Is Part Of: Pattern recognition. Volume 119(2021)
• Publication Date: 2021-11
• Subjects: Action recognition -- Temporal attention -- Convolutional neural network -- Weakly-supervised learning -- Video analysis -- Video classification; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108068
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose weakly-supervised temporal attention 3D network for human action recognition, called TA3DNet. TA3DNet consists of two subnetworks: temporal frame selection and weakly supervised temporal attention. We accelerate 3D convolutional neural networks (3D CNNs) by temporally assigning different importance to each frame We adopt weakly-supervised learning to successfully train an action recognition model from given labels. Abstract: From a series of observations, we have inferred that human actions in videos are defined by a set of significant frames. In this paper, we propose a weakly-supervised temporal attention 3D network for human action recognition, called as TA3DNet, to accelerate 3D convolutional neural networks (3D CNNs) by temporally assigning different importance to each frame. First, we obtain short-term frames with long-term connection by regularly or randomly skipping frames to avoid temporal redundancy, and apply 3D convolutional layers to extract features for action recognition. Then, we apply a temporal attention module to assign different weights to each frame. We train the temporal attention module in a weakly-supervised manner that updates weights based on only class labels without event information and extra labels. Thus, TA3DNet reduces the number of input frames and constructs a lightweight network for action recognition. Experimental results demonstrate that TA3DNet achieves high performance on two challenging datasets (UCF101 and HMDB51) and outperforms state-of-the-art methods for action recognition.