Semantics-enhanced early action detection using dynamic dilated convolution. (August 2023)

• Record Type: Journal Article
• Authors: Korban, Matthew; Li, Xin
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Early action detection -- Action semantics -- Dilated convolutional network; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109595
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A new Dynamic Dilated Convolutional Network that better handles temporal scale variance in incomplete actions. A new Semantic Referencing Module that helps detect ambiguous similar-looking incomplete actions. A new early action detection pipeline achieving state-of-the-art accuracy. Abstract: This paper proposes a new pipeline to perform early action detection from skeleton-based untrimmed videos. Our pipeline includes two new technical components. The first is a new Dynamic Dilated Convolutional Network (DDCN), which supports dynamic temporal sampling and makes feature learning more robust against temporal scale variance in action sequences. The second is a new semantic referencing module, which uses identified objects in the scene and their co-existence relationship with actions to adjust the probabilities of inferred actions. Such semantic guidance can help distinguish many ambiguous actions, which is a core challenge in the early detection of incomplete actions. Our pipeline achieves state-of-the-art performance in early action detection in two widely used skeleton-based untrimmed video benchmarks. The source codes are available at: https://github.com/Powercoder64/DDCN_SRM .