Semantics-enhanced early action detection using dynamic dilated convolution. (August 2023) - Korban, Matthew; Li, Xin
Highlights: A new Dynamic Dilated Convolutional Network that better handles temporal scale variance in incomplete actions. A new Semantic Referencing Module that helps detect ambiguous similar-looking incomplete actions. A new early action detection pipeline achieving state-of-the-art accuracy. Abstract: This paper proposes a new pipeline to perform early action detection from skeleton-based untrimmed videos. Our pipeline includes two new technical components. The first is a new Dynamic Dilated Convolutional Network (DDCN), which supports dynamic temporal sampling and makes feature learning more robust against temporal scale variance in action sequences. The second is a new semantic referencing module, which uses identified objects in the scene and their co-existence relationship with actions to adjust the probabilities of inferred actions. Such semantic guidance can help distinguish many ambiguous actions, which is a core challenge in the early detection of incomplete actions. Our pipeline achieves state-of-the-art performance in early action detection in two widely used skeleton-based untrimmed video benchmarks. The source codes are available at: https://github.com/Powercoder64/DDCN_SRM .