2PESNet: Towards online processing of temporal action localization. (November 2022)

• Record Type: Journal Article
• Authors: Kim, Young Hwi; Nam, Seonghyeon; Kim, Seon Joo
• Is Part Of: Pattern recognition. Volume 131(2022)
• Publication Date: 2022-11
• Subjects: Online video understanding -- Temporal action localization; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108871
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The new task which aims to detect instance-level actions instantly from the streaming videos. 2-pass action instance detection method that can be applied in the online setting. Two-stage action end detection module to gain the accurate action end point with low delay of response. Extensive experiments on THUMOS'14 and ActivityNet v1.3 that show effectiveness and robustness of the model. Abstract: Existing online video processing methods such as online action detection focus on a frame-level understanding for high responsiveness. However, it has a fundamental limitation in that it lacks instance-level understanding of videos, making it difficult to be applied to higher-level vision tasks. The instance-level action detection, known as Temporal Action Localization (TAL), have limitations when applying to the online settings. In this work, we introduce a new task that aims to detect action instances of videos in an online setting, named Online Temporal Action Localization (OnTAL). To tackle this problem, we propose a 2-Pass End/Start detection Network (2PESNet) that detects action instances by effectively finding the start and end of an action instance. Additionally, we propose a two-stage action end detection method to further improve the performance. Extensive experiments on THUMOS'14 and ActivityNet v1.3 demonstrate that our model is able to take both accuracy and responsiveness when predicting action instances from streaming videos.