Multi-stream CNN: Learning representations based on human-related regions for action recognition. (July 2018)

• Record Type: Journal Article
• Authors: Tu, Zhigang; Xie, Wei; Qin, Qianqing; Poppe, Ronald; Veltkamp, Remco C.; Li, Baoxin; Yuan, Junsong
• Is Part Of: Pattern recognition. Volume 79(2018:Jul.)
• Page Start: 32
• Page End: 43
• Publication Date: 2018-07
• Subjects: Convolutional Neural Network -- Action recognition -- Multi-Stream -- Motion salient region; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2018.01.020
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Presenting a multi-stream CNN architecture to incorporate multiple complementary features trained in appearance and motion networks. Demonstrating that using full-frame, human body, and motion-salient body part regions together is effective to improve recognition performance. Proposing methods to detect the actor and motion-salient body part precisely. Verifying that high-quality flow is critically important to learn accurate video representations for action recognition. Abstract: The most successful video-based human action recognition methods rely on feature representations extracted using Convolutional Neural Networks (CNNs). Inspired by the two-stream network (TS-Net), we propose a multi-stream Convolutional Neural Network (CNN) architecture to recognize human actions. We additionally consider human-related regions that contain the most informative features. First, by improving foreground detection, the region of interest corresponding to the appearance and the motion of an actor can be detected robustly under realistic circumstances. Based on the entire detected human body, we construct one appearance and one motion stream. In addition, we select a secondary region that contains the major moving part of an actor based on motion saliency. By combining the traditional streams with the novel human-related streams, we introduce a human-related multi-stream CNN (HR-MSCNN) architecture that encodes appearance, motion, and the captured tubes of the human-related regions. Comparative evaluation on the JHMDB, HMDB51, UCF Sports and UCF101 datasets demonstrates that the streams contain features that complement each other. The proposed multi-stream architecture achieves state-of-the-art results on these four datasets.