Few-shot activity recognition with cross-modal memory network. (December 2020)

• Record Type: Journal Article
• Authors: Zhang, Lingling; Chang, Xiaojun; Liu, Jun; Luo, Minnan; Prakash, Mahesh; Hauptmann, Alexander G.
• Is Part Of: Pattern recognition. Volume 108(2020:Dec.)
• Publication Date: 2020-12
• Subjects: Few-shot learning -- Activity recognition -- Cross-modal memory; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107348
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose an end-to-end framework for few-shot activity recognition, which consists deep embedding module, cross-modal memory module, and the few-shot activity recognition module. We design an innovative cross-modal memory structure, where each memory slot is an visual-textual embedding pair that stores the multi-modal semantic information for one activity attribute. We conduct extensive experiments on datasets HMDB51 and UCF101 to illustrate the effectiveness and superiority of the cross-modal memory network. Abstract: Deep learning based action recognition methods require large amount of labelled training data. However, labelling large-scale video data is time consuming and tedious. In this paper, we consider a more challenging few-shot action recognition problem where the training samples are few and rare. To solve this problem, memory network has been designed to use an external memory to remember the experience learned in training and then apply it to few-shot prediction during testing. However, existing memory-based methods just update the visual information with fixed label embeddings in the memory, which cannot adapt well to novel activities during testing. To alleviate the issue, we propose a novel end-to-end cross-modal memory network for few-shot activity recognition. Specifically, the proposed memory architecture stores the dynamic visual and textual semantics for some high-level attributes related to human activities. And the learned memory can provide effective multi-modal information for new activity recognition in the testing stage. Extensive experimental results on two video datasets, including HMDB51 and UCF101, indicate that our method could achieve significant improvements over other previous methods.