Semantic action recognition by learning a pose lexicon. (December 2017)

• Record Type: Journal Article
• Authors: Zhou, Lijuan; Li, Wanqing; Ogunbona, Philip; Zhang, Zhengyou
• Is Part Of: Pattern recognition. Volume 72(2017:Dec.)
• Page Start: 548
• Page End: 562
• Publication Date: 2017-12
• Subjects: Lexicon -- Semantic pose -- Visual pose -- Action recognition; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2017.06.035
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel semantic representation, pose lexicon, is proposed for action recognition. An extended hidden Markov alignment model is developed to learn a pose lexicon. A semantic action recognition method that is capable of zero-shot recognition is developed upon the lexicon. The efficacy of the proposed learning and recognition algorithms were evaluated on five datasets using cross-subject, cross-dataset and zero-shot protocols. Abstract: This paper proposes a semantic representation, pose lexicon, for action recognition. The lexicon is composed of a set of semantic poses, a set of visual poses and a probabilistic mapping between the visual and semantic poses. Specially, an action can be represented by a sequence of semantic poses extracted from an associated textual instruction. Visual frames of the action are considered to be generated from a sequence of hidden visual poses. To learn the lexicon, a visual pose model is learned from training samples by a Gaussian Mixture model to characterize the likelihood of an observed visual frame being generated by a visual pose. A pose lexicon model is also learned by an extended hidden Markov alignment model to encode the probabilistic mapping between hidden visual poses and semantic poses sequences. With the lexicon, action classification is formulated as a problem of finding the maximum posterior probability of a given sequence of visual frames that fits to a given sequence of semantic poses through the most likely visual pose and alignment sequences. The efficacy of the proposed method was evaluated on MSRC-12, WorkoutSU-10, WorkoutUOW-18, Combined-15 and Combined-17 action datasets using cross-subject, cross-dataset and zero-shot protocols.