Skeleton-based relational reasoning for group activity analysis. (February 2022)

• Record Type: Journal Article
• Authors: Perez, Mauricio; Liu, Jun; Kot, Alex C.
• Is Part Of: Pattern recognition. Volume 122(2022)
• Publication Date: 2022-02
• Subjects: Group activity recognition -- Skeleton information -- Relational network -- Attention mechanisms; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108360
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Group activity recognition using skeletons to reason about the actors interactions. Novel method describes the interactions via learned relations between human joints. Usage of previously unexplored interactions between humans and objects. Leveraging of individual action information for higher quality feature learning. Tailored attention mechanism over relations gives more importance to key individuals. Abstract: Research on group activity recognition mostly leans on the standard two-stream approach (RGB and Optical Flow) as their input features. Few have explored explicit pose information, with none using it directly to reason about the persons interactions. In this paper, we leverage the skeleton information to learn the interactions between the individuals straight from it. With our proposed method GIRN, multiple relationship types are inferred from independent modules, that describe the relations between the body joints pair-by-pair. Additionally to the joints relations, we also experiment with the previously unexplored relationship between individuals and relevant objects (e.g. volleyball). The individuals distinct relations are then merged through an attention mechanism, that gives more importance to those individuals more relevant for distinguishing the group activity. We evaluate our method in the Volleyball dataset, obtaining competitive results to the state-of-the-art. Our experiments demonstrate the potential of skeleton-based approaches for modeling multi-person interactions.