Relational graph neural network for situation recognition. (December 2020)

• Record Type: Journal Article
• Authors: Jing, Ya; Wang, Junbo; Wang, Wei; Wang, Liang; Tan, Tieniu
• Is Part Of: Pattern recognition. Volume 108(2020:Dec.)
• Publication Date: 2020-12
• Subjects: Situation recognition -- Relationship modeling -- Graph neural network -- Reinforcement learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107544
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a novel relational graph neural network for situation recognition, which explicitly models the triplet relationships between the activity and the objects. We propose a progressive supervised learning method which incrementally adds the weights to the cross entropy loss. To harmonize the training and testing procedures, we use a policy-gradient method to directly optimize the non-differentiable value-all metric. We conduct extensive experiments to quantitatively evaluate our proposed method on the only available dataset Imsitu. Experimental results demonstrate that our proposed method performs much better than other state-of-the-art methods. Abstract: Recently, situation recognition as a new challenging task for image understanding has gained great attention, which needs to simultaneously predict the main activity (verb) and its associated objects (noun entities) in a structured and detailed way. Several methods have been proposed to handle this task, but usually they cannot effectively model the relationships between the activity and the objects. In this paper, we propose a Relational Graph Neural Network (RGNN) for situation recognition, which builds a neural graph on the activity and the objects, and models the triplet relationships between the activity and pairs of objects through message passing between graph nodes. Moreover, we propose a two-stage training strategy to optimize the model. A progressive supervised learning is first adopted to obtain an initial prediction for the activity and the objects. Then, the initial predictions are refined by using a policy-gradient method to directly optimize the non-differentiable value-all metric. To verify the effectiveness of our method, we perform extensive experiments on the Imsitu dataset which is currently the only available dataset for situation recognition. Experimental results show that our approach outperforms the state-of-the-art methods on verb and value metrics, and demonstrates better relationships between the activity and the objects.