A question-guided multi-hop reasoning graph network for visual question answering. Issue 2 (March 2023)

• Record Type: Journal Article
• Authors: Xu, Zhaoyang; Gu, Jinguang; Liu, Maofu; Zhou, Guangyou; Fu, Haidong; Qiu, Chen
• Is Part Of: Information processing & management. Volume 60:Issue 2(2023)
• Publication Date: 2023-03
• Subjects: Visual question answering -- Multi-hop reasoning -- Reasoning graph network; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2022.103207
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: Visual Question Answering (VQA) requires reasoning about the visually-grounded relations in the image and question context. A crucial aspect of solving complex questions is reliable multi-hop reasoning, i.e., dynamically learning the interplay between visual entities in each step. In this paper, we investigate the potential of the reasoning graph network on multi-hop reasoning questions, especially over 3 "hops." We call this model QMRGT: A Question-Guided Multi-hop Reasoning Graph Network. It constructs a cross-modal interaction module (CIM) and a multi-hop reasoning graph network (MRGT) and infers an answer by dynamically updating the inter-associated instruction between two modalities. Our graph reasoning module can apply to any multi-modal model. The experiments on VQA 2.0 and GQA (in fully supervised and O.O.D settings) datasets show that both QMRGT and pre-training V&L models+MRGT lead to improvement on visual question answering tasks. Graph-based multi-hop reasoning provides an effective signal for the visual question answering challenge, both for the O.O.D and high-level reasoning questions. Highlights: We explore the "hop" distribution of GQA datasets spanning five types of questions. We design a multi-hop reasoning graph network to generate question instructions. We study the strength of our model in multi-hop reasoning and visual-explainable.