A question-guided multi-hop reasoning graph network for visual question answering. Issue 2 (March 2023) - Xu, Zhaoyang; Gu, Jinguang; Liu, Maofu; Zhou, Guangyou; Fu, Haidong; Qiu, Chen
Abstract: Visual Question Answering (VQA) requires reasoning about the visually-grounded relations in the image and question context. A crucial aspect of solving complex questions is reliable multi-hop reasoning, i.e., dynamically learning the interplay between visual entities in each step. In this paper, we investigate the potential of the reasoning graph network on multi-hop reasoning questions, especially over 3 "hops." We call this model QMRGT: A Question-Guided Multi-hop Reasoning Graph Network. It constructs a cross-modal interaction module (CIM) and a multi-hop reasoning graph network (MRGT) and infers an answer by dynamically updating the inter-associated instruction between two modalities. Our graph reasoning module can apply to any multi-modal model. The experiments on VQA 2.0 and GQA (in fully supervised and O.O.D settings) datasets show that both QMRGT and pre-training V&L models+MRGT lead to improvement on visual question answering tasks. Graph-based multi-hop reasoning provides an effective signal for the visual question answering challenge, both for the O.O.D and high-level reasoning questions. Highlights: We explore the "hop" distribution of GQA datasets spanning five types of questions. We design a multi-hop reasoning graph network to generate question instructions. We study the strength of our model in multi-hop reasoning and visual-explainable.