Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa. (June 2023)

• Record Type: Journal Article
• Authors: Zeng, Gangyan; Zhang, Yuan; Zhou, Yu; Yang, Xiaomeng; Jiang, Ning; Zhao, Guoqing; Wang, Weiping; Yin, Xu-Cheng
• Is Part Of: Pattern recognition. Volume 138(2023)
• Publication Date: 2023-06
• Subjects: Textvqa -- End-to-end -- Scene text reading -- Scene text reasoning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109337
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: In TextVQA task, text reading and text reasoning modules can be mutually reinforced. A visually enhanced text embedding is proposed to suppress the cumulative error propagation from reading errors. Two schemes are developed to improve text reading via the contextual information in the downstream reasoning. Our method achieves state-of-the-art results in terms of accuracy and robustness. Abstract: Text-based visual question answering (TextVQA), which answers a visual question by considering both visual contents and scene texts, has attracted increasing attention recently. Most existing methods employ an optical character recognition (OCR) module as a pre-processor to read texts, then combine it with a visual question answering (VQA) framework. However, inaccurate OCR results may lead to cumulative error propagation, and the correlation between text reading and text-based reasoning is not fully exploited. In this work, we integrate OCR into the flow of TextVQA, targeting the mutual reinforcement of OCR and VQA tasks. Specifically, a visually enhanced text embedding module is proposed to predict semantic features from the visual information of texts, by which texts can be reasonably understood even without accurate recognition. Further, two elaborate schemes are developed to leverage contextual information in VQA to modify OCR results. The first scheme is a reading modification module that adaptively selects the answer results according to the contexts. Second, we propose an efficient end-to-end text reading and reasoning network, where the downstream VQA signal contributes to the optimization of text reading. Extensive experiments show that our method outperforms existing alternatives in terms of accuracy and robustness, whether ground truth OCR annotations are used or not.