Bidirectional difference locating and semantic consistency reasoning for change captioning. Issue 5 (19th January 2022)

• Record Type: Journal Article
• Authors: Sun, Yaoqi; Li, Liang; Yao, Tingting; Lu, Tongyv; Zheng, Bolun; Yan, Chenggang; Zhang, Hua; Bao, Yongjun; Ding, Guiguang; Slabaugh, Gregory
• Other Names: Wang Meng guestEditor.; Zhang Chi guestEditor.; Hao Shijie guestEditor.; Yu Jun guestEditor.; Mu Tingting guestEditor.
• Is Part Of: International journal of intelligent systems. Volume 37:Issue 5(2022)
• Page Start: 2969
• Page End: 2987
• Publication Date: 2022-01-19
• Subjects: change captioning -- semantic consistency reasoning -- spatial‐channel attention; Artificial intelligence -- Periodicals; Expert systems (Computer science) -- Periodicals; Intelligence artificielle -- Périodiques; Systèmes experts (Informatique) -- Périodiques; 006.3
• DOI: 10.1002/int.22821
• Languages: English
• ISSNs: 0884-8173

Abstract

Abstract: Change captioning is an emerging task to describe the changes between a pair of images. The difficulty in this task is to discover the differences between the two images. Recently, some methods have been proposed to address this problem. However, they all employ unidirectional difference localization to identify the changes. This can lead to ambiguity about the nature of the changes. Instead, we propose a framework with bidirectional difference localization and semantic consistency reasoning to describe the image changes. First, we locate the changes in the two images by capturing bidirectional differences. Then we design a decoder with spatial‐channel attention to generate the change caption. Finally, we introduce semantic consistency reasoning to constrain our bidirectional difference localization module and spatial‐channel attention module. Extensive experiments on three public data sets show that the performance of our proposed model outperforms the state‐of‐the‐art change captioning models by a large margin.