Dual Global Enhanced Transformer for image captioning. (April 2022)

• Record Type: Journal Article
• Authors: Xian, Tiantao; Li, Zhixin; Zhang, Canlong; Ma, Huifang
• Is Part Of: Neural networks. Volume 148(2022)
• Page Start: 129
• Page End: 141
• Publication Date: 2022-04
• Subjects: Image captioning -- Transformer -- Global information -- Visual attention -- Reinforcement learning; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.01.011
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Transformer-based architectures have shown great success in image captioning, where self-attention module can model source and target interaction (e.g., object-to-object, object-to-word, word-to-word). However, the global information is not explicitly considered in the attention weight calculation, which is essential to understand the scene content. In this paper, we propose Dual Global Enhanced Transformer (DGET) to incorporate global information in the encoding and decoding stages. Concretely, in DGET, we regard the grid feature as the visual global information and adaptively fuse it into region features in each layer by a novel Global Enhanced Encoder (GEE). During decoding, we proposed Global Enhanced Decoder (GED) to explicitly utilize the textual global information. First, we devise the context encoder to encode the existing caption generated by classic captioner as a context vector. Then, we use the context vector to guide the decoder to generate accurate words at each time step. To validate our model, we conduct extensive experiments on the MS COCO image captioning dataset and achieve superior performance over many state-of-the-art methods. Highlights: We use grid-level features to enhance the representation of region-level features. We use the existing captions generated by the classical captioner to guide the prediction of the decoder. We design three different context encoder to explore the impact of embedding existing captions on performance.