Visual enhanced gLSTM for image captioning. (1st December 2021)

• Record Type: Journal Article
• Authors: Zhang, Jing; Li, Kangkang; Wang, Zhenkun; Zhao, Xianwen; Wang, Zhe
• Is Part Of: Expert systems with applications. Volume 184(2021)
• Publication Date: 2021-12-01
• Subjects: Image caption -- Visual enhanced-gLSTM -- Bag of words -- Region of interest -- Salient region; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.115462
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: A visual enhanced guiding long short-term memory is proposed for image captioning. Visual features combined with text is used to guide long short-term memory. Visual information is added to the model for avoiding gradient diminishing. Region based visual enhancement method by region of interest or salient region is proposed. Image based visual enhancement method by visual words is proposed. Abstract: For reducing the negative impact of the gradient diminishing on guiding long-short term memory (gLSTM) model in image captioning, we propose a visual enhanced gLSTM model for image caption generation. In this paper, the visual features of image's region of interest (RoI) are extracted and used as guiding information in gLSTM, in which visual information of RoI is added to gLSTM for generating more accurate image captions. Two visual enhanced methods based on region and entire image are proposed respectively. Among them the visual features from the important semantic region by CNN and the full image visual features by visual words are extracted to guide the LSTM for generating the most important semantic words. Then the visual features and text features of similar images are respectively projected to the common semantic space to obtain visual enhancement guiding information by canonical correlation analysis, and added to each memory cell of gLSTM for generating caption words. Compared with the original gLSTM method, visual enhanced gLSTM model focuses on important semantic region, which is more in line with human perception of images. Experiments on Flickr8k dataset illustrate that the proposed method can achieve more accurate image captions, and outperform the baseline gLSTM algorithm and other popular image captioning methods.