Feedback evaluations to promote image captioning. Issue 13 (23rd September 2020)

• Record Type: Journal Article
• Authors: He, Jun; Zhao, Yijia; Sun, Bo; Yu, Lejun
• Is Part Of: IET image processing. Volume 14:Issue 13(2020)
• Page Start: 3021
• Page End: 3027
• Publication Date: 2020-09-23
• Subjects: feature extraction -- neural nets -- learning (artificial intelligence) -- image retrieval -- text analysis -- information retrieval; feedback evaluations -- policy gradient problem -- retrieval model -- discriminability score -- image captioning evaluation metrics -- caption‐generating process -- auxiliary retrieval loss -- ARL -- generated caption -- feedback evaluation method -- evaluation reward -- captioning process -- captioning model; Image processing -- Periodicals; 621.36705
• DOI: 10.1049/iet-ipr.2019.1317
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract : Image captioning can be treated as a policy gradient problem. A retrieval model to obtain the discriminability score to distinguish between two images, given the caption for one of them, has been proposed previously; the discriminability score and one of the image captioning evaluation metrics were optimised using policy gradient. Based on this, two methods to evaluate the caption and caption‐generating process, referred to as feedback evaluations, are proposed in this study. The results of the evaluations were used to improve the model. First, an auxiliary retrieval loss (ARL) is introduced to evaluate the generated caption to improve the discriminability of the model. ARL has been utilised as a feedback evaluation method because it calculates similarity between the generated caption and convolutional neural network features. With ARL, a higher similarity and better discriminability were achieved. Second, an evaluation reward is introduced to evaluate the captioning process. With ER, the overall evaluation metrics can be improved. A policy gradient was used, and a captioning model could be trained by jointly adjusting the captioning process and captioning itself. The attention long short‐term memory network was trained with ARL and ER successively and it demonstrated state‐of‐the‐art performance on the COCO database.