Visual-Text Reference Pretraining Model for Image Captioning. (21st January 2022)

• Record Type: Journal Article
• Authors: Li, Pengfei; Zhang, Min; Lin, Peijie; Wan, Jian; Jiang, Ming
• Other Names: Doulamis Anastasios D. Academic Editor.
• Is Part Of: Computational intelligence and neuroscience. Volume 2022(2022)
• Publication Date: 2022-01-21
• Subjects: Neurosciences -- Data processing -- Periodicals; Computational intelligence -- Periodicals; Computational neuroscience -- Periodicals; 612.80285
• DOI: 10.1155/2022/9400999
• Languages: English
• ISSNs: 1687-5265

Abstract

Abstract : People can accurately describe an image by constantly referring to the visual information and key text information of the image. Inspired by this idea, we propose the VTR-PTM (Visual-Text Reference Pretraining Model) for image captioning. First, based on the pretraining model (BERT/UNIML), we design the dual-stream input mode of image reference and text reference and use two different mask modes (bidirectional and sequence to sequence) to realize the VTR-PTM suitable for generating tasks. Second, the target dataset is used to fine tune the VTR-PTM. To the best of our knowledge, VTR-PTM is the first reported pretraining model to use visual-text references in the learning process. To evaluate the model, we conduct several experiments on the benchmark datasets of image captioning, including MS COCO and Visual Genome, and achieve significant improvements on most metrics. The code is available at https://github.com/lpfworld/VTR-PTM .