Bi-Attention enhanced representation learning for image-text matching. (August 2023)

• Record Type: Journal Article
• Authors: Tian, Yumin; Ding, Aqiang; Wang, Di; Luo, Xuemei; Wan, Bo; Wang, Yifeng
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Image-text matching -- Bi-attention -- Polynomial loss; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109548
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel image-text representation learning network BAERL is proposed. It captures both inter- and intra-modality correlations between image regions and words. Self-similarity polynomial loss is used to enhance the matching performance. Extensive experimental results demonstrate its superior performance. Abstract: Image-text matching has become a research hotspot in recent years. The key point of image-text matching is to accurately measure the similarity between an image and a sentence. However, most existing methods either focus on the inter-modality similarities between regions in image and words in text or the intra-modality similarities within image regions or words, such that they cannot well exploit detailed correlations between images and texts. Furthermore, existing methods typically train their models using a triplet ranking loss, which relies on the similarity of randomly sampled triples. Since the weights of positive and negative samples are not adjusted, it cannot provide enough gradient information for training, resulting in slow convergence and limited performance. To address the above problems, we propose an image-text matching method named Bi-Attention Enhanced Representation Learning (BAERL). It builds a self-attention learning sub-network to exploit intra-modality correlations within image regions or words and a co-attention learning sub-network to exploit inter-modality correlations between image regions and words. Then, representations obtained by two sub-networks capture holistic correlations between images and texts. In addition, BAERL uses the self-similarity polynomial loss instead of triplet ranking loss to train the model. The self-similarity polynomial loss can adaptively assign appropriate weights to different pairs based on their similarity scores so as to further improve the retrieval performance. Experiments on two benchmark datasets demonstrate the superior performance of the proposed BAERL method over several state-of-the-art methods.