Re-ranking image-text matching by adaptive metric fusion. (August 2020)

• Record Type: Journal Article
• Authors: Niu, Kai; Huang, Yan; Wang, Liang
• Is Part Of: Pattern recognition. Volume 104(2020:Aug.)
• Publication Date: 2020-08
• Subjects: Image-text matching -- Re-ranking method -- Adaptive metric fusion; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107351
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The uni-modal re-ranking methods are not suitable for the image-text matching. Fusing different metrics can provide a comprehensive similarity evaluation. Considering two directions in image-text matching improves re-ranking performance. Alleviating differences of visual and textual feature spaces improves performance. Our re-ranking method outperforms other methods on two representative benchmarks. Abstract: Image-text matching has drawn much attention recently with the rapid growth of multi-modal data. Many effective approaches have been proposed to solve this challenging problem, but limited effort has been devoted to re-ranking methods. Compared with the uni-modal re-ranking methods, modality heterogeneity is the major difficulty when designing a re-ranking method in the cross-modal field, which mainly lies in two aspects of different visual and textual feature spaces and different distributions in inverse directions. In this paper, we propose a heuristic re-ranking method called Adaptive Metric Fusion (AMF) for image-text matching. The method can obtain a better metric by adaptively fusing metrics based on two modules: 1) Cross-modal Reciprocal Encoding, which considers ranks in inverse directions to comprehensively evaluate a metric. The sentence retrieval and image retrieval have different distribution characteristics and galleries in different modalities, thus it is necessary to exploit them simultaneously for appropriate metric fusion. 2) Query Replacement Gap, which quantifies the gap between cross-modal and uni-modal similarities to alleviate the influence of different visual and textual feature spaces on the fused metric. The proposed re-ranking method can be implemented in an unsupervised way without requiring any human interaction or annotated data, and can be easily applied to any initial ranking result. Extensive experiments and analysis validate the effectiveness of our method on the large-scale MS-COCO and Flickr30K datasets.