Self-Attention based fine-grained cross-media hybrid network. (October 2022)

• Record Type: Journal Article
• Authors: Shan, Wei; Huang, Dan; Wang, Jiangtao; Zou, Feng; Li, Suwen
• Is Part Of: Pattern recognition. Volume 130(2022)
• Publication Date: 2022-10
• Subjects: Fine-Grained -- Cross-Media -- Retrieval -- Attention; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2022.108748
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Propose a common attention space learning method for different feature spaces. Jointly learn the relative position encoding of local features and the spatial relationship between features. Achieve state-of-the-art performance on fine-grained cross-media retrieval task. Abstract: Due to the heterogeneity gap, the data representations of different types of media are inconsistent. It is challenging to measure the fine-grained gap between different media. To this end, we propose a self-attention-based hybrid network to learn the common representations of different media data. Specifically, we first utilize a local self-attention layer to learn the common attention space between different media data. Then we propose a similarity concatenation method to understand the content relationship between features. To further improve the robustness of the model, we also learn a local position encoding to capture the spatial relationships between features. Therefore, our proposed approach can effectively reduce the gap between different feature distributions on cross-media retrieval tasks. Extensive experiments and ablation studies demonstrate that our proposed method achieves state-of-the-art performance. The source code and models are publicly available at: https://github.com/NUST-Machine-Intelligence-Laboratory/SAFGCMHN .