AE-Net: Fine-grained sketch-based image retrieval via attention-enhanced network. (February 2022)

• Record Type: Journal Article
• Authors: Chen, Yangdong; Zhang, Zhaolong; Wang, Yanfei; Zhang, Yuejie; Feng, Rui; Zhang, Tao; Fan, Weiguo
• Is Part Of: Pattern recognition. Volume 122(2022)
• Publication Date: 2022-02
• Subjects: Fine-grained sketch-based image retrieval (FG-SBIR) -- Residual channel attention -- Local self-spatial attention -- Contrastive learning -- Spatial sequence transformer; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108291
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel FG-SBIR model with Attention-enhanced Network (AE-Net) is established, which pays more attention to the fine-grained details of the sketches and images. We introduce three modules, i.e., the Residual Channel Attention module, Local Self-attention mechanism, and Spatial Sequence Transformer to mine the fine-grained details of the sketches and images in all dimensions. Mutual Loss is proposed to improve the traditional Triplet Loss and restrain the distance relations among the sketches/images in a single modality. Abstract: In this paper, we investigate the task of Fine-grained Sketch-based Image Retrieval (FG-SBIR), which uses hand-drawn sketches as input queries to retrieve the relevant images at the fine-grained instance level. The sketches and images come from different modalities, thus the similarity computation needs to consider both fine-grained and cross-modal characteristics. Existing solutions only focus on fine-grained details or spatial contexts, while ignoring the channel context and spatial sequence information. To mitigate such challenging problems, we propose a novel deep FG-SBIR model, which aims at inferring attention maps along channel dimension and spatial dimension, improving modules of channel attention and spatial attention, and exploring Transformer to enhance the model's ability for constructing and understanding spatial sequence information. We focus not only on the correlation information between two modalities of sketch and image, but also on the discrimination information inside the single modality. Mutual Loss is especially proposed to enhance the traditional triplet loss, and promote the internal discrimination ability of the model on a single modality. Extensive experiments show that our AE-Net obtains promising results on Sketchy, which is the largest public dataset available for FG-SBIR at present.