Sketch-based image retrieval with deep visual semantic descriptor. (April 2018)

• Record Type: Journal Article
• Authors: Huang, Fei; Jin, Cheng; Zhang, Yuejie; Weng, Kangnian; Zhang, Tao; Fan, Weiguo
• Is Part Of: Pattern recognition. Volume 76(2018:Apr.)
• Page Start: 537
• Page End: 548
• Publication Date: 2018-04
• Subjects: Sketch-based image retrieval (SBIR) -- Deep learning -- Deep visual semantic descriptor -- Sketch-like transformation -- Re-ranking optimization -- Multiple feature fusion -- Accelerated hierarchical K-means clustering; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2017.11.032
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Novel Real-time SBIR is built by employing multimodal deep sketch-image matching. Deep Visual Semantic Descriptor is created to bridge sketch-image multimodal gap. Sketch-like Transformation is established to improve sketch-image resemblance. Re-ranking Optimization is introduced to characterize sketch-image correlation. Abstract: Sketch-based Image Retrieval (SBIR) has received a lot of attentions recently. In this paper we aim to enhance SBIR with deep visual semantic descriptor and related optimization mechanisms. Our scheme significantly differs from other earlier work in: 1) A feature representation via deep visual semantic descriptor is established to bridge the gap between sketches and images, which can encode both low-level local features and high-level semantic features; 2) A clustering-based re-ranking optimization is introduced to further improve SBIR by dynamically adjusting the correlations of images in the ranking list. The main contribution of our work is that we effectively apply the deep visual semantic descriptor to enable deep sketch-image matching, which has provided a more reasonable base for us to fuse local low-level visual features with high-level semantic features by determining an optimal correlated mapping. Our experiments on a large number of public data have obtained very positive results.