A deformable CNN-based triplet model for fine-grained sketch-based image retrieval. (May 2022)

• Record Type: Journal Article
• Authors: Zhang, Xianlin; Shen, Mengling; Li, Xueming; Feng, Fangxiang
• Is Part Of: Pattern recognition. Volume 125(2022)
• Publication Date: 2022-05
• Subjects: Freehand sketches -- FG-SBIR -- Semantic attributes -- Deformable CNNs -- Preprocessing; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2021.108508
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a new way to generate pseudo sketches, which can be used alone as an artistic creation tool, at the preprocessing step. Experiments consistently illustrate the superior performance of the proposed method. We propose a novel multi-task FG-SBIR structure which takes advantage of deformable convolutional neural networks while, at the same time, taking into consideration of freehand sketch attributes to reduce the semantic gap. We build a new clothing fine grained sketch dataset, which has 2000 sketch-image pairs and rich semantic attribute annotations for FG-SBIR, for the first time. Abstract: With the popularity of electronic touch-screen and pressure sensing devices, fine-grained sketch based image retrieval (FG-SBIR) has become a research hotspot. In this paper, we stress the core problems of FG-SBIR: a. how to reduce the difference between the non-homogenous of heterogeneous media, and b. how to improve the distinguishability of sketch features. Specifically, a sketch generation model is first proposed to replace the conventional pre-processing of roughly extracting image edges, moreover, this model can alleviate the dilemma of sketch data scarcity. We then construct a novel FG-SBIR model which takes advantage of deformable convolutional neural network while taking into consideration of semantic attributes together. In addition, we build a fine-grained clothing sketch-image dataset, which has rich attribute annotations, for the first time. Extensive experiments exhibit that our proposed model achieves a better performance in improving the retrieval accuracy over the state-of-the-art baselines.