A study of deep single sketch-based modeling: View/style invariance, sparsity and latent space disentanglement. (August 2022)

• Record Type: Journal Article
• Authors: Zhong, Yue; Gryaditskaya, Yulia; Zhang, Honggang; Song, Yi-Zhe
• Is Part Of: Computers & graphics. Volume 106(2022)
• Page Start: 237
• Page End: 247
• Publication Date: 2022-08
• Subjects: Deep sketch-based modeling -- Single-view reconstruction; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2022.06.005
• Languages: English
• ISSNs: 0097-8493

Abstract

Abstract: Deep image-based modeling has received a lot of attention in recent years. Sketch-based modeling in particular has gained popularity given the ubiquitous nature of touchscreen devices. In this paper, we (i) study and compare diverse single-image reconstruction methods on sketch input, comparing the different 3D shape representations: multi-view, voxel- and point-cloud-based, mesh-based and implicit ones; and (ii) analyze the main challenges and requirements of sketch-based modeling systems. We introduce the regression loss and provide two variants of its formulation for the two most promising 3D shape representations: point clouds and signed distance functions. We show that this loss can increase general reconstruction accuracy, and the view- and style-robustness of the reconstruction methods. Moreover, we demonstrate that this loss can benefit the disentanglement of latent space to view-invariant and view-specific information, resulting in further improved performance. To address the figure-ground ambiguity typical for sparse freehand sketches, we propose a two-branch architecture that exploits sparse user labeling. We hope that our work will inform future research on sketch-based modeling. Graphical abstract: Highlights: Sketch-based modeling robust to view and style change. A new formulation of the regression loss fitted to work with SDFs. Latent space disentanglement with regression loss increases reconstruction accuracy. Challenges inherent to sketch input in the context of deep-reconstruction methods. Auxiliary network learns to predict foreground mask supportting user sparse labels.