Learning Style Compatibility Between Objects in a Real‐World 3D Asset Database. (14th November 2019)

• Record Type: Journal Article
• Authors: Liu, Yifan; Tang, Ruolan; Ritchie, Daniel
• Is Part Of: Computer graphics forum. Volume 38:Number 7(2019)
• Page Start: 775
• Page End: 784
• Publication Date: 2019-11-14
• Subjects: Categories and Subject Descriptors (according to ACM CCS): I.3.8 [Computer Graphics]: Applications—3D Model Search -- I.5.1 [Pattern Recognition]: Models—Neural Networks -- I.2.10 [Artificial Intelligence]: Vision and Scene Understanding—Perceptual Reasoning; Computer graphics -- Periodicals; 006.605
• DOI: 10.1111/cgf.13879
• Languages: English
• ISSNs: 0167-7055

Abstract

Abstract: Large 3D asset databases are critical for designing virtual worlds, and using them effectively requires techniques for efficient querying and navigation. One important form of query is search by style compatibility : given a query object, find others that would be visually compatible if used in the same scene. In this paper, we present a scalable, learning‐based approach for solving this problem which is designed for use with real‐world 3D asset databases; we conduct experiments on 121 3D asset packages containing around 4000 3D objects from the Unity Asset Store. By leveraging the structure of the object packages, we introduce a technique to synthesize training labels for metric learning that work as well as human labels. These labels can grow exponentially with the number of objects, allowing our approach to scale to large real‐world 3D asset databases without the need for expensive human training labels. We use these synthetic training labels in a metric learning model that analyzes the in‐engine rendered appearance of an object—combining geometry, material, and texture—whereas prior work considers only object geometry, or disjoint geometry and texture features. Through an ablation experiment, we find that using this representation yields better results than using renders which lack texture, materiality, or both.