Deep learning-based mobile augmented reality for task assistance using 3D spatial mapping and snapshot-based RGB-D data. (August 2020)

• Record Type: Journal Article
• Authors: Park, Kyeong-Beom; Choi, Sung Ho; Kim, Minseok; Lee, Jae Yeol
• Is Part Of: Computers & industrial engineering. Volume 146(2020)
• Publication Date: 2020-08
• Subjects: Mobile augmented reality (AR) -- Spatial mapping -- Deep learning-based AR -- Task assistance; Engineering -- Data processing -- Periodicals; Industrial engineering -- Periodicals; 620.00285
• DOI: 10.1016/j.cie.2020.106585
• Languages: English
• ISSNs: 0360-8352

Abstract

Highlights: Deep learning-based mobile AR for smart task assistance. Efficient and automatic 3D spatial mapping using deep learning-based instance segmentation and ICP matching in mobile AR. Utilizing single snapshot-based RGB-D data for spatial mapping instead of cluttered point cloud data. Formal evaluation of the proposed approach through quantitative and qualitative analyses. Abstract: This paper proposes a new deep learning-based mobile AR for intelligent task assistance by conducting 3D spatial mapping without pre-registration using AR markers, which can match virtual AR objects to their corresponding physical objects automatically and accurately using single snapshot-based RGB-D data. Firstly, the proposed approach applies a deep learning-based instance segmentation method to the snapshot-based RGB-D data to detect real object instances and to segment their surrounding regions in 3D point cloud data. Then, an iterative closest point (ICP) algorithm is used to perform a 3D spatial mapping between the segmented point cloud of the real object and its corresponding virtual model. Therefore, the virtual information can be seamlessly and automatically synchronized with its corresponding real object. To prove the effectiveness of the proposed method, we performed comparative experiments quantitatively and qualitatively, which evaluated the accuracy, basic task performance, and usability. Experimental results verify that the proposed deep learning-based 3D spatial mapping approach is more accurate and more suitable for mobile AR-based visualization and interaction than previous studies. We have also implemented several applications in actual working situations, which verifies the applicability and extensibility of the proposed approach.