A novel MR remote collaboration system using 3D spatial area cue and visual notification. (April 2023)

• Record Type: Journal Article
• Authors: Zhang, Xiangyu; Bai, Xiaoliang; Zhang, Shusheng; He, Weiping; Wang, Shuxia; Yan, Yuxiang; Yu, Quan; Liu, Liwei
• Is Part Of: Journal of manufacturing systems. Volume 67(2023)
• Page Start: 389
• Page End: 409
• Publication Date: 2023-04
• Subjects: AMI ambient intelligence -- AR Augmented reality -- CAD Computer-aided design -- HMD Head-mounted device -- M Mean -- MEC mobile edge computing -- MR Mixed reality -- MRTK Mixed Reality Toolkit -- NUC Next unit of computing -- PCL Point cloud library -- RANSAC Random sample consensus -- SE Standard error -- SUS System usability scale -- VR Virtual reality -- 3DS 3D sketch cue -- 3DSA 3D sketch cue combines 3D spatial area cue -- 3DSAN 3D sketch cue combines 3D spatial area cue and visual notification; Remote collaboration -- Mixed reality -- 3D sketch cue -- 3D spatial area cue -- Visual notification -- Hand gesture -- MR assembly; Manufacturing processes -- Periodicals; Production engineering -- Data processing -- Periodicals; Robots, Industrial -- Periodicals; Production, Technique de la -- Informatique -- Périodiques; Robots industriels -- Périodiques; Electronic journals; 670.42
• DOI: 10.1016/j.jmsy.2023.02.013
• Languages: English
• ISSNs: 0278-6125

Abstract

Abstract: In remote collaboration for emergency maintenance or training, since local workers may lack relevant experience and knowledge, it is very important to support remote experts to annotate 3D spatial areas for local workers in their workspace, such as risky spatial areas that need to be avoided for safe operation when assembling replacement parts. Recently, many researchers have studied various visual communication cues for expressing spatial information in Mixed Reality (MR) remote collaboration systems for physical tasks. However, the expression and transmission of 3D spatial area information have not been well explored, especially in tasks that need to annotate specific spatial areas in 3D space (such as risky 3D spatial areas). In the study, we developed a novel MR remote collaboration system that supports remote experts using 3D spatial area cues and visual notification to create Augmented Reality (AR) instructions for local workers on the shared 3D stereoscopic scene of the local workspace in the Virtual Reality (VR) space. We performed a user study to explore the effect of three interfaces on remote collaborative physical assembly tasks in the workspace with risky 3D spatial areas: 3D sketch cue (3DS) which is the baseline solution, 3D sketch cue combines the novel 3D spatial area cue (3DSA), and 3D sketch cue combines 3D spatial area cue and visual notification (3DSAN). We found the 3DSAN can significantly improve collaboration efficiency, reduce operation errors, and improve the usability and collaboration experience (especially the level of awareness), among industrial remote collaborative physical tasks requiring 3D spatial area reference. Highlights: Development of a novel MR remote collaboration system which supports 3D spatial area cue and visual notification. Performing a user study to explore the effect of adding 3D spatial area cue and visual notification in MR remote system. Discussion the benefits of adding 3D spatial area cue and visual notification to improve MR remote collaboration.