AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop. (August 2022)

• Record Type: Journal Article
• Authors: Li, Chengxi; Zheng, Pai; Li, Shufei; Pang, Yatming; Lee, Carman K.M.
• Is Part Of: Robotics and computer-integrated manufacturing. Volume 76(2022)
• Publication Date: 2022-08
• Subjects: Augmented reality -- Digital twin -- Collaborative manufacturing system -- Reinforcement learning -- Human-in-the-loop control; Robots, Industrial -- Periodicals; Computer integrated manufacturing systems -- Periodicals; Robotics -- Periodicals; Robots industriels -- Périodiques; Productique -- Périodiques; Robotique -- Périodiques; 670.285
• DOI: 10.1016/j.rcim.2022.102321
• Languages: English
• ISSNs: 0736-5845

Abstract

Abstract: The teleoperation and coordination of multiple industrial robots play an important role in today's industrial internet-based collaborative manufacturing systems. The user-friendly teleoperation approach allows operators from different manufacturing domains to reduce redundant learning costs and intuitively control the robot in advance. Nevertheless, only a few preliminary works have been introduced very recently, let alone its effective implementation in the manufacturing scenarios. To address the gap, this research proposes a novel multi-robot collaborative manufacturing system with human-in-the-loop control by leveraging the cutting-edge augmented reality (AR) and digital twin (DT) techniques. In the proposed system, the DTs of industrial robots are firstly mapped to physical robots and visualize them in the AR glasses. Meanwhile, a multi-robot communication mechanism is designed and implemented, to synchronize the state of robots in the twin. Moreover, a reinforcement learning algorithm is integrated into the robot motion planning to replace the conventional kinematics-based robot movement with corresponding target positions. Finally, three interactive AR-assisted DT modes, including real-time motion control, planned motion control, and robot monitoring mode are generated, which can be readily switched by human operators. Two experimental studies are conducted on (1) a single robot with a commonly used peg-in-hole experiment, and (2) the motion planning of multi-robot collaborative tasks, respectively. From the experimental results, it can be found that the proposed system can well handle the multi-robot teleoperation tasks with high efficiency and owns great potentials to be adopted in other complicated manufacturing scenarios in the near future. Highlights: Presented a multi-robot multi-client-based communication mechanism to synchronize teleoperation states. Introduced three typical augmented reality-assisted digital twin-enabled human-in-the-loop robot control modes. Proposed a RL algorithm for motion control and planning of the multi-robots with learning intelligence.