System for augmented human–robot interaction through mixed reality and robot training by non-experts in customer service environments. (16th February 2020)

• Record Type: Journal Article
• Authors: El Hafi, L.; Isobe, S.; Tabuchi, Y.; Katsumata, Y.; Nakamura, H.; Fukui, T.; Matsuo, T.; Garcia Ricardez, G. A.; Yamamoto, M.; Taniguchi, A.; Hagiwara, Y.; Taniguchi, T.
• Is Part Of: Advanced robotics. Volume 34:Number 3/4(2020)
• Page Start: 157
• Page End: 172
• Publication Date: 2020-02-16
• Subjects: Service robotics competition -- human–robot interaction -- unsupervised multimodal learning -- spatial concepts formation -- mixed reality interface; Robotics -- Periodicals; Robotics -- Japan -- Periodicals; Robotics; Japan; Periodicals; 629.89205
• DOI: 10.1080/01691864.2019.1694068
• Languages: English
• ISSNs: 0169-1864

Abstract

Abstract : Human–robot interaction during general service tasks in home or retail environment has been proven challenging, partly because (1) robots lack high-level context-based cognition and (2) humans cannot intuit the perception state of robots as they can for other humans. To solve these two problems, we present a complete robot system that has been given the highest evaluation score at the Customer Interaction Task of the Future Convenience Store Challenge at the World Robot Summit 2018, which implements several key technologies: (1) a hierarchical spatial concepts formation for general robot task planning and (2) a mixed reality interface to enable users to intuitively visualize the current state of the robot perception and naturally interact with it. The results obtained during the competition indicate that the proposed system allows both non-expert operators and end users to achieve human–robot interactions in customer service environments. Furthermore, we describe a detailed scenario including employee operation and customer interaction which serves as a set of requirements for service robots and a road map for development. The system integration and task scenario described in this paper should be helpful for groups facing customer interaction challenges and looking for a successfully deployed base to build on. GRAPHICAL ABSTRACT: