Digital twin improved via visual question answering for vision-language interactive mode in human–machine collaboration. (January 2021)

• Record Type: Journal Article
• Authors: Wang, Tian; Li, Jiakun; Kong, Zhaoning; Liu, Xin; Snoussi, Hichem; Lv, Hongqiang
• Is Part Of: Journal of manufacturing systems. Volume 58(2021)Part B
• Page Start: 261
• Page End: 269
• Publication Date: 2021-01
• Subjects: Digital twin -- Human–machine collaboration -- Visual question answer -- Deep learning; Manufacturing processes -- Periodicals; Production engineering -- Data processing -- Periodicals; Robots, Industrial -- Periodicals; Production, Technique de la -- Informatique -- Périodiques; Robots industriels -- Périodiques; Electronic journals; 670.42
• DOI: 10.1016/j.jmsy.2020.07.011
• Languages: English
• ISSNs: 0278-6125

Abstract

Highlights: A digital twin model is proposed for human–machine collaboration. Visual question answering technology is employed to enhance the digital twin. A new VQA model, video-text concatenation (VTC) network, is proposed. Abstract: The human–machine collaboration system is a key means of manufacturing. Its surveillance, prognostic, and health management are related to safety and manufacturing persistence. This paper begins with the mission requirements of intelligent manufacturing. The study is based on the visual question answering (VQA) technology with a digital twin to increase efficiency. The research contents are as follows: (1) A method of modeling human–machine collaboration based on digital twins is proposed. (2) A VQA is adopted in the digital twin. The video and neural language are considered. (3) VQA technology is introduced into the modeling of the human–machine collaboration system for consistent integration. With VQA technology, humans and machines can collaborate. Human–machine interaction and product counting are implemented in a case study to provide a comprehensive perception.