Real-time monitoring for manual operations with machine vision in smart manufacturing. (October 2022)

• Record Type: Journal Article
• Authors: Lou, Ping; Li, Ji; Zeng, YuHang; Chen, Bing; Zhang, Xiaomei
• Is Part Of: Journal of manufacturing systems. Volume 65(2022)
• Page Start: 709
• Page End: 719
• Publication Date: 2022-10
• Subjects: Manual action recognition -- Real-time operation monitoring -- Deep learning -- Repetition counting -- Object detection; Manufacturing processes -- Periodicals; Production engineering -- Data processing -- Periodicals; Robots, Industrial -- Periodicals; Production, Technique de la -- Informatique -- Périodiques; Robots industriels -- Périodiques; Electronic journals; 670.42
• DOI: 10.1016/j.jmsy.2022.10.015
• Languages: English
• ISSNs: 0278-6125

Abstract

Abstract: Online real-time production process monitoring is the basis for intelligent manufacturing refinement management. This paper proposes a contactless monitoring framework with machine vision to realize real-time production/assembly process monitoring and repetition counting. In addition, a two-stage approach is presented to solve the issue of manual operation recognition and counting by pieces. In the first stage, a high-efficient method for object detection and classification, YOLOv4-tiny, is used to recognize manual operations/sub-operations and classify the sub-operations of the operations using monitoring video. And a Sliding Window Counter algorithm is utilized to find appropriate boundary points of different categories of the sub-operations for counting in the second stage. A dataset of repetitive manual operations is constructed for training this approach. The experiment results on the production/assembly line operations indicate that the approach can identify and count worker operations in real time. Highlights: The contactless and online perception method is presented to monitor manual operations. The lightweight object detection method is used to recognize the sub-operations at high-speed. A state detecting method with a sliding window is put forward for counting by pieces in real-time.