Pose-guided matching based on deep learning for assessing quality of action on rehabilitation training. (February 2022)

• Record Type: Journal Article
• Authors: Qiu, Yuhang; Wang, Jiping; Jin, Zhe; Chen, Honghui; Zhang, Mingliang; Guo, Liquan
• Is Part Of: Biomedical signal processing and control. Volume 72(2022)Part A
• Publication Date: 2022-02
• Subjects: Rehabilitation training -- Eight-section brocade -- Pose-guided matching -- Siamese convolutional neural network -- Attention module; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2021.103323
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: Current pose assessment systems ignore the similarity between comparative poses. A new pose evaluation system that can provide comprehensive feedback was proposed. The proposed pose matching method is feasible on the eight-section brocade dataset. The proposed method is one-stage and it has been deployed in software for testing. Abstract: The application of pose assessment on rehabilitation training has gradually received attention in recent years. However, current evaluation indicators of these methods are mostly based on the score or scoring function that defined by users, which is too subjective and hard to be used by patients directly. In this paper, we conceptualized a new idea for pose matching, namely pose-guided matching that aims at providing objective and accurate score, feedback and guidance (i.e. guided) to the patients when the pose is compared to the standard pose. More specifically, we proposed a pair-based Siamese Convolutional Neural Network (SCNN) abbreviated ST-AMCNN to realize the idea of pose-guided matching on the eight-section brocade dataset which is one of the most representative traditional rehabilitation exercises in China. We simplified the multi-stages pose matching by merging two standalone modules (i.e. alignment and matching module) into a one-stage task. Such that, only one loss function is required to tune, which reduces the computational complexity. On top of the Spatial Transformer Networks (STN) employed as an alignment module, we proposed a new Attention-based Multi-Scale Convolution (AMC) to match different posture parts (i.e. multi-scale). Furthermore, the proposed AMC can assign more weight to useful pose features as opposed to other irrelevant features e.g. background features for performance gain. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is adopted to visualize the matching result for the learner. Experimental results indicate that ST-AMCNN achieves a competitive performance than the state-of-the-art models and can provide accurate feedback for learners on rehabilitation training. Simultaneously, the proposed method is also deployed in client software for testing.