Automated classification of hand gestures using a wristband and machine learning for possible application in pill intake monitoring. (June 2022)

• Record Type: Journal Article
• Authors: Moccia, Sara; Solbiati, Sarah; Khornegah, Mahshad; Bossi, Federica FS; Caiani, Enrico G
• Is Part Of: Computer methods and programs in biomedicine. Volume 219(2022)
• Publication Date: 2022-06
• Subjects: Hand-gesture classification -- Monitoring medical adherence -- Wearable sensors -- Machine learning -- Deep learning -- CNN-LSTM; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2022.106753
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: A learning-based pipeline for hand-gesture classification using wristband inertial sensors was proposed. Multi-gesture and 'drink' gesture recognition resulted in 90.5% and 92.5% f1-scores. This could benefit healthcare applications where the drinking action becomes a clinical indicator, such as pill intake. Abstract: Background: Thanks to the increased interest towards health and lifestyle, a larger adoption in wearable devices for activity tracking is present among the general population. Wearable devices such as smart wristbands integrate inertial units, including accelerometers and gyroscopes, which can be utilised to perform automatic classification of hand gestures. This technology could also find an important application in automatic medication adherence monitoring. Accordingly, this study aims at comparing the performance of several Machine-Learning (ML) and Deep-Learning (DL) approaches for the automatic identification of hand gestures, with a specific focus on the drinking gesture, commonly associated to the action of oral intake of a pill-packed medication. Methods: A method to automatically recognize hand gestures in daily living is proposed in this work. The method relies on a commercially available wristband sensor (MetaMotionR, MbientLab Inc.) integrating tri-axial accelerometer and gyroscope. Both ML and DL algorithms were evaluated for both multi-gesture (drinking, eating, pouring water, opening a bottle, typing, answering a phone, combing hair, and cutting) and binary gesture (drinking versus other gestures) classification from wristband sensor signals. Twenty-two participants were involved in the experimental analysis, performing a 10 min acquisition in a laboratory setting. Leave-one-subject-out cross validation was performed for robust performance assessment. Results: The highest performance was achieved using a convolutional neural network with long- short term memory (CNN-LSTM), with a median f1-score of 90.5 [first quartile: 84.5; third quartile: 92.5]% and 92.5 [81.5;98.0]% for multi-gesture and binary classification, respectively. Conclusions: Experimental results showed that hand gesture classification with ML/DL from wrist accelerometers and gyroscopes signals can be performed with reasonable accuracy in laboratory settings, paving the way for a new generation of medical devices for monitoring medical adherence.