Real-time computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements using a contactless sensor and system of systems approach. (July 2020)

• Record Type: Journal Article
• Authors: Nguyen, Tan-Nhu; Dakpé, Stéphanie; Ho Ba Tho, Marie-Christine; Dao, Tien-Tuan
• Is Part Of: Computer methods and programs in biomedicine. Volume 191(2020)
• Publication Date: 2020-07
• Subjects: Real time computer vision system -- Rigid head movements -- Non-rigid facial mimic movements -- Contactless kinect sensor -- System of systems; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2020.105410
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: Accurate generation process for subject specific head model with texture information. Tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements in real time. System of systems approach in developing a real-time computer vision system. Multi-level evaluation of the developed computer vision system. Abstract: Background and Objective: Head and facial mimic animations play important roles in various fields such as human-machine interactions, internet communications, multimedia applications, and facial mimic analysis. Numerous studies have been trying to simulate these animations. However, they hardly achieved all requirements of full rigid head and non-rigid facial mimic animations in a subject-specific manner with real-time framerates. Consequently, this present study aimed to develop a real-time computer vision system for tracking simultaneously rigid head and non-rigid facial mimic movements. Methods: Our system was developed using the system of systems approach. A data acquisition sub-system was implemented using a contactless Kinect sensor. A subject-specific model generation sub-system was designed to create the geometrical model from the Kinect sensor without texture information. A subject-specific texture generation sub-system was designed for enhancing the reality of the generated model with texture information. A head animation sub-system with graphical user interfaces was also developed. Model accuracy and system performances were analyzed. Results: The comparison with MRI-based model shows a very good accuracy level (distance deviation of ~1 mm in neutral position and an error range of [2–3 mm] for different facial mimic positions) for the generated model from our system. Moreover, the system speed can be optimized to reach a high framerate (up to 60 fps) during different head and facial mimic animations. Conclusions: This study presents a novel computer vision system for tracking simultaneously subject-specific rigid head and non-rigid facial mimic movements in real time. In perspectives, serious game technology will be integrated into this system towards a full computer-aided decision support system for facial rehabilitation. Graphical abstract: Image, graphical abstract