High-fidelity 3D real-time facial animation using infrared structured light sensing system. (May 2022)

• Record Type: Journal Article
• Authors: Ye, Yuping; Song, Zhan; Zhao, Juan
• Is Part Of: Computers & graphics. Volume 104(2022)
• Page Start: 46
• Page End: 58
• Publication Date: 2022-05
• Subjects: Structured light -- Facial animation -- Non-rigid registration; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2022.03.007
• Languages: English
• ISSNs: 0097-8493

Abstract

Abstract: The acquisition of 3D facial models is crucial in the gaming and film industries. In this study, we developed a facial acquisition system based on infrared structured light sensors to obtain facial-expression models with high fidelity and accuracy. First, we employed time-multiplexing structured light to obtain an accurate and dense point cloud. The template model was then warped to the captured facial expression. A model-tracking method based on optical flow was applied to track the registered 3D model displacement. Finally, the live 3D mesh was textured using high-resolution images captured by three color cameras. Experiments were conducted on human faces to demonstrate the performance of the proposed system and methods. Graphical abstract: Highlights: A high-fidelity realtime facial animation pipeline based on an infrared structured light imaging system is built. The facial 3D imaging system enables the game and film makers to capture dense and accurate facial models. A nonrigid registration running on GPU is proposed to fast blend the target model according to captured models. Varies experimental results have been presented to verify our system's efficiency and accuracy.