Going beyond free viewpoint: creating animatable volumetric video of human performances. Issue 6 (7th September 2020)

• Record Type: Journal Article
• Authors: Hilsmann, Anna; Fechteler, Philipp; Morgenstern, Wieland; Paier, Wolfgang; Feldmann, Ingo; Schreer, Oliver; Eisert, Peter
• Is Part Of: IET computer vision. Volume 14:Issue 6(2020)
• Page Start: 350
• Page End: 358
• Publication Date: 2020-09-07
• Subjects: image texture -- video signal processing -- computer animation -- computational geometry -- neural nets; semantic enrichment -- geometric animation ability -- parametric shape adaptive full human body model -- hybrid geometry -- video‐based animation approach -- classical CG animation -- video‐based textures -- autoencoder‐based approach -- deformation properties -- hybrid animation -- human performances -- high‐quality animatable volumetric video content -- free‐viewpoint volumetric video -- animation properties -- high‐quality data -- actor performance -- temporal consistency -- 3D data -- automatic rigging -- pose editing -- facial expressions; Computer vision -- Periodicals; Pattern recognition systems -- Periodicals; 006.37
• DOI: 10.1049/iet-cvi.2019.0786
• Languages: English
• ISSNs: 1751-9632

Abstract

Abstract : An end‐to‐end pipeline for the creation of high‐quality animatable volumetric video of human performances is presented. Going beyond the application of free‐viewpoint video, the authors allow re‐animation and alteration of an actor's performance through the enrichment of the captured data with semantics and animation properties. Hybrid geometry‐ and video‐based animation methods are applied that allow a direct animation of the high‐quality data itself instead of creating a CG model that resembles the captured data. Semantic enrichment and animation are achieved by establishing temporal consistency followed by automatic rigging of each 3D frame using a parametric human body model. The hybrid approach combines the flexibility of classical CG animation with the realism of real captured data. For the face, coarse movements are modelled in the geometry only, while very fine and subtle details, often lacking in purely geometric methods, are captured in video textures, which can interactively be combined to form new facial expressions. On top of that, regions that are challenging to synthesise, such as the teeth or the eyes, are learned and filled in realistically in an autoencoder‐based approach. This study covers the full pipeline from capturing, volumetric video production, and enrichment with semantics for the final hybrid animation.