Cartoonish sketch-based face editing in videos using identity deformation transfer. (April 2019)

• Record Type: Journal Article
• Authors: Zhao, Long; Han, Fangda; Peng, Xi; Zhang, Xun; Kapadia, Mubbasir; Pavlovic, Vladimir; Metaxas, Dimitris N.
• Is Part Of: Computers & graphics. Volume 79(2019)
• Page Start: 58
• Page End: 68
• Publication Date: 2019-04
• Subjects: Video editing -- Sketch-based modeling -- Shape deformation -- Deformation transfer -- 3D morphable model; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2019.01.004
• Languages: English
• ISSNs: 0097-8493

Abstract

Highlights: A novel sketch-based facial identity editing framework for videos. A robust 2D to 3D sketch-based identity deformation transfer algorithm. A new contour-based interface for accurate 3D model refinement. Decent results can be easily produced by both professional and amateur users. Abstract: We address the problem of using hand-drawn sketches to create exaggerated deformations to faces in videos, such as enlarging the shape or modifying the position of eyes or mouth. This task is formulated as a 3D face model reconstruction and deformation problem. We first recover the facial identity and expressions from the video by fitting a face morphable model for each frame. At the same time, user's editing intention is recognized from input sketches as a set of facial modifications. Then a novel identity deformation algorithm is proposed to transfer these facial deformations from 2D space to the 3D facial identity directly while preserving the facial expressions. After an optional stage for further refining the 3D face model, these changes are propagated to the whole video with the modified identity. Both the user study and experimental results demonstrate that our sketching framework can help users effectively edit facial identities in videos, while high consistency and fidelity are ensured at the same time. Graphical abstract: