PhD Dissertation Defense | Luming Ma
Thursday, April 2, 2020
2:30 pm - 4:30 pm
Acquisition and editing of facial performance is an essential and challenging task in computer graphics, with broad applications in films, cartoons, VR systems, and electronic games. The creation of high-resolution, realistic facial animations often involves controlled lighting setups, multiple cameras, active markers, depth sensors, and substantial post-editing from experienced artists. This dissertation focuses on the capture and manipulation of facial performance from regular RGB video. First, we propose a novel method to reconstruct high-resolution facial geometry and appearance in real-time by capturing an individual-specific face model with fine-scale details, based on monocular RGB video input. Our approach can produce results close to off-line methods and better than previous real-time methods. On top of the reconstruction method, we propose two manipulation approaches upon facial expressions and facial appearance, namely facial expression transformation and face swapping. Our method automatically transforms the source expression in an input video clip to a specified target expression through the combination of the 3D face reconstruction, the learned bi-directional expression mapping, and automatic lip correction. It can be applied to new users with different identities, ages, speeches, and expressions, and without additional training. More importantly, unlike existing deep-learning-based methods, our method does not need to pre-train any models- - i.e., pre-collecting a large image/video dataset of the source or target face for model training is not required.
- Online - https://teams.microsoft.com/l/team/19%3afb9d5754f8484440abed99b071905485%40thread.tacv2/conversations?groupId=066e3443-0db2-4ae8-8a2c-4939ba9fd00c&tenantId=170bbabd-a2f0-4c90-ad4b-0e8f0f0c4259 (access code: ml36l0i)