Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras

Dos Santos Brito, Caio Jos�; Mitchell, Kenny

Abstract

Preparing datasets for use in the training of real-time face tracking algorithms for HMDs is costly. Manually annotated facial landmarks are accessible for regular photography datasets, but introspectively mounted cameras for VR face tracking have incompatible requirements with these existing datasets. Such requirements include operating ergonomically at close range with wide angle lenses, low-latency short exposures, and near infrared sensors. In order to train a suitable face solver without the costs of producing new training data, we automatically repurpose an existing landmark dataset to these specialist HMD camera intrinsics with a radial warp reprojection. Our method separates training into local regions of the source photos, \ie mouth and eyes for more accurate local correspondence to the mounted camera locations underneath and inside the fully functioning HMD. We combine per-camera solved landmarks to yield a live animated avatar driven from the user's face expressions. Critical robustness is achieved with measures for mouth region segmentation, blink detection and pupil tracking. We quantify results against the unprocessed training dataset and provide empirical comparisons with commercial face trackers.

Citation

Dos Santos Brito, C. J., & Mitchell, K. (2019, November). Recycling a Landmark Dataset for Real-time Facial Capture and Animation with Low Cost HMD Integrated Cameras. Presented at VRCAI ’19, Brisbane, QLD, Australia

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: VRCAI ’19
• Start Date: Nov 14, 2019
• End Date: Nov 16, 2019
• Acceptance Date: Oct 3, 2019
• Online Publication Date: Nov 15, 2019
• Publication Date: 2019-11
• Deposit Date: Nov 4, 2019
• Publicly Available Date: Nov 5, 2019
• Publisher: Association for Computing Machinery (ACM)
• Book Title: VRCAI '19: The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry
• ISBN: 978-1-4503-7002-8
• DOI: https://doi.org/10.1145/3359997.3365690
• Keywords: real-time, facial capture, virtual reality, HMD, data preparation
• Public URL: http://researchrepository.napier.ac.uk/Output/2284374