Ye Pan
Emotional Voice Puppetry
Pan, Ye; Zhang, Ruisi; Cheng, Shengran; Tan, Shuai; Ding, Yu; Mitchell, Kenny; Yang, Xubo
Authors
Ruisi Zhang
Shengran Cheng
Shuai Tan
Yu Ding
Prof Kenny Mitchell K.Mitchell2@napier.ac.uk
Professor
Xubo Yang
Abstract
The paper presents emotional voice puppetry, an audio-based facial animation approach to portray characters with vivid emotional changes. The lips motion and the surrounding facial areas are controlled by the contents of the audio, and the facial dynamics are established by category of the emotion and the intensity. Our approach is exclusive because it takes account of perceptual validity and geometry instead of pure geometric processes. Another highlight of our approach is the generalizability to multiple characters. The findings showed that training new secondary characters when the rig parameters are categorized as eye, eyebrows, nose, mouth, and signature wrinkles is significant in achieving better generalization results compared to joint training. User studies demonstrate the effectiveness of our approach both qualitatively and quantitatively. Our approach can be applicable in AR/VR and 3DUI, namely, virtual reality avatars/self-avatars, teleconferencing and in-game dialogue.
Citation
Pan, Y., Zhang, R., Cheng, S., Tan, S., Ding, Y., Mitchell, K., & Yang, X. (in press). Emotional Voice Puppetry. IEEE Transactions on Visualization and Computer Graphics, https://doi.org/10.1109/tvcg.2023.3247101
Journal Article Type | Article |
---|---|
Acceptance Date | Jan 30, 2023 |
Online Publication Date | Feb 22, 2023 |
Deposit Date | Feb 24, 2023 |
Publicly Available Date | Feb 27, 2023 |
Journal | IEEE Transactions on Visualization and Computer Graphics |
Print ISSN | 1077-2626 |
Electronic ISSN | 1941-0506 |
Publisher | Institute of Electrical and Electronics Engineers |
Peer Reviewed | Peer Reviewed |
DOI | https://doi.org/10.1109/tvcg.2023.3247101 |
Keywords | Virtual reality, audio, emotion, character animation |
Files
Emotional Voice Puppetry (accepted version)
(5.7 Mb)
PDF
You might also like
Generating real-time detailed ground visualisations from sparse aerial point clouds
(2022)
Conference Proceeding
MienCap: Performance-based Facial Animation with Live Mood Dynamics
(2022)
Conference Proceeding
Collimated Whole Volume Light Scattering in Homogeneous Finite Media
(2022)
Journal Article
Embodied online dance learning objectives of CAROUSEL +
(2021)
Conference Proceeding