DeFT-Net: Dual-Window Extended Frequency Transformer for Rhythmic Motion Prediction

Ademola, Adeyemi; Sinclair, David; Koniaris, Babis; Hannah, Samantha; Mitchell, Kenny

Abstract

Enabling online virtual reality (VR) users to dance and move in a way that mirrors the real-world necessitates improvements in the accuracy of predicting human motion sequences paving way for an immersive and connected experience. However, the drawbacks of latency in networked motion tracking present a critical detriment in creating a sense of complete engagement, requiring prediction for online synchronization of remote motions. To address this challenge, we propose a novel approach that leverages a synthetically generated dataset based on supervised foot anchor placement timings of rhythmic motions to ensure periodicity resulting in reduced prediction error. Specifically, our model compromises a discrete cosine transform (DCT) to encode motion, refine high frequencies and smooth motion sequences and prevent jittery motions. We introduce a feed-forward attention mechanism to learn based on dual-window pairs of 3D key points pose histories to predict future motions. Quantitative and qualitative experiments validating on the Human3.6m dataset result in observed improvements in the MPJPE evaluation metrics protocol compared with prior state-of-the-art.

Citation

Ademola, A., Sinclair, D., Koniaris, B., Hannah, S., & Mitchell, K. (2024, September). DeFT-Net: Dual-Window Extended Frequency Transformer for Rhythmic Motion Prediction. Presented at EG UK Computer Graphics & Visual Computing (2024), London, UK

• Presentation Conference Type: Conference Paper (published)
• Conference Name: EG UK Computer Graphics & Visual Computing (2024)
• Start Date: Sep 12, 2024
• End Date: Sep 13, 2024
• Acceptance Date: Jul 24, 2024
• Online Publication Date: Oct 1, 2024
• Publication Date: 2024
• Deposit Date: Aug 20, 2024
• Publicly Available Date: Oct 1, 2024
• Publisher: Eurographics Association
• Peer Reviewed: Peer Reviewed
• Book Title: Computer Graphics & Visual Computing (CGVC) 2024
• ISBN: 9783038682493
• DOI: https://doi.org/10.2312/cgvc.20241220
• Keywords: Computing methodologies → Machine Learning; Motion Processing; Virtual Reality
• Publisher URL: https://diglib.eg.org/communities/20818d8d-d709-42e3-9adc-c29719de3e99
• Other Repo URL: https://github.com/CarouselDancing/DeFt-net