Skip to main content

Research Repository

Advanced Search

Intermediated Reality: A Framework for Communication Through Tele-Puppetry

Casas, Llogari; Mitchell, Kenny



We introduce Intermediated Reality (IR), a framework for intermediated communication enabling collaboration through remote possession of entities (e.g., toys) that come to life in mobile Mediated Reality (MR). As part of a two-way conversation, each person communicates through a toy figurine that is remotely located in front of the other participant. Each person's face is tracked through the front camera of their mobile devices and the tracking pose information is transmitted to the remote participant's device along with the synchronized captured voice audio, allowing a turn-based interactive avatar chat session, which we have called ToyMeet. By altering the camera video feed with a reconstructed appearance of the object in a deformed pose, we perform the illusion of movement in real-world objects to realize collaborative tele-present augmented reality (AR). In this turn based interaction, each participant first sees their own captured puppetry message locally with their device's front facing camera. Next, they receive a view of their counterpart's captured response locally (in AR) with seamless visual deformation of their local 3D toy seen through their device's rear facing camera. We detail optimization of the animation transmission and switching between devices with minimized latency for coherent smooth chat interaction. An evaluation of rendering performance and system latency is included. As an additional demonstration of our framework, we generate facial animation frames for 3D printed stop motion in collaborative mixed reality. This allows a reduction in printing costs since the in-between frames of key poses can be generated digitally with shared remote review.

Journal Article Type Article
Acceptance Date Jul 3, 2019
Online Publication Date Jul 23, 2019
Publication Date Jul 23, 2019
Deposit Date Aug 8, 2019
Publicly Available Date Aug 8, 2019
Journal Frontiers in Robotics and AI
Electronic ISSN 2296-9144
Publisher Frontiers Media
Peer Reviewed Peer Reviewed
Volume 6
Keywords augmented reality, collaborative mixed reality, real-time graphics, human-computer interaction, tele-puppetry
Public URL
Contract Date Aug 8, 2019


You might also like

Downloadable Citations