Intermediated Reality

Casas, Llogari

Abstract

This thesis explores technical solutions to address the gap between the virtual and physical worlds towards photo-realistic interactive Augmented Reality (AR). As mobile network bandwidth increases, latencies reduce and graphics processing power becomes more efficient, this work tackles the challenge of convincingly re-animating physical objects remotely through digital displays. A framework for distributed Intermediated Reality (IR) communication is introduced, and forms the structure of the constituent methods developed for seamless collaboration through the remote possession of entities that come to life in mobile AR.

To perform such augmentation in an unnoticeable way, a method of deforming surface camera samples for seamless animations of physical objects with background inpainting is first introduced. This technique, in combination with a method to retarget the proximate appearance of real shadows to deformed virtual shadows and a method to perform environment illumination estimation using inconspicuous flat Fresnel lenses, brings real-world props to life in a compelling and practical way. Each method is integrated together to perform in real-time with analysis and evaluations using metric comparisons to expected ground truth renderings are provided.

Intermediated Reality can be applied to a variety of industries and scenarios beyond communication. This thesis presents applications in the movie industry and computer games sectors. For example, an approach to reduce the number of physical configurations needed for a stop-motion animation movie by generating the in-between frames digitally in AR is demonstrated. AR-generated frames preserve its natural appearance and achieve smooth transitions between real-world key-frames and digitally generated in-betweens. Further, the techniques extend across the entire Reality-Virtuality Continuum to target new game experiences called Multi-Reality games. This gaming experience makes an evolutionary step toward the convergence of real and virtual game characters for visceral digital experiences.

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: SA '19: SIGGRAPH Asia 2019
• Start Date: Nov 17, 2019
• End Date: Nov 21, 2019
• Acceptance Date: Sep 16, 2019
• Online Publication Date: Nov 17, 2019
• Publication Date: Nov 17, 2019
• Deposit Date: Oct 29, 2020
• Publisher: Association for Computing Machinery (ACM)
• Book Title: SA '19: SIGGRAPH Asia 2019 Doctoral Consortium
• ISBN: 9781450370219
• DOI: https://doi.org/10.1145/3366344.3366629
• Public URL: http://researchrepository.napier.ac.uk/Output/2696616