Skip to main content

Research Repository

Advanced Search

IRIDiuM: immersive rendered interactive deep media

Koniaris, Babis; Israr, Ali; Mitchell, Kenny; Huerta, Ivan; Kosek, Maggie; Darragh, Karen; Malleson, Charles; Jamrozy, Joanna; Swafford, Nick; Guitian, Jose; Moon, Bochang


Ali Israr

Ivan Huerta

Maggie Kosek

Charles Malleson

Joanna Jamrozy

Nick Swafford

Jose Guitian

Bochang Moon


Compelling virtual reality experiences require high quality imagery as well as head motion with six degrees of freedom. Most existing systems limit the motion of the viewer (prerecorded fixed position 360 video panoramas), or are limited in realism, e.g. video game quality graphics rendered in real-time on low powered devices. We propose a solution for presenting movie quality graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive deep media representation, we display the content in real-time according to the tracked head pose. For each frame, we generate a set of 360-degree images (colors and depths) using cameras placed in selected locations within a small view volume surrounding a central viewing position. We employ a parallax masking technique which minimizes the rendering work required for the additionally visible surfaces in viewing locations around the main viewpoint. At run-time, a decompression and rendering algorithm fetches the appropriate surface data in real-time and projects them to the eye positions as the user moves within the tracked view volume. To further illustrate this ability for interactivity and embodiment within VR movies, we track the full upper body using our sparse sensor motion capture solver allowing users to see themselves in the virtual world. Here, both head and upper body are tracked in realtime using data from IMU (Inertial Measurement Unit) and EMG (Electromyogram) sensors. Our real-time solver, Triduna Live uses a physics-based approach to robustly estimate pose from a few sensors. Hand gesture and object grasping motions are detected from the EMG data and combined with the tracked body position to control gameplay seamlessly integrated within the deep media environment.

Presentation Conference Type Conference Paper (Published)
Conference Name ACM SIGGRAPH 2016 VR Village on - SIGGRAPH '16
Start Date Jul 24, 2017
End Date Jul 27, 2017
Acceptance Date Apr 2, 2016
Online Publication Date Jul 24, 2016
Publication Date Jul 24, 2016
Deposit Date Dec 12, 2017
Publicly Available Date Dec 14, 2017
Publisher Association for Computing Machinery (ACM)
ISBN 9781450343770
Keywords real-time rendering, VR, viewer tracking, interaction
Public URL
Contract Date Dec 12, 2017


IRIDiuM: immersive rendered interactive deep media (17.7 Mb)

Copyright Statement
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for thirdparty components of this work must be honored. For all other uses, contact the owner/author(s).
c 2016 Copyright held by the owner/author(s).

You might also like

Downloadable Citations