In-situ Playback of Immersive Recordings

Independent Research - Spring 2024 - XR Collaboratory, Cornell Tech

Student: Amitesh Sinha

Video | Report | APK

Overview

XRC Students sp2024-ir19-sinha is a video playback system for VR experiences. It provides a comprehensive set of components that enable seamless video playback and control in virtual reality (VR) environments. With this package, developers can effortlessly incorporate video playback functionalities into their VR applications, allowing users to interact with videos using a variety of input modalities.

Core Functionality

  1. Record: Seamless record the VR experience.
  2. Play-Pause: Toggle video playback between play and pause states.
  3. Seek Timeline: Users can smoothly scrub through the video timeline to navigate to specific frames or moments. The package handles timeline scrubbing efficiently, ensuring smooth and precise control over video playback.
  4. Change Scrub Speed: Users can adjust the scrub speed according to their preference. The package provides options for increasing or decreasing the scrubbing speed, enabling a personalized video browsing experience.

Installation instructions

Note

This package extends the Ultimate Replay 3.0 asset (UPM format), an access to the purchased asset is required. You also need edu.cornell.xrc.core as a dependency. Follow these installation instructions.

To install this package, follow these steps:

  1. In the Unity Editor, click on Window > Package Manager
  2. Click the + button and choose Add package from git URL option
  3. Paste this URL https://github.com/xrc-students/xrc-students-sp2024-ir19-sinha.git in the prompt and click on Add
  4. You might have to authenticate as this is a private repository
  5. The package should be installed into your project
  6. You can download XRC Playback Tool from under Samples for the demo scene

Requirements

This version of XRC Students sp2024-ir19-sinha is compatible with the following versions of the Unity Editor:

  • 2022.3 and later

Dependencies

XRC Students su2023-in04-sinha package has the following dependencies which are automatically added to your project when installing:

Samples

The Playback Samples are a great resource to explore the package. XRC Playback Tool/Scenes/XRC Playback Tool is where you should start to get a feel for all the components in the package.

Package Demo

This video will walk you through the demo scene shipped with the package to better understand the features.

Getting Started

  1. Attach the Replay Transform component to the objects you want to record.

    Replay Transform Component

  2. The Playback component that comes with the package exposes all the functionality needed for a convenient and flexible record & replay experience. These include:

    • Properties: Includes playback duration, current playback time, playback speed, and status indicators for recording and replaying.
    • Methods: Public methods available for user interaction include start recording, stop recording, start playback, pause playback, seek playback, and change playback speed.
    • Events: System events such as recording started, recording stopped, replay started, and replay stopped are triggered as appropriate to inform other system components or the user directly

  3. The PlaybackInput and PlaybackFeedback components are bespoke implementations for showcasing the use of the package. They can be used as-is or you can build your own input/feedback using the Playback component.

Input

Following are the input controls provided in the sample scene:

Input Action Functionality
Left Controller Primary Button Start/Stop Recording
Right Controller Primary Button Start/Pause Playback
Right Controller Thumbstick Along Y axis Increase/Decrease Playback Speed
Right Controller Secondary Button + Right Controller Along X axis Seek Timeline

Feedback

Following feedback comes out of the box with the PlaybackFeedback component:

  1. Every gameobject with the replay transform component attached will have a trail renderer that will show the trajectory of the object during playback.

    Trajectory

  2. If the feedback object (prefab provided) and the anchor object (left controller by default) are set in the component, then the feedback object will change color depending on the current state

    Record Mode

Limitations

We are aware of the following limitations in this package:

  1. When you hit replay button, the scale/orientation changes of Grab Move are reset