TECHNOLOGY

SOFTWARE

border2021 MR System Configuration (Visual Part Only)

We used Varjo XR-1 for HMD. There were a few choices of development environment with Varho XR-1, such as Native SDK, Unity XR SDK, Unreal SDK, and etc, and we decided to develop contents for “border2021” using Varjo Native SDK implemented as openFrameworks addon. The purposes behind this decision were to efficiently migrate the visual assets developed for “border 2015” and to handle special tasks related to the HMD camera.

The special tasks are:
Transformation of HMD camera’s texture.
Recording of the HMD camera footage and posture information from the first half of the performance onto the memory, and playing it back in reverse at the end of the performance.
Those processes were not feasible with Unity XR SDK or Unreal SDK.

We also developed our own special plugin using Varjo Tracking Plugin to process HMD posture information within VICON’s coordination system. This allowed us to integrate Varjo’s system and VICON Tracker.

Additionally, we prepared our own software for calibrating the coordinate system of both Varjo and Vicon so that the real objects and ones in CG would overlap perfectly.

Two WiFi APs are installed to ensure redundancy. In addition, the latency of posture information transmission was minimized by separating the APs for data transmission and video transmission.

Video for monitoring and video for online streaming on each machine were controlled by OBS.

We developed a system that allows us to effectively monitor the operational status of each application and the hardware status of the PC, which led to quick recovery on unexpected problems.

border2021 HMD Monitoring App Screen

HARDWARE

Photo: Muryo Homma(Rhizomatiks)

—WHILL

The WHILL Model CR, a research and development model of the personal mobility device WHILL, was adapted.
The operation of WHILL was controlled remotely by transmitting control commands　wirelessly to the Model CR's serial port.
To enable one Xbee transmitter to control two WHILLs, we developed our own wireless device of receiver side mounted on WHIL, to which ID numbers were assigned so that it can send control commands.The laptop was mounted on the back of the WHILL, and the battery at the bottom powered the laptop and the Varjo XR-1.

The total power consumption of the devices was about 300W, and the battery capacity was 1200wh. Therefore, the battery was replaced at the end of the third performance.
In order to track the position of the WHILL, we set up a pole, which has reflective markers on the end on the back of the WHILL . The pole served as an antenna for Xbee communication and a headphone rack.

—HMD

Varjo XR-1 equipped with a stereo camera is adapted. We designed our own marker jigs for tracking with motion capture system using VICON.

—Omni White Box

White cube with 4-wheel omni-wheel. Wirelessly controlled by 2.4GHz Xbee.

ONLINE

—Realtime Mutli-Angle Streaming

Gnzo's Multi-Angle Streaming System was adapted for online streaming of multiple viewpoints of the venue from which viewers can select their own viewpoints. Viewers were able to experience this performance online by selecting from six different angles in real time, including a bird's eye view of the entire venue, the control screen of actual operation systems, and the real time footage feed from head-mounted display.

—Original Payment System

Special online viewing page was developed so the payment system can be linked to various payment services (Amazon Pay was used for this project), and the IDs being used for watching streaming were monitored to prevent unauthorized access.