Traditional Performing Arts Meets Mixed Reality

Thanks to the rapid advancement of technology, life seems to be changing every day. The first of its kind is something we have been saying a lot when it comes to Mixed Reality. The next first of its kind application and use case with Mixed Reality is happening in the space of theater and performing arts.

October 3rd (Monday) to 25th (Tuesday) 2022 marked the first use of Mixed Reality in theater and performing arts. In Japan at the at Osaka Shochikuza Theatre, a Kabuki (song and dance) performance of a “Japanese ghost story (J horror) Kabuki” seem to be a normal show for most. In the back seats behind the audience featured select participants who experienced the show wearing Microsoft’s Mixed Reality headset the HoloLens. They experienced “a new form of theatergoing”, in which subtitles and video production are superimposed on the main stage.

According to Mr. Tsuyoshi Tomita of the production company Shochiku Co., Ltd., how this idea came to fruition is they have been “working on a next-generation entertainment creation business that fuses the real and the virtual.” They explored the potential use of AR (Augmented Reality) but did not like the results. When they discovered Mixed Reality, they quickly realized MR was the answer. They could enhance the viewing experience without taking away from the normal show.

The selection of Mixed Reality was the easy part. How to use MR to add new virtual elements became a big challenge for developers. The show producers hired a couple of companies to help with the MR design and CG production. But they struggled with the first attempt to fuse MR and reality. They tried many different combinations such as subtitles, videos, and the combination of both subtitles and videos to improve upon the live performance. Finally, they developed two major systems. One is designed to display visual effects and subtitles and another a simultaneous display system. The reason behind the second application is each audience member has a different vantage point based on where they sit. To overcome this and create a consistent experience for each user a custom display had to be designed for each audience member.

During the show the selected participants started off watching the show without the headset. The reason for this is they wanted participants to experience what the show is normally like. After the first act, participants were asked to wear the headset and view the show in Mixed Reality. The results were the best outcome they could have hoped for. The application and headset did not interfere with the show, they enhanced the viewing.