Immersive storytelling in virtual reality: A breakdown of Beyond the Stars VR

We’ve been working with VR for a long time now and in the initial concepting phase we decided that it would be a great medium to get the kids really immersed in the story we were telling. The basic flow is that the kids start by watching the animated short film which introduces them to the story and the characters of BTS. Then they play the VR experience, which virtually puts them in the world we created, making it believable and gives them agency to make choices and interact with the world around them. That world is then translated into the storybook and mobile game which they use as part of their curriculum in school. It’s worth noting that none of these kids have tried VR before, so it would hopefully be very exciting for them!

Story

The story puts you in the shoes of a Fijian student at the end of a school day. You begin in your classroom, the last person out. You see your school bag on the table and it seems to be moving. As you go to pick it up it bursts open, unravelling into a magic Masi (which is a Fijian cloth) floating in front of you. The Masi flies around the room and opens a portal in the classroom floor. It then swan dives into the hole, calling you to follow it down. When you jump you are left with little visibility, with only a small light radiating from a staff that’s stuck into the ground. When you pick it up, the area around you is illuminated and your Masi friend comes back to play!

You are led through the cave with the help of the magic Masi and your staff, which allows you to teleport to different locations. The walls of the cave play animated murals and a friendly voice tells you the story of the lost guardians, who helped protect the land and its people. You have been chosen as the hero to bring back the health and prosperity people once had but have long forgotten.

You are led out of the cave and brought to an ancient ruin surrounded by statues of the guardians. You use your magic staff to rekindle the power of the guardians. Then, you must make a choice about which guardian will assist you on this journey. After you choose, your magic Masi tumbles back into the form of your school bag and changes colour based on the guardian you choose. The bag opens and out comes the Beyond the Stars storybook, which every child gets in physical form to use at school and at home. This is where the VR experience ends and the hero’s journey begins!

Pre-production

In the pre-production phase we had a few questions to answer; Who is this for? What do we want the player to feel? How do we get the player to see what we want them to see? There’s always a danger in VR storytelling that the player will miss something that was important to the story. It doesn’t matter how much work you put into it, if they miss something important it never existed in their experience. In a lot of ways there’s no getting around this, but you can do things that help.

Using attractors to get people’s attention is one solution that we used. It involves guiding the player through the design of the environment using audio triggers and animations to catch their attention. Not putting so much importance on one particular part and looking at the experience as a whole is also a good approach. “Did they miss something? Yes. Did they still come away with the same feeling and sense of the story? Hopefully yes!” It’s a tricky thing to balance and get right, but not putting all your eggs in one basket can help.

The kids we made this for are in Year 4 and are between 8 and 9 years old, so we thought it would be best to go with a stylised, cartoony aesthetic. Shifting the mood throughout the experience was also important to us, as it helped illustrate the story and really push the sense of magic and fantasy. At the start of the experience we wanted to make the scene warm, inviting and somewhat familiar. Then as the player jumps down through a portal and into a cave we shift to a more mysterious, magical environment.

Production

The production followed a pretty traditional pipeline, starting with a blockout of the environment to get a general layout, scale and feeling of the space. We wanted the player to explore but we needed to balance that with keeping them on track with our linear story. The player area is limited to a 3×3 metre space, and we didn’t want the kids running into walls and getting hurt. Our solution was to use a few different techniques. We limited the players movement through the design of the environment. For example, when you jump down the hole, you land on a sandbank partly surrounded by water. We also used fog to limit the player’s visibility and then animated it to move away and become more transparent, revealing more once the player had finished watching the animated murals.

For locomotion through the cave we used a teleportation mechanic in the magic staff for locomotion, which reinforced the story and fantasy and also kept the exploratory element in the experience. In the process we found that pressing a button and jumping to a new area could leave the player a little lost and disorientated. Instead we opted for a technique that moves the player quickly to the next area (think Star Wars when the ship goes into hyperdrive). This did take some tweaking to make sure the player didn’t feel sick, but overall it worked really well.

Lighting

Lighting and post-processing was a crucial element to this project. The classroom uses an intense warm, directional light coming through the side window, looking too bright to see anything outside. This adds to the atmosphere and really puts the player’s focus on the scene around them. In the cave we shift to colder blues, purples and greens to push the sense of fantasy and make the player feel like they’ve crossed into another world.

We also made use of Unreal’s baked emissive lighting feature which allows an emissive material to be used as a light. This meant we could add different colours and bleed them together really nicely to create dynamic colour and highlight key areas. We placed different coloured spheres around the cave to achieve this. The fog sheets also helped in isolating certain areas in the cave so the player doesn’t have too much visual stimulus all at once.

The lighting in the final scene was, again, pretty simple. A directional light was used and a few spot lights put focus on the play area and to highlight the guardian statues with a rim light. We also gave the town’s houses an emissive window material to make them feel a bit more alive.

The only dynamic light throughout the experience was the one attached to the magic staff. We wanted to give the light a very hard edge to add to the toon style, so we used a light function which measured the distance from the light to the casted surface and used some math to make the edge hard.

Materials

I got to work on some cool materials in this project so I’ll go into some of these now. The cave rock material is pretty simple, but because it’s used throughout the entire cave I wanted to break up the similarity and add more interest. To do this I added moss on top of the base material. I wanted to keep it on style so I didn’t add too much detail, just making the edges of the moss a brighter green and a subtle normal map I made using Quixel from a simple gaussian noise texture. Then I used a world aligned blend node and applied it to the up side of the mesh so the moss always sits on top of the rock, even if you change the rotation. To finish I added the functionality to remove the moss through vertex painting so I could paint out where I didn’t want the moss visible and break up the similarity between rocks.

In Fiji, classrooms are usually covered in posters. I wanted to re-create this and make a quick, easy way of putting different posters around the room and keep it optimised. I did this through the material by creating a texture with multiple poster designs laid out on a grid. Then I added a parameter to move the UV’s in steps of the grid based on the object position through per instance random. That way when I duplicated a poster and moved it to a new position, the UV’s would move to a new position of the texture and add a different poster design.

The waterfall was a really fun asset to make. I wanted to make it super cartoony, and through some research I found a great breakdown of the VFX from a game called ‘Rime’ by VFX Artist Simon Schreibt. It gave me lots of tips to make the style of waterfall I wanted, and I’d encourage anyone interested to check it out. To begin I started with the waterfall mesh, which is pretty much just a tube that’s scaled and tapers out at the bottom with a sharp curve at the top.

After this I took the mesh into the engine and started working on the material. Starting with the water foam using a panning noise texture as a base, I wanted to change the speed of the water effect in certain places. So back in Maya I stretched the UV’s at the curve of the waterfall and at the bottom end of the waterfall to mimic the water going faster in those places and make it a bit more dynamic.

Then I wanted the foam to have a high contrast against the water for that cartoony style. I also wanted it to be controlled by vertex painting so I could paint where I wanted it visible. For the contrast I used a power node and a cheap contrast node to control the sharpness of the black and white values. For the vertex paint I used a lerp node with my foam mask going into one input, 0 in the other and used the vertex paint node as the alpha. I also added another panning noise texture which only affects the foam and inputs into the bump offset of the material, which displaced the foam giving it some volume and movement. The water splashing at the end is just a particle emitter which uses a mesh that looks like a splash and changes in scale as it emits.

Sequencer

For the animated rock murals we utilized Unreal Sequencer. This is a multi-track animation editor within Unreal, which is generally used for in-game cinematics. It allows you to create tracks for animations, triggering audio and lots more. If you’re familiar with Adobe’s After Effects it’s a similar tool. Our lead animator Nicky Tunpitcha and I worked on the design and animations. Nicky worked in After Effects and then I would re-create the animations within Unreal, adding audio and animating material parameters to make them appear, glow and trigger particle effects. It was a really great pipeline!

Nvidia Flow

Nvidia Flow is a real-time fluid simulator, particularly used for simulating fire and smoke. It was over a lunch conversation that me and technical director Jack Condon said it would be awesome to try using it to make some interactive fog in the cave. This was a stretch goal and we weren’t sure if we were going to get time to even attempt it. Plus, at the time it wasn’t even available for our version of the game engine. The week before the project’s deadline it was released for our version and we decided to work over the weekend to get it in. I’m really glad we got to and I think it adds a really magical element to the experience.

I had never used Nvidia Flow before and that soon became apparent! It took a lot of tweaking and hair pulling to get it to the desired low hanging fog effect I was after. But I got there in the end and it was definitely the icing on the cake. I won’t go into too much detail because, to be honest, I was quite delirious at the time. It was one of those things that once I got it looking good I never touched it again.

What I loved about it was it’s curve based colouring. It allowed me to make some really nice fantasy colours come through the fog and the colour was also affected by velocity – as you interact with the fog the colours will change and get more intense. Pretty cool stuff and very fun to play with.

If you have a room-scale VR kit and an Nvidia card, you can try Beyond the Stars VR for yourself on Steam. If not, you can also take a closer look at the experience in this trailer video. And if you’d like to know more about the wider project, this blog post is a great place to start.

Subscribe below to get the latest updates and insights from Steve and the rest of the team.