The future of storytelling through interactivity + virtual reality

It begins with Richard Parker, a seventeen-year-old orphan who knew nothing of sailing, but when he stumbled upon the Mignonette, a small yacht bound for Australia, he knew it was his ticket to a grand Victorian adventure. The yacht’s captain, was Thomas Dudley (by all accounts  an upstanding gentleman), and two other crew members including first mate Stephens and able Sailor Brooks.

A few months into the voyage, just off the Cape of Good Hope, an unseen wave hit the ship as Stephens was adjusting the ship’s course. No one knows why, but the Mignonette began to capsize.

Dudley loosened the lifeboat in time for everyone to scramble aboard, but not fast enough to grab more than two tins of turnips. The first night alone they had to fight off a shark and it was four more days until Dudley finally opened the can of turnips, on the fifth day they were lucky enough to catch a turtle.  But eight more days went by with no food and next to no water.

Imagine being in that lifeboat…Who would you be thinking about? What would you do?

Parker, being a sailing noob, did the one thing sailors knew never to do – he drank his fill of seawater. He was left dried out, slipping in and out of consciousness

Nineteen days since the shipwreck, after discussing their wives and kids Dudley suggested they draw lots to see who should die to save the rest. But Brooks refused, so no lots were drawn.

The next morning, with no hope in sight, Dudley sent Brooks to take a nap and motioned to Stephens that the boy Parker should be killed.  Dudley walked up to the boy, said a prayer, it was said Parker mumbled ”me?” before Dudley drove a penknife into his neck.

For four more days, the three men survived off Parker before being rescued and becoming part of one of the most famous legal cases of the era.

True story.

And it’s a story my lecturer used to make sure I deeply understood what utilitarianism means. Can you kill one for the good of the many? A lesson and story I’ve never forgotten.

Stories are beautiful Trojan horses seducing us into confronting questions we don’t really want to think about, or much ever have anything to do with personally. It hijacks our empathy to question our own values.

Was Dudley right to kill Parker?

Above is a mural by the brilliant artist Jo Ley, it hangs in our office. It has six different islands, each one representing a new era of storytelling, the technologies and characters that defined it. From our mysterious oral past, to the printing press, to the moving image where our fictional worlds jumped off the page.

The point here is we still use every one of these mediums today to tell each other stories, to try and explain this complicated world to each other.

The value of each new island has always been to enrich the stories of the last to new audiences, and give storytellers of that age a new toolkit to communicate in powerful ways. In the hands of the masters – The Shakespeare’s, The Orwell’s, The Hitchcock’s, audiences have been shaped to see the world through their eyes.In their hands, story has been the most effective way we have understood the human condition and the world we live in.

In our lifetime, we can see a 6th island start to emerge, that of interactivity, whose voice in storytelling grows stronger every year. What’s currently being done and what’s to be done in the future we’ll find incredibly interesting.

So what does interactivity mean for storytelling? What’s its value? Since the first time you ran Mario off the ledge, you became an interactor, and the pleasure you felt, interacting with a fictional world in a meaningful way is called agency.

It is a genuinely new experience in media that is uniquely enabled by interactivity.  Be it interactive sculptures, apps, games, or virtual reality. If someone is interacting with your work, you hope it’s in some way meaningful – it should be the goal of all good interaction design.

It’s easiest manifestation is scoring something like this:

A simple game we created for Energizer.  Where she intends to step on an icon, and it responds by giving her a nice juicy animation with sound and is meaningful because her score accumulates.

Her intention in the fictional world has been acted on and rewarded. That sense of agency becomes harder to design when you have 100+ interactors in an environment like we did at Vivid and pretty much guess work when you have 10,000 people in a stadium-wide game.

But the goal remains the same, make your interactors feel like they matter, like they exist in this fictional space. After 30 years, games in procedural environments are getting really really good at this, so by the end of these experiences you feel like you saved the princess, you are the Pokemon master, only you are protecting your clan.

What’s interesting is as we’ve improved at designing for agency through interaction, immersion, graphics, and sound, interactors are beginning to expect the next evolution of agency. What Janet Murray, former MIT Professor and now Associate Dean at Georgia Tech, calls dramatic agency.

Simply the state much more like our own where your interactions go beyond rewarding your actions with a sense you exist in this fictional world, but the reward of your actions change the very story, or outcomes of the world you’re interacting with.

Jump back into the lifeboat with me and the desperate crew of the Mignonette…

You are Brooks the Sailor, you have been sitting in this stinking boat for nineteen days. As the men talk of their wives and kids back home, Dudley the Captain comes to you and suggests we all draw straws.

But instead of choosing to refuse, you choose to agree. Who draws the short straw? Who lives and who dies? How does that change the very meaning of the story?

Interactivities innate affordance for dramatic agency is putting one hand on the holy grail of interactive storytelling. It gives us a sense of a world, like our own that can go more than one way. To ask how can my actions make this story, this situation different. It also presents the possibility of paying again through Parker’s eyes, through Dudley’s eyes.

Interactivity gives us a groundhog day tool to replay the story from multiple perspectives and compare the outcomes. All the while using story’s innate power of empathy to feel beyond a single character’s perspective. This affordance can fundamentally enhance how we use story to understand and make for a more emphatic society.

The most interesting question to me is not is this future possible, as the work of Michael Mateus and others like him who are developing playable models of social physics to accommodate such interaction is already proving it is.

But the more interesting question is, if we ever experience and interact with Hamlet on the Holodeck, it will not be the triumph of Shakespearian poetry we will admire but the Shakespearean eloquence of the scripting and logic rules written for the AI that generates these deeply human experiences and how we feel when a computer has made us cry.

This is where the clouds begin to part and we see what this 6th island can look like where interactivity doesn’t just impact how we experience stories but what use stories for.

So we’ve gone through how we use story to communicate meaning with each other and how interactivity could enhance our capacity to investigate and understand that meaning.

One of the building blocks to Janet Murray’s Hamlet on the Holodeck is playable social physics models to evolve stories around our actions, surely the other building block of our time is virtual reality (VR). What does VR mean for story and the 6th island?

Just as interactivities goal is agency, the very first and the very last virtual reality systems will all share a common goal – to immerse the user so well, they achieve a state of presence. Which is the state you’re are all in now, you feel like you exist wherever you’re sitting or standing.

Why VR today is getting so hyped (arguably over-hyped) is how close the technology can come to creating this sensation right out of the box. Its key affordance is, it’s very very immersive.

Where Tolkien writes pages and pages to describe the shire, and Jackson uses shot after shot to take you there. You put a headset on and you’re instantly immersed in Bag End. To reach presence in Middle Earth though is the job of the designer and the technology. The more both work together to meet the audience’s expectations of the fictional reality they occupy, the more present they can feel.

Ben’s immersed in this world, but he can hear it, walk around and his actions are met by the physical and social feedback of socially trying to kill each other. Because even the basic headsets can afford this feeling of presence. I think we need to think of VR as a fundamentally interactive medium, which it currently isn’t.

Last year, we were lucky enough to produce a 360 live-action Wallaby experience, where audiences could feel what a Wallaby felt like before the game – the training, the sheds, running onto the stadium, the whole kit and kaboodle.

You really do feel immersed in all these environments, but here’s the problem. There’s a point in the clip where Stephen Moore, the Wallabies Captain, is talking to you, right to you. But I’m looking around wherever I want, my backs turned to him. The real Stephen Moore would knock me out, but here I am dancing with no consequence, which breaks the experience the believability of the character and the respect of the experience.

We explored this problem in a Dreamtime story about Tibrogargan, a father of a family of mountain giants and how he tries to save them from a mega Tsunami. We experimented with by removing the user from the story, as an observer. But you still feel so immersed, like you’re there and should be able to interact with the world and its characters.

So when this mega wave is approaching and it towers in the sky, you feel like, you feel like the movie Scream, where she’s about to get knifed and you’re screaming at her to run, but she can’t hear you. But you’re in the room as well and you can’t run! Same thing with Tibrogargan, when you feel the weight of the wave about to crash down on you.

This is the major hurdle we are discovering as we all invent the medium of VR. We have always been protected from story by the 4th wall, the magical divide between you and the drama. Be it the page, the stage or the screen. With VR, the very value of story, giving us license to deal with dangerous and difficult issues suddenly comes into question.

Let’s return to our lifeboat for the last time.

You are Captain Dudley, you feel the rock of the waves, hold your hands up in front of your face to see there’s just bones, burns, and blisters, where your healthy fingers used to be. It’s been 20 days. You know the cabin boy has to die, otherwise, you will never see your wife and kids again. You pick up the penknife and shuffle over to the boy. He tries to move away, but he’s too weak.

Now if this was film, we would have a long shot, close up of Dudley’s resolute face, close up of Parker’s eyes panicked and terrified. And a long shot 20 meters from the boat as the deed is done.

But you’re here, you’re in the boat, the knife is in your hands and Parker’s eyes are staring right into yours. You can’t look away. Can you bring yourself to kill one to save many?

Just like our own reality, virtual reality has no walls. Story has given us the language of meaning. Interactivity is giving us control of the story and VR will put us inside the story.

Which all makes the future look very interesting to us at S1T2.