Incorporating HCI to Tell Better Immersive Stories

“Production for VR” is still an all-inclusive, yet elusive concept, as evidenced by the diversity of panelists at the talk of the same name at SIGGRAPH this year. Works presented ranged across animation, live action and computer generated content.  On the panel were Larry Cutler (Baobab, “Invasion!”) Alex Henning and Tzuen Wu (Magnopus, “The Argos Files”) and Joe Farrell (Tangerine, “Jungle Book: Through Mowgli’s Eyes), and myself (Sunchaser, “Kanju”).

For VR productions, I frequently see content developers look to film and television directors to create 360 experiences. They tell great stories after all! Problem is that those extremely talented people are great at telling linear stories in squares, but immersive entertainment involves usability constraints and interactions. As a result, the  experiences tend to end up being confusing, disorienting or overwhelming. So I took a different angle for the tech-savvy, engineering-inspired, boundary-pushing audience at SIGGRAPH.

I suggested that we need to look outside film and television for storytelling solutions as the language of immersion evolves. Specifically, Human-Computer Interaction (HCI). HCI is a computer science field that combines the fields of psychology, computer science, and design to optimize user experiences. With my VR documentary “Kanju” as an example, I discussed how we can implement HCI techniques to tell better immersive stories.

A little background on the project (there’s a 10-part series on Creating Kanju on this blog, if you’re interested): the narrative storyline for “Kanju” required audiences to follow an immersive live action experience that spanned three separate African countries and included four distinct storylines in a 360 environment. “Kanju” premiered at Tribeca Film Festival where it screened on a head-mounted display (HMD) device to audiences, many of whom had never experienced virtual reality, immersive environments, or HMDs.

To create a clear and compelling narrative experience for audiences in the immersive environment, key HCI principles were combined with traditional storytelling techniques to create what I call Human-Centered Storytelling. Some of the principles discussed were:

  1. Know Thy User (for they are not you) – this is more of a mantra than a principle, but if you spend hours a day in an HMD or shooting 360 content, you are not your average user. We tend to predict and guess how users will respond to interfaces (or content), but user testing is far more helpful and accurate.
  2. Visibility – the more visible functions are, the more likely users will know what to do next. In VR, our environments are filled with content the users can’t see. Start where the user starts, use visual cues inside environments.
  3. Constraint – restricting user interactions through physical, logical, or cultural constraints. In the square content, this constraint is the frame. In immersive content, we need to go deeper to seamlessly guide our audiences.
  4. Cognitive Load – the total amount of mental effort being used in the working memory. Novice VR users experience cognitive overload – sensory input from 360×180 degrees, spatialized sound, haptic input devices. Minimize the “noise” and reduce unnecessary elements.
  5. Consistency – Similar operations and elements achieve similar tasks. This consistency elevates storytelling and provides a foundation for audiences’ expectations.The results at Tribeca, Nantucket, SIGGRAPH and other events screening “Kanju” around the world have been overwhelming positive, including retention of storylines, clearly following the story across locations and characters, and not feeling overwhelmed. Let’s keep looking outside traditional storytelling for immersive answers.

    You can read the full abstract from the paper here.

Leave a Reply

Your email address will not be published. Required fields are marked *