Accessing the Alternate Realities of VR, AR, and 360
When our reality is not enough, we invent new ones. It’s the way of humans. We’ve imagined new worlds and scenes in art and literature and film, and as technology advanced we created immersive new realities to explore. “The goal of immersion isn’t new,” says Dov Jacobson from GamesThatWork, in his article 12 Steps Towards Immersion. “Renaissance painters achieved it by inventing perspective. Audiophiles pursue it by piling up speakers. Early movies, grainy, jumpy and silent, were stunningly immersive to their new audiences. People ran screaming from the theater when an oncoming locomotive was projected on the screen.” With the massive technological boosts of the 21st century, immersive alternate realities have taken a giant leap forward, allowing us to step out of our everyday reality into worlds and experiences specially crafted for our enjoyment. “We are like the dreamer who dreams and then lives in the dream,” as the story goes.
While the paths for immersion are ever-growing and boundless, they’re currently broken down into three major categories: 360 Video (360), Augmented Reality (AR), and Virtual Reality (VR). These three arenas often overlap with each other and with our everyday reality to form mixed-reality experiences for consumers and audiences. Separate or united, these three realms of alternate realities form the future of immersive content, and numerous Atlanta creators and companies are diving in to create content that gives viewers and clients a new perspective.
Considered the kid brother of immersive content, 360 video is by far the most accessible of the immersive media to date, with users able to engage on headsets, mobile phones, or desktop/ laptop computers using point-and-drag navigation. 360 drops you in the center of the action, playing the scene out around you on all sides: you can’t move freely in the space, but you can pan and tilt and rotate to your heart’s content to take in the content around you. 360 video and photography became popularized most recently by Facebook, enabling users to spin around while holding their phones, taking in the content provided.
For some, it may seem like a flash in the pan, but for others the data says otherwise. “I don’t want to say any of that’s a gimmick,” says Spaceman Digital’s creative director Nathan Smith. “If you look at Google’s metrics, the interaction numbers on 360 video are through the roof compared to traditional video. Shares are up, too. They tested the same spot in 360 and traditional video at the same time; the play-through was lower on 360, but earned action metrics and shares were above traditional.” The key to this number boost is engagement. When the content prompts the user to interact, they’re more likely to stick around to explore. Interactivity breeds interest.
But it doesn’t come without hard work. “The process is six times as hard,” says Annie Eaton, co-founder and CEO at Futurus. “We have to capture up, down, left, right, front and back. Whereas, in traditional filming, the camera person is only filming front.” Futurus works in both 360 and VR to create marketing, training and testing experiences that immerse the subject in their surroundings, allowing for a deeper learning experience. “We have to produce the experience more like a stage show, where the audience is watching and can look all around them. It’s up to the producer and development team to pay attention to every detail and create an environment that reacts as good, if not better, than it does in real life.”
Jon Milavec of Mixed Bag Media can relate to the struggle. Making the changeover from traditional filmmaking to 360 filmmaking, he experienced the learning curve firsthand. “From a pure production standpoint, you can’t have all your gear and crew right by the camera,” he explains. “There’s an invisible wall, traditionally, that your stuff and crew can be behind. In 360, Nathan Smith you see everything. On our first big project there was a lot of choreography and blocking. ‘Where can the audio guy hide? What lights can we have? Where can people hide? Around corners?’ Lots of logistics for people and equipment.”
Learning the challenges of 360 posed for interesting solutions, however. “Some cameras literally see 360 degrees, some see 360 in horizontal and 240 in vertical, so there’s dead space at the bottom,” Milavec continues, describing his experience creating a video for Northside Hospital as part of their Celebrating Nurses Sponsorship with the Atlanta Journal-Constitution. “We used that dead space to our advantage. The camera was on a tripod, and we had our lighting director tied around the tripod holding a directional light (with another light mounted to the tripod) to light the talent. The talent had wireless microphones. With all of that, everything looked like any pro production. It’s hard to shoot real places and make it look like traditional production, but it can be done. It’s just about reimagining your space.”
For 360 filmmakers like Jason Drakeford, the challenge is both exciting and frustrating. “You have to take into account where the viewer is gazing, use different techniques to guide the viewer, such as motion graphics or user-interface animation,” explains Drakeford. “You lose a little of the magic of giving the audience your unique perspective, but I’m working towards changing that.” Drakeford (along with his long-time collaborator Thomas Nybo) uses the 360 medium for documentary features, dropping viewers into locations such as Syria and Afghanistan to tell the stories of the countries and their citizens. “Not just watching, but experiencing the reality — this is what initially pulled me into it,” he says. For Drakeford and Nybo’s film Interrupture, the viewer experiences a mixture of immersive video and photography that follows two 11-year old Syrian refugee girls.
Drakeford’s approach is very narrative, coming from a filmmaker background. “It’s not interactive, and it’s more of a film-like experience,” he describes. “It’s passive VR. The director displays their vision and story, a unique perspective told in a traditional film structure.” Drakeford foregoes 360 camera rigs, however, choosing to “stitch” together footage and assets into an immersive experience.
With companies like Nektr, though, taking the traditional approach can be intuitive. “There are simple ways to jump in,” offers Nektr’s innovation director, Jami Becker. “Get a cheaper 360 camera and experiment. Use your traditional software like Premiere to edit it with plugins like Metal. To do huge VR products you need some heavy firepower, but just to get started and have fun and learn you can use inexpensive things. Things exist that are just expansions of what traditional filmmakers use.” With a long personal work history in alternate realities (including The Sims and MechWarrior), Becker partnered with Jeff Levy to form Nektr. “We created a really cool 360 and VR tour of a cinemographic space of the Carlos Museum [at Emory],” Becker provides as an example. “We built a virtual gallery for them, because most museums have items they can’t show to the public. We wanted to apply the tech in a meaningful way, so the virtual gallery allowed us (and them) to show off art that’s usually just filed away in a box in a basement. We tour you through the museum in 360, but then you go into a virtual space for the other artwork — a space that doesn’t actually exist — so you can see it all.”
Nektr has kept it untraditional as well, though, experimenting with the parameters of 360. “There’s a group immersive experience, like projecting it on interior domes so we can all be in the 360 arena together. We’ve looked at that technology, we’ve done some displays on balls/spheres. We created a 360 experience that plays on a ball, and that’s a really cool evolution. It changes the concept of what a display is. You change the screen to be the room.”
“Don’t be intimidated,” offers Milavec. “It’s not hard to buy a $500 camera and use the app on your phone to learn how to shoot and how the world of 360 works. 360 and VR is not for everybody, but it can be for a lot of people depending on needs. Depending on what you want a client, buyer, donor or student to see, it could be the perfect thing to use.”
Smith from Spaceman agrees. He’s utilized 360 video on many behind-the-scenes segments to help viewers better explore the space of a set during productions. “Use the medium for what it is, don’t just pigeonhole the use of 360 or VR into every application. Use it when it helps your audience see something better. You’re going to get a lot of crap on the internet when people can just stand in a park and spin 360. That’s not the technology’s real future. Using it where it’s most beneficial is best.”
“Not just watching, but experiencing the reality...”
Many creators and companies view 360 as a stepping stone to VR rather than a viable medium in itself. Disagreeing with this assessment, however, is Sprocket Creative, who believe so strongly in the 360 medium that they’ve dedicated an entire sector of their business to prototyping how 360 can work—and work for anyone. “The content matters more than the medium to folks,” says creative director Dean Velez. “You have to have content that people care about, first and foremost.” Making the move from broadcast to digital, Sprocket considered immersive content to be part of the digital medium, and saw 360 video sliding under everyone’s radar as “just a fad thing.” They began exploring the terrain with content for Cartoon Network Latin America, but to better learn the world of 360, they created their own proprietary content: Victoria Frankenstein.
Serving as Sprocket’s research and development project, Victoria Frankenstein is “our sandbox,” according to executive producer Billy Reese. “Here we can experiment, we can make mistakes. It’s new, so there’s a lot of testing to be done.”
The Sprocket team designed Victoria without the usual animation engines of Unity or Unreal, opting for more traditional motion graphics, then built a world to showcase the talent. “No one was biting, so we had to create a world of possibilities to show them the possibilities,” says Velez. “We built an already finished product and showed the clients the components. And it changed everything. Now it’s a finished product they can see and explore and understand. And then we took the whole experience and adapted it over four other pieces— children, travel, corporate, event—to show people that it’s not married to the narrative; the strategy can work with anything. The goal of Victoria was not to show off Victoria, it was to show off the mechanism and how it can work for anything
Sprocket’s innovation with 360 prompted Turner Classic Movies to bring them on for another project: Noir Alley: 360° of Noir. “With TCM and the way we’ve engaged with folks for 23 years, 360 medium seems like a great opportunity to further connect and embrace fans in this world,” says TCM’s senior director of marketing, Steve Denker. “It’s exciting because you can both look all around AND be a part of the story. It feels like as people are looking to tell better stories and experience more of the story, this is a great opportunity.” Sprocket made sure the opportunities were endless. “We gamified the video to keep audience engagement through social media posts,” recounts Velez, “and we built more social media strategy along with the 360 video to keep the engagement going. We pulled 2D movie posters from the 360 video, we pulled segments of video out for 2D on-air campaign — all of which saved money for TCM. TCM was more than willing to play in these theoretical parameters. They’re proof that if you do a strategy the right way, people will see what it truly is and latch onto it.” 360 can be more than just a look around you. it can be an entire world unto itself.
Augment Your World
Differing from its siblings, augmented reality is the more fluid of the group, allowing the user to be half in and half out of normal reality. You don’t truly escape with augmented reality, you reshape the world around you instead. The concept has floated throughout human history since the early 20th Century with Sensorama, arguably the first augmentation of viewing entertainment. But AR began to gain traction in the 1970s, and really kicked into gear for the public in the 1990s-2000s.
It’s been a part of weather shows and sports shows for years, but many are only just now beginning to grasp how AR amplifies the experience for the user. “Things in a scene can trigger AR experiences; movie posters can pop up AR trailers of the film,” suggest Becker. “It doesn’t have to just be marketing, it can be part of storytelling. Going beyond that: If I want to share my story, I put out stickers about my story or a T-shirt or something, and the AR makes my film appear on your phone via my shirt. Now it’s a distribution method, too. AR is so much more.”
Fervently behind that sentiment are John Rich and Jerry Hudson, vice presidents of future experiences at Moxie Interactive. “This watershed moment is about to happen,” says Rich. “Once Apple announced the AR kit, they became the largest platform of mixed reality. When everyone gets the new iPhone, there’ll be tens of millions of people in this marketplace capable of experiencing mixed reality. AR has been around for years now, but there’s a barrier of an app or limiting the experience to a moment. The effort it took to engage wasn’t a good value exchange. It put people off due to the quality of experience, but this mass audience will bring a shift. People will expect most things to have a mixed reality layer to it, like how everyone expects things to be smartphone ready.”
Rich and Hudson spend their days thinking of how to get clients to adapt to future technologies before they flood the market and become obsolete, or less cutting-edge (so that they don’t become “Amazon’d or Uber’d,” in Rich’s words). The duo built an AR poster for Moe’s Southwest Grill that allowed customers in line at the restaurant to interact with the art via their phones, playing music and viewing content.
“2D is dying, it’s all about 3D.”
Hudson loves the potential for AR integration into the consumer’s everyday world. “You can consume a commercial or a trailer from a poster. Content gets attached to inanimate objects.” And as technology advances from phones to wearable glasses, the AR potential is boundless. “We’re hoping Apple’s glasses are wirelessly tethered to the phone, so you don’t need a high-end computer. That’s the next step into everything being on the glasses. Flat rectangles are unnatural and suck compared to gesture cues, voice-activation, and wearables. 2D is dying, it’s all about 3D.”
For now, companies like Bark Bark are working within the available technologies to create AR content for users, but they’re happy to expand forward and beyond. “Frankly, it doesn’t matter what the platform or the technology is: We’re agnostic,” says Tabitha Mason-Elliott, head of production for Bark Bark. “We’re more interested in learning about new technologies to evaluate how we can apply it to the marketing goals of our clients. Ultimately, we have to focus on making content that achieves brand directives and connects to their audiences.”
Bark Bark has worked with big names to create 360 and AR content, creating a second-screen interactive app for installations at the National Center for Civil and Human Rights, and partnering with Mountain Dew and The Walking Dead to create AR commercials for the two brands’ AR app game. Users could scan the Mountain Dew commercials to unlock content in the game.
It’s fun and exciting, but for Mason- Elliott, it’s about the quality of the content first and foremost. “You have to determine how to best use mixed reality as an effective storytelling tool that entertains audiences and accomplishes something for the brand," she says. "People create a lot of hype around new technologies, but mixed reality can’t fix subpar creative. The creative and execution will always be the main differentiator and having solid content is key to making technology last. In the long term, we still need to get more people exploring it and using it. I’m hoping to see a tipping point where it’s at least on the minds of our clients from the outset of thinking about a project.”
We Are VR
The crown jewel of alternate realities, virtual reality makes the public the most excited. It feels the most like the future; the user puts on a headset to fully immerse themselves in a new world.
The definition of virtual reality, to differentiate it from 360 video, can get a little hazy. Ultimately, it’s about user immersion — i.e. making the viewer feel like they’re really dropped into the content.
The thing that appeals to most people about VR currently is its growing accessibility on the creation side. “VR is doing what the smartphone did: It changed how we interacted with the world and made everything accessible,” says Laura Hall, a virtual-reality advocate who operates under the handle The VR Girl. “You have to get behind the technology before it’s controlled by people with more money. For $1,200, anyone can now own the tools to create content in VR. There are all kinds of apps for content creation and viewing. It’s catching up.”
With the virtual-reality playing field opening up, the real question on the minds of content creators has been how to best utilize the medium. Is VR a standalone medium like 360? Does it serve best in conjunction with other media like AR? Or is it a combination of the two, and more? Every creator tackles it differently. “In the near term, I see companion programs with television and movie releases becoming commonplace,” says Eaton. “In the long term, I would hope that we would consume more full-length content on these immersive devices, creating a truer and more connected experience.”
Evolved Cinema has taken VR as a chance to help build brands outside of the box. “It wasn’t so much a shift in the workflow as it was a shift in how we had to think about the content,” says Hunter Hughes, director and cinematographer at Evolved. “We didn’t know what we were getting into when we first jumped into it, but we kept a frank perspective with how we approached it. If you asked me a year and a half ago if I saw us producing a gaming-like experience, I’d say we weren’t equipped or positioned for that, but when a client came in with a need, we learned what we needed to learn to make it work.” Evolved recently tackled VR branding with a home improvement company, creating an immersive experience that puts the user on a forklift in the store, retrieving an item as the shelves collapse. “We used hi-res texture plates that we captured inside the store,” explains Hughes. “We used digital assets they provided like boxes and labels and CAD files. We went to one of their stores to capture binaural audio of the experience — human perspective audio with silicone ear receptors. It was extremely effective.” For Hughes, it was about creating the most realistic virtual experience possible.
For Trick 3D, there wasn’t just one route. They’ve created branded VR content and narrative VR content, but they’ve also worked VR into the preproduction side of the industry. “People say that there’s not a consumer base and that it’s mostly for gamers,” says Trick 3D founder Chad Eikhoff. “That’s still up in the air. We don’t know how consumers will ultimately use VR, or how that will shape the future of VR. But in terms of the technology, it’s capable of monumentally shifting the way content is created. It’s already happening; it doesn’t require a consumer marketplace. That’s true for us working with big brands like Delta and entertainment content like [the Make-a- Wish Foundation collaboration feature] Zayden’s Wish. You get value out of it as a production tool.”
Using VR as a production tool, Trick 3D can mock up a scene for a production while the scene is still in its concept phase. This previsualization tool allows the directors and DPs to don a headset and step into the middle of a virtual rendering of their scene. Here, they can teleport around the area to get a sense of best camera angles and lighting before ever setting foot on location, potentially saving significant production costs by not having extraneous equipment. “This pushes everything up in the pipeline. The director or DP don’t just show up on set and figure it out while everyone holds on,” explains Eikhoff. “Everything’s planned out upfront, since the DP and director can go into the environment and move around like they’re really there.”
Eikhoff sees a solid future for VR, with the first step being integrating VR content as an extension of existing content.
With the hit Christmas special An Elf’s Story, Trick 3D designed an entire virtual Santa’s Village in conjunction with the film. Fans of the film could explore the virtual village, getting a more in-depth look at the world of An Elf’s Story. The added bonus of creating this world, of course, was that Trick 3D now also had the setting for the film that they could navigate and shoot at any angle. Eikhoff wants this mentality to expand significantly. “Hollywood’s first big play is location-based entertainment. Get people into traditional theaters, but then in the lobby they offer an extended narrative experience. People come out of those movies and they want to experience those worlds. You see characters enter a new world and you want to do that, so the burden is on the narrative to build that interesting world and entice people to want to get inside of it. They get the story through linear, traditional narrative, and then you give them an opportunity to further engage the world. It may be the same characters, it may be different characters.”
Rich and Hudson are excited for the advancements in VR as companion elements. “Tons of companies are jumping into this space,” says Hudson. “The major movie theaters are looking at it. We’ve seen full-length motion pictures put into VR. John Gaeta is doing VR for Disney and Star Wars. Companion pieces will turn into the main focus.”
Adult Swim also feels similarly, creating VR content in conjunction with its existing intellectual property (IP), such as the Rick and Morty VR experience. “This furthers the narrative, too. We don’t just do it to do it,” says associate creative director Shawn Moore. “Most marketers and agencies jumped into VR because it was new, so you got VR test drives… Those were cool, but it didn’t really help sell cars, because the experience was just fun. It had no real connection; it didn’t persuade people any more than actually driving the car would. Right now you need an IP that can stand on its own well, then look at VR as an extension of that world.”
Moore and Adult Swim have also used VR to create expanded experiences with music offered from the network. The Rick and Morty VR experience dropped fans of the show inside the established world, while the characters guided them through the experience. “From storytelling, you need to learn how to get people to focus,” says Moore. “I need you to look behind you for this. Rick and Morty worked well in that experience because Rick prompted you and auditory cues prompted you. Having a guide is really beneficial. It makes looking around you natural and keeps the viewer on track.”
Eikhoff agrees: “Being just an observer in a virtual space is awkward unless the narrative supports that notion.” To expand VR into a full medium of storytelling, the notion of narrative must be reimagined.
One studio reimagining the narrative—and reimagining production— is Studio Disrupt, a fully-VR production company based in Atlanta. Fronted by Jak Wilmont, Disrupt is something new: a wholly-VR company that solely creates original content, not producing in other mediums or for other IPs and brands. Coming from a video game narrative background (known as machinima), Wilmont dove into the virtual world, producing immersive films like Please State Your Name and La Camila. “I want to provide the viewer the chance to completely escape into another world,” says Wilmont, “so the more they can look at and the more angles they can view, the more opportunity they can have to come back and re-watch or see things differently from their friends.” When asked why Disrupt sticks to original content only, Wilmont replies, “I want to figure out how best to tell a story before I branch out to other people’s stuff.”
Disrupt plays in the new medium well, learning as the art evolves. “The hardest part is pacing,” admits Wilmont. “When you write a story, you imagine how it plays out. But I had to throw it out and consider how someone actually ACTS it out or how they observe it instead. I think more interactivity will be key. The coolest thing is to have a story that is different based on your interaction. Right now, it’s just about figuring out the technical side, but down the road I want to build to that.”
Alternate realities can be overwhelming and scary to some, which breeds trepidation to adapt. “When I first got into it, I was kind of scared of it,” admits Spaceman’s Smith. “I thought it’d ruin society. But I found myself getting a good taste once in a while and not sitting with the headset on all the time. Now that it’s growing, the content creators will be in a shuffle for the next decade as technology evolves, trying to figure out how to make the best content for it. People think it’ll go away like 3D did in home viewing, but that’s from the early bad experiences.”
“VR has a marketing problem: How do you get the rest of the world to see it?” asks The VR Girl. “If we want VR to go mainstream, we need to show how it can mesh with real life. Gaming has been a big factor, but that’s not how VR goes mainstream. Everyone wants to know what VR can do for them, and it’s more than gaming and storytelling. You can create from VR as well; you have a tool of content creation. I put my goggles on and grab my controls and create a little thing. It’s easy to learn, easy to do. If industries don’t pivot into VR, they’ll be left behind.”
“Talking about virtual reality is like
dancing about architecture.”
Hudson agrees with the marketing issue. “Until you experience it, you don’t really understand it. It’s like Chris Milk said during a TED Talk: ‘Talking about virtual reality is like dancing about architecture.’” Seeing is believing, for users and for clients. “We created a VR experience called Stay Alive Live for the shared platform Twitch,” recounts Rich. “Each round you need to stay alive, but everyone in Twitch can participate. It’s the first Twitch-Plays VR game that we know of. The guys at Twitch thought it was far off in the future, that VR was a five-year plan…and now we’re aiming for this year. It’s sooner than you think.”
Becker thinks that Atlanta is perfect proving ground for alternate realities. “The cool thing we’re seeing is high commercialization in Atlanta and lots of vision. So many industries are experimenting right now, and that wasn’t traditional in the past. They’re willing to do that here because there’s been an entrepreneurial slant to corporations. They realize what they’re missing out on, so they’re willing to jump into the technology boost because it’s more accessible.”
So don’t be afraid to dive into these technologies. The beauty of their newness is that everyone’s learning together. “Don’t get constrained by this being film or a game,” says Velez. “This is not a game, this is its own genre, and when we all really figure it out, it’ll be amazing. Everyone’s a professional the moment you touch it, though, because everyone’s still learning it. It’s scary, and if you’re stupid enough, it’s awesome to play with.”
Wilmont agrees. “Just keep your mind open. It’s so early, people are trying to figure out what it can be instead of just experimenting with it. We have so much time still. There are no VR experts, so there’s no limit. I hope I never get complacent.”
“And it goes beyond the film and television industry,” adds Hughes. “This is human evolution being impacted by this technology. You can talk about the experience, but you don’t fully understand it until you put on a headset and change your entire perspective. It’s unnerving. I gave this VR presentation speaking, and I told everyone, ‘Look, I have no idea what I’m doing and neither does anyone else. And if they tell you they do then they’re lying because no one is prepared for this totally different way of thinking.’”