Surprising Secrets and Actionable Examples of Immersive Experiences
Immersive experience may be a trendy buzzword, but it’s also one of Gartner’s top ten tech trends for 2018, along with blockchain, and intelligent apps. It’s definitely worth exploring the topic and how it relates to virtual reality (VR), augmented reality (AR), and occurrences in everyday life. If you’re still not sure if you may have taken part in an immersive experience, our extensive list of examples will help clear that up for you. Chances are that you have.
What Is an Immersive Experience?
A truly immersive experience is one that creates a sense of flow. The actual experience can be something simple like an engrossing book, or complex like strapping on a VR helmet and flying over the ocean while a pod of whales rise to the surface below. The best-known immersive experiences are found in AR and VR, which can mimic, replace, or enhance the real world. Here’s how:
Virtual reality mimics or replaces activities and places in the real world, but currently requires a headset to experience. Headsets block out most of the visual and auditory sensations from the real word and send sights and sounds via a view screen and earphones. The Oculus Rift, HTC Vive, Microsoft HoloLens, Samsung Gear, and Google Cardboard are some commercially available VR headsets that you can use to experience VR. The first three are complete units, the last two require inserting a smartphone, which functions as the view screen.
There are two main types of VR experiences. The first uses 360-degree cameras to film some portion of the real world (but digitally-created elements can be added later), such as travel videos and apartment tours. The second kind of VR is computer-generated (real-world elements can be added, too). The experience never existed in the real world and you’re experiencing a completely fabricated environment virtually. VR games are a prime example.
Augmented reality enhances the real world by overlaying it with virtual elements. Examples of AR experiences are Pokémon Go, Ikea Place, or Topology Eyewear. In these instances, you’ll need to use a smartphone or a tablet to view the result.
Like VR, AR has a couple variants. The first option overlays a digital layer on an image of the world in real-time, so as the viewing device moves around in space, the generated objects react in sync. AR games use this method. The second variant adds elements to a saved picture or video. Many makeup and hairstyle apps use this version of AR.
Mixed reality (MR), which is a bit of a combo between AR and VR, has a couple of definitions. Some people say MR is the same as AR. Others say MR is when digital objects are embedded into the real world. Still, others it’s an umbrella term that incorporates AR, VR, and everything in between. However, MR is defined as “extended reality.”
On the web, a page that uses a photo or video as the entire background, as well as including elements such as sound to draw the user in can be called an immersive experience. Some examples of this kind of experience are the University of Texas’s Long Horn Storm (which is no longer active, but it may be the first example of this, developed by a company called Right Here Interactive in 2008), the Google Arts and Culture site, France’s Control Films, and a New York Times article about an avalanche. These may become more prevalent, as HTML5 spreads. Responsive design will help bring these experiences to mobile devices.
We can also find immersive experiences in the real world, like a 1950s-themed party that engages guests with rich details like bottles of Nehi soda (aka RC cola) and egg-salad sandwiches served by people in poodle skirts or men wearing fedoras while engaging in activities like a sock hop. Another real-world example would be taking a trip to a theme park where everything is designed to keep attention away from what’s outside the gates, or an escape room where a group has a set time to figure out how to get out of a room before some pretend fate befalls them.
Examples of Immersive Experiences
The list below includes ideas and situations that represent a broad range of applications immersive experiences.
The first commercial VR headset, the Forte VFX1, came out in 1995. However, VR is still not widely available, mostly because the technology to create it hasn’t been easily available and the cost to consumers is high. Both of those barriers are beginning to fall away making adoption more accessible.
- Games: Probably the best-known use of VR at the moment. Some well-known VR games are Star Trek: Bridge Crew and Doom VFR.
- Battlefield Simulations: The Department of Defense was an early adopter of VR and used it to train soldiers in battlefield situations and other precarious environments without actually putting them in danger.
- Healthcare: Medial personal can practice operations and learn new techniques before trying them on a live patient.
- Vehicles: Drivers and pilots can use VR to learn the layout of a new vehicle or practice dangerous maneuvers. Pilots have long used full-motion simulators that incorporated some VR-like features, but the improvements in sound and visual quality have made a huge difference. Vehicles can also be operated remotely or be used in building experiences for driverless cars, such as those being developed by Uber, Lyft, Google, and others.
- Exposure Therapy: Doctors and therapists can help people with phobias or trauma by gradually exposing them to their triggers.
- Physical Therapy: Patients can opt in for remote sessions that provide quick feedback.
- Drone Operations: A drone operator can tune out their environment and focus on the area where the drone they are piloting is flying. Consumer drones can also be operated with a VR interface. Data gathered via drone can also be analyzed in a VR environment.
- Equipment Repair: Mechanics and others who maintain and repair equipment (e.g. cars, robots, electronics, etc.) can use VR for training to prepare them for many more situations than might arise in the real world.
- Cooking: Chefs can learn how to prepare dishes and eliminate food waste by practicing techniques before using real ingredients.
- Sciences: Biologists can immerse themselves in the structure of a cell, mice can be run through a virtual maze, or rocket designs can be tested in a virtual wind tunnel. Check out what’s at the bottom of the ocean with this app.
- Arcades: Play some immersive games solo or with friends. Arcades are part of a concept called Location-Based VR (LBVR). One advantage of LBVR is that people can experience VR without having to buy a VR headset.
Real Estate: Explore the inside of an apartment or house from another location, or even before it’s built. Dogu Taskiran is the co-founder and CEO of Stambol Studios, a creative technology studio specializing in immersive content and interactive applications for real estate marketing, enterprise and brand engagement applications through VR and AR. Taskiran says, “We get all the architectural diagrams, 2D models, floor plans, and we create the 3D visualizations for them. We go to the property site, take 360 photos for multiple levels of a high rise so we can showcase different views. Then we create the 3D models and combine it with the view photos so that when you’re inside the VR model you see outside, and when your outside it you can see inside as well.”
- Meditation: A guided meditation experience can be created in a virtual world to help users practice and reach nirvana.
- Education: VR has many education applications, as well as helping students improve their performance. Students can virally leave earth to learn about the solar system, and study real-world building techniques in virtual space. VR can even help students with special needs.
- Storytelling: Choose your own adventure style storytelling leaves the pages of a book and finds a home in the virtual world. Users get one step closer to the real world, changing the way stories are created and.
- Movies: While full-length VR movies are probably still a long way off, a VR presentation won an Oscar this year for the first time. The entire experience of movie going will have to be remade (imagine a room full of people in VR headsets wandering around and bumping into each other). People who work in Hollywood, such as video editors who may need to work in VR while working on a VR film, will also have to adapt.
- Adult Content: The adult film industry has been on the cutting edge of many technological changes, such as streaming video, online payments, and video chat. VR is no different.
- Sports: Some sporting events are now broadcast in VR, allowing fans to follow the action and see the game from many angles.
- Concerts: Fans can watch their favorite performers from anywhere on the planet from their own home.
- Art: Enter the mind of an artist.
- Product Development: Prototypes can be built in a virtual environment. Taskiran explains, “Ford has been prototyping their products for more than 20 years using virtual reality, analyzing different aspects of it, even looking at reflections on the paint. It’s expensive to build a car prototype, but with VR, it’s all within reach at a much lower cost. You can change it on the fly. Go into the engine of a car that you’ve been designing and see if the pieces are going to fit together and make better decisions. You can collaborate between teams in different parts of the world. One engineer could be in Australia the other in San Francisco, and they could work inside the same product.”
- Music: Musicians can collaborate and practice with players in other locations.
- Travel: Explore rental cars or hotel rooms, in advance to make sure you like them. Virtually visit places you can’t get to in person.
- Construction Management: VR can be used to collaboratively model buildings, as well as for training machine operators.
- City Planning: See the impact of new buildings on existing structures.
- Business Operations: VR can be used in a range of applications, from meetings to interviews.
Augmented Reality Immersive Experiences
In the late 60s at Harvard, computer scientist Ivan Sutherland demonstrated the first use of AR when he displayed simple wireframes via a head-mounted display. However, the first time most people experienced AR was in the late 90s when the first down line in football started being shown in yellow during televised games. A BMW Mini ad in a German magazine in 2008 allowed readers with a webcam to view a 3D rendering of a car on the magazine page; later that year, Wikitude launched an AR travel guide for mobile devices. Apple released its ARKit for developers, so more AR options should be available as people find ways to deploy it. Wingnut AR (founded by Peter Jackson of Lord of the Rings trilogy fame) is a leader in using ARKit to develop compelling content. The equivalent of ARKit for the Android OS is ARCore.
A concept related to AR is diminished reality and refers to removing real-world items from the AR viewer. Sonia Schechter is the CMO of Marxent Labs, a company that builds AR 3D space planning and customization software for full-line furniture companies, as well as DIY centers, home improvement stores, and homebuilders. The company has a patent pending on diminished reality. Schechter says, “Diminished reality allows you to select an object in space, say a couch in a room, circle it, and eliminate it so you’re looking at an empty room. The couch is still there, but you don’t see it in the AR viewer.”
Here are some more examples of how AR enables immersive experiences:
- Air Traffic Control: Still in the testing phase, AR may be able to help air traffic controllers manage the plane traffic, and may contribute most when the weather is bad.
- Games: Besides the well-known Pokémon Go, there are many other AR games, like Ingress and Hide the Hubu.
- Theater: AR can be used to assist people with hearing impairments, make performances interactive, add elements to the performance, and even incorporate elements of gaming.
- Messaging: Snapchat popularized AR in messaging apps, by adding visual effects to user videos and photos.
- Movies: As well as being used as part of a movie plot, AR can be used to add effects to movies or make marketing more immersive.
- Sports: Games can be enhanced in many ways, like a virtual scoreboard, seeing a player’s stats when they are on the field, or creating fan competitions during time outs.
- Concerts: Special effects can be added to the stage via AR to enhance the experience of a live show.
- Art: AR can make exhibitions interactive.
- Video Conferences: Calls could be augmented in to make them more productive and interactive.
- Theme Parks: New entertainment options can be brought to a day at the theme park.
- Tourism: Visitors can see custom information about attractions and events.
- Customer Service: Customers can view troubleshooting tips and steps while looking at the product that’s causing problems.
- Equipment Repair: Having a manual or a video right in front of a repair person’s eyes at they work on a machine can improve speed and efficiency.
Shopping: Shoppers can scan a bar code and get directed to matching accessories or related items, or find out if a piece of clothing is available in their size. AR-enabled mirrors can show shoppers what items look like on them even if they are not in stock. Cortney Harding is the founder and principal of Friends with Holograms, which works with retailers to create personalized AR and VR shopping experiences. Harding shares, “Our goal is to take everything that’s good about online shopping: the recommendation engine, personalization, and data, and take that into the offline world. It’s still fun to go to the store with friends, a great social experience, but lacks a lot of the convenience for people used to online commerce.”
- Style: Use AR to test different hair and makeup styles before committing.
- Healthcare: Project a patient’s medical information while performing a surgery. Taskiran adds, “You could have doctors in the operating room looking at a patient with all of the X-rays and MRI data overlaid on top of them, helping the doctor make better decisions.”
- Public Spaces: Currently AR in public spaces is used to enhance existing art. AR could also replace signs and plaques describing historical events or buildings.In the future, the art could be entirely virtual. “What that will do is create much more open space in communities,” says Harding. “You can see things everywhere that are all personalized to you. You can have blank space in cities again, which is something that’s at a premium right now.”
- Remodeling: See what different layout and finish options look like in a space before deciding which to buy.
- Exercise and Fitness: Turn your workout into a game or see your heart rate and other stats in real-time without having to look down at your mobile device.
Other Kinds of Immersive Experiences
VR and AR aren’t the only immersive experiences. Indeed, there are plenty of immersive experiences happening in the world around you.
Interactive Plays: Tony n’ Tina’s Wedding which was first staged in the late 80s in New York, is a perfect example of an immersive experience. During the performance, audience members are the “guests” at the wedding and some are invited (or coerced) to dance or get involved in vignettes. They are also fed a meal as if they were actually at a wedding.
Movies: People interact with events on the screen, like The Rocky Horror Picture Show (which is an example of an immersive experience that evolved, rather than one that’s designed). Perhaps inspired by Rocky Horror, companies like BBQ Films create experiences around well-known movies and TV shows, such as Teenage Mutant Ninja Turtles, that include related activities for the audience and actors in costume interacting with them before a showing.
Escape Rooms: Participants are locked in a room and must figure out how to get out with tools and hints provided by the creators. These events started as simple video games that have since escaped into the real world.
Cruises: Some cruise lines, such as Disney, make the time guests spend on their ships into a full-on immersion in their world. Sonia Schechter explains, “The Disney cruise is an incredible example of immersion. It doesn’t use emerging technologies, but it does everything tech does. Immersion is about forgetting where you are and being able to replace your reality with something else. Disney is the top company at replacing reality. The whole thing is designed, from the music to the smells to the way things are named, to how people are trained. They really map this journey that keeps people completely immersed throughout their entire experience.”
Theme Parks: Disneyland may be considered the universal example of an immersive experience.
Corporate Events: Build employee morale by plunging them into a speakeasy, a spaceship, or any other environment you can imagine.
What It takes to Make an Experience Immersive
Since virtual reality is what most people think of when imagining immersive experiences, and it’s also the hardest to do, the information in this section applies mostly to VR. Remember, consumers of VR experiences need a fast internet connection, a VR headset, and a decent computer. Regardless of platform, immersive experiences tell a good story to engage the audience.
Harding states, “You need a good story. You need to engage people. You need to have some form of interactivity. And you need a reason to make something a VR experience. I’ve seen a lot of VR, like travel videos, where someone put a 360-degree camera next to the Eiffel Tower and you’re in Paris, but why are you in Paris?”
Give the audience something to do or a role to fill. Harding says, “One of the Fifty Shades of Grey movies developed a 360-degree experience of a masquerade party, which seems like a cool idea. Since no one acknowledged or interacted with you, the effect it gave was that you were the uncool kid at the junior high school dance.”
When setting out to create a VR immersive experience there are a few rules to follow. Be realistic (or get as close as the technology allows). Make sure that items that should be on the ground aren’t floating in the air. Customize elements that match the theme (don’t show the Millennium Falcon in a Star Trek game) to provide a sense of place that won’t pull the user back to the real world. Set clear expectations. Harding adds, “The worst thing you can do in VR is drop someone in a place and expect them to get it. That’s not something we’re used to and its also kind of boring.”
Additionally, it’s important to involve as many senses as possible. While taste, smell, and touch are still in nascent stages, you can use sight and sound effectively. Video quality and integrity must be high to pull this off. Pixel resolution, frame rate, color accuracy, contrast, and brightness all must be attended to during development. In the same vein, high audio quality matters too. The sample rate and precision of sound as well as clarity and surround sound contribute to the experience the designer is trying to create.
Keep the hardware and supporting tech for the experience hidden. The science fiction author and futurist Arthur C. Clarke is known for his three laws about the future. The best-known law is the third: “Any sufficiently advanced technology is indistinguishable from magic.” While an immersive experience isn’t magic (at least not in the literal sense), hiding the puppet strings as much as possible will add to the illusion.
Take an end-to-end approach when creating your experience by thinking holistically. With that in mind, don’t make users struggle. Schechter advises, “Make the experience really easy and intuitive to use. When things are difficult, challenging, or uncomfortable, it breaks the suspension of disbelief. Ensuring that the experience is comfortable and makes sense without a learning curve is important.”
In addition to technical considerations, using the right hardware, software, and content, creating a good experience requires teamwork. While a single person can do it, a team with varied skills will be a boon. Some people that can help:
- Data scientists
- Information architects
- Data analysts
User-Centric Design Leads to Immersion
Immersive experience designers, like all designers, need to keep users in mind. Clearly identify the audience who will be the primary users. Harding says, “For an older audience, it’s making sure that it’s easy to use and well-explained. A lot of the time people assume a high-level of comfort, and that limits your audience. Tech designers sometimes forget that.”
Benefits of an Immersive Experience Done Properly
Entertainment experiences have different goals than therapeutic experiences; shopping apps do different things than training apps. In the end, being involved in an immersive experience can change the audience physically, socially, and cognitively. This list of benefits is primarily concerned with VR.
- Emotional engagement and empathy
- Bonding with friends
- A sense of accomplishment
Training and Education:
- Behavioral changes
- Increased efficiency and precision
- New skills and capabilities or improvement of existing skills
- Increased engagement
- Improved decision making
- VR falls into the sweet spot Edgar Dale’s cone of learning. Therefore, retention will be better than most other learning techniques.
Therapy and Mediation:
- Behavioral changes and improved patient outcomes
- Reduction of PTSD
- Reduction of chronic pain
- Coping skills
- Improved decision making
What Not to Do with AR and VR
Like many technologies, VR and AR have both positive and negative facets. It’s up to society and government to monitor the misuse of a technology and take steps to stop it. Some ways these technologies can be abused:
- VR can be used as a torture device.
- Like the internet itself, VR will have trolls.
- AR can be used to present false information.
- As rendering engines become capable of creating more and more realistic scenes, what’s real and what’s not are becoming harder to distinguish.
- VR can be used to create fake memories.
The future of immersive experiences will follow the technology that enables them. The 2018 movie Ready Player One (based on the novel of the same name by Ernest Cline), shows a likely future for VR: A fully immersive world, where users interact in real time with each other (via avatars) and with digital objects and characters rendered with full sensory immersion (including touch, smell, and taste) via advanced technology (body suits that allow the body to act in the real world that cause reactions in the virtual world, and vice versa). How far off general availability of this scenario might be (both in terms of time and in accuracy) is anyone’s guess.
Improvements in tech will continue to make VR and AR more engaging. Here’s what we can expect to see evolve in the future:
- Haptic Feedback: A simple example of haptic feedback is when a phone or tablet vibrates or beeps in response to taps on the screen. A more advanced idea, still in development is a glove that allows a handshake in a virtual world to feel real. Getting feedback for actions that the user didn’t originate is even more advanced, but a jacket that can squeeze a user in response to a virtual hug, make them feel a punch to the gut, or simulate a snake crawling across their chest is a step in that direction. Schechter says “In five or ten years, you’ll use haptics to imitate what it feels like to sit in a chair or to better understand textures if you don’t have samples you can touch.”
- Human Augmentation and Brain/Computer Interfaces: Advancements will help enable faster reactions, more natural interfaces and interactions, and allow more user control.
- Lightfield Displays: Projected images in space in front of users.
- Retinal Displays: Images could project right onto a user’s retina.
- Neuromorphic Engineering: This technology involves computer systems that are based on the brain.
- Affective Computing: Allows a VR simulation to react to a user’s emotions to improve the experience.
- Eye Tracking and Improved Voice Recognition and Control: These options can make user interfaces more natural and responsive.
- Specialized Processors (e.g. system on a chip): Drive improvements in computing speed, the way that specialized graphic chips have improved PC game play.
- HDR Visuals: High Dynamic Range (HDR) will provide further improvements to visual quality.
- No More Apps: WebAR will no longer require a dedicated app to use AR. Instead, it will use a mobile device’s browser. This evolution allows you to send links to someone’s device, so they can see the digital layer without needing an app. This concept will expand the ease-of-use, the ubiquity, and the popularity of AR.
- Devices: Some sort of headset that allows both VR and AR may be the main way to view immersive experiences in the future. Harding explains, “It’s foolish to think that in five years we’ll all be wearing this big headset. We’ll probably have smart glasses that will have a setting where you can interact more or less with the real world. If you’re out running errands and you want AR capabilities, you set your glasses to one end of the spectrum and you do your errands wearing your glasses that have a digital layer like on your phone. At home, if you want to play a game, you put your glasses setting to the other side and it blocks everything else out.”Magic Leap is working on a headset that may be the first hardware to fill this space. Apple recently announced they are planning a headset with both AR and VR capabilities.
- Cost: VR headsets are becoming more affordable, which will help expand the audience. Oculus recently announced a new device with a list price under $200 (no smartphone required), the Google Daydream is available for under $100 (ditto), and there are a number of other headsets available for not much more. Schechter says, “It’ll get to the point where these things are very accessible, much easier to use, and not require so much capital output. To date it’s been risky to create VR content because the audience is very niche. But once there’s a broader audience, the content will get better, and applications will become more immersive.”
Immersive Experiences and the Future of Content
Many news outlets have been experimenting with immersive content. The New York Times has been publishing stories formatted for VR regularly for a few years, from short daily pieces to more in-depth stories. These stories can make some news feel more real and present, but the format is not ideal for every kind of story (yes, for seeing the result of an earthquake in a far off country, but not for an in-depth profile of a world leader or a stock market analysis). The cost to produce VR stories is high (in both time and money), so they haven’t become a large percentage of news. Until VR equipment becomes more accessible and the cost to produce them drops, they will remain a niche part of newsgathering.
The annual Tribeca Film Festival has some VR offerings; both rendered and filmed experiences that may be available in the near future are highlighted there.
Questions About Immersive Experiences
Who is Robert Scoble?
He’s a writer and tech executive, known for his work in AR and VR as well as other emerging tech fields. See what he had to say as a Futurecast guest.
What is heterogeneous computing?
A computing system that uses multiple types of processors is called heterogeneous. These systems are designed to handle specific tasks. In AR and VR, they can speed processing time and improve video and sound quality. The Qualcomm Snapdragon 820 is a chipset designed with VR in mind.
What are cognitive technologies?
A subset of AI, cognitive technologies mimic functions of the human brain to perform or supplement tasks that have traditionally required human intelligence, such as planning for the future, drawing conclusions from partial or uncertain information, and learning. These tools can be used in VR and AR.
What are virtual senses?
It’s experiencing things with your senses in VR, and to a lesser extent, in AR. Sight and sound have already been done, touch is next with advances in haptic technology. Taste and smell will take longer to figure out.
What is neuromorphic engineering?
Also originating from the AI field, neuromorphic engineering is the process of designing computer systems based on natural systems like brains. These systems will make reactions in VR worlds more natural and faster.
What are neural sensors?
These are devices that attach to someone’s head (or implanted) that can detect certain patterns generated by the brain. These are a key part of brain/computer interfaces.
What’s a digital twin?
Sometimes called digital shadows, a digital twin is a copy of a person, thing, or even process from the real world mirrored in a virtual world. The copies can be linked to their real-world counterpart and receive status updates and notifications of problems, or they can be a product prototype. A digital twin can also be disconnected and used for training.
What is sensory extension?
It’s a design paradigm that emphasizes computers as tools to augment or improve human senses rather than seeing them as just tools for teaching or education. In the same way that a microscope or telescope extend sight, or glasses to correct it, computers can do the same for sight and other senses.
How does VR impact memory retention?
It’s an established science that playing a video game helps memory (including those with cognitive disabilities). Combine that with how VR fits into Dale’s cone of learning, and VR has the possibility to generate even better results.