Published: May 10, 2019
Updated: Oct 15, 2019

Designing for Mixed Reality: How to Think Differently

Author: Ed Schmit

Two people looking at a virtual city model with a smartphone

Developing products for mixed reality (MR) poses interesting challenges for designers. As the disruptors, they’ll have to blaze a trail for the rest of the industry to follow. Read more about how thinking differently about MR can help.

Mixed reality (MR) is going to be a big part of our future lives, both at home and in the office. Companies are expanding their reality development focus from video games to MR experiences in healthcare, architecture, education, and business.

Ed SchmitAuthor: Ed Schmit, AVP Product Marketing Management, AT&T Developer Program

Ed tracks new technologies for the AT&T Developer Program. His specialties include network technologies, technology enablement, and strategic marketing.

Because of its myriad applications, designing for MR environments and applications poses interesting challenges for designers and business people. Lillian Xiao, a UX designer based in San Francisco, talked about the challenges in a post for The Startup. “(In) MR, the blend of physical and digital presents a unique set of design challenges, and best practices are yet to be discovered,” she wrote.

Everyone from the designers, subject matter experts, business sponsors, users, and consumers will have a say in how MR is shaped because it will impact them in different ways. As the early adopters, it currently falls on developers’ shoulders to create and manage a set of best practices for MR development, deployment, and maintenance.

How can developers keep everyone’s needs and objectives in mind as they move forward with this innovative and disruptive technology?

How Design Evolved from 2D to Mixed Reality

Broadly speaking, “design” started in the physical world. Whether we’re talking about graphic design, product design, computer design, or UX design, it all started with people, pens (or pencils) and paper. Everyone could contribute since, at its most basic level, design could be done with pen + paper or marker + whiteboard.

Ideas were shared and prototypes were created, and along the way, workflows were designed and design best practices codified. Everyone agreed with these practices and followed them.

Enter the computer and the world of computer-aided design (CAD). Initially used to assist designers (architects and engineers, mainly) with electronic drafting, CAD software eventually led to automated everything: inventory lists, auto layout of integrated circuits, interference checking, and most importantly, a myriad of other calculations that were still done by hand. Using CAD software, designers and their employers saved time, effort, and money, so they could push the boundaries of their designs.

To keep pace with this rate of design change, the makeup of the design team evolved as well. Product managers started working alongside business leaders and pure designers to guide products through their lifecycle from conception to delivery. They helped tie product concepts to business objectives, ensuring designers thought beyond the aesthetics and inner workings of their products. Designers were gaining an unprecedented perspective of how their work would impact the world outside their office.

The combination of the evolution of technology and the broader view of how digital products could be used lead designers to push even further. Virtual reality (VR), augmented reality (AR), and now MR experiences are rolling out to the consumer faster than before, as it becomes less expensive to create and experience. Consumers are now able to experience products that were seen as science-fiction just a decade ago, leading designers to continue to push the boundaries of MR design.

The way designers respond to consumer demands and how they adapt their design practices will dictate just how well MR is adopted by the world at large.

The Traditional Design Process

Design teams have been using traditional processes to work through their projects, from ideation to prototyping and execution. Teams work through a problem, share and evaluate ideas, and ultimately, reach a solution that meets customer needs and business objectives.

Today, there are well-established design and development methods for building experiences on laptops, desktops, smartphones, and tablets. Design and product teams have clear expectations of what’s necessary to iterate an idea from start to finish and release a product for customers.

Design teams are often a mix of development, design, research, sales, marketing, and management, and all roles are welcome to participate in the design process. People only need to understand a rough framework of the final product to contribute an idea and establish technical requirements or define potential user behavior.

The Challenges of Traditional Design for Mixed Reality

When it comes to designing for mixed reality, however, it’s more complicated.

  • 2D design tools often limit the ideas MR demands: Traditional 2D tools like pen + paper cannot handle the 3D ideas MR requires since not everyone can draw well enough to explain ideas fully.
  • 3D design tools are not built for general product iteration: The 3D design tools currently available are mainly built for gaming and engineering markets since they were the early adopters to technology-based product design. They are not ready to deal with the business objectives and workflows outside of these markets.
  • 3D design tools require a high degree of training and proficiency: Fleshing out even the most basic product design in a 3D design tool requires a lot of training and proficiency.
  • 3D design tools can be expensive: The design software itself may be expensive to purchase as well, increasing the cost to 3D product design. This is why many companies buy few licenses for the software and train so few people to use it.
  • Collaboration is stalled by design bottlenecks: “We’re just waiting for the designer to finish their prototype” is a common refrain in MR development teams. They want to work together to create their next product, and they may even be used to doing so. However, when specialized skills or tools are isolated with certain roles, it becomes harder to collaborate.
  • Designers are overloaded: Because of the dependency on designers familiar with 3D tools, they’re overloaded with work. Teams rely on designers to solve the technical implementation of the product and create a foundation for the user experience as they attempt to flesh out the team’s ideas in code. The design becomes less of a collaborative process and relies heavily on the designer and their tool to complete.
  • MR product timelines stretch out too long: Low-tech and 2D design processes worked well for design teams because it removed any barriers to design from the entire workflow. The lack of barriers to ideas made it easier to iterate through the workflow. MR products, however, add complexity to each stage, thereby lengthening the entire process. Companies are not able to capitalize on consumer trends and deliver products that end up failing because consumers have already moved on to something else.

The very characteristics of MR that make it an intriguing and disruptive experience are increasing the barriers to entry not only from an end-user perspective, but also a product development one. What can teams do to optimize the design process for mixed reality, so they can deliver MR products at a faster rate than they have been?

Start Thinking Differently about the MR Design Process

To get back to the collaborative, barrier-free nature of the traditional design process, MR design teams must create a hybrid design process. That means retaining the traditional design process and then adding on the 3D techniques and activities the team needs to use to embrace designing for mixed reality. This way, all members of the team, regardless of role and discipline, can be part of the design process, discovering opportunities, identifying potential obstacles, and making significant contributions towards the product design — just like they used to.

How 3D Techniques can Enhance your Design Process for MR

Design teams can add many different techniques to their MR design process, including bodystorming, acting and physical feedback, and storyboarding.


The entertainment industry has a long history of storyboarding, where scenes are sketched out on large boards independent of the main script or story. Directors and writers use them to arrange the flow of the story and help plan their shooting schedules at a lower level. Production staff use them to plan and create the aesthetic look and feel of the scenes.

  • High-level storyboarding: By experiencing the design from a high level (as if you were the camera shooting the scene), teams can better describe the big picture components of the experience you’re creating. E.g. The user walks into the office and sees [object] sitting on their desk.
  • Low-level storyboarding: By experiencing the design from a lower level (as if you were the user), teams get a better feel for that first-person experience. Teams will have a better understanding of how their designs will interact with the physical space and how users may interact with the digital aspects of the experience.


Bodystorming lets designers get out of their heads and into the experience they’re designing. It’s the 3D equivalent of sketching out an idea on a napkin to see if it would work. By quickly setting up the experience with props at hand, MR design teams can immediately see if an idea is worth pursuing or not. Detailed designs or coding isn’t necessary with bodystorming since it’s only being used to vet ideas as viable or not. Bodystorming takes the design experience out of the hands of the design tool experts and puts it back into the entire team’s hands. The deep dive into the technical and user requirements will be done later. 


Acting takes bodystorming to the next level, by having the design team mime interactions in a real-world environment set-up to mimic the MR one. We all do a version of this when we’re telling a story or explaining something: using our hands, bodies, and sometimes even props, to fully flesh out the story to our listeners.

By physically acting out what they want users to do in the experience, using props, motions, and other people, designers will gain a deeper understanding of their idea. They’ll identify obstacles and problems they’ll need to address in the prototyping phase without the added cost of time and effort. It also helps designers gain empathy for users because they’ll be going through a basic version of the experience.

Skills Beyond Design for Mixed Reality

Adding these new techniques to the MR design process helps the entire team and prepares them for future product evolution. Ideas and work are shared by everyone, instead of being isolated with particular roles, disciplines, and skills.

Encourage more ideation at the start of the design process

Because of their proficiency and skill with design tools like Unity and Unreal Engine, designers can contribute their unique perspective at the start of the process. They may identify new solutions to problems or uncover opportunities for future enhancements that can be explored later.

Deepen collaboration between technical and design roles

Design teams may have deep-seated roles and responsibilities, however, these new design techniques will break down communication and work silos, so everyone works more collaboratively. Team members will gain a deeper understanding of each member’s role and skills and make the entire workflow more efficient. Information will flow back and forth between the teams more easily, instead of in one direction.

Develop the design team’s skillset beyond the usual

A more collaborative design team that uses these new techniques encourages members to move out of their comfort zone and take on tasks they’re not used to. Like having a UX designer lead a bodystorming session or having a business leader act out an MR experience. Each person will bring their own perspective to the task, leading to the discovery of future enhancements or an ingenious solution to a problem that stumped the rest of the team. Innovation and creativity are spawned by this time of boundary-pushing and could lead to future MR product innovation as well.

Shorten MR development timelines

When there is more efficient collaboration and communication within the MR design team, problems are identified and solved earlier in the process. Which means the entire design process is shortened. Stakeholders can raise questions and offer help to other team members more easily, removing bottlenecks and allowing the entire team to work more efficiently.

Looking Ahead to the Future of MR Design

As mixed reality gains a deeper foothold in our technology-savvy world and the technology that powers it becomes cheaper, the possibilities genuinely become endless.

  • Business leaders will push for MR in data and cloud services, taking advantage of natural language processing and real-time computer vision to monitor and maintain the mountains of data our devices produce each day.
  • This will lead MR designers further into the world of business, medicine, and architecture as these industries are currently underserviced, but could also gain from the benefits enjoyed by the gaming industry.
  • Star Trek‘s holodeck experience may be closer to us than ever before as MR designers take on more social scenarios for their products, allowing consumers to interact with digital products in a more immersive experience. More companies are jumping into the holographic technology market, leading to competition and technological advances.
  • Designing environments will become more immersive as well, as the teams use their own technology and ideas to make their work more efficient and interactive, moving beyond the traditional interfaces we see today (keyboards, cameras, and projection goggles).

MR designers are paving the way for this industry, developing new workflows, processes, and best practices as they forge the way. With the exciting possibilities MR has for everyone, they’ve got ample incentive to get it right. The future of mixed reality design looks quite bright (and won’t require shades).

Share this post

by good on 05.22.2019 02:38 PM


by HOOD on 05.22.2019 03:16 PM

Nice info!!!

by HOOD on 05.22.2019 03:25 PM

Nice info!!!!

by HOOD on 05.22.2019 03:28 PM

nice post

by HOOD on 05.22.2019 03:29 PM

amazingly awesome

by Charlie on 06.11.2019 10:45 AM

Looks great!

by Deepak Velayudhan on 06.24.2019 03:16 AM

Now Mixed Reality has entered the normal lives of every people. It has now got a major influence on the lives of teens as they are more revolving around with technology.

Leave a Comment:
Sign in is required to comment on this blog. Not a member? Join now.