If extended reality (XR) offerings – including virtual reality (VR), augmented reality (VR) and mixed reality (MR) – are to realize their full market potential, the organizations participating in this complex sector would benefit to double down on efforts to establish industry-wide open technical specifications and business conventions to integrate operations across the entire ecosystem.

As it stands today, the outlook for this market is already bright. According to new research by global technology intelligence firm ABI Research, nearly 40 million AR smart glasses will ship by 2027. The total augmented and mixed reality market will surpass $220 billion in the same year.

Meanwhile, consumer spending on the VR segment alone is expected to reach a staggering $20 billion in 2027, according to Omdia’s Consumer VR Headset and Content Revenue Forecast.

Realizing – and perhaps surpassing – these forecasts, however, will require a higher level of coordination and collaboration than is taking place today, says Dhananjay (DJ) Lal Senior Director, of advanced research and development for media intellectual property and technologies with Adeia, in an interview for journalists.

Here is what he had to say:

Q: What is the current state of the XR ecosystem?

DJ Lal: Extended reality, or XR, is an umbrella term for computer-generated mediums that can create virtual environments and integrate them with our physical world to create new user experiences. It is the union of multiple technologies such as 3D gaming engines, volumetric and light field capture, low latency transmission, edge computing, virtual reality, augmented reality, holographic displays and more.

XR has been used in different ways over the past decade as software and technology companies developed their own devices, platforms and apps to create or enable immersive experiences for gaming, media consumption and general-purpose or work-related applications. It typically involves the integration of devices, which can range from extremely sophisticated headsets and consoles to smartphones and commercially available eyewear.

The market today, however, is highly fragmented. Companies in this sector are inclined to focus on specific ecosystem segments. For example, manufacturers may improve computing performance to maximize user experiences with their consoles. Other companies may focus on integrating apps and platforms to deliver interactive content on their platforms. Still others have focused on the role of networks in providing the enabling connectivity that makes it possible to interact with cloud-based services and engage with other remote users.

Few have focused on creating an open value chain that opens the door for organizations – and users – to participate in the content capture, distribution and consumption of XR experiences.

As a result, the market is composed of islands of activity that cannot integrate with each other. This has served to limit the potential of XR. It creates a zero-sum game that limits the growth of the market. Unless consumers have significant disposable income, they are forced to choose from a short list of proprietary offerings. These hardware purchases often lead to software platform lock-in, resulting in limited choices for content consumption. It prevents the broader market from having cost-effective access to the vast array of available innovations and offerings.

This creates friction and increases frustration for consumers. Having the best device is wonderful, but if abundant content is not available for users to access easily, the market will have difficulty getting consumers to participate.

Q: What can the industry do to address this fractured environment?

Lal: All key players must come together for XR experiences to flourish. Device makers, content creators, audio and video compression technologists, 5G and broadband network operators, streaming service providers, edge computing manufacturers – even players active in AI and machine learning – must coexist, coordinate and collaborate on standards-based principles to enable the best end-user experiences.

The good news is that this has begun to happen.

There is growing momentum around initiatives like OpenXR, a royalty-free standards community for VR and AR platforms for developing content that is agnostic of the underlying target device. It is advocated by a working group managed by the Khronos Group consortium. Formats like USD (Universal Scene Description) and ITMF (Immersive Technology Media Format) have also been gaining acceptance for describing 3D scene graphs in the recent past. Finally, there is the Metaverse Standards Forum, set up in 2022, that aims to build an open, interoperable Metaverse by coordinating requirements and resources to foster standards evolution.

A growing number of industry leaders in the segment support these initiatives, as they advocate for coordination and collaboration to bridge gaps that enable cross-platform communication and connection.

It is important to acknowledge that XR is still in its infancy, and therefore subject to incremental and potentially disruptive innovations. The need of the hour is not to specify mature standards, but to foster collaboration among organizations that are working on similar problems. This entails specifying open interfaces, even as organizations compete for technical superiority. Sometimes, this may mean open-sourcing software with basic capabilities to the community of practice, while reserving a more advanced version of the software in the proprietary domain.

Q: What challenges do organizations in the XR experience community face when capturing, creating and distributing a cohesive experience for end users?

Lal: The most challenging aspect is getting collaborative interaction between key players in each stage of the content lifecycle, starting with content creation and capture and then moving through compression, distribution and consumption.

It is much easier said than done.

In XR, the process often involves synthesizing content using a 3D graphics engine or capturing content using volumetric or light field cameras. Some of the most complicated challenges revolve around creating or capturing photo-realistic 3D content, such as ray-tracing image synthesis or light field capture. This is because it involves a massive amount of computation and data compared to traditional media production. Compression of such formats requires significant consideration – and coordination – of the optimal settings and codec parameters, if not entirely new codecs. Many technologies are involved in just this portion of the journey.

Ensuring that the final content is compatible with the technologies of the next stage – distribution – requires broadband capacity at the wide-area network (WAN), local-area network (LAN) and personal connectivity levels.

Fortunately, there have been significant advances in communication networks over the past few years. By leveraging multi-gigabit fiber and DOCSIS networks, 5G fixed and mobile networks, the latest generation Wi-Fi – such as Wi-Fi 6/6E/7 – and edge computing technologies across the XR community, the industry can ensure the delivery of high-quality experiences for latency-sensitive applications.

The final step – in this oversimplified example – involves how consumer devices render immersive visual representations. The community must work toward developing negotiation protocols for parameters such as bandwidth, latency and compute such that the content delivered is matched to the capabilities of the display device.

As we envision and prototype XR applications, such as virtual social worlds with large-scale active participation, we must tackle another challenge: how to create communities and tribes that offer value for individuals and enterprises. This requires the ecosystem to address a range of technological challenges, such as server allocation of new users entering virtual worlds, as well as linking the real world with the virtual world through the creation of Digital Twins or Real-World Assets as NFTs (RWANFTs). It also encompasses platform enhancements for creating marketplaces that are far more immersive than today’s ecommerce websites.

Q: What behaviors need to change for companies to operate in ways that create a more cohesive experience?

Lal: A broad goal of creating seamless experiences across platform transitions is paramount for the whole ecosystem. If the headset of company X cannot render an XR experience created by company Y, consumers will be forced to purchase many devices instead of one. Imagine buying a different TV for each streaming channel or owning multiple mobile phones to perform separate tasks – such as texting, calling, or watching video content.

Today’s dynamics for TV must happen in XR markets. Getting there requires open technologies and standards.

Key players must be acknowledged and credited for sharing their perspectives on how to shape the ecosystem for great customer experiences. The entire value chain must commit to collaborative development where every player moves in the same direction avoiding metaphorical “walled garden” approaches. And this must be done pragmatically, such that innovation and technological competitive advantages are not stifled. The key lies in advocacy groups working together and ensuring that customer experience remains in focus, even as organizations protect and monetize their innovations.

Q: What are the economic consequences or implications?

Lal: If collaboration is successful across the value chain – from content creation, distribution, and consumption – the entire ecosystem will be a much more stable environment, spurring growth and increased revenue streams.

Open interfaces limit the costs of developing solutions while improving time to market. As a result, entry price points for consumer devices are lowered, expanding the addressable market by making XR services more affordable, dramatically improving adoption rates.

Finally, collaboration provides the impetus for the industry to harvest value from applications by improving existing content and trialing new experiences – such as interactions in social worlds, or world-scale visual positioning and augmented reality, or 3D product marketplaces, and much more.

Q: How does Adeia fit into this ecosystem?

Lal: Adeia has over 30 years of experience creating innovative solutions for all aspects of the entertainment experience, including discovery, search, recommendations, DVR, VOD, OTT, multi-screen, personalization, data analytics, advertising, imaging, content storage and high-performance computing.

We are uniquely positioned to provide solutions to companies across all sectors of the XR ecosystem.

We’re excited to share our expertise in areas that include data storage and handling, compression, transmission, high-performance computing, imaging, rendering and machine learning with entities in the immersive media value-chain as we continue to help content creators, content distributors, device manufacturers – and other technical solution providers – deliver the most immersive, high quality XR experiences possible.

Read more about immersive experiences in our white paper, Immersive Media: Bringing Science Fiction to Commercial Reality.

‍