True Spatial Audio in Gaming, XR, and Beyond

Spatial Audio and the Magic Within

The immersive potential of multimedia content - spanning extended reality (XR), gaming, the social web, music, and cinema - is significantly amplified by the integration of spatial audio technology. Nowadays, there are many different terms, such as 3D audio, immersive audio, ambisonics, or spatial audio, for the same or a similar perceptual goal: mimicking the natural way humans perceive sound in the environment, creating an auditory experience that is as real as the visual stimuli it accompanies. This article will focus on how spatial audio impacts our experience with media. We'll explore how it affects things like our sense of space, emotions, well-being, and how stories are told. Then, we’ll discuss the use of spatial audio in gaming and introduce atmoky and the tools it provides for game engines and audio middleware.

Enhancing Spatial Awareness, Immersion, and Engagement

It makes intuitive sense that replicating a three-dimensional soundscape brings us closer to a real-life listening experience. But science backs this up with fascinating evidence. Research by Kobayashi, Ueno, & Ise (2015)1 delves into how spatialized sounds significantly enhance the feeling of "being there" in virtual environments, not just psychologically but also physiologically.

Their study used 3D sound reproduction to simulate sounds approaching the listener, comparing it to non-spatialized audio. The results are compelling:

• Deeper Psychological Presence: Participants reported a much stronger sense of presence with spatial audio, confirming that accurate sound placement heightens the immersive experience of virtual worlds.
• Physiological Response: Even more intriguing, spatialized sounds triggered greater activation of the sympathetic nervous system, similar to how someone invading your personal space in real life would. This suggests our bodies respond to meticulously crafted virtual sounds in ways akin to real-world interactions.
• Mind and Body in Sync: The study found a significant correlation between the psychological and physiological measurements, highlighting that our emotional and bodily responses to virtual environments are deeply intertwined.

Overall, this research illuminates the profound impact of sound quality on the realism and emotional engagement of virtual environments. Well-crafted auditory cues aren't just perceived but are felt by the body, enhancing the overall virtual experience.

Similar findings are echoed by Hendrix & Barfield (1996)2, who explored how the presence or absence of spatialized sound affects user experience in virtual spaces. Unsurprisingly, they found that spatial audio significantly elevates the sense of presence compared to flat audio. But even more interestingly, spatial audio fundamentally changes how users interact with virtual environments. Participants felt sounds emanated from specific locations, making the experience not only more realistic but also increasing their engagement with the virtual world – a crucial aspect for platform builders.

Improved Concentration and Wellbeing

Beyond the immersive thrills, spatial audio offers surprising benefits for our mental well-being. Studies have shown it can boost our concentration and reduce stress.

Imagine trying to pick out a specific voice in a crowded room. With regular audio, it can be a struggle. However, research by Kilgore & Chignell (2005)3 found that spatial audio makes identifying voices significantly easier, suggesting improved focus and reduced mental strain. It's like our brains have less work to do when sounds are presented in a natural, 3D environment.

The benefits extend beyond just everyday tasks. A study by Greenberg et al. (2021)4 explored the use of spatial audio in reducing stress and negative moods. The results were particularly promising for individuals with anxiety or depression, highlighting the potential of spatial audio as a therapeutic tool.

Furthermore, research by Hale et al. (2005)5 demonstrated that training with spatial audio cues in virtual environments led to decreased mental demand and enhanced concentration. This suggests that incorporating spatial audio into educational and training contexts can lead to more effective learning experiences with less mental strain.

So, the next time you experience the wonders of spatial audio, remember that it's not just about creating a cool soundscape – it's about making your mind sharper and calmer in the process. Platforms that atmoky is proud to work within that context are VRelax, YourHaven, and Portal.

Spatial Audio in Gaming

Game sound design has long held a significant role in storytelling, shaping emotions and driving narratives forward. Spatial audio doesn't tell the stories – but, it makes us feel them. Placing sounds within the virtual space creates a powerful emotional connection that traditional audio simply cannot match. This is why spatial audio is especially a driver in gaming and XR.

Acoustic Ecology in Games: Grimshaw & Schott (2007)6 delve into the concept of "acoustic ecology" in first-person shooter games, arguing that sound is not merely an add-on, but an integral part of the gameplay experience. The way sounds are positioned and interact within the virtual world significantly influences player awareness, decision-making, and overall immersion. This highlights how sophisticated sound design in games goes beyond creating atmosphere; it actively shapes how players interact with the world and ultimately elevates the entire gameplay experience.

However, crafting immersive soundscapes within game engines presents challenges beyond the traditional Digital Audio Workstations (DAWs) used for creating and editing audio assets.

One key hurdle is the limited availability of dedicated spatial audio plugins for game engines like Unity and Unreal Engine. These engines, while powerful for game development, often lack robust native spatial audio capabilities. Additionally, game environments demand real-time audio processing, as opposed to the pre-rendered audio editing typical in DAWs. High-performance real-time audio processing is crucial for key elements in game sound design like dynamic sound object positioning, dynamic reverb, and soundscape rendering (e.g. ambisonics or multi-channel content), which need to constantly adapt to player movement and head-tracking.

The game development landscape features two major engines: Unity and Unreal Engine. While Unreal Engine offers some built-in spatial audio tools (MetaSounds), the native capabilities and the channel width are even more limited within Unity, where you have only a few native immersive audio features. For projects with demanding audio requirements, dedicated audio middleware solutions like FMOD and Wwise are often employed. These offer extensive audio manipulation tools and integrate seamlessly with major game engines. But even there, the solutions for high-quality and intuitive spatial audio processing are limited. Open-source solutions for spatial audio also exist, but they often lack the precision and control required for professional game development. Also, most available libraries are deprecated, are not cross-platform compatible (i.e. do not sound the same on all platforms), and they don’t offer support or the integration of advanced features. atmoky addresses this gap with its atmoky trueSpatial plugins. They seamlessly integrate with major game engines and middlewares like Unity, FMOD, and Wwise, empowering developers to leverage advanced spatial audio capabilities within their projects.

atmoky trueSpatial: Bridging the Gap for High-Quality Spatial Audio

Recognizing the limitations of native audio tools in game engines and the complexities of audio middleware, atmoky developed trueSpatial to provide a powerful and accessible solution for creating immersive audio experiences. Here's a summary of the capabilities of the plugin bundle:

• Binaural Rendering and Perception Modelling: At the core of trueSpatial lies an advanced binaural rendering engine that incorporates a perception model. This ensures minimal sound coloration and maximizes the immersive experience, even for complex ambisonic scenes.
• Externalization Booster: When using headphones, the lack of simulated room acoustics often leads to sounds appearing inside the head, diminishing the sense of externalization. trueSpatial's externalizer addresses this by providing a convincing out-of-head experience.
• Signal Processing and Efficiency: trueSpatial features a highly optimized convolution engine that allows for real-time adjustments without compromising on performance, even in scenes with numerous audio sources. Even if audio processing has a footprint nowhere near the visuals, it is a plus to offer high-performing solutions.
• Advanced Audio Object Manipulation: The plugin offers sophisticated control over audio objects' directivity patterns and occlusion effects, mimicking real-world sound behavior and enhancing the believability of virtual spaces.

• Near-Field Effects: When sound sources are close to the listener, specific near-field effects come into play. These effects influence how we perceive distance and create a sense of intimacy. Studies by Spagnol et al. (2017) and Hyodo et al. (2021) highlight the importance of accurately recreating these effects for improved spatial accuracy and deeper emotional impact.

Games already building with atmoky are:

• Breachers 
• VailVR
• TinHearts

For a full overview of the features and how to use the plugin go here: atmoky trueSpatial for Unity

How this can sound? Here is a demo of the functionalities:

Cross-Platform and More

Seamless Experiences Across Devices: The Power of atmoky

At atmoky, we understand the importance of delivering consistent, high-quality audio experiences. That's why our trueSpatial engine is built to sound the same on every platform it supports! This means sound designers can create immersive soundscapes with the confidence that users will hear them exactly as intended, regardless of whether they're playing on a PC, console, or mobile device.

Beyond Gaming: True Spatial Audio for the Web and Social Interactions

Our commitment to cross-platform compatibility extends beyond the world of gaming and XR. We've successfully brought spatial audio experiences to the web, opening up exciting possibilities for:

• Real-time voice communication: Imagine attending virtual meetings or social gatherings where voices sound natural and spatially realistic, enhancing communication and connection. See atmoky Engage for more information and feel free to try out our demo app meet.atmoky.com

  Video URL

  atmoky showroom on arrival.space

• Immersive web experiences: Web-based concerts, virtual tours, and interactive installations can now leverage the power of spatial audio to create truly captivating experiences. See Sonic Meds - Enchanted Forest by Jelmer Althuis 

  Video URL

  atmoky trueSpatial for web - enchanted forest

We are also proud to be integrated into an innovative social web platform, where we’ve also built our showroom as well.

This cross-platform flexibility empowers creators to deliver innovative and immersive experiences across a wider range of platforms and use cases.

atmoky and Our Journey

Before we started atmoky, we’d been at the IEM in Graz and researched advanced spatial audio technology. We’ve also built the IEM plugins, which are now among the most used tools for producing and working with spatial audio. Based on that knowledge and expertise we set out to take it to the next level and to offer a flexible and more versatile alternative to the renowned solutions that push from the entertainment and loudspeaker-base rendering. 

• 1

  Kobayashi, Ueno, & Ise (2015): Kobayashi, M., Ueno, K., & Ise, S. (2015). The Effects of Spatialized Sounds on the Sense of Presence in Auditory Virtual Environments: A Psychological and Physiological Study. PRESENCE: Teleoperators and Virtual Environments, 24, 163-174. https://doi.org/10.1162/PRES_a_00226.

• 2

  Hendrix, C., & Barfield, W. (1996). The Sense of Presence within Auditory Virtual Environments. Presence: Teleoperators & Virtual Environments, 5, 290-301. https://doi.org/10.1162/pres.1996.5.3.290. 

• 3

  Kilgore, R., & Chignell, M. (2005). Simple Visualizations Enhance Speaker Identification when Listening to Spatialized Voices. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49, 615 - 618. https://doi.org/10.1177/154193120504900403. 

• 4

  Greenberg, D., Bodner, E., Shrira, A., & Fricke, K. (2021). Decreasing Stress Through a Spatial Audio and Immersive 3D Environment: A Pilot Study With Implications for Clinical and Medical Settings. Music & Science, 4. https://doi.org/10.1177/2059204321993992. 

• 5

  Hale, K., Stanney, K., & Malone, L. (2009). Enhancing virtual environment spatial awareness training and transfer through tactile and vestibular cues. Ergonomics, 52, 187 - 203. https://doi.org/10.1080/00140130802376000. 

• 6

  Grimshaw, M., & Schott, G. (2007). Situating gaming as a sonic experience: the acoustic ecology of first-person shooters. https://wlv.openrepository.com/bitstream/handle/2436/35996/Grimshaw_DiGRA2007.pdf?sequence=1