Audio Design in XR: Practical Advice, Common Mistakes, and Techniques

With such a rapid changing industry, it’s hard to follow what’s going on, or whats the right way of doing things. Often times creators fall into the trap of focusing on the newest coolest technology instead of focusing on the art itself. Over the years I learned that the important thing is to remember what you’re trying to convey as an artist, and pick the technology based on that. So, even though the tech is changing every year- the creative work stays the same.

Defining Spatial Audio

There have been so many ways to describe spatial audio, everyone you meet will have their own! When Apple announced that they support spatial audio, it was a very different product from what Dolby offered with their Atmos system. Between game devs using spatial audio for directionality in video games, to musicians using spatial audio to calibrate a big speaker array - it was hard to define what spatial audio is. It became one of those loaded terms that everyone use as a buzz word to enhance their product. 

At the core essence of it - spatial audio is a technology that provides the illusion that sound is coming from one thing (an object in the room, a character on the screen), while actually coming the 2 headphones on your ears, or the speakers in the room.

Spatial audio creates a 3 dimensional, multidirectional audio image that, if done right - immerse the listener in an experience.

Spatial Music

This is the one category that every musician is trying to get into when the first hear of spatial audio. How to take their music sound from different places in the room to create a spatial, dolby atmos mix. I’ve seen everyone from indie artists to recording experts and everything in between trying to apply spatialization on their music mixes. Here are the goods and the bads : 

When working with contemporary music, with full DAW productions - listeners don’t really like a spatial mix. When Apple introduced ‘spatial music’ for the first time - it was a cool feature, but most users intentionally looked for how to turn it off, and maybe rightfully so. The genre of contemporary music was designed to be heard with 2 good headphones, or maybe in a loud party, where the sound goes through your entire body with loud speakers pushing sound waves through the space. When we hear a new song, the last thing we want to think about is what direction is each stem coming from, especially when that music is electronic. We want to be immersed in the song, and not look for the direction of the drums or the guitar. Spatial audio can be a curse when used incorrectly, and in those types of productions I would say it’s making more harm than good. The only time that spatial mixes work well on well produced music, is when artists take known experiences and put them in a new context, usually with the support of a big company. A good example for it would be Dolby studios, who did a mix for Elton John’s Rocket man in spatial audio. The audience is familiar with the song, and therefore - the experience of feeling in outer space, listening to something familiar - actually worked very well. 

The other type of musical experience that works well in spatial setting, is choirs and classical ensembles. There are few reason why it works well. Classical music was always performed live, and is written in a way that incorporate the space in the actual piece. Classical composers had to think about the type of space the piece would be performed in, and use the reflections of sound as part of the piece. The style of the music took into consideration where people would be standing or sitting while listening to it, and where every musician would sit. Another reason why choirs and a-capella music works so well, is because our brains can easily detect human voice in space, so placing singers in different sides creates a great effect that actually comes through in spatial audio! 

Bandwidth Limitation

With new tech, comes new limitations. Unfortunately, most XR headsets has limited abilities when it comes to audio, and the speaker response is not great (to put it lightly). To get the full range of audio, some creators choose to author their experiences using headphones, but for those who are publishing on the various app stores or streaming channels - there is no way to control how the users will listen to the experience. This is a known issue nowadays, not just in XR. 

In the past, musicians could count of the fact that at least most music fans would get a decent audio system to listen to their music at home. Nowadays audio producers have to account for audio quality that can range from thousand dollar speakers to a pair of cheap headphones and everything in between, plus - each streaming service has their own compression rate, so there’s basically no way of knowing how a song would sound like on the other end. With XR, the problem is less wide compare to traditional music, but it still exist. Most XR headset rely on very simple speakers that barely cover the ears. When looking at AR devices - the speakers don’t cover the ear at all, to allow open-ear experience. The distance that the sound has to travel between the speaker and the listener’s ear is already causing a significant loss of frequencies. Add to that the size and quality of the speakers (depending on the headset) and the bandwidth that musicians are left with is really limited. The good this about XR, is that if the experience is offered mainly on 1 headset (lets say, Quest) - you can design an experience knowing that your bandwidth is limited, but create one mix that would actually sound pretty good with the existing speakers. The rule of thumb is - always test your sound on the headset, never trust the computer monitors.

Sound Design

When it comes to sound design, it can be tricky in XR. Mostly because the sound design needs to match the aesthetics of the experience. XR devices are usually small and portable, that means that they have limited CPU and GPU power, so most experiences are not going to be high end experience in terms of graphics. The sound has to follow the same artistic idea as the visuals, and in most cases, creators have to choose what’s important in the experience, and where to put their efforts in, plus - follow with the overall aesthetic of the visual design.

When using synthesizers, the sounds can be harsh, on unpleasant. When I started working in XR, I thought the solution might be to work with mostly sine waves. They always sound nice and pleasant to the ears. I learned really quickly that sine waves do not spatialize well. In fact - they don’t spatialize at all. A sine wave doesn’t have enough info to carry through and change based on the parallax of the ears, so as a result - it’s spatial-less. 

To design a sound that stands out in XR, the best things to do is to layer the sound with different textures, to create an in-depth sound. It makes it easier to spatialize it, and masks nicely the bandwidth limitations.

Haptics

Haptics is one of the secret tools of the XR audio creator. Haptics are always considered a separate category from audio, but when you think about it - sound is the movement of air. If you want to be more precise - it’s the vibration of air. Haptics - is vibration. The vibrations are usually small enough that we do not perceive them as audible tone, but it’s definitely in the same category with sound. It’s even defines by the same characteristics of sound. Amplitude, frequency and duration. Haptics can be used to enhance interactions in XR by pairing sound with feel.

With such a rapid changing industry, there are lots of opportunity for creativity and it’s easy to get lost. Technology was never suppose to get in the way of creativity, it was suppose to be a tool, something that allows us to create new things but instead - we are caught up on terminology. Remember to use the medium for what it’s good for, and make it really shine. Maybe in the future we will see a new genre of music emerge, a genre that conveys the modern styles of audio production and the new technology for immersive computing.