Relevance of 3D audio for the immersive live experience

Welcome to the world of spatial audio; 3D audio; or immersive audio. I, Martin Rieger from VRTonung, will take you on our acoustic journey as a 3D audio producer. If possible, without getting hung up on the theoretical concepts of surround sound. Rather, today we're talking about the actual meaning of 3D audio in a practical sense.

But although 3D audio is still relatively new, there are already many myths surrounding this technology. So let's start with a simple explanation and work our way up to the practical applications in the context of live experiences.

What is 3D audio?

The concept behind spatial audio is simple, yet revolutionary. It describes a playback method that makes it possible to make sound perceptible not only from the front, left, and right (as with stereo) or from behind (as with surround sound), but also from above or below (3D sound).

But why is this considered a game-changer? We humans naturally hear in three dimensions. We automatically locate the source of the sound when someone is standing behind us and speaking, without having to see that person. The technical possibility of artificially recreating this natural sound impression opens up completely new audiovisual experiences.

In the context of spatial audio, the term "immersive audio" is often used. This refers to an enveloping, natural sound that allows us to immerse ourselves in digital reality and forget the physical world around us - similar to the immersion offered by VR goggles with visual content, but in terms of sound.

Ideally, all the senses should be involved in this immersive experience. Although image and sound are enough to captivate us for most media, it turns out that even stereo or mono can be immersive if they are implemented well. At a DJ performance in a disco, for example, people get carried away by the music, even if it has no more than two audio channels. But it can be even more immersive, but more on that later.

That's why I prefer the term "3D audio", as it evokes the idea of using a technology that allows sounds to be experienced from all directions.

How can you listen to 3D audio?

Let's not get confused by all the 3D audio formats such as Dolby Atmos, MPEG-H, etc. But we do need certain devices to be able to enjoy 3D sound at home or events.

• many speakers around us
• normal headphones

Let's take a brief look at the feeders in detail. Spoiler: Many artists and sound people are currently throwing themselves into the subject of loudspeakers and surround systems. However, I find two-channel music playback, which can create a spatial listening impression even with standard headphones, much more exciting.

Of course, there are other possibilities, such as headphones with head tracking or VR glasses. These devices appear in my 3D audio matrix and are often associated with 3 or 6 degrees of freedom. But let's concentrate on the two "simplest" devices for 3D sound.

Immersive speaker systems

The good old method, but with a futuristic twist. Imagine sitting in your living room, surrounded by speakers that are so cleverly placed that the sound comes not only from the front, back, left, and right, but also from above and below! Yes, this means a bit of effort in terms of setup, but the result is worth it. You'll hear planes flying over your head as if you were right under the flight path - without the inconvenience of a real plane, of course.

This works quite well in the home cinema sector with the right soundbar or smart speakers. If you want to read more about this, you can find out how it works here. There's no getting around the topic of Dolby Atmos here.

In the context of live music, however, you have a different room that you want to fill with sound. That's why there are some other manufacturers from the hi-res audio sector, but more on that later.

Immersive audio with headphones

Now it's getting personal. For those of you who prefer to immerse yourself in your little world, headphones are the key to happiness. For me, this makes the topic more accessible than a dozen speakers that you would otherwise need.

But how does it work? Without going into too much technical detail, these technologies use clever audio processing to create a three-dimensional sound space within your headphones. You can record the sound with special microphones, for example, or "spatialize" it in post-production.

It feels as if the sounds are coming from different directions and distances - a real audiovisual illusion!

Real on-site: Immersive sound reinforcement

Let's put spatial sound into practice and imagine a concert or live event in a typical location. Here I would go into detail again using loudspeakers and headphones to show how the 3D format can be utilized.

Immersive sound reinforcement with loudspeakers

In traditional sound reinforcement systems, especially in stereo, the best listening experience is often found in the middle between the two main speakers. The reason for this is the phenomenon of the phantom sound source, where the sounds from both speakers combine in the center to create a seemingly unified sound image. However, as soon as you move away from this central point, the sound field begins to diverge and you end up with a mash of sound in one ear.

Detection and elimination of phantom sound sources

On-site immersive sound breaks through the traditional boundaries of the audio experience by using innovative speaker arrangements. Effectively, you don't just have speakers to the left and right of the stage, but distributed more centrally. The aim is to create a sound field in which the position of the audience is no longer decisive for the sound quality.

Advantage for the listener

One of the key aspects here is matching what you hear with what you see, especially in live performances. For example, if a musician is playing on the left side of the stage, the sound should also come from that direction. This not only increases spatial perception, but also enables the audience to identify the individual instruments more clearly and with greater differentiation - a clear advantage over frontal sound reinforcement through the "wall of sound" of conventional sound reinforcement methods.

The hardware offers room for creativity

In addition, the spatial arrangement of loudspeakers around the audience opens up new possibilities for creative experiments. Even if a performance takes place mainly in front of the audience on a stage, interesting and unexpected effects can be achieved through the targeted placement of loudspeakers around the audience.

Here, elements of a song are usually moved around wildly. Of course, you shouldn't overdo it, but whatever works is allowed. This can enhance the immersive experience by using spatial sound to create a deeper level of engagement and experience.

Immersive headphone events

The concept of immersive spatial audio can not only be applied to traditional loudspeaker systems, but also finds innovative use in headphone-based applications such as silent discos. This technology allows DJs and performers to transport the audience into a whole new world of sound by providing each listener individually with spatial sound via headphones. It may seem simple at first glance - the DJ connects and the experience can begin. But the possibilities opened up by the integration of 3D audio go far beyond that. See: Disco headphones: A new dimension for live events with 3D audio

© Christiane Mudra

A particularly impressive example is the use of 3D audio in conjunction with visuals or even in virtual reality (VR), where the boundaries between acoustic perception and visual experience become blurred. A specific project I was involved in illustrates the potential of 3D audio streaming technology:

A theater production that offered a three-dimensional sound experience via headphones and thus drew the audience into the action on stage in a completely new way. Through the precise placement of sounds in three-dimensional space, emotions, and atmospheres could be conveyed in a way that would not be possible with conventional sound methods. You can find more information about this project on my website.

These developments open up a world of new creative possibilities for artists and technicians to fascinate and engage audiences in unprecedented ways.

Mobile immersive experiences

But you don't have to be tied to one event location. Mobile installations or even open-air experiences with an immersive character are becoming increasingly common. The focus here is not on sound from a specific location. Instead, visitors can move freely around the space, pushing the technology further into the background.

Immersive sound installation

3D audio is revolutionizing the way sound is used in a variety of places by creating immersive environments that take visitors on an auditory journey. In museums, 3D audio enables exhibits and artworks to be brought to life through appropriate soundscapes, creating a deeper understanding and a stronger emotional connection.

Churches can also be transformed with 3D audio to enhance spiritual, historical, or artistic narratives. By placing speakers at strategic points within the space, an acoustic environment is created that utilizes the architecture and sacred atmosphere of the church. In most cases, projections in the form of videos are projected onto the wall to provide an immersive experience.

Here I had a small project at the Deutsches Museum with a combination of tablets, VR glasses, and other technology. So it can be quite a hardware battle - as long as it's worth it. See: Quintessence: A multi-sensory, immersive sound installation

© Martin Rieger

Sound-Walks

Audio guides are well known and are currently experiencing a renaissance. So-called sound walks use spatial audio in combination with GPS or motion data. During a sound walk, participants are guided through a specially curated audio tour that is played through headphones. These tours are often synchronized with the physical environment in which they take place and offer a unique opportunity to rediscover places through the combination of real-world and sonic design.

Through the use of 3D audio, sounds are positioned in space to interact with the participants' movements. As they traverse different physical environments, the sounds can change, shift, or evolve, leading to a deep immersion in the audiovisual narrative. Mostly in combination with a smartphone and head tracking.

Augmented reality (AR) plays a crucial role in the further development of sound walks. Through AR, visual elements synchronized with spatial audio can be overlaid on top of the real world to create an even more immersive experience. For example, historical scenes or fictional characters could appear in the environment, synchronized with the 3D audio, to tell stories in a way that is both aurally and visually engaging.

Here is a video of how you could imagine a sound walk through Leipzig. You can also see and hear an immersive audio pioneer sound installation:

Virtual realities, spatial computing, Metaverse, XR

In the era of digital innovation (the headline contains all the terms that somehow mean the same thing), spatial audio opens up new dimensions for streaming and digital events by creating an immersive soundscape that pushes the boundaries of traditional audio experiences. Online events - whether conferences, concerts, or theater performances - are enhanced by the use of spatial audio, giving participants the feeling of being in the middle of the action. My article from Event-Partner highlights how spatial audio can enhance online experiences. There I already discuss video live streams, video conferencing, and virtual event platforms.

But due to the hype surrounding the new Apple Vision Pro, I would like to return to the subject of virtual festivals in 3D space:

The evolution of spatial audio also plays a crucial role in the development of Metaverse festivals. The Metaverse - a collectively shared virtual reality - provides a platform for festivals and events that are free from physical constraints. Here, participants from around the world can come together in virtual environments brought to life through spatial audio. Examples such as Fortnite, which has already hosted concerts in its virtual world, or avatar-based events, show the potential of this technology to usher in a new era of festival experience - in some cases already with millions of users worldwide able to use music streaming simultaneously.

However, the realization of festivals in the metaverse with spatial audio also faces challenges. Even with the best technology, it is not so easy to integrate the right 3D compatibility for the sound. From a creative point of view, the focus is usually simply placed on filming, with 2-channel stereo sound supposedly being the right choice for the sound.

I have already reported several times on my blog about the various platforms and 360 video productions in combination with 3D music.

Relevance of 3D audio and explanations for the future

The introduction of 3D audio for the general public is associated with both technical and financial challenges. The extent to which the cost of implementing 3D audio systems is justified compared to traditional stereo sound is currently the subject of intense debate. While stereo audio has been established and cost-effective for decades, spatial audio requires a significant investment in new technologies, both for consumers and producers.

© KI-generiert von Martin Rieger

Finally, a critical reflection on the limitations of 3D audio is essential to provide a balanced perspective. This includes discussing possible undesirable effects or technical and creative challenges that may arise when implementing spatial audio. Such consideration helps to keep expectations realistic and to find applications in the field of live experiences that also have real added value.

In summary, the future of 3D audio is promising and is being fueled by innovations in other areas, from Dolby Atmos Music to AirPods. The ongoing development and integration of spatial audio into our daily lives has the potential to fundamentally change the way we experience and interact with sound.