ROOMS for your ears

Binaural Stereo

Binaural sound production has transitioned from being a privilege of large industries to becoming an accessible immersive technology. To make it even more accessible, professors of the Institut für Elektronische Musik und Akustik (IEM), Gerhard Eckel, Franz Zotter, and Matthias Frank created the course „Binaural production for sound artists“. Being an educational part of the ROOMS project, the course aims to connect technical and artistic approaches, as well as to provide practical tools for sound creators to explore.

The project leader of ROOMS, Alisa Kobzar, conducted interviews with the authors of the course about the content and development of the course.

Alisa: It is common to think that binaural audio is a type of stereo format, associated with headphones. But what is unique about the binaural stereo?

Franz: Superficially, binaural stereo resembles the typical stereo format by its two channels. Most headphone listeners until now might confuse both because they listen to their loudspeaker-stereo music on headphones. But binaural stereo, which was already briefly famous in the 1980s, directly delivers the signals that would be received at the ears in a 3D sound field. Sounds can appear in all surrounding directions and heights, not just in front as in loudspeaker stereo.

Alisa: Sounds exciting! It seems that now the headphone listeners are dominating the loudspeakers-listeners. Can you say that the binaural format gives more 3D impressions than stereo or 5.1?

Franz: It is quite affordable to consume most music and audio nowadays via earbuds and a smartphone or laptop. And binaural stereo is a perfect fit for most applications and can deliver more 3D impressions on headphones than possible on a stereo or 5.1 loudspeaker system.

Alisa: In which contexts could one use the binaural stereo?

Matthias: Many current music producers pursue delivering their immersive audio as a target format to be listened to on headphones, although the production itself is typically done in studios with surrounding loudspeaker systems. In the future, binaural stereo can be expected to be even more important to the consumer than it is now because what is common now in open-world virtual reality games will be getting increasingly popular and widespread. There, interactively rendered 3D sound environments mimicking a static outside world will be the goal, and smartphones can do the required audio processing and position sensing. There are many interesting interactive applications also for the internet, the Graz-based company atmoky is working on navigable, virtual 3D audio worlds that are navigable in the web browser.

For sound artists, binaural stereo is great also for simulating how their music would sound on different loudspeaker systems and in different electroacoustic performance spaces. This is one of the main focuses of the ROOMS project (details see below).

Creative potentials of binaural format

The Institute of Electronic Music and Acoustics (IEM) is a hub for diverse research, with two branches directly relevant to the "Binaural Production for Sound Artists" course: "Spatial Audio" and "Sound and Space." The course's video lecturers are experts from both fields. The collaboration between the technical and artistic departments of IEM has led to several binaural sound art projects, like "tingles & clicks" of ORF musikprotokoll in 2020, the camera-tracked 3D audio player, the HOAST 3D audio and video streaming library just mentioning some. Additionally, the IEM offers a free Plug-in Suite widely used in the course. Various publications, including "Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality" by F. Zotter and M. Frank, enrich the IEM's theoretical and practical knowledge.

Alisa: Given this rich environment, it prompts the question: what should sound artists consider while producing for the binaural format?

Gerhard: While it is interesting to explore the immersive possibilities of the binaural format, i.e. allow for sounds to be localized anywhere around the listener, at various distances, with different orientations of the sound source, and in different acoustic environments, the degree of immersion can even be enhanced by contrasting binaural sound layers with non-binaural ones. More specifically, the sound’s location could be:

• inside one’s head: when sending the same signal to both ears, which creates the surreal sensation of a sound located inside the listener’s head.
• only in one ear: when presenting a signal only to one ear (which does not have an equivalent in an unmediated listening situation, i.e. without wearing headphones).
• in-between #1 and #2: within a large range of possibilities between the mentioned extremes, allowing for hybrid spatial audio experiences and morphing between them.
• between acoustic environments: binaural sound lets you feel like you're in different rooms, which is only possible with headphones because they block out other sounds. This is unlike loudspeakers, which don't isolate you from your surroundings. Transitions between acoustic environments can be used as strong dramaturgical devices, for instance evoking in the listener a sense of movement through different spaces.
• as a shared experience: using open-back headphones enables the composer to combine individual binaural sound projection with shared loudspeaker projection, opening up yet another range of sound dramaturgical devices in concert situations.

IR - a technical or compositional tool?

One of the main tools of production for virtual spaces is IR or impulse response. It is a necessary part of the convolution process and convolution plugins, used in the course.

Alisa: While it is explained in detail in the video course, Franz, could you briefly tell us what is the main purpose of IR?

Franz: Impulse responses (IRs) are a very powerful characterization of the response of a system to an impulse. In rooms, the Room Impulse Response (RIR) is powerful in characterizing what its acoustics do to the signal of a loudspeaker when received. To our ears, the binaural room impulse responses (BRIRs) characterize what will arrive through the room to a listener’s ears when a loudspeaker plays an impulsive click, an impulse. We use a dummy head, a mannequin head with two pinnae, and ear microphones to measure that.

The cool thing about impulse responses is that we can decompose a signal into a sequence of closely spaced impulses with different amplitudes. In this way, we also know what each of these tiny signal bits will produce as a weighted impulse response at the ears. Adding up this sequence of signal-weighted and time-shifted binaural impulse responses is called convolution, and it gives us the signals we can play back on headphones. Nothing new, really, to the researcher and many artistic researchers, but still quite a powerful thing to have in audio and music production.

Binaural room impulse responses virtualize the behavior of a room with many loudspeakers for one listening position. 
It allows us to try out how different signals would sound when played through the loudspeakers in the virtual room.

This is still amazing, also after working for more than 20 years in the field. If interested: check out our video lecture series and its examples.

Alisa: From the technical point of view sounds very interesting! And which artistic implications of the IRs could you imagine?

Gerhard: I can think of the following two implications:

• Simulation of target acoustic environment. IRs are a great tool to work with electroacoustic setups (loudspeakers and microphones arranged in a room) off-site. The only thing you need are good headphones. Consider you want to produce a site-specific electroacoustic work and have only limited access to the site during the composition process. If you can make binaural impulse response measurements of the setup of your choice, they can provide you with a very reliable simulation of the most important acoustic properties of the situation you are composing for. Even if not all details can be covered by such a simulation, the advantage of being able to create in the target environment as opposed to in the studio is enormous.
• IRs as a creative space. There is the possibility to compose with the IRs themselves, convolving sound material with the measured IRs to achieve effects other than the auralisation of a particular electroacoustic environment. Imagine you convolve a sound again and again with the IR of a particular loudspeaker/microphone setup in a room. This would be (at least conceptually) equivalent to the approach Alvin Lucier used in his seminal piece “I Am Sitting in a Room” from 1969. In the video course, I explore this idea and provide the means to compose with such concepts.

Course: „Binaural production for sound artists“

„Binaural production for sound artists“ was an online course by the IEM for composers and sound artists on working with 3D spatial audio using only their headphones and a laptop. It covered the necessary theory and practice of working with up-to-date 3D audio production tools and focused on the binaural stereo format as a target. The goal of the course was to apply the acquired knowledge and skills in the theory and practice of spatial sound in binaural representation in personal artistic practice. 

The pilot project’s phase (implemented during the spring of 2022) included 12 participants - electronic/electroacoustic composers, sound artists, and musicians from Ukraine born after 1981 and students of the University of Music and Performing Arts Graz (KUG) in Austria with experience in writing electroacoustic, computer music or sound design.

During the 30 days of access to the video tutorial series (hosted on the Moodle platform), participants were always able to stay in touch with the course teachers - by using the discussion forum or through personal consultation related to the course. The training material consisted of tutorial videos, supplementary content, and tests to sequentially unlock the next tutorial.

The final work of the course was a sound composition or arrangement of one’s own composition or sound installation (up to ten minutes long), which was published on the website of the ROOMS project and its playlist.

At the end of the course, several works were selected from among the final works to receive special awards.

Here are some of the final works:

After the pilot phase of the educational part of the ROOMS project was over, a video tutorial series was opened for public access. As a part of the course, Gerhard Eckel developed a SuperCollider tool, covered in the last 4 units of the video tutorial series.

Alisa: Gerhard, can you describe the creative potential of a tool you developed in SuperCollider?

Gerhard: While preparing the course I was looking for an example of employing convolution with room impulse responses for other purposes than auralisation. Alvin Lucier’s piece “I Am Sitting in a Room”, which has been an inspiration for many composers and sound artists for more than half a century now, seemed an appealing starting point. Based on a performance of the piece in the IEM CUBE, I created an environment to simulate that performance using signal processing code written in the SuperCollider language.

All operations are performed in non-real time and can therefore very easily be modified and extended by the composer. The provided SuperCollider classes allow for producing structured corpora of iteratively convolved signals, which can be browsed with a graphical tool illustrating the topological relations of the speakers and microphones used to measure the impulse responses. This conveys an experiential understanding of the sound transformations and their dependence on the minutest details of the speaker/microphone configurations, which is also the source for one of the fascinations afforded by Lucier’s piece. By linking back nowadays possibilities of easily working with impulse responses and convolution to Lucier’s pioneering work of sound art, I hope to inspire composers and sound artists to further explore the underlying concepts artistically.

Alisa: What should be a person's background to make the best use of the course?

Matthias: You can make the best use of the course if you have basic knowledge about sound processing with computers, i.e. if you know what a DAW is and can operate it. Furthermore, experience with sound processing environments such as Pure Data, SuperCollider, or Max/MSP will be very helpful. It is an advantage to have experienced a performance of “I Am Sitting in a Room” by Alvin Lucier. In terms of hardware, a laptop, and good stereo headphones are all that are required.

Alisa: And where could the knowledge gained during the course be applied?

Franz: In your artistic practice as a composer and sound artist, as a producer of radio drama and sound for theatre - to name just a few examples. The knowledge and skills will be useful anywhere, where the particularities of acoustic environments and their simulation can be used creatively.

Postskriptum: ROOMS Project

© IEM

Screenshot from the video course „Binaural production for sound artists“

The ROOMS was initially planned as an international double residency hosted by the IEM KUG in Graz, Austria, and the Creative Sensor NGO within the expansive grounds of the Vymiriuvach Innovation Park in Zhytomyr, Ukraine. However, the onset of the pandemic and subsequent war in Ukraine prompted the ROOMS team to adapt its original offline format into a multifaceted project. This transformation led to the development of two main components:

• An educational aspect is represented by a freely available online video tutorial series.
• A shelter renovation initiative focused on the urgent refurbishment of a bomb shelter in Zhytomyr.

The project derived its name from its central concept of exploring virtualized spaces. The video tutorial series, titled "Binaural Production for Sound Artists" concentrates on utilizing binaural stereo as the primary format for both theoretical and practical creative elements. This choice aimed to enhance course accessibility, requiring only a laptop, headphones, and an internet connection. The primary language of the course is English, with subsequent additions of English and Ukrainian transcriptions to all video content following the pilot phase.

For more information, visit the project's website.

The project was funded by the House of Europe Grant for International Collaborations. Most of the production costs for the video tutorial series were donated to the shelter renovation initiative.

Authors of the course "Binaural Production for Sound Artists"

© Barbara Becker, Franz Zotter, Matthias Frank

Gerhard Eckel, Franz Zotter, Matthias Frank

Gerhard Eckel is a Professor of Computer Music and Multimedia at the IEM (KUG) in Austria, an affiliate professor at the KTH Royal Institute of Technology, and a visiting professor at the Royal College of Music (KMH) in Stockholm. Besides his artistic work and teaching, he leads publicly funded transdisciplinary research projects and supervises scholarly and artistic doctoral research. He is an artist using sound to explore ways of world-making. 

Franz Zotter is an Associate Professor at the IEM (KUG). He does research in Audio Engineering. In 2019 together with M. Frank he wrote the book “Ambisonics. A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality”, published by Springer. In 2012 he was awarded the Lothar-Cremer medal by the German Acoustical Society (DEGA) for his work on this PhD topic and Ambisonics. He is a member of the DEGA, AES, and VDT.

Matthias Frank is an Assistant Professor at the IEM (KUG). He is a specialist in spatial sound recording and playback and an active researcher and developer of virtual and augmented reality in the field of acoustics. In 2019 together with F. Zotter, he finished writing the open-access book “Ambisonics. A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality”, published by Springer. He is a member of the DEGA, AES, and VDT.