Sensor-based Instruments in a context of free improvised music

Free improvised music

Free improvised music as its genre and subculture developed slowly through the second half of the 20th century, even though taking its roots from different genres, it established nowadays as its musical genre and influenced the growth of various communities for improvised music. As Edward Neeman indicates: “The late 1950s and 1960s saw the birth of several groups devoted to performing improvisation in an unrestricted style. Classical musicians were inspired by the freedoms of their jazz colleagues and sought to recapture that spontaneity in their music. Many groups, particularly those with jazz roots, cited John Cage as an important influence.” (Neeman, 2014, p. 12). Cage, as Neeman also mentions, was quite skeptical regarding improvised music, using the concept of aleatoric, composing free-for-interpretation graphical scores, and inventing happenings. In the interview with Steve Sweeney, he says: “Improvisation [...] is something that I want to avoid.” (Sweeney, Cage, 1990, p. 472).

Among famous improvised collectives, who were pioneers of free improvised music in the middle of the 20th century, it is important to mention SME (Spontaneous Music Ensemble), which was active in England in the years from 1965 to 1994, AMM (also based in England, active since 1965), The Music Improvisation Company (British ensemble, active at the end of the 60s and beginning of the 70s, launched by Evan Parker, Derek Bailey, Jamie Muir and Hugh Davies, English composer, who was also the inventor of experimental musical instruments), Iskra 1903 (British music collective, active in the 1970s), The Art Ensemble of Chicago (Chicago based ensemble for improvised music in the 60s, grew from a community of AACM - Association for the Advancement of Creative Musicians).

© Ralph-Quinke, ECM-JDG-Masthead

The Art Ensemble of Chicago

Limitations of the human being as a starting point

Parallel to the community growth of free improvised music in the mid-20th century a development of sensor-based instruments and interfaces started to establish and become more powerful. However, sensor-based interfaces are still relatively young and novel to musicians compared to classical instruments. The development of sensor-based interfaces is primarily focused on personal usage by the developers, who typically build their instruments for their use. According to Tanaka, most sensor-based instruments are highly personalized and are developed for personal usage by artists, who also serve as the inventors of their instruments. This makes such instruments into more closed systems, which can be operated mostly by its inventors. (s. Tanaka, 2009, p. 243). Nevertheless, there is also usage of these instruments by groups of musicians:  "These instruments were at once idiosyncratic instruments highly personalized to a particular artist's needs and at the same time general platforms for groups of artists to explore and extend.” (Tanaka, 2009, p. 243).

The development of new instruments and interfaces for musical expressions would be not possible without such personalities as Oskar Sala, Leon Theremin, Maurice Martenot, and others, who distinguished their inventions from traditional acoustic musical instruments and worked instead on development of new interfaces, more suitable and more experimental. As Bert Borger in his paper “Physical Interfaces in the Electronic Arts” insists: “Instrument designers like Leon Theremin, Maurice Martenot, Oskar Sala and Hugh Le Caine seemed to exploit the freedom of design offered by the new medium, which imposed far less on the design of the shape and dimensions of the instrument than the traditional instruments did. They could take the properties and possibilities (and limitations) of the human being as a starting point for the design.” (Bongers, 2000, p. 42). If Leon Theremin’s instrument was a new interface for that time, which allowed the performers to use gestures to create sound, Ondes Martenot in comparison to Theremin, used a combination of a new interface (metal ring) with a classical piano keyboard to manipulate the sound. 

© Wikipedia, CC BY-SA 3.0

Les Ondes Martenot

Cracklebox and the Hands

One of the iconic persons for the sensor music scene was Michel Waisvisz, Dutch composer and inventor of experimental musical instruments, such as Cracklebox and the Hands. Atau Tanaka in his paper “Sensor-based musical instruments and interactive music” describes the origin of the STEIM’s (Studio for Electro Instrumental Music, based in Amsterdam, Netherlands, launched in 1969) related movement of the new musical instrument’s inventions connected to the series of instrument building workshops, initiated by Michel Waisvisz. As one of the outcomes of such workshop Michel Waisvisz’s instrument Cracklebox was developed, Tanaka: "It was with the arrival of the late Michel Waisvisz in 1973 that STEIM took on an instrument-building workshop approach, first producing the Cracklebox, an instrument by which the performer becomes part of the electrical circuitry of the instrument through touch and human conductivity." (Tanaka, 2009, p. 243). Later Michel Waisvisz recorded his free improvised album, played with the Cracklebox instrument “Crackle”, released in 1978. (Crackle, 1978).

With the Hands, a sensor-based instrument, Michel Waisvisz performed numerous concerts, establishing it as his trademark and primarily associating it with his name.

© Cover, Discogs / The Hands, Luka Ivanovic

NIME and community

Nowadays the community, related to the development and research of new sensor-based instruments is closely connected to the NIME (New Interfaces for Musical Expressions) conference, which was launched in early 2001 in the USA.

Since then the NIME conference has become a platform for developers, researchers, makers, and inventors from all over the world, who build and present yearly a growing number of new instruments and interfaces. In the paper “Design for Longevity: Ongoing Use of Instruments from NIME 2010-2014” (Morreale;  McPerson, 2017) Morreale and McPerson analyze the practical side of the usage of newly developed instruments by performers and inventors. The main question they try to answer in their study is to find out how many instruments were used more than once and if these instruments are ready for ongoing use at the moment. The results of this study show, that 47.1% of DMIs (Digital Musical Instruments), presented at the NIME conferences from 2010 - 2014 are not ready for ongoing use, the rest of the DMIs are ready for ongoing use but would require from its developers for a few hours (31.5%) to more than few weeks preparation time (20%). (s. Morreale;  McPerson, 2017, p. 194). This kind of study shows the problem around newly developed sensor-based instruments: the complexity of the system behind the instrument, which consists not only of hardware but out of software, computer, system, and mapping of the data. As Tanaka writes about sensor-based instruments: “An instrument could be a piece of hardware or a software program but could also be a reconfigurable hardware/software system. The composition could be specifically built into a system as much as a single instrument could execute a family of works.” (Tanaka, 2009, p. 254). Tanaka observes that sensor-based instruments should keep their own identity and still be flexible at the same time.

Limits, characteristics and identity of new invented instrument

In his paper Tanaka brings out interesting questions related to the invention of new instruments: what exactly is a sensor-based instrument, “[w]here does the instrument end and the composition begin? Is the entire system specific for one musical work, or can parts of it [...] be altered or generalized to create different works for the same instrument?” (Tanaka, 2009, p. 238). Tanaka mentions that “music for violin differs from music for flute, even if the two instruments share similar tonal ranges.” (Tanaka, 2009, p. 239), pointing that each sensor-based instrument can also have its own limits and characteristics. If we bring the question of sensor-based instruments in a context of free improvised music performed collectively, a logical question appears: which characteristics should sensor-based instruments have and which limitations, to be flexible enough for the live situation approach in free improvisation with a group of musicians, but not to lose their own identity?

One of the fundamentals for the free improvised music is its immediacy. According to Neeman “[i]mprovisation is the creation of something new in the moment of performance. It exists exclusively in the present, both as a creative impulse and as a musical experience; the power of its immediacy cannot be fully captured for future reinterpretation.” (Neeman, 2014, p. 34). Acoustic instruments, due to their long history of existence, managed over the years and centuries to adapt to the different styles, and genres’ changes and have proven their ability to change the musical context. Musical flexibility, which is so important for free improvised music, is not an easy point to achieve for sensor-based instruments. However, electronic instruments like Ondes Martenot and Theremin were well integrated into the orchestral music by various composers, like Edgar Varese, Olivier Messiaen. (s. Tanaka, 2009, p. 249). For instance, Ondes Martenot was popularized by the famous British indie band Radiohead, which used it in their songs and even took them on tour for their live performances. Jonny Greenwood invested in the manufacturing of a limited edition of this rare instrument. Musicians, like Barbara Buchhol or Pamelia Stickney (known also as Pamela Kurstin), introduced Theremin to the context of free-improvised and jazz music, performing it live with various jazz formations with John Zorn, Ravi Coltrane, and others.

Bob Ostertag

American musician and composer Bob Ostertag is famous for his collaborations with various musicians (John Zorn, Anthony Braxton, Fred Frith, Roscoe Mitchell) in the field of free improvised music, using different self-made instruments, as well as already existing gestural interfaces, like for example Buchla’s Lightning. For his improvised album “Getting a head” (Ostertag, 1980), recorded in the 1980 together with Charles K. Noyes and Fred Frith, Bob built a tape-recorder based instrument, which was controlled by the balloons filled with helium. "The improvised music scene in general assimilated performers on keyboard samplers. From this practice, Ostertag extended this tradition to perform in ensembles with Fred Frith and other notable improvising musicians, using the Buchla Lightning infrared wand controller to articulate and sculpt output from a sampler." (Tanaka, 2009, p. 250). Buchla’s Lightning is a gestural based instrument, similar to another famous instrument Mathews’ Radio Baton (s. Mathews, 1991), allows its performer to manipulate and control sounds with gestures. This gave Ostertag on one side certain freedom in working with samples during free improvisation, but at the same time helped to create his own individual way of performance. 

© Buchla

Buchla’s Lightning III. (Matrixsynth)

Sensor ensembles and improvised music

The tradition of sensor ensembles and musical groups who work with new invented electronic instruments in a context of free improvised or composed music goes back to the 1990s, as Tanaka claims: “The Hub and Sensorband were early examples of network ensembles and sensor-instrument groups, respectively. They inspired a series of groups, including the German group Tryonis and the Australian group Metraform. However, the number of ensembles dedicated to computer-based instruments remains small.” (Tanaka, 2009, p. 250). And it remains small, fifteen years later. Even growth in the market of commercially produced sensor-based tools for musicians, performers, and dancers, like Instruments of Things, Genki’s Wave ring, or Keith McMillen’s sensor keyboards and controllers, did not change this tendency. 

The Sensorband, established in 1990 by three composers and performers - Edwin van der Heide, Zbigniew Karkowski, and Atau Tanaka - initiated a significant tradition of ensembles employing sensor-based and digital musical instruments in live performances. Throughout the 1990s, the trio not only utilized sensor-based and gestural instruments but also developed a distinctive interactive instrument/installation known as Soundnet. This construction, measuring 11x11 meters, required all three performers to climb and interact with it. Bert Bongers observes that “[t]he Soundnet's immense size introduces an enormous challenge of instrumental virtuosity. [...] It is a multi-user instrument where each performer is at the mercy of the other's actions. In this way, the conflict of control versus uncontrollability becomes a central conceptual focus of Soundnet.” (Bongers; Sensorband, 1998, p. 17). After active tour years with the ensemble Sensorband and its disbanding in 2003 Atau Tanaka, together with his colleagues Laurent Dailleau and Cecile Babiole, created a new sensor ensemble Sensors_Sonics_Sights. In the new ensemble Tanaka continues to work with his sensor instrument BioMuse, while Cecile Babiole works with ultrasonic distance sensors and Laurent Dailleau uses Theremin. (s. Tanaka, 2009, p. 250).

Composing for sensor-based instruments

The development of computer music and electro-acoustic composition in the 2000s influenced the development of laptop orchestras and ensembles. The Princeton Laptop Orchestra, also known as PLOrk, was founded in 2005 by Perry Cook and Dan Trueman. The Tradition of laptop orchestras developed fast also in European countries and became a part of computer music studies in various universities. 

The institute of Electro-acoustic Music in Graz, better known as IEM, a part of the University of Music and performing arts, was founded in 1965 and has since that time been established as a strong institution in the field of computer music. The IEM collected a powerful community around itself. During my study for computer music at the IEM, together with my colleagues Jonathan Carter and Takuto Fukuda I co-founded sensor trio Elektrichka, which was active for five years. During this time each of us developed his sensor-based instrument (Sensor Staff, Sensor Box, and Sensor Board), which we used for our live music performances in the sensor trio. We composed music pieces for our ensemble, wrote music for sensor-based instruments, and experimented with different forms and formats. We have opted for graphical scores as a logical method of notation for our music. However, while most of the composed pieces featured a discernible structure, they also incorporated numerous moments of free improvisation. Typically, each piece comprised structured segments, prepared and pre-programmed sound synthesis, and an audio aesthetic. Keeping the structure as a crucial part of the composition, was one of the ways John Cage was integrating improvised music into his composition, as Feisst writes on improvisation in his music: “The only category-defying improvisation was structure. This view implies that for him structure was the foundation of a piece and that after the structure was organized, improvisation could be used as a compositional means, and that material and method could be handled more freely.” (Feisst, 2009)

After the active years of the project Elektrichka, I decided first to concentrate on solo performances with my sensor-based instrument Sensor Board and found certain limitations in further development for the years. The more I tended to work in the field of free improvised music, the more limitations with the sensor-based instrument I was acquainted with. Among such limitations, I would notice several, such as the absence of a plug-and-play solution. Similar to the research about sensor instruments, presented at NIME conferences over the years, a certain amount of preparation time was needed before each show, which made it complicated to have the instrument ready for each spontaneous show. Another important limitation was a lack of a clear sound identity for my instrument, the flexibility of various sound syntheses and the usage of always new gestures for each new piece made it harder for the audience to keep the coherence between the gestures and produced sounds.

© BlablaTrains

blablaTrains, NIME 2019

My colleague from the Elektrichka Trio Takuto Fukuda together with Ana Dall'Ara co-founded another sensor-based ensemble blablaTrains, with which they are actively performing in various venues, investigating gestures in electronic music. In their project, two performers combine the Sensor Box, developed by Takuto Fukuda, and augmented Theremin, used by Ana Dall'Ara as an interface. The duo works with structured compositions for sensor instruments, leaving space for free improvised parts inside the structure. (s. blablaTrains)

Audio-visual improvisation with sensor-based instruments

Together with Ukrainian visual artist Svitlana Zhitnia, also known as Zhi Zhi Visuals, I co-founded a new sensor-based duo called Zhizhi Nini, dedicated to creating audio-visual compositions. Zhizhi Nini integrates sensor-based instruments with game controllers in their setup. Each instrument not only manipulates sounds but also influences live visuals, allowing for complete control over all aspects of their audio-visual performances. Instead of the Sensor Board for the Zhizhi Nini setup, sensor wristbands are used, which help the duo to concentrate more on gestures while working with sounds and visuals.

Audio-visual compositions, created for the live performances of the Zhizhi Nini project, consist mostly of free improvisation, limited only to certain audio-visual aesthetics, which differ from piece to piece. 

Seeking to differentiate myself from laptop-centric music, I began exploring methods to exclusively utilize sensor-based interfaces, aiming to eliminate additional interfaces previously employed for sound selection, composition, and parameter adjustments such as volume during live performances with wrist sensors. Consequently, I embarked on devising a solution for crafting my user interface based on sensor technology and gesture recognition. This endeavor led me to delve into machine learning, enabling more tailored gesture recognition for my purposes.

By generating diverse patterns of hand positions analyzed in real-time by a Pure Data patch leveraging machine learning libraries, I have developed a bespoke user menu complete with sub-menus. This menu facilitates the selection of compositions/sound banks and seamless transitions between performance and user menu modes, all controlled solely by gestures. In performance mode, one hand is dedicated to manipulating various parameters such as volume, pan/spatialization of sound (particularly when working with ambisonics systems), and effects, while the other hand remains free to trigger sounds.

Current point

My current research and work in the field of sensor-based instruments within the Zhizhi Nini project focuses on establishing gesture communication between two performers, which would also serve as a tool for free improvisation. For example, one gesture might mute the other performer during live performances, while other gestures could signal a solo part or invite collaboration in a specific section. Developing this new mode of communication for performers on stage is intended not only to enhance live concerts but also to introduce another layer of performativity based on non-verbal gesture communication.

In my opinion, sensor-based instruments play a significant role in the further advancement of both free improvised music and ensemble performances. Free improvised music presents new challenges for developers, while performers using sensor-based instruments in ensemble settings expand the range of sound aesthetics and overall performativity of ensembles.