John Greschak - Technology-Assisted Conducting of Polytempo Music: Some Possibilities

Technology-Assisted Conducting of Polytempo Music
Some Possibilities

by John Greschak

August 30, 2001

Introduction
Semi-Automatic Conducting
Content-Rich Audio Cues
Visual Cues from Artificial Conductors
Tactile Cues
Multi-Sensory Cueing
Wireless Connections
Coda
References
Update History

Introduction

This article proposes some suggestions as to how technology might be used to coordinate the performance of polytempo music (i.e. music in which two or more tempi occur simultaneously). It expounds on some of the possibilities that were presented in the brief article Viewing a Finale Document on a Wearable Computer for Performance Purposes in which a particular technology-based conducting system was put forward for consideration.

For an overview of the ways in which technology has been used to coordinate polytempo music, see the section Technology-Assisted Conducting in the article Facilitating the Performance of Polytempo Music: An Overview.

Semi-Automatic Conducting

Various automated conducting systems have been used to coordinate the performance of polytempo music. In these systems, the cue-signal mechanism operates autonomously. For example, for systems in which cue signals are sounded from audio tape, the tape player operates independently of the performers or conductor (between the times at which playback is started and stopped). Similarly, for MIDI-based systems where cue signals are sounded by sequencer-controlled synthesizers, the sequencer operates in a self-directed manner, outside of the control of anyone and unaffected by the performance.

For fully automated conducting systems such as these, the cue-signal mechanism is in effect, the main conductor, and all performers and conductors must follow its lead. Consequently, the pace of cue signals, and thus the overall tempo of the performance, is not controlled by anyone during the performance. Instead, the tempo is set prior to a performance at the time at which the cue signals are encoded on an audio tape or in a MIDI file.

However, this need not be the case. Technology exists that could enable a human conductor or lead performer to serve as the main conductor. For example, a human conductor could be equipped with a wired baton (e.g. one of the available Baton Interfaces such as the Mathews/Boie Radio Baton or the Buchla Lightning) or a lead performer could be equipped with a suitable controller such as a wired foot pedal. Another device could receive the beat messages sent from the human leader's controller, and transmit appropriate beat-cue signals to each performer. The beat-cue information which is supplied to each performer would be derived from, but not necessarily equal to, the beat established by the human leader. In a semi-automatic conducting system such as this, the cue-signal mechanism would serve as an automated assistant conductor.

Alternatively, the cue-signal mechanism might automatically determine the appropriate pacing of beat-cue information by applying a beat detection algorithm to a collection of audio signals from some selected performers, or from the composite audio output of the entire ensemble.

For such systems, before the performance, one would need to specify the times at which conductor beats are to occur. For example, with the radio baton, this could be done by inserting appropriate note-on MIDI messages for MIDI key numbers in the range 0 through 11. This feature of the radio baton is described in the section "Updates on the Radio-Baton Program" by Max V. Mathews on the page "Controllers for Computers and Musical Instruments" that is available at http://www-ccrma.stanford.edu/overview/controllers.html.

When doing this, one might specify conductor beat times that coincide with recognizable beats of the overall resultant tempo that is heard when the simultaneous different tempi of the work are performed. Or, one might choose to have the times be irregularly spaced. For example, this might be done to avoid boredom on the part of the main conductor, or to exaggerate or diminish the impact of certain anticipated tempo changes that might be caused by the conductor. For example, if one expects that a conductor will play a given passage faster than desired, this tendency might be counterbalanced by increasing the duration between consecutive conductor beats during the passage. To this end, one might experiment with various changes in the duration between consecutive conductor beats or the time of such beats relative to beats in the overall resultant tempo, and observe the effect that such variations have on the conductor and the tempo of the performance.

Content-Rich Audio Cues

In an automated conducting system where audio cues are employed, cue signals are provided to performers (or conductors) through headphones. For most systems of this type, audio cue signals consist of a sequence of "clicks", each of which is sounded at the time at which a downbeat is to occur. For most systems, the click sound is constant throughout. That is, all clicks have the same pitch, duration, intensity, timbre and spatial position. For example, each click might be an audio sample of a cowbell that is centered in the stereo field.

In a few cases, clicks have been used to communicate information other than beat times. For example, in works by Emmanuel Ghent, audio cue signals are used for the following purposes: to indicate downbeats and upbeats (with signals for downbeats being longer than those for upbeats), to establish a new tempo before notes are to be played (as a few preparatory beats), to signal the entire rhythmic pattern of a complicated phrase before it is to be played (e.g. for klangfarbenmelodie that has an aperiodic rhythm), and to provide a reassuring checkpoint cue or an alerting cue just before the end of a long rest (in the form of a rapid succession of three cue signals).

In Larry Austin's Life Pulse Prelude, the click pitch and the duration between clicks is varied to distinguish the first and last beats of a measure from intermediate beats. For example, the sequence of click pitches used for the 31 beats of a 31/8 measure is |C---c---c---c---c---c---c---ceg| where 'C' is the pitch that is an octave above the pitch 'c', the pitches 'e' and 'g' are in the octave between 'c' and 'C', and the symbol '-' represents a beat for which no click is sounded. Similarly, the sequence used for 13/4 measures is |Ccccccccccceg|. In each case, a high 'C' is used to distinguish clicks that coincide with the first beat of a measure, and the pitches 'e' and 'g' are used to signal the last two beats of each measure.

Beyond this, audio cue signals could be used to transmit other types of messages as well. For example, audio cues could provide textual information in the form of words that are spoken automatically, such as a lead-in measure consisting of the words "mark 95...one, two, three, four", or instructions such as the words "con sordino" to remind a given performer to play a particular passage with the mute on. Similarly, expression markings or performance notes could be spoken as cues. (Note: For spoken cues, it might be possible to use existing software that is capable of automatically converting text to speech at specified times.) Also, representational sounds could be used to communicate various messages. For example, the sound of a rapid succession of two taps of a baton on the edge of a table might be used as an alerting cue. And, audio cues could indicate the pitch, duration, intensity and timbre of notes that are to be played. Pitch cues might be especially useful for microtonal works in which unfamiliar pitches are employed, or for tuning purposes. Some audio cues could be spoken into a microphone by a conductor and then transmitted directly to the headphones of a given performer.

Additionally, audio cues could consist of music (with no clicks) to which a performer would add his part. To guide a performer, a composer might specify the intended relationship between the cue music and the added part. For example, the added part might be thought of as a solo that is to be added to an accompaniment, as an accompaniment to be added to a solo, or as a part to be blended into the cue music. The cue music could be different for each performer and would not be heard by the audience. In addition to providing a pulse, it could serve to put the performer in a particular frame of mind for a given phrase that is to be performed. This approach would have the advantage of providing audio cues that are similar to those that a performer would hear ordinarily when performing in a traditional group.

Potentially, audio cue signals could be disadvantageous. Since they are transmitted via the same medium (i.e. sound) through which the creative output of the performer is carried, they might be distracting. For example, it is possible that some performers would find it difficult to create a non-metronomic style of artistic expression while listening to (and following) a sequence of evenly spaced click sounds. Also, headphones or earphones that are used to transmit cue signals can act as an unwanted attenuator or filter, affecting a performer's perception of other sounds such as those emitted from the performer's own instrument as well as sounds from other performers in an ensemble, all of which might be used by a performer to control pitch, intensity and timbre.

There is another difficulty associated with the use of audio cues. For most works, consideration must be given to the relative loudness of cues and musical sounds. Soft audio cues can be difficult to hear over musical sounds. And, loud cues can drown out musical sounds or bleed from headphones into a neighboring microphone used for amplification or recording purposes. For works that vary over a wide dynamic range, it might be necessary to vary the intensity of cue signals throughout the course of a performance.

Visual Cues from Artificial Conductors

Visual cues are often used in performance, especially to communicate a pulse. For example, performers follow a tempo established by the motions of a conductor or the gestures of a lead performer such as the first violinist in a string quartet. Given this, one would expect that automatically generated visual cues might be used for a similar purpose in a conducting system for coordinating the performance of polytempo music. But, outside of a few examples where blinking-light metronomes have been employed, visual cues have rarely been used in this way.

With existing technology, it would be possible to devise an artificial conductor that would imitate gestures typical of human conductors, to indicate the time at which downbeats are to occur. For example, an artificial conductor could be realized as a hologram to be displayed before a performer or subgroup of performers. Or, it could be a more abstract representation such as a colored disc that moves up and down on a standard computer monitor, like the tip of a human conductor's baton. Or, a video recording of a human conductor could be used. (Note: A video recording of a human conductor has been used to coordinate the performance of electro-acoustic music. For example, see the project Animated Visual Score for Electro-Acoustic Music by Chad Kirby that was done during the Fall of 1998 at the research center CARTAH of the University of Washington.) Another possibility would be to use robot technology to simulate a human conductor. (Note: A robot conductor was used by artists Rob Gonsalves and William Tremblay in their work Y2K Pops (1999) of which an image is available at http://members.aol.com/y2kpops/show_front_wide.jpg.)

A visual mechanism such as this might have the following advantages over audio systems that employ a sequence of click sounds for the same purpose:

With visual cues, beat times would be indicated by gestures similar to those that are used to coordinate a traditional ensemble. Because of the familiarity of this type of cueing mechanism, performers might willingly accept it and adapt to it easily.

With visual cues that imitate a human conductor, the precise time at which a downbeat is to occur is relatively indistinct, and as a result, players have some freedom to choose the times at which downbeats are to occur in their own part. With visual cues such as this, the moment at which one beat ends and another beat begins cannot be sharply delineated because the speed of a conductor's downward motion must gradually decelerate and accelerate about the extreme lowest position in order to change direction. Under such conditions, a performer may play a beat at any time while the conductor is in the neighborhood of the bottom of a down stroke, and still feel that he is in conformance with the conductor's indication. One consequence of this would be that it might be easy for a player to deviate from a metronomic style of performance when following visual cues that imitate a human conductor.

It is possible to predict the onset of a pending downbeat, when downbeats are indicated by the motion of some object such as a conductor's baton that moves up and down in such a way that the lowest point of each stroke occurs at the same vertical position. For a given downbeat, one may estimate the time at which the baton will arrive at its low point by observing the vertical position from which the down stroke began and the speed at which the baton descends to the low point. This can be beneficial, especially when the tempo is to be changed as in an accelerando or decelerando.

Another possible advantage of using visual beat cues relates to the ease with which a performer might synchronize to such cues. This stems from observations made by Edward T. Hall in a chapter titled "Rhythm and Body Movement" (p.61-73) of his book Beyond Culture (1976). There, Hall notes that the members of a group of people automatically synchronize their body motions among themselves when they are interacting. For example, he describes a study in which the analysis of a video recording of children playing in a playground revealed that the children's motions were synchronized as if they were all dancing to the same song, although no music was playing and the children were not singing together. And throughout, one of the children moved about the group affecting the motions of neighboring children, much like a conductor. To play a note on an instrument, one must move in some way. Usually, the time at which a note is to occur is related in a simple way to the times at which downbeat and upbeat gestures are made by the conductor and performers. Thus, in performing a work, each member of an ensemble synchronizes their motions to those of a conductor, lead performer, or in some cases, another performer playing a part that serves as a cue. From Hall's observations, one might hypothesize that people are naturally suited to the task of synchronizing their motions to those of others, and consequently, performers might be coordinated with ease by using visual cues that imitate the gestures of a conductor.

Visual cues do not have the disadvantages of audio cues that were cited in the previous section. Specifically, since visual cues are silent and are perceived by one's eyes rather than ears: they may be perceived regardless of the loudness of the music; they do not interfere with a performer's perception of the sound of their own instrument or the sounds of others; and they do not generate unwanted sonic noise in a recording or performance situation. (Here, it is assumed that the mechanism used to generate visual cues is silent.)

For systems in which an artificial conductor is some type of image, musical scores could be included in the image, as well. To cue the entry of notes and beats, or to indicate the current position in a score, a "follow-the-bouncing-ball" approach could be used whereby some moving object such as a scrolling vertical line or bouncing disc could be displayed. As noted in the article Viewing a Finale Document on a Wearable Computer for Performance Purposes, additional benefits would come from integrating the display of musical scores into an automated conducting system. For example, it might facilitate the use of multi-colored scores, and musicians could continue playing through page turns, which could be accomplished almost immediately upon request (e.g. by using a foot mouse) or automatically. The designers of such a system could draw from the experience of recent explorations on the use of computer monitors to display scores for the performance of monotempo music. These would include the following efforts:

The MOODS Project.

Paolo Nesi

University of Florence

Managing Music in Orchestras

The Muse

Designing the muse: A Digital Music Stand for the Symphony Musician

http://www.idsa.org/whatis/seewhat/idea/winners/muse.htm

A Study on the Automation of Page Turns

Page Turning - Score Automation for Musicians

The Chamber Orchestra of the Bamberg Symphony Orchestra.

Notebook statt Notenblatt - Bamberger Symphoniker lesen im Konzertsaal vom Computer ab / Bildschirm mit dezenter Hintergrundbeleuchtung

Noten aus dem Notebook

http://info.bamberg.de/magazin/kultur/symphoniker/SiemensFoto.jpg

Harry Connick Jr..

A Chart A Day

Concert Technology Debuts in Ames

Also, along with beat-cues and scores, visual cues could include electronic messages consisting of text, signs or symbols that would be sent by a conductor to be displayed before a given performer. For this, it might be possible to use a speech recognition system to translate commands spoken by a conductor into electronic messages.

Wearable computer displays might be used in place of standard computer monitors to display beat-cue information, scores and messages. As side benefits, music stands and monitors could be eliminated from the stage which would result in a simpler setup with an aesthetically pleasing clean appearance, and musicians equipped with wearable computers would be mobile (although a given performer's mobility would still be limited by the mobility of their instrument). For more information on wearable computers, see: Wearables Links.

Tactile Cues

Tactile cues have rarely been used for the purpose of coordinating the performance of polytempo music, if at all.

It is possible to perceive beats with the tactile sense. For example, one is able to measure a person's pulse rate by placing the fingertips on various locations on the body (e.g. at the wrist, neck, top of the foot, or temple near the ear). And, during a performance, especially if it is loud, one can sense the vibrations and literally feel the beat. Deaf musicians such as percussionist Evelyn Glennie use the tactile sense extensively in live performances. This suggests that it might be possible to transmit beat cues as periodic movements or pulsations of some device that could be felt during a performance.

For example, an inflatable rubber cuff similar to that which is used for measuring blood pressure could be placed around a leg above the ankle. With a solenoid-operated pump, a small amount of air could be pumped into the cuff on downbeats and released on upbeats. The resulting pressure changes in the cuff would be felt by the performer. Alternatively, a band with a winding mechanism (like a tourniquet) could be used to produce pressure changes. With systems of this type, safeguards would be needed to ensure that the pressure placed on a performer would not stop blood flow, cause injury or be painful in any way.

Alternatively, a solenoid-based device could be attached to the front of a performer's thigh and positioned so that its line of motion is perpendicular to the surface of the leg. The solenoid could be thrust slightly toward the leg on downbeats, and withdrawn on upbeats. Safeguards would be needed to ensure that the trusting would not cause pain.

The devices suggested here are somewhat related to those that have been proposed to provide haptic feedback to a musician that is using an electronic controller. For example, see the section "Incorporating Haptic Feedback into Music Controllers" by Sile O'Modhrain on the page "Controllers for Computers and Musical Instruments" that is available at http://www-ccrma.stanford.edu/overview/controllers.html.

The tactile sense might be of relatively limited use for receiving musical cues, because, as compared to the visual and auditory senses, it is capable of perceiving much less complex information. However, tactile cues might be very helpful in a multi-sensory cueing system where they could reinforce simultaneous visual and audio cues.

Multi-Sensory Cueing

In a traditional performance by a Western music ensemble, coordination is accomplished with a complex system of multimodal cues that are transmitted simultaneously. Each ensemble member sees their own part score as well as gestures of the conductor, lead performer, other performers, and the audience. Each hears the sound that they generate together with that of other performers. And, they feel the vibrations of that sound.

Most automated conducting systems use audio or visual cues, but usually not both. Automated conducting systems could be developed that use audio and visual cues in combination, along with tactile cues. Such a system could simulate much of what a given musician perceives in a traditional performance, but on an individual basis with independent cues for each musician. Along with cues supplied by an automated conducting system, all of the traditional audio, visual and tactile cues (mentioned in the previous paragraph) could be used as well.

Any work in this area would be related to and could benefit from that which is currently being done in the areas of multimodal interface design, multimodal presentation or output, and multimodal virtual reality systems.

Wireless Connections

All connections could be wireless. This was suggested by Emmanuel Ghent in the article "Programmed Signals to Performers: A New Compositional Resource" (1967) for an automated conducting system involving audio cue signals derived from tape to be distributed to musicians through miniature earphones. Ghent's suggestion could be extended to include the connection between a controller that a conductor might use to transmit beat messages (e.g. a baton), and the device to which such messages would be sent. Also, regardless of the mode of cue signals (e.g. audio, visual or tactile), such signals could be transmitted to musicians by wireless means.

Coda

Most likely, any automated conducting system that is adopted for use by traditional Western music ensembles, if any system is adopted at all, will satisfy the following criteria: it will be the simplest system that accomplishes the task, it will be robust, it will gather and present information in a way that seems natural and is least disturbing, it will be suitable for ensembles ranging in size from a duo to a full orchestra, it will not detract from the ambiance of the performance environment, and it will be relatively inexpensive. In short, probably, it will be a system that represents an evolutionary step in performance technology that is simple, reliable, scalable, aesthetically pleasing and affordable.

References

Austin, Larry 1985. “Charles Ives's Life Pulse Prelude for Percussion Orchestra: A Realization for Modern Performance from Sketches for his Universe Symphony.” Percussive Notes: Research Edition 23(6): 58-84.

Ghent, Emmanuel 1967. “Programmed Signals to Performers: A New Compositional Resource.” Perspectives of New Music 6(1): 96-106. Reprinted in Boretz, Benjamin and Edward T. Cone (Eds.) 1976. Perspectives on Notation and Performance. New York: Norton.

Greschak, John 2000. “Viewing a Finale Document on a Wearable Computer for Performance Purposes.” Finale: IWBNI Suggestion No. 644. Available: http://www.johnpaulandbertha.org/wbjp/notation/finale/iwbni/fs644.htm.

Greschak, John 2001. “Facilitating the Performance of Polytempo Music: An Overview.” Polytempo Music Articles. Available: http://www.johnpaulandbertha.org/wbjp/polytempo/ptperf.htm.

Hall, Edward T. 1976. Beyond Culture. Garden City: Doubleday.

Ligeti, Lukas 2000. “Beta Foly: Experiments with Tradition and Technology in West Africa.” Leonardo Music Journal 10: 41-47. Available: http://muse.jhu.edu/demo/lmj/10.1ligeti.html.

Update History

August 30, 2001: Completed the first version of this page.