Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation (Ip 2005)

Summary:
Cyber Composer is a music generation system that combines capturing hand gesture commands with music theory to produce an interactive approach to compositions. The paper supplies much background information in music theory, most of which defines variables of music which need to be set in a certain way to achieve a certain musical style. Several rules were introduced that shape music production based on classical music theory. The system determines which diatomic scale to use based on what musical key is specified. Given an initial chord, the degree of affinity between chord pairs is considered when determining subsequent chords. Cadence is avoided in general, allowing the user to trigger it. Harmony notes are used for the melody note at the accented beat to serve as a balance between using too many or too few harmony notes.
The system itself uses two CyberGloves, each with a polhemus receiver, to track hand and finger motion, position, and orientation. Hand signals are translated into motion triggers by the main program module and sent to the melody-generation module where MIDI output is produced according to the style template and music-theory-based rules. The user defines tempo and key before the interactive portion of the composition to be used in background music generation. During the interactive composition, melody notes are triggered when the wrist is straightened. Pitch is controlled by the relative height of the right hand compared to its position during the previous notes. Force of a note is determined by finger extension, while volume is controlled by finger flexion. Closing the left hand fingers triggers cadence. There was no concrete measurement of the system's ability to follow through on claims of allowing laypeople to express music in an interactive, innovative, and intuitive way.

Discussion:
Waving the wrist at every melody note seems like it could get tiring over the duration of a long song. Perhaps fixing the rhythm to remain constant (until recognition of a certain finger indicating that change was desired) would make the composition experience less repetitious. This change makes sense since rhythm is somewhat temporally consistent, meaning that it doesn't change a lot from moment to moment, and a change is usually not followed quickly by another change.
There is no real user study to speak of, but the creative nature of the system does not lend itself well to analysis. The approach is fairly novel, and the gestures control a variety of musical aspects, even if the mapping from gestures to their effects are not the most intuitive. Perhaps a user study should be done to see what gestures people would perform when wanting to signal changes in variables used by Cyber Composer when generating music.
As an improvement to MIDI-based sounds, a system could be devised to play back sound samples of an instrument using a library such as FMOD. For holding longer notes, a loop point within each sound sample could be set to allow arbitrarily long, yet fluid, note samples to be generated.