Virtual Instrumentation



I started on the design of the Virtual World, and on learning how to play it, within only a month of the performances, so I had to become completely immersed in the creation process.

I had originally thought the piece would be an elaborate VR "demo", or explication, with clear visual cues for the music, easy to use interfaces, and lots of funny, Rube Goldberg-like tricks. But, as I worked on the World, a mood, or an essence, started to emerge, and furthermore it was true to my emotional and spiritual experience at the time. This was unexpected and exciting, even if the content was not cheerful. So, I went with a darker and more intuitive process instead of falling in line with the more familiar computer culture of clarity and light humor. There have only been rare occasions when I felt I was programming in an intuitive way, and this was one.

Don't expect the instruments to be immediately understandable, or imagine that they are easy to play. Rather, they emerged from a creative process I cannot fully explain, and I learned to play them. I don't think the two esthetics I'm distinguishing must be mutually exclusive, of course, but the intuitive side of the equation can't be reliably willed into action. A synthesis of clarity and mood will come by grace, when it comes.

The first instrument played is called the Rhythm Gimbal. A gimbal is a common mechanical construction of a hierarchy of rotating joints. The Rhythm Gimbal resembles a gyroscope. When it is still it is completely white and completely silent. When I pick it up and move it, it begins to emit sound. Actually the sound is created by the rings rubbing against each other- they also change color at contact. Once set in motion, the Rhythm Gimbal will slow down, but will take a long time to stop completely. If I give the Rhythm Gimbal a good spin as I release it, it emits an extra set of noises which are more tinkly, and which slow down as the instrument winds down. Thus, unless I carefully release it without any spin, it will continue to make sounds when I'm not looking at it. The "background" sound heard while I am playing the other instruments comes from the Rhythm Gimbal.

The primary (non-tinkly) Rhythm Gimbal sound is an interpolation of a choir and an orchestra and some other stuff. The harmony is generated by the momentum with which internal parts of the instrument hit each other after it has been released by the hand. Each ring transmits spin to the ring outside it, creating a complex motion, like pendulums hung on pendulums. The rings have beads on them. When the beads collide, they change color, and also force a change in harmony. You know those old attractions at amusement park arcades, where you hammer a target on the ground with a giant mallet and see how high you can send a puck on a big vertical ruler? The internal collisions of the Rhythm Gimbal are flinging virtual pucks around the circle of fifths, and up the harmonic series, in much the same way. A note is added to the harmony when the two types of puck reach it approximately at once. All the harmony, and all the rhythmic texture, comes out of this process. A slow twist of the Gimbal will tend to choose notes that are close to each other harmonically, such as musical fifths. What's interesting about the Gimbal is the music that it turns out when it is spun with a greater energy. There is a range of harmonic and textural style that can be explored by practicing the spinning of the Gimbal, varying from a very open, harmonic, and calm sound, to a crazed dissonance. My favorite is the in-between zone, which can sound like a cross between late Scriabin and the Barber Adagio (no kidding).

I was almost appalled for a while at how good this simple gizmo was at generating harmony. Is this all a composer's brain does? But the Gimbal can't be described properly as an algorithmic music generator. For example, I don't think an explicit style of initialization could be used to find the right parameters to make it sing. There is a necessary element of intuitive performance in the weird harmonies of this curious instrument.

Every note of The Sound of One Hand is generated by my hand movements, as they are transmitted through the virtual instruments: There are no pre-determined sequences or groupings of notes. The musical content is entirely improvised with the sole exception of the timbral range of the instruments. This does not mean that I can make any arbitrary music, any more than I could with any other musical instruments. I can't get a specific chord out of the Rhythm Gimbal reliably, but I can get a feel out of a chord progression, because I can influence when chords change and how radical the change will be. This does not feel like less control to me, but rather like a different kind of control. The test of an instrument is not what it can do, but: Can you become infinitely more sensitive to it as you explore and learn? A piano is like this. Only some mental objects can fail this test. A good instrument has a depth that the body can learn and the mind cannot. I believe it is entirely possible for the mind to invent such instruments.

Hidden mechanisms in Virtual Reality are just invisible objects. While I was developing this World, I would make the harmonic structure visible- it looks like a bunch of notes crawling on rings and up a pole. I made it mostly invisible for the performance as a visual design decision. There is one part which is still visible, though, and that is the large blue ring with tuning forks on it. Each of the tuning forks has a T-shaped thing on the base and rings on the arms. These objects are the storage of the current legal tonic and chords for progressions. You can see them moving as the harmony changes.

The CyberXylo is a mallet instrument. It's notes are taken from the tuning forks on the blue ring, so it is always harmonious with the Rhythm Gimbal. The mallet retains angular momentum, with some friction, when it is released. Thus it is possible to set it spinning so that it will continue to hit the keys of the CyberXylo on its own for a while. The spin is of poor mathematical quality: it increments rotations instead of using quaternians. This creates wild, unnatural spinning patterns. With practice, enthusiastic spins of the mallet close to the keys can be a source of remarkable rhythms.

The Cybersax is the most ergonomically complex instrument. When the instrument is grabbed, it turns to gradually become held by your hand, and tries to avoid passing through fingers on the way. Once you are holding it, the positions of your virtual fingers continue to respond to your physical ones, but are adjusted to be properly placed on the sax keys. This is an example of a "simulation of control" that is critical in the design of virtual hand tools, especially when force-feedback is not available.

There are three musical registers, soprano, alto, and bass, located along the main tube. Each register consists of a set of shiny sax-like keys. The notes played by the keys come from the current set of legal notes defined by the Rhythm Gimbal, so it will not clash with the other instruments. Once grabbed, it is possible to slide between registers by jerking the hand towards a targeted register. The momentum of your slide helps determine which notes will be associated with the keys until you slide again (if, for example, you approach the soprano register from the alto with greater force you will choose a set of notes that are higher up in pitch). You can play freely without dropping the horn by mistake (this was a hard quality to achieve). In the upper register it is possible to play two melodies at once, by modulating with the thumb. The orientation, or twist, of the horn in space controls the timbre, mix, and other properties of the sound. Other elements of the design include the obscene, wagging tail/mouthpiece and the throbbing bell. The Cybersax sound and geometric construction were partly inspired by a bizarre bamboo saxophone I have that came from Thailand. It is jointed at the top just like the Cybersax's tail.

Computer music must use instruments which are built out of concepts of what music is. This is a drastic departure from the "dumb" instruments of the past. A piano doesn't know what a note is, it just vibrates when struck. A sensitivity, and a sense of awe, at the mystery that surrounds life is at the heart of both science and art, and instruments with mandatory concepts built in can dull this sensitivity by providing an apparently non-mysterious setting for activity. This can lead to "nerdy" or bland art. It is interesting to hide oneself behind a piano, as opposed to a computer, but only because a piano is made of resonant materials, not of concepts. In order for computer art, or music, to work you have to be extra careful to put people and human contact at the center of attention.

I was delighted to discover that The Sound of One Hand created an unusual status relationship between the performer, the audience, and the technology. The usual use of rare and expensive high technology in performance is to create a spectacle that elevates the status of the performer. The performer is made relatively invulnerable, while the audience is supposed to be awestruck. This is what rock concerts and the Persian Gulf War have in common. The Sound of One Hand creates quite a different situation. The audience watches me contort myself in all manners as I navigate the space and handle the virtual instruments, but I am wearing EyePhones. Five thousand people watch me, as I display sometimes awkward poses, but I can't see them, or know what I look like to them. I was vulnerable, and very human, despite the technology. This created a more authentic setting for music. If you have played music, especially improvised music, in front of an audience, you know the kind of vulnerability I am talking about, the vulnerability that precedes an authentic performance.

About that contorting... I am using point-flying in the performance. This is a technique of navigating where you point where you want to go and this causes you to fly there. I dislike point-flying in industrial applications of VR. It requires skill and uses up your hand. I used it in this case because I did want the un-constrained, skillful type of navigation; It allowed me to choreograph a tour of the asteroid along with the performance. I was completely shocked and embarrassed when I got lost in my own World during one of the performances!

Another human element of the piece is its physicality. The Sound of One Hand is in the tradition of the Theremin in that the interface is primarily physical instead of mental. Although the instruments were made of information, the music was primarily made of gesture.

The equipment I used was, for the most part, not state-of-the-art, but about a year out of date. The synthesizers and Head Mounted Display were '92 models, but the graphics engine, tracker and DataGlove were all older. I think you have to actively avoid using the latest gear, in doing art, to avoid getting caught up in technology for its own sake.

(The HMD I used deserve to be noted, since the performance will probably turn to be the last time it will have been seen. I used a prototype XVR EyePhone from VPL, my favorite HMD thus far, which is apparently not going into production. Sigh.)

The software was quite current, however. The piece was written entirely in Body Electric, a visual programming language for Virtual Reality. I am extremely fond of this software working environment, which was designed primarily by Chuck Blanchard. You hook up visual diagrams to control what happens in the Virtual world and see the effect immediately. All the music and physics was done in "B.E."; I could never have made this thing in "C". Visually and sculpturally, the World took advantage of every trick available for real time rendering, including radiosity, fog, texture mapping, environment mapping, and morphing. The color "flaking" effect results from a bug seen when the color of hardware fog on the SGI is gradually changed (I set up a very slow-moving bouncing ball in the cylinder of red/green/blue color space as a chooser for the fog color). I sculpted all the parts of the world except for the illuminated skeletal hand that sprouts from the asteroid wall, which is from a Magnetic Resonance scan of a patient's hand taken at the Veterans Administration hospital in Palo Alto. It was originally used in research on surgical simulation. Just for the record, the asteroid is hollow, and about twelve feet in diameter, although the dense fog makes it look and feel immense. It has a big crack in the side, through which a pair of fireflies constantly frolic, and along with the skeletal hand, it has big red ginger plant growing inside, and also a few spotlights. The instruments are generally kept inside. During the performance I spin the Gimbal at one point and fly out through the crack for a while to let the audience see how lonely the asteroid is, surrounded by absolute void.

The sounds were created on two sampler/synthesizers. I decided not to use the wonderful 3D sound capabilities of Virtual Reality, since they are intended primarily for headphone use, and I didn't want the audience to be trying to hear something.

In many ways, The Sound of One Hand was a bigger leap into the unknown than all of the weird "experimental" performances I had been involved in New York in the late 70's. I had absolutely no idea if the piece would take on a mood or a meaning or if the audience would find the experience comprehensible. The performance turned out to be a cheerful, therapeutic event for me. It was a sort of a technological blues, a bleak work that I could play happily. It was a chance to work on a purely creative project with the VPL family, a chance to treat all of VPL's stuff as a given set of (reliable!) raw materials instead of as work to do, a chance to practice what I preach about virtual tool design, a chance to use VR just for beauty, and a chance to be musical in front of my ridiculously political professional peer community. It was also a celebration of not having to run VPL anymore. The audiences were incredibly responsive, and I didn't hear anyone describe the piece as a demo; It was experienced as music. We had a blast putting on this performance. Hope the tape gives you some idea.


This article originally appeared in the 10th Anniversary issue of the Whole Earth Review.

Go back to Jaron's home page.