Reel Ear: Music Theory / Ear
Training / Improvisation
Words: A soundscape application used in performance installations by the artist duo greyworld. The application uses camera and manual tracking of up to 6 participants in a square space, and based upon their position, creates audio for each person to experience. The space used in the first showing was about ten meters on a side, though this could be a square of any size provided the camera can track the light sources well enough. Rectangle spaces could also be used with a change in how the overall GL scene is scaled/stretched in X or Y.
The sound heard by each participant comes from three layers:
---Participant Words Layer :: allows each participant to record their own phrase that they can place somewhere in the environment. Once placed, these phrases are played back from disk, so more recordings can be made.
---Pre-recorded Words Layer :: consists of words that live or exist at various points in the space, and can repeat or change the words spoken. Any folder with audio files can be loaded into a dropdown menu.
---Four underlying Environments :: Cave, Forest, Water, Wind :: add background sounds as well as moderate the effect levels used on each person's audio. Any .aiff or .wav file can be used.
Participants wear wireless headphones and explore the space through movement; what each person hears depends upon their proximity to the sounds in the other layers. There are global falloff controls for each layer, so each sound group has its own proximity effect :: distance away versus volume level. There are 12 separate channels of audio going out, grouped into 6 stereo pairs; the controller person can monitor any pair or pair combination, or all 6 pairs at once, and can save audio recordings of the overall mix.
Effects (delay and panning) are moderated by the person's position relative to the four Environments, but can also be controlled individually and globally. You can also generate background noise, which really adds to some of the Environments. There's also a 6-voice synthesizer (carrier/phase/ring mod) you can mess with...it's pretty simple, but it has timed ramps for its frequencies, so some really nice evolving sounds are possible.
A visualizer (created using Jitter's OpenGL functions) shows the location of all sounds and participants, and allows manual tracking of participants with the mouse, or with multiple mice. Keyboard controls are also provided for the first four participants. A multi-touch interface could also be used for tracking, given the right communication.
Musical Skates: An application which uses sensors to turn ice skates into musical instruments, for skating practice, musical experimentation, and performance.
My client has developed a system for sensing a number of physical parameters of the movements of ice skaters: their speed, direction, the proximity of the left and right skates to each other, being on the inside or outside edge of the skates, the angle of the edges, the skater's arm position, and so on. The data streams from these sensors are transmitted wirelessly to a receiver at a central computer. The application gathers these streams (from up to 8 skaters at once) and transforms them into various musical sounds, with control over instruments, pitches, volumes, and so on. In this way, a team of skaters can CREATE the music they are skating to, rather than following pre-recorded music...or venture anywhere in between, as each experience can inform the other. The sensor data can be used to control lighting or projections in the ice arena as well, so the skaters could control these aspects of the performance if this was desired.
Combine a really talented team of skaters with a few choreographers, musicians, show designers, and producers, and this instrument could really be something!
United Taps Tap Dance Analyzer: An application which senses incoming foot-tapping data from tap dance students, then displays the taps in separate tracks for each student, for timing, rhythm practice, experimentation, and performance.
The music and dance possibilities for this project are really exciting. Similar to multitrack music applications, this program will allow users to create patterns of note beats, then play them back in a variety of ways, with numerous real-time controls over them. Their incoming tap beats will appear on top of their beat track, so they can see how their timing compares to the set pattern. Up to 16 tracks will be supported, with control over appearance, motion, beat patterns, and sequences of these patterns as larger units.
Many combinations of tracks will be useful---for example, having 8 tracks could mean you have 8 people, it could mean 4 people with 2 tracks each, it could mean 2 people with 4, or other combinations. Different movements can be matched to different tracks, or maybe each of one's limbs follows its own track...like you're tapping on a drum kit, or just on your thighs! The possibilities for rhythmic practice and experimentation, whether through dance movements or other means, really are endless. And the interplay between using the program as a follower and as a creator is so interesting to experience, especially in groups.
The grid-type of track in the program is purposefully limited in one key way: it uses a sixteenth note as its smallest unit of rhythmic time. This means that rhythms like triplets and other more complex subdivisions are not directly possible, at least the way other applications generally handle this. However, if one thinks additively---build complexity by combination, rather than subdivision---many fresh rhythmic ideas are possible. The controls for the beats in each track allow you to make your patterns very dynamic: able to be changed / shifted / randomized / sequenced easily, freely or in time with the overall tempo. This interaction is much like playing an instrument or a video game, and can add another dimension to the experience: with a practiced teacher operating the program while others follow along, that person can can act as coach, facilitator, challenger, conductor, co-player, and more. Or, you can set up your own set of challenges by "playing" a session, then follow along with your motions as it plays back. And you can always change what track or tracks you're following simply by where you look, so actions like having all members of a group do something in unison is as easy as saying "now everyone follow track 4", then "back to your own tracks". In this way, even the simplest beat pattern can be used for many different purposes.
Research Interests in education :: the arts, mathematics, science, and how they all operate and guide interactive media
ambient, interactive, audiovisual experiences :: axion theory
16-step sequencer designed for touch-screens
Sine Wave Explorer
OpenGL / Jitter Gallery
Evolution UC33e Controller Patches
A couple projects that never quite made it:
3D Model Viewer: An application for orthodontists to view and manipulate 3D virtual models of upper and lower sets of teeth. While designed for this purpose, the viewer can be used to view up to 6 .obj (Wavefront Object Model) files at a time, in different window layouts. I'm looking forward to building this one, because it will be a useful and fun tool for all kinds of 3D exploration, especially using models and grids. Plenty of interesting free models are available online, and free converters can turn lots of formats into .obj (which is required). And you'll be able to texture the models with any image or movie file, which can look amazing...lots of room for creativity there.
Astronomy Project: An application that calculates the heliocentric planetary positions for a range of years, listing one measurement per day.
Using a series of equations, this program determines the heliocentric positions of all 9 planets (yes, Pluto is back!) and stores these "Latitude/Longitude" pairs in a text file. These values can then be read by a spreadsheet or database program.
What can I say? I've found my software of choice. The more I use it, the more convinced I am that it can really and truly *do just about anything* in the digital realm ... limited only by one's imagination, patience, and dedication. My experimentation with digital media-manipulation / program writing in Flash and Director was enjoyable and fruitful while I did it, but looking back I really wish I had discovered Max first. It pays off to research these things and to keep up on what's going on...
Cycling '74 :: the great folks behind the magic
For what I wanted to do---design interfaces for the creation, manipulation, and recording of media---it has been a far more intuitive, flexible, and extensible program to use. Of course there will be pros and cons to any program, and these can change from project to project. But in my experience, Max/MSP/Jitter (MMJ) is great for 90-plus percent of my experimentation---which itself has expanded tremendously, largely due to the ease of working in the environment.
Is it "programming"? Of course. But there's a much bigger discussion there. You could say that writing an email is "programming", but perhaps many don't see it like that. Using software to create new media types, even a simple instant message, is *programmatic interaction through a tool* which has already been programmed (in that sense of the word) by the creators. But inside the program, there are a ton of options for the user, and in large applications, this is pretty much infinite. Areas such as "Customize", "Preferences", or "Macros" push the fuzziness of these boundaries even further ...
... so, my interests have been to explore that spectrum (or network) of possible "connection levels" between the user, the program, and the programmers. Using MMJ allows me to play all three roles. Just like the potentially infinite choices one could go through when organizing a kitchen, planting a garden, decorating a dorm room, writing a song, painting a picture, or typing a letter---to name a few out of a billion---you can choose how you want the interface to look, what you want it to do, what freedoms and limitations it will have, and so on. The most interesting (and challenging) thing is that you are constantly learning while you're doing it :: learning how to make the logic behind this or that interaction, what interface elements are more useful than others (as you play the "user" role and test it), what additional media could be utilized, how best to explain the patch with hints or examples, etc... etc...........
What I love about MMJ is that you can start at any layer and build your own networks of information and functions, gradually digging into more complex possibilities as you learn or think of other things to add. All the while, the interface is growing along with the functionality; each dynamically determines the design of the other, in many interesting ways ... paralleling the multiple roles one plays as designer : programmer : user.
Often, the design of an interface is quite important in determining how usable it is -- removing extraneous functions can clean things up and make the interfaces more intuitive, even at the expense of features. It's always tough to determine the right balance between something that does a simple function well versus something that does a whole lot, but in a complex interface. This is an eternal trade-off, but one that constantly pushes me to evolve in many areas of creativity and craft.
If you're interested in the truly fascinating programming environment that is Max MSP Jitter, there is a free, fully-functional 30-day trial available on the Cycling '74 site. Students can also get a 9-month fully-functional version very affordably. Thank you Cycling!
Also definitely check out the forums :: they have been invaluable to me for years, being the source of many fresh ideas (not to mention clever techniques or solutions to problems). There's the complete range of experience and expertise there, and many example patches get posted all the time. My username on the forums is seejayjames.
If you do get into using Max, do yourself a huge favor and take *extra time* to read the manuals, try the tutorials, check out every help patch as you use new objects, and read what messages can be sent to each object. Don't worry what you do or don't understand at a given point; the more possibilites you try, the more connections you'll make between your ideas. Don't fall into the trap (as I continue to do in spite of myself) of too much bit-by-bit experimentation. Some of this is obviously necessary to understand how each piece works, and lots of times you just want to play around with a very specific idea (like testing the range of effects on a piece of audio). Ideally, creating patches should approach 90% thought and 10% programming, where you know what you can achieve and how, then you think through the best way to do it before you grab the mouse. Once it's made, a good patch is 98% play and 2% tweak.
When you get small functions working, make them nice-looking and Save them into a "snippets" file. Then use copies of it in your patches as you need them. This really will save a lot of time, but it will also help you understand an absolutely essential concept :: processes can be as modular as you want. In other words, how you connect controls together can exist in any configuration, but at each point of connection, *that object is simply doing it's thing*, based upon the interaction or messages it receives. And even more basic than that, it's always operating on numbers :: single numbers, lists, streams, numbers that represent text / pixels / samples / MIDI notes / 3D positioning / effect levels / and / pretty / much / everything / else ... so if you understand how to work some number magic, you'll be a true DJ ... ... data jockey!