Music Box: Turning London's Royal Albert Hall into an interactive instrument. A project done with the artist duo greyworld. Featured at TEDx Albertopolis.
Words: A soundscape application used in performance installations by the artist duo greyworld.
Reel Ear: Music Theory / Ear Training / Improvisation, created for the Reel Space Music School, Valencia, Spain.
Musical Skates: An application which uses sensors to turn ice skates into musical instruments, for skating practice, musical experimentation, and performance.
At this point, this project is on hold, but I am excited for the possibilities. Read on!
My client has developed a system for sensing a number of physical parameters of the movements of ice skaters: their speed, direction, the proximity of the left and right skates to each other, being on the inside or outside edge of the skates, the angle of the edges, the skater's arm position, and so on. The data streams from these sensors are transmitted wirelessly to a receiver at a central computer. The application gathers these streams (from up to 8 skaters at once) and transforms them into various musical sounds, with control over instruments, pitches, volumes, and so on. In this way, a team of skaters can CREATE the music they are skating to, rather than following pre-recorded music...or venture anywhere in between, as each experience can inform the other. The sensor data can be used to control lighting or projections in the ice arena as well, so the skaters could control these aspects of the performance if this was desired.
Combine a really talented team of skaters with choreographers, musicians, show designers, and producers, and this project could really be something!
United Taps Tap Dance Analyzer: An application which senses incoming foot-tapping data from tap dance students, then displays the taps in separate tracks for each student, for timing, rhythm practice, experimentation, and performance.
At this point, this project is on hold, but I am excited for the possibilities. Read on!
Similar to multitrack music applications, this program will allow users to create patterns of note beats, then play them back in a variety of ways, with numerous real-time controls over them. Their incoming tap beats will appear on top of their beat track, so they can see how their timing compares to the set pattern. Up to 16 tracks will be supported, with control over appearance, motion, beat patterns, and sequences of these patterns as larger units.
Many combinations of tracks will be useful---for example, having 8 tracks could mean you have 8 people, it could mean 4 people with 2 tracks each, it could mean 2 people with 4, or other combinations. Different movements can be matched to different tracks, or maybe each of one's limbs follows its own track...like you're tapping on a drum kit, or just on your thighs! The possibilities for rhythmic practice and experimentation, whether through dance movements or other means, really are endless. And the interplay between using the program as a follower and as a creator is so interesting to experience, especially in groups.
Note that the grid track in the program is purposefully limited in one key way: it uses a sixteenth note as its smallest unit of rhythmic time. This means that rhythms like triplets and other more complex subdivisions are not directly possible, at least the way other applications generally handle this. However, if one thinks additively---build complexity by combination, rather than subdivision---many fresh rhythmic ideas are possible. The controls for the beats in each track allow you to make your patterns very dynamic: able to be changed / shifted / randomized / sequenced easily, freely or in time with the overall tempo. This interaction is much like playing an instrument or a video game, and can add another dimension to the experience: with a practiced teacher operating the program while others follow along, that person can can act as coach, facilitator, challenger, conductor, co-player, and more. Or, you can set up your own set of challenges by "playing" a session, then follow along with your motions as it plays back. And you can always change what track or tracks you're following simply by where you look, so actions like having all members of a group do something in unison is as easy as saying "now everyone follow track 4", then "back to your own tracks". In this way, even the simplest beat pattern can be used for many different purposes.
16-step sequencer designed for touch-screens
Sine Wave Explorer
OpenGL / Jitter Gallery
Evolution UC33e Controller Patches
A couple projects that didn't make it past the prototype phase, but were interesting to work on:
3D Model Viewer: An application for orthodontists to view and manipulate 3D virtual models of upper and lower sets of teeth. While designed for this purpose, the viewer can be used to view up to 6 .obj (Wavefront Object Model) files at a time, in different window layouts. I'm looking forward to building this one, because it will be a useful and fun tool for all kinds of 3D exploration, especially using models and grids. Plenty of interesting free models are available online, and free converters can turn lots of formats into .obj (which is required). And you'll be able to texture the models with any image or movie file, which can look amazing...lots of room for creativity there.
Astronomical Bodies: An application that calculates the heliocentric planetary positions for a range of years, listing one measurement per day.
Using a series of equations, this program determines the heliocentric positions of all 9 planets (yes, Pluto is back!) and stores these "Latitude/Longitude" pairs in a text file. These values can then be read by a spreadsheet or database program.
Max/MSP/Jitter (or just “Max”) is a graphical programming environment. You place objects into a “patch” and hook them together in various configurations, based on the inlets and outlets on the objects. Some objects are “black boxes”, which perform a function or set of functions; others are interface elements such as buttons, toggles, sliders, and meters, which display and send out values of various kinds. Connections are of three kinds: messages, signal-rate audio, and matrix video. These data types correspond directly to Max, MSP, and Jitter in the program's name. However, the underlying framework is Max, while MSP and Jitter objects and functions can be thought of as extension libraries operating within Max.
Most everything in Max is active at all times, including when it is in Edit Mode, unless processes are specifically stopped by the user. This allows true real-time testing of functions, unlike most environments in which the code must be compiled before running. Additionally, because the environment is graphical, it is not possible to make the same kinds of syntax errors one can make in text-based code, which eliminates a lot of potential problems. That said, other kinds of errors (such as hooking up cords incorrectly) are certainly possible!
The underlying idea behind Max is to allow users to create interfaces which contain the visible (interactive) and invisible (functional) elements alongside each other. The main elements one uses are numbers and messages; the ways these elements flow through the patch determines what it does. So, for example, an interface slider might output numbers from 0 to 127. These can be used for any parameter which needs a range: the volume level of an audio clip, the amount of red hue in a video segment, the frequency of a sine wave, or which of 128 presets one is activating. As another example, a toggle sends a 0 when off, and a 1 (or another non-zero number if desired) when on. This number can be used to turn audio on and off, to open and close a gate for messages, to activate or deactivate an effect, or to toggle any boolean (true/false) function. While such uses are similar in most programming environments, in Max they are particularly simple and fast to implement, as no extra variables need to be stored, modified, or accessed---all changes occur in the objects themselves, via messages.
Sample Max Course
Assuming 16 weeks, 3 hours of class time per week
Note: The included tutorials can definitely be helpful, but students should avoid basing one's learning on them completely. It is better to build patches from scratch, then look at finished patches once one has a grasp on the fundamentals of the particular objects. Help patches for each object should be studied upon first using any given object, then revisited multiple times as needed.
Weeks 1-2: Getting Started
The Interface: Patching, Objects, Messages, Patch cords, the Object Inspector, the Patcher Inspector, and the Max Window
Making a basic patch: press a button to make a light blink and play MIDI notes; include the capability to change light color, note pitch, and note velocity; explore other parameters which could be changed
Extend the patch by including automatic start with [loadbang] and automatic play with [metro]; include methods to change tempo, show different tempo-regulation techniques; get messages printing in the Max window and in the patch, for monitoring values and debugging
More extensions of this patch as time permits
Weeks 3-5: Diving in
Gradually add various control-based objects, such as [trigger], [gate], [switch], [uzi], [line], [counter], and others
Introduce math operators: traditional ones as well as [scale] and [expr]
Explore more interface objects: drop-down menu, multislider, radio buttons, tabs, matrixctrl, LCD, function, preset
Combine interface objects with control objects to make more complex patches; these can do a variety of things, but simply getting things to blink and make noise can be sufficient to indicate that the logic is correct
Work with the [preset] object to store and recall patch settings; mention [pattrstorage] (an advanced preset system) but don't delve into it yet
Work with encapsulation (subpatches only for now) for organization and cleanliness
Work with hiding objects and patch cords, and Presentation Mode, for more organization and cleanliness
Work with [send] and [receive] to cut down on the number of patch cords needed
A brief demonstration of [thispatcher] and how to control the patch window and objects within it via script messages
Get input from MIDI controllers into Max and explore what they could be used for (beyond the normal uses)
Projects: Create several more complex patches which utilize a number of these new objects. At this point, appearance is not highly important, nor is having the patch “do something interesting”. What matters is an understanding of the interface and the processes behind them. Don't start patches by saying “I want to create something that does X, Y, and Z”. Start with “Here are some objects which I know how to use. Here are some user-interface objects. What can I build with these?” From there, and a basic control design such as blinking lights or playing notes, build an interactive instrument. Gradually introduce more MIDI and graphical interface objects to expand the instrument's palette.
Weeks 6-7: Audio (MSP)
Exploration of audio: signal flow, [buffer~], and control objects
Build a variety of simple synthesizers using the included basic waveforms; include many interface elements to change parameters in real-time
Audio file playback with [sfplay~]: longer clips, typically not looped
Audio file playback with [groove~]: shorter clips, typically looped
Using [waveform~] to control playback
Audio input from microphone or line in
Pitch shifting with [gizmo~]
EQ filtering with [filtergraph~]
Delay lines with [tapin~] and [tapout~]
Signal routing with [gate~], [switch~], and [matrix~]
Other objects as time permits
Project: Build an audio patch which incorporates as many of these elements as possible, while still remaining usable. Most parts will have been built in draft form during the demonstrations of the various objects. Students will add features and expand the patch's possibilities. There must be a synthesis section as well as a loop-playback section, as these are fundamentally different ways of creating audio. Filters and effects should work with both of these sections.
Weeks 8-9: Video (Jitter)
Exploration of video: the Jitter Matrix and matrix data cords
Separating red, green, blue, and alpha channels, and manipulating each
Exploration of Jitter effects (too many to list)
Compositing multiple matrices into one output
Pre-recorded movie playback
Project: Build a “video instrument” patch which includes a number of Jitter effects. Use pre-recorded movies as well as live camera input. Chain effects in any way you wish, including routing effected matrices back into the effects chain for “feedback”. Audio is not necessary for this project, but can be included (if so, keep controls simple).
Week 10: OpenGL and Jitter Physics
A brief introduction to the second side of Jitter, the OpenGL objects for rendering and physics. Basic shapes, importing and morphing models, using textures and lighting, etc. Using [jit.gl.gridshape] for visual structure. Particle systems. The [jit.gl.sketch] object, which allows arbitrary drawing of primitives.
Week 11: Network communication and hardware input (if hardware is available)
A brief introduction to network communication objects: [udpsend], [udpreceive], [jit.net.send], [jit.net.recv]. Ideas for multi-user interaction across computers. Managing latency and data.
Hardware input: Arduino, MAKE Controller, other controllers.
Week 12: Advanced objects and techniques
Demonstrate [pattrstorage] for more advanced preset storage, including interpolation.
Demonstrate encapsulation using objects and [bpatcher]: using separate files for organization, cloning, and layering one's control structures.
Finish showing any other objects or techniques which seem appropriate.
Weeks 13-16: Student-directed projects, with demonstrations during the final week.
Students plan a final project drawing from the entire semester. It can be as broad or as focused as desired, and can draw from previous projects. However, to get a good grade, it must demonstrate advanced ability in several areas: interface design, patch functionality, and the use of objects and techniques beyond that already done in class. It must be bug-free and well-commented, with Help features as appropriate so that a complete novice to Max can use the interface easily, and be published in forms which anyone with a PC or Mac can view the work without issue.
Max is incredible. The more I use it, the more convinced I am that it can really and truly *do just about anything* in the digital realm ... limited only by one's imagination, patience, and dedication. My experimentation with digital media-manipulation / program writing in Flash and Director was enjoyable and fruitful while I did it, but looking back I really wish I had discovered Max first. Of course, all that said, a big limitation is not publishing to mobile platforms. Maybe someday, but that's a very tall order.
Cycling '74 :: the great folks behind the magic
For what I wanted to do---design interfaces for the creation, manipulation, and recording of media---it has been a far more intuitive, flexible, and extensible program to use. Of course there will be pros and cons to any program, and these can change from project to project. But in my experience, Max/MSP/Jitter (MMJ) is great for 90-plus percent of my experimentation---which itself has expanded tremendously, largely due to the ease of working in the environment.
Is it "programming"? Of course. But there's a much bigger discussion there. You could say that writing an email is "programming", but perhaps many don't see it like that. Using software to create new media types, even a simple instant message, is *programmatic interaction through a tool* which has already been programmed (in that sense of the word) by the creators. But inside the program, there are a ton of options for the user, and in large applications, this is pretty much infinite. Areas such as "Customize", "Preferences", or "Macros" push the fuzziness of these boundaries even further ...
... so, my interests have been to explore that spectrum (or network) of possible "connection levels" between the user, the program, and the programmers. Using MMJ allows me to play all three roles. Just like the potentially infinite choices one could go through when organizing a kitchen, planting a garden, decorating a dorm room, writing a song, painting a picture, or typing a letter---to name a few out of a billion---you can choose how you want the interface to look, what you want it to do, what freedoms and limitations it will have, and so on. The most interesting (and challenging) thing is that you are constantly learning while you're doing it :: learning how to make the logic behind this or that interaction, what interface elements are more useful than others (as you play the "user" role and test it), what additional media could be utilized, how best to explain the patch with hints or examples, etc... etc...........
What I love about MMJ is that you can start at any layer and build your own networks of information and functions, gradually digging into more complex possibilities as you learn or think of other things to add. All the while, the interface is growing along with the functionality :: each dynamically determines the design of the other, in many interesting ways ... paralleling the multiple roles one plays as designer : programmer : user.
Often, the design of an interface is quite important in determining how usable it is -- removing extraneous functions can clean things up and make the interfaces more intuitive, even at the expense of features. It's always tough to determine the right balance between something that does a simple function well versus something that does a whole lot, but in a complex interface. This is an eternal trade-off, but one that constantly pushes me to evolve in many areas of creativity and craft.
If you're interested in the fascinating programming environment that is Max, do check out the free, 30-day download on the Cycling '74 site. Student and educational institution discounts abound for this software as well...thank you Cycling!
Also definitely check out the forums :: they have been invaluable to me for years, being the source of many fresh ideas (not to mention clever techniques or solutions to problems). There's the complete range of experience and expertise there, and many example patches get posted all the time. My username on the forums is seejayjames.
If you do get into using Max, do yourself a HUGE favor and take *extra time* to read the manuals, try the tutorials, check out every help patch as you use new objects, and read what messages can be sent to each object. Don't worry about what you do or don't understand at a given point. The more possibilites you try, the more connections you'll make between your ideas. Don't fall into the trap of too much bit-by-bit experimentation. Some of this is obviously necessary to understand how each piece works, and lots of times you just want to play around with a very specific idea...like testing the range of effects on a piece of audio. That is totally fine, and one of the main reasons Max is so powerful and easy to use! But eventually, this can definitely detract you from learning greater ideas in the programming realm, which opens up yet new ideas.
Ideally, creating patches should approach 90% thought and 10% programming, where you know what you can achieve and how, then you think through the best way to do it before you grab the mouse. Once it's made, a good patch is 98% play and 2% tweak.
When you get small functions working, make them nice-looking and save them into a "snippets" file. Then use copies of it in your patches as you need them. This really will save a lot of time, but it will also help you understand an absolutely essential concept :: processes can be as modular as you want. In other words, how you connect controls together can exist in any configuration, but at each point of connection, *that object is simply doing it's thing*, based upon the interaction or messages it receives. And even more basic than that, it's always operating on numbers :: single numbers, lists, streams, numbers that represent text / pixels / samples / MIDI notes / 3D positioning / effect levels / and / pretty / much / everything / else ... so if you understand how to work some number magic, you'll be a true DJ ... ... data jockey!