I’ll start. I’ve posted periodically here about my dream live-looping “theremin for Ableton” project. Here’s the pitch: imagine you’re standing in front of your TV, being watched by Kinect (hooked up to a Windows PC). In one hand you’ve got a microphone, in the other hand a Wiimote. There’s a sphere in the center of the TV. You point the Wiimote at the sphere and pull the trigger; it turns red. Now you’re being recorded. You sing a measure or two and release the trigger. A second sphere splits off of the center one and floats off to the side, vibrating in time with what you sang. That second sphere is the sound you just made. You can point at it and press A to select it, then you can warp its sound (pitch, tempo, whatever) by pressing the cross-pad in different directions and waving your arms around. Hold the trigger while doing this to make the warping stick (e.g. to record the effects you’re applying). Point back at the center sphere (which represents your microphone) to apply effects to your voice or to record additional loops.
Basically, this is a gestural direct manipulation Kinect/Wiimote interface for doing live-looping performance, where the sounds you make become visible things you can grab and further manipulate. It’s kind of like a holodeck symphony, implemented as a Windows XNA app.
Ultimately I’d like the sound spheres/blobs to be colorized based on an FFT spectrum analysis (so you can visually see what the sound sounds like), to have a beat indicator so you can beatmatch different sounds, and to have sound effects alter the visual appearance (apply reverb and the sphere gets some mirror-image copies; apply stereo and the sphere distorts to show where the stereo is going; apply pitch shifting and watch the spectrum change). The PC should also have two video outputs, so you can perform in front of an audience, facing the audience, with a monitor in front of you (showing an over-the-shoulder third-person Kinect wireframe and the sound spheres you’re conducting, including various UI elements – this is what you watch while performing), and a projector in back of you (showing an audience-viewpoint rendering of the same scene, with your wireframe conducting the sound spheres, and considerably more flash and dazzle). The audience should be able to see as well as hear the whole soundscape you’re creating.
It’s been very slow to get going, partly because I know almost nothing about audio processing. So over the last year I’ve been researching Ableton Live (which is cool – I even splurged and bought a copy – but which doesn’t seem to have the real-time flexibility that I want for this), ASIO for low-latency audio, various random audio and FFT libraries (NAudio, Math.NET), ways to get Wiimote data to a PC via .NET (WiimoteLib), and so forth.
I’ve also been thinking hard about how the hell to architect this thing so that it’s reasonably pleasant to prototype with. Got a tiny hierarchical state machine library and a tiny XNA scene graph library, developing plans for how to represent the evolving performance (so it can be saved and replayed), that sort of thing.
I asked around about how to use ASIO from .NET recently, and got two solid pointers to the BASS audio library, which truly seems like the shit. Dropped their sample code into my XNA game app and it worked the first time. Documentation seems good, forums are active, this thing looks like it will save me months. So that’s awesome.
The one big remaining missing piece is the Windows Kinect SDK, which is supposed to be out more or less anytime now. I also need to buy a bluetooth receiver and an actual Wiimote. Once I have those things, all the pieces will be in place, and it’s just a matter of flinging a lot of C# around and prototyping like hell while trying not to get my goddamned right-hand coordinate spaces mixed up. (I have never been that much of a math nut…)
Anyway, that’s the hacking project I can talk about. There’s another one I’m being mentored on at work, but that one is still hush-hush.
What about you all?