Interactive DAW

Where it came from

We came in knowing we wanted to do a hardware project, and when looking at the available hardware, we found an ultrasonic distance sensor and the first thing that came to mind was a theremin. A musical instrument designed to change pitch and play notes via hand gestures. So then we discussed and came to an idea of making digital audio workstations (DAWs) more interactive by making it possible to choose instruments, record, and play them all with just your hands, one tapping beats the other controlling pitch, something not that far from the theremin inspiration.

What it does

Interactive DAW lets you make music with two hands using only a webcam, laptop and our distance sensor pi setup. The camera hand selects instruments with different hand gestures, triggers quantized hits with a pinch, and a fist changes modes between instrument select and play mode, and a special gesture in play mode allows for recording and playback to the DAW directly. The pitch hand controls note pitch by distance: hold it at a certain distance, then pinch with the camera hand to produce the corresponding frequency and hold for note length.

How it works

The laptop node runs OpenCV and MediaPipe Hands behind a small state machine: instrument select, then play, then recording. Finger count chooses the instrument, a pinch triggers or holds a note, and a fist transitions between states. Events are timestamped and sent into the DAW through a MIDI adapter. We used Reaper as the base DAW with loopMIDI creating the MIDI ports, which let Reaper respond to recording, playback and instrument selection through its Actions system.

The pi node listens to the ultrasonic sensor through pigpiod and turns the readings into numbers, filtering implausible values and cutting noise so that hand movements register cleanly. Distances go from the pi to the laptop over OSC across an ethernet link between the two.

On the hardware side the sensor runs on a Raspberry Pi with a voltage divider on the ECHO pin down to 3.3 V and a common ground, and its readings are smoothed and mapped to pitch. That module plugs into the software path without touching the camera side at all.

What was hard

The Raspberry Pi setup ate a lot of our time. Flaky card readers and a stubborn micro SD made flashing and booting painful, and getting into the pi headless without an easy way to ssh in was its own problem until we found an ethernet cord.

The other real hardware problem was power. The sensor’s ECHO pin puts out 5 V and the pi’s GPIO takes 3.3 V, so we went digging through the spare hardware for resistors that would give us a two thirds ratio, and got lucky enough to find three 1k ohm resistors to build the divider out of.

Integration was the part we expected to be hardest, and it was. We planned out how the sensor, camera, controller and DAW would connect before building it, and moved pieces around to keep latency down, including running the camera off the laptop rather than the pi.

What’s next

More instrument states, since we only had time for four. More control for the player: mute, clear, undo, volume, reverb. Running the whole thing independently of a laptop and against any DAW rather than just ours. And there is no reason this has to stay hand controlled, making music from dance or whole body movement is a real possibility for the same setup.