Context
I worked with artist Seiji Oda to create a tool that allowed him to create real time, generative audio to accompnay his Tai Chi flows
Role
I was the sole developer on this project and collaborated with Seiji on sound design
Inspiration
The inspiration for this project originally came from a 17-year-old YouTube video titled 'Daft Hands'. In it, someone twists and flips their hands to reveal lyrics synchronized to Daft Punk's "Harder, Better, Faster, Stronger." I wondered if there was a way to create an interactive version of this video, where a user could simply mimic the hand shapes and trigger the respective lyric.
I started looking into how this could be accomplished, and came across the MediaPipe library. Using pose landmark detection on a live feed from my web cam, I was able to to build a rough prototype:
As music is increasingly ‘happening’ online, I saw an opportunity to build something for artists that would help them to connect with listeners in real time and in real life. By adding a new, embodied dimension to live performance, we could ‘reclaim’ the visceral feeling of music that sometimes gets lost behind a screen.
In brainstorming how to go about this, I sort of stumbled across Tai Chi as a good use case for pose detection, as the movements are slow and unique and therefore easily picked up by computer vision. I just so happened to see Seiji Oda share a post of his Tai Chi routines, and I reached out to him to collaborate.
The idea, then, was to create a system that would generate/modulate sound in response to Seiji’s movements.
Process
I built the original ‘Daft Hands’ prototype in Javascript (my most familiar language), but for this next iteration I wanted to challenge myself to use Python (both as a challenge and because I had found some Python-specific libraries that might be helpful).
Using a library called Mido, I was able to send signal from a video feed into Ableton’s Max For Live. From there, I mapped each pose landmark (in this case, head, shoulders, knees, etc) to a different granular synth in Ableton, experimenting with automating parameters like filter cutoff, reverb size, and track panning.
With videos that Seiji had sent me, I designed an Ableton session that could be configured to different ambiances/styles, before ultimately meeting with Seiji to film our first live performance. You can watch/listen to the result below: