Moflo - Extracting Rhythm from Video
HistoryThis page is the reference point for my project for Mus 220C. The project builds upon the work I did previously for 220B. In that project I explored using video input to control music with several mappings:
- color -> pitch
- object position -> pitch, loudness
- motion -> tempo
Of all of these the motion->tempo axis is the most engaging, though object position -> pitch/loudness has some potential as well. For this quarter I’ll focus on the rhythmic aspect mostly, and leave the more instrumental aspects out.
New questions
The main goal of this project is to create an unstable feedback loop between a dancer and a music synthesis system.
The 220B version of Moflo showed that video can be used to drive music synthesis.
In 220B I started looking at the following types of tempo interaction:
- video events triggering musical events (engaging)
- locking a tempo to rhytmic motion (needs work)
- “boredom” detectors (overly simple)
Deliverables
The goal of the class for me will be to develop the apparatus a bit more. The following are key steps:
- display chuck rhythm events in the Processing window so we can see how well rhythm tracking is working
- improve the lock-to-motion algorithm, probably by using published beat-detection techniques
- replace the synthesized music from 220B (which emphasized timbre) with samples, midi events or loops so music sounds natural without so much effort
- develop the notion of a loop library and a boredom detector that moves between loops
Week 1 Status
Project overview and direction. Fixing bugs left over from last quarter.
Week 2 Status
Read some online papers on beat detection. Fixed camera timing bug and brighness.
Week 3 - 7
Planned to drop the class so I could focus more on work, but the registrars office isn’t playing along.
Week 8.1
Added plots and instrumentation from Chuck -> processing so I can see what the problems with the triggering are. Observations:
- the previous trigger method where any division of the video space could generate a trigger was not working reliably so i changed the global trigger to be based on the sum of all divisions
- adding the chuck->processing graphs shows that there were a few bugs in the trigger code fixed that
Week 8.2
Added some additional plots (bpm, trigger plot colors) to look at the trigger methods better. Got improved response by reducing the holdoff time to 150msec. This will allow resolution of 8th notes at 180bpm. The 9-10fps we are getting is still a limitation, but the basic system here is good enough to prototype.
Week 8.2