Ideation:
As someone who is often extremely visually and aesthetically driven, I wanted to push myself and do something new and add new elements to my work. I decided to explore generative music alongside visualization.
Tech Stack:
- P5.js
- ML5
- KNN
- Tone.js
Process:
At first I wanted to do something with bytebeats which is byte music generated purely from math. It's a technique that I had encountered in many advanced shader programs displayed on shadertoy. Within a few hours I realized that I was waaaaaaay too bad at math to utilize this effectively. This was especially true because most documentation on beatbyte is limited and very old and not very accessible. At most there were a few webpages that helped create beatbyte music but no API's or raw code examples and tutorials.
Due to my inability to really utilize beatbyte for music generation I pivoted to the popular tone.js. I wasn't particularly interested in generating real music rather I just wanted to transmute imagery into noise. I mapped frequencies for my synth to generate to the result scores from the labels from the KNN algorithm and went from there.
For the visual aspect of the project I figured I'd brush up on some old classics and play with 2D curl noise and flow fields. I created some particles and generated a flow field for them to move around in with some curl noise applied as well.
As for the KNN and machine learning part of this project, based off Moon's base code I began training a very simple model based on colors. I literally googled a pictures of red, green, blue, yellow, orange and purple and held them up to cover the entirety of webcam when training each label. By training the model on pure color I was able to get a rather unique and relatively randomized output as in realtime usage the video camera would have a variety of colors, especially when moving around or incorporating lots of movement in your video feed the result scores get very jumbled and creating an interesting noisescape.
Live Site:
References:
https://www.youtube.com/watch?v=sZBfLgfsvSk&ab_channel=BarneyCodes
https://www.youtube.com/watch?v=ddVrGY1dveY&ab_channel=DavidBouchard
