Week 4: Drawing – Log

One of my favorite songs of last year was “Make Me Feel” by Janelle Monáe. It’s a pop masterpiece, and Monáe expresses her queerness beautifully in the music video.

For this project, I’m using p5.js to make a site site centered around the Make Me Feel music video, with a dynamic music visualizer as the video plays.

View App

Speaking of the music video, let’s show it onscreen. While I first did this with a simple HTML <video> tag & positioned it on top of the canvas with CSS, I soon realized I’d want to work with it in the p5 context. Luckily, p5 has a createVideo function, although oddly it works nothing like sounds or images.

let vid, song, analyzer, fft

function preload() {
  song = loadSound('makemefeel.m4a')
}

function setup() {
  createCanvas(1024, 768)

  vid = createVideo('makemefeel.mp4')
  vid.hide()
  vid.pause()
  vid.position(width / 2, height / 2)
  vid.size(512, 384)

  ellipseMode(CORNER)

  analyzer = new p5.Amplitude()
  analyzer.setInput(song)

  fft = new p5.FFT()
  fft.setInput(song)
}

function mouseClicked() {
  if (song.isPlaying()) {
    song.pause()
    vid.pause()
  } else {
    song.play()
    vid.play()
    vid.volume(0)
  }
  vid.volume(0)
}

function draw() {
  background(0)
  noStroke()

  if (song.isPlaying()) {
    // Volume indicator
    let rms = analyzer.getLevel()
    fill(255)
    ellipse(width / 2, height / 2, 16 + rms * 512, 16 + rms * 512)

    // Frequency graph
    let spectrum = fft.analyze()
    fill('#fae')
    beginShape()
    vertex(0, height)
    for (i = 0; i < spectrum.length; i++) {
      vertex(i + 1, map(spectrum[i], 0, 255, height, 0))
    }
    endShape()
  } else {
    fill(255)
    textSize(72)
    textAlign(CENTER)
    textFont('Futura-Bold')
    text('CLICK TO PLAY', width / 2, height / 2)
  }
}

Great! I discovered through this that the p5 video APIs are pretty basic & not well-documented, but we’re going to continue nonetheless…

Confetti

The video & the rest of the sketch are really disconnected, though. What if we pulled colors out of the video to use in the background?

This seemed pretty tricky, but p5 has functions to help! This tutorial was super useful.

Here’s how they describe the getPixels() helper:

Pixel data in .pixels is arranged such that the red, green, blue and alpha values of each pixel are stored as separate items. The first four items are the RGBA values for the pixel at 0,0; the second set of four items are the RGBA values for the pixel 1,0; the next four are for the pixel at 2,0; etc. When a row of pixels on the screen ends, the pixel data starts over again at 0,1 (and then 1,1 and 2,1, etc).
The two for loops iterate from zero to the width and height in both dimensions. The expression ((y * width) + x) * 4 gives the offset of the four values that correspond to the color of the pixel at x and y. Then, I set the fill color to the red value for that pixel by getting the value at vid.pixels[offset], the green value for that pixel by evaluating vid.pixels[offset+1], and the blue value by evaluating vid.pixels[offset + 2].

Ok, that was intense but makes sense. I decided colored confetti will be this simplest implementation, since the particles can directly map to pixels, no filtering or processing needed.

I looked at their code for making a video pixelated & used it to write a confetti implementation. The main difference is that instead of lining up the pixels over the original video, we want them elsewhere on the screen, a basic version of which we can create with random(0, {width,height}).

// Add confetti of current colors
vid.loadPixels()
for (let y = 0; y < height; y += 16) {
  for (let x = 0; x < width; x += 16) {
    let offset = (y * width + x) * 4
    fill(vid.pixels[offset], vid.pixels[offset + 1], vid.pixels[offset + 2])
    rect(random(0, width), random(0, height), 16, 16)
  }
}

This is a start, but:

Many of the pixels are black, from the background of the video. That’s ugly.
Having them as squares ties them too directly to the pixel idea. Circles?
The random movement is so fast & distracts the video. Can they fade out?

Filtering confetti by color

Let’s first fix the black pixels issue. Basically, if the pixel’s RGB values are [0, 0, 0], we should cut it. So in the nested loops, we can skip any pixels where this is the case:

// Filter out black pixels
if (vid.pixels[offset] == 0) {
	continue
}

Again, good start, but this has issues too. Like the bouncing ball in class, it’s leaving near-black colors, which aren’t great either. Since we need the pixel color for both the filtering & the filling, let’s pull it into a separate variable to simplify the code, too.

// Add confetti of current colors
vid.loadPixels()
const px = vid.pixels
for (let y = 0; y < height; y += 16) {
  for (let x = 0; x < width; x += 16) {
    let offset = (y * width + x) * 8
    // Filter out black pixels
    const rgb = [px[offset], px[offset + 1], px[offset + 2]]
    if (rgb.map(value => value < 24).includes(true)) {
      continue
    }
    fill(rgb[0], rgb[1], rgb[2])
    rect(random(0, width), random(0, height), 16, 16)
  }
}

Except…we also need to filter out white pixels. The first way that came to mind was to check if all the RGB values summed was equal to the sum of [255, 255, 255].

Now, JavaScript doesn’t include a sum() function in its standard library, and I want to avoid adding a utility library like Lodash for the time being. We can make use of the reduce operator, starting at 0, to calculate the sum, like this:

// Filter out white or near-black pixels
const rgb = [px[offset], px[offset + 1], px[offset + 2]]
const white = rgb.reduce((a, b) => a + b, 0) === 255 * 3
const black = rgb.map(value => value < 24).includes(true)
if (white || black) continue

Filtering is finished now!

Making the confetti circular

This is a lot simpler than the filtering. I switched rect to ellipse & added ellipseMode(CORNER) to the setup function to keep the coordinate system consistent with the rectangles we were using before.

Oh wait, changing the ellipse mode broke the volume indicator. Always something…

Changing the volume indicator

Aren’t we getting a little side-tracked? Yes. But that’s how I work, so we’re not waiting to go down this road.

My first thought was to make the the volume indicator super small & in the corner. But that’s boring, & Janelle deserves better.

I had an idea in the meantime to saturate the colors more depending on the volume, but since I think that would require converting RGB to HSL, increasing saturation, & converting back again, let’s skip that for today.

More simply, what if we applied the concept of the volume indicator to the confetti themselves? It’ll be similar code for calculating the size, but have a smaller minimum size, and not give too much influence to the volume. Here was the code for the former volume indicator:

ellipse(width / 2, height / 2, 16 + rms * 512, 16 + rms * 512)

Attempt #1:

const size = 16 + rms
ellipse(random(0, width), random(0, height), size, size)

Actually, rms is usually a value around .03xx, so let’s convert it to a small integer:

const size = 8 + rms * 768
ellipse(random(0, width), random(0, height), size, size)

Fab.

Fading out the confetti

Thinking back the class snow example, can we just add an alpha channel to the fill? As in: fill(rgb[0], rgb[1], rgb[2], 100). Yep, it works!

Coloring the volume graph

Dynamically finding a dominant color out of all the pixels would be memory-intensive & frustrating to code, but luckily my simpler idea first.

I set let dominant = [] before the confetti, then after the filtering, set dominant = [rgb[0], rgb[1], rgb[2]], and changed the frequency graph to use that color. It’s not an ideal implementation, but it’s 90%.

At this point, my “now playing” code looks like this:

let rms = analyzer.getLevel()

// Add confetti of current colors
vid.show()
vid.loadPixels()
let dominant = []
const px = vid.pixels
for (let y = 0; y < height; y += 16) {
  for (let x = 0; x < width; x += 16) {
    let offset = (y * width + x) * 8
    // Filter out white or near-black pixels
    const rgb = [px[offset], px[offset + 1], px[offset + 2]]
    if (rgb[0] === undefined) continue
    const white = rgb.reduce((a, b) => a + b, 0) === 255 * 3
    const black = rgb.map(val => val < 24).includes(true)
    if (white || black) continue
    fill(rgb[0], rgb[1], rgb[2], 128)
    dominant = [rgb[0], rgb[1], rgb[2]]
    const size = 8 + rms * 768
    const particleX = random(0, width / 64) * 64
    const particleY = random(0, height / 64) * 64
    ellipse(particleX, particleY, size, size)
  }
}

// Frequency graph
let spectrum = fft.analyze()
fill(dominant[0], dominant[1], dominant[2])
beginShape()
vertex(0, height)
for (i = 0; i < spectrum.length; i++) {
  vertex((i + 1) * 1.5, map(spectrum[i], 0, 512, height, 0))
}
endShape()

Finishing up

A few things left!

I used the dominant color to change the background of the paused screen:

if (dominant[0]) {
  background(dominant[0], dominant[1], dominant[2])
}

I centered the canvas on the webpage (using flexbox):

body {
  margin: 0;
  padding: 0;
  display: flex;
  min-height: 100vh;
  background: #000;
}
canvas {
  display: block;
  margin: auto;
}

But then made the canvas fullscreen:

function setup() {
  createCanvas(windowWidth, windowHeight)
}

I added a rounded rectangular frame to the video:

// Add rounded frame
fill(0)
rect(width / 4 - 48, 128 + 40, 768 + 96, 512 - 80, 24)

I fixed a few bugs, like the audio & video tracks playing simultaneously (set the video’s volume to 0), adjusted the confetti size some more, & stopped the filtering from generating particles with undefined fills.

This app doesn’t work well on mobile browsers at all, but that’s a bit outside the scope of this prototype.

Conclusion

We did it!!