Making music using new sounds generated with machine learning

I’m fascinated with this idea, every time we see a new technology come along artists take interesting bits of that technology and turn it into new ways of making music. From ancient instruments… through to the 808 drum machine, transforming the technology into something of artistic merit. My name is Doug, I’m a research scientist
on the Google Brain team and I run a project called Magenta. We’re a research group, dedicated to understanding how to use machine learning in the space of the arts. We’re not trying to make music making easier, we’re trying to build some sort of machine learning tool that gives musicians new ways
to express themselves. We’re using neural networks in a novel sort
of way. We’re using neural networks to generate sound. Not the actual note that’s playing but the sound of the of the instrument. And one of the first projects we worked on was NSynth, which stands for Neural Synthesizer. The NSynth algorithm learns the core aspects of what makes each sound sound like it does and then combine those characteristics, drawing a new sound that’s not just a blending of the two original sounds. OK, so the first surprise for me is that NSynth works at all because it’s technically a very complicated algorithm and one that was hard to train. And so, we’ve needed to go from heavy math and big computers and lots of code to something musicians could play. And so another group at Google called Creative Lab took the sounds from NSynth and created a musical instrument. We call it NSynth Super. That’s like wild to me. Here’s a flute. Here’s a snare. I guess in the middle, this is what it sounds like. Now it does feel like we’re turning a corner of what could be new possibilities. It could generate a sound that might inspire us. The fun part is though you think you know
what you’re doing, there’s some weird interaction happening that can give you something totally unexpected. Wait, why did that happen that way? This is our first foray into actually building
tools for musical creativity. I would love to see people using NSynth Super in ways that surprise us all and gives us some new bit of musical joy in the world.