AI made this music video | What happens when OpenAI's CLIP meets BigGAN?
deep learning machine learning explained neural networks ai artificial intelligence paper deep learning tutorial what is deep learning introduction to deep learning ai generated music video ai music video deep learning music video ai music video generator music video generator openai clip openai clip music video biggan music video clip biggan biggan clip stylegan clip imagenet song imagenet classes lyrics stylegan music gan interpolation be my weasel
#artificialintelligence #musicvideo #clip

I used OpenAI's CLIP model and BigGAN to create a music video that goes along with the lyrics of a song that I wrote. The song lyrics are made from ImageNet class labels, and the song itself is performed by me on a looper.

OUTLINE:
0:00 - Intro
1:00 - AI-generated music video for "be my weasel"
3:50 - How it was made
7:30 - My looping gear
9:35 - AI-generated music video #2
12:45 - Outro & Credits

Code and references: https://github.com/yk/clip_music_video

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 I wrote a song with lyrics made from ImageNet class labels and then I used OpenAI's clip model together with a bigGAN and a back propagation procedure to generate a music video that fits the lyrics of the song. The song is performed on a live looper and the lyrics mean absolutely nothing. I hope you think this is as cool as I do. Enjoy! On a larger screen with my head in a guillotine My hair smells like an old dish rack, my face looks like a used dorm, not my spine is like a horizontal bar. These are just some things you'll find on ImageNet. A thousand cups of joy but mostly things to pack. Be my weasel, be my dick, be my badger on an offshore rig. Find a beagle, catch a slug, bring them all to my whiskey jar. Watch out for the king snake, the vine snake, the green snake, and don't forget the night snake, the sea snake and the pug. Find a beagle, catch a slug, bring them all to my whiskey jar. And here I sit in my rocking chair looking for my purple hair. What's inside that wooden chest? Maybe it is my bulletproof vest. Here aboard a collie cry a bird. Bernie's mountain dog goes by and all the while two hummingbirds stay near. Those are just some things you'll find on ImageNet. A thousand cups of joy but mostly things to pack. Be my weasel, be my dick, be my badger on an offshore rig. Find a beagle, catch a slug, bring them all to my whiskey jar. Watch out for the king snake, the vine snake, the green snake, and don't forget the night snake, the sea snake and the pug. Find a beagle, catch a slug, bring them all to my whiskey jar. Be my weasel, be my dick, be my badger on an offshore rig. Find a beagle, catch a slug, bring them all to my whiskey jar. So how was this all made? See if you want AI to generate you images you have to have a model that learned from a data set. In our case this is a generative adversarial model or a GAN. GANs are amazingly good at producing high quality images. The cool thing about a GAN is that it's a very simple model. The cool thing about a GAN is that what you need to do is you need to sample a point in what's called the latent space and then you'll get out a picture in picture space. Now if you have two points in latent space you can also go from one to the other in a stepwise fashion. We call that interpolation or traversal. If you sequence those pictures one after another it gives you a video of morphing one picture into the other. We came up with a picture for each line of lyric and then we simply traversed the latent space in sync with the music in order to produce this video. But how did we even get the initial pictures and how did we make them fit the text? That's where OpenAI's CLIP model comes in. So CLIP is a model that takes a piece of text and a picture and it will give you a number telling you how well the two fit together or not. Now that in itself will not be useful but the useful part comes when you realize that the picture part of the pipeline is fully differentiable. That means we can back propagate the error signal all the way to the image space. So what we do in practice is we take CLIP and we put a piece of text, in our case one line of lyrics, for the picture. We don't just put a picture we actually put the output of a GAN. In our case we use BigGAN that has been trained on a variety of images and can produce amazing images by itself. We take the output of BigGAN and feed it into the input of CLIP and now that we have all of this we back propagate the error that CLIP tells us through the image part of CLIP, through the GAN into the latent space of the GAN. So in essence we start off with a random picture that might not fit the text at all but then through back propagation over many hundreds of steps we find a point in the input space of the GAN that more and more and more makes the CLIP model happy. So this doesn't always give you very realistic images, however it usually gives you pretty cool images. Like this one is the spine being a horizontal bar, not exactly horizontal but still very very cool. And this here is the face being a used doormat. I think this is amazing. So we feed each line of lyrics through this system, get out a point in the latent space that gives us a picture that is fitting to that line of lyrics. And then with all these points in the latent space all we need to do is traverse them in order synchronized up with the music and we have ourselves a music video. So for the song itself I took ImageNet lyrics and made them into a song text. This isn't because I'm superbly musically talented or anything but usually YouTube and music copyright aren't best friends. I just wanted to avoid all of that stuff and so I came up with my own song. So the lyrics mean absolutely nothing, there's no hidden meaning. I struggled already enough to actually find some rhymes and yeah that's what came out. The song is played in a loop fashion so all the songs are produced by me in some form or another. My gear is I use a Boss VE2 as a voice processor for harmonies. Though I only use it at the very end in this song. I use a Boss RC500 for looping. That's pretty new to me and I still have my troubles with it. And a Boss Octave OC5 pedal. In order to simulate a bass with my guitar. My guitar is a little Martin electroacoustic guitar. It sounds pretty good honestly. The flaw in this setup is probably the microphone I used to record this with as it is an iPad microphone and I didn't have anything else. I guess I could have used this one. Yeah I was pretty stupid for not thinking of that. I can't whistle anymore. And yes I did buy this combo after I saw Ed Sheeran perform live. Absolutely amazing. So usually I'm pretty comfortable playing in front of people. I have terrible stage fright but I do overcome it pretty quickly. Cameras is a different thing. As soon as a camera is rolling like my brain just turns off. So this was certainly my 20th attempt or so at recording this song and not even now I have it down. So forgive a little bit of cracks in voices and my whistling was a bit tired at this point. I hope you still enjoy it. I'm going to let the play the song one more time with a different generation of music video. Girl and man, my sleeping bag, all dressed up in my shower cap. Soon I'll be on a larger screen with my head in a guillotine. My hair smells like an old dish rack, my face looks like a used dorm. That my spine is like a horizontal bar. These are just some things you'll find on ImageNet. A thousand cuts of joy, but mostly things to pet. Be my weasel, be my pig, be my badger, on an offshore rig. Find a beagle, catch a sloth, bring them all to my whiskey jug. Watch out for the king snake, the vine snake, the green snake. And don't forget the night snake, the sea snake and the pug. Find a beagle, catch a sloth, bring them all to my whiskey jug. And here I sit in my rocking chair, looking for my purple hair. What's inside that wooden chest? Maybe it is my bulletproof vest. I hear a border collie cry, a birdie's mountain dog goes by, and all the wild two hummingbirds stay near. Those are just some things you'll find on ImageNet. A thousand cuts of joy, but mostly things to pet. Be my weasel, be my pig, be my badger, on an offshore rig. Find a beagle, catch a sloth, bring them all to my whiskey jug. Watch out for the king snake, the vine snake, the green snake. And don't forget the night snake, the sea snake and the pug. Find a beagle, catch a sloth, bring them all to my whiskey jug. Be my weasel, be my pig, be my badger, on an offshore rig. Find a beagle, catch a sloth, bring them all to my whiskey jug. Be my weasel, be my pig, be my badger, on an offshore rig. Find a beagle, catch a sloth, bring them all to my whiskey jug. Thank you so much for watching. Of course, this is not all my work. It's built upon the work of many great people. And I'll link to as much as I can in the description of the video. So please check this out. A lot of people have worked very hard. And I'm simply building on top of them. And the same people are actually pushing the state of the art of what's possible with the clip model to an entirely new level that you wouldn't believe how cool this is. So check it out. I've also linked my code that I've used to produce the music video. You can produce your own if you want to or play around with it. Special thanks to JR for helping me with the code, to Lance for editing and to you for watching. Ciao!
 I wrote a song with lyrics made from ImageNet class labels and then I used
 OpenAI's clip model together with a bigGAN and a back propagation procedure
 to generate a music video that fits the lyrics of the song. The song is performed
 on a live looper and the lyrics mean absolutely nothing. I hope you think this
 is as cool as I do. Enjoy!
 On a larger screen with my head in a guillotine
 My hair smells like an old dish rack, my face looks like a used dorm, not my spine
 is like a horizontal bar. These are just some things you'll find on
 ImageNet. A thousand cups of joy but mostly things to pack. Be my weasel, be my dick,
 be my badger on an offshore rig. Find a beagle, catch a slug, bring them all to my whiskey jar.
 Watch out for the king snake, the vine snake, the green snake, and don't forget
 the night snake, the sea snake and the pug. Find a beagle, catch a slug, bring them all to my whiskey jar.
 And here I sit in my rocking chair looking for my purple hair. What's inside
 that wooden chest? Maybe it is my bulletproof vest. Here aboard a collie cry a bird.
 Bernie's mountain dog goes by and all the while two hummingbirds stay near. Those are
 just some things you'll find on ImageNet. A thousand cups of joy but mostly things to pack.
 Be my weasel, be my dick, be my badger on an offshore rig. Find a beagle, catch a slug,
 bring them all to my whiskey jar. Watch out for the king snake, the vine snake, the green snake,
 and don't forget the night snake, the sea snake and the pug. Find a beagle, catch a slug, bring them all to my whiskey jar.
 Be my weasel, be my dick, be my badger on an offshore rig. Find a beagle, catch a slug, bring them all to my whiskey jar.
 So how was this all made? See if you want AI to generate you images you have to have a model that learned from a data set. In our case this is a generative adversarial model or a GAN. GANs are amazingly good at producing high quality images. The cool thing about a GAN is that it's a very simple model.
 The cool thing about a GAN is that what you need to do is you need to sample a point in what's called the latent space and then you'll get out a picture in picture space. Now if you have two points in latent space you can also go from one to the other in a stepwise fashion. We call that interpolation or traversal.
 If you sequence those pictures one after another it gives you a video of morphing one picture into the other. We came up with a picture for each line of lyric and then we simply traversed the latent space in sync with the music in order to produce this video. But how did we even get the initial pictures and how did we make them fit the text?
 That's where OpenAI's CLIP model comes in. So CLIP is a model that takes a piece of text and a picture and it will give you a number telling you how well the two fit together or not. Now that in itself will not be useful but the useful part comes when you realize that the picture part of the pipeline is fully differentiable. That means we can back propagate the error signal all the way to the image space.
 So what we do in practice is we take CLIP and we put a piece of text, in our case one line of lyrics, for the picture. We don't just put a picture we actually put the output of a GAN. In our case we use BigGAN that has been trained on a variety of images and can produce amazing images by itself.
 We take the output of BigGAN and feed it into the input of CLIP and now that we have all of this we back propagate the error that CLIP tells us through the image part of CLIP, through the GAN into the latent space of the GAN.
 So in essence we start off with a random picture that might not fit the text at all but then through back propagation over many hundreds of steps we find a point in the input space of the GAN that more and more and more makes the CLIP model happy.
 So this doesn't always give you very realistic images, however it usually gives you pretty cool images. Like this one is the spine being a horizontal bar, not exactly horizontal but still very very cool.
 And this here is the face being a used doormat. I think this is amazing.
 So we feed each line of lyrics through this system, get out a point in the latent space that gives us a picture that is fitting to that line of lyrics.
 And then with all these points in the latent space all we need to do is traverse them in order synchronized up with the music and we have ourselves a music video.
 So for the song itself I took ImageNet lyrics and made them into a song text. This isn't because I'm superbly musically talented or anything but usually YouTube and music copyright aren't best friends.
 I just wanted to avoid all of that stuff and so I came up with my own song.
 So the lyrics mean absolutely nothing, there's no hidden meaning. I struggled already enough to actually find some rhymes and yeah that's what came out.
 The song is played in a loop fashion so all the songs are produced by me in some form or another. My gear is I use a Boss VE2 as a voice processor for harmonies.
 Though I only use it at the very end in this song. I use a Boss RC500 for looping.
 That's pretty new to me and I still have my troubles with it. And a Boss Octave OC5 pedal.
 In order to simulate a bass with my guitar.
 My guitar is a little Martin electroacoustic guitar. It sounds pretty good honestly.
 The flaw in this setup is probably the microphone I used to record this with as it is an iPad microphone and I didn't have anything else.
 I guess I could have used this one. Yeah I was pretty stupid for not thinking of that.
 I can't whistle anymore.
 And yes I did buy this combo after I saw Ed Sheeran perform live.
 Absolutely amazing. So usually I'm pretty comfortable playing in front of people. I have terrible stage fright but I do overcome it pretty quickly.
 Cameras is a different thing. As soon as a camera is rolling like my brain just turns off.
 So this was certainly my 20th attempt or so at recording this song and not even now I have it down.
 So forgive a little bit of cracks in voices and my whistling was a bit tired at this point.
 I hope you still enjoy it. I'm going to let the play the song one more time with a different generation of music video.
 Girl and man, my sleeping bag, all dressed up in my shower cap.
 Soon I'll be on a larger screen with my head in a guillotine.
 My hair smells like an old dish rack, my face looks like a used dorm.
 That my spine is like a horizontal bar. These are just some things you'll find on ImageNet.
 A thousand cuts of joy, but mostly things to pet.
 Be my weasel, be my pig, be my badger, on an offshore rig.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 Watch out for the king snake, the vine snake, the green snake.
 And don't forget the night snake, the sea snake and the pug.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 And here I sit in my rocking chair, looking for my purple hair.
 What's inside that wooden chest? Maybe it is my bulletproof vest.
 I hear a border collie cry, a birdie's mountain dog goes by, and all the wild two hummingbirds stay near.
 Those are just some things you'll find on ImageNet.
 A thousand cuts of joy, but mostly things to pet.
 Be my weasel, be my pig, be my badger, on an offshore rig.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 Watch out for the king snake, the vine snake, the green snake.
 And don't forget the night snake, the sea snake and the pug.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 Be my weasel, be my pig, be my badger, on an offshore rig.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 Be my weasel, be my pig, be my badger, on an offshore rig.
 Find a beagle, catch a sloth, bring them all to my whiskey jug.
 Thank you so much for watching. Of course, this is not all my work.
 It's built upon the work of many great people.
 And I'll link to as much as I can in the description of the video.
 So please check this out. A lot of people have worked very hard.
 And I'm simply building on top of them.
 And the same people are actually pushing the state of the art
 of what's possible with the clip model to an entirely new level
 that you wouldn't believe how cool this is. So check it out.
 I've also linked my code that I've used to produce the music video.
 You can produce your own if you want to or play around with it.
 Special thanks to JR for helping me with the code, to Lance for editing
 and to you for watching. Ciao!