Jukebox: A Generative Model for Music (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper music vae vq-vae latent codes quantization sound lyrics sinatra kanye transformer openai
This generative model for music can make entire songs with remarkable quality and consistency. It can be conditioned on genre, artist, and even lyrics.

Blog: https://openai.com/blog/jukebox/
Paper: https://cdn.openai.com/papers/jukebox.pdf
Code: https://github.com/openai/jukebox/

Abstract:
We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multiscale VQ-VAE to compress it to discrete codes, and modeling those using autoregressive Transformers. We show that the combined model at scale can generate high-fidelity and diverse songs with coherence up to multiple minutes. We can condition on artist and genre to steer the musical and vocal style, and on unaligned lyrics to make the singing more controllable. We are releasing thousands of non cherry-picked samples, along with model weights and code.

Authors: Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Alright, so what you're hearing is the OpenAI jukebox. This paper came out and it is a surprisingly good quality generative model for music, including lyrics, so including singing, which I believe is pretty novel. And the fact that it works so well and has musical consistency throughout entire songs is something that is very, very novel and cool. Alright, so we're looking at the paper, it's called Jukebox, a generative model for music, and it's by Profola Darival, Heiwo Jun, Christine Pine, Jongwoo Kim, Alec Radford, and Ilya Sutskever of OpenAI. So the model here is not very novel, but I think the way they set it up makes a lot of sense and we'll quickly go through it and look at what comes out of this. This is not a very technical paper, but I think it's well written and their main thing is these VQVAE models. So these models, you might know what a variational autoencoder is. So in a variational autoencoder I have an input, let's say an image, here of our typical cat, and I put it through what is called an encoder in order to get a hidden representation. Now this hidden representation, usually called something like Z or H, is then in a classic autoencoder I would here have a decoder and then I train that decoder to match that original picture as closely as possible. So I train these two neural networks, the encoder and the decoder, to just basically compress to this representation and then reproduce again the original image. And thereby I think I can learn a good hidden representation. Now in a variational autoencoder what happens is there is an in-between step, namely this Z representation here is not directly fed to the decoder, but this representation is actually used to parameterize Gaussians. So in the easiest case, let's say we have six dimensions here in the hidden representation, the first two are used to parameterize the first Gaussian, one is going to be the mean and one is going to be the standard deviation of that Gaussian, and the second two are going to parameterize the second Gaussian and the third are going to parameterize the third one. So we have now three Gaussians and then from those three Gaussian distributions we're going to sample a three-dimensional vector and then we're going to feed that three-dimensional vector to the decoder. So every input image in essence is giving us a distribution in the latent space and not just a single vector, it's just giving us a vector that describes an entire multivariate Gaussian distribution. And then we train the decoder basically to reconstruct the encoder given that distribution. So the variational autoencoder has improved over the classic autoencoder because it tends to circumvent some of the shortcomings, but still variational autoencoders have their problems and in terms of generative model for images, people as you know have gone to things like GANs and so on. But here this new thing is called the VQVAE and that's because I believe this stands for vector quantization. Not entirely sure honestly, but so what it does is it takes the input right here and again it maps it to a latent code. And these latent code I believe are called H. So it maps it to a vector called H. Now here is where it gets different. We do have a so called codebook here. So a codebook is just a list of vectors. This is the codebook and these are here called E. So what we'll do is we'll simply look for H which one's the closest neighbor in the codebook that we have of hidden vectors and we'll map it to that and say this here is the closest neighbor. So we'll basically quantize the hidden representation to these codes. So we end up only with vectors that are these codes. And now instead of saving the vector H, first of all we get a super compressed representation because if these are like 16 codebook vectors we can just simply enumerate them. And then we can simply encode the image as one or a sequence of these indices here. But second of all it tends to bring a more kind of a diversity and accuracy if we then decode from these code vectors. Now of course everything is trained here. So the encoder is trained and the codebooks themselves are also trained and the decoder is also trained to give you the maximum kind of benefit. So this is what is described here. What they have is first of all their loss for these VQVAEs and you'll see how they are using them to generate music in a second. Their loss is part this reconstruction loss which you can see here. This is the original image. How far is it away from the decoded hidden quantized representation? This E here, this is the quantized codebook vector that belongs to the hidden representation of X. So this is your standard reconstruction loss. The next part of the loss is this codebook loss. Now the codebook loss is for training these codebook vectors. So it basically pulls the codebook vector closer to the actual hidden representation. So that is where you train these. This here is a stop gradient. So basically it just means that you want your codebook vectors to be better representing of the data that you feed in because otherwise it would be useless codebook vectors. And the third part of their loss is this commit loss right here. And that is exactly the opposite. Now you put the stop gradient on the codebook vector and you simply want to pull the hidden representation closer to the codebook vector such that, imagine the encoder must learn to approximately hit one of these codebook vectors. Otherwise it cannot really learn something meaningfully. It must learn to deal with the codebook vectors that are there and to approximately map the input into the vicinity of one of the codebook vectors. Otherwise there is no information flowing. So that is how you train things. You train the encoder and the decoder to reconstruct. You train the codebook vectors to represent the data and you also train the encoder to make good use of the codebook vectors. Basically now that I think about it I am pretty sure in the reconstruction loss you might only train the decoder because you do have this quantization step in between in this thing here. So technically you could not back propagate through that. So that is how you train the individual parts of a VQVAE and we will see how they use it in order to produce music. Alright, so what they do is they start off with a sample of music and they send it through this thing, through this architecture and at the end they are trying to reconstruct the same thing they had at the beginning. So you see this overarching autoencoder architecture. That is why it is an autoencoder. You try to reconstruct your input and thereby you try to learn something about the data because a model that can compress and then uncompress all of your data has learned something useful about it. Now you have these VQVAEs here in the middle but you have them at different scales. So you will have three of those here. There is a coarse scale one, middle scale one and a high frequency one. So three VQVAEs right here. And you train each one of them separately. And the difference between them is that because this is a continuous signal you cannot just encode a continuous signal because it is an audio waveform. What you have to do is you have to go through it in some sort of a stepwise fashion. You have to divide it into individual pieces and encode each of those pieces as one hidden vector, or one element of the codebook. And the size here, the precise size of how you go through the audio, that is different between these different scales. So in this scale right here you go through the audio in very small steps. This as you can imagine gives you the best reconstruction. So this is the audio. The audio is like this. And you just take part of it, like from here to here, and you encode it into a single hidden vector, which is this brown thing here. Then you take the next one, the next slice and you encode it and that will give you this blue thing here. Then you run this sequence through the vector quantization step where each of those will be mapped. So you have a codebook here, right? You have your codebook. And you look up the first one and you decide, ah, this probably goes here to this code vector. So you put that code vector into the first place. You take the second place and you might decide, no, that's this one. So you put this one into the second place. And the third one you might decide, ah, no, that also is closest to the first codebook vector. So you again put the first codebook vector into that slot. Now that doesn't mean that it's the same music, but of course the decoder now is going to look at the entire sequence and can decide, ah, probably this isn't the exact same note as here, but it might decide, you know, that the chord played will repeat or something like this. So there's this vector quantization step and the codebook look up. Sorry, yeah, this minimizes which vector of the codebook is closest and the codebook look up, I think we'll just replace then the code, this vector with the actual codebook things. And so there's this slight difference here. So Z here, as you can see, is the argmin K. So the argument that is the actual number K, which codebook vector is the closest. And then this EZT will be the actual vectors. So this here is actually what I described right here. But this is, I think, a detail. And you're going to do this at different scales. Now you can imagine that the bottom one is going to give you the best, most faithful reconstruction when you decode it, right? But it is also going to learn about the kind of details in the music, the short term details. Whereas this coarse grained one, it can learn things about longer range compositions. It might not produce as correct of a reconstruction, but it can learn long range dependencies, such as the structure of a song or the structure of a verse or something like this. So these are independent of each other. And they make an argument as to why. So people have tried to kind of share these architectures, but have found that mainly the models will basically ignore the top two and only go over via the coarse grained ones. So that's why they completely separate these at this stage of training. Right, so we have trained three different VAEs at three different scales of music to always reconstruct the input. What does that give us? That gives us a distribution right here. That gives us a way to take a piece of music and map it to this hidden space, to this very compressed representation in this quantized world, right? I've said before, this is a very compressed representation of your data. Why can you do that? Sorry, what's that useful for? What you can do now is you can try to sample in that hidden space. So instead of sampling music, we have no clue of how to sample music unless we are given some music. What we can do is we can say maybe this thing here, because it's compressed, it kind of... So if we just sample a waveform, it's very unlikely that it's music. But if we sample these hidden things, you know, it's quite likely that if we feed it through the decoder, something will come out. And even better, maybe our data set in this hidden space follows a kind of a simpler distribution, one that we could learn, right? So we're trying, we're going to try to learn a prior distribution over the distribution of codebook vectors. And that is naturally going to be a joint distribution between the top, middle and bottom VQVAEs. And we can decompose this into the following thing simply by applying the standard probabilistic algebra transformation. And we can then, they say, we train separate models, sorry about that, we train separate models for the top level prior, the top, the middle, and the bottom. So what that means is basically, these are now neural networks. If you read something like this in a paper like this, this is going to be a neural network that takes the right side as an input and produces the left side, right? So you start out with this one, this is a neural network that simply takes as an input, sorry, I'm going to draw this neural network, takes as an input, maybe something like a Gaussian super prior, right? You sample from that and that will as an output give you this Z top. Then the next neural network will take this as an input and will give you Z middle as an output. And then the final neural network will input the two of those and give you Z top. And you can train these neural networks simply by kind of training a prior to produce this thing right here. You'd simply use your data, compress it to the hidden space, and then train a neural network to produce that distribution. And you can do this in any number of ways. You can use classic VAEs, you can use, sorry, you can use here, they say we use transformers with sparse attention, as they are currently state of the art in autoregressive modeling. And they say we propose a simplified version, which we call the scalable transformer that is easier to implement and scale. But they see, you see, they model this distribution with these scalable transformers. All right, so now what do we have? We have a way to sample these hidden vectors, right? So we don't need, we don't need this part anymore. This part, sorry about that, this part here was just used for training. We can, we now have our transformers. They take nothing as an input or they take like a Gaussian as an input and they can directly output this hidden representation. So we could technically sample from that and then just push it through this decoder of the VQVA. But the question is, which of the three do we take? And wouldn't it be great if we can combine them? Because if we simply sample these, this higher scale one, we just get not very long range dependencies, right? Because that's what the VQVA learned. If we just sample this one, then we just get a coarse music and we can sample all three, but they will just give us three different tracks of music. So we want to combine the three decoders into one somehow. And that's, we do this through these up samplers. So what we'll use, what we'll target actually is this bottom one. We target this one because this one gives us the best quality music, right? Because it was trained with the shortest time scale. We're going to try to take the other signals and influence it. So we'll start with the top level prior, right? That will produce us, these transformers will give us a sequence, a sequence of tokens in the hidden space that is very coarse, as you can see here. And then we'll feed that into a up sampler. And these up samplers again are on the neural networks that can connect the different scales with each other. All right, so you can connect this to this. It's basically like conditioning the model that produces the sequence on this sequence right here. And again, we use an up sampler to up sample this to the finest scale. And that we feed in the bottom scale, and then we get our music. Now throughout all of this, you have conditioning information here, which is a bit of an addition to the model. So the conditioning information can be things like artist, genre, and timing. And this is, it appears to be pretty important because you kind of, first of all, want some variety. And then second of all, you sort of want to control what music is produced. And you don't just want to train this model for one single artist, because you have much more data across all of music. So this conditioning information is just included here via another neural network. And you can find all the architectures for all of these models in the paper. It's not particularly important, I believe, how exactly you include them, but the fact that you do. The last thing is, what they do is they do this kind of windowed sampling. So in order to produce music, you're going to have to produce these slices of music right here. But you sort of have a maximum length here that your models can handle. And this is usually not the length of a song. You may know transformers and so on. They usually have token limits of like 512 tokens. In terms of audio, that's not that much. So what you do is this windowed sampling, where you sample something, and then you condition basically on the first part, and then you just sample the next thing, and then you again condition on the first part, and then you just sample the next thing, the next few ones. And that guarantees that each of the sampling steps is basically conditioned on what comes before, as you see up here. So you would always sort of condition on a part, produce the next part. All right. And they say you can also basically condition on... So you can feed in, you don't have to sample the very first one, you can also feed in an existing song in order to prime the system. So what you can do is if you have a beginning of a song, then you can let the system finish the song by simply taking the song, running it through the encoder that we produced during training, right? You get these hidden representations, so you don't actually have to sample them from your prior. And then you run this generation process as if this came out of your prior instead of what you sampled. Okay. So let's have a look at how that sounds, or listen. This is an explorer, they release many, many samples from this. And the part here where we're going to listen is called no lyrics conditioning. So as you can hear, this is already pretty good music and the genre is American folk. The singer is Pete Seeger. This already sounds very authentic, but you can hear that the lyrics are just kind of mumbly, right? And that's because the model is basically asked to come up with lyrics as pure audio waveforms. And that results in some subpar lyrics. Basically it just produces phonemes that sound like the singer. It doesn't produce entire words. And of course it also doesn't produce sentences that make any sort of sense. And that's why they're building in an additional thing to do lyrics conditioning. So with lyrics conditioning, the idea is that in the conditioning information, you also add lyrics. So here is plus text. So you add text, and then the model is basically can also look at the text. Now you never, you still, you still, so even before we had music with lyrics and the decoder was always asked to reconstruct that. And so none of that changes. That's why it has learned to produce phonemes, right? But now the decoder can also, and also the encoder, the system can look at the lyrics that you provide right here in order to help with its decoding. So technically it could learn to bypass the encoding of the exact way the lyrics are uttered. And it could just look at the text that you provide. Now this of course requires that during training you provide the lyrics of the song that are actually that you feed in. But also it means that during decoding, if you sample, you can then provide your own lyrics and look what happens. So they say they provide lyrics, they always have to provide lyrics for chunks of audio. So our data set includes song level lyrics, but to make it easier, we train on shorter 24 second chunks of audio. And this is partly to make it easier for the model, but also partly because those appear to be the limitations of these systems, right? If you have transformers in there and whatnot, 24 seconds of raw audio waveform is a lot. So they have this problem of they have a song from here to here, and they have the lyrics, blah, blah, blah. And they need to know which lyrics belong to which part of the song. And usually it's monotonic, right? And linear because you get the lyrics from some lyrics website, this blah, blah, blah. But you don't know particularly to which 24 second chunk they belong. So they say, first of all, they started with simply linearly aligning the lyrics, but then they had some, they had some problems with fast songs. So they had some heuristic here. But ultimately, the decoder needs to learn to attend to these lyrics. And these the graphics like this you see here is the music token position and lyrics token position. Here you see the the system learns that for example, if it has this music token needs to attend to this token in the lyric. So you can by inspecting these attention heads that you have on the lyrics text in the system, you can see which lyrics the model is paying attention to. And the fact that it learns to pay linearly attention to these things is kind of a confirmation because you you don't you give the whole text or at least the 24 second chunks of audio, you give that at once as a as a text, right. And the fact that it learns to linearly attend to the tokens is a confirmation that it actually includes that information into the coding. And that is a pretty gives you pretty much better results. So we can maybe go to classic pop. So this are unseen lyrics. So the model has never seen these lyrics, right? It was just asked to produce classic pop in the style of Frank Sinatra with these lyrics. And that's what it came up with. That is pretty, pretty, pretty cool. I think they also have re renditions where they basically feed I believe feed in the original lyrics, we conditioned on lyrics seen during training. And they have fun songs. And in the fun songs, I like the hip hop in the style of Kanye West, where they provide the lyrics of Eminem's lose yourself. I'm grooving. I don't know what you're thinking, but this is cool. And they can also, as we said, do these completions where they start with part of a song. And just I have to do this. I have to do the hi there. So the first version of this video was copystriked because what you would hear would be the original never going to give you up like 10 seconds of it, and then followed by what the model continues with. So as a substitute, you're now going to have to listen to me. I hope that suffices. Almost as good, almost as good as the original. So as you can see, this the results here are pretty, pretty cool. And I want to show you one last thing, and that is this Christmas song in the style of Frank Sinatra. I believe it's this one right here. And the special thing here is, it's again classic pop in the style of Frank Sinatra. And you see on the bottom here, you see on the bottom which of the lyrics it's attending to. And you see, you know, this this graph right here that shows you that first, it's attending linearly through the lyrics, but then it kind of jumps around and attends to different things because it doesn't it doesn't it doesn't just continue. So this is great. So it kind of falls out of this linearly attending to the lyrics. And probably because there was sort of a pause in the lyrics. And maybe this is just more than one audio window. So it doesn't have this autoregressive property anymore. And then it doesn't find the proper place to attend anymore. And just, again, comes up with sort of babbles, but it sounds pretty, pretty cool. Yeah, so this they have released many, many samples here, some cherry picked and just a lot of samples with unseen lyrics, rerenditions, and so on. This all is very cool. They have their training setup described, I believe they also release their code. Many more results in the paper of how to make this thing work if you want to do that yourself. And with that, I invite you to read the paper. If you're still here, please subscribe if you like this content, leave a comment and bye bye.
 Alright, so what you're hearing is the OpenAI jukebox.
 This paper came out and it is a surprisingly good quality generative model for music,
 including lyrics, so including singing, which I believe is pretty novel.
 And the fact that it works so well and has musical consistency throughout entire songs
 is something that is very, very novel and cool.
 Alright, so we're looking at the paper, it's called Jukebox, a generative model for music,
 and it's by Profola Darival, Heiwo Jun, Christine Pine, Jongwoo Kim, Alec Radford, and Ilya
 Sutskever of OpenAI.
 So the model here is not very novel, but I think the way they set it up makes a lot of
 sense and we'll quickly go through it and look at what comes out of this.
 This is not a very technical paper, but I think it's well written and their main thing
 is these VQVAE models.
 So these models, you might know what a variational autoencoder is.
 So in a variational autoencoder I have an input, let's say an image, here of our typical
 cat, and I put it through what is called an encoder in order to get a hidden representation.
 Now this hidden representation, usually called something like Z or H, is then in a classic
 autoencoder I would here have a decoder and then I train that decoder to match that original
 picture as closely as possible.
 So I train these two neural networks, the encoder and the decoder, to just basically
 compress to this representation and then reproduce again the original image.
 And thereby I think I can learn a good hidden representation.
 Now in a variational autoencoder what happens is there is an in-between step, namely this
 Z representation here is not directly fed to the decoder, but this representation is
 actually used to parameterize Gaussians.
 So in the easiest case, let's say we have six dimensions here in the hidden representation,
 the first two are used to parameterize the first Gaussian, one is going to be the mean
 and one is going to be the standard deviation of that Gaussian, and the second two are going
 to parameterize the second Gaussian and the third are going to parameterize the third
 one.
 So we have now three Gaussians and then from those three Gaussian distributions we're going
 to sample a three-dimensional vector and then we're going to feed that three-dimensional
 vector to the decoder.
 So every input image in essence is giving us a distribution in the latent space and
 not just a single vector, it's just giving us a vector that describes an entire multivariate
 Gaussian distribution.
 And then we train the decoder basically to reconstruct the encoder given that distribution.
 So the variational autoencoder has improved over the classic autoencoder because it tends
 to circumvent some of the shortcomings, but still variational autoencoders have their
 problems and in terms of generative model for images, people as you know have gone to
 things like GANs and so on.
 But here this new thing is called the VQVAE and that's because I believe this stands for
 vector quantization.
 Not entirely sure honestly, but so what it does is it takes the input right here and
 again it maps it to a latent code.
 And these latent code I believe are called H. So it maps it to a vector called H. Now
 here is where it gets different.
 We do have a so called codebook here.
 So a codebook is just a list of vectors.
 This is the codebook and these are here called E. So what we'll do is we'll simply look for
 H which one's the closest neighbor in the codebook that we have of hidden vectors and
 we'll map it to that and say this here is the closest neighbor.
 So we'll basically quantize the hidden representation to these codes.
 So we end up only with vectors that are these codes.
 And now instead of saving the vector H, first of all we get a super compressed representation
 because if these are like 16 codebook vectors we can just simply enumerate them.
 And then we can simply encode the image as one or a sequence of these indices here.
 But second of all it tends to bring a more kind of a diversity and accuracy if we then
 decode from these code vectors.
 Now of course everything is trained here.
 So the encoder is trained and the codebooks themselves are also trained and the decoder
 is also trained to give you the maximum kind of benefit.
 So this is what is described here.
 What they have is first of all their loss for these VQVAEs and you'll see how they are
 using them to generate music in a second.
 Their loss is part this reconstruction loss which you can see here.
 This is the original image.
 How far is it away from the decoded hidden quantized representation?
 This E here, this is the quantized codebook vector that belongs to the hidden representation
 of X.
 So this is your standard reconstruction loss.
 The next part of the loss is this codebook loss.
 Now the codebook loss is for training these codebook vectors.
 So it basically pulls the codebook vector closer to the actual hidden representation.
 So that is where you train these.
 This here is a stop gradient.
 So basically it just means that you want your codebook vectors to be better representing
 of the data that you feed in because otherwise it would be useless codebook vectors.
 And the third part of their loss is this commit loss right here.
 And that is exactly the opposite.
 Now you put the stop gradient on the codebook vector and you simply want to pull the hidden
 representation closer to the codebook vector such that, imagine the encoder must learn
 to approximately hit one of these codebook vectors.
 Otherwise it cannot really learn something meaningfully.
 It must learn to deal with the codebook vectors that are there and to approximately map the
 input into the vicinity of one of the codebook vectors.
 Otherwise there is no information flowing.
 So that is how you train things.
 You train the encoder and the decoder to reconstruct.
 You train the codebook vectors to represent the data and you also train the encoder to
 make good use of the codebook vectors.
 Basically now that I think about it I am pretty sure in the reconstruction loss you might
 only train the decoder because you do have this quantization step in between in this
 thing here.
 So technically you could not back propagate through that.
 So that is how you train the individual parts of a VQVAE and we will see how they use it
 in order to produce music.
 Alright, so what they do is they start off with a sample of music and they send it through
 this thing, through this architecture and at the end they are trying to reconstruct
 the same thing they had at the beginning.
 So you see this overarching autoencoder architecture.
 That is why it is an autoencoder.
 You try to reconstruct your input and thereby you try to learn something about the data
 because a model that can compress and then uncompress all of your data has learned something
 useful about it.
 Now you have these VQVAEs here in the middle but you have them at different scales.
 So you will have three of those here.
 There is a coarse scale one, middle scale one and a high frequency one.
 So three VQVAEs right here.
 And you train each one of them separately.
 And the difference between them is that because this is a continuous signal you cannot just
 encode a continuous signal because it is an audio waveform.
 What you have to do is you have to go through it in some sort of a stepwise fashion.
 You have to divide it into individual pieces and encode each of those pieces as one hidden
 vector, or one element of the codebook.
 And the size here, the precise size of how you go through the audio, that is different
 between these different scales.
 So in this scale right here you go through the audio in very small steps.
 This as you can imagine gives you the best reconstruction.
 So this is the audio.
 The audio is like this.
 And you just take part of it, like from here to here, and you encode it into a single hidden
 vector, which is this brown thing here.
 Then you take the next one, the next slice and you encode it and that will give you this
 blue thing here.
 Then you run this sequence through the vector quantization step where each of those will
 be mapped.
 So you have a codebook here, right?
 You have your codebook.
 And you look up the first one and you decide, ah, this probably goes here to this code vector.
 So you put that code vector into the first place.
 You take the second place and you might decide, no, that's this one.
 So you put this one into the second place.
 And the third one you might decide, ah, no, that also is closest to the first codebook
 vector.
 So you again put the first codebook vector into that slot.
 Now that doesn't mean that it's the same music, but of course the decoder now is going to
 look at the entire sequence and can decide, ah, probably this isn't the exact same note
 as here, but it might decide, you know, that the chord played will repeat or something
 like this.
 So there's this vector quantization step and the codebook look up.
 Sorry, yeah, this minimizes which vector of the codebook is closest and the codebook look
 up, I think we'll just replace then the code, this vector with the actual codebook things.
 And so there's this slight difference here.
 So Z here, as you can see, is the argmin K.
 So the argument that is the actual number K, which codebook vector is the closest.
 And then this EZT will be the actual vectors.
 So this here is actually what I described right here.
 But this is, I think, a detail.
 And you're going to do this at different scales.
 Now you can imagine that the bottom one is going to give you the best, most faithful
 reconstruction when you decode it, right?
 But it is also going to learn about the kind of details in the music, the short term details.
 Whereas this coarse grained one, it can learn things about longer range compositions.
 It might not produce as correct of a reconstruction, but it can learn long range dependencies,
 such as the structure of a song or the structure of a verse or something like this.
 So these are independent of each other.
 And they make an argument as to why.
 So people have tried to kind of share these architectures, but have found that mainly
 the models will basically ignore the top two and only go over via the coarse grained ones.
 So that's why they completely separate these at this stage of training.
 Right, so we have trained three different VAEs at three different scales of music to
 always reconstruct the input.
 What does that give us?
 That gives us a distribution right here.
 That gives us a way to take a piece of music and map it to this hidden space, to this very
 compressed representation in this quantized world, right?
 I've said before, this is a very compressed representation of your data.
 Why can you do that?
 Sorry, what's that useful for?
 What you can do now is you can try to sample in that hidden space.
 So instead of sampling music, we have no clue of how to sample music unless we are given
 some music.
 What we can do is we can say maybe this thing here, because it's compressed, it kind of...
 So if we just sample a waveform, it's very unlikely that it's music.
 But if we sample these hidden things, you know, it's quite likely that if we feed it
 through the decoder, something will come out.
 And even better, maybe our data set in this hidden space follows a kind of a simpler distribution,
 one that we could learn, right?
 So we're trying, we're going to try to learn a prior distribution over the distribution
 of codebook vectors.
 And that is naturally going to be a joint distribution between the top, middle and bottom
 VQVAEs.
 And we can decompose this into the following thing simply by applying the standard probabilistic
 algebra transformation.
 And we can then, they say, we train separate models, sorry about that, we train separate
 models for the top level prior, the top, the middle, and the bottom.
 So what that means is basically, these are now neural networks.
 If you read something like this in a paper like this, this is going to be a neural network
 that takes the right side as an input and produces the left side, right?
 So you start out with this one, this is a neural network that simply takes as an input,
 sorry, I'm going to draw this neural network, takes as an input, maybe something like a
 Gaussian super prior, right?
 You sample from that and that will as an output give you this Z top.
 Then the next neural network will take this as an input and will give you Z middle as
 an output.
 And then the final neural network will input the two of those and give you Z top.
 And you can train these neural networks simply by kind of training a prior to produce this
 thing right here.
 You'd simply use your data, compress it to the hidden space, and then train a neural
 network to produce that distribution.
 And you can do this in any number of ways.
 You can use classic VAEs, you can use, sorry, you can use here, they say we use transformers
 with sparse attention, as they are currently state of the art in autoregressive modeling.
 And they say we propose a simplified version, which we call the scalable transformer that
 is easier to implement and scale.
 But they see, you see, they model this distribution with these scalable transformers.
 All right, so now what do we have?
 We have a way to sample these hidden vectors, right?
 So we don't need, we don't need this part anymore.
 This part, sorry about that, this part here was just used for training.
 We can, we now have our transformers.
 They take nothing as an input or they take like a Gaussian as an input and they can directly
 output this hidden representation.
 So we could technically sample from that and then just push it through this decoder of
 the VQVA.
 But the question is, which of the three do we take?
 And wouldn't it be great if we can combine them?
 Because if we simply sample these, this higher scale one, we just get not very long range
 dependencies, right?
 Because that's what the VQVA learned.
 If we just sample this one, then we just get a coarse music and we can sample all three,
 but they will just give us three different tracks of music.
 So we want to combine the three decoders into one somehow.
 And that's, we do this through these up samplers.
 So what we'll use, what we'll target actually is this bottom one.
 We target this one because this one gives us the best quality music, right?
 Because it was trained with the shortest time scale.
 We're going to try to take the other signals and influence it.
 So we'll start with the top level prior, right?
 That will produce us, these transformers will give us a sequence, a sequence of tokens in
 the hidden space that is very coarse, as you can see here.
 And then we'll feed that into a up sampler.
 And these up samplers again are on the neural networks that can connect the different scales
 with each other.
 All right, so you can connect this to this.
 It's basically like conditioning the model that produces the sequence on this sequence
 right here.
 And again, we use an up sampler to up sample this to the finest scale.
 And that we feed in the bottom scale, and then we get our music.
 Now throughout all of this, you have conditioning information here, which is a bit of an addition
 to the model.
 So the conditioning information can be things like artist, genre, and timing.
 And this is, it appears to be pretty important because you kind of, first of all, want some
 variety.
 And then second of all, you sort of want to control what music is produced.
 And you don't just want to train this model for one single artist, because you have much
 more data across all of music.
 So this conditioning information is just included here via another neural network.
 And you can find all the architectures for all of these models in the paper.
 It's not particularly important, I believe, how exactly you include them, but the fact
 that you do.
 The last thing is, what they do is they do this kind of windowed sampling.
 So in order to produce music, you're going to have to produce these slices of music right
 here.
 But you sort of have a maximum length here that your models can handle.
 And this is usually not the length of a song.
 You may know transformers and so on.
 They usually have token limits of like 512 tokens.
 In terms of audio, that's not that much.
 So what you do is this windowed sampling, where you sample something, and then you condition
 basically on the first part, and then you just sample the next thing, and then you again
 condition on the first part, and then you just sample the next thing, the next few ones.
 And that guarantees that each of the sampling steps is basically conditioned on what comes
 before, as you see up here.
 So you would always sort of condition on a part, produce the next part.
 All right.
 And they say you can also basically condition on...
 So you can feed in, you don't have to sample the very first one, you can also feed in an
 existing song in order to prime the system.
 So what you can do is if you have a beginning of a song, then you can let the system finish
 the song by simply taking the song, running it through the encoder that we produced during
 training, right?
 You get these hidden representations, so you don't actually have to sample them from your
 prior.
 And then you run this generation process as if this came out of your prior instead of
 what you sampled.
 Okay.
 So let's have a look at how that sounds, or listen.
 This is an explorer, they release many, many samples from this.
 And the part here where we're going to listen is called no lyrics conditioning.
 So as you can hear, this is already pretty good music and the genre is American folk.
 The singer is Pete Seeger.
 This already sounds very authentic, but you can hear that the lyrics are just kind of
 mumbly, right?
 And that's because the model is basically asked to come up with lyrics as pure audio
 waveforms.
 And that results in some subpar lyrics.
 Basically it just produces phonemes that sound like the singer.
 It doesn't produce entire words.
 And of course it also doesn't produce sentences that make any sort of sense.
 And that's why they're building in an additional thing to do lyrics conditioning.
 So with lyrics conditioning, the idea is that in the conditioning information, you also
 add lyrics.
 So here is plus text.
 So you add text, and then the model is basically can also look at the text.
 Now you never, you still, you still, so even before we had music with lyrics and the decoder
 was always asked to reconstruct that.
 And so none of that changes.
 That's why it has learned to produce phonemes, right?
 But now the decoder can also, and also the encoder, the system can look at the lyrics
 that you provide right here in order to help with its decoding.
 So technically it could learn to bypass the encoding of the exact way the lyrics are uttered.
 And it could just look at the text that you provide.
 Now this of course requires that during training you provide the lyrics of the song that are
 actually that you feed in.
 But also it means that during decoding, if you sample, you can then provide your own
 lyrics and look what happens.
 So they say they provide lyrics, they always have to provide lyrics for chunks of audio.
 So our data set includes song level lyrics, but to make it easier, we train on shorter
 24 second chunks of audio.
 And this is partly to make it easier for the model, but also partly because those appear
 to be the limitations of these systems, right?
 If you have transformers in there and whatnot, 24 seconds of raw audio waveform is a lot.
 So they have this problem of they have a song from here to here, and they have the lyrics,
 blah, blah, blah.
 And they need to know which lyrics belong to which part of the song.
 And usually it's monotonic, right?
 And linear because you get the lyrics from some lyrics website, this blah, blah, blah.
 But you don't know particularly to which 24 second chunk they belong.
 So they say, first of all, they started with simply linearly aligning the lyrics, but then
 they had some, they had some problems with fast songs.
 So they had some heuristic here.
 But ultimately, the decoder needs to learn to attend to these lyrics.
 And these the graphics like this you see here is the music token position and lyrics token
 position.
 Here you see the the system learns that for example, if it has this music token needs
 to attend to this token in the lyric.
 So you can by inspecting these attention heads that you have on the lyrics text in the system,
 you can see which lyrics the model is paying attention to.
 And the fact that it learns to pay linearly attention to these things is kind of a confirmation
 because you you don't you give the whole text or at least the 24 second chunks of audio,
 you give that at once as a as a text, right.
 And the fact that it learns to linearly attend to the tokens is a confirmation that it actually
 includes that information into the coding.
 And that is a pretty gives you pretty much better results.
 So we can maybe go to classic pop.
 So this are unseen lyrics.
 So the model has never seen these lyrics, right?
 It was just asked to produce classic pop in the style of Frank Sinatra with these lyrics.
 And that's what it came up with.
 That is pretty, pretty, pretty cool.
 I think they also have re renditions where they basically feed I believe feed in the
 original lyrics, we conditioned on lyrics seen during training.
 And they have fun songs.
 And in the fun songs, I like the hip hop in the style of Kanye West, where they provide
 the lyrics of Eminem's lose yourself.
 I'm grooving.
 I don't know what you're thinking, but this is cool.
 And they can also, as we said, do these completions where they start with part of a song.
 And just I have to do this.
 I have to do the hi there.
 So the first version of this video was copystriked because what you would hear would be the original
 never going to give you up like 10 seconds of it, and then followed by what the model
 continues with.
 So as a substitute, you're now going to have to listen to me.
 I hope that suffices.
 Almost as good, almost as good as the original.
 So as you can see, this the results here are pretty, pretty cool.
 And I want to show you one last thing, and that is this Christmas song in the style of
 Frank Sinatra.
 I believe it's this one right here.
 And the special thing here is, it's again classic pop in the style of Frank Sinatra.
 And you see on the bottom here, you see on the bottom which of the lyrics it's attending
 to.
 And you see, you know, this this graph right here that shows you that first, it's attending
 linearly through the lyrics, but then it kind of jumps around and attends to different things
 because it doesn't it doesn't it doesn't just continue.
 So
 this is great.
 So it kind of falls out of this linearly attending to the lyrics.
 And probably because there was sort of a pause in the lyrics.
 And maybe this is just more than one audio window.
 So it doesn't have this autoregressive property anymore.
 And then it doesn't find the proper place to attend anymore.
 And just, again, comes up with sort of babbles, but it sounds pretty, pretty cool.
 Yeah, so this they have released many, many samples here, some cherry picked and just
 a lot of samples with unseen lyrics, rerenditions, and so on.
 This all is very cool.
 They have their training setup described, I believe they also release their code.
 Many more results in the paper of how to make this thing work if you want to do that yourself.
 And with that, I invite you to read the paper.
 If you're still here, please subscribe if you like this content, leave a comment and
 bye bye.