DDPM - Diffusion Models Beat GANs on Image Synthesis (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper diffusion models diffusion model ddpm ddim denoising autoencoders generative models generative models deep learning gan alternatives alternatives to gans computer vision generative machine learning image generation openai diffusion openai gan variational autoencoder log likelihood variational lower bound
#ddpm #diffusionmodels #openai

GANs have dominated the image generation space for the majority of the last decade. This paper shows for the first time, how a non-GAN model, a DDPM, can be improved to overtake GANs at standard evaluation metrics for image generation. The produced samples look amazing and other than GANs, the new model has a formal probabilistic foundation. Is there a future for GANs or are Diffusion Models going to overtake them for good?

OUTLINE:
0:00 - Intro & Overview
4:10 - Denoising Diffusion Probabilistic Models
11:30 - Formal derivation of the training loss
23:00 - Training in practice
27:55 - Learning the covariance
31:25 - Improving the noise schedule
33:35 - Reducing the loss gradient noise
40:35 - Classifier guidance
52:50 - Experimental Results

Paper (this): https://arxiv.org/abs/2105.05233
Paper (previous): https://arxiv.org/abs/2102.09672
Code: https://github.com/openai/guided-diffusion

Abstract:
We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for sample quality using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128×128, 4.59 on ImageNet 256×256, and 7.72 on ImageNet 512×512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.85 on ImageNet 512×512. We release our code at this https URL

Authors: Alex Nichol, Prafulla Dhariwal

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello! These are generated images from a new model, actually a new class of model. It's been around for a while, but for the first time this new class of model has been pushed to the point where the images they produce not only look really nice and look like something you've come to expect from the latest and greatest GAN models, but also they are better in the standard metrics we use to evaluate GANs, specifically here in the FID, the fresher inception distance. The paper we're going to talk about today is called Diffusion Models Beat GANs on Image Synthesis. It's by Prof. Dariwall and Alex Nicole of OpenAI. Already in the title they're pulling no punches, just be like this beats GANs. In this paper they're mainly talking about improvements to this new class of models which they call diffusion models. I would like to dive a bit more into what diffusion models are instead of just telling you what the improvements of this paper are, because I think most people haven't come in contact with these types of models yet. They thoroughly reference another paper which is called Improved Denoising Diffusion Probabilistic Models by themselves. In this paper they more develop these new models than in the other paper. The paper here, as you can see, is just three months younger than the other paper. This is really close. I think this paper is more insightful into what these models are. That being said, by the name Improved right here you can also see that this is not the seminal paper of these types of models. If you're interested in that you have to go back even further. However we're going to look at this and we're going to look at the new paper and see what are all the things that lead to this new class of models being better than GANs. Specifically we're going to talk about DDPMs, Denoising Diffusion Probabilistic Models. They're a bit like a variational auto encoder, a little bit. We'll go through that. If you feel that this was helpful please do share it out. It's been a pleasure bringing this to a lot of people and if you do it will just be more people. We'll have more fun. They say that Denoising Diffusion Probabilistic Models, DDPMs, are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications DDPMs can also achieve competitive log likelihoods while maintaining high sample quality. In this paper they take these models, these DDPM models, and they say look we can push those models to push their log likelihood. There are a number of metrics that generative models track. It's not as easy as the validation set accuracy in a classifier. Log likelihood is one of the metrics that these models track. Here they say well we can get competitive log likelihood while maintaining high sample quality, which is a nice way of saying we don't beat GANs yet. In the next paper, the one I showed you before, they actually do beat GANs on the standard metrics and also the samples look quite impressive. The DDPMs have been around before but they go into a quick overview right here, which is what I think is quite appropriate for us to dive in. The philosophy here or the whole purpose behind this is they say let's imagine I have an image of my house right here. I have an image of a house and I define a process, what they call a forward noising process. This forward noising process takes the image and it just adds a little bit of noise to it, like epsilon noise that's sampled from some standard distribution like a Gaussian. You just sample a bit of noise and you just add it to that image. You have the same house but there'll be a bit of noise on it. Then you do it again. You sample another bit of noise and you do it again. As you do this over many steps, and here they actually notice that the previous authors were using a thousand steps and if they just increase that to four thousand steps, the log likelihoods go better. In any case you do this for many steps, thousands of steps in this first instance. You do this, what are you gonna end up with? The argument here is that if you do this for so many times, for so long, over so many steps, you're going to end up with random noise itself. This is ish according to some kind of normal distribution. You just assume. You can actually prove this that if you do enough steps, like if you do infinitely many steps, it goes actually towards just noise. Whenever you're done with this, there is no more information about the original image than actually sampling from this distribution right here. You have successfully defined a process that takes you from the image space. This here is from the data space that takes you from the data space to a known distribution, which is the normal distribution. Now here is the logic. If we could invert this, if we just somehow could invert this mapping, if we could have a process that knows if I give you an image with some noise, can you tell me what image that came from? Is that doable? It's not, it's not, it's thinkable. If I give you this image with some specks of noise on it and I ask you, could you please give me, I tell you, I'm the oracle, I tell you, look I've taken some image that already had a bit of noise on it, but I've added more. I've taken an image, I've added some noise. What was the original image that I, I don't tell you what the noise is, right? I just tell you the noise comes from whatever, a normal distribution, I've added it. What was the original image? Now you looking at this image, you'll see, you know, this could be a house. So not quite sure, but you know, this might be something like this might be the original image and this here I'm not really sure about if this is noise. So you're gonna sort of revert that process a little bit, right? Knowing that this is how the image came to be, you as a human, if I told you, you could approximately reverse that process. That of course requires you to know something about these images, right? That like, it requires you to know what a house looks like and when you see something like this, that well, you know, probably because I don't tell you which ones are the noise and which ones aren't. So that's the trick, right? If I just told you, well, all the orange stuff is noise, right? But you just see, you just see this all in mono color, but you know kind of, okay, so this here looks like it's from the image itself, but then this here is just kind of a spec and that just kind of, might just be noise, maybe not, right? But then this here, I'm pretty sure it's just noise and not part of the original image. So you could do that and the question is, can we learn a function that does this reverse process? If we can do so, right? If we can learn a function, function of course that's going to be some kind of neural network-ish thing. We can learn a function where I give you an image with noise and I tell you, by the way, so this is maybe time step zero, this is t equals zero, t equals one, t equals two, and so on. Well, you can't see that. If I tell you, okay, here is an image, this happened at t equals 50, can you give me the t equals 49 image that this came from? Alright, and this is the whole principle. We're going to, we can generate training data for this neural network very easily because we just take data and we run them through the noise process forward, right? Then we have plenty of training data for every step of this pipeline, right? In fact, we don't train a, we don't train a different phi function for every step. As you can see, the phi function simply takes the time or can take the time as an input. It's certainly possible otherwise, or it's possible to not tell it at all, right? Then you, it has no clue. So yeah, if you do this, you can generate training data and then the idea is you can just run this process in reverse and arrive at the original sample. And even more, because this here is actually the normal distribution, you can now sample random noise from that normal distribution, right? You can feed it to this process and this process, who has learned to map the data distribution to the normal distribution and can reverse that process, will give you some sort of data distribution sample for your input that you sampled from the normal distribution. All right, this is the idea and it's quite tricky to get this to work, as you can imagine, but let's not forget that GANs also have been quite tricky to get to work. It's just maybe there has been a bit more work going into GANs. All right, so formally this goes as follows. We define this forward-noising process, right? We sample this from the data distribution. We sample x0 from the data distribution. We define this forward-noising process Q, which produces x1 through xt, so capital T as the end here. And we, by adding Gaussian noise at time t with some variance, okay, so you can have, you can have, it's zero mean Gaussian noise, I believe, maybe. Yeah, it's, well, you scale, but you define this variance schedule right here. That's also your choice, right? You choose what kind of noise you want to add, but ultimately you take, ultimately, the distribution of the things you produce via that noising process, given that you start at the data sample x0, you simply define as this product of distributions. So you start with, this just means you start with x0 and then you go from x0 to x1 and then you go from x1 to x2 and so on, okay? And each of these steps is an independent application of noise. As you can see here, this is one of those steps. So what you're saying is that the distribution of the next sample right here is going to be a normal distribution that's going to be centered at this thing right here and its variance is this thing right here. So you can see that the assumption here is you use noise that has a diagonal covariance matrix, okay? This is, I guess it's reasonable. It certainly makes computing things easier, right? The other thing here is that you can see this Gaussian is centered at the last sample but down scaled by this factor right here and I think, like, this is a choice again by the modelers but I think this is also due to the fact that makes computation easier because I guess if you don't have this then, you know, you start somewhere and you add noise and you sample something, you add noise, you sample something, maybe this would grow indefinitely and you sort of need to rescale things such that you can make this statement right here. Given sufficiently large T and a well behaved schedule of beta, the latent XT, so the very last step, is nearly an isotropic Gaussian distribution, okay? That's the entire point. So if you do it like this, which is a choice, but if you do it like this then at the end if you do enough steps, infinitely many steps, then you end up at an isotropic Gaussian distribution. Thus, if we know the exact reverse distribution, we can sample from the Gaussian and run the process in reverse to get a sample from the data distribution. Then they say, however, since the reverse distribution depends on the entire data distribution, we approximate it using a neural network as follows. So this statement can be a bit weird in the first instance. This depends on the entire data distribution, right? Because it's very close to this thing right here and this thing right here depends on nothing, right? This you just define, you just say I'm gonna add random noise to something and that's my next distribution. It only depends on the input image right here. The way to see it, that this depends, the reverse depends on the entire data distribution, is exactly what I said before. If I give you the, like if I give you this picture, I'm not gonna actually tell you right where the noise is. So I give you this picture and I tell you this is a drawing from a very small child, because that's my drawing level, and I've just added a bunch of noise to it. Could you tell me what the original drawing was? This is very different from me saying here is a drawing from a small child, please add noise to it. That's easy, I just did this, right? I was just called, I just did it. But if I tell you what was the original image, you have to take into account the entire world. You know about how small children draw, what kind of motives they usually draw and so on, and that's how you are able to come up by saying well it was probably something like this. This needs your knowledge of the entire data distribution. That's why they say it right here. So they say well we can't, we like, we can't just have the entire data distribution otherwise, you know, we wouldn't even have the problem in the first place. So what we can do is we can approximate one of these steps using a neural network. So we have a neural network that takes as an input, as I said, it takes as an input the noised version of the image and it gives you as an output, it's a bit like this is, it gives you, I told you give me the image that this came from, in this case what they want is give me a distribution over images where that could have come from, right? And again they say this, they model this as a Gaussian right here and the neural network will produce the mean and the covariance matrix given the image. So the neural network is supposed to look at the image and decide okay what's the Gaussian distribution of images where that probably came from? And this is a strong assumption, right? The fact for example that you know this is a Gaussian distribution, like this is adequately modeled as a Gaussian distribution, it's a strong assumption that you can only make because you make these very small steps. Because nothing, I mean nothing stops you from actually doing this in one step, right? Nothing stops you from taking, you know, the data distribution just adding like a wild bunch of noise because then you're also approximately normally distributed. Maybe not, I don't know, you maybe end up at some other distribution. But I mean certainly if you, like you can do the reverse, also you can train a neural network to do it in one step. In fact that's a little bit what GANs do, right? But if you want to do this in this sort of manner where you model all the distributions, notice this is a very different language than GANs. Here it's all kind of in the distributional semantics. If you want to do this and you want to say well I modeled the reverse as a normal distribution, this is just not true if you took large enough steps, right? But if you take very tiny steps you can adequately make sort of the argument that the normal distribution is kind of okay for this to work. And of course it makes life easier after that. So they need the tiny steps because in the tiny steps they're able to sort of, the modeling assumptions hold, also I guess it works better. And then you can define the loss function right here. So they say the combination of QMP is a variational autoencoder and we can write the variational lower bound as follows. So I'm not sure if I have ever gone over variational autoencoders, but they, it's very much, it's very similar to here. What you can do is you can define this variational lower bound which essentially boils down to saying I would like the distribution that I want a model and the thing I actually output to be close together, right? So this is the reverse process that my neural network does and this is the thing that I actually would like to model, okay? And we're going to, this is the thing that needs the entire data distribution. We're going to look at that in just a second. So yeah there are some other terms here but you can get around that and the last term right here, like the last term, you just assume that's kind of a Gaussian. So really it comes down to does the distribution that your neural network outputs match what you, what it actually is? And here you can see the sort of proxy for well this needs the whole data distribution is the following. If I tell you that this is the process by which I derive the data, right? And I ask you what is the reverse distribution of one of these steps? You can't possibly compute that, right? Accurately because you don't know the data distribution. However what you can do is for this particular sample you can compute it if I tell you that you know this is the process by which I derived it and also if I actually give you x0 right here. If I give you that then you can do, you can do, you can calculate and that's what they show here, you can actually calculate this distribution. You can say what is the actual distribution I'd like to model and that's going to be a normal distribution but what just, it makes sense right? In this case like if this is, if this is the forward process and I give you x0, if you already know the result you can calculate the distribution. So that's what they derive right here and that is dependent of course on your noise scale which is like all over the place in this, in these formulas but you can calculate that and this is a Gaussian and they model the output of the neural network as a Gaussian so these KL divergences just they become really easy to calculate and then you have a loss function. So now they say how do we, how do we actually train this thing in practice? Because it turned out in the last papers that this thing right here, the actual variational lower bound isn't too effective. I think that's what they're saying. So yeah what the, what the authors here say is they go back to previous paper and say the previous paper found that modeling the noise here is the best way to do it. So the question is how exactly, what exactly does the neural network do? Like the neural network could do many things, it it could actually just predict this mean parameter which we've talked about right? The neural network could simply, I give you an image and you tell me what's the most probable image where it comes from or sort of the mean and also give me the covariance but also what you could do is you could just model the noise, that's a different thing. You could model the noise and that's equivalent from a computational perspective right or from a conceptual perspective. If I give you again this image you can either tell me where it came from or equivalently you can tell me what's the noise that I've added right and you tell me what this, you've probably added this noise. It's a, this is a both the same from an information perspective, however the authors previously noted that the modeling the noise is better just from a neural network training standpoint. In fact they make a point here to define a new loss function that simply estimates, that simply says well the noise that I output from the neural network should approximately match the actual noise that I've added right because I know what noise I sampled in my forward noise process and that works better. However these authors here say okay this does not tell you anything about the covariance because that only tells you something about the mean and the old authors found that we don't actually need the covariance we just we fix it and that works a lot better or equally well to actually learning it and the authors here say maybe they've you know missed something maybe they've missed the opportunity to learn the covariance so this was a little bit of a rant but to repeat we define this noising process and then we try to learn a neural network that reverts that noising process. In order to do so we train a neural network to reverse each of the little steps that we do right here and the way we do it is the neural network will predict the distribution of the predecessor so given a noised image the neural network will output the distribution modeled as a normal distribution over where that noisy image probably came from and it the previous authors have said well there are two things to model there is the mean and the covariance and we find first of all if we just fix the covariance that's enough right we fix the covariance matrix to the noise scale that we know we applied and good enough we don't actually need to model the the true covariance matrix just from an empirical standpoint and then when we model the mean we don't model the mean directly we actually model the noise and which is equivalent but it works better from a neural network standpoint. The authors now say maybe you've missed an opportunity learning that covariance matrix because it's one thing to say this is probably a Gaussian right it's another thing to say this is probably a Gaussian with completely isotropic covariance matrix and you would expect the second one is easier but also it's more wrong so that's what we're that's what we go about here so they say can we improve the log likelihood right here and the first topic they go into is learning this covariance matrix and what they discover I want to say is that if you fix the covariance matrix right here you have to know what scale to fix it at which is dependent on the the noise that you applied in the forward process right so you applied some noise and you can calculate what the average covariance of the reverse step should be at that particular time step and in fact you can derive an upper and a lower bound so if beta here is their schedule for noise then these are the two bounds so this this is the actual beta you used in that step the noise scale and this is sort of an accumulated noise scale up until that step these are the two bounds in which in which the noise can be right the noise level or the covariance and the previous author said well we can use either one of them it's actually fine it doesn't matter and these authors say okay look at this right here this is the ratio between the two so the ratio between the upper and the lower bound as a function of the diffusion step now especially if you go to a large amount of step size you see this immediately clamps at one right so there is like almost no difference between the upper and the lower bound which is probably why the other authors estimated it didn't matter now these authors go further and they say well if you just try to learn like a number neural networks are kind of bad at regression right so if you tell neural network learn me any number on the number string whatever you call that in English if there me any number like here's one here's two here's three like here's 500 any number whatsoever but however the only actual right answers are going to be a tiny tiny sliver between like like the ratio between them is going to be a tiny tiny sliver somewhere in in like three orders of magnitude down the neural networks going to have trouble hitting these correctly so the way they do it is they reparameterize the the how they predict the covariance matrix in fact what they come up with is they simply learn an interpolation parameter V right here to interpolate between the upper and the lower bound and that turns out to be quite a good decision because now the neural network can predict a number V for each dimension which is between 0 and 1 right and that's neural networks can predict stuff between 0 and 1 they're pretty good at it and the whole rest the whole scale issue will be taken care of by interpolating between the two valid bounds so this this is one thing they're able to learn the covariance matrix now and that boosts them a bit and then they also look at the noising process right here and they say well if you look at this and this is something I find a bit shady they say if you look at this and this top row is what is currently done with the noise schedule that is usually defined it's just kind of noisy a bit too much right like from here on out there's just noise right could we not schedule this a little bit such that the drop-off is more gradual that might help a lot and so they come up with a new schedule that does this now this seems very subjective right you know this is you as a human looking at it they they do some experiments here where they say we measure the inception distance as we just leave away a fraction of the reverse diffusion process so they wonder how many of these steps can we just leave away and still end up with something that's fine like can we can we just skip the first step of the reverse process and start here can we skip five steps and start here it turns out in the linear schedule you're just able to skip a lot more steps which gives you an indication that those steps weren't really helpful and it'd probably be better that you define a schedule where all of the steps are helpful so that's what they what they come up with you can see the linear schedule right here is dumping pretty fast like it goes down pretty fast while their new cosine schedule is much much slower like this these are now actual practical considerations that are just done by kind of looking evaluating a bit empirically and then going and saying well can't we do something better now this something better they admit that themselves isn't by no means the best thing you can do it's just something better like ultimately you would want the same step in the noising process probably to contribute equally to the quality of the entire system you know but that's what they do the last thing is very similar they say we reduce the gradient noise so they observe if they use they have now two loss functions right they have the loss the original loss function where you simply look at the L2 distance between the noise and the predicted noise like no variational lower bound yada KL divergence and who needs that crap right that's what they call the simple objective now the simple objective doesn't contain the covariance so what they would like to do is they would like to go back to the variational objective and that's the blue line here I know you can't really read it but that's the blue line here and you can see only is it pretty noisy it's also well okay I guess it's like it's pretty noisy the loss curve if they mix the variational objective together with the simple objective they get a better loss curve you see that right here this this is this hybrid loss it's the orange loss it it's still noisy their new loss which they call resampled loss that's again the variational lower bound loss but in a sampled in a different way is the green line which is much much smoother and also lower and that comes from this fact right here if you look at the sorry not from this right here is it okay so they what they say is if you look at the process like this noise process here and you look at where the actual loss comes from where does the the majority of the loss contribution come from they notice that the majority of the loss contribution comes from the first step so there is a real imbalance of how much these individual steps in the noising process differ from like contribute to the overall loss and say well if you know if we just add all of them up equally right because what do you need to do to train these neural networks you need to start off with a clean image then sample some step like some step you say okay I'm gonna now train the t equals 205 network right so you add noise 205 times you can do this in one go by the way but essentially you add noise 205 times you get here right you add noise once more to here and now you have your if your training sample right here you can calculate the the distribution you want to match by also including this one as we discussed and you good right so this is one training sample the next training sample is you select a different t and you produce another training sample it's one now if the first few steps are much more important than you know the step at t equals 5000 and you're just sampling t uniform you will end up with you know a correct on probably unbiased estimate of your laws oh sorry of your loss however it will be super duper noisy so they're saying can't we just focus a bit on where a loss actually occurs so they devise a scheme to do important sampling notice that the different terms of of the variational around have greatly different magnitudes and figure two where's which one's figure or figure two figure two oh there we go that was the plot so here is the step in the noising process and here is the loss term magnitude and you can see that the the first few steps they have a really lot like a larger loss this is a log scale right on the left then the last ones so they devise an important sampling scheme to counter that this is not specific right to this particular technique you can use this anywhere where different samples have very different contributions to loss you can choose to focus on the ones where the loss is high and I will not give you that will give you a biased estimate of your loss however it might decrease your variance by quite a bit and that's what they they end up with they in this paper they end up with something that's competitive but not better than the best GANs however it already it already looks pretty good they also investigate model size but I don't want to go into this I actually want to jump quickly into this next paper where they improve again on their models to make them actually better than GANs and the improvements right here are much more I don't know I want to say boring because it's like okay architecture improvements so we're going through the same process that we've gone through with GANs where it's like well here's a tweak here's a tweak here is an architecture a better architecture here is kind of a better loss function regularizer whatnot and it's quite conceivable right that this these models here come to the level of GANs now whether they are actually you know better than GANs like I think this is remains to be seen because you know it also depends quite a bit on how much compute you put into this and then you also have to see that here you have to it went when you want to sample a sample you have to input the sample and then do this denoising process a bunch of times like thousands of times until you end up with the data sample now they do have a kind of a trick going into another model class where you only have to have they say 25 of these steps so it's pretty cool but still like that's 25 forward passes through this neural network that predicts the denoising where again is just like you sample once the latent you you ship it through the GAN and you end up with a you end up with a sample and I'm actually wondering if GANs could take some sort of lesson from here we'll we'll look at this after we look at this right here which is what I think is the kind of cool improvement that they do in the new paper which is where they say classifier guidance so they say if you use GANs for conditional image synthesis so if you if you conditionally if you use a GAN to create images that are of a particular class condition on a class label they make heavy use of class label okay so they say it makes sense to explore different ways to condition diffusion models on class labels we already incorporate class information into normalization layers so you have different normalization layers for different classes here we explore a different approach exploiting a classifier to improve a diffusion generator as they say the kind of a previous work two previous works show one way to achieve this we're in a pre-trained diffusion model can be conditioned using the gradients of a classifier in particular we can train a classifier and on noisy images and then use the gradients to guide the diffusion sampling process towards an arbitrary class label in this section we first review two ways of driving conditional sampling processes we then describe how we use such classifiers in practice to improve sample quality so the idea here is that if you have class labels together with your data set you can train a classifier on not only the data set but also noisy samples of that data set right and then you can use that classifier in order to guide the process so this is what we're dealing with right here they say well instead of simply reverting the process which would be this part right here like instead of simply reverting the noise process if I tell you what label that image is from like what class that image is from can you do a better job right so if I in our original example if I tell you if I give you a noisy picture of a house and I tell you about by the way this is a house you're much more able to tell me what the original image was or alternatively what the noise is that I've added to the image so if you write this as a as a distribution as we did so far you can say if you want you want to predict the previous image from the next image and the class label and you can pull this apart into these two components which is the old component like how likely is the previous image given the noisy version times the what they I think what they call this this the prior right yeah they call this prior you can see that if you just like kind of ship this out it just it just swaps well I don't know how to explain this properly but I mean this is this is just probability manipulation so if you have a probability product between whatever we had before and how likely is that is the class label under this so this is sort of you want an image that makes sense given the noisy image but you also want you want an image that's that Mac that is a high probability of being of the class that you want to produce and of course this is exactly a classifier on the right which you can use so since we it since our model of so the question is what are these two things and can we sort of derive an easy form how we can work with this so the first thing we've already seen and we model this as a normal distribution and if we know the mean and covariance of that thing the the log is simply this form so you should recognize this as being just the form of the normal distribution this here is the normalization constant if you work in log space that is added and it is a constant so if you're just interesting in minimizing a function you might as well leave it away the second part is a bit more tricky but you can say well this distribution right here I can do a Taylor expansion around the predicted mean right then the first order Taylor expansion which becomes this so this is it's just kind of a vector form of the Taylor expansion if you've never seen it so this is this is f of x 0 right here and this is the this is f of x 1 this is the derivative at the point x 0 how do you say it is the derivative according to X at X 0 times X minus X 0 right here it's the same thing okay so what you end up with is this form right here and if you calculate this through what you end up with is the entire distributions of the product of the two things in log space looks like this and therefore therefore the distribution that you're looking at is a distribution you're saying here somewhere is the image that is the noisy version you ask your two models you ask your first model well what's what's an image or where does this likely come from and that model tells you well it's probably from here and the the covariance is like so like I think that's where it it came from when it was noised and the other model simply shifts that towards it says well but if you shift it a bit like this and it actually comes from here then it's much more likely under the classifier that's what you have you have the predicted mean right here that says where does it probably come from given that I've had a noise and this part right here says so the G is the gradient of the classifier with respect to the input this says well but if I shift it like this a little bit it becomes much more likely under the class and given that you've already told me what the class label is right I'm just gonna choose I'm I'm gonna choose to shift over here so this is what the classifier buys you the classifier will tell you without the classifier I think it comes from here but now that I know it comes from this class I can refine my belief of where it came from and that's how you become more accurate like if this is really the class it came from you're gonna be more accurate right given that the assumptions of the Taylor expansion hold now here as you can see we're really kind of getting close to the land of the GANs okay now if as soon as you have something like this where you derive the gradient of a model right of a classifier model with respect to its input and you use that gradient to sort of guide your search that is it's it's very close to a GAN it's very close to models that do score matching actually this very bad at explaining score matching but it is exactly sort of this you use the gradient of the log probability in order to model a distribution and I wonder if GANs can't sort of take a bit of a lesson from here like I wonder what happens if you don't have a GAN that just goes from noise to data but again like like here you have like little GANs or the discriminators at intermediate steps right that do their discrimination you can generate training data pretty easily again by doing this reverse noising process you can generate training data and you just have like little discriminators that discriminate between true data that was actually noised and data that you just produced and by you just produced I don't know what I'm just coming up with this right now this is not a prepared thing by the way you could probably use your existing model to somehow forward propagate and then you noise whatever that is right and then you have generated data and true data in all their noisy fashion and you can do discriminator at each level I'm not sure maybe it works maybe it won't I'm just saying maybe there is a way to get sort of the best out of both worlds because this this here like if this weren't a class label but kind of a label of true and fake data this would very much look like again and maybe we don't need all of this distribution distribution Schmistribution I guess it's a forever war between people who do formally correct their things and people who just throw everything out that doesn't contribute to the end quality in any case they also go into this DDIM models which are different class of models very close here but they do they they say to this and we use a score based conditioning trick adapted from these other papers which can leverage is the connection between diffusion models and score matching so there is an actual formal connection and you can use that to kind of actually what I said right now get rid of the noise in the system and directly sort of directly predict the predecessors and that will still end up at a formally correct thing and that allows you I think with this trick they don't have to sample as much or they they only use 25 reverse steps instead of 4000 which is important right and the last thing they discover if they discover like a hyper parameter like if you scale classifier gradients like this you have to observe that the classifier gradients are in log scale so technically the way multiplication behaves with a log is it becomes an exponent right here and that simply means that this distribution also you know the normalization that distribution is going to be more or less peaky and define depending on that hyper parameter and they notice that you can make it sort of more peaky and then the sample quality becomes higher right I think they a issue that the variational auto encoders had for a long time is that they were sort of blurry and so on and you know this is this is a little bit I think how that might be fixed though this is you know the classifier gradients so you want to make the classifier gradients more peaky which means that you get a stronger signal for from them which apparently results in better things so here all the results you see whenever they say 80m that's their model they have several variations namely this dash G here is the classifier guided version and whenever they say 25 steps that is the version without the noise with the trick connection to score matching yep so you can see in sort of the FID scores they do beat a big GAN on these tasks yeah maybe they you know the GANs will one up taking some tricks from here or maybe it's quite possible that these models will go beyond GANs because we've poured a lot of effort into GANs and not so much yet into these models into the denoising models and you know the samples look pretty good so the left is GAN and the middle here it's a bit small but the middle here is is their model and I have actually like I've gone through this entire image net class I've looked at every single image to try to find these images and I can I can tell you that the images are not in the training or the validation data set here are these are images from the actual data set they're pretty close but still I always fear a little bit that you know at some point a model is just gonna learn to copy the data all right so that was it I know this video is already too long if you're still here thank you I hope you've enjoyed this and I'll see you next time bye bye
 Hello! These are generated images from a new model, actually a new class of model.
 It's been around for a while, but for the first time this new class of model has
 been pushed to the point where the images they produce not only look really
 nice and look like something you've come to expect from the latest and
 greatest GAN models, but also they are better in the standard metrics we use to
 evaluate GANs, specifically here in the FID, the fresher inception distance.
 The paper we're going to talk about today is called
 Diffusion Models Beat GANs on Image Synthesis. It's by Prof. Dariwall and
 Alex Nicole of OpenAI. Already in the title they're pulling no punches,
 just be like this beats GANs. In this paper they're mainly talking about
 improvements to this new class of models which they call diffusion models.
 I would like to dive a bit more into what diffusion models are instead of
 just telling you what the improvements of this paper are, because I think most
 people haven't come in contact with these types of models yet. They thoroughly
 reference another paper which is called Improved Denoising Diffusion
 Probabilistic Models by themselves. In this paper they more
 develop these new models than in the other paper. The paper here, as you
 can see, is just three months younger than the other paper. This is
 really close. I think this paper is more insightful into what these models are.
 That being said, by the name Improved right here you can also see
 that this is not the seminal paper of these types of models. If you're
 interested in that you have to go back even further. However we're going to
 look at this and we're going to look at the new paper and see what are all the
 things that lead to this new class of models being better than GANs.
 Specifically we're going to talk about DDPMs, Denoising Diffusion
 Probabilistic Models. They're a bit like a variational auto
 encoder, a little bit. We'll go through that.
 If you feel that this was helpful please do share it out. It's been
 a pleasure bringing this to a lot of people and if you do it will just be
 more people. We'll have more fun. They say that Denoising Diffusion
 Probabilistic Models, DDPMs, are a class of generative models which have recently
 been shown to produce excellent samples. We show that with a few simple
 modifications DDPMs can also achieve competitive log likelihoods while
 maintaining high sample quality. In this paper they take these models, these
 DDPM models, and they say look we can push those models to push their
 log likelihood. There are a number of metrics that generative models track.
 It's not as easy as the validation set accuracy in a classifier. Log
 likelihood is one of the metrics that these models track. Here they say
 well we can get competitive log likelihood while maintaining high sample
 quality, which is a nice way of saying we don't beat GANs yet. In the next
 paper, the one I showed you before, they actually do beat GANs on
 the standard metrics and also the samples look quite impressive.
 The DDPMs have been around before but they go into a quick overview right
 here, which is what I think is quite appropriate for us to dive in.
 The philosophy here or the whole purpose behind this is they
 say let's imagine I have an image of my house right here.
 I have an image of a house and I define a process, what they call a forward
 noising process. This forward noising process takes the image and it just
 adds a little bit of noise to it, like epsilon noise that's sampled from some
 standard distribution like a Gaussian. You just sample a bit of noise and
 you just add it to that image. You have the same house but there'll be a
 bit of noise on it. Then you do it again. You sample another bit of
 noise and you do it again. As you do this
 over many steps, and here they actually notice that the previous
 authors were using a thousand steps and if they just increase that to four
 thousand steps, the log likelihoods go better. In any case you do
 this for many steps, thousands of steps in this first instance. You do this, what
 are you gonna end up with? The argument here is that if you do this for
 so many times, for so long, over so many steps, you're going to end up with random
 noise itself. This is ish according to some kind of normal
 distribution. You just assume. You can actually prove this that if you
 do enough steps, like if you do infinitely many steps, it goes actually
 towards just noise. Whenever you're done with this, there is no more
 information about the original image than actually sampling from this
 distribution right here. You have successfully defined a process that
 takes you from the image space. This here is from the data space that
 takes you from the data space to a known distribution, which is the normal
 distribution. Now here is the logic. If we could invert this, if we
 just somehow could invert this mapping, if we could have a process that knows
 if I give you an image with some noise, can you tell me what image that came
 from? Is that doable? It's not, it's not, it's thinkable. If I give you
 this image with some specks of noise on it and I ask you, could you
 please give me, I tell you, I'm the oracle, I tell you, look I've taken
 some image that already had a bit of noise on it, but I've added more.
 I've taken an image, I've added some noise. What was the original image that I, I
 don't tell you what the noise is, right? I just tell you the noise comes from
 whatever, a normal distribution, I've added it. What was the original image?
 Now you looking at this image, you'll see, you know, this could be a house. So not
 quite sure, but you know, this might be something like this might be the
 original image and this here I'm not really sure about if this is noise. So
 you're gonna sort of revert that process a little bit, right? Knowing that this is
 how the image came to be, you as a human, if I told you, you could
 approximately reverse that process. That of course requires you to know something
 about these images, right? That like, it requires you to know what a house looks
 like and when you see something like this, that well, you know, probably because
 I don't tell you which ones are the noise and which ones aren't. So that's
 the trick, right? If I just told you, well, all the orange stuff is noise, right? But
 you just see, you just see this all in mono color, but you know kind of, okay, so
 this here looks like it's from the image itself, but then this here is just kind
 of a spec and that just kind of, might just be noise, maybe not, right? But then
 this here, I'm pretty sure it's just noise and not part of the original image.
 So you could do that and the question is, can we learn a function that does this
 reverse process? If we can do so, right? If we can learn a function, function of
 course that's going to be some kind of neural network-ish thing. We can learn a
 function where I give you an image with noise and I tell you, by the way, so this
 is maybe time step zero, this is t equals zero, t equals one, t equals two, and so
 on. Well, you can't see that. If I tell you, okay, here is an image, this happened
 at t equals 50, can you give me the t equals 49 image that this came from?
 Alright, and this is the whole principle. We're going to, we can generate training
 data for this neural network very easily because we just take data and we run
 them through the noise process forward, right? Then we have plenty of training
 data for every step of this pipeline, right? In fact, we don't train a, we don't
 train a different phi function for every step. As you can see, the phi function
 simply takes the time or can take the time as an input. It's certainly possible
 otherwise, or it's possible to not tell it at all, right? Then you, it has no clue.
 So yeah, if you do this, you can generate training data and then the
 idea is you can just run this process in reverse and arrive at the original
 sample. And even more, because this here is actually the normal distribution, you
 can now sample random noise from that normal distribution, right? You can feed
 it to this process and this process, who has learned to map the data distribution
 to the normal distribution and can reverse that process, will give you some
 sort of data distribution sample for your input that you sampled from the
 normal distribution. All right, this is the idea and it's quite tricky to get
 this to work, as you can imagine, but let's not forget that GANs also have been
 quite tricky to get to work. It's just maybe there has been a bit more work
 going into GANs. All right, so formally this goes as follows. We define this
 forward-noising process, right? We sample this from the data distribution. We
 sample x0 from the data distribution. We define this forward-noising process Q,
 which produces x1 through xt, so capital T as the end here. And we, by
 adding Gaussian noise at time t with some variance, okay, so you can have, you
 can have, it's zero mean Gaussian noise, I believe, maybe. Yeah, it's, well, you
 scale, but you define this variance schedule right here. That's also your
 choice, right? You choose what kind of noise you want to add, but ultimately
 you take, ultimately, the distribution of the things you produce via that
 noising process, given that you start at the data sample x0, you simply define as
 this product of distributions. So you start with, this just means you start
 with x0 and then you go from x0 to x1 and then you go from x1 to x2 and so on,
 okay? And each of these steps is an independent application of noise. As you
 can see here, this is one of those steps. So what you're saying is that the
 distribution of the next sample right here is going to be a normal
 distribution that's going to be centered at this thing right here and its
 variance is this thing right here. So you can see that the assumption here is you
 use noise that has a diagonal covariance matrix, okay? This is, I guess it's
 reasonable. It certainly makes computing things easier, right? The other thing here
 is that you can see this Gaussian is centered at the last sample but down
 scaled by this factor right here and I think, like, this is a choice again by the
 modelers but I think this is also due to the fact that makes computation easier
 because I guess if you don't have this then, you know, you start somewhere and
 you add noise and you sample something, you add noise, you sample something, maybe
 this would grow indefinitely and you sort of need to rescale things such that
 you can make this statement right here. Given sufficiently large T and a well
 behaved schedule of beta, the latent XT, so the very last step, is nearly an
 isotropic Gaussian distribution, okay? That's the entire point. So if you do it
 like this, which is a choice, but if you do it like this then at the end if you
 do enough steps, infinitely many steps, then you end up at an isotropic Gaussian
 distribution. Thus, if we know the exact reverse distribution, we can sample from
 the Gaussian and run the process in reverse to get a sample from the data
 distribution. Then they say, however, since the reverse distribution depends
 on the entire data distribution, we approximate it using a neural network as
 follows. So this statement can be a bit weird in the first instance. This
 depends on the entire data distribution, right? Because it's very close to
 this thing right here and this thing right here depends on nothing, right?
 This you just define, you just say I'm gonna add random noise to something and
 that's my next distribution. It only depends on the input image right here.
 The way to see it, that this depends, the reverse depends on the entire data
 distribution, is exactly what I said before. If I give you the, like if I
 give you this picture, I'm not gonna actually tell you right where the noise
 is. So I give you this picture and I tell you this is a drawing from a
 very small child, because that's my drawing level, and I've just added a
 bunch of noise to it. Could you tell me what the original drawing was? This
 is very different from me saying here is a drawing from a small child, please add
 noise to it. That's easy, I just did this, right? I was just called, I just did it.
 But if I tell you what was the original image, you have to take into account the
 entire world. You know about how small children draw, what kind of
 motives they usually draw and so on, and that's how you are able to come up by
 saying well it was probably something like this.
 This needs your knowledge of the entire data distribution. That's
 why they say it right here. So they say well we can't, we like, we can't just
 have the entire data distribution otherwise, you know, we wouldn't even
 have the problem in the first place. So what we can do is we can approximate one
 of these steps using a neural network. So we have a neural network that
 takes as an input, as I said, it takes as an input the noised version of the image
 and it gives you as an output, it's a bit like this is, it gives you, I told you
 give me the image that this came from, in this case what they want is give me a
 distribution over images where that could have come from, right? And again
 they say this, they model this as a Gaussian right here and the neural
 network will produce the mean and the covariance matrix given the image. So the
 neural network is supposed to look at the image and decide okay what's the
 Gaussian distribution of images where that probably came from? And this is a
 strong assumption, right? The fact for example that you know this is a Gaussian
 distribution, like this is adequately modeled as a Gaussian
 distribution, it's a strong assumption that you can only make because you make
 these very small steps. Because nothing, I mean nothing stops you from actually
 doing this in one step, right? Nothing stops you from taking, you know,
 the data distribution just adding like a wild bunch of noise because then you're
 also approximately normally distributed. Maybe not, I don't know, you maybe
 end up at some other distribution. But I mean certainly if you, like you can do
 the reverse, also you can train a neural network to do it in one step. In fact
 that's a little bit what GANs do, right? But if you want to do this in this sort
 of manner where you model all the distributions, notice this is a very
 different language than GANs. Here it's all kind of in the
 distributional semantics. If you want to do this and you want to say well I
 modeled the reverse as a normal distribution, this is just not true if
 you took large enough steps, right? But if you take very tiny steps you can
 adequately make sort of the argument that the normal distribution is kind of
 okay for this to work. And of course it makes life easier after that. So they
 need the tiny steps because in the tiny steps they're able to sort of, the
 modeling assumptions hold, also I guess it works better. And then you can
 define the loss function right here. So they say the combination of QMP is a
 variational autoencoder and we can write the variational lower bound as follows.
 So I'm not sure if I have ever gone over variational autoencoders, but
 they, it's very much, it's very similar to here. What you can do is you can
 define this variational lower bound which essentially boils down to saying I
 would like the distribution that I want a model and the thing I actually output
 to be close together, right? So this is the reverse process that my neural
 network does and this is the thing that I actually would like to model, okay? And
 we're going to, this is the thing that needs the entire data distribution. We're
 going to look at that in just a second. So yeah there are some other terms here
 but you can get around that and the last term right here, like the
 last term, you just assume that's kind of a Gaussian. So really it comes down to
 does the distribution that your neural network outputs match what you, what it
 actually is? And here you can see the sort of proxy for well this needs the
 whole data distribution is the following. If I tell you that this is
 the process by which I derive the data, right? And I ask you what is the reverse
 distribution of one of these steps? You can't possibly compute that, right?
 Accurately because you don't know the data distribution. However what you can
 do is for this particular sample you can compute it if I tell you that you know
 this is the process by which I derived it and also if I actually give you x0
 right here. If I give you that then you can do, you can do, you can calculate and
 that's what they show here, you can actually calculate this distribution. You
 can say what is the actual distribution I'd like to model and that's going to be
 a normal distribution but what just, it makes sense right? In this case like if
 this is, if this is the forward process and I give you x0, if you already know
 the result you can calculate the distribution. So that's what they derive
 right here and that is dependent of course on your noise scale which is like
 all over the place in this, in these formulas but you can calculate that and
 this is a Gaussian and they model the output of the neural network as a
 Gaussian so these KL divergences just they become really easy to calculate and
 then you have a loss function. So now they say how do we, how do we actually
 train this thing in practice? Because it turned out in the last papers that this
 thing right here, the actual variational lower bound isn't too effective. I think
 that's what they're saying. So yeah what the, what the authors here say is they go
 back to previous paper and say the previous paper found that modeling the
 noise here is the best way to do it. So the question is how exactly, what exactly
 does the neural network do? Like the neural network could do many things, it
 it could actually just predict this mean parameter which we've talked about right?
 The neural network could simply, I give you an image and you tell me what's the
 most probable image where it comes from or sort of the mean and also give me the
 covariance but also what you could do is you could just model the
 noise, that's a different thing. You could model the noise and that's
 equivalent from a computational perspective right or from a conceptual
 perspective. If I give you again this image you can either tell me where it
 came from or equivalently you can tell me what's the noise that I've added
 right and you tell me what this, you've probably added this noise. It's a, this is
 a both the same from an information perspective, however the authors
 previously noted that the modeling the noise is better just from a neural
 network training standpoint. In fact they make a point here to define a new loss
 function that simply estimates, that simply says well the noise that I output
 from the neural network should approximately match the actual noise
 that I've added right because I know what noise I sampled in my forward
 noise process and that works better. However these authors here say okay this
 does not tell you anything about the covariance because that only tells you
 something about the mean and the old authors found that we don't actually
 need the covariance we just we fix it and that works a lot better or equally
 well to actually learning it and the authors here say maybe they've you know
 missed something maybe they've missed the opportunity to learn the covariance
 so this was a little bit of a rant but to repeat we define this noising process
 and then we try to learn a neural network that reverts that noising
 process. In order to do so we train a neural network to reverse each of the
 little steps that we do right here and the way we do it is the neural network
 will predict the distribution of the predecessor so given a noised image the
 neural network will output the distribution modeled as a normal
 distribution over where that noisy image probably came from and it the previous
 authors have said well there are two things to model there is the mean and
 the covariance and we find first of all if we just fix the covariance that's
 enough right we fix the covariance matrix to the noise scale that we know
 we applied and good enough we don't actually need to model the the true
 covariance matrix just from an empirical standpoint and then when we model the
 mean we don't model the mean directly we actually model the noise and which is
 equivalent but it works better from a neural network standpoint. The authors
 now say maybe you've missed an opportunity learning that covariance
 matrix because it's one thing to say this is probably a Gaussian right it's
 another thing to say this is probably a Gaussian with completely isotropic
 covariance matrix and you would expect the second one is easier but also it's
 more wrong so that's what we're that's what we go about here so they say can we
 improve the log likelihood right here and the first topic they go into is
 learning this covariance matrix and what they discover I want to say is that if
 you fix the covariance matrix right here you have to know what scale to fix it at
 which is dependent on the the noise that you applied in the forward process right
 so you applied some noise and you can calculate what the average covariance of
 the reverse step should be at that particular time step and in fact you can
 derive an upper and a lower bound so if beta here is their schedule for noise
 then these are the two bounds so this this is the actual beta you used in that
 step the noise scale and this is sort of an accumulated noise scale up until that
 step these are the two bounds in which in which the noise can be right the
 noise level or the covariance and the previous author said well we can use
 either one of them it's actually fine it doesn't matter and these authors say
 okay look at this right here this is the ratio between the two so the ratio
 between the upper and the lower bound as a function of the diffusion step now
 especially if you go to a large amount of step size you see this immediately
 clamps at one right so there is like almost no difference between the upper
 and the lower bound which is probably why the other authors estimated it
 didn't matter now these authors go further and they say well if you just
 try to learn like a number neural networks are kind of bad at regression
 right so if you tell neural network learn me any number on the number
 string whatever you call that in English if there me any number like here's one
 here's two here's three like here's 500 any number whatsoever but however the
 only actual right answers are going to be a tiny tiny sliver between like like
 the ratio between them is going to be a tiny tiny sliver somewhere in in like
 three orders of magnitude down the neural networks going to have trouble
 hitting these correctly so the way they do it is they reparameterize the the
 how they predict the covariance matrix in fact what they come up with is they
 simply learn an interpolation parameter V right here to interpolate between the
 upper and the lower bound and that turns out to be quite a good decision
 because now the neural network can predict a number V for each dimension
 which is between 0 and 1 right and that's neural networks can predict
 stuff between 0 and 1 they're pretty good at it and the whole rest the whole
 scale issue will be taken care of by interpolating between the two valid
 bounds so this this is one thing they're able to learn the covariance matrix now
 and that boosts them a bit and then they also look at the noising process right
 here and they say well if you look at this and this is something I find a
 bit shady they say if you look at this and this top row is what is currently
 done with the noise schedule that is usually defined it's just kind of noisy
 a bit too much right like from here on out there's just noise right could we
 not schedule this a little bit such that the drop-off is more gradual that might
 help a lot and so they come up with a new schedule that does this now this
 seems very subjective right you know this is you as a human looking at it
 they they do some experiments here where they say we measure the inception
 distance as we just leave away a fraction of the reverse diffusion
 process so they wonder how many of these steps can we just leave away and still
 end up with something that's fine like can we can we just skip the first step
 of the reverse process and start here can we skip five steps and start here it
 turns out in the linear schedule you're just able to skip a lot more steps which
 gives you an indication that those steps weren't really helpful and it'd probably
 be better that you define a schedule where all of the steps are helpful so
 that's what they what they come up with you can see the linear schedule right
 here is dumping pretty fast like it goes down pretty fast while their new cosine
 schedule is much much slower like this these are now actual practical
 considerations that are just done by kind of looking evaluating a bit
 empirically and then going and saying well can't we do something better now
 this something better they admit that themselves isn't by no means the best
 thing you can do it's just something better like ultimately you would want
 the same step in the noising process probably to contribute equally to the
 quality of the entire system you know but that's what they do the last thing
 is very similar they say we reduce the gradient noise so they observe if they
 use they have now two loss functions right they have the loss the original
 loss function where you simply look at the L2 distance between the noise and
 the predicted noise like no variational lower bound yada KL divergence and who
 needs that crap right that's what they call the simple objective now the simple
 objective doesn't contain the covariance so what they would like to do is they
 would like to go back to the variational objective and that's the blue line here
 I know you can't really read it but that's the blue line here and you can see
 only is it pretty noisy it's also well okay I guess it's like it's pretty noisy
 the loss curve if they mix the variational objective together with the
 simple objective they get a better loss curve you see that right here this this
 is this hybrid loss it's the orange loss it it's still noisy their new loss which
 they call resampled loss that's again the variational lower bound loss but in
 a sampled in a different way is the green line which is much much smoother
 and also lower and that comes from this fact right here if you look at the sorry
 not from this right here is it okay so they what they say is if you look at the
 process like this noise process here and you look at where the actual loss comes
 from where does the the majority of the loss contribution come from they notice
 that the majority of the loss contribution comes from the first step so
 there is a real imbalance of how much these individual steps in the noising
 process differ from like contribute to the overall loss and say well if you know
 if we just add all of them up equally right because what do you need to do to
 train these neural networks you need to start off with a clean image then sample
 some step like some step you say okay I'm gonna now train the t equals 205
 network right so you add noise 205 times you can do this in one go by the way but
 essentially you add noise 205 times you get here right you add noise once more
 to here and now you have your if your training sample right here you can
 calculate the the distribution you want to match by also including this one as
 we discussed and you good right so this is one training sample the next training
 sample is you select a different t and you produce another training sample
 it's one now if the first few steps are much more important than you know the
 step at t equals 5000 and you're just sampling t uniform you will end up with
 you know a correct on probably unbiased estimate of your laws oh sorry of your
 loss however it will be super duper noisy so they're saying can't we just
 focus a bit on where a loss actually occurs so they devise a scheme to do
 important sampling notice that the different terms of of the variational
 around have greatly different magnitudes and figure two where's which
 one's figure or figure two figure two oh there we go that was the plot so here is
 the step in the noising process and here is the loss term magnitude and you can
 see that the the first few steps they have a really lot like a larger loss
 this is a log scale right on the left then the last ones so they devise an
 important sampling scheme to counter that this is not specific right to this
 particular technique you can use this anywhere where different samples have
 very different contributions to loss you can choose to focus on the ones where
 the loss is high and I will not give you that will give you a biased estimate of
 your loss however it might decrease your variance by quite a bit and that
's what they they end up with they in this paper they end up with something
 that's competitive but not better than the best GANs however it already it
 already looks pretty good they also investigate model size but I don't want
 to go into this I actually want to jump quickly into this next paper where they
 improve again on their models to make them actually better than GANs and the
 improvements right here are much more I don't know I want to say boring because
 it's like okay architecture improvements so we're going through the same process
 that we've gone through with GANs where it's like well here's a tweak here's a
 tweak here is an architecture a better architecture here is kind of a better
 loss function regularizer whatnot and it's quite conceivable right that this
 these models here come to the level of GANs now whether they are actually you
 know better than GANs like I think this is remains to be seen because you know
 it also depends quite a bit on how much compute you put into this and then you
 also have to see that here you have to it went when you want to sample a sample
 you have to input the sample and then do this denoising process a bunch of times
 like thousands of times until you end up with the data sample now they do have a
 kind of a trick going into another model class where you only have to have they
 say 25 of these steps so it's pretty cool but still like that's 25 forward
 passes through this neural network that predicts the denoising where again is
 just like you sample once the latent you you ship it through the GAN and you end
 up with a you end up with a sample and I'm actually wondering if GANs could
 take some sort of lesson from here we'll we'll look at this after we look at this
 right here which is what I think is the kind of cool improvement that they do in
 the new paper which is where they say classifier guidance so they say if you
 use GANs for conditional image synthesis so if you if you conditionally if you
 use a GAN to create images that are of a particular class condition on a class
 label they make heavy use of class label okay so they say it makes sense to
 explore different ways to condition diffusion models on class labels we
 already incorporate class information into normalization layers so you have
 different normalization layers for different classes here we explore a
 different approach exploiting a classifier to improve a diffusion
 generator as they say the kind of a previous work two previous works show
 one way to achieve this we're in a pre-trained diffusion model can be
 conditioned using the gradients of a classifier in particular we can train a
 classifier and on noisy images and then use the gradients to guide the diffusion
 sampling process towards an arbitrary class label in this section we first
 review two ways of driving conditional sampling processes we then describe how
 we use such classifiers in practice to improve sample quality so the idea here
 is that if you have class labels together with your data set you can train
 a classifier on not only the data set but also noisy samples of that data set
 right and then you can use that classifier in order to guide the process
 so this is what we're dealing with right here they say well instead of simply
 reverting the process which would be this part right here like instead of
 simply reverting the noise process if I tell you what label that image is from
 like what class that image is from can you do a better job right so if I in
 our original example if I tell you if I give you a noisy picture of a house and
 I tell you about by the way this is a house you're much more able to tell me
 what the original image was or alternatively what the noise is that
 I've added to the image so if you write this as a as a distribution as we did so
 far you can say if you want you want to predict the previous image from the next
 image and the class label and you can pull this apart into these two
 components which is the old component like how likely is the previous image
 given the noisy version times the what they I think what they call this this
 the prior right yeah they call this prior you can see that if you just like
 kind of ship this out it just it just swaps well I don't know how to explain
 this properly but I mean this is this is just probability manipulation so if you
 have a probability product between whatever we had before and how likely is
 that is the class label under this so this is sort of you want an image that
 makes sense given the noisy image but you also want you want an image that's
 that Mac that is a high probability of being of the class that you want to
 produce and of course this is exactly a classifier on the right which you can
 use so since we it since our model of so the question is what are these two
 things and can we sort of derive an easy form how we can work with this so the
 first thing we've already seen and we model this as a normal distribution and
 if we know the mean and covariance of that thing the the log is simply this
 form so you should recognize this as being just the form of the normal
 distribution this here is the normalization constant if you work in
 log space that is added and it is a constant so if you're just interesting
 in minimizing a function you might as well leave it away the second part is a
 bit more tricky but you can say well this distribution right here I can do a
 Taylor expansion around the predicted mean right then the first order Taylor
 expansion which becomes this so this is it's just kind of a vector form of the
 Taylor expansion if you've never seen it so this is this is f of x 0 right here
 and this is the this is f of x 1 this is the derivative at the point x 0 how do
 you say it is the derivative according to X at X 0 times X minus X 0 right here
 it's the same thing okay so what you end up with is this form right here and if
 you calculate this through what you end up with is the entire distributions of
 the product of the two things in log space looks like this and therefore
 therefore the distribution that you're looking at is a distribution you're
 saying here somewhere is the image that is the noisy version you ask your two
 models you ask your first model well what's what's an image or where does
 this likely come from and that model tells you well it's probably from here
 and the the covariance is like so like I think that's where it it came from when
 it was noised and the other model simply shifts that towards it says well but if
 you shift it a bit like this and it actually comes from here then it's much
 more likely under the classifier that's what you have you have the predicted
 mean right here that says where does it probably come from given that I've had
 a noise and this part right here says so the G is the gradient of the classifier
 with respect to the input this says well but if I shift it like this a little
 bit it becomes much more likely under the class and given that you've already
 told me what the class label is right I'm just gonna choose I'm I'm gonna
 choose to shift over here so this is what the classifier buys you the
 classifier will tell you without the classifier I think it comes from here
 but now that I know it comes from this class I can refine my belief of where it
 came from and that's how you become more accurate like if this is really the
 class it came from you're gonna be more accurate right given that the
 assumptions of the Taylor expansion hold now here as you can see we're really
 kind of getting close to the land of the GANs okay now if as soon as you have
 something like this where you derive the gradient of a model right of a
 classifier model with respect to its input and you use that gradient to sort
 of guide your search that is it's it's very close to a GAN it's very close to
 models that do score matching actually this very bad at explaining score
 matching but it is exactly sort of this you use the gradient of the log
 probability in order to model a distribution and I wonder if GANs can't
 sort of take a bit of a lesson from here like I wonder what happens if you don't
 have a GAN that just goes from noise to data but again like like here you have
 like little GANs or the discriminators at intermediate steps right that do
 their discrimination you can generate training data pretty easily again by
 doing this reverse noising process you can generate training data and you just
 have like little discriminators that discriminate between true data that was
 actually noised and data that you just produced and by you just produced I
 don't know what I'm just coming up with this right now this is not a prepared
 thing by the way you could probably use your existing model to somehow
 forward propagate and then you noise whatever that is right and then you have
 generated data and true data in all their noisy fashion and you can do
 discriminator at each level I'm not sure maybe it works maybe it won't I'm just
 saying maybe there is a way to get sort of the best out of both worlds because
 this this here like if this weren't a class label but kind of a label of true
 and fake data this would very much look like again and maybe we don't need all
 of this distribution distribution Schmistribution I guess it's a forever
 war between people who do formally correct their things and people who just
 throw everything out that doesn't contribute to the end quality in any
 case they also go into this DDIM models which are different class of models very
 close here but they do they they say to this and we use a score based
 conditioning trick adapted from these other papers which can leverage is the
 connection between diffusion models and score matching so there is an actual
 formal connection and you can use that to kind of actually what I said right now
 get rid of the noise in the system and directly sort of directly predict the
 predecessors and that will still end up at a formally correct thing and that
 allows you I think with this trick they don't have to sample as much or they
 they only use 25 reverse steps instead of 4000 which is important right and the
 last thing they discover if they discover like a hyper parameter like if
 you scale classifier gradients like this you have to observe that the classifier
 gradients are in log scale so technically the way multiplication
 behaves with a log is it becomes an exponent right here and that simply
 means that this distribution also you know the normalization that distribution
 is going to be more or less peaky and define depending on that hyper parameter
 and they notice that you can make it sort of more peaky and then the sample
 quality becomes higher right I think they a issue that the variational auto
 encoders had for a long time is that they were sort of blurry and so on and
 you know this is this is a little bit I think how that might be fixed though
 this is you know the classifier gradients so you want to make the
 classifier gradients more peaky which means that you get a stronger signal for
 from them which apparently results in better things so here all the results
 you see whenever they say 80m that's their model they have several
 variations namely this dash G here is the classifier guided version and
 whenever they say 25 steps that is the version without the noise with the trick
 connection to score matching yep so you can see in sort of the FID scores they
 do beat a big GAN on these tasks yeah maybe they you know the GANs will one up
 taking some tricks from here or maybe it's quite possible that these models
 will go beyond GANs because we've poured a lot of effort into GANs and not so
 much yet into these models into the denoising models and you know the
 samples look pretty good so the left is GAN and the middle here it's a bit small
 but the middle here is is their model and I have actually like I've gone
 through this entire image net class I've looked at every single image to try to
 find these images and I can I can tell you that the images are not in the
 training or the validation data set here are these are images from the actual
 data set they're pretty close but still I always fear a little bit that you know
 at some point a model is just gonna learn to copy the data all right so that
 was it I know this video is already too long if you're still here thank you I
 hope you've enjoyed this and I'll see you next time bye bye