GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper openai glide diffusion clip-guided diffusion diffusion models clip-guided diffusion models generative models image to text generate image from text ai text to image machine learning text to image text 2 image classifier-free guidance noise process posterior variational lower bound log likelihood dalle dall-e ai drawing ai images
#glide #openai #diffusion

Diffusion models learn to iteratively reverse a noising process that is applied repeatedly during training. The result can be used for conditional generation as well as various other tasks such as inpainting. OpenAI's GLIDE builds on recent advances in diffusion models and combines text-conditional diffusion with classifier-free guidance and upsampling to achieve unprecedented quality in text-to-image samples.

Try it yourself: https://huggingface.co/spaces/valhalla/glide-text2im

OUTLINE:
0:00 - Intro & Overview
6:10 - What is a Diffusion Model?
18:20 - Conditional Generation and Guided Diffusion
31:30 - Architecture Recap
34:05 - Training & Result metrics
36:55 - Failure cases & my own results
39:45 - Safety considerations

Paper: https://arxiv.org/abs/2112.10741
Code & Model: https://github.com/openai/glide-text2im

More diffusion papers:
https://arxiv.org/pdf/2006.11239.pdf
https://arxiv.org/pdf/2102.09672.pdf

Abstract:
Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at this https URL.

Authors: Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today we'll look at Glide towards photo-realistic image generation and editing with text-guided diffusion models by Alex Nicol, Prafula Darewal, Aditya Ramesh and others of OpenAI. This paper on a high level, well, I'll just show you what you can do. I'm sure you've all seen this paper in one way or another. It is another paper that generates images given a piece of text, but this time it's not a GAN or anything like this or a VQVAE. This time it is a diffusion model. This is a different class of models and we'll go into what they are and how they work. But essentially you can see right here that the model that turns out of this and of course this being OpenAI, they train this on a massive scale and this model is really big, but what comes out of it is very, very much better than for example Dali, which always had this kind of blurriness to it. You can see right here a crayon drawing of a space elevator, pixel art, corgi pizza. So this is trained on a big scrape of images from the internet and as you can see the outputs are pretty stunning. So it gets, for example, the shadows right here, it gets them correctly, even the red on blue blending, it gets different styles like the Salvador Dali style. It combines different concepts, although maybe you know this has been seen on the internet somewhere, but it is able to combine different concepts. And given that these are diffusion models, you can actually do a bunch of more stuff with them. For example, inpainting is immediately accessible to this model. Now usually inpainting is accessible to diffusion models, however, they actually train an inpainting model on top of this. But in essence, a lot of stuff would be accessible. So this is now possible where you say, okay, I only want to change a part of the image like this part right here, you give a text saying a man wearing a white hat and the model generates the man wearing a white hat. This is very cool. You can do things like this where you first, so the pictures here are a bit confusing, but you first generate an image from a text prompt, like a cozy living room, then you get this living room and then here the user would annotate this window sort of would draw over it and will give the next text prompt. The next text prompt will be a painting of a corgi on the wall above the couch. And the model it's an inpainting, so this is the inpainting mode, the model would only be able to paint the green area. So it would sort of try to conform to the text using only the green area. And therefore, it would make this corgi picture on the wall right here, then the user goes further and says, well, now I'm going to paint this area right here. And I'm going to issue the prompt around coffee table in front of a couch, and the model will generate it and so on. You can see that this enables sort of an interactive creation of these scenery at the end, the couch, the couch in the corner of the room, so changing the entire wall right here, you can see the back of the room has some space. And now it's being changed to a wall. So this is the kind of stuff that's possible. Editing right here. Even what's this this sort of sketch editing where you don't only mask, but along with the mask, you provide sort of like a sketch as you can see right here. So this part here is blue, and then the part here is white. And that's also the mask that the the picture receives. And you can see that only one cloud in the sky today, it's sort of, you can guide even more so you can guide with text and you can guide with sketch color, and so on. So this is a very, very, very cool model, you can see the quality is very, very good. Here is for example, a comparison. These are real images from the MS, MS Marco data set, MS Coco, sorry. This is a data set of pictures with associated labels, so text descriptions of the picture. So you have some ground truth. So the ground truth here will be this one. And the label is a green train coming down the tracks. You can see Dali generates something neat, but it's sort of blurry. It's kind of cartoonish, as all the Dali pictures are if you look in this row. The last one's pretty good, but all the other ones are sort of elephants are more like blobs. And we've seen this in the Dali paper. It was impressive at the time, but this is way more impressive. And then their best model, this clip, sorry, this glide model with classifier free guidance, you can see right here, it generates like a high quality train that fits the image description. And you can see in the entire row right here, it's pretty good at doing that. So there are a lot of components to this model. And we're going to explore them a little bit. OpenAI has released in classic OpenAI fashion, they've released like a small, very filtered version of that model because they're worried about safety. Like anyone's going to believe them after GPT-2. They've just been doing this every single model, right? They're just like, oh, no safety, people can make deep fakes. Oh, no, like, no one's made a deep fake. Like GPT-2, all the worries, they were just not true. No one has used GPT-2 to spread around fake news. And no one like no one's going to use this model substantially to make very misleading pictures. But we'll get to that as well. All right, so what is a diffusion model? And that's sort of at the core of this thing right here. A diffusion model is a different type of generative model than maybe you're used to from like a GAN or a VQVAE. So in a GAN, a GAN is probably the closest right here. So again, it's sort of like a neural network with a bunch of layers. And what you do is you sample from some sort of a distribution, you sample some noise, right, you sample some noise, you get some noise vector. So here's a vector with just complete noise, every entry is noise. You put it through the network, the network generates pretty picture, and you train the model using a discriminator. In this case, you train the model to produce pretty pictures, given the noise and the noise acts sort of as a source of randomness. So the mapping is clear, you train to map from noise to picture. Now, a diffusion model goes in almost like a different direction. So what you do is during training, you have a data set, and you take an image. So from from a data set, you have a data set, you take an image out of it. Let's say this is your trusty, trusty cat, ta-da. And you're going to, you're going to put noise onto this image. So you're going to add noise and noise, let's represent that with sigma. No, I think they do, they do epsilon or eta in this in this paper right here. So you add that, and then you get a slightly noisy version of this. Let's just, let's just wiggle a bit, wiggle, wiggle, wiggle. And you do it again. So through adding noise, and you add lots and lots and lots of noise, okay, so every time you add a tiny, tiny bit of noise. And that means that more and more your picture is just going to be blurry and blurry and blurry. Now, if you do this for long enough, in the limit, you can prove that obviously, if you do this infinitely many times, what comes out at the end is going to be just nor normally distributed, if your noise is normally distributed, and you scale every time correctly, then whatever turns out is going to be normally distributed with some parameters here. So this right here is going to be a known distribution, if you, if you add noise for long enough, if you destroy all of the information that the picture has, then you'll end up with sort of an entry in a known distribution. However, every step that you do right here is very small, every step, you just add a little bit of noise. So technically, it's possible for a model to look at this picture right here, which is kind of a bit of a blurry version of the cat and predict and learn to predict the more sharp version of the cat. Okay, this is a foundation of many, many sort of denoising models, many up sampling models, super resolution models, what have you, okay, they do this in one step. But essentially here, we say the individual step is small enough such that the model can technically predict the can technically learn to reconstruct it. However, if we do it for long enough in, you know, going to infinity, the we are at a known distribution, namely the standard normal distribution. And these two things together mean that, well, if we have trained the model to reconstruct the individual steps, what we can technically do is we can now go ahead sample from this known distribution, right, because ultimately, we want to sample from the data distribution. But that's hard because we don't know it. But here we can just sample some noise from a known distribution, then put it through this process of reconstruction, all the way, all the steps that we did up here during training. During training, we just noise the noise and noise the images again and again. And again, we trained the neural network to for every step to reconstruct the previous step. So we can now just put it through this series of trained neural networks. In fact, it's just going to be one neural network that gets the index of the step as a parameter and outcomes an image, right outcomes a true data image. If these two things up here hold, then this should be possible. This is the basis for these diffusion models. So specifically, given a sample, that's what they say here, given a sample from the data distribution, this is x zero. So this is the data distribution, we produce a Markov chain of latent variables x one to xt, with everyone being a more noisy version, and xt finally being of a like a known distribution, because we do it infinitely, or a large number of times by progressively adding Gaussian noise to the sample. So you can see right here, we take xt minus one, we scale it down a bit, because if you wouldn't do that, the sort of the image would just increase in scale over because we just keep adding stuff. But this it's just a rescaling that there's nothing more happening here. So we, we, we add noise, this here is the mean of a distribution, the covariance matrix here is a diagonal, which essentially means we just add a bit of noise of the scale of alpha t. No, sorry, we just add a bit of noise, we rescale by alpha t, which is a scaling factor. And that's how we obtain the next step, the xt. So get we do this enough. So we take xt for the next step, we plug it in here, and then we obtain xt plus one, and so on. So if the magnitude of the noise added at each step is small enough, the posterior is well, well approximated by a diagonal Gaussian. That's what they say right here. So what does this mean, the posterior, it means that this is the reverse step, right, I have xt, and I'm looking to recreate xt minus one. So if the noise is small enough, then the posterior is well approximated by a diagonal Gaussian, and we have a hope to learn it with a neural network, right. Furthermore, if the magnitude of the total noise added throughout the chain is large enough, then the last step is well approximated by a known by a standard normal distribution. These properties suggest learning a model for this posterior, right, we have xt, we want to reconstruct xt minus one to approximate the true posterior. Okay, so we are going to learn a neural network that it doesn't exactly reconstruct the image, but this is a variational model. So what we're going to do is we're going to plug in xt into a neural network, the neural network is going to predict the mean and the covariance matrix of the next step. So we're going to do this, of the next step up the chain of the next step of the denoising chain. And then we can use this to produce samples, we simply sorry, we start we start with Gaussian noise, which is the end, and we gradually reduce the noise in a sequence of steps until we are at the data distribution, or at least the predicted data distribution. So this is not a new idea. This has been and I think I have the references open. This has been explored previously. For example, this is just an example right here. Denoising diffusion probabilistic models is one of the papers that introduced lots of these things you can see right here. These have still been trained on like just images as such. So this is the left is trained on a face data set, the right is trained on CIFAR 10. This is unconditional generation without a text prompt or anything like this. But you can see the same principle applies, we simply add noise during training and we learn a neural network to remove the noise to predict what the image would look like one noise step less. Here already, there was an invention that the paper here would make use of namely the loss function right here, we're going to look at that in just a second. So that's the second. So they say, while there exists a tractable variational lower bound, better results arise from optimizing a surrogate objective, which reways the term in the variational lower bound. So the loss we're going to optimize right here is during training, if you can see right here, what during training, we train the neural network to reconstruct one of these steps, right, each sample in training is going to be some image x t minus one, and some image x t, and we're going to reconstruct, we're going to train the neural network to predict x t minus one from x t or the variational sort of the distribution of that. So this is a training sample. Now, how do we get the training sample, what we can do is we can take x zero right here, and we could go through and add and add and add noise. But since we always add the Gaussian noise, we can simply do this in one step. There's nothing depending intermediately right here. So we do it in one step, right here, and then we add another bit of noise. That's how we get the two samples. And then rather than predicting the image itself, what these models do is they will predict the noise. So what we actually predict is going to be the noise, the noise epsilon here, which we can calculate by x t minus x t minus one. So this is our prediction target. This is our loss function, the network is supposed to output this right here. And of course, we know the true one. See the network will try to output this given x t and an index into which step it is. So we're going to tell the network, by the way, here's the noise. Here's the number of steps we're into this process. And we're going to train the network to read to say, what was the noise that was added, it's a bit easier, just, I think it's just like a scaling, scaling property, because this is going to have sort of zero mean and unit variance. So it's easier to predict for a neural network. So that is one of that is very standard in diffusion models. The next thing they introduce is guided diffusion. By the way, they also mentioned somewhere that they they learn the covariance matrix. Yes, there's another paper that also learns the covariance matrix. This first paper just fixed it at a diagonal. But then there is another paper that improved upon that, called improved denoising diffusion probabilistic model, interestingly, by the same authors here. And they, they show a method to learn this covariance matrix, which is mostly a scaling issue, because there is a narrow band that is a valid covariance matrix. And they show up with the correct parameterization, they can in fact, learn it and get better, better performance. But this just for reference, it's not super important right here. The second part is more important. So this is guided diffusion. So what we can do here is we can build a model, let's just assume we have images and we have class labels for the images, let's leave away the text right now. Okay, so we have a class label for for here. So this has a class label of cat, for example, there's also dog and so on. So what we can do is we can train the neural network here, you know, each step we train it to reconstruct one step. So that's going to predict the noise that was added, given the image xt, given the index t, what we can also do is we can say, by the way, it's also we give it the label y, so y, in this case is cat. So we can train a class conditional model. And that, you know, has some some advantages, we know class conditional GANs work quite well. So if you give it the class label as an input, you can often improve that. And you would do that by either embedding the class label as a one hot vector into the network or something like this. Now with the text model, it's a bit more tricky, right. But what you can do as you let's say this here, this here is some sort of a neural network, right. So xt goes in, this is xt goes into an encoder with a bunch of layers, maybe the t itself also goes in here as some sort of a float or an embedding a one hot vector or something like this. And the class label could also go in here, right. However, if you have text, what you can do is let's say you don't have this, but now you have a text description, they call this C. So you can first put the text description through an its own network, and then combine the embeddings. So either put the embeddings here as sort of a class embedding, or you can put the embeddings into each layer right here in this stack. And I think they do both. In any case, you can embed the text right here of the image, because their data set always has images and text together. So that's what I said at the beginning. So you can take this text, you can put it through an encoder itself, you can input it into this process right here. This is the network that is going to ultimately predict the added noise, given an image. And yeah, the network can take inspiration to take can learn from the text. So if it sees this picture right here, for example, that but in a very noisy way, and it has the text information, a couch in the corner of a room, it's obviously going to perform better than if it wouldn't have the text. And ultimately, that's going to unlock the capability that we can input a text at the very beginning, and then the model guided by this text will produce a living room, sorry, a couch in the corner of a room. So now, is this enough? And the answer is not yet. So class conditional models are working fine. However, it's better if you do what's called guided diffusion. So in guided diffusion, we not only want to make our models class conditional, but we want to, we want to guide them even more, we want to push them into a direction. And this is called guided diffusion. And one way to do it is to say, well, I have an additional classifier. I have a classifier, for example, an image net classifier, right. And if I want to push my diffusion process towards a particular label, I can take that image net classifier, and I can go along the gradient of that. This is very much like things like deep dream work, or this is essentially clip, clip guided diffusion is this but with clip. So I have the clip model. And if you don't know what the clip model is, this is a model where you input an image, and a piece of text, da da da da da, and it tells you how good, how good do the so let's put that as sigmoid, is do these two things fit together well or not. Now, if you think about the gradient of this, with respect to the image, then you can see that you can push the diffusion process into a direction where the image would fit together with the text more because you go along the gradient of that. It's kind of you construct an adversarial example towards this classifier. So this is one way of doing it, but it means that you have to have some sort of an external classifier to go by. There is also a method called classifier free guidance. And this was introduced by Hoenn Solomons. And this is where you sort of use the models own knowledge about its class conditioning in order to do this guidance. And this is a bit weird. And I feel like I feel like I feel like this shouldn't really work. And I feel the fact that this works appears to be a little bit of just a a little bit of just a hint that our current models aren't making use of the data fully, because we have to do these tricks at inference time. So it's more pointing towards us not really being the masters of these technologies yet, rather than this being some sort of an intrinsically good thing to do. But essentially, what we want to do is during training, we train these class conditional things, right, we train, let's produce the noise that was added to xt in the last step, conditioned on y, and y here could be a class label, y could be the input text, y could be, you know, pretty much any conditioning information. And then every we also alongside that, sometimes we don't provide that label at all. We don't just don't provide the label, which essentially means that we are training an unconditional generator. So we just simply forget the fact that we have labels, we simply train the image generation model unconditional. So we just give the model xt, we ask, here is just some image without description without nothing, what was the noise added to this image. And now at inference, so we just train the model in both ways. During training, we sometimes just leave away the label. This could be beneficial, as this part, in fact, would be the opportunity to bring more data into the picture, right? Let's say I have only part of my data is labeled and part of my data is on the label unlabeled, we could actually in here, bring in the unlabeled data, and therefore get more data into the system than we usually had. But given that they probably have enough data with their giant image caption data set here, by the way, it's the same data set they used for Dali. Given that it's probably they just leave away the text at during during training for some of the they say right here, for the label with a fixed probability during training. Now during inference, you can do something with that. What you can do during inference, you can say, well, if I am in the situation where I have an image and a label, and I asked my model to generate the noise, what I can do is I can do a little bit like the same thing I did with the clip guiding. So here I let my model predict the unnoised version. But I also push it into the direction that clip tells me would be a good image. So it's two things. This is given the image, what would be the unnoisy or the less noisy version. And this one would be, well, in general, which image would be sort of appropriate for this piece of text, and mix the two objectives. This is very much the same. So if you unpack this, you can see that this right here, unconditionally asks, given this image, which is the less noisy version of the image, or give me the noise that is was added to the image. And then you push it into this direction right here. And you can see this is the difference between the noise that the model predicts unconditionally, and the noise that the model predicts conditioned on the label. So this is a direction, this direction points very much into the direction of the noise that was specifically added to the label, right. So it's the difference between the conditional and unconditional prediction, we add that to the predicted noise right here. So the model predicts okay, this is the noise that was added. And the conditional model predicts this one, and this one, and then we simply push the prediction into this direction. You can see right here, there's a scalar s involved, s obviously must be larger than one. Because if s is smaller, like, this is what we would predict, usually the conditional one. So now, if s is larger than one, we're going to predict something more up here. And notice the difference if we didn't have this, if we didn't have this, we would simply predict this point right here, we wouldn't know which one which direction was a better direction. Because we also have the unconditional point right here, we can clearly say that this direction is probably the direction that goes into the direction of the conditioning information. So we can choose to sort of overdo it. Again, I think that is, that's kind of a trick around the fact that we don't know, we don't know how to handle the information very well quite yet. I'm not sure about it. It seems like you wouldn't even have to seems like you wouldn't even have to do this necessarily what you could also do if you want to go further, you could take sort of inspiration from the contrastive learning communities, and maybe do some hard some, you can also replace this part, and this part, by the way, so these parts, you could replace sort of by an expectation of these noises over some labels y hat or y prime. So and which means you could just sample some other text or some other conditioning information randomly, and get an expectation, you could also do hard negative sampling. So you could take labels that are fairly close, or you could take labels that are kind of confusing, and try to differentiate yourself. There's a lot of possibilities here. I can see that but still it feels like a bit of a trick. Yeah, so good. That's what they do. They do clip guidance. So they do this classifier free guidance, which turns out to be the better variant. And they also do the clip guidance, which is what we discussed before, except with clip, you can see they've just replaced the gradient of a classifier with the gradient of the clip model, the clip model is simply an inner product between an embedding of the image and embedding of the text. And they say the reason probably that the class for free guidance works better is because the clip, sort of the diffusion models, what they do is they find like adversarial examples to clip and not necessarily good, good pictures. Now I don't know if the classifier free guidance would also be something that could replace sort of the the current notebooks that are flying around where clip is used clip guided diffusion and VQV VQGAN plus clip. But I'm not sure because the VQGAN it seems already restricts the already restricts the space of images such that it's not that easy to find adversarial examples because it always has to go through the vector quantization. Okay, that's the model. Like the model is nothing else. It's a diffusion model. All right, this has existed before. It is conditioned on conditioning information, the diffusion model itself is conditioned, in this case on text that goes through a transformer encoder, which is the blue thing right here. This embeddings are then sort of concatenated into the process of this diffusion model. The diffusion model is a model that for one of these steps predicts sort of tries to predict the reverse. It's the same model for each step. It just gets as an additional conditioning information which step it's currently trying to reconstruct. It always reconstructs the noise that was added. Training data generation is pretty easy. You simply add noise to an image and then you add a bit more and then the difference between that is the target to predict. Then at inference time, at inference time, they also do this guided diffusion. That's either going to be achieved by clip and the disadvantage of that is that you have to have an additional classifier like clip. Not only that, but in fact the classifier has also had to be trained on noisy images because otherwise noisy images are going to be out of its distribution. So they do in fact train noised clip versions. The disadvantage as I said is you need this additional model that's trained on noisy data. The advantage is that you get to bring additional information here. You get to potentially even bring additional data sets that was used to train these other classifiers. You can use multiple classifiers, whatever. They also do classifier-free guidance. These two things, they don't use them together, clip guidance and classifier-free. They use them either or. The classifier-free guidance is more like a hack where you alongside the conditional denoising train an unconditional denoising. So you train the model also to sometimes not be conditioned and then you push it into the direction away from the unconditioned towards the conditioned and beyond to make it extra conditioned, I guess. The disadvantage here is that it seems like a hack. The advantage is that there's potential maybe to do some some hard negative sampling and also it doesn't require an extra model on the side. And also in the unconditional training, you might bring in additional data that has no label. So training happens. It's a 3.5 billion parameter, a text conditional diffusion model at 64 by 64 resolution. This is way smaller than Dali, by the way. And this is cool. And a 1.5 billion parameter text conditional upsampling diffusion model to increase the resolution. So it's a two-stage process. The diffusion model itself is at a 64 by 64 resolution and then they have an upsampling model. It's also text conditional, but it is an... So this is purely an diffusion upsampling model. It's very much the same principle, except that it now doesn't go... It doesn't go from noisy image or sorry, from pure noise to image. It goes from low resolution image to high resolution image. And alongside of that, they train a noised clip model, which is the classifier that they're going to need to do guidance. Well, they describe here a little bit of the architectures. We're not super interested, at least I'm not super interested in the architectures. They're way big models. As I said, they release the small models. They don't release the big models. They don't release the big models. And they explicitly train for inpainting, even though you could do it with diffusion models without training. But they say if you train it, it behaves a bit better. So during training, they would sort of mask out random parts of the images and then use diffusion to reconstruct those. And yeah, the results are the results that we've already seen. These are pretty interesting. They do studies with it. So they do studies on these datasets. So as they increase the guidance scales, the guidance scales are like the only handle they have at inference time to trade off diversity and sort of adherence to the dataset. And it turns out that the classifier free guidance, as you can see right here, is behaving better. This is the frontier right here. These always trade off two different metrics in the MSCoco dataset here. Precision recall, inception score, and FID. And you can see the only time the clip guidance is better than classifier free guidance is when you directly look at the clip score. That's why they say probably the clip guidance simply finds adversarial examples towards clip. They also let humans rate the pictures in terms of photorealism and caption similarity. And you can see that the classifier free guidance wins both times. And that's pretty much it. They show some failure cases, which I also find pretty interesting. So an illustration of a cat that has eight legs is not not a thing. A bicycle that has continuous tracks instead of wheels. It seemed a bit like Dali as a model was more sort of sensitive or was more respondent to text itself, so to the prompt. Whereas here it seems it's more like generating realistic images that has some sort of the words. So the words kind of match with the text. A mouse hunting a lion, not happening. Also a car with a car with triangular wheels. Also not happening as you can see. I myself have tried the small model a little bit and you can see you can you can try it yourself. I'll put a link a link up. There is a Gradio space by the user Valhalla. Thanks a lot for creating that. So here is balloon race. You can see that works pretty well. A drawing of a tiny house. That's also okay. A hidden treasure on a tropical island. I mean it's a tropical island right but yeah. All the elephants had left a long time ago. Now only a few vultures remain and it's just kind of a bunch of elephants. So well the elephants are kind of walking away a little bit right. Yeah. Attention is all you need obviously. Oddly Russian vibes from this picture. And this one is glory to the party. And I guess party is just sort of equated with birthday cake or so. So the sort of text sensitivity of this model might not be as good but there might be opportunity to fiddle here. The samples as such, they look they look pretty pretty cool. It's also not clear how much of a difference this is between the small model and the large model or how much effort into diffusion is put. They also say they release the model they release is sort of a model on a filtered version of a data set. And the filtered version removes for example, removes hate symbols and anything to do with people. So they say it's not as easy to generate deep fakes. Yeah. And where was yeah I think the the coolest one is where you can do this interactively. That is that is a pretty cool one. I want to look at lastly where we're sorry for the scrolling around safety consideration. So there's so like they say as a result releasing our model without safeguards would significantly reduce skills required to create convincing disinformation or deep fakes. And they say they only release the small model they say this somewhere. Where is it? Well in any case, they only release a small model, but I just want everyone to remember GPT two. And it was exactly the same. And to my knowledge, cheap it there is there is not the world is not in chaos right now because people have used GPT two, which is sort of public by now and can be easily used in the future. So I think that's a good point. And I think that's a good point, but if the world is not actively trained by anyone, the world is not in chaos because people have access to GPT two, it's, it's not the case. And I don't know why they do it because for PR reasons, or because they want to kind of sell it, sell the larger model, sell access to it, I mean that's all fine, but don't tell me this is safety considerations. And yeah, the fact is, deep fakes in the future, it's going to be easier. But it's kind of we have to the answer is not to not release the models and techniques. The answer is to educate people that hey, look not everything you see on a picture, especially if it looks like it's up sampled from 64 by 64. Not everything you see on there might be entirely real, right? Things can be altered, things can be photoshopped, things can be created like this. It's the same as people have learned that not everything that's written in an email is true, and people will simply have to adapt. That's going to be the only way. Not giving people access to these things seems to be kind of futile. But as I said, I don't believe for a second that actual safety considerations were the reason for this. In any case, let me know what you think. And that was it from me. Try the try out the model and maybe you'll find something cool. Bye bye.
 Hello there! Today we'll look at Glide towards photo-realistic image generation and editing
 with text-guided diffusion models by Alex Nicol, Prafula Darewal, Aditya Ramesh and others of OpenAI.
 This paper on a high level, well, I'll just show you what you can do. I'm sure you've all seen this
 paper in one way or another. It is another paper that generates images given a piece of text,
 but this time it's not a GAN or anything like this or a VQVAE. This time it is a diffusion model.
 This is a different class of models and we'll go into what they are and how they work. But essentially
 you can see right here that the model that turns out of this and of course this being OpenAI,
 they train this on a massive scale and this model is really big, but what comes out of it is very,
 very much better than for example Dali, which always had this kind of blurriness to it.
 You can see right here a crayon drawing of a space elevator, pixel art, corgi pizza. So this is
 trained on a big scrape of images from the internet and as you can see the outputs are pretty stunning.
 So it gets, for example, the shadows right here, it gets them correctly, even the red on blue
 blending, it gets different styles like the Salvador Dali style. It combines different concepts,
 although maybe you know this has been seen on the internet somewhere, but it is able to combine
 different concepts. And given that these are diffusion models, you can actually do a bunch
 of more stuff with them. For example, inpainting is immediately accessible to this model. Now
 usually inpainting is accessible to diffusion models, however, they actually train an inpainting
 model on top of this. But in essence, a lot of stuff would be accessible. So this is now possible
 where you say, okay, I only want to change a part of the image like this part right here,
 you give a text saying a man wearing a white hat and the model generates the man wearing a white hat.
 This is very cool. You can do things like this where you first, so the pictures here are a bit
 confusing, but you first generate an image from a text prompt, like a cozy living room, then you get
 this living room and then here the user would annotate this window sort of would draw over it
 and will give the next text prompt. The next text prompt will be a painting of a corgi on the wall
 above the couch. And the model it's an inpainting, so this is the inpainting mode, the model would
 only be able to paint the green area. So it would sort of try to conform to the text using only the
 green area. And therefore, it would make this corgi picture on the wall right here, then the user goes
 further and says, well, now I'm going to paint this area right here. And I'm going to issue the
 prompt around coffee table in front of a couch, and the model will generate it and so on. You can
 see that this enables sort of an interactive creation of these scenery at the end, the couch,
 the couch in the corner of the room, so changing the entire wall right here, you can see the back
 of the room has some space. And now it's being changed to a wall. So this is the kind of stuff
 that's possible. Editing right here. Even what's this this sort of sketch editing where you don't
 only mask, but along with the mask, you provide sort of like a sketch as you can see right here.
 So this part here is blue, and then the part here is white. And that's also the mask that the
 the picture receives. And you can see that only one cloud in the sky today, it's sort of, you can
 guide even more so you can guide with text and you can guide with sketch color, and so on. So this is
 a very, very, very cool model, you can see the quality is very, very good. Here is for example,
 a comparison. These are real images from the MS, MS Marco data set, MS Coco, sorry. This is a data
 set of pictures with associated labels, so text descriptions of the picture. So you have some
 ground truth. So the ground truth here will be this one. And the label is a green train coming
 down the tracks. You can see Dali generates something neat, but it's sort of blurry. It's
 kind of cartoonish, as all the Dali pictures are if you look in this row. The last one's pretty
 good, but all the other ones are sort of elephants are more like blobs. And we've seen this in the
 Dali paper. It was impressive at the time, but this is way more impressive. And then their best
 model, this clip, sorry, this glide model with classifier free guidance, you can see right here,
 it generates like a high quality train that fits the image description. And you can see in the
 entire row right here, it's pretty good at doing that. So there are a lot of components to this
 model. And we're going to explore them a little bit. OpenAI has released in classic OpenAI fashion,
 they've released like a small, very filtered version of that model because they're worried
 about safety. Like anyone's going to believe them after GPT-2. They've just been doing this every
 single model, right? They're just like, oh, no safety, people can make deep fakes. Oh, no,
 like, no one's made a deep fake. Like GPT-2, all the worries, they were just not true. No one has
 used GPT-2 to spread around fake news. And no one like no one's going to use this model substantially
 to make very misleading pictures. But we'll get to that as well. All right, so what is a diffusion
 model? And that's sort of at the core of this thing right here. A diffusion model is a different type
 of generative model than maybe you're used to from like a GAN or a VQVAE. So in a GAN, a GAN is
 probably the closest right here. So again, it's sort of like a neural network with a bunch of layers.
 And what you do is you sample from some sort of a distribution, you sample some noise, right,
 you sample some noise, you get some noise vector. So here's a vector with just complete noise,
 every entry is noise. You put it through the network, the network generates pretty picture,
 and you train the model using a discriminator. In this case, you train the model to produce
 pretty pictures, given the noise and the noise acts sort of as a source of randomness. So the
 mapping is clear, you train to map from noise to picture. Now, a diffusion model goes in almost like
 a different direction. So what you do is during training, you have a data set, and you take an
 image. So from from a data set, you have a data set, you take an image out of it. Let's say this
 is your trusty, trusty cat, ta-da. And you're going to, you're going to put noise onto this image. So
 you're going to add noise and noise, let's represent that with sigma. No, I think they do,
 they do epsilon or eta in this in this paper right here. So you add that, and then you get a slightly
 noisy version of this. Let's just, let's just wiggle a bit, wiggle, wiggle, wiggle. And you do
 it again. So through adding noise, and you add lots and lots and lots of noise, okay, so every
 time you add a tiny, tiny bit of noise. And that means that more and more your picture is just
 going to be blurry and blurry and blurry. Now, if you do this for long enough, in the limit,
 you can prove that obviously, if you do this infinitely many times, what comes out at the end
 is going to be just nor normally distributed, if your noise is normally distributed, and you scale
 every time correctly, then whatever turns out is going to be normally distributed with some
 parameters here. So this right here is going to be a known distribution, if you, if you
 add noise for long enough, if you destroy all of the information that the picture has, then
 you'll end up with sort of an entry in a known distribution. However, every step that you do
 right here is very small, every step, you just add a little bit of noise. So technically,
 it's possible for a model to look at this picture right here, which is kind of a bit of a blurry
 version of the cat and predict and learn to predict the more sharp version of the cat. Okay,
 this is a foundation of many, many sort of denoising models, many up sampling models,
 super resolution models, what have you, okay, they do this in one step. But essentially here,
 we say the individual step is small enough such that the model can technically predict the
 can technically learn to reconstruct it. However, if we do it for long enough in, you know, going
 to infinity, the we are at a known distribution, namely the standard normal distribution.
 And these two things together mean that, well, if we have trained the model to reconstruct the
 individual steps, what we can technically do is we can now go ahead sample from this known
 distribution, right, because ultimately, we want to sample from the data distribution. But that's
 hard because we don't know it. But here we can just sample some noise from a known distribution,
 then put it through this process of reconstruction, all the way, all the steps that we did up here
 during training. During training, we just noise the noise and noise the images again and again.
 And again, we trained the neural network to for every step to reconstruct the previous step. So
 we can now just put it through this series of trained neural networks. In fact, it's just going
 to be one neural network that gets the index of the step as a parameter and outcomes an image,
 right outcomes a true data image. If these two things up here hold, then this should be possible.
 This is the basis for these diffusion models. So specifically, given a sample, that's what they say
 here, given a sample from the data distribution, this is x zero. So this is the data distribution,
 we produce a Markov chain of latent variables x one to xt, with everyone being a more noisy
 version, and xt finally being of a like a known distribution, because we do it infinitely, or a
 large number of times by progressively adding Gaussian noise to the sample. So you can see right
 here, we take xt minus one, we scale it down a bit, because if you wouldn't do that, the sort of the
 image would just increase in scale over because we just keep adding stuff. But this it's just a
 rescaling that there's nothing more happening here. So we, we, we add noise, this here is the mean
 of a distribution, the covariance matrix here is a diagonal, which essentially means we just add
 a bit of noise of the scale of alpha t. No, sorry, we just add a bit of noise, we rescale by alpha t,
 which is a scaling factor. And that's how we obtain the next step, the xt. So get we do this enough.
 So we take xt for the next step, we plug it in here, and then we obtain xt plus one, and so on.
 So if the magnitude of the noise added at each step is small enough, the posterior is well,
 well approximated by a diagonal Gaussian. That's what they say right here. So what does this mean,
 the posterior, it means that this is the reverse step, right, I have xt, and I'm looking to recreate
 xt minus one. So if the noise is small enough, then the posterior is well approximated by a
 diagonal Gaussian, and we have a hope to learn it with a neural network, right.
 Furthermore, if the magnitude of the total noise added throughout the chain is large enough,
 then the last step is well approximated by a known by a standard normal distribution.
 These properties suggest learning a model for this posterior, right, we have xt, we want to
 reconstruct xt minus one to approximate the true posterior. Okay, so we are going to learn a neural
 network that it doesn't exactly reconstruct the image, but this is a variational model. So what
 we're going to do is we're going to plug in xt into a neural network, the neural network is going to
 predict the mean and the covariance matrix of the next step. So we're going to do this,
 of the next step up the chain of the next step of the denoising chain. And then we can use this to
 produce samples, we simply sorry, we start we start with Gaussian noise, which is the end,
 and we gradually reduce the noise in a sequence of steps until we are at the data distribution,
 or at least the predicted data distribution. So this is not a new idea. This has been and
 I think I have the references open. This has been explored previously. For example, this is just an
 example right here. Denoising diffusion probabilistic models is one of the papers that introduced
 lots of these things you can see right here. These have still been trained on like just images as
 such. So this is the left is trained on a face data set, the right is trained on CIFAR 10. This
 is unconditional generation without a text prompt or anything like this. But you can see the same
 principle applies, we simply add noise during training and we learn a neural network to remove
 the noise to predict what the image would look like one noise step less. Here already, there was
 an invention that the paper here would make use of namely the loss function right here, we're going
 to look at that in just a second. So that's the second. So they say, while there exists a tractable
 variational lower bound, better results arise from optimizing a surrogate objective, which reways the
 term in the variational lower bound. So the loss we're going to optimize right here is during
 training, if you can see right here, what during training, we train the neural network to reconstruct
 one of these steps, right, each sample in training is going to be some image x t minus one,
 and some image x t, and we're going to reconstruct, we're going to train the neural network to predict
 x t minus one from x t or the variational sort of the distribution of that. So this is a training
 sample. Now, how do we get the training sample, what we can do is we can take x zero right here,
 and we could go through and add and add and add noise. But since we always add the Gaussian noise,
 we can simply do this in one step. There's nothing depending intermediately right here.
 So we do it in one step, right here, and then we add another bit of noise. That's how we get the
 two samples. And then rather than predicting the image itself, what these models do is they will
 predict the noise. So what we actually predict is going to be the noise, the noise epsilon here,
 which we can calculate by x t minus x t minus one. So this is our prediction target. This is our
 loss function, the network is supposed to output this right here. And of course, we know the true
 one. See the network will try to output this given x t and an index into which step it is. So we're
 going to tell the network, by the way, here's the noise. Here's the number of steps we're into this
 process. And we're going to train the network to read to say, what was the noise that was added,
 it's a bit easier, just, I think it's just like a scaling, scaling property, because this is going
 to have sort of zero mean and unit variance. So it's easier to predict for a neural network.
 So that is one of that is very standard in diffusion models. The next thing
 they introduce is guided diffusion. By the way, they also mentioned somewhere that they
 they learn the covariance matrix. Yes, there's another paper that also learns the covariance
 matrix. This first paper just fixed it at a diagonal. But then there is another paper that
 improved upon that, called improved denoising diffusion probabilistic model, interestingly,
 by the same authors here. And they, they show a method to learn this covariance matrix, which is
 mostly a scaling issue, because there is a narrow band that is a valid covariance matrix. And they
 show up with the correct parameterization, they can in fact, learn it and get better,
 better performance. But this just for reference, it's not super important right here.
 The second part is more important. So this is guided diffusion. So what we can do here is we can
 build a model, let's just assume we have images and we have class labels for the images, let's
 leave away the text right now. Okay, so we have a class label for for here. So this has a class
 label of cat, for example, there's also dog and so on. So what we can do is we can train the neural
 network here, you know, each step we train it to reconstruct one step. So that's going to predict
 the noise that was added, given the image xt, given the index t, what we can also do is we can
 say, by the way, it's also we give it the label y, so y, in this case is cat. So we can train a
 class conditional model. And that, you know, has some some advantages, we know class conditional
 GANs work quite well. So if you give it the class label as an input, you can often improve that.
 And you would do that by either embedding the class label as a one hot vector into the network
 or something like this. Now with the text model, it's a bit more tricky, right. But what you can do
 as you let's say this here, this here is some sort of a neural network, right. So xt goes in, this is
 xt goes into an encoder with a bunch of layers, maybe the t itself also goes in here as some sort
 of a float or an embedding a one hot vector or something like this. And the class label could
 also go in here, right. However, if you have text, what you can do is let's say you don't have this,
 but now you have a text description, they call this C. So you can first put the text description
 through an its own network, and then combine the embeddings. So either put the embeddings here
 as sort of a class embedding, or you can put the embeddings into each layer right here in this
 stack. And I think they do both. In any case, you can embed the text right here of the image,
 because their data set always has images and text together. So that's what I said at the beginning.
 So you can take this text, you can put it through an encoder itself, you can input it into this
 process right here. This is the network that is going to ultimately predict the added noise,
 given an image. And yeah, the network can take inspiration to take can learn from the text. So
 if it sees this picture right here, for example, that but in a very noisy way, and it has the text
 information, a couch in the corner of a room, it's obviously going to perform better than if it
 wouldn't have the text. And ultimately, that's going to unlock the capability that we can input
 a text at the very beginning, and then the model guided by this text will produce a living room,
 sorry, a couch in the corner of a room. So now, is this enough? And the answer is not yet. So
 class conditional models are working fine. However, it's better if you do what's called
 guided diffusion. So in guided diffusion, we not only want to make our models class conditional,
 but we want to, we want to guide them even more, we want to push them into a direction.
 And this is called guided diffusion. And one way to do it is to say, well, I have an additional
 classifier. I have a classifier, for example, an image net classifier, right. And if I want to push
 my diffusion process towards a particular label, I can take that image net classifier, and I can
 go along the gradient of that. This is very much like things like deep dream work, or this is
 essentially clip, clip guided diffusion is this but with clip. So I have the clip model. And if
 you don't know what the clip model is, this is a model where you input an image, and a piece of
 text, da da da da da, and it tells you how good, how good do the so let's put that as sigmoid,
 is do these two things fit together well or not. Now, if you think about the gradient of this,
 with respect to the image, then you can see that you can push the diffusion process into a direction
 where the image would fit together with the text more because you go along the gradient of that.
 It's kind of you construct an adversarial example towards this classifier. So this is one way of
 doing it, but it means that you have to have some sort of an external classifier to go by.
 There is also a method called classifier free guidance. And this was introduced by Hoenn
 Solomons. And this is where you sort of use the models own knowledge about its class conditioning
 in order to do this guidance. And this is a bit weird. And I feel like I feel like I feel like this
 shouldn't really work. And I feel the fact that this works appears to be a little bit of just a
 a little bit of just a hint that our current models aren't making use of the data fully,
 because we have to do these tricks at inference time. So it's more pointing towards us not really
 being the masters of these technologies yet, rather than this being some sort of an intrinsically
 good thing to do. But essentially, what we want to do is during training, we train these class
 conditional things, right, we train, let's produce the noise that was added to xt in the last step,
 conditioned on y, and y here could be a class label, y could be the input text, y could be,
 you know, pretty much any conditioning information. And then every we also alongside that,
 sometimes we don't provide that label at all. We don't just don't provide the label, which
 essentially means that we are training an unconditional generator. So we just simply
 forget the fact that we have labels, we simply train the image generation model unconditional.
 So we just give the model xt, we ask, here is just some image without description without nothing,
 what was the noise added to this image. And now at inference, so we just train the model in both
 ways. During training, we sometimes just leave away the label. This could be beneficial, as this part,
 in fact, would be the opportunity to bring more data into the picture, right? Let's say I have only
 part of my data is labeled and part of my data is on the label unlabeled, we could actually in here,
 bring in the unlabeled data, and therefore get more data into the system than we usually had. But
 given that they probably have enough data with their giant image caption data set here,
 by the way, it's the same data set they used for Dali.
 Given that it's probably they just leave away the text at during during training for some of the
 they say right here, for the label with a fixed probability during training.
 Now during inference, you can do something with that. What you can do during inference,
 you can say, well, if I am in the situation where I have an image and a label, and I asked my model to
 generate the noise, what I can do is I can do a little bit like the same thing I did with the
 clip guiding. So here I let my model predict the unnoised version. But I also push it into
 the direction that clip tells me would be a good image. So it's two things. This is given the image,
 what would be the unnoisy or the less noisy version. And this one would be, well, in general,
 which image would be sort of appropriate for this piece of text, and mix the two objectives.
 This is very much the same. So if you unpack this, you can see that this right here,
 unconditionally asks, given this image, which is the less noisy version of the image,
 or give me the noise that is was added to the image. And then you push it into this direction
 right here. And you can see this is the difference between the noise that the model predicts
 unconditionally, and the noise that the model predicts conditioned on the label. So this is a
 direction, this direction points very much into the direction of the noise that was specifically
 added to the label, right. So it's the difference between the conditional and
 unconditional prediction, we add that to the predicted noise right here. So the model predicts
 okay, this is the noise that was added. And the conditional model predicts this one, and this
 one, and then we simply push the prediction into this direction. You can see right here, there's a
 scalar s involved, s obviously must be larger than one. Because if s is smaller, like, this is what
 we would predict, usually the conditional one. So now, if s is larger than one, we're going to
 predict something more up here. And notice the difference if we didn't have this, if we didn't
 have this, we would simply predict this point right here, we wouldn't know which one which
 direction was a better direction. Because we also have the unconditional point right here,
 we can clearly say that this direction is probably the direction that goes into the direction of the
 conditioning information. So we can choose to sort of overdo it. Again, I think that is, that's kind
 of a trick around the fact that we don't know, we don't know how to handle the information very well
 quite yet. I'm not sure about it. It seems like you wouldn't even have to seems like you wouldn't
 even have to do this necessarily what you could also do if you want to go further, you could take
 sort of inspiration from the contrastive learning communities, and maybe do some hard some, you can
 also replace this part, and this part, by the way, so these parts, you could replace sort of by an
 expectation of these noises over some labels y hat or y prime. So and which means you could just
 sample some other text or some other conditioning information randomly, and get an expectation,
 you could also do hard negative sampling. So you could take labels that are fairly close,
 or you could take labels that are kind of confusing, and try to differentiate yourself.
 There's a lot of possibilities here. I can see that but still it feels like a bit of a trick.
 Yeah, so good. That's what they do. They do clip guidance. So they do this classifier free guidance,
 which turns out to be the better variant. And they also do the clip guidance, which is what we
 discussed before, except with clip, you can see they've just replaced the gradient of a classifier
 with the gradient of the clip model, the clip model is simply an inner product between an
 embedding of the image and embedding of the text. And they say the reason probably that the class
 for free guidance works better is because the clip, sort of the diffusion models, what they do is
 they find like adversarial examples to clip and not necessarily good, good pictures.
 Now I don't know if the classifier free guidance would also be something that could replace sort
 of the the current notebooks that are flying around where clip is used clip guided diffusion
 and VQV VQGAN plus clip. But I'm not sure because the VQGAN it seems already restricts the
 already restricts the space of images such that it's not that easy to find
 adversarial examples because it always has to go through the vector quantization.
 Okay, that's the model. Like the model is nothing else. It's a diffusion model. All right,
 this has existed before. It is conditioned on conditioning information, the diffusion model
 itself is conditioned, in this case on text that goes through a transformer encoder, which is the
 blue thing right here. This embeddings are then sort of concatenated into the process of this
 diffusion model. The diffusion model is a model that for one of these steps predicts sort of tries
 to predict the reverse. It's the same model for each step. It just gets as an additional
 conditioning information which step it's currently trying to reconstruct. It always reconstructs the
 noise that was added. Training data generation is pretty easy. You simply add noise to an image and
 then you add a bit more and then the difference between that is the target to predict. Then at
 inference time, at inference time, they also do this guided diffusion. That's either going to be
 achieved by clip and the disadvantage of that is that you have to have an additional classifier
 like clip. Not only that, but in fact the classifier has also had to be trained on noisy images
 because otherwise noisy images are going to be out of its distribution. So they do in fact train
 noised clip versions. The disadvantage as I said is you need this additional model that's trained
 on noisy data. The advantage is that you get to bring additional information here. You get to
 potentially even bring additional data sets that was used to train these other classifiers. You
 can use multiple classifiers, whatever. They also do classifier-free guidance. These two things,
 they don't use them together, clip guidance and classifier-free. They use them either or.
 The classifier-free guidance is more like a hack where you alongside the conditional denoising train
 an unconditional denoising. So you train the model also to sometimes not be conditioned and then you
 push it into the direction away from the unconditioned towards the conditioned and beyond
 to make it extra conditioned, I guess. The disadvantage here is that it seems like a hack.
 The advantage is that there's potential maybe to do some some hard negative sampling and also it
 doesn't require an extra model on the side. And also in the unconditional training, you might
 bring in additional data that has no label. So training happens. It's a 3.5 billion parameter,
 a text conditional diffusion model at 64 by 64 resolution. This is way smaller than Dali, by the way.
 And this is cool. And a 1.5 billion parameter text conditional upsampling diffusion model to increase
 the resolution. So it's a two-stage process. The diffusion model itself is at a 64 by 64 resolution
 and then they have an upsampling model. It's also text conditional, but it is an... So this is purely
 an diffusion upsampling model. It's very much the same principle, except that it now doesn't go...
 It doesn't go from noisy image or sorry, from pure noise to image. It goes from low resolution image
 to high resolution image. And alongside of that, they train a noised clip model, which is the
 classifier that they're going to need to do guidance. Well, they describe here a little bit of
 the architectures. We're not super interested, at least I'm not super interested in the architectures.
 They're way big models. As I said, they release the small models. They don't release the big models.
 They don't release the big models. And they explicitly train for inpainting, even though you could do it
 with diffusion models without training. But they say if you train it, it behaves a bit better.
 So during training, they would sort of mask out random parts of the images and then use diffusion
 to reconstruct those. And yeah, the results are the results that we've already seen. These are
 pretty interesting. They do studies with it. So they do studies on these datasets. So as they increase
 the guidance scales, the guidance scales are like the only handle they have at inference time
 to trade off diversity and sort of adherence to the dataset. And it turns out that the classifier
 free guidance, as you can see right here, is behaving better. This is the frontier right here.
 These always trade off two different metrics in the MSCoco dataset here. Precision recall,
 inception score, and FID. And you can see the only time the clip guidance is better than classifier
 free guidance is when you directly look at the clip score. That's why they say probably the clip
 guidance simply finds adversarial examples towards clip. They also let humans rate the pictures in
 terms of photorealism and caption similarity. And you can see that the classifier free guidance
 wins both times. And that's pretty much it. They show some failure cases, which I also find
 pretty interesting. So an illustration of a cat that has eight legs is not not a thing.
 A bicycle that has continuous tracks instead of wheels. It seemed a bit like Dali as a model
 was more sort of sensitive or was more respondent to text itself, so to the prompt. Whereas here
 it seems it's more like generating realistic images that has some sort of the words. So the
 words kind of match with the text. A mouse hunting a lion, not happening. Also a car with
 a car with triangular wheels. Also not happening as you can see. I myself have tried the small
 model a little bit and you can see you can you can try it yourself. I'll put a link a link up.
 There is a Gradio space by the user Valhalla. Thanks a lot for creating that. So here is balloon
 race. You can see that works pretty well. A drawing of a tiny house. That's also okay. A hidden treasure
 on a tropical island. I mean it's a tropical island right but yeah. All the elephants had left a long
 time ago. Now only a few vultures remain and it's just kind of a bunch of elephants. So well the
 elephants are kind of walking away a little bit right. Yeah. Attention is all you need obviously.
 Oddly Russian vibes from this picture. And this one is glory to the party. And I guess party
 is just sort of equated with birthday cake or so. So the sort of text sensitivity of this model
 might not be as good but there might be opportunity to fiddle here. The samples as such,
 they look they look pretty pretty cool. It's also not clear how much of a difference this is between
 the small model and the large model or how much effort into diffusion is put. They also say they
 release the model they release is sort of a model on a filtered version of a data set.
 And the filtered version removes for example, removes hate symbols and anything to do with people.
 So they say it's not as easy to generate deep fakes. Yeah. And where was yeah I think the the
 coolest one is where you can do this interactively. That is that is a pretty cool one. I want to look
 at lastly where we're sorry for the scrolling around safety consideration. So there's so like
 they say as a result releasing our model without safeguards
 would significantly reduce skills required to create convincing disinformation or deep fakes.
 And they say they only release the small model they say this somewhere.
 Where is it? Well in any case, they only release a small model, but I just want everyone to remember
 GPT two. And it was exactly the same. And to my knowledge, cheap it there is there is not the
 world is not in chaos right now because people have used GPT two, which is sort of public by now and
 can be easily used in the future. So I think that's a good point. And I think that's a good
 point, but if the world is not actively trained by anyone, the world is not in chaos because
 people have access to GPT two, it's, it's not the case. And I don't know why they do it because
 for PR reasons, or because they want to kind of sell it, sell the larger model, sell access to it,
 I mean that's all fine, but don't tell me this is safety considerations. And yeah, the fact is,
 deep fakes in the future, it's going to be easier.
 But it's kind of we have to the answer is not to not release the models and techniques.
 The answer is to educate people that hey, look not everything you see on a picture,
 especially if it looks like it's up sampled from 64 by 64.
 Not everything you see on there might be entirely real, right?
 Things can be altered, things can be photoshopped, things can be created like this.
 It's the same as people have learned that not everything that's written in an email is true,
 and people will simply have to adapt.
 That's going to be the only way.
 Not giving people access to these things seems to be kind of futile.
 But as I said, I don't believe for a second that actual safety considerations were the reason
 for this. In any case, let me know what you think.
 And that was it from me.
 Try the try out the model and maybe you'll find something cool.
 Bye bye.