OpenAI DALL·E: Creating Images from Text (Blog Post Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gpt gpt-3 visual transformer transformer transformers attention mechanism vqvae vq vae vq-vae codebook relaxation gumbel text images nlp natural language processing autoregressive grid encoder decoder gpt3 avocado chair porcupine sphere animations fisheye text to image image captioning openai sutskever dali dalle walle vector quantized hierarchical gan generative likelihood
#openai #science #gpt3

OpenAI's newest model, DALL·E, shows absolutely amazing abilities in generating high-quality images from arbitrary text descriptions. Like GPT-3, the range of applications and the diversity of outputs is astonishing, given that this is a single model, trained on a purely autoregressive task. This model is a significant step towards the combination of text and images in future AI applications.

OUTLINE:
0:00 - Introduction
2:45 - Overview
4:20 - Dataset
5:35 - Comparison to GPT-3
7:00 - Model Architecture
13:20 - VQ-VAE
21:00 - Combining VQ-VAE with GPT-3
27:30 - Pre-Training with Relaxation
32:15 - Experimental Results
33:00 - My Hypothesis about DALL·E's inner workings
36:15 - Sparse Attention Patterns
38:00 - DALL·E can't count
39:35 - DALL·E can't global order
40:10 - DALL·E renders different views
41:10 - DALL·E is very good at texture
41:40 - DALL·E can complete a bust
43:30 - DALL·E can do some reflections, but not others
44:15 - DALL·E can do cross-sections of some objects
45:50 - DALL·E is amazing at style
46:30 - DALL·E can generate logos
47:40 - DALL·E can generate bedrooms
48:35 - DALL·E can combine unusual concepts
49:25 - DALL·E can generate illustrations
50:15 - DALL·E sometimes understands complicated prompts
50:55 - DALL·E can pass part of an IQ test
51:40 - DALL·E probably does not have geographical / temporal knowledge
53:10 - Reranking dramatically improves quality
53:50 - Conclusions & Comments

Blog: https://openai.com/blog/dall-e/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 A sphere made of Swiss cheese. A sphere with a texture of Swiss cheese. And there you have it. Beautiful, very appetizing Swiss cheese balls. My Swiss heart had just skipped a beat out of this monstrosity. What's even cooler than a sphere made of Swiss cheese is a torus made of denim. These images are so cool. A torus made of denim. And the point here is that these images aren't photoshopped or sort of human created. They are AI generated. And they are generated by this new model that OpenAI released a blog post about. It's called Dali. And it can, what it can do is it can take a piece of text such as the one on top here. The fact that I can select is simply the fact that they don't give you access to the model. They just give you access of a bunch of things that they've tried. But the model can take any piece of text and it can output a picture that matches that text. So here you got a torus made of toothpaste. And the quality of these images is super astounding. And what's even more astounding is sort of the range of capabilities that this model has. So the model can do various things such as so in here the input is an illustration of a baby daikon radish in a tutu walking a dog. And you see an illustration of a baby daikon radish in a tutu walking a dog. The outputs are just adorable. These are generated by the AI. The same for an armchair in the shape of an avocado, a storefront that has the word OpenAI written on it. I've tried reverse image searching some of these images and I could not find them on the internet. So it's definitely not just a model sort of outputting an image it found somewhere. These are actually generated images. And the astounding thing is that it's the same model that outputs all of these different images. It's not one model here trained on illustrations and one model trained on chairs. It's a single model that can take in a piece of text and optionally part of an image or none of an image and it will output the image either it continues the image you already give part of or it just generates the image by itself. So the model is called Dali. And this is just a blog post for now by OpenAI. They say they'll follow this up with a paper. And if the paper brings substantially new things, I think I'll make a video on it. But today we're just going to look at what this model can do, how it works, how it probably works. And we can take some guesses of what we can read in the paper once it's out. In fact, OpenAI has brought out two new models along with this Dali model. They've also released a blog post and a paper about a model called Clip, which is more of a sort of a classifier, not exactly a classifier. It's sort of a it connects text and images in a different way. It's not a generative model. And we're going to look at that in a different video. But you can see the clear trend right here is that OpenAI is looking into connecting text and images. So they say Dali, which is an, this is a, and I think an homage to Salvador Dali and mixed with the character Wally. So they say it's a 12 billion parameter version of GPT-3. So you know, it's more like, it's more like not GPT-3. That was more than 10 times larger, but it's a 12 billion parameter version of GPT-3 trained to generate images from text descriptions using a data set of text image pairs. We found that it has diverse set of capabilities, including creating anthropomorphized versions of animals and objects, combining unrelated concepts in plausible ways, rendering text and applying transformations to existing images. So a lot of the things they don't tell us here, especially the data set, like how did they get the data set? Nobody knows. They don't say this. They simply say it's a data set of text image pairs. And they sort of allude to the fact that they have large pieces of data, especially in the clip. Then they allude to the fact that you can just find data that connects text and images on the internet. And it's true if you if you search, if you scrape the correct websites, and do it in sort of a smart fashion, you can find a lot of data where there's an image and there's a piece of text describing that image. And we have to assume that they sort of scrape the internet for something like this. I don't think they have a lot of human explicitly human labeled data for this type of thing. So we'll just assume that they have like a huge data set. And of course, they train a huge model on it, a 12 billion parameter version of GPT three GPT three is the famous model, the famous text generation model by open AI. And you can sort of see the same things right here. So GPT three, my hypothesis was that it sort of smartly mixes the training data rather than remember the training data, it sort of remembers it and then smartly interpolates between it. And I think you can sort of see the same kind of things right here in that these are all definitely pictures that you could imagine in the real world. But they have, you know, they have, for example, they're changed to open AI in here, there are surely chairs that sort of look like this. So it just kind of mixes a chair with an avocado in a plausible way. I'm not saying this to denigrate the model, I'm saying that, I mean, this is seriously cool, the fact that it can do that. So they say like GPT three, Dulli is a transformer language model. Now, this is very, very interesting, the fact that it's a transformer language model, it receives both the text and the image as a single stream of data containing up to 1000 and 1280 tokens, and it's trained using maximum likelihood to generate all of the tokens one after another. Okay, this training procedure allows Dulli not only to generate images from scratch, but also to regenerate any rectangular region of an existing image that extends to the bottom right corner in a way that is consistent with the text prompt. And they say a little bit more here on the right. And they also say a little bit more down on the bottom. So I'm going to try to take a stab of explaining how this model works with the full knowledge that I might be wrong once the paper comes out. And for that, we have to go back a little bit and look at the models it draws from, namely the VQ VAE. So the vector quantized VAE literature, so VQ VAE will consider this to be sort of the inspiration of or one of the necessary ingredients of this model. So if we combine VQ VAE with something like GPT three, we get Dulli. That's my that's my hypothesis for today. Why combining these two models? So GPT three is extremely good at modeling language, right? So if I have a piece of text, let's go down here for a minute. And let's say I have a cat set on the mat. A transformer will be very good at understanding this sentence and being able to complete it. So if I cross out this and ask a transformer to continue the sentence, it will be able to continue the sentence just fine if it is if it is trained well. And that's exactly how GPT three works. Now imagine that I don't have a piece of text, but I have some sort of a description of an image, right? And let's say I have, I have a box. Here is a box. And the box which is going to be a VQ VAE can take in a description of an image in words, but not exactly words that humans understand. But let's say there is an image language, sort of like a programming language, okay. And you input symbols into the image, let's say, it's a bit like Egyptian hieroglyphs, maybe. So here is the here is the this, this hieroglyph thing, and then there is the sun, the sun thing. And then there is the tree, the word for tree, like the hieroglyph for tree. And I input that here. And the output will be an image where I don't know, there the sun is shining. Yes, I draw some like a child, it has a little smile, okay, deal with it. And there is a tree, maybe not exactly the tree from the hieroglyphs, but like some sort of some sort of tree that fits. And then there is some human in the scene, maybe the human sits here, the human sits at the tree, you know, relaxing, chilling. Okay, so this, now the image on the right is consistent of pixels, right. And modeling pixels with a transformer is very, very hard, because in the case of our model right here, it's something like 256 by 256 pixels. That would mean the transformer would have to generate 256 times 256, which is like two to the two to the 16. This is just too much for a transformer to model the pixels individually. So there are multiple ways around this, for example, modeling little regions right here, which are not really satisfactory. So what this model does is it sort of it doesn't try to model the picture as such, it tries to predict to predict these hieroglyphs right here, it tries to predict sort of a language that this box can understand and produce a picture from, okay, so its task is going to be given some sort of a given some sort of a text prefix. So a human in a sunny field, sunny day or on a sunny day, chilling under a tree. So this piece of text followed. So the model is trained to take this piece of text and output this sequence of hieroglyphs. Okay, so this sequence of hieroglyphs outputting from this piece of text. And that's something a transformer can do if you have a vocabulary right here. So if you have a fixed list of hieroglyphs that you could use, right, so in there there is the human is in there. That's a worse Egyptian. And then the pyramid is in here as well, some that you need, some that you don't need. So if there is a vocabulary, the transformer is going to be pretty, pretty good at generating this thing. So you need two parts. The first part right here is a transformer language model, a GPT-3 thing that can input a sequence of text, and it can output a sequence of text, which is just in a different vocabulary, namely this picture vocabulary. And then in the step two, you need a box that takes in this picture vocabulary and actually produces an images and image right here. So as I already said, this part is taken over by GPT, GPT-3, like the custom GPT model they built for this. And this part is taken over by something like a VQVAE, the generator part of it. So what is a VQVAE? A VQVAE is, and you will be able to see that. So the box that we're going to need is this box right here, from here up to where the image is. And this thing right here is going to be that vocabulary. So what does a VQVAE do? It takes the image here on the left, you can see that here's the encoder, it takes the image, it encodes it into a latent space. Now what a VAE would do, or what an autoencoder would do, is it would encode the image into a latent space, and then it would decode it again into and try to reproduce the same image. And then you assume that whatever is in the middle right here is a sensible representation, a latent representation of that image, right? If you can train this model, you're going to get some sort of a representation in the middle that describes the image, otherwise you couldn't reproduce the image. And there have been many models built on this concept. Now this model right here, it turns out that the classic autoencoder doesn't work too well. But this model works quite formidably. So what you're going to have is you're going to have this vocabulary right here. It's also called a codebook. Let's call it a codebook. So the codebook is also the vocabulary. So what you're saying is that you can't just output any latent encoding. So the encoder outputs a continuous vector. But what you're saying is it has to be one of those. Like there are a number of vectors that you have at your disposal, Mr. or Miss Encoder or Mrs. Encoder. There is a number of vectors that you have at your disposal. You can only choose those. You can't choose any vector that you want, right? So in your latent space, you can't just choose any latent space. There's this, there's this, there's this, there's this, there's this, there's this, you have to choose one of them. And if you choose something in between, which you'll inevitably will because this, all of our neural networks output continuous values, we're just going to have to use the same codebook. And in this case, we're just going to clamp you, we're just going to find the nearest one in our codebook. And we'll just say, well, we, we just make it such that you as if you had output that one. So the encoder can only hit one of those codebook vectors. And then you feed these codebook vectors to the decoder. And the decoder just decodes from these codebook vectors. Okay. And this should make output as project almost very good paying attention. And then we can can simply write the decoder in anyữa like way. So I've Stanford and bizarre and execute suite programming are out to be much, much, much better than simply doing the auto encoder thing continuously. So imagine that this codebook vocabulary is sort of like a vocabulary of This is a cat. And you don't just encode this into one of these words. What you do is you split the image up into a grid. It's not as fine as pixels. It's fairly, it's okay large. So in their experiments, they're going to use something like 32 by 32 grids, which is also what Dolly uses. Every image is described by 1024 tokens. That's 32 by 32 tokens. And then you're going to encode, you're going to make an encoder such that when this grid is through the encoder, this thing here corresponds to one of the code vectors and this thing here corresponds to another one. So you have your big vocabulary right here. And this is the red vector, this is the blue vector, this is the green vector, and you're going to just describe the image regions with these codebook vectors, like such. Now, the fact that you have a lot of these vectors, you have in fact, you have 8092 vectors in Dolly. And the image only consists of 1024 tokens. So, you know, it's conceivable, like, it's not like here where you have to reuse the same token over and over again. But one of these tokens could, for example, be sky. So maybe this is the thing that sort of describes sky. So what you'll have is like this thing and this thing and this thing and this thing should be approximately sky. Right. And then maybe the red one is is, I don't know, animal. And the blue one is vegetation. And the green one is some something else. So you can see if you feed this to a model that has to make a picture from it, it can just look at this and it's sort of like a description, a low resolution description of an image is not exactly a down sampled image. It's a it's a description because these things here contain a lot of information by themselves. OK, it's just that you can't choose any vector in latent space. You have to choose one of those vectors in the codebook. So that's a vector quantized VAE. And they train everything at the same time. So they train the encoder and decoder with this straight through estimator because this nearest neighbor computation isn't exactly differentiable. They also train the codebook to match the outputs of the encoder. So you can train that or you can just take the the exponential average of the encoder outputs. And that's the VQVAE, which is developed more in VQVAE 2. So this is VQVAE 2. I've linked the papers. VQVAE. What's the writing of 3 to the version two of it does the same thing, but in multi scale. So here you can see that in the encoder, you you you take the image and you put it at multiple resolutions. So this is large resolution. This is low resolution. Then you use the vector quantization to encode this into this grid and encode this into the codebook vectors. So again, here maybe you have red, red, red. This is red and this is the green one and so on. So you each square has to choose one of these eight thousand vectors to represent itself. And then you do this sort of hierarchical thing where you use the deep a decoder on this level to produce a slightly higher resolution image. But then you quantize again and you use a decoder at a next level to produce an even higher resolution image. So you can see that this hierarchical models, usually these hierarchical models, if you want good high resolution images, you sort of need them. So you can see that the the top decoder here outputs something quite blocky. And then every every additional one adds a sort of details to the image. It's pretty impressive as such. And you can see that the training right here of the VQVA. These are these are papers from last year or the years before. So this has been known. What Dali does is from what I can gather from the blog post right here. The images are preprocessed to 256 to 256 during training, similar to VQVA each image is compressed to a 32 by 32 grid of discrete latent codes using a discrete VAE that we pre trained using a continuous relaxation. OK, there's a lot of there's a lot of stuff here. So the VAE is pre trained. And they're saying they're saying also down here that their model uses maximum likelihood to to generate all of the tokens one after another. It's decoder only and so on. So probably this whole pipeline here is pre trained. They pre train a VAE a discrete VAE. And then they simply the Dali model simply has to learn how to produce the tokens. Right. The Dali model simply has to learn how to produce these hieroglyphs. And the box is fixed. The box is not changed. It's possible that they also train the decoder here. So the decoder. But I don't know, I can't tell this from the blog post. What's certainly is that they what's certain is that they don't train the encoder. So what you would do in a single step of Dali is you would have your text right here. Blah, blah, blah. And you would have a partial image. OK, you would input this text and the partial image to Dali. The partial image is any image where you've blacked out the bottom right. And they do the bottom right simply. It's the same as you do left to right by text. So you do sort of top left to bottom right. And yeah, it's good because you can always flip an image. Maybe not actually. But it's just a bias that you have to provide the model with in order to do autoregressive training. Right. So here is the image of that cat. Right. I. And you black out the bottom right. You can black out the whole image if you want the model to produce images unconditionally. All right. So you black all of this out. Cool. So now what you do is these here, they are already. They are already words. Right. You tokenize those token, token, token, and you go into your vocabulary of text. Right. So there is a vocabulary of text somewhere. There's blah. And you encode all of these using that vocabulary. So this is maybe word 34. So this is word 34, 34, 34. You go to your image. You rasterize this according to your definition. OK. And then you go and run this through this encoder that you trained. So you run it through the box and the box will tell you for each of this grid outputs, the box will tell you, well, in my in my vocabulary of image pieces, this here is number two. This here is number four. This is two again. This is 35 and so on. So you do this left to right, top to bottom, and then you put it right here. OK. So this is followed by an image of two, four, two, 35. And what you ask the model to do is simply to predict from all of this. And the model knows that this is text and this is images. From all of this, predict the next token, which would be this token right here. So you want to predict this one right here. What is it? And that's how you train the model. Right. And once it gets that, you can ask it to predict the next one and so on. And in this way, you can let it generate an entire image at inference time. And you know, you can train this. They say all these tokens are generated autoregressively. Now, in my understanding, this is all the model does, because once you have that token, so if the model says this is number seven, you go back to your box and you say, please. It's a different box. This is the encoder. This is the encoder of the VQVA. And now you go to your decoder that you've also pre-trained. Right. So this is a different box. And you ask it, I have this image, right? I have two, four, two, 35 and seven. Please generate an image for me for that. Or maybe you want to wait until you have the complete image. Right. So you have the complete image and you give this to your decoder. These are now that these hieroglyphs, right? So you have the box and the box produces an image. And the box says, well, OK, this cat here probably reproduces the ears fairly well, because you can describe them sort of exactly. Maybe you also want to copy that over or something. But then it says, well, it's a cat. So I'm going to, you know, maybe this. If the model has done a good job, there should be some sort of a cat. Right. And the model, you know, maybe in these hieroglyphs, it's even described how the cat looks like. The cat looks straight ahead as whiskers, as eyes and so on. OK. So I'm going to guess that the part on top that is trained and the part on bottom is pre-trained. With the option that the decoder part could also be trained at training time. At the same time, they train this language model on top. So they make some further inferences right here. They say each image is compressed in latent codes using a discrete V that we pre-trained using a continuous relaxation. We found that training using the relaxation obviates the need for an explicit codebook, EMA loss or tricks like dead code revival and can scale up to large vocabulary sizes. And this is the part where I am a bit confused. So clearly they say they have a vocabulary individual domain. OK, there are 8192. Well, I don't know my powers of two 8192 different words in the codebook. So there must be a codebook. But they say there this obviates the need for an explicit codebook. So I don't really know what to make of that. I can tell you what a continuous relaxation might look like. So this is from a different paper that they linked of the concrete random variables. So if you have an operation such as this, like a discrete random variable, you need to take an argmax of it. What you'll have is you'll have some sort of logits, right? There may be like this and you take the argmax of it, which means that you put it into a distribution where it's just one value. And this is sort of the same operation as we do in the VQVAE, where we assign each each output of the encoder to the nearest codebook vector. We say you can only have one of the codebook vectors. That's it. Right. Now, what you want to do when you relax this is you want to say, well, instead of that, what you could do is you could just kind of take that codebook vector a lot, but also, you know, take a little bit of the others. So more than doing a hard assignment to a codebook vector, right? So here would be the output of your encoder and you hard assign it to the nearest neighbor. You want to say, well, I'm going to soft assign it to all the ones. It's sort of like the difference between k nearest neighbor and a Gaussian mixture model, as I understand. Not what they do here, but it's analogous to that. And with that, they don't need an explicit codebook. And I don't know what that means. What I can imagine is that they don't actually train the codebook vectors. Maybe they just quantized to some prefixed schema, or I just don't understand what they do. Yeah, here is an illustration of these discrete random variables. So you want to get to a point when when you sample the variable, as you drop your temperature, it more and more approaches this fixed sampling. Like you can be either here or here or here with the sort of masses that are indicated by the size of the circle. But as you increase the temperature, you go more to a mixture. So yeah, you can be at the corner, but you can also be kind of in this region or in this region or in this region. As you increase the temperature, you can see the the distribution becomes more of a mixture distribution. And the mixture distribution, any mixture distribution with a temperature other than zero, of course, now all of a sudden has sort of a defined gradient. Whereas these discrete random variables, they do not have a gradient. And that's the reason why the VQVAE needs to do this straight through estimator right here, because this hard assignment to the codebook does not have a gradient defined. With the soft relaxation, you do have a gradient. And maybe they just mean they don't need they don't need this hard assignment to the codebook. I'm not sure. Or maybe they just they quantize in a different way. Maybe they go back to a continuous latent space. Yeah, I can imagine they they might go back to a continuous latent space. But somehow, somehow, they still do this a form of quantization. This could be a fixed quantization. Like you say, OK, you can choose any of the bases vectors and some mixtures that we define between them. Or they define it via moving averages or they define it via batch statistics or I don't know. If you know, let me know in the comments to the video. Right. So this was my take on what the model does and what is probably behind it. Now, let's look at some more examples right here, because these are fun. So they they say it can sort of control attributes. So you see these, it's, for example, a pentagonal green clock. And you see it's not always pentagonal. It's sometimes hexagonal and sometimes heptagonal and whatnot. But in general, what it does well is sort of color and also kind of object description. So lunch box, it gets and green it gets. What it can't do super well is stuff like counting. So I have sort of a hypothesis. I have multiple hypotheses about here. Just see what's in all of these examples, how the text prompt is phrased. So it says a pentagonal green lunchbox, a green lunchbox in the shape of a pentagon. This is quite unusual way to phrase the prompt. And by the way, all these criticisms that I'm leveraging here, most of them are actually admitted and discussed in this blog post. It's actually it's pretty cool and pretty self, let's say, self critical of them. So it's this is I've you know, I thought of these things and then I read the little text. And then they they already describe what I concluded. It's sad. But yeah, it's pretty cool of them because the current climate is sort of make your research look as as cool and flawless as possible. This goes a bit against it. So they say that the images here aren't cherry picked. And I totally believe this. So they have a little trick that they do. They output, I think, five hundred and twelve images from their model because they can sample and then they re rank them using this other model that they've released this clip model. And this clip model is a pretty good re ranker. So you give it a piece of text and an image and sort of tells you how well they fit together. And so the outputs that you see here are re ranked by this model. So you see are strictly the best outputs according to that model. So it's not cherry picked by humans, but it's cherry picked by a very good model. And the second thing is that the text prompt here is absolutely cherry picked. Right. By the way, this is phrased. You can see that it is very, very brittle. Probably the model. I can't test it, but probably it's very brittle in how exactly you phrase this text prompt. And I'm going to guess they have tried a lot of things before they've released these few examples right here that they show. And they've made sure that they work. So, yeah, just keep in mind that this is very brittle. And we already know this from like GPT three. We know that the input might seem the same to a human, just phrased differently in some cases. And yet the model will output completely different things. And we know that a lot of these GPT three examples are very, very constructed in terms of the input prompt. So, yeah, the other thing is the model, as I said, it can do colors and it can do colors and textures pretty well. So we've already seen the things made of things. So the sphere made of noodles that actually probably exists, the sphere made of guacamole. However, it's not super good at counting, for example. And I have a sort of multiple hypothesis. So these image models, they tend to be very good at sort of style and texture. Style and texture are the domain of these image models, like anywhere where there's like a convolution. And by the way, they use in the VQVAE model. No, not in the VQVAE. In this transformer for images, they don't do full attention. What they do is each one of the image tokens can attend to each of the text tokens such as this. But the image tokens, they can only sort of attend in the grid layer by layer. In one layer, they can attend sort of to the row of other image elements. In another layer, they can attend to the same column. And in even another layer, they can attend to sort of the the surroundings of them, like a convolution. So they can attend to, let's say, their couple of neighbors right here. So it's not full attention, yet in every layer, every image token can attend to all the text tokens. So yeah, in these models, what you typically see is that textures and style is pretty good. However, global correspondences are not as good. And that's what you see a lot in these face models where the left and the right earring don't match and things like this. So global correspondences are not so good. And you would actually expect that objects aren't as good as well. Right. So here, this is still a clock. This is still a light bulb. This is still a stop sign. Right. So it somehow gets the objects correct, which in my hypothesis, it shouldn't because this is some sort of a global structure. However, I think that's just a matter of how the data set is collected. The data sets are probably we humans, we take pictures of objects. Right. So the fundamental structures in these data sets is the object. So it makes sense that it learns that we humans, we don't we don't take pictures and we often don't describe the count in them. So I can get that the model has a harder time to learn that and actually focuses just on the object as a global thing. The count would be a global thing. Right. But it's not that prominent in the data. And the rest is a local thing like the color, the texture and so on. Yeah. The cube made of porcupine. So you can see here that this this counting. So two is often quite good. Actually, here it mixes up glasses and glasses. Right. So two often works. However, if you go if you go past two, it often gets it wrong. So five, you'll get anything from three to seven clocks and so on. So I'm going to also guess it's very brittle. Like they're not here. Yes, they're sitting on a table. But if you take a object that's not that often on a table like a club, you'll see that it's pretty unrecognizable whether or not it's on a table. Five, four clubs. So, you know, the model is prone to ignoring part of its input if the likelihood in another part is larger. Also, it can't do things like this. You know, a stack of three cubes, a red cube is on the top sitting on a green cube. It often gets the order wrong, like it gets the cubes on top of each other. However, it often gets it wrong when it comes to, you know, the order, the global things. As I said, anything global that is not what the object is tends to be weak. Anything local tends to be strong in these models. And that's just a matter of how they're built and how the data is. So they say the image can render new views. And here is where I'm not as convinced. So here you have like an extreme close up view of a cubby cub, cabby bar, sorry, of a fox. They're close up. Sometimes they're extreme close up. Right. You can see that it gets like forest. It gets it gets pretty well. But then you say, OK, a ground level view like, and then you say, OK, an aerial view. Maybe some of them are aerial views. Some of them aren't. What's pretty cool is things like a OK, a fish eye lens view. I mean, that's that's pretty cool. And a they have some of them, a bottom view or a rear view. Yeah, the rear view works better. So it does understand these these kind of things like what's the rear of a fox and what's the front of a fox. Though, as you can also see, not always texture. It's very good at texture. So here something made of voxels can do that perfectly. An owl made of voxels like this looks like it comes straight from Minecraft. Right. Absolutely, absolutely cool. Even X-Ray sometimes doesn't always get the bones right. But yeah, as I said, style structure. Very cool. So here is an example of a completion. So they give the text prompt a photograph of a bust of Homer and the image, the top part of the image. And they say, well, it can describing a well-known figure. It can complete the figure. I don't agree that it completes Homer. Like it completes it probably just sees this bust and this and it just completes whatever fits. I don't I have not studied Homer as a historic person or busts of him. But, you know, I disagree that this depicts largely the same person very often. You can see here there is sometimes there is even, you know, there's completely unrelated stuff. There is that lady with the pearl earring by Vermeer somewhere in there and so on. And what I also like in this kind of this this one, you know, the game draw something where or, you know, pictionary and so on, there are people when they can't draw something, they just kind of write it on the picture. It's like, ah, screw it. Now, this is right. This is Homer. This is Homer. Now, I don't care what you say. This is Homer. But, you know, it does, you know, it does. So when you say Cleopatra, it it goes more into the into sort of the female direction Medusa. It has some though. I'm pretty sure Medusa has the snake, the snake hair. No, maybe Venus. Yeah, somewhat somewhat. It they test a lot of things like can it do mirror reflections? And you can see right here, they say it can do reflections on the ground pretty well, but it can't do reflections, for example, in a mirror, because in a lot of these pictures, the object like here would actually have to be in front of the mirror. However, in the fewest amount of pictures, the object mirrored is actually also in front of the mirror. So this kind of global correspondence isn't given as much. However, there is a fair bit of reflection on the ground, so to say. So, you know, that's pretty cool, but it's also probably very, very common in datasets. Yeah, cross section view of a walnut. So they sort of implore, sorry, explore the model, what it can do. And here you can see that, you know, if something is common in the dataset, you know, like the cross section view of human head, there are a lot of pictures of that right in the dataset. However, if it comes to cross section view of a where, where did I see the airplane? There is an airplane somewhere. It's less, it's less so. So you can see that this is still it is. So here it probably doesn't really know how that looks, because, you know, they probably on the image, on the Internet, even on the whole Internet, pictures of cross sections of airplanes or any sections of airplanes. Are not really distributed often. So it sort of just focuses on airplane. And then with cross section, it probably knows that it should somehow display some of the interior. So it just kind of produces some stuff that matches this thing. As I said, if if it can't make the likelihood high of all of the things, what it tends to do is just focus on one of the things and just make that likelihood high, which is reasonable for a model. A macro photo, macro photographs of stuff. These are pretty cool. This is what you would find in some image galleries. Absolutely. Then it can do various things like style transfer. And here is where it shines. Right. So you can have different paintings of different objects in different styles. So here you can like have an owl sitting in the forest in the morning. And you can have this as a painting, as a painting in the pop art style and so on. It's very, very impressive. So I absolutely glory actually, too, like as a postage stamp. These are these are these are absolutely amazing. And yeah, you can have stuff like stained glass windows. And this is yeah, it's where the model shines. And even here a storefront that has the word Opnea written on it. So just right now, just look at how convoluted this text prompt has to be for them to get this to work. It's impressive. But the text prompt has to be repeated and reformulated a bunch of times and so on. My personal favorite is the pie torch chips. They're crunchy. You get a piece of back prop in every package. So you can see it sometimes misses like this is perch, perch chips and so on. It sometimes misses. But it is pretty cool that it basically can do OCR, right. Or reverse OCR. You can you give it a piece of text and it sort of makes a picture with that on it. It's very, very impressive, even though, as we said, like the global the global correspondences are not always there. They do implore like fashion, a skirt like here that the yellow skirt, then, you know, these mannequins. And here they have a loft bedroom with a white bed next to a nightstand. There is a fish tank standing beside the bed and they give sort of the beginning of the image. And here's what the model comes up with. And, you know, you can imagine that there are a lot of pictures like this in the data set. So the model might be pretty good at stuff like this, though I have found their king bed next to, yeah, let's say the nightstand with the telescope. The telescope beside the bed, it just, you know, that beside like there's a telescope. Sometimes it's on the bed. Sometimes it's next to it. There are some weird telescopes around. Well, this is a lot of telescopes. That's a weird telescope. But, you know, the quality is pretty impressive. This is absolutely nitpicking that I'm doing here. Combining unrelated concepts. We've already seen the armchair in the shape of an avocado. They also have a snail made of harp. Though my personal favorite is the penguin made of garlic. The penguin made of garlic. This. Perfect, right? Absolutely adorable. And just qualitatively like this. This would take a human like you would pay a high quality, highly educated Photoshop artist quite a bit of money to get this sort of output. Right. And these models, they shine at this sort of style transfer texture stuff. And here you have the illustrations. You can have any kind of illustrations like the illustration of a baby shark with a mustache. Holding. There's holding an umbrella somewhere. Playing it. Running. Riding a unicycle. It's just it's just nice. And as I said, this is the same model that can do all of this stuff. And these are samples. They're just samples. They're not cherry picked. However, they are re-ranked. Remember that. So they can do hybrids of images, hybrids of different giraffe and turtle and so on. And they do sort of implore the model a little bit more where they, as I said, they give this cat on the top and they say they want the exact same cat on the top as a photo colored blue on the bottom. So you can see that doesn't always work. Right. But in a surprising amount of times, it actually does work. Sometimes it's just like a blue pot. But you can you can see it's not the finished model yet. However, it is a step into the direction that shows us that this is definitely, definitely possible. It can even do some of these progressive matrices where it fills in the bottom right. However, they do mention it's very, very finicky with respect to whether or not, for example, if you invert the color. So if you look at the bottom right of any of these things, if I invert the colors, the output sort of changes and it's often also not right. However, sometimes it is actually right, which is crazy because in some of these things, you have to do some crazy sort of inference that we usually we usually do these things in IQ tests. So I don't know the debate about what is intelligence goes on. They say it has geographic knowledge. However, I'm not sure it has geographic knowledge as it just associates words with particular images like they say, OK, this is a photo of food of China. OK, maybe you just not sure this classifies as geographic knowledge. He said he's yeah, also this temporal knowledge, a photo of a phone from the 20s. OK, you know, and then the different time periods, 60s, 70s, 80s, future and so on, like distant future. Like, wow, these phones, I particularly so I like the usually this stuff. It's it's pretty OK, right? But it's not temporal knowledge. It just associates a bunch of tokens with some sort of style of computer. Today's computer, the future computer, the distant future computer. Please know, please, please, please don't give me that. I don't want to. I don't want that. I love the action movie poster because so the style is correct. But it just says action movie in the future. Yeah, they do get sort of the kind of some of the styles. It just it just says action movie like this is like a like a naggy, naggy child like I'm hungry. Hi, hungry. I'm dead. All right. So they also have a summary right here and they do show what it means that they they use this clip to rerank. So on the left here, you can see just eight samples straight up from the model. And they're not too bad. But, you know, you increase the quality by sort of sampling more and then taking the best eight as you go to the right here, according to the reranker. So I'm going to guess they decided on five twelve because that was sort of, you know, it gives you already pretty diverse, pretty good, pretty high quality outputs right here. All right. So just lastly, shout out to the the the authors right here. The primary authors are deter mesh, Mikhail Pavlov, Gabrielle Goh and Scott Ray with a I guess the secondary supporting authors and most of open eye behind them. I don't know how they work. I would encourage you to go look at the model. It's pretty cool. Try out all these inputs. As I said, these are the inputs are simply restricting you because they don't trust you with their model. Yet, right in the real model, you can input any piece of text that you want and you will get out an image. And the fact that you have to select the stuff here is simply because that's the stuff they tried. That's the stuff their PR department has signed off on. Right. And so so you get to see that because as I said, they're not like this is at the same time, this is a PR dilemma when you release a generative model because it could release. They discussed this a little bit in the blog post. You know, it could release like very problematic images in a classifier. It's not as pronounced. It's also sometimes dangerous, but not as dangerous as if you have a generative model. That's the first thing. And the second thing is there is I mean, there is money in this definitely, definitely money to be made in this. So, you know, we'll see whether or not we get the full model or not. All right. With that, that was it for me. I hope you enjoy the blog post. I hope you enjoyed the video. If you did, let me know. Shared out. Subscribe if you haven't and bye bye.
 A sphere made of Swiss cheese. A sphere with a texture of Swiss cheese.
 And there you have it. Beautiful, very appetizing Swiss cheese balls. My Swiss heart had just
 skipped a beat out of this monstrosity. What's even cooler than a sphere made of Swiss cheese
 is a torus made of denim. These images are so cool. A torus made of denim. And the point here is
 that these images aren't photoshopped or sort of human created. They are AI generated. And they are
 generated by this new model that OpenAI released a blog post about. It's called Dali. And it can,
 what it can do is it can take a piece of text such as the one on top here. The fact that I can select
 is simply the fact that they don't give you access to the model. They just give you access
 of a bunch of things that they've tried. But the model can take any piece of text and it can output
 a picture that matches that text. So here you got a torus made of toothpaste. And the quality
 of these images is super astounding. And what's even more astounding is sort of the range of
 capabilities that this model has. So the model can do various things such as so in here the input is
 an illustration of a baby daikon radish in a tutu walking a dog. And you see an illustration of a
 baby daikon radish in a tutu walking a dog. The outputs are just adorable. These are generated
 by the AI. The same for an armchair in the shape of an avocado, a storefront that has the word
 OpenAI written on it. I've tried reverse image searching some of these images and I could not
 find them on the internet. So it's definitely not just a model sort of outputting an image it found
 somewhere. These are actually generated images. And the astounding thing is that it's the same
 model that outputs all of these different images. It's not one model here trained on illustrations
 and one model trained on chairs. It's a single model that can take in a piece of text and
 optionally part of an image or none of an image and it will output the image either it continues
 the image you already give part of or it just generates the image by itself. So the model is
 called Dali. And this is just a blog post for now by OpenAI. They say they'll follow this up with a
 paper. And if the paper brings substantially new things, I think I'll make a video on it. But today
 we're just going to look at what this model can do, how it works, how it probably works. And we
 can take some guesses of what we can read in the paper once it's out. In fact, OpenAI has brought
 out two new models along with this Dali model. They've also released a blog post and a paper
 about a model called Clip, which is more of a sort of a classifier, not exactly a classifier.
 It's sort of a it connects text and images in a different way. It's not a generative model.
 And we're going to look at that in a different video. But you can see the clear trend right here
 is that OpenAI is looking into connecting text and images. So they say Dali, which is an, this is a,
 and I think an homage to Salvador Dali and mixed with the character Wally. So they say it's a 12
 billion parameter version of GPT-3. So you know, it's more like, it's more like not GPT-3. That was
 more than 10 times larger, but it's a 12 billion parameter version of GPT-3 trained to generate
 images from text descriptions using a data set of text image pairs. We found that it has diverse
 set of capabilities, including creating anthropomorphized versions of animals and
 objects, combining unrelated concepts in plausible ways, rendering text and applying transformations
 to existing images. So a lot of the things they don't tell us here, especially the data set,
 like how did they get the data set? Nobody knows. They don't say this. They simply say it's a data
 set of text image pairs. And they sort of allude to the fact that they have large pieces of data,
 especially in the clip. Then they allude to the fact that you can just find data that connects
 text and images on the internet. And it's true if you if you search, if you scrape the correct
 websites, and do it in sort of a smart fashion, you can find a lot of data where there's an image
 and there's a piece of text describing that image. And we have to assume that they sort of scrape
 the internet for something like this. I don't think they have a lot of human explicitly human
 labeled data for this type of thing. So we'll just assume that they have like a huge data set.
 And of course, they train a huge model on it, a 12 billion parameter version of GPT three GPT three
 is the famous model, the famous text generation model by open AI. And you can sort of see the
 same things right here. So GPT three, my hypothesis was that it sort of smartly mixes the training
 data rather than remember the training data, it sort of remembers it and then smartly interpolates
 between it. And I think you can sort of see the same kind of things right here in that these are
 all definitely pictures that you could imagine in the real world. But they have, you know, they have,
 for example, they're changed to open AI in here, there are surely chairs that sort of look like
 this. So it just kind of mixes a chair with an avocado in a plausible way. I'm not saying this to
 denigrate the model, I'm saying that, I mean, this is seriously cool, the fact that it can do that.
 So they say like GPT three, Dulli is a transformer language model. Now, this is very,
 very interesting, the fact that it's a transformer language model, it receives both the text and the
 image as a single stream of data containing up to 1000 and 1280 tokens, and it's trained using maximum
 likelihood to generate all of the tokens one after another. Okay, this training procedure allows Dulli
 not only to generate images from scratch, but also to regenerate any rectangular region of an existing
 image that extends to the bottom right corner in a way that is consistent with the text prompt.
 And they say a little bit more here on the right. And they also say a little bit more down on the
 bottom. So I'm going to try to take a stab of explaining how this model works with the full
 knowledge that I might be wrong once the paper comes out. And for that, we have to go back a
 little bit and look at the models it draws from, namely the VQ VAE. So the vector quantized VAE
 literature, so VQ VAE will consider this to be sort of the inspiration of or one of the necessary
 ingredients of this model. So if we combine VQ VAE with something like GPT three, we get Dulli.
 That's my that's my hypothesis for today. Why combining these two models? So GPT three is
 extremely good at modeling language, right? So if I have a piece of text, let's go down here for a
 minute. And let's say I have a cat set on the mat. A transformer will be very good at understanding
 this sentence and being able to complete it. So if I cross out this and ask a transformer to continue
 the sentence, it will be able to continue the sentence just fine if it is if it is trained
 well. And that's exactly how GPT three works. Now imagine that I don't have a piece of text,
 but I have some sort of a description of an image, right? And let's say I have, I have a box.
 Here is a box. And the box which is going to be a VQ VAE can take in a description of an image in
 words, but not exactly words that humans understand. But let's say there is an image language,
 sort of like a programming language, okay. And you input symbols into the image, let's say,
 it's a bit like Egyptian hieroglyphs, maybe. So here is the here is the this, this hieroglyph thing,
 and then there is the sun, the sun thing. And then there is the tree, the word for tree, like the
 hieroglyph for tree. And I input that here. And the output will be an image where I don't know,
 there the sun is shining. Yes, I draw some like a child, it has a little smile, okay, deal with it.
 And there is a tree, maybe not exactly the tree from the hieroglyphs, but like some sort of some
 sort of tree that fits. And then there is some human in the scene, maybe the human sits here,
 the human sits at the tree, you know, relaxing, chilling. Okay, so this, now the image on the
 right is consistent of pixels, right. And modeling pixels with a transformer is very, very hard,
 because in the case of our model right here, it's something like 256 by 256 pixels. That would mean
 the transformer would have to generate 256 times 256, which is like two to the two to the 16. This
 is just too much for a transformer to model the pixels individually. So there are multiple ways
 around this, for example, modeling little regions right here, which are not really satisfactory.
 So what this model does is it sort of it doesn't try to model the picture as such, it tries
 to predict to predict these hieroglyphs right here, it tries to predict sort of a language
 that this box can understand and produce a picture from, okay, so its task is going to be given some
 sort of a given some sort of a text prefix. So a human in a sunny field, sunny day or on a sunny day,
 chilling under a tree. So this piece of text followed. So the model is trained to take this
 piece of text and output this sequence of hieroglyphs. Okay, so this sequence of hieroglyphs
 outputting from this piece of text. And that's something a transformer can do if you have a
 vocabulary right here. So if you have a fixed list of hieroglyphs that you could use, right,
 so in there there is the human is in there. That's a worse Egyptian. And then the pyramid
 is in here as well, some that you need, some that you don't need. So if there is a vocabulary,
 the transformer is going to be pretty, pretty good at generating this thing. So you need
 two parts. The first part right here is a transformer language model, a GPT-3 thing that can
 input a sequence of text, and it can output a sequence of text, which is just in a different
 vocabulary, namely this picture vocabulary. And then in the step two, you need a box that takes
 in this picture vocabulary and actually produces an images and image right here. So as I already
 said, this part is taken over by GPT, GPT-3, like the custom GPT model they built for this.
 And this part is taken over by something like a VQVAE, the generator part of it. So what is
 a VQVAE? A VQVAE is, and you will be able to see that. So the box that we're going to need is
 this box right here, from here up to where the image is. And this thing right here is going to
 be that vocabulary. So what does a VQVAE do? It takes the image here on the left, you can see
 that here's the encoder, it takes the image, it encodes it into a latent space. Now what a
 VAE would do, or what an autoencoder would do, is it would encode the image into a latent space,
 and then it would decode it again into and try to reproduce the same image. And then you assume
 that whatever is in the middle right here is a sensible representation, a latent representation
 of that image, right? If you can train this model, you're going to get some sort of a
 representation in the middle that describes the image, otherwise you couldn't reproduce the image.
 And there have been many models built on this concept. Now this model right here, it turns out
 that the classic autoencoder doesn't work too well. But this model works quite formidably. So
 what you're going to have is you're going to have this vocabulary right here. It's also called a
 codebook. Let's call it a codebook. So the codebook is also the vocabulary.
 So what you're saying is that you can't just output any latent encoding. So the encoder outputs a
 continuous vector. But what you're saying is it has to be one of those. Like there are a number
 of vectors that you have at your disposal, Mr. or Miss Encoder or Mrs. Encoder. There is a number of
 vectors that you have at your disposal. You can only choose those. You can't choose any vector
 that you want, right? So in your latent space, you can't just choose any latent space. There's this,
 there's this, there's this, there's this, there's this, there's this, you have to choose one of them.
 And if you choose something in between, which you'll inevitably will because this, all of our
 neural networks output continuous values, we're just going to have to use the same codebook.
 And in this case, we're just going to clamp you, we're just going to find the nearest one in our
 codebook. And we'll just say, well, we, we just make it such that you as if you had output that
 one. So the encoder can only hit one of those codebook vectors. And then you feed these codebook
 vectors to the decoder. And the decoder just decodes from these codebook vectors. Okay. And
 this should make output as project almost very good paying attention. And then we can can
 simply write the decoder in anyữa like way. So I've Stanford and bizarre and execute suite
 programming are out to be much, much, much better than simply doing the auto encoder thing
 continuously. So imagine that this codebook vocabulary is sort of like a vocabulary of
 This is a cat.
 And you don't just encode this into one of these words.
 What you do is you split the image up into a grid.
 It's not as fine as pixels.
 It's fairly, it's okay large.
 So in their experiments, they're going to use something like 32 by 32 grids,
 which is also what Dolly uses.
 Every image is described by 1024 tokens. That's 32 by 32 tokens.
 And then you're going to encode, you're going to make an encoder such that
 when this grid is through the encoder,
 this thing here corresponds to one of the code vectors
 and this thing here corresponds to another one.
 So you have your big vocabulary right here.
 And this is the red vector, this is the blue vector,
 this is the green vector,
 and you're going to just describe the image regions with these codebook vectors, like such.
 Now, the fact that you have a lot of these vectors,
 you have in fact, you have 8092 vectors in Dolly.
 And the image only consists of 1024 tokens.
 So, you know, it's conceivable, like, it's not like here where you have to reuse the same token over and over again.
 But one of these tokens could, for example, be sky.
 So maybe this is the thing that sort of describes sky.
 So what you'll have is like this thing and this thing and this thing and this thing should be approximately sky.
 Right. And then maybe the red one is is, I don't know, animal.
 And the blue one is vegetation.
 And the green one is some something else.
 So you can see if you feed this to a model that has to make a picture from it,
 it can just look at this and it's sort of like a description, a low resolution description of an image is not exactly a down sampled image.
 It's a it's a description because these things here contain a lot of information by themselves.
 OK, it's just that you can't choose any vector in latent space.
 You have to choose one of those vectors in the codebook.
 So that's a vector quantized VAE.
 And they train everything at the same time.
 So they train the encoder and decoder with this straight through estimator because this nearest neighbor computation isn't exactly differentiable.
 They also train the codebook to match the outputs of the encoder.
 So you can train that or you can just take the the exponential average of the encoder outputs.
 And that's the VQVAE, which is developed more in VQVAE 2.
 So this is VQVAE 2. I've linked the papers.
 VQVAE.
 What's the writing of 3 to the version two of it does the same thing, but in multi scale.
 So here you can see that in the encoder, you you you take the image and you put it at multiple resolutions.
 So this is large resolution.
 This is low resolution.
 Then you use the vector quantization to encode this into this grid and encode this into the codebook vectors.
 So again, here maybe you have red, red, red.
 This is red and this is the green one and so on.
 So you each square has to choose one of these eight thousand vectors to represent itself.
 And then you do this sort of hierarchical thing where you use the deep a decoder on this level to produce a slightly higher resolution image.
 But then you quantize again and you use a decoder at a next level to produce an even higher resolution image.
 So you can see that this hierarchical models, usually these hierarchical models, if you want good high resolution images, you sort of need them.
 So you can see that the the top decoder here outputs something quite blocky.
 And then every every additional one adds a sort of details to the image.
 It's pretty impressive as such.
 And you can see that the training right here of the VQVA.
 These are these are papers from last year or the years before.
 So this has been known.
 What Dali does is from what I can gather from the blog post right here.
 The images are preprocessed to 256 to 256 during training, similar to VQVA each image is compressed to a 32 by 32 grid of discrete latent codes using a discrete VAE that we pre trained using a continuous relaxation.
 OK, there's a lot of there's a lot of stuff here.
 So the VAE is pre trained.
 And they're saying they're saying also down here that their model uses maximum likelihood to to generate all of the tokens one after another.
 It's decoder only and so on.
 So probably this whole pipeline here is pre trained.
 They pre train a VAE a discrete VAE.
 And then they simply the Dali model simply has to learn how to produce the tokens.
 Right. The Dali model simply has to learn how to produce these hieroglyphs.
 And the box is fixed.
 The box is not changed.
 It's possible that they also train the decoder here.
 So the decoder.
 But I don't know, I can't tell this from the blog post.
 What's certainly is that they what's certain is that they don't train the encoder.
 So what you would do in a single step of Dali is you would have your text right here.
 Blah, blah, blah.
 And you would have a partial image.
 OK, you would input this text and the partial image to Dali.
 The partial image is any image where you've blacked out the bottom right.
 And they do the bottom right simply.
 It's the same as you do left to right by text.
 So you do sort of top left to bottom right.
 And yeah, it's good because you can always flip an image.
 Maybe not actually.
 But it's just a bias that you have to provide the model with in order to do autoregressive training.
 Right.
 So here is the image of that cat.
 Right.
 I.
 And you black out the bottom right.
 You can black out the whole image if you want the model to produce images unconditionally.
 All right.
 So you black all of this out.
 Cool.
 So now what you do is these here, they are already.
 They are already words.
 Right.
 You tokenize those token, token, token, and you go into your vocabulary of text.
 Right.
 So there is a vocabulary of text somewhere.
 There's blah.
 And you encode all of these using that vocabulary.
 So this is maybe word 34.
 So this is word 34, 34, 34.
 You go to your image.
 You rasterize this according to your definition.
 OK.
 And then you go and run this through this encoder that you trained.
 So you run it through the box and the box will tell you for each of this grid outputs,
 the box will tell you, well, in my in my vocabulary of image pieces,
 this here is number two.
 This here is number four.
 This is two again.
 This is 35 and so on.
 So you do this left to right, top to bottom, and then you put it right here.
 OK.
 So this is followed by an image of two, four, two, 35.
 And what you ask the model to do is simply to predict from all of this.
 And the model knows that this is text and this is images.
 From all of this, predict the next token, which would be this token right here.
 So you want to predict this one right here.
 What is it?
 And that's how you train the model.
 Right.
 And once it gets that, you can ask it to predict the next one and so on.
 And in this way, you can let it generate an entire image at inference time.
 And you know, you can train this.
 They say all these tokens are generated autoregressively.
 Now, in my understanding, this is all the model does, because once you have that token,
 so if the model says this is number seven, you go back to your box and you say, please.
 It's a different box.
 This is the encoder.
 This is the encoder of the VQVA.
 And now you go to your decoder that you've also pre-trained.
 Right.
 So this is a different box.
 And you ask it, I have this image, right?
 I have two, four, two, 35 and seven.
 Please generate an image for me for that.
 Or maybe you want to wait until you have the complete image.
 Right.
 So you have the complete image and you give this to your decoder.
 These are now that these hieroglyphs, right?
 So you have the box and the box produces an image.
 And the box says, well, OK, this cat here probably reproduces the ears fairly well,
 because you can describe them sort of exactly.
 Maybe you also want to copy that over or something.
 But then it says, well, it's a cat.
 So I'm going to, you know, maybe this.
 If the model has done a good job, there should be some sort of a cat.
 Right.
 And the model, you know, maybe in these hieroglyphs, it's even described how the cat looks like.
 The cat looks straight ahead as whiskers, as eyes and so on.
 OK.
 So I'm going to guess that the part on top that is trained and the part on bottom is pre-trained.
 With the option that the decoder part could also be trained at training time.
 At the same time, they train this language model on top.
 So they make some further inferences right here.
 They say each image is compressed in latent codes using a discrete V that we pre-trained using a continuous relaxation.
 We found that training using the relaxation obviates the need for an explicit codebook,
 EMA loss or tricks like dead code revival and can scale up to large vocabulary sizes.
 And this is the part where I am a bit confused.
 So clearly they say they have a vocabulary individual domain.
 OK, there are 8192.
 Well, I don't know my powers of two 8192 different words in the codebook.
 So there must be a codebook.
 But they say there this obviates the need for an explicit codebook.
 So I don't really know what to make of that.
 I can tell you what a continuous relaxation might look like.
 So this is from a different paper that they linked of the concrete random variables.
 So if you have an operation such as this, like a discrete random variable, you need to take an argmax of it.
 What you'll have is you'll have some sort of logits, right?
 There may be like this and you take the argmax of it, which means that you put it into a distribution where it's just one value.
 And this is sort of the same operation as we do in the VQVAE, where we assign each each output of the encoder to the nearest codebook vector.
 We say you can only have one of the codebook vectors.
 That's it. Right.
 Now, what you want to do when you relax this is you want to say, well, instead of that, what you could do is you could just kind of take that codebook vector a lot, but also, you know, take a little bit of the others.
 So more than doing a hard assignment to a codebook vector, right?
 So here would be the output of your encoder and you hard assign it to the nearest neighbor.
 You want to say, well, I'm going to soft assign it to all the ones.
 It's sort of like the difference between k nearest neighbor and a Gaussian mixture model, as I understand.
 Not what they do here, but it's analogous to that.
 And with that, they don't need an explicit codebook.
 And I don't know what that means.
 What I can imagine is that they don't actually train the codebook vectors.
 Maybe they just quantized to some prefixed schema, or I just don't understand what they do.
 Yeah, here is an illustration of these discrete random variables.
 So you want to get to a point when when you sample the variable, as you drop your temperature, it more and more approaches this fixed sampling.
 Like you can be either here or here or here with the sort of masses that are indicated by the size of the circle.
 But as you increase the temperature, you go more to a mixture.
 So yeah, you can be at the corner, but you can also be kind of in this region or in this region or in this region.
 As you increase the temperature, you can see the the distribution becomes more of a mixture distribution.
 And the mixture distribution, any mixture distribution with a temperature other than zero, of course, now all of a sudden has sort of a defined gradient.
 Whereas these discrete random variables, they do not have a gradient.
 And that's the reason why the VQVAE needs to do this straight through estimator right here, because this hard assignment to the codebook does not have a gradient defined.
 With the soft relaxation, you do have a gradient.
 And maybe they just mean they don't need they don't need this hard assignment to the codebook.
 I'm not sure. Or maybe they just they quantize in a different way.
 Maybe they go back to a continuous latent space.
 Yeah, I can imagine they they might go back to a continuous latent space.
 But somehow, somehow, they still do this a form of quantization.
 This could be a fixed quantization.
 Like you say, OK, you can choose any of the bases vectors and some mixtures that we define between them.
 Or they define it via moving averages or they define it via batch statistics or I don't know.
 If you know, let me know in the comments to the video.
 Right. So this was my take on what the model does and what is probably behind it.
 Now, let's look at some more examples right here, because these are fun.
 So they they say it can sort of control attributes.
 So you see these, it's, for example, a pentagonal green clock.
 And you see it's not always pentagonal.
 It's sometimes hexagonal and sometimes heptagonal and whatnot.
 But in general, what it does well is sort of color and also kind of object description.
 So lunch box, it gets and green it gets.
 What it can't do super well is stuff like counting.
 So I have sort of a hypothesis.
 I have multiple hypotheses about here.
 Just see what's in all of these examples, how the text prompt is phrased.
 So it says a pentagonal green lunchbox, a green lunchbox in the shape of a pentagon.
 This is quite unusual way to phrase the prompt.
 And by the way, all these criticisms that I'm leveraging here, most of them are actually admitted and discussed in this blog post.
 It's actually it's pretty cool and pretty self, let's say, self critical of them.
 So it's this is I've you know, I thought of these things and then I read the little text.
 And then they they already describe what I concluded.
 It's sad. But yeah, it's pretty cool of them because the current climate is sort of make your research look as as cool and flawless as possible.
 This goes a bit against it.
 So they say that the images here aren't cherry picked.
 And I totally believe this.
 So they have a little trick that they do.
 They output, I think, five hundred and twelve images from their model because they can sample and then they re rank them using this other model that they've released this clip model.
 And this clip model is a pretty good re ranker.
 So you give it a piece of text and an image and sort of tells you how well they fit together.
 And so the outputs that you see here are re ranked by this model.
 So you see are strictly the best outputs according to that model.
 So it's not cherry picked by humans, but it's cherry picked by a very good model.
 And the second thing is that the text prompt here is absolutely cherry picked.
 Right.
 By the way, this is phrased.
 You can see that it is very, very brittle.
 Probably the model.
 I can't test it, but probably it's very brittle in how exactly you phrase this text prompt.
 And I'm going to guess they have tried a lot of things before they've released these few examples right here that they show.
 And they've made sure that they work.
 So, yeah, just keep in mind that this is very brittle.
 And we already know this from like GPT three.
 We know that the input might seem the same to a human, just phrased differently in some cases.
 And yet the model will output completely different things.
 And we know that a lot of these GPT three examples are very, very constructed in terms of the input prompt.
 So, yeah, the other thing is the model, as I said, it can do colors and it can do colors and textures pretty well.
 So we've already seen the things made of things.
 So the sphere made of noodles that actually probably exists, the sphere made of guacamole.
 However, it's not super good at counting, for example.
 And I have a sort of multiple hypothesis.
 So these image models, they tend to be very good at sort of style and texture.
 Style and texture are the domain of these image models, like anywhere where there's like a convolution.
 And by the way, they use in the VQVAE model.
 No, not in the VQVAE. In this transformer for images, they don't do full attention.
 What they do is each one of the image tokens can attend to each of the text tokens such as this.
 But the image tokens, they can only sort of attend in the grid layer by layer.
 In one layer, they can attend sort of to the row of other image elements.
 In another layer, they can attend to the same column.
 And in even another layer, they can attend to sort of the the surroundings of them, like a convolution.
 So they can attend to, let's say, their couple of neighbors right here.
 So it's not full attention, yet in every layer, every image token can attend to all the text tokens.
 So yeah, in these models, what you typically see is that textures and style is pretty good.
 However, global correspondences are not as good.
 And that's what you see a lot in these face models where the left and the right earring don't match and things like this.
 So global correspondences are not so good.
 And you would actually expect that objects aren't as good as well.
 Right. So here, this is still a clock.
 This is still a light bulb.
 This is still a stop sign.
 Right. So it somehow gets the objects correct, which in my hypothesis, it shouldn't because this is some sort of a global structure.
 However, I think that's just a matter of how the data set is collected.
 The data sets are probably we humans, we take pictures of objects.
 Right. So the fundamental structures in these data sets is the object.
 So it makes sense that it learns that we humans, we don't we don't take pictures and we often don't describe the count in them.
 So I can get that the model has a harder time to learn that and actually focuses just on the object as a global thing.
 The count would be a global thing. Right. But it's not that prominent in the data.
 And the rest is a local thing like the color, the texture and so on.
 Yeah. The cube made of porcupine.
 So you can see here that this this counting.
 So two is often quite good.
 Actually, here it mixes up glasses and glasses. Right.
 So two often works.
 However, if you go if you go past two, it often gets it wrong.
 So five, you'll get anything from three to seven clocks and so on.
 So I'm going to also guess it's very brittle.
 Like they're not here.
 Yes, they're sitting on a table.
 But if you take a object that's not that often on a table like a club,
 you'll see that it's pretty unrecognizable whether or not it's on a table.
 Five, four clubs.
 So, you know, the model is prone to ignoring part of its input if the likelihood in another part is larger.
 Also, it can't do things like this.
 You know, a stack of three cubes, a red cube is on the top sitting on a green cube.
 It often gets the order wrong, like it gets the cubes on top of each other.
 However, it often gets it wrong when it comes to, you know, the order, the global things.
 As I said, anything global that is not what the object is tends to be weak.
 Anything local tends to be strong in these models.
 And that's just a matter of how they're built and how the data is.
 So they say the image can render new views.
 And here is where I'm not as convinced.
 So here you have like an extreme close up view of a cubby cub, cabby bar, sorry, of a fox.
 They're close up. Sometimes they're extreme close up. Right.
 You can see that it gets like forest. It gets it gets pretty well.
 But then you say, OK, a ground level view like, and then you say, OK, an aerial view.
 Maybe some of them are aerial views. Some of them aren't.
 What's pretty cool is things like a OK, a fish eye lens view.
 I mean, that's that's pretty cool.
 And a they have some of them, a bottom view or a rear view.
 Yeah, the rear view works better.
 So it does understand these these kind of things like what's the rear of a fox and what's the front of a fox.
 Though, as you can also see, not always texture.
 It's very good at texture.
 So here something made of voxels can do that perfectly.
 An owl made of voxels like this looks like it comes straight from Minecraft.
 Right. Absolutely, absolutely cool.
 Even X-Ray sometimes doesn't always get the bones right.
 But yeah, as I said, style structure.
 Very cool. So here is an example of a completion.
 So they give the text prompt a photograph of a bust of Homer and the image, the top part of the image.
 And they say, well, it can describing a well-known figure.
 It can complete the figure.
 I don't agree that it completes Homer.
 Like it completes it probably just sees this bust and this and it just completes whatever fits.
 I don't I have not studied Homer as a historic person or busts of him.
 But, you know, I disagree that this depicts largely the same person very often.
 You can see here there is sometimes there is even, you know, there's completely unrelated stuff.
 There is that lady with the pearl earring by Vermeer somewhere in there and so on.
 And what I also like in this kind of this this one, you know, the game draw something where or, you know,
 pictionary and so on, there are people when they can't draw something, they just kind of write it on the picture.
 It's like, ah, screw it. Now, this is right.
 This is Homer. This is Homer.
 Now, I don't care what you say.
 This is Homer. But, you know, it does, you know, it does.
 So when you say Cleopatra, it it goes more into the into sort of the female direction Medusa.
 It has some though.
 I'm pretty sure Medusa has the snake, the snake hair.
 No, maybe Venus. Yeah, somewhat somewhat.
 It they test a lot of things like can it do mirror reflections?
 And you can see right here, they say it can do reflections on the ground pretty well,
 but it can't do reflections, for example, in a mirror, because in a lot of these pictures,
 the object like here would actually have to be in front of the mirror.
 However, in the fewest amount of pictures, the object mirrored is actually also in front of the mirror.
 So this kind of global correspondence isn't given as much.
 However, there is a fair bit of reflection on the ground, so to say.
 So, you know, that's pretty cool, but it's also probably very, very common in datasets.
 Yeah, cross section view of a walnut.
 So they sort of implore, sorry, explore the model, what it can do.
 And here you can see that, you know, if something is common in the dataset, you know,
 like the cross section view of human head, there are a lot of pictures of that right in the dataset.
 However, if it comes to cross section view of a where, where did I see the airplane?
 There is an airplane somewhere.
 It's less, it's less so.
 So you can see that this is still it is.
 So here it probably doesn't really know how that looks, because, you know, they probably on the image,
 on the Internet, even on the whole Internet, pictures of cross sections of airplanes or any sections of airplanes.
 Are not really distributed often.
 So it sort of just focuses on airplane.
 And then with cross section, it probably knows that it should somehow display some of the interior.
 So it just kind of produces some stuff that matches this thing.
 As I said, if if it can't make the likelihood high of all of the things,
 what it tends to do is just focus on one of the things and just make that likelihood high,
 which is reasonable for a model.
 A macro photo, macro photographs of stuff.
 These are pretty cool.
 This is what you would find in some image galleries.
 Absolutely.
 Then it can do various things like style transfer.
 And here is where it shines.
 Right.
 So you can have different paintings of different objects in different styles.
 So here you can like have an owl sitting in the forest in the morning.
 And you can have this as a painting, as a painting in the pop art style and so on.
 It's very, very impressive.
 So I absolutely glory actually, too, like as a postage stamp.
 These are these are these are absolutely amazing.
 And yeah, you can have stuff like stained glass windows.
 And this is yeah, it's where the model shines.
 And even here a storefront that has the word Opnea written on it.
 So just right now, just look at how convoluted this text prompt has to be for them to get this to work.
 It's impressive.
 But the text prompt has to be repeated and reformulated a bunch of times and so on.
 My personal favorite is the pie torch chips.
 They're crunchy.
 You get a piece of back prop in every package.
 So you can see it sometimes misses like this is perch, perch chips and so on.
 It sometimes misses.
 But it is pretty cool that it basically can do OCR, right.
 Or reverse OCR.
 You can you give it a piece of text and it sort of makes a picture with that on it.
 It's very, very impressive, even though, as we said, like the global the global correspondences are not always there.
 They do implore like fashion, a skirt like here that the yellow skirt, then, you know, these mannequins.
 And here they have a loft bedroom with a white bed next to a nightstand.
 There is a fish tank standing beside the bed and they give sort of the beginning of the image.
 And here's what the model comes up with.
 And, you know, you can imagine that there are a lot of pictures like this in the data set.
 So the model might be pretty good at stuff like this, though I have found their king bed next to, yeah, let's say the nightstand with the telescope.
 The telescope beside the bed, it just, you know, that beside like there's a telescope.
 Sometimes it's on the bed.
 Sometimes it's next to it.
 There are some weird telescopes around.
 Well, this is a lot of telescopes.
 That's a weird telescope.
 But, you know, the quality is pretty impressive.
 This is absolutely nitpicking that I'm doing here.
 Combining unrelated concepts.
 We've already seen the armchair in the shape of an avocado.
 They also have a snail made of harp.
 Though my personal favorite is the penguin made of garlic.
 The penguin made of garlic.
 This.
 Perfect, right?
 Absolutely adorable.
 And just qualitatively like this.
 This would take a human like you would pay a high quality,
 highly educated Photoshop artist quite a bit of money to get this sort of output.
 Right.
 And these models, they shine at this sort of style transfer texture stuff.
 And here you have the illustrations.
 You can have any kind of illustrations like the illustration of a baby shark with a mustache.
 Holding.
 There's holding an umbrella somewhere.
 Playing it.
 Running.
 Riding a unicycle.
 It's just it's just nice.
 And as I said, this is the same model that can do all of this stuff.
 And these are samples.
 They're just samples.
 They're not cherry picked.
 However, they are re-ranked.
 Remember that.
 So they can do hybrids of images, hybrids of different giraffe and turtle and so on.
 And they do sort of implore the model a little bit more where they, as I said, they give this cat on the top and they say they want the exact same cat on the top as a photo colored blue on the bottom.
 So you can see that doesn't always work.
 Right.
 But in a surprising amount of times, it actually does work.
 Sometimes it's just like a blue pot.
 But you can you can see it's not the finished model yet.
 However, it is a step into the direction that shows us that this is definitely, definitely possible.
 It can even do some of these progressive matrices where it fills in the bottom right.
 However, they do mention it's very, very finicky with respect to whether or not, for example, if you invert the color.
 So if you look at the bottom right of any of these things, if I invert the colors, the output sort of changes and it's often also not right.
 However, sometimes it is actually right, which is crazy because in some of these things, you have to do some crazy sort of inference that
 we usually we usually do these things in IQ tests.
 So I don't know the debate about what is intelligence goes on.
 They say it has geographic knowledge.
 However, I'm not sure it has geographic knowledge as it just associates words with particular images like they say, OK, this is a photo of food of China.
 OK, maybe you just not sure this classifies as geographic knowledge.
 He said he's yeah, also this temporal knowledge, a photo of a phone from the 20s.
 OK, you know, and then the different time periods, 60s, 70s, 80s, future and so on, like distant future.
 Like, wow, these phones, I particularly so I like the usually this stuff.
 It's it's pretty OK, right?
 But it's not temporal knowledge.
 It just associates a bunch of tokens with some sort of style of computer.
 Today's computer, the future computer, the distant future computer.
 Please know, please, please, please don't give me that.
 I don't want to. I don't want that.
 I love the action movie poster because so the style is correct.
 But it just says action movie in the future.
 Yeah, they do get sort of the kind of some of the styles.
 It just it just says action movie like this is like a like a naggy, naggy child like I'm hungry.
 Hi, hungry. I'm dead.
 All right. So they also have a summary right here and they do show what it means that they they use this clip to rerank.
 So on the left here, you can see just eight samples straight up from the model.
 And they're not too bad. But, you know, you increase the quality by sort of sampling more and then taking the best eight as you go to the right here, according to the reranker.
 So I'm going to guess they decided on five twelve because that was sort of, you know, it gives you already pretty diverse, pretty good, pretty high quality outputs right here.
 All right. So just lastly, shout out to the the the authors right here.
 The primary authors are deter mesh, Mikhail Pavlov, Gabrielle Goh and Scott Ray with a I guess the secondary supporting authors and most of open eye behind them.
 I don't know how they work. I would encourage you to go look at the model.
 It's pretty cool. Try out all these inputs. As I said, these are the inputs are simply restricting you because they don't trust you with their model.
 Yet, right in the real model, you can input any piece of text that you want and you will get out an image.
 And the fact that you have to select the stuff here is simply because that's the stuff they tried.
 That's the stuff their PR department has signed off on. Right.
 And so so you get to see that because as I said, they're not like this is at the same time, this is a PR dilemma when you release a generative model because it could release.
 They discussed this a little bit in the blog post. You know, it could release like very problematic images in a classifier.
 It's not as pronounced. It's also sometimes dangerous, but not as dangerous as if you have a generative model.
 That's the first thing. And the second thing is there is I mean, there is money in this definitely, definitely money to be made in this.
 So, you know, we'll see whether or not we get the full model or not.
 All right. With that, that was it for me. I hope you enjoy the blog post. I hope you enjoyed the video.
 If you did, let me know. Shared out. Subscribe if you haven't and bye bye.