Parti - Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper diffusion models what is deep learning deep learning tutorial introduction to deep learning generative models parti google parti google party google pathways google imagen image dalle dalle2 dalle 2 dall e 2 dall e 2 vs graphic designer anubis
#parti #ai #aiart 

Parti is a new autoregressive text-to-image model that shows just how much scale can achieve. This model's outputs are crips, accurate, realistic, and can combine arbitrary styles, concepts, and fulfil even challenging requests.

OUTLINE:
0:00 - Introduction
2:40 - Example Outputs
6:00 - Model Architecture
17:15 - Datasets (incl. PartiPrompts)
21:45 - Experimental Results
27:00 - Picking a cherry tree
29:30 - Failure cases
33:20 - Final comments

Website: https://parti.research.google/
Paper: https://arxiv.org/abs/2206.10789
Github: https://github.com/google-research/parti

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Not a day goes by in AI research in which we don't get a new image generation model these days. So take a look at the top row right here and listen to the prompt that generated them. Oil on canvas painting of a blue night sky with roiling energy. A fuzzy and bright yellow crescent moon shining at the top. Below the exploding yellow stars and radiating swirls of blue, a distant village sits quietly on the right. Connecting earth and sky is a flame-like cypress tree with curling and swaying branches on the left. A church spire rises as a beacon over rolling blue hills. That is a 67 word description of Starry Night by Vincent van Gogh. And it is also the prompt that generated the top row of images. And the paper does this to show that image generation models, specifically this one, they have become super duper capable of incorporating not only wild concepts, as you can see here, co-locating the Eiffel Tower with the Sydney skyline and fireworks and whatnot, but also, you know, minute details about things in the image and where things are and how things look. So we've gone from essentially conditional GANs where we could create one of 10 classes to something where we can input like a little essay about what we want to see and get it out. So this is by a group of researchers out of Google Research. And they are a parallel work to the Imogen model that you might have seen. So this model or the paper is called Scaling Autoregressive Models for Content-Rich Text to Image Generation. But the model is called, let me grab if I can, let me grab a pen. The model is called PARTI. And I have no clue how to pronounce this. This could be party. Maybe the pronunciation is on the art or on the part because it's pathways like it's, or partai or I have no idea. Let's call it PARTI. And PARTI is a model that generates images from text as we have so many models. However, it doesn't do this in the same style as like Imogen, which is a diffusion model. It is an autoregressive model. So here you can see a bunch of other outputs like this. This is insane. Look at the left side right here. A photo of a frog reading the newspaper named Toaday. The newspaper is named Toaday. Like how crazy is that? That in itself is pretty funny. But we know that these image to sorry, these text to image models are pretty bad at spelling stuff in images. Well, not this model, as you can see right here, it gets it completely right. It doesn't always get it right, but it gets it right often enough. Or this one, portrait of a statue of the Egyptian god Anubis wearing aviator goggles like another connoisseur of fine eyewear. White t-shirt and a leather jacket. The city of Los Angeles is in the background. High res DSLR photograph. That's literally that's the academic version of the Unreal Engine trick right here. And you can see the images spot on. So this requires a lot of knowledge, not only of, you know, what a DSLR photograph is, but also how the skyline of Los Angeles looks, how the Egyptian got an Anubis looks right. And the composition of things together like these, this god was never in a leather jacket depicted, I guess, maybe on the internet you'll find anything. But you can see a bunch of more examples right here. I specifically love the thing on the left side here. You can see that they generated images. So the prompt is three quarters front view of a XYZ coming around a curve in a mountain road looking over a green valley on a cloudy day. So X here is any of the colors blue, red and yellow. Y is any of the numbers. 1977, 1997 and 2017. And Z is any of these car types. And now look that the model can essentially track the historical evolution of these cars. So not only does it know what a Porsche is, it also knows how a Porsche in 77 looked like. Maybe it's not exactly the correct year, but this is pretty crazy. You can see a bunch more examples right here. They do a lot of examples with animals. I specifically like the raccoon here in the style of cubism. So this is going to be very, very powerful technology. We can immediately see that, you know, the quality of these models gets fast, gets quickly, sorry, gets well, gets better so quickly that in the foreseeable future, we're going to have super powerful tools to just create and edit images from text. Look at the left side here, a giant cobra snake made from salad. You know, I'm sure they even say these are cherry picked, but still this is insane. Now, I would love to tell you that behind all of this cool development is a really cool idea like is a smart architecture and something like this. But I'm afraid it is not. It is simply scale and not simply scale. I mean, you have to have the sort of correct base architecture. There is nothing like particularly there's no cool invention in architecture or a neat trick involved or anything like this. It's really just plug basic things together, make them really big, train them for long on a lot of data and you'll get quality. So this is the model overview right here, the overview of this party or part time model. This is, as I already said, in contrast to image and it is an auto regressive model, so not a diffusion model. What happens is that on this side here, you have this VQ GAN image encoder and decoder. Well, they don't call them encoder and decoder, they call them tokenizer and de tokenizer. So if you are not aware, auto regressive models, they work on tokens. Now, tokens in usually in natural language processing are words or part of words. So these would be tokens, token one, token two, and so on until token N. And then what you would try to do is you would try always to predict the next token. That's what makes it auto regressive. You feed in parts of a token sequence, like parts of a sentence, you try to predict the next one. That's exactly what you see right here in the architecture. So you pass in the start of sentence token, you try to predict the first token, then you pass in the first token. And then from these two, you try to predict the second token. And then you put that here from these three, you try to predict the third token and so on. That's the auto regressivity. In text, that works well. However, in images, it's not quite obvious how to do that. That's why you first need to get from the image space to the token space. So we need a way for any given image that we get out a sequence of tokens. And it can't be the pixels themselves. We would like to have tokens that are kind of latent and have sort of a bit of meaning, not just individual pixels, because that, first of all, is too many pixels. And second of all, there's not too much, let's say, information in the single pixel. So what we do is we have these image tokenizer and detokenizer. This is a VQGAN that is powered by a vision transformer. So essentially, this is a model that takes this image, it ships it through a bunch of layers. And at the end, so let's say the image at the beginning has a bunch of rows, a bunch of columns with its pixels. This goes through a series of maybe downscalings and so on. No, actually, it's because it's a vision transformer. It probably even tokenizes, like it patches the image at the very beginning. So these would be image patches. Then these are transformed by a transformer to a latent space. Maybe they are compressed. And then you get tokens. So at the end, you can take these things right here or the things that correspond to them in the latent representation. You can take those as image tokens and you can unroll essentially this image and then feed it into this model. Hey, just a short interjection here from Janek from the future. The idea, I forgot, the idea behind the whole setup here is behind the whole VQGAN is obviously that these things here are tokens, which means that they come from a set vocabulary. So the way you train a VQGAN isn't just to give you this latent representation of like token like things, but then you also quantize them. So there is also a vocabulary somewhere where you have a set defined set of tokens. I believe in their case, they have like 8,000 tokens or so. And your image, your image tokens must be of these 8,000. So the image has a bunch of tokens, but they all must be one of the things in the vocabulary here. Now, the vocabulary is also learned. There are some techniques by which to learn the vocabulary. But this quantization is actually what then enables you to treat essentially to treat it as a sequence of language tokens, which also come from a vocabulary. All right. Back to Janek in the past. The image tokenizer is trained as an as it says here as a VQGAN, which means that you encode and then you decode again and you try to get out the same image. And at the end, this representation here in the middle is really valuable because it's a tokenized representation of an image. So you put that into the transformer right here. And this is, as we said, an autoregressive model. So it gets as an input, obviously, the sequence so far, it tries to predict the next image token, but also gets as an input, the text. So this is the prompt that the user puts in. So the prompt is encoded in a transformer encoder and is then fed in as a side input as a target for attention. So whenever in the layer here you have queries, keys and values, I'm going to guess the query can also look at the transformer encoder. The query can also look at the keys right here. So over here, you'd only have keys and values. If you don't know what the attend, what this all of this means, I have a video on attention is all you need where you can learn how attention mechanisms work. So essentially, the way this is trained is the following. You attach a sentence here or a description of an image and you attach an image right here. The image is then patched. It is fed through the VQGAN encoder. Its latent representation is obtained. That latent representation is put here. And then you essentially train a decoder language model that has cross attention into the text representation of the prompt. So you simply train this thing right here like you would train a GPT model or any other model. And this thing right here is trained, as I said, as an imagery construction model. And this thing right here is trained, I guess, jointly with this. Actually don't know. This could this could not be true, but I think it is true. I think it is trained jointly. So that's the model, as I said, is very basic. I wish I could tell you something more interesting right here, but I can't. It's a standard, you know, bunch of transformers in sequence. Essentially, every single component right here is a transformer. And because every single thing is a transformer, you can scale this thing by a lot. By the way, here you can see a bunch of the I'm not going to go into the architectural details. Quite quite as much. But they do also train an up sampler. So they have images of resolution 256 by 256. Ultimately, they do train an up sampler as well, where so here this is the up sampler super resolution up sampler where they can go from their pipeline, which does 256 by 256 to a 1024 by 1024. Picture essentially. But this is just up sampling. Right. So there is, I mean, technically no extra information right here. This doesn't get to look at the prompt or anything like this. It simply gets to look at this image and then make a four times larger image out of that. So where did we leave off? Oh, yeah, I also wanted to say if you now want to get an image out of this thing, so not training, but inference. What you do is you attach only the prompt right here. You encode the prompt, you put the start of sentence token right here. You let the model generate one. Then you put that here, too. Then you put that here, three and so on. You let the model generate the image tokens here. You take those image tokens, you feed, you arrange it into the latent representation of the VQ again. And you use the decoder right here in order to generate the final image. So that's the whole flow. And then you put it through the super resolution if you want that. Here you can see the basics, the basic architectural layouts. So there is the smallest model has 350 million parameter. You can see it has 12 encoder and 12 decoder layer. It's pretty standard transformer scaling laws right here. I mean, scaling laws, pretty standard transformer architectural laws. They go through a 750 million parameter model, 3 billion. And the last one here has 20 billion parameters. So that's a decently sized model. It's not as large as the large language models. And they do use things like sparse con attention and things like this. But it is, you know, it's pretty large, I would say. You could not run that at home very easily. So where does that get us? They have a big description right here how they solve this architecturally, how they short the model, how they use parallelism, which is very interesting. I'm just not an expert at it. So if you're interested, I'll leave you to read this part. I found the at least the drawings are pretty cool. So apparently this the signal is routed like, you know, like so, like so and so. So like in like a snake type of arrangement so that always you can pipeline so that always one thing is essentially busy as you send data to the next thing and so on. But as I said, I'm not the expert in this and I'd rather want to get to the other things, which are the data sets that they use. So they have three data sets, three main data sets right here. One is Emma's Coco. Now, Emma's Coco, as they show right here for the image on the right hand side, it simply says a bowl of broccoli and apples with a utensil. So it just kind of is a high level description of what's in the image. Like an image, simple image caption right for this image right here. Whereas the localized narratives data set, you can see that its description is way longer. It's more linguistically prosaic, but it is also much more descriptive of the actual image. Like so the top is if you want to tell someone what's in an image and the bottom is more like if you want to like really paint the picture, like no pun intended. Or if you want to describe the picture to someone so that they could maybe recreate it in some way. And it turns out that we are now at the point with these image generation models where they are so good that we need data sets like the bottom one to really push them to their limits. And not only that, but the authors here find that there are even problems with that because these image data sets, they're always created in a way that an image is given and then the humans are asked to write a description, which is really good because then you have image and description together. However, the authors here know that this prevents, for example, fantasy pictures like we saw before the raccoon and cubism that it doesn't exist. So it can't be in any data set or anubis in a leather jacket doesn't exist. So it can't be in any data set. So while we rely on generalization during training for the model to learn these things, we actually need data sets like that to evaluate whether they can really do these things. Right. Otherwise, we're left with sort of subjective evaluation. So they come up with their own data set, which is called party prompts. That's actually also the thing they release as far as I understand. And obviously, as all of the recent works in big models, this thing isn't released. There's no code. There's no I mean, the code would be trivial. There's no weights. There's no training recipe. There's no some of the data sets are proprietary, if I understand correctly. So the paper is more open about what they do, but still that there is no way of accessing this. So party prompts. This is a data set that essentially only consists of prompts. So there's no images in this data set. And I believe the only way you can really assess thing is you can let the model generate stuff and then you can let humans rate it. That's essentially it. The party prompts. It is pretty interesting because they create these prompts by letting the prompt engineers sort of they choose, for example, a challenge. So the challenge might be perspective. Right. Which could be, you know, I need a prompt that asks for some object in some in some specific perspective that is unusual. Or, yeah, quantity. Like I need a prompt that a that asks for a given number of things because we know that these models, they're not super good at counting. Right. I mean, we also thought the models aren't super good at spelling. And now it turns out, well, if we just make them bigger, they are. So, you know, I'm fairly confident they're going to be good at counting in short while. That's the challenge. There's also, if I recall correctly, this is this upper table right here, like categories. So there are categories, animals, there are categories, illustrations and so on. So you can see this is a diverse set of category challenge combinations and they make a bunch of prompts for each one. I think they have about 1600 prompts in total in this party prompt eval set, which is a pretty neat thing to have. Even if it comes without images. So now they train the thing with their whole architectural shebangs with the parallelism and the pipelining and the yada, yada, yada on TPU v4, I think. So this is a huge operation. So what does that give us? I want to just jump the evals here on the metrics because yes, yes, yes, they're very good, very good. They're also very good as rated by humans, humans, very good, which is what's interesting is they have, for example, a retrieval baseline, which simply retrieves images from the training data set. And even if the if the obviously image text match, the party model wins because you can actually create an image and not retrieve one. But even in image realism, you can see the retrieval is only slightly higher in realism, right? Every single image is real that the retrieval retrieves. And still the humans rate the realism of party almost the same, which is quite speaking for the model. The loss curves are also pretty interesting, especially interesting that the 20 billion model here, it takes quite a time to come down here. Right. It kind of has to get surpassed by the three billion model initially and then overtakes it, which maybe means that we haven't exactly found the right training recipes yet for these largest of models. So this now is the cool part where they put the model, the models next to one another. So this is the same prompt with all of these different models. And you can just see where scale gets you. This is a portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House, holding a sign on the chest that says welcome friends. And you can see my this these these things right here, this and this there may be like Dolly Mini kind of style pictures. And there are also that scale. All right. And then we go to the three B model. And this is something that would be familiar maybe from something like Dolly or Dolly, maybe between Dolly and Dolly to write these things. You can see they're bad at spelling. But as soon as you go bigger, all of a sudden, welcome friends. But a boom. There it is. Not bad at spelling anymore. All you need to scale. That's crazy. The sign very deep learning. Look, as the model learns to spell, initially, it can only do Russian or whatever. And and just eventually it would actually be funny if that was like actual Russian and it said very deep learning. Can you imagine how crazy that would be? In any case, and also the Grand Canyon, right? So there's kind of structure here and so on. But this very, very deep learning. Perfect. A blue Porsche parked in front of a yellow brick wall. You can see it doesn't always work. But it works better and better and better with scale. Crazy. And here this is like maybe like is this a direct shot at Gary Marcus? Because the challenge is like an an astronaut riding a horse. So astronaut riding a horse in the forest, even the three billion model. Oh, no, it's going to be a horse riding an astronaut, which is going to come up later. And I promise it's going to be funny. But yeah, an astronaut riding a horse in the water in front of them, water lilies and so on. A map of the United States made out of sushi. So as you can see, these these results are fairly insane. Infinity, the back of a violin, four cats surrounding a dog. So now they're really testing these individual categories. Infinity is an abstract concept. Back of violin is perspective. Four cats surrounding a dog is this quantity metric. You can you can see there are four cats, right? So, yeah, I'm pretty confident that with with scale, these types of problems are going to be solved. Scroll gives an apple to a bird. Yeah, so what's interesting is they have this narrative of what they call growing a cherry tree. So obviously, these samples here are cherry picked, which means that they take out whatever they think are good samples to present in the paper. However, they detail fairly extensively how they arrive at this thing. So what they do is they don't just come up with these long prompts by themselves. Well, these aren't long. OK. But, you know, these long prompts with Anubis in front in a leather jacket in front of Los Angeles skyline, they don't just come up with them on the spot. They have a process of coming up with them and the process is detailed here. So, for example, they have this idea of combining like a sloth with a van. Right. So they start by just exploring the model and entering things like a smiling sloth, like what comes out. Right. And a van parked on grass. There are always good images and bad images that turn out and they sort of learn how to have to tweak the prompt to get what they want. Once they're happy, they go on. So they modify the prompt a bit. So there is the smiling sloth wearing a leather jacket, a cowboy hat and a kilt or wearing a bow tie and holding a quarterstaff. So they kind of explore, they go more and more, as you can see, as you go down this tree, this cherry tree, as they call it, they go down and down. They detail well. Sometimes there's problems. This one, I believe, has two arms on this side and so on. So, but still they refine and refine and refine. They finally try to combine them. Right. Yeah. Here is a combination. They refine again. They try to combine the two prompts again. And at the end, they get to something that they might be happy with. For example, the thing here on the left, like this one right here. But I found this pretty interesting, like this process of arriving at these things. So you can't just enter any old long sentence and expect the model to do well. But what turns, what might, what will work often better, at least as they describe it, is to go through this process right here, which also means that full artistic freedom is a bit away. So it is almost like, yes, you are guiding the model with your inputs, but also the model is kind of guiding you by what it does well and what it doesn't do well if you go via this process. And if you don't go via this process, then I guess you can expect that you can expect that it might not work as well. So they also have some big failure cases, which is pretty cool. For example, the failure cases like color bleeding, where you describe the color of one of the things in the image and sort of the other take on that color. There's also counting failures and so on, localization failures. For example, here the prompt is, the prompt is, Oh yeah, the Great Pyramid of Giza situated in front of Mount Everest. That's the bottom two pictures should be that. You can see this. Okay, I mean, this isn't, this isn't too bad. But this here is just like the pyramid with sort of a Mount Everest cover. Right. You can see these models, they sometimes if they can't fulfill the prompt directly, they'll kind of mix. They'll just try to get it done somehow and get it really close in text embedding space. That's exactly what you can see right here. There's a bunch, a bunch of examples. And this one, I told you, it's the horse riding on an astronaut. So they have to actually specify the horse is sitting on an astronaut because the riding is just, is just riding indicates too much that the horse is on the bottom. But I just found the horse riding on the astronaut to be absolutely hilarious, especially this one. Yeah, but all in all, I guess what I wanted to say is that this is complaining on a very, very high level. Right. The paper itself is like moving the goal posts already by sort of criticizing itself for, oh, well, I specified like nine apples in a perfect arrangement. I don't have or write 10 red apples and it's only eight red apples. Like what a loser model. Look at that. I mean, this is it is crazy good how these models are and the failure cases here are, you know, yes, they're failure cases. But I don't think that if you told me three, four years ago that this is the type of error that we're at solving that I would have said, yeah, I believe that I would have way guessed. We're still at the point where, you know, we we have mode collapses. We can't create most of the text stuff. We have artifacts and all kinds of things. And I think this is yeah, it's it's kind of mind blowing how fast the progress here is. Obviously, half a year ago or so. Yeah, I would have expected something like this, but I believe, yeah, a lot of people must be very surprised and including me. Yeah, like spelling mistakes, like complaining that, you know, sometimes text is still not spelled right. Like, you know, even though, right, Dali couldn't do it at all. And now this thing is doing it almost perfectly, as you can see right here, combining abstract concepts. Look at the thing on top. It's it's insane. Or here like, oh, this leg is in the behind the race car. Come on. This is better than I guess anyone had expected. So, yeah, I don't want to waste your time too much more. I just thought this was absolutely cool. And I'm very excited to see where this is going next. Of course, huge bummer that we don't get access to this. I hope this finds its way into some products that we can use. As you know, I'm all for these companies making making money with their inventions. I mean, I think it's cool that they are inventing and, you know, if they want to make some cash off of it, you know, good for them. But I do hope that we actually get to use it. And I it's going to be a fun future where for every presentation or anything, if you need like an illustration, you just you just type it. You don't go to the Internet to search an appropriate stock photo. You just type it. It's so cool. Or you want to change something in a picture. You just erase it. You just say, whatever here, change that part to something else. So cool. No Photoshop skills anymore. No drawing skills anymore. Just you and your mind and your creativity. All right. That was it. As I said, the paper presented in this new system is fairly simple. All it does is scale a bunch of transformers in sequence. Essentially, I presented a evaluation benchmark, these party prompts, and it presented their model, which is ridiculously insane. That was it for me. Let me know what you think and I'll see you around. Bye bye.
 Not a day goes by in AI research in which we don't get a new image generation model these days.
 So take a look at the top row right here and listen to the prompt that generated them.
 Oil on canvas painting of a blue night sky with roiling energy.
 A fuzzy and bright yellow crescent moon shining at the top.
 Below the exploding yellow stars and radiating swirls of blue, a distant village sits quietly on the right.
 Connecting earth and sky is a flame-like cypress tree with curling and swaying branches on the left.
 A church spire rises as a beacon over rolling blue hills.
 That is a 67 word description of Starry Night by Vincent van Gogh.
 And it is also the prompt that generated the top row of images.
 And the paper does this to show that image generation models, specifically this one, they have become super duper capable of incorporating not only wild concepts,
 as you can see here, co-locating the Eiffel Tower with the Sydney skyline and fireworks and whatnot,
 but also, you know, minute details about things in the image and where things are and how things look.
 So we've gone from essentially conditional GANs where we could create one of 10 classes to something where we can input like a little essay about what we want to see and get it out.
 So this is by a group of researchers out of Google Research.
 And they are a parallel work to the Imogen model that you might have seen.
 So this model or the paper is called Scaling Autoregressive Models for Content-Rich Text to Image Generation.
 But the model is called, let me grab if I can, let me grab a pen.
 The model is called PARTI.
 And I have no clue how to pronounce this.
 This could be party.
 Maybe the pronunciation is on the art or on the part because it's pathways like it's, or partai or I have no idea.
 Let's call it PARTI.
 And PARTI is a model that generates images from text as we have so many models.
 However, it doesn't do this in the same style as like Imogen, which is a diffusion model.
 It is an autoregressive model.
 So here you can see a bunch of other outputs like this.
 This is insane.
 Look at the left side right here.
 A photo of a frog reading the newspaper named Toaday.
 The newspaper is named Toaday.
 Like how crazy is that?
 That in itself is pretty funny.
 But we know that these image to sorry, these text to image models are pretty bad at spelling stuff in images.
 Well, not this model, as you can see right here, it gets it completely right.
 It doesn't always get it right, but it gets it right often enough.
 Or this one, portrait of a statue of the Egyptian god Anubis wearing aviator goggles like another connoisseur of fine eyewear.
 White t-shirt and a leather jacket.
 The city of Los Angeles is in the background.
 High res DSLR photograph.
 That's literally that's the academic version of the Unreal Engine trick right here.
 And you can see the images spot on.
 So this requires a lot of knowledge, not only of, you know, what a DSLR photograph is, but also how the skyline of Los Angeles looks, how the Egyptian got an Anubis looks right.
 And the composition of things together like these, this god was never in a leather jacket depicted, I guess, maybe on the internet you'll find anything.
 But you can see a bunch of more examples right here.
 I specifically love the thing on the left side here.
 You can see that they generated images.
 So the prompt is three quarters front view of a XYZ coming around a curve in a mountain road looking over a green valley on a cloudy day.
 So X here is any of the colors blue, red and yellow.
 Y is any of the numbers.
 1977, 1997 and 2017.
 And Z is any of these car types.
 And now look that the model can essentially track the historical evolution of these cars.
 So not only does it know what a Porsche is, it also knows how a Porsche in 77 looked like.
 Maybe it's not exactly the correct year, but this is pretty crazy.
 You can see a bunch more examples right here.
 They do a lot of examples with animals.
 I specifically like the raccoon here in the style of cubism.
 So this is going to be very, very powerful technology.
 We can immediately see that, you know, the quality of these models gets fast, gets quickly, sorry, gets well, gets better so quickly that in the foreseeable future,
 we're going to have super powerful tools to just create and edit images from text.
 Look at the left side here, a giant cobra snake made from salad.
 You know, I'm sure they even say these are cherry picked, but still this is insane.
 Now, I would love to tell you that behind all of this cool development is a really cool idea like is a smart architecture and something like this.
 But I'm afraid it is not.
 It is simply scale and not simply scale.
 I mean, you have to have the sort of correct base architecture.
 There is nothing like particularly there's no cool invention in architecture or a neat trick involved or anything like this.
 It's really just plug basic things together, make them really big, train them for long on a lot of data and you'll get quality.
 So this is the model overview right here, the overview of this party or part time model.
 This is, as I already said, in contrast to image and it is an auto regressive model, so not a diffusion model.
 What happens is that on this side here, you have this VQ GAN image encoder and decoder.
 Well, they don't call them encoder and decoder, they call them tokenizer and de tokenizer.
 So if you are not aware, auto regressive models, they work on tokens.
 Now, tokens in usually in natural language processing are words or part of words.
 So these would be tokens, token one, token two, and so on until token N.
 And then what you would try to do is you would try always to predict the next token.
 That's what makes it auto regressive.
 You feed in parts of a token sequence, like parts of a sentence, you try to predict the next one.
 That's exactly what you see right here in the architecture.
 So you pass in the start of sentence token, you try to predict the first token, then you pass in the first token.
 And then from these two, you try to predict the second token.
 And then you put that here from these three, you try to predict the third token and so on.
 That's the auto regressivity.
 In text, that works well. However, in images, it's not quite obvious how to do that.
 That's why you first need to get from the image space to the token space.
 So we need a way for any given image that we get out a sequence of tokens.
 And it can't be the pixels themselves.
 We would like to have tokens that are kind of latent and have sort of a bit of meaning, not just individual pixels,
 because that, first of all, is too many pixels.
 And second of all, there's not too much, let's say, information in the single pixel.
 So what we do is we have these image tokenizer and detokenizer.
 This is a VQGAN that is powered by a vision transformer.
 So essentially, this is a model that takes this image, it ships it through a bunch of layers.
 And at the end, so let's say the image at the beginning has a bunch of rows, a bunch of columns with its pixels.
 This goes through a series of maybe downscalings and so on.
 No, actually, it's because it's a vision transformer.
 It probably even tokenizes, like it patches the image at the very beginning.
 So these would be image patches.
 Then these are transformed by a transformer to a latent space.
 Maybe they are compressed.
 And then you get tokens.
 So at the end, you can take these things right here or the things that correspond to them in the latent representation.
 You can take those as image tokens and you can unroll essentially this image and then feed it into this model.
 Hey, just a short interjection here from Janek from the future.
 The idea, I forgot, the idea behind the whole setup here is behind the whole VQGAN is obviously that these things here are tokens,
 which means that they come from a set vocabulary.
 So the way you train a VQGAN isn't just to give you this latent representation of like token like things, but then you also quantize them.
 So there is also a vocabulary somewhere where you have a set defined set of tokens.
 I believe in their case, they have like 8,000 tokens or so.
 And your image, your image tokens must be of these 8,000.
 So the image has a bunch of tokens, but they all must be one of the things in the vocabulary here.
 Now, the vocabulary is also learned.
 There are some techniques by which to learn the vocabulary.
 But this quantization is actually what then enables you to treat essentially to treat it as a sequence of language tokens, which also come from a vocabulary.
 All right.
 Back to Janek in the past.
 The image tokenizer is trained as an as it says here as a VQGAN, which means that you encode and then you decode again and you try to get out the same image.
 And at the end, this representation here in the middle is really valuable because it's a tokenized representation of an image.
 So you put that into the transformer right here.
 And this is, as we said, an autoregressive model.
 So it gets as an input, obviously, the sequence so far, it tries to predict the next image token, but also gets as an input, the text.
 So this is the prompt that the user puts in.
 So the prompt is encoded in a transformer encoder and is then fed in as a side input as a target for attention.
 So whenever in the layer here you have queries, keys and values, I'm going to guess the query can also look at the transformer encoder.
 The query can also look at the keys right here.
 So over here, you'd only have keys and values.
 If you don't know what the attend, what this all of this means, I have a video on attention is all you need where you can learn how attention mechanisms work.
 So essentially, the way this is trained is the following.
 You attach a sentence here or a description of an image and you attach an image right here.
 The image is then patched.
 It is fed through the VQGAN encoder.
 Its latent representation is obtained.
 That latent representation is put here.
 And then you essentially train a decoder language model that has cross attention into the text representation of the prompt.
 So you simply train this thing right here like you would train a GPT model or any other model.
 And this thing right here is trained, as I said, as an imagery construction model.
 And this thing right here is trained, I guess, jointly with this.
 Actually don't know.
 This could this could not be true, but I think it is true.
 I think it is trained jointly.
 So that's the model, as I said, is very basic.
 I wish I could tell you something more interesting right here, but I can't.
 It's a standard, you know, bunch of transformers in sequence.
 Essentially, every single component right here is a transformer.
 And because every single thing is a transformer, you can scale this thing by a lot.
 By the way, here you can see a bunch of the I'm not going to go into the architectural details.
 Quite quite as much.
 But they do also train an up sampler.
 So they have images of resolution 256 by 256.
 Ultimately, they do train an up sampler as well, where so here this is the up sampler super resolution up sampler where they can go from their pipeline, which does 256 by 256 to a 1024 by 1024.
 Picture essentially.
 But this is just up sampling.
 Right.
 So there is, I mean, technically no extra information right here.
 This doesn't get to look at the prompt or anything like this.
 It simply gets to look at this image and then make a four times larger image out of that.
 So where did we leave off?
 Oh, yeah, I also wanted to say if you now want to get an image out of this thing, so not training, but inference.
 What you do is you attach only the prompt right here.
 You encode the prompt, you put the start of sentence token right here.
 You let the model generate one.
 Then you put that here, too.
 Then you put that here, three and so on.
 You let the model generate the image tokens here.
 You take those image tokens, you feed, you arrange it into the latent representation of the VQ again.
 And you use the decoder right here in order to generate the final image.
 So that's the whole flow.
 And then you put it through the super resolution if you want that.
 Here you can see the basics, the basic architectural layouts.
 So there is the smallest model has 350 million parameter.
 You can see it has 12 encoder and 12 decoder layer.
 It's pretty standard transformer scaling laws right here.
 I mean, scaling laws, pretty standard transformer architectural laws.
 They go through a 750 million parameter model, 3 billion.
 And the last one here has 20 billion parameters.
 So that's a decently sized model.
 It's not as large as the large language models.
 And they do use things like sparse con attention and things like this.
 But it is, you know, it's pretty large, I would say.
 You could not run that at home very easily.
 So where does that get us?
 They have a big description right here how they solve this architecturally, how they short the model, how they use parallelism, which is very interesting.
 I'm just not an expert at it.
 So if you're interested, I'll leave you to read this part.
 I found the at least the drawings are pretty cool.
 So apparently this the signal is routed like, you know, like so, like so and so.
 So like in like a snake type of arrangement so that always you can pipeline so that always one thing is essentially busy as you send data to the next thing and so on.
 But as I said, I'm not the expert in this and I'd rather want to get to the other things, which are the data sets that they use.
 So they have three data sets, three main data sets right here.
 One is Emma's Coco.
 Now, Emma's Coco, as they show right here for the image on the right hand side, it simply says a bowl of broccoli and apples with a utensil.
 So it just kind of is a high level description of what's in the image.
 Like an image, simple image caption right for this image right here.
 Whereas the localized narratives data set, you can see that its description is way longer.
 It's more linguistically prosaic, but it is also much more descriptive of the actual image.
 Like so the top is if you want to tell someone what's in an image and the bottom is more like if you want to like really paint the picture, like no pun intended.
 Or if you want to describe the picture to someone so that they could maybe recreate it in some way.
 And it turns out that we are now at the point with these image generation models where they are so good that we need data sets like the bottom one to really push them to their limits.
 And not only that, but the authors here find that there are even problems with that because these image data sets, they're always created in a way that an image is given and then the humans are asked to write a description,
 which is really good because then you have image and description together.
 However, the authors here know that this prevents, for example, fantasy pictures like we saw before the raccoon and cubism that it doesn't exist.
 So it can't be in any data set or anubis in a leather jacket doesn't exist.
 So it can't be in any data set.
 So while we rely on generalization during training for the model to learn these things, we actually need data sets like that to evaluate whether they can really do these things.
 Right. Otherwise, we're left with sort of subjective evaluation.
 So they come up with their own data set, which is called party prompts.
 That's actually also the thing they release as far as I understand. And obviously, as all of the recent works in big models, this thing isn't released.
 There's no code. There's no I mean, the code would be trivial. There's no weights.
 There's no training recipe.
 There's no some of the data sets are proprietary, if I understand correctly.
 So the paper is more open about what they do, but still that there is no way of accessing this.
 So party prompts. This is a data set that essentially only consists of prompts.
 So there's no images in this data set.
 And I believe the only way you can really assess thing is you can let the model generate stuff and then you can let humans rate it.
 That's essentially it.
 The party prompts. It is pretty interesting because they create these prompts by letting the prompt engineers sort of they choose, for example, a challenge.
 So the challenge might be perspective. Right.
 Which could be, you know, I need a prompt that asks for some object in some in some specific perspective that is unusual.
 Or, yeah, quantity.
 Like I need a prompt that a that asks for a given number of things because we know that these models, they're not super good at counting.
 Right. I mean, we also thought the models aren't super good at spelling.
 And now it turns out, well, if we just make them bigger, they are.
 So, you know, I'm fairly confident they're going to be good at counting in short while.
 That's the challenge.
 There's also, if I recall correctly, this is this upper table right here, like categories.
 So there are categories, animals, there are categories, illustrations and so on.
 So you can see this is a diverse set of category challenge combinations and they make a bunch of prompts for each one.
 I think they have about 1600 prompts in total in this party prompt eval set, which is a pretty neat thing to have.
 Even if it comes without images.
 So now they train the thing with their whole architectural shebangs with the parallelism and the pipelining and the yada, yada, yada on TPU v4, I think.
 So this is a huge operation. So what does that give us?
 I want to just jump the evals here on the metrics because yes, yes, yes, they're very good, very good.
 They're also very good as rated by humans, humans, very good, which is what's interesting is they have, for example, a retrieval baseline, which simply retrieves images from the training data set.
 And even if the if the obviously image text match, the party model wins because you can actually create an image and not retrieve one.
 But even in image realism, you can see the retrieval is only slightly higher in realism, right?
 Every single image is real that the retrieval retrieves.
 And still the humans rate the realism of party almost the same, which is quite speaking for the model.
 The loss curves are also pretty interesting, especially interesting that the 20 billion model here, it takes quite a time to come down here.
 Right. It kind of has to get surpassed by the three billion model initially and then overtakes it, which maybe means that we haven't exactly found the right training recipes yet for these largest of models.
 So this now is the cool part where they put the model, the models next to one another.
 So this is the same prompt with all of these different models.
 And you can just see where scale gets you.
 This is a portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House, holding a sign on the chest that says welcome friends.
 And you can see my this these these things right here, this and this there may be like Dolly Mini kind of style pictures.
 And there are also that scale.
 All right. And then we go to the three B model.
 And this is something that would be familiar maybe from something like Dolly or Dolly, maybe between Dolly and Dolly to write these things.
 You can see they're bad at spelling. But as soon as you go bigger, all of a sudden, welcome friends.
 But a boom. There it is. Not bad at spelling anymore.
 All you need to scale. That's crazy. The sign very deep learning.
 Look, as the model learns to spell, initially, it can only do Russian or whatever.
 And and just eventually it would actually be funny if that was like actual Russian and it said very deep learning.
 Can you imagine how crazy that would be?
 In any case, and also the Grand Canyon, right?
 So there's kind of structure here and so on. But this very, very deep learning.
 Perfect.
 A blue Porsche parked in front of a yellow brick wall. You can see it doesn't always work.
 But it works better and better and better with scale.
 Crazy. And here this is like maybe like is this a direct shot at Gary Marcus?
 Because the challenge is like an an astronaut riding a horse.
 So astronaut riding a horse in the forest, even the three billion model.
 Oh, no, it's going to be a horse riding an astronaut, which is going to come up later.
 And I promise it's going to be funny.
 But yeah, an astronaut riding a horse in the water in front of them, water lilies and so on.
 A map of the United States made out of sushi.
 So as you can see, these these results are fairly insane. Infinity, the back of a violin, four cats surrounding a dog.
 So now they're really testing these individual categories. Infinity is an abstract concept.
 Back of violin is perspective. Four cats surrounding a dog is this quantity metric.
 You can you can see there are four cats, right?
 So, yeah, I'm pretty confident that with with scale, these types of problems are going to be solved.
 Scroll gives an apple to a bird.
 Yeah, so what's interesting is they have this narrative of what they call growing a cherry tree.
 So obviously, these samples here are cherry picked, which means that they take out whatever they think are good samples to present in the paper.
 However, they detail fairly extensively how they arrive at this thing.
 So what they do is they don't just come up with these long prompts by themselves.
 Well, these aren't long. OK.
 But, you know, these long prompts with Anubis in front in a leather jacket in front of Los Angeles skyline, they don't just come up with them on the spot.
 They have a process of coming up with them and the process is detailed here.
 So, for example, they have this idea of combining like a sloth with a van.
 Right. So they start by just exploring the model and entering things like a smiling sloth, like what comes out.
 Right. And a van parked on grass.
 There are always good images and bad images that turn out and they sort of learn how to have to tweak the prompt to get what they want.
 Once they're happy, they go on.
 So they modify the prompt a bit.
 So there is the smiling sloth wearing a leather jacket, a cowboy hat and a kilt or wearing a bow tie and holding a quarterstaff.
 So they kind of explore, they go more and more, as you can see, as you go down this tree, this cherry tree, as they call it, they go down and down.
 They detail well.
 Sometimes there's problems.
 This one, I believe, has two arms on this side and so on.
 So, but still they refine and refine and refine.
 They finally try to combine them.
 Right. Yeah.
 Here is a combination.
 They refine again.
 They try to combine the two prompts again.
 And at the end, they get to something that they might be happy with.
 For example, the thing here on the left, like this one right here.
 But I found this pretty interesting, like this process of arriving at these things.
 So you can't just enter any old long sentence and expect the model to do well. But what turns, what might, what will work often better, at least as they describe it, is to go through this process right here, which also means that full artistic freedom is a bit away.
 So it is almost like, yes, you are guiding the model with your inputs, but also the model is kind of guiding you by what it does well and what it doesn't do well if you go via this process.
 And if you don't go via this process, then I guess you can expect that you can expect that it might not work as well.
 So they also have some big failure cases, which is pretty cool. For example, the failure cases like color bleeding, where you describe the color of one of the things in the image and sort of the other take on that color.
 There's also counting failures and so on, localization failures.
 For example, here the prompt is, the prompt is,
 Oh yeah, the Great Pyramid of Giza situated in front of Mount Everest. That's the bottom two pictures should be that. You can see this. Okay, I mean, this isn't, this isn't too bad.
 But this here is just like the pyramid with sort of a Mount Everest cover. Right. You can see these models, they sometimes if they can't fulfill the prompt directly, they'll kind of mix.
 They'll just try to get it done somehow and get it really close in text embedding space. That's exactly what you can see right here.
 There's a bunch, a bunch of examples. And this one, I told you, it's the horse riding on an astronaut.
 So they have to actually specify the horse is sitting on an astronaut because the riding is just, is just riding indicates too much that the horse is on the bottom.
 But I just found the horse riding on the astronaut to be absolutely hilarious, especially this one.
 Yeah, but all in all, I guess what I wanted to say is that this is complaining on a very, very high level. Right.
 The paper itself is like moving the goal posts already by sort of criticizing itself for, oh, well, I specified like nine apples in a perfect arrangement.
 I don't have or write 10 red apples and it's only eight red apples. Like what a loser model. Look at that.
 I mean, this is it is crazy good how these models are and the failure cases here are, you know, yes, they're failure cases.
 But I don't think that if you told me three, four years ago that this is the type of error that we're at solving that I would have said, yeah, I believe that I would have way guessed.
 We're still at the point where, you know, we we have mode collapses. We can't create most of the text stuff.
 We have artifacts and all kinds of things. And I think this is yeah, it's it's kind of mind blowing how fast the progress here is.
 Obviously, half a year ago or so. Yeah, I would have expected something like this, but I believe, yeah, a lot of people must be very surprised and including me.
 Yeah, like spelling mistakes, like complaining that, you know, sometimes text is still not spelled right.
 Like, you know, even though, right, Dali couldn't do it at all. And now this thing is doing it almost perfectly, as you can see right here, combining abstract concepts.
 Look at the thing on top. It's it's insane. Or here like, oh, this leg is in the behind the race car.
 Come on. This is better than I guess anyone had expected.
 So, yeah, I don't want to waste your time too much more. I just thought this was absolutely cool.
 And I'm very excited to see where this is going next. Of course, huge bummer that we don't get access to this.
 I hope this finds its way into some products that we can use. As you know, I'm all for these companies making making money with their inventions.
 I mean, I think it's cool that they are inventing and, you know, if they want to make some cash off of it, you know, good for them.
 But I do hope that we actually get to use it. And I it's going to be a fun future where for every presentation or anything, if you need like an illustration, you just you just type it.
 You don't go to the Internet to search an appropriate stock photo. You just type it. It's so cool. Or you want to change something in a picture.
 You just erase it. You just say, whatever here, change that part to something else. So cool.
 No Photoshop skills anymore. No drawing skills anymore. Just you and your mind and your creativity.
 All right. That was it. As I said, the paper presented in this new system is fairly simple. All it does is scale a bunch of transformers in sequence.
 Essentially, I presented a evaluation benchmark, these party prompts, and it presented their model, which is ridiculously insane.
 That was it for me. Let me know what you think and I'll see you around. Bye bye.