Image GPT: Generative Pretraining from Pixels (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper openai gpt2 gpt3 bert transformer attention is all you need attention mechanism multi-head attention pixel rnn pixel cnn pretraining representation linear probe fine-tuning cifar10 cifar100 imagenet cnn convolutional neural network autoregressive
BERT and GPT-2/3 have shown the enormous power of using generative models as pre-training for classification tasks. However, for images, pre-training is usually done with supervised or self-supervised objectives. This paper investigates how far you can get when applying the principles from the world of NLP to the world of images.

OUTLINE:
0:00 - Intro & Overview
2:50 - Generative Models for Pretraining
4:50 - Pretraining for Visual Tasks
7:40 - Model Architecture
15:15 - Linear Probe Experiments
24:15 - Fine-Tuning Experiments
30:25 - Conclusion & Comments

Paper:
https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf
Blog: https://openai.com/blog/image-gpt/
Code: https://github.com/openai/image-gpt

Abstract:
Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models can learn useful representations for images. We train a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels, we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99.0% accuracy with full finetuning, matching the top supervised pre-trained models. An even larger model trained on a mixture of ImageNet and web images is competitive with self-supervised benchmarks on ImageNet, achieving 72.0% top-1 accuracy on a linear probe of our features.

Authors: Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, Ilya Sutskever

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Okay, I'm sure many of you have already seen this because it was rather widely announced but the OpenAI team has announced a new model that produces pictures instead of text. So as you can see right here on the left you'll always see like a half a picture and on the right is the ground truth. So they took this picture, they simply cut the bottom half right here and then they let the model sort of imagine what they cut away and what it comes up with is pretty cool I have to say. Like look at the birds, like this is just awesome. But the special thing about this isn't that it simply completes pictures, the special thing about it is it does it one pixel by pixel. So basically it goes at this pixel right here and asks okay what's that pixel and then what's that pixel and so on. So it is basically like a language model but for pixels in that it goes over the images in order basically like this or like always from left to right, left to right, left to right and it has no clue of the spatial relations between the pixels. It needs to learn that by itself as opposed to a convolutional neural network which is specifically designed such that if you want to predict this pixel right here then it's specifically designed to say okay the most important information is probably around that pixel and then some like other important information is wider around that pixel. So CNNs are built with this in mind whereas this model right here which is also known as image GPT doesn't have any of that. It's simply a transformer model that goes over these pixels one by one and we'll see how that's done. There are some more examples right here. Particularly cool is the cat and you see that there is the beginning of this little white thing right here which is this card and the completions of the model. Yes very interesting. The model can of course as a language model can also sample by itself just random images. You sample them once through and this is what it comes up with. So these are pretty good quality images for a model that just produces one pixel by one pixel. Now this is a pixel. Now this idea of one pixel by pixel isn't new. This has been around before but the investigation here is basically how much can we how far can we push these generative models for pre-training. Hi there this is Janek from Post Production. I've realized that I've forgotten to even read the name of the paper. So it's called generative pre-training from pixels by Mark Chen, Alec Radford, Rowan Child, Jeff Wu, Hewon Ju, Prafula Dariwal, David Luan and Ilya Sotskyver. And since Henry AI Labs has already made a video on this, this video is going to be more of kind of a rumble rant about what I find interesting about the paper and some thoughts about it rather than like a classic explanation. I hope you still enjoy that. So what you saw on the right wasn't even the this isn't the final result the supposed result this is simply the pre-training task. It's fun to look at it but the actual object objective of the paper is the following. What if we train we pre-train on a large data set to generate what good images like these or we to complete images like these and then we fine-tune on a classification task. And the answer is here they say on C410 we achieve the 96.3% accuracy with a linear probe outperforming a supervised wide resonant and a 99% accuracy with full fine-tuning matching the top supervised pre-trained models. An even larger model trained on a mixture of ImageNet and web images is competitive with self supervised benchmarks on ImageNet achieving 72 top one accuracy on a linear probe of our features. So the goal here is that you have a data set that you want to train a classifier on. So usually you have a data set and the data set has images and you put them through like a convolutional neural network and then you have to classify the image into one of I don't know how many classes on C410 that's 10 classes on ImageNet it's a thousand. And the data set is these images together with these labels. Now the idea of pre-training is that you somewhere have a bigger data set that is sort of similar to the small data set but yeah it's similar enough such that the network could learn something. So what you want to do first is you first want to take the large data set, train this network right here and then in a second step fine-tune the network on this smaller data set and you sort of hope that what you learn from the large data set right here transfers over a little bit of knowledge. You already have a little bit of knowledge and you can make better use of the data that you have right here. Now the question is how do you do this pre-training and of course this has a long tradition, well long for maybe two or three years right now in the language community where people they pre-train these large models like we've just seen GPT-3 or BERT was one of them. They pre-train these large transformer models on text and then to fine-tune them on classification tasks for text and that's what this paper is doing right here. They pre-train a transformer that is a GPT-2 scale model, they pre-train it on image generation and then they fine-tune it or transfer learn it to classification tasks. And the point of the paper is to say that like in text data, in text data we have made pretty good experiences with doing this, with pre-training a generative model and then fine-tuning on a classification task. While so far in images all we've ever done is we've pre-trained these pre-training tasks which usually is a classification task or like a self-supervised task with a contrastive loss or something like this. What they're doing new is the generative modeling as a pre-training. And again this isn't entirely new but they show that if you throw a lot of computers at it and lots of data and a model then that can work equally well to these self-supervised tasks. So their model as I said is pretty pretty simple. They take an image and they unroll the image. Now a fully unrolled image on let's say ImageNet has 224 squared pixels and that times three right because you have three color channels. That's too large even for an open AI supercomputer. So what they do is first they downscale the image. So they downscale it's not as drastic as here where you just get a three by three image but they do downscale it to like a 32 by 32 or a 64 by 64. Then they unroll it which simply means they go through the image like this and make a sequence out of it because their models are naturally made for text sequences. They simply put the image into a text sequence. They further simplify this by reducing the three color channels to a single one. So they have their own color representation and basically yeah they reduce the three color channels to one channel that simply indexes the color in their color representation. And they say it's still pretty good. It's pretty faithful. So ultimately they end up with like a 32 squared length representation of their image. And then they do one of two things. They either do autoregressive generative pre-training which is the sort of GPT-2 style pre-training. And the idea here is that you always want to predict the next pixel of a sequence. So you can see right here that's the sequence that you input. And you always want to predict what is the next pixel. And in this case you see that we've already predicted everything. Here we've already predicted everything up to this red pixel. So you want to know what's this next pixel, this thing right here. What's this going to be? And the diagram here basically shows you how the attention flows. So every position in this transformer, and if you don't know what a transformer is, I haven't made a video about attention is all you need where these are explained. But briefly every position here can sort of send information only in one direction. So you train all of these in parallel. And when you predict this pixel right here, you only want information from whatever was before that pixel. Otherwise the model could cheat, right? Otherwise the model could simply learn to copy over the value. But the attention pattern here is simply to show you that this is autoregressive and it's in one direction. So you always want to predict the next pixel. And then from all of this you want to predict the next pixel. And then from all of this you want to predict the next pixel. This is in contrast to this objective here that comes from BERT. And I've also made a video on BERT. What you do in BERT is you simply take that image and you cross the block out two of the pixels, or many of the pixels, and you simply ask your network to reconstruct those pixels. And now you can see the attention flows in all directions. BERT, the B stands actually for bidirectional. So this is the contrast to the autoregressive pre-training framework. Now these two things have been applied in text both. The autoregressive is usually easier to actually make it produce something, like we saw producing these images, because you can always just predict the next pixel and then the next and then the next and then the next. Whereas in BERT it's a bit more unclear how you would produce things in a consistent manner. Because the predictions of these two pixels right here, they are independent. It's one forward pass and then both of these are predicted. But other papers have tried to solve this like this, not XLNet. I forget its name. It's something with an X. But these are the two objectives they look at. And it turns out they sort of trade off a bit. They work equally well, or a bit better and a bit worse, depending on the task. So once they have done this, they simply feed images. And you'll notice that you don't need any labels for this. So what you'll do is simply input an image and then simply take away half of it like this, and then predict that pixel. And then you want to predict that pixel and then you want to predict that pixel. That's all like you do with text. And in BERT you simply input an image, cross out pixels and then predict them. So you don't need labels for this. And that's why you can do it with this big data set. And you can do it in an unsupervised fashion. So you can just crawl the internet for images and just feed this into there. And it will sort of learn to produce these images. Now the question is, if you learn to produce these images, does that help you for classification? And they have two methods of assessing this. The bottom one here is the fine-tuning method. So this is supposed to be the representation you learn in the different layers of the network. So this is supposed to be this thing right here. What you'll do is you'll simply fine-tune. That means you on top of this representation you add a classification head that has two outputs, cat or dog, and you train this entire network on your small data set that we discussed before. So you train the entire network, all of the parameters. This is called fine-tuning. In contrast to that, what you can do is you can simply add this classification head with two outputs and then only train this classification head. And that won't perform as well, but it gives you sort of a better idea of how good is the representation that this network right here learned. And on top of that, so if you spin this idea further, you can actually go and do this at any intermediate layer right here. So you can forward propagate until layer two right here, and then here you add your classification head into the two classes and you only train the classification head. That being said, you can also do this with fine-tuning, but in this case this is called a linear probe. And it is often used to assess how good the A representation in intermediate layers is. Whereas what it actually does is assessing how linearly classifiable a representation is, which isn't the same as how useful or how informative, but it is one way to assess these things. So these are the two things they assess. So as for datasets, for C410 they use like C410 and C4100 as datasets and the STL10. And there you have to keep in mind the pre-training is done on ImageNet for those. So you pre-train on ImageNet without the labels and then you transfer learn or fine-tune or linear probe on these small datasets. Whereas later we're going to look at ImageNet and there the pre-training as I understand it is done on ImageNet itself, but also a wider collection of a hundred million or so images from the web, from the internet. Okay, so as you can see right here this is what happens if you do this linear probing. And you can see it works pretty well. So you get like a 95-96% accuracy with linear probes. This is very powerful. So it's not easy to get 96% on C410. I mean current state-of-the-art is like 99%, but still 96% is pretty good. And this is the entire network. There is this big giant network that you input your image into and then there is this one linear layer that does the classification. And all of this right here has not been trained with classification in mind. It simply has been trained to reproduce images. It hasn't even been trained on C410 as far as I understand. It's been trained on ImageNet. So this is to stress how cool or how significant this result is basically. That just a linear probe on top of that will give you such a good accuracy. And the second thing that is obvious right here is this bottom axis is the layer. So this is the layer where they attach the linear probe. And usually if you pre-train a network with a classification task in mind, so you pre-train it with the labels or maybe even without the labels in a self supervised way or something like this, usually the last layer has the best representation for classification. But here the special thing is that the intermediate layers in the middle have the best representation. You can see that the representation quality in terms of linear probing falls off as they sort of it falls off as they go into higher layers. And this is consistent across the datasets as you can see. And the idea here is or the way they interpret it is that if you have an image right here and you've blocked part of it, so you've blocked this and this, wrong way around this, so you've generated everything and now your task is to predict the next pixel. So you're trained to predict this next pixel right here. And the idea is that as you put the image through the network, what it will do is sort of, since the first layers they're going to be, if you're going to be similar to a CNN, they're going to be doing some low-level feature transformation thing. But also the last layers, they're going to really care about what's the exact pixel that goes here. Since it's their job to do that, they're going to care what color does it need to have, what exact luminosity and so on, how does it fit in with the previous pixels and so on. So that's also good. But it's not just low-level information and consistency with other pixels or something like this. At some point if you want to generate consistent images, and we saw that this model can generate consistent images, at some point there needs to be some kind of a notion of the global information in the picture, because the images are consistent throughout. So there needs to be some notion of what is in that image as a whole. And that's the exact information that we need for classification. And the only way that could actually be is here in the middle, since you know that's the place. So the hypothesis is that these models somehow learn a higher level of representation of global information somewhere in the middle before they then specify that information again down to predict the actual pixel. And that's why the best representations for classification are in the middle. So this is one of the interesting findings of this paper. I mean it's cool that they can reach a good accuracy, but to recognize that maybe in these generative models they have some intermediate stage where they represent the global information, and that will actually make the best representation. The second cool thing right here is that you can see they have different sizes of models. So the IGPT-L I believe is something like 60 layers, then this is like 48 layers, and this is 32 layers. So these are all on the scale of GPT-2, either a little bigger or a little smaller. It's not like a GPT-3 scale where you need a ginormous supercomputer, though they do do a lot of computation. But this still sort of fits within hardware of a standard size and not like exascale. What's interesting right here is that you can see the larger models, they reach a lower validation loss. So here is the validation loss. The larger model, if you train them on, so these checkpoints here are always after the same amount of steps. The larger models do reach a lower validation loss right here, as you can see. So this is the large, this is the medium, this is the small. And also you can see that on this axis the linear probe accuracy. So this is whenever you go and you find the best intermediate layer for linear probing, you probe it and you record the accuracy. So you can see a general trend as your validation loss goes down, the linear probe accuracy goes up. So there is a connection like it is in text models. In text models there's a connection of the perplexity of your language model and the quality of the representation you get for downstream tasks. In this model it seems to be the exact same thing. There is a connection between reaching lower validation loss and reaching a higher performance on classification. So that's one interesting thing, the general trend to up to the upper right corner. The other interesting and even arguably even more interesting thing is that if you look at the same validation loss. So at this point all of these models have the same validation loss, yet still the bigger model is better. You can see right here the bigger model outperforms the smaller model even though they have the same validation loss on the image modeling task. And this is also something that OpenAI in their text papers has stressed, that the larger models they seem to be somehow more capable of forming good representations even if they have the same loss. So again this could just be sort of a training data, better training data remembering thing. And when I said that in GPT-3 I didn't actually mean explicit remembering of training data. I meant kind of a fuzzy remembering of training data. I formulate that in the comments but I feel a lot of people have misunderstood me there. Here I think it's a much harder to estimate what's going on also since image pixels. Humans don't have a super good model on image pixels in their head as we have about text. As you can see if you then fine-tune, so for now we've just do linear probing, if you fine-tune these architectures then you reach like a 99% accuracy on C410 which is on par with the best models that we have. So G-Pipe is supervised, pre-trained on ImageNet but also I guess uses a bunch of data augmentation while these image GPT it uses minimal data augmentation I think. They simply random crop a little bit and that's about it. So they also experiment around with this BERT objective. So until now this was all this autoregressive objective and I feel that OpenAI people are a bit more of a fan of the autoregressive objective just given what they've done so far in their papers. And you can see here comparison of the two objectives on C410 and on ImageNet. Again C410 is pre-trained with ImageNet and ImageNet itself is pre-trained with like a larger collection of images from the web. All the pre-training is done without labels. Now the blue is what you can reach with a linear probe and the orange is then on top of that what you can reach by fine-tuning. So no linear probe but fine-tuning. I have to say that the fine-tuning is always done at the end. So even though the linear probe can be attached anywhere in between and it's often useful to do that as we saw because the in-between layers are the best. They say they tried fine-tuning also from in-between but it always worked out best whenever you fine-tune. Whenever you fine-tune you take actually the last layer. So that kind of gives you an idea that the model is then... What seems to be important is this coming up with the higher level representation and then once you fine-tune you're probably able to push that representation through to the end because of your training signal. But if you hadn't done the pre-training you wouldn't even have that higher level representation and then the signal I guess is not strong enough to back propagate through the whole model. It would be very interesting if they investigate, if they do this linear probe analysis again after they fine-tune the model. And to see if then still it is the intermediate layers that have the best representation or if now the best representation in a linear probe sense shifted towards the end. I'm gonna guess it's shifted towards the end but I sort of want to even see if the accuracy of the linear probe in the middle, does it keep the same? So does the curve go like this? This is the linear probe when you simply pre-train. This is linear probe accuracy. The question would be does it change to be like this or does it change to be like this? This is supposed to be the same at the end. So basically does it stay as good as it is but simply get better at the end or does the representation like in this curve, does the good representation now shift towards the end and leave the lower layer with even more capacity to do some low-level stuff? Yeah, maybe they've done this. I haven't seen it. And as you can see these BERT and autoregressive objective, they sort of trade off. So the BERT it tends to do poorly in the linear probe setting but then it catches up during fine-tuning. In C410 almost being at the level of the autoregressive and in in ImageNet actually outperforming it. This darker thing here it simply means that you average across different maskings of BERT because I guess even in classification it's not entirely clear how to get a signal out of BERT because they don't do this CLS vector with BERT. What they do for classification and linear probing and that's written up here, they simply take the average pooling of all the representations of the sequence. And the last thing that I've also forgotten, there's a lot of stuff, when they fine-tune, while fine-tuning the classification loss yields reasonable downstream performance, we find empirically that the joint objective, the generative objective and the classification objective works even better. So even when you fine-tune with this model you have to keep the generative modeling part, the generative loss around and then it performs even more better, more well, whatever that word is. So that's also something to think about. I think this paper right here it kind of lays down a lot of cool things that you can think about and it gives rise to a lot of hypotheses of how does this stuff work, why does this stuff work. I don't even think that the numbers are the most important thing, it's mostly the fact of the effects and what does it mean. Okay, so this was my take on it. It's more kind of a my rant of what I find special about this paper than about the actual paper. You can look at the paper, their numbers are pretty good. On ImageNet they do not reach the same like super-duper performance as they do on C410 and I guess that's probably because they have to downscale the ImageNet images way more than they have to downscale the C410 images because those are of course only 32 by 32. So because they have to downscale so much they lose probably a lot of information and I would be interested to see if there is a way to involve convolutions in all of this. So to do the downscaling that in a learned manner with convolutions or something. I'm sure this has all been done already, I'm just lazy to look it up. Yeah, so I invite you to look at their blog post where they have these samples. They look pretty funny and these full samples up here look fairly cool for what it's trained to do and that it has no spatial awareness whatsoever. It simply uses learned position encodings. And yeah, check it out, that was it from me. Bye bye.
 Okay, I'm sure many of you have already seen this because it was rather widely
 announced but the OpenAI team has announced a new model that produces
 pictures instead of text. So as you can see right here on the left you'll always
 see like a half a picture and on the right is the ground truth. So they took
 this picture, they simply cut the bottom half right here and then they let the
 model sort of imagine what they cut away and what it comes up with is pretty cool
 I have to say. Like look at the birds, like this is just awesome. But the special
 thing about this isn't that it simply completes pictures, the special thing
 about it is it does it one pixel by pixel. So basically it goes at this pixel
 right here and asks okay what's that pixel and then what's that pixel and
 so on. So it is basically like a language model but for pixels in that it
 goes over the images in order basically like this or like always from left to
 right, left to right, left to right and it has no clue of the spatial relations
 between the pixels. It needs to learn that by itself as opposed to a
 convolutional neural network which is specifically designed such that if you
 want to predict this pixel right here then it's specifically designed to say
 okay the most important information is probably around that pixel and then some
 like other important information is wider around that pixel. So CNNs are
 built with this in mind whereas this model right here which is also known as
 image GPT doesn't have any of that. It's simply a transformer model that
 goes over these pixels one by one and we'll see how that's done. There are some
 more examples right here. Particularly cool is the cat and you see that there
 is the beginning of this little white thing right here which is this card and
 the completions of the model. Yes very interesting. The model can of course as a
 language model can also sample by itself just random images. You sample them once
 through and this is what it comes up with. So these are pretty good quality
 images for a model that just produces one pixel by one pixel. Now this is a
 pixel. Now this idea of one pixel by pixel isn't new. This has been around
 before but the investigation here is basically how much can we how far can we
 push these generative models for pre-training. Hi there this is Janek
 from Post Production. I've realized that I've forgotten to even read the name of
 the paper. So it's called generative pre-training from pixels by Mark Chen,
 Alec Radford, Rowan Child, Jeff Wu, Hewon Ju, Prafula Dariwal, David Luan and
 Ilya Sotskyver. And since Henry AI Labs has already made a video on this, this
 video is going to be more of kind of a rumble rant about what I find
 interesting about the paper and some thoughts about it rather than like a
 classic explanation. I hope you still enjoy that. So what you saw on the right
 wasn't even the this isn't the final result the supposed result this is
 simply the pre-training task. It's fun to look at it but the actual object
 objective of the paper is the following. What if we train we pre-train on a large
 data set to generate what good images like these or we to complete images like
 these and then we fine-tune on a classification task. And the answer is
 here they say on C410 we achieve the 96.3% accuracy with a linear probe
 outperforming a supervised wide resonant and a 99% accuracy with full
 fine-tuning matching the top supervised pre-trained models. An even larger model
 trained on a mixture of ImageNet and web images is competitive with self
 supervised benchmarks on ImageNet achieving 72 top one accuracy on a
 linear probe of our features. So the goal here is that you have a data set that
 you want to train a classifier on. So usually you have a data set and the
 data set has images and you put them through like a convolutional neural
 network and then you have to classify the image into one of I don't know how
 many classes on C410 that's 10 classes on ImageNet it's a thousand. And the
 data set is these images together with these labels. Now the idea of pre-training
 is that you somewhere have a bigger data set that is sort of similar to the small
 data set but yeah it's similar enough such that the network could learn
 something. So what you want to do first is you first want to take the large data
 set, train this network right here and then in a second step fine-tune the
 network on this smaller data set and you sort of hope that what you learn from
 the large data set right here transfers over a little bit of knowledge. You
 already have a little bit of knowledge and you can make better use of the data
 that you have right here. Now the question is how do you do this pre-training
 and of course this has a long tradition, well long for maybe two or three years
 right now in the language community where people they pre-train these large
 models like we've just seen GPT-3 or BERT was one of them. They pre-train these
 large transformer models on text and then to fine-tune them on classification
 tasks for text and that's what this paper is doing right here. They pre-train a
 transformer that is a GPT-2 scale model, they pre-train it on image generation
 and then they fine-tune it or transfer learn it to classification tasks. And the
 point of the paper is to say that like in text data, in text data we have made
 pretty good experiences with doing this, with pre-training a generative
 model and then fine-tuning on a classification task. While so far in
 images all we've ever done is we've pre-trained these pre-training tasks
 which usually is a classification task or like a self-supervised task with a
 contrastive loss or something like this. What they're doing new is the
 generative modeling as a pre-training. And again this isn't entirely new but
 they show that if you throw a lot of computers at it and lots of data and a
 model then that can work equally well to these self-supervised tasks. So their
 model as I said is pretty pretty simple. They take an image and they unroll the
 image. Now a fully unrolled image on let's say ImageNet has 224 squared
 pixels and that times three right because you have three color channels.
 That's too large even for an open AI supercomputer. So what they do is first
 they downscale the image. So they downscale it's not as drastic as here
 where you just get a three by three image but they do downscale it to like a 32 by
 32 or a 64 by 64. Then they unroll it which simply means they go through the
 image like this and make a sequence out of it because their models are naturally
 made for text sequences. They simply put the image into a text sequence. They
 further simplify this by reducing the three color channels to a single one. So
 they have their own color representation and basically yeah they reduce the
 three color channels to one channel that simply indexes the color in their color
 representation. And they say it's still pretty good. It's pretty faithful. So
 ultimately they end up with like a 32 squared length representation of their
 image. And then they do one of two things. They either do autoregressive
 generative pre-training which is the sort of GPT-2 style pre-training. And the
 idea here is that you always want to predict the next pixel of a sequence. So
 you can see right here that's the sequence that you input.
 And you always want to predict what is the next pixel. And in this case you
 see that we've already predicted everything. Here we've already predicted
 everything up to this red pixel. So you want to know what's this next pixel, this
 thing right here. What's this going to be? And the diagram here basically shows
 you how the attention flows. So every position in this transformer, and if you
 don't know what a transformer is, I haven't made a video about attention is
 all you need where these are explained. But briefly every position here can sort
 of send information only in one direction. So you train all
 of these in parallel. And when you predict this pixel right here, you only
 want information from whatever was before that pixel. Otherwise the model
 could cheat, right? Otherwise the model could simply learn to copy over the
 value. But the attention pattern here is simply to show you that this is
 autoregressive and it's in one direction. So you always want to predict
 the next pixel. And then from all of this you want to predict the next pixel. And
 then from all of this you want to predict the next pixel. This is in
 contrast to this objective here that comes from BERT. And I've also made a
 video on BERT. What you do in BERT is you simply take that image and you cross
 the block out two of the pixels, or many of the pixels, and you simply ask your
 network to reconstruct those pixels. And now you can see the attention flows
 in all directions. BERT, the B stands actually for bidirectional. So this is the
 contrast to the autoregressive pre-training framework. Now these two
 things have been applied in text both. The autoregressive is usually easier
 to actually make it produce something, like we saw producing these images,
 because you can always just predict the next pixel and then the next and then
 the next and then the next. Whereas in BERT it's a bit more unclear how you
 would produce things in a consistent manner. Because the predictions of these
 two pixels right here, they are independent. It's one forward pass and
 then both of these are predicted. But other papers have tried to solve this
 like this, not XLNet. I forget its name. It's something with an X.
 But these are the two objectives they look at. And it turns out they
 sort of trade off a bit. They work equally well, or a bit better and a bit worse,
 depending on the task. So once they have done this, they simply feed images.
 And you'll notice that you don't need any labels for this. So what you'll do is
 simply input an image and then simply take away half of it like this, and then
 predict that pixel. And then you want to predict that pixel and then you want to
 predict that pixel. That's all like you do with text. And in BERT you simply
 input an image, cross out pixels and then predict them. So you don't need labels
 for this. And that's why you can do it with this big data set. And you can do it
 in an unsupervised fashion. So you can just crawl the internet for images and
 just feed this into there. And it will sort of learn to produce these images.
 Now the question is, if you learn to produce these images, does that
 help you for classification? And they have two methods of assessing
 this. The bottom one here is the fine-tuning method. So this is
 supposed to be the representation you learn in the different layers of the
 network. So this is supposed to be this thing right here. What you'll do is
 you'll simply fine-tune. That means you on top of this representation you add a
 classification head that has two outputs, cat or dog, and you train this entire
 network on your small data set that we discussed before. So you train the
 entire network, all of the parameters. This is called fine-tuning. In contrast
 to that, what you can do is you can simply add this
 classification head with two outputs and then only train this classification head.
 And that won't perform as well, but it gives you sort of a better idea of
 how good is the representation that this network right here learned. And on top of
 that, so if you spin this idea further, you can actually go and do this at any
 intermediate layer right here. So you can forward propagate until layer two
 right here, and then here you add your classification head into the two
 classes and you only train the classification head. That being said, you
 can also do this with fine-tuning, but in this case this is called a linear probe.
 And it is often used to assess how good the A representation in intermediate
 layers is. Whereas what it actually does is assessing how linearly classifiable a
 representation is, which isn't the same as how useful or how informative, but it
 is one way to assess these things. So these are the two things they
 assess. So as for datasets, for C410 they use like C410 and C4100 as
 datasets and the STL10. And there you have to keep in mind the pre-training
 is done on ImageNet for those. So you pre-train on ImageNet without the
 labels and then you transfer learn or fine-tune or linear probe on these
 small datasets. Whereas later we're going to look at ImageNet and there the
 pre-training as I understand it is done on ImageNet itself, but also a wider
 collection of a hundred million or so images from the web, from the internet.
 Okay, so as you can see right here this is what happens if you do this linear
 probing. And you can see it works pretty well. So you get like a 95-96% accuracy
 with linear probes. This is very powerful. So it's not easy to get 96% on
 C410. I mean current state-of-the-art is like 99%, but still 96% is pretty
 good. And this is the entire network. There is this big giant network that you
 input your image into and then there is this one linear layer that does the
 classification. And all of this right here has not been trained with
 classification in mind. It simply has been trained to reproduce images. It
 hasn't even been trained on C410 as far as I understand. It's been trained on
 ImageNet. So this is to stress how cool or how significant this result is
 basically. That just a linear probe on top of that will give you such a good
 accuracy. And the second thing that is obvious right here is this bottom axis
 is the layer. So this is the layer where they attach the linear probe. And usually
 if you pre-train a network with a classification task in mind, so you
 pre-train it with the labels or maybe even without the labels in a self
 supervised way or something like this, usually the last layer has the best
 representation for classification. But here the special thing is that the
 intermediate layers in the middle have the best representation. You can see that
 the representation quality in terms of linear probing falls off as they sort of
 it falls off as they go into higher layers. And this is consistent across the
 datasets as you can see. And the idea here is or the way they interpret it
 is that if you have an image right here and you've blocked part of it,
 so you've blocked this and this, wrong way around this, so you've generated
 everything and now your task is to predict the next pixel. So you're
 trained to predict this next pixel right here. And the idea is that as you put
 the image through the network, what it will do is sort of, since the first
 layers they're going to be, if you're going to be similar to a CNN, they're
 going to be doing some low-level feature transformation thing. But also the
 last layers, they're going to really care about what's the exact pixel that goes
 here. Since it's their job to do that, they're going to care what color
 does it need to have, what exact luminosity and so on, how does it fit in
 with the previous pixels and so on. So that's also good. But it's not
 just low-level information and consistency with other pixels or
 something like this. At some point if you want to generate consistent images, and
 we saw that this model can generate consistent images, at some point there
 needs to be some kind of a notion of the global information in the picture,
 because the images are consistent throughout. So there needs to be some
 notion of what is in that image as a whole. And that's the exact
 information that we need for classification. And the only way that
 could actually be is here in the middle, since you know that's the place. So the
 hypothesis is that these models somehow learn a higher level of
 representation of global information somewhere in the middle before they then
 specify that information again down to predict the actual pixel. And that's why
 the best representations for classification are in the middle. So this
 is one of the interesting findings of
 this paper. I mean it's cool that they can reach a good accuracy, but to recognize
 that maybe in these generative models they have some intermediate stage
 where they represent the global information, and that will actually make
 the best representation. The second cool thing right here is that you can
 see they have different sizes of models. So the IGPT-L I believe is something
 like 60 layers, then this is like 48 layers, and this is 32 layers. So
 these are all on the scale of GPT-2, either a little bigger or a
 little smaller. It's not like a GPT-3 scale where you need a ginormous
 supercomputer, though they do do a lot of computation. But this still sort of fits
 within hardware of a standard size and not like exascale. What's interesting
 right here is that you can see the larger models, they reach a lower
 validation loss. So here is the validation loss. The larger model, if you train them
 on, so these checkpoints here are always after the same amount of steps. The
 larger models do reach a lower validation loss right here, as you can see.
 So this is the large, this is the medium, this is the small. And also you can see
 that on this axis the linear probe accuracy. So this is whenever you go
 and you find the best intermediate layer for linear probing, you probe it and you
 record the accuracy. So you can see a general trend as your validation loss
 goes down, the linear probe accuracy goes up. So there is a connection like it is
 in text models. In text models there's a connection of the perplexity of your
 language model and the quality of the representation you get for downstream
 tasks. In this model it seems to be the exact same thing. There is a connection
 between reaching lower validation loss and reaching a higher performance on
 classification. So that's one interesting thing, the general trend to up to the
 upper right corner. The other interesting and even arguably even more
 interesting thing is that if you look at the same validation loss. So at this
 point all of these models have the same validation loss, yet still the bigger
 model is better. You can see right here the bigger model outperforms the
 smaller model even though they have the same validation loss on the image
 modeling task. And this is also something that OpenAI in their text
 papers has stressed, that the larger models they seem to be somehow more
 capable of forming good representations even if they have the
 same loss. So again this could just be sort of a training data,
 better training data remembering thing. And when I said that in GPT-3 I didn't
 actually mean explicit remembering of training data. I meant kind of a fuzzy
 remembering of training data. I formulate that in the comments but
 I feel a lot of people have misunderstood me there. Here I think it's
 a much harder to estimate what's going on also since image pixels. Humans
 don't have a super good model on image pixels in their head as we have about
 text. As you can see if you then fine-tune, so for now we've just do
 linear probing, if you fine-tune these architectures then you reach like a 99%
 accuracy on C410 which is on par with the best models that we have. So G-Pipe
 is supervised, pre-trained on ImageNet but also I guess uses a bunch of data
 augmentation while these image GPT it uses minimal data augmentation I think.
 They simply random crop a little bit and that's about it. So they also experiment
 around with this BERT objective. So until now this was all this
 autoregressive objective and I feel that OpenAI people are a bit more of a fan of
 the autoregressive objective just given what they've done so far in their papers.
 And you can see here comparison of the two objectives on C410 and on ImageNet.
 Again C410 is pre-trained with ImageNet and ImageNet itself is pre-trained
 with like a larger collection of images from the web. All the pre-training is
 done without labels. Now the blue is what you can reach with a linear probe and
 the orange is then on top of that what you can reach by fine-tuning. So no
 linear probe but fine-tuning. I have to say that the fine-tuning is always done
 at the end. So even though the linear probe can be attached
 anywhere in between and it's often useful to do that as we saw because the
 in-between layers are the best. They say they tried fine-tuning also from
 in-between but it always worked out best whenever you fine-tune. Whenever you
 fine-tune you take actually the last layer. So that kind of gives you an idea
 that the model is then... What seems to be important is this coming up
 with the higher level representation and then once you fine-tune you're probably
 able to push that representation through to the end because of your training
 signal. But if you hadn't done the pre-training you wouldn't even have
 that higher level representation and then the signal I guess is not strong
 enough to back propagate through the whole model. It would be very interesting
 if they investigate, if they do this linear probe analysis again after they
 fine-tune the model. And to see if then still it is the intermediate layers
 that have the best representation or if now the best representation in a linear
 probe sense shifted towards the end. I'm gonna guess it's shifted towards the end
 but I sort of want to even see if the accuracy of the linear probe in the
 middle, does it keep the same? So does the curve go like this? This is the
 linear probe when you simply pre-train. This is linear probe accuracy. The
 question would be does it change to be like this or does it change to be like
 this? This is supposed to be the same at the end. So basically does it stay as
 good as it is but simply get better at the end or does the representation like
 in this curve, does the good representation now shift towards the
 end and leave the lower layer with even more capacity to do some low-level
 stuff? Yeah, maybe they've done this. I haven't seen it. And as you can see
 these BERT and autoregressive objective, they sort of trade off. So the BERT it
 tends to do poorly in the linear probe setting but then it catches up during
 fine-tuning. In C410 almost being at the level of the autoregressive and in
 in ImageNet actually outperforming it. This darker thing here it
 simply means that you average across different maskings of BERT because I
 guess even in classification it's not entirely clear how to get a signal out
 of BERT because they don't do this CLS vector with BERT. What they do for
 classification and linear probing and that's written up here, they simply take
 the average pooling of
 all the representations of the sequence. And the last thing that I've
 also forgotten, there's a lot of stuff, when they fine-tune, while fine-tuning
 the classification loss yields reasonable
 downstream performance, we find empirically that the joint objective, the
 generative objective and the classification objective works even
 better. So even when you fine-tune with this model you have to keep the
 generative modeling part, the generative loss around and then it performs even
 more better, more well, whatever that word is. So that's also something to think
 about. I think this paper right here it kind of lays down a lot of cool
 things that you can think about and it gives rise to a lot of hypotheses of how
 does this stuff work, why does this stuff work. I don't even think that the
 numbers are the most important thing, it's mostly the fact of the effects and
 what does it mean. Okay, so this was my take on it. It's more kind of a my
 rant of what I find special about this paper than about the actual paper. You
 can look at the paper, their numbers are pretty good. On ImageNet they do not
 reach the same like super-duper performance as they do on C410 and I
 guess that's probably because they have to downscale the ImageNet images way
 more than they have to downscale the C410 images because those are of course
 only 32 by 32. So because they have to downscale so much they lose probably a
 lot of information and I would be interested to see if there is a way to
 involve convolutions in all of this. So to do the downscaling that in a
 learned manner with convolutions or something. I'm sure this has all been
 done already, I'm just lazy to look it up. Yeah, so I invite you to look at their
 blog post where they have these samples. They look pretty funny and these
 full samples up here look fairly cool for what it's trained to do
 and that it has no spatial awareness whatsoever. It simply uses learned
 position encodings. And yeah, check it out, that was it from me. Bye bye.