OpenAI CLIP: ConnectingText and Images (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper openai sutskever radford meme dalle dall-e images vision text nlp natural language processing resnet vision transformer transformer visual transformer sota state of the art zero shot zero-shot few shot few-shot unsupervised contrastive simclr efficientnet noisy student representation embedding latent natural language prompt engineering bias scale distribution shift
#ai #openai #technology

Paper Title: Learning Transferable Visual Models From Natural Language Supervision
CLIP trains on 400 million images scraped from the web, along with text descriptions to learn a model that can connect the two modalities. The core idea is a contrastive objective combined with a large batch size. The resulting model can be turned into arbitrary zero-shot classifiers for new image & text tasks.

OUTLINE:
0:00 - Introduction
3:15 - Overview
4:40 - Connecting Images & Text
9:00 - Building Zero-Shot Classifiers
14:40 - CLIP Contrastive Training Objective
22:25 - Encoder Choices
25:00 - Zero-Shot CLIP vs Linear ResNet-50
31:50 - Zero-Shot vs Few-Shot
35:35 - Scaling Properties
36:35 - Comparison on different tasks
37:40 - Robustness to Data Shift
44:20 - Broader Impact Section
47:00 - Conclusion & Comments

Paper: https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf
Blog: https://openai.com/blog/clip/
Code: https://github.com/openai/CLIP

Abstract:
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on.

Authors: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 So here you see a classifier that takes a look at this image and assigns one of many, many labels, actually one of 101 labels, as you can see here. And one of the labels is a photo of guacamole, a type of food, and it assigns a really high probability to that, as opposed to like the second prediction, which is ceviche. So, you know, classifier, pretty good. Okay. Take a look at this classifier. Out of 397 labels, it correctly identifies that this is a television studio. You can go on right here. And so this is a photo of an airplane. Whenever there's a green bar at the top, it means that the respective classifier has this correctly. Whenever there is an orange bar, it's an incorrect label with the green bar being the correct label. So you can see here these classifiers perform sometimes pretty well on these examples and sometimes not. But what you can distinctly see is that these are all from different data sets. So different tasks. There is a satellite image. There is a car and you're supposed to classify which car it is. Not only that, it is a car. So very diverse set of tasks. And the interesting thing is that this is all the same classifier. So this classifier is it's not even fine tuned. It is a zero shot classifier that handles all of these different training data sets. Sorry, not training data sets. All of these different test data sets in one go. So that's already pretty cool. But what you may have noticed is that the labels aren't labels that you would usually see in a classifier. So these 101 labels here, they are, it says it here, Wacomole. That's the label. Interestingly, the label the classifier assigns is not just the word. It's the a photo of Wacomole, a type of food. That's the label the classifier assigns. And the second highest label is a photo of ceviche, a type of food. It's not always a photo, though it is often a photo. But here you can see, for example, the label that the classifier assigns is a centered satellite photo of permanent crop land, where the the correct label here is the annual crop land, which is down here. Again, the label is longer. So there's something interesting going on here. It's the same classifier. It's zero shots. So that means the classifier is not trained on these data sets. It's not trained to fulfill these tasks, yet still it seems to perform okay. And the labels are quite weird. So this is this is a new paper by OpenAI, which we're going to look at today. You can see it's a pretty long paper, but we'll cut it short, I promise. And it's called Learning Transferable Visual Modes from Natural Language Supervision. And the model colloquially or also in this paper is referred to as CLIP. So this is the model has been released along with the DALI model, which can do the chair made of avocado and so on. The DALI model is a generative model that generates images. CLIP is a more of a I want I want to say discriminative model, but CLIP is a model that takes in images and text and connects them in a non generative way. So we're going to see what that entails. It's by Alec Radford and Jong-Woo Kim and others, as I said, of OpenAI. So the idea here is to connect text and images. And this has been done in a in a number of ways previously, even in this way, it has been done in one fashion or another. I find the introduction and discussion of related related works in this paper to be very, very thorough and and superb. So they do assign a lot of credit to people who have had the various ideas. So the goal here is that we want to get a model that can represent images and text really, really well. OK, so how do we connect images and text? First of all, what what if what if we have a data set of images and text? OK, so they construct a new data set where there is an image, something like this, a cat and a text, a little piece of text to it. Like my my cute cat images and text like this you'll find on, you know, for example, social media. You can scrape that Pinterest, what not flicker. People write descriptions along with their pictures. So it's pretty easy to get these pairs of images and text from the Internet without having to label them. Right. So one motivation of doing this kind of work is if we train a image classifier model, we always need labeled examples into, you know, into a very predefined set of classes. So an image that we have a thousand classes or twenty two thousand respectively. And MNIST we have ten. However, if we could just somehow learn to connect images with the text that comes along, we wouldn't be bound by the classifier labels and we could get very good representations. So the original idea or one of the original idea is we take the image and we predict, predict the text from the image. Of course, Dali goes the other way. So Dali some somehow goes the other way, taking the text and predicting the image. But the idea is if we can take an image and from it predict the text, what we get out of it is not only a model that can label images, but what we hope to get out of it is this process right here may be very, very good representer. So if this is like the image goes into a neural network with a bunch of layers and then outcomes, you know, the text, my cat and so on, then somewhere in here in the intermediate representation of the neural network, there must be a pretty, pretty good representation of what is in the image. So not not only, you know, the pixel values, but there must be actually some kind of representation of the concept of cat, because otherwise it could not predict the word cat at the end. OK, so the idea is to get a really good representer and then you could take that representation and fine tune it to other tasks and so on. So that's one of the ideas that we're going to work off here. And it turns out this is pretty useful. There have been papers before predicting the simply predicting the caption of images, but it doesn't work too well. So what this model here is going for and we'll simply we'll simply let's look at this graph right here. So they tried first to predict the text and you can see that zero shot and we're going to to look at what exactly zero shot image net accuracy means in this context. But you can see here that they had some success with using a transformer language model to predict the text and images and evaluating that on on image net. However, they seem to have more success by using just a bag of words prediction. So what that means is you're not trying to predict the exact words. You're simply trying to predict which words occur in the description. So you see the photo if you predict cat and my and cute in any not non ordered you're already correct. And that already gives a sort of a better efficiency. You can see the models here. They tend to go up, but it's questionable if that will ever reach the orange line. And with their new objective with what this paper suggests, you can see right here the contrastive method. You get a way bigger performance. So we'll look at what this zero shot accuracy means and why it might be that these simply predicting the text from an image might not be a good enough idea. So let's say we have a model that can do this. We have a model that can take an image and it can predict the text that appears in it. Most of the time, this model right here is also going to give you something like a probability, like a likelihood. So if this is a transformer, you can you can ask for its logits and then you can compute the likelihood of a given label. So if you have such a model, what you can do is exactly what what they allude to right here. If you have an image task and you have a you have a model that can predict the text of an image, you can take that image and you can run this sort of through your image and through your encoding pipeline. And then you can ask the model instead of predicting a text, you can ask the model how likely is the text dog? How likely is the text cat for this image? How likely is the text mouse? And then you can you get some sort of likelihood. Right. So maybe it says dog is this likely cat is this likely mouse is this likely. And immediately you have built a classifier. So I hope you can see if if I have a model that can predict how likely a piece of text goes with an image, I can by simply asking my model for each of the for each of the classes that are possible in the task, I immediately get a classifier out of that. I mean, I have to normalize or something by that, but I immediately get a classifier. And now you can already see why we might want to phrase the things a bit. So I don't want to just put dog and cat right here, even though those are the labels in that task. Right. If if I had an image net classifier, I would put here I would put all of the 1000 possible classes and ask the model for each. How likely is that label to go with this image and the model can produce text, but the model can not only produce the single word dog. The model can also tell me how likely is the phrase a photo of a dog, a photo of a dog, or how likely is the phrase a photo of a cat and so on. Right. So and you can you can see that this result here, the classifier result, it might change actually, depending on how you phrase. So here you can use the exact same classes as you used above. But by rephrasing the prompt, so to say, you might get a better quality classifier or a worse quality classifier. So if you already know that your images are all photographs and you will get a better accuracy because simply, you know, the model, if you you might get a better accuracy by asking the model, hey, how likely is the phrase a photo of a dog going with this image versus the phrase a photo of a cat that might give you a better signal. So less noise in whatever you get as an output than simply going with the single word. Because again, this model is trained to predict this just from a data set scrape from the Internet. So how often do people post something, I don't know, on Instagram of their cat and simply write cat with it? Whereas, you know, maybe they they write here's a photo of my cat. Right. So the phrase photo of a cat is or they do like hashtag photo hashtag cat or something like this. So that's why these classifiers at the bottom, they were constructed from the labels of the data set, but with a prompt that has been adapted by humans to work, you know, find to work particularly well on that data set. So we're sort of back to prompt engineering here. So this is how we go from a model that can assess predict text to a classifier. And that's a zero shot classifier. We don't need to train this classifier on the actual task. We simply need to restrict its possible outputs to the classes at hand. Right. This is a bit it's a bit like a tiny bit like like, you know, in in Q learning in where for in in each step you ask your model. Well, what if I do action one and then the model tells you what that's five good probably your Q value is five and then he says, well, what if I do action two and then your model says, well, that's seven good and so on. So it's it's sort of a similar concept in except, you know, Q learning. We usually train end to end with an actual classifier. But I said simply predicting text objective might not be good enough. Right. So we're going to retain this property of being able to zero shot classifier. But we're going to now switch out our task of how we get to such a model. So instead of predicting text, what does clip do clip does the following. So what we're going to do is we're going to take the image right here and we're going to pass it through an image encoder. And that gives us an image representation. So a vector in some latent space. So this is image one and then image two right here would be image two here. OK, so we have a mini batch of images and that's important. Then we're going to take the text and feed it to the text encoder. Also obtaining a representation for the text, a single vector for this entire text right here. And then, of course, if we go to the second sample in the mini batch, we get the second representation. And the batch is, of course, in the training data set. We know that the first the first text goes with the first image. The second text goes with the second image, the third text goes with the third image, because that's how we scraped it from the Internet. And then what we ask the model to do is simply to tell us not so previously we tried to predict from the image the text, right? We went through the image encoder and from this representation here, we try to predict the text. So we no longer do that. What we're trying to do is simply ask. Ask the model which for so for this representation, which of these texts is most appropriate to that particular image. OK, so this is why it's called a contrastive objective. We know because this is training data, we of course know that image one goes with description one. And image two goes with description two. But we're going to train this in the way that we feed in this image and we ask it to which of all of these texts right here, to which of all of these is this image the closest? And we're going to train it such that it is maximally close to the correct one and minimally and far away from all the other. So this this is why it's contrastive. It contrasts what we know goes together, right? The diagonal elements in this matrix with what we know doesn't go together. Actually, we don't know if a different description wouldn't fit the same image, but we can safely assume that a random piece of text, since we do the mini batches randomly, a random piece of text will probably not go with this particular image, at least not as well as the piece of text that we founded with on the Internet. Okay, so you get what you get is effectively for each input, you get a classification task in this direction. You can see right here for image three, there is one correct text that it goes with. And for each text, you get a classification task in this direction. By the way, this is simply an inner product right here, right? You simply trying to maximize the inner product of things that go together and minimize the inner product of things that don't go together. So you you multiply the two for the inner product, you interpret that as a log it, and then you do a softmax classification in this direction and the softmax classification in this direction. So this is a symmetric loss from the text and image perspective. And yeah, so it's a classification problem, classification problem viewed from two different angles. So you can immediately see that this relies on having large enough mini batches, right? So the larger your mini batch, as your mini batch size approximates the entire data set, your representations are going to be more and more detailed, right? So you want to so pepper the Aussie pop being close together to this particular image means that in the ideal case, it is close to this image and far away from anything else in the data set. And as an approximation, far away from anything in this particular mini batch. And at inference time, you do very much what we did so far. So you take if you want to build an image classifier. And the interesting thing is you can also build a text classifier, right? If you have multiple images to go with a text, then you you can do that. It's entirely symmetric. But in this case, you take an image, you put it through the image encoder, you get a representation here, you get all the labels of your classification tasks, right? So this is the label is this right here, you engineer a prompt and that you do as a human, right? This is heuristic. This you as a human think, aha, OK, I'm going to put whatever this is here. You encode all of these labels in their prompt context through the text encoder. And you get the representations here and you simply ask to which of these labels is it closest, right? So is the inner product the highest? And then and that's how you obtain the label. Zero training needed on the actual task, right? So the data set that you do this with can be an entirely different data set that then you do this with. And this is extremely, extremely interesting. I've actually seen some some posts on Twitter and Reddit where people use this to guide a style again to produce given pictures with given descriptions and so on. So the possibilities for this are pretty, pretty huge. OK, so that's that's the model, the model. It encodes images and codes text. It does this contrastive objective. What goes together? What needs a part? And now you see why this might be a better representer than, for example, simply pre-training a model on an image classification task. Because if you pre-train a model on an image classification task, it is going to simply lump together every all the dogs. You know, if this is if this is your classification task, it's going to lump together all the dogs because there's no need to differentiate the individual dogs from each other. Right. It's going to lump all of them together and forget that they are actually different. Right. It's also going to forget everything that doesn't concern the immediate classification problem. Whereas this model here, this model is specific as as it gets better and better, it will pick up at more of the text. Right. So in this case, maybe if the model is pretty weak still, it will focus on this pup. And that's about the same as saying, OK, it's a classifier of a dog. But then we can also see pup if it incorporates that, if it gets better. Well, it can differentiate it from other dogs. And by the way, it's a pup. So it's a young dog. It can also learn, eventually learn its actual name. Right. And and so on. So you can see this as the model gets stronger, can pick up more and more nuances of the data set. So they test this and they tested fairly, fairly, fairly extensively. And I don't think we'll have to go through all of it for me to convince you that this is a good idea. You're going to maybe see it approximately or immediately. So, yes, so they use different, different types of yes. That's what I wanted to say. They use different types of encoders for the image encoder. So for the text encoder, this is a transformer. So trans former, it's not a particularly big transformer even. And they simply take the end of sentence token, the representation of that at the end. And that's their vector. If you don't know what a transformer is, I've done many, many videos on transformers. Find one of them, any of them for the image encoder, they test out a bunch of different things. So they test out a bunch of variants of Resnet. I've done a video on that. And they also test out a bunch of variants of the visual transformer, the VIT that has recently been popularized. I've also made a video on that. So that's why their model shows up in sort of different flavors and sort of different, different points here. They scale the amount of data, I believe, with the model. So they scale everything together, compute data and model size. And that's why you see different variants of the same model. They also do ensembling. So, you know, you have to engineer these prompts. And what you can do is you can engineer better prompts and that will gain performance. And you can also ensemble over prompts. And you can see right here that that gets you both an efficiency gain. If you want to stay at the same performance and also sorry, yeah. And also it gives you a performance improvement for the same compute with the same model. Right. So here the corresponding dots are the same model. That's why they have the same compute. So that's just one of the fun things you can do. And again, I think prompt engineering will become quite a bit more relevant. So here you can see you can see the comparison. Zero shot clip is competitive with a fully supervised baseline. Right. So the baseline here isn't too good. So it's a fully supervised linear classifier fitted on ResNet 50 features on 16 datasets, including ImageNet. So the ResNet 50 is a popular architecture. It's not nowhere near the absolute best we have, but it's a popular architecture. So this ResNet 50, what it's what it has been trained on is that's been trained on ImageNet. Right. So you get so and that results in a neural network with a bunch of layers, including a classification layer at the end, right into a thousand classes. So what you do is you pre train this on ImageNet and then you simply take this part right here up until the last layer. And you take it. So that's this part right here. And you assume that this has a sort of a good representational power since it can do ImageNet. And then you simply train a new linear classifier on top that does the classification into whatever new task you want. So this is called it's called linear probing. So linear probing, you can also do it in the middle sort of. But in this case, they mean linear probing at the second to last layer, like before the classification layer. So you assume that whatever this is, is a good representation function. You keep it constant and then you train a linear probe on top of it. This is compared to fine tuning where you would fine tune the entire network on your new task. But they elect to do most of their experiments with linear probing since it gives you a better indication of the representational power of the bases. So here they compare to ImageNet, right? So on six and that is including ImageNet. So for ImageNet, you would expect ResNet-50 to perform quite well because it's been its representational base has been trained on ImageNet and training a linear classifier on top. It should simply give you back the performance that it had on ImageNet. And here you can see how zero shot clip compares to linear probe on ResNet-50, right? Zero shot clip compared to an actual trained thing. Not the best, but a trained thing. And you can see that on many, many, many data sets clip outperforms the ResNet-50. Zero shot, right? So no training required beyond the pre-training. That being said, the pre-training is huge. But it's similar to GPT-3, right? You train it once, huge training, but then you can do lots of things. ImageNet, interestingly, you see right here only it's actually improving ImageNet over ResNet-50. Crazy, right? Whereas, so ResNet-50 still better in various other tasks. So this is not to say that this is the new state of the art or anything, except in STL-10 where it actually appears to be the new state of the art against all the previously, including all the supervised whatever, it's the new state of the art on this data set. And the reason is this STL-10 data set, it has very few training examples per class only. So supervised is very difficult. Transfer learning is kind of difficult. As I understand it, it's not that similar to ImageNet. So that transfer learning is kind of different. So this really seems to be this zero shot clip objective seems to be good if you have images that are sort of natural, that happen a lot on the internet, but are not really like ImageNet. So there exists quite a number of those and that you have few labeled examples of if any, right? So that's a good application domain. However, on more specialized things, they say things like tumor classification and so on, but on the other hand, in terms of the satellite images, this clip objective still does pretty poorly, probably because, you know, that's not the type of images you find on the internet with a piece of text. Super interesting. MNIST, one of the easiest tasks in deep learning. It also quite underperforms in this thing. So they compare to ResNet-50 and also to visual N-grams right here, and they discuss the importance of the different data sets. Oh, I found this to be very interesting. Most standard image classification data sets treat the information, naming or describing classes, which enables natural language based zero shot transfer as an afterthought. The vast majority of data sets annotate images with just a numeric ID of the label and contain a file mapping these IDs back to their names in English. Some data sets, such as Flowers and the GTSRB, that's a German transport street sign or data set. I don't exactly know, don't appear to include this mapping at all in their released versions, preventing zero shot transfer entirely. So what these authors had to do is they had to look at the classes and then sort of label them themselves because their model works on language, whereas this street sign data set probably just came with this is sign type one, this is sign type two. They have a footnote here. Alec learned much more about flower species and German traffic signs over the course of this project than he originally anticipated. I love that. I love a bit of humor in the papers. And so I made this meme where the street sign is specifically tractors and trucks with an authorized loaded weight of more than 3.5 tons prohibited. I wonder actually how the model does on exactly this sign, but yeah, we'll find out. By the way, the clip model is available in not the big one, but a small one is available, actually trained. So you can test it out and maybe we'll do a video on it where we actually do something with it. So here you can see that if they compare their model to few shot linear probes. So here they compare zero shot clip with few shot linear probes. So before we compare to linear probe, which mean means we just trained this linear classifier, but we did it on the whole data set. OK, so here we simulate only having very few examples per class, which is where pre training really comes in. And you can see that zero shot clip outperforms a lot of models if you only give them very few labeled examples per class. In fact, it is comparative to a 16. It is comparative to a 16 label bit M. So this is one of the best models that is currently in the public and that is doing this transfer learning. So if you transfer learn with a linear probe, again, this is not fine tuning with a linear probe on 16 samples per class with this model, you are still only as good as the zero shot, no training at all of the clip model. That is pretty, pretty interesting and pretty cool. The other noteworthy thing is that if you linearly probe the clip model, you way outperform the the largest models there. And also, what is also interesting is that when you do labeled examples for clip, when you do linear probe on clip, the performance decreases first and only increases once you get to like four labeled examples per class. And that, you know, is is pretty intuitive when you think about it. So what you're doing is so in clip, the zero shot classifier is actually a different one than the linear classifier. So the zero shot classifier is in a way already trained. So it has already trained this sort of last layer, whereas if you do linear probing, you throw that away, you know, the whole part where you encode the text and you blah, blah, blah, you throw that away. And you simply do the old school. So the linear probe here, this is no more of that is which text is close. This is simply I take this I throw away the last layer, I put in a new last layer and I do my original classification task. And of course, this layer right here is initialized randomly and it's going to require some training. And maybe, you know, one example per class isn't enough. It's just going to pick up on some spurious correlation in the feature. And it's going that's why it's getting worse initially. But it recovers at four examples per class and it severely outperforms the other models. So we'll forgive it. They do discover in various experiments here that it is very, very different from data set to data set how this model performs zero shot, how it performs versus linear probing. They they find that they find that very often in in in some data sets that are far away from sort of natural images, they perform worse in again, in some data sets, they require lots of labels to match zero shot performance. So it is really a study into sort of I want to say it's a study into what kind of images appear on the Internet. They do. Interestingly, there is a trend in machine learning that if you give more data and compute, then your error goes down even with the same type of models. And that seems to hold pretty well here, as you can see here, as they scale up. This is the same. This is a ResNet backbone. As you scale that up, zero shot clip performance scales smoothly as a function of model compute. However, they do note that there is a whole bunch of variations of the curve you're seeing as the average. But for the individual tasks in their task data sets, it varies wildly. So there's a lot of noise here. This could be because of how the data sets are selected. This could be because of how the prompts are engineered. There are still a lot on known right here. They compare various other things like linear probe linear probe performance of clip models in comparison with state of the art computer vision models. And they do outperform all of these other models, as you can see here. So there is 12 data sets in previous experiments, but the 12 are still sort of similar to ImageNet. But if you include more data sets, of course, that's sort of a selection bias or whatnot. But then this model severely outperforms all of the other models. So the red models here are the red ones are the clip models compared to the other ones. So, yeah, this seems to be a step forward in the sort of in the sort of building classifiers for the average user. So I can now go ahead, take this model and build my own classifier pretty, pretty easily. They also make some interesting discoveries in terms of robustness and robustness to perturbations. So previously, all these models, they sort of pre trained on ImageNet and so on. And people have discovered that as soon as you go away from ImageNet, these the performance of these models decreases heavily. So if, for example, ImageNet V2 is just ImageNet, but is it they try to collect. I made a video about that, by the way, they try to collect ImageNet as closely as possible to the original test set. They try to collect a new test set and immediately the performance of all the classifiers dropped in the light of this just slightly data shifted data set. And if you if you sort of try to go away a little bit further, so you just have sketches of these objects, you sort of have this this adversarial placement of objects you can see right here. It's pretty it's pretty mean, but still a human could do this right. You see right here that these are just variations on the themes of ImageNet. They have the same classes. So a classifier trained on ImageNet should be able to also classify these images. So here they compare zero shot clip to models that have been trained on ImageNet. And they find that zero shot clip, even though it matches. So this zero shot clip matches the performance of ImageNet. By the way, huge achievement, right? This is a fully trained model on ImageNet. And this is a not the state of the art, but respectable top one performance on ImageNet. And zero shot classifier matches that performance. This is crazy. You can see as this classifier degrades, degrades, degrades, degrades, degrades, as you go to harder and harder data sets that are all technically ImageNet images like in the same classes. This classifier, it sometimes even gets better. But it keeps up its performance, which you can see here the difference between it gets just larger and larger. So the clip is way more robust. And of course, this model right here is trained to predict these specific types of images. So it knows very well how to keep them apart. The only thing it has to do as a classifier of ImageNet is keep apart the individual instances of exactly those classes and exactly this data set. So it forgets about everything else. Right. And as a result, it has never seen a sketch. It like a banana is yellow. What are you talking about? So it heavily degrades. Right. And whereas clip, it simply knows how to sort of connect images to text. So while clip realizes that, of course, both are described as banana, it somehow has to account for the fact that there are also lemons in here. Right. It has to somehow represent that. It has to represent that this is a bunch of fruit and that this is here. Maybe a high grade picture like on a magazine where this here might be more of a sort of random GoPro fallen into some bunch of bananas. It has to somehow represent all of this if it performs well on its task and thereby its representation will be nuanced enough such that it can transfer more easily. It picks up on different features than only distinguishing banana from other classes in the ImageNet data set. And that results. So here is the curve in that if you had the ideally robust model, you'd have this right here. So the exact same performance on the natural distortions than on ImageNet in the original ImageNet. You can see that all of the standard ImageNet training examples, including all the robustness techniques that barely lift away from this curve, are massively outperformed by a zero. Again, a zero shot classifier that hasn't even been trained on ImageNet. And the fact that it hasn't been trained on ImageNet might be one of the things that it actually is very helpful. So they do some investigation into it, including that you can in fact adapt to ImageNet. So you can in I think that's a linear probe. If you linear probe clip, you can improve the performance on ImageNet where interestingly you can improve the performance on ImageNet by doing a linear probe on top of clip. This is logistic regression clip while only mildly degrading your performance on these other data sets. So there seems to be a value to only to just having the representation. So the representation itself seems to be more stable. So you can see as you adapt to ImageNet, this performance improves massively, but it only degrades a little bit across the other data sets. So that means, yeah, as I said, the representation itself is more nuanced, such that even if you train a linear classifier on pure classification, you'll still keep up the performance on the other tasks. You can also adapt to class shift. So by better prompt sort of prompt engineering for some of these subtasks. But I think that's a sort of a minor thing. All right. Yeah, I don't want to go too much. They also compare to humans, which is very interesting. And they discover that samples that are hard for the clip model are also hard for the human model. They do some sort of duplicate detection from their training data set because their training data set is 400 million images together with text. Right. So it's conceivable that there's some duplicates, but they find even if there is, there's generally not a problem. And they have like a three or four page broader impact section, as you can see right here, which is so if you read it, it reads sort of like, yeah, there are problems with these models. We are better than other models, but we're still not good enough or things like this. Or they always they were like, yeah, this is of course we're better like they're better at everything. But then again, you know, this is only preliminary more study is needed and so on. But I so they have some fairly interesting, interesting results. So they what they do is since there is such a focus on prompt engineering, right, it actually matters what you give to the model as possible labels. So this is no longer fixed labels. You can give any labels. So they have these data sets where you, you know, for example, this fair face, fair face race, where you try to categorize faces into different ethnicities or races. These seven things that are given here, they also include some non human categories. Or is it so they include they include categories such as here, such as animal chimpanzee gorilla or Angutan. And they also include sort of crime categories like thief, suspicious person, criminal. And then they research how how the model misbehaves. And these models, they do do a fair bit of, you know, kind of misclassification right here, as you can see. They also so they notice that the misclassification is especially there for younger people. So these are the ages of people. And here are the misclassification rates. You can see the misclassifications are mostly for younger people, then they simply add a child category. And then the misclassification for young people all of a sudden drops because the model now has the option to classify them as a child. So this, I think the result of the paper and especially of the broader impact section, one of the results is that it matters a lot how you engineer the prompts, which is something we already knew. But of course, this can be particularly, particularly crucial in some applications in some concerning applications. That's kind of one of their points right here. You can see that the paper is huge and it also has a huge appendix. And they do, as I said, a lot more experiments right here. But all in all, this is a very, very cool approach, I feel. And it's, as I said, a step towards making it easier for the, you know, the everyday person to build their own classifier for, you know, you can do quite niche tasks. As long as they're sort of natural images, this will work fairly, fairly well. I think it's pretty cool. It gives, it gives a little bit of more freedom in how you work with these models. And I'm excited for people to come up with ideas of how to use this, how to connect this to other models, such as you can connect it, as we already saw with Dolly, you can connect it with StyleGAN, as some people are doing. And sure, you can connect it to something like GPT-3 and it's going to be an exciting world. All right. That was it for me. Thanks. Bye bye.
 So here you see a classifier that takes a look at this image and assigns one of many, many labels, actually one of 101 labels, as you can see here.
 And one of the labels is a photo of guacamole, a type of food, and it assigns a really high probability to that, as opposed to like the second prediction, which is ceviche.
 So, you know, classifier, pretty good. Okay. Take a look at this classifier. Out of 397 labels, it correctly identifies that this is a television studio.
 You can go on right here. And so this is a photo of an airplane. Whenever there's a green bar at the top, it means that the respective classifier has this correctly.
 Whenever there is an orange bar, it's an incorrect label with the green bar being the correct label.
 So you can see here these classifiers perform sometimes pretty well on these examples and sometimes not.
 But what you can distinctly see is that these are all from different data sets. So different tasks. There is a satellite image.
 There is a car and you're supposed to classify which car it is. Not only that, it is a car. So very diverse set of tasks.
 And the interesting thing is that this is all the same classifier. So this classifier is it's not even fine tuned.
 It is a zero shot classifier that handles all of these different training data sets. Sorry, not training data sets. All of these different test data sets in one go.
 So that's already pretty cool. But what you may have noticed is that the labels aren't labels that you would usually see in a classifier.
 So these 101 labels here, they are, it says it here, Wacomole. That's the label.
 Interestingly, the label the classifier assigns is not just the word. It's the a photo of Wacomole, a type of food.
 That's the label the classifier assigns. And the second highest label is a photo of ceviche, a type of food.
 It's not always a photo, though it is often a photo. But here you can see, for example, the label that the classifier assigns is a centered satellite photo of permanent crop land,
 where the the correct label here is the annual crop land, which is down here. Again, the label is longer.
 So there's something interesting going on here. It's the same classifier. It's zero shots. So that means the classifier is not trained on these data sets.
 It's not trained to fulfill these tasks, yet still it seems to perform okay. And the labels are quite weird.
 So this is this is a new paper by OpenAI, which we're going to look at today.
 You can see it's a pretty long paper, but we'll cut it short, I promise.
 And it's called Learning Transferable Visual Modes from Natural Language Supervision.
 And the model colloquially or also in this paper is referred to as CLIP.
 So this is the model has been released along with the DALI model, which can do the chair made of avocado and so on.
 The DALI model is a generative model that generates images.
 CLIP is a more of a I want I want to say discriminative model, but CLIP is a model that takes in images and text and connects them in a non generative way.
 So we're going to see what that entails. It's by Alec Radford and Jong-Woo Kim and others, as I said, of OpenAI.
 So the idea here is to connect text and images.
 And this has been done in a in a number of ways previously, even in this way, it has been done in one fashion or another.
 I find the introduction and discussion of related related works in this paper to be very, very thorough and and superb.
 So they do assign a lot of credit to people who have had the various ideas.
 So the goal here is that we want to get a model that can represent images and text really, really well.
 OK, so how do we connect images and text?
 First of all, what what if what if we have a data set of images and text?
 OK, so they construct a new data set where there is an image, something like this, a cat and a text, a little piece of text to it.
 Like my my cute cat images and text like this you'll find on, you know, for example, social media.
 You can scrape that Pinterest, what not flicker.
 People write descriptions along with their pictures.
 So it's pretty easy to get these pairs of images and text from the Internet without having to label them.
 Right. So one motivation of doing this kind of work is if we train a image classifier model,
 we always need labeled examples into, you know, into a very predefined set of classes.
 So an image that we have a thousand classes or twenty two thousand respectively.
 And MNIST we have ten.
 However, if we could just somehow learn to connect images with the text that comes along,
 we wouldn't be bound by the classifier labels and we could get very good representations.
 So the original idea or one of the original idea is we take the image and we predict, predict the text from the image.
 Of course, Dali goes the other way.
 So Dali some somehow goes the other way, taking the text and predicting the image.
 But the idea is if we can take an image and from it predict the text,
 what we get out of it is not only a model that can label images,
 but what we hope to get out of it is this process right here may be very, very good representer.
 So if this is like the image goes into a neural network with a bunch of layers and then outcomes, you know, the text, my cat and so on,
 then somewhere in here in the intermediate representation of the neural network,
 there must be a pretty, pretty good representation of what is in the image.
 So not not only, you know, the pixel values, but there must be actually some kind of representation of the concept of cat,
 because otherwise it could not predict the word cat at the end.
 OK, so the idea is to get a really good representer and then you could take that representation and fine tune it to other tasks and so on.
 So that's one of the ideas that we're going to work off here.
 And it turns out this is pretty useful.
 There have been papers before predicting the simply predicting the caption of images, but it doesn't work too well.
 So what this model here is going for and we'll simply we'll simply let's look at this graph right here.
 So they tried first to predict the text and you can see that zero shot and we're going to to look at what exactly zero shot image net accuracy means in this context.
 But you can see here that they had some success with using a transformer language model to predict the text and images and evaluating that on on image net.
 However, they seem to have more success by using just a bag of words prediction.
 So what that means is you're not trying to predict the exact words.
 You're simply trying to predict which words occur in the description.
 So you see the photo if you predict cat and my and cute in any not non ordered you're already correct.
 And that already gives a sort of a better efficiency.
 You can see the models here.
 They tend to go up, but it's questionable if that will ever reach the orange line.
 And with their new objective with what this paper suggests, you can see right here the contrastive method.
 You get a way bigger performance.
 So we'll look at what this zero shot accuracy means and why it might be that these simply predicting the text from an image might not be a good enough idea.
 So let's say we have a model that can do this.
 We have a model that can take an image and it can predict the text that appears in it.
 Most of the time, this model right here is also going to give you something like a probability, like a likelihood.
 So if this is a transformer, you can you can ask for its logits and then you can compute the likelihood of a given label.
 So if you have such a model, what you can do is exactly what what they allude to right here.
 If you have an image task and you have a you have a model that can predict the text of an image, you can take that image and you can run this sort of through your image and through your encoding pipeline.
 And then you can ask the model instead of predicting a text, you can ask the model how likely is the text dog?
 How likely is the text cat for this image?
 How likely is the text mouse?
 And then you can you get some sort of likelihood.
 Right. So maybe it says dog is this likely cat is this likely mouse is this likely.
 And immediately you have built a classifier.
 So I hope you can see if if I have a model that can predict how likely a piece of text goes with an image, I can by simply asking my model for each of the for each of the classes that are possible in the task, I immediately get a classifier out of that.
 I mean, I have to normalize or something by that, but I immediately get a classifier.
 And now you can already see why we might want to phrase the things a bit.
 So I don't want to just put dog and cat right here, even though those are the labels in that task.
 Right. If if I had an image net classifier, I would put here I would put all of the 1000 possible classes and ask the model for each.
 How likely is that label to go with this image and the model can produce text, but the model can not only produce the single word dog.
 The model can also tell me how likely is the phrase a photo of a dog, a photo of a dog, or how likely is the phrase a photo of a cat and so on.
 Right. So and you can you can see that this result here, the classifier result, it might change actually, depending on how you phrase.
 So here you can use the exact same classes as you used above.
 But by rephrasing the prompt, so to say, you might get a better quality classifier or a worse quality classifier.
 So if you already know that your images are all photographs and you will get a better accuracy because simply, you know, the model, if you you might get a better accuracy by asking the model,
 hey, how likely is the phrase a photo of a dog going with this image versus the phrase a photo of a cat that might give you a better signal.
 So less noise in whatever you get as an output than simply going with the single word.
 Because again, this model is trained to predict this just from a data set scrape from the Internet.
 So how often do people post something, I don't know, on Instagram of their cat and simply write cat with it?
 Whereas, you know, maybe they they write here's a photo of my cat. Right. So the phrase photo of a cat is or they do like hashtag photo hashtag cat or something like this.
 So that's why these classifiers at the bottom, they were constructed from the labels of the data set, but with a prompt that has been adapted by humans to work, you know, find to work particularly well on that data set.
 So we're sort of back to prompt engineering here.
 So this is how we go from a model that can assess predict text to a classifier.
 And that's a zero shot classifier. We don't need to train this classifier on the actual task.
 We simply need to restrict its possible outputs to the classes at hand. Right.
 This is a bit it's a bit like a tiny bit like like, you know, in in Q learning in where for in in each step you ask your model.
 Well, what if I do action one and then the model tells you what that's five good probably your Q value is five and then he says, well, what if I do action two and then your model says, well, that's seven good and so on.
 So it's it's sort of a similar concept in except, you know, Q learning.
 We usually train end to end with an actual classifier.
 But I said simply predicting text objective might not be good enough. Right.
 So we're going to retain this property of being able to zero shot classifier.
 But we're going to now switch out our task of how we get to such a model.
 So instead of predicting text, what does clip do clip does the following.
 So what we're going to do is we're going to take the image right here and we're going to pass it through an image encoder.
 And that gives us an image representation.
 So a vector in some latent space.
 So this is image one and then image two right here would be image two here.
 OK, so we have a mini batch of images and that's important.
 Then we're going to take the text and feed it to the text encoder.
 Also obtaining a representation for the text, a single vector for this entire text right here.
 And then, of course, if we go to the second sample in the mini batch, we get the second representation.
 And the batch is, of course, in the training data set.
 We know that the first the first text goes with the first image.
 The second text goes with the second image, the third text goes with the third image, because that's how we scraped it from the Internet.
 And then what we ask the model to do is simply to tell us not so previously we tried to predict from the image the text, right?
 We went through the image encoder and from this representation here, we try to predict the text.
 So we no longer do that.
 What we're trying to do is simply ask.
 Ask the model which for so for this representation, which of these texts is most appropriate to that particular image.
 OK, so this is why it's called a contrastive objective.
 We know because this is training data, we of course know that image one goes with description one.
 And image two goes with description two.
 But we're going to train this in the way that we feed in this image and we ask it to which of all of these texts right here, to which of all of these is this image the closest?
 And we're going to train it such that it is maximally close to the correct one and minimally and far away from all the other.
 So this this is why it's contrastive.
 It contrasts what we know goes together, right?
 The diagonal elements in this matrix with what we know doesn't go together.
 Actually, we don't know if a different description wouldn't fit the same image, but we can safely assume that a random piece of text, since we do the mini batches randomly, a random piece of text will probably not go with this particular image, at least not as well as the piece of text that we founded with on the Internet.
 Okay, so you get what you get is effectively for each input, you get a classification task in this direction.
 You can see right here for image three, there is one correct text that it goes with.
 And for each text, you get a classification task in this direction.
 By the way, this is simply an inner product right here, right?
 You simply trying to maximize the inner product of things that go together and minimize the inner product of things that don't go together.
 So you you multiply the two for the inner product, you interpret that as a log it, and then you do a softmax classification in this direction and the softmax classification in this direction.
 So this is a symmetric loss from the text and image perspective.
 And yeah, so it's a classification problem, classification problem viewed from two different angles.
 So you can immediately see that this relies on having large enough mini batches, right?
 So the larger your mini batch, as your mini batch size approximates the entire data set,
 your representations are going to be more and more detailed, right?
 So you want to so pepper the Aussie pop being close together to this particular image means that in the ideal case,
 it is close to this image and far away from anything else in the data set.
 And as an approximation, far away from anything in this particular mini batch.
 And at inference time, you do very much what we did so far.
 So you take if you want to build an image classifier.
 And the interesting thing is you can also build a text classifier, right?
 If you have multiple images to go with a text, then you you can do that.
 It's entirely symmetric. But in this case, you take an image, you put it through the image encoder, you get a representation here,
 you get all the labels of your classification tasks, right?
 So this is the label is this right here, you engineer a prompt and that you do as a human, right?
 This is heuristic. This you as a human think, aha, OK, I'm going to put whatever this is here.
 You encode all of these labels in their prompt context through the text encoder.
 And you get the representations here and you simply ask to which of these labels is it closest, right?
 So is the inner product the highest? And then and that's how you obtain the label.
 Zero training needed on the actual task, right?
 So the data set that you do this with can be an entirely different data set that then you do this with.
 And this is extremely, extremely interesting.
 I've actually seen some some posts on Twitter and Reddit where people use this to guide a style again to produce given pictures with given descriptions and so on.
 So the possibilities for this are pretty, pretty huge.
 OK, so that's that's the model, the model. It encodes images and codes text.
 It does this contrastive objective. What goes together? What needs a part?
 And now you see why this might be a better representer than, for example, simply pre-training a model on an image classification task.
 Because if you pre-train a model on an image classification task, it is going to simply lump together every all the dogs.
 You know, if this is if this is your classification task, it's going to lump together all the dogs because there's no need to differentiate
 the individual dogs from each other. Right.
 It's going to lump all of them together and forget that they are actually different.
 Right. It's also going to forget everything that doesn't concern the immediate classification problem.
 Whereas this model here, this model is specific as as it gets better and better, it will pick up at more of the text.
 Right. So in this case, maybe if the model is pretty weak still, it will focus on this pup.
 And that's about the same as saying, OK, it's a classifier of a dog.
 But then we can also see pup if it incorporates that, if it gets better.
 Well, it can differentiate it from other dogs. And by the way, it's a pup. So it's a young dog.
 It can also learn, eventually learn its actual name. Right.
 And and so on. So you can see this as the model gets stronger, can pick up more and more nuances of the data set.
 So they test this and they tested fairly, fairly, fairly extensively.
 And I don't think we'll have to go through all of it for me to convince you that this is a good idea.
 You're going to maybe see it approximately or immediately.
 So, yes, so they use different, different types of yes.
 That's what I wanted to say. They use different types of encoders for the image encoder.
 So for the text encoder, this is a transformer. So trans former, it's not a particularly big transformer even.
 And they simply take the end of sentence token, the representation of that at the end.
 And that's their vector. If you don't know what a transformer is, I've done many, many videos on transformers.
 Find one of them, any of them for the image encoder, they test out a bunch of different things.
 So they test out a bunch of variants of Resnet. I've done a video on that.
 And they also test out a bunch of variants of the visual transformer, the VIT that has recently been popularized.
 I've also made a video on that.
 So that's why their model shows up in sort of different flavors and sort of different, different points here.
 They scale the amount of data, I believe, with the model.
 So they scale everything together, compute data and model size.
 And that's why you see different variants of the same model.
 They also do ensembling. So, you know, you have to engineer these prompts.
 And what you can do is you can engineer better prompts and that will gain performance.
 And you can also ensemble over prompts.
 And you can see right here that that gets you both an efficiency gain.
 If you want to stay at the same performance and also sorry, yeah.
 And also it gives you a performance improvement for the same compute with the same model.
 Right. So here the corresponding dots are the same model.
 That's why they have the same compute.
 So that's just one of the fun things you can do.
 And again, I think prompt engineering will become quite a bit more relevant.
 So here you can see you can see the comparison.
 Zero shot clip is competitive with a fully supervised baseline.
 Right. So the baseline here isn't too good.
 So it's a fully supervised linear classifier fitted on ResNet 50 features on 16 datasets, including ImageNet.
 So the ResNet 50 is a popular architecture.
 It's not nowhere near the absolute best we have, but it's a popular architecture.
 So this ResNet 50, what it's what it has been trained on is that's been trained on ImageNet.
 Right. So you get so and that results in a neural network with a bunch of layers,
 including a classification layer at the end, right into a thousand classes.
 So what you do is you pre train this on ImageNet and then you simply take this part right here up until the last layer.
 And you take it.
 So that's this part right here. And you assume that this has a sort of a good representational power since it can do ImageNet.
 And then you simply train a new linear classifier on top that does the classification into whatever new task you want.
 So this is called it's called linear probing.
 So linear probing, you can also do it in the middle sort of.
 But in this case, they mean linear probing at the second to last layer, like before the classification layer.
 So you assume that whatever this is, is a good representation function.
 You keep it constant and then you train a linear probe on top of it.
 This is compared to fine tuning where you would fine tune the entire network on your new task.
 But they elect to do most of their experiments with linear probing since it gives you a better indication of the representational power of the bases.
 So here they compare to ImageNet, right?
 So on six and that is including ImageNet.
 So for ImageNet, you would expect ResNet-50 to perform quite well because it's been its representational base has been trained on ImageNet and training a linear classifier on top.
 It should simply give you back the performance that it had on ImageNet.
 And here you can see how zero shot clip compares to linear probe on ResNet-50, right?
 Zero shot clip compared to an actual trained thing.
 Not the best, but a trained thing.
 And you can see that on many, many, many data sets clip outperforms the ResNet-50.
 Zero shot, right?
 So no training required beyond the pre-training.
 That being said, the pre-training is huge.
 But it's similar to GPT-3, right?
 You train it once, huge training, but then you can do lots of things.
 ImageNet, interestingly, you see right here only it's actually improving ImageNet over ResNet-50.
 Crazy, right?
 Whereas, so ResNet-50 still better in various other tasks.
 So this is not to say that this is the new state of the art or anything,
 except in STL-10 where it actually appears to be the new state of the art against all the previously,
 including all the supervised whatever, it's the new state of the art on this data set.
 And the reason is this STL-10 data set, it has very few training examples per class only.
 So supervised is very difficult.
 Transfer learning is kind of difficult.
 As I understand it, it's not that similar to ImageNet.
 So that transfer learning is kind of different.
 So this really seems to be this zero shot clip objective seems to be good
 if you have images that are sort of natural, that happen a lot on the internet,
 but are not really like ImageNet.
 So there exists quite a number of those and that you have few labeled examples of if any, right?
 So that's a good application domain.
 However, on more specialized things, they say things like tumor classification and so on,
 but on the other hand, in terms of the
 satellite images, this clip objective still does pretty poorly,
 probably because, you know, that's not the type of images you find on the internet with a piece of text.
 Super interesting. MNIST, one of the easiest tasks in deep learning.
 It also quite underperforms in this thing.
 So they compare to ResNet-50 and also to visual N-grams right here,
 and they discuss the importance of the different data sets.
 Oh, I found this to be very interesting.
 Most standard image classification data sets treat the information, naming or describing classes,
 which enables natural language based zero shot transfer as an afterthought.
 The vast majority of data sets annotate images with just a numeric ID of the label
 and contain a file mapping these IDs back to their names in English.
 Some data sets, such as Flowers and the GTSRB,
 that's a German transport street sign or data set.
 I don't exactly know, don't appear to include this mapping at all in their released versions,
 preventing zero shot transfer entirely.
 So what these authors had to do is they had to look at the classes
 and then sort of label them themselves because their model works on language,
 whereas this street sign data set probably just came with this is sign type one, this is sign type two.
 They have a footnote here.
 Alec learned much more about flower species and German traffic signs
 over the course of this project than he originally anticipated.
 I love that. I love a bit of humor in the papers.
 And so I made this meme where the street sign is specifically
 tractors and trucks with an authorized loaded weight of more than 3.5 tons prohibited.
 I wonder actually how the model does on exactly this sign, but yeah, we'll find out.
 By the way, the clip model is available in not the big one, but a small one is available, actually trained.
 So you can test it out and maybe we'll do a video on it where we actually do something with it.
 So here you can see that if they compare their model to few shot linear probes.
 So here they compare zero shot clip with few shot linear probes.
 So before we compare to linear probe, which mean means we just trained this linear classifier,
 but we did it on the whole data set.
 OK, so here we simulate only having very few examples per class, which is where pre training really comes in.
 And you can see that zero shot clip outperforms a lot of models if you only give them very few labeled examples per class.
 In fact, it is comparative to a 16.
 It is comparative to a 16 label bit M.
 So this is one of the best models that is currently in the public and that is doing this transfer learning.
 So if you transfer learn with a linear probe, again, this is not fine tuning with a linear probe on 16 samples per class with this model,
 you are still only as good as the zero shot, no training at all of the clip model.
 That is pretty, pretty interesting and pretty cool.
 The other noteworthy thing is that if you linearly probe the clip model, you way outperform the the largest models there.
 And also, what is also interesting is that when you do labeled examples for clip, when you do linear probe on clip,
 the performance decreases first and only increases once you get to like four labeled examples per class.
 And that, you know, is is pretty intuitive when you think about it.
 So what you're doing is so in clip, the zero shot classifier is actually a different one than the linear classifier.
 So the zero shot classifier is in a way already trained.
 So it has already trained this sort of last layer, whereas if you do linear probing, you throw that away, you know, the whole part where you encode the text and you blah, blah, blah, you throw that away.
 And you simply do the old school.
 So the linear probe here, this is no more of that is which text is close.
 This is simply I take this I throw away the last layer, I put in a new last layer and I do my original classification task.
 And of course, this layer right here is initialized randomly and it's going to require some training.
 And maybe, you know, one example per class isn't enough.
 It's just going to pick up on some spurious correlation in the feature.
 And it's going that's why it's getting worse initially.
 But it recovers at four examples per class and it severely outperforms the other models.
 So we'll forgive it.
 They do discover in various experiments here that it is very, very different from data set to data set how this model performs zero shot, how it performs versus linear probing.
 They they find that they find that very often in in in some data sets that are far away from sort of natural images, they perform worse in again, in some data sets, they require lots of labels to match zero shot performance.
 So it is really a study into sort of I want to say it's a study into what kind of images appear on the Internet.
 They do. Interestingly, there is a trend in machine learning that if you give more data and compute, then your error goes down even with the same type of models.
 And that seems to hold pretty well here, as you can see here, as they scale up.
 This is the same. This is a ResNet backbone.
 As you scale that up, zero shot clip performance scales smoothly as a function of model compute.
 However, they do note that there is a whole bunch of variations of the curve you're seeing as the average.
 But for the individual tasks in their task data sets, it varies wildly.
 So there's a lot of noise here. This could be because of how the data sets are selected.
 This could be because of how the prompts are engineered.
 There are still a lot on known right here.
 They compare various other things like linear probe linear probe performance of clip models in comparison with state of the art computer vision models.
 And they do outperform all of these other models, as you can see here.
 So there is 12 data sets in previous experiments, but the 12 are still sort of similar to ImageNet.
 But if you include more data sets, of course, that's sort of a selection bias or whatnot.
 But then this model severely outperforms all of the other models.
 So the red models here are the red ones are the clip models compared to the other ones.
 So, yeah, this seems to be a step forward in the sort of in the sort of building classifiers for the average user.
 So I can now go ahead, take this model and build my own classifier pretty, pretty easily.
 They also make some interesting discoveries in terms of robustness and robustness to perturbations.
 So previously, all these models, they sort of pre trained on ImageNet and so on.
 And people have discovered that as soon as you go away from ImageNet, these the performance of these models decreases heavily.
 So if, for example, ImageNet V2 is just ImageNet, but is it they try to collect.
 I made a video about that, by the way, they try to collect ImageNet as closely as possible to the original test set.
 They try to collect a new test set and immediately the performance of all the classifiers dropped in the light of this just slightly data shifted data set.
 And if you if you sort of try to go away a little bit further, so you just have sketches of these objects,
 you sort of have this this adversarial placement of objects you can see right here.
 It's pretty it's pretty mean, but still a human could do this right.
 You see right here that these are just variations on the themes of ImageNet.
 They have the same classes. So a classifier trained on ImageNet should be able to also classify these images.
 So here they compare zero shot clip to models that have been trained on ImageNet.
 And they find that zero shot clip, even though it matches.
 So this zero shot clip matches the performance of ImageNet.
 By the way, huge achievement, right? This is a fully trained model on ImageNet.
 And this is a not the state of the art, but respectable top one performance on ImageNet.
 And zero shot classifier matches that performance. This is crazy.
 You can see as this classifier degrades, degrades, degrades, degrades, degrades,
 as you go to harder and harder data sets that are all technically ImageNet images like in the same classes.
 This classifier, it sometimes even gets better.
 But it keeps up its performance, which you can see here the difference between it gets just larger and larger.
 So the clip is way more robust.
 And of course, this model right here is trained to predict these specific types of images.
 So it knows very well how to keep them apart.
 The only thing it has to do as a classifier of ImageNet is keep apart the individual instances of exactly those classes and exactly this data set.
 So it forgets about everything else. Right.
 And as a result, it has never seen a sketch.
 It like a banana is yellow. What are you talking about?
 So it heavily degrades. Right.
 And whereas clip, it simply knows how to sort of connect images to text.
 So while clip realizes that, of course, both are described as banana, it somehow has to account for the fact that there are also lemons in here.
 Right. It has to somehow represent that.
 It has to represent that this is a bunch of fruit and that this is here.
 Maybe a high grade picture like on a magazine where this here might be more of a sort of random GoPro fallen into some bunch of bananas.
 It has to somehow represent all of this if it performs well on its task and thereby its representation will be nuanced enough such that it can transfer more easily.
 It picks up on different features than only distinguishing banana from other classes in the ImageNet data set.
 And that results. So here is the curve in that if you had the ideally robust model, you'd have this right here.
 So the exact same performance on the natural distortions than on ImageNet in the original ImageNet.
 You can see that all of the standard ImageNet training examples, including all the robustness techniques that barely lift away from this curve, are massively outperformed by a zero.
 Again, a zero shot classifier that hasn't even been trained on ImageNet.
 And the fact that it hasn't been trained on ImageNet might be one of the things that it actually is very helpful.
 So they do some investigation into it, including that you can in fact adapt to ImageNet.
 So you can in I think that's a linear probe.
 If you linear probe clip, you can improve the performance on ImageNet where interestingly you can improve the performance on ImageNet by doing a linear probe on top of clip.
 This is logistic regression clip while only mildly degrading your performance on these other data sets.
 So there seems to be a value to only to just having the representation. So the representation itself seems to be more stable.
 So you can see as you adapt to ImageNet, this performance improves massively, but it only degrades a little bit across the other data sets.
 So that means, yeah, as I said, the representation itself is more nuanced, such that even if you train a linear classifier on pure classification, you'll still keep up the performance on the other tasks.
 You can also adapt to class shift. So by better prompt sort of prompt engineering for some of these subtasks.
 But I think that's a sort of a minor thing.
 All right. Yeah, I don't want to go too much. They also compare to humans, which is very interesting.
 And they discover that samples that are hard for the clip model are also hard for the human model.
 They do some sort of duplicate detection from their training data set because their training data set is 400 million images together with text.
 Right. So it's conceivable that there's some duplicates, but they find even if there is, there's generally not a problem.
 And they have like a three or four page broader impact section, as you can see right here, which is so if you read it, it reads sort of like, yeah, there are problems with these models.
 We are better than other models, but we're still not good enough or things like this.
 Or they always they were like, yeah, this is of course we're better like they're better at everything. But then again, you know, this is only preliminary more study is needed and so on.
 But I so they have some fairly interesting, interesting results.
 So they what they do is since there is such a focus on prompt engineering, right, it actually matters what you give to the model as possible labels.
 So this is no longer fixed labels. You can give any labels.
 So they have these data sets where you, you know, for example, this fair face, fair face race, where you try to categorize faces into different ethnicities or races.
 These seven things that are given here, they also include some non human categories.
 Or is it so they include they include categories such as here, such as animal chimpanzee gorilla or Angutan.
 And they also include sort of crime categories like thief, suspicious person, criminal.
 And then they research how how the model misbehaves.
 And these models, they do do a fair bit of, you know, kind of misclassification right here, as you can see.
 They also so they notice that the misclassification is especially there for younger people.
 So these are the ages of people. And here are the misclassification rates.
 You can see the misclassifications are mostly for younger people, then they simply add a child category.
 And then the misclassification for young people all of a sudden drops because the model now has the option to classify them as a child.
 So this, I think the result of the paper and especially of the broader impact section, one of the results is that it matters a lot how you engineer the prompts,
 which is something we already knew. But of course, this can be particularly, particularly crucial in some applications in some concerning applications.
 That's kind of one of their points right here. You can see that the paper is huge and it also has a huge appendix.
 And they do, as I said, a lot more experiments right here.
 But all in all, this is a very, very cool approach, I feel. And it's, as I said, a step towards making it easier for the, you know, the everyday person to build their own classifier for, you know, you can do quite niche tasks.
 As long as they're sort of natural images, this will work fairly, fairly well. I think it's pretty cool.
 It gives, it gives a little bit of more freedom in how you work with these models.
 And I'm excited for people to come up with ideas of how to use this, how to connect this to other models, such as you can connect it, as we already saw with Dolly, you can connect it with StyleGAN, as some people are doing.
 And sure, you can connect it to something like GPT-3 and it's going to be an exciting world. All right. That was it for me. Thanks. Bye bye.