VirTex: Learning Visual Representations from Textual Annotations (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper cnn visual resnet caption nlp transformer vasvani attention text coco imagenet convolutional neural network adaptation transfer learning quality unsupervised self-supervised
Pre-training a CNN backbone for visual transfer learning has recently seen a big push into the direction of incorporating more data, at the cost of less supervision. This paper investigates the opposite: Visual transfer learning by pre-training from very few, but very high-quality samples on an image captioning task.

OUTLINE:
0:00 - Intro & Overview
1:00 - Pre-Training for Visual Tasks
3:40 - Quality-Quantity Tradeoff
5:50 - Image Captioning
8:35 - VirTex Method
14:30 - Linear Classification
20:30 - Ablations
22:05 - Fine-Tuning
25:45 - Attention Visualization
27:30 - Conclusion & Remarks

Paper: https://arxiv.org/abs/2006.06666
Code: https://github.com/kdexd/virtex

Abstract:
The de-facto approach to many vision tasks is to start from pretrained visual representations, typically learned via supervised training on ImageNet. Recent methods have explored unsupervised pretraining to scale to vast quantities of unlabeled images. In contrast, we aim to learn high-quality visual representations from fewer images. To this end, we revisit supervised pretraining, and seek data-efficient alternatives to classification-based pretraining. We propose VirTex -- a pretraining approach using semantically dense captions to learn visual representations. We train convolutional networks from scratch on COCO Captions, and transfer them to downstream recognition tasks including image classification, object detection, and instance segmentation. On all tasks, VirTex yields features that match or exceed those learned on ImageNet -- supervised or unsupervised -- despite using up to ten times fewer images.

Authors: Karan Desai, Justin Johnson

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at Vertex Learning Visual Representations from Textual Annotations by Karen Desai and Justin Johnson of the University of Michigan. So this paper at its core is pretty simple. On a high level it proposes to take the task of image captioning, which is where you're given an image and you're asked to produce a caption for the image, and basically train a model to do this, and then just take the visual part of it as a baseline to transfer learn on other visual tasks. And that appears to work surprisingly well if you don't have much data. So if you don't have much data to pre-train on, this appears to work very well. Alright, as always, if you like content like this, then consider sharing it out, subscribing to the channel, or tell me what you think in the comments. So as I already said, the idea here is pretty simple. So people have been looking for pre-training tasks for visual tasks. So a visual task is anything where the input is an image, and then you usually have some sort of neural network that processes the image, and then at the end you can have many things. So you could have a classifier that classifies the image into one of many classes. If you know ImageNet, that's a thing. So if there's a cat here, then the ImageNet classifier here would say cat. Or you could have something like an object detector that tries to predict on the image where the cat is, like with a bounding box. You could have a semantic segmentation where it's like all of these pixels here are cats, and maybe all of these pixels here are sky. And so it labels every pixel. There's many visual tasks that you can formulate, and they all sort of share the same architecture. And specifically, they all share this part right here. If you will, this is the visual encoder. It's usually a convolutional neural network. And what's really different between the tasks is mostly this last part here that does the actual task. But this is often called the backbone. So this is the backbone. And the idea now is, if I have a bunch of these tasks, sometimes I don't have many labels for these tasks. I don't have many labeled images so that I could train this big architecture from scratch, like in medical images or just in domains where you don't have many images. So couldn't I somehow come up with a method to create this backbone beforehand? So to create backbone given another dataset. And the simplest variant here is you take a big image dataset, such as ImageNet, and then you train a classifier, like we said, to predict some classes on it. And then because an ImageNet has a lot of images, then this is your backbone. And then whenever you have a different task, you simply take the backbone, transfer it over, and then train the other. Basically, you continue training on the other task. That's called transfer learning. The question is, how do you get a good backbone? So if you train on something like ImageNet, then this is of course a supervised task. You have a very good learning signal, but even ImageNet has like 1 million images. But for example, the internet has many more images. So what you could do is you could train on this much bigger dataset that you collected from the internet. Let's call it internet. But there you don't have labels, right? So what you'll have to resort to is instead of supervised learning is self supervised learning, where you have an image and maybe you rotate it to the right. So here is our cat. You rotate it to the right. And then you have a classifier that predicts that this image was rotated to the right. And then that will become your backbone. These self supervised methods, they work very well. There is a different number of them. For example, MoCo, things like this. And there's also a number of techniques that do supervised pre training and then transfer learning. You can maybe watch my video on big transfer, which is a very large attempt to do to pre train a backbone for visual tasks. All right. Now, you can see right here that the sort of direction is that the more data the better. So that's sort of the idea here that ImageNet is a big data set, we can train a really good backbone. But you know, the internet is an even bigger data set, we don't have labels. So there's a trade off. But we potentially can train an even better visual backbone to then transfer learn with. This paper goes into a different direction. They say, look, if you go in this direction right here, you get more images, but you get less information per image. So with ImageNet, at least you have the label, right per image. But if you simply take a photo of the internet, you don't even have to label you have to resort to self supervised. What if we go into the other direction, and we look for images that have very high quality annotations, but maybe we don't have as many? Can we can we do the same thing? Can we learn good backbones by trading off quality for quantity in this case, and their quantity and quality trade off is they go for descriptions. So they'll go for something like this, where you'll have an image, and you'll have a caption for the image. And so they show these on a line here, semantically dense, semantically sparse, but their task is going to be caption generation. So their back their mod, their task is given an image, I want to produce a caption. And there are data sets that you can train this from in a supervised fashion, which of course, these are very expensive to create. I mean, if you want to create an ImageNet data set, then you have to label each image. But if you want to create a caption data set, that's even harder because human really needs to sit down, look at the image. And in ImageNet, everything is like one class. But here you need to look at the image. And then you'll have to come up with like an adequate description. Here the adequate description is an orange, sorry, an orange and white, an orange and white cat near a plate, and the white cake. Okay. So that's, that's the caption right here. And of course, the caption is ambiguous. So you'll have to collect multiple captions per image. And you'll have to make sure that the humans that do this do a good job and so on. So this these are very, very expensive data sets, but they are very high quality. If you think of what does what does a single label, let's just take ImageNet, ImageNet has a single label per class. Let's say this is cat or cake for that matter. It just sort of gives you very few bits of information. But if you consider the text here, an orange cat and a white cat, an orange and white cat, you know that there is a cat, right? You know that it's one cat, you know what its color is, orange and white, then you know that there is a white cake, right? So you know the other object. And you know the relation, they are near each other. Okay, same for here, a brown and white puppy. So this is one object and the description of the object. There is a there are apples, there is a green lawn, and the relations between them are also clear. The puppy is lying on the green lawn and looking at the apples. So the information in captions is so much more dense than just labels. And that's the that's the backdrop here to say, Hey, can't we can't we do? Can't we pre train a backbone from maybe a small data set, but that has so much information, like a caption date, image caption data set. Okay, so their method is nothing more. They train image captioning, and then they use the visual backbone for transfer learning. So this is the model, there's an image, the image goes into this visual backbone right here, which is a resin at 50. So this is a very, very standard convolutional neural network. And that gives you these features. So these features are seven by seven by 2048. This is the standard output of a resin at 50. And then from this part on, they do a linear projection, such that they can now input it into a language model. So they have visual features. And now they feed those into the language model. And the language model is just a transformer, actually two transformers. So one transformer, they're both autoregressive, one transformer tries to predict the caption in a forward way. And the other transformer tries to predict the caption in a backward way. And that's down here. So in this direction is backward because the caption has been reversed. If you don't know what a transformer is, I've made several videos on transformers. The first one is attention is all you need. And that's sort of the same, the same kind of transformer they use here. So as you can see right here, you have this multi-head attention, the layer normalization attention from the decoder. Now the difference between the original Vasvani attention is all you need transformer. And this one is that in the original transformer, you had, for example, if you had a machine translation task, you would have the French, maybe a French sentence over here. And then you would have the beginnings of German sentence here, right? This is what you have already produced. And now you're asking what should the next word be. And the architecture was such that there is a decoder transformer right here and that there is an encoder transformer that encodes whatever you already had. And then at some point there is this cross attention, right? There is the signal from the decoder going into the encoder and the encoder incorporating that. And then at the end right here, the encoder would predict or the entire transformer would predict what the next word will be. The only difference right here is that the decode this, sorry, I mixed this up. This is the decoder. This is the encoder. The only difference right here is that this encoder is no longer a transformer, but is this ResNet, this ResNet 50. Okay, because now you don't have an image as a, you can think of it like a translation task. You want to translate from images to text. Okay, so your input is going to be an image and the signal is going like it would go in the original transformer into the decoder. It would come from the image. So from these visual features goes here. So in this drawing, this thing is going in here. And then you simply predict the next word and you do it in both directions. And the reason you can do it in both directions here, this wasn't, is not the case, of course, if you have a decoder like a standard transformer task, because you don't need to do inference at this, you just need to do training. And training you can do using teacher forcing. And so you can do this in a bi directional way. You don't need, you don't need this at inference time. So at inference time, you simply cut off this part right here. That's your visual backbone. Okay. And these features here, those are going to be the features that you then train your task on. And sometimes you fine tune this or sometimes you keep it frozen, you can choose that. Alright, so convolutional network to encode the images that gives you features, visual features. Those visual features go into two transformers, both try to predict the caption of the image, one in a forward motion, one in a backward motion. And you train it to predict as accurately as possible the gold standard captions that you have in your data set. That's it. If you train this model well, that means the model can produce accurate captions for these images, which means that it has learned something meaningful about the image to the degree of course, that the original caption that was in your data set was a good descriptive caption. But we're just we're going to assume that the in these data sets, this is the case. Alright, that's what they do. Now, interesting thing here is that in their standard in their standard in their standard setup, they only have one of these transformer layers. So of these things right here, they only have one. And that's like I think it's like 2000 units wide, but or sorry, the hidden dimension is 2000 units or 2048. But they only have one layer. So what that means is that this transformer is not very powerful. So most that you force most of the power to come from the visual encoder, the visual encoder had basically has to do most of the work. And then the transformer is going to simply be a very shallow language model on top of that. And that of course makes your visual backbone even better. Alright, we can pretty much skip the rest. That's the idea. Like that there's nothing more to it. You train this from the beginning, you don't use any pre trained, whatever you train this from scratch. And then you use this. And then the first experiment, they simply train a linear classifier on top of that representation. So they freeze the backbone, and then they use a linear classifier. And they compare this to baselines. So one of the baseline is image net supervised, where you use the same backbone, but you train it on image net in a supervised fashion. Okay, and then you use that backbone to transfer out of the text, it's kind of like what big transfer does, but just on the regular 1000 class image net baseline. Then you have the sort of the unsupervised pre training ones. So moco, so pearly pearly some, I want to go into pearl, but moco is this momentum contrast, which is one of these supervised methods that has been shown to work really, really well. And this is also moco en is trained on image net, but now without the labels, because moco is unsupervised. And moco cocoa is trained on the cocoa data set. And the cocoa data set is what this paper here, the vertex paper uses cocoa is this image captioning data set. Now what's important to note is that cocoa has about 10% only of the images of image net. So it's considerably smaller. Now let's see how these things fair. Right here, you can see on the x axis, the number of images, okay, the number of images that the data set or that the pre training method trains on. Now, of course, some of these are going to be capped because for some data sets, there are just not more images available, right? So they're going to be capped here, the ones that are training on cocoa and the ones that are training on image net are going to be capped here. And you can already see that the vertex outperforms the image net supervised baseline by pretty much when you only give it this many images. Okay, so the way you do it is, in this case, you simply train these models. Now the brown one is when you take one caption per image, but the data set actually has more more than one caption per image. So when you use more than one, you can still boost your performance a bit. And that works way better than when you do the supervised pre training on image net, which would get you here with about the same amount of images. Now, when you use all of image net, you can see here you can get to a similar performance right here, but you have to use a 10 times bigger data set to get there. Right, so this already shows you sort of the advantage here. Now also consider the difference to the unsupervised ones. So if you look at the same amount of images, the unsupervised self supervised baselines are even lower. But if you go to more images, they sort of get closer to image net. And in their own papers, there are there are some evidence that if you self supervised train for long enough, you can actually surpass image net supervised pre training, but I'm not so sure that that's really the case. But you can see here the trade off between higher quality information, but smaller data sets versus lower quality information, but more data per data set. And yeah, if I guess if you were to if you were to pre train these self supervised methods with lots more data in a self supervised manner, they would maybe end up even higher than image net. Now this graph here is sort of the same thing where they also train a linear classifier. And you can see right here that now the image net supervised baseline is outperforming vertex by a lot. So what's happening here? Now this here is actually this is on image net. So the task here that you transfer learn is image net. Here it was like a neutral task Pascal VOC. None of these methods have trained on Pascal. They simply have trained on their own data set. These have trained on cocoa. This has trained on image net. And then they have transfer learned to Pascal. Now, the task is actually the transfer learning task is image net. So naturally, the the thing that was pre trained in a supervised fashion on image net is going to have a huge advantage in this task, because it basically has already learned the task beforehand, whereas the vertex, it has pre trained on cocoa, not on image net. And you can see, if you give it the same amount of images for pre training, it can actually it's it's fairly close to the image net baseline. So that's pretty respectable right there. Now, again, of course, if you use more images on the same data set that you then train for, then of course, the the image that baseline is going to outperform it. But so pretty cool to see here that in this smaller image regime, and also consider this down here, if you go even an order of magnitude lower, it's really shining that if you have higher quality information, and you make use of it, you don't need as many images. And now we knew this for a long time. But this now is showing the same for transfer learning for visual transfer learning. So this was when we froze the backbone. And then we trained a linear classifier on top, they go, they make a short excursion here and show how different parts of their model affect their final performance. And they find that, for example, the by captioning, which I believe is the is forward and backward captioning significantly helps, for example, compared to only forward captioning. And they also find that it's significantly outperforms other pre training tasks that they could do. And they also investigate whether how big their models should be. So here, this is their baseline model. Oh, I was I was wrong, actually, they the it's one layer of with 1024. You can see as you make the layer bigger and bigger, that generally helps. But I guess they decided against it because the gains are too little to, to afford to make it worth. And also if you make the network deeper here, you make the transformer have more layers, the performance goes up. But again, the gains are marginal. So I guess they're going to leave it away. So their baseline, as you can see, is these resin at 50 with the one layer of 1024 size. So this is now the last task, it's the fine tuning task. So this is what most people would do is they would train a backbone, and then they would fine tune it on a different data set on or on a different task where they don't have much labels. And here the situation looks a bit different. So if you look at, for example, a task on cocoa, so there are several tasks on cocoa, one of them is image captioning, which they use for peritra for pre training. If you do other tasks on cocoa, you can see right here that compared to the supervised baseline, this vertex, it performs about the same or maybe a bit worse. But what you can see is it performs significantly better than, for example, moco that was only trained on cocoa. So again, this shows that if you have the same data set, higher quality information makes it worth it. And it's even better, as you can see, on moco that was trained on image net is just not quite as good as the supervised baseline. But all of them, of course, are better than just a randomly initialized network that is trained from scratch. I mean, that's the entire point of transfer learning, that you are better than simply learning from scratch. And this shows throughout this experiment, except in this LVS masking task, where they do outperform the other the other things, the other methods significantly. Now the lower numbers on this tasks task also means that the task is harder than these tasks right here. And therefore, there are more gains to be made. And therefore, you could hypothesize that the bigger, the more quality information that you input can be used in a better way. So maybe more complex, also, the more complex a task is might also have an influence on how well the transfer learning works, if you come from a high quality transfer learning task versus a low quality transfer learning tasks. Yeah, so the lastly compare here with the again with Pascal VOC object detection, and these iNaturalist classification, where I believe this is also a transfer learning task with fine tuning. And as you can see, they can also hold up against the supervised baseline, or even outperform it at sometimes the green triangles mean that they outperform it by a significant margin. But then on this task right here, they again lag behind. So I think the point of the paper isn't really to show that that this is the best thing ever. But the point of the paper is to show that you can go about pre trainings, basically, the the common assumption is that you need more and more and more and more data for your model to learn about the data set. And they conclude here, no, actually, you can do with with very few data points as long as they have high quality annotations. Okay, so I think that's the point of the of the paper that and they don't always outperform the other baselines and whatnot. But they keep, they keep the performance the same, which basically means that this is an option. Here is a pretty cool result where they visualize the attention of their image captioning model, because they train an image captioning model. And you can really see that the image captioning model learns something meaningful about the image. So when it's a bird flying, the attention is mainly on the bird, as you can see, then over the the attention widens out over the image, air. So over the air, the attention is here in the sky and on the on the ocean. And then it goes near the ocean. And then the attention is on the ocean itself. As you can see, so they have a bunch of these images and they're they're pretty cool here a dog, so focused on the dog riding on and then you can see the attention going down because on is riding on means probably there's something below the dog. A surfboard. Now the attention is fully on the surfboard in. So as soon as you say in the attention, as you can see, it widens out. So I think that's that's fairly cool, fairly cool demonstration that the model understands sort of the the in relation, namely, if it is focused on something, and that something is in something else, then it widens the attention out to see what it is in, okay, the ocean, and then it focuses the attention on the ocean. So that's, that's a pretty, that's a pretty cool result. I guess we already knew this because we could train image captioning models before. It's just to show that it actually makes sense to use them as a pre training task for backbones. Now, what's the future of this, the authors here in their introduction, they make a claim that this has a good future because they here they only train on this small data set, right, it's smaller than image net, as you can see here, and they already get the same performance as if you train on the whole image net data set in a supervised fashion. Of course, they're also supervised, but they have 10 times less images. And they they say something to the effect of you do you know, it would be pretty easy to collect more data for this task, because the internet is full of images. And mostly these images have like some text with them. They, you know, they have these descriptions or they have text around it, people write something about the images, you could like mine Twitter, and then the responses when someone posts an image might tell you something about the image. But this definitely counteracts their this definitely counteracts their notion that these are very high quality labels, right? Their entire point here was that these annotations, these, these data sets with these these image captioning data sets like Coco, they have very, very high quality annotations. So this this text here is very high quality is really a descriptive text of the image that tries to capture what a human can see visually in the image. And as soon as you go out to the internet and collect a text around images, that's not going to be the case that information is again going to be quite low quality. And so I doubt that the performance here would hold up or that the claim you can easily, you know, you can easily create more data for this task holds up. So that's a bit my worry about the future of this, but it's definitely cool and definitely shows these quality quantity trade off very well. Alright, that was my two cents to the paper. I invite you to read it and tell me in the comments what you think about it and I'll see you next time.
 Hi there! Today we're looking at Vertex Learning Visual Representations from Textual Annotations
 by Karen Desai and Justin Johnson of the University of Michigan. So this paper at its core is pretty
 simple. On a high level it proposes to take the task of image captioning, which is where
 you're given an image and you're asked to produce a caption for the image, and basically
 train a model to do this, and then just take the visual part of it as a baseline to transfer
 learn on other visual tasks. And that appears to work surprisingly well if you don't have
 much data. So if you don't have much data to pre-train on, this appears to work very
 well. Alright, as always, if you like content like this, then consider sharing it out, subscribing
 to the channel, or tell me what you think in the comments. So as I already said, the
 idea here is pretty simple. So people have been looking for pre-training tasks for visual
 tasks. So a visual task is anything where the input is an image, and then you usually
 have some sort of neural network that processes the image, and then at the end you can have
 many things. So you could have a classifier that classifies the image into one of many
 classes. If you know ImageNet, that's a thing. So if there's a cat here, then the ImageNet
 classifier here would say cat. Or you could have something like an object detector that
 tries to predict on the image where the cat is, like with a bounding box. You could have
 a semantic segmentation where it's like all of these pixels here are cats, and maybe all
 of these pixels here are sky. And so it labels every pixel. There's many visual tasks that
 you can formulate, and they all sort of share the same architecture. And specifically, they
 all share this part right here. If you will, this is the visual encoder. It's usually a
 convolutional neural network. And what's really different between the tasks is mostly this
 last part here that does the actual task. But this is often called the backbone. So
 this is the backbone. And the idea now is, if I have a bunch of these tasks, sometimes
 I don't have many labels for these tasks. I don't have many labeled images so that I
 could train this big architecture from scratch, like in medical images or just in domains
 where you don't have many images. So couldn't I somehow come up with a method to create
 this backbone beforehand? So to create backbone given another dataset. And the simplest variant
 here is you take a big image dataset, such as ImageNet, and then you train a classifier,
 like we said, to predict some classes on it. And then because an ImageNet has a lot of
 images, then this is your backbone. And then whenever you have a different task, you simply
 take the backbone, transfer it over, and then train the other. Basically, you continue training
 on the other task. That's called transfer learning. The question is, how do you get
 a good backbone? So if you train on something like ImageNet, then this is of course a supervised
 task. You have a very good learning signal, but even ImageNet has like 1 million images.
 But for example, the internet has many more images. So what you could do is you could
 train on this much bigger dataset that you collected from the internet. Let's call it
 internet. But there you don't have labels, right? So what you'll have to resort to is
 instead of supervised learning is self supervised learning, where you have an image and maybe
 you rotate it to the right. So here is our cat. You rotate it to the right. And then
 you have a classifier that predicts that this image was rotated to the right. And then that
 will become your backbone. These self supervised methods, they work very well. There is a different
 number of them. For example, MoCo, things like this. And there's also a number of techniques
 that do supervised pre training and then transfer learning. You can maybe watch my video on
 big transfer, which is a very large attempt to do to pre train a backbone for visual
 tasks. All right. Now, you can see right here that the sort of direction is that the more
 data the better. So that's sort of the idea here that ImageNet is a big data set, we can
 train a really good backbone. But you know, the internet is an even bigger data set, we
 don't have labels. So there's a trade off. But we potentially can train an even better
 visual backbone to then transfer learn with. This paper goes into a different direction.
 They say, look, if you go in this direction right here, you get more images, but you get
 less information per image. So with ImageNet, at least you have the label, right per image.
 But if you simply take a photo of the internet, you don't even have to label you have to resort
 to self supervised. What if we go into the other direction, and we look for images that
 have very high quality annotations, but maybe we don't have as many? Can we can we do the
 same thing? Can we learn good backbones by trading off quality for quantity in this case,
 and their quantity and quality trade off is they go for descriptions. So they'll go for
 something like this, where you'll have an image, and you'll have a caption for the image.
 And so they show these on a line here, semantically dense, semantically sparse, but their task
 is going to be caption generation. So their back their mod, their task is given an image,
 I want to produce a caption. And there are data sets that you can train this from in
 a supervised fashion, which of course, these are very expensive to create. I mean, if you
 want to create an ImageNet data set, then you have to label each image. But if you want
 to create a caption data set, that's even harder because human really needs to sit down,
 look at the image. And in ImageNet, everything is like one class. But here you need to look
 at the image. And then you'll have to come up with like an adequate description. Here
 the adequate description is an orange, sorry, an orange and white, an orange and white cat
 near a plate, and the white cake. Okay. So that's, that's the caption right here. And
 of course, the caption is ambiguous. So you'll have to collect multiple captions per image.
 And you'll have to make sure that the humans that do this do a good job and so on. So this
 these are very, very expensive data sets, but they are very high quality. If you think
 of what does what does a single label, let's just take ImageNet, ImageNet has a single
 label per class. Let's say this is cat or cake for that matter. It just sort of gives
 you very few bits of information. But if you consider the text here, an orange cat and
 a white cat, an orange and white cat, you know that there is a cat, right? You know
 that it's one cat, you know what its color is, orange and white, then you know that there
 is a white cake, right? So you know the other object. And you know the relation, they are
 near each other. Okay, same for here, a brown and white puppy. So this is one object and
 the description of the object. There is a there are apples, there is a green lawn, and
 the relations between them are also clear. The puppy is lying on the green lawn and looking
 at the apples. So the information in captions is so much more dense than just labels. And
 that's the that's the backdrop here to say, Hey, can't we can't we do? Can't we pre train
 a backbone from maybe a small data set, but that has so much information, like a caption
 date, image caption data set. Okay, so their method is nothing more. They train image captioning,
 and then they use the visual backbone for transfer learning. So this is the model, there's
 an image, the image goes into this visual backbone right here, which is a resin at 50.
 So this is a very, very standard convolutional neural network. And that gives you these features.
 So these features are seven by seven by 2048. This is the standard output of a resin at
 50. And then from this part on, they do a linear projection, such that they can now
 input it into a language model. So they have visual features. And now they feed those into
 the language model. And the language model is just a transformer, actually two transformers.
 So one transformer, they're both autoregressive, one transformer tries to predict the caption
 in a forward way. And the other transformer tries to predict the caption in a backward
 way. And that's down here. So in this direction is backward because the caption has been reversed.
 If you don't know what a transformer is, I've made several videos on transformers. The first
 one is attention is all you need. And that's sort of the same, the same kind of transformer
 they use here. So as you can see right here, you have this multi-head attention, the layer
 normalization attention from the decoder. Now the difference between the original Vasvani
 attention is all you need transformer. And this one is that in the original transformer,
 you had, for example, if you had a machine translation task, you would have the French,
 maybe a French sentence over here. And then you would have the beginnings of German sentence
 here, right? This is what you have already produced. And now you're asking what should
 the next word be. And the architecture was such that there is a decoder transformer right
 here and that there is an encoder transformer that encodes whatever you already had. And
 then at some point there is this cross attention, right? There is the signal from the decoder
 going into the encoder and the encoder incorporating that. And then at the end right here, the encoder
 would predict or the entire transformer would predict what the next word will be. The only
 difference right here is that the decode this, sorry, I mixed this up. This is the decoder.
 This is the encoder. The only difference right here is that this encoder is no longer a transformer,
 but is this ResNet, this ResNet 50. Okay, because now you don't have an image as a,
 you can think of it like a translation task. You want to translate from images to text.
 Okay, so your input is going to be an image and the signal is going like it would go in
 the original transformer into the decoder. It would come from the image. So from these
 visual features goes here. So in this drawing, this thing is going in here. And then you
 simply predict the next word and you do it in both directions. And the reason you can
 do it in both directions here, this wasn't, is not the case, of course, if you have a
 decoder like a standard transformer task, because you don't need to do inference at
 this, you just need to do training. And training you can do using teacher forcing. And so you
 can do this in a bi directional way. You don't need, you don't need this at inference time.
 So at inference time, you simply cut off this part right here. That's your visual backbone.
 Okay. And these features here, those are going to be the features that you then train your
 task on. And sometimes you fine tune this or sometimes you keep it frozen, you can choose
 that. Alright, so convolutional network to encode the images that gives you features,
 visual features. Those visual features go into two transformers, both try to predict
 the caption of the image, one in a forward motion, one in a backward motion. And you
 train it to predict as accurately as possible the gold standard captions that you have in
 your data set. That's it. If you train this model well, that means the model can produce
 accurate captions for these images, which means that it has learned something meaningful
 about the image to the degree of course, that the original caption that was in your data
 set was a good descriptive caption. But we're just we're going to assume that the in these
 data sets, this is the case. Alright, that's what they do. Now, interesting thing here
 is that in their standard in their standard in their standard setup, they only have one
 of these transformer layers. So of these things right here, they only have one. And that's
 like I think it's like 2000 units wide, but or sorry, the hidden dimension is 2000 units
 or 2048. But they only have one layer. So what that means is that this transformer is
 not very powerful. So most that you force most of the power to come from the visual
 encoder, the visual encoder had basically has to do most of the work. And then the transformer
 is going to simply be a very shallow language model on top of that. And that of course makes
 your visual backbone even better. Alright, we can pretty much skip the rest. That's the
 idea. Like that there's nothing more to it. You train this from the beginning, you don't
 use any pre trained, whatever you train this from scratch. And then you use this. And then
 the first experiment, they simply train a linear classifier on top of that representation.
 So they freeze the backbone, and then they use a linear classifier. And they compare
 this to baselines. So one of the baseline is image net supervised, where you use the
 same backbone, but you train it on image net in a supervised fashion. Okay, and then you
 use that backbone to transfer out of the text, it's kind of like what big transfer does,
 but just on the regular 1000 class image net baseline. Then you have the sort of the unsupervised
 pre training ones. So moco, so pearly pearly some, I want to go into pearl, but moco is
 this momentum contrast, which is one of these supervised methods that has been shown to
 work really, really well. And this is also moco en is trained on image net, but now without
 the labels, because moco is unsupervised. And moco cocoa is trained on the cocoa data
 set. And the cocoa data set is what this paper here, the vertex paper uses cocoa is this image
 captioning data set. Now what's important to note is that cocoa has about 10% only of
 the images of image net. So it's considerably smaller. Now let's see how these things fair.
 Right here, you can see on the x axis, the number of images, okay, the number of images
 that the data set or that the pre training method trains on. Now, of course, some of
 these are going to be capped because for some data sets, there are just not more images
 available, right? So they're going to be capped here, the ones that are training on cocoa
 and the ones that are training on image net are going to be capped here. And you can already
 see that the vertex outperforms the image net supervised baseline by pretty much when
 you only give it this many images. Okay, so the way you do it is, in this case, you simply
 train these models. Now the brown one is when you take one caption per image, but the data
 set actually has more more than one caption per image. So when you use more than one,
 you can still boost your performance a bit. And that works way better than when you do
 the supervised pre training on image net, which would get you here with about the same
 amount of images. Now, when you use all of image net, you can see here you can get to
 a similar performance right here, but you have to use a 10 times bigger data set to
 get there. Right, so this already shows you sort of the advantage here. Now also consider
 the difference to the unsupervised ones. So if you look at the same amount of images,
 the unsupervised self supervised baselines are even lower. But if you go to more images,
 they sort of get closer to image net. And in their own papers, there are there are some
 evidence that if you self supervised train for long enough, you can actually surpass
 image net supervised pre training, but I'm not so sure that that's really the case. But
 you can see here the trade off between higher quality information, but smaller data sets
 versus lower quality information, but more data per data set. And yeah, if I guess if
 you were to if you were to pre train these self supervised methods with lots more data
 in a self supervised manner, they would maybe end up even higher than image net. Now this
 graph here is sort of the same thing where they also train a linear classifier. And you
 can see right here that now the image net supervised baseline is outperforming vertex
 by a lot. So what's happening here? Now this here is actually this is on image net. So
 the task here that you transfer learn is image net. Here it was like a neutral task Pascal
 VOC. None of these methods have trained on Pascal. They simply have trained on their
 own data set. These have trained on cocoa. This has trained on image net. And then they
 have transfer learned to Pascal. Now, the task is actually the transfer learning task
 is image net. So naturally, the the thing that was pre trained in a supervised fashion
 on image net is going to have a huge advantage in this task, because it basically has already
 learned the task beforehand, whereas the vertex, it has pre trained on cocoa, not on image
 net. And you can see, if you give it the same amount of images for pre training, it can
 actually it's it's fairly close to the image net baseline. So that's pretty respectable
 right there. Now, again, of course, if you use more images on the same data set that
 you then train for, then of course, the the image that baseline is going to outperform
 it. But so pretty cool to see here that in this smaller image regime, and also consider
 this down here, if you go even an order of magnitude lower, it's really shining that
 if you have higher quality information, and you make use of it, you don't need as many
 images. And now we knew this for a long time. But this now is showing the same for transfer
 learning for visual transfer learning. So this was when we froze the backbone. And then
 we trained a linear classifier on top, they go, they make a short excursion here and show
 how different parts of their model affect their final performance. And they find that,
 for example, the by captioning, which I believe is the is forward and backward captioning
 significantly helps, for example, compared to only forward captioning. And they also
 find that it's significantly outperforms other pre training tasks that they could do. And
 they also investigate whether how big their models should be. So here, this is their baseline
 model. Oh, I was I was wrong, actually, they the it's one layer of with 1024. You can see
 as you make the layer bigger and bigger, that generally helps. But I guess they decided
 against it because the gains are too little to, to afford to make it worth. And also if
 you make the network deeper here, you make the transformer have more layers, the performance
 goes up. But again, the gains are marginal. So I guess they're going to leave it away.
 So their baseline, as you can see, is these resin at 50 with the one layer of 1024 size.
 So this is now the last task, it's the fine tuning task. So this is what most people would
 do is they would train a backbone, and then they would fine tune it on a different data
 set on or on a different task where they don't have much labels. And here the situation looks
 a bit different. So if you look at, for example, a task on cocoa, so there are several tasks
 on cocoa, one of them is image captioning, which they use for peritra for pre training.
 If you do other tasks on cocoa, you can see right here that compared to the supervised
 baseline, this vertex, it performs about the same or maybe a bit worse. But what you can
 see is it performs significantly better than, for example, moco that was only trained on
 cocoa. So again, this shows that if you have the same data set, higher quality information
 makes it worth it. And it's even better, as you can see, on moco that was trained on image
 net is just not quite as good as the supervised baseline. But all of them, of course, are
 better than just a randomly initialized network that is trained from scratch. I mean, that's
 the entire point of transfer learning, that you are better than simply learning from scratch.
 And this shows throughout this experiment, except in this LVS masking task, where they
 do outperform the other the other things, the other methods significantly. Now the lower
 numbers on this tasks task also means that the task is harder than these tasks right
 here. And therefore, there are more gains to be made. And therefore, you could hypothesize
 that the bigger, the more quality information that you input can be used in a better way.
 So maybe more complex, also, the more complex a task is might also have an influence on
 how well the transfer learning works, if you come from a high quality transfer learning
 task versus a low quality transfer learning tasks.
 Yeah, so the lastly compare here with the again with Pascal VOC object detection, and
 these iNaturalist classification, where I believe this is also a transfer learning task
 with fine tuning. And as you can see, they can also hold up against the supervised baseline,
 or even outperform it at sometimes the green triangles mean that they outperform it by
 a significant margin. But then on this task right here, they again lag behind. So I think
 the point of the paper isn't really to show that that this is the best thing ever. But
 the point of the paper is to show that you can go about pre trainings, basically, the
 the common assumption is that you need more and more and more and more data for your model
 to learn about the data set. And they conclude here, no, actually, you can do with with very
 few data points as long as they have high quality annotations. Okay, so I think that's
 the point of the of the paper that and they don't always outperform the other baselines
 and whatnot. But they keep, they keep the performance the same, which basically means
 that this is an option. Here is a pretty cool result where they visualize the attention
 of their image captioning model, because they train an image captioning model. And you can
 really see that the image captioning model learns something meaningful about the image.
 So when it's a bird flying, the attention is mainly on the bird, as you can see, then
 over the the attention widens out over the image, air. So over the air, the attention
 is here in the sky and on the on the ocean. And then it goes near the ocean. And then
 the attention is on the ocean itself. As you can see, so they have a bunch of these images
 and they're they're pretty cool here a dog, so focused on the dog riding on and then you
 can see the attention going down because on is riding on means probably there's something
 below the dog. A surfboard. Now the attention is fully on the surfboard in. So as soon as
 you say in the attention, as you can see, it widens out. So I think that's that's fairly
 cool, fairly cool demonstration that the model understands sort of the the in relation, namely,
 if it is focused on something, and that something is in something else, then it widens the attention
 out to see what it is in, okay, the ocean, and then it focuses the attention on the ocean.
 So that's, that's a pretty, that's a pretty cool result. I guess we already knew this
 because we could train image captioning models before. It's just to show that it actually
 makes sense to use them as a pre training task for backbones. Now, what's the future
 of this, the authors here in their introduction, they make a claim that this has a good future
 because they here they only train on this small data set, right, it's smaller than image
 net, as you can see here, and they already get the same performance as if you train on
 the whole image net data set in a supervised fashion. Of course, they're also supervised,
 but they have 10 times less images. And they they say something to the effect of you do
 you know, it would be pretty easy to collect more data for this task, because the internet
 is full of images. And mostly these images have like some text with them. They, you know,
 they have these descriptions or they have text around it, people write something about
 the images, you could like mine Twitter, and then the responses when someone posts an image
 might tell you something about the image. But this definitely counteracts their this
 definitely counteracts their notion that these are very high quality labels, right? Their
 entire point here was that these annotations, these, these data sets with these these image
 captioning data sets like Coco, they have very, very high quality annotations. So this
 this text here is very high quality is really a descriptive text of the image that tries
 to capture what a human can see visually in the image. And as soon as you go out to the
 internet and collect a text around images, that's not going to be the case that information
 is again going to be quite low quality. And so I doubt that the performance here would
 hold up or that the claim you can easily, you know, you can easily create more data
 for this task holds up. So that's a bit my worry about the future of this, but it's definitely
 cool and definitely shows these quality quantity trade off very well. Alright, that was my
 two cents to the paper. I invite you to read it and tell me in the comments what you think
 about it and I'll see you next time.