Big Self-Supervised Models are Strong Semi-Supervised Learners (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper cnn resnet simclr simclr2 simclrv2 simclr v2 v2 hinton geoff brain wide deep convolutional convolutions self-supervised contrastive moco momentum projection semi-supervised unsupervised distillation teacher student
This paper proposes SimCLRv2 and shows that semi-supervised learning benefits a lot from self-supervised pre-training. And stunningly, that effect gets larger the fewer labels are available and the more parameters the model has.

OUTLINE:
0:00 - Intro & Overview
1:40 - Semi-Supervised Learning
3:50 - Pre-Training via Self-Supervision
5:45 - Contrastive Loss
10:50 - Retaining Projection Heads
13:10 - Supervised Fine-Tuning
13:45 - Unsupervised Distillation & Self-Training
18:45 - Architecture Recap
22:25 - Experiments
34:15 - Broader Impact

Paper: https://arxiv.org/abs/2006.10029
Code: https://github.com/google-research/simclr

Abstract:
One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to most previous approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of a big (deep and wide) network during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLR), supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9\% ImageNet top-1 accuracy with just 1\% of the labels (≤13 labeled images per class) using ResNet-50, a 10× improvement in label efficiency over the previous state-of-the-art. With 10\% of labels, ResNet-50 trained with our method achieves 77.5\% top-1 accuracy, outperforming standard supervised training with all of the labels.

Authors: Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we'll look at big self-supervised models are strong semi-supervised learners by Ting Chen, Simon Kornblith, Kevin Swirsky, Mohamed Nourouzi and Jeffrey Hinton of Google Brain. So this paper on a high level, it's also known as Sinclair v2, demonstrates that if you want to do semi-supervised learning, that you're very well served by starting out with self-supervised learning, and then doing fine tuning much like NLP models do, rather than the kind of semi-supervised approach that image tasks had so far. And they present this Sinclair v2, which is an improvement over the Sinclair approach to self-supervised pre-training, and they demonstrate it outperforms a lot of the baselines. Alright, so if you like content like this, don't forget to share it out, and leave a like and tell me what you think in the comments. So this paper, it sort of is kind of a club together thing of different things. So they present this new method, like this Sinclair v2, which is a modification of Sinclair, and we'll go over that, but they also try to make like a scientific claim, namely that somehow bigger models are better for this pathway of learning, and we'll try to untangle all of these things. So first of all, we're in the semi-supervised learning regime right here. This basically means that you have a data set, and you only have labels for a part of that data set. So this could be like here, the bottom 10% or so, because labels might be expensive to get. And so you only have a few of them, but you have much more data that's unlabeled. Now sometimes this problem is formulated as this here is your data set, and then this here is like a different data set, but one that's close enough such that you can learn from it. And that's usually in NLP. You'll have your data set is like a sentiment classification task, but you have all of Wikipedia that is not labeled, but it's just text. So you can sort of pre-train on it. In this case, we'll be in a situation where we'll artificially construct a small data set. So this entire thing here is going to be the ImageNet data set, and this right here is going to be our labeled portion, like we have labels. Now usually one has labels for ImageNet as well, but we artificially restrict ourselves to simulate a situation where we have lots of data and we only have a fixed budget. So we can only, because to obtain labels, oftentimes you have to ask humans to label images. And let's say we are a company and we've collected this big data set, but we only have like maybe 500 bucks on Amazon Mechanical Turk, and we only managed to get a very like 1% of our data set labeled. Now we're in the regime of semi-supervised learning. This is slightly different from what NLP does. As I said, in NLP, usually assume you have different data sets, the large one being a different distribution, and in the semi-supervised regime, you often assume that it is actually the same data distribution, but you only have labels for some of them. But there should be a fair bit of overlap between the two things. So I've recently made a video about OpenAI's ImageGPT that kind of goes into the same direction as this work right here that basically says pre-training on unlabeled data, like this whole data set without the labels, can be a very good preconditioner for fine tuning later. And this paper says the same thing. So basically, in the good old days, what you would do is you would devise a method that somehow takes, you know, takes in a device, a method that takes in a mini batch. And in the mini batch, you'd have your data samples, and then some of them would be labeled, right here, you'd have a Y and here you'd have a Y, but most of them would be not labeled. And you'd have like some sort of loss function that would put special weight on the ones that are labeled or somehow handle these ones that are unlabeled in a way, you might be doing like some sort of a consistency loss such that if they are very nearest near neighbors to these in the feature space, they should have similar labels or things like this. So these semi supervised methods, they basically try to solve the problem at once. But while taking data that is labeled and not labeled, this paper goes into a different direction. This paper says, first, we should, it's actually three stages right here, and they have a diagram, so I don't need to draw. They have a three stage approach. Three stages. The one on the left is unsupervised pre training. So they say, let's forget about the labels right now, even like your unlabeled data. So even the data where we have the labels, let's forget about the labels. And let's just do unsupervised pre training. Now unsupervised pre training in this kind of setting is also known as self supervised pre training. And this first stage is done using a contrastive loss, and that's very similar to sim clear to this contrastive loss. So what you'll do, and they describe it very, very well here. So what you'll do is given a randomly sampled mini batch of images, each image is augmented twice using random crop color distortion and Gaussian blur, creating two views of the same example. Okay, so you have an image in your mini batch. Each image you take and you make two versions of it. And each version you crop, you random crop somewhere. So version one could be random cropped here. Version two could be random cropped here. And then you put some Gaussian blur on it and so on. So a little bit of, as you can see, random crop color distortion, Gaussian blur. So what you'll want is two different versions of the same image. Each of these versions has been augmented in a different way, cropped in a different way, blurred in a different way. It's two slightly different versions of the same image. And now you want to enforce, you want to put this through your network. So ultimately, as you can see on the right side here, what you want to end up is a network. And then, okay, we'll forget about this right now. What you want to train is this network right here, actually including these projection layers. We'll get to them later. This is the network that you want to train. So you want to put, you take your unlabeled data, you take an image, you make two versions of it. And you put those through the network, right, until the end right here. So you'll get Z1, Z2. These are the outputs of the network for the two images. And then what you want to do is you want to take another image that's not this image, and also put it through the network, maybe also augmented first. And then you have Z3. So now you have the outputs of two things that are supposed to come from the same image and one thing that's supposed to come from a different image. And now your loss is simply going to be make those two things close together and push those two things apart, or those three actually. So the loss, and this is the contrastive loss of self supervised learning. As you know, you don't need any labels right here. You simply say the things that come from the same image should be close together. And the things that come from different images should be far apart. And this relies heavily on these data augmentations that you do right here. They also employ some other tricks like the momentum encoder from MoCo, from momentum contrast and so on. But this is the main part. So you can pull a lot of strings here to get like another percent of performance. But ultimately, they won't see the similarity of ZI and ZJ, which are the outputs of the same image to be close together. And then this down here, they want to be far apart, ZI with ZK, where K is all the other images. Okay. And you can do this in a mini batch fashion. So this is self supervised learning. And the reason why you do this is you don't need labels. And it tends, we know it tends to give very, very good representations. So I'm past that. So what this network here will learn will be very good for some reason. We still don't exactly know why combining augmentation with the self supervised loss with contrastive loss, for example, gives such good performance. There have been papers recently that modify the loss and so on. But it's not super well understood yet. But if you do it like this, the network here will give you already very, very good representation. And we know this because we can take a network like this and then simply train a linear classifier on top of that on a data set and achieve very, very good performance. And mind you, you have trained it with unlabeled data, right? So the network has never been trained to solve like ImageNet classification. It has simply been trained to look at the pictures and determine if two versions of a picture come from the same picture or from different pictures. And now, if you simply train a linear classifier on top of these representations, you're doing extremely well already. So we know these representations, they actually learn something about these images. So that's the first part. Then stage two, let's cancel all that. Stage two is you want to do supervised fine tuning. Now you already see that the arrow here coming out is not this task agnostic big CNN. The arrow is actually coming out of those yellow boxes. And the yellow boxes are these projection heads. So in the original SimClear paper, what they did was they wanted originally, they wanted to train this network right here. This is like a ResNet-50. It's pretty standard in these kind of self-supervised approaches and so on to train or these few label approaches to train a standardized network. And this is like a ResNet-50. So in the original SimClear paper, they said we want to make ResNet-50 as strong as possible. But in order to do this loss right here, we are going to attach this projection head just to because the dimensionality here I think is like 2048. And we want to do this inner product in a lower dimension of like maybe 256 or so. So these are just multi-layer perceptrons. These are just fully connected layers that compress the representation down to that. And once we're done with the unsupervised pre-training, we're going to throw those away, right? And this ResNet is the thing that we really care about. Now here they claim, OK, it actually works better. And they have experiments to prove this or to show this if you use one, if you actually leave one of these layers here. So in the end, I guess they converge on three projection head layers. And then they only throw away the top two. And like they make this big deal out of the fact where, you know, I can just call this part right here now the encoder. And I don't so I don't know exactly like I don't see the giant deal here. Like you just made your network one layer bigger. And now you consider that to be your encoder. And the projection head is now two layers. And that will be much easier than calling the projection head three layers. But we leave one layer and we train from the middle layer. In any case, they have this layer, additional layer right here compared to the old Sinclair. And then the representation of that goes into supervised fine tuning. Now, this is pretty easy. This is exactly what it sounds like. So now you use only only the data set that has labels. So the part of the data set that has labels, and you do the fine tuning and fine tuning is simply supervised learning. You train this network in a supervised fashion on that small fraction of data that has cloud class labels. And that already performs pretty well. And they show this in experiments. But then you can go a step further and do what's known as distillation or self training. And what's distillation or self training? It's so distillation is when you have a network that you call the teacher network. And that network has been trained to do some classification maybe into three classes pretty, pretty well. Okay. But now this is very large and you want maybe a smaller model. So you just want like this tiny model because you want to ship it on a mobile device, right? But it's also supposed to do this. And you know that if you just directly train this, which is called the student model, it doesn't perform as well as the teacher model. There is a better way. If you have the teacher model, you can sort of transfer the knowledge to the student model. You can distill the knowledge. And how do you do that? You do that by, so what would you do in supervised training? In supervised training, you would take an image, put it in, and then put the label that comes along with the image. You put it up here and you compare the output to the label and that gives you the loss function. Right? So you do that right here. If you distill, you put the image into both. Now the teacher is already trained. So its output will be a distribution over classes. It won't be a single label. It will be like, okay, 90% class one, 10% class two, 0% class three, something like this. And now you take this as like a pseudo label, this entire distribution, and you put it here and you compare the output of the student to that of the teacher and that's your loss function. So this kind of, the teacher might have learned to put some nuance into the classification to say, well, I'm pretty sure this is class one, but I'm not 100% sure. And it can transfer that knowledge to the student. And that makes the student better than had you just trained it from the beginning from, with just the labels. Right? So this is distillation and you can do this even what they call self distillation here or self training. So apparently this even helps if the teacher is, if the student model is the same as the teacher model. Now why does it help in this case? And I think it is not exactly the case in this case because they always say their teacher model has this extra projection layer. Right? And then the student model doesn't have that even if they do self training. But why does it help in this case? I mean, it's, it's kind of shocking and I'm pretty sure it helps in any case, but in this particular case it helps because now you're using the unlabeled data again. So you have a teacher model and the teacher model is trained first using unsupervised like this is the teacher model right here using unsupervised training. Then the teacher model is further fine tuned on the small data. Right? So it is now already pretty good at the task, but how can you get a student model that's even better than the teacher model? It's by using again this unlabeled data. You have this giant amount of data. So what you'll do is you take an image from the unlabeled data and you ask the teacher model, teacher model, what do you think about that image? Right? And the teacher model will give you a prediction. Like let's say again, this 90%, 10%, 0% and then you take the student model, you input that image and you compare its output to what the teacher said. So this combines the teacher model. You freeze the teacher model, right? The teacher model is only trained until here. You take it from here. The student model is now able to take basically the teacher. It takes everything that the teacher model knows, not only about this data, but about all the data. So it kind of gets to ask the teacher model, what do you think about this? What do you think about this? What do you think about this? And it can incorporate all that knowledge about all of this unlabeled data. And that's why the student model here in the end, if it's the same size, will probably end up even better than the teacher model. So distillation, I think also is still kind of a mystery of why you get a better model or, I mean, to make it smaller, if you make it a lot smaller, usually you don't end up with a better model, but you end up with a pretty good model that you couldn't have gotten by just training the small model. So that's already pretty cool. But why you get a better model when they're the same size, I don't think that's well understood yet. So that's the three stage approach. So recap, first, use all of the data without labels to do unsupervised or self supervised contrastive pre-training. Second, use only the data that has labels to do fine tuning. Third, either distill the learned classifier to a smaller model or distill it to a model of the same size. Then in both cases, you would again use the unlabeled, all of the unlabeled data. And that's the three step approach. That's SEMCLEAR v2 in all of its form. So they go into fine tuning right here. And yeah, so they say again, we elaborate with a three layer projection head. So that's the three layer projection head. This here is the output of ResNet-50, where Sigma is a ReLU non-linearity and we ignore the bias term for brevity, blah, blah, blah, blah, blah. So they contrast this here. For fine tuning, SEMCLEAR uses this right here, which is just, it's basically just a classifier on top of the output of the ResNet-50. This is fine tuning from the input layer of the projection head. To fine tune from the first layer of the projection head, we have a new encoder function as this, which is ResNet followed by fully connected layers. And you see they take the ResNet-50 output and they ship it through the first projection layer and then there is a task specific classifier. Now, again, why, I don't even see why they make like this ginormous deal out of it, especially, especially since the last layer of the ResNet-50. I'm not, okay, here is, I'm not entirely sure, but are they taking the log? No, they're probably not taking the log. It's okay. But it's, yeah, it's just weird. Like is there even a non-linearity at the end right here? Or is this really just like two matrix multiplications in a row, which I'm going to guess there's a big chance that that's the case, that the last layer of this encoder is actually not even followed by non-linearity and therefore you'll just kind of make the dimension different. And I don't see why you can't just incorporate this into the model and have to like say it over and over again that this is a new special thing, right? Again, this is equivalent of tuning from a middle layer of the projection head instead of the output layer. Okay, you just make your model a bit bigger. Yeah. So the third step is self-training or knowledge distillation. And they give two variants right here. This variant, as you can see here, this is just the cross entropy. But instead of having labels right here, Y, you have what the teacher model thinks Y is given X. Okay, that's cross entropy, but not with the true labels, but with the output of the teacher model. And you can even mix that. So you can, as you can see right here, you can mix this with an actual supervised loss. So this would be the supervised loss, whatever. Yeah, I guess that I was wrong. That wasn't, I guess P of Y is always one in that case. But they don't use this particular kind, I think, except in one of the ablations. So how does this work? It works pretty well. And so one of their experiments, as you see up here, it works pretty well in that if you have 1% of the labels, only 1% of ImageNet labels, which they say is smaller or equal than 13 images per class, so there's a thousand classes and you only have 13 labels per class or less. If you, and they differentiate, if your encoder that you train is a ResNet 50, then you get, and you can see the dashed line here is a supervised baseline. You almost get to the supervised baseline with 1% of the labels. And if you actually have a larger ResNet, then you get to the supervised performance without 99% of the labels. And if you have, excuse me, 10% of the labels, you pass the supervised baseline. So the supervised baseline is on 100% of the labels, mind you, and you only have 10% and this outperforms the supervised baseline. Now of course, you could, here you could have another graphic where you show, oh, 100%. What if we, you know, what if we do the whole procedure with 100% of the labels? So first we don't label the data, we do supervised, self-supervision, then we fine tune on a 100% of the data. And then we do this distillation again, you would of course be even better. And I think they have this somewhere in a table, but this is already pretty, pretty impressive. And another claim they make right here is about the model sizes. So and this figure is description, this now relates to the title. They say bigger models yield larger gains when fine tuning with fewer labeled examples. So there are three comparative statement words in one sentence. Let's unpack this. Bigger models yield larger gains. So the bigger the model, the better the good, let's say, when fine tuning with fewer labeled examples. Let's just look at the graph. It's pretty, it's really clear. So here we have number of parameters going over. So these are the different models they look at, how many parameters they have to do this whole procedure. And here is the relative improvement in percent over the top ImageNet 1 top accuracy. So if you do this whole thing with 100% of the labels, right, I'm going to guess this here, this here is where they start out. And you can see as you grow your models, you grow the performance. And this, this is just by increasing the model size, right, you have the same data set, you have the same amount of labels, you have the same number of steps that you train for, and so on, just by the fact that you make your model bigger, you gain in performance. Okay, now you can see that these curves here are above one another. And these curves refer to getting small, less and less labels. Okay, so if you only have 10% of the labels, your relative gains are larger. That doesn't mean that you perform better with 10% of the labels than with 100% of the labels, that would be like ridiculous. Well, I guess in this day and age, nothing is ridiculous. But for now, we're still performing better by having more labels if we do the same procedure, right? It's not like here. So here, this baseline, the supervised baseline only does supervised training, right? So that's why we can outperform it with less of labels. But here, we do the same procedure. This is relative improvement, right? So this right here, the starting point would be if you had 10% of labels and a 25 million model, parameter model. And this right here, for example, is if you have the same amount of labels, but a 200 million parameter model. And this is relative improvement, okay? But what the graph says is that the relative improvement is larger, the relative improvement is higher, the more parameters you have, which is the more you go to the right. And that effect in itself is higher, the fewer labels you have, which is the different graphs. And you can see that right here. So if you have fewer and fewer labels, it becomes more and more important that you have bigger models. And that's really counterintuitive, right? Because you would expect that the bigger models, they can overfit much more easily to the fewer labels. But that doesn't seem the case. So this self supervision, it really seems to be sort of a counter to this notion of overfitting. And if you have larger and larger models, that's what they argue in the paper, you might be able to learn more and more features that might be useful for classification. So if you have a larger model, you might, you're going to learn more kinds of features, and then you're going to outperform because you have more chance that these features are going to be useful for classification. And I don't think they really make a statement as to why that happens more with the, if you have less labels. So let's think about this. If I have very few labels, very, very few labels, why does it help me even more if I have a big model? Well, with the same argumentation, we could say, and maybe they actually say this already. So I might be copying them involuntarily. Maybe with fewer and fewer labels, like let's say we have all the labels, that's probably too many, right? If we can learn a task with some accuracy, we probably had too many labels. Okay. It's like, if we can't learn a task, we know we have too few. Somewhere there is a border where we have enough, but that's like kind of one number. And everything else is too many, technically speaking, like learning theoretically speaking. So usually we have too many labels. And what does that mean? That probably means that there are multiple ways. Like if we have too many labels, there are multiple different features we can pick up to learn. There are multiple different paths to learn our goals. So if we have ImageNet, and like there's this weird task to recognize a three, and we get lots and lots and lots of examples of threes, right? We can decide on a feature. We can say, oh, all the threes that I see, they have this bow down here, or all the threes that I see, they have this bend here, and so on. But if I only have very few labels, there might only be like a single feature that is even theoretically possible to learn from the labels I'm given. And therefore, if I have a bigger model in cell in pre-training, because the pre-training happens with the same amount of data, right? If I have a bigger model that does the self-supervised pre-training, it's going to learn more features. And then there's a higher chance that that one feature that these very few labels that I am able to learn something from is going to be in these features. So that's kind of how I make sense of it in combination with what they're saying right here. Okay, so this was the main points. They do a lot of empirical studies showing the effects of these sizes. They stress that it's important to have both deep and wide networks. And they also do this additional attention mechanism over the convolution filters. I don't want to go into that particularly. But they also do linear evaluation compared to supervised, compared to fine tuning with 100% of the labels. So they do a very thorough empirical investigation. And yeah, I do appreciate that. And they kind of show the same things. And here they show the number of layers in the projection head. So as you increase the number of layers in the projection head and train from the optimal layer in the middle, your performance goes up, as you can see. But it also this effect is stronger when you have fewer labels, right? You can see the differences here are greater than the differences here or even here when you have 100% of the labels. So the fewer labels, the fewer the labels, the more benefit you have from the architecture right here. And here they show that it's not always optimal to train from the last projection layer, but here the first one. So I guess they converge on three projection layers, and you always want to keep the first one around after self supervised training, as we mentioned before. They investigate different distillation losses and show that it is actually important that you do the distillation loss on labeled and unlabeled sets. You can see here if you only train with the labels after fine tuning, you get poor performance. If you do the label and distillation loss, but only do it on the data set where you have labels, then you get more performance. If you do label and distillation loss, but also include your unlabeled data, you get even more performance. And then if you do that, but you don't do the label loss. So before we've seen you can mix the distillation loss with the label loss, if you have lots of labels, then you drop in performance again. And you can see right here, the drop in performance is proportional to how many labeled examples you have. And that's natural, right? If you have the labels, you can actually mix that information in with the distillation loss and that will make you better. And here they drop 0.1% and here they drop less than 1% by leaving away the label. But their point basically is that it is more important to distill using also unlabeled data, then it is to distill, including the label loss. And it's much easier to not include the label loss. So they don't do it, I guess. All right, so I think that was it. They compare, as I said, they compare like self distillation, where you distill into an equally sized model and down distillation, where you distill into a smaller model, maybe that's vice versa. And they do a lot of comparison to other methods. So this is a very thorough work, I feel. And yeah, if you want more about the exact experiments, I invite you to look at the paper. And let's just have a final look at the broader impact statement right here. So the broader, remember the broader impact statement is supposed to force you to think about how society might be impacted at large by your work. So it says, the finding described in this paper can potentially be harnessed to improve accuracy in any application or computer vision, where it is more expensive or difficult to label additional data than to train larger models. Such applications are clearly beneficial to society. For example, in medical applications where acquiring high quality labels requires careful annotation by clinicians, better semi supervised learning approaches can potentially help save lives. Application of computer vision to agriculture can increase crop yields, which may help to improve availability of food. However, we also recognize that our approach can become a potential component of harmful surveillance systems. Moreover, there is an entire industry built around human labeling services and technology that reduces the need for these services could lead to short term loss of income for some of those currently employed or contracted to provide labels. So ask yourself how much of that statement has to do with the actual novelty of this paper? And the answer is of course, zero, right? Like you can replace like our method in this thing with like machine learning or computer vision in general, like, oh, really SIMClear V2 specifically can increase crop yields? Like that specific invention of this paper will lead to higher crop yields, will lead to surveillance systems. So I'm, yeah, you know, I think like, I'm not gonna get too upset about these. I mean, this, I think it's quite funny. But just, again, I wonder whether the people advocating for these things are happy with these statements, because clearly, clearly, this is just a template that you copy paste from paper to paper, replacing like a few words. And if it's computer vision, you're like, oh, my deep fakes. And if it's an NLP, it's like, oh, I'm a fake news. And yeah, I wonder if really anything like particularly is has I wonder whether these people are happy now. Yeah, I just I wonder. And if, if they are, I wonder whether it's really for the reason that they claim that, oh, now we have a statement here of how it impacts society, because I could have told you that before. I even read the title of the paper, right, what the broader impact statement is going to be. In any case, rant too long, check out paper, share it out, leave a like, comment if you disagree or agree. And yeah, bye bye.
 Hi there, today we'll look at big self-supervised models are strong semi-supervised learners
 by Ting Chen, Simon Kornblith, Kevin Swirsky, Mohamed Nourouzi and Jeffrey Hinton of Google
 Brain.
 So this paper on a high level, it's also known as Sinclair v2, demonstrates that if you want
 to do semi-supervised learning, that you're very well served by starting out with self-supervised
 learning, and then doing fine tuning much like NLP models do, rather than the kind of
 semi-supervised approach that image tasks had so far.
 And they present this Sinclair v2, which is an improvement over the Sinclair approach
 to self-supervised pre-training, and they demonstrate it outperforms a lot of the baselines.
 Alright, so if you like content like this, don't forget to share it out, and leave a
 like and tell me what you think in the comments.
 So this paper, it sort of is kind of a club together thing of different things.
 So they present this new method, like this Sinclair v2, which is a modification of Sinclair,
 and we'll go over that, but they also try to make like a scientific claim, namely that
 somehow bigger models are better for this pathway of learning, and we'll try to untangle
 all of these things.
 So first of all, we're in the semi-supervised learning regime right here.
 This basically means that you have a data set, and you only have labels for a part of
 that data set.
 So this could be like here, the bottom 10% or so, because labels might be expensive to
 get.
 And so you only have a few of them, but you have much more data that's unlabeled.
 Now sometimes this problem is formulated as this here is your data set, and then this
 here is like a different data set, but one that's close enough such that you can learn
 from it.
 And that's usually in NLP.
 You'll have your data set is like a sentiment classification task, but you have all of Wikipedia
 that is not labeled, but it's just text.
 So you can sort of pre-train on it.
 In this case, we'll be in a situation where we'll artificially construct a small data
 set.
 So this entire thing here is going to be the ImageNet data set, and this right here is
 going to be our labeled portion, like we have labels.
 Now usually one has labels for ImageNet as well, but we artificially restrict ourselves
 to simulate a situation where we have lots of data and we only have a fixed budget.
 So we can only, because to obtain labels, oftentimes you have to ask humans to label
 images.
 And let's say we are a company and we've collected this big data set, but we only have like maybe
 500 bucks on Amazon Mechanical Turk, and we only managed to get a very like 1% of our
 data set labeled.
 Now we're in the regime of semi-supervised learning.
 This is slightly different from what NLP does.
 As I said, in NLP, usually assume you have different data sets, the large one being a
 different distribution, and in the semi-supervised regime, you often assume that it is actually
 the same data distribution, but you only have labels for some of them.
 But there should be a fair bit of overlap between the two things.
 So I've recently made a video about OpenAI's ImageGPT that kind of goes into the same direction
 as this work right here that basically says pre-training on unlabeled data, like this
 whole data set without the labels, can be a very good preconditioner for fine tuning
 later.
 And this paper says the same thing.
 So basically, in the good old days, what you would do is you would devise a method that
 somehow takes, you know, takes in a device, a method that takes in a mini batch.
 And in the mini batch, you'd have your data samples, and then some of them would be labeled,
 right here, you'd have a Y and here you'd have a Y, but most of them would be not labeled.
 And you'd have like some sort of loss function that would put special weight on the ones
 that are labeled or somehow handle these ones that are unlabeled in a way, you might be
 doing like some sort of a consistency loss such that if they are very nearest near neighbors
 to these in the feature space, they should have similar labels or things like this.
 So these semi supervised methods, they basically try to solve the problem at once.
 But while taking data that is labeled and not labeled, this paper goes into a different
 direction.
 This paper says, first, we should, it's actually three stages right here, and they have a diagram,
 so I don't need to draw.
 They have a three stage approach.
 Three stages.
 The one on the left is unsupervised pre training.
 So they say, let's forget about the labels right now, even like your unlabeled data.
 So even the data where we have the labels, let's forget about the labels.
 And let's just do unsupervised pre training.
 Now unsupervised pre training in this kind of setting is also known as self supervised
 pre training.
 And this first stage is done using a contrastive loss, and that's very similar to sim clear
 to this contrastive loss.
 So what you'll do, and they describe it very, very well here.
 So what you'll do is given a randomly sampled mini batch of images, each image is augmented
 twice using random crop color distortion and Gaussian blur, creating two views of the same
 example.
 Okay, so you have an image in your mini batch.
 Each image you take and you make two versions of it.
 And each version you crop, you random crop somewhere.
 So version one could be random cropped here.
 Version two could be random cropped here.
 And then you put some Gaussian blur on it and so on.
 So a little bit of, as you can see, random crop color distortion, Gaussian blur.
 So what you'll want is two different versions of the same image.
 Each of these versions has been augmented in a different way, cropped in a different
 way, blurred in a different way.
 It's two slightly different versions of the same image.
 And now you want to enforce, you want to put this through your network.
 So ultimately, as you can see on the right side here, what you want to end up is a network.
 And then, okay, we'll forget about this right now.
 What you want to train is this network right here, actually including these projection
 layers.
 We'll get to them later.
 This is the network that you want to train.
 So you want to put, you take your unlabeled data, you take an image, you make two versions
 of it.
 And you put those through the network, right, until the end right here.
 So you'll get Z1, Z2.
 These are the outputs of the network for the two images.
 And then what you want to do is you want to take another image that's not this image,
 and also put it through the network, maybe also augmented first.
 And then you have Z3.
 So now you have the outputs of two things that are supposed to come from the same image
 and one thing that's supposed to come from a different image.
 And now your loss is simply going to be make those two things close together and push those
 two things apart, or those three actually.
 So the loss, and this is the contrastive loss of self supervised learning.
 As you know, you don't need any labels right here.
 You simply say the things that come from the same image should be close together.
 And the things that come from different images should be far apart.
 And this relies heavily on these data augmentations that you do right here.
 They also employ some other tricks like the momentum encoder from MoCo, from momentum
 contrast and so on.
 But this is the main part.
 So you can pull a lot of strings here to get like another percent of performance.
 But ultimately, they won't see the similarity of ZI and ZJ, which are the outputs of the
 same image to be close together.
 And then this down here, they want to be far apart, ZI with ZK, where K is all the other
 images.
 Okay.
 And you can do this in a mini batch fashion.
 So this is self supervised learning.
 And the reason why you do this is you don't need labels.
 And it tends, we know it tends to give very, very good representations.
 So I'm past that.
 So what this network here will learn will be very good for some reason.
 We still don't exactly know why combining augmentation with the self supervised loss
 with contrastive loss, for example, gives such good performance.
 There have been papers recently that modify the loss and so on.
 But it's not super well understood yet.
 But if you do it like this, the network here will give you already very, very good representation.
 And we know this because we can take a network like this and then simply train a linear classifier
 on top of that on a data set and achieve very, very good performance.
 And mind you, you have trained it with unlabeled data, right?
 So the network has never been trained to solve like ImageNet classification.
 It has simply been trained to look at the pictures and determine if two versions of
 a picture come from the same picture or from different pictures.
 And now, if you simply train a linear classifier on top of these representations, you're doing
 extremely well already.
 So we know these representations, they actually learn something about these images.
 So that's the first part.
 Then stage two, let's cancel all that.
 Stage two is you want to do supervised fine tuning.
 Now you already see that the arrow here coming out is not this task agnostic big CNN.
 The arrow is actually coming out of those yellow boxes.
 And the yellow boxes are these projection heads.
 So in the original SimClear paper, what they did was they wanted originally, they wanted
 to train this network right here.
 This is like a ResNet-50.
 It's pretty standard in these kind of self-supervised approaches and so on to train or these few
 label approaches to train a standardized network.
 And this is like a ResNet-50.
 So in the original SimClear paper, they said we want to make ResNet-50 as strong as possible.
 But in order to do this loss right here, we are going to attach this projection head just
 to because the dimensionality here I think is like 2048.
 And we want to do this inner product in a lower dimension of like maybe 256 or so.
 So these are just multi-layer perceptrons.
 These are just fully connected layers that compress the representation down to that.
 And once we're done with the unsupervised pre-training, we're going to throw those away,
 right?
 And this ResNet is the thing that we really care about.
 Now here they claim, OK, it actually works better.
 And they have experiments to prove this or to show this if you use one, if you actually
 leave one of these layers here.
 So in the end, I guess they converge on three projection head layers.
 And then they only throw away the top two.
 And like they make this big deal out of the fact where, you know, I can just call this
 part right here now the encoder.
 And I don't so I don't know exactly like I don't see the giant deal here.
 Like you just made your network one layer bigger.
 And now you consider that to be your encoder.
 And the projection head is now two layers.
 And that will be much easier than calling the projection head three layers.
 But we leave one layer and we train from the middle layer.
 In any case, they have this layer, additional layer right here compared to the old Sinclair.
 And then the representation of that goes into supervised fine tuning.
 Now, this is pretty easy.
 This is exactly what it sounds like.
 So now you use only only the data set that has labels.
 So the part of the data set that has labels, and you do the fine tuning and fine tuning
 is simply supervised learning.
 You train this network in a supervised fashion on that small fraction of data that has cloud
 class labels.
 And that already performs pretty well.
 And they show this in experiments.
 But then you can go a step further and do what's known as distillation or self training.
 And what's distillation or self training?
 It's so distillation is when you have a network that you call the teacher network.
 And that network has been trained to do some classification maybe into three classes pretty,
 pretty well.
 Okay.
 But now this is very large and you want maybe a smaller model.
 So you just want like this tiny model because you want to ship it on a mobile device, right?
 But it's also supposed to do this.
 And you know that if you just directly train this, which is called the student model, it
 doesn't perform as well as the teacher model.
 There is a better way.
 If you have the teacher model, you can sort of transfer the knowledge to the student model.
 You can distill the knowledge.
 And how do you do that?
 You do that by, so what would you do in supervised training?
 In supervised training, you would take an image, put it in, and then put the label that
 comes along with the image.
 You put it up here and you compare the output to the label and that gives you the loss function.
 Right?
 So you do that right here.
 If you distill, you put the image into both.
 Now the teacher is already trained.
 So its output will be a distribution over classes.
 It won't be a single label.
 It will be like, okay, 90% class one, 10% class two, 0% class three, something like
 this.
 And now you take this as like a pseudo label, this entire distribution, and you put it here
 and you compare the output of the student to that of the teacher and that's your loss
 function.
 So this kind of, the teacher might have learned to put some nuance into the classification
 to say, well, I'm pretty sure this is class one, but I'm not 100% sure.
 And it can transfer that knowledge to the student.
 And that makes the student better than had you just trained it from the beginning from,
 with just the labels.
 Right?
 So this is distillation and you can do this even what they call self distillation here
 or self training.
 So apparently this even helps if the teacher is, if the student model is the same as the
 teacher model.
 Now why does it help in this case?
 And I think it is not exactly the case in this case because they always say their teacher
 model has this extra projection layer.
 Right?
 And then the student model doesn't have that even if they do self training.
 But why does it help in this case?
 I mean, it's, it's kind of shocking and I'm pretty sure it helps in any case, but in this
 particular case it helps because now you're using the unlabeled data again.
 So you have a teacher model and the teacher model is trained first using unsupervised
 like this is the teacher model right here using unsupervised training.
 Then the teacher model is further fine tuned on the small data.
 Right?
 So it is now already pretty good at the task, but how can you get a student model that's
 even better than the teacher model?
 It's by using again this unlabeled data.
 You have this giant amount of data.
 So what you'll do is you take an image from the unlabeled data and you ask the teacher
 model, teacher model, what do you think about that image?
 Right?
 And the teacher model will give you a prediction.
 Like let's say again, this 90%, 10%, 0% and then you take the student model, you input
 that image and you compare its output to what the teacher said.
 So this combines the teacher model.
 You freeze the teacher model, right?
 The teacher model is only trained until here.
 You take it from here.
 The student model is now able to take basically the teacher.
 It takes everything that the teacher model knows, not only about this data, but about
 all the data.
 So it kind of gets to ask the teacher model, what do you think about this?
 What do you think about this?
 What do you think about this?
 And it can incorporate all that knowledge about all of this unlabeled data.
 And that's why the student model here in the end, if it's the same size, will probably
 end up even better than the teacher model.
 So distillation, I think also is still kind of a mystery of why you get a better model
 or, I mean, to make it smaller, if you make it a lot smaller, usually you don't end up
 with a better model, but you end up with a pretty good model that you couldn't have gotten
 by just training the small model.
 So that's already pretty cool.
 But why you get a better model when they're the same size, I don't think that's well understood
 yet.
 So that's the three stage approach.
 So recap, first, use all of the data without labels to do unsupervised or self supervised
 contrastive pre-training.
 Second, use only the data that has labels to do fine tuning.
 Third, either distill the learned classifier to a smaller model or distill it to a model
 of the same size.
 Then in both cases, you would again use the unlabeled, all of the unlabeled data.
 And that's the three step approach.
 That's SEMCLEAR v2 in all of its form.
 So they go into fine tuning right here.
 And yeah, so they say again, we elaborate with a three layer projection head.
 So that's the three layer projection head.
 This here is the output of ResNet-50, where Sigma is a ReLU non-linearity and we ignore
 the bias term for brevity, blah, blah, blah, blah, blah.
 So they contrast this here.
 For fine tuning, SEMCLEAR uses this right here, which is just, it's basically just a
 classifier on top of the output of the ResNet-50.
 This is fine tuning from the input layer of the projection head.
 To fine tune from the first layer of the projection head, we have a new encoder function as this,
 which is ResNet followed by fully connected layers.
 And you see they take the ResNet-50 output and they ship it through the first projection
 layer and then there is a task specific classifier.
 Now, again, why, I don't even see why they make like this ginormous deal out of it, especially,
 especially since the last layer of the ResNet-50.
 I'm not, okay, here is, I'm not entirely sure, but are they taking the log?
 No, they're probably not taking the log.
 It's okay.
 But it's, yeah, it's just weird.
 Like is there even a non-linearity at the end right here?
 Or is this really just like two matrix multiplications in a row, which I'm going to guess there's
 a big chance that that's the case, that the last layer of this encoder is actually not
 even followed by non-linearity and therefore you'll just kind of make the dimension different.
 And I don't see why you can't just incorporate this into the model and have to like say it
 over and over again that this is a new special thing, right?
 Again, this is equivalent of tuning from a middle layer of the projection head instead
 of the output layer.
 Okay, you just make your model a bit bigger.
 Yeah.
 So the third step is self-training or knowledge distillation.
 And they give two variants right here.
 This variant, as you can see here, this is just the cross entropy.
 But instead of having labels right here, Y, you have what the teacher model thinks Y is
 given X.
 Okay, that's cross entropy, but not with the true labels, but with the output of the teacher
 model.
 And you can even mix that.
 So you can, as you can see right here, you can mix this with an actual supervised loss.
 So this would be the supervised loss, whatever.
 Yeah, I guess that I was wrong.
 That wasn't, I guess P of Y is always one in that case.
 But they don't use this particular kind, I think, except in one of the ablations.
 So how does this work?
 It works pretty well.
 And so one of their experiments, as you see up here, it works pretty well in that if you
 have 1% of the labels, only 1% of ImageNet labels, which they say is smaller or equal
 than 13 images per class, so there's a thousand classes and you only have 13 labels per class
 or less.
 If you, and they differentiate, if your encoder that you train is a ResNet 50, then you get,
 and you can see the dashed line here is a supervised baseline.
 You almost get to the supervised baseline with 1% of the labels.
 And if you actually have a larger ResNet, then you get to the supervised performance
 without 99% of the labels.
 And if you have, excuse me, 10% of the labels, you pass the supervised baseline.
 So the supervised baseline is on 100% of the labels, mind you, and you only have 10% and
 this outperforms the supervised baseline.
 Now of course, you could, here you could have another graphic where you show, oh, 100%.
 What if we, you know, what if we do the whole procedure with 100% of the labels?
 So first we don't label the data, we do supervised, self-supervision, then we fine tune on a 100%
 of the data.
 And then we do this distillation again, you would of course be even better.
 And I think they have this somewhere in a table, but this is already pretty, pretty
 impressive.
 And another claim they make right here is about the model sizes.
 So and this figure is description, this now relates to the title.
 They say bigger models yield larger gains when fine tuning with fewer labeled examples.
 So there are three comparative statement words in one sentence.
 Let's unpack this.
 Bigger models yield larger gains.
 So the bigger the model, the better the good, let's say, when fine tuning with fewer labeled
 examples.
 Let's just look at the graph.
 It's pretty, it's really clear.
 So here we have number of parameters going over.
 So these are the different models they look at, how many parameters they have to do this
 whole procedure.
 And here is the relative improvement in percent over the top ImageNet 1 top accuracy.
 So if you do this whole thing with 100% of the labels, right, I'm going to guess this
 here, this here is where they start out.
 And you can see as you grow your models, you grow the performance.
 And this, this is just by increasing the model size, right, you have the same data set, you
 have the same amount of labels, you have the same number of steps that you train for, and
 so on, just by the fact that you make your model bigger, you gain in performance.
 Okay, now you can see that these curves here are above one another.
 And these curves refer to getting small, less and less labels.
 Okay, so if you only have 10% of the labels, your relative gains are larger.
 That doesn't mean that you perform better with 10% of the labels than with 100% of the
 labels, that would be like ridiculous.
 Well, I guess in this day and age, nothing is ridiculous.
 But for now, we're still performing better by having more labels if we do the same procedure,
 right?
 It's not like here.
 So here, this baseline, the supervised baseline only does supervised training, right?
 So that's why we can outperform it with less of labels.
 But here, we do the same procedure.
 This is relative improvement, right?
 So this right here, the starting point would be if you had 10% of labels and a 25 million
 model, parameter model.
 And this right here, for example, is if you have the same amount of labels, but a 200
 million parameter model.
 And this is relative improvement, okay?
 But what the graph says is that the relative improvement is larger, the relative improvement
 is higher, the more parameters you have, which is the more you go to the right.
 And that effect in itself is higher, the fewer labels you have, which is the different graphs.
 And you can see that right here.
 So if you have fewer and fewer labels, it becomes more and more important that you have
 bigger models.
 And that's really counterintuitive, right?
 Because you would expect that the bigger models, they can overfit much more easily to the fewer
 labels.
 But that doesn't seem the case.
 So this self supervision, it really seems to be sort of a counter to this notion of
 overfitting.
 And if you have larger and larger models, that's what they argue in the paper, you might
 be able to learn more and more features that might be useful for classification.
 So if you have a larger model, you might, you're going to learn more kinds of features,
 and then you're going to outperform because you have more chance that these features are
 going to be useful for classification.
 And I don't think they really make a statement as to why that happens more with the, if you
 have less labels.
 So let's think about this.
 If I have very few labels, very, very few labels, why does it help me even more if I
 have a big model?
 Well, with the same argumentation, we could say, and maybe they actually say this already.
 So I might be copying them involuntarily.
 Maybe with fewer and fewer labels, like let's say we have all the labels, that's probably
 too many, right?
 If we can learn a task with some accuracy, we probably had too many labels.
 Okay.
 It's like, if we can't learn a task, we know we have too few.
 Somewhere there is a border where we have enough, but that's like kind of one number.
 And everything else is too many, technically speaking, like learning theoretically speaking.
 So usually we have too many labels.
 And what does that mean?
 That probably means that there are multiple ways.
 Like if we have too many labels, there are multiple different features we can pick up
 to learn.
 There are multiple different paths to learn our goals.
 So if we have ImageNet, and like there's this weird task to recognize a three, and we get
 lots and lots and lots of examples of threes, right?
 We can decide on a feature.
 We can say, oh, all the threes that I see, they have this bow down here, or all the threes
 that I see, they have this bend here, and so on.
 But if I only have very few labels, there might only be like a single feature that is
 even theoretically possible to learn from the labels I'm given.
 And therefore, if I have a bigger model in cell in pre-training, because the pre-training
 happens with the same amount of data, right?
 If I have a bigger model that does the self-supervised pre-training, it's going to learn more features.
 And then there's a higher chance that that one feature that these very few labels that
 I am able to learn something from is going to be in these features.
 So that's kind of how I make sense of it in combination with what they're saying right
 here.
 Okay, so this was the main points.
 They do a lot of empirical studies showing the effects of these sizes.
 They stress that it's important to have both deep and wide networks.
 And they also do this additional attention mechanism over the convolution filters.
 I don't want to go into that particularly.
 But they also do linear evaluation compared to supervised, compared to fine tuning with
 100% of the labels.
 So they do a very thorough empirical investigation.
 And yeah, I do appreciate that.
 And they kind of show the same things.
 And here they show the number of layers in the projection head.
 So as you increase the number of layers in the projection head and train from the optimal
 layer in the middle, your performance goes up, as you can see.
 But it also this effect is stronger when you have fewer labels, right?
 You can see the differences here are greater than the differences here or even here when
 you have 100% of the labels.
 So the fewer labels, the fewer the labels, the more benefit you have from the architecture
 right here.
 And here they show that it's not always optimal to train from the last projection layer, but
 here the first one.
 So I guess they converge on three projection layers, and you always want to keep the first
 one around after self supervised training, as we mentioned before.
 They investigate different distillation losses and show that it is actually important that
 you do the distillation loss on labeled and unlabeled sets.
 You can see here if you only train with the labels after fine tuning, you get poor performance.
 If you do the label and distillation loss, but only do it on the data set where you have
 labels, then you get more performance.
 If you do label and distillation loss, but also include your unlabeled data, you get
 even more performance.
 And then if you do that, but you don't do the label loss.
 So before we've seen you can mix the distillation loss with the label loss, if you have lots
 of labels, then you drop in performance again.
 And you can see right here, the drop in performance is proportional to how many labeled examples
 you have.
 And that's natural, right?
 If you have the labels, you can actually mix that information in with the distillation
 loss and that will make you better.
 And here they drop 0.1% and here they drop less than 1% by leaving away the label.
 But their point basically is that it is more important to distill using also unlabeled
 data, then it is to distill, including the label loss.
 And it's much easier to not include the label loss.
 So they don't do it, I guess.
 All right, so I think that was it.
 They compare, as I said, they compare like self distillation, where you distill into
 an equally sized model and down distillation, where you distill into a smaller model, maybe
 that's vice versa.
 And they do a lot of comparison to other methods.
 So this is a very thorough work, I feel.
 And yeah, if you want more about the exact experiments, I invite you to look at the paper.
 And let's just have a final look at the broader impact statement right here.
 So the broader, remember the broader impact statement is supposed to force you to think
 about how society might be impacted at large by your work.
 So it says, the finding described in this paper can potentially be harnessed to improve
 accuracy in any application or computer vision, where it is more expensive or difficult to
 label additional data than to train larger models.
 Such applications are clearly beneficial to society.
 For example, in medical applications where acquiring high quality labels requires careful
 annotation by clinicians, better semi supervised learning approaches can potentially help save
 lives.
 Application of computer vision to agriculture can increase crop yields, which may help to
 improve availability of food.
 However, we also recognize that our approach can become a potential component of harmful
 surveillance systems.
 Moreover, there is an entire industry built around human labeling services and technology
 that reduces the need for these services could lead to short term loss of income for some
 of those currently employed or contracted to provide labels.
 So ask yourself how much of that statement has to do with the actual novelty of this
 paper?
 And the answer is of course, zero, right?
 Like you can replace like our method in this thing with like machine learning or computer
 vision in general, like, oh, really SIMClear V2 specifically can increase crop yields?
 Like that specific invention of this paper will lead to higher crop yields, will lead
 to surveillance systems.
 So I'm, yeah, you know, I think like, I'm not gonna get too upset about these.
 I mean, this, I think it's quite funny.
 But just, again, I wonder whether the people advocating for these things are happy with
 these statements, because clearly, clearly, this is just a template that you copy paste
 from paper to paper, replacing like a few words.
 And if it's computer vision, you're like, oh, my deep fakes.
 And if it's an NLP, it's like, oh, I'm a fake news.
 And yeah, I wonder if really anything like particularly is has I wonder whether these
 people are happy now.
 Yeah, I just I wonder.
 And if, if they are, I wonder whether it's really for the reason that they claim that,
 oh, now we have a statement here of how it impacts society, because I could have told
 you that before.
 I even read the title of the paper, right, what the broader impact statement is going
 to be.
 In any case, rant too long, check out paper, share it out, leave a like, comment if you
 disagree or agree.
 And yeah, bye bye.