Self-training with Noisy Student improves ImageNet classification (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper ssl semi-supervised transfer learning cnn resnet efficientnet noise augmentation data augmentation randaugment dropout stochastic depth google distillation self-training knowledge distillation imagenet unsupervised unlabeled unlabelled jft
The abundance of data on the internet is vast. Especially unlabeled images are plentiful and can be collected with ease. This model investigates a new method for incorporating unlabeled data into a supervised learning pipeline. First, a teacher model is trained in a supervised fashion. Then, that teacher is used to label the unlabeled data. Next, a larger student model is trained on the combination of all data and achieves better performance than the teacher by itself.

OUTLINE:
0:00 - Intro & Overview
1:05 - Semi-Supervised & Transfer Learning
5:45 - Self-Training & Knowledge Distillation
10:00 - Noisy Student Algorithm Overview
20:20 - Noise Methods
22:30 - Dataset Balancing
25:20 - Results
30:15 - Perturbation Robustness
34:35 - Ablation Studies
39:30 - Conclusion & Comments

Paper: https://arxiv.org/abs/1911.04252
Code: https://github.com/google-research/noisystudent
Models: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet

Abstract:
We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.
Noisy Student Training extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. Models are available at this https URL. Code is available at this https URL.

Authors: Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar (preferred to Patreon): https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at self-training with noisy student improves ImageNet classification by Qi Zi Sie, Min Tan Luong, Edward Hovey and Quoc Vy Lue. So this paper takes an ImageNet classifier that's been trained on the ImageNet data set and uses that classifier as a teacher model to label a whole bunch of unlabeled images and then it trains a student model that is larger than the original teacher model on those teacher labeled images and that turns out to improve the classification on the ImageNet validation set. Now that there is a couple of things that make this all work and today we're going to explore how this paper does it and what they say is important. If you enjoy content like this as always don't hesitate to share it out there tell your friends about it and if you're not subscribed yet then do so. I would appreciate that and you'll get more content so win-win. So this this paper is about semi-supervised learning in effect. So it's at the intersection actually of semi-supervised learning, knowledge distillation and transfer learning. So what do we mean by semi-supervised learning? Usually in supervised learning you'll have some sort of data set and the data set will contain, let's say it's an ImageNet, it's image data set. So the data set will contain images. This is an image with like some sort of cat on it and it will contain the labels according to that. So cat. Now in semi-supervised learning you assume that, so this is supervised learning, in semi-supervised learning you assume that only part of your data set has the labels. So like only this part down here has the labels and the upper part does not have the labels. So that's semi-supervised learning. It's often the case when it's very expensive to get labels so you can only get labels for a couple of images in your data set. But very often in semi-supervised learning you still assume it's the same data set. There is a slightly different setup here that's called transfer learning. So in transfer learning what you'll have is you'll have your data set that has the labels but it's very small. So you'll notice I've drawn it smaller. That means you have very little. That is also the case when it's very expensive to get labels but also it's expensive to get the data itself. This is often the case like say in medical data where not only is it expensive to get labels for like a CT scan it's actually expensive to get the CT scan. So what the goal in transfer learning is is to say well I do have only this small data set but I do have this giant other data set over here. Now can't I... it's not the same. Maybe they're not CT so these are CT scans. Maybe these are X-rays right? They're fairly similar. Similar technology. If you slice the CT it'll give you sort of an X-ray. Can I you know train my model, pre-train my model on X-ray data and then fine-tune it on the CT data? So that's called transfer learning usually. Now this can be done with or without labels. So it can be that for the X-ray data set you do have the labels or you don't have the labels. There are techniques for all of those. Now what we're going to look at today is kind of the situation right here. It's the transfer learning situation where you do not have the labels for this X-ray data set. But other than in this X-ray example what we're going to look at is the small data set is going to be our ImageNet database. So our original picture with label database. So you'll see immediately the difference here is that in the transfer learning setting we usually assume that the data set we want to train on is fairly small. Here you know ImageNet is already sizable. But what we have is we have a much larger database of unlabeled images that we can just get from the internet. So we can scrape the internet for any kind of pictures and that will be our unlabeled data set. Now what we'll try to do is somehow incorporate this unlabeled data set here into the training process to get better on the ImageNet data set. So this is the problem statement is you have the ImageNet data set and you have a second much larger data set of unlabeled images and you somehow want to make use of them. So I hope you see how this is sort of connected to the others. It's essentially sort of a transfer semi-supervised learning setting but with the exception that usually in transfer learning you assume that the label data set is like super small. Which is not the case here and that's going to result in us being able to apply a different technique. So this different technique is called the noisy student. Now usually what you might do in a transfer learning setting is you might want to start with that big data set because that's the data set that's sizable enough to allow you to train a really big model on it and then you fine-tune and you sort of hope that the information transfers over. Here on the other hand what we want to do is we start with the ImageNet data set. So first we train this in a supervised learning fashion into our model. Now this model is going to be called the teacher model. We know how to do this, we know how to train ImageNet models. So we can train this into a teacher model that has a reasonable accuracy on the ImageNet data set. Step two we're going to take that big data set over here and use the teacher model to label the unlabeled images. So for each image coming in here the teacher will say that's a cat. So that gives you the big data set where now you have images along with labels. Just the labels aren't true labels, they're generated by the teacher. And then in the third step you train this big data set, you train on this big data set and that's what you call your student model. And then the student model in this paper will see how can we make it such that the student is then better at the original ImageNet task than the teacher ever was. Which seems counterintuitive at first because all of the information that the student is trained from is basically what the teacher already knows. All the labels here come from the teacher, therefore the student shouldn't be able to outperform the teacher. But in this case the student will be able to outperform the teacher and their argument here is that this is mainly due to the fact that you use noise in this training procedure. So when you train the student what you'll do is you'll use noise and one of the types of noise is that you severely augment this data right here in order to train the student. Now we've known for a long time that data augmentation, for example in the frameworks of self-supervised learning and so on, can have a very large benefit to training. And here the fact that we incorporate this extra data and we use noise and augmentations on it is going to result in a student that can sort of learn more about the data than the teacher did know. Okay this is basically it and as you can see this is kind of their main final results where they say on ImageNet our top one accuracy sort of increases right here and even on these kind of subsets of ImageNet or these are sort of corrupted sets of ImageNet they make even more substantial improvements as you can see here. Now we'll go into what these corrupted subsets are but you know just for now these here are very difficult variants of ImageNet. They can be severely corrupted or distorted and so on and you can see that the model improves severely over the previous state of the art which basically means that this model is more robust and that's a direct consequence of the noise. Now one last thing I should say is that the student here is also larger than the teacher so that's also one thing that makes the student better. So what you will make is the student model is larger than the teacher model as a model as the architecture. So in combination with the noise right here with the noise in combination that means the student model is probably able to capture more of the variance of the data. It's larger it has more parameters it can learn more about the data together with the noise it can probably be a more robust and that's what makes it generalize better and we'll also see as we see here it's more robust to these transformations and it's also going to be more robust to adversarial perturbations. So the technique again is illustrated here as as we said it's pretty simple. First so step one step one train the teacher model with labeled data as you would. Step two you infer the pseudo labels on unlabeled data. Step three you make a student you make sorry we'll step three over here train an equal or a larger student model with combined data and noise injected. So they don't they use the original labeled data here and the pseudo labeled data right here in order to train the student but still this the student doesn't have more information more label information than the teacher had it simply has this teacher labeled teacher labeled unlabeled data also to train on. Now the crucial part here is well first of all that the student can be larger and second of all that there can be noise and the noise comes in three different forms. So first of all you use data augmentation which we've already seen this is sort of like random cropping or mild rotations color jitter whatever they use a rand augment here which is a specific technique to apply these augmentations they use dropout which is a fairly old technique where you in the student model that you train you randomly drop out connections which makes it more robust and more generalizing and then you also use stochastic depth. Now stochastic depth is a technique when you train a model what you'll do during training instead of always passing your data forward through the layers like this you use some sort of a dropout but with entire layers so what you'll do is you'll pass your data forward and then randomly you'll skip a layer and then pass it forward again. Now these these might seem weird first because yeah it might seem weird but in if you know that most models especially computer vision models nowadays are residual networks which means that their layers look like so you have the input and you have some computation and then you have the output and then there is already a residual connection that basically adds the original signal together to the result of the computation. So all you do in this stochastic layer dropout or this stochastic depth right here is you basically disable you you disable this connection right here and all the signal has to flow through here. If you read the residual the ResNet original ResNet paper they make it pretty clear why the residual connection is a good idea basically they say these computations here they if you have a very deep network each layer only has to basically do very a little bit of computation that that can be bypassed fairly efficiently for a lot of data points so it's not that hurtful to bypass a layer and in this case they actually use it to just bypass some of these small computations and inject some more robustness into the student model. So with these three strategies to bring noise into the training process one is on the data and two is on the student model itself they train the student model and then fourth and this is what we didn't have before four or maybe we put four here make the student a new teacher so now you can iterate you can use the student model that you just trained to again label the unlabeled data and then you can use another student model again under the influence of noise to train from that student model and so on and you can go on and they do up to like three iterations of this where they always take the new the student as the new teacher and then use a new student model to train from that teacher and they get better and better as they do this of course there's like a diminishing returns but it's pretty impressive that this even works right the new students in fact aren't even larger than the old students it's just that the students are larger than the original teacher model in most of these cases so here's the algorithm written down you'll require labeled images right here and unlabeled images which are the ones with the tilde so first you learn the teacher model which minimizes the cross entropy on labeled images this we already know this right this is the label this is the image according to the label and you train the teacher model which is this thing here and you can see here noised so already in the teacher training process you want to introduce this noise you want to introduce these data augmentations these are as I said these are standard techniques to make models more robust and therefore more generalizable yeah we know from these from these self-supervised papers that these augmentations are very powerful and the way you design them basically if you one of these augmentations is a random crop which means if you have an image you randomly crop out like part of that image and then that's your training sample and not the entire thing so by doing this you basically teaching the model to ignore the exact location and scale of things on an image and you can do this because you as a human know that you know I can zoom in I can zoom out into something and it won't change what's on the picture and so that's you use these augmentations to kind of heuristically tell the model what it should be invariant to and that is that is a very powerful technique to regularize basically to to robustify these deep methods and this is used the same here so already in the teacher model we train with this noise and then step two use a normal ie not noise teacher model to generate soft or hard pseudo labels for the clean ie not distorted unlabeled images and this is important they stress this here that when you when you label the unlabeled images you want to use the model that is without the noise and you do it on the not distorted unlabeled images so when you infer the labels it's very important that you have clean accurate labels without any sort of noise in them so label noise is not something that they have found to help in this case so not label noise on the teacher that is so you can see right here on the unlabeled images will use that teacher model without the noise to infer the labels now they say these can be hard model hard labels or soft labels so what does that mean if we generate hard pseudo labels that means that the y here is simply going to be either 0 or 1 or 2 or 3 and so on so just the index of the class whichever class is most likely that's going to be our label this is exactly how the supervised datasets come right so this is what you will think first when you see that however soft pseudo labels means that the y will be a distribution so instead of being of class 0 it will be sort of let's say 90% of class 0 but also 5% class 1 and 5% class 2 right so you'll output the distribution instead of the just the label and they have found that the soft pseudo labels work slightly slightly better than the hard pseudo labels okay thanks so that they use the soft pseudo labels here because they work slightly better but you can do it with hard or soft labels the important thing is that you use the teacher to generate as accurate as possible labels for your unlabeled data then third we've already seen this learn an equal or larger student model which minimizes the cross entropy loss on labeled images and unlabeled images with noise added to the student model so as you can see labeled images and unlabeled images so we're in this semi semi supervised learning setting right now you take in both together with noise and noise here is in bold which means they stress it again this is important so you can see that the loss is composed of two different things these are the true images of your original model and you use that and this means you noise the student model and that that noise can be on the data or in the model itself and here also the unlabeled images that you have labeled with the teacher you do the exact same thing so you train on both of these data sets and step four is if you want to do iterative training use the student as a teacher and go back to step two now they have some more tricks when they do this iterative training they also up the batch size during the iterative training and so on so they do a lot of things to make the student learn something more something better than the teacher and I think this the whole paper it doesn't it doesn't state it explicitly but I think the whole paper everything they do here is to kind of force or allow the student to become better than the teacher by by giving more noise by making the student larger by making the batch size for the student larger and so on so you you want to sort of inject as much invariance as you can and that will make the student learn more so they say here noising student when the student is deliberately noised in its it is trained to be consistent to the teacher that is not noised when it generates the pseudo labels in our experiments we use two types of noise input noise and model noise all right first data augmentation is an important noising method in noisy student training because it forces the student to ensure prediction consistency across augmented versions of an image specifically in our method the teacher produces high quality pseudo labels by reading in clean images while the student is required to produce to reproduce those labels with augmented images as an input second when dropout and stochastic depth function are used as noise the teacher behaves like an ensemble at inference time when it generates pseudo labels whereas the student behaves like a single model in other words the student is forced to mimic a more powerful ensemble model we present an ablation study so this it's a bit weird what they say here don't be confused you use the dropout and the stochastic depth on the student model and they they say here if you do this the teacher behaves like an ensemble at inference time whereas the student behaves like a single model and yeah it's it's a bit of a weird formulation but it's it's true like the teacher the teacher will produce these same the label for different pathways through the students if you use dropout and kind of stochastic depth and therefore the student is kind of required to approximate each time each forward pass has a different forward pass through the layers through the connections with dropout and it's forced to approximate that teacher label with all of these different things so you see that you you put in a lot of a lot of techniques so they have even other techniques there is one additional trick and it's not and it's not one actually they have so many tricks and if you look at their experimental setup that it's crazy like they describe exactly we reduce the learning rate like this and the batch size like this and so on so to get state-of-the-art on image net it's not enough to just have a good idea of a new thing to do what you you you have to have the good idea and then execute it almost like really well because you have to regard all of these additional tricks that people have figured out over the years in any case they say it works better with an additional trick data filtering and balancing specifically we filter images that the teacher model has low confidence on since they are usually out of domain images so that goes to a point where if you see we have this image net label data set right and we have the larger data set now the larger data set simply contains images and there is no guarantee that the images are actually of the classes that we have in the image net data set right here we have a thousand classes here there's no guarantee that these images fit into any of those classes yet we still ask the teacher model to put them in some of these classes now you can filter out part of those images if you can look at the teacher model and you look at its confidence so when it outputs a distribution if if there's just two labels let's say if it outputs a distribution like this that's wildly different than if it outputs a distribution like this both are class one labels but one is much more confident than the other so what you want to do is you want to filter out these low confidence labels because you know the model isn't really sure but it has to assign a class but that's usually an indication that it is an out of domain image so if they filter this it works better and then also to ensure that the distribution of the unlabeled images match that of the training set we also need to balance the number of unlabeled images for each class as all classes in image net have a similar number of labeled images for this purpose we duplicate images in classes where there are not enough images for classes where we have too many images we take the images with the highest confidence okay so this is just another technique this has basically nothing to do with their core idea but this is just another thing where they say okay we can treat this big thing that we scrape from the internet you know we can somehow filter and balance it smartly and that will work even better alright so let's go into the experiments of course there so what they do I think where is the graphic what they do is they take an image net sorry they take an efficient net right here and they trade they first train an efficient net a smaller efficient net as we said for to be the teacher and then they train a larger efficient net for the student the best model in our experiments is a result of three iterations of putting back the student as a new teacher we first train an efficient net B7 on image net as the teacher model so you can see in the table right here what the B7 achieves the efficient net B7 here you can see it has 66 million parameters which is fairly small compared to these other kind of previous state-of-the-art methods on image net right so they first train this and that will achieve something like an 85% accuracy now if you just train a larger model this efficient net L2 right here that has you can see 480 million parameters so a lot of more million parameters but you just train it on the same data set on image net you will get a 0.5% improvement and you can see that here with noisy student training with the exact same model so it has the same amount of parameters you'll actually get an 88.4 so I like a more than a 3% improvement and that's what the same model just with this different training procedure and inputting these 300 million unlabeled images that you have laying around but the all the information about all the label information comes from the image net data set and comes from this efficient net B7 teacher model so that's basically you can it's a testament that out of this out of this 85 you can make this 88 just by smartly using the information that the model that this model has learned about the data and transferring it to new data so they train an efficient net B7 that's the small model as a teacher model then by using the B7 model as the teacher we trained an efficient net L2 model with the unlabeled batch size set to 14 times the labeled batch size and they stressed that it's important that you up the batch size that's another thing that makes the student learn more than the teacher then we trained a new efficient net so by the way these 14 times it's also it can be done because now you have more data right so you can also up the batch size then we trained a new efficient net L2 model with the efficient net L2 model as the teacher lastly we iterated again and used an unlabeled batch size of 28 times the labeled batch size the detailed result of the three iterations and so okay so you can see that it's a fairly complicated procedure but you can gain and gain and gain by simply up upping the by simply upping the or iterating on this procedure and I think they have it somewhere here yes so as you can see if iteration one you train the efficient net L2 you started with the B7 you train the efficient at a two with a batch size 14 times larger and you gain significantly right this gains about 2% over the original efficient net then you iterate again with the same batch size and you get like a 5.5% improvement and you iterate again with an even larger batch size and you get a point three percent improvement so there is diminishing returns but still you can see that you know the more with the introduction of noise with the introduction of the larger model with the introduction of the larger batch size these are all things that help the student basically become better than the teacher all right so they do a bunch of other experiments so their main comparison is right here where they say look if we if even if we train the same model with this noisy student training we can make you know pretty large gains over the model over the same model where we do not train it with this noisy student training so this really seems to help you know due to the noise due to the additional data they do a lot of ablation studies so that's pretty interesting and they also do these studies on this special image net data set for example image net see you can see that there are quite a bit of distortions right here I don't even see if you can see it on this video but this is a swing so the swing right here is like something like this but you almost can't see it and you see that the bold on the left is always the prediction of their model while the thing on the right is the prediction of the original model so this model they claim is significantly more robust to these kinds of perturbations and they do an analysis of this where they show yes in fact it is so I think we've already seen this at the beginning that the noisy student is significantly more robust to these perturbations and they also test this to adversarial perturbations so right here you can see that the original model drops pretty quickly as you increase the epsilon the epsilon is kind of the strength of the adversarial perturbation and the noisy the original model drops very quickly to you know fairly low accuracy while as the noisy student training drops much much less quickly now this is another testament to the fact that what you do I think what's happening is you have your data space right and you have your data points in it now when you do the like normal data augmentation what you'll do is you not only force the model to predict those points correctly but you'll sort of make a bit of a cloud around them and you force the model to predict that cloud correctly now if you introduce more data and you do even more noise what you do is you'll make these clouds kind of larger and that means the model is more robust to any sort of perturbations in these clouds right and and that means it's probably also going to be more robust to adversarial perturbations so that's sort of how you can think of this this introduction of noise to make it more generalizable how does this generalize better so if you think of this data point right here if I'm looking to generalize that means you know I have this IID data set so probably my test data is going to be related to the training data so I might get a data point that's fairly close to that data point and generalizing means I classify it correctly now if this cloud is very small like it is here my decision boundary could be like here right and even though the terrestres data set is fairly close to the original training data point it's it won't be classified incorrectly however if my original cloud during training is larger you can see if I train a model it can maybe put the decision boundary here and then my test data point will be included in on that same side so that's kind of the idea behind generalizing better of course that's a vast simplification and also to say that this here is an FGSM attack so this is kind of the weakest attack in the adversarial perturbation spectrum they do say under a stronger attack PGD which is a fairly strong attack with 10 iterations at epsilon equals 16 noisy student training improves efficient netl2 accuracy from 1.1% to 4.4% now this I'm like you know 1.1% really means the model is almost like dead this is lower this is like random performance and 4.4% is still a bit above random performance but yeah you could probably you could probably get there by simply using any sort of noise in that case but still you can see that it is more robust to especially to natural distortions and therefore it generalizes better as I said they do quite a bit of drop sorry not drop out at ablation studies to figure out where exactly the performance comes from and the answer is it pretty much comes from all the things that they've described so here you can see the effect of that extra data set and you can see pretty much with that extra data set all the all the situations improve here you can see what do you what is happening when you do not augment the student when you do not data augment you can immediately see that the accuracy drops and then when you do not augment and also don't use these model noises then the performance drops again and lastly when you use the teacher but you noise the teacher you can see also here the performance is dropping from the original quite a bit so all of these things kind of contribute and they do much more ablations and they have listed their findings here so using a large teacher model with better performance leads to better result so you know as the original teacher you should use as good as possible a teacher model you can find second a large amount of unlabeled data is necessary for better performance okay so if you want to do this you better get a large large amount of extra data because that's one thing that makes the student perform better soft pseudo labels work better than hard pseudo labels for out of the main data in certain cases fourth a large student model is important to enable the student to learn a more powerful model okay so because usually this knowledge distillation is what it this is usually called knowledge distillation if you use a teacher model to train a student model and it is often used when the student model is smaller than the teacher because you want to kind of become more efficient to you from so the teacher is large or make the student small and you usually sacrifice some accuracy and here they say if you want to gain some accuracy you need a large student model it can't be like a small one number five data balancing is useful for small models number six joint training on labeled data and unlabeled data out performs the pipeline that first pre trains with unlabeled data and then fine-tunes on labeled data so this is in contrast to like what people have done before in the self supervised learning and so on where it's always kind of pre training then fine-tuning or in the in the transfer learning setting seven using a large ratio between unlabeled batch size and label batch size enables models to train longer on unlabeled data to it to achieve a higher accuracy okay we've already seen that they have used that and number eight training the student from scratch is sometimes better than initializing the student with the teacher and the student initialized with the teacher still requires a large number of training epochs to perform well this is fairly interesting because it kind of alludes to the fact that the minima in weight space if so if this is of course the case if the student model is the same as the teacher model so in like iteration two or three or whatnot it means that you know in weight space if we look at you know you might want to start the student here and the minimum is right here and you might want to think that if I learn the same thing then the minima are fairly close together right so the the teachers minima might be here and the student minima might be fairly close so it might be beneficial if I if I start not over here but actually start at the teachers minimum but this doesn't always seem to be the case and that is a fairly interesting observation because it kind of means that we're talking about different minima here we're talking about the student model learning different things and that's what we've discussed already the student model kind of learns to be robust and that's probably a minimum that's fairly far away in weight space at least in in a sort of energy landscape weight space might be the case that it needs to actually overcome kind of a hill here even though the minimum might be close there's lots of research in like how minima are distributed in these weight spaces which I don't want to go into right here but it is a fairly interesting observation that it's not always helpful to initialize the teacher sorry the student at the teachers optimum okay so this was the paper and you know this is this is the type of research where I do appreciate kind of the these large labs taking it on because they have the resources to do all of these ablations all of these different models cross them with these giant data sets and so on which I guess university labs just would not have and this is a fairly thorough paper really investigating which parts of the pipeline you know do something and which ones don't and usually I I'm fairly critical of pipelines that have like 50 billion tricks because you never know where the improvement exactly is coming from but you can sort of mitigate that criticism by doing all of these kind of ablations on the different parts and really showing look this is important but this is also important but this is also important but this is also important so yeah that was my two cents to this paper I hope you enjoyed this and I'll see you next time bye bye
 Hi there, today we'll look at self-training with noisy student improves
 ImageNet classification by Qi Zi Sie, Min Tan Luong, Edward Hovey and Quoc Vy Lue.
 So this paper takes an ImageNet classifier that's been trained on the
 ImageNet data set and uses that classifier as a teacher model to label
 a whole bunch of unlabeled images and then it trains a student model that is
 larger than the original teacher model on those teacher labeled images and that
 turns out to improve the classification on the ImageNet validation set. Now that
 there is a couple of things that make this all work and today we're going to
 explore how this paper does it and what they say is important. If you enjoy
 content like this as always don't hesitate to share it out there tell your
 friends about it and if you're not subscribed yet then do so. I would
 appreciate that and you'll get more content so win-win. So this this paper is
 about semi-supervised learning in effect. So it's at the intersection
 actually of semi-supervised learning, knowledge distillation and transfer
 learning. So what do we mean by semi-supervised learning? Usually in
 supervised learning you'll have some sort of data set and the data set will
 contain, let's say it's an ImageNet, it's image data set. So the data set will
 contain images. This is an image with like some sort of cat on it and it will
 contain the labels according to that. So cat. Now in semi-supervised learning you
 assume that, so this is supervised learning, in semi-supervised learning you
 assume that only part of your data set has the labels. So like only this part
 down here has the labels and the upper part does not have the labels. So that's
 semi-supervised learning. It's often the case when it's very expensive to get
 labels so you can only get labels for a couple of images in your data set. But
 very often in semi-supervised learning you still assume it's the same data set.
 There is a slightly different setup here that's called transfer learning.
 So in transfer learning what you'll have is you'll have your data set that has
 the labels but it's very small. So you'll notice I've drawn it smaller. That means
 you have very little. That is also the case when it's very expensive to get
 labels but also it's expensive to get the data itself. This is often the case
 like say in medical data where not only is it expensive to get labels for like a
 CT scan it's actually expensive to get the CT scan. So what the goal in transfer
 learning is is to say well I do have only this small data set but I do have
 this giant other data set over here. Now can't I... it's not the same.
 Maybe they're not CT so these are CT scans. Maybe these are X-rays right?
 They're fairly similar. Similar technology. If you slice the CT it'll
 give you sort of an X-ray. Can I you know train my model, pre-train my model on
 X-ray data and then fine-tune it on the CT data? So that's called transfer
 learning usually. Now this can be done with or without labels. So it can be that
 for the X-ray data set you do have the labels or you don't have the labels.
 There are techniques for all of those. Now what we're going to look at today is
 kind of the situation right here. It's the transfer learning situation where
 you do not have the labels for this X-ray data set. But other than in this
 X-ray example what we're going to look at is the small data set is going to be
 our ImageNet database. So our original picture with label database. So you'll
 see immediately the difference here is that in the transfer learning setting we
 usually assume that the data set we want to train on is fairly small. Here you
 know ImageNet is already sizable. But what we have is we have a much larger
 database of unlabeled images that we can just get from the internet. So we can
 scrape the internet for any kind of pictures and that will be our unlabeled
 data set. Now what we'll try to do is somehow incorporate this unlabeled data
 set here into the training process to get better on the ImageNet data set.
 So this is the problem statement is you have the ImageNet data set and you
 have a second much larger data set of unlabeled images and you somehow want to
 make use of them. So I hope you see how this is sort of connected to the others.
 It's essentially sort of a transfer semi-supervised learning setting but with
 the exception that usually in transfer learning you assume that the
 label data set is like super small. Which is not the case here and that's going to
 result in us being able to apply a different technique. So this different
 technique is called the noisy student. Now usually what you might do in a
 transfer learning setting is you might want to start with that big data set
 because that's the data set that's sizable enough to allow you to train a
 really big model on it and then you fine-tune and you sort of hope that the
 information transfers over. Here on the other hand what we want to do is we
 start with the ImageNet data set. So first we train this in a supervised
 learning fashion into our model. Now this model is going to be called the teacher
 model. We know how to do this, we know how to train ImageNet models. So we can
 train this into a teacher model that has a reasonable accuracy on the ImageNet
 data set. Step two we're going to take that big data set over here and use the
 teacher model to label the unlabeled images. So for each image coming
 in here the teacher will say that's a cat. So that gives you the big data
 set where now you have images along with labels. Just the labels aren't true
 labels, they're generated by the teacher. And then in the third step you train
 this big data set, you train on this big data set and that's what you call your
 student model. And then the student model in this paper will see how can we make
 it such that the student is then better at the original ImageNet task than the
 teacher ever was. Which seems counterintuitive at first because all of
 the information that the student is trained from is basically what the
 teacher already knows. All the labels here come from the teacher, therefore the
 student shouldn't be able to outperform the teacher. But in this case the student
 will be able to outperform the teacher and their argument here is that this is
 mainly due to the fact that you use noise in this training procedure. So when
 you train the student what you'll do is you'll use noise and one of the types of
 noise is that you severely augment this data right here in order to train the
 student. Now we've known for a long time that data augmentation, for example in
 the frameworks of self-supervised learning and so on, can have a very large
 benefit to training. And here the fact that we incorporate this extra data
 and we use noise and augmentations on it is going to result in a student that can
 sort of learn more about the data than the teacher did know. Okay this is
 basically it and as you can see this is kind of their main final results where
 they say on ImageNet our top one accuracy sort of increases right here
 and even on these kind of subsets of ImageNet or these are sort of corrupted
 sets of ImageNet they make even more substantial improvements as you can see
 here. Now we'll go into what these corrupted subsets are but you know just
 for now these here are very difficult variants of ImageNet. They can be
 severely corrupted or distorted and so on and you can see that the model
 improves severely over the previous state of the art which basically means
 that this model is more robust and that's a direct consequence of the noise.
 Now one last thing I should say is that the student here is also larger than the
 teacher so that's also one thing that makes the student better. So what you
 will make is the student model is larger than the teacher model as a model as the
 architecture. So in combination with the noise right here with the noise in
 combination that means the student model is probably able to capture more of the
 variance of the data. It's larger it has more parameters it can learn more about
 the data together with the noise it can probably be a more robust and that's
 what makes it generalize better and we'll also see as we see here it's more
 robust to these transformations and it's also going to be more robust to
 adversarial perturbations. So the technique again is illustrated here as
 as we said it's pretty simple. First so step one step one train the teacher
 model with labeled data as you would. Step two you infer the pseudo labels on
 unlabeled data. Step three you make a student you make sorry we'll step three
 over here train an equal or a larger student model with combined data and
 noise injected. So they don't they use the original labeled data here and the
 pseudo labeled data right here in order to train the student but still this the
 student doesn't have more information more label information than the teacher
 had it simply has this teacher labeled teacher labeled unlabeled data also to
 train on. Now the crucial part here is well first of all that the student can
 be larger and second of all that there can be noise and the noise comes in
 three different forms. So first of all you use data augmentation which we've
 already seen this is sort of like random cropping or mild rotations color jitter
 whatever they use a rand augment here which is a specific technique to apply
 these augmentations they use dropout which is a fairly old technique where
 you in the student model that you train you randomly drop out connections which
 makes it more robust and more generalizing and then you also use
 stochastic depth. Now stochastic depth is a technique when you train a model what
 you'll do during training instead of always passing your data forward through
 the layers like this you use some sort of a dropout but with entire layers so
 what you'll do is you'll pass your data forward and then randomly you'll skip a
 layer and then pass it forward again. Now these these might seem weird first
 because yeah it might seem weird but in if you know that most models especially
 computer vision models nowadays are residual networks which means that their
 layers look like so you have the input and you have some computation and then
 you have the output and then there is already a residual connection that
 basically adds the original signal together to the result of the
 computation. So all you do in this stochastic layer dropout or this
 stochastic depth right here is you basically disable you you disable this
 connection right here and all the signal has to flow through here. If you read the
 residual the ResNet original ResNet paper they make it pretty clear why the
 residual connection is a good idea basically they say these computations
 here they if you have a very deep network each layer only has to basically
 do very a little bit of computation that that can be bypassed fairly efficiently
 for a lot of data points so it's not that hurtful to bypass a layer and in
 this case they actually use it to just bypass some of these small computations
 and inject some more robustness into the student model. So with these three
 strategies to bring noise into the training process one is on the data and
 two is on the student model itself they train the student model and then fourth
 and this is what we didn't have before four or maybe we put four here make the
 student a new teacher so now you can iterate you can use the student model
 that you just trained to again label the unlabeled data and then you can use
 another student model again under the influence of noise to train from that
 student model and so on and you can go on and they do up to like three
 iterations of this where they always take the new the student as the new
 teacher and then use a new student model to train from that teacher and they get
 better and better as they do this of course there's like a diminishing
 returns but it's pretty impressive that this even works right the new students
 in fact aren't even larger than the old students it's just that the students are
 larger than the original teacher model in most of these cases so here's the
 algorithm written down you'll require labeled images right here and unlabeled
 images which are the ones with the tilde so first you learn the teacher model
 which minimizes the cross entropy on labeled images this we already know this
 right this is the label this is the image according to the label and you
 train the teacher model which is this thing here and you can see here noised so
 already in the teacher training process you want to introduce this noise you
 want to introduce these data augmentations these are as I said these
 are standard techniques to make models more robust and therefore more
 generalizable yeah we know from these from these self-supervised papers that
 these augmentations are very powerful and the way you design them basically if
 you one of these augmentations is a random crop which means if you have an
 image you randomly crop out like part of that image and then that's your training
 sample and not the entire thing so by doing this you basically teaching the
 model to ignore the exact location and scale of things on an image and you can
 do this because you as a human know that you know I can zoom in I can zoom out
 into something and it won't change what's on the picture and so that's you
 use these augmentations to kind of heuristically tell the model what it
 should be invariant to and that is that is a very powerful technique to
 regularize basically to to robustify these deep methods and this is used
 the same here so already in the teacher model we train with this noise and then
 step two use a normal ie not noise teacher model to generate soft or hard
 pseudo labels for the clean ie not distorted unlabeled images and this is
 important they stress this here that when you when you label the unlabeled
 images you want to use the model that is without the noise and you do it on the
 not distorted unlabeled images so when you infer the labels it's very important
 that you have clean accurate labels without any sort of noise in them so
 label noise is not something that they have found to help in this case so not
 label noise on the teacher that is so you can see right here on the unlabeled
 images will use that teacher model without the noise to infer the labels
 now they say these can be hard model hard labels or soft labels so what does
 that mean if we generate hard pseudo labels that means that the y here is
 simply going to be either 0 or 1 or 2 or 3 and so on so just the index of the
 class whichever class is most likely that's going to be our label this is
 exactly how the supervised datasets come right so this is what you will think
 first when you see that however soft pseudo labels means that the y will be a
 distribution so instead of being of class 0 it will be sort of let's say 90%
 of class 0 but also 5% class 1 and 5% class 2 right so you'll output the
 distribution instead of the just the label and they have found that the soft
 pseudo labels work slightly slightly better than the hard pseudo labels okay
 thanks so that they use the soft pseudo labels here because they work slightly
 better but you can do it with hard or soft labels the important thing is that
 you use the teacher to generate as accurate as possible labels for your
 unlabeled data then third we've already seen this learn an equal or larger
 student model which minimizes the cross entropy loss on labeled images and
 unlabeled images with noise added to the student model so as you can see labeled
 images and unlabeled images so we're in this semi semi supervised learning
 setting right now you take in both together with noise and noise here is in
 bold which means they stress it again this is important so you can see that
 the loss is composed of two different things these are the true images of your
 original model and you use that and this means you noise the student model and
 that that noise can be on the data or in the model itself and here also the
 unlabeled images that you have labeled with the teacher you do the exact same
 thing so you train on both of these data sets and step four is if you want to do
 iterative training use the student as a teacher and go back to step two now they
 have some more tricks when they do this iterative training they also up the
 batch size during the iterative training and so on so they do a lot of things to
 make the student learn something more something better than the teacher and I
 think this the whole paper it doesn't it doesn't state it explicitly but I think
 the whole paper everything they do here is to kind of force or allow the student
 to become better than the teacher by by giving more noise by making the student
 larger by making the batch size for the student larger and so on so you you want
 to sort of inject as much invariance as you can and that will make the student
 learn more so they say here noising student when the student is deliberately
 noised in its it is trained to be consistent to the teacher that is not
 noised when it generates the pseudo labels in our experiments we use two
 types of noise input noise and model noise all right first data augmentation
 is an important noising method in noisy student training because it forces the
 student to ensure prediction consistency across augmented versions of an image
 specifically in our method the teacher produces high quality pseudo labels by
 reading in clean images while the student is required to produce to
 reproduce those labels with augmented images as an input second when dropout
 and stochastic depth function are used as noise the teacher behaves like an
 ensemble at inference time when it generates pseudo labels whereas the
 student behaves like a single model in other words the student is forced to
 mimic a more powerful ensemble model we present an ablation study so this it's a
 bit weird what they say here don't be confused you use the dropout and the
 stochastic depth on the student model and they they say here if you do this
 the teacher behaves like an ensemble at inference time whereas the student
 behaves like a single model and yeah it's it's a bit of a weird formulation
 but it's it's true like the teacher the teacher will produce these same the
 label for different pathways through the students if you use dropout and kind of
 stochastic depth and therefore the student is kind of required to
 approximate each time each forward pass has a different forward pass through the
 layers through the connections with dropout and it's forced to approximate
 that teacher label with all of these different things so you see that you you
 put in a lot of a lot of techniques so they have even other techniques there is
 one additional trick and it's not and it's not one actually they have so many
 tricks and if you look at their experimental setup that it's crazy like
 they describe exactly we reduce the learning rate like this and the batch
 size like this and so on so to get state-of-the-art on image net it's not
 enough to just have a good idea of a new thing to do what you you you have to
 have the good idea and then execute it almost like really well because you have
 to regard all of these additional tricks that people have figured out over the
 years in any case they say it works better with an additional trick data
 filtering and balancing specifically we filter images that the teacher model has
 low confidence on since they are usually out of domain images so that goes to a
 point where if you see we have this image net label data set right and we
 have the larger data set now the larger data set simply contains images and
 there is no guarantee that the images are actually of the classes that we have
 in the image net data set right here we have a thousand classes here there's no
 guarantee that these images fit into any of those classes yet we still ask the
 teacher model to put them in some of these classes now you can filter out
 part of those images if you can look at the teacher model and you look at its
 confidence so when it outputs a distribution if if there's just two
 labels let's say if it outputs a distribution like this that's wildly
 different than if it outputs a distribution like this both are class
 one labels but one is much more confident than the other so what you
 want to do is you want to filter out these low confidence labels because you
 know the model isn't really sure but it has to assign a class but that's usually
 an indication that it is an out of domain image so if they filter this it
 works better and then also to ensure that the distribution of the unlabeled
 images match that of the training set we also need to balance the number of
 unlabeled images for each class as all classes in image net have a similar
 number of labeled images for this purpose we duplicate images in classes
 where there are not enough images for classes where we have too many images we
 take the images with the highest confidence okay so this is just another
 technique this has basically nothing to do with their core idea but this is just
 another thing where they say okay we can treat this big thing that we scrape from
 the internet you know we can somehow filter and balance it smartly and that
 will work even better alright so let's go into the experiments of course there
 so what they do I think where is the graphic what they do is they take an
 image net sorry they take an efficient net right here and they trade they first
 train an efficient net a smaller efficient net as we said for to be the
 teacher and then they train a larger efficient net for the student the best
 model in our experiments is a result of three iterations of putting back the
 student as a new teacher we first train an efficient net B7 on image net as the
 teacher model so you can see in the table right here what the B7 achieves the
 efficient net B7 here you can see it has 66 million parameters which is fairly
 small compared to these other kind of previous state-of-the-art methods on
 image net right so they first train this and that will achieve something like an
 85% accuracy now if you just train a larger model this efficient net L2 right
 here that has you can see 480 million parameters so a lot of more million
 parameters but you just train it on the same data set on image net you will get
 a 0.5% improvement and you can see that here with noisy student training with
 the exact same model so it has the same amount of parameters you'll actually get
 an 88.4 so I like a more than a 3% improvement and that's what the same
 model just with this different training procedure and inputting these 300
 million unlabeled images that you have laying around but the all the
 information about all the label information comes from the image net
 data set and comes from this efficient net B7 teacher model so that's
 basically you can it's a testament that out of this out of this 85 you can make
 this 88 just by smartly using the information that the model that this
 model has learned about the data and transferring it to new data so they
 train an efficient net B7 that's the small model as a teacher model then by
 using the B7 model as the teacher we trained an efficient net L2 model with
 the unlabeled batch size set to 14 times the labeled batch size and they stressed
 that it's important that you up the batch size that's another thing that
 makes the student learn more than the teacher then we trained a new efficient
 net so by the way these 14 times it's also it can be done because now you
 have more data right so you can also up the batch size then we trained a new
 efficient net L2 model with the efficient net L2 model as the teacher
 lastly we iterated again and used an unlabeled batch size of 28 times the
 labeled batch size the detailed result of the three iterations and so okay so
 you can see that it's a fairly complicated procedure but you can gain
 and gain and gain by simply up upping the by simply upping the or iterating on
 this procedure and I think they have it somewhere here yes so as you can see if
 iteration one you train the efficient net L2 you started with the B7 you
 train the efficient at a two with a batch size 14 times larger and you gain
 significantly right this gains about 2% over the original efficient net then you
 iterate again with the same batch size and you get like a 5.5% improvement and
 you iterate again with an even larger batch size and you get a point three
 percent improvement so there is diminishing returns but still you can
 see that you know the more with the introduction of noise with the
 introduction of the larger model with the introduction of the larger batch
 size these are all things that help the student basically become better than the
 teacher all right so they do a bunch of other experiments so their main
 comparison is right here where they say look if we if even if we train the
 same model with this noisy student training we can make you know pretty
 large gains over the model over the same model where we do not train it with this
 noisy student training so this really seems to help you know due to the noise
 due to the additional data they do a lot of ablation studies so that's pretty
 interesting and they also do these studies on this special image net data
 set for example image net see you can see that there are quite a bit of
 distortions right here I don't even see if you can see it on this video but this
 is a swing so the swing right here is like something like this but you almost
 can't see it and you see that the bold on the left is always the prediction of
 their model while the thing on the right is the prediction of the original model
 so this model they claim is significantly more robust to these kinds
 of perturbations and they do an analysis of this where they show yes in fact it
 is so I think we've already seen this at the beginning that the noisy student is
 significantly more robust to these perturbations and they also test this to
 adversarial perturbations so right here you can see that the original model
 drops pretty quickly as you increase the epsilon the epsilon is kind of the
 strength of the adversarial perturbation and the noisy the original model drops
 very quickly to you know fairly low accuracy while as the noisy student
 training drops much much less quickly now this is another testament to the
 fact that what you do I think what's happening is you have your data space
 right and you have your data points in it now when you do the like normal data
 augmentation what you'll do is you not only force the model to predict those
 points correctly but you'll sort of make a bit of a cloud around them and you
 force the model to predict that cloud correctly now if you introduce more data
 and you do even more noise what you do is you'll make these clouds kind of
 larger and that means the model is more robust to any sort of perturbations in
 these clouds right and and that means it's probably also going to be more
 robust to adversarial perturbations so that's sort of how you can think of this
 this introduction of noise to make it more generalizable how does this
 generalize better so if you think of this data point right here if I'm
 looking to generalize that means you know I have this IID data set so
 probably my test data is going to be related to the training data so I might
 get a data point that's fairly close to that data point and generalizing means I
 classify it correctly now if this cloud is very small like it is here my decision
 boundary could be like here right and even though the terrestres data set is
 fairly close to the original training data point it's it won't be classified
 incorrectly however if my original cloud during training is larger you can see if
 I train a model it can maybe put the decision boundary here and then my test
 data point will be included in on that same side so that's kind of the idea
 behind generalizing better of course that's a vast simplification and also
 to say that this here is an FGSM attack so this is kind of the weakest attack in
 the adversarial perturbation spectrum they do say under a stronger attack
 PGD which is a fairly strong attack with 10 iterations at epsilon equals 16
 noisy student training improves efficient netl2 accuracy from 1.1% to 4.4%
 now this I'm like you know 1.1% really means the model is almost like dead
 this is lower this is like random performance and 4.4% is still a bit above
 random performance but yeah you could probably you could probably get there by
 simply using any sort of noise in that case but still you can see that it is
 more robust to especially to natural distortions and therefore it generalizes
 better as I said they do quite a bit of drop sorry not drop out at ablation
 studies to figure out where exactly the performance comes from and the answer is
 it pretty much comes from all the things that they've described so here you can
 see the effect of that extra data set and you can see pretty much with that
 extra data set all the all the situations improve here you can see what
 do you what is happening when you do not augment the student when you do not data
 augment you can immediately see that the accuracy drops and then when you do not
 augment and also don't use these model noises then the performance drops again
 and lastly when you use the teacher but you noise the teacher you can see also
 here the performance is dropping from the original quite a bit so all of these
 things kind of contribute and they do much more ablations and they have listed
 their findings here so using a large teacher model with better performance
 leads to better result so you know as the original teacher you should use as
 good as possible a teacher model you can find second a large amount of unlabeled
 data is necessary for better performance okay so if you want to do this you better
 get a large large amount of extra data because that's one thing that makes the
 student perform better soft pseudo labels work better than hard pseudo
 labels for out of the main data in certain cases fourth a large student
 model is important to enable the student to learn a more powerful model okay so
 because usually this knowledge distillation is what it this is usually
 called knowledge distillation if you use a teacher model to train a student model
 and it is often used when the student model is smaller than the teacher
 because you want to kind of become more efficient to you from so the teacher is
 large or make the student small and you usually sacrifice some accuracy and here
 they say if you want to gain some accuracy you need a large student model
 it can't be like a small one number five data balancing is useful for small
 models number six joint training on labeled data and unlabeled data out
 performs the pipeline that first pre trains with unlabeled data and then
 fine-tunes on labeled data so this is in contrast to like what people have done
 before in the self supervised learning and so on where it's always kind of
 pre training then fine-tuning or in the in the transfer learning setting seven
 using a large ratio between unlabeled batch size and label batch size enables
 models to train longer on unlabeled data to it to achieve a higher accuracy okay
 we've already seen that they have used that and number eight training the
 student from scratch is sometimes better than initializing the student with the
 teacher and the student initialized with the teacher still requires a large
 number of training epochs to perform well this is fairly interesting because
 it kind of alludes to the fact that the minima in weight space if so if this is
 of course the case if the student model is the same as the teacher model so in
 like iteration two or three or whatnot it means that you know in weight space
 if we look at you know you might want to start the student here and the minimum
 is right here and you might want to think that if I learn the same thing then
 the minima are fairly close together right so the the teachers minima might
 be here and the student minima might be fairly close so it might be beneficial
 if I if I start not over here but actually start at the teachers minimum
 but this doesn't always seem to be the case and that is a fairly interesting
 observation because it kind of means that we're talking about different
 minima here we're talking about the student model learning different things
 and that's what we've discussed already the student model kind of learns to be
 robust and that's probably a minimum that's fairly far away in weight space
 at least in in a sort of energy landscape weight space might be the case
 that it needs to actually overcome kind of a hill here even though the minimum
 might be close there's lots of research in like how minima are distributed in
 these weight spaces which I don't want to go into right here but it is a fairly
 interesting observation that it's not always helpful to initialize the
 teacher sorry the student at the teachers optimum okay so this was the
 paper and you know this is this is the type of research where I do appreciate
 kind of the these large labs taking it on because they have the resources to do
 all of these ablations all of these different models cross them with these
 giant data sets and so on which I guess university labs just would not have and
 this is a fairly thorough paper really investigating which parts of the
 pipeline you know do something and which ones don't and usually I I'm fairly
 critical of pipelines that have like 50 billion tricks because you never know
 where the improvement exactly is coming from but you can sort of mitigate that
 criticism by doing all of these kind of ablations on the different parts and
 really showing look this is important but this is also important but this is
 also important but this is also important so yeah that was my two cents
 to this paper I hope you enjoyed this and I'll see you next time bye bye