Big Transfer (BiT): General Visual Representation Learning (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google brain cnn convolutional neural network resnet residual network pretraining finetuning vtab imagenet cifar state of the art pretrained computer vision
One CNN to rule them all! BiT is a pre-trained ResNet that can be used as a starting point for any visual task. This paper explains what it takes to pre-train such a large model and details how fine-tuning on downstream tasks is done best.

Paper: https://arxiv.org/abs/1912.11370
Code & Models: TBA

Abstract:
Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

Authors: Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're talking about Big Transfer, General Visual Representation Learning by Alexander Kolesnikov, Lukas Baier, Yao-Ai Chai and others of Google Brain. So this paper is basically an application slash engineering paper for the community and it is about the task of transfer learning for visual tasks. So what does that mean? In a visual task the meaning is basically that the input is an image. So it could be a classifier where you have an image and you have to say that this is a cat. Or it could be, let's say, a medical image of a lung and you have to point out where the defect in the lung is or if there is a defect in the lung or something like this. As we all know this field is pretty much dominated by CNNs, by convolutional neural networks that take in these images through many layers of convolution. Especially residual networks are doing particularly well on these tasks. The problem of course is that in some tasks you have lots of data and that's fine because CNNs need lots of data to train. But in some tasks, especially these medical tasks, you only have like very small database. Look at this small database. You only have very few labeled samples where the model could learn from. And that just is not enough to learn these big models that would perform well. So you will have to settle for a less performing model. Now the solution or one of the solutions is transfer learning. In transfer learning what you do is you take a large data set, for example the ImageNet data set. You have this big data set right here and you train your CNN on that. And then you take that CNN and you do what's called a fine-tuning step on this small data set. So you take the CNN that you gained from the large data set as a starting point and then you just train for a few steps. You just kind of adapt it to this final data set that you actually want to train it on. And that usually helps. And why does that help? Because you sort of hope that the large data set and the small data set are at least somewhat overlapping in their... So the images in the large data set are somewhat similar to the images in the small data set. It doesn't need to be super similar but just somewhat. And you hope that the features that the CNN learns from the large data set are useful in the small data set. Because then if that is the case when you fine-tune on the small data set, that's this step down here, that's called a fine-tuning. When you fine-tune you can pretty much reuse those features. You only have to adjust them a little bit. And you just have to learn how to map the features to the output, which now is of course different than in the original task. But you won't have to rediscover the features. So that's why transfer learning can help. So the first phase is called pre-training. The second phase is called fine-tuning. Now the ultimate goal in this is the following. Imagine you have like a giant database of data. This is giant. Look at the size comparison to the others. And so you have this big big database of images. And you train a CNN on that big database of labeled images. Now what you're hoping is that you can do this once and then this one CNN trained on this giant data set will become the starting point for all kinds of small tasks now. So basically you can post this on a repository online and everyone that has a visual task will not train from scratch. But they will basically take this one CNN as a starting point. It is very similar to what people are doing right now with BERT or generally these transformer language models. You never want to train them from scratch. You always want to train from a pre-trained state that someone else has done. Because usually the big work is now shifted to the pre-training. So the goal is to find this one universal starting point for visual learning. And of course no better place to do this than Google. They certainly do have a giant databases of images. They certainly have lots of computation which we're going to see is very necessary for something like this. Now they do train three different models. Their model is called BIT and they train three different variants BIT. Small, medium and large. So the L model is trained on 300 million images. The medium model is trained on 14 million images. So this is the I think it's called JFT dataset. This here is called the ImageNet 21k dataset which looks pretty funky. It has like objects in front of weird backgrounds and stuff like that. And the small is simply trained on the 1.3 million ImageNet dataset. So I mean just look at this. We're in a situation now where the small model is trained is pre-trained on ImageNet just for reference. If you had imagined this five years ago this you would not have maybe you would have guessed it but it's still impressive. So they do release these two models here the medium and the small one pre-trained I believe. They don't release the large one which maybe that's the price we have to pay for getting the medium and the small one. The fact that now Google can use this in their products because they have probably spent a considerable amount of money in doing this. I'm not sure this is a philosophical discussion whether in the interest of science they should really solve. Because they do give the sort of exact training protocol. You just need the money basically. Alright but that's not topic of this video. So the models here are all pretty much just residual networks. They're all these ResNet 152 I think x4 which means that basically scale the width of each layer by a number of four from the original ResNet architecture and that's pretty much it. This is the architecture. There's nothing really new in this paper. The paper just details what exactly you have to do. Which things exactly matter when you pre-train these things and which ones don't. Therefore I believe it is a pretty good paper and I think that these models here the M and S models and maybe someone else trains an L model and releases it will sort of become the standard like we have in BERT now. So whenever you have a visual task you're just gonna start from those in practice. So this I think is mainly relevant for people in practice. Alright here you can see these models. First of all excellent excellent not labeling of your x-axis. Absolutely beautiful. The x-axis I believe is the number of samples per data class. So now they take their pre-trained model this bit L and they fine-tune it on these datasets. So ImageNet is one of the tasks they fine tune on or CIFAR 10, CIFAR 100 and so on. And first of all look on the right side this full thing. This is when you take the entire dataset. So often they outperform, they get state-of-the-art on the full datasets. Now they do compare against what they call generalist models. So generalist models are ones that have this particular training protocol where they train on one big giant database and then fine-tune to all the other tasks. They do not achieve state-of-the-art on all datasets in what they call specialist models. The specialist models would be such models that have this exact task in mind and therefore they don't care about other tasks. They outperform some of these specialist models but not all of them. So this is not the new state of the art in everything but it is in this transfer learning regime. And I think even more important if you see on the left this is in the small label regime. So here you have something like 100 or 25 or 10 or even 5 labels per class. And if you take 5 labels per class for CIFAR 10, this model, so of course you have to pre-train it first on the big dataset, but just taking 5 labels per class you still get like 94% accuracy on CIFAR 10. And that's pretty good. That is pretty impressive especially if you compare it to this baseline model here which is a ResNet pre-trained on just the ImageNet dataset. So that really shows you the power of pre-training with full data. So one thing they say is that in their big dataset in their 300 million images they make sure to remove all the images that then appear in the downstream tasks. Because otherwise it is fairly conceivable that this database here is just scraped from the internet and of course these tasks are often, like CIFAR 10, are also scraped from the internet and also ImageNet. And it is entirely conceivable of course that the test data is already here and they say we remove images but I think they just remove exact duplicates. So it could still be that someone has taken ImageNet and then kind of recoded it into another color scheme or whatnot or just compressed it a bit more and then they find these images on the web. So it's a little shaky this whole thing because these datasets might just be part of one another but you know given the results I do generally believe the improvements here. But yeah. So I guess what we need is like people to actually go out with cameras and shoot new pictures for a new test set. But in any case let's dive into how to pre-train something like this. So they divide their findings up in two parts. How to pre-train and how to fine-tune. So how to transfer to downstream tasks. And the methods they find are surprisingly easy. They say there are two components to pre-training. The first component is scale. So you have to have a lot of data and a lot of models and that is a pretty important recognition. So down here they have this ablation where they scale up the model and scale up the data. So look at this for example. You can see here you have the different datasets to pre-train on. So this is the small dataset, the medium dataset and the large dataset. So in this direction you have dataset size. Then here you have accuracy not labeled. Again I guess we can understand accuracy. That's fine. And there we have the different models. Now the larger the dot here is the larger the model architecture. And you can see within the individual bins the larger the model the better performance you usually get. But as you can see like here this improvement in the large models isn't as much as when you have much data. And you can also see by the slope of the line here the larger amounts of data help more when you have larger models. So only scaling up the data is not as effective as scaling up the data and the model at the same time. And in some cases like in this small architecture here it actually hurts to incorporate more data. At least they say that. And you can also see that here. And here it just doesn't help as much anymore if you incorporate more data. So if your model is too small you can't handle the big data. Of course there are weird effects like here the performance goes down and then up with the larger data. So this might actually be an effect of the images in these data sets being somewhat qualitatively different also with respect to the task that you are training for. But in general it holds that you need a combination of data set size and model size to go up. And this I think might be an indication of where we are in Belkin's double descent curve. So if you look at the researcher Mikhail Belkin and other people also research in this area they have this sort of empirical finding and hypothesis that if you plot a graph and here is the number of parameters in relation to the data set size. It's a number of parameters in relation to the size of data. And here is your validation loss. Then what happens as you have very little parameters you can add more parameters to your model to get better validation loss. This is you know we get a better model and we train that and we get better. And then at some point you'll start to overfit. We've all learned this in our general machine learning course and there is a point here, what is called the interpolation threshold, where you have this is one so the number of parameters is equal to the number of data points which is just interpolating your training data. Sorry the data point here that's train. But then the discovery sort of is that this comes down again and it stays down. So as you go up in number of data points, sorry number of parameters with the same data set you're perfectly fitting the training data set. You're past the number of data points in your model but still your validation loss comes down and there's various hypotheses why this could happen. And here we find ourselves maybe in this sort of situation where if you have a model right here and you want to scale it you want to add more data you can't just keep the model constant because if you add more data that will shift you to the left here because you add more data but you keep the number of parameters the same so this number will shift to the left and you actually go up in your validation loss. So maybe this is actually what's happening right here the fact that the model is too small. This is just a hypothesis by me. So if you want to up your number of data points you also have to up your number of parameters and that will keep it going and maybe these models here are more on this side of this interpolation threshold and the models where it doesn't happen might be more over here. Though that is a big thing to assume. Maybe not. Now that I think about it since they have even more parameters here they would be even more here somewhere so maybe you add a bunch of data it's just not as bad. There might be some weird interactions here. Like this. Who knows? Let's just skip this. In any case the message here is you need more model and more data at the same time. Alright then there is a second message a second recipe for pre-training. There we are. The second method is group normalization and weight standardization. So they criticize batch norm. Batch norm has of course been used a lot. That is where if you have a batch of data and you put it through your layers and it has some intermediate representation what you want to do is you want to calculate sort of the mean and variance of your data in each of the features and then make it such that it's nice mean one and standard deviation. So mean zero and standard deviation of one. That is called batch norm but of course it is dependent on your batch size. So it is dependent on how many data points you have because that's how well you can estimate these mean and variance parameters. And what people do nowadays is they take these batches and they group them into different groups and they distribute those groups onto many many machines which is called data parallelism especially with TPUs nowadays. You can just distribute everything to so many TPUs. I believe they say they distribute to something like 500 TPUs. They have a batch size of I think 4,000 and they distribute to 500 TPUs so that leaves them with eight samples per batch. So this is eight and eight is just not very good for batch norm and if you have to if you want to circumvent that you need to in each layer globally sync with all of the other workers your batch norm parameters and that slows you down. So people have gone around this using what they call group normalization and weight standardization. So these two techniques of weight standardization is a addition to group normalization. They don't require the other samples in the batch. They work on a per sample basis and they normalize the features within groups of each channel. So the group normalization groups together different features within a sample and then normalizes across that and the weight standardization is a bit like standardizing the features but it's standardizes the weights to be of a normal distribution. And just suffice to say these are standard techniques that you can build in and they allow you to not have to synchronize constantly between your workers at the training time which makes everything a lot faster and also not a problem that you just have eight samples per worker. So that's what they do. They do large data large models and group normalization with weight standardization. That's how they pre-train and then how do they fine-tune. They say they have a rule to select hyperparameters. They call the bit hyper rule and that's just sort of a formula of how you have one hyper parameter. So you have one I guess it's a hyper hyper parameter and that hyper hyper parameter you run through their rule and the rule will tell you what each of the hyper parameters should be. So it's like a lookup table basically. It's oh you set this one number and we give you the rest of the hyper parameters and that one rule works pretty well. So you only have to find for fine-tuning you only have to grid search over one hyper parameter. It's not really grid anymore is it? And then they basically decide on the training schedule length resolution and whether to do mixup regularization. Mixup is a technique that can help when you have very little data and what it does is it interpolates between data points and also trains on kind of like data points from half this class and half that class just to make more data available. But they all have this packed into this rule and they of course the exact settings of this rule are presented. So you can look it up then they have a data pre-processing, resize the image to a square, crop out small random square, randomly horizontally flip the image at training time. So they basically describe a standard training protocol here. You don't want to go mix it too up too much. The only thing they say surprisingly we do not use any form any of the following forms of regularization during downstream tuning, weight decay to zero, weight decay to initial parameters or dropout. I think they only use weight decay during pre-training and that's it. So let's look at some of the graphs. We've already seen some. Here is where they pretty much outperform the generalist, these generalist models on all of these tasks including this visual task adaptation benchmark. I've made a video about this. This is a benchmark that includes 19 different visual tasks from all over the place and they have significant improvement here as you can see. They do not always outperform these specialist models but as you can see they outperform for example this on the flowers data set and they come pretty close. Here you can also see how much they improve when pre-training on the larger data set. So far people have basically pre-trained on this ImageNet data set and now that they pre-train on the larger one of course they gain a lot of performance and the largest one isn't even in this table. So what I finally want to look at is this visual task adaptation benchmark. This consists of 19 tasks and they're divided into natural tasks which are kind of natural images and then specialized tasks which are let's say the medical images and not really natural and then structured tasks and the structured tasks isn't simply labeling or locating something. It is tasks where you have to maybe reason about something. So let's say there is an image and there is a cup right here and there is a glass right here and the question is what's to the left of the glass and there's a bunch of other stuff around here and you have to say the cup. So it sort of requires a structured understanding of the image and you can see the main performance boost here comes in the natural images which is to be expected. So you only get what you feed in and this 300 million image data set I'm pretty sure that's just a web scrape of photos or mainly photos. So the main improvement you're gonna get is on pictures that are similar to that as we said at the beginning and these natural tasks have images like that and you can see that the model here improves extremely in that category, improves slightly in this specialized thing and only improves a little bit in the structured tasks. So this as I said is to be expected. Just know if you use this model know what is in there. You have to know what it does, what it does well. It does well on natural images that are similar to what it was pre-trained on. Okay so they do have some analysis here and we've already went to most of them. I find this to be pretty impressive. So they say when they apply the standard computational budget of ImageNet pre-training when they scale up to the larger data set it seems detrimental. As you can see right here the performance actually goes down when you go to the larger data set. Only if you train longer then your improves. The axis labeling is just amazing here. Standard, long, longer. Oh how long you train for? Longer. Thanks. But I guess the point is taken that you have to invest more computation along with your bigger model and bigger data set. Sorry it's the same model but the bigger data set. They also make some other points here that if you for example if you decrease your learning rate too early or set your weight decay parameter differently that also hurts you. So on the right here you see a smaller weight decay initially looks better. So initially you're higher but through the training you end up at a worse place than a higher setting right here. I mean they make a big point out of this but who's to say that someone else doesn't come with like a ten times longer training and figures out that ultimately you start off like this and then maybe goes up super high. So to me the lessons learned here is pretty much that there's always a way to get more performance out of more compute and probably there is a way to schedule all of these things because that's combined with decaying learning rate and so on. There's probably a way to schedule these things with the current method. So with this particular method that would end up somewhere here we just haven't found it yet because it's so complex. I would guess that is the case. Here they make an interesting point that if you decay the learning rate too early then you also end up at a worse place. So this this dashed researcher the noob. So after eight GPU weeks which come on what is that eight GPU weeks that's just eight GPUs for a week. That's nothing nothing. It looks like this right it looks fairly flat and this researcher now decides to decay the learning rate and that results in this thing here. So decays the learning rate here here and here. Sorry not here. So the case learning right here and then it flattens out again and then decays the learning rate again ends up at this level. Yet if you train for longer you can see right here if you look over eight months you can see that there is a slight upward trend still and it hasn't converged yet and you can if you decrease the learning rate only later and always wait for this to fully converge then you will end up at a better place right here above 70. Again who's to say that if I just wait here there isn't a slight upward trend if I wait for eight GPU years or eight GPU solar system births then there might be even a better point to decay finally decay the learning rate and then go up. I mean again this this researcher here only takes point five million steps where you take two million. So that's the first point. The second point is ImageNet or visual state-of-the-art research is now officially out of the hands of academia. This is it. If you see things like if you see a paper dissing on people that only wait eight GPU weeks to decrease their learning rate for the first time and advocating that you should you know at least wait until eight GPU months actually they wait twice as long it's over that's it. Yeah bye bye. Maybe maybe you know you want to do some theory or something yeah bye bye. What I find interesting is the mistakes so since on CIFAR-10 they reach like 99.4 percent there's only a handful of mistakes that they're still making because it's not that large of a data set and they do classify it so red in particular I think means red is the ground truth label is correct but green is the machine is correct and the ground truth label is wrong and you can see there is a fair number of green things here right so the model says ship and the label says cat and the model says bird and the label says cat clearly this this would be one weird cat so it gets to the point where you you also have to expect these errors to be in the training set so it could just be that the model here doesn't necessarily even make those mistakes but it's just somewhat consistent with the training set in making the mistakes and also here on image net they have selected ones where you know the model says notebook but it's actually laptop and the model says mouse but it's actually spacebar you know the model says Alp and it's ski so or here the model the model says candle but it's a this is a dishwasher what so you see that the the the types of mistakes here we get to very quirky very fine-grained points in these models last thing I want to show I have never seen these image net 21k images these are just funky like look at that so here's the state-of-the-art previously I think said triceratops and the new model now says bit else as starfish good job bit L you wow probably the correct label would just be weird and this no okay I don't want to rag on this too much this is a cool paper I believe this will be the new starting point for a lot of practitioners in when they do visual tasks I always as always invite you to check out the paper subscribe to the channel leave a like leave a comment if you want I do read them usually and bye bye
 Hi there! Today we're talking about Big Transfer, General Visual Representation
 Learning by Alexander Kolesnikov, Lukas Baier, Yao-Ai Chai and others of Google
 Brain. So this paper is basically an application slash engineering paper for
 the community and it is about the task of transfer learning for visual tasks. So
 what does that mean? In a visual task the meaning is basically that the input is
 an image. So it could be a classifier where you have an image and you have to
 say that this is a cat. Or it could be, let's say, a medical image of a lung and
 you have to point out where the defect in the lung is or if there is a defect
 in the lung or something like this. As we all know this field is pretty much
 dominated by CNNs, by convolutional neural networks that take in these
 images through many layers of convolution. Especially residual networks
 are doing particularly well on these tasks. The problem of course is that in
 some tasks you have lots of data and that's fine because CNNs need lots of
 data to train. But in some tasks, especially these medical tasks, you only
 have like very small database. Look at this small database. You only have very
 few labeled samples where the model could learn from. And that just is not
 enough to learn these big models that would perform well. So you will have to
 settle for a less performing model. Now the solution or one of the solutions is
 transfer learning. In transfer learning what you do is you take a large data set,
 for example the ImageNet data set. You have this big data set right here and you
 train your CNN on that. And then you take that CNN and you do what's called a
 fine-tuning step on this small data set. So you take the CNN that you gained from
 the large data set as a starting point and then you just train for a few steps.
 You just kind of adapt it to this final data set that you actually want to train
 it on. And that usually helps. And why does that help? Because you sort of hope
 that the large data set and the small data set are at least somewhat
 overlapping in their... So the images in the large data set are somewhat
 similar to the images in the small data set. It doesn't need to be super similar
 but just somewhat. And you hope that the features that the CNN learns from the
 large data set are useful in the small data set. Because then if that is the
 case when you fine-tune on the small data set, that's this step down here,
 that's called a fine-tuning. When you fine-tune you can pretty much reuse
 those features. You only have to adjust them a little bit. And you just
 have to learn how to map the features to the output, which now is of course
 different than in the original task. But you won't have to rediscover the
 features. So that's why transfer learning can help. So the first phase is called
 pre-training. The second phase is called fine-tuning. Now the ultimate goal in
 this is the following. Imagine you have like a giant database of data.
 This is giant. Look at the size comparison to the others. And so you have
 this big big database of images. And you train a CNN on that big database of
 labeled images. Now what you're hoping is that you can do this once and then this
 one CNN trained on this giant data set will become the starting point for all
 kinds of small tasks now. So basically you can post this on a repository online
 and everyone that has a visual task will not train from scratch. But they will
 basically take this one CNN as a starting point. It is very similar to
 what people are doing right now with BERT or generally these transformer
 language models. You never want to train them from scratch. You always want to
 train from a pre-trained state that someone else has done. Because usually
 the big work is now shifted to the pre-training. So the goal is to find
 this one universal starting point for visual learning. And of course no better
 place to do this than Google. They certainly do have a giant databases of
 images. They certainly have lots of computation which we're going to see is
 very necessary for something like this. Now they do train three different models.
 Their model is called BIT and they train three different variants BIT. Small,
 medium and large. So the L model is trained on 300 million images. The medium
 model is trained on 14 million images. So this is the I think it's called JFT
 dataset. This here is called the ImageNet 21k dataset which looks pretty funky. It
 has like objects in front of weird backgrounds and stuff like that. And the
 small is simply trained on the 1.3 million ImageNet dataset. So I mean just
 look at this. We're in a situation now where the small model is trained is
 pre-trained on ImageNet just for reference. If you had imagined this five
 years ago this you would not have maybe you would have guessed it but it's still
 impressive. So they do release these two models here the medium and the small one
 pre-trained I believe. They don't release the large one which maybe that's the
 price we have to pay for getting the medium and the small one. The fact that
 now Google can use this in their products because they have probably
 spent a considerable amount of money in doing this. I'm not sure this is a
 philosophical discussion whether in the interest of science they should really
 solve. Because they do give the sort of exact training protocol. You just need
 the money basically. Alright but that's not topic of this video. So the
 models here are all pretty much just residual networks. They're all
 these ResNet 152 I think x4 which means that basically scale the width of
 each layer by a number of four from the original ResNet architecture and that's
 pretty much it. This is the architecture. There's nothing really new in
 this paper. The paper just details what exactly you have to do. Which
 things exactly matter when you pre-train these things and which ones don't.
 Therefore I believe it is a pretty good paper and I think that these
 models here the M and S models and maybe someone else trains an L model and
 releases it will sort of become the standard like we have in BERT now. So
 whenever you have a visual task you're just gonna start from those in practice.
 So this I think is mainly relevant for people in practice. Alright here you can
 see these models. First of all excellent excellent not labeling of your x-axis.
 Absolutely beautiful. The x-axis I believe is the number of samples per
 data class. So now they take their pre-trained model this bit L and they
 fine-tune it on these datasets. So ImageNet is one of the tasks they fine
 tune on or CIFAR 10, CIFAR 100 and so on. And first of all look on the right side
 this full thing. This is when you take the entire dataset. So often they
 outperform, they get state-of-the-art on the full datasets. Now they do compare
 against what they call generalist models. So generalist models are ones that have
 this particular training protocol where they train on one big giant database
 and then fine-tune to all the other tasks. They do not achieve state-of-the-art
 on all datasets in what they call specialist models. The specialist models
 would be such models that have this exact task in mind and therefore they
 don't care about other tasks. They outperform some of these specialist
 models but not all of them. So this is not the new state of the
 art in everything but it is in this transfer learning regime. And I think
 even more important if you see on the left this is in the small label regime.
 So here you have something like 100 or 25 or 10 or even 5 labels per class. And
 if you take 5 labels per class for CIFAR 10, this model, so of course you have to
 pre-train it first on the big dataset, but just taking 5 labels per class you
 still get like 94% accuracy on CIFAR 10. And that's pretty good. That is pretty
 impressive especially if you compare it to this baseline model here which is a
 ResNet pre-trained on just the ImageNet dataset. So that really shows you the
 power of pre-training with full data. So one thing they say is that in their big
 dataset in their 300 million images they make sure to remove all the images that
 then appear in the downstream tasks. Because otherwise it is fairly
 conceivable that this database here is just scraped from the internet and of
 course these tasks are often, like CIFAR 10, are also scraped from the internet
 and also ImageNet. And it is entirely conceivable of course that the
 test data is already here and they say we remove images but I think they just
 remove exact duplicates. So it could still be that someone has taken
 ImageNet and then kind of recoded it into another color scheme or whatnot or
 just compressed it a bit more and then they find these images on the web.
 So it's a little shaky this whole thing because these datasets might just be
 part of one another but you know given the results I do generally believe the
 improvements here. But yeah. So I guess what we need is like people to actually
 go out with cameras and shoot new pictures for a new test set. But in any
 case let's dive into how to pre-train something like this. So they divide their
 findings up in two parts. How to pre-train and how to fine-tune. So how to
 transfer to downstream tasks. And the methods they find are surprisingly
 easy. They say there are two components to pre-training. The first component is
 scale. So you have to have a lot of data and a lot of models and that is a pretty
 important recognition. So down here they have this ablation where they scale up
 the model and scale up the data. So look at this for example. You can see here you
 have the different datasets to pre-train on. So this is the small dataset, the
 medium dataset and the large dataset. So in this direction you have dataset size.
 Then here you have accuracy not labeled. Again I guess we can understand
 accuracy. That's fine. And there we have the different models. Now the
 larger the dot here is the larger the model architecture. And you can see
 within the individual bins the larger the model the better performance you
 usually get. But as you can see like here this improvement in the large models
 isn't as much as when you have much data. And you can also see by the slope of the
 line here the larger amounts of data help more when you have larger models. So
 only scaling up the data is not as effective as scaling up the
 data and the model at the same time. And in some cases like in this small
 architecture here it actually hurts to incorporate more data. At least they say
 that. And you can also see that here. And here it just doesn't help as much anymore
 if you incorporate more data. So if your model is too small you can't handle the
 big data. Of course there are weird effects like here the performance goes
 down and then up with the larger data. So this might actually be an effect of the
 images in these data sets being somewhat qualitatively different also with
 respect to the task that you are training for. But in general it holds that
 you need a combination of data set size and model size to go up. And this I think
 might be an indication of where we are in Belkin's double descent curve. So if
 you look at the researcher Mikhail Belkin and other people also
 research in this area they have this sort of empirical finding and hypothesis
 that if you plot a graph and here is the number of parameters in relation
 to the data set size. It's a number of parameters in relation to the size
 of data. And here is your validation loss. Then what happens as you have very
 little parameters you can add more parameters to your model to get better
 validation loss. This is you know we get a better model and we train
 that and we get better. And then at some point you'll start to overfit.
 We've all learned this in our general machine learning course and there is a
 point here, what is called the interpolation threshold, where you have
 this is one so the number of parameters is equal to the number of data points
 which is just interpolating your training data. Sorry the data point here
 that's train. But then the discovery sort of is that this comes down again and it
 stays down. So as you go up in number of data points, sorry number of parameters
 with the same data set you're perfectly fitting the training data set. You're
 past the number of data points in your model but still your validation
 loss comes down and there's various hypotheses why this could happen. And
 here we find ourselves maybe in this sort of situation where if you have a
 model right here and you want to scale it you want to add more data you can't
 just keep the model constant because if you add more data that will shift you to
 the left here because you add more data but you keep the number of
 parameters the same so this number will shift to the left and you actually go up
 in your validation loss. So maybe this is actually what's happening right here the
 fact that the model is too small. This is just a hypothesis by me. So if you want
 to up your number of data points you also have to up your number of
 parameters and that will keep it going and maybe these models here are more on
 this side of this interpolation threshold and the models where it
 doesn't happen might be more over here. Though that is a big thing to assume.
 Maybe not. Now that I think about it since they have even more parameters
 here they would be even more here somewhere so maybe you add a bunch of
 data it's just not as bad. There might be some weird interactions here.
 Like this. Who knows? Let's just skip this. In any case the message here is you
 need more model and more data at the same time. Alright then there is a second
 message a second recipe for pre-training. There we are. The second method is group
 normalization and weight standardization. So they criticize batch norm. Batch norm
 has of course been used a lot. That is where if you have a batch of data and
 you put it through your layers and it has some intermediate
 representation what you want to do is you want to calculate sort of the mean
 and variance of your data in each of the features and then make it such that it's
 nice mean one and standard deviation. So mean zero and standard deviation of one.
 That is called batch norm but of course it is dependent on your batch size. So it
 is dependent on how many data points you have because that's how well you can
 estimate these mean and variance parameters. And what people do nowadays
 is they take these batches and they group them into different groups and
 they distribute those groups onto many many machines which is called data
 parallelism especially with TPUs nowadays. You can just distribute
 everything to so many TPUs. I believe they say they distribute to something
 like 500 TPUs. They have a batch size of I think 4,000 and they
 distribute to 500 TPUs so that leaves them with eight samples per batch.
 So this is eight and eight is just not very good for batch norm and if you have
 to if you want to circumvent that you need to in each layer globally sync with
 all of the other workers your batch norm parameters and that slows you down. So
 people have gone around this using what they call group normalization and weight
 standardization. So these two techniques of weight standardization is a
 addition to group normalization. They don't require the other samples in the
 batch. They work on a per sample basis and they normalize the features within
 groups of each channel. So the group normalization groups together
 different features within a sample and then normalizes across that and the
 weight standardization is a bit like standardizing the features but it's
 standardizes the weights to be of a normal distribution. And just suffice to
 say these are standard techniques that you can build in and they allow you to
 not have to synchronize constantly between your workers at the training
 time which makes everything a lot faster and also not a problem that you just
 have eight samples per worker. So that's what they do. They do large data
 large models and group normalization with weight standardization. That's how
 they pre-train and then how do they fine-tune. They say they have a rule to
 select hyperparameters. They call the bit hyper rule and that's just sort of a
 formula of how you have one hyper parameter. So you have one I guess it's
 a hyper hyper parameter and that hyper hyper parameter you run through their
 rule and the rule will tell you what each of the hyper parameters should be.
 So it's like a lookup table basically. It's oh you set this one
 number and we give you the rest of the hyper parameters and that one rule works
 pretty well. So you only have to find for fine-tuning you only have to grid
 search over one hyper parameter. It's not really grid anymore is it? And then they
 basically decide on the training schedule length resolution and whether to
 do mixup regularization. Mixup is a technique that can help when you have
 very little data and what it does is it interpolates between data points and
 also trains on kind of like data points from half this class and half that class
 just to make more data available. But they all have this packed into this rule
 and they of course the exact settings of this rule are presented. So you can look
 it up then they have a data pre-processing, resize the image to a
 square, crop out small random square, randomly horizontally flip the image at
 training time. So they basically describe a standard training protocol here. You
 don't want to go mix it too up too much. The only thing they say surprisingly we
 do not use any form any of the following forms of regularization during downstream
 tuning, weight decay to zero, weight decay to initial parameters or dropout. I think
 they only use weight decay during pre-training and that's it. So let's
 look at some of the graphs. We've already seen some. Here is where they pretty much
 outperform the generalist, these generalist models on all of these tasks
 including this visual task adaptation benchmark. I've made a video about this.
 This is a benchmark that includes 19 different visual tasks from all over the
 place and they have significant improvement here as you can see. They do
 not always outperform these specialist models but as you can see they
 outperform for example this on the flowers data set and they come pretty
 close. Here you can also see how much they improve when pre-training on the
 larger data set. So far people have basically pre-trained on this ImageNet
 data set and now that they pre-train on the larger one of course they gain a lot
 of performance and the largest one isn't even in this table. So what I
 finally want to look at is this visual task adaptation benchmark. This consists
 of 19 tasks and they're divided into natural tasks which are kind of natural
 images and then specialized tasks which are let's say the medical images and not
 really natural and then structured tasks and the structured tasks isn't simply
 labeling or locating something. It is tasks where you have to maybe reason
 about something. So let's say there is an image and there is a cup right here and
 there is a glass right here and the question is what's to the left of the
 glass and there's a bunch of other stuff around here and you have to say
 the cup. So it sort of requires a structured understanding of the image
 and you can see the main performance boost here comes in the natural images
 which is to be expected. So you only get what you feed in and this 300 million
 image data set I'm pretty sure that's just a web scrape of photos or mainly
 photos. So the main improvement you're gonna get is on pictures that are similar
 to that as we said at the beginning and these natural tasks have images like
 that and you can see that the model here improves extremely in that category,
 improves slightly in this specialized thing and only improves a little bit in
 the structured tasks. So this as I said is to be expected. Just know if you
 use this model know what is in there. You have to know what it does, what it does
 well. It does well on natural images that are similar to what it was pre-trained
 on. Okay so they do have some analysis here and we've already went to most of
 them. I find this to be pretty impressive. So they say when they apply
 the standard computational budget of ImageNet pre-training when they scale
 up to the larger data set it seems detrimental. As you can see right here
 the performance actually goes down when you go to the larger data set. Only if
 you train longer then your improves. The axis labeling is just amazing here.
 Standard, long, longer. Oh how long you train for? Longer. Thanks. But I guess the
 point is taken that you have to invest more computation along with your
 bigger model and bigger data set. Sorry it's the same model but the bigger data
 set. They also make some other points here that if you for example if you
 decrease your learning rate too early or set your weight decay parameter
 differently that also hurts you. So on the right here you see a smaller weight
 decay initially looks better. So initially you're higher but through the
 training you end up at a worse place than a higher setting right here. I
 mean they make a big point out of this but who's to say that someone else
 doesn't come with like a ten times longer training and figures out that
 ultimately you start off like this and then maybe goes up super high. So to me
 the lessons learned here is pretty much that there's always a way to
 get more performance out of more compute and probably there is a way to schedule
 all of these things because that's combined with decaying learning rate and
 so on. There's probably a way to schedule these things with the current
 method. So with this particular method that would end up somewhere here we just
 haven't found it yet because it's so complex. I would guess that is the
 case. Here they make an interesting point that if you decay the learning rate too
 early then you also end up at a worse place. So this this dashed researcher
 the noob. So after eight GPU weeks which come on what is that eight GPU weeks
 that's just eight GPUs for a week. That's nothing nothing. It looks like this
 right it looks fairly flat and this researcher now decides to decay the
 learning rate and that results in this thing here. So decays the learning rate
 here here and here. Sorry not here. So the case learning right here and then it
 flattens out again and then decays the learning rate again ends up at this
 level. Yet if you train for longer you can see right here if you look over eight
 months you can see that there is a slight upward trend still and it hasn't
 converged yet and you can if you decrease the learning rate only later
 and always wait for this to fully converge then you will end up at a
 better place right here above 70. Again who's to say that if I just wait here
 there isn't a slight upward trend if I wait for eight GPU years or eight GPU
 solar system births then there might be even a better point to decay finally
 decay the learning rate and then go up. I mean again this this researcher here
 only takes point five million steps where you take two million. So that's the
 first point. The second point is ImageNet or visual state-of-the-art
 research is now officially out of the hands of academia. This is it. If
 you see things like if you see a paper dissing on people that only wait eight
 GPU weeks to decrease their learning rate for the first time and advocating
 that you should you know at least wait until eight GPU months actually they
 wait twice as long it's over that's it. Yeah bye bye. Maybe maybe you know you
 want to do some theory or something yeah bye bye. What I find interesting is the
 mistakes so since on CIFAR-10 they reach like 99.4 percent there's only a
 handful of mistakes that they're still making because it's not that large of a
 data set and they do classify it so red in particular I think means red is the
 ground truth label is correct but green is the machine is correct and the
 ground truth label is wrong and you can see there is a fair number of green
 things here right so the model says ship and the label says cat and the model
 says bird and the label says cat clearly this this would be one weird cat so it
 gets to the point where you you also have to expect these errors to be in the
 training set so it could just be that the model here doesn't necessarily even
 make those mistakes but it's just somewhat consistent with the training
 set in making the mistakes and also here on image net they have selected ones
 where you know the model says notebook but it's actually laptop and the model
 says mouse but it's actually spacebar you know the model says Alp and it's ski
 so or here the model the model says candle but it's a this is a dishwasher
 what so you see that the the the types of mistakes here we get to very quirky
 very fine-grained points in these models last thing I want to show I have never
 seen these image net 21k images these are just funky like look at that so
 here's the state-of-the-art previously I think said triceratops and the new
 model now says bit else as starfish good job bit L you wow probably the correct
 label would just be weird and this no okay I don't want to rag on this too
 much this is a cool paper I believe this will be the new starting point for a lot
 of practitioners in when they do visual tasks I always as always invite you to
 check out the paper subscribe to the channel leave a like leave a comment if
 you want I do read them usually and bye bye