Do ImageNet Classifiers Generalize to ImageNet? (Paper Explained)
deep learning machine learning imagenet cifar10 cifar10.1 generalization overfitting mturk arxiv vision models research hardness accuracy classifier resnet
Has the world overfitted to ImageNet? What if we collect another dataset in exactly the same fashion? This paper gives a surprising answer!

Paper: https://arxiv.org/abs/1902.10811
Data: https://github.com/modestyachts/ImageNetV2

Abstract:
We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

Authors: Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there today we're looking at to do image net classifiers Generalized to image net by Benjamin wrecked Rebecca are Olaf's Ludwig Schmidt and Vyshal Shankar So the premise of this paper is pretty simple We've been training models on image net now for a while Almost ten years to be exact image net is this data set with a lot of images millions of images categorized into many thousands of categories now the The classic part of image net that people know is that has about 1.5 million images in 1,000 different classes and this Image net was one of the main data sets now or has been for the last few years And as you can see on the right here the error rate year after year Was pretty much I think cut in half every year since 2012 When the first net Alex net was using deep learning instead of the classical visual computer vision approaches so we've been training on image net for a while and the idea or the question this paper asks if we Collect a second test set right so for image net we have a train and a test set If we now collect a second test set here test v2 right if We have a model that was trained on training here and evaluated on test Does it also perform well on this second test set right? the idea of this being that maybe over the years we've Tuned our hyper parameters and all such that the models perform well on that particular test set Let's call this v1 right and it might not it might not be as successful on a new test set So this paper goes about collecting a test set to image net in Exactly the way that the v1 test set was collected right? So they they try to match exactly the process of how v1 was collected To create another test set and then they evaluate models on that new test set They do this not only for image net but also for c4 10 which is a much smaller data set But also a lot of computer vision algorithms are evaluated on c4 10 So let's just put up a hypothesis here The hypothesis is that we have pretty much over fitted to image net by now. This is a very prestigious Number to get if you have state-of-the-art on image net and therefore Tuning your hyper parameters and your learning rate and everything such that it performs well on the test set v1 is very likely So this paper has the most important plots are like this It has two axes and on the bottom axis is the performance on v1 right and so performance and that means Accuracy basically so accuracy and on line two is v2 accuracy now Here is one Here is zero This line here means that if a model is performing 50% accuracy on v1 it is also performing 50% accuracy on v2 So being on this line would basically mean we have not over fitted and the model is performing equally well on both both sets So what what now if we assume that we have over fitted if we have over fitted? We would assume that you know the models that perform really poorly might also you know They perform kind of they're not really over fitted but over the years as we've gotten better on v1 We stray away from this so we get better on v1 But we don't really get better on v2 right and that means we've over fitted to v1 and this might even go down right the more The more kind of we might overfit to v1 The worse we're actually going to get on v2 So this this is kind of a meta over fitting so this is what we would expect if we over fit right over over fit to v1 And I think this was the initial hypothesis behind the people that ran this experiment to check Can we see an effect like this or is the effect more? A continuous one where we don't over fit and what they found was neither And these are basically these interesting plots here So again the dashed line here would be the not over fitting line So what they find if you for example look at image net every dot here is a model right? So this model right here is performing with like a 67 percent accuracy On v1 and it is performing with something like a 53 percent accuracy on v2 If you look at this line here what that means is that Every model kind of drops By about this much right so Not only not we don't see This and we don't see this but we see this line is shifted down And if you look closely especially on c410 you can see the line rather than being tilted like this is actually tilted A bit slanted upwards right? So the the the the angle is not is higher is greater the slope is greater than the one-to-one slope This is extremely interesting If you think about what does that mean? It means that if you Take a model right right here If you look at its order It's it's it's a this Let's let's look at this model here. This model is number one Best model in the world right it will still be number one on v2 This model here is number number three rank three on v1. It will also be rank three on v2 right? So the order of models is is pretty much constant So if if a model is doing well on v1 it is also doing well on v2 In relation to other models but every model experiences this drop in in accuracy And the most interesting part is that and again you can see this more here The the better you're doing the smaller this drop gets right this drop here is smaller than this drop here This is exactly counter to the notion of overfitting Where it seems the more accurate you get on v1 the more you're able to close this gap between v1 and v2 And if you extrapolate here you might as well think that once we are at at or actually sorry 100% is already here If you could go higher maybe or maybe you can see here that in the end these will actually converge But nevertheless if the models that are doing better on v1 aren't only not overfit But they are actually experience less of a drop with regards to a new test set So they generalize better to the new test set than the the worst models right And that is crazy and it is not only neural networks right so up here for up here you have the the deep neural networks and whatnot But you can also go with I believe some of these or even further down here are k nearest neighbor sorry k nearest neighbor classifiers and things like this So it doesn't seem to be a property of neural networks it really seems to be a property of the data set And this paper first of all goes over how they collected these and second of all their hypotheses and investigations into why this phenomenon exists Why we are all of a sudden worse on the new test set but completely worse in a different way than we expected compared to the original test set All right so they first say potential causes of accuracy drops and they propose a model They say here are two here is the entire difference between two data sets with regards to a classifier It can be decomposed into three different gaps and you see the first and the last part here are the ones from the left side So this is an expanding sum in the middle So there is the generalization gap the generalization gap refers to the gap that you have between different data sets of the same distribution So these this is like you know from the train versus test set you train on the training set and then you have a generalization gap to the test set In this case the generalization gap simply refers to the difference between the generalization to the first and to the second set They argue that this isn't really an issue here because they say they can put up confidence intervals So if those were identically distributed how much would the generalization gap be at maximum given some kind of confidence interval 95 confidence interval would only give you plus minus a one percent difference in generalization gap So they rule out that this is the reason for the big discrepancy Then have two others they have the adaptivity gap and the distribution gap So the adaptivity gap is what we hypothesized at the beginning It is the overfitting to the first data set or to one of the two data sets So if you have a big adaptivity gap then you have fitted much more to one than to the other data set Now because of the shape of the curve being the way it is they also rule out the adaptivity gap And we went over why right because it would look completely different than it does Now the only thing remaining here is this distribution gap So they explain that this difference here most likely comes from the fact that the old and the new test set have a different distribution And they go into why that is And I'm going to compress their hypothesis of why that is into a short summary Let's say we won't go over the entire paper They basically say that the Mechanical Turk part of the processing pipeline has a very big influence So what happens when you collect an ImageNet test set? You start with Flickr This is a big image database and the images as far as I can understand they are tagged and you can search for them and so on So they start by going to Flickr and searching for images And their ground truth class labels come, you may know this, from a system called WordNet And WordNet is sort of a linguistic classification of words into groups So it would have hierarchical, it would have animals, animal being a word and then below animal it would have dog and then it would have terrier And it would have these hierarchical groupings of these words And they search on Flickr for images and then they put the images to a human rater So the human rater on a system called Mechanical Turk, you may know it, you can just sign up there and do these kind of tasks They present the human with a grid of images and a class, terrier And they say please select all the images where a terrier appears, so the human might select this, this, this and this one And that will give you what they call a selection frequency So, selection frequency, so how often was a particular image selected given that class And of course the higher, so you do this over many, sorry, the selection frequency is across many of these Mechanical Turk workers So if for a given image the selection frequency is high, let's say going towards 1.0, that means every single human selected that image to be in this class So you can be pretty sure it's in the class, if it goes towards 0, then you can be pretty sure it's not in the class And this selection frequency criteria, the paper thinks that this is the main criteria why the datasets are of different difficulties, let's say Because even though they try to match the process exactly, even the questions they pose to the Turk workers They even restricted their flicker date range to the date range where the original image net was collected They still think that there is a difference in how the Mechanical Turk workers basically rated the images Or then after that how they were selected using the selection frequency So what they do is they do different, they select different images depending on criteria so they can test these hypotheses Their original V2 test set is called matched frequency Now what you do in matched frequency is you kind of play a little game What you do is every now and then you will implant an image here of the V1 test set So thereby, of the V1 test set, either of this class or of another class So you can kind of do a quality control From this you can now find out what is the selection frequency of images in the V1 class for Terrier And then you can simply select the same one in the V2 So if you know the selection frequency for V1 was 0.8, you can just put the threshold here at 0.8 And you know, you can be reasonably sure that you have selected a similar difficulty Or so you would think, right? So if they do it like this, then they get this drop that you saw at the beginning They also do this threshold 0.7, which I guess is arbitrary-ish And then they also say top images, where they say For each class we chose the 10 images with the highest selection frequency So these would be the sort of easiest ones And if you look at the graphs, and this is ImageNet for these different datasets So if you do the threshold 0.7 that they selected, now the old line was somewhere here Now the new line is much closer, right? You see that here And if you do these top images, so you just select the easy ones The new line is actually above, right? This is now Note that the red line here is above the black line, while here it's below It is still extremely interesting that still there is this almost linear relationship between the V1 and V2 accuracies And even here on this easier dataset there is So they basically hypothesize by thresholding differently for the new dataset You have a very good grip on the difficulty of the new dataset So this process of matching the selection frequency, so this matched frequency dataset It might actually not result in the same difficulty in dataset They do have some more experiments where they experiment with different difficulties So let's actually jump down there, it's a bit of a jump because there are over 70 pages in this paper The appendix has its own content directory, that's how crazy that is But I want to show you these plots, so here what they do is they do different bins of this set So they have bins of easy samples, less easy samples and so on And you can see that you have a pretty good grip on where this line is So if you only take the easy samples you're up here, and this is about what we saw before If this is the old line, it's the red line, right? If you take the entire new test set If you just take bin the second hardest bin you're somewhere here or here or here So you have a good hold on where this line is But that still doesn't explain that if you try to follow the protocol exactly Why does the accuracy drop? That is still a mystery Even though they say here is a variable that influences this a lot If they try to set the variable as it was set in v1, it is not equal And still mystery remains, I would say So the last thing they do is they try to just come up with a model for this So their hypothesis now is that the new test set just is harder And they have an analytical kind of a formal model of why if you assume certain things This results in this line, right? The really interesting thing about the paper is that the accuracies, they all fall exactly on this line There's this linear relationship, especially if you do like probit scaling of accuracies There's this line, and so they put up a model where you say What if we assume that each example i has a difficulty, right? It's just a number how difficult it is And each model j has a probability of correctly classifying an image with difficulty tau here Given by this function, right? So this here is the probability that the model will classify an image correctly, given that it's tau hard And so this is an increasing function And they put up this following parameterization This is the CDF of the... So they put up a model for this function now, right? For this, they say if we assume that it is like this That each model has a sort of a skill number So each model has a skill, and each image has a difficulty that is tau If the skill is higher than the tau, probably it will classify correctly If the skill is lower, probably, then this number is negative, it will classify it incorrectly So this is the CDF of a normal distribution, it goes something like this, right? And if the zero point is here, so if this number here is zero, it's like 50-50 Whether it will classify it correctly So if you assume this, and you assume a bunch of other Gaussian error distributions Then the performance of a model on the test set V2 is exactly the performance of the model under test set V1 Times this scalar here, plus this scalar here, which is a linear relationship So they put up a model for this Of course this doesn't explain anything, right? This doesn't explain the phenomena, but it still gives a clue of why the linear relationship here might result from the test set having a different difficulty setting Or a different difficulty properties So they go on, after discussing related work, they go on to say what can one do And suggestions for future research, I especially like the super holdout So if you ever make a data set, then make a super holdout set And once you're almost out of your career, just come up with it and say, oh I have this lost data set here that I made way back It will be fantastic Alright, so I think this paper is very interesting And I think everyone that sees and reads this comes up with their own hypothesis of why this is and what's going on here They have investigated a lot of this Especially I want to highlight an experiment where they taken part of V2 here So they split this one into a train and a test And they put this and this training together into like a super train So you train on both things together and they see whether it improves at this test set You would think that if you put this training in there that it would improve And it does improve, but it improves by like a miniscule amount So they've done a whole bunch of experiments like this to investigate what's going on This is all in this 70 page appendix that you can go over Alright, that was what I had to say for this paper If you like this video, consider subscribing and comment what you think I usually answer or like or read most comments Thanks for listening, bye bye
 Hi there today we're looking at to do image net classifiers
 Generalized to image net by Benjamin wrecked Rebecca are Olaf's Ludwig Schmidt and Vyshal Shankar
 So the premise of this paper is pretty simple
 We've been training models on image net now for a while
 Almost ten years to be exact image net is this data set with a lot of images millions of images
 categorized into many thousands of categories now the
 The classic part of image net that people know is that has about 1.5 million images in
 1,000 different classes and
 this
 Image net was one of the main data sets now or has been for the last few years
 And as you can see on the right here the error rate year after year
 Was pretty much I think cut in half every year since 2012
 When the first net Alex net was using deep learning instead of the classical
 visual computer vision approaches
 so we've been training on image net for a while and the
 idea or the question this paper asks if we
 Collect a second test set right so for image net we have a train and a test set
 If we now collect a second test set here test v2
 right if
 We have a model that was trained on training here and evaluated on test
 Does it also perform well on this second test set right?
 the idea of this being that maybe over the years we've
 Tuned our hyper parameters and all such that the models perform well on that particular test set
 Let's call this v1 right and it might not it might not be as successful on a new test set
 So this paper goes about collecting a test set to image net in
 Exactly the way that the v1 test set was collected right?
 So they they try to match exactly the process of how v1 was collected
 To create another test set and then they evaluate models on that new test set
 They do this not only for image net but also for c4 10 which is a much smaller data set
 But also a lot of computer vision algorithms are evaluated on c4 10
 So let's just put up a hypothesis here
 The hypothesis is that we have pretty much over fitted to image net by now. This is a very prestigious
 Number to get if you have state-of-the-art on image net and therefore
 Tuning your hyper parameters and your learning rate and everything such that it performs well on the test set v1 is very likely
 So this paper has the most important plots are like this
 It has two axes and on the bottom axis is the performance on v1 right and
 so performance
 and that means
 Accuracy basically so accuracy and on line two is v2 accuracy
 now
 Here is one
 Here is zero
 This line here means that if a model is performing
 50% accuracy on v1 it is also performing 50% accuracy on v2
 So being on this line would basically mean we have not over fitted and the model is performing equally well on both
 both sets
 So what what now if we assume that we have over fitted if we have over fitted?
 We would assume that you know the models that perform really poorly might also you know
 They perform kind of they're not really over fitted but over the years as we've gotten better on v1
 We stray away from this so we get better on v1
 But we don't really get better on v2 right and that means we've over fitted to v1 and this might even go down right the more
 The more kind of we might overfit to v1
 The worse we're actually going to get on v2
 So this this is kind of a meta over fitting so this is what we would expect if we over fit right over
 over fit to v1
 And I think this was the initial hypothesis behind the people that ran this experiment to check
 Can we see an effect like this or is the effect more?
 A continuous one where we don't over fit and what they found was neither
 And these are basically these interesting plots here
 So again the dashed line here would be the not over fitting line
 So what they find if you for example look at image net every dot here is a model right?
 So this model right here is performing with like a 67 percent accuracy
 On v1 and it is performing with something like a 53 percent accuracy on v2
 If you look at this line here what that means is that
 Every model kind of drops
 By about this much
 right
 so
 Not only not we don't see
 This and we don't see this but we see this line is shifted down
 And if you look closely especially on c410 you can see the line rather than being tilted like this is actually tilted
 A bit slanted upwards right?
 So the the the the angle is not is higher is greater the slope is greater than the one-to-one slope
 This is extremely interesting
 If you think about what does that mean?
 It means that if you
 Take a model right right here
 If you look at its order
 It's it's it's a this
 Let's let's look at this model here. This model is number one
 Best model in the world right it will still be number one on v2
 This model here is number number three rank three on v1. It will also be rank three on v2 right?
 So the order of models is is pretty much constant
 So if if a model is doing well on v1 it is also doing well on v2
 In relation to other models but every model experiences this drop in in accuracy
 And the most interesting part is that and again you can see this more here
 The the better you're doing the smaller this drop gets right this drop here is smaller than this drop here
 This is exactly counter to the notion of overfitting
 Where it seems the more accurate you get on v1 the more you're able to close this gap between v1 and v2
 And if you extrapolate here you might as well think that once we are at at or actually sorry 100% is already here
 If you could go higher maybe or maybe you can see here that in the end these will actually converge
 But nevertheless if the models that are doing better on v1 aren't only not overfit
 But they are actually experience less of a drop with regards to a new test set
 So they generalize better to the new test set than the the worst models right
 And that is crazy and it is not only neural networks right so up here for up here you have the the deep neural networks and whatnot
 But you can also go with I believe some of these or even further down here are k nearest neighbor sorry k nearest neighbor classifiers and things like this
 So it doesn't seem to be a property of neural networks it really seems to be a property of the data set
 And this paper first of all goes over how they collected these and second of all their hypotheses and investigations into why this phenomenon exists
 Why we are all of a sudden worse on the new test set but completely worse in a different way than we expected compared to the original test set
 All right so they first say potential causes of accuracy drops and they propose a model
 They say here are two here is the entire difference between two data sets with regards to a classifier
 It can be decomposed into three different gaps and you see the first and the last part here are the ones from the left side
 So this is an expanding sum in the middle
 So there is the generalization gap the generalization gap refers to the gap that you have between different data sets of the same distribution
 So these this is like you know from the train versus test set you train on the training set and then you have a generalization gap to the test set
 In this case the generalization gap simply refers to the difference between the generalization to the first and to the second set
 They argue that this isn't really an issue here because they say they can put up confidence intervals
 So if those were identically distributed how much would the generalization gap be at maximum given some kind of confidence interval
 95 confidence interval would only give you plus minus a one percent difference in generalization gap
 So they rule out that this is the reason for the big discrepancy
 Then have two others they have the adaptivity gap and the distribution gap
 So the adaptivity gap is what we hypothesized at the beginning
 It is the overfitting to the first data set or to one of the two data sets
 So if you have a big adaptivity gap then you have fitted much more to one than to the other data set
 Now because of the shape of the curve being the way it is they also rule out the adaptivity gap
 And we went over why right because it would look completely different than it does
 Now the only thing remaining here is this distribution gap
 So they explain that this difference here most likely comes from the fact that the old and the new test set have a different distribution
 And they go into why that is
 And I'm going to compress their hypothesis of why that is into a short summary
 Let's say we won't go over the entire paper
 They basically say that the Mechanical Turk part of the processing pipeline has a very big influence
 So what happens when you collect an ImageNet test set? You start with Flickr
 This is a big image database and the images as far as I can understand they are tagged and you can search for them and so on
 So they start by going to Flickr and searching for images
 And their ground truth class labels come, you may know this, from a system called WordNet
 And WordNet is sort of a linguistic classification of words into groups
 So it would have hierarchical, it would have animals, animal being a word and then below animal it would have dog and then it would have terrier
 And it would have these hierarchical groupings of these words
 And they search on Flickr for images and then they put the images to a human rater
 So the human rater on a system called Mechanical Turk, you may know it, you can just sign up there and do these kind of tasks
 They present the human with a grid of images and a class, terrier
 And they say please select all the images where a terrier appears, so the human might select this, this, this and this one
 And that will give you what they call a selection frequency
 So, selection frequency, so how often was a particular image selected given that class
 And of course the higher, so you do this over many, sorry, the selection frequency is across many of these Mechanical Turk workers
 So if for a given image the selection frequency is high, let's say going towards 1.0, that means every single human selected that image to be in this class
 So you can be pretty sure it's in the class, if it goes towards 0, then you can be pretty sure it's not in the class
 And this selection frequency criteria, the paper thinks that this is the main criteria why the datasets are of different difficulties, let's say
 Because even though they try to match the process exactly, even the questions they pose to the Turk workers
 They even restricted their flicker date range to the date range where the original image net was collected
 They still think that there is a difference in how the Mechanical Turk workers basically rated the images
 Or then after that how they were selected using the selection frequency
 So what they do is they do different, they select different images depending on criteria so they can test these hypotheses
 Their original V2 test set is called matched frequency
 Now what you do in matched frequency is you kind of play a little game
 What you do is every now and then you will implant an image here of the V1 test set
 So thereby, of the V1 test set, either of this class or of another class
 So you can kind of do a quality control
 From this you can now find out what is the selection frequency of images in the V1 class for Terrier
 And then you can simply select the same one in the V2
 So if you know the selection frequency for V1 was 0.8, you can just put the threshold here at 0.8
 And you know, you can be reasonably sure that you have selected a similar difficulty
 Or so you would think, right?
 So if they do it like this, then they get this drop that you saw at the beginning
 They also do this threshold 0.7, which I guess is arbitrary-ish
 And then they also say top images, where they say
 For each class we chose the 10 images with the highest selection frequency
 So these would be the sort of easiest ones
 And if you look at the graphs, and this is ImageNet for these different datasets
 So if you do the threshold 0.7 that they selected, now the old line was somewhere here
 Now the new line is much closer, right? You see that here
 And if you do these top images, so you just select the easy ones
 The new line is actually above, right? This is now
 Note that the red line here is above the black line, while here it's below
 It is still extremely interesting that still there is this almost linear relationship between the V1 and V2 accuracies
 And even here on this easier dataset there is
 So they basically hypothesize by thresholding differently for the new dataset
 You have a very good grip on the difficulty of the new dataset
 So this process of matching the selection frequency, so this matched frequency dataset
 It might actually not result in the same difficulty in dataset
 They do have some more experiments where they experiment with different difficulties
 So let's actually jump down there, it's a bit of a jump because there are over 70 pages in this paper
 The appendix has its own content directory, that's how crazy that is
 But I want to show you these plots, so here what they do is they do different bins of this set
 So they have bins of easy samples, less easy samples and so on
 And you can see that you have a pretty good grip on where this line is
 So if you only take the easy samples you're up here, and this is about what we saw before
 If this is the old line, it's the red line, right? If you take the entire new test set
 If you just take bin the second hardest bin you're somewhere here or here or here
 So you have a good hold on where this line is
 But that still doesn't explain that if you try to follow the protocol exactly
 Why does the accuracy drop? That is still a mystery
 Even though they say here is a variable that influences this a lot
 If they try to set the variable as it was set in v1, it is not equal
 And still mystery remains, I would say
 So the last thing they do is they try to just come up with a model for this
 So their hypothesis now is that the new test set just is harder
 And they have an analytical kind of a formal model of why if you assume certain things
 This results in this line, right?
 The really interesting thing about the paper is that the accuracies, they all fall exactly on this line
 There's this linear relationship, especially if you do like probit scaling of accuracies
 There's this line, and so they put up a model where you say
 What if we assume that each example i has a difficulty, right?
 It's just a number how difficult it is
 And each model j has a probability of correctly classifying an image with difficulty tau here
 Given by this function, right?
 So this here is the probability that the model will classify an image correctly, given that it's tau hard
 And so this is an increasing function
 And they put up this following parameterization
 This is the CDF of the... So they put up a model for this function now, right?
 For this, they say if we assume that it is like this
 That each model has a sort of a skill number
 So each model has a skill, and each image has a difficulty that is tau
 If the skill is higher than the tau, probably it will classify correctly
 If the skill is lower, probably, then this number is negative, it will classify it incorrectly
 So this is the CDF of a normal distribution, it goes something like this, right?
 And if the zero point is here, so if this number here is zero, it's like 50-50
 Whether it will classify it correctly
 So if you assume this, and you assume a bunch of other Gaussian error distributions
 Then the performance of a model on the test set V2 is exactly the performance of the model under test set V1
 Times this scalar here, plus this scalar here, which is a linear relationship
 So they put up a model for this
 Of course this doesn't explain anything, right?
 This doesn't explain the phenomena, but it still gives a clue of why the linear relationship here might result from the test set having a different difficulty setting
 Or a different difficulty properties
 So they go on, after discussing related work, they go on to say what can one do
 And suggestions for future research, I especially like the super holdout
 So if you ever make a data set, then make a super holdout set
 And once you're almost out of your career, just come up with it and say, oh I have this lost data set here that I made way back
 It will be fantastic
 Alright, so I think this paper is very interesting
 And I think everyone that sees and reads this comes up with their own hypothesis of why this is and what's going on here
 They have investigated a lot of this
 Especially I want to highlight an experiment where they taken part of V2 here
 So they split this one into a train and a test
 And they put this and this training together into like a super train
 So you train on both things together and they see whether it improves at this test set
 You would think that if you put this training in there that it would improve
 And it does improve, but it improves by like a miniscule amount
 So they've done a whole bunch of experiments like this to investigate what's going on
 This is all in this 70 page appendix that you can go over
 Alright, that was what I had to say for this paper
 If you like this video, consider subscribing and comment what you think
 I usually answer or like or read most comments
 Thanks for listening, bye bye