FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
deep learning machine learning arxiv google semi-supervised unlabeled augmentation research randaugment
FixMatch is a simple, yet surprisingly effective approach to semi-supervised learning. It combines two previous methods in a clever way and achieves state-of-the-art in regimes with few and very few labeled examples.

Paper: https://arxiv.org/abs/2001.07685
Code: https://github.com/google-research/fixmatch

Abstract:
Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at this https URL.

Authors: Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi, today we're looking at FixMatch simplifying semi-supervised learning with consistency and confidence by Kyuk Son, David Berthelot and others of Google research. So this paper concerns semi-supervised learning. So what does semi-supervised learning mean? In semi-supervised learning you have a data set of labeled samples. So right, you have this data set of X's and corresponding Y labels. But this data set sometimes is very small. Now you have a much bigger data set of unlabeled examples, just X's with no labels, right? So you don't know what the labels of the unlabeled examples are, but what you would like to do is you would like to use this really large data set in order to help you with learning the association between the data points and the labels. So for example, in this case you would have something like an image classification data set. And I'm going to take the example here of medical data. So you have pictures of lungs. Let's draw a lung here. That is an ugly lung. You have pictures of lungs and whether or not they have a tumor in them. So medical data is very hard to get, especially labeled medical data. Because first of all you need the data itself, but then you also need at least one, but ideally three radiologists to look at whether or not this is a good or a bad image and label it. So it's usually very expensive to collect that data. But you might have plenty of unlabeled data, right? You might just be able to go through some database and find like anonymized, undiagnosed lung scans somewhere lying around. The same with image, like other images. So labeling images is pretty human intensive, but the internet contains like a whole bunch of unlabeled images. So the task of semi-supervised learning is how do you use this unlabeled data set in order to make your classification on the labeled data set easier. And FixMatch combines two approaches to this in a smart way, namely consistency and confidence approach. So what does... we'll jump right into the method. So basically what you want to do is you want to say my loss that I optimize, this is my loss, consists of two parts, namely a supervised loss, which is your classic classification loss, plus an unsupervised loss, right? And then you have like some sort of a trade-off parameter in front. Now your supervised loss here, this is just the cross entropy, let's call it H, between your predicted labels and the actual true labels, right? And the predicted labels, they can be, you know, kind of a distribution over labels. Now the magic of course is here in the unsupervised loss. And this unsupervised loss, this is what's described here in this part, right? So the unsupervised loss is going to be this H between P and Q, and we'll see what P and Q is. So if for the unsupervised loss you of course want to start with an unlabeled example, then you have the same sample go into two different pipelines. In the first pipeline up here, what you do is you so called weakly augmented. And here we're dealing with images, so we have to talk about image augmentation. So image augmentation has long been used in supervised learning to kind of give you more, it's kind of a cheat to give you more training data. So if you have an image, right, of let's say our famous cat, you can obtain more training data if you, for example, by random cropping. So you can random crop, let's say we just take this bottom right corner here, and then we enlarge it to the original size, right? Then it is still sort of a cat, but it's just a part of a cat, right? But usually that helps because you say, okay, my image data set is just pictures of animals, right? It's entirely conceivable that someone held the camera like this or like this, right? So technically in terms of generalizing to a test set, these both data points should be valid. So I'm just going to add both to my training data. So you can see how from one training data point you can get many training data points just by doing this cropping. What you can also do is you can flip it left right, right? You just swap the pixels left right, and usually these kind of... So a cat that has a little dark spot here is still a cat when it has the little dark spot over there, right? But to your classifier, those are two different samples. So you can do many of those things, and they have two kind of augmentations. They have what they call weakly augmented and strongly augmented, right? So in the weakly augmented pipeline, I think they just they crop and they shift and they rotate or something like this. So you can see here this horsey here, it is something like it's cropped here about, then it is turned slightly to the left, and then... Yeah, I think that's it. So they crop, they rotate, and then they also flip horizontally at random in like 50% of the time. So these are what's called weakly augmented. The goal here is just to kind of obtain a bit more training data, alright? So you run this through your model, through your classification model as you would a regular sample, and you get a prediction. Now from your prediction, you can take the highest prediction here, and that is going to be your pseudo-label. So this is P of Y, this is your distribution that you estimate, right? So and this, if you just take the max, this is going to be your Y hat, right? And this is what they call a pseudo-label, sorry. You'll see why it is called a pseudo-label. So the other pipeline here is the strong augmentation pipeline. Now in weak augmentation, we just wanted to get some more training data in strong augmentation. Now the goal is to really screw up that picture to the point where it's still, you know, you could recognize in the same class, but you can see here the augmentations, they go wild. So you play around with the color, with the hue, you play around with the light intensity, right? With the contrast, you can do many, many things. You can see this image looks basically nothing like this image, but you can still kind of recognize it as a horse. But the strongly augmented data is much more distorted than the weakly augmented data. And that's the point. So also you send the strongly augmented data through the model, and again you get a prediction, right? And now the trick is you take the label from here, and you take that as if it were the true label, right? You take that as if it were the true label, and you form a loss from this prediction being the model prediction, as if this thing here that also comes from the model, as if that was the true label, right? That's why it's called a pseudo label, because it is a label that you produce from the model itself. Now of course if these were to be the same picture, it would be kind of pointless, right? That's why you see there needs to be a weakly and a strongly augmented pipeline. I'm pretty sure if you want a more basic version of this, make this just clean, so no augmentation, and make this augmented, right? That's how you can think of it. The fact that there is weak and here strong augmentation I think is just your classic trick to get more training data. But in essence you can think of it as this is here, the clean thing, you just want to produce a label, and then you want that an augmented version of the image has the same label. Now you can think of it shortly, what does this model learn? If you just have this, you remember. I think the important thing is always to remember that there are two components here, right? There is first the supervised loss, this is the important one ultimately, because we have the true labels, right? And then second there is the unsupervised loss, which is just an auxiliary loss that is supposed to just kind of tune our model to the nature of the data, right? So don't forget that this down here just concerns the unsupervised part of that loss. So if you think what does the model actually learn whenever you train it like this, it basically learns to revert this strong augmentation, right? So it basically says, hey model, whenever I give you a weak augmented image and I distort it heavily, right? Whenever I give you an image and I distort it heavily, I want the label to be the same. So the model basically learns that whatever the image, the whatever the image, the model at the end of the training will be able to basically map any strongly augmented picture to the same class as a weakly augmented picture if it comes from the same source, right? So the model basically learns to ignore these kinds of augmentations. That's what this loss over here does. It basically says these sorts of augmentations, these sorts of distortions of images, please ignore those because I always want you to output the same label here in the prediction here as if I had not distorted or just weakly distorted the image. So that's what you have to keep in mind that this loss is designed to make the model not distinguish between differently augmented versions of the same image. And interestingly, that really seems to help with the supervised loss, right? My kind of hypothesis is that all these methods, what they're kind of trying to do is to just tune the neural network to the, let's say the orders of magnitude of the input data and also to the kinds of augmentations that the humans come up with. And that's a very important point. So the augmentations, and here we said, you know, it's kind of a rotation and the crop, the kind of augmentation really seemed to play a role. So this paper finds that on CIFAR-10, where the state of the art I believe is something like 96, 97 percent accuracy, on CIFAR-10 with just 250 labeled examples, right? Now the usual data set size is about 50,000. It goes to 94.9%. So almost 95 percent accuracy with the state of the art being like 97. This is incredible with just 250 labeled examples. Crazy, right? And with only four labels per class, it gets 88.6 percent. So that's just 40 images with labels. They get 88.6 percent of accuracy compared to the 97 percent that you get with like 50,000 images. That is pretty pretty cool, right? Simply by having all other images not labeled but pseudo labeled and consistency regularized, right? So the two things that are combined by FixMatch again are consistency regularization, which basically it means that the model should output similar predictions when fed perturbed versions of the same image, right? They're really forthcoming that they are not the ones who invented this. They just combine the consistency regularization with the pseudo labeling. Now the pseudo labeling they have also not invented. The pseudo labeling leverages the idea that we should use the model itself to obtain artificial labels for unlabeled data. We've seen a lot of papers in the last few months or years where it's like the teacher teaches the student and then the student teaches the teacher model again and so on. So they simply combine the two methods in a clever way. They have one last thing that is not in this drawing, namely they only use the pseudo label. They have a break right here and they only use the pseudo label if the confidence, so if this P of Y here is above a certain threshold. So they don't take all the pseudo labels but they only take the labels where the model is fairly sure about, right? So they have actually an ablation study where they show that this is reasonably important. And if you go down here where they say ablation, where is it? Ablation study, oh yeah something I also find cool. If you just give one image per class, one image per class, ten images that are labeled, it still gets like 78% accuracy. I think the images are chosen as good representations of their class but still one image per class. Pretty pretty cool. An important part of this is the ablation study where they say okay we want to tease apart why this algorithm, why this semi-supervised learning technique works so well. And they find several important factors. They find for example that their augmentation strategy is extremely important. So how they augment the images is very important. You see here the error of this 4.8% on the 250 label split. If you change up the augmentation strategies your error gets higher, right? So they say we use this cutout and we measure the effect of cutout. We find that both cutout and CCT augment are required to obtain the best performance. Removing either results in a comparable increase in error rate. Now you've seen before for example they went from this 93, sorry, 93 point something percent to 94 point something percent from the previous state-of-the-art semi-supervised learning. And here they find that simply changing the augmentation strategy changes the error by more than a percent. So you can just see this in context of what's important here. They say again the ratio of unlabeled data seems pretty important. We observe a significant decrease in error rates by using large amounts of unlabeled data. Then the optimizer and learning rate schedule seems to be very important as well in that they use this, they say SGD with momentum works much better than Adam and then they use this decreasing learning rate schedule, this cosine learning rate schedule. So there seem to be a lot of things, a lot of hyperparameters that are fairly important here. And you can see that the gains are substantial sometimes but they aren't like through the roof substantial, where you can make a good argument that it is unclear how much really comes from this clever combination that FixMatch proposes and how much also just comes from whether or not you set the hyperparameters correctly and exactly how much computation are you able to throw at selecting your hyper parameters. So that seems to be a bit of a pain point for me. They also say we find that tuning the weight decay is exceptionally important for low label regimes. Choosing a value that is just one order of magnitude larger or smaller than optimal can cost 10 percentage points or more. And so that all of that seems to me that this kind of research where you're nibbling for half or single percentage points in accuracy while a single misstep in a choice of hyper parameter might cost you 10 times that gain is a bit sketchy. Now I recognize they get numbers like no one else has gotten before but where exactly the gains come from and if the gains really come from this architecture or actually just more from throwing computers at it I don't know. Alright with that I hope you enjoyed this and I invite you to check out the paper. Bye bye.
 Hi, today we're looking at FixMatch simplifying semi-supervised learning
 with consistency and confidence by Kyuk Son, David Berthelot and others of
 Google research. So this paper concerns semi-supervised learning. So what does
 semi-supervised learning mean? In semi-supervised learning you have a
 data set of labeled samples. So right, you have this data set of X's and
 corresponding Y labels. But this data set sometimes is very small. Now you have a
 much bigger data set of unlabeled examples, just X's with no labels, right?
 So you don't know what the labels of the unlabeled examples are, but
 what you would like to do is you would like to use this really large data set
 in order to help you with learning the association between the data points and
 the labels. So for example, in this case you would have something like an
 image classification data set. And I'm going to take the example here of
 medical data. So you have pictures of lungs. Let's draw a lung here. That is an
 ugly lung. You have pictures of lungs and whether or not they have
 a tumor in them. So medical data is very hard to get, especially
 labeled medical data. Because first of all you need the data itself, but
 then you also need at least one, but ideally three radiologists
 to look at whether or not this is a good or a bad image and label it. So it's
 usually very expensive to collect that data. But you might have plenty of
 unlabeled data, right? You might just be able to go through some
 database and find like anonymized, undiagnosed lung scans somewhere lying
 around. The same with image, like other images. So labeling images is pretty
 human intensive, but the internet contains like a whole bunch of unlabeled
 images. So the task of semi-supervised learning is how do you use this
 unlabeled data set in order to make your classification on the labeled data set
 easier. And FixMatch combines two approaches to this in a smart way, namely
 consistency and confidence approach. So what does... we'll jump right
 into the method. So basically what you want to do is you want to say my loss
 that I optimize, this is my loss, consists of two parts, namely a
 supervised loss, which is your classic classification loss, plus an
 unsupervised loss, right? And then you have like some sort of a trade-off
 parameter in front. Now your supervised loss here, this is just the
 cross entropy, let's call it H, between your predicted labels and the
 actual true labels, right? And the predicted labels, they can be, you know,
 kind of a distribution over labels. Now the magic of course is here in the
 unsupervised loss. And this unsupervised loss, this is what's described here in
 this part, right? So the unsupervised loss is going to be this H between P and Q,
 and we'll see what P and Q is. So if for the unsupervised loss you of course
 want to start with an unlabeled example, then you have the same sample go into
 two different pipelines. In the first pipeline up here, what you do is you so
 called weakly augmented. And here we're dealing with images, so we have to talk
 about image augmentation. So image augmentation has long been used in
 supervised learning to kind of give you more, it's kind of a cheat to give you
 more training data. So if you have an image, right, of let's say our famous cat,
 you can obtain more training data if you, for example, by random cropping. So you
 can random crop, let's say we just take this bottom right corner here, and then
 we enlarge it to the original size, right? Then it is still sort of a cat, but it's
 just a part of a cat, right? But usually that helps because you say, okay,
 my image data set is just pictures of animals, right? It's entirely conceivable
 that someone held the camera like this or like this, right? So technically in
 terms of generalizing to a test set, these both data points should be valid.
 So I'm just going to add both to my training data. So you can see how from
 one training data point you can get many training data points just by doing this
 cropping. What you can also do is you can flip it left right, right? You just
 swap the pixels left right, and usually these kind of... So a cat that has a
 little dark spot here is still a cat when it has the little dark spot over
 there, right? But to your classifier, those are two different samples. So you can do
 many of those things, and they have two kind of augmentations. They have what
 they call weakly augmented and strongly augmented, right? So in the weakly
 augmented pipeline, I think they just they crop and they shift and they
 rotate or something like this. So you can see here this horsey here, it is
 something like it's cropped here about, then it is turned slightly to the left,
 and then... Yeah, I think that's it. So they crop, they rotate, and then they also flip
 horizontally at random in like 50% of the time. So these are what's called
 weakly augmented. The goal here is just to kind of obtain a bit more training
 data, alright? So you run this through your model, through your classification
 model as you would a regular sample, and you get a prediction. Now from your
 prediction, you can take the highest prediction here, and that is going to be
 your pseudo-label. So this is P of Y, this is your distribution that you
 estimate, right? So and this, if you just take the max, this is going to be
 your Y hat, right? And this is what they call a pseudo-label, sorry. You'll see why
 it is called a pseudo-label. So the other pipeline here is the strong
 augmentation pipeline. Now in weak augmentation, we just wanted to get some
 more training data in strong augmentation. Now the goal is to really
 screw up that picture to the point where it's still, you know, you could recognize
 in the same class, but you can see here the augmentations, they go wild. So you
 play around with the color, with the hue, you play around with the light intensity,
 right? With the contrast, you can do many, many things. You can see this image
 looks basically nothing like this image, but you can still kind of recognize it
 as a horse. But the strongly augmented data is much more distorted than the
 weakly augmented data. And that's the point. So also you send the strongly
 augmented data through the model, and again you get a prediction, right? And now
 the trick is you take the label from here, and you take that as if it
 were the true label, right? You take that as if it were the true label, and you
 form a loss from this prediction being the model prediction, as if this thing
 here that also comes from the model, as if that was the true label, right? That's
 why it's called a pseudo label, because it is a label that you produce from the
 model itself. Now of course if these were to be the same picture, it would be kind
 of pointless, right? That's why you see there needs to be a weakly and a
 strongly augmented pipeline. I'm pretty sure if you want a more basic version
 of this, make this just clean, so no augmentation, and make this augmented,
 right? That's how you can think of it. The fact that there is weak and
 here strong augmentation I think is just your classic trick to get more
 training data. But in essence you can think of it as this is here, the clean
 thing, you just want to produce a label, and then you want that an augmented
 version of the image has the same label. Now you can think of it shortly, what
 does this model learn? If you just have this, you remember. I think the important
 thing is always to remember that there are two components here, right? There is
 first the supervised loss, this is the important one ultimately, because we have
 the true labels, right? And then second there is the unsupervised loss, which is
 just an auxiliary loss that is supposed to just kind of tune our model to the
 nature of the data, right? So don't forget that this down here just
 concerns the unsupervised part of that loss. So if you think what does the model
 actually learn whenever you train it like this, it basically learns to
 revert this strong augmentation, right? So it basically says, hey model, whenever I
 give you a weak augmented image and I distort it heavily, right? Whenever I
 give you an image and I distort it heavily, I want the label to be the same.
 So the model basically learns that whatever the image, the whatever the
 image, the model at the end of the training will be able to basically map
 any strongly augmented picture to the same class as a weakly augmented
 picture if it comes from the same source, right? So the model basically learns to
 ignore these kinds of augmentations. That's what this loss over here does. It
 basically says these sorts of augmentations, these sorts of distortions
 of images, please ignore those because I always want you to output the same label
 here in the prediction here as if I had not distorted or just weakly distorted
 the image. So that's what you have to keep in mind that this
 loss is designed to make the model not distinguish between differently
 augmented versions of the same image. And interestingly, that really seems to help
 with the supervised loss, right? My kind of hypothesis is that all
 these methods, what they're kind of trying to do is to just tune the neural
 network to the, let's say the orders of magnitude of the input data and also
 to the kinds of augmentations that the humans come up with. And that's a very
 important point. So the augmentations, and here we said, you know, it's kind of a
 rotation and the crop, the kind of augmentation really seemed to play a
 role. So this paper finds that on CIFAR-10, where the state of the art I believe is
 something like 96, 97 percent accuracy, on CIFAR-10 with just 250 labeled
 examples, right? Now the usual data set size is about 50,000. It goes to 94.9%.
 So almost 95 percent accuracy with the state of the art being like 97.
 This is incredible with just 250 labeled examples. Crazy, right? And with
 only four labels per class, it gets 88.6 percent. So that's just 40 images with
 labels. They get 88.6 percent of accuracy compared to the 97 percent that
 you get with like 50,000 images. That is pretty pretty cool, right? Simply by
 having all other images not labeled but pseudo labeled and consistency
 regularized, right? So the two things that are combined by FixMatch again
 are consistency regularization, which basically it means that the model
 should output similar predictions when fed perturbed versions of the same image,
 right? They're really forthcoming that they are not the ones who
 invented this. They just combine the consistency regularization with the
 pseudo labeling. Now the pseudo labeling they have also not invented. The pseudo
 labeling leverages the idea that we should use the model itself to obtain
 artificial labels for unlabeled data. We've seen a lot of papers in the last
 few months or years where it's like the teacher teaches the student and then the
 student teaches the teacher model again and so on. So they simply combine
 the two methods in a clever way. They have one last thing that is not in this
 drawing, namely they only use the pseudo label. They have a break right here and
 they only use the pseudo label if the confidence, so if this P of Y here is
 above a certain threshold. So they don't take all the pseudo labels but they only
 take the labels where the model is fairly sure about, right? So they have
 actually an ablation study where they show that this is reasonably
 important. And if you go down here where they say ablation, where is it?
 Ablation study, oh yeah something I also find cool. If you just give one
 image per class, one image per class, ten images that are labeled, it still gets
 like 78% accuracy. I think the images are chosen as good representations of their
 class but still one image per class. Pretty pretty cool. An important part of
 this is the ablation study where they say okay we want to tease apart why this
 algorithm, why this semi-supervised learning technique works so well. And
 they find several important factors. They find for example that their
 augmentation strategy is extremely important. So how they augment the
 images is very important. You see here the error of this 4.8% on the
 250 label split. If you change up the augmentation
 strategies your error gets higher, right?
 So they say we use this cutout and we measure the effect of cutout. We find
 that both cutout and CCT augment are required to obtain the best performance.
 Removing either results in a comparable increase in error rate. Now you've
 seen before for example they went from this 93, sorry, 93 point something
 percent to 94 point something percent from the previous state-of-the-art
 semi-supervised learning. And here they find that simply changing the
 augmentation strategy changes the error by more than a percent. So you can just
 see this in context of what's important here. They say again the ratio
 of unlabeled data seems pretty important. We observe a significant decrease in
 error rates by using large amounts of unlabeled data. Then the
 optimizer and learning rate schedule seems to be very important as well in
 that they use this, they say SGD with momentum works much better than Adam and
 then they use this decreasing learning rate schedule, this cosine learning rate
 schedule. So there seem to be a lot of things, a lot of hyperparameters that are
 fairly important here. And you can see that the gains are substantial sometimes
 but they aren't like through the roof substantial, where you can make a good
 argument that it is unclear how much really comes from this clever
 combination that FixMatch proposes and how much also just comes from
 whether or not you set the hyperparameters correctly and exactly how
 much computation are you able to throw at selecting your hyper
 parameters. So that seems to be a bit of a pain point for me. They also
 say we find that tuning the weight decay is exceptionally important for low label
 regimes. Choosing a value that is just one order of magnitude larger or
 smaller than optimal can cost 10 percentage points or more. And so that
 all of that seems to me that this kind of research where you're
 nibbling for half or single percentage points in accuracy while a single
 misstep in a choice of hyper parameter might cost you 10 times that gain is
 a bit sketchy. Now I recognize they get numbers like no one else has gotten
 before but where exactly the gains come from and if the gains really come from
 this architecture or actually just more from throwing computers at it I don't
 know. Alright with that I hope you enjoyed this and I invite you to check
 out the paper. Bye bye.