Supervised Contrastive Learning
deep learning machine learning supervised learning classification classifier labels pretraining unsupervised self-supervised representation learning representations hidden space loss function google mit imagenet
The cross-entropy loss has been the default in deep learning for the last few years for supervised learning. This paper proposes a new loss, the supervised contrastive loss, and uses it to pre-train the network in a supervised fashion. The resulting model, when fine-tuned to ImageNet, achieves new state-of-the-art.

https://arxiv.org/abs/2004.11362

Abstract:
Cross entropy is the most widely used loss function for supervised training of image classification models. In this paper, we propose a novel training methodology that consistently outperforms cross entropy on supervised learning tasks across different architectures and data augmentations. We modify the batch contrastive loss, which has recently been shown to be very effective at learning powerful representations in the self-supervised setting. We are thus able to leverage label information more effectively than cross entropy. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. In addition to this, we leverage key ingredients such as large batch sizes and normalized embeddings, which have been shown to benefit self-supervised learning. On both ResNet-50 and ResNet-200, we outperform cross entropy by over 1%, setting a new state of the art number of 78.8% among methods that use AutoAugment data augmentation. The loss also shows clear benefits for robustness to natural corruptions on standard benchmarks on both calibration and accuracy. Compared to cross entropy, our supervised contrastive loss is more stable to hyperparameter settings such as optimizers or data augmentations.

Authors: Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, Dilip Krishnan

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at supervised contrastive learning by people from Google Research and MIT. Now this paper proposes a new loss for supervised learning. And you might recognize that this is a big claim. So forever now we've basically used this cross entropy loss in order to do supervised training of neural networks. This paper proposes to replace that with the supervised contrastive loss. And let's jump straight into the results here. They say our supervised contrastive loss outperforms the cross entropy loss with standard data augmentations such as auto augment and rand augment. So these are some of the previous state of the art data augmentation techniques used together with the cross entropy loss. And they say their supervised contrastive loss outperforms them. You can see here on ImageNet, which is the biggest vision benchmark or the most famous one, this new loss, the supervised contrastive loss, outperforms these other methods by something like a percent. One percent is a big improvement on ImageNet right now. So they claim it is a big claim, right? We recognize if this is true, this could be a game changer basically for all of supervised learning. And supervised learning is really the only thing right now in deep learning that works. So it could revolutionize the field. So here's the but. It is actually not a new loss to replace the cross entropy loss. And they do come about this pretty quickly. I don't think they're dishonest or lying or anything here. But it is sort of if you start reading you like what this is a new loss. It is not. It is a new way of pre training the network for a classification task. And so let's look into this. So if you look at what does what does it mean to build a classifier, this is what you usually do. You do supervised cross entropy training, you have an image and the image here is of a dog, you put it through your network, and you obtain a representation. So the representation here are is this last layer, or the second to last layer. And you put that through a classification layer and then a softmax. And what you get as an output is basically a probability distribution. And let's say you have three classes here. There's dog, there's cat, and there's horse. And let's say the network doesn't yet isn't yet trained very well. So the probability for dog here is fairly low. So this is basically what the network thinks of that image, like which class does it belong to with what probability. They also have this label right here. So the label dog for that image, what you do with that is you do a one hot vector. So that would look like this. So the one is at the position where the correct class is. And then the cross entropy loss takes all of this and does the following. There's a sum over all your classes. In this case, you have three classes. And let's call these the labels L. And you want to always take the label of the class times the log probability that the network thinks belongs to this class. So you can quickly see that this if the label is zero, so for all the incorrect classes, that means this entire term drops away. And only if the label is one, so only the correct class that will result in the log probability of the class where the label is the correct label. So in order to make this a loss, you actually have to put a negative sign in front of here because you want to this so this entire thing reduces to the log probability of the correct class. This is what you want to maximize. Therefore, you if you want to minimize something you need. So you minimize the negative log probability of the correct class, which means you maximize the probability. If you've never looked at the cross entropy loss like this, it is important to notice that you're going to say, hey, all this does is pull this here up, right? And it doesn't do anything to the other ones. But you have to realize that this softmax operation, since this is a probability distribution, all of this is normalized to sum up to one. So implicitly, you will push these down through the normalization, right? So what this does is it pushes the correct class up, and it pushes the other classes down. So this, to look at this is going to be important later. Because if you look at what this representation here does, so again, you have the network produces a representation here. This is 2000 dimensional, and then it does, it adds on top this classification layer, this classification layer is simply a linear layer, and then a softmax on top. So how you have to imagine this is that there is a representation space, this 2000 dimensional space, and the representations are made in such a way that the labels such that sorry, let's have three classes here. The representations are made in such a way that a linear classifier can separate them correctly, right? So here, this would be like a boundary. And then this would be another boundary. And this maybe would be another decision boundary. So you can see that linear classifier can separate the classes well. That is the goal. If you use this softmax cross entropy loss, that is implicitly what will happen in the representation space W. All it cares about is that the classes are on one side of the decision boundary and everything else is on the other side of a decision boundary. So if you have the network isn't trained very well at the beginning, and you maybe have a sample of the green class here, it will push the network such that the representation of that sample will go onto the other side of this decision boundary and it will push the decision boundary at the same time to make that happen more easily. Right, so it will optimize all of this at the same time. That's what you do. That's how you optimize the representations. So this work here, and another work has said, wouldn't it be great if the representation and decision boundaries weren't just trained at the same time for this, but we learn good representations first, such that classifying them becomes very simple. And in essence, what this paper says is, if we have a representation space W, shouldn't images of the same class, shouldn't we just make them close together? So without caring about decision boundaries, we just want them to be close to each other. And we want them to be far apart from other classes. If that happens, you can see that a linear classifier is going to have a very easy time separating these classes later. So that's exactly what this paper does. It has a pre training stage and a training stage. So in the pre training stage, this is over here, supervised contrastive. In the pre training stage, it simply tries to learn these representations, like over like down here, such that without the decision boundaries, class thing, images of the same class are close together, and images of different classes are far apart, which notice the subtle difference right to the cross entropy loss where you just care about them being on one or the other side of a decision boundary. And in stage this, so this stage one, and then in stage two, and there is where where it comes in, you basically freeze the network. So you freeze these weights down here, these are frozen, you don't train them anymore. All you train is this one classification layer. So the represent you actually freeze also the representation layer here, you only train the classifier on top in stage two, but you train it using softmax and using the cross entropy loss. So you you train the classifier in the old cross entropy way, using just normal supervised learning. So what we see here is that the stage one pre training is is what's training the network and the cross entropy loss only trains the classifier. Right, so let's look at how this pre training actually work what is using what it's using is a method called contrastive pre training. Now in contrastive pre training, and they have a little diagram up here, what this does is if you look at the classic way of doing contrastive pre training, you have to go to the unsupervised pre training literature, people have kind of discovered that they can improve a neural network by pre training it first in an unsupervised way. This is also called some of these methods are called self supervised. So the advantage here of self supervised or unsupervised pre training is that you don't need labels. What you want to do is simply to make the representation space somewhat meaningful, right? So you simply want the network to learn representations of images that are somehow meaningful, right? That are there. And here's how you do it. So you want to take an image like this dog here. And then you want to randomly augment this image, which just means you want to produce different versions of the same image. In this case down here, this is a random crop, it's cropped about here, it's still the same image but it's kind of a different version of it. In the case here, you can see that it's flipped left right and the brightness is slightly increased. So these are just different versions of the same image. And what you also want are what's called negatives. Natives are simply different images from your data set, right? For example, this or this or this, you don't care as long as they're different, right? You just sample a bunch. And what you want, so your embedding space and they make a big deal here that they are normalized and that seems to work better. But this is not necessary for the idea to work. The big idea is here that if you have an image right here, let's say this is the dog, and the blue dots here are the augmented versions of the same dog, and the green dots are all the other images in the data set. What you want is that all the images that come from the original same image are pulled close together and everything else is pushed apart. Right? So that's why these are called positives and these are called negatives. So the contrastive training basically means that you always want to have a set that you pull together in representation space and a set called the negatives that you push apart. So the network basically learns about these random transformations that you have here. The network learns what it means to come from the same image. It learns to be robust to these kind of transformations. It learns about the data in general and how to spread the data and embedding space with these transformations. So this usually ends up in a pretty good representation space and people have been using this in recent years in order to gain significant improvements. Now the problem here, if you specifically do this to pre-train a classifier is the thing they show on the right. So on the left here you have a picture of a dog. But if you just do this self-supervised, you do it without the labels. So it can happen that this image here shows up in the negatives, but it is also of a dog. And now this image here is going to end up maybe being this image here. And you see what happens to it. It's a green one. So it's going to get pushed apart. And this is going to make the entire task for the later classifier much harder because if they are pushed apart from each other, how is a linear classifier going to have them on the same side of the decision boundary while having everything else on a different side? So the task here is implicitly making the task for the later classifier harder by pushing apart samples that should be of the same class. And so this is not happening if you introduce labels to the pre-training objective. That's what they do, the supervised contrastive objective. Now you still, all you want to do is here, we're going to draw the same embedding space and we're going to draw this original dog image. And we're going to draw the augmented version of the original dog image. But now we also have the following. We also have these images, which are images of the same class. So we're going to put them in black here. And let's say the augmented versions around them in smaller black dots, augmented versions of those, right? You can augment them as well. And then you have the negative samples. And the negative samples are not just any images, but just images of different classes. So you just go over your mini batch and all everything that's of the same class becomes positives, including their augmentations, and everything that is not in the same class becomes negatives. And also you can augment them as well. So now we have a bunch of things in our embedding space. And our objective is simply going to be, again, we want to push away all the images that are not of the same class as our original, as our red original image, which is called the anchor. So all of this needs to be pushed away. But now we want to pull together all the augmented versions of the original image, but also we want to pull together all of the other images of the same class, including also their augmented versions. So all of this is going to be pulled together. So not only does the network learn about these augmentations, which again, for this idea, the augmentations aren't even necessary. The network learns a representation space where images of the same class are close together, which again is going to make the task of later linear classifiers that needs to separate this class from other classes very, very easy. And again, the other images aren't just going to be pushed away, but if they're from the same class, let's say this and this image are from the same class, all of those are going to be pushed apart from our red dot, but by themselves being pushed together to their own cluster here of their own class. I hope this makes sense. And I hope the difference to the cross entropy objective is sort of clear. The cross entropy objective simply from the beginning just cares about which side of the decision boundary you're on. While this pre training objective first cares to put things close together that are in the same class, and then the decision classifier will have a much easier time. The reason why this works better than the because because it's not entirely clear from the beginning that why this should work better because it's working with the same information. It's just because people have generally found that these pre training contrastive pre training objectives, they just are somewhat better at exploiting the information in the data set than if you just hammer on hammer with the contrastive sorry with the cross entropy loss from the beginning. So but it is not fully explained yet why this works better because it's working with the same data. Again, the difference here is that the previous methods of contrastive pre training the self supervised ones, they did not have access to the labels. And the advantage of that is you can have a giant database of unlabeled additional data that you do the pre training on. Whereas here we do the pre training, including the labels. So here, the label dog is an intrinsic part because we need to know which of the samples we need to pull together. But that also means we cannot leverage the maybe that we have more unlabeled data and unlabeled data is pretty cheap to obtain. So that's the advantages and disadvantages here. So this new loss, so they they do compare this here. And usually in these contrastive objectives, you have somewhat like two encoders, one to encode the the anchor and one to encode the augmented versions. And this one is like a momentum with shared weights and so on. All of this isn't really important. If you want to look into that look into papers like momentum contrast, or I did one on curl for reinforcement learning. I think the the general gist of it is clear. So they compare the formulation of their loss to the self supervised one, usually it takes the form of things like this. So one is the the anchor here, and then the zji would be the positive example. And you see here that the inner product between the anchor and the positive example, sorry about that, the inner product should be high, because here the loss is the negative of whatever is here. So if you minimize the loss, you say I want the inner product between my anchor, and whatever is the positive sample to be high, and everything else here, which includes the thing on the top, but it also includes everything else, I want the inner product to be low, and which is exactly the thing where you push you pull together the positives, and you push apart everything else. That, that is the standard objective that you had before they, they extend this, but it looks almost the same. So compared to the unsupervised objective now, first of all, they extend this such that you can have more than one positive sample. Now this is also possible in the unsupervised way. So they just augmented by this. And they also now this is the crucial part, they include the labels into the pre turning objective. So they say everywhere where I and J have the same label should be maximized in the inner product, so should be pulled together, while everything else is being pushed apart. Yeah, so they say we generalize to an arbitrary number of positives. And they also say contrastive power increases with more negatives. I think that's just a finding that they have that when they add more negatives, so when they increase the batch size, that contrastive power increases. They do analyze their gradient, which I find is pretty neat. You can already see that if you formulate a loss, of course, the gradient is going to go in the negative direction, but they make it clear that if you look at the gradient for the positive cases, what appears is this one minus p ij quantity and the p ij quantity is exactly the inner product between i and j normalized, of course. So if you minimum, so the gradient is going to point into the negative direction of that for the positives, which means you're going to pull them together. And it's going to push into this direction for the negative classes, which means you push them apart. And they also analyze what happens with relation to hardness. So they say there are two kinds of, if you just look at the positive samples, there are two kinds. There are easy positives where the network has already learned to match them closely, where the inner product is almost one. If you look at them, that means the p ij quantity is large, right? Because that is basically the inner product. And you look at this term, this term is exactly what we saw in the gradient. Then you see that this here, since this is one, this entire thing is zero. This is also high. This is close to one. So this entire thing is zero. This is almost zero. But if you have a hard positive where the network hasn't learned yet to align the inner product properly or align the representation properly, then the angle between the things again, these are normalized, the angle is they're approximately orthogonal. So the gradient magnitude is going to be this here is going to be approximately zero. So this is close to one and this here, since this is also zero is also close to one. So this is going to be larger than zero, which means that their loss focuses on the examples that are that the network cannot yet represent well, according to their objective, which makes sense, right? First of all, but second of all, it that is exactly the same thing as in the cross entropy loss. So if you look at the cross entropy loss and you have a situation where the network is really good already for a given sample, so it already puts a dog into the dog class, then the gradient will not be pulling much for that sample. It might mainly focuses on where you're still wrong. So it is like I appreciate the analysis, but it is not a notable difference. I think what they want to show is that their loss, if you do gradient descent really does what it is supposed to do, namely, first of all, it does this pulling together pushing apart of inner products for the positive and negative samples. And it mainly focuses on samples where you not yet have found a good representation to align them with others. It focuses on pairs that are not yet correctly close or together or far apart. They also connect this to the triplet loss, where they can show after some approximation, that if their loss only has one positive and one negative sample, it is going to be proportional to the triplet loss. The triplet loss is basically where you have an image and you find one positive, I think that's going to be of the same class right here, and you find one negative of a different class and you try to push those apart while pulling those together. The problem here, they say, is the problem of hard negative sampling. In order for this to make sense, you need the negative sample to be what's called a hard negative sample. So this is called this hard negative mining, because you only have one negative sample, you better make this something where the network can learn from. And if it's too easy, the network can't learn anything. And thereby you have the problem of hard negative mining, where you often have to filter through your mini batch or even through your data set to find a good negative sample to go along with this pair of positive samples. But I don't really see how their method, except that it has a bunch of positives and negative samples, except for that, which I guess you could also apply to the triplet loss. There's not really a difference here. Again, if your method is a contrastive method, you do have the problem that if you simply sample at random, your negative samples are going to become easier and easier over the training over the course of training. And you get the problem of at some point, you're going to have to do actively sample hard negatives. I think this paper just gets around it by having huge batch sizes. So yeah, but again, they do get state of the art on ImageNet for these types of networks and augmentation strategies. And they do look at how their loss appears to be more hyperparameter stable. So if they change out the augmentation, if they change the optimizer or the learning rate, you can see here that the spread in accuracy is much smaller than for the cross entropy loss except here, but it is hard to compare variances of things that don't have the same means in terms of accuracy. So take this on the right here with a grain of salt. They also evaluate this on corrupted ImageNet. So there's an ImageNet data set where it has several levels of corruptedness of the data set. And you can see your accuracy goes down, but the accuracy for the cross entropy loss goes down faster than for the supervised contrastive loss. You see they start together like this and they go further apart. Now it is not clear to me whether that's just an effect. Like if you just trained a supervised contrastive loss also to this level, whether it would fall off at the same speed or whether because it is the supervised contrastive loss, it would kind of match that curve. It's not clear whether that's really an effect of the difference of the losses or it's just an effect of the fact that they aren't the same accuracy to begin with. Again, this kind of shifting, you can't really compare things that have different means in the first place. But it is an interesting finding that their method is more stable to these corruptions. I just want to point out at the end, their training details just highlight they train for up to 700 epochs during the pre-training stage, which is, I think, standard but mad. And they trained up models with batch sizes up to 8192. So you need like a super TPU cluster to run these kind of things. And I am never exactly trusting of numbers like this. Even though it's kind of a good improvement, it is still like a 1% improvement. And in these small numbers, I just feel there might be a big effect that things like batch sizes and how much you put into computing, how much compute you put into it and what else you're doing. There might be so much influence of that, that I first want to see this replicated multiple times across the entire field before I'm going to really trust that this is a good thing to do. Alright, so I hope you like this. If you're still here, thank you. Consider subscribing. If you have a comment, please leave it. I usually read them. And with that, bye bye.
 Hi there, today we're looking at supervised contrastive learning by people from Google
 Research and MIT.
 Now this paper proposes a new loss for supervised learning.
 And you might recognize that this is a big claim.
 So forever now we've basically used this cross entropy loss in order to do supervised training
 of neural networks.
 This paper proposes to replace that with the supervised contrastive loss.
 And let's jump straight into the results here.
 They say our supervised contrastive loss outperforms the cross entropy loss with standard data
 augmentations such as auto augment and rand augment.
 So these are some of the previous state of the art data augmentation techniques used
 together with the cross entropy loss.
 And they say their supervised contrastive loss outperforms them.
 You can see here on ImageNet, which is the biggest vision benchmark or the most famous
 one, this new loss, the supervised contrastive loss, outperforms these other methods by
 something like a percent.
 One percent is a big improvement on ImageNet right now.
 So they claim it is a big claim, right?
 We recognize if this is true, this could be a game changer basically for all of supervised
 learning.
 And supervised learning is really the only thing right now in deep learning that works.
 So it could revolutionize the field.
 So here's the but.
 It is actually not a new loss to replace the cross entropy loss.
 And they do come about this pretty quickly.
 I don't think they're dishonest or lying or anything here.
 But it is sort of if you start reading you like what this is a new loss.
 It is not.
 It is a new way of pre training the network for a classification task.
 And so let's look into this.
 So if you look at what does what does it mean to build a classifier, this is what you usually
 do.
 You do supervised cross entropy training, you have an image and the image here is of
 a dog, you put it through your network, and you obtain a representation.
 So the representation here are is this last layer, or the second to last layer.
 And you put that through a classification layer and then a softmax.
 And what you get as an output is basically a probability distribution.
 And let's say you have three classes here.
 There's dog, there's cat, and there's horse.
 And let's say the network doesn't yet isn't yet trained very well.
 So the probability for dog here is fairly low.
 So this is basically what the network thinks of that image, like which class does it belong
 to with what probability.
 They also have this label right here.
 So the label dog for that image, what you do with that is you do a one hot vector.
 So that would look like this.
 So the one is at the position where the correct class is.
 And then the cross entropy loss takes all of this and does the following.
 There's a sum over all your classes.
 In this case, you have three classes.
 And let's call these the labels L. And you want to always take the label of the class
 times the log probability that the network thinks belongs to this class.
 So you can quickly see that this if the label is zero, so for all the incorrect classes,
 that means this entire term drops away.
 And only if the label is one, so only the correct class that will result in the log
 probability of the class where the label is the correct label.
 So in order to make this a loss, you actually have to put a negative sign in front of here
 because you want to this so this entire thing reduces to the log probability of the correct
 class.
 This is what you want to maximize.
 Therefore, you if you want to minimize something you need.
 So you minimize the negative log probability of the correct class, which means you maximize
 the probability.
 If you've never looked at the cross entropy loss like this, it is important to notice
 that you're going to say, hey, all this does is pull this here up, right?
 And it doesn't do anything to the other ones.
 But you have to realize that this softmax operation, since this is a probability distribution,
 all of this is normalized to sum up to one.
 So implicitly, you will push these down through the normalization, right?
 So what this does is it pushes the correct class up, and it pushes the other classes
 down.
 So this, to look at this is going to be important later.
 Because if you look at what this representation here does, so again, you have the network
 produces a representation here.
 This is 2000 dimensional, and then it does, it adds on top this classification layer,
 this classification layer is simply a linear layer, and then a softmax on top.
 So how you have to imagine this is that there is a representation space, this 2000 dimensional
 space, and the representations are made in such a way that the labels such that sorry,
 let's have three classes here.
 The representations are made in such a way that a linear classifier can separate them
 correctly, right?
 So here, this would be like a boundary.
 And then this would be another boundary.
 And this maybe would be another decision boundary.
 So you can see that linear classifier can separate the classes well.
 That is the goal.
 If you use this softmax cross entropy loss, that is implicitly what will happen in the
 representation space W. All it cares about is that the classes are on one side of the
 decision boundary and everything else is on the other side of a decision boundary.
 So if you have the network isn't trained very well at the beginning, and you maybe have
 a sample of the green class here, it will push the network such that the representation
 of that sample will go onto the other side of this decision boundary and it will push
 the decision boundary at the same time to make that happen more easily.
 Right, so it will optimize all of this at the same time.
 That's what you do.
 That's how you optimize the representations.
 So this work here, and another work has said, wouldn't it be great if the representation
 and decision boundaries weren't just trained at the same time for this, but we learn good
 representations first, such that classifying them becomes very simple.
 And in essence, what this paper says is, if we have a representation space W, shouldn't
 images of the same class, shouldn't we just make them close together?
 So without caring about decision boundaries, we just want them to be close to each other.
 And we want them to be far apart from other classes.
 If that happens, you can see that a linear classifier is going to have a very easy time
 separating these classes later.
 So that's exactly what this paper does.
 It has a pre training stage and a training stage.
 So in the pre training stage, this is over here, supervised contrastive.
 In the pre training stage, it simply tries to learn these representations, like over
 like down here, such that without the decision boundaries, class thing, images of the same
 class are close together, and images of different classes are far apart, which notice the subtle
 difference right to the cross entropy loss where you just care about them being on one
 or the other side of a decision boundary.
 And in stage this, so this stage one, and then in stage two, and there is where where
 it comes in, you basically freeze the network.
 So you freeze these weights down here, these are frozen, you don't train them anymore.
 All you train is this one classification layer.
 So the represent you actually freeze also the representation layer here, you only train
 the classifier on top in stage two, but you train it using softmax and using the cross
 entropy loss.
 So you you train the classifier in the old cross entropy way, using just normal supervised
 learning.
 So what we see here is that the stage one pre training is is what's training the network
 and the cross entropy loss only trains the classifier.
 Right, so let's look at how this pre training actually work what is using what it's using
 is a method called contrastive pre training.
 Now in contrastive pre training, and they have a little diagram up here, what this does
 is if you look at the classic way of doing contrastive pre training, you have to go to
 the unsupervised pre training literature, people have kind of discovered that they can
 improve a neural network by pre training it first in an unsupervised way.
 This is also called some of these methods are called self supervised.
 So the advantage here of self supervised or unsupervised pre training is that you don't
 need labels.
 What you want to do is simply to make the representation space somewhat meaningful,
 right?
 So you simply want the network to learn representations of images that are somehow meaningful, right?
 That are there.
 And here's how you do it.
 So you want to take an image like this dog here.
 And then you want to randomly augment this image, which just means you want to produce
 different versions of the same image.
 In this case down here, this is a random crop, it's cropped about here, it's still the same
 image but it's kind of a different version of it.
 In the case here, you can see that it's flipped left right and the brightness is slightly
 increased.
 So these are just different versions of the same image.
 And what you also want are what's called negatives.
 Natives are simply different images from your data set, right?
 For example, this or this or this, you don't care as long as they're different, right?
 You just sample a bunch.
 And what you want, so your embedding space and they make a big deal here that they are
 normalized and that seems to work better.
 But this is not necessary for the idea to work.
 The big idea is here that if you have an image right here, let's say this is the dog, and
 the blue dots here are the augmented versions of the same dog, and the green dots are all
 the other images in the data set.
 What you want is that all the images that come from the original same image are pulled
 close together and everything else is pushed apart.
 Right?
 So that's why these are called positives and these are called negatives.
 So the contrastive training basically means that you always want to have a set that you
 pull together in representation space and a set called the negatives that you push apart.
 So the network basically learns about these random transformations that you have here.
 The network learns what it means to come from the same image.
 It learns to be robust to these kind of transformations.
 It learns about the data in general and how to spread the data and embedding space with
 these transformations.
 So this usually ends up in a pretty good representation space and people have been using this in recent
 years in order to gain significant improvements.
 Now the problem here, if you specifically do this to pre-train a classifier is the thing
 they show on the right.
 So on the left here you have a picture of a dog.
 But if you just do this self-supervised, you do it without the labels.
 So it can happen that this image here shows up in the negatives, but it is also of a dog.
 And now this image here is going to end up maybe being this image here.
 And you see what happens to it.
 It's a green one.
 So it's going to get pushed apart.
 And this is going to make the entire task for the later classifier much harder because
 if they are pushed apart from each other, how is a linear classifier going to have them
 on the same side of the decision boundary while having everything else on a different
 side?
 So the task here is implicitly making the task for the later classifier harder by pushing
 apart samples that should be of the same class.
 And so this is not happening if you introduce labels to the pre-training objective.
 That's what they do, the supervised contrastive objective.
 Now you still, all you want to do is here, we're going to draw the same embedding space
 and we're going to draw this original dog image.
 And we're going to draw the augmented version of the original dog image.
 But now we also have the following.
 We also have these images, which are images of the same class.
 So we're going to put them in black here.
 And let's say the augmented versions around them in smaller black dots, augmented versions
 of those, right?
 You can augment them as well.
 And then you have the negative samples.
 And the negative samples are not just any images, but just images of different classes.
 So you just go over your mini batch and all everything that's of the same class becomes
 positives, including their augmentations, and everything that is not in the same class
 becomes negatives.
 And also you can augment them as well.
 So now we have a bunch of things in our embedding space.
 And our objective is simply going to be, again, we want to push away all the images that are
 not of the same class as our original, as our red original image, which is called the
 anchor.
 So all of this needs to be pushed away.
 But now we want to pull together all the augmented versions of the original image, but also we
 want to pull together all of the other images of the same class, including also their augmented
 versions.
 So all of this is going to be pulled together.
 So not only does the network learn about these augmentations, which again, for this idea,
 the augmentations aren't even necessary.
 The network learns a representation space where images of the same class are close together,
 which again is going to make the task of later linear classifiers that needs to separate
 this class from other classes very, very easy.
 And again, the other images aren't just going to be pushed away, but if they're from the
 same class, let's say this and this image are from the same class, all of those are
 going to be pushed apart from our red dot, but by themselves being pushed together to
 their own cluster here of their own class.
 I hope this makes sense.
 And I hope the difference to the cross entropy objective is sort of clear.
 The cross entropy objective simply from the beginning just cares about which side of the
 decision boundary you're on.
 While this pre training objective first cares to put things close together that are in the
 same class, and then the decision classifier will have a much easier time.
 The reason why this works better than the because because it's not entirely clear from
 the beginning that why this should work better because it's working with the same information.
 It's just because people have generally found that these pre training contrastive pre training
 objectives, they just are somewhat better at exploiting the information in the data
 set than if you just hammer on hammer with the contrastive sorry with the cross entropy
 loss from the beginning.
 So but it is not fully explained yet why this works better because it's working with the
 same data.
 Again, the difference here is that the previous methods of contrastive pre training the self
 supervised ones, they did not have access to the labels.
 And the advantage of that is you can have a giant database of unlabeled additional data
 that you do the pre training on.
 Whereas here we do the pre training, including the labels.
 So here, the label dog is an intrinsic part because we need to know which of the samples
 we need to pull together.
 But that also means we cannot leverage the maybe that we have more unlabeled data and
 unlabeled data is pretty cheap to obtain.
 So that's the advantages and disadvantages here.
 So this new loss, so they they do compare this here.
 And usually in these contrastive objectives, you have somewhat like two encoders, one to
 encode the the anchor and one to encode the augmented versions.
 And this one is like a momentum with shared weights and so on.
 All of this isn't really important.
 If you want to look into that look into papers like momentum contrast, or I did one on curl
 for reinforcement learning.
 I think the the general gist of it is clear.
 So they compare the formulation of their loss to the self supervised one, usually it takes
 the form of things like this.
 So one is the the anchor here, and then the zji would be the positive example.
 And you see here that the inner product between the anchor and the positive example, sorry
 about that, the inner product should be high, because here the loss is the negative of whatever
 is here.
 So if you minimize the loss, you say I want the inner product between my anchor, and whatever
 is the positive sample to be high, and everything else here, which includes the thing on the
 top, but it also includes everything else, I want the inner product to be low, and which
 is exactly the thing where you push you pull together the positives, and you push apart
 everything else.
 That, that is the standard objective that you had before they, they extend this, but
 it looks almost the same.
 So compared to the unsupervised objective now, first of all, they extend this such that
 you can have more than one positive sample.
 Now this is also possible in the unsupervised way.
 So they just augmented by this.
 And they also now this is the crucial part, they include the labels into the pre turning
 objective.
 So they say everywhere where I and J have the same label should be maximized in the
 inner product, so should be pulled together, while everything else is being pushed apart.
 Yeah, so they say we generalize to an arbitrary number of positives.
 And they also say contrastive power increases with more negatives.
 I think that's just a finding that they have that when they add more negatives, so when
 they increase the batch size, that contrastive power increases.
 They do analyze their gradient, which I find is pretty neat.
 You can already see that if you formulate a loss, of course, the gradient is going to
 go in the negative direction, but they make it clear that if you look at the gradient
 for the positive cases, what appears is this one minus p ij quantity and the p ij quantity
 is exactly the inner product between i and j normalized, of course.
 So if you minimum, so the gradient is going to point into the negative direction of that
 for the positives, which means you're going to pull them together.
 And it's going to push into this direction for the negative classes, which means you
 push them apart.
 And they also analyze what happens with relation to hardness.
 So they say there are two kinds of, if you just look at the positive samples, there are
 two kinds.
 There are easy positives where the network has already learned to match them closely,
 where the inner product is almost one.
 If you look at them, that means the p ij quantity is large, right?
 Because that is basically the inner product.
 And you look at this term, this term is exactly what we saw in the gradient.
 Then you see that this here, since this is one, this entire thing is zero.
 This is also high.
 This is close to one.
 So this entire thing is zero.
 This is almost zero.
 But if you have a hard positive where the network hasn't learned yet to align the inner
 product properly or align the representation properly, then the angle between the things
 again, these are normalized, the angle is they're approximately orthogonal.
 So the gradient magnitude is going to be this here is going to be approximately zero.
 So this is close to one and this here, since this is also zero is also close to one.
 So this is going to be larger than zero, which means that their loss focuses on the examples
 that are that the network cannot yet represent well, according to their objective, which
 makes sense, right?
 First of all, but second of all, it that is exactly the same thing as in the cross entropy
 loss.
 So if you look at the cross entropy loss and you have a situation where the network is
 really good already for a given sample, so it already puts a dog into the dog class,
 then the gradient will not be pulling much for that sample.
 It might mainly focuses on where you're still wrong.
 So it is like I appreciate the analysis, but it is not a notable difference.
 I think what they want to show is that their loss, if you do gradient descent really does
 what it is supposed to do, namely, first of all, it does this pulling together pushing
 apart of inner products for the positive and negative samples.
 And it mainly focuses on samples where you not yet have found a good representation to
 align them with others.
 It focuses on pairs that are not yet correctly close or together or far apart.
 They also connect this to the triplet loss, where they can show after some approximation,
 that if their loss only has one positive and one negative sample, it is going to be proportional
 to the triplet loss.
 The triplet loss is basically where you have an image and you find one positive, I think
 that's going to be of the same class right here, and you find one negative of a different
 class and you try to push those apart while pulling those together.
 The problem here, they say, is the problem of hard negative sampling.
 In order for this to make sense, you need the negative sample to be what's called a
 hard negative sample.
 So this is called this hard negative mining, because you only have one negative sample,
 you better make this something where the network can learn from.
 And if it's too easy, the network can't learn anything.
 And thereby you have the problem of hard negative mining, where you often have to filter through
 your mini batch or even through your data set to find a good negative sample to go along
 with this pair of positive samples.
 But I don't really see how their method, except that it has a bunch of positives and negative
 samples, except for that, which I guess you could also apply to the triplet loss.
 There's not really a difference here.
 Again, if your method is a contrastive method, you do have the problem that if you simply
 sample at random, your negative samples are going to become easier and easier over the
 training over the course of training.
 And you get the problem of at some point, you're going to have to do actively sample
 hard negatives.
 I think this paper just gets around it by having huge batch sizes.
 So yeah, but again, they do get state of the art on ImageNet for these types of networks
 and augmentation strategies.
 And they do look at how their loss appears to be more hyperparameter stable.
 So if they change out the augmentation, if they change the optimizer or the learning
 rate, you can see here that the spread in accuracy is much smaller than for the cross
 entropy loss except here, but it is hard to compare variances of things that don't have
 the same means in terms of accuracy.
 So take this on the right here with a grain of salt.
 They also evaluate this on corrupted ImageNet.
 So there's an ImageNet data set where it has several levels of corruptedness of the data
 set.
 And you can see your accuracy goes down, but the accuracy for the cross entropy loss goes
 down faster than for the supervised contrastive loss.
 You see they start together like this and they go further apart.
 Now it is not clear to me whether that's just an effect.
 Like if you just trained a supervised contrastive loss also to this level, whether it would
 fall off at the same speed or whether because it is the supervised contrastive loss, it
 would kind of match that curve.
 It's not clear whether that's really an effect of the difference of the losses or it's just
 an effect of the fact that they aren't the same accuracy to begin with.
 Again, this kind of shifting, you can't really compare things that have different means in
 the first place.
 But it is an interesting finding that their method is more stable to these corruptions.
 I just want to point out at the end, their training details just highlight they train
 for up to 700 epochs during the pre-training stage, which is, I think, standard but mad.
 And they trained up models with batch sizes up to 8192.
 So you need like a super TPU cluster to run these kind of things.
 And I am never exactly trusting of numbers like this.
 Even though it's kind of a good improvement, it is still like a 1% improvement.
 And in these small numbers, I just feel there might be a big effect that things like batch
 sizes and how much you put into computing, how much compute you put into it and what
 else you're doing.
 There might be so much influence of that, that I first want to see this replicated multiple
 times across the entire field before I'm going to really trust that this is a good thing
 to do.
 Alright, so I hope you like this.
 If you're still here, thank you.
 Consider subscribing.
 If you have a comment, please leave it.
 I usually read them.
 And with that, bye bye.