Learning To Classify Images Without Labels (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper ethz clustering self-supervision self-labeling entropy dot product representation learning cnns convolutional neural network deep cluster nce noise contrastive estimation unsupervised overcluster imagenet cifar10 nearest neighbors
How do you learn labels without labels? How do you classify images when you don't know what to classify them into? This paper investigates a new combination of representation learning, clustering, and self-labeling in order to group visually similar images together - and achieves surprisingly high accuracy on benchmark datasets.

OUTLINE:
0:00 - Intro & High-level Overview
2:15 - Problem Statement
4:50 - Why naive Clustering does not work
9:25 - Representation Learning
13:40 - Nearest-neighbor-based Clustering
28:00 - Self-Labeling
32:10 - Experiments
38:20 - ImageNet Experiments
41:00 - Overclustering

Paper: https://arxiv.org/abs/2005.12320
Code: https://github.com/wvangansbeke/Unsupervised-Classification

Abstract:
Is it possible to automatically classify images without the use of ground-truth annotations? Or when even the classes themselves, are not a priori known? These remain important, and open questions in computer vision. Several approaches have tried to tackle this problem in an end-to-end fashion. In this paper, we deviate from recent works, and advocate a two-step approach where feature learning and clustering are decoupled. First, a self-supervised task from representation learning is employed to obtain semantically meaningful features. Second, we use the obtained features as a prior in a learnable clustering approach. In doing so, we remove the ability for cluster learning to depend on low-level features, which is present in current end-to-end learning approaches. Experimental evaluation shows that we outperform state-of-the-art methods by huge margins, in particular +26.9% on CIFAR10, +21.5% on CIFAR100-20 and +11.7% on STL10 in terms of classification accuracy. Furthermore, results on ImageNet show that our approach is the first to scale well up to 200 randomly selected classes, obtaining 69.3% top-1 and 85.5% top-5 accuracy, and marking a difference of less than 7.5% with fully-supervised methods. Finally, we applied our approach to all 1000 classes on ImageNet, and found the results to be very encouraging. The code will be made publicly available.

Authors: Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Check out these clusters of images right here. And just have a look at how all of them are pretty much showing the same object. So here's balloons, here's birds, here's sharks or other fish. These are images from the ImageNet dataset. And you can see that these clusters are pretty much the object classes themselves. There's all the frogs right here, all the people that have caught fish. So the astonishing thing about this is that these clusters have been obtained without any labels of the ImageNet dataset. Of course, the dataset has labels, but this method doesn't use the labels. It learns to classify images without labels. So today we're looking at this paper, Learning to Classify Images Without Labels, by Wouter von Ganzbecke, Simon Vandenhende, Stamatios Georgoulis, Mark Prozémans and Luke van Gaal. And on a high-level overview, they have a three-step procedure. Basically, first, they use self-supervised learning in order to get good representations. Second, they do a clustering. So they do a sort of k-nearest neighbor clustering, but they do clustering on top of those things. But they do it in a kind of special way. And then third, they do a refinement through self-labeling. So if you know what all of these are, you basically understand the paper already. But there's a bit of tricky steps in there. And it's pretty cool that at the end it works out like you just saw. So before we dive in, as always, if you're here and not subscribed, then please do. And if you like the video, share it out. And leave a comment if you feel like commenting. Cool. So as we already stated the problem, they ask, is it possible to automatically classify images without the use of ground truth annotations? Or even when the classes themselves are not known a priori? Now, you might think that this is outrageous. How can you classify when you don't even know what the classes are and so on? So the way you have to imagine it going forward, and they don't explicitly explain it, but it's assumed that if you have a dataset, and you learn to classify it, what basically that means is you cluster it, right? You put some of the data points in the same clusters. And then, of course, the dataset, I'm going to draw the same dataset right here, the same dataset would have an actual classification thing. So this would be class 0, this here may be class 1, and this here might be class 2. Now, you can't possibly know how the classes are called or something, which one is the first, which one is the second. So at test time, basically, if you have a method like this that doesn't use labels, what you're going to do is you're basically going to find, you're going to be as generous as possible in the assignment of these and say, oh, look, if I assign this here to cluster 0 and this here to cluster 2 and this here to cluster 1, and I just carry over the labels, what would my accuracy be under that labeling? So you're as generous as possible with the assignments of the labels. So that's how it's going to work, right? That's what you have to keep in mind. We're basically developing an algorithm that gives us this kind of clustering of the data. And then if that clustering partitions the data in the same way as the actual labeling would, the actual labeling with the test labels, then we think it's a good algorithm. OK, so they claim they have a... OK, in this paper, we deviate from recent works and advocate a two-step approach. And it's actually a three-step approach, but where feature learning and clustering are decoupled. OK, why is that? So they argue what you could do, what people have done is... And I'm going to... Well, this is just a wall of text. So what you could do is you could just basically cluster the data. Like who says you can't use clustering algorithms? And then the question is, what do you cluster them by? Like you need a distance. So if I have points in 2D, it sort of makes sense to use the Euclidean distance here. But if I have images of cats and dogs and whatnot, then the Euclidean distance between the pixels is really not a good thing. But also, so you might think we could actually... We could use a deep neural network and then basically send the image, that's the image right here, send the image through the deep neural network and then either take this last state right here. So it goes through and through and through. And we could get take either of the hidden states or we could just take, you know, the last state, that is the sort of hidden representation right here and do the clustering with that. But then of course, the question is, what do you, which neural network do you take? How do you train that neural network? And there have been a few approaches such as a deep cluster, which try to formulate basically an objective for that neural network where you first, you send all the images through, right? You send a bunch of images through to get you in embedding space, to get you points. And then in embedding space, you think, well, the features that are in the embedding space, they are somehow latent and they... If basically the entire thing is, if this neural network was used to classify images, you would have a classification head on top. And a classification head, this is like a five class classification head, is nothing else than a linear classifier boundary that you put on top of this hidden representation. So if you were to use this neural network for classification, it must be possible to draw a linear boundary between the classes. And therefore, the either things like the inner product distance or the Euclidean distance must make sense in that space. They don't make sense in the picture space, but they must make sense in the hidden representation space, because what you're going to do with them is exactly linear classification. The last classification head of a neural network is just a linear classifier. So the assumption is that, and the conclusion is, well, in this space, you should be able to cluster by Euclidean distance. So what deep cluster does, like is first get the representations. You start off with a random neural network, then cluster these representations, then basically label, self label the images in a way. Now, way over simplifying that technique right here. But you have these alternative steps of clustering and then kind of finding better representation and then clustering these representations. And what it basically says is that the CNN itself is such a is like a prior, because it's the translation of it and works very good for very well for natural images. So the CNN itself will lead to good representations if we do it in this way. And they have some good results there. But this paper argues that if you do that, then the the algorithm tends to focus a lot on very low level features. So if the pixel on the bottom right here is blue, right, then you can. And the neural network, by chance, puts two of those images where the blue pixel on the bottom right, it puts them close together. Then in the next step, it will, because they're close together, will cluster them together. And then it will basically feed back the new representation should put the two in the same class, right? It will feed back that it should focus even more on that blue pixel. So it's very, very dependent on initializations and it can jump super easily onto these low level features that have nothing to do with with the high level task you're ultimately trying to solve, which is to classify these images later. So what this paper does is it says we can eliminate this. We can eliminate this the fact that these methods will produce will produce neural networks that focus on low level features. And how do we do that? We do that by representation learning. So representation learning, you might know this as self supervised learning. And this is the task they solve in the first step of their objective. So let's go through this. This right here is an image. Now, the T is a transformation of that image. And in self supervised learning, there are several methods that you can transform an image. So, for example, you can random crop an image. You can just cut out like a piece right here and scale that up to be as large as the original image. Or you can use, for example, data augmentation, which means you take the image and you basically so if there is, I don't know, the cat right here, you kind of convolve it with something. So there's like a very squiggly cat. OK, I'm terrible. You can you can rotate it, for example. So it's like this. OK, so these are all these are all sets, including the crop sets of this transformation T. So you transform it in some way and you want after you've transformed it, you send your original image. That should be read. You send your original image and the transformed image through a neural network, each one by themselves. OK, and then after this, you say the hidden representation here should be close to each other. OK, this is this is basically the self supervised training task. It's been shown to work very, very well as a pre training method for classification neural networks. You have an image and its augmented version and you minimize the inner product or the Euclidean distance between the two versions in the hidden space. And the rationale is exactly the same. The rationale is that this hidden space, of course, should be linearly classifiable. And so the distance between those should be close. And the rationale between having these tasks is that, well, if I flip the image, right, if I flip the image to the right, it cannot focus on the pixel on the bottom right anymore, because that's not going to be the pixel on the bottom right here. And I'm not always going to flip it into the same direction. And sometimes I'm going to crop it so it also can't focus on the pixel on the bottom right, because in the crop, that pixel is like out here. It's not even in the crop. But basically what you're looking to do with the self supervised methods is you are looking to destroy this low level information. That's that's all you're looking to build a pipeline of a neural network here that destroys deliberately low level information. And you do that by coming up with tasks like this self supervision tasks that just that deliberately exclude this information from being used. I think that's what's going on generally in the self supervised learning thing. OK, so this here, as you can see, is the neural network that you train. You send both images, the original and the augmented version, through the same neural network, and then you minimize some distance, which is usually like the inner product or the Euclidean distance in this embedding space. OK, and what you train, you can see right here, you train the parameters of this neural network. So the transformations are fixed or sampled and the distance is fixed. You train the neural networks such that your embeddings minimize this task. Now, this is nothing new. This has been this has been used for a couple of years now to get better representation. Self supervised learning is a thing. But they basically say we can use this as an initialization step for this clustering procedure, because if we don't do that, we we focus on these low level features. OK, and notice you don't need any labels for this procedure. That's why it's called self supervised. OK, so the second second part is the clustering. Now they cluster, but they don't just cluster these representations. That would be that doesn't perform very well in their in their experiments. What they instead do is they minimize this entire objective right here and we'll go through it step by step. So they train a new neural network. OK, this thing right here, this is a new neural network. So first you have you already have the neural network, which was called. What was it even called? The one that gives you the embedding with the theta. OK, it's called five theta. It's the same architecture. I think they initialize one with the other. So in step one, you get five theta five theta goes give from from X gives you a representation of X. OK, let's call it hidden X. So that's the self supervised learning. But in step two, you train an entirely new neural network, this five data here, and you initialize it with this one. But now you train it to do the following again. You want to minimize. Sorry, you want to maximize the inner product right here. See, that's the inner product. You want to maximize the inner product between two things. Now, that's the same thing as before. We want to minimize the distance between two things and the dot product distance. In that case, you maximize the dot product between two things. And the two things are two images that go through the same neural network as before. Right. This and this. Now, what's different here is that here we input an one image of the data set. That's the same as before. OK, so we input one image. But here before in the self supervised learning, we input an augmented version of that. And now we input something else. We input this K right here. Now, what's K? What K comes from this neighbor set of X. OK, this is the set of neighbors of X. And these neighbors are determined with respect to this neural network right here. So what you do after step one is you take your neural network with the good embeddings. And here is your data set X. Your data set X. This should be another. Your data set X is this list basically of all the images in your data set. And what you're going to do is you're going to take all of them using that neural network that you just trained and embed them into a latent space right here. OK. This is the latent space where you have done the self supervised training. And now for each image right here. So if this is X, I, you're going to find its K nearest neighbors. And they use I think they use five as a benchmark. So you're going to find its nearest neighbors. It's five nearest neighbors. And you do this for each image. So this image has these five nearest neighbors. So in step two, what you're trying to do is you're going to try to pull together each image and its nearest neighbors in that in this this not in this space directly, but you determine which ones are the nearest neighbor from this neural network and you keep it constant. That's how you determine what the nearest neighbors are in the first task. And that is your NX set for X, I. And in the second step, you're trying to make the representations of any image and its nearest neighbors closer to each other. OK, so with with this thing right here, you maximize the inner product between X in after this neural network and a nearest neighbor of X that was was a nearest neighbor after the first task. Now, the way they cluster here is not just again by putting it into an embedding space like we saw before. But this thing right here, this neural network, as you can see here, is is a C dimensional vector in zero one. Now, C is the number of classes that you can either know that. So you don't know which classes which you don't have labels, but you could know how many classes there are. Or you could just guess how many classes there are. And as long as you as you overguess, you can still like build super clusters later. So this they simply say it's in zero one, but they also say it performs a soft assignment. So we're also going to assume that this is normalized. So for each for each data point X here, you're going to you're going to have an image. You're going to put it through this new neural network. Okay, this new neural network new, and it's going to tell you it's going to give you basically a histogram. Let's say class one, two or three, we guess there are three class and it's going to give you an assignment of the three. And you also take a nearest neighbor. Here is your data set. You also take a nearest neighbor of that. So you look for this set N of X and you take a nearest neighbor. Maybe that's that's a maybe that's a dog. I can't I really can't draw dog. Yeah, that's the best I can do. I'm sorry. And you also put that through the same network. And you're saying since they were nearest neighbor in task one, they must share some sort of interesting high level features because that's what the first task was for. Therefore, I want to make them closer together in in the in the light of these of this neural network right here. So this is also going to give you an assignment like maybe like this. Okay. And now you object you you train this network right here to basically match these two distributions. Okay. So this is this is now a classifier into C classes, but we guess C and we don't have labels. We simply our label is going to be my neighbors from the first task must have the same labels. That's our label. Now they say they also have this term right here, which is the entropy over assignments. Okay. As you can see, so they minimize the following. They minimize this quantity, which has a negative in front of it. So that means they maximize this log inner product. And they also maximize the entropy because sorry. So they minimize this thing. But the entropy is a negative quantity. Right. So they maximize the entropy because here's a plus. And now they minimize the entropy. Let's see what they say by minimizing the following objective. Now entropy is the sum of the negative sum of P log P. And this if this is P. Yes, this is the probability that an image is going to be assigned to cluster C over the entire data set. So they're going to. Yes, so it's negative. This quantity negative. Minus P log P. And this is the entropy. So they're going to minimize the entropy. Let's see what they say. We include an entropy term. The second term in equation two. Which spreads the predictions uniformly across clusters C. OK. So what we want is a uniform assignment over cluster, which means we should maximize the entropy. Oh, yes. OK. They minimize this thing. And this here is the negative entropy. Right. So they want basically what they want over the whole data set that not all of the images are going to be in the same cluster. This is cluster one. And then this is cluster two. And then this is cluster three. So that term counteracts that basically the more evenly spread the entire data set distribution is the the higher the entropy, the lower the negative entropy. And that's the goal right here. I'm sorry. This this was I was confused by the too many negative signs. And then you minimize the entire thing. All right. Now, they say they say a different thing right here. They say here this bracket denotes the dot product operator. As we saw, it's the dot product between these two distributions right here. The first term in equation two imposes this neural network to make consistent predictions for a sample XI and its neighboring samples, the neighbors of XI. And here is an interesting thing. Note that the dot product will be maximal when the predictions are one hot. That means confident and assigned to the same cluster consistent. So they basically say the objective encourages confidence because it encourages predictions to be one hot and it encourages consistency because it you know the because the distributions need to be the same. They should be in the same cluster. Right now, I agree with the consistency. Like if you make the inner product high, then of the of two of these histograms, of course, they look the same. Right. Because these are ultimately vectors. These are three dimensional vectors. Let's call them two dimensional vectors. Right. So here is class one. Here's class two. If you make the inner product small or high, they will agree on their predictions. But I disagree that this encourages anything to be one hot. Like in my mind, if you have two vectors, they're both zero one times zero one. The inner product is going to be one. And if you have two assignments that are point five and point five, then it is also going to result in an in an inner product of it. Zero point five. Right. It's also going to to be no. So what's the inner product here? The inner product is point five times point five plus point five times point five, which is point five. Am I dumb? An embarrassingly long time later. Oh, it's because the L1 norm. OK, OK, we got it. We got it. I am I am OK. I am too dumb. Yes, of course, I was thinking of these vectors being normalized in L2 space where their inner products would always be one. But of course, if you have assignments between classes and it's a probability distribution, a histogram, then all of the prob possible assignments lie on this on this thing right here. Now, the inner product with yourself, of course, is the length of the vector and the length of a vector that points to one class or the other class is longer than a vector that points in between. So, OK, I see. That's where they get this. That's where they get this must be one hot from. So, OK, I'll give that to them. It is actually encouraging one hot predictions as long as these things are normalized in L1 space, which they probably are because they're histograms. Right. Yes, that was that was dumbness of me. I was trying to make a counter example. I'm like, wait a minute, this counter example is a counter example to my counter example. OK, so, yeah, that's that. So, as you can see, they are, of course, correct here and they now make the first experiments. So they say basically after the first step of the self supervised training, they can already retrieve sort of nearest neighbors and the nearest neighbors. The nearest neighbors of these images right here are the ones that you see on the right. And after the self supervised one, these nearest neighbors are already pretty good at sharing the high level features actually crazy, crazy good. Right. This flute here is in different sizes. As you can see, the fishes aren't aren't all exactly the same. The birds. So you can see it really focuses on sort of higher level features, but I guess it's really dependent on this higher level task. And they were they also investigate this quantitatively, but I just want to focus on how good is this after only the self supervised thing. And now they do this clustering and they can already sort of could already evaluate it right here because now they have a clustering. Right. After this step, they've basically pulled together the neighbors and they have this neural network that is now assigning classes. So they could already evaluate this and they are going to do that. But that's not good enough yet. Then they do a third step, which is fine tuning through self labeling. Now self labeling is pretty much exactly what it's what it says. It's you label your own data with your own classifier. Now that might be a bit outrageous. But it's basically saying, wait a minute, if I label my own data and learn a classifier on these labels, isn't isn't it just going to come out the same? And the answer is no. Right. If you have a data set because your classifier doesn't give you just first of all, if your classifier is something like this. Right. Just happens to be and you label and you learn a new classifier. It is going to be more like this. Right. Because it sort of maximizes a lot of classifiers maximize these distances between the classes. So even if it's like that and then the second step they do is they say, OK, there are some points where we are actually more confident about such as this one. We're more confident about that one. Also this one. And then this one here is pretty close. Like we're not super neither this one, but we're very confident about these two. So we're only going to use the ones where we are in fact confident about to learn to learn the new classifier. Or basically we you can also weigh them and so on. But they go by confidence right here, as you can see in this final algorithm. So this is the entire algorithm. And I got kicked away. Our algorithm. There we go. All right. So semantic clustering by adopting nearest neighbors, their scan algorithm. So in the first step, you do this pretext task. This is the self supervision, the representation learning. For your entire data set. No, sorry. This is this year. Optimize, optimize the neural network with task T. That's just self supervised representation learning. OK, then the second thing we're going to determine the nearest neighbor set for each X. Now they also in that step, they also augment the data. They do heavy data augmentation and so on. Also in this in the third step in the self labeling, they do data augmentation. There's a lot of tricks in here, but ultimately the base algorithm goes like this. So you find your neighboring sets for each X. And then what you do while your clustering loss decreases, you update this clustering neural network by with this loss that we saw. So this is the loss where you make the nearest neighbors closer to each other while still keeping the entropy high. OK, and then in the last after you've done this. You go through and you say, while the length of Y increases, what's why? Why is all the data points that are above a certain threshold? Now you're going to filter the data set that is above a certain threshold. And that's your data set Y. And you train this same neural network. You basically fine tune it with the cross entropy loss on your own labels. So now you only have labels Y. OK, so it's not it's not labels. You have the cross entropy loss between the assignments of this and the assignments of your data set. OK, so you basically do the same task, but you filter by confidence. And they use a threshold, I think, of point seven or something like this. Now let's go into the experiments, the experiments or look as follows. So they do some ablations to find out where in their methods kind of the the gains come from and will just quickly go through them. If they just do these self supervision at the beginning and then just do K means clustering on top of that, that will give them on C for 10 a thirty five point nine percent accuracy. So not very good. So the clustering, you can't just cluster on top of these representations and then be done. If they do what they say, so this is sample and batch entropy loss. This basically means you do not care about the nearest neighbors. You do this entire thing, but you only make an image close to the prediction, close to itself and its augmentations. So you don't use any nearest neighbor information also doesn't work. I wouldn't pay too much attention that the numbers are 10, 20 or 30. It just doesn't work. Now, if you use the scan loss, you all of a sudden you get into a regime where there is actual signal. So this is this is now significantly above the this is significantly above random guessing. And if you use strong data augmentation, as I said, is a lot of this is has these tricks in it of what kind of data augmentation you do and so on. So never forget that that these papers, besides their idea, they put in all the tricks they can. So you get 10 percent more. And then if you do this self labeling step, you get another 10 percent more. And this is fairly respectable, like eighty three point five without ever seeing labels. It's fairly good. But of course, there are only 10 classes right here. So keep that in mind. But they will do it on ImageNet later. And they investigate what kind of self supervision tasks at the beginning are important. And they investigate things like ROTNET feature decoupling and noise contrastive estimation, which noise contrastive estimation is the best. And noise contrastive estimation, I think, is just where you as we said, you input an image and then it's kind of noisy versions with augmented in various ways. And then you classify them together. This has been like this. These methods have been very successful in the last few years. Yeah, so this they have various investigations into their algorithm. I want to point out this here. This is the accuracy versus confidence after the complete clustering step. So this is now the third step, the self labeling. And you can see right here as this confidence of the network goes up, the actual accuracy goes up as well. So that means the network after the clustering is really more confident about the points that it can classify more accurately. There's like a correlation between where the network is confident and the actual label of the point, which is remarkable because it has never seen the label. But also see how sort of the range here is quite small. So with the standard augmentation, it goes like from here to here. So where you set that threshold is fairly important and might be quite brittle here because you need to set the threshold right such that some points are below it and some are above it. And you don't want to pull in points where you're not because if you pull in points from here, you're only you only have the correct label for 75 percent or something like them of them. And that means if you now self label and learn on them, you're going to learn the wrong signal. So this this step seems fairly brittle, honestly, but I don't know, of course. They go on and investigate various things such as how many clusters do you need or how many nearest neighbors? Sorry. Do you need this number K here? And you can see that if you have zero neighbors, then you're doing a lot worse than if you have, let's say, five nearest neighbors. So the jump here, as you can see, is fairly high in all the data sets. But after that, it sort of doesn't really matter much. So it seems like five nearest neighbors should be enough for most things. And here they just show that when they remove the false positives, that their algorithm actually converges to the correct clustering, the correct accuracy, which is not surprising. Like if you remove the wrong samples that are wrong, then the rest of the samples are going to be right. I think that's just showing that it doesn't go into some kind of crazy downward spiral loop or something like this. But still, it's just kind of funny. OK, so they do investigate how much they improve. And they improve by quite a lot above the kind of previous methods. So they have a lot of previous methods. But even this includes things like K means and so on, GANs, deep cluster that we spoke about. And this method, it already gets, as you can see, fairly close to good accuracy. So you have like 88.6% accuracy. And that's fairly remarkable on C410 without seeing the labels. But we'll go on. And now they go into ImageNet. Now ImageNet, of course, has way more classes. It has 1,000 classes compared to C410's 10 classes. So if you think clustering 10 classes might, and they're fairly apart from each other, might work with various techniques, ImageNet, 1,000 classes, that's way more difficult. Now they do sub sample this to 5100 and 200 classes. And they get OK accuracy. As you can see, they get 81% for 50 classes where a supervised baseline would get 86%. Into 200 classes, they get 69% where a supervised baseline would get 76%. So it's fairly, it's there. And that's quite remarkable for these low number of classes. And they figure out that if they look for the samples that are kind of in the most of the middle of their cluster, they get these prototypes right here. You can see all of these images. If you know ImageNet, some of the images really only have a part of the object and so on. So here with the prototypical things, you really get center clear shot of the object with clearly visible features and so on. So this sort of repeats the fact that this clustering really does go on that sort of semantic information. Of course, the labels here are from the test label set. The network can't figure that out. And then they go for 1,000 classes. And in 1,000 classes, it doesn't really work because there might be just too many confusions right here. But they do have this confusion matrix of their method. And it shows that the confusion matrix is pretty much a block diagonal along these super clusters right here. So you can see the dogs, the network confuses the dogs fairly often and then insects with each other, but not really across here. Which is still quite remarkable. But I mean, that's you get the same thing for a lot of these methods. So I don't I don't know how much different this would be in other methods. But certainly it's interesting to look at. Now, they go into one last thing, and that is what if we don't know how many clusters there are, right? If we don't know anything. So say so far, we have assumed to to have knowledge about the number of ground truth classes. The model predictions were valid losing the Hungarian matching algorithm. We already saw this in the DETR by Facebook, if you remember. However, what happens if the number of clusters does not match the number of ground truth classes anymore? So they now say table three reports the results when we overestimate the number of ground truth classes by a factor of two. OK, so now they build just 20 classes for C for 10 instead of 10 classes. And we'll look at table three real quick. Where's table three? This is table three. OK, so when they over cluster, you get the thing here on the bottom. And you can see there is a drop in accuracy right here. Now, what I don't actually they don't actually say how they do the over cluster matching. So if you imagine if I now have, I don't know, six clusters, but I need to assign them to three clusters, you know, here. Do I still use this most optimistic thing? So do I still use I think they still use this most optimistic matching right where you assign everything to its best fitted cluster. You compute all the permutations and then you give it the best benefit of the doubt. Now, if you imagine the situation where I over cluster to the point that I have each image in its own cluster and I run this algorithm to evaluate my clustering, I give it basically the most beneficial view, then I would get 100 percent accuracy. OK, so like in one of in these over cluster approach, I would sort of expect that you actually get a better score because you can like there is more generosity of the matching algorithm involved. Now, that's counteracted by the fact that you can't group together things that obviously have similar features because they are in the same class. So there's kind of two forces pulling here. But I was kind of astounded that it's going down and the evaluation method of this matching algorithm, it sort of breaks down when you have more classes, at least in my opinion. Yeah, but but it's interesting to see that you can just overshoot and but then you need some sort of heuristic to reconcile that. In any case, I think this paper is pretty cool. It brings together a lot of things that were already present and introduces this kind of this step approach. But what you have to keep in mind and by the way, there's lots of samples down here. What you have to keep in mind is there are a lot of hyperparameters in here. There are like this threshold and you know, the first of all, yeah, the number of classes, the thresholds, the architectures and so on. And all of this has been tuned to get these numbers really high. Right. All of these steps, all of the augmentations and so on, the chosen data augmentations. It has been chosen to get this number as high as possible. So, you know, to interpret this as, oh, look, we can classify without knowing the labels is, you know, yes, in this case, but the hyperparameter choices of the algorithm are all informed by the labels. So it is still very, very unclear of how this method will actually work when you really don't have the labels, when you actually have to choose the hyperparameters in absence of anything. And yeah, I think the future might tell if they continue to work on this. All right. Thanks for listening, looking, watching and bearing with me through my wrestling with various math, basic math in this video. I wish you a good day and bye bye.
 Hi there! Check out these clusters of images right here. And just have a look at how all of them are pretty much showing the same object.
 So here's balloons, here's birds, here's sharks or other fish. These are images from the ImageNet dataset.
 And you can see that these clusters are pretty much the object classes themselves.
 There's all the frogs right here, all the people that have caught fish.
 So the astonishing thing about this is that these clusters have been obtained without any labels of the ImageNet dataset.
 Of course, the dataset has labels, but this method doesn't use the labels. It learns to classify images without labels.
 So today we're looking at this paper, Learning to Classify Images Without Labels, by Wouter von Ganzbecke, Simon Vandenhende, Stamatios Georgoulis, Mark Prozémans and Luke van Gaal.
 And on a high-level overview, they have a three-step procedure. Basically, first, they use self-supervised learning in order to get good representations.
 Second, they do a clustering. So they do a sort of k-nearest neighbor clustering, but they do clustering on top of those things.
 But they do it in a kind of special way. And then third, they do a refinement through self-labeling.
 So if you know what all of these are, you basically understand the paper already.
 But there's a bit of tricky steps in there. And it's pretty cool that at the end it works out like you just saw.
 So before we dive in, as always, if you're here and not subscribed, then please do. And if you like the video, share it out.
 And leave a comment if you feel like commenting. Cool.
 So as we already stated the problem, they ask, is it possible to automatically classify images without the use of ground truth annotations?
 Or even when the classes themselves are not known a priori?
 Now, you might think that this is outrageous. How can you classify when you don't even know what the classes are and so on?
 So the way you have to imagine it going forward, and they don't explicitly explain it, but it's assumed that if you have a dataset,
 and you learn to classify it, what basically that means is you cluster it, right?
 You put some of the data points in the same clusters.
 And then, of course, the dataset, I'm going to draw the same dataset right here, the same dataset would have an actual classification thing.
 So this would be class 0, this here may be class 1, and this here might be class 2.
 Now, you can't possibly know how the classes are called or something, which one is the first, which one is the second.
 So at test time, basically, if you have a method like this that doesn't use labels, what you're going to do is you're basically going to find,
 you're going to be as generous as possible in the assignment of these and say,
 oh, look, if I assign this here to cluster 0 and this here to cluster 2 and this here to cluster 1,
 and I just carry over the labels, what would my accuracy be under that labeling?
 So you're as generous as possible with the assignments of the labels.
 So that's how it's going to work, right? That's what you have to keep in mind.
 We're basically developing an algorithm that gives us this kind of clustering of the data.
 And then if that clustering partitions the data in the same way as the actual labeling would,
 the actual labeling with the test labels, then we think it's a good algorithm.
 OK, so they claim they have a...
 OK, in this paper, we deviate from recent works and advocate a two-step approach.
 And it's actually a three-step approach, but where feature learning and clustering are decoupled.
 OK, why is that? So they argue what you could do, what people have done is...
 And I'm going to...
 Well, this is just a wall of text. So what you could do is you could just basically cluster the data.
 Like who says you can't use clustering algorithms?
 And then the question is, what do you cluster them by? Like you need a distance.
 So if I have points in 2D, it sort of makes sense to use the Euclidean distance here.
 But if I have images of cats and dogs and whatnot, then the Euclidean distance between the pixels is really not a good thing.
 But also, so you might think we could actually...
 We could use a deep neural network and then basically send the image, that's the image right here,
 send the image through the deep neural network and then either take this last state right here.
 So it goes through and through and through.
 And we could get take either of the hidden states or we could just take, you know, the last state,
 that is the sort of hidden representation right here and do the clustering with that.
 But then of course, the question is, what do you, which neural network do you take?
 How do you train that neural network?
 And there have been a few approaches such as a deep cluster,
 which try to formulate basically an objective for that neural network where you first, you send all the images through, right?
 You send a bunch of images through to get you in embedding space, to get you points.
 And then in embedding space, you think, well, the features that are in the embedding space,
 they are somehow latent and they...
 If basically the entire thing is, if this neural network was used to classify images,
 you would have a classification head on top.
 And a classification head, this is like a five class classification head,
 is nothing else than a linear classifier boundary that you put on top of this hidden representation.
 So if you were to use this neural network for classification, it must be possible to draw a linear boundary between the classes.
 And therefore, the either things like the inner product distance or the Euclidean distance must make sense in that space.
 They don't make sense in the picture space, but they must make sense in the hidden representation space,
 because what you're going to do with them is exactly linear classification.
 The last classification head of a neural network is just a linear classifier.
 So the assumption is that, and the conclusion is, well, in this space, you should be able to cluster by Euclidean distance.
 So what deep cluster does, like is first get the representations.
 You start off with a random neural network, then cluster these representations,
 then basically label, self label the images in a way.
 Now, way over simplifying that technique right here.
 But you have these alternative steps of clustering and then kind of finding better representation and then clustering these representations.
 And what it basically says is that the CNN itself is such a is like a prior,
 because it's the translation of it and works very good for very well for natural images.
 So the CNN itself will lead to good representations if we do it in this way.
 And they have some good results there.
 But this paper argues that if you do that, then the the algorithm tends to focus a lot on very low level features.
 So if the pixel on the bottom right here is blue, right, then you can.
 And the neural network, by chance, puts two of those images where the blue pixel on the bottom right, it puts them close together.
 Then in the next step, it will, because they're close together, will cluster them together.
 And then it will basically feed back the new representation should put the two in the same class, right?
 It will feed back that it should focus even more on that blue pixel.
 So it's very, very dependent on initializations and it can jump super easily onto these low level features that have nothing to do with with the high level task you're ultimately trying to solve, which is to classify these images later.
 So what this paper does is it says we can eliminate this.
 We can eliminate this the fact that these methods will produce will produce neural networks that focus on low level features.
 And how do we do that? We do that by representation learning.
 So representation learning, you might know this as self supervised learning.
 And this is the task they solve in the first step of their objective.
 So let's go through this. This right here is an image.
 Now, the T is a transformation of that image.
 And in self supervised learning, there are several methods that you can transform an image.
 So, for example, you can random crop an image.
 You can just cut out like a piece right here and scale that up to be as large as the original image.
 Or you can use, for example, data augmentation, which means you take the image and you basically so if there is, I don't know, the cat right here, you kind of convolve it with something.
 So there's like a very squiggly cat. OK, I'm terrible.
 You can you can rotate it, for example.
 So it's like this. OK, so these are all these are all sets, including the crop sets of this transformation T.
 So you transform it in some way and you want after you've transformed it, you send your original image.
 That should be read. You send your original image and the transformed image through a neural network, each one by themselves.
 OK, and then after this, you say the hidden representation here should be close to each other.
 OK, this is this is basically the self supervised training task.
 It's been shown to work very, very well as a pre training method for classification neural networks.
 You have an image and its augmented version and you minimize the inner product or the Euclidean distance between the two versions in the hidden space.
 And the rationale is exactly the same.
 The rationale is that this hidden space, of course, should be linearly classifiable.
 And so the distance between those should be close.
 And the rationale between having these tasks is that, well, if I flip the image, right, if I flip the image to the right,
 it cannot focus on the pixel on the bottom right anymore, because that's not going to be the pixel on the bottom right here.
 And I'm not always going to flip it into the same direction.
 And sometimes I'm going to crop it so it also can't focus on the pixel on the bottom right, because in the crop, that pixel is like out here.
 It's not even in the crop.
 But basically what you're looking to do with the self supervised methods is you are looking to destroy this low level information.
 That's that's all you're looking to build a pipeline of a neural network here that destroys deliberately low level information.
 And you do that by coming up with tasks like this self supervision tasks that just that deliberately exclude this information from being used.
 I think that's what's going on generally in the self supervised learning thing.
 OK, so this here, as you can see, is the neural network that you train.
 You send both images, the original and the augmented version, through the same neural network, and then you minimize some distance,
 which is usually like the inner product or the Euclidean distance in this embedding space.
 OK, and what you train, you can see right here, you train the parameters of this neural network.
 So the transformations are fixed or sampled and the distance is fixed.
 You train the neural networks such that your embeddings minimize this task.
 Now, this is nothing new. This has been this has been used for a couple of years now to get better representation.
 Self supervised learning is a thing.
 But they basically say we can use this as an initialization step for this clustering procedure,
 because if we don't do that, we we focus on these low level features.
 OK, and notice you don't need any labels for this procedure.
 That's why it's called self supervised.
 OK, so the second second part is the clustering.
 Now they cluster, but they don't just cluster these representations.
 That would be that doesn't perform very well in their in their experiments.
 What they instead do is they minimize this entire objective right here and we'll go through it step by step.
 So they train a new neural network. OK, this thing right here, this is a new neural network.
 So first you have you already have the neural network, which was called.
 What was it even called? The one that gives you the embedding with the theta.
 OK, it's called five theta. It's the same architecture.
 I think they initialize one with the other.
 So in step one, you get five theta five theta goes give from from X gives you a representation of X.
 OK, let's call it hidden X. So that's the self supervised learning.
 But in step two, you train an entirely new neural network, this five data here,
 and you initialize it with this one. But now you train it to do the following again.
 You want to minimize. Sorry, you want to maximize the inner product right here.
 See, that's the inner product. You want to maximize the inner product between two things.
 Now, that's the same thing as before. We want to minimize the distance between two things and the dot product distance.
 In that case, you maximize the dot product between two things. And the two things are two images
 that go through the same neural network as before. Right. This and this.
 Now, what's different here is that here we input an one image of the data set.
 That's the same as before. OK, so we input one image.
 But here before in the self supervised learning, we input an augmented version of that.
 And now we input something else. We input this K right here. Now, what's K?
 What K comes from this neighbor set of X. OK, this is the set of neighbors of X.
 And these neighbors are determined with respect to this neural network right here.
 So what you do after step one is you take your neural network with the good embeddings.
 And here is your data set X. Your data set X. This should be another.
 Your data set X is this list basically of all the images in your data set.
 And what you're going to do is you're going to take all of them using that neural network that you just trained
 and embed them into a latent space right here. OK.
 This is the latent space where you have done the self supervised training.
 And now for each image right here. So if this is X, I, you're going to find its K nearest neighbors.
 And they use I think they use five as a benchmark. So you're going to find its nearest neighbors.
 It's five nearest neighbors. And you do this for each image.
 So this image has these five nearest neighbors. So in step two, what you're trying to do is you're going to try to pull together
 each image and its nearest neighbors in that in this this not in this space directly,
 but you determine which ones are the nearest neighbor from this neural network and you keep it constant.
 That's how you determine what the nearest neighbors are in the first task.
 And that is your NX set for X, I. And in the second step, you're trying to make the representations
 of any image and its nearest neighbors closer to each other.
 OK, so with with this thing right here, you maximize the inner product between X in after this neural network
 and a nearest neighbor of X that was was a nearest neighbor after the first task.
 Now, the way they cluster here is not just again by putting it into an embedding space like we saw before.
 But this thing right here, this neural network, as you can see here, is is a C dimensional vector in zero one.
 Now, C is the number of classes that you can either know that.
 So you don't know which classes which you don't have labels, but you could know how many classes there are.
 Or you could just guess how many classes there are.
 And as long as you as you overguess, you can still like build super clusters later.
 So this they simply say it's in zero one, but they also say it performs a soft assignment.
 So we're also going to assume that this is normalized.
 So for each for each data point X here, you're going to you're going to have an image.
 You're going to put it through this new neural network.
 Okay, this new neural network new, and it's going to tell you it's going to give you basically a histogram.
 Let's say class one, two or three, we guess there are three class and it's going to give you an assignment of the three.
 And you also take a nearest neighbor.
 Here is your data set.
 You also take a nearest neighbor of that.
 So you look for this set N of X and you take a nearest neighbor.
 Maybe that's that's a maybe that's a dog.
 I can't I really can't draw dog.
 Yeah, that's the best I can do.
 I'm sorry.
 And you also put that through the same network.
 And you're saying since they were nearest neighbor in task one, they must share some sort of interesting high level features
 because that's what the first task was for.
 Therefore, I want to make them closer together in in the in the light of these of this neural network right here.
 So this is also going to give you an assignment like maybe like this.
 Okay.
 And now you object you you train this network right here to basically match these two distributions.
 Okay. So this is this is now a classifier into C classes, but we guess C and we don't have labels.
 We simply our label is going to be my neighbors from the first task must have the same labels.
 That's our label.
 Now they say they also have this term right here, which is the entropy over assignments.
 Okay.
 As you can see, so they minimize the following.
 They minimize this quantity, which has a negative in front of it.
 So that means they maximize this log inner product.
 And they also maximize the entropy because sorry.
 So they minimize this thing.
 But the entropy is a negative quantity.
 Right.
 So they maximize the entropy because here's a plus.
 And now they minimize the entropy.
 Let's see what they say by minimizing the following objective.
 Now entropy is the sum of the negative sum of P log P.
 And this if this is P.
 Yes, this is the probability that an image is going to be assigned to cluster C over the entire data set.
 So they're going to.
 Yes, so it's negative.
 This quantity negative.
 Minus P log P.
 And this is the entropy.
 So they're going to minimize the entropy.
 Let's see what they say.
 We include an entropy term.
 The second term in equation two.
 Which spreads the predictions uniformly across clusters C.
 OK.
 So what we want is a uniform assignment over cluster, which means we should maximize the entropy.
 Oh, yes.
 OK.
 They minimize this thing.
 And this here is the negative entropy.
 Right.
 So they want basically what they want over the whole data set that not all of the images are going to be in the same cluster.
 This is cluster one.
 And then this is cluster two.
 And then this is cluster three.
 So that term counteracts that basically the more evenly spread the entire data set distribution is the the higher the entropy, the lower the negative entropy.
 And that's the goal right here.
 I'm sorry.
 This this was I was confused by the too many negative signs.
 And then you minimize the entire thing.
 All right.
 Now, they say they say a different thing right here.
 They say here this bracket denotes the dot product operator.
 As we saw, it's the dot product between these two distributions right here.
 The first term in equation two imposes this neural network to make consistent predictions for a sample XI and its neighboring samples, the neighbors of XI.
 And here is an interesting thing.
 Note that the dot product will be maximal when the predictions are one hot.
 That means confident and assigned to the same cluster consistent.
 So they basically say the objective encourages confidence because it encourages predictions to be one hot and it encourages consistency because it you know the because the distributions need to be the same.
 They should be in the same cluster.
 Right now, I agree with the consistency.
 Like if you make the inner product high, then of the of two of these histograms, of course, they look the same.
 Right.
 Because these are ultimately vectors. These are three dimensional vectors.
 Let's call them two dimensional vectors.
 Right. So here is class one.
 Here's class two.
 If you make the inner product small or high, they will agree on their predictions.
 But I disagree that this encourages anything to be one hot.
 Like in my mind, if you have two vectors, they're both zero one times zero one.
 The inner product is going to be one.
 And if you have two assignments that are point five and point five, then it is also going to result in an in an inner product of it.
 Zero point five.
 Right. It's also going to to be no.
 So what's the inner product here?
 The inner product is point five times point five plus point five times point five, which is point five.
 Am I dumb?
 An embarrassingly long time later.
 Oh, it's because the L1 norm.
 OK, OK, we got it.
 We got it.
 I am I am OK.
 I am too dumb.
 Yes, of course, I was thinking of these vectors being normalized in L2 space where their inner products would always be one.
 But of course, if you have assignments between classes and it's a probability distribution, a histogram, then all of the prob possible assignments lie on this on this thing right here.
 Now, the inner product with yourself, of course, is the length of the vector and the length of a vector that points to one class or the other class is longer than a vector that points in between.
 So, OK, I see.
 That's where they get this.
 That's where they get this must be one hot from.
 So, OK, I'll give that to them.
 It is actually encouraging one hot predictions as long as these things are normalized in L1 space, which they probably are because they're histograms.
 Right.
 Yes, that was that was dumbness of me.
 I was trying to make a counter example.
 I'm like, wait a minute, this counter example is a counter example to my counter example.
 OK, so, yeah, that's that.
 So, as you can see, they are, of course, correct here and they now make the first experiments.
 So they say basically after the first step of the self supervised training, they can already retrieve sort of nearest neighbors and the nearest neighbors.
 The nearest neighbors of these images right here are the ones that you see on the right.
 And after the self supervised one, these nearest neighbors are already pretty good at sharing the high level features actually crazy, crazy good.
 Right. This flute here is in different sizes.
 As you can see, the fishes aren't aren't all exactly the same.
 The birds.
 So you can see it really focuses on sort of higher level features, but I guess it's really dependent on this higher level task.
 And they were they also investigate this quantitatively, but I just want to focus on how good is this after only the self supervised thing.
 And now they do this clustering and they can already sort of could already evaluate it right here because now they have a clustering.
 Right. After this step, they've basically pulled together the neighbors and they have this neural network that is now assigning classes.
 So they could already evaluate this and they are going to do that.
 But that's not good enough yet.
 Then they do a third step, which is fine tuning through self labeling.
 Now self labeling is pretty much exactly what it's what it says.
 It's you label your own data with your own classifier.
 Now that might be a bit outrageous.
 But it's basically saying, wait a minute, if I label my own data and learn a classifier on these labels, isn't isn't it just going to come out the same?
 And the answer is no.
 Right. If you have a data set because your classifier doesn't give you just first of all, if your classifier is something like this.
 Right.
 Just happens to be and you label and you learn a new classifier.
 It is going to be more like this.
 Right. Because it sort of maximizes a lot of classifiers maximize these distances between the classes.
 So even if it's like that and then the second step they do is they say, OK, there are some points where we are actually more confident about such as this one.
 We're more confident about that one.
 Also this one.
 And then this one here is pretty close.
 Like we're not super neither this one, but we're very confident about these two.
 So we're only going to use the ones where we are in fact confident about to learn to learn the new classifier.
 Or basically we you can also weigh them and so on.
 But they go by confidence right here, as you can see in this final algorithm.
 So this is the entire algorithm.
 And I got kicked away.
 Our algorithm.
 There we go.
 All right.
 So semantic clustering by adopting nearest neighbors, their scan algorithm.
 So in the first step, you do this pretext task.
 This is the self supervision, the representation learning.
 For your entire data set.
 No, sorry.
 This is this year.
 Optimize, optimize the neural network with task T.
 That's just self supervised representation learning.
 OK, then the second thing we're going to determine the nearest neighbor set for each X.
 Now they also in that step, they also augment the data.
 They do heavy data augmentation and so on.
 Also in this in the third step in the self labeling, they do data augmentation.
 There's a lot of tricks in here, but ultimately the base algorithm goes like this.
 So you find your neighboring sets for each X.
 And then what you do while your clustering loss decreases, you update this clustering neural network by with this loss that we saw.
 So this is the loss where you make the nearest neighbors closer to each other while still keeping the entropy high.
 OK, and then in the last after you've done this.
 You go through and you say, while the length of Y increases, what's why?
 Why is all the data points that are above a certain threshold?
 Now you're going to filter the data set that is above a certain threshold.
 And that's your data set Y.
 And you train this same neural network.
 You basically fine tune it with the cross entropy loss on your own labels.
 So now you only have labels Y.
 OK, so it's not it's not labels.
 You have the cross entropy loss between the assignments of this and the assignments of your data set.
 OK, so you basically do the same task, but you filter by confidence.
 And they use a threshold, I think, of point seven or something like this.
 Now let's go into the experiments, the experiments or look as follows.
 So they do some ablations to find out where in their methods kind of the the gains come from and will just quickly go through them.
 If they just do these self supervision at the beginning and then just do K means clustering on top of that,
 that will give them on C for 10 a thirty five point nine percent accuracy.
 So not very good.
 So the clustering, you can't just cluster on top of these representations and then be done.
 If they do what they say, so this is sample and batch entropy loss.
 This basically means you do not care about the nearest neighbors.
 You do this entire thing, but you only make an image close to the prediction, close to itself and its augmentations.
 So you don't use any nearest neighbor information also doesn't work.
 I wouldn't pay too much attention that the numbers are 10, 20 or 30.
 It just doesn't work.
 Now, if you use the scan loss, you all of a sudden you get into a regime where there is actual signal.
 So this is this is now significantly above the this is significantly above random guessing.
 And if you use strong data augmentation, as I said, is a lot of this is has these tricks in it of what kind of data augmentation you do and so on.
 So never forget that that these papers, besides their idea, they put in all the tricks they can.
 So you get 10 percent more.
 And then if you do this self labeling step, you get another 10 percent more.
 And this is fairly respectable, like eighty three point five without ever seeing labels.
 It's fairly good.
 But of course, there are only 10 classes right here.
 So keep that in mind.
 But they will do it on ImageNet later.
 And they investigate what kind of self supervision tasks at the beginning are important.
 And they investigate things like ROTNET feature decoupling and noise contrastive estimation,
 which noise contrastive estimation is the best.
 And noise contrastive estimation, I think, is just where you as we said, you input an image and then it's kind of noisy versions with augmented in various ways.
 And then you classify them together.
 This has been like this.
 These methods have been very successful in the last few years.
 Yeah, so this they have various investigations into their algorithm.
 I want to point out this here.
 This is the accuracy versus confidence after the complete clustering step.
 So this is now the third step, the self labeling.
 And you can see right here as this confidence of the network goes up, the actual accuracy goes up as well.
 So that means the network after the clustering is really more confident about the points that it can classify more accurately.
 There's like a correlation between where the network is confident and the actual label of the point, which is remarkable because it has never seen the label.
 But also see how sort of the range here is quite small.
 So with the standard augmentation, it goes like from here to here.
 So where you set that threshold is fairly important and might be quite brittle here because you need to set the threshold right such that some points are below it and some are above it.
 And you don't want to pull in points where you're not because if you pull in points from here, you're only you only have the correct label for 75 percent or something like them of them.
 And that means if you now self label and learn on them, you're going to learn the wrong signal.
 So this this step seems fairly brittle, honestly, but I don't know, of course.
 They go on and investigate various things such as how many clusters do you need or how many nearest neighbors?
 Sorry. Do you need this number K here?
 And you can see that if you have zero neighbors, then you're doing a lot worse than if you have, let's say, five nearest neighbors.
 So the jump here, as you can see, is fairly high in all the data sets.
 But after that, it sort of doesn't really matter much.
 So it seems like five nearest neighbors should be enough for most things.
 And here they just show that when they remove the false positives, that their algorithm actually converges to the correct clustering, the correct accuracy, which is not surprising.
 Like if you remove the wrong samples that are wrong, then the rest of the samples are going to be right.
 I think that's just showing that it doesn't go into some kind of crazy downward spiral loop or something like this.
 But still, it's just kind of funny.
 OK, so they do investigate how much they improve.
 And they improve by quite a lot above the kind of previous methods.
 So they have a lot of previous methods.
 But even this includes things like K means and so on, GANs, deep cluster that we spoke about.
 And this method, it already gets, as you can see, fairly close to good accuracy.
 So you have like 88.6% accuracy.
 And that's fairly remarkable on C410 without seeing the labels.
 But we'll go on.
 And now they go into ImageNet.
 Now ImageNet, of course, has way more classes.
 It has 1,000 classes compared to C410's 10 classes.
 So if you think clustering 10 classes might, and they're fairly apart from each other,
 might work with various techniques, ImageNet, 1,000 classes, that's way more difficult.
 Now they do sub sample this to 5100 and 200 classes.
 And they get OK accuracy.
 As you can see, they get 81% for 50 classes where a supervised baseline would get 86%.
 Into 200 classes, they get 69% where a supervised baseline would get 76%.
 So it's fairly, it's there.
 And that's quite remarkable for these low number of classes.
 And they figure out that if they look for the samples that are kind of in the most of the middle of their cluster,
 they get these prototypes right here.
 You can see all of these images.
 If you know ImageNet, some of the images really only have a part of the object and so on.
 So here with the prototypical things, you really get center clear shot of the object with clearly visible features and so on.
 So this sort of repeats the fact that this clustering really does go on that sort of semantic information.
 Of course, the labels here are from the test label set.
 The network can't figure that out.
 And then they go for 1,000 classes.
 And in 1,000 classes, it doesn't really work because there might be just too many confusions right here.
 But they do have this confusion matrix of their method.
 And it shows that the confusion matrix is pretty much a block diagonal along these super clusters right here.
 So you can see the dogs, the network confuses the dogs fairly often and then insects with each other, but not really across here.
 Which is still quite remarkable.
 But I mean, that's you get the same thing for a lot of these methods.
 So I don't I don't know how much different this would be in other methods.
 But certainly it's interesting to look at.
 Now, they go into one last thing, and that is what if we don't know how many clusters there are, right?
 If we don't know anything.
 So say so far, we have assumed to to have knowledge about the number of ground truth classes.
 The model predictions were valid losing the Hungarian matching algorithm.
 We already saw this in the DETR by Facebook, if you remember.
 However, what happens if the number of clusters does not match the number of ground truth classes anymore?
 So they now say table three reports the results when we overestimate the number of ground truth classes by a factor of two.
 OK, so now they build just 20 classes for C for 10 instead of 10 classes.
 And we'll look at table three real quick.
 Where's table three?
 This is table three.
 OK, so when they over cluster, you get the thing here on the bottom.
 And you can see there is a drop in accuracy right here.
 Now, what I don't actually they don't actually say how they do the over cluster matching.
 So if you imagine if I now have, I don't know, six clusters, but I need to assign them to three clusters, you know, here.
 Do I still use this most optimistic thing?
 So do I still use I think they still use this most optimistic matching right where you assign everything to its best fitted cluster.
 You compute all the permutations and then you give it the best benefit of the doubt.
 Now, if you imagine the situation where I over cluster to the point that I have each image in its own cluster
 and I run this algorithm to evaluate my clustering, I give it basically the most beneficial view, then I would get 100 percent accuracy.
 OK, so like in one of in these over cluster approach, I would sort of expect that you actually get a better score
 because you can like there is more generosity of the matching algorithm involved.
 Now, that's counteracted by the fact that you can't group together things that obviously have similar features because they are in the same class.
 So there's kind of two forces pulling here.
 But I was kind of astounded that it's going down and the evaluation method of this matching algorithm,
 it sort of breaks down when you have more classes, at least in my opinion.
 Yeah, but but it's interesting to see that you can just overshoot and but then you need some sort of heuristic to reconcile that.
 In any case, I think this paper is pretty cool.
 It brings together a lot of things that were already present and introduces this kind of this step approach.
 But what you have to keep in mind and by the way, there's lots of samples down here.
 What you have to keep in mind is there are a lot of hyperparameters in here.
 There are like this threshold and you know, the first of all, yeah, the number of classes, the thresholds, the architectures and so on.
 And all of this has been tuned to get these numbers really high.
 Right. All of these steps, all of the augmentations and so on, the chosen data augmentations.
 It has been chosen to get this number as high as possible.
 So, you know, to interpret this as, oh, look, we can classify without knowing the labels is, you know,
 yes, in this case, but the hyperparameter choices of the algorithm are all informed by the labels.
 So it is still very, very unclear of how this method will actually work when you really don't have the labels,
 when you actually have to choose the hyperparameters in absence of anything.
 And yeah, I think the future might tell if they continue to work on this.
 All right. Thanks for listening, looking, watching and bearing with me through my wrestling with various math,
 basic math in this video. I wish you a good day and bye bye.