DINO: Emerging Properties in Self-Supervised Vision Transformers (Facebook AI Research Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial what is deep learning introduction to deep learning facebook facebook ai fair byol swav self supervised learning unsupervised feature learning unsupervised machine learning feature engineering stop gradient dino self distillation self-distillation segmentation maps visual transformer visual transformer self supervised imagenet
#dino #facebook #selfsupervised

Self-Supervised Learning is the final frontier in Representation Learning: Getting useful features without any labels. Facebook AI's new system, DINO, combines advances in Self-Supervised Learning for Computer Vision with the new Vision Transformer (ViT) architecture and achieves impressive results without any labels. Attention maps can be directly interpreted as segmentation maps, and the obtained representations can be used for image retrieval and zero-shot k-nearest neighbor classifiers (KNNs).

OUTLINE:
0:00 - Intro & Overview
6:20 - Vision Transformers
9:20 - Self-Supervised Learning for Images
13:30 - Self-Distillation
15:20 - Building the teacher from the student by moving average
16:45 - DINO Pseudocode
23:10 - Why Cross-Entropy Loss?
28:20 - Experimental Results
33:40 - My Hypothesis why this works
38:45 - Conclusion & Comments

Paper: https://arxiv.org/abs/2104.14294
Blog: https://ai.facebook.com/blog/dino-paws-computer-vision-with-self-supervised-transformers-and-10x-more-efficient-training
Code: https://github.com/facebookresearch/dino

My Video on ViT: https://youtu.be/TrdevFK_am4
My Video on BYOL: https://youtu.be/YPfUiOMYOEE

Abstract:
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

Authors: Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, I hope you have all seen this. This is a new system by Facebook AI and what you're seeing here is a visualization of the attention maps of that neural network. In the middle is a supervised baseline and on the right is this new system called Dino. It's not as much a system as it is a methodology for unsupervised pre-training of visual transformers. And you can see that this system has neither been trained to learn what a dog is nor has it been trained to do any sort of segmentation. Yet if you look at the attention maps, it clearly can track objects, it knows what to pay attention to in the images, and it can do much more than that. So here you can see that it can sort of track objects behind occlusion. So the ship goes behind the waves, the horse goes behind the grass. And you can see in the attention map that this is well reflected. You can do more than that though, even. So if you use this feature representation that this model gives you for ImageNet, then as the model gets trained and you represent ImageNet and its feature space, it will cluster the images of the same class, it will cluster them together, which is already pretty cool because it has no labels at training time. But also it will cluster similar classes with each other, which speaks to the fact that this might be the next step in unsupervised representation learning for images. And specifically, it appears that the features that come out of a network that is trained with Dyno are extremely valuable for the kinds of things we are interested in when working with natural images. So this is image retrieval and classification. So this system, let's just switch over to the paper right here. The paper is called Emerging Properties in Self-Supervised Vision Transformers. It presents a system called Dyno. It's by Mathilde Caron, Hugo Duvron, Ishan Misra, Hervé Gégou, Julien Mayral, Piotr Boyanowski and Armand Joulin of Facebook Air Research, Indria and Sorbonne University. You can see a bit more here in these pictures, where again, this is the self-attention. So the attention map from a vision transformer that was trained with Dyno and no supervision. And you can clearly see that in all the cases, the attention falls on what you would consider as a human, the relevant things in the image. Now, I have my hypotheses why this is the case, like completely without labels, and we'll see about that. But the representations that come out of the systems are really useful. For example, you can fine tune linear classifiers on top of these representations and that gives you really good image classifiers. They do that with ImageNet. You can use these for image retrieval because similar images are clustered together. You can use even do zero-shot classification simply by doing a k-nearest neighbor classifier in that feature space. And yeah, here you can also do some sort of proto image segmentation by looking at the attention maps. You don't even have to do something special to visualize this like you have to do in CNNs. The attention map directly gives you the the sort of segmentation map or or something pretty close to it. As an overview, this system Dyno is simply a they push the self-supervised learning and they specifically make the case that self-supervised and visual transformer. They go together really well and they, as I said, the Dyno is called self-distillation with no labels. So that is Dyno. And yeah, they they push various kind of metrics in in self-supervised systems or, you know, then linear classifier trained on top of them. For example, 80.1 percent top one on ImageNet in linear evaluation with the with a visual transformer base. And a quick overview over the system is right here. So two things they say are important next to all the other self-supervised systems. First of all, they do they have a kind of student teacher. That's the self-distillation part. The teacher is a momentum teacher and it does this centering and it also does sharpening in the softmax right here. And then there is no contrastive learning. There's no negative samples that the sharpening and the centering sort of take care of keeping the model from mode collapse or from collapsing. Also, there's no batch norm. So if those things don't don't mean anything to you, maybe you stay tuned. We'll we'll discuss them in a bit more detail as we go through the paper. If you like paper summaries like this and other content, for example, our cooking video, feel free to share this out and tell your friends about it. By the way, the cooking video did terribly. I don't know why. I guess I guess my YouTuber skills are just not not not on par. But yeah, I don't know. Yeah. If anyone has any ideas. All right, let's dive in. So vision transformers are a new thing, right? Vision transformers. I've also made a video about vision transformers. They are the easy, the simple application of the transformer architecture, which was prevalent in natural language processing with the introduction of attention is all you need and follow up papers, BERT, and so on. And applying this to images. And the concept is very simple. You have an image and you divide this into patches. So you divide the image into patches. And then you simply unroll that array sort of so you unroll that array so you have patch patch patch patch and so on. And then you simply consider this as a sequence, like a sentence like, Hello, my name is, and so on. You simply consider the sequence of patches as word embeddings. So there's like one I think there is one fully connected layer to actually get the word embedding or the token embedding. And then you put a transformer as you would in NLP. So there is a transformer here. And you do whatever you do with a transformer. So usually, if you don't know, people prepend a special token. That special token is usually called something where I'm going to draw this. That special token is usually called CLS token. And that is also passed through the transformer and the transformer in its base configuration. It sort of keeps it keeps the length of the sequence the same. It's actually not necessary to do this, but that's just how we do things. So for every input token, you'll get a corresponding output token or output embedding or output signal, whatever you want to call it. And such that none of the input tokens is, you know, kind of preferred because every input token sort of refers to some little patch here in the image. If you want to say something about the entire image, you don't want to prefer any one of them. So what you do is you have this special token, the CLS token, which is associated with no location in the image. And that's ultimately what you use to classify the image or also here to do representation learning. So the representation we're looking to get out is the final layer embedding of the CLS token. And that through the transformer architecture had aggregated all the information or we hope so from all the visual tokens in the image. So that's a visual transformer. Now, what do we do with it in this dino architecture? I've already shown you this picture. Let's go a little bit deeper into that. Self supervised learning naturally means you have no labels. And in this case, you don't even have a negative sample mechanism or a contrastive learning mechanism. So what you want to do is you want to train a model that sort of gives you gives you sensible representations. And that is easier said than done if you have no labels. Now, the when you do contrastive learning, the goal is that you have an image and you just take two patches from the image, let's say, and you have another image and you take a patch from that. And now you have what's called your anchor. This is your anchor. And then you have patch, patch A from the same patch B. Now you present the model, all the three patches, and you tell it which one is the anchor. And it needs to decide is the patch A or patch B from the same image. And you can see how this objective can give you a sort of representation because the model learns what kind of stuff is likely to be in the same image. This is not the case right here. We don't do contrastive learning. We don't have negative samples. We only we take one image and then we augment that image in different ways. Now, augmentations are a kind of a science by itself. I think they say they follow the paper BYOL in terms of augmentations. I've also made a video on that. Essentially, what you do is you do various random perturbations of the image. You might flip it. You might apply some color jitter. You might apply like some solarization, anything like this. Anything you can do to make the image different, but that you're relatively sure that, you know, it still looks like the same. Like you would still recognize it as the same image. So a part of these augmentations are also crops. What I've shown you here are crops of the same image. They do something special right here. When they have an image, they crop in two different ways. One they call global crops. And these are crops which generally cover more than 50 percent of the image. Whereas the other ones they called local crops. And these are crops that cover less than 50 percent of the image. This is going to be important in one while. These are global and these are local crops of the same image. Exactly. Keep that in mind. Now we have to understand what's up with this student and this teacher. So what we ideally want to do is we want to have two different augmentations of the same image. So here we have an image and you can see we make two different versions of that image. Now this could be two different crops and then we apply two different color jitters. We apply two different random rotations and so on. We just want two different versions of the same image. And our goal finally is going to be, here you can see the loss, is that the representation we get out of it is the same. So we teach the network that look these two things they might look different, but they are in fact the same. They are from their crops, differently augmented, differently cropped, but from the same image. So the easiest thing would be to just pass the two through the same network, but that does not work. So if you don't have negative samples, your main goal is to avoid what's called collapse. If the network just maps everything to the same representation, then it always wins. It always is like, well, you know, okay, the two things are the same because everything is the same. You don't want that. So a trick is to have two different models. One you call the student and one you call the teacher. And they're called student and teacher because from distillation. So in distillation, what you usually have is you have a data set and then you train a big model, which is the teacher. And now what you want to do is you want to make that model maybe smaller, right? Such that it runs on a mobile phone. And that's then the student. And there is a procedure where you take the data set and you take the teacher model. You sort of transfer the knowledge from the teacher model to the student model while using. You can use the data set to do so. And that usually works better than training the student model from scratch. It's very interesting why that even works. But this process is called distillation. So that's why it's called teacher and student. However, in this case, it's kind of a self distillation. So the teacher and the student, they're not big or small. They're the same architectures. In fact, we only train the student. OK, and the teacher is made from the student. So here is where the terms break down a bit like. So in the distillation sense, the teacher is the teacher in the distillation. But now it breaks down because the teacher is constructed from the student. So we have a teacher. We train the student to predict the same thing as the teacher does. Like learning from the teacher. But then at the same time, after we have done, after we've updated the student, we then have we then build the teacher from the new student. And the way we do this, you can see right here, is by exponentially moving average. So we keep the teacher model. And then as we update the student model, we simply update the teacher a little bit into the direction of the student model. And there is also a schedule associated with this exponentially moving average, like how much the exponential decay is and so on. This seems all to be loaded with hyperparameters. But again, the results are really cool. And it I guess it's yet going to turn out how sensitive to hyperparameters this whole setup is. They do make ablations, but we'll see how other people with other data sets fare. All right, so we have the teacher that is built from the student exponentially moving average. And we want to make the two predict the same represents or the same output for different augmentations of the same image. In fact, here you see it's even a bit more complicated. So this is the pseudo code. So we want to augment the image. We get two different versions of the image. We push both of these versions through the student and through the teacher. And then we want if you if you can if you can track if you can track that. But T1 is the X1 that went through the teacher. That needs to be the same as X2 that went through the student. And then the image X2 went through the teacher should be the same as X1 going through the student. So we want to augment the image differently two times. Then that gives us two different views of the same image. Then we want to run them through both through the teacher and student. And then we want sort of everything to be consistent with everything else. So we want the one augmentation in the one model to be consistent with another augmentation through another model. Now, there are two more things here. The first one is the centering, what's called centering. And that's what something the teacher does. And also something they say in the text is that in the teacher, they only use the global cropping, whereas in the student, they use both the global and the local cropping. So the student uses both and the teacher only uses the global crops. So essentially, if the student gets a local crop and the teacher gets a global crop, the goal here is that both things predict the same representation. And that means the student has somehow learned that, you know, whatever I see here is a little piece of whatever the teacher has, even though it doesn't, I should reformulate this because it doesn't see what the teacher has. So the student somehow has to from a very small sub patch, it has to know it has to output something that it would that itself or the teacher, which is itself averaged, would also output if it sees more context in the image. So you train the network to for all of these crops and for all the different augmentations, output the same thing without knowing what the other thing is. And I think that is the advantage to contrastive representations, honestly, because in contrastive representation, in contrastive learning, you sort of contrast with the negative with the negative samples. And here it's really like you don't know anything and you need to output something. And that needs to match whatever whatever you yourself would output if you saw a different part of the image. So you have no choice but to output, you know, either the same thing all the time, which is prevented here, or to output something that's on the image. And you can't just output something that's only in your patch, right? Otherwise, another patch wouldn't show the same thing. Like if you if there's like there's like a little tiny structure here, you would not output that because the other patches don't have it. However, if there is something big in the image, right, like, you know, our traditional cat right here. And you recognize that because you see a little cat ear. If you output a representation for cat and, you know, since you would also do this for the other ear and for the paws and so on, you this whiskers, you then would you then win like your loss is small. So you're intrinsically pushed towards outputting something that describes the image as a whole. Right. And that differentiates it from other images. So what what encourages you to be different? That's this centering. And also in the softmax, there is a there is a sharpening. So first of all, the centering is simply what you do in the teacher. You keep a running average here. Again, you can see that you can keep a running average of all the representations that the teacher sees. But you just you keep you keep that as a list or a running list, all the representations that the teacher sees running average. And you simply subtract that from the logits down here. That's that's centering. It's something like a normalization, but not really. What it does is it it keeps the keeps the logits sort of close in a in a range that's manageable. And and has some variance and so on. And, you know, within as a proxy, it also does that to the student because the student is trained to be like the teacher. So centering is a bit like a normalization here. And then the second thing is that there is a different parameter in the softmax as a temperature parameter. So the softmax function is at the end. And that has a temperature. Where is it? Where are you? This is the softmax function. You can see it has a temperature parameter. Right. And that temperature is much lower for the teacher than for the student. And they call this sharpening. Now, why is there even a softmax? That's what I asked myself. Like, if you think of a of what you do with a representation, usually when you do something like a contrastive loss, you may just do a contrastive loss or a self supervised loss on the representation itself. Like you do cross product or not cross product, inner product, or you do L2 distance between the representations or something. Here we do cross entropy and the cross entropy after a softmax. And the way I interpret this is the following. A softmax is like what you get out is a normalized distribution. Right. However, we have no class labels here. So what you do is you simply choose. You choose a number, any number. Right. This is you as an implementer of this algorithm, choose what dimension you want to output here. Now, after the softmax, whatever you input is going to be a distribution over the amount of things that you have input. So and you can interpret this as classes. Right. There's class zero, one, two, three, and so on. And you're going to get class zero is probability 10 percent, class one, zero percent, class two, 40 percent, and so on. You don't know what it means, but you know, you you get this as an output and the teacher having this sharpening, it will have a much more peaked distribution. So for the same thing, it might have a distribution that's not as much class zero, not as much class one, very much class two. All right. This even goes off screen for you. Yeah. Very much class two and so on. And since this is the since the teacher is the target for the student, you see here is a stop gradient. The student is sort of this is a common, I guess, I guess this is a common trick in distillation. Like the teacher is very sure. And that means the student gets a better learning signal to match the teacher. So this this sharpening of the teacher gives is less noisy for the student. And also, I think it also helps prevent this. I'm not sure. So they speak of sharpening and centering and one, I think one they claim furthers collapse, probably the sharpening and one prevents it, which might be the centering. I might mix them up. But, you know, one sort of reduces the noise but encourages. I think the sharpening must reduce noise, but encourage collapse. And then the centering counteracts that, counteracts the collapse. Yeah, probably. Though there is an argument to be made that the sharpening might also counter collapse because, oh, yes, that's what they say. Now, I remember. So they say the sharp. So they they say naturally this would then be biased towards the uniform distribution with the centering, I believe. But the sharpening then counteracts that again. It's in the text somewhere. I'm more interested in why this is even a softmax in the first place. So I interpret this as you force the model to come up with an with an K dimensional classification problem by itself. And it has to choose by itself what the classes are. Right. So it has to somehow make representations that allow itself to come up with a classification problem that it can solve. And I think that's that's pretty smart. You know, you instead of giving it a classification problem, you simply ask it to come up with one. Now, this could go horribly wrong. Right. But apparently, if you do it like this, it goes well. So that's the Dino architecture. Again, we augment image, we augment it in different ways. We pull we put all the things through the student and through the teacher. The teacher is an exponential moving average of the student. That gives us different representations of different augmentations of the same image. We require the representations to be the same in terms of their. So we take the representations, we ship them through a classifier, through a softmax into a distribution. We require the outputs to be the same of the student and the teacher. While the teacher has centering, which is centering the logits by an exponential running average of all the representations it has ever seen. And also it has a sharper softmax. All of this together. And yeah, the teacher has a stop gradient. So it's we train the student of this together, gives us a system that comes up with good representations and does not collapse. Now, what does this buy us? It buys us what I've essentially shown you at the beginning. And also it buys us k nearest neighbor classification, which are zero shot classifiers. Like right now I can I can pump this through the system, pump a data set through the system. I can come with a new image and I can simply do k nearest neighbor. I don't even have to train the network anymore. I can come with a new data set. I can do image retrieval. I can do linear classification on top of the representation. And all of this works much better than previous systems, no matter the architecture. But it seems to work especially well with the visual transformers down here. If you see this, for example, compared to the to the best Resnets. So there is this five percent difference in linear evaluation, which, you know, this is 25 percent error. This is 20 percent error on ImageNet. And there is even a bigger difference when you look at k nearest neighbor classification, which is the rightmost column. They do a lot of experiments, as I said, in image retrieval. In copy detection, which is really interesting. That's, I think, where you where you want to realize if if someone has taken an image and made another image out of it. You know, I don't know if that's a good if that's such a good thing, given that the entire meme culture relies on it. If you look at this CLS token, right, the CLS token is ultimately where the representation that you take comes out. If you look at the attention heads of that and you visualize the attention maps, it gives you this this not only this segmentation map, but like, yeah, like not only does it tell you where to look, but it even seems to be sort of segmenting the individual objects here in the horse. You can you can see the straps of the horse. You can see. Sorry, this is a zebra. Yeah, you can see there in the trucks, you can see the roads is or the wheels are separate from the truck and so on. They do ablations. They compare it with sort of supervised baselines. You can see this works much better. And what I think is pretty cool is down here in the appendix somewhere. Yeah, they have more of these attention maps compared to supervised attention maps. And this, I mean, the comparison is very, very strong. Yeah. Because, yeah, so compared to supervised what I think is happening that if you give the these things a supervised problem, they, you can see they do pay attention, for example, here they pay attention to whatever the cat's face or something and the ears. You can see that the cat shape. However, there is this thing like there is the shortcut learning, which is, I think, a data set problem. But also, supervised system just stops kind of learning once it has mastered the task or it might it might try out various optimizations for the task that you give it. Right. And these optimizations, I think, are what, you know, pop up all over the place with these little specks of attention that it also does. You know, these, it might not make sense in this particular image, but, you know, the same attention pattern or the same thing to pay attention to might make a lot of sense in like three other images in the data set. So that's why that's there. Whereas if you do this unsupervised, there is no there is no hyper optimization on a single task. There is no real like there is only there's like especially if you have also more images, which you can do in unsupervised. Right. You can also can't hyper optimize for individual samples and so on. So that's one thing. And here is this complete map of ImageNet, I think. And maybe you can't read it, but like here's Tractor and right next to it is like Harvester and Trasher. There's Minibus down here. So all of these like the vehicles are clustered together. There is kind of butcher shop and grocery store right next to each other. This, you know, it appears to be really, really good representations. Now, the question is why? Right. That's that's the question. So this this was the paper I encourage you to go read the experiment section and so on. It's it's very cool. Cool ablations. They show why exactly they use this loss and what happens without the momentum of the teacher and so on. But what interests me is why does this give you such extraordinary representations in unsupervised fashion? And I am sort of I have two hypothesis or two things that I think contribute mostly to this. So if we look at the question of why, right, the first thing I think is the augmentations, the augmentations. Yeah, the augmentations have played a large role, not as much in an LP and LP. We do it a little bit differently, but augmentations in computer vision and self-supervised learning have a central role. And it's really important that you have the correct ones, which is a thing they also say right here. Right. They really stress that this multi crop augmentation is quite important. So augmentations seem to be central. And to me, augmentations are a bit like that's where you put the that's where you put the human prior. That's where you tell the model what it should pay attention to and what it shouldn't pay attention to. Right. Because all the things you destroy with an augmentation, like you make the color brighter, that's you tell the model color doesn't matter. Right. Or brightness variations don't matter. So by augmenting, you tell the model what it should and shouldn't or what it shouldn't pay attention to, essentially. So all the things that it's the same if you have an if you have a data set of dogs and cats. Right. And, you know, you tell it, you know, this is a dog, this is a dog, this is a dog. Essentially, you tell it you shouldn't pay attention to, you know, what is different in these images. You should only pay attention to what is the same. And the augmentations, that's kind of where the knowledge goes in. So if we want to go towards fully, let's say fully autonomous self supervised learning, that's what we need to get rid of. We need to get rid of the augmentations or we need to get rid of us designing augmentations for the domain. If we want this to be, you know, domain agnostic and also if we want better image representations, because the probability that we as humans exactly capture the correct augmentations is zero. Right. We seem to capture pretty good ones. But, you know, the probability we have the best ones is like zero. OK. The second thing, and this is a thing that's, I think, more hidden is the data set. And what I mean is how the data set is constructed. So these things are often trained on something like ImageNet data set. And you can see in these pictures, there always seems to be like an object of interest in these in these pictures. Right. Even if you train this from pictures in the wild, like you scrape pictures from Instagram or whatever, the way people don't take pictures of random things. People, if you're, you know, it would be pretty weird to have a picture and, you know, there's just like dirt road. Like it's just like dirt road. And here's like, you know, a bit of grass. And you post this on social media and you're like, whoa, look at this. So by how you construct the data set, even if you scrape it from the Internet, by how humanity takes pictures, you are implicitly telling the model what's important. So the model learns how to say this, how you make the data set speaks a lot about where your attention goes. And that's what you feed the model. Right. So these things, these self supervised methods in this way, they rely a lot on data set construction. So we shouldn't expect this to transfer to domains where we get like random IID data from the world because these things aren't IID. We tell the model pretty clearly by the data we give it. What's important? What isn't? So that is a little bit of my opinion. And I think that's correct. Right. I think the model, if we have self supervised learning, the information should be taken from the data set. Right. So that the model should look at the data and say, you know, what seems to be given how this data set is, what seemed to be the important things in there? I am more a fan of getting rid of the augmentations. So that's my opinion. If you want more experiments, it's you know, it's also faster and has less parameters and and so on. But again, Dino is a method of self supervised learning where and they their argument is that it combines naturally well with the vision transformer. Right. That was it for me. Check out paper, check out blog, subscribe, share and bye bye.
 Hello there, I hope you have all seen this. This is a new system by Facebook AI and what you're seeing here is a visualization of the attention maps of that neural network.
 In the middle is a supervised baseline and on the right is this new system called Dino.
 It's not as much a system as it is a methodology for unsupervised pre-training of visual transformers.
 And you can see that this system has neither been trained to learn what a dog is nor has it been trained to do any sort of segmentation.
 Yet if you look at the attention maps, it clearly can track objects, it knows what to pay attention to in the images, and it can do much more than that.
 So here you can see that it can sort of track objects behind occlusion. So the ship goes behind the waves, the horse goes behind the grass.
 And you can see in the attention map that this is well reflected. You can do more than that though, even.
 So if you use this feature representation that this model gives you for ImageNet, then as the model gets trained and you represent ImageNet and its feature space,
 it will cluster the images of the same class, it will cluster them together, which is already pretty cool because it has no labels at training time.
 But also it will cluster similar classes with each other, which speaks to the fact that this might be the next step in unsupervised representation learning for images.
 And specifically, it appears that the features that come out of a network that is trained with Dyno are extremely valuable for the kinds of things we are interested in when working with natural images.
 So this is image retrieval and classification. So this system, let's just switch over to the paper right here.
 The paper is called Emerging Properties in Self-Supervised Vision Transformers. It presents a system called Dyno.
 It's by Mathilde Caron, Hugo Duvron, Ishan Misra, Hervé Gégou, Julien Mayral, Piotr Boyanowski and Armand Joulin of Facebook Air Research, Indria and Sorbonne University.
 You can see a bit more here in these pictures, where again, this is the self-attention.
 So the attention map from a vision transformer that was trained with Dyno and no supervision.
 And you can clearly see that in all the cases, the attention falls on what you would consider as a human, the relevant things in the image.
 Now, I have my hypotheses why this is the case, like completely without labels, and we'll see about that.
 But the representations that come out of the systems are really useful. For example, you can fine tune linear classifiers on top of these representations and that gives you really good image classifiers.
 They do that with ImageNet. You can use these for image retrieval because similar images are clustered together.
 You can use even do zero-shot classification simply by doing a k-nearest neighbor classifier in that feature space.
 And yeah, here you can also do some sort of proto image segmentation by looking at the attention maps.
 You don't even have to do something special to visualize this like you have to do in CNNs.
 The attention map directly gives you the the sort of segmentation map or or something pretty close to it.
 As an overview, this system Dyno is simply a they push the self-supervised learning and they specifically make the case that self-supervised and visual transformer.
 They go together really well and they, as I said, the Dyno is called self-distillation with no labels.
 So that is Dyno. And yeah, they they push various kind of metrics in in self-supervised systems or, you know, then linear classifier trained on top of them.
 For example, 80.1 percent top one on ImageNet in linear evaluation with the with a visual transformer base.
 And a quick overview over the system is right here.
 So two things they say are important next to all the other self-supervised systems.
 First of all, they do they have a kind of student teacher.
 That's the self-distillation part.
 The teacher is a momentum teacher and it does this centering and it also does sharpening in the softmax right here.
 And then there is no contrastive learning.
 There's no negative samples that the sharpening and the centering sort of take care of keeping the model from mode collapse or from collapsing.
 Also, there's no batch norm.
 So if those things don't don't mean anything to you, maybe you stay tuned.
 We'll we'll discuss them in a bit more detail as we go through the paper.
 If you like paper summaries like this and other content, for example, our cooking video, feel free to share this out and tell your friends about it.
 By the way, the cooking video did terribly.
 I don't know why. I guess I guess my YouTuber skills are just not not not on par.
 But yeah, I don't know. Yeah.
 If anyone has any ideas. All right, let's dive in.
 So vision transformers are a new thing, right?
 Vision transformers. I've also made a video about vision transformers.
 They are the easy, the simple application of the transformer architecture, which was prevalent in natural language processing with the introduction of attention is all you need and follow up papers, BERT, and so on.
 And applying this to images.
 And the concept is very simple.
 You have an image and you divide this into patches.
 So you divide the image into patches.
 And then you simply unroll that array sort of so you unroll that array so you have patch patch patch patch and so on.
 And then you simply consider this as a sequence, like a sentence like, Hello, my name is, and so on.
 You simply consider the sequence of patches as word embeddings.
 So there's like one I think there is one fully connected layer to actually get the word embedding or the token embedding.
 And then you put a transformer as you would in NLP.
 So there is a transformer here.
 And you do whatever you do with a transformer.
 So usually, if you don't know, people prepend a special token.
 That special token is usually called something where I'm going to draw this.
 That special token is usually called CLS token.
 And that is also passed through the transformer and the transformer in its base configuration.
 It sort of keeps it keeps the length of the sequence the same.
 It's actually not necessary to do this, but that's just how we do things.
 So for every input token, you'll get a corresponding output token or output embedding or output signal, whatever you want to call it.
 And such that none of the input tokens is, you know, kind of preferred because every input token sort of refers to some little patch here in the image.
 If you want to say something about the entire image, you don't want to prefer any one of them.
 So what you do is you have this special token, the CLS token, which is associated with no location in the image.
 And that's ultimately what you use to classify the image or also here to do representation learning.
 So the representation we're looking to get out is the final layer embedding of the CLS token.
 And that through the transformer architecture had aggregated all the information or we hope so from all the visual tokens in the image.
 So that's a visual transformer. Now, what do we do with it in this dino architecture?
 I've already shown you this picture. Let's go a little bit deeper into that.
 Self supervised learning naturally means you have no labels. And in this case, you don't even have a negative sample mechanism or a contrastive learning mechanism.
 So what you want to do is you want to train a model that sort of gives you gives you sensible representations.
 And that is easier said than done if you have no labels.
 Now, the when you do contrastive learning, the goal is that you have an image and you just take two patches from the image, let's say,
 and you have another image and you take a patch from that.
 And now you have what's called your anchor. This is your anchor. And then you have patch, patch A from the same patch B.
 Now you present the model, all the three patches, and you tell it which one is the anchor.
 And it needs to decide is the patch A or patch B from the same image.
 And you can see how this objective can give you a sort of representation because the model learns what kind of stuff is likely to be in the same image.
 This is not the case right here. We don't do contrastive learning. We don't have negative samples.
 We only we take one image and then we augment that image in different ways.
 Now, augmentations are a kind of a science by itself. I think they say they follow the paper BYOL in terms of augmentations.
 I've also made a video on that. Essentially, what you do is you do various random perturbations of the image.
 You might flip it. You might apply some color jitter. You might apply like some solarization, anything like this.
 Anything you can do to make the image different, but that you're relatively sure that, you know, it still looks like the same.
 Like you would still recognize it as the same image. So a part of these augmentations are also crops.
 What I've shown you here are crops of the same image. They do something special right here.
 When they have an image, they crop in two different ways. One they call global crops.
 And these are crops which generally cover more than 50 percent of the image.
 Whereas the other ones they called local crops. And these are crops that cover less than 50 percent of the image.
 This is going to be important in one while. These are global and these are local crops of the same image.
 Exactly. Keep that in mind. Now we have to understand what's up with this student and this teacher.
 So what we ideally want to do is we want to have two different augmentations of the same image.
 So here we have an image and you can see we make two different versions of that image.
 Now this could be two different crops and then we apply two different color jitters.
 We apply two different random rotations and so on. We just want two different versions of the same image.
 And our goal finally is going to be, here you can see the loss, is that the representation we get out of it is the same.
 So we teach the network that look these two things they might look different, but they are in fact the same.
 They are from their crops, differently augmented, differently cropped, but from the same image.
 So the easiest thing would be to just pass the two through the same network, but that does not work.
 So if you don't have negative samples, your main goal is to avoid what's called collapse.
 If the network just maps everything to the same representation, then it always wins.
 It always is like, well, you know, okay, the two things are the same because everything is the same.
 You don't want that. So a trick is to have two different models.
 One you call the student and one you call the teacher.
 And they're called student and teacher because from distillation.
 So in distillation, what you usually have is you have a data set and then you train a big model, which is the teacher.
 And now what you want to do is you want to make that model maybe smaller, right?
 Such that it runs on a mobile phone. And that's then the student.
 And there is a procedure where you take the data set and you take the teacher model.
 You sort of transfer the knowledge from the teacher model to the student model while using.
 You can use the data set to do so.
 And that usually works better than training the student model from scratch.
 It's very interesting why that even works. But this process is called distillation.
 So that's why it's called teacher and student.
 However, in this case, it's kind of a self distillation.
 So the teacher and the student, they're not big or small.
 They're the same architectures. In fact, we only train the student.
 OK, and the teacher is made from the student.
 So here is where the terms break down a bit like.
 So in the distillation sense, the teacher is the teacher in the distillation.
 But now it breaks down because the teacher is constructed from the student.
 So we have a teacher. We train the student to predict the same thing as the teacher does.
 Like learning from the teacher.
 But then at the same time, after we have done, after we've updated the student,
 we then have we then build the teacher from the new student.
 And the way we do this, you can see right here, is by exponentially moving average.
 So we keep the teacher model.
 And then as we update the student model, we simply update the teacher a little bit
 into the direction of the student model.
 And there is also a schedule associated with this exponentially moving average,
 like how much the exponential decay is and so on.
 This seems all to be loaded with hyperparameters.
 But again, the results are really cool.
 And it I guess it's yet going to turn out how sensitive to hyperparameters this whole setup is.
 They do make ablations, but we'll see how other people with other data sets fare.
 All right, so we have the teacher that is built from the student exponentially moving average.
 And we want to make the two predict the same represents or the same output
 for different augmentations of the same image.
 In fact, here you see it's even a bit more complicated.
 So this is the pseudo code.
 So we want to augment the image.
 We get two different versions of the image.
 We push both of these versions through the student and through the teacher.
 And then we want if you if you can if you can track if you can track that.
 But T1 is the X1 that went through the teacher.
 That needs to be the same as X2 that went through the student.
 And then the image X2 went through the teacher should be the same as X1 going through the student.
 So we want to augment the image differently two times.
 Then that gives us two different views of the same image.
 Then we want to run them through both through the teacher and student.
 And then we want sort of everything to be consistent with everything else.
 So we want the one augmentation in the one model to be consistent with another augmentation through another model.
 Now, there are two more things here.
 The first one is the centering, what's called centering.
 And that's what something the teacher does.
 And also something they say in the text is that in the teacher, they only use the global cropping,
 whereas in the student, they use both the global and the local cropping.
 So the student uses both and the teacher only uses the global crops.
 So essentially, if the student gets a local crop and the teacher gets a global crop,
 the goal here is that both things predict the same representation.
 And that means the student has somehow learned that, you know,
 whatever I see here is a little piece of whatever the teacher has,
 even though it doesn't, I should reformulate this because it doesn't see what the teacher has.
 So the student somehow has to from a very small sub patch,
 it has to know it has to output something that it would that itself or the teacher,
 which is itself averaged, would also output if it sees more context in the image.
 So you train the network to for all of these crops and for all the different augmentations,
 output the same thing without knowing what the other thing is.
 And I think that is the advantage to contrastive representations, honestly,
 because in contrastive representation, in contrastive learning,
 you sort of contrast with the negative with the negative samples.
 And here it's really like you don't know anything and you need to output something.
 And that needs to match whatever whatever you yourself would output if you saw a different part of the image.
 So you have no choice but to output, you know, either the same thing all the time,
 which is prevented here, or to output something that's on the image.
 And you can't just output something that's only in your patch, right?
 Otherwise, another patch wouldn't show the same thing.
 Like if you if there's like there's like a little tiny structure here,
 you would not output that because the other patches don't have it.
 However, if there is something big in the image, right, like, you know, our traditional cat right here.
 And you recognize that because you see a little cat ear.
 If you output a representation for cat and, you know,
 since you would also do this for the other ear and for the paws and so on, you this whiskers,
 you then would you then win like your loss is small.
 So you're intrinsically pushed towards outputting something that describes the image as a whole.
 Right. And that differentiates it from other images.
 So what what encourages you to be different?
 That's this centering. And also in the softmax, there is a there is a sharpening.
 So first of all, the centering is simply what you do in the teacher.
 You keep a running average here.
 Again, you can see that you can keep a running average of all the representations that the teacher sees.
 But you just you keep you keep that as a list or a running list,
 all the representations that the teacher sees running average.
 And you simply subtract that from the logits down here.
 That's that's centering. It's something like a normalization, but not really.
 What it does is it it keeps the keeps the logits sort of close in a in a range that's manageable.
 And and has some variance and so on.
 And, you know, within as a proxy, it also does that to the student because the student is trained to be like the teacher.
 So centering is a bit like a normalization here.
 And then the second thing is that there is a different parameter in the softmax as a temperature parameter.
 So the softmax function is at the end.
 And that has a temperature. Where is it? Where are you?
 This is the softmax function. You can see it has a temperature parameter.
 Right. And that temperature is much lower for the teacher than for the student.
 And they call this sharpening. Now, why is there even a softmax?
 That's what I asked myself. Like, if you think of a of what you do with a representation,
 usually when you do something like a contrastive loss, you may just do a contrastive loss or a self supervised loss on the representation itself.
 Like you do cross product or not cross product, inner product, or you do L2 distance between the representations or something.
 Here we do cross entropy and the cross entropy after a softmax.
 And the way I interpret this is the following.
 A softmax is like what you get out is a normalized distribution. Right.
 However, we have no class labels here. So what you do is you simply choose.
 You choose a number, any number. Right.
 This is you as an implementer of this algorithm, choose what dimension you want to output here.
 Now, after the softmax, whatever you input is going to be a distribution over the amount of things that you have input.
 So and you can interpret this as classes. Right.
 There's class zero, one, two, three, and so on.
 And you're going to get class zero is probability 10 percent, class one, zero percent, class two, 40 percent, and so on.
 You don't know what it means, but you know, you you get this as an output and the teacher having this sharpening,
 it will have a much more peaked distribution.
 So for the same thing, it might have a distribution that's not as much class zero, not as much class one, very much class two.
 All right. This even goes off screen for you. Yeah.
 Very much class two and so on.
 And since this is the since the teacher is the target for the student, you see here is a stop gradient.
 The student is sort of this is a common, I guess, I guess this is a common trick in distillation.
 Like the teacher is very sure.
 And that means the student gets a better learning signal to match the teacher.
 So this this sharpening of the teacher gives is less noisy for the student.
 And also, I think it also helps prevent this. I'm not sure.
 So they speak of sharpening and centering and one, I think one they claim furthers collapse, probably the sharpening and one prevents it,
 which might be the centering. I might mix them up.
 But, you know, one sort of reduces the noise but encourages.
 I think the sharpening must reduce noise, but encourage collapse.
 And then the centering counteracts that, counteracts the collapse.
 Yeah, probably.
 Though there is an argument to be made that the sharpening might also counter collapse because, oh, yes, that's what they say.
 Now, I remember. So they say the sharp.
 So they they say naturally this would then be biased towards the uniform distribution with the centering, I believe.
 But the sharpening then counteracts that again. It's in the text somewhere.
 I'm more interested in why this is even a softmax in the first place.
 So I interpret this as you force the model to come up with an with an K dimensional classification problem by itself.
 And it has to choose by itself what the classes are. Right.
 So it has to somehow make representations that allow itself to come up with a classification problem that it can solve.
 And I think that's that's pretty smart.
 You know, you instead of giving it a classification problem, you simply ask it to come up with one.
 Now, this could go horribly wrong. Right.
 But apparently, if you do it like this, it goes well.
 So that's the Dino architecture.
 Again, we augment image, we augment it in different ways.
 We pull we put all the things through the student and through the teacher.
 The teacher is an exponential moving average of the student.
 That gives us different representations of different augmentations of the same image.
 We require the representations to be the same in terms of their.
 So we take the representations, we ship them through a classifier, through a softmax into a distribution.
 We require the outputs to be the same of the student and the teacher.
 While the teacher has centering, which is centering the logits by an exponential running average of all the representations it has ever seen.
 And also it has a sharper softmax.
 All of this together.
 And yeah, the teacher has a stop gradient.
 So it's we train the student of this together, gives us a system that comes up with good representations and does not collapse.
 Now, what does this buy us?
 It buys us what I've essentially shown you at the beginning.
 And also it buys us k nearest neighbor classification, which are zero shot classifiers.
 Like right now I can I can pump this through the system, pump a data set through the system.
 I can come with a new image and I can simply do k nearest neighbor.
 I don't even have to train the network anymore.
 I can come with a new data set.
 I can do image retrieval.
 I can do linear classification on top of the representation.
 And all of this works much better than previous systems, no matter the architecture.
 But it seems to work especially well with the visual transformers down here.
 If you see this, for example, compared to the to the best Resnets.
 So there is this five percent difference in linear evaluation, which, you know, this is 25 percent error.
 This is 20 percent error on ImageNet.
 And there is even a bigger difference when you look at k nearest neighbor classification, which is the rightmost column.
 They do a lot of experiments, as I said, in image retrieval.
 In copy detection, which is really interesting.
 That's, I think, where you where you want to realize if if someone has taken an image and made another image out of it.
 You know, I don't know if that's a good if that's such a good thing, given that the entire meme culture relies on it.
 If you look at this CLS token, right, the CLS token is ultimately where the representation that you take comes out.
 If you look at the attention heads of that and you visualize the attention maps, it gives you this this not only this segmentation map, but like, yeah, like not only does it tell you where to look, but it even seems to be
 sort of segmenting the individual objects here in the horse.
 You can you can see the straps of the horse.
 You can see. Sorry, this is a zebra.
 Yeah, you can see there in the trucks, you can see the roads is or the wheels are separate from the truck and so on.
 They do ablations. They compare it with sort of supervised baselines.
 You can see this works much better.
 And what I think is pretty cool is down here in the appendix somewhere.
 Yeah, they have more of these attention maps compared to supervised attention maps.
 And this, I mean, the comparison is very, very strong.
 Yeah.
 Because, yeah, so compared to supervised what I think is happening that if you give the these things a supervised problem, they, you can see they do pay attention, for example, here they pay attention to whatever the cat's face or something and the ears.
 You can see that the cat shape. However, there is this thing like there is the shortcut learning, which is, I think, a data set problem.
 But also, supervised system just stops kind of learning once it has mastered the task or it might it might try out various optimizations for the task that you give it.
 Right. And these optimizations, I think, are what, you know, pop up all over the place with these little specks of attention that it also does.
 You know, these, it might not make sense in this particular image, but, you know, the same attention pattern or the same thing to pay attention to might make a lot of sense in like three other images in the data set.
 So that's why that's there.
 Whereas if you do this unsupervised, there is no there is no hyper optimization on a single task.
 There is no real like there is only there's like especially if you have also more images, which you can do in unsupervised.
 Right. You can also can't hyper optimize for individual samples and so on.
 So that's one thing. And here is this complete map of ImageNet, I think.
 And maybe you can't read it, but like here's Tractor and right next to it is like Harvester and Trasher.
 There's Minibus down here. So all of these like the vehicles are clustered together.
 There is kind of butcher shop and grocery store right next to each other.
 This, you know, it appears to be really, really good representations.
 Now, the question is why? Right. That's that's the question.
 So this this was the paper I encourage you to go read the experiment section and so on.
 It's it's very cool. Cool ablations.
 They show why exactly they use this loss and what happens without the momentum of the teacher and so on.
 But what interests me is why does this give you such extraordinary representations in unsupervised fashion?
 And I am sort of I have two hypothesis or two things that I think contribute mostly to this.
 So if we look at the question of why, right, the first thing I think is the augmentations, the augmentations.
 Yeah, the augmentations have played a large role, not as much in an LP and LP.
 We do it a little bit differently, but augmentations in computer vision and self-supervised learning have a central role.
 And it's really important that you have the correct ones, which is a thing they also say right here.
 Right. They really stress that this multi crop augmentation is quite important.
 So augmentations seem to be central.
 And to me, augmentations are a bit like that's where you put the that's where you put the human prior.
 That's where you tell the model what it should pay attention to and what it shouldn't pay attention to.
 Right. Because all the things you destroy with an augmentation, like you make the color brighter, that's you tell the model color doesn't matter.
 Right. Or brightness variations don't matter.
 So by augmenting, you tell the model what it should and shouldn't or what it shouldn't pay attention to, essentially.
 So all the things that it's the same if you have an if you have a data set of dogs and cats.
 Right. And, you know, you tell it, you know, this is a dog, this is a dog, this is a dog.
 Essentially, you tell it you shouldn't pay attention to, you know, what is different in these images.
 You should only pay attention to what is the same.
 And the augmentations, that's kind of where the knowledge goes in.
 So if we want to go towards fully, let's say fully autonomous self supervised learning, that's what we need to get rid of.
 We need to get rid of the augmentations or we need to get rid of us designing augmentations for the domain.
 If we want this to be, you know, domain agnostic and also if we want better image representations,
 because the probability that we as humans exactly capture the correct augmentations is zero.
 Right. We seem to capture pretty good ones.
 But, you know, the probability we have the best ones is like zero.
 OK. The second thing, and this is a thing that's, I think, more hidden is the data set.
 And what I mean is how the data set is constructed.
 So these things are often trained on something like ImageNet data set.
 And you can see in these pictures, there always seems to be like an object of interest in these in these pictures.
 Right. Even if you train this from pictures in the wild, like you scrape pictures from Instagram or whatever,
 the way people don't take pictures of random things.
 People, if you're, you know, it would be pretty weird to have a picture and, you know, there's just like dirt road.
 Like it's just like dirt road. And here's like, you know, a bit of grass.
 And you post this on social media and you're like, whoa, look at this.
 So by how you construct the data set, even if you scrape it from the Internet, by how humanity takes pictures,
 you are implicitly telling the model what's important.
 So the model learns how to say this, how you make the data set speaks a lot about where your attention goes.
 And that's what you feed the model. Right.
 So these things, these self supervised methods in this way, they rely a lot on data set construction.
 So we shouldn't expect this to transfer to domains where we get like random IID data from the world because these things aren't IID.
 We tell the model pretty clearly by the data we give it. What's important? What isn't?
 So that is a little bit of my opinion. And I think that's correct. Right.
 I think the model, if we have self supervised learning, the information should be taken from the data set. Right.
 So that the model should look at the data and say, you know, what seems to be given how this data set is,
 what seemed to be the important things in there? I am more a fan of getting rid of the augmentations.
 So that's my opinion. If you want more experiments, it's you know, it's also faster and has less parameters and and so on.
 But again, Dino is a method of self supervised learning where and they their argument is that it combines naturally well with the vision transformer.
 Right. That was it for me. Check out paper, check out blog, subscribe, share and bye bye.