BYOL: Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deepmind ucl representation moco momentum contrast simclr encoder augmentation mixup randaugment crop random crop jitter flip unsupervised self-supervised cnn resnet latent contrastive online target exponential moving average negatives
Self-supervised representation learning relies on negative samples to keep the encoder from collapsing to trivial solutions. However, this paper shows that negative samples, which are a nuisance to implement, are not necessary for learning good representation, and their algorithm BYOL is able to outperform other baselines using just positive samples.

OUTLINE:
0:00 - Intro & Overview
1:10 - Image Representation Learning
3:55 - Self-Supervised Learning
5:35 - Negative Samples
10:50 - BYOL
23:20 - Experiments
30:10 - Conclusion & Broader Impact

Paper: https://arxiv.org/abs/2006.07733

Abstract:
We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods intrinsically rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches 74.3% top-1 classification accuracy on ImageNet using the standard linear evaluation protocol with a ResNet-50 architecture and 79.6% with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks.

Authors: Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hello there! Today we're looking at Bootstrap Your Own Latent, a new approach to self-supervised learning by researchers of DeepMind and Imperial College. So almost no day goes by where we don't hear some sort of new self-supervised algorithm right here. This paper on a high level tries to get rid of the necessary negative samples when doing the contrastive loss for self-supervised learning. They basically combine momentum contrast and same clear and then remove the negative samples. That seems to work pretty well even though it's magic. So yeah, if you want to see how it's done, stick around, share the video out. If you want other people to see how it's done, leave a comment. This one I really don't get what's going on. So if you have ideas, put them there. I'll read them through. It'll be fun. Alright, so they say we introduce Bootstrap Your Own Latent or B.O.L. a new approach to self-supervised image representation learning. Image representation learning is the simple task of taking an image and then feeding it through a function which is usually like a neural network. Let's just say this is a neural network and in fact all of these, the community has sort of standardized this to be, most of the time it's something like a ResNet-50. So what you want to do is you want to train a neural network like a ResNet-50 to give you a good representation of the image. So this would be like H and H is a vector and H is a representation of this image and the representation should be such that you can then take this representation and solve many tasks with it, which either can be like linear, you can put a linear classifier on top of the H or you can fine-tune the entire architecture to solve some other tasks. The idea is if you have a large data set here you may use this data set to train these good representations of these images and then you can transfer this to a task where you might not have as much data. Because you don't have as much data it's not enough to completely train an architecture like this, but it is enough to take an architecture that's been trained with the large data set and just adapt it to your small data set. That usually tends to work pretty well. This is called transfer learning. This step here is called fine-tuning sometimes and it's sort of the approach that comes from natural language processing from these big transformers like BERT where you first train on a really big data set that might not be the data set that you want in the end but it's really big so you can sort of learn a lot of things from that data set and then the only thing left to do is to fine-tune it to basically adapt it to the nuances of your data set but it will have learned most things already and that's called representation learning. The goal is to learn a good representation. The self-supervised here is also important because representation learning can be as easy as if this here is ImageNet. The ImageNet data set contains like a million images all with labels. You can simply train your ResNet-50 to predict the class. This is called supervised pre-training or supervised representation learning and that works pretty well but you need a labeled data set. In self-supervised learning you do not need labels. What you do is you do self-supervision and self-supervision there are many ways to do self-supervision but what we'll see in this particular paper is that you will take an image and you'll make different variants of that same image. You'll take the image and you'll make many many variants of it. Let's just say two. You have some procedure to sort of change the picture a little bit but it's essentially still the same and you do that through data augmentation. This could be a random crop or you color jitter or you rotate it or something like this and then you exploit the fact that you know that these two things they should be still sort of the same image. Once you send them through your encoder the representations of the two images they should be fairly close. Now let's actually read on right here. Bjoll relies on two neural networks referred to as online and target networks that interact and learn from each other. From an augmented view of an image we train the online network to predict the target representation of the same image under a different augmented view. That's sort of what we saw. We have the same image under a different augmented view. What does it mean? You make two versions of the same image. One that are slightly different and then their representation should be close. Until this point we have always thought that this would degenerate. If you think of this neural network that does this encoding to the hidden space, this ResNet-50 right here, if you simply want to make the two representations close, what's the best thing it can do? It can simply map all the hidden, it can simply have the constant function h equals zero or something like this. Just a constant function. Because then this loss here is always going to be zero. Like perfect. No matter what image comes in, if you always map it to the same thing you will always be close in representation space and therefore you always win. That doesn't learn a really good representation. What people have done is they have included so-called negative samples where you'll say I'll take a different image from this data set but it's a different image than this image. I also do some data augmentation with that image and then I send this through the same encoder to also give me an h. This is the h, let's call that h original. This is h plus because it's the same image but slightly differently augmented. And this is h minus which is a different image. Now the task is let's make those two very similar to each other but let's distance them from this other one. We want this to be as far away as possible and these two to be close to each other. Now the network can't simply map everything to a constant function anymore. It needs to actually do something to make these be close together and this be far apart. The combination of this together with the augmentation procedure that goes into augmenting the images has been sort of a good combo to learn good representations. A lot of papers have alluded to the fact that this is... The negative samples are to not have these degeneracy, so to not have the simple solutions. But the fact that the representation then is actually good, like is good for image tasks down the line, probably comes from the fact of these augmentations right here. There's a lot of evidence from the fact that depending on which augmentations we choose, these representations are going to be better or worse. For example random cropping of an image, so the random sub, like taking a random crop from the image, tends to be very very beneficial. So here this is the same image twice, right? Let's say we take a random crop here and one up here. Maybe there's an overlap here in the middle, right? So it sort of needs to understand that these random crops sort of needs to communicate between these two places in these random crops. So the representation has to somehow make sure that the object that is overlapping here is somehow represented, but it can't represent it just as a pixel value, because it doesn't know where the crops come from. So there's a lot of evidence that these representations are the thing that's responsible for making the representation so good. Okay, now this paper simply says do we really need these negative samples right here? Let's just get rid of them. And with a couple of tricks this seems to work. And this is what seems like magic to me, because as we go forward, think of it, nothing keeps this model right here from doing the degenerate solution h equals constant. Nothing, right? Now for some reason it doesn't do that. And I have the feeling that this is a super delicate balance that you have to do, because when you train, when you start out, it's probably not the constant function, right? It's probably some distribution. And then simply by the fact that you train it and kind of keep it in the... So this is certainly an optimal solution, but you might be like in some sort of local minimum once you start training and you simply don't get out of it during training. And that's why the network has an easier time step by step as it updates itself in very small incremental steps. It has an easier time actually going for the good representation than it has to see this solution right here and converge to that. But yeah, it seems delicate. So what are they doing? They are taking that idea of taking an input image right here. And so by the way, why is it important that there are no negative samples? Because now the question is always, oh, where do you get these negative samples from? Right? Should they be uniformly sampled? Should we keep a buffer? Should we order them? There is this task of hard negative mining where you say, oh, any old negative won't do. It's actually better if we take negatives that are, you know, just hard enough. There is a curriculum learning problems and so on. So it would be best to actually just get rid of these negative things. So that's why we want to get rid of them. So that's the approach. BYOL. Bootstrap your own latent. There is the input image. You take one image at a time and you apply two different random augmentations to it. Right? So you create two slightly different variants of that image through augmentation. And again, this can be something like a random crop. It can be a horizontal flip randomly. You color jitter, you solarize, you blur, and so on. There are all these variants of data augmentation. And the fact that down the line, the representation of these two things has to be close to each other. I think these random, these augmentations here are responsible to make the representations powerful. The fact that later down the line, the network has to sort of learn to ignore these. It has to learn that, oh, you know, it doesn't matter where in the image this object is, because it's been random cropped for different, you know, at different locations. It doesn't matter where in the image this object is. I simply need to have my hidden representation have this particular object in the image. And that's what makes it powerful. Okay, I've said that enough now. Then you have these two slightly different versions. And then you map it through your encoder. Okay, let's go the top path first. You see the bottom path has the same encoder, but the parameters are different. And this is going to be one of the crucial elements right here. So this here are your actual parameters that you learn. And this here are what are called the target parameters. Now after each, and you can see this for all of these components right here. So what happens is that the target parameters are basically a copy of these what's what are called the online parameters. Okay, so after each step, you copy over from the online parameters, you copy over to the target parameters, you never learn the target parameters, you simply copy them after each step. Now you don't copy them outright, what you do is you do an exponential moving average. So the target parameters are always going to be sort of a lagging average of your online parameters. And that idea comes from the momentum contrast principle, where the reasoning sort of behind it is that you need a kind of a stable you kind of need a stable representation as a target. But I think it hasn't been fully explored or explained why exactly that is so helpful. But we just know that if if we have the target to be not the same as the the online parameters, but actually a kind of a stable version of the past of the online parameters, then that tends to work well. Again, it's kind of the same principle as with the augmentations. With the augmentations, we have two different versions of the same image. And now with this procedure here, we sort of have two different versions of the same neural network, but they're slightly different, right. This idea has been around for much longer, like the first queue, deep queue networks, and so on. They had the same principles where they had the the network that they actually learned and then the target network that is copied over every such and such episodes, and so on. So this, this seems to work seems to be a fundamental principle that seems to work. All right, so we take our two slightly different augmented versions of the same image, and we run them through our two slightly different encoders to obtain two representations. Now this thing right here, that's going to be our representer. So after this procedure, we discard the entire thing right here, except that. So this here is your whatever your ResNet 50. Okay, after that follows a projection. And the projection is is here to reduce the dimensionality. And honestly, I'm actually not sure why it is here. Because you can do it without, like technically, the algorithm doesn't require this projection. So you can imagine the algorithm without the projection. But just really quickly, the projection simply brings down the representation, which is like 2048 dimensional that comes out of the ResNet 50. It has, it is a two layer neural network that first pumps this up to like 4092, and then compresses it down to 256 dimensions. Okay, so that's the projection network. Again, there is a part that's learned and then the target projector is simply the exponential moving average of the online projector. But again, this is why exactly this is here, probably simply because it works, right? But probably because there is no distinction because you don't have different losses, you simply back propagate through everything and then train everything. So there is no logical distinction between the projection and the representation other than you have a different dimensionality. But maybe that's the point here that you make a different dimensionality, even though you could you could do the rest in this 2048 space. Yeah, so for now, just this doesn't exist. Let's just say this doesn't exist. And we just work with this representation here. Let's call this Z, Z prime. Okay, so what happens is we take the representation. And now we have one neural network, the predictor right here, that takes the representation of one of the image versions. And it simply tries to predict the representation of the other image version. So what you want is that q of z equals z prime. Okay, and if we expand that is that q of f of z is equal to f target of z prime. And if we expand that even further, you can see that q, I'll just write q and f for now q of f of a, which is an augmentation at an augmentation of z should be one bracket two bracket three bracket should be f of a of z sorry not see that's the image x. Alright, so this makes a lot of sense. You're simply with q. Since these are all different here, so f is the target instead of the online parameters, a is also different, it's a different augmentation that you do, but the x is the same. Okay, so the queue simply tries to somehow negate this augmentation and this difference between the target and the online parameters. But you don't tell the queue which augmentation was used. And you don't tell the queue what are the exact parameters of that network. So what the queue has to do is it has to somehow it's like it's like it has to take its best guess, right? So basically the queue is trained to output the expected value of the representation right the expected value of the representation f of a of x under all of the different possible image augmentations. And that's why it learns to ignore these augmentations. So your entire goal with these methods is you learn to ignore these augmentations. So you want to learn some method that is independent of the augmentations. So by crafting the augmentations in a smart way, we can make these representations contain a lot of semantic information, because what we want to do with the augmentation is basically we want to destroy all the non-segmented information. So non-semantic information. And random cropping is one of those methods. Horizontal flipping is one of those methods, because we say, well, whether an image goes left to right or right to left, most of the time the semantics are the same. The pixels are different, but the semantics are the same. So by putting an augmentation in there, we learn to ignore that augmentation, because our representation now needs to be predictable. We learn Q to predict the representation under the expectation of our augmentations. And that means it can't be dependent on one particular augmentation. It learns to ignore it. So that's basically what's happening here. Again, there is nothing keeping this from simply collapsing it to a trivial solution. And it's probably a combination of the initialization and the learning procedure itself, that it goes on in little, little steps, one by one, that keeps it in the realm of rather having to... Like it's easier to learn a good representation than it is to collapse to that solution. Okay? So again, components is image, then you augment differently, then you run it through different encoders, but the encoders are similar in the fact that one is the exponential moving average of the other. And then you try to predict one from the other. And that ultimately makes the representation be independent of the augmentation. And that means that the representation can only include things that are not destroyed by the augmentations. And if you construct the augmentations smartly, that means you only retain the semantic information. That's it. So the loss function is pretty simple. As you can see right here, what you want is, and this bar is a normalization, what you want is the L2 norm between this representation be close to the Q of that representation. So the Q simply tries to predict the other representation. And you do that for both ways. So you once stick the image in here and try to predict the other one, and you do it vice versa. So you get two loss components each time. It's a symmetric loss. And that's it. That's the method. And they beat all the other self-supervised methods, and they get pretty close to the supervised representation learning method. As you can see right here, as the number of parameters goes up in their model, so one of them is ResNet-50, but I'm gonna guess this one right here. But you can also get two higher architectures, and then it appears to work even better and come even closer to this supervised baseline. This could be because if you have more parameters technically in a supervised method, you would also need more labeled images maybe, and therefore it doesn't scale as well. I don't know. There is a lot of unclarity in this research. All they show is that their numbers are good, which is cool, right? And it's cool that you don't need the negative samples anymore, and it actually doesn't collapse when you do that kind of stuff. But there's a lot of, I don't know, there's a lot of things here. For example, we use a batch size of 4096 split over 512 TPUv3 cores. With this setup, training takes approximately eight hours for ResNet-50. So they train eight hours on 512 TPUs. Just imagine that! So that's sort of crazy amount of computation, again, going into these models. And then the second thing here is that you can see that there are some things missing right here, and there are all these annotations, which probably means that they take these numbers from those papers. Now, they allude to the fact that they try to follow their protocol as closely as possible, but I mean, that's never given. Or almost never, unless they release the exact code, and even then there are still going to be differences. You'd have to replicate the exact thing on the exact same number of TPU cores and whatnot. So I highly like these numbers seem to be... I'm not sure, especially if you then go and look, and at some point they actually do reproduce the SimClear baseline. So you can see right here that they have a own implementation of SimClear, and they actually compare this to the numbers that they find in the SimClear paper. And you can see, for example, here there's like four percentage points that their implementation of SimClear gains above this implementation. And if you look at this supervised baseline, that's also from that paper. And there is a graph further down where they also implement their own version of the... their own version of the supervised baseline. I forget... here. So you can see that between the supervised in that paper and the supervised of them, sometimes there's like a giant gap right here for the same model, it seems. So all of these numbers, I'm not sure you should put too much weight on the fact that this is now outperforming the other methods. I would not put... like unless this is like super duper replicated very often, I would not put a lot of weight on the fact that it is better. What I would put a lot of weight on is the fact that it works at all and achieves, you know, good performance. And there is more. They make... they have like experiments right here that show that their method, the BYOL, is much more resistant to like changes in hyperparameters. So here you can see that it falls off much later when you reduce the batch size, which makes sense, right? Because SimClear is one of these methods that uses negative samples. And for negative samples, it uses the other samples in the mini batch. Now if you have less samples in the mini batch, that means you have a less representative distribution of your entire data set as negative samples. And therefore, if you increase... decrease the mini batch, then this drops off. And also they show that, for example, their method is much more robust to the removal of a couple of these image augmentations. So all of this I find actually pretty cool. But the actual numbers here... first, I'm not super duper interested that they get like two or one points more in something, but they do perform like a lot of experiments. And that... it shows that you can apply the method to different things. It's not only like in one setting, so that's pretty cool. It works at least... you can say it works at least as well as other methods. And it is a lot easier because you don't have this negative sample things. Now the last quarrel I have with the paper, and where is it? Where is it? Somewhere they say that we release the code... they release the pseudo code. They don't release the code. They release the pseudo code in the appendix. So I mean, there are reasons why you sometimes want to release pseudo code. And that's if an algorithm is so high level and so simple in its high levelity and so modular to be fleshed out that you can't... like it makes more sense. But here it's like pseudo code in jacks. And come on... is it really that competitively advantageous to retain your code? It's just not reproducible with this. You know that they have like 50 billion hacks in their code. And yeah, so DeepMind has this history of just not releasing... like publishing behind paywalls and just giving pseudo code that has lots of mistakes in them. Like the new zero pseudo code, you can't even like run it in its basic form if you fill in the things. It's a bit annoying. In any way, the method itself seems promising for representation learning, as I said, especially because it's pretty simple. It still heavily relies on these augmentation methods. So and that's what they say right here. Nevertheless, BYOL remains dependent on existing sets of augmentations that are specific to vision applications. To generalize BOL to other modalities, it is necessary to obtain similarly suitable augmentations for each of them. Designing such augmentations may require significant effort and expertise. Therefore automating the search for these augmentations would be an important next step to generalize BOL to other modalities. And I'm not sure if you can do this automating the search for these augmentations. I guess you can do it if you have like a supervised data set and then you can search and then you can use those augmentations for the unsupervised. But it seems a bit bootstrap-y, no pun intended right here. I think the power of these representations again comes from the fact that we have these augmentations carefully constructed. So oh yes, the last thing broader impact statement. Just read this like try to estimate the perplexity of this broader impact statement. Let's go. The presented research should be categorized as research in the field of unsupervised learning. This work may inspire new algorithms, theoretical and experimental investigation. The algorithm presented here can be used for many different vision applications and a particular use may have both positive or negative impacts, which is known as the dual use problem. Besides as vision data sets could be biased, the representation learned by BOL could be susceptible to replicate these biases. Like come on. So people who advocated for making everyone do this. Is this what you wanted? Is this like is this a satisfactory result for you? And if you have this as a reviewer, is this okay or not? I mean let's just cross out some words here. Blank, like field, let's just put field. Or machine learning. Why not? Machine learning. Machine learning. This work inspire new algorithms? Yes. The algorithm presented here can be used for many different machine learning applications and a particular use may have both negative effects. Besides as data sets could be biased, the representation learned by this paper could be susceptible to replicate these biases. Well there is a copy-paste thing that you can apparently put into any and all papers that you write from now on. And hey DeepMind is doing it. So you know, there you go. Okay maybe a bit cynical but I'm like I told you this would happen. I told you. And you know. Okay so that was it for my comments right here. They do have like a giant ton of experiments and I appreciate that right. They really try to show that it works in many different situations and yeah yet to solve why this doesn't collapse but apparently it doesn't. So try it out. Give it a try and I'll see you next time. Bye bye.
 Hello there! Today we're looking at Bootstrap Your Own Latent, a new approach
 to self-supervised learning by researchers of DeepMind and Imperial
 College. So almost no day goes by where we don't hear some sort of new
 self-supervised algorithm right here. This paper on a high level tries to get
 rid of the necessary negative samples when doing the contrastive loss for
 self-supervised learning. They basically combine momentum contrast and
 same clear and then remove the negative samples. That seems to work pretty
 well even though it's magic. So yeah, if you want to see how it's done, stick
 around, share the video out. If you want other people to see how it's done, leave
 a comment. This one I really don't get what's going on. So if you have
 ideas, put them there. I'll read them through. It'll be fun.
 Alright, so they say we introduce Bootstrap Your Own Latent or B.O.L. a new
 approach to self-supervised image representation learning. Image
 representation learning is the simple task of taking an image and then feeding
 it through a function which is usually like a neural network. Let's just
 say this is a neural network and in fact all of these, the community has sort of
 standardized this to be, most of the time it's something like a ResNet-50.
 So what you want to do is you want to train a neural network like a ResNet-50
 to give you a good representation of the image. So this would be like H and H is a
 vector and H is a representation of this image and the representation should be
 such that you can then take this representation and solve many tasks with
 it, which either can be like linear, you can put a linear classifier on top of
 the H or you can fine-tune the entire architecture to solve some other tasks.
 The idea is if you have a large data set here you may use this data set to train
 these good representations of these images and then you can transfer
 this to a task where you might not have as much data. Because you
 don't have as much data it's not enough to completely train an architecture like
 this, but it is enough to take an architecture that's been trained with
 the large data set and just adapt it to your small data set. That usually
 tends to work pretty well. This is called transfer learning. This step here
 is called fine-tuning sometimes and it's sort of the approach that comes from
 natural language processing from these big transformers like BERT where you
 first train on a really big data set that might not be the data set that you
 want in the end but it's really big so you can sort of learn a lot of things
 from that data set and then the only thing left to do is to fine-tune it to
 basically adapt it to the nuances of your data set but it will have learned
 most things already and that's called representation learning. The goal is
 to learn a good representation. The self-supervised here is also important
 because representation learning can be as easy as if this here is ImageNet.
 The ImageNet data set contains like a million images all with labels. You can
 simply train your ResNet-50 to predict the class. This is called
 supervised pre-training or supervised representation learning and that works
 pretty well but you need a labeled data set. In self-supervised learning you do
 not need labels. What you do is you do self-supervision and self-supervision
 there are many ways to do self-supervision but what we'll
 see in this particular paper is that you will take an image and you'll make
 different variants of that same image. You'll take the image and you'll make
 many many variants of it. Let's just say two. You have some procedure to
 sort of change the picture a little bit but it's essentially still the same and
 you do that through data augmentation. This could be a random crop or you
 color jitter or you rotate it or something like this and then you exploit
 the fact that you know that these two things they should be still sort of the
 same image. Once you send them through your encoder the
 representations of the two images they should be fairly close. Now let's
 actually read on right here. Bjoll relies on two neural networks referred to as
 online and target networks that interact and learn from each other. From an
 augmented view of an image we train the online network to predict the target
 representation of the same image under a different augmented view. That's
 sort of what we saw. We have the same image under a different
 augmented view. What does it mean? You make two versions of
 the same image. One that are slightly different and then their representation
 should be close. Until this point we have always thought that this would
 degenerate. If you think of this neural network that does this
 encoding to the hidden space, this ResNet-50 right here, if you
 simply want to make the two representations close, what's the best
 thing it can do? It can simply map all the hidden, it can simply have the
 constant function h equals zero or something like this. Just a constant
 function. Because then this loss here is always going to be zero. Like perfect.
 No matter what image comes in, if you always map it to the same thing you
 will always be close in representation space and therefore you always win.
 That doesn't learn a really good representation. What people have
 done is they have included so-called negative samples where you'll say I'll
 take a different image from this data set but it's a different
 image than this image. I also do some data augmentation with that
 image and then I send this through the same encoder to also give me an h.
 This is the h, let's call that h original. This is h plus because it's the same
 image but slightly differently augmented. And this is h minus which is a different
 image. Now the task is let's make those two very similar to each other but
 let's distance them from this other one. We want this to be as far away
 as possible and these two to be close to each other. Now the network can't simply
 map everything to a constant function anymore. It needs to actually do
 something to make these be close together and this be far apart. The
 combination of this together with the augmentation procedure that goes into
 augmenting the images has been sort of a good combo to learn good
 representations. A lot of papers have alluded to the fact that this is...
 The negative samples are to not have these degeneracy, so to not have
 the simple solutions. But the fact that the representation then is actually good,
 like is good for image tasks down the line, probably comes from the
 fact of these augmentations right here. There's a lot of evidence from the
 fact that depending on which augmentations we choose, these
 representations are going to be better or worse. For example random cropping of
 an image, so the random sub, like taking a random crop from the image, tends to be
 very very beneficial. So here this is the same image twice, right? Let's
 say we take a random crop here and one up here. Maybe there's an
 overlap here in the middle, right? So it sort of needs to understand that these
 random crops sort of needs to communicate between these two places in
 these random crops. So the representation has to somehow make sure that the
 object that is overlapping here is somehow represented, but it can't
 represent it just as a pixel value, because it doesn't know where the crops
 come from. So there's a lot of evidence that these representations are the thing
 that's responsible for making the representation so good. Okay, now this
 paper simply says do we really need these negative samples right here? Let's
 just get rid of them. And with a couple of tricks this seems to work. And this
 is what seems like magic to me, because as we go forward, think of it,
 nothing keeps this model right here from doing the degenerate solution
 h equals constant. Nothing, right? Now for some reason it doesn't do that. And I
 have the feeling that this is a super delicate balance that you have to do,
 because when you train, when you start out, it's probably not the constant
 function, right? It's probably some distribution. And then simply by the
 fact that you train it and kind of keep it in the... So this is certainly an
 optimal solution, but you might be like in some sort of local minimum once you
 start training and you simply don't get out of it during training. And that's why
 the network has an easier time step by step as it updates itself in very small
 incremental steps. It has an easier time actually going for the good
 representation than it has to see this solution right here and converge to that.
 But yeah, it seems delicate. So what are they doing? They are taking that idea of
 taking an input image right here. And so by the way, why is it important that
 there are no negative samples? Because now the question is always, oh, where do
 you get these negative samples from? Right? Should they be uniformly sampled?
 Should we keep a buffer? Should we order them? There is this task of hard negative
 mining where you say, oh, any old negative won't do. It's actually better if we take
 negatives that are, you know, just hard enough. There is a curriculum
 learning problems and so on. So it would be best to actually just get rid of these
 negative things. So that's why we want to get rid of them. So that's the approach.
 BYOL. Bootstrap your own latent. There is the input image. You take one image at a
 time and you apply two different random augmentations to it. Right? So you create
 two slightly different variants of that image through augmentation. And again,
 this can be something like a random crop. It can be a horizontal flip randomly.
 You color jitter, you solarize, you blur, and so on. There are all these variants of
 data augmentation. And the fact that down the line, the representation of
 these two things has to be close to each other. I think these random, these
 augmentations here are responsible to make the representations powerful.
 The fact that later down the line, the network has to sort of learn to ignore
 these. It has to learn that, oh, you know, it doesn't matter where in the image this
 object is, because it's been random cropped for different, you know, at
 different locations. It doesn't matter where in the image this object is. I
 simply need to have my hidden representation have this particular
 object in the image. And that's what makes it powerful. Okay, I've said that
 enough now. Then you have these two slightly different versions. And then you
 map it through your encoder. Okay, let's go the top path first. You see the bottom
 path has the same encoder, but the parameters are different. And this is
 going to be one of the crucial elements right here. So this here are your actual
 parameters that you learn. And this here are what are called the target
 parameters. Now after each, and you can see this for all of these components
 right here. So what happens is that the target parameters are basically a copy
 of these what's what are called the online parameters. Okay, so after each
 step, you copy over from the online parameters, you copy over to the target
 parameters, you never learn the target parameters, you simply copy them after
 each step. Now you don't copy them outright, what you do is you do an
 exponential moving average. So the target parameters are always going to be sort
 of a lagging average of your online parameters. And that idea comes from the
 momentum contrast principle, where the reasoning sort of behind it is that you
 need a kind of a stable you kind of need a stable representation as a target. But
 I think it hasn't been fully explored or explained why exactly that is so helpful.
 But we just know that if if we have the target to be not the same as the the
 online parameters, but actually a kind of a stable version of the past of the
 online parameters, then that tends to work well. Again, it's kind of the same
 principle as with the augmentations. With the augmentations, we have two
 different versions of the same image. And now with this procedure here, we sort of
 have two different versions of the same neural network, but they're slightly
 different, right. This idea has been around for much longer, like the first
 queue, deep queue networks, and so on. They had the same principles where they had
 the the network that they actually learned and then the target network that
 is copied over every such and such episodes, and so on. So this, this seems
 to work seems to be a fundamental principle that seems to work. All right, so
 we take our two slightly different augmented versions of the same image, and
 we run them through our two slightly different encoders to obtain two
 representations. Now this thing right here, that's going to be our representer.
 So after this procedure, we discard the entire thing right here, except that. So
 this here is your whatever your ResNet 50. Okay, after that follows a projection.
 And the projection is is here to reduce the dimensionality. And honestly, I'm
 actually not sure why it is here. Because you can do it without, like
 technically, the algorithm doesn't require this projection. So you can
 imagine the algorithm without the projection. But just really quickly, the
 projection simply brings down the representation, which is like 2048
 dimensional that comes out of the ResNet 50. It has, it is a two layer neural
 network that first pumps this up to like 4092, and then compresses it down to 256
 dimensions. Okay, so that's the projection network. Again, there is a part that's
 learned and then the target projector is simply the exponential moving average of
 the online projector. But again, this is why exactly this is here, probably
 simply because it works, right? But probably because there is no
 distinction because you don't have different losses, you simply back propagate
 through everything and then train everything. So there is no logical
 distinction between the projection and the representation other than you have a
 different dimensionality. But maybe that's the point here that you make a
 different dimensionality, even though you could you could do the rest in this
 2048 space. Yeah, so for now, just this doesn't exist. Let's just say this
 doesn't exist. And we just work with this representation here. Let's call this Z, Z
 prime. Okay, so what happens is we take the representation. And now we have one
 neural network, the predictor right here, that takes the representation of one of
 the image versions. And it simply tries to predict the representation of the
 other image version. So what you want is that q of z equals z prime. Okay, and if
 we expand that is that q of f of z is equal to f target of z prime. And if we
 expand that even further, you can see that q, I'll just write q and f for now
 q of f of a, which is an augmentation at an augmentation of z should be one
 bracket two bracket three bracket should be f of a of z sorry not see that's the
 image x. Alright, so this makes a lot of sense. You're simply with q. Since these
 are all different here, so f is the target instead of the online parameters,
 a is also different, it's a different augmentation that you do, but the x is the
 same. Okay, so the queue simply tries to somehow negate this augmentation and this
 difference between the target and the online parameters. But you don't tell the
 queue which augmentation was used. And you don't tell the queue what are the
 exact parameters of that network. So what the queue has to do is it has to
 somehow it's like it's like it has to take its best guess, right? So basically
 the queue is trained to output the expected value of the representation
 right the expected value of the representation f of a of x under all of
 the different possible image augmentations. And that's why it learns
 to ignore these augmentations. So your entire goal with these methods is you
 learn to ignore these augmentations. So you want to learn some method that is
 independent of the augmentations. So by crafting the augmentations in a smart
 way, we can make these representations contain a lot of semantic information,
 because what we want to do with the augmentation is basically we want to
 destroy all the non-segmented information. So non-semantic information.
 And random cropping is one of those methods. Horizontal flipping is one of
 those methods, because we say, well, whether an image goes left to right or
 right to left, most of the time the semantics are the same. The pixels are
 different, but the semantics are the same. So by putting an augmentation in there,
 we learn to ignore that augmentation, because our representation now needs to
 be predictable. We learn Q to predict the representation under the
 expectation of our augmentations. And that means it can't be dependent on one
 particular augmentation. It learns to ignore it. So that's basically what's
 happening here. Again, there is nothing keeping this from simply collapsing it
 to a trivial solution. And it's probably a combination of the initialization and
 the learning procedure itself, that it goes on in little, little steps, one by
 one, that keeps it in the realm of rather having to... Like it's easier to learn a
 good representation than it is to collapse to that solution. Okay? So again,
 components is image, then you augment differently, then you run it through
 different encoders, but the encoders are similar in the fact that one is the
 exponential moving average of the other. And then you try to predict one from the
 other. And that ultimately makes the representation be independent of the
 augmentation. And that means that the representation can only include things
 that are not destroyed by the augmentations. And if you construct the
 augmentations smartly, that means you only retain the semantic information.
 That's it. So the loss function is pretty simple. As you can see right here, what
 you want is, and this bar is a normalization, what you want is the L2
 norm between this representation be close to the Q of that
 representation. So the Q simply tries to predict the other representation. And you
 do that for both ways. So you once stick the image in here and try to predict the
 other one, and you do it vice versa. So you get two loss components each time.
 It's a symmetric loss. And that's it. That's the method. And they beat all the
 other self-supervised methods, and they get pretty close to the supervised
 representation learning method. As you can see right here, as the
 number of parameters goes up in their model, so one of them is ResNet-50, but
 I'm gonna guess this one right here. But you can also get two higher
 architectures, and then it appears to work even better and come even closer to
 this supervised baseline. This could be because if you have more
 parameters technically in a supervised method, you would also need more labeled
 images maybe, and therefore it doesn't scale as well. I don't know. There is a
 lot of unclarity in this research. All they show is that their numbers are
 good, which is cool, right? And it's cool that you don't need the
 negative samples anymore, and it actually doesn't collapse when you do that kind
 of stuff. But there's a lot of, I don't know, there's a lot of things here. For
 example, we use a batch size of 4096 split over 512 TPUv3 cores. With this
 setup, training takes approximately eight hours for ResNet-50. So they train eight
 hours on 512 TPUs. Just imagine that! So that's sort of crazy amount of
 computation, again, going into these models. And then the second thing here is
 that you can see that there are some things missing right here, and there are
 all these annotations, which probably means that they take these numbers
 from those papers. Now, they allude to the fact that they try to follow their
 protocol as closely as possible, but I mean, that's never given.
 Or almost never, unless they release the exact code, and even then there
 are still going to be differences. You'd have to replicate the
 exact thing on the exact same number of TPU cores and whatnot. So I highly
 like these numbers seem to be... I'm not sure, especially if you then go and look,
 and at some point they actually do reproduce the SimClear baseline. So you
 can see right here that they have a own implementation of SimClear, and they
 actually compare this to the numbers that they find in the SimClear paper. And
 you can see, for example, here there's like four percentage points that
 their implementation of SimClear gains above this implementation. And if you
 look at this supervised baseline, that's also from that paper. And there is a
 graph further down where they also implement their own version of the...
 their own version of the supervised baseline. I forget... here. So you can see
 that between the supervised in that paper and the supervised of them,
 sometimes there's like a giant gap right here for the same model, it seems. So all
 of these numbers, I'm not sure you should put too much weight on the fact
 that this is now outperforming the other methods. I would not put... like unless
 this is like super duper replicated very often, I would not put a lot of weight on
 the fact that it is better. What I would put a lot of weight on is the fact that
 it works at all and achieves, you know, good performance. And there is more. They
 make... they have like experiments right here that show that their method, the
 BYOL, is much more resistant to like changes in hyperparameters. So here you
 can see that it falls off much later when you reduce the batch size, which
 makes sense, right? Because SimClear is one of these methods that uses negative
 samples. And for negative samples, it uses the other samples in the mini batch. Now
 if you have less samples in the mini batch, that means you have a less
 representative distribution of your entire data set as negative samples. And
 therefore, if you increase... decrease the mini batch, then this drops off. And also
 they show that, for example, their method is much more robust to the removal of a
 couple of these image augmentations. So all of this I find actually pretty cool.
 But the actual numbers here... first, I'm not super duper interested that they
 get like two or one points more in something, but they do perform like a lot
 of experiments. And that... it shows that you can apply the method to different
 things. It's not only like in one setting, so that's pretty cool. It works at least...
 you can say it works at least as well as other methods. And it is a lot easier
 because you don't have this negative sample things. Now the last quarrel I
 have with the paper, and where is it? Where is it? Somewhere they say that we
 release the code... they release the pseudo code. They don't release the code.
 They release the pseudo code in the appendix. So I mean, there are reasons why
 you sometimes want to release pseudo code. And that's if an algorithm is so
 high level and so simple in its high levelity and so modular to be fleshed
 out that you can't... like it makes more sense. But here it's like pseudo code in
 jacks. And come on... is it really that competitively advantageous to retain
 your code? It's just not reproducible with this. You know that they
 have like 50 billion hacks in their code. And yeah, so DeepMind has this history of
 just not releasing... like publishing behind paywalls and just giving pseudo
 code that has lots of mistakes in them. Like the new zero pseudo code, you can't
 even like run it in its basic form if you fill in the things. It's a bit
 annoying. In any way, the method itself seems promising for representation
 learning, as I said, especially because it's pretty simple. It still heavily
 relies on these augmentation methods. So and that's what they say right here.
 Nevertheless, BYOL remains dependent on existing sets of
 augmentations that are specific to vision applications. To generalize BOL to
 other modalities, it is necessary to obtain similarly suitable augmentations
 for each of them. Designing such augmentations may require significant
 effort and expertise. Therefore automating the search for these
 augmentations would be an important next step to generalize BOL to other
 modalities. And I'm not sure if you can do this automating the search for these
 augmentations. I guess you can do it if you have like a supervised data set and
 then you can search and then you can use those augmentations for the
 unsupervised. But it seems a bit bootstrap-y, no pun intended right here. I
 think the power of these representations again comes from the
 fact that we have these augmentations carefully constructed. So oh yes, the last
 thing broader impact statement. Just read this like try to estimate the
 perplexity of this broader impact statement. Let's go. The presented
 research should be categorized as research in the field of unsupervised
 learning. This work may inspire new algorithms, theoretical and experimental
 investigation. The algorithm presented here can be used for many different
 vision applications and a particular use may have both positive or negative
 impacts, which is known as the dual use problem. Besides as vision data sets
 could be biased, the representation learned by BOL could be susceptible to
 replicate these biases. Like come on. So people who advocated for making everyone
 do this. Is this what you wanted? Is this like is this a satisfactory result for
 you? And if you have this as a reviewer, is this okay or not? I mean let's just
 cross out some words here. Blank, like field, let's just put field. Or
 machine learning. Why not? Machine learning. Machine learning. This work
 inspire new algorithms? Yes. The algorithm presented here can be used for many
 different machine learning applications and a particular use may have both negative
 effects. Besides as data sets could be biased, the representation learned by this
 paper could be susceptible to replicate these biases. Well there is a copy-paste
 thing that you can apparently put into any and all papers that you write from
 now on. And hey DeepMind is doing it. So you know, there you go. Okay maybe a bit
 cynical but I'm like I told you this would happen. I told you. And you know.
 Okay so that was it for my comments right here. They do have like a giant ton of
 experiments and I appreciate that right. They really try to show that it works in
 many different situations and yeah yet to solve why this doesn't collapse but
 apparently it doesn't. So try it out. Give it a try and I'll see you next time. Bye bye.