Yann LeCun - Self-Supervised Learning: The Dark Matter of Intelligence (FAIR Blog Post Explained)
deep learning machine learning explained neural networks artificial intelligence deep learning tutorial what is deep learning introduction to deep learning what is self supervised learning self supervised learning self-supervised learning self-supervised learning yann lecun yann lecun yann lecun energy based models energy based models energy based machine learning energy based models deep learning byol contrastive learning bert noise contrastive estimation
#selfsupervisedlearning #yannlecun #facebookai

Deep Learning systems can achieve remarkable, even super-human performance through supervised learning on large, labeled datasets. However, there are two problems: First, collecting ever more labeled data is expensive in both time and money. Second, these deep neural networks will be high performers on their task, but cannot easily generalize to other, related tasks, or they need large amounts of data to do so. In this blog post, Yann LeCun and Ishan Misra of Facebook AI Research (FAIR) describe the current state of Self-Supervised Learning (SSL) and argue that it is the next step in the development of AI that uses fewer labels and can transfer knowledge faster than current systems. They suggest as a promising direction to build non-contrastive latent-variable predictive models, like VAEs, but ones that also provide high-quality latent representations for downstream tasks.

OUTLINE:
0:00 - Intro & Overview
1:15 - Supervised Learning, Self-Supervised Learning, and Common Sense
7:35 - Predicting Hidden Parts from Observed Parts
17:50 - Self-Supervised Learning for Language vs Vision
26:50 - Energy-Based Models
30:15 - Joint-Embedding Models
35:45 - Contrastive Methods
43:45 - Latent-Variable Predictive Models and GANs
55:00 - Summary & Conclusion

Paper (Blog Post): https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence
My Video on BYOL: https://www.youtube.com/watch?v=YPfUiOMYOEE

ERRATA:
- The difference between loss and energy: Energy is for inference, loss is for training.
- The R(z) term is a regularizer that restricts the capacity of the latent variable. I think I said both of those things, but never together.
- The way I explain why BERT is contrastive is wrong. I haven't figured out why just yet, though :)

Video approved by Antonio.

Abstract:
We believe that self-supervised learning (SSL) is one of the most promising ways to build such background knowledge and approximate a form of common sense in AI systems.

Authors: Yann LeCun, Ishan Misra

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there. Today we're looking at Self-Supervised Learning, the Dark Matter of Intelligence. This was written by Jan LeCun and Ishan Misra of Facebook AI Research. And it is not a paper, it is more a blog post shared on the Facebook AI blog. And it outlines the current state of self-supervised learning, what it is and what it can do, why the authors think it is important. It goes over things like BERT, goes over things like Contrastive Learning, energy-based models, GANs and so on. And at the end it gives a bunch of recommendations for the way to go forward. On a high level the main recommendation is that we should build latent variable prediction models that are not trained contrastively. And we'll go through all of what this means in this article. So we'll go through the article. I'll switch over to here where it's a bit of a more legible format. And as always, if you like content like this, if you enjoy it, share it out. Don't hesitate to tell a friend about it. All right, let's do it. They say in recent years the AI field has made tremendous progress in developing AI systems that can learn from massive amounts of carefully labeled data. So the keywords here are massive amounts. Yes, we got that. But carefully labeled data. Of course, we all know that supervised learning has worked very well if you have enough labeled data. And that's exactly the problem. In order to push machine learning to more, to higher abilities, it seems like what we need is first of all bigger architectures, which we can do by just building bigger computers. But we also need more data. The problem here is that we need orders of magnitude more data and labeling that data is going to be very, very expensive. And therefore, we're looking for methods that can do without labeled data, that can learn most of what they learn from non labeled data, and then apply that to a little bit of labeled data in order to learn a task. But this is not the only thing. So the need the expansiveness of labeling is not the only thing that they criticize here. They say this paradigm of supervised learning has a proven track record for training specialist models that perform extremely well on the tasks they were trained to do. So this is another criticism right here. Namely, that if we train something in a supervised fashion with labels, it will become or it might become very good, but it will be very good at that particular task. And it won't be super good at other tasks, such as, you know, tasks that are tasks that are relatively neighboring to the field that we're concerned about. They go on they say that supervised learning is a bottleneck for building more intelligent generalist models that can do multiple tasks and acquire new skills without massive amounts of labeled data. This is into the direction of Francois Chollet, who defines intelligence as the efficiency with which you transform new data into new skills. And this is reflected here in this article by Jan Lecoe. And I'm sorry, Ishan, but Jan Lecoe just has the big name. And unfortunately, you're a bit in his shadow here. But I'm fairly confident these that Jan Lecoe is not just on this for the name, because the arguments in this article he has raised in many talks that I've seen of him in the past few years. So it is it is really kind of a condensing of all of these talks in this here. But back to the paper, this acquiring new skills without massive amounts of labeled data. They say that has to be our goal, because it is impossible to label everything in the world. And there are also some tasks where there is not enough labeled data, like translation systems for low resource languages. So they make two observations right here. First of all, they say, look, here, for example, if we show just a few drawings of cows to small children, they'll eventually be able to recognize any cow they see. By contrast, AI systems trained with supervised learning require many examples of carmages and might still fail to classify cows in unusual situations, such as lying on a beach. What are you doing, silly cow? Don't lie on a beach. So this is another point, right? These these AI systems, they take so much more data than humans to learn new skills. And they ask why the short answer is that humans rely on their previously acquired knowledge of how the world works. So they make this, they make this argument here that there is a thing like common knowledge about the world or common sense forms the bulk of biological intelligence in both humans and animals. Humans are animals. Okay, this common sensibility is taken for granted, but has remained an open challenge in AI research. Common sense, they say is the dark matter of artificial intelligence. So they point out that you have this common sense that you learn simply by interacting with the world. They say as babies, we learn how the world works largely by observations, you form predictive models about the world, you learn concepts such as object permanence and gravity. And later in life, you you even act in the world. Now they're not going into this acting in the world. But their point is that throughout your life, you just observe the world and you build these predictive models. And that's how you will learn about how the world works. I'm not entirely sure that things like gravity are learned in this way. I think there's some evidence that at least part of it is biological or at least you're extremely biologically predetermined to learn about things like object permanence and gravity. But the point is taken that there is something built into you either from experience or from biology that allows you that is kind of this common sense. And that allows you to acquire new tasks with extremely few additional samples because you bring in this knowledge about the world. So their core claim here is that we believe that self supervised learning is one of the most promising ways to build such background knowledge and approximate a form of common sense in AI systems. They say the way we're going to get AI systems to also have this common sense knowledge is by doing self supervised learning. Right, so they give some examples of self supervised learning. They also contrast it with unsupervised learning, where the difference that so they say unsupervised learning is a bit of a misnomer. Learning is never really unsupervised. Self supervised learning specifically means that you generate the label out of the data itself. So what could that be? You know, for example, in in BERT, the language model, you might have a sentence like this is a cat. And this is a sentence from the data set. Now in self supervised learning, you would somehow need to come up with an input sample and a label for that input sample, just by just using this text, right in a supervised in a supervised data set, you would have some label associated with this. And this could be anything depending on what the task is, like, this could be labels could be annotations for what kind of words these words are, label could be whether or not the sentence is a positive or negative sentence. But in self supervised learning, you can do something like this. And here's what BERT does, they cross out a word like this a, so this now becomes the input sample x, and the label is going to be whatever was missing here. So the label will be the word a. Now, the task of the machine learning system is given x, figure out what is y. Okay, so figure out that at this particular place in the sentence, there should be the word a. Now BERT does a bit more sophisticated things like it also replaces tokens and so on. But ultimately, what you want is for any for any corrupted input to for the system to output the uncorrupted output. And thereby, the system will learn about the world, it will maybe not about the world, but it will learn about language. If it wants to do this task correctly, it needs to learn that if you have a this is construction, there should probably be some kind of specifier for what comes next right here. And then cat is some sort of an object or animal. So given all of this evidence, you only have very few possibilities like a or my or this is a one this is two cat. No, this is your cat. Something like this, but all the other words in the language cannot be. So they formulate self supervised learning as obtaining supervisory signals from the data itself. That's why it's not unsupervised. It is self supervised because you create the label from the data. And the important part here is and I think that's often neglected in the self supervised things is that the way you create the label from the data that is human specified, right, this this step right here, that needs I can I draw a light bulb. That needs a human idea, like how could we create a label and an input data point given a data point. So we shift the burden of the human from labeling the data explicitly to simply saying to simply constructing the method of how to obtain labels from data. This is still building in substantial human bias, but it is much more scalable. If I have one method to create labels, I can apply it to an entire data set. Whereas if I create labels myself, I can go through every single data point. But it's not unsupervised because the supervision is in the process that creates the label. So they say leverage the underlying structure of the data. The general technique of self supervised learning is to predict any unobserved or hidden part or property of the input from any observed or unhidden part of the input. So the general recipe or one, I would say one general recipe because it's not the general recipe, even though they claim it here, I would say one general recipe is that if you have an input, you just hide part of it. And then you have the model predict that hidden part. They give a bunch of examples here. This is quite a cryptic drawing, I think. So these are three examples of what you could do if you have data and this yet time or space, I would claim it's easiest if you think of this as a video sequence. So this is a video sequence and the frames are all they're stacked like this frame, frame, frame. Okay, and it goes up until here. So what you're going to do, what you can do option one is you simply take the past, you define a time point t right here, and you take the past, and that's the observed part. And you take the future, which you have in your data set, but you don't show it to the model. So the model is supposed to predict the future from the past. This in video, you can understand it. This is also what for example, GP, the GPT model, like GPT three does exactly this, it takes in a past words so far, and it predicts the next word or the next few words. The second part is, you don't have to necessarily predict the future, you can also just leave away a bunch of frames in the middle, somewhere at different parts. Now, what the model has to do is has to reason about a part, let's say, a part of the model, let's say this part right here, it has to reason, given the surrounding evidence. So it takes all the evidence into account. And it reasons what kind of frames could have been left out there. In again, in video in NLP land, this would be something like BERT. So BERT is trained in this objective, as a as a masked language model. And then the last one is really quite specific, I think, to something like video, maybe also different modalities, but doesn't apply super well to NLP. Maybe you could though. But this is where if you imagine this being your frames, you not only do you leave away these frames right here, but you also would leave away part of the frames that you observe. So in these frames, you would simply only observe the bottom right thing right here, and you would not observe everything else. So not only do you have to reason about what goes into the missing slot, but you also have to reason about what goes into the parts of the frames you don't observe. And as you can see here, these can be different parts throughout the video. So I think it's just it just makes a point that this can be quite general. So in general, you just hide parts of your input, and you re predict them from a model. And that means the model, you know, if it can, for example, if it can predict the future of a video from the past, given, you know, certain input, it will necessarily have to learn something about how the world works, or at least about how the world looks through a video lens. Right? If it does this task, well, it has a lot of prop captured a lot of properties of how the world looks in in video. And that is much more rich information than simply giving a label to train on. And the hope is that by learning all of these different things that are necessary to predict the future well from the past, the model will learn such a useful representation that adapting this model to solve any labeled supervised task is going to be really quick because it also it already has very, very good representation of the data. And the common thing here is that, okay, in order to predict the order from the past to the future, there can be there can be numerous features that are helpful, right, there are all of these features that are very helpful to predict the future from the past. Now, if I have any supervised task, right, I have, for example, the past, and then I want to determine if I don't know what what can we determine from a video, if this is a happy video, right, is this a happy video or not? The core assumption here is that since you know, predicting the future from the past has sort of the structure of the world built in and since our supervised task is not task is probably a function of a subset of that structure, like whether or not it's a happy video probably depends on whether or not in the future, someone will fall off a cliff or not, right. So a subset of these things in combination are going to be relevant for that task. So they can be adapted. Since the representation is already there, they can be adapted pretty rapidly, while the ones that are not important can maybe be overwritten and relearned to get some additional signal from the from the input that was not learned in the in the self supervised training. So the goal is, again, by learning to predict the hidden inputs from the non hidden inputs, you learn about the structure of the data. By learning about the structure of the data, you get useful representations. And by having useful representations, you can adapt very quickly to new tasks. That's the that's the sort of argument here. So why don't we do this all the time, every time everywhere, they go into self supervised learning for language versus vision. So in language, this is uber duper successful, while in vision, I think in vision, it's fairly successful too. But there is a challenge when you think about language versus vision, specifically in terms of this hiding, hiding parts of the inputs and then reconstructing them. So there are two there are two different things that we need to consider here. The first thing the first problem is dimensionality. Dimensionality. And the second thing we need to consider is uncertainty. Okay, so dimensionality in NLP is what's our dimensionality, if you think of this problem, again, this is a cat, this thing right here. How do we do it in BERT, like we mask out the word, and then we feed this sentence, we feed it through a big neural network that is BERT. And then at the end, at this position, we attach a classification head. So this is a classifier that classifies into the whole vocabulary. So what we end up with is we have our whole vocabulary. So there is the word a, there is the word is there is the word cat, there is the word dog, there is the word mom. There are all these words, right, we can actually enumerate all of these words. And because we can enumerate them, we can let the model output a distribution. So maybe it says, well, the word a is, you know, super likely, the word is not so likely the word cat, it appears in the sentence, you know, the observed sentence, so it might be a bit like the word dog, the word mom, not really, and so on. So what we get is a discrete probability distribution. Note that the dimensionality, even though it's sometimes large, so this can be something like 30k, it's still countable, we can still do a classification into 30,000 different classes, especially if we use word pieces, we don't have out of vocabulary, we can actually choose our vocabulary size. Second of all, we can actually represent our uncertainty. Notice that not all the weight here is on the word a, especially if there is also like your which is also possible, but in this case, not correct, the model can express the fact that it thinks that both words could fit into this thing. So if there is this is zero, this is one over here, probably adds up to more than one. In any case, you can see that the top prediction here is only maybe point four, in probability, so the model can represent uncertainty by simply not allocating all of the classification mask to a single thing. So these two things are solved pretty well. Dimensionality is in a high but not too high, and uncertainty can be represented. Now what about computer vision? And that's where they they have this diagram right here, that sort of is supposed to sort of detail what I what I just said in that NLP tasks, these masked prediction tasks, they have they're rather discrete, okay. They have relatively less, well, they're relatively low dimensional, and have less uncertainty. I'm not really sure if the less uncertainty and they have a better I would say they have a better way of representing uncertainty. And then the fact that they have less uncertainty simply comes from the fact that they are more discrete and low dimensional than other problems. So what do I mean by more discrete, lower dimensional and so on, if you look at vision problems, if you think, what do I need to do to predict a video, right. And let's, let's even go, let's even go simpler than that. Let's take a common task in self supervised learning. So I have an image, the images of a cat, let's say, like I know you're surprised. Ears, eyes, let's that is a cruel cat. Okay, so that is one cat, okay. And I mask away part of an image. So I simply cut out this part here. And my model is supposed to reconstruct the part of the image that I just created. So my model is supposed to reconstruct the part from the known parts. That is a self supervised task is exactly in the category of what they suggest here. Now, can we do the same thing as we do in the NLP thing? Remember, in the NLP thing, we made a model that output a classifier over all the possible things that could go in there. Like, no, we cannot. Well, first of all, how many things are there that can go there? Well, infinity, because this is a continuous problem, right. So if I give you a patch, and you know, the here is a part of the head, this and maybe the whiskers, you can see this, it could technically be right, but it could also be that the cat here, because we don't know, right, an equally likely continuation is that the cat is like holding a wine glass right here that is filled with wine. We don't we don't know, right. An equally likely continuation, like there are infinitely many likely continuations for this for filling in. And that's a bit the same as in the NLP task, because there are multiple words that could fill that slot, but way less. Plus, we can we will never be able to enumerate all of the different patches that could and could not go in there, right. We can't we can't even enumerate all the ones that could go in there. And it's completely impossible to list all the ones that are both possible and non possible. So we could build a classifier on top of it. So we simply cannot, like this, this we cannot build a classifier, this is not possible in the vision case. So it is too high dimensional. And also, there is no good way of representing uncertain, there's much more. And now I get it. Well, well, I think the dimensionality has a direct effect on the uncertainty. So what people do, or what people can do is they say, let's not build a classifier, let's actually just predict what is there, right, because I can do a neural network like a CNN, something like this layer, layer, layer, layer, layer, layer, layer, like a unit with some skip connections right here, right. And I can actually try to train my model to just reconstruct that part, right? Like, how hard is this? Like we said at the beginning, instead of this is a this is a very terrible cut, but you know, the model is not trained super well. So it only has one eye. The model isn't helped me. The model isn't trained super well. So I can just program or I can train my model to reconstruct. But now, all my model can do is it can output one thing, it can only output one completion. If I don't have a classifier, where I can represent my probability distribution, I can only output a thing. And since there are many, I have no way of representing many. And I can't really output the mean of them, because the mean of these two pictures is going to be not a real picture, because it's like a half transparent wine glass, right. So that's certainly invalid. So you can, as you can see, the fact that we can't build an explicit classifier means we have to predict directly. But then since we can't predict directly, we have no way of representing uncertainty. So I wouldn't call this more uncertainty, I would call it that computer vision has less of a possibility to represent uncertainty directly. I think that's something they say in the text, actually. So that is the problem with computer vision. Now, what do people do to tackle this? And the answer is going to be contrastive learning. But they go there in a bit. First, they make an excursion to energy based models. So here they say a unified view of self supervised methods, even though I thought this hiding part of the input was already the unified view, but in any case, they say there is a way to think about self supervised learning within the unified framework of an energy based model. Now, short pre thing here from me, I know this energy based model, and you will see what it is in a second. I think that is just kind of a, it doesn't tell me anything like the term energy based model, it can just be applied to anything like any problem like energy based model simply means loss function, right? But yeah, let's, so an energy based model is a trainable system that given two inputs x and y tells us how incompatible they are with each other. For example, x could be a short video clip and y another proposed video clip. The machine would tell us to what extent y is a good continuation for x. To indicate the incompatibility between x and y, the machine produces a single number called an energy. If the energy is low, x and y are deemed compatible. If it is high, they are deemed incompatible. So this is kind of a physics approach to the thing. So if you again, think of this as your video, and you want to predict the future from the past, what an energy based model would do is it would, it had two components. So the main component would be this energy function right here, and the energy function would tell you how well x and y fit together. So now it's, you can actually put both frameworks in this. So if you predict y, right, if you if your model actually predicts the continuation, then your energy function could simply be something like the L2 loss between the actual true, between the true continuation in your data and the one you predicted. However, if you do, if you could, if you could do the classifier approach, and you could actually list all the video sequences that are possible, then your energy function could be something like could be the classifier loss. But you know, again, so if you think about this, then anything is an energy based model, right, a classification problem is an energy based model. Because if I have an image here of my trusty cat, and I have the label cat, right, my f of x and y is simply if I define my energy function as my cross entropy between, you know, as my classification cross entropy of cat, given all the other labels, that is an energy based model, right. So I don't see why we need to frame this as energy based model if we can simply say loss function like beats me. But in any case, I guess the sort of physics approach here is just another way of thinking about it. But I dare anyone to bring me a thing that is not an energy based model in machine learning. I might have just summoned some I might have just summoned some demons here. Okay, so they go back and say, well, look, the the an early example of this are these Siamese networks that have recently become fashionable again. And that is where you do the following. So now we switch away from predicting this hidden part from the unhidden part. And we go more into the predicting a hidden property part. So here you can see you have two different crops of an image. And this is the most popular self supervised task for computer vision, you have an image of something like the sun. And you crop it twice in different locations. So you crop it here, you crop it here. And what your what your model needs to do is it needs to figure out that these two patches come from the same image. If it can do that, then it will have learned some good representation. And if you regularize correctly, then it learns an even better representation. So here it needs to figure out that these two chess looking things actually come from a similar picture. And the hope is so okay, what do they do, they feed each of the ones through the same encoder, right, and the W in the middle means that the weights of the encoder are shared. So you obtain two hidden representation. And then this here, this could simply be, you know, like the inner product between H and H prime, or like the negative inner product, if you want to actually make it as an energy. So, or maybe one over the inner product, however, you formulate it. But what this will do is it will tell the model, if two things come from the same image, you better have representations for them, these H that agree with each other, which means that they are close in the inner product space, they have a high inner product. If this is the case, right, then it means that you have learned something useful about the world, because you can tell me when two crops are from the same image. And the hope is that the model will learn that, oh, wait, if you know, if the model wants to do this, well, it needs to learn, aha, there are chess pieces in here, it can simply compare, maybe it can compare these pixels, okay, that will work. But if you compare this pixel and this pixel, that won't work. So it needs to learn something more sophisticated actually needs to learn that are chess pieces in here, if it wants to do a good job and differentiate representations from those with crops from different images, like if we have a crop from the sun right here, what we want is that the inner product between these two is high, but the inner product between any with anyone with the part of the sun picture is low. Okay, so we train it like this. And this is exactly where the contrastive learning goes. So these Siamese networks, they look fun. But without the part I just outlined without the contrastive part, they fall into danger of collapse. So if I only ever input two crops from the same image and say, please make the hidden representation such that the inner product is high. What I what I will end up with is a model that simply collapses and always gives me the same hidden representation for every single image, because that satisfies the constraint, right. And that's what they point out here. This phenomenon, like the network could happily ignore their inputs and always produce identical output embeddings. This phenomenon is called a collapse. When a collapse occurs, the energy is not higher for non matching x and y than it is for matching x and y. So they say the the easy part is the easy part is that when vectors, when x and y are slightly different versions of the same image, the system is trained to produce a low energy. Okay, so now that's easy. The difficult part is to train the model so that it produces a high energy for images that are different. Now what counts as different and non different here again is much of human supervision. So this task of cropping that has fundamental assumptions that you know, for example, in one image, there is largely one object or one topic that we're interested in, right, if this is a map, and we actually want to differentiate the places, it's a pretty bad task to do this cropping. Also, what people do a lot is color jittering color, inversions, brightness modifications, all of these is human intuition, human supervision that the color shouldn't matter, the brightness shouldn't matter, and so on. And the more things you give to the model like this, the more you bake in your assumptions. So again, we we move from supervised learning, where we tell the model, here's the correct label, here's the correct label, to self supervised learning, where we tell the model sort of we tell the model what what kind of transformations should and shouldn't matter. And the model has to figure out itself, how to create the representation such that these constraints hold. So now they go into the solutions for collapse, they say there are avoid there are two techniques to avoid collapse, one is contrastive methods, and the other one is regularization methods. So contrastive methods, they actually have this graphic right here. As you can see, so their point is that if we talk about energy based models, we want energy to be low on x y pairs that we as humans define match. So this could be because we crop them from the same image, or we actually it is the same image, but slightly distorted in different ways. So we as humans, we simply determine these two things match, or it is the uncorrupted and the corrupted version of the same sentence and birds training. And these here are represented by the blue points. So we want the energy to go down on the blue points, but we want the energy to go up everywhere else, right everywhere where it doesn't match, we want the energy to be high. Now, what could we do, we could simply, you know, push down here, because we can create lots of examples, right, we can create lots of samples, where x and y match, because we don't need labels anymore, we can create the labels ourselves. So we can create lots and lots and lots and lots of image crop pairs that match, right. So the pushing down isn't the problem, the pushing up is the problem. Now, if you see this graphic, you might say, why don't I just, you know, enumerate, kind of go through here, and I push up on all the green places, right, I push just up and up here and up here, up here. The problem with that is that the higher dimensionality, the less possible that is. And here is where the graphic tricks you into thinking that it's a good idea when it's actually not like, you will not be able to enumerate all the green dots, even around the blue dots, like it's just not possible because the dimensionality is so high. If you have a dot in 512 dimensions, that is a vector with 512 entries, right 512 entries. Now, you would need to, let's say, if you were just to look around a data point, you would need to jiggle the first dimension, maybe to the left and to the right, and the second dimension, and the third dimension, and you need to do this all combinatorically. So you would need to do this one to the right, this one to the left, this one to the left, and then this one to the right, this one to the right, this one to the left, and so on. You need to do it in different magnitudes here. Sometimes you need to keep them constant. It's just not possible. So what do people do in these contrastive methods? They say, well, we can't push up on all the points. But what we can do is we can sample. And that's why you see the green things epileptically jumping around in that we can sample the green points instead of enumerating them, we simply sample them, and that's where we push up. And that is a difficult task to do. So it is difficult to come up with examples with sense, with meaningful negative examples, because so what people do in this task right here is what I just said. Well, here are two images that fit, right? This is a blue point. And here are two images that don't fit. So this is a green point. However, as we already saw, there are many, many more green points than blue points. And most green points are really far apart from the blue points. If I just take any image right here, it might be way too easy for the model. So the best thing would be to give the model sort of a curriculum, or at least what we call hard negatives. But that is computationally very expensive, because we have to go search for hard negatives, like images that are close, but not, but still different, would be best for the model. But we don't have that all we can do is sort of randomly sample crops from other images, because we don't have labels, we have no clue if you know, two images are the same or not, we just scrape them from Instagram, come on. All looks all the same to me. So the problem here is that if we just do it randomly, then most of the green points will actually be pretty far apart. And that means we just have to train for a long, long time. So contrastive methods, they work in computer vision right now. However, coming up with incompatible pairs that will shape the energy in a suitable way is challenging and expensive computationally, at least in vision systems, right? The method used to train NLP systems by maxing or substituting some input words belongs to the category of contrastive methods, but they don't use joint embedding architecture. Instead, they use a predictive architecture. Okay, so that's saying that if you look at what, you know, BERT does with this masking one thing out, and then classify directly, that is technically contrastive, because what you do in a classification model is you push up, like these are all the possibilities, and what you do during training is you push up on the class that is correct, and you push down on the classes that are not correct. That's what the cross entropy loss does. So technically, it is a contrastive method. However, you do this in this sort of predictive framework, you don't do it via this method of having shared embeddings. And that's because you can actually enumerate all the things that you could do. So with the contrastive methods for vision, we can do the same thing now. What we can do here, if you think about this problem again, of we cannot possibly enumerate all possible pictures that go here, but what we can do is we can enumerate a couple, and then simply classify which ones are good and which ones aren't. And that's exactly what these contrastive methods do that we just looked at, right? So we sample the green points, we sample also the blue points, and then we simply either classify between the green and the blue points, or, you know, we make their inner product go high at the end, these are not so much different objectives, whether or not it's really a classification loss or not. The point here is that first they obtain shared embeddings, they obtain some sort of embedding right here, and then they make the embedding agree or not agree. So they quickly go into the class, and then so they quickly go into what BERT is. BERT is usually called a denoising autoencoder. So what you have is you start off with a data point with the uncorrupted version, you corrupt it, and that's the part where you mask out some parts, you can see this right here, you mask them out. And then you have a prediction for what should go in the blanks. And the loss here is simply the classification loss, this is just your cross entropy loss that goes here. A vast language model, which is an instance of a denoising autoencoder, itself an instance of a contrastive self-supervised learning. However, there is another way, there is another. So here they talked about there are two ways where we in which we can combat this, right? There are two categories, sorry about that, there are two categories. So this is category one is contrastive methods, where we classify some against others, either all of them or a sample of them. However, the other one is what they call this this predictive architecture. Oh, sorry. No. Predictive architecture of this type can produce only a single prediction for a given output. Since the model must be able to predict multiple possible outcomes, the prediction is not a single set of words, but a series of scores for every word in the vocabulary for each missing word location. So that's still BERT. BERT, which can give you uncertainty by simply telling how likely each word is. And here they say we cannot use this trick for images because we cannot enumerate all possible images. Is there a solution for this problem? The short answer is no. There are interesting ideas in this direction, but they've not yet led to results that are as good as joint embedding architectures. One interesting avenue is latent variable predictive architectures. So that what you see down here, this is a latent variable predictive architectures. So it goes down, this is the description that goes down here, latent variable predictive models contain an extra input variable Z. It is called latent because its value is never observed with a properly trained model as latent variable varies over a given set. The output prediction varies over the set of plausible predictions compatible with the input X and they name generative adversarial models here. So this is a bit confusing, but so up here is the loss. This is a loss. And here you have this new variable Z and this Z comes from a domain right here where it can move around. And by moving around Z, you actually move around the output Y right here. So they represent this as this curvy boy here. So maybe Z is here and that represents a point here on the manifold. But as you move Z like to the right, then you move along this manifold right here. So this is a way in which a model can for a given X, you can see here X is mixed with Z, X is first you obtain a representation for X, then it's mixed with Z. For a given X, you can produce many different outputs by simply varying Z. And if you sample a bunch of these Z and then calculate sort of an average loss over them maybe or just a loss per sample, then eventually, you'll train your model to not only you know, handle this one prediction, but handle many different predictions. Now, you might know GANs. So GANs are simply when you do not have so when you say again, simply cuts off this here. So GANs only have the Z variable. And then they produce this set of outputs. And the this is the discriminator right here that decides between the real image and the produced image, of course. The last thing here is that this R is the regularization on Z. I believe they never I don't think they ever pointed out what the R is. But they also don't think they ever point out what this regularization is they talk up here about. So I'm going to assume that refers to the R right here. And now it gets a little bit it gets a little bit confusing. So they say down here. They say first of all, they say non-contrastive methods applied to joint embedding architectures is possibly the hottest topic in self supervised learning for vision at the moment. Domain is still largely unexplored, but it seems very promising. So non-contrastive methods, which means they don't need negative samples, but they still do joint embedding. So they take two different things that come like from the same image, they jointly embed them, but they don't have negative samples, like the original Siamese networks, but you need to avoid collapse. And these models right here, for example, there's Bjoel, which I have made a video about, you can check that out. I think they argue that batch norm for some reason avoids this collapse if they build in batch norm, but also there are other architectures, right, but they all they they are in the beginning. And so they say rather than doing non-contrastive joint embedding, maybe we should do essentially what BERT is doing, but for vision. So perhaps a better alternative in the long run will be to devise non-contrastive methods with latent variable predictive models. So predictive is, you know, we predict the output directly like BERT does, but we can't envision because we can't enumerate all the possibilities, so we can't represent uncertainty. So what we should do is we should do this latent variable thing where we deterministically predict, right, this is deterministic, we deterministically predict the embedding, and then from the embedding, we construct fuzzily, like with the by sampling z, like we sample z from this ground distribution, we construct this entire set of outputs, and that will represent our possibilities, like our uncertainty, that will represent all the things that could fill the gap that we're trying to predict. So they say that may be the way forward. And then I say something confusing, the main obstacle is that they require a way to minimize the capacity of the latent variable, the volume of the set over which the latent variable can vary limits the volume of the outputs that take a low energy, but minimizing this volume one automatically shapes the energy in the right way, which sort of means that, yes, if I have to limit this capacity of this latent variable, right, because otherwise the latent variable could contain all the information, like in a GAN, the latent variable contains all the information, and it's only actually limited by the by the generator, right, by what the generators weights are. So the latent variable contains all of the information, so technically, a GAN, something like a style GAN could happily ignore the input right here. And it could still produce pretty good images. And you have to do tricks in order to make the model actually pay attention to the input and not only pay attention to the latent variable. So you can regularize, you can constrain this latent variable such that the model pays attention to the input. And why do we want the model to pay attention to the input? Because the entire reason is that we want to use this embedding right here, then for future supervised learning, like this embedding, that's actually the goal of self supervised learning. There you see why GANs probably cannot give us super good embeddings, because GANs just have the part on the right. Okay. But something like an info GAN, or like, as we said, like a style GAN that takes an input could technically already give us is technically a model about something like this. So here they say, so so that's, you know, you limit the the capacity of the latent variable, but then they go on and say, a successful example of such a method is the variational autoencoder, the VAE, in which the latent variable is made fuzzy, which limits its capacity. Okay, and the here is where I, I was I was confused, but the VAE have not yet been shown to produce good representations for downstream visual tasks. Okay. Another successful example is sparse modeling, but its use has been limited to simple architectures. No perfect recipe seems to exist to limit the capacity of the latent variables. Now, I get that limiting capacity. However, in a variational encoder, it is not exactly the latent variable that is made fuzzy. It is actually the embedding, right? If you think here, in a variational autoencoder, what you do is you have whatever your image, and then you have your encoder, and then you predict in the latent space, you predict Gaussian distributions, like you predict the mean and you predict the standard deviation of a Gaussian distribution, and then you sample from that Gaussian, that is a horrible Gaussian, you sample from that Gaussian distribution, and due to the reparameterization trick, you can actually simply sample from a standard Gaussian down here, like that is at zero and has standard deviation one, and that will be your z variable, and then you can simply do z times, sorry, z times sigma plus mu, and that will be sampling essentially from the, that will be sampling from that respective Gaussian. So in this way, the variable z is not made fuzzy. What is actually made fuzzy is this here, and this here comes from h, right? This is h, this is the embedding, gives rise to these mu and sigma, and these are made fuzzy because they're multiplied by a stochastic variable. So I'm a little bit confused about this paragraph right here, because a VAE, I don't think that it limits the capacity of the latent variable, and it fuzzes the latent variable, but I might be wrong, or they actually mean something else by latent variable, they actually mean the embedding here, in that case, it might make sense again. However, then it doesn't make super much sense to limit its capacity. And I've also looked at this sparse model, in which simply seems to be kind of sparse encoding of images, it's a really old paper from 1969, but sorry, 96, 96, not that old. Yeah, but okay, I'm simply going to interpret this as, in order to obtain a meaningful representation h down here, we need to limit the capacity of the latent variable right here, because otherwise, the model will simply ignore the input and not build a good representation for it. So they argue that an architecture like this, an architecture like a VAE, like an Infogan, or something like this, could potentially be the next step, if we can make it work. The challenge in the next few of the next few years may be to devise non-contrastive methods for latent variable energy based model that successfully produce good representation of image, video, speech and other signals and yield top performance in downstream supervised tasks without requiring large amounts of labeled data. So in German, we have a saying that what they want is which means the egg laying wool milk pig. So he can do anything and everything and it costs nothing. So that's what they mean. Again, some of these things like energy based model, like anything is an energy based model, I just don't find this to be super discriminating in its meaning of what that is. Lastly, they talk a bit about their new model called a seer, which is a self supervised model, but it's just like a giant confinet trained on a billion images. Oh, but you know, they open sourced it. Thank you. You open source the code. So I can totally train my own billion parameter on a on a billion random public Instagram images because my Raspberry Pi just technically has that capacity. So thanks. But you know, no, but I'm joking a little bit, at least better than OpenAI. And at the end, they go into how they use other ways of self supervised learning at Facebook. All right, that was my overview over this article. I hope you got at least something from it as a high level overview, they first say self supervised learning is maybe the way to get this common sense into AI systems. Then they go into what is self supervised learning, they define it first as predicting hidden parts from unhidden parts. And later, they say it can be viewed as an energy based model that they point out that there's a crucial distinction between tasks like language and vision because vision is much more high dimensional gives you much less of a way to represent uncertainty. Then they go on and say, well, the contrastive methods, they're not going to be very useful, but the contrastive methods handle part of that, they handle this not, they handle this part of the dimensionality that you can enumerate all of the possible things. However, they are prone to collapse. Sorry, no, the Siamese networks are prone to collapse, the contrastive methods fix that. However, because you have to sample from such a high dimensional space, and that is really hard, it takes a lot of data. And what we could do is we could do these predictive models that directly classify the output, or directly predict the output, right, you predict the missing frame, you predict the missing word. But we do it in this way, where you not only do you predict a single thing, but you predict an entire set by means of these latent variable predictive models. And that they say is maybe the way forward, even though it doesn't work too well yet, like VAEs work. But the problem is, they don't have this ability to generate good representations for supervised learning, that just doesn't work too well yet. Alright, that was it. If you liked it, leave a like, subscribe, share, doubt, tell me what you think in the comments, and bye bye.
 Hello there. Today we're looking at Self-Supervised Learning, the Dark Matter of Intelligence.
 This was written by Jan LeCun and Ishan Misra of Facebook AI Research. And it is not a paper,
 it is more a blog post shared on the Facebook AI blog. And it outlines the current state
 of self-supervised learning, what it is and what it can do, why the authors think it is
 important. It goes over things like BERT, goes over things like Contrastive Learning, energy-based
 models, GANs and so on. And at the end it gives a bunch of recommendations for the way to go
 forward. On a high level the main recommendation is that we should build latent variable prediction
 models that are not trained contrastively. And we'll go through all of what this means in this
 article. So we'll go through the article. I'll switch over to here where it's a bit of a more
 legible format. And as always, if you like content like this, if you enjoy it, share it out. Don't
 hesitate to tell a friend about it. All right, let's do it. They say in recent years the AI
 field has made tremendous progress in developing AI systems that can learn from massive amounts of
 carefully labeled data. So the keywords here are massive amounts. Yes, we got that. But carefully
 labeled data. Of course, we all know that supervised learning has worked very well if you have enough
 labeled data. And that's exactly the problem. In order to push machine learning to more,
 to higher abilities, it seems like what we need is first of all bigger architectures, which we can
 do by just building bigger computers. But we also need more data. The problem here is that we need
 orders of magnitude more data and labeling that data is going to be very, very expensive. And
 therefore, we're looking for methods that can do without labeled data, that can learn most of what
 they learn from non labeled data, and then apply that to a little bit of labeled data in order to
 learn a task. But this is not the only thing. So the need the expansiveness of labeling is not the
 only thing that they criticize here. They say this paradigm of supervised learning has a proven track
 record for training specialist models that perform extremely well on the tasks they were trained to
 do. So this is another criticism right here. Namely, that if we train something in a supervised
 fashion with labels, it will become or it might become very good, but it will be very good at
 that particular task. And it won't be super good at other tasks, such as, you know, tasks that are
 tasks that are relatively neighboring to the field that we're concerned about. They go on they say
 that supervised learning is a bottleneck for building more intelligent generalist models that
 can do multiple tasks and acquire new skills without massive amounts of labeled data. This is
 into the direction of Francois Chollet, who defines intelligence as the efficiency with which you
 transform new data into new skills. And this is reflected here in this article by Jan Lecoe. And
 I'm sorry, Ishan, but Jan Lecoe just has the big name. And unfortunately, you're a bit in his shadow
 here. But I'm fairly confident these that Jan Lecoe is not just on this for the name, because
 the arguments in this article he has raised in many talks that I've seen of him in the past few
 years. So it is it is really kind of a condensing of all of these talks in this here. But back to
 the paper, this acquiring new skills without massive amounts of labeled data. They say that
 has to be our goal, because it is impossible to label everything in the world. And there are also
 some tasks where there is not enough labeled data, like translation systems for low resource languages.
 So they make two observations right here. First of all, they say, look,
 here, for example, if we show just a few drawings of cows to small children, they'll eventually be
 able to recognize any cow they see. By contrast, AI systems trained with supervised learning
 require many examples of carmages and might still fail to classify cows in unusual situations,
 such as lying on a beach. What are you doing, silly cow? Don't lie on a beach. So this is another
 point, right? These these AI systems, they take so much more data than humans to learn new skills.
 And they ask why the short answer is that humans rely on their previously acquired knowledge of how
 the world works. So they make this, they make this argument here that there is a thing like common
 knowledge about the world or common sense forms the bulk of biological intelligence in both humans
 and animals. Humans are animals. Okay, this common sensibility is taken for granted, but has remained
 an open challenge in AI research. Common sense, they say is the dark matter of artificial intelligence.
 So they point out that you have this common sense that you learn simply by interacting with the
 world. They say as babies, we learn how the world works largely by observations, you form predictive
 models about the world, you learn concepts such as object permanence and gravity. And later in life,
 you you even act in the world. Now they're not going into this acting in the world. But their point
 is that throughout your life, you just observe the world and you build these predictive models. And
 that's how you will learn about how the world works. I'm not entirely sure that things like
 gravity are learned in this way. I think there's some evidence that at least part of it is
 biological or at least you're extremely biologically predetermined to learn about
 things like object permanence and gravity. But the point is taken that there is something built into
 you either from experience or from biology that allows you that is kind of this common sense. And
 that allows you to acquire new tasks with extremely few additional samples because you bring in this
 knowledge about the world. So their core claim here is that we believe that self supervised learning
 is one of the most promising ways to build such background knowledge and approximate a form of
 common sense in AI systems. They say the way we're going to get AI systems to also have this common
 sense knowledge is by doing self supervised learning. Right, so they give some examples of
 self supervised learning. They also contrast it with unsupervised learning, where the difference
 that so they say unsupervised learning is a bit of a misnomer. Learning is never really unsupervised.
 Self supervised learning specifically means that you generate the label out of the data itself.
 So what could that be? You know, for example, in in BERT, the language model, you might have a
 sentence like this is a cat. And this is a sentence from the data set. Now in self supervised learning,
 you would somehow need to come up with an input sample and a label for that input sample, just by
 just using this text, right in a supervised in a supervised data set, you would have some label
 associated with this. And this could be anything depending on what the task is, like, this could
 be labels could be annotations for what kind of words these words are, label could be whether or
 not the sentence is a positive or negative sentence. But in self supervised learning,
 you can do something like this. And here's what BERT does, they cross out a word like this a,
 so this now becomes the input sample x, and the label is going to be whatever was missing here.
 So the label will be the word a. Now, the task of the machine learning system is given x, figure
 out what is y. Okay, so figure out that at this particular place in the sentence, there should be
 the word a. Now BERT does a bit more sophisticated things like it also replaces tokens and so on.
 But ultimately, what you want is for any for any corrupted input to for the system to output the
 uncorrupted output. And thereby, the system will learn about the world, it will maybe not about
 the world, but it will learn about language. If it wants to do this task correctly, it needs to
 learn that if you have a this is construction, there should probably be some kind of specifier
 for what comes next right here. And then cat is some sort of an object or animal. So given all of
 this evidence, you only have very few possibilities like a or my or this is a one this is two cat.
 No, this is your cat. Something like this, but all the other words in the language cannot be. So they
 formulate self supervised learning as obtaining supervisory signals from the data itself. That's
 why it's not unsupervised. It is self supervised because you create the label from the data. And
 the important part here is and I think that's often neglected in the self supervised things is that
 the way you create the label from the data that is human specified, right, this this step right here,
 that needs I can I draw a light bulb. That needs a human idea, like how could we create a label and
 an input data point given a data point. So we shift the burden of the human from labeling the data
 explicitly to simply saying to simply constructing the method of how to obtain labels from data.
 This is still building in substantial human bias, but it is much more scalable. If I have one method
 to create labels, I can apply it to an entire data set. Whereas if I create labels myself, I can
 go through every single data point. But it's not unsupervised because the supervision is in the
 process that creates the label. So they say leverage the underlying structure of the data.
 The general technique of self supervised learning is to predict any unobserved or hidden part or
 property of the input from any observed or unhidden part of the input. So the general recipe or one,
 I would say one general recipe because it's not the general recipe, even though they claim it here,
 I would say one general recipe is that if you have an input, you just hide part of it.
 And then you have the model predict that hidden part. They give a bunch of examples here. This is
 quite a cryptic drawing, I think. So these are three examples of what you could do if you have
 data and this yet time or space, I would claim it's easiest if you think of this as a video
 sequence. So this is a video sequence and the frames are all they're stacked like this frame,
 frame, frame. Okay, and it goes up until here. So what you're going to do, what you can do option
 one is you simply take the past, you define a time point t right here, and you take the past,
 and that's the observed part. And you take the future, which you have in your data set,
 but you don't show it to the model. So the model is supposed to predict the future from the past.
 This in video, you can understand it. This is also what for example, GP, the GPT model,
 like GPT three does exactly this, it takes in a past words so far, and it predicts the next word
 or the next few words. The second part is, you don't have to necessarily predict the future,
 you can also just leave away a bunch of frames in the middle, somewhere at different parts. Now,
 what the model has to do is has to reason about a part, let's say, a part of the model,
 let's say this part right here, it has to reason, given the surrounding evidence. So it takes all
 the evidence into account. And it reasons what kind of frames could have been left out there.
 In again, in video in NLP land, this would be something like BERT. So BERT is trained in
 this objective, as a as a masked language model. And then the last one is really quite specific,
 I think, to something like video, maybe also different modalities, but doesn't apply super
 well to NLP. Maybe you could though. But this is where if you imagine this being your frames,
 you not only do you leave away these frames right here, but you also would leave away
 part of the frames that you observe. So in these frames, you would simply only observe the bottom
 right thing right here, and you would not observe everything else. So not only do you have to reason
 about what goes into the missing slot, but you also have to reason about what goes into the parts of
 the frames you don't observe. And as you can see here, these can be different parts throughout the
 video. So I think it's just it just makes a point that this can be quite general. So in general,
 you just hide parts of your input, and you re predict them from a model. And that means the
 model, you know, if it can, for example, if it can predict the future of a video from the past,
 given, you know, certain input, it will necessarily have to learn something about how the world works,
 or at least about how the world looks through a video lens. Right? If it does this task, well,
 it has a lot of prop captured a lot of properties of how the world looks in in video. And that is
 much more rich information than simply giving a label to train on. And the hope is that by
 learning all of these different things that are necessary to predict the future well from the
 past, the model will learn such a useful representation that adapting this model to solve any labeled
 supervised task is going to be really quick because it also it already has very, very good
 representation of the data. And the common thing here is that, okay, in order to predict the order
 from the past to the future, there can be there can be numerous features that are helpful, right,
 there are all of these features that are very helpful to predict the future from the past.
 Now, if I have any supervised task, right, I have, for example, the past, and then I want to determine
 if I don't know what what can we determine from a video, if this is a happy video, right, is this
 a happy video or not? The core assumption here is that since you know, predicting the future from
 the past has sort of the structure of the world built in and since our supervised task is not
 task is probably a function of a subset of that structure, like whether or not it's a happy video
 probably depends on whether or not in the future, someone will fall off a cliff or not, right. So
 a subset of these things in combination are going to be relevant for that task. So they can be
 adapted. Since the representation is already there, they can be adapted pretty rapidly,
 while the ones that are not important can maybe be overwritten and relearned to get some additional
 signal from the from the input that was not learned in the in the self supervised training.
 So the goal is, again, by learning to predict the hidden inputs from the non hidden inputs,
 you learn about the structure of the data. By learning about the structure of the data,
 you get useful representations. And by having useful representations, you can adapt very quickly
 to new tasks. That's the that's the sort of argument here. So why don't we do this all the
 time, every time everywhere, they go into self supervised learning for language versus vision.
 So in language, this is uber duper successful, while in vision, I think in vision, it's fairly
 successful too. But there is a challenge when you think about language versus vision, specifically
 in terms of this hiding, hiding parts of the inputs and then reconstructing them. So there are two
 there are two different things that we need to consider here. The first thing the first problem
 is dimensionality. Dimensionality. And the second thing we need to consider is uncertainty.
 Okay, so dimensionality in NLP is what's our dimensionality, if you think of this problem,
 again, this is a cat, this thing right here. How do we do it in BERT, like we mask out the word,
 and then we feed this sentence, we feed it through a big neural network that is BERT.
 And then at the end, at this position, we attach a classification head. So this is a classifier
 that classifies into the whole vocabulary. So what we end up with is we have our whole vocabulary.
 So there is the word a, there is the word is there is the word cat, there is the word dog,
 there is the word mom. There are all these words, right, we can actually enumerate all of these
 words. And because we can enumerate them, we can let the model output a distribution. So maybe it
 says, well, the word a is, you know, super likely, the word is not so likely the word cat,
 it appears in the sentence, you know, the observed sentence, so it might be a bit like the word dog,
 the word mom, not really, and so on. So what we get is a discrete probability distribution.
 Note that the dimensionality, even though it's sometimes large, so this can be something like
 30k, it's still countable, we can still do a classification into 30,000 different classes,
 especially if we use word pieces, we don't have out of vocabulary, we can actually choose our
 vocabulary size. Second of all, we can actually represent our uncertainty. Notice that not all
 the weight here is on the word a, especially if there is also like your which is also possible,
 but in this case, not correct, the model can express the fact that it thinks that both words
 could fit into this thing. So if there is this is zero, this is one over here, probably adds up to
 more than one. In any case, you can see that the top prediction here is only maybe point four,
 in probability, so the model can represent uncertainty by simply not allocating all of
 the classification mask to a single thing. So these two things are solved pretty well.
 Dimensionality is in a high but not too high, and uncertainty can be represented.
 Now what about computer vision? And that's where they they have this diagram right here,
 that sort of is supposed to sort of detail what I what I just said in that NLP tasks,
 these masked prediction tasks, they have they're rather discrete, okay.
 They have relatively less, well, they're relatively low dimensional, and have less uncertainty.
 I'm not really sure if the less uncertainty and they have a better I would say they have a better
 way of representing uncertainty. And then the fact that they have less uncertainty simply comes from
 the fact that they are more discrete and low dimensional than other problems. So what do I
 mean by more discrete, lower dimensional and so on, if you look at vision problems, if you think,
 what do I need to do to predict a video, right. And let's, let's even go, let's even go simpler
 than that. Let's take a common task in self supervised learning. So I have an image,
 the images of a cat, let's say, like I know you're surprised. Ears, eyes, let's that is a cruel cat.
 Okay, so that is one cat, okay. And I mask away part of an image. So I simply cut out this part
 here. And my model is supposed to reconstruct the part of the image that I just created.
 So my model is supposed to reconstruct the part from the known parts. That is a self supervised
 task is exactly in the category of what they suggest here. Now, can we do the same thing as
 we do in the NLP thing? Remember, in the NLP thing, we made a model that output a classifier
 over all the possible things that could go in there. Like, no, we cannot. Well, first of all,
 how many things are there that can go there? Well, infinity, because this is a continuous problem,
 right. So if I give you a patch, and you know, the here is a part of the head, this and maybe
 the whiskers, you can see this, it could technically be right, but it could also be
 that the cat here, because we don't know, right, an equally likely continuation is that the cat is
 like holding a wine glass right here that is filled with wine. We don't we don't know, right.
 An equally likely continuation, like there are infinitely many likely continuations for this for
 filling in. And that's a bit the same as in the NLP task, because there are multiple words that
 could fill that slot, but way less. Plus, we can we will never be able to enumerate all of the
 different patches that could and could not go in there, right. We can't we can't even enumerate
 all the ones that could go in there. And it's completely impossible to list all the ones that
 are both possible and non possible. So we could build a classifier on top of it. So we simply
 cannot, like this, this we cannot build a classifier, this is not possible in the vision
 case. So it is too high dimensional. And also, there is no good way of representing uncertain,
 there's much more. And now I get it. Well, well, I think the dimensionality has a direct effect on
 the uncertainty. So what people do, or what people can do is they say, let's not build a classifier,
 let's actually just predict what is there, right, because I can do a neural network like a CNN,
 something like this layer, layer, layer, layer, layer, layer, layer, like a unit with some skip
 connections right here, right. And I can actually try to train my model to just reconstruct that
 part, right? Like, how hard is this? Like we said at the beginning, instead of this is a this is a
 very terrible cut, but you know, the model is not trained super well. So it only has one eye.
 The model isn't helped me. The model isn't trained super well. So I can just program or I can train
 my model to reconstruct. But now, all my model can do is it can output one thing, it can only output
 one completion. If I don't have a classifier, where I can represent my probability distribution,
 I can only output a thing. And since there are many, I have no way of representing many. And
 I can't really output the mean of them, because the mean of these two pictures is going to be not
 a real picture, because it's like a half transparent wine glass, right. So that's certainly invalid.
 So you can, as you can see, the fact that we can't build an explicit classifier means we have to
 predict directly. But then since we can't predict directly, we have no way of representing uncertainty.
 So I wouldn't call this more uncertainty, I would call it that computer vision has less
 of a possibility to represent uncertainty directly. I think that's something they say in the text,
 actually. So that is the problem with computer vision. Now, what do people do to tackle this?
 And the answer is going to be contrastive learning. But they go there in a bit. First,
 they make an excursion to energy based models. So here they say a unified view of self supervised
 methods, even though I thought this hiding part of the input was already the unified view, but in any
 case, they say there is a way to think about self supervised learning within the unified framework
 of an energy based model. Now, short pre thing here from me, I know this energy based model,
 and you will see what it is in a second. I think that is just kind of a, it doesn't tell me anything
 like the term energy based model, it can just be applied to anything like any problem like energy
 based model simply means loss function, right? But yeah, let's, so an energy based model is a
 trainable system that given two inputs x and y tells us how incompatible they are with each other.
 For example, x could be a short video clip and y another proposed video clip. The machine would
 tell us to what extent y is a good continuation for x. To indicate the incompatibility between
 x and y, the machine produces a single number called an energy. If the energy is low, x and y
 are deemed compatible. If it is high, they are deemed incompatible. So this is kind of a physics
 approach to the thing. So if you again, think of this as your video, and you want to predict the
 future from the past, what an energy based model would do is it would, it had two components.
 So the main component would be this energy function right here, and the energy function
 would tell you how well x and y fit together. So now it's, you can actually put both frameworks
 in this. So if you predict y, right, if you if your model actually predicts the continuation,
 then your energy function could simply be something like the L2 loss between the actual true,
 between the true continuation in your data and the one you predicted. However, if you do, if you
 could, if you could do the classifier approach, and you could actually list all the video sequences
 that are possible, then your energy function could be something like could be the classifier loss.
 But you know, again, so if you think about this, then anything is an energy based model, right,
 a classification problem is an energy based model. Because if I have an image here of my trusty cat,
 and I have the label cat, right, my f of x and y is simply if I define my energy function as my
 cross entropy between, you know, as my classification cross entropy of cat, given all the other labels,
 that is an energy based model, right. So I don't see why we need to frame this as energy based
 model if we can simply say loss function like beats me. But in any case, I guess the sort of
 physics approach here is just another way of thinking about it. But I dare anyone to bring me
 a thing that is not an energy based model in machine learning. I might have just summoned some
 I might have just summoned some demons here. Okay, so they go back and say, well, look, the the
 an early example of this are these Siamese networks that have recently become fashionable again.
 And that is where you do the following. So now we switch away from predicting this hidden part
 from the unhidden part. And we go more into the predicting a hidden property part. So here you
 can see you have two different crops of an image. And this is the most popular self supervised task
 for computer vision, you have an image of something like the sun. And you crop it twice in different
 locations. So you crop it here, you crop it here. And what your what your model needs to do is it
 needs to figure out that these two patches come from the same image. If it can do that, then
 it will have learned some good representation. And if you regularize correctly, then it learns
 an even better representation. So here it needs to figure out that these two chess looking things
 actually come from a similar picture. And the hope is so okay, what do they do, they feed each of the
 ones through the same encoder, right, and the W in the middle means that the weights of the encoder
 are shared. So you obtain two hidden representation. And then this here, this could simply be,
 you know, like the inner product between H and H prime, or like the negative inner product,
 if you want to actually make it as an energy. So, or maybe one over the inner product, however,
 you formulate it. But what this will do is it will tell the model, if two things come from the same
 image, you better have representations for them, these H that agree with each other, which means
 that they are close in the inner product space, they have a high inner product. If this is the case,
 right, then it means that you have learned something useful about the world, because you can
 tell me when two crops are from the same image. And the hope is that the model will learn that,
 oh, wait, if you know, if the model wants to do this, well, it needs to learn, aha, there are
 chess pieces in here, it can simply compare, maybe it can compare these pixels, okay, that will work.
 But if you compare this pixel and this pixel, that won't work. So it needs to learn something
 more sophisticated actually needs to learn that are chess pieces in here, if it wants to do a
 good job and differentiate representations from those with crops from different images, like if
 we have a crop from the sun right here, what we want is that the inner product between these two
 is high, but the inner product between any with anyone with the part of the sun picture is low.
 Okay, so we train it like this. And this is exactly where the contrastive learning goes.
 So these Siamese networks, they look fun. But without the part I just outlined without the
 contrastive part, they fall into danger of collapse. So if I only ever input two crops from the same
 image and say, please make the hidden representation such that the inner product is high.
 What I what I will end up with is a model that simply collapses and always gives me the same
 hidden representation for every single image, because that satisfies the constraint, right.
 And that's what they point out here. This phenomenon, like the network could happily
 ignore their inputs and always produce identical output embeddings. This phenomenon is called a
 collapse. When a collapse occurs, the energy is not higher for non matching x and y than it is for
 matching x and y. So they say the the easy part is the easy part is that when vectors, when x and y
 are slightly different versions of the same image, the system is trained to produce a low energy.
 Okay, so now that's easy. The difficult part is to train the model so that it produces a high energy
 for images that are different. Now what counts as different and non different here again is much of
 human supervision. So this task of cropping that has fundamental assumptions that you know, for
 example, in one image, there is largely one object or one topic that we're interested in, right, if
 this is a map, and we actually want to differentiate the places, it's a pretty bad task to do this
 cropping. Also, what people do a lot is color jittering color, inversions, brightness modifications,
 all of these is human intuition, human supervision that the color shouldn't matter, the brightness
 shouldn't matter, and so on. And the more things you give to the model like this, the more you bake
 in your assumptions. So again, we we move from supervised learning, where we tell the model,
 here's the correct label, here's the correct label, to self supervised learning, where we tell the
 model sort of we tell the model what what kind of transformations should and shouldn't matter.
 And the model has to figure out itself, how to create the representation such that these constraints
 hold. So now they go into the solutions for collapse, they say there are avoid there are two
 techniques to avoid collapse, one is contrastive methods, and the other one is regularization
 methods. So contrastive methods, they actually have this graphic right here. As you can see,
 so their point is that if we talk about energy based models, we want energy to be low
 on x y pairs that we as humans define match. So this could be because we crop them from the same
 image, or we actually it is the same image, but slightly distorted in different ways. So we as
 humans, we simply determine these two things match, or it is the uncorrupted and the corrupted
 version of the same sentence and birds training. And these here are represented by the blue points.
 So we want the energy to go down on the blue points, but we want the energy to go up
 everywhere else, right everywhere where it doesn't match, we want the energy to be high. Now,
 what could we do, we could simply, you know, push down here, because we can create lots of examples,
 right, we can create lots of samples, where x and y match, because we don't need labels anymore,
 we can create the labels ourselves. So we can create lots and lots and lots and lots of image
 crop pairs that match, right. So the pushing down isn't the problem, the pushing up is the problem.
 Now, if you see this graphic, you might say, why don't I just, you know, enumerate, kind of go
 through here, and I push up on all the green places, right, I push just up and up here and up here,
 up here. The problem with that is that the higher dimensionality, the less possible that is. And
 here is where the graphic tricks you into thinking that it's a good idea when it's actually not like,
 you will not be able to enumerate all the green dots, even around the blue dots, like it's just
 not possible because the dimensionality is so high. If you have a dot in 512 dimensions,
 that is a vector with 512 entries, right 512 entries. Now, you would need to, let's say,
 if you were just to look around a data point, you would need to jiggle the first dimension,
 maybe to the left and to the right, and the second dimension, and the third dimension,
 and you need to do this all combinatorically. So you would need to do this one to the right,
 this one to the left, this one to the left, and then this one to the right,
 this one to the right, this one to the left, and so on. You need to do it in different magnitudes
 here. Sometimes you need to keep them constant. It's just not possible. So what do people do
 in these contrastive methods? They say, well, we can't push up on all the points. But what we can
 do is we can sample. And that's why you see the green things epileptically jumping around in that
 we can sample the green points instead of enumerating them, we simply sample them,
 and that's where we push up. And that is a difficult task to do. So it is difficult to come up
 with examples with sense, with meaningful negative examples, because so what people do
 in this task right here is what I just said. Well, here are two images that fit, right? This is a blue
 point. And here are two images that don't fit. So this is a green point. However, as we already saw,
 there are many, many more green points than blue points. And most green points are really far apart
 from the blue points. If I just take any image right here, it might be way too easy for the model.
 So the best thing would be to give the model sort of a curriculum, or at least what we call hard
 negatives. But that is computationally very expensive, because we have to go search for
 hard negatives, like images that are close, but not, but still different, would be best for the
 model. But we don't have that all we can do is sort of randomly sample crops from other images,
 because we don't have labels, we have no clue if you know, two images are the same or not, we just
 scrape them from Instagram, come on. All looks all the same to me. So the problem here is that if we
 just do it randomly, then most of the green points will actually be pretty far apart. And that means
 we just have to train for a long, long time. So contrastive methods, they work in computer vision
 right now. However, coming up with incompatible pairs that will shape the energy in a suitable
 way is challenging and expensive computationally, at least in vision systems, right?
 The method used to train NLP systems by maxing or substituting some input words belongs to the
 category of contrastive methods, but they don't use joint embedding architecture. Instead, they use a
 predictive architecture. Okay, so that's saying that if you look at what, you know, BERT does with
 this masking one thing out, and then classify directly, that is technically contrastive,
 because what you do in a classification model is you push up, like these are all the possibilities,
 and what you do during training is you push up on the class that is correct, and you push down on
 the classes that are not correct. That's what the cross entropy loss does. So technically, it is a
 contrastive method. However, you do this in this sort of predictive framework, you don't do it via
 this method of having shared embeddings. And that's because you can actually enumerate all the things
 that you could do. So with the contrastive methods for vision, we can do the same thing now.
 What we can do here, if you think about this problem again, of we cannot possibly enumerate
 all possible pictures that go here, but what we can do is we can enumerate a couple, and then
 simply classify which ones are good and which ones aren't. And that's exactly what these
 contrastive methods do that we just looked at, right? So we sample the green points, we sample
 also the blue points, and then we simply either classify between the green and the blue points,
 or, you know, we make their inner product go high at the end, these are not so much different
 objectives, whether or not it's really a classification loss or not. The point here is
 that first they obtain shared embeddings, they obtain some sort of embedding right here, and
 then they make the embedding agree or not agree. So they quickly go into the class, and then
 so they quickly go into what BERT is. BERT is usually called a denoising autoencoder. So what
 you have is you start off with a data point with the uncorrupted version, you corrupt it, and that's
 the part where you mask out some parts, you can see this right here, you mask them out. And then
 you have a prediction for what should go in the blanks. And the loss here is simply the
 classification loss, this is just your cross entropy loss that goes here. A vast language model,
 which is an instance of a denoising autoencoder, itself an instance of a contrastive self-supervised
 learning. However, there is another way, there is another. So here they talked about there are two
 ways where we in which we can combat this, right? There are two categories, sorry about that, there
 are two categories. So this is category one is contrastive methods, where we classify some
 against others, either all of them or a sample of them. However, the other one is what they call
 this this predictive architecture. Oh, sorry. No. Predictive architecture of this type can produce
 only a single prediction for a given output. Since the model must be able to predict multiple
 possible outcomes, the prediction is not a single set of words, but a series of scores for every
 word in the vocabulary for each missing word location. So that's still BERT. BERT, which can
 give you uncertainty by simply telling how likely each word is. And here they say we cannot use this
 trick for images because we cannot enumerate all possible images. Is there a solution for this
 problem? The short answer is no. There are interesting ideas in this direction, but they've
 not yet led to results that are as good as joint embedding architectures. One interesting avenue
 is latent variable predictive architectures. So that what you see down here, this is a latent
 variable predictive architectures. So it goes down, this is the description that goes down here,
 latent variable predictive models contain an extra input variable Z. It is called latent because its
 value is never observed with a properly trained model as latent variable varies over a given set.
 The output prediction varies over the set of plausible predictions compatible with the input
 X and they name generative adversarial models here. So this is a bit confusing, but so up here
 is the loss. This is a loss. And here you have this new variable Z and this Z comes from a domain
 right here where it can move around. And by moving around Z, you actually move around the output Y
 right here. So they represent this as this curvy boy here. So maybe Z is here and that represents
 a point here on the manifold. But as you move Z like to the right, then you move along this manifold
 right here. So this is a way in which a model can for a given X, you can see here X is mixed with
 Z, X is first you obtain a representation for X, then it's mixed with Z. For a given X, you can
 produce many different outputs by simply varying Z. And if you sample a bunch of these Z and then
 calculate sort of an average loss over them maybe or just a loss per sample, then eventually,
 you'll train your model to not only you know, handle this one prediction, but handle many
 different predictions. Now, you might know GANs. So GANs are simply when you do not have so when you
 say again, simply cuts off this here. So GANs only have the Z variable. And then they produce this
 set of outputs. And the this is the discriminator right here that decides between the real image
 and the produced image, of course. The last thing here is that this R is the regularization on Z.
 I believe they never I don't think they ever pointed out what the R is. But they also don't
 think they ever point out what this regularization is they talk up here about. So I'm going to assume
 that refers to the R right here. And now it gets a little bit it gets a little bit confusing.
 So they say down here.
 They say first of all, they say non-contrastive methods applied to joint embedding architectures
 is possibly the hottest topic in self supervised learning for vision at the moment. Domain is still
 largely unexplored, but it seems very promising. So non-contrastive methods, which means they don't
 need negative samples, but they still do joint embedding. So they take two different things that
 come like from the same image, they jointly embed them, but they don't have negative samples,
 like the original Siamese networks, but you need to avoid collapse. And these models right here,
 for example, there's Bjoel, which I have made a video about, you can check that out. I think they
 argue that batch norm for some reason avoids this collapse if they build in batch norm, but also
 there are other architectures, right, but they all they they are in the beginning.
 And so they say rather than doing non-contrastive joint embedding, maybe we should do essentially
 what BERT is doing, but for vision. So perhaps a better alternative in the long run will be to
 devise non-contrastive methods with latent variable predictive models. So predictive is,
 you know, we predict the output directly like BERT does, but we can't envision because we can't
 enumerate all the possibilities, so we can't represent uncertainty. So what we should do is
 we should do this latent variable thing where we deterministically predict, right, this is
 deterministic, we deterministically predict the embedding, and then from the embedding, we construct
 fuzzily, like with the by sampling z, like we sample z from this ground distribution,
 we construct this entire set of outputs, and that will represent our possibilities, like our
 uncertainty, that will represent all the things that could fill the gap that we're trying to predict.
 So they say that may be the way forward. And then I say something confusing, the main obstacle is
 that they require a way to minimize the capacity of the latent variable, the volume of the set over
 which the latent variable can vary limits the volume of the outputs that take a low energy,
 but minimizing this volume one automatically shapes the energy in the right way, which sort
 of means that, yes, if I have to limit this capacity of this latent variable, right, because
 otherwise the latent variable could contain all the information, like in a GAN, the latent variable
 contains all the information, and it's only actually limited by the by the generator, right,
 by what the generators weights are. So the latent variable contains all of the information, so
 technically, a GAN, something like a style GAN could happily ignore the input right here. And
 it could still produce pretty good images. And you have to do tricks in order to make the model
 actually pay attention to the input and not only pay attention to the latent variable. So you can
 regularize, you can constrain this latent variable such that the model pays attention to the input.
 And why do we want the model to pay attention to the input? Because the entire reason is that
 we want to use this embedding right here, then for future supervised learning, like this embedding,
 that's actually the goal of self supervised learning. There you see why GANs probably cannot
 give us super good embeddings, because GANs just have the part on the right. Okay. But something
 like an info GAN, or like, as we said, like a style GAN that takes an input could technically
 already give us is technically a model about something like this. So here they say,
 so so that's, you know, you limit the the capacity of the latent variable, but then they go on and
 say, a successful example of such a method is the variational autoencoder, the VAE, in which the
 latent variable is made fuzzy, which limits its capacity. Okay, and the here is where I,
 I was I was confused, but the VAE have not yet been shown to produce good representations for
 downstream visual tasks. Okay. Another successful example is sparse modeling, but its use has been
 limited to simple architectures. No perfect recipe seems to exist to limit the capacity of the latent
 variables. Now, I get that limiting capacity. However, in a variational encoder, it is not
 exactly the latent variable that is made fuzzy. It is actually the embedding, right? If you think
 here, in a variational autoencoder, what you do is you have whatever your image, and then you have
 your encoder, and then you predict in the latent space, you predict Gaussian distributions, like
 you predict the mean and you predict the standard deviation of a Gaussian distribution, and then you
 sample from that Gaussian, that is a horrible Gaussian, you sample from that Gaussian distribution,
 and due to the reparameterization trick, you can actually simply sample from a standard Gaussian
 down here, like that is at zero and has standard deviation one, and that will be your z variable,
 and then you can simply do z times, sorry, z times sigma plus mu, and that will be sampling essentially
 from the, that will be sampling from that respective Gaussian. So in this way, the variable z is not
 made fuzzy. What is actually made fuzzy is this here, and this here comes from h, right? This is
 h, this is the embedding, gives rise to these mu and sigma, and these are made fuzzy because they're
 multiplied by a stochastic variable. So I'm a little bit confused about this paragraph right here,
 because a VAE, I don't think that it limits the capacity of the latent variable, and it fuzzes
 the latent variable, but I might be wrong, or they actually mean something else by latent variable,
 they actually mean the embedding here, in that case, it might make sense again. However, then
 it doesn't make super much sense to limit its capacity. And I've also looked at this sparse
 model, in which simply seems to be kind of sparse encoding of images, it's a really old paper from
 1969, but sorry, 96, 96, not that old. Yeah, but okay, I'm simply going to interpret this as,
 in order to obtain a meaningful representation h down here, we need to limit the capacity of the
 latent variable right here, because otherwise, the model will simply ignore the input and not build
 a good representation for it. So they argue that an architecture like this, an architecture like a VAE,
 like an Infogan, or something like this, could potentially be the next step, if we can make it work.
 The challenge in the next few of the next few years may be to devise non-contrastive methods
 for latent variable energy based model that successfully produce good representation of image,
 video, speech and other signals and yield top performance in downstream supervised tasks without
 requiring large amounts of labeled data. So in German, we have a saying that what they want is
 which means the egg laying wool milk pig. So he can do anything and everything and it costs nothing.
 So that's what they mean. Again, some of these things like energy based model, like anything is
 an energy based model, I just don't find this to be super discriminating in its meaning of what that
 is. Lastly, they talk a bit about their new model called a seer, which is a self supervised model,
 but it's just like a giant confinet trained on a billion images. Oh, but you know, they open sourced
 it. Thank you. You open source the code. So I can totally train my own billion parameter on a
 on a billion random public Instagram images because my Raspberry Pi just technically has that
 capacity. So thanks. But you know, no, but I'm joking a little bit, at least better than OpenAI.
 And at the end, they go into how they use other ways of self supervised learning at Facebook.
 All right, that was my overview over this article. I hope you got at least something from it as a
 high level overview, they first say self supervised learning is maybe the way to get this common sense
 into AI systems. Then they go into what is self supervised learning, they define it first as
 predicting hidden parts from unhidden parts. And later, they say it can be viewed as an energy based
 model that they point out that there's a crucial distinction between tasks like language and vision
 because vision is much more high dimensional gives you much less of a way to represent uncertainty.
 Then they go on and say, well, the contrastive methods, they're not going to be very useful,
 but the contrastive methods handle part of that, they handle this not, they handle this
 part of the dimensionality that you can enumerate all of the possible things. However, they are
 prone to collapse. Sorry, no, the Siamese networks are prone to collapse, the contrastive methods
 fix that. However, because you have to sample from such a high dimensional space, and that is
 really hard, it takes a lot of data. And what we could do is we could do these predictive models
 that directly classify the output, or directly predict the output, right, you predict the missing
 frame, you predict the missing word. But we do it in this way, where you not only do you predict a
 single thing, but you predict an entire set by means of these latent variable predictive models.
 And that they say is maybe the way forward, even though it doesn't work too well yet, like VAEs
 work. But the problem is, they don't have this ability to generate good representations for
 supervised learning, that just doesn't work too well yet. Alright, that was it. If you liked it,
 leave a like, subscribe, share, doubt, tell me what you think in the comments, and bye bye.