Concept Learning with Energy-Based Models (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper openai ebm energy function gradient descent relational neural network latent attention entities spatial relation inference time reasoning demonstration
This is a hard paper! Energy-functions are typically a mere afterthought in current machine learning. A core function of the Energy - its smoothness - is usually not exploited at inference time. This paper takes a stab at it. Inferring concepts, world states, and attention masks via gradient descent on a learned energy function leads to an interesting framework with many possibilities.

Paper: https://arxiv.org/abs/1811.02486
Blog: https://openai.com/blog/learning-concepts-with-energy-functions/
Videos: https://sites.google.com/site/energyconceptmodels/

Abstract:
Many hallmarks of human intelligence, such as generalizing from limited experience, abstract reasoning and planning, analogical reasoning, creative problem solving, and capacity for language require the ability to consolidate experience into concepts, which act as basic building blocks of understanding and reasoning. We present a framework that defines a concept by an energy function over events in the environment, as well as an attention mask over entities participating in the event. Given few demonstration events, our method uses inference-time optimization procedure to generate events involving similar concepts or identify entities involved in the concept. We evaluate our framework on learning visual, quantitative, relational, temporal concepts from demonstration events in an unsupervised manner. Our approach is able to successfully generate and identify concepts in a few-shot setting and resulting learned concepts can be reused across environments. Example videos of our results are available at this http URL

Authors: Igor Mordatch

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, what you're seeing here is an energy-based model that learns the concept of a shape from a demonstration on the left. So on the left you can see a demonstration of data points sampled from a shape, in these cases circles or squares, and then the corresponding energy function that the model infers from that. And then it can replicate that shape on the right using that energy function. So the paper we're going to analyze today is called Concept Learning with Energy-Based Models by Igor Mordac of OpenAI. And this is a very cool paper, or at least I think it's a very cool paper, but it is also a very hard paper. So therefore first I want to kind of make a bit of an introduction into the concepts that we are facing in this paper. So the first thing you need to know are energy functions or energy-based models. What is an energy function? An energy function, sometimes called E, is simply a function with one or multiple inputs, let's call them X. And you can make the... if the energy function is happy with X it will be the value 0. And if the energy function is not happy with X it will be a high value, like larger than 0. So this is happy, this is not happy. So let's give some examples of this. We can formulate almost any machine learning problem in terms of an energy function. Let's say we have a classifier. The classifier takes as an input an image here, maybe of a cat, and a label. So if the label is cat then the energy will be 0 if the energy function is of course working correctly. And if we give the energy function the same image but we give it a wrong label, dog, then it is very high. In the case of the classifier of course we can simply take the loss function as the energy function and we automatically get an energy-based model. So the loss function here would be something like the negative log probability of the correct class. But in any case it is just going to be a high number, let's call it 10 to the 9. So the energy function says, this is very bad, this thing here is very bad, the entire thing you input. It won't tell you yet what's bad about it. So that also means you can change any of the two things to make the classifier happy. Now usually we're concerned with changing the label. It's like, tell me which other label do I need to input to make you happy? And if we make the labels differentiable, of course we never input a true label, we actually input like a distribution, softmax distribution over labels, and that's differentiable. We can use gradient descent to update the dog label, we can use gradient descent to find a label that would make the energy function more happy. So we could use gradient descent to get the cat level if we had a good classifier. But we can also optimize the image to make it compatible with the dog label. That's things that if you ever saw deep dream or something like this, those models do exactly that, they optimize the input image for a particular label. And there you can view the entire neural network including the loss function as the energy function. So what's another example? Another example is, let's say you have a k-means model, and the energy function simply input a data point. And for the data point, what you're going to do is you're going to find the min cluster index, the min k, over, you know, you have your multiple clusters here and your data point might be here, so you're going to find the cluster that's closest and then the distance here, this distance d, will be the energy of that. So the model is very happy when your data point comes from one of the clusters, but your model is not happy when the data point is far away. And that would be the cost function of the k-means function. So that's an energy-based model too. Now currently energy-based models have come into fashion through things like GANs or any sort of noise contrastive estimation. So in a GAN, what you have is you have a discriminator. And the discriminator will basically learn a function to differentiate data from non-data. So that by itself is an energy function. The discriminator will learn a function and that function will be low wherever the discriminator thinks there is data. So it will usually do this around the data points, so the data points form the valleys right here. And then the generator will basically take that discriminator function and will try to infer points that are also in these valleys, to produce points that are also in the valleys. And then you basically have an energy learning competition. The discriminator now tries to push down on the energy where the true data is and push up on the energy where the generated data is. And that will give you basically a steeper energy-based function in the future. So in this case the discriminator neural network is the energy function. And the degenerator just tries to produce data that is compatible with that energy function. So I hope that the concept of what an energy function is is a bit clear. Again any machine learning problem can be formulated in terms of an energy function. Now what is not done so far is what we alluded to a little bit before in the classifier example and also here. So right now when we want to train again we simply take the generator to produce data. Now what's the generator's goal? The generator's goal is to hit those valleys in the energy function. And we produce a generator in one shot to produce this data. But what we could also do is of course we could just start somewhere. Let's say here we pick a random data point and then we use gradient descent because the energy function in this case is smooth. We use gradient descent to just drop down this valley and then find ourselves in this valley. So without ever training a generator we can use this methods to produce points that are in the valley of the energy function. And I don't know if people... I guess people have trained GANs like this. The reason why it doesn't work let's say in the real world is because that procedure will just produce adversarial examples for the discriminator. And those usually look like nothing like data. Because if you keep the discriminator just stable and gradient descent against it what you'll get isn't really qualitatively good. But in principle if the discriminator was a good energy function for the data to describe the data we could use gradient descent. The same up here. In order to find a good label for an image given that we have a good energy function, we could simply gradient descent on the label in order to find a better label. So in this paper we're going to have a situation where we say we're given an energy function and we're given a bunch of inputs. They are then called X, A, and W. And if I have my energy function already, if I have given my energy function and I have given two of those three things, any two, I can infer the last thing simply by gradient descent on my energy function. Because I know the energy function is zero when the energy function is happy with the input. So when all of these things agree, basically the energy function is happy, it will output zero otherwise it will output a high value. Therefore if I'm given any of those two, any two of those three things, I can find a compatible third thing by descending. And then of course over here in these machine learning problems, the task was always actually to learn an energy function. So usually in the training data set we are given images and labels and we want to learn this energy function which would be parameterized. So we want to learn the parameters. And the same here in our general case if we are now given three things but we are not given the parameters of the energy function, we don't know what those are. As long as we're given all of the inputs in our training data set, and our training data set guarantees these are actually, you know, these are inputs that are compatible with each other, the energy function should be low, we can simply gradient descent on the parameters of the energy function. So in a sense there are four things, right? There are these three inputs and then there are the parameters of the energy function. If we're given any three of those four, we can gradient descent on the rest. And that's going to be the basis. So the X here is going to be the so-called state. And the state in this paper is going to be images of entities. The entities, sorry it's not going to be images, but the entities are these little circles that you're going to see. And each of those entities can have an X position, a Y position, and I believe a color. So R, G and B. So each of those can have that. And then the concatenation of all of those attributes is one big vector and that is your X, that's your state. So state is number of entities and their attributes. A is going to be an attention mask over the state. So A is going to be... here you have four entities, so A will have four entries telling you which of these entities you should pay attention to right now. And W is going to be a concept vector so called. So W is going to be the embedding of a concept. Now what a concept is in this case is very general. I can give you an example. One concept is do the entities that the A pays attention to, are they close to each other? So in this case you see we have two entities that A has a high value on and this is this ball up here and this ball down here. Now if the concept vector is the embedding for the concept of being close to each other then the energy function would be very happy if those two things are close to each other and it would be very unhappy if those two things aren't close to each other. But in the very same situation, so the same X, the same attention mask, but a different concept, so a different W vector right here, then the energy function would be maybe very happy if the two things are far apart and maybe unhappy if the two things are close. So the question is always how are the three things that you put into the energy function compatible with each other and given all but one of these things you can infer the other. So let's say you have a perfect energy function for this situation. You're just given the energy function, you can trust it. And you are given, let's make an example, you are given the X, so you're given the state, I'm going to draw the state down here, right? Okay, this is the state and you're given the W and the W is the embedding, it's a vector but in embedding space, but the embedding is for a line, right? So the geometric unit of a line. Now your task is to find A, the attention mask that will make the energy function happy. And as you can see right here, what you would do is you would put a lot of weight on this, this, this and this ball and no weight on that ball, because those make a line. And since everything here is differentiable, so the state is differentiable, the attention is differentiable and the concepts are vectors, they're differentiable, you can use gradient descent to find that. Another example, if you're given again the same W, so line, and you are given this following thing and you are given, now you're given the attention on these three and you say please find the X, please find the X, the state that makes this energy function happy. Now this here you would call the starting state, the X zero, your task is going to be find the X one, find the state, how do you have to change this state such that the energy function is happy? And of course the answer is going to be is to push this ball here inward until it is in the middle of the two others, so the three form a line. Right, these three form a line. You don't have to do anything to this ball up here, because there is no attention on it. And the attention, it's only, is the concept for the things that you put attention on and the state, are those three in agreement and the energy function is happy. Okay, we have covered the basics. Now let's dive into the paper. I think this is the longest introduction ever, but I think it will pay off once you see. So they specifically, or this author, I think it's a single author, identifies two different things that you can do with an energy function here. Of course you can do more as we saw, but they identify two. So here is where you have given the initial state and an attention mask and you want to find the x1, the state that satisfies the concept and attention the most. This the author calls generation. As you can see here, these four things that you have the attention on are pushed around until they make a square, because the concept right now is square. And in the other case, where you are given this x0 and x1, just call this x right here, just call this thing x. If you're given those two, and you are given the concept square, and you're tasked with finding a, the attention mask, of course you're going to put the attention on these right here. And that is going to happen through gradient descent. Again, we're not learning a model to give you that attention. Like in a GAN, we're learning a generator to just one shot give it to you. Right now, what we're going to do is we're going to gradient descent optimize on our smooth energy function to give us that perfect attention mask that satisfies the energy function. Alright, so this is the difference right here. Gradient descent is part of the output procedure of the model. Usually we just use it to learn, and we learn a one-shot model. But here gradient descent is part of the model. So they introduce energy functions here, and they say, okay, we can have a policy on x. So if we're given a concept W, and if we're given an A, we can have a policy over x, which basically means we can find x's that are compatible with that by running gradient descent here. You see there is an xk minus one, and we are running gradient descent on the energy function with respect to x to find a better x that satisfies the energy function given those inputs. And the same if we want to find an attention mask, we are running gradient descent on the attention mask, again, in order to satisfy the same energy function. So you see the inputs are both times the same. The concept here we can input square, here we can input square, but the difference is what we're running gradient descent on and what we keep constant. And I would get, I would add a third line here actually, because we can also, if we're given an x and an a, we can also infer a W. And that's going to be an integral part. So if I have this right here, and this situation, and I have, say I have attention on these four, now I can ask the model, so I'm given x and I'm given a, I can ask the model to infer W. And the model should ideally output, ha, this is square. Now the model isn't going to output square, the model is going to output a vector representation of square. So the model is going to output square but as a vector of numbers, because that's how we've trained it. W is an embedding. But what we can then do later is we can say, okay, I'm not going to tell you it's a square, you just come up with a vector W to describe this situation. And now I'm going to take that vector W that you came up with, miss, mister or missus model, and I'm going to take, tell you a new situation. This situation right here. And I'm going to now give you x, and I'm going to give you the W that you yourself have output, and now please tell me what's the a. And then the model is of course supposed to tell you, oh these four here are the a. So without ever telling that it should be a square, what you can do is you can let the model infer a W from one example situation, and then transfer that W to a new situation. So it can identify, you can just say whatever concept I have up here, please apply that same concept, which is the W down here. And this is the entire paper now. This is the concept learning through energy-based models. Okay, so that is kind of a third line I would add down here. You can infer a concept vector if you're given the X and the a. So in order to do all this, their energy function is going to be a so-called relational neural network. So what you'll have is you'll have a simple neural network, a multi-layer perceptron that always connects two entities to each other with the concept vector, and then this is I believe a sigmoid that connects the attention masks of the two, and you simply sum over all pairs of two entries in your model, and then you send that through an MLP, sorry, through an MLP again. This I believe is not so important, it's just important that they can feed this entire situation, the X, the a, and the W, they can basically feed into a neural network, and the neural network comes up with a number of how well those three things fit together. And then you can transfer these concepts. That's pretty cool. Now the only question is, of course, we've always said we're given an energy function, we're just, we just have it. But of course, this is a neural network, and the neural network has parameters, and the parameters, we don't know what good parameters are at the beginning. So we need to train this thing. And again, the reason why these are toy problems right here is, I mean, we'll get to why it's computational, but this is kind of a new field, I believe in machine learning, at least I come from classical machine learning, and we only ever have used, like SGD to train, and we only ever have produced models that one shot produce something. And here, we, this is a, I believe this is a new concept where you use gradient descent as part of the output. And that makes a lot of trouble. So that's why we work in toy problems. So what this, this here is the situation I described. You have a demo event where you're given the X and the A, and you're supposed to infer the W. So the question here is, what's the W? And the model will come up with a W, and you're not going to do anything, you know, right now, you're simply going to take that W and tell it, oh, well, here is a so-called test event. So please apply the W you came up with in this test event. And please find me the A, in this case, that satisfies the W and the X I give you here. And of course, the A right here is, as you can see, even you don't know that it's a square. And the actual concept here is move the gray ball to the middle of the square, right? That is it here. But no one has told me this. I just looked at the picture. So the correct answer here would be to place attention on those four things, and then to take this thing and move it to the middle right here, in this over here. So that would be the correct answer. Now, the question is, how do you train something like this? And they show that they, so this is the loss function right here. The loss function is they give you a concept and an initial situation, and you're supposed to infer the X1 and the A. And the loss function is simply the negative log likelihood of that. But what does that mean? So we'll make it easier. If you have this procedure right here, where you have a demo event, this up here, this is demo, and this is a test event. How are you going, this entire procedure, how are you going to learn the energy function? Well, in this case, this entire procedure, this entire thing is one training sample. But usually we have input and label. And now here, it's much more complicated because, so we have input, okay, that's this X and this A, cool. But then we have SGD as integral part of the procedure to determine the W. And now what we could do is just apply a loss to the W, but we don't because we don't know what the embedding space for the concepts is. We could maybe train a classifier, but in this case, we want to train the ability to transfer these concepts. So our training sample needs to be one time transferring a concept. So SGD for one is part of our process here. And not only that, but then this X here, of course, is also part of our training sample, right? This appears X0 and this here is X1. And now we need to find this A, this attention mask. And that is an SGD again. Remember, inferring anything through the energy function is a gradient descent process. So ultimately, our one training example consists of X0, A at the beginning, so let's call that A0. It consists of the SGD procedure to find W, it consists of X1, and it consists of the SGD procedure to find A, the A1, the output A. And then that will give us the output A, the A1. So this here is our input in the classical machine learning. This would be our X, and this here would be our label Y. And that's what we train on. We train. So such that the output right here, the A, this is of course, sorry, this is of course the Y hat. This is what we predict. And in the training sample, we just write a little generator that will, you know, make this situation that knows what the concept is, right? It will say, okay, I'm gonna make an example for a square, then it will make this, will make the attention mask for a square, and then it will make the new situation again with a square, but not tell us the attention mask there, and it will make the attention mask into the true Y. So at the end, we can compare what our model output, the attention mask we output here, without ever knowing that this should be a square, right? And we have the true label, which comes out of the generator that at the beginning decided that it should be a square. And then the loss, the distance between those two, that's our loss. This is an, this is an enormous procedure to get a loss. And most crucially, you have to back propagate through optimization procedures. And this is something that we just can't do yet in our models. If you take an image, a ResNet 50, right, right now, we do one forward propagation to get a label. In this procedure, if you had to back propagate through the optimization procedure, for each sample, you would need to basically back propagate through 50 forward passes of the ResNet, if you if your optimization procedure is 50 steps long, and that is just not feasible right now. So that's why we don't do it. But I believe maybe once we find a smart way of back propping through optimization procedures, a whole lot of these things will become the new a new wave in machine learning. I really I'm excited by this. I'm pretty sure it doesn't work yet. And this is very fiddly, fiddly work. But I'm excited by the prospect that we can do this. So this is the training procedure, right? You are given x0, x1, and a, and you optimize in order to infer the concept behind it, right? The generator that your level generator of your training data, it knows the concept, it has a concept in mind when it generated this, but you're not telling your model what the concept is, it needs to infer that. And then using the model, the thing that the model inferred, you can either give it x0 and x1 and infer a, or you can give it the x and the a and infer x, you can do either of those, right? These are called identification or generation, respectively. And then you compare the output here to what the generator at the beginning thought, again, it's not telling you it's that's because that's the label. And you compare this to that. And that will be your loss to train your energy function parameters. So your training samples, if you think of this entire thing as one forward pass of the model, then it's just classic machine learning, right? You have a training sample, which is one forward pass and you have a corresponding label that you infer. So let's jump to the experiments right here. The experiments are actually pretty cool. So what they've done is, for example, taken the concept of being far apart from something now being far apart so that the little x needs to be as far away as possible from the ball that has the attention on it. So if you do generation, and you start the little x right here, and you ask the model, where please infer the next state of the world, it will push that little x away right here. And in color, you can see the energy function values of the position of the x. So it pushes it away from this thing. But if you take the same concept embedding, the concept embedding of being far away, but you don't do generation, you do identification, which means you infer the a, then it will simply tell you that this ball right here is the furthest away from the x. So you can do all sorts of things like this and transferring concepts. I find this here pretty interesting. So they have two different concepts. One concept is red as an identification. You need to identify the red ball. But the other concept is you need to turn something red, right? You need to take a ball that is maybe now blue, and of course the color, you can gradient descent on the colors, you need to make it red. And since the energy function, it just takes three input x, a and w. You're not gonna tell it right now in which situation you are. It has to create this w embedding space through learning. And if you do it with those two concepts, then it will put the make something red concept and the is something red concepts in the same places. So this is a PCA. And in blue, I think these blue is the attention codes for identify the red things. And in red are the generation code for make something red and they will be put in the same place, which is pretty cool. It means that the energy function really learns the feature of something being red. I find this pretty neat. And then here they have some experiments where they basically show we need that gradient descent optimization procedure because only after many steps will the energy function basically be aligned with the concept that you want. So if you have a zero-shot model, like just one forward pass as we do here, you'll see that the energy function that is supposed to make a circle from samples, right? This is the example concept right here. If you just have a one-shot model, it will, it cannot or in this case at least, it doesn't learn to one-shot produce. Only if you optimize for a few steps will it get this. So you optimize at inference time and that seems to be very important. You can see again here demonstrations of this. So the example is this and then the model as you can see after 20 steps learn optimizes the points to go to these locations. Whereas after only one step it didn't do that yet. So there are complex things at work here. And this column here is where you don't have a relational neural network. So you can't basically capture dependencies between things. So you have no chance of making a square because you don't know where the things are in relation to each other. But that's more of an engineering question. Their point is basically that if you have models that do optimization at inference time, they are much more powerful than models that just do a one-shot forward pass. It's sort of like an autoregressive model in NLP versus a non autoregressive model that produces all words at once. If you produce all words of a sentence at once, no word can depend on any other word and you can just come produce independent or you can just produce independent things which will make the sentence often not make any sense. They also have this KL objective which is a regularizer which I believe that's just a trial and error they built it in because. But it is a regularizer. I don't want to really go into that. And then they do demonstration and they reenact it on a robot. The demonstration here is that there is a situation where two things have attention on and you're supposed to move something into the middle of the two things. So that's the concept. You don't tell the robot the concept. It needs to learn that from data and then infer that this is the concept that you want and then transfer that to the other environment. Now you know there's this robot environment but ultimately they still encode the positions of these things and the position of that. And really all you have to do different here is that instead of moving this actuator directly you need to calculate what you need to do to the individual joints in the robot. So I think this is maybe because it's open AI and it needs to you know look roboty and stuff but the problem here is not really different. It's not real-world transfer or anything. So yeah let's go through some of the things they can learn with this. So you can see here they can learn these regional geometric shapes and on the left is the example event that the model needs to take the concept from. Now this is I believe very much identification. So what they did is they trained with a data set where all of these appear. So there are squares, there are lines, there are circles. So this is maybe my criticism here that it is not so much to generally infer a concept. It is more like identify the concept. So the model basically just needs to decide is this line, is this circle or is this square because those things were in the training data set. It would be nice to see how this generalizes to general concepts or if we can even make that if we can have a zero-shot concept inference and then transfer those concepts to other things. Maybe that's already happening. I don't know. So here the spatial arrangement is to either be close to something or to be between two things. So if the attention is on two things you want in between. So you see the top ones are the demonstrations. It needs to recognize the concept and it needs to basically optimize to fulfill that concept. Shapes. So to make shapes is... Oh yeah there's a triangle. Again this very much I believe relies on recognition and not actual understanding of what a triangle is. Here you have proximity being closer being far apart. What else is cool? Oh yeah you have the recognition for the same task. You need to identify the ball that is closer. Here you really also see the optimization procedure in action. Where for example at the beginning of each flicker you kind of see the attention being everywhere and then stabilizing to one or two points. So if two points are equally close or far apart you'll see the attention being on multiple points. Which is pretty cool right? So that means the model really learns this concept. Here's the count quantity. So you can either have one two or larger than three or something. Yeah that seems like they tried three and four and didn't work so they just said we'll just do larger than three. And here is this robot thing where it also always needs to move in between. Now this is the part that I'm not really impressed with but you know whatever you want. Okay I hope this was a good introduction to energy functions. What you can do with them. What I think of them. And of this paper it is a pretty cool paper. Yes it only works on toy problems so far but I believe this is one interesting direction of future machine learning and something yet to be very much explored. If you like this content please subscribe, tell all of your friends about it, share and I'll see you next time. Bye bye!
 Hi there, what you're seeing here is an energy-based model that learns the
 concept of a shape from a demonstration on the left. So on the left you can see a
 demonstration of data points sampled from a shape, in these cases circles or
 squares, and then the corresponding energy function that the model infers
 from that. And then it can replicate that shape on the right using that energy
 function. So the paper we're going to analyze today is called Concept Learning
 with Energy-Based Models by Igor Mordac of OpenAI. And this is a very cool paper,
 or at least I think it's a very cool paper, but it is also a very hard paper.
 So therefore first I want to kind of make a bit of an introduction into the
 concepts that we are facing in this paper. So the first thing you need to
 know are energy functions or energy-based models. What is an energy
 function? An energy function, sometimes called E, is simply a function with one
 or multiple inputs, let's call them X. And you can make the... if the energy
 function is happy with X it will be the value 0. And if the energy function is
 not happy with X it will be a high value, like larger than 0. So this is happy, this
 is not happy. So let's give some examples of this. We can formulate almost any
 machine learning problem in terms of an energy function. Let's say we have a
 classifier. The classifier takes as an input an image here, maybe of a cat, and a
 label. So if the label is cat then the energy will be 0 if the energy function
 is of course working correctly. And if we give the energy function the same
 image but we give it a wrong label, dog, then it is very high. In the case of the
 classifier of course we can simply take the loss function as the energy function
 and we automatically get an energy-based model. So the loss function here
 would be something like the negative log probability of the correct class.
 But in any case it is just going to be a high number, let's call it 10 to the 9.
 So the energy function says, this is very bad, this thing here is
 very bad, the entire thing you input. It won't tell you yet what's bad about it.
 So that also means you can change any of the two things to make the classifier
 happy. Now usually we're concerned with changing the label. It's like, tell me
 which other label do I need to input to make you happy? And if we make the labels
 differentiable, of course we never input a true label, we actually input like a
 distribution, softmax distribution over labels, and that's differentiable.
 We can use gradient descent to update the dog label, we can use gradient descent
 to find a label that would make the energy function more happy. So we could
 use gradient descent to get the cat level if we had a good classifier. But we can
 also optimize the image to make it compatible with the dog label.
 That's things that if you ever saw deep dream or something like this, those
 models do exactly that, they optimize the input image for a particular label.
 And there you can view the entire neural network including the loss function
 as the energy function. So what's another example? Another example is, let's say
 you have a k-means model, and the energy function simply input a data point.
 And for the data point, what you're going to do is you're going to find the min
 cluster index, the min k, over, you know, you have your multiple clusters here and
 your data point might be here, so you're going to find the cluster that's closest
 and then the distance here, this distance d, will be the energy of that. So the
 model is very happy when your data point comes from one of the clusters, but your
 model is not happy when the data point is far away. And that would be the cost
 function of the k-means function. So that's an energy-based model too. Now
 currently energy-based models have come into fashion through things like GANs or
 any sort of noise contrastive estimation. So in a GAN, what you
 have is you have a discriminator. And the discriminator will basically learn a
 function to differentiate data from non-data. So that by itself is an energy
 function. The discriminator will learn a function and that function will be low
 wherever the discriminator thinks there is data. So it will usually do
 this around the data points, so the data points form the valleys right here. And
 then the generator will basically take that discriminator function and will try
 to infer points that are also in these valleys, to produce points that are also
 in the valleys. And then you basically have an energy learning competition. The
 discriminator now tries to push down on the energy where the true data is and
 push up on the energy where the generated data is. And that will give you
 basically a steeper energy-based function in the future. So in this
 case the discriminator neural network is the energy function. And the
 degenerator just tries to produce data that is compatible with that energy
 function. So I hope that the concept of what an energy function is is a bit
 clear. Again any machine learning problem can be formulated in terms of
 an energy function. Now what is not done so far is what we alluded to a little
 bit before in the classifier example and also here. So right now when we want to
 train again we simply take the generator to produce data. Now what's the
 generator's goal? The generator's goal is to hit those valleys in the energy
 function. And we produce a generator in one shot to produce this data. But
 what we could also do is of course we could just start somewhere. Let's say
 here we pick a random data point and then we use gradient descent because the
 energy function in this case is smooth. We use gradient descent to just drop
 down this valley and then find ourselves in this valley. So without ever training
 a generator we can use this methods to produce points that are in the valley of
 the energy function. And I don't know if people... I guess people have
 trained GANs like this. The reason why it doesn't work let's say in the real
 world is because that procedure will just produce adversarial examples for
 the discriminator. And those usually look like nothing like data. Because if you
 keep the discriminator just stable and gradient descent against it what you'll
 get isn't really qualitatively good. But in principle if the discriminator was a
 good energy function for the data to describe the data we could use gradient
 descent. The same up here. In order to find a good label for an image given
 that we have a good energy function, we could simply gradient
 descent on the label in order to find a better label. So in this
 paper we're going to have a situation where we say we're given an energy
 function and we're given a bunch of inputs. They are then called X, A, and W.
 And if I have my energy function already, if I have given my energy function and I
 have given two of those three things, any two, I can infer the last thing
 simply by gradient descent on my energy function. Because I know the energy
 function is zero when the energy function is happy with the input.
 So when all of these things agree, basically the energy function is happy, it
 will output zero otherwise it will output a high value. Therefore if I'm given any
 of those two, any two of those three things, I can find a compatible third
 thing by descending. And then of course over here in these machine learning
 problems, the task was always actually to learn an energy function. So
 usually in the training data set we are given images and labels and we want to
 learn this energy function which would be parameterized. So we want to learn the
 parameters. And the same here in our general case if we are now given three
 things but we are not given the parameters of the energy function, we
 don't know what those are. As long as we're given all of the inputs in our
 training data set, and our training data set guarantees these are actually, you
 know, these are inputs that are compatible with each other, the energy
 function should be low, we can simply gradient descent on the parameters of
 the energy function. So in a sense there are four things, right? There are these
 three inputs and then there are the parameters of the energy function. If
 we're given any three of those four, we can gradient descent on the rest. And
 that's going to be the basis. So the X here is going to be the so-called state.
 And the state in this paper is going to be images of entities. The entities,
 sorry it's not going to be images, but the entities are these little circles
 that you're going to see. And each of those entities can have an X position, a
 Y position, and I believe a color. So R, G and B. So each of those can have that.
 And then the concatenation of all of those attributes is one big vector and
 that is your X, that's your state. So state is number of entities and their
 attributes. A is going to be an attention mask over the state. So A is
 going to be... here you have four entities, so A will have four entries telling you
 which of these entities you should pay attention to right now. And W is going to
 be a concept vector so called. So W is going to be the embedding of a concept.
 Now what a concept is in this case is very general. I can give you an example.
 One concept is do the entities that the A pays attention to, are they
 close to each other? So in this case you see we have two entities that A has a
 high value on and this is this ball up here and this ball down here. Now if the
 concept vector is the embedding for the concept of being close to each other
 then the energy function would be very happy if those two things are close to
 each other and it would be very unhappy if those two things aren't close to each
 other. But in the very same situation, so the same X, the same attention mask, but
 a different concept, so a different W vector right here, then the energy
 function would be maybe very happy if the two things are far apart and maybe
 unhappy if the two things are close. So the question is always how are the three
 things that you put into the energy function compatible with each other and
 given all but one of these things you can infer the other. So let's say you
 have a perfect energy function for this situation.
 You're just given the energy function, you can trust it. And you are given, let's
 make an example, you are given the X, so you're given the state, I'm going to draw
 the state down here, right? Okay, this is the state and you're given the W and the
 W is the embedding, it's a vector but in embedding space, but the
 embedding is for a line, right? So the geometric unit of a line.
 Now your task is to find A, the attention mask that will make the energy function
 happy. And as you can see right here, what you would do is you would put a lot of
 weight on this, this, this and this ball and no weight on that ball, because those
 make a line. And since everything here is differentiable, so the state is
 differentiable, the attention is differentiable and the concepts are
 vectors, they're differentiable, you can use gradient descent to find that. Another
 example, if you're given again the same W, so line, and you are given this
 following thing and you are given, now you're given the attention on these
 three and you say please find the X, please find the X, the state that makes
 this energy function happy. Now this here you would call the starting state, the X
 zero, your task is going to be find the X one, find the state, how do you have to
 change this state such that the energy function is happy? And of course the
 answer is going to be is to push this ball here inward until it is in the
 middle of the two others, so the three form a line. Right, these three form a
 line. You don't have to do anything to this ball up here, because
 there is no attention on it. And the attention, it's only, is the concept for
 the things that you put attention on and the state, are those three in agreement
 and the energy function is happy. Okay, we have covered the basics. Now let's
 dive into the paper. I think this is the longest introduction ever, but
 I think it will pay off once you see. So they specifically, or this
 author, I think it's a single author, identifies two different things that you
 can do with an energy function here. Of course you can do more as we saw, but they
 identify two. So here is where you have given the initial state and an attention
 mask and you want to find the x1, the state that satisfies the concept and
 attention the most. This the author calls generation. As you can see here,
 these four things that you have the attention on are pushed around until
 they make a square, because the concept right now is square. And in the other
 case, where you are given this x0 and x1, just call this x right here, just call
 this thing x. If you're given those two, and you are given the concept square, and
 you're tasked with finding a, the attention mask, of course you're going to
 put the attention on these right here. And that is going to happen through
 gradient descent. Again, we're not learning a model to give you that
 attention. Like in a GAN, we're learning a generator to just one shot give it to
 you. Right now, what we're going to do is we're going to gradient descent
 optimize on our smooth energy function to give us that perfect attention mask
 that satisfies the energy function. Alright, so this is the difference right
 here. Gradient descent is part of the output procedure of the model. Usually we
 just use it to learn, and we learn a one-shot model. But here gradient descent
 is part of the model. So they introduce energy functions here, and they say, okay,
 we can have a policy on x. So if we're given a concept W, and if we're given an
 A, we can have a policy over x, which basically means we can find x's that are
 compatible with that by running gradient descent here. You see there is an xk
 minus one, and we are running gradient descent on the energy function with
 respect to x to find a better x that satisfies the energy function given
 those inputs. And the same if we want to find an attention mask, we are running
 gradient descent on the attention mask, again, in order to satisfy the same
 energy function. So you see the inputs are both times the same. The concept here
 we can input square, here we can input square, but the difference is what we're
 running gradient descent on and what we keep constant. And I would get, I would
 add a third line here actually, because we can also, if we're given an x and an
 a, we can also infer a W. And that's going to be an integral part. So if I
 have this right here, and this situation, and I have, say I have attention on these
 four, now I can ask the model, so I'm given x and I'm given a, I can ask the
 model to infer W. And the model should ideally output, ha, this is square. Now
 the model isn't going to output square, the model is going to output a vector
 representation of square. So the model is going to output square but as a
 vector of numbers, because that's how we've trained it. W is an embedding. But
 what we can then do later is we can say, okay, I'm not going to tell you it's a
 square, you just come up with a vector W to describe this situation. And now I'm
 going to take that vector W that you came up with, miss, mister or missus model,
 and I'm going to take, tell you a new situation. This situation right here. And
 I'm going to now give you x, and I'm going to give you the W that you
 yourself have output, and now please tell me what's the a. And then the model is of
 course supposed to tell you, oh these four here are the a. So without
 ever telling that it should be a square, what you can do is you can let the model
 infer a W from one example situation, and then transfer that W to a new
 situation. So it can identify, you can just say whatever concept I have up here,
 please apply that same concept, which is the W down here. And this is the entire
 paper now. This is the concept learning through energy-based models. Okay, so that
 is kind of a third line I would add down here. You can infer a concept vector if
 you're given the X and the a. So in order to do all this, their energy function is
 going to be a so-called relational neural network. So what you'll have is
 you'll have a simple neural network, a multi-layer perceptron that always
 connects two entities to each other with the concept vector, and then this is I
 believe a sigmoid that connects the attention masks of the two, and you
 simply sum over all pairs of two entries in your model, and then you send that
 through an MLP, sorry, through an MLP again. This I believe is not so important,
 it's just important that they can feed this entire situation, the X, the a, and
 the W, they can basically feed into a neural network, and the neural network
 comes up with a number of how well those three things fit together. And then you
 can transfer these concepts. That's pretty cool. Now the only question is, of
 course, we've always said we're given an energy function, we're just, we just have
 it. But of course, this is a neural network, and the neural network has
 parameters, and the parameters, we don't know what good parameters are at the
 beginning. So we need to train this thing. And again, the reason why these are toy
 problems right here is, I mean, we'll get to why it's computational, but this is
 kind of a new field, I believe in machine learning, at least I come from classical
 machine learning, and we only ever have used, like SGD to train, and we only ever
 have produced models that one shot produce something. And here, we, this is a,
 I believe this is a new concept where you use gradient descent as part of the
 output. And that makes a lot of trouble. So that's why we work in toy problems. So
 what this, this here is the situation I described. You have a demo event where
 you're given the X and the A, and you're supposed to infer the W. So the question
 here is, what's the W? And the model will come up with a W, and you're not going to
 do anything, you know, right now, you're simply going to take that W and tell it,
 oh, well, here is a so-called test event. So please apply the W you came up with in
 this test event. And please find me the A, in this case, that satisfies the W and
 the X I give you here. And of course, the A right here is, as you can see, even you
 don't know that it's a square. And the actual concept here is move the gray
 ball to the middle of the square, right? That is it here. But no one has told me
 this. I just looked at the picture. So the correct answer here would be to place
 attention on those four things, and then to take this thing and move it to the
 middle right here, in this over here. So that would be the correct answer. Now,
 the question is, how do you train something like this? And they show
 that they, so this is the loss function right here. The loss function is they
 give you a concept and an initial situation, and you're supposed to infer
 the X1 and the A. And the loss function is simply the negative log likelihood of
 that. But what does that mean? So we'll make it easier. If you have this
 procedure right here, where you have a demo event, this up here, this is demo,
 and this is a test event. How are you going, this entire procedure, how are you
 going to learn the energy function? Well, in this case, this entire procedure, this
 entire thing is one training sample. But usually we have input and
 label. And now here, it's much more complicated because, so we have input, okay,
 that's this X and this A, cool. But then we have SGD as integral part of the
 procedure to determine the W. And now what we could do is just apply a loss to
 the W, but we don't because we don't know what the embedding space for the concepts
 is. We could maybe train a classifier, but in this case, we want to train the
 ability to transfer these concepts. So our training sample needs to be one time
 transferring a concept. So SGD for one is part of our process here. And not only
 that, but then this X here, of course, is also part of our training sample, right?
 This appears X0 and this here is X1. And now we need to find this A, this
 attention mask. And that is an SGD again. Remember, inferring anything through the
 energy function is a gradient descent process. So ultimately, our one training
 example consists of X0, A at the beginning, so let's call that A0. It
 consists of the SGD procedure to find W, it consists of X1, and it consists of
 the SGD procedure to find A, the A1, the output A. And then that will give us the
 output A, the A1. So this here is our input in the classical machine learning. This
 would be our X, and this here would be our label Y. And that's what we train on.
 We train. So such that the output right here, the A, this is of course, sorry, this
 is of course the Y hat. This is what we predict. And in the training sample, we
 just write a little generator that will, you know, make this situation that knows
 what the concept is, right? It will say, okay, I'm gonna make an example for a
 square, then it will make this, will make the attention mask for a square, and then
 it will make the new situation again with a square, but not tell us the
 attention mask there, and it will make the attention mask into the true Y. So at
 the end, we can compare what our model output, the attention mask we output
 here, without ever knowing that this should be a square, right? And we have the
 true label, which comes out of the generator that at the beginning decided
 that it should be a square. And then the loss, the distance between those two,
 that's our loss. This is an, this is an enormous procedure to get a loss. And
 most crucially, you have to back propagate through optimization procedures.
 And this is something that we just can't do yet in our models. If you take an
 image, a ResNet 50, right, right now, we do one forward propagation to get a
 label. In this procedure, if you had to back propagate through the optimization
 procedure, for each sample, you would need to basically back propagate through
 50 forward passes of the ResNet, if you if your optimization procedure is 50
 steps long, and that is just not feasible right now. So that's why we don't do it.
 But I believe maybe once we find a smart way of back propping through optimization
 procedures, a whole lot of these things will become the new a new wave in
 machine learning. I really I'm excited by this. I'm pretty sure it doesn't work
 yet. And this is very fiddly, fiddly work. But I'm excited by the prospect that
 we can do this. So this is the training procedure, right? You are given x0, x1,
 and a, and you optimize in order to infer the concept behind it, right? The
 generator that your level generator of your training data, it knows the concept,
 it has a concept in mind when it generated this, but you're not telling
 your model what the concept is, it needs to infer that. And then using the
 model, the thing that the model inferred, you can either give it x0 and x1 and
 infer a, or you can give it the x and the a and infer x, you can do either of
 those, right? These are called identification or generation,
 respectively. And then you compare the output here to what the generator at the
 beginning thought, again, it's not telling you it's that's because that's
 the label. And you compare this to that. And that will be your loss to train your
 energy function parameters. So your training samples, if you think of this
 entire thing as one forward pass of the model, then it's just classic machine
 learning, right? You have a training sample, which is one forward pass and you
 have a corresponding label that you infer. So let's jump to the experiments
 right here. The experiments are actually pretty cool. So what they've done is, for
 example, taken the concept of being far apart from something now being far apart
 so that the little x needs to be as far away as possible from the ball that has
 the attention on it. So if you do generation, and you start the little x
 right here, and you ask the model, where please infer the next state of the world,
 it will push that little x away right here. And in color, you can see the energy
 function values of the position of the x. So it pushes it away from this thing. But
 if you take the same concept embedding, the concept embedding of being far away,
 but you don't do generation, you do identification, which means you infer the a,
 then it will simply tell you that this ball right here is the furthest away
 from the x. So you can do all sorts of things like this and transferring
 concepts. I find this here pretty interesting. So they have two different
 concepts. One concept is red as an identification. You need to identify the
 red ball. But the other concept is you need to turn something red, right? You
 need to take a ball that is maybe now blue, and of course the color, you can
 gradient descent on the colors, you need to make it red. And since the energy
 function, it just takes three input x, a and w. You're not gonna tell
 it right now in which situation you are. It has to create this w embedding
 space through learning. And if you do it with those two concepts, then it will put
 the make something red concept and the is something red concepts in the same
 places. So this is a PCA. And in blue, I think these blue is the attention codes
 for identify the red things. And in red are the generation code for make
 something red and they will be put in the same place, which is pretty cool. It
 means that the energy function really learns the feature of something being
 red. I find this pretty neat. And then here they have some
 experiments where they basically show we need that gradient descent
 optimization procedure because only after many steps will the energy
 function basically be aligned with the concept that you want. So if you have a
 zero-shot model, like just one forward pass as we do here, you'll see that the
 energy function that is supposed to make a circle from samples, right? This is the
 example concept right here. If you just have a one-shot model, it will, it cannot
 or in this case at least, it doesn't learn to one-shot produce. Only if you
 optimize for a few steps will it get this. So you optimize at inference time
 and that seems to be very important. You can see again here demonstrations of
 this. So the example is this and then the model as you can see after 20 steps
 learn optimizes the points to go to these locations. Whereas after only one
 step it didn't do that yet. So there are complex things at work here. And this
 column here is where you don't have a relational neural network. So you can't
 basically capture dependencies between things. So you have no chance of
 making a square because you don't know where the things are in relation to each
 other. But that's more of an engineering question. Their point is basically that
 if you have models that do optimization at inference time, they are much more
 powerful than models that just do a one-shot forward pass. It's sort of like
 an autoregressive model in NLP versus a non autoregressive model that produces
 all words at once. If you produce all words of a sentence at once, no word can
 depend on any other word and you can just come produce independent or you can
 just produce independent things which will make the sentence often not make
 any sense. They also have this KL objective which is a regularizer which I
 believe that's just a trial and error they built it in because. But it is a
 regularizer. I don't want to really go into that. And then they do
 demonstration and they reenact it on a robot. The demonstration here is that
 there is a situation where two things have attention on and you're supposed to
 move something into the middle of the two things. So that's the concept. You don't
 tell the robot the concept. It needs to learn that from data and then infer that
 this is the concept that you want and then transfer that to the other
 environment. Now you know there's this robot
 environment but ultimately they still encode the positions of these things and
 the position of that. And really all you have to do different here is that
 instead of moving this actuator directly you need to calculate what you
 need to do to the individual joints in the robot. So I think this is maybe
 because it's open AI and it needs to you know look roboty and stuff but the
 problem here is not really different. It's not real-world
 transfer or anything. So yeah let's go through some of the things they can
 learn with this. So you can see here they can learn these regional geometric
 shapes and on the left is the example event that the model needs to take the
 concept from. Now this is I believe very much identification. So what
 they did is they trained with a data set where all of these appear. So
 there are squares, there are lines, there are circles. So this is maybe my
 criticism here that it is not so much to generally infer a concept. It is more
 like identify the concept. So the model basically just needs to decide is this
 line, is this circle or is this square because those things were in
 the training data set. It would be nice to see how this generalizes to general
 concepts or if we can even make that if we can have a zero-shot concept
 inference and then transfer those concepts to other things. Maybe that's
 already happening. I don't know. So here the spatial arrangement is to
 either be close to something or to be between two things. So if the attention
 is on two things you want in between. So you see the top ones are the
 demonstrations. It needs to recognize the concept and it needs to basically
 optimize to fulfill that concept. Shapes. So to make shapes is...
 Oh yeah there's a triangle. Again this very much I believe
 relies on recognition and not actual understanding of what a triangle is. Here
 you have proximity being closer being far apart. What else is cool? Oh yeah you
 have the recognition for the same task. You need to identify the ball that
 is closer. Here you really also see the optimization procedure in action.
 Where for example at the beginning of each flicker you kind of see the
 attention being everywhere and then stabilizing to one or two points. So if
 two points are equally close or far apart you'll see the attention being on
 multiple points. Which is pretty cool right? So that means the model really
 learns this concept. Here's the count quantity. So you can either have one
 two or larger than three or something. Yeah that seems like they tried three
 and four and didn't work so they just said we'll just do larger than three.
 And here is this robot thing where it also always needs to move in between.
 Now this is the part that I'm not really impressed with but you know
 whatever you want. Okay I hope this was a good introduction to energy
 functions. What you can do with them. What I think of them. And of this paper it is
 a pretty cool paper. Yes it only works on toy problems so far but I believe this
 is one interesting direction of future machine learning and something yet to be
 very much explored. If you like this content please subscribe, tell all of
 your friends about it, share and I'll see you next time. Bye bye!