Dynamics-Aware Unsupervised Discovery of Skills (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl deep rl control planning world model dads skills latent high level unsupervised tree search deep reinforcement learning mujoco ant google
This RL framework can discover low-level skills all by itself without any reward. Even better, at test time it can compose its learned skills and reach a specified goal without any additional learning! Warning: Math-heavy!

OUTLINE:
0:00 - Motivation
2:15 - High-Level Overview
3:20 - Model-Based vs Model-Free Reinforcement Learning
9:00 - Skills
12:10 - Mutual Information Objective
18:40 - Decomposition of the Objective
27:10 - Unsupervised Skill Discovery Algorithm
42:20 - Planning in Skill Space
48:10 - Conclusion

Paper: https://arxiv.org/abs/1907.01657
Website: https://sites.google.com/view/dads-skill
Code: https://github.com/google-research/dads

Abstract:
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Authors: Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Take a look at this humanoid right here. It walks from one checkpoint to another checkpoint and then to the next checkpoint and so on. And that is its task. It gets reward from walking from checkpoint to checkpoint. Take a look at this ant. This is called the ant. It also walks from checkpoint to checkpoint. Now we've seen a lot of reinforcement learning algorithms in this environment. It's called Mukojo, where you basically teach these little things to walk around. So what's the impressive part here? The impressive part is that at training time, this ant has never ever seen what a checkpoint is and has never gotten any reward from walking from checkpoint to another checkpoint. Actually, it has never gotten any reward for anything that is given from the environment. It has discovered the skill of walking by itself. And then at test time, there is no additional learning when it goes from checkpoint to checkpoint. It simply composes the skills that it knows from its unsupervised discovery phase in order to go from checkpoint to checkpoint. So here you can see this paper basically proposes to learn these skills in a completely unsupervised way at the beginning, sort of, so in the training phase, it learns these skills, you can see these skills that the humanoid has learned. And then all you have to do at test time is to compose these skills to reach a given goal. And these are the things that the ant has learned. Watch out, this is trippy. You can see it has learned various walks, various ways of walking here. And if you know anything about this environment, it's actually not that easy to make the ant walk by itself. So the discovery here that these skills that are discovered are various ways of walking is actually already pretty impressive. And the last thing here, this cheetah, of course, also has to has learned to walk back, forward, going to jump around, and so on. So we're going to dive into this paper. It's called Dynamics Aware, Unsupervised Discovery of Skills by Archie Sharma and other people of Google Brain. So this was published at iClear 2020. And on a high level, I already said it's basically proposing to learn unsupervised skills, and then to compose these skills in a model based planning method at test time to reach a given goal without additional training, without additional training on the reward that you give at test time. As always, if you like videos like this, you're very welcome to subscribe and share it to everyone you know. Yeah, okay, that's live in. So they say, conventionally, model based reinforcement learning aims to learn a global model for the dynamics of the environment, which is not exactly true, right. So we have we dive into model based and model free reinforcement learning. Model based reinforcement learning basically means that you have a model of the environment. A example for this is, let's say, tic tac toe. So in tic tac toe, I know, like I have nine actions at my disposal. And if I take action, let's say I take action zero, which is to make a let's say I'm the X player, so I take action zero. And if I you know, number my things correctly, then that will result in this state of the world. Okay, so I know exactly how the world will look when I take a given action. And what that allows me to do is that allows me to actually plan. So I can now plan ahead, I can say what would happen if I took action zero, so I can do this in my mind. And then what would happen if I took action one, I can be like, okay, that's going to happen. And I can do this with many things. And I can, in my mind, continue this and basically roll out the entire games or and then only do the given action that has led to the best result at the end, right. So this is model model based reinforcement learning means you have a model of the environment, you know, what's going to happen when you perform given actions. And you can also combine this with machine learning, like, you know, alpha, alpha go alpha zero, or so they have models of the games they're playing, they know what's going to happen. But it's very intractable to basically go down this entire tree and plan out everything. So they combine it with machine learning. It doesn't change that it's model based. Now in, in opposition to that in model three, in in reinforcement learning, what you do, you are this agent, there's the environment, and you simply have to do an action, I do action zero, and the environment just gives you back a reward and the next observation. And it you have basically no clue how the environment will change. If you do something, all these and all these, these agents do or the classic model free agents do is basically they're trying to have a neural network somewhere in them. And you put the observation in here and outcomes in action. And you can do this in various ways into Q learning or policy gradient or actor critic and so on. But ultimately, it's simply mapping the up the current observation and maybe the last few to the best action to take without explicitly modeling what happens in the environment. Now when they say model based reinforcement learning, what they mean is technically what you can do if you're in if you are in the model free, if you're in this situation, what you could do is you could say, well, since these model based RL techniques tend to work better, I could hear inside the agent, I could try to learn a model of the environment, E prime, and I could try to basically learn what happens in my environment when I do a certain action. And then I could use that model right here, in order to do this planning that I know from up here. Okay, so in this case, they go for exactly this they go for, let's learn a model of the environment, this is not an exact model, it's a learned model. And then let's use that to plan. Now this usually has a bunch of, you know, very a bunch of things that go against it, namely, if this model right here is bad, then the planning in the model will often accumulate and even exaggerate the errors that are in this model. So it's sometimes very hard to learn a model of the world and then use that for planning. And I've recently done a paper where, where curious AI takes the noise in auto encoders to regularize exactly such a planning procedure to counter this. And this paper right here is a different approach of combining this learned model. This learned model. So, okay. That was about the first sentence. They say it aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks, which is true, right? If I have a model of the environment, then I could just use it to plan. I wouldn't even have to do anything fancy anymore, right? If I have a model of how my tic tac toe works, I can just plan, plan my way to success. And I can do this all for zero style. Or if the if this state tree is small enough, I can actually just use a planner. And I don't even have to, I don't have to do anything anymore. If I have a good model. They say however, learning an accurate model for complex dynamical systems is difficult. And even then the model might not generalize well outside the distribution of states on which it was trained. So this is another problem. If you learn a model, it's only going to be valid in a certain range. Okay. And say in this work, we combine model based learning with model free learning of the model free learning is of primitives that make model based planning easy. So what they attempt to do is they attempt in an unsupervised fashion to learn so called set of skills. And the set of skills could be something like walk forward, walk backward, stay put, jump. So they attempt to learn things like this in a model free way, that the model is simply asked to come up with these things or the agent is simply asked to come up with these things. And then in in stage two, a planner can use these skills and decompose a plan. Now this plan here, the special thing about this plan, this planner, it doesn't operate in the space of actions of like small scale actions, it actually operates in the space of these skills right here. So here would be walk forward, walk back. So and with if we have a good enough model of the environment, it will tell us if I walk forward in this situation, what will happen? Okay, so I can walk forward. And then after that, I could walk backward, what's going to happen? And if I have a good model of the environment over the and the actions now are these macro actions of these skills, then I can use planning to reach my goal. Okay, so the question is, how do we come up with useful skills that the planner can then use? So they need to be somewhat diverse, right? But also, and here is the the crucial part and the the sort of contribution of this paper, they say, how can we discover skills whose outcomes are easy to predict? And this is how they counteract this notion here, that if your environment model is crap, then you're you're, it can't basically can't be used for planning, you'll just make it worse. So what they say is that these things right here, these skills that we learn, we will learn them in a way that make them easy to predict. So it makes it easy to predict what will happen after I do them. So they must be at the same time diverse. So only you know, if you stay put, it's pretty easy to predict what's going to happen like nothing. Okay. But we're going to see in the exact objective that they have to be sort of diverse. But also, so only one of them can be stay put, the other ones have to do something else. But also, they should be easily predictable by the environment model. And if you learn the skills such that they're easily predictable, your environment model will make less errors, and then you can use it for planning. Okay, let's dive in. They do actually open source code, and they have more of these videos, if you want to check it out, I'll link everything in the description. Okay, let's actually dive into the meat right here. They say they do maximize the mutual information. And we're going to see between between what and what they want to maximize the mutual information. If you don't know what the mutual information is, the mutual information is a quantity. The mutual information between x and y is a quantity, that's the entropy of x minus the entropy of x conditioned on y, or you can also decompose it in the other way around entropy of y minus entropy of y conditioned on x. And we're going to they apply this, where is it equation two right here. Okay, so what they want is the following, they want to maximize the mutual information. And the mutual information basically means how much does one variable tell me about the other variable, okay? The mutual information between the skill, and we're going to see what that is, and the next state. So what is a skill? A skill is one entry in our table right here. So these here are skills, skills. And they're in their index by z. Now z here, it seems like they're discrete, right? But in this case, they would be z would be a continuous vector. But it's easier if you imagine that so each one of this is like z one, z two, and so on. They're going to be continuous, but in our case, we'll just think of a discrete set of skills to be learned. Okay, so they say we want to maximize the mutual information between the skill that's currently in action. So in every, every time the agent has to like choose a skill and saying like, okay, now I'm going to walk forward. And it's in a given state. And what you want to say is you have to maximize the mutual information between the skill and the next state, which means that it means two things, which you can see right here, you can decompose it in two different ways. One way is the following is to say, if I know which state I'm in, what's the entropy over my? Okay, that's the wrong bad way for me leading it. It's if I knew, if I know these two things, so if I know the state I'm in, and the next state that I'm going to write, I can in hindsight, I look back, if I know both things, what can I say about this skill z right here, that I couldn't say just from the starting state. So the starting state, let's say is the person right here, and the end state is the person a little bit more over there. So that's, that's called this forward, it's the person is looking to the right. Okay. Now, if I only show you if I only show you the state on the left, the starting state, what can you tell me which action is going to follow right here? Basically, you can tell me much it could be any action like walk forward, walk back, stay put. But if I also show you the next state, then you can pretty confidently say, ah, I know what you did, you did walk forward. Okay, so this, this, in this situation, we would have a high mutual information between z and s prime in the formulation here, if you decompose it in the other way, this is equivalent, but it's a different way of thinking about it, it means when I show you these two things, how much more can you tell me about the next state than if I only show you this. So in this formulation, what we would do is, I would say I tell you the human is here looking to the right, what can you tell me about the next state? And you like what I, I couldn't tell you very much, right? It can be anything. But if I then tell you the action is walk forward, then you could say, ah, now I now I get it, it's probably going to be something like this. Okay. This also would be a high mutual information. So you see that the the task of maximizing this mutual information is good, because what happens if I don't, if I have a low mutual information, if I have a low mutual information, it would mean that I could predict the next state just as well from the current state. It doesn't make a difference whether you give me the skill or not, it would not make a difference. And that is only the case if my skills are basically either all the same or all pretty, pretty random and pretty useless. Right? So if if all my skills are basically walking backwards, then I don't, you don't have to tell me which skill you do, I'm going to know that the next state is like this. So you can see that the objective of maximizing the mutual information between the skill and the next state is going to result in a situation where these skills are going to be first of all, diverse. And second of all, easy to easy to predict. Okay. And to see this, yeah, we only have to imagine what would happen, what would happen in a in a situation where the skills weren't diverse or weren't easy to predict. And you'll get exactly the situation where the information of the skill doesn't help you in predicting the next state, because yeah, either it's obvious or it's random. Okay, so we agree that it makes sense to maximize the mutual information. And they decompose this into two objectives. So they say the mutual information will stay is basically what you'll have to do is you'll have to it decomposes into two terms where you can into two terms in a lower bound in the mutual information. And this is kind of the this is sort of the standard variational approximation literature. If you're into that read up on variational auto encoders and things like this. Basically, the two steps here are you tighten the variational lower bound and you maximize the approximate lower bound. Okay, so you have the mutual information, and you can lower bound it. Okay, you can you can lower bound it by this quantity. Now the if since this is a lower bound, you can prove that this is a lower bound, it means that the higher you make this term, then the the more basically, okay, if it's a lower, I don't know how to formulate, but it should be fairly obvious if this if this thing on the right is a lower bound to the mutual information, then maximizing the thing on the right will maximize the thing on the left. And it will do so very so imagine I is up here. And this E on the right side is down here, it's a lower bound, right? It's lower than I. So if I maximize E, well, I haven't really done anything to I but if I maximize it even more up to here, and since it's a lower bound, I know now my I must be at least higher than this. Okay, so maximizing a lower bound to a quantity will ultimately increase the quantity. But you can also the efficiency by which it does this depends on how tight the bound is. So if the bound is very tight, like this, you see I is not much above E. If the bound is very tight, then maximizing E will result in a faster maximization of I. Okay, so you can do two things, you can maximize the quantity that is the lower bound, or you can tighten the bound. And here we can see that the difference that the tightness of the bound depends on this quantity right here, which is the KL difference between this and this. So yeah, let's watch this in the context of we can actually go go through it on the high level. If you've never done this variational approximation sort of math, then this might be a bit informative. Okay, so the thing right here is just pops out of the definition of the mutual information. It's the it's basically the differences of the entropy's, which the entropy's are log quantities, right? So if you have a log A minus log B, that you can also write this as the log of the fraction of A over B, that's just a property of the log. And so it's expectations over logs, these entropy's. So you can write it as this thing right here. Okay. And this basically says, this is very high, if or very low, depending so you need to whether or something is lower high always will depend on what you exactly you have to consider. But ultimately, what you'll want is the ratio between this quantity, which is the probability of the next state given the current state and the skill you're taking divided by just the probability over the next state, given the the current state, in expectation over all the skills, current states and next states. Now what they're saying is, this here, this is basically the environment, right? This is if you are in a state and you perform a skill, what's the next state? That's the true environment. P here is the true environment, which we don't know, right? We don't know what the environment's going to do. But we would like to learn a model for the environment. And this model for the environment is now Q, Q, theta, phi, theta, phi, Greek letter. So Q phi here is going to be a neural network that will approximate the environment. And in this probabilistic framework, it is going to be a learned distribution that will approximate the distribution of P. So we approximate it by this. But now this is here it says equal equal, right? This is not equal, because this is just an approximation. So the equality must become must be basically compensated for by this term right here, you can see this here is expanded into these two, you can go through the exact definitions and see why this is an equality. But basically, you can say that the mutual information is this expectation, or it is this expectation. But now you have to correct for the fact that here you only have an approximation. And you have to correct for the fact by exactly the amount by which the approximation is different than the quantity that you're approximating. And this is this key KL divergence right here. So the KL divergence basically measures how different two distributions are. It's sort of a distance, not exactly distance, but sort of a distance between these two distributions right here, it says, here's the real world. And here is your estimate of the real world, how much do they disagree, and that quantity plus, then you can replace your world, the exact world distribution by your approximate distribution. And you still are equal to the mutual information. And now the basically the trick is you say, oh, the KL divergence is always positive, it's a quantity that it can only be a positive number. So if I leave it away, certainly, this is only this is going to be a lower bound to the quantity. Okay. All right. So two tasks right here. First of all, tighten the variational bound, which means make this quantity small, make your approximate world model as close as possible to the real world. How do we do this neural network? Okay, you input trajectories. I was in this state, I performed this skill, and I ended up in this state. Sorry, that's this and then you simply match your neural network simply matches what happens in the real world, it learns the transition function, basically. So that's, that's the tightening of the variational bound. And the second step is this right here to to maximize the approximate lower bound, right? The first step was Titan variation lower bound, that basically means make your world model more accurate. And the second is Titan that maximize the approximate lower bound. Now this is going to part, this is going to be the part that says, now given that I already have a better world model right here, can I improve my can I sort of improve my skills such that they become easier to predict and more diverse? Can I improve my skills such that this mutual information right here gets to be high as high as possible? Okay. So this is sort of an alternating thing. And you can see this in this very, very, very, very confusing diagram, honestly. So what are you going to do in this algorithm? First of all, in each episode, you're going to select a skill at random. And as I said, these skills, they're not predefined. So no one tells the agent to walk forward, it simply says, okay, you have like, in a discrete case, you would have like, you have five skill slots, right? And the only thing I require is that they're sort of, you know, consistent over time. So skill one is always going to be sort of the same thing and skill two, but agent, you can basically decide what skill one is, right? But make the skill such that it's predictable and that the different skills are diverse. Okay, so you're going to sample one of the skills, like skill zero or whatnot. And then you're going to do two things. First of all, you're going to learn these skill dynamics, which is you're going to learn your approximate model of the world. Okay. And how do you do that? Basically, here, you're the agent and the agent will. So what does the agent have to do? The agent will take in the skill Z and it will take in the current state of the world and it will output an action. Now, this is the model free part, right? So the agent that somehow has to come up with saying, ah, skill zero, that's walking forward. And in this situation, walking forward means I have to lift my leg or something like this. So you're going to take your skill, you're going to with your agent perform an action based on that skill and the current state of the world. Then the environment is going to give you the next state right here. And from those things, you can now learn your world model. You know, I was in state S, I performed action a but I performed action a based on skills Z. And then I ended up in state S prime. And I can learn a model of the world, right? This is a triple, I can do supervised learning of a world model. Now here, they do probabilistic learning, but and we're going to see in a second how that works. But ultimately, they approximate the world with their model. Cool. So that's the this outer loop. And then what are they going to do next, they're going to use that world model to determine a reward for the agent, and the reward for the agent for taking the action. So the reward is going to be, oh, agent, you took action a. Now, what's your reward for doing this? This is the model free reinforcement learning part, your reward is going to be very high, if if this was very predictable. And if it is also diverse, right, so now, the agent has to sort of max sort of the agent has to go and make this quantity very high this, we want the outcome of these actions to be predictable, and dive and the actions themselves to be diverse. It is, I'm sorry, it's very hard to keep all of this very straight. Okay. But ultimately, two steps, learn world model from the experience that you've generated. And second thing, learn the agent such that it maximizes this this quantity that we've seen before. And you do this via giving the agent a reward that is proportional to the mutual information. And we've already seen that we can approximate the mutual information by by this quantity here. Okay. So learn world model, and make the agent go higher mutual information, two steps. Okay, learn world model is very, very classic, you can say, okay, I need to improve, I need to minimize this KL divergence. So I need the gradient with respect to the parameters of my world model, I can write down the KL divergence like this. And then since I can do this reverse, so log a over b is log a minus log b. And since the world doesn't depend on the parameters of my model, this will simply give me this thing right here, which is the gradient of the log probability basically, of my neural network. And this can be just optimized straightforward, this is a neural network, optimize it with gradient descent. These are the inputs, this is the output. Now, okay, you, this is all probability distributions, but ultimately, you can you can do it pretty straightforward. Okay, so corresponds to maximizing the likelihood of the samples from P under Q. Now, the second step maximize the approximate lower bound. Okay. So after they say after fitting Q, after improving our world model, we can optimize pi pi is the agent that actually takes the actions based on the skill. So it's given a skill, and it needs to perform an action. And it needs to maximize this quantity, as we've seen, needs to maximize the mutual information between if I know the action and if I don't, or the mutual information between the skill and the next state. I say note, this is a reinforcement learning style optimization, with a reward function of this quantity. However, so you look at the quantity that they need right here, the quantity is going to be this thing. And this thing is just, I feed the skill and the state into my world model. And I look what what comes out of the world model. So this I can compute, right? But this thing right here, I can't compute because this is this is what happens in the world when I'm in state s, and I just run my agent over in expectation over all the skills. So this I don't know, they have a log of this is intractable. And then so we approximate the reward function for pi as this thing right here. Now, first, let's look at what this thing is. So the reward of taking action a, and action a is based on skill z, right? So skill z was fed into the agent, the agent comes up with action a, oh, you want me to walk forward in this situation, okay, I'm gonna lift my leg. That's the action. Okay. So the reward for this action, given this skill, and given the current state is going to be what it's going to be very high, if this here is very high. So it's going to be very high if the probability so this s prime is the state you ended up in, right? So after taking the action, you ended up in s prime. So if what does it mean when this quantity is very high? It means that my world model q, that is approximating the world, thinks that this state is very probable if you were in this state and are given the skill z. So this basically means that the neural network can predict with very high accuracy, what's going to happen if you are in this state and are given this skill to perform, right? This is one of the things that we want. Now, what is it divided by? It's divided by this. And you can see here, the z i are other skills. So it is, what does this mean? This is almost the same quantity. It means how well can the same neural network predict the next state if you were given a different skill. So it means if I'm here, and I ended up here, how well can you predict it if I tell you that I walked forward? And here you ask, well, how well can you predict it if I told you you walked backward, if I told you you jumped, if I told you, and so on. So you basically aggregate over all the other skills you could perform. And each time you ask the neural network, well, how likely is it that you end up in the state that I ended up with? So what does it mean if this quantity is high, or sorry, if the entire sum here is high? That means that the skill doesn't really give you much information, the neural network is very good, no matter which skill you selected, right? It's very accurate in predicting the next state doesn't really matter. The skill doesn't really matter. And this is what we don't want, right? We want that the skills are very diverse, right? So the top part is, they're easy, it's easy to predict what will happen if you perform a given skill. And we divide this by the bottom part. And this makes it such that these skills are very diverse, because if they're not diverse, then it doesn't really matter which one you perform. And then this quantity on the bottom will be very high, but we divide by it. So we want, we want it to be low. Okay, now the reward is going to be the log of this fraction here. And this makes sense, right intuitively, but they're going to try to motivate this mathematically. And for motivating this mathematically, of course, they need to approximate this quantity right here. This quantity is the denominator. So they this denominator is a proc is an approximation to this. It's an approximation. As you can see here, this is sort of sort of a sample based approximation to the transition from s to s prime under the distribution of z. But what you want is just is the transition from s to s prime, not in your approximation, but in the real world. And they formulate this, they say, okay, we can decompose it as such as a as an integral over this conditional right here. So they bring in the z variable. And then they say, well, this, this is approximately approximately we can replace this here by this. And we can replace this here by this. They say, well, since the this is a an approximation, the two, this is the world model is an approximation to the real world, we can sort of replace that. And then this is the this is the part that doesn't convince me they say, well, this P z of s, we can just replace it by P z. Now this is it's very tricky to see what these quantities are. Ultimately, it ends up being that right here. But it's it's it's so tricky. So they say we replace P z given s by P of z. And, okay, let's think about this for a second. What does the top the bottom quantity is simply the distribution over your skills. And depending on how you sample them, this could be like a uniform distribution over your skills, like that's fine. But what's the top thing, the top thing, basically, we can use base formula to reformulate it. It's P of s given z times P of right times P of z divided by P of s. So this quantity depends on multiple things. Here's that prior again. And this means what's the general distribution of states? What's the general distribution of states? If your agent acts in the world, right, this? And this, we don't know, we don't we don't know. And also this right here, what's the distribution in the true world? What's the probability of a state given a given given that you were acting under a skill z? And this is also something we don't know, because we don't know the world, we we don't have the world model. So you run into the same problem again and again, that you're trying to approximate this. And they want to make this so mathematically rigorous, but ultimately, and they go in the appendix, they go through various ways that they could solve this. But ultimately, they just say, well, this is approximately the same. So this right here, it basically means what skills, if you're if you're in a certain state, what skills brought you here, basically, what what skills brought you here, what's the distribution of skills that brought you to this state, and they say, well, we're just going to approximate that by the prior distribution over our skills, basically disregard the state here. And this seems overly shaky. Like, as I said, the entire paper makes sense. But I just feel it's trying to be overly mathematical, and then run into a point where you can't be and then they're just like, okay, we'll, we'll just replace it. And then sort of things break down, like, you can only be overly mathematical to some degree, it doesn't really fit. But okay, so this is how you discover the skills, you maximize these quantities, alternately, you learn the world model, and you improve your your skills by making them diverse and easily predictable. So how do you then plan using these skills? This is the second part of the paper, and this is just as complicated as the first part. So they say given the learned skills, so the learned skills are policies over action given the DZ, right? So now you know how to like walk forward and walk back and so on. And now you're, you're placed in a world, and you're given this checkpoint, it says, well, walk there. And you want this to do this using planning, you don't want to learn anymore, you simply want to plan. Okay, what do you do? And as I said, this is even more so what you want to do is you want to do something like model predictive control, but not over actions, but over your learned skills. So you have this planner in the NPC, and the planner will in its head roll out a number of different, a number of different plans, it will kind of explore a bunch of different, different plans Z, it will roll them out, I'll say, okay, if I do this, and this, and this, and this, what will happen using its world model that it has learned, it will observe what's going to be the reward in each of these cases. Now, they say here, access the environment reward, but can also be estimated. And this is another sort of, I feel weak point of this in that they now assume they have the true reward function. But they don't have a world model, right? They don't have the world model, but they assume that they can sort of always ask for the true reward, which is probably not the case if you had a true world, but it could be the case, the reward could be something like, well, if you're over there, you get higher reward, but you don't exactly know how to get over there. In any case, so they roll out a bunch of trajectories in their head, they kind of plan forward, see what's going to happen if they do this or that, or this or that. And then they choose the best one of these forward thoughts, and they execute it in the real world, right? So they say, well, I'm going to use choose the skill walk forward. So the agent is now going to be tasked with walking forward. And it's going to do that in the real world for a certain amount of steps, like 10 steps of walking forward, after 10 steps of walking forward, you go back and say, I'm in this new situation right here, what should I do? And again, the planner is going to be like, if you first walk forward, and then walk back, where are you going to be, and so on. So the planner will always plan, basically to go from where you are to the checkpoint using a composition of the skills that you have learned. So the planner may be fine. Okay, if I first walk forward, walk back a bit, and so on, I'm going to get to the goal, I'm going to reach the goal. Now please agent execute this first thing, walk forward, the agent executes it, and maybe it won't, you know, it won't do as well, it will maybe end up here. And then it says, well, I'm here now, please plan again. So it plans again, it says, okay, I can still kind of walk back, I'll be here, here, but then I have to do something else. So now walk back. And okay, so this is what's going to happen. But it is going to happen in a weird way. Namely, what we keep are normal, since everything is continuous, we'll keep normal distributions of all our future steps. So we don't say, okay, I go here, and then I go here. What you'll say is I approximately go here. And after that, I'll approximately go here. And you will do it in such a way that the peak of this normal distribution is going to be the highest, where you think you will get the most reward. If you follow this trajectory, like if you follow this trajectory, you get a very high reward. And if I follow a trajectory that maybe goes here, I won't get a high reward. If it actually turns out in your imagination that you do get a high reward for this trajectory, you'll change this distribution, such that the peak is here. And of course, the tighter the peak is, the more sure you are. So you sort of are looking, if you look out into the world, you want the closest steps to be very peaky. And then as you look out, they can be more sort of broad. And that's how you plan ahead, you keep doing a step. So if you go from here to finally you choose, I want to go here, where the tip is the highest, go here, then you imagine forward again, you refine these distributions over the future. And then you take the next step that gets you to the where the highest peak is right here, basically. And so on. This is simply planning in a continuous domain, it is pretty analogous to how you would plan in like, alpha go, if you or tic tac toe, if you had a planner. But since everything's continuous, it makes it just so much harder. So they yeah, they always update these distributions, as you can see here, to the skill that gave you a high reward in your imagination, compared to the rewards of the other plans that you had. Okay, well, this was a long, long way until we got here. But if you recap, so first, they in an unsupervised fashion, learn these low level skills, such that they're easily predictable by their own world model, and diverse. And then in the second step, they can use that to, to do basically planning. So they first learn these skills, and then the planner composes them to make the agent do something. And again, the agent will never have to learn how to do this go from checkpoint to checkpoint, because the planner can just compose these low level skills. So they have these experiments right here. And we won't go through the experiment because this video is already very, very long. But they basically show that they they're learned things, actually, their learned skills do end up being very diverse, do end up predictable, have a high variance, and so on, they have to give certain priors to it to make it actually work in a real setting. But the results you can actually see in these videos and in the graphs, I'm about you to check out the paper if you're still here. Thanks for being here, I hope this this work was one of the most more complicated and mathy papers we looked at. But I think I still think it's fun. And I still think the outcome is pretty impressive right here, how you can use math to derive basically these intuitive, very intuitive objectives to learn. It's also pretty cool. Alright, that was it for me. And bye bye.
 Hi there! Take a look at this humanoid right here. It walks from one checkpoint to another
 checkpoint and then to the next checkpoint and so on. And that is its task. It gets reward
 from walking from checkpoint to checkpoint. Take a look at this ant. This is called the
 ant. It also walks from checkpoint to checkpoint. Now we've seen a lot of reinforcement learning
 algorithms in this environment. It's called Mukojo, where you basically teach these little
 things to walk around. So what's the impressive part here? The impressive part is that at
 training time, this ant has never ever seen what a checkpoint is and has never gotten
 any reward from walking from checkpoint to another checkpoint. Actually, it has never
 gotten any reward for anything that is given from the environment. It has discovered the
 skill of walking by itself. And then at test time, there is no additional learning when
 it goes from checkpoint to checkpoint. It simply composes the skills that it knows from
 its unsupervised discovery phase in order to go from checkpoint to checkpoint. So here
 you can see this paper basically proposes to learn these skills in a completely unsupervised
 way at the beginning, sort of, so in the training phase, it learns these skills, you can see
 these skills that the humanoid has learned. And then all you have to do at test time is
 to compose these skills to reach a given goal. And these are the things that the ant has
 learned. Watch out, this is trippy. You can see it has learned various walks, various
 ways of walking here. And if you know anything about this environment, it's actually not
 that easy to make the ant walk by itself. So the discovery here that these skills that
 are discovered are various ways of walking is actually already pretty impressive. And
 the last thing here, this cheetah, of course, also has to has learned to walk back, forward,
 going to jump around, and so on. So we're going to dive into this paper. It's called
 Dynamics Aware, Unsupervised Discovery of Skills by Archie Sharma and other people of
 Google Brain. So this was published at iClear 2020. And on a high level, I already said
 it's basically proposing to learn unsupervised skills, and then to compose these skills in
 a model based planning method at test time to reach a given goal without additional training,
 without additional training on the reward that you give at test time. As always, if
 you like videos like this, you're very welcome to subscribe and share it to everyone you
 know. Yeah, okay, that's live in. So they say, conventionally, model based reinforcement
 learning aims to learn a global model for the dynamics of the environment, which is
 not exactly true, right. So we have we dive into model based and model free reinforcement
 learning. Model based reinforcement learning basically means that you have a model of the
 environment. A example for this is, let's say, tic tac toe. So in tic tac toe, I know,
 like I have nine actions at my disposal. And if I take action, let's say I take action
 zero, which is to make a let's say I'm the X player, so I take action zero. And if I
 you know, number my things correctly, then that will result in this state of the world.
 Okay, so I know exactly how the world will look when I take a given action. And what
 that allows me to do is that allows me to actually plan. So I can now plan ahead, I
 can say what would happen if I took action zero, so I can do this in my mind. And then
 what would happen if I took action one, I can be like, okay, that's going to happen.
 And I can do this with many things. And I can, in my mind, continue this and basically
 roll out the entire games or and then only do the given action that has led to the best
 result at the end, right. So this is model model based reinforcement learning means you
 have a model of the environment, you know, what's going to happen when you perform given
 actions. And you can also combine this with machine learning, like, you know, alpha, alpha
 go alpha zero, or so they have models of the games they're playing, they know what's going
 to happen. But it's very intractable to basically go down this entire tree and plan out everything.
 So they combine it with machine learning. It doesn't change that it's model based. Now
 in, in opposition to that in model three, in in reinforcement learning, what you do,
 you are this agent, there's the environment, and you simply have to do an action, I do
 action zero, and the environment just gives you back a reward and the next observation.
 And it you have basically no clue how the environment will change. If you do something,
 all these and all these, these agents do or the classic model free agents do is basically
 they're trying to have a neural network somewhere in them. And you put the observation in here
 and outcomes in action. And you can do this in various ways into Q learning or policy
 gradient or actor critic and so on. But ultimately, it's simply mapping the up the current observation
 and maybe the last few to the best action to take without explicitly modeling what happens
 in the environment. Now when they say model based reinforcement learning, what they mean
 is technically what you can do if you're in if you are in the model free, if you're in
 this situation, what you could do is you could say, well, since these model based RL techniques
 tend to work better, I could hear inside the agent, I could try to learn a model of the
 environment, E prime, and I could try to basically learn what happens in my environment when
 I do a certain action. And then I could use that model right here, in order to do this
 planning that I know from up here. Okay, so in this case, they go for exactly this they
 go for, let's learn a model of the environment, this is not an exact model, it's a learned
 model. And then let's use that to plan. Now this usually has a bunch of, you know, very
 a bunch of things that go against it, namely, if this model right here is bad, then the
 planning in the model will often accumulate and even exaggerate the errors that are in
 this model. So it's sometimes very hard to learn a model of the world and then use that
 for planning. And I've recently done a paper where, where curious AI takes the noise in
 auto encoders to regularize exactly such a planning procedure to counter this. And this
 paper right here is a different approach of combining this learned model. This learned
 model. So, okay. That was about the first sentence. They say it aims to learn a global
 model for the dynamics of the environment. A good model can potentially enable planning
 algorithms to generate a large variety of behaviors and solve diverse tasks, which is
 true, right? If I have a model of the environment, then I could just use it to plan. I wouldn't
 even have to do anything fancy anymore, right? If I have a model of how my tic tac toe works,
 I can just plan, plan my way to success. And I can do this all for zero style. Or if the
 if this state tree is small enough, I can actually just use a planner. And I don't even
 have to, I don't have to do anything anymore. If I have a good model. They say however,
 learning an accurate model for complex dynamical systems is difficult. And even then the model
 might not generalize well outside the distribution of states on which it was trained. So this
 is another problem. If you learn a model, it's only going to be valid in a certain range.
 Okay. And say in this work, we combine model based learning with model free learning of
 the model free learning is of primitives that make model based planning easy. So what they
 attempt to do is they attempt in an unsupervised fashion to learn so called set of skills.
 And the set of skills could be something like walk forward, walk backward, stay put, jump.
 So they attempt to learn things like this in a model free way, that the model is simply
 asked to come up with these things or the agent is simply asked to come up with these
 things. And then in in stage two, a planner can use these skills and decompose a plan.
 Now this plan here, the special thing about this plan, this planner, it doesn't operate
 in the space of actions of like small scale actions, it actually operates in the space
 of these skills right here. So here would be walk forward, walk back. So and with if
 we have a good enough model of the environment, it will tell us if I walk forward in this
 situation, what will happen? Okay, so I can walk forward. And then after that, I could
 walk backward, what's going to happen? And if I have a good model of the environment
 over the and the actions now are these macro actions of these skills, then I can use planning
 to reach my goal. Okay, so the question is, how do we come up with useful skills that
 the planner can then use? So they need to be somewhat diverse, right? But also, and
 here is the the crucial part and the the sort of contribution of this paper, they say, how
 can we discover skills whose outcomes are easy to predict? And this is how they counteract
 this notion here, that if your environment model is crap, then you're you're, it can't
 basically can't be used for planning, you'll just make it worse. So what they say is that
 these things right here, these skills that we learn, we will learn them in a way that
 make them easy to predict. So it makes it easy to predict what will happen after I do
 them. So they must be at the same time diverse. So only you know, if you stay put, it's pretty
 easy to predict what's going to happen like nothing. Okay. But we're going to see in the
 exact objective that they have to be sort of diverse. But also, so only one of them
 can be stay put, the other ones have to do something else. But also, they should be easily
 predictable by the environment model. And if you learn the skills such that they're
 easily predictable, your environment model will make less errors, and then you can use
 it for planning. Okay, let's dive in. They do actually open source code, and they have
 more of these videos, if you want to check it out, I'll link everything in the description.
 Okay, let's actually dive into the meat right here. They say they do maximize the mutual
 information. And we're going to see between between what and what they want to maximize
 the mutual information. If you don't know what the mutual information is, the mutual
 information is a quantity. The mutual information between x and y is a quantity, that's the
 entropy of x minus the entropy of x conditioned on y, or you can also decompose it in the
 other way around entropy of y minus entropy of y conditioned on x. And we're going to
 they apply this, where is it equation two right here. Okay, so what they want is the
 following, they want to maximize the mutual information. And the mutual information basically
 means how much does one variable tell me about the other variable, okay? The mutual information
 between the skill, and we're going to see what that is, and the next state. So what
 is a skill? A skill is one entry in our table right here. So these here are skills, skills.
 And they're in their index by z. Now z here, it seems like they're discrete, right? But
 in this case, they would be z would be a continuous vector. But it's easier if you imagine that
 so each one of this is like z one, z two, and so on. They're going to be continuous,
 but in our case, we'll just think of a discrete set of skills to be learned. Okay, so they
 say we want to maximize the mutual information between the skill that's currently in action.
 So in every, every time the agent has to like choose a skill and saying like, okay, now
 I'm going to walk forward. And it's in a given state. And what you want to say is you have
 to maximize the mutual information between the skill and the next state, which means
 that it means two things, which you can see right here, you can decompose it in two different
 ways. One way is the following is to say, if I know which state I'm in, what's the entropy
 over my? Okay, that's the wrong bad way for me leading it. It's if I knew, if I know these
 two things, so if I know the state I'm in, and the next state that I'm going to write,
 I can in hindsight, I look back, if I know both things, what can I say about this skill
 z right here, that I couldn't say just from the starting state. So the starting state,
 let's say is the person right here, and the end state is the person a little bit more
 over there. So that's, that's called this forward, it's the person is looking to the
 right. Okay. Now, if I only show you if I only show you the state on the left, the starting
 state, what can you tell me which action is going to follow right here? Basically, you
 can tell me much it could be any action like walk forward, walk back, stay put. But if
 I also show you the next state, then you can pretty confidently say, ah, I know what you
 did, you did walk forward. Okay, so this, this, in this situation, we would have a high
 mutual information between z and s prime in the formulation here, if you decompose it
 in the other way, this is equivalent, but it's a different way of thinking about it,
 it means when I show you these two things, how much more can you tell me about the next
 state than if I only show you this. So in this formulation, what we would do is, I would
 say I tell you the human is here looking to the right, what can you tell me about the
 next state? And you like what I, I couldn't tell you very much, right? It can be anything.
 But if I then tell you the action is walk forward, then you could say, ah, now I now
 I get it, it's probably going to be something like this. Okay. This also would be a high
 mutual information. So you see that the the task of maximizing this mutual information
 is good, because what happens if I don't, if I have a low mutual information, if I have
 a low mutual information, it would mean that I could predict the next state just as well
 from the current state. It doesn't make a difference whether you give me the skill or
 not, it would not make a difference. And that is only the case if my skills are basically
 either all the same or all pretty, pretty random and pretty useless. Right? So if if
 all my skills are basically walking backwards, then I don't, you don't have to tell me which
 skill you do, I'm going to know that the next state is like this. So you can see that the
 objective of maximizing the mutual information between the skill and the next state is going
 to result in a situation where these skills are going to be first of all, diverse. And
 second of all, easy to easy to predict. Okay. And to see this, yeah, we only have to imagine
 what would happen, what would happen in a in a situation where the skills weren't diverse
 or weren't easy to predict. And you'll get exactly the situation where the information
 of the skill doesn't help you in predicting the next state, because yeah, either it's
 obvious or it's random. Okay, so we agree that it makes sense to maximize the mutual
 information. And they decompose this into two objectives. So they say the mutual information
 will stay is basically what you'll have to do is you'll have to it decomposes into two
 terms where you can into two terms in a lower bound in the mutual information. And this
 is kind of the this is sort of the standard variational approximation literature. If you're
 into that read up on variational auto encoders and things like this. Basically, the two steps
 here are you tighten the variational lower bound and you maximize the approximate lower
 bound. Okay, so you have the mutual information, and you can lower bound it. Okay, you can
 you can lower bound it by this quantity. Now the if since this is a lower bound, you can
 prove that this is a lower bound, it means that the higher you make this term, then the
 the more basically, okay, if it's a lower, I don't know how to formulate, but it should
 be fairly obvious if this if this thing on the right is a lower bound to the mutual information,
 then maximizing the thing on the right will maximize the thing on the left. And it will
 do so very so imagine I is up here. And this E on the right side is down here, it's a lower
 bound, right? It's lower than I. So if I maximize E, well, I haven't really done anything to
 I but if I maximize it even more up to here, and since it's a lower bound, I know now my
 I must be at least higher than this. Okay, so maximizing a lower bound to a quantity
 will ultimately increase the quantity. But you can also the efficiency by which it does
 this depends on how tight the bound is. So if the bound is very tight, like this, you
 see I is not much above E. If the bound is very tight, then maximizing E will result
 in a faster maximization of I. Okay, so you can do two things, you can maximize the quantity
 that is the lower bound, or you can tighten the bound. And here we can see that the difference
 that the tightness of the bound depends on this quantity right here, which is the KL
 difference between this and this. So yeah, let's watch this in the context of we can
 actually go go through it on the high level. If you've never done this variational approximation
 sort of math, then this might be a bit informative. Okay, so the thing right here is just pops
 out of the definition of the mutual information. It's the it's basically the differences of
 the entropy's, which the entropy's are log quantities, right? So if you have a log A
 minus log B, that you can also write this as the log of the fraction of A over B, that's
 just a property of the log. And so it's expectations over logs, these entropy's. So you can write
 it as this thing right here. Okay. And this basically says, this is very high, if or very
 low, depending so you need to whether or something is lower high always will depend on what you
 exactly you have to consider. But ultimately, what you'll want is the ratio between this
 quantity, which is the probability of the next state given the current state and the
 skill you're taking divided by just the probability over the next state, given the the current
 state, in expectation over all the skills, current states and next states. Now what they're
 saying is, this here, this is basically the environment, right? This is if you are in
 a state and you perform a skill, what's the next state? That's the true environment. P
 here is the true environment, which we don't know, right? We don't know what the environment's
 going to do. But we would like to learn a model for the environment. And this model
 for the environment is now Q, Q, theta, phi, theta, phi, Greek letter. So Q phi here is
 going to be a neural network that will approximate the environment. And in this probabilistic
 framework, it is going to be a learned distribution that will approximate the distribution of
 P. So we approximate it by this. But now this is here it says equal equal, right? This is
 not equal, because this is just an approximation. So the equality must become must be basically
 compensated for by this term right here, you can see this here is expanded into these two,
 you can go through the exact definitions and see why this is an equality. But basically,
 you can say that the mutual information is this expectation, or it is this expectation.
 But now you have to correct for the fact that here you only have an approximation. And you
 have to correct for the fact by exactly the amount by which the approximation is different
 than the quantity that you're approximating. And this is this key KL divergence right here.
 So the KL divergence basically measures how different two distributions are. It's sort
 of a distance, not exactly distance, but sort of a distance between these two distributions
 right here, it says, here's the real world. And here is your estimate of the real world,
 how much do they disagree, and that quantity plus, then you can replace your world, the
 exact world distribution by your approximate distribution. And you still are equal to the
 mutual information. And now the basically the trick is you say, oh, the KL divergence
 is always positive, it's a quantity that it can only be a positive number. So if I leave
 it away, certainly, this is only this is going to be a lower bound to the quantity. Okay.
 All right. So two tasks right here. First of all, tighten the variational bound, which
 means make this quantity small, make your approximate world model as close as possible
 to the real world. How do we do this neural network? Okay, you input trajectories. I was
 in this state, I performed this skill, and I ended up in this state. Sorry, that's this
 and then you simply match your neural network simply matches what happens in the real world,
 it learns the transition function, basically. So that's, that's the tightening of the variational
 bound. And the second step is this right here to
 to maximize the approximate lower bound, right? The first step was Titan variation lower bound,
 that basically means make your world model more accurate. And the second is Titan that
 maximize the approximate lower bound. Now this is going to part, this is going to be
 the part that says, now given that I already have a better world model right here, can
 I improve my can I sort of improve my skills such that they become easier to predict and
 more diverse? Can I improve my skills such that this mutual information right here gets
 to be high as high as possible? Okay. So this is sort of an alternating thing. And you can
 see this in this very, very, very, very confusing diagram, honestly. So what are you going to
 do in this algorithm? First of all, in each episode, you're going to select a skill at
 random. And as I said, these skills, they're not predefined. So no one tells the agent
 to walk forward, it simply says, okay, you have like, in a discrete case, you would have
 like, you have five skill slots, right? And the only thing I require is that they're sort
 of, you know, consistent over time. So skill one is always going to be sort of the same
 thing and skill two, but agent, you can basically decide what skill one is, right? But make
 the skill such that it's predictable and that the different skills are diverse. Okay, so
 you're going to sample one of the skills, like skill zero or whatnot. And then you're
 going to do two things. First of all, you're going to learn these skill dynamics, which
 is you're going to learn your approximate model of the world. Okay. And how do you do
 that? Basically, here, you're the agent and the agent will. So what does the agent have
 to do? The agent will take in the skill Z and it will take in the current state of the
 world and it will output an action. Now, this is the model free part, right? So the agent
 that somehow has to come up with saying, ah, skill zero, that's walking forward. And in
 this situation, walking forward means I have to lift my leg or something like this. So
 you're going to take your skill, you're going to with your agent perform an action based
 on that skill and the current state of the world. Then the environment is going to give
 you the next state right here. And from those things, you can now learn your world model.
 You know, I was in state S, I performed action a but I performed action a based on skills
 Z. And then I ended up in state S prime. And I can learn a model of the world, right? This
 is a triple, I can do supervised learning of a world model. Now here, they do probabilistic
 learning, but and we're going to see in a second how that works. But ultimately, they
 approximate the world with their model. Cool. So that's the this outer loop. And then what
 are they going to do next, they're going to use that world model to determine a reward
 for the agent, and the reward for the agent for taking the action. So the reward is going
 to be, oh, agent, you took action a. Now, what's your reward for doing this? This is
 the model free reinforcement learning part, your reward is going to be very high, if if
 this was very predictable. And if it is also diverse, right, so now, the agent has to sort
 of max sort of the agent has to go and make this quantity very high this, we want the
 outcome of these actions to be predictable, and dive and the actions themselves to be
 diverse. It is, I'm sorry, it's very hard to keep all of this very straight. Okay. But
 ultimately, two steps, learn world model from the experience that you've generated. And
 second thing, learn the agent such that it maximizes this this quantity that we've seen
 before. And you do this via giving the agent a reward that is proportional to the mutual
 information. And we've already seen that we can approximate the mutual information by
 by this quantity here. Okay. So learn world model, and make the agent go higher mutual
 information, two steps. Okay, learn world model is very, very classic, you can say,
 okay, I need to improve, I need to minimize this KL divergence. So I need the gradient
 with respect to the parameters of my world model, I can write down the KL divergence
 like this. And then since I can do this reverse, so log a over b is log a minus log b. And
 since the world doesn't depend on the parameters of my model, this will simply give me this
 thing right here, which is the gradient of the log probability basically, of my neural
 network. And this can be just optimized straightforward, this is a neural network, optimize it with
 gradient descent. These are the inputs, this is the output. Now, okay, you, this is all
 probability distributions, but ultimately, you can you can do it pretty straightforward.
 Okay, so corresponds to maximizing the likelihood of the samples from P under Q. Now, the second
 step maximize the approximate lower bound. Okay. So after they say after fitting Q, after
 improving our world model, we can optimize pi pi is the agent that actually takes the
 actions based on the skill. So it's given a skill, and it needs to perform an action.
 And it needs to maximize this quantity, as we've seen, needs to maximize the mutual information
 between if I know the action and if I don't, or the mutual information between the skill
 and the next state. I say note, this is a reinforcement learning style optimization,
 with a reward function of this quantity. However, so you look at the quantity that they need
 right here, the quantity is going to be this thing. And this thing is just, I feed the
 skill and the state into my world model. And I look what what comes out of the world model.
 So this I can compute, right? But this thing right here, I can't compute because this is
 this is what happens in the world when I'm in state s, and I just run my agent over in
 expectation over all the skills. So this I don't know, they have a log of this is intractable.
 And then so we approximate the reward function for pi as this thing right here. Now, first,
 let's look at what this thing is. So the reward of taking action a, and action a is based
 on skill z, right? So skill z was fed into the agent, the agent comes up with action
 a, oh, you want me to walk forward in this situation, okay, I'm gonna lift my leg. That's
 the action. Okay. So the reward for this action, given this skill, and given the current state
 is going to be what it's going to be very high, if this here is very high. So it's going
 to be very high if the probability so this s prime is the state you ended up in, right?
 So after taking the action, you ended up in s prime. So if what does it mean when this
 quantity is very high? It means that my world model q, that is approximating the world,
 thinks that this state is very probable if you were in this state and are given the skill
 z. So this basically means that the neural network can predict with very high accuracy,
 what's going to happen if you are in this state and are given this skill to perform,
 right? This is one of the things that we want. Now, what is it divided by? It's divided by
 this. And you can see here, the z i are other skills. So it is, what does this mean? This
 is almost the same quantity. It means how well can the same neural network predict the
 next state if you were given a different skill. So it means if I'm here, and I ended up here,
 how well can you predict it if I tell you that I walked forward? And here you ask, well,
 how well can you predict it if I told you you walked backward, if I told you you jumped,
 if I told you, and so on. So you basically aggregate over all the other skills you could
 perform. And each time you ask the neural network, well, how likely is it that you end
 up in the state that I ended up with? So what does it mean if this quantity is high, or
 sorry, if the entire sum here is high? That means that the skill doesn't really give you
 much information, the neural network is very good, no matter which skill you selected,
 right? It's very accurate in predicting the next state doesn't really matter. The skill
 doesn't really matter. And this is what we don't want, right? We want that the skills
 are very diverse, right? So the top part is, they're easy, it's easy to predict what will
 happen if you perform a given skill. And we divide this by the bottom part. And this makes
 it such that these skills are very diverse, because if they're not diverse, then it doesn't
 really matter which one you perform. And then this quantity on the bottom will be very high,
 but we divide by it. So we want, we want it to be low. Okay, now the reward is going to
 be the log of this fraction here. And this makes sense, right intuitively, but they're
 going to try to motivate this mathematically. And for motivating this mathematically, of
 course, they need to approximate this quantity right here. This quantity is the denominator.
 So they this denominator is a proc is an approximation to this. It's an approximation. As you can
 see here, this is sort of sort of a sample based approximation to the transition from
 s to s prime under the distribution of z. But what you want is just is the transition
 from s to s prime, not in your approximation, but in the real world. And they formulate
 this, they say, okay, we can decompose it as such as a as an integral over this conditional
 right here. So they bring in the z variable. And then they say, well, this, this is approximately
 approximately we can replace this here by this. And we can replace this here by this.
 They say, well, since the this is a an approximation, the two, this is the world model is an approximation
 to the real world, we can sort of replace that. And then this is the this is the part
 that doesn't convince me they say, well, this P z of s, we can just replace it by P z. Now
 this is it's very tricky to see what these quantities are. Ultimately, it ends up being
 that right here. But it's it's it's so tricky. So they say we replace P z given s by P of
 z. And, okay, let's think about this for a second. What does the top the bottom quantity
 is simply the distribution over your skills. And depending on how you sample them, this
 could be like a uniform distribution over your skills, like that's fine. But what's the top thing,
 the top thing, basically, we can use base formula to reformulate it. It's P of s given z times
 P of right times P of z divided by P of s. So this quantity depends on multiple things.
 Here's that prior again. And this means what's the general distribution of states? What's
 the general distribution of states? If your agent acts in the world, right, this? And this, we
 don't know, we don't we don't know. And also this right here, what's the distribution in the true
 world? What's the probability of a state given a given given that you were acting under a skill
 z? And this is also something we don't know, because we don't know the world, we we don't have
 the world model. So you run into the same problem again and again, that you're trying to approximate
 this. And they want to make this so mathematically rigorous, but ultimately, and they go in the
 appendix, they go through various ways that they could solve this. But ultimately, they just say,
 well, this is approximately the same. So this right here, it basically means what skills,
 if you're if you're in a certain state, what skills brought you here, basically, what what
 skills brought you here, what's the distribution of skills that brought you to this state, and
 they say, well, we're just going to approximate that by the prior distribution over our skills,
 basically disregard the state here. And this seems overly shaky. Like, as I said, the entire paper
 makes sense. But I just feel it's trying to be overly mathematical, and then run into a point
 where you can't be and then they're just like, okay, we'll, we'll just replace it. And then sort
 of things break down, like, you can only be overly mathematical to some degree, it doesn't really fit.
 But okay, so this is how you discover the skills, you maximize these quantities, alternately,
 you learn the world model, and you improve your your skills by making them diverse and easily
 predictable. So how do you then plan using these skills? This is the second part of the paper,
 and this is just as complicated as the first part. So they say given the learned skills,
 so the learned skills are policies over action given the DZ, right? So now you know how to like
 walk forward and walk back and so on. And now you're, you're placed in a world, and you're given
 this checkpoint, it says, well, walk there. And you want this to do this using planning, you don't
 want to learn anymore, you simply want to plan. Okay, what do you do? And as I said, this is even
 more so what you want to do is you want to do something like model predictive control, but not
 over actions, but over your learned skills. So you have this planner in the NPC, and the planner will
 in its head roll out a number of different, a number of different plans, it will kind of
 explore a bunch of different, different plans Z, it will roll them out, I'll say, okay, if I do this,
 and this, and this, and this, what will happen using its world model that it has learned,
 it will observe what's going to be the reward in each of these cases. Now, they say here, access
 the environment reward, but can also be estimated. And this is another sort of, I feel weak point of
 this in that they now assume they have the true reward function. But they don't have a world model,
 right? They don't have the world model, but they assume that they can sort of always ask for the
 true reward, which is probably not the case if you had a true world, but it could be the case,
 the reward could be something like, well, if you're over there, you get higher reward,
 but you don't exactly know how to get over there. In any case, so they roll out a bunch of trajectories
 in their head, they kind of plan forward, see what's going to happen if they do this or that,
 or this or that. And then they choose the best one of these forward thoughts, and they execute it in
 the real world, right? So they say, well, I'm going to use choose the skill walk forward. So the agent
 is now going to be tasked with walking forward. And it's going to do that in the real world for
 a certain amount of steps, like 10 steps of walking forward, after 10 steps of walking forward,
 you go back and say, I'm in this new situation right here, what should I do? And again, the planner
 is going to be like, if you first walk forward, and then walk back, where are you going to be,
 and so on. So the planner will always plan, basically to go from where you are to the
 checkpoint using a composition of the skills that you have learned. So the planner may be fine. Okay,
 if I first walk forward, walk back a bit, and so on, I'm going to get to the goal, I'm going to
 reach the goal. Now please agent execute this first thing, walk forward, the agent executes it,
 and maybe it won't, you know, it won't do as well, it will maybe end up here. And then it says, well,
 I'm here now, please plan again. So it plans again, it says, okay, I can still kind of walk back,
 I'll be here, here, but then I have to do something else. So now walk back. And okay, so this is what's
 going to happen. But it is going to happen in a weird way. Namely, what we keep are normal,
 since everything is continuous, we'll keep normal distributions of all our future steps. So we don't
 say, okay, I go here, and then I go here. What you'll say is I approximately go here. And after that,
 I'll approximately go here. And you will do it in such a way that the peak of this normal distribution
 is going to be the highest, where you think you will get the most reward. If you follow this
 trajectory, like if you follow this trajectory, you get a very high reward. And if I follow a
 trajectory that maybe goes here, I won't get a high reward. If it actually turns out in your
 imagination that you do get a high reward for this trajectory, you'll change this distribution,
 such that the peak is here. And of course, the tighter the peak is, the more sure you are. So
 you sort of are looking, if you look out into the world, you want the closest steps to be very peaky.
 And then as you look out, they can be more sort of broad. And that's how you plan ahead, you keep
 doing a step. So if you go from here to finally you choose, I want to go here, where the tip is the
 highest, go here, then you imagine forward again, you refine these distributions over the future.
 And then you take the next step that gets you to the where the highest peak is right here, basically.
 And so on. This is simply planning in a continuous domain, it is pretty analogous to how you would
 plan in like, alpha go, if you or tic tac toe, if you had a planner. But since everything's
 continuous, it makes it just so much harder. So they yeah, they always update these distributions,
 as you can see here, to the skill that gave you a high reward in your imagination,
 compared to the rewards of the other plans that you had. Okay, well, this was a long, long way
 until we got here. But if you recap, so first, they in an unsupervised fashion, learn these low
 level skills, such that they're easily predictable by their own world model, and diverse. And then in
 the second step, they can use that to, to do basically planning. So they first learn these
 skills, and then the planner composes them to make the agent do something. And again, the agent will
 never have to learn how to do this go from checkpoint to checkpoint, because the planner
 can just compose these low level skills. So they have these experiments right here. And we won't
 go through the experiment because this video is already very, very long. But they basically show
 that they they're learned things, actually, their learned skills do end up being very diverse,
 do end up predictable, have a high variance, and so on, they have to give certain priors to it
 to make it actually work in a real setting. But the results you can actually see in these videos
 and in the graphs, I'm about you to check out the paper if you're still here. Thanks for being here,
 I hope this this work was one of the most more complicated and mathy papers we looked at. But I
 think I still think it's fun. And I still think the outcome is pretty impressive right here, how you
 can use math to derive basically these intuitive, very intuitive objectives to learn. It's also
 pretty cool. Alright, that was it for me. And bye bye.