Planning to Explore via Self-Supervised World Models (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper rl deep rl deep reinforcement learning novelty curiosity intrinsic reward dreamer planet control walker run forward imaginary imagination planning google neural network actor critic uncertainty information gain mutual information model
What can an agent do without any reward? Explore the world! While many formulations of intrinsic rewards exist (Curiosity, Novelty, etc.), they all look back in time to learn. Plan2Explore is the first model that uses planning in a learned imaginary latent world model to seek out states where it is uncertain about what will happen.

OUTLINE:
0:00 - Intro & Problem Statement
3:30 - Model
5:10 - Intrinsic Motivation
9:05 - Planning in Latent Space
11:15 - Latent Disagreement
16:30 - Maximizing Information Gain
21:00 - More problems with the model
26:45 - Experiments
32:10 - Final Comments

Paper: https://arxiv.org/abs/2005.05960
Website: https://ramanans1.github.io/plan2explore/
Code: https://github.com/ramanans1/plan2explore

Abstract:
Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at this https URL

Authors: Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at planning to explore via self-supervised world models by Ramanan Sekar, Ole Rybkin, Kostas Danilidis, Peter Abil, Danijar Hafner and Depak Patak. So this is a paper that concerns reinforcement learning and specifically self-supervised reinforcement learning. So what do they mean? Here's a graphic right here. In reinforcement learning, usually you have an environment and an agent. So you have this environment and let's ignore the without rewards for now. And you have the agent and the agent needs to interact with the environment in order to achieve a maximum reward. So the reward is given by a certain task. You have to do something in this environment. In this case, they consider these types of tasks where I think the top task might be called run forward. So your reward is more the further you go with this walker. And how you can influence the walker is you can sort of give a bit of force onto its joints right here and you have a bunch of sensors. So the main task is actually to keep it to balance it on its feet and then sort of walk forward such that it never falls over. Otherwise, you get negative reward. You lose. So in this case, what they want to do is they want to say, wait, if we just train a reinforcement learning agent for each of these tasks individually, that will use a lot of data. And basically, we can't reuse the learned reinforcement learning agent for each of these individual tasks. It's sort of like if you have many, you know, image tasks or NLP tasks, you don't want to learn one model for each one individually, but you might do something like a common joint pre-training. And this is exactly this for reinforcement learning. And it's even called self supervised, right? Like we are used to in the in the classification setting, self supervised learning. What does it mean? So it means that at first you're in an environment without rewards. So basically, the agent is just dropped in an environment and there's no rewards. It can just do actions and observes states from this environment. And after that, after a while of that, then the tasks come in. So task A, task B and task C are three different tasks, all in the same environment, but all requiring the agent to do different things like running forward or running backwards or do a front flip or things like this. So the how fast the agent can adapt to these individual tasks very much depends on what it has learned during this phase where there were no rewards, right? So the agent is tasked to just explore the world via what they call here task agnostic exploration to explore the world to learn something about the world in order then to generalize to these tasks. And in their case, they learn this global world model. So the agent is supposed to learn somehow how the world works, right? And this is this is the way that this agent is then able to adapt really quickly. So in essence, what does this agent do? The agent works as follows. It gets an input observation, and it runs that through an encoder, which is usually something like a convolutional neural network. And that will give you a set of features, right? So this is sort of an embedding of the state that you're in. And that you can incorporate into a latent state at time t. Now, usually in these RL algorithms, or what can happen is that you incorporate the last latent state, so the latent state from the step here also goes into the latent state of the next step, right? So here was the sorry, the last observation observation comes in features, latent state and so on, ultimately, and then and so on and comes in from here. And there's usually like an RNN going over the time steps. But ultimately here, the agent has to decide on an action using this policy network. Now how is this trained this policy network has to come up with an action, but there are no rewards. So usually, we would train this policy network with like an actor critic method. So we would also train some sort of a value function, and then the policy would try to maximize the value function. And if we don't have rewards, how are we going to do that? So people have thought about this for a bit. And people have come up with things like intrinsic motivation. Intrinsic motivation is a term where you're trying to say something like, if you're in a room right here, like this, your agent is right here, then you just you know, you do something. And maybe your agent goes down here. If your agent were to go down there again, it would sort of not really learn anything because it has now already gone there and has already learned from those states. So you might want to explore some different space, right, like here. And in the next episode, you might want to explore this room right here. So this this notion of intrinsic motivation to explore, it has a bunch of different formulations of how exactly you can formulate it. But just imagine basically, the entire state is filled with a bunch of coins, and I'm going to draw this as green dots, sort of like Pac-Man. And everything is filled with these green dots, right? And what the agent wants to do if it has no rewards, it will simply collect those green dots. And once one is collected, so if I go here, I'll collect all these green dots, these are now no longer there. So that area doesn't give me any reward anymore. So you can imagine sort of like this. So as an intrinsic reward, you simply reward the agent every time it finds itself in a new state that it hasn't seen before. So you train it to seek out novel states that usually, when you just have like an actor critic method, and that's what this paper here criticizes, if you have it's called retrospective novelty. That means if you train a model free algorithm, which is an actor critic, right, if we just plug in into here, something like a three C, that will simply have a policy and a value function. And in this case, if we train it on intrinsic reward, the policy will simply tell you where to go to find more green stuff. But you can only train it. So you use this to run an episode. And then you observe how many green things you found in that episode, right, if your episode goes here, and then you put that back into your buffer to learn from. But at that point, you've already collected the green things, right. So the reward signal is actually a bit off because you want to train your agent that it should seek out novel things. But as soon as you've explored them, they're not really novel anymore, because you have now explored them. But still, you're going to train your agent, telling your agent that this area right here has lots of has given me lots of rewards. So the agent is going to be encouraged to repeat that. They say this right here, the retrospective novelty model free exploration methods not only require large amounts of experience to adapt to downstream tasks, they can also be inefficient during exploration. These agents usually first act in the environment, collect trajectories, and then calculate an intrinsic reward as the agent's current estimate of novelty. This approach misses out on efficiency by operating retrospectively. That is, the novelty of inputs is computed after the agent has already reached them. Hence it seeks out previously novel inputs that have already been visited and would not be novel anymore. Instead, one should directly seek out future inputs that are expected to be novel. Now, so what this paper is doing, it's basically saying, can we build a model that estimates the future novelty of a state that we maybe haven't seen so far? And here is where that goes. So what do they do in this policy? So the policy isn't just trained to maximize the novelty in the world. Instead, sorry, it uses planning. It uses planning in latent space. So what this model does is it learns a world model in latent space. The world model takes as input these features that you saw right here that the encoder gives you and it predicts the future hidden latent states. These are the things you saw here. So these things that are always made by incorporating the new features with the old state, it tries to predict it. So technically, these things here should have like some sort of a, no, this is actually exact here, but these here should have some sort of a tick or something to indicate these are estimated. These are estimated future states. And this model right here, this is an estimate. This is a world model. They use Dreamer for this. And I have made a video about Dreamer. Dreamer tries exactly that. It tries to estimate what is the future, but not in actual world space, but in latent space. And yeah, so it tries to estimate its own future. And the cool thing here is that this is probabilistic or you can make it probabilistic. So you can technically from this one age that you have here, you can run out many futures in your imagination. And since you don't need the observations, you only need the latent space, you can simply forward roll your RNN and sample from it and you have many trajectories in the future. Now the fact that you have many trajectories leads to even a different thing. So what you can do for each of these hidden states, they have a head here that predicts the so-called latent disagreement. What does this do? This consists of a whole bunch of models. These are ensemble models. They're the same for each time step. But what they take in, they take in the latent state of the model and the action that you're about to do, the action that you imagine you would do. So this is the imagined state and this is the imagined action in that state. And then it will compute the next features. So the next, whatever in the next step would be the age. So right, right, where do we put it? Whatever in the next, so if I have this age and I have this state, it tells me if I do action A1 and if I were to execute this in the real world, what would be my next age that I would get? So basically by performing an action, I will get the next observation and I will encode that to get the next features. And this small model would try to predict what are the features of the next state if I were to execute this action in this state. Okay, so it's kind of a future predictor. But also not in observation space, but in latent space. So it tries to predict the latent features of the next observation. And the split here, you might think that there is a bit of a, it's like almost the same, this latent state here and the features. But as we discussed before, the latent state can incorporate the last or the history of latent states while the features simply are only a function of the current observation. And that's why they predict the features. They really want to predict the observation. But history has sort of shown that if you try to predict, for example, the pixels of the observation, that won't serve you really well. And therefore, what you need to do is you need to predict the latent features of the observation that works much better. So they have a bunch of these models right here. They have a bunch of models with different parameterizations. They instantiate k different models of that. And they all run the same. So these are all the same inputs through these different models. Now, these different models have been initialized at different points. So they will make slightly different predictions. And the crucial part is, so if it's really deterministic what the next state is going to be, right? So say you're in this state, you perform this action. And so if you have a ball in your hand and you drop the ball, then the ball is going to fall down. Really deterministic. That means these next these estimated next features, if the models are any good, they all agree. And this variance here between the estimates is very, very small. And now if if the uncertainty over the next state is very high, and this can be due to two facts either, it is actually uncertain what's going to happen. So maybe you have a really a piece of paper and you drop that and due to the wind, you can't know what's happening. Or because your model has simply not learned yet what's going to happen. In either of those cases, you don't know what's going to happen. Therefore, these these predictions here, sorry, are going to be very different from each other. And because of that, this variance will be high. And this variance you take as the intrinsic reward. So in each step, you basically try to predict over the next actions you can do, which ones leads me to a leads me to a situation where I don't know what's going to happen, where I cannot really predict the variance in my prediction is high. So I really don't know what's going to happen. And that is going to be the states you seek out. Okay, so this is the core of the paper. Basically, you do this planning in latent space in order to find the states or action that leads you to a state where you don't know what's going to happen. And you measure that by trying to predict it using slightly different models. And if they disagree a whole bunch, then you can use you sort of you say, I don't know what's going to happen. And therefore, I want to go there because I want to learn about that state. Now this, this is the this is the entire thing. It has a bunch of problems, as you can imagine. So this is the reasoning behind it, right? Now they try to make a deal out of basically their latent disagreement here agrees with minima, sorry, maximizing the expected information gain. They go into the theory right here and say, okay, if I have a state and an action, and I had and this W are the dynamics parameters of the world. So the W characterizes how the world works. And the H here is the next state, or sorry, the features of the next observation. And the I is the mutual information between H and W. So this right here measures how much information of the next state is contained in the dynamics of the world. If this is really low, and I have a good world model, then I should be able to predict the next state really well. And this, this, they say, okay, selecting the most promising data during exploration. We want to select the action that maximizes this information gain. So the, the, the more the mutual information here, we want to select the action that maximizes that they decompose this mutual information into two things. They decompose it into this thing right here, which is simply the entropy of the next state given the current state and action. This is simply the total uncertainty, including the fact that it could actually be stochastic like dropping a paper, and the fact that you haven't learned yet what happens like if you drop a ball, but you haven't learned that yet, that is also uncertain. So this is that part, this is the total uncertainty minus this right here. And this is the uncertainty if you know the dynamics, right? So this is the wind, basically, in the paper example. So you want something where the total uncertainty is high, but the the kind of uncertainty of the of the stochasticity of the world is low. If you maximize this quantity here, this total quantity, sorry, if you maximize this entire quantity, because I called one of them total, that means you are going to seek out actions where what's left is only the uncertainty that you yourself don't know, right? You say, well, this state has a pretty high total uncertainty, but it's not due to the fact that the world itself is uncertain. It must be due to the fact that I don't know yet. And they make the claim that their model is actually going after these things. And they say, okay, because we have these these Gaussians here as our estimators, they somehow reduce to this total to this uncertainty, but only basically by taking Gaussians, they assume, they just assume that this quantity here is constant. At least that's how I understand it. They basically assume that every transition in the world has about the same amount of uncertainty. And therefore, we can just focus on the total amount of uncertainty, right? So if we can't predict, if we can't predict the next state, a, we can predict the next state a better than the next state B, and both have about the same amount of intrinsic uncertainty, stochasticity in the world, that must mean we should go to B because that's where our model hasn't learned yet. Now, of course, in the real world, that is absolutely not the case. And I think this model works mainly because they tested in these transitions, or in these environments where that might be very close to accurate, that actually most of the transitions have the same stochasticity as any other transition. The second, the second part, why this is a bit difficult is because you have to somehow keep this latent, sorry, this, this couple of models right here that make this disagreement prediction. So you rely on the fact that you can capture disagreement by looking how those models disagree with each other. And again, they employ Gaussians here. But it is not said that these things, these things will actually give you the true disagreement among themselves. If you initialize them wrongly, they might miss like if, if your distribution has three modes, they act, they might just for all of them, focus on one of them. And then your disagreement will be completely out of whack, or you could initialize them not far enough for or too close together. That's the same thing. So it all depends on kind of how you manage to handle this uncertainty right here. So all of this seems a bit problematic, but the whole setup is pretty cool. Because imagine like all of this is shifting constantly, right? The the policy here tries to maximize these rewards. And that's something I don't understand. In the paper, they make it sort of explicitly clear that the policy tries to maximize this quantity right here, the next uncertainty. The planning objective is to maximize expected novelty, or it, which is this thing right here. However, I, I don't actually see why in that case, you'd need planning. Because with planning, your goal is sort of to look ahead more than one step. So what I would expect is that they somehow have the aggregated somehow that they not don't maximize this, but somehow they maximize the future right of t prime of r t prime, I, they somehow maximize the yes, they somehow maximize the future, the total future, maybe with a disagreement, like if it was a reward, and you actually want to maximize the total reward across your episode, I would imagine they use planning to maximize the total future uncertainty that they encounter because right here you have your trajectories. And as they say it, they only maximize the uncertainty after the first step. So this here might be, you know, even intrinsically uncertain or a bit uncertain. But if you go down the path here, there might be a state where that's super uncertain, and you would like to find that right through your different rollouts. So I'm not sure that the paper is correct or consistent here, actually. I might be wrong, though, they do have the code, which is a really good thing. So I'll link to the code and you can go and explore that. They do have this algorithm down here, which is pretty much I mean, this is saying just nothing. Still exploring, do train the world model, train the latent disagreement ensemble, train the policy in imagination of like, rather, rather, it helps a bit. Okay. But one other thing right here, the policy that tries to maximize the reward, right, so you use planning to look ahead where the uncertainty is. But how do you do the planning? You need a policy in imagination space, right? This latent disagreement policy here is used to train how you act in latent space, right? How this action that you imagine comes to be, you can't plan in imagination space and in imagination space use planning again, it's just an infinite recursion. At some point, you need a model that tells you what to do. And in imagination, they just use an actor critic model, you see they have a value function here. They just use an actor critic model to basically one shot predict the next best action to get you to the next step. So as they themselves rag on these model free methods, because they only look ahead, how is that not exactly the same as me ragging on the fact that they use model free and imagination space, because your world model certainly is retrospective, your world model learns from the past, right? So the model free method that learns on your imagined world model learns from retrospective imagination. And therefore, it itself has sort of the same problem, just one layer deeper that it learns from retrospective data and not from data ahead, because your uncertainty about the future might just be because of your retrospect, exactly is because of your retrospective data. I see the value in having this uncertainty. But I think there are other methods that also do model free, and don't just maximize an intrinsic reward, but actually maximize a sort of uncertainty. Okay, enough ragging, let's go to the experiment. So the cool thing you can do with this is what's called zero shot performance. So what they do is in the first step, they do this, they just learn task agnostic, just explorer. Without task, then second, they go and into their buffer. So when they explore, they all save, they save what they do, there is no reward, but they just save, they store their episodes, right? And then someone comes with a task, and the task is simply, they specify like you have to run forward, and they go to this buffer, and they now label every episode with its reward. So this is different, this is like offline reinforcement learning, right? So basically, it is how well they call it zero shot, but it is how well can an algorithm that has explored with this kind of self supervision, perform in offline reinforcement learning on the trajectories that it has already experienced, which is different from performing the same trajectories with the reward, because you would learn from the reward, and you would learn to seek out different, like your experience would be different. If you were going after a reward. So this is harder. So they compare this to dreamer, and dreamer is a fully supervised method, the dreamer is actually cheating, dreamer actually goes after the reward. And all the other methods here, they don't have a reward, and they're just zero shot offline reinforcement learning generalized to these methods. And you see the green is the plan to explore, and that outperforms almost all the other methods right here, down here, and even comes close to this, to the dreamer that goes up here. It's seen pretty much every graphic that dreamer is the one that's able to cheat, right? So it is performing pretty well. But then the the zero shot generalized plan to explore is sometimes on par and certainly outperforms the other intrinsic reward methods. Now how does that go about when you try actually when you allow the model to fine tune on the task. So you can see the performance on few shot adaptation from raw pixels without state space input. So basically, you learn without reward for this many steps right here, until this shaded area back here. This is how when you do no reward, and then all of a sudden, now you say, okay, now, I'll give you reward. Now please learn. This many steps, you have this many steps where you can learn from the reward. So now we're no longer in this offline RL setting. This is now online RL. But we've been pre trained with all of this experience that we had without the reward. Again, the orange here is the cheater. So the orange is is cheating. And now we don't so before the graphs were higher, because we've, we've actually at each step, for example, how this works is you train until here without reward, and then you do this offline RL offline RL training. And that's how this point comes about. Now, I think in the graph down here, they they don't do that. So they just measure how well you're doing in the task. And of course, if after this many steps, you've never looked at the reward, you haven't been able to look at the reward, your reward will be fairly low, right at the task, because you don't know what to do to get the high reward. The dreamer again, this orange line is able to cheat. That's why it just is basically straight line or goes up at the beginning, it's able to look at the reward from the beginning. And it's here as a baseline comparison. So you see as soon as you give the reward to the models, they generally shoot up. And this plan to explore generally shoots up much harder than these others, as you can see, pretty much everywhere. And again, it gets competitive and here even outperforms the dreamer. Why could it outperform the supervised method? Maybe because this method here is sort of confused or is stuck in a local optimum, which can happen very easily in reinforcement learning. Whereas the plan to explore has never seen the reward, therefore hasn't tried to just single mindedly maximize the reward and has explored a bunch of different things to do in the world. And now we can use that knowledge to outperform the plan, the dreamer, the baseline. So the other thing I would like to draw your attention to here is that sometimes you see that the plan to explore or the other curiosity methods actually get a reward before the reward kicks in as we saw here, right? For example, right here. And this tells me that this is probably a property of the environment itself, namely these reinforcement learning environments, they don't really have much noise going on, right? They pretty much just have, it's a simulator with one figure that can walk or not. And therefore, it might be that the only interesting thing to do in these models is to actually perform one of these tasks. And that's why it might be that sometimes they already get a reward. So it's true that they don't see the reward for this entire duration, but also implicitly via the developers building the simulator, they have made it such that the only interesting thing to do is the same thing as getting a reward, right? So I'm sort of skeptical that this is like a general exploration policy, because also in the real world, there are just combinatorically hugely many, many actions to do many paths to follow. And if you just go by what do I not know yet, I think you can't you can't put that all into one model is just too much. And the states where you really were really something interesting happens are so few and far in between, and that it doesn't compare to the amount of states where you simply don't know most states, you don't know what's going to happen, but probably nothing's nothing interesting is going to happen just different things, which will screw over this method completely. In any case, they Yeah, sorry, this is just this is another experiment. They have a bunch of other experiments. And yeah, that that was my this was my review of the paper. Tell me if you agree or disagree or if I've misunderstood something that's entirely possible. I'm just always a bit skeptical of these things a bit. So the experiments, they're very compute intensive, of course, so you never know there and then these specific environments right here. You never know there and then the fact that the real world actually has very different stochasticity, which they simply assume away right here. Yeah, but other than that, big props to the fact that the code is out. And as I said, leave a comment if you agree or disagree. Please subscribe and share this video if you liked it, and I'll see you next time. Bye bye.
 Hi there! Today we're looking at planning to explore via self-supervised world models
 by Ramanan Sekar, Ole Rybkin, Kostas Danilidis, Peter Abil, Danijar Hafner and Depak Patak.
 So this is a paper that concerns reinforcement learning and specifically self-supervised
 reinforcement learning.
 So what do they mean?
 Here's a graphic right here.
 In reinforcement learning, usually you have an environment and an agent.
 So you have this environment and let's ignore the without rewards for now.
 And you have the agent and the agent needs to interact with the environment in order
 to achieve a maximum reward.
 So the reward is given by a certain task.
 You have to do something in this environment.
 In this case, they consider these types of tasks where I think the top task might be
 called run forward.
 So your reward is more the further you go with this walker.
 And how you can influence the walker is you can sort of give a bit of force onto its joints
 right here and you have a bunch of sensors.
 So the main task is actually to keep it to balance it on its feet and then sort of walk
 forward such that it never falls over.
 Otherwise, you get negative reward.
 You lose.
 So in this case, what they want to do is they want to say, wait, if we just train a reinforcement
 learning agent for each of these tasks individually, that will use a lot of data.
 And basically, we can't reuse the learned reinforcement learning agent for each of these
 individual tasks.
 It's sort of like if you have many, you know, image tasks or NLP tasks, you don't want to
 learn one model for each one individually, but you might do something like a common joint
 pre-training.
 And this is exactly this for reinforcement learning.
 And it's even called self supervised, right?
 Like we are used to in the in the classification setting, self supervised learning.
 What does it mean?
 So it means that at first you're in an environment without rewards.
 So basically, the agent is just dropped in an environment and there's no rewards.
 It can just do actions and observes states from this environment.
 And after that, after a while of that, then the tasks come in.
 So task A, task B and task C are three different tasks, all in the same environment, but all
 requiring the agent to do different things like running forward or running backwards
 or do a front flip or things like this.
 So the how fast the agent can adapt to these individual tasks very much depends on what
 it has learned during this phase where there were no rewards, right?
 So the agent is tasked to just explore the world via what they call here task agnostic
 exploration to explore the world to learn something about the world in order then to
 generalize to these tasks.
 And in their case, they learn this global world model.
 So the agent is supposed to learn somehow how the world works, right?
 And this is this is the way that this agent is then able to adapt really quickly.
 So in essence, what does this agent do?
 The agent works as follows.
 It gets an input observation, and it runs that through an encoder, which is usually
 something like a convolutional neural network.
 And that will give you a set of features, right?
 So this is sort of an embedding of the state that you're in.
 And that you can incorporate into a latent state at time t.
 Now, usually in these RL algorithms, or what can happen is that you incorporate the last
 latent state, so the latent state from the step here also goes into the latent state
 of the next step, right?
 So here was the sorry, the last observation observation comes in features, latent state
 and so on, ultimately, and then and so on and comes in from here.
 And there's usually like an RNN going over the time steps.
 But ultimately here, the agent has to decide on an action using this policy network.
 Now how is this trained this policy network has to come up with an action, but there are
 no rewards.
 So usually, we would train this policy network with like an actor critic method.
 So we would also train some sort of a value function, and then the policy would try to
 maximize the value function.
 And if we don't have rewards, how are we going to do that?
 So people have thought about this for a bit.
 And people have come up with things like intrinsic motivation.
 Intrinsic motivation is a term where you're trying to say something like, if you're in
 a room right here, like this, your agent is right here, then you just you know, you do
 something.
 And maybe your agent goes down here.
 If your agent were to go down there again, it would sort of not really learn anything
 because it has now already gone there and has already learned from those states.
 So you might want to explore some different space, right, like here.
 And in the next episode, you might want to explore this room right here.
 So this this notion of intrinsic motivation to explore, it has a bunch of different formulations
 of how exactly you can formulate it.
 But just imagine basically, the entire state is filled with a bunch of coins, and I'm going
 to draw this as green dots, sort of like Pac-Man.
 And everything is filled with these green dots, right?
 And what the agent wants to do if it has no rewards, it will simply collect those green
 dots.
 And once one is collected, so if I go here, I'll collect all these green dots, these are
 now no longer there.
 So that area doesn't give me any reward anymore.
 So you can imagine sort of like this.
 So as an intrinsic reward, you simply reward the agent every time it finds itself in a
 new state that it hasn't seen before.
 So you train it to seek out novel states that usually, when you just have like an actor
 critic method, and that's what this paper here criticizes, if you have it's called retrospective
 novelty.
 That means if you train a model free algorithm, which is an actor critic, right, if we just
 plug in into here, something like a three C, that will simply have a policy and a value
 function.
 And in this case, if we train it on intrinsic reward, the policy will simply tell you where
 to go to find more green stuff.
 But you can only train it.
 So you use this to run an episode.
 And then you observe how many green things you found in that episode, right, if your
 episode goes here, and then you put that back into your buffer to learn from.
 But at that point, you've already collected the green things, right.
 So the reward signal is actually a bit off because you want to train your agent that
 it should seek out novel things.
 But as soon as you've explored them, they're not really novel anymore, because you have
 now explored them.
 But still, you're going to train your agent, telling your agent that this area right here
 has lots of has given me lots of rewards.
 So the agent is going to be encouraged to repeat that.
 They say this right here, the retrospective novelty model free exploration methods not
 only require large amounts of experience to adapt to downstream tasks, they can also be
 inefficient during exploration.
 These agents usually first act in the environment, collect trajectories, and then calculate an
 intrinsic reward as the agent's current estimate of novelty.
 This approach misses out on efficiency by operating retrospectively.
 That is, the novelty of inputs is computed after the agent has already reached them.
 Hence it seeks out previously novel inputs that have already been visited and would not
 be novel anymore.
 Instead, one should directly seek out future inputs that are expected to be novel.
 Now, so what this paper is doing, it's basically saying, can we build a model that estimates
 the future novelty of a state that we maybe haven't seen so far?
 And here is where that goes.
 So what do they do in this policy?
 So the policy isn't just trained to maximize the novelty in the world.
 Instead, sorry, it uses planning.
 It uses planning in latent space.
 So what this model does is it learns a world model in latent space.
 The world model takes as input these features that you saw right here that the encoder gives
 you and it predicts the future hidden latent states.
 These are the things you saw here.
 So these things that are always made by incorporating the new features with the old state, it tries
 to predict it.
 So technically, these things here should have like some sort of a, no, this is actually
 exact here, but these here should have some sort of a tick or something to indicate these
 are estimated.
 These are estimated future states.
 And this model right here, this is an estimate.
 This is a world model.
 They use Dreamer for this.
 And I have made a video about Dreamer.
 Dreamer tries exactly that.
 It tries to estimate what is the future, but not in actual world space, but in latent space.
 And yeah, so it tries to estimate its own future.
 And the cool thing here is that this is probabilistic or you can make it probabilistic.
 So you can technically from this one age that you have here, you can run out many futures
 in your imagination.
 And since you don't need the observations, you only need the latent space, you can simply
 forward roll your RNN and sample from it and you have many trajectories in the future.
 Now the fact that you have many trajectories leads to even a different thing.
 So what you can do for each of these hidden states, they have a head here that predicts
 the so-called latent disagreement.
 What does this do?
 This consists of a whole bunch of models.
 These are ensemble models.
 They're the same for each time step.
 But what they take in, they take in the latent state of the model and the action that you're
 about to do, the action that you imagine you would do.
 So this is the imagined state and this is the imagined action in that state.
 And then it will compute the next features.
 So the next, whatever in the next step would be the age.
 So right, right, where do we put it?
 Whatever in the next, so if I have this age and I have this state, it tells me if I do
 action A1 and if I were to execute this in the real world, what would be my next age
 that I would get?
 So basically by performing an action, I will get the next observation and I will encode
 that to get the next features.
 And this small model would try to predict what are the features of the next state if
 I were to execute this action in this state.
 Okay, so it's kind of a future predictor.
 But also not in observation space, but in latent space.
 So it tries to predict the latent features of the next observation.
 And the split here, you might think that there is a bit of a, it's like almost the same,
 this latent state here and the features.
 But as we discussed before, the latent state can incorporate the last or the history of
 latent states while the features simply are only a function of the current observation.
 And that's why they predict the features.
 They really want to predict the observation.
 But history has sort of shown that if you try to predict, for example, the pixels of
 the observation, that won't serve you really well.
 And therefore, what you need to do is you need to predict the latent features of the
 observation that works much better.
 So they have a bunch of these models right here.
 They have a bunch of models with different parameterizations.
 They instantiate k different models of that.
 And they all run the same.
 So these are all the same inputs through these different models.
 Now, these different models have been initialized at different points.
 So they will make slightly different predictions.
 And the crucial part is, so if it's really deterministic what the next state is going
 to be, right?
 So say you're in this state, you perform this action.
 And so if you have a ball in your hand and you drop the ball, then the ball is going
 to fall down.
 Really deterministic.
 That means these next these estimated next features, if the models are any good, they
 all agree.
 And this variance here between the estimates is very, very small.
 And now if if the uncertainty over the next state is very high, and this can be due to
 two facts either, it is actually uncertain what's going to happen.
 So maybe you have a really a piece of paper and you drop that and due to the wind, you
 can't know what's happening.
 Or because your model has simply not learned yet what's going to happen.
 In either of those cases, you don't know what's going to happen.
 Therefore, these these predictions here, sorry, are going to be very different from each other.
 And because of that, this variance will be high.
 And this variance you take as the intrinsic reward.
 So in each step, you basically try to predict over the next actions you can do, which ones
 leads me to a leads me to a situation where I don't know what's going to happen, where
 I cannot really predict the variance in my prediction is high.
 So I really don't know what's going to happen.
 And that is going to be the states you seek out.
 Okay, so this is the core of the paper.
 Basically, you do this planning in latent space in order to find the states or action
 that leads you to a state where you don't know what's going to happen.
 And you measure that by trying to predict it using slightly different models.
 And if they disagree a whole bunch, then you can use you sort of you say, I don't know
 what's going to happen.
 And therefore, I want to go there because I want to learn about that state.
 Now this, this is the this is the entire thing.
 It has a bunch of problems, as you can imagine.
 So this is the reasoning behind it, right?
 Now they try to make a deal out of basically their latent disagreement here agrees with
 minima, sorry, maximizing the expected information gain.
 They go into the theory right here and say, okay, if I have a state and an action, and
 I had and this W are the dynamics parameters of the world.
 So the W characterizes how the world works.
 And the H here is the next state, or sorry, the features of the next observation.
 And the I is the mutual information between H and W. So this right here measures how much
 information of the next state is contained in the dynamics of the world.
 If this is really low, and I have a good world model, then I should be able to predict the
 next state really well.
 And this, this, they say, okay, selecting the most promising data during exploration.
 We want to select the action that maximizes this information gain.
 So the, the, the more the mutual information here, we want to select the action that maximizes
 that they decompose this mutual information into two things.
 They decompose it into this thing right here, which is simply the entropy of the next state
 given the current state and action.
 This is simply the total uncertainty, including the fact that it could actually be stochastic
 like dropping a paper, and the fact that you haven't learned yet what happens like if you
 drop a ball, but you haven't learned that yet, that is also uncertain.
 So this is that part, this is the total uncertainty minus this right here.
 And this is the uncertainty if you know the dynamics, right?
 So this is the wind, basically, in the paper example.
 So you want something where the total uncertainty is high, but the the kind of uncertainty of
 the of the stochasticity of the world is low.
 If you maximize this quantity here, this total quantity, sorry, if you maximize this entire
 quantity, because I called one of them total, that means you are going to seek out actions
 where what's left is only the uncertainty that you yourself don't know, right?
 You say, well, this state has a pretty high total uncertainty, but it's not due to the
 fact that the world itself is uncertain.
 It must be due to the fact that I don't know yet.
 And they make the claim that their model is actually going after these things.
 And they say, okay, because we have these these Gaussians here as our estimators, they
 somehow reduce to this total to this uncertainty, but only basically by taking Gaussians, they
 assume, they just assume that this quantity here is constant.
 At least that's how I understand it.
 They basically assume that every transition in the world has about the same amount of
 uncertainty.
 And therefore, we can just focus on the total amount of uncertainty, right?
 So if we can't predict, if we can't predict the next state, a, we can predict the next
 state a better than the next state B, and both have about the same amount of intrinsic
 uncertainty, stochasticity in the world, that must mean we should go to B because that's
 where our model hasn't learned yet.
 Now, of course, in the real world, that is absolutely not the case.
 And I think this model works mainly because they tested in these transitions, or in these
 environments where that might be very close to accurate, that actually most of the transitions
 have the same stochasticity as any other transition.
 The second, the second part, why this is a bit difficult is because you have to somehow
 keep this latent, sorry, this, this couple of models right here that make this disagreement
 prediction.
 So you rely on the fact that you can capture disagreement by looking how those models disagree
 with each other.
 And again, they employ Gaussians here.
 But it is not said that these things, these things will actually give you the true disagreement
 among themselves.
 If you initialize them wrongly, they might miss like if, if your distribution has three
 modes, they act, they might just for all of them, focus on one of them.
 And then your disagreement will be completely out of whack, or you could initialize them
 not far enough for or too close together.
 That's the same thing.
 So it all depends on kind of how you manage to handle this uncertainty right here.
 So all of this seems a bit problematic, but the whole setup is pretty cool.
 Because imagine like all of this is shifting constantly, right?
 The the policy here tries to maximize these rewards.
 And that's something I don't understand.
 In the paper, they make it sort of explicitly clear that the policy tries to maximize this
 quantity right here, the next uncertainty.
 The planning objective is to maximize expected novelty, or it, which is this thing right
 here.
 However, I, I don't actually see why in that case, you'd need planning.
 Because with planning, your goal is sort of to look ahead more than one step.
 So what I would expect is that they somehow have the aggregated somehow that they not
 don't maximize this, but somehow they maximize the future right of t prime of r t prime,
 I, they somehow maximize the yes, they somehow maximize the future, the total future, maybe
 with a disagreement, like if it was a reward, and you actually want to maximize the total
 reward across your episode, I would imagine they use planning to maximize the total future
 uncertainty that they encounter because right here you have your trajectories.
 And as they say it, they only maximize the uncertainty after the first step.
 So this here might be, you know, even intrinsically uncertain or a bit uncertain.
 But if you go down the path here, there might be a state where that's super uncertain, and
 you would like to find that right through your different rollouts.
 So I'm not sure that the paper is correct or consistent here, actually.
 I might be wrong, though, they do have the code, which is a really good thing.
 So I'll link to the code and you can go and explore that.
 They do have this algorithm down here, which is pretty much I mean, this is saying just
 nothing.
 Still exploring, do train the world model, train the latent disagreement ensemble, train
 the policy in imagination of like, rather, rather, it helps a bit.
 Okay.
 But one other thing right here, the policy that tries to maximize the reward, right,
 so you use planning to look ahead where the uncertainty is.
 But how do you do the planning?
 You need a policy in imagination space, right?
 This latent disagreement policy here is used to train how you act in latent space, right?
 How this action that you imagine comes to be, you can't plan in imagination space and
 in imagination space use planning again, it's just an infinite recursion.
 At some point, you need a model that tells you what to do.
 And in imagination, they just use an actor critic model, you see they have a value function
 here.
 They just use an actor critic model to basically one shot predict the next best action to
 get you to the next step.
 So as they themselves rag on these model free methods, because they only look ahead, how
 is that not exactly the same as me ragging on the fact that they use model free and imagination
 space, because your world model certainly is retrospective, your world model learns
 from the past, right?
 So the model free method that learns on your imagined world model learns from retrospective
 imagination. And therefore, it itself has sort of the same problem, just one layer deeper
 that it learns from retrospective data and not from data ahead, because your uncertainty
 about the future might just be because of your retrospect, exactly is because of your
 retrospective data.
 I see the value in having this uncertainty.
 But I think there are other methods that also do model free, and don't just maximize an
 intrinsic reward, but actually maximize a sort of uncertainty.
 Okay, enough ragging, let's go to the experiment.
 So the cool thing you can do with this is what's called zero shot performance.
 So what they do is in the first step, they do this, they just learn task agnostic, just
 explorer.
 Without task, then second, they go and into their buffer.
 So when they explore, they all save, they save what they do, there is no reward, but
 they just save, they store their episodes, right?
 And then someone comes with a task, and the task is simply, they specify like you have
 to run forward, and they go to this buffer, and they now label every episode with its
 reward.
 So this is different, this is like offline reinforcement learning, right?
 So basically, it is how well they call it zero shot, but it is how well can an algorithm
 that has explored with this kind of self supervision, perform in offline reinforcement learning
 on the trajectories that it has already experienced, which is different from performing the same
 trajectories with the reward, because you would learn from the reward, and you would
 learn to seek out different, like your experience would be different.
 If you were going after a reward.
 So this is harder.
 So they compare this to dreamer, and dreamer is a fully supervised method, the dreamer
 is actually cheating, dreamer actually goes after the reward.
 And all the other methods here, they don't have a reward, and they're just zero shot
 offline reinforcement learning generalized to these methods.
 And you see the green is the plan to explore, and that outperforms almost all the other
 methods right here, down here, and even comes close to this, to the dreamer that goes up
 here.
 It's seen pretty much every graphic that dreamer is the one that's able to cheat, right?
 So it is performing pretty well.
 But then the the zero shot generalized plan to explore is sometimes on par and certainly
 outperforms the other intrinsic reward methods.
 Now how does that go about when you try actually when you allow the model to fine tune on the
 task.
 So you can see the performance on few shot adaptation from raw pixels without state space
 input.
 So basically, you learn without reward for this many steps right here, until this shaded
 area back here.
 This is how when you do no reward, and then all of a sudden, now you say, okay, now, I'll
 give you reward.
 Now please learn.
 This many steps, you have this many steps where you can learn from the reward.
 So now we're no longer in this offline RL setting.
 This is now online RL.
 But we've been pre trained with all of this experience that we had without the reward.
 Again, the orange here is the cheater.
 So the orange is is cheating.
 And now we don't so before the graphs were higher, because we've, we've actually at each
 step, for example, how this works is you train until here without reward, and then you do
 this offline RL offline RL training.
 And that's how this point comes about.
 Now, I think in the graph down here, they they don't do that.
 So they just measure how well you're doing in the task.
 And of course, if after this many steps, you've never looked at the reward, you haven't been
 able to look at the reward, your reward will be fairly low, right at the task, because
 you don't know what to do to get the high reward.
 The dreamer again, this orange line is able to cheat.
 That's why it just is basically straight line or goes up at the beginning, it's able to
 look at the reward from the beginning.
 And it's here as a baseline comparison.
 So you see as soon as you give the reward to the models, they generally shoot up.
 And this plan to explore generally shoots up much harder than these others, as you can
 see, pretty much everywhere.
 And again, it gets competitive and here even outperforms the dreamer.
 Why could it outperform the supervised method?
 Maybe because this method here is sort of confused or is stuck in a local optimum, which
 can happen very easily in reinforcement learning.
 Whereas the plan to explore has never seen the reward, therefore hasn't tried to just
 single mindedly maximize the reward and has explored a bunch of different things to do
 in the world.
 And now we can use that knowledge to outperform the plan, the dreamer, the baseline.
 So the other thing I would like to draw your attention to here is that sometimes you see
 that the plan to explore or the other curiosity methods actually get a reward before the reward
 kicks in as we saw here, right?
 For example, right here.
 And this tells me that this is probably a property of the environment itself, namely
 these reinforcement learning environments, they don't really have much noise going on,
 right?
 They pretty much just have, it's a simulator with one figure that can walk or not.
 And therefore, it might be that the only interesting thing to do in these models is to actually
 perform one of these tasks.
 And that's why it might be that sometimes they already get a reward.
 So it's true that they don't see the reward for this entire duration, but also implicitly
 via the developers building the simulator, they have made it such that the only interesting
 thing to do is the same thing as getting a reward, right?
 So I'm sort of skeptical that this is like a general exploration policy, because also
 in the real world, there are just combinatorically hugely many, many actions to do many paths
 to follow.
 And if you just go by what do I not know yet, I think you can't you can't put that all into
 one model is just too much.
 And the states where you really were really something interesting happens are so few and
 far in between, and that it doesn't compare to the amount of states where you simply don't
 know most states, you don't know what's going to happen, but probably nothing's nothing
 interesting is going to happen just different things, which will screw over this method
 completely.
 In any case, they Yeah, sorry, this is just this is another experiment.
 They have a bunch of other experiments. And yeah, that that was my this was my review
 of the paper. Tell me if you agree or disagree or if I've misunderstood something that's
 entirely possible.
 I'm just always a bit skeptical of these things a bit.
 So the experiments, they're very compute intensive, of course, so you never know there and then
 these specific environments right here.
 You never know there and then the fact that the real world actually has very different
 stochasticity, which they simply assume away right here.
 Yeah, but other than that, big props to the fact that the code is out.
 And as I said, leave a comment if you agree or disagree.
 Please subscribe and share this video if you liked it, and I'll see you next time.
 Bye bye.