EfficientZero: Mastering Atari Games with Limited Data (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper muzero alphazero berkeley pieter abbeel dreamer dreamerv2 atari reinforcement learning deep reinforcement learning world model learned world model latent world model alphago deep rl model-based reinforcement learning how does muzero work efficientzero efficientzero model atari 100k sample-efficient reinforcement learning
#efficientzero #muzero #atari

Reinforcement Learning methods are notoriously data-hungry. Notably, MuZero learns a latent world model just from scalar feedback of reward- and policy-predictions, and therefore relies on scale to perform well. However, most RL algorithms fail when presented with very little data. EfficientZero makes several improvements over MuZero that allows it to learn from astonishingly small amounts of data and outperform other methods by a large margin in the low-sample setting. This could be a staple algorithm for future RL research.

OUTLINE:
0:00 - Intro & Outline
2:30 - MuZero Recap
10:50 - EfficientZero improvements
14:15 - Self-Supervised consistency loss
17:50 - End-to-end prediction of the value prefix
20:40 - Model-based off-policy correction
25:45 - Experimental Results & Conclusion

Paper: https://arxiv.org/abs/2111.00210
Code: https://github.com/YeWR/EfficientZero
Note: code not there yet as of release of this video

Abstract:
Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 190.4% mean human performance and 116.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

Authors: Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we're going to look at Mastering Atari Games with Limited Data by Waziru Yeh, Shahuwa Liu, Tanahar Kuretach, Pietra Biel and Yang Gao. This paper presents the Efficient Zero model, which is a model that can do reinforcement learning with severely limited data. So the paper tackles the Atari 100k benchmark, which means to learn Atari, the Atari benchmark as a reinforcement learning task, as for example, deep Q networks did, but you only get 100k transitions. This is about it's about two days worth of real time data to work with. And after that, the model supposedly be able to play Atari. So this is a variant on mu zero mu zero, which is an insanely data intensive reinforcement learning algorithm. And it introduces various tricks and various amendments to mu zero to make it more sample efficient. So when we look at this paper, you can see the gist of it right here. If you do this Atari 100k benchmark, you can see that a lot of the other reinforcement learning algorithm, they fail to even reach human level performance. Whereas this new algorithm out competes not only the other RL algorithms on in this low data regime, but also the humans. Here they say, efficient zeros performance is close to DQN performance at 200 million frames while we consume 500 times less data. Efficient zeros, low sample complexity, and high performance can bring RL closer to real world applicability. They even say we implement their algorithm in an easy to understand manner. And it is available at this GitHub address. So this code is out there, especially if you want to do reinforcement learning, but you don't have as much computer time or money. This might be for you. So we'll go through the paper, we'll see what the improvements are. There's not a single improvement. There are many improvements, three big ones to be exact. And yeah, if you like content like this, don't hesitate to subscribe and tell your friends, and family and professors, I guess. Alright, so we'll first take a small look at what mu zero does. Just as a recap, I have done a video on mu zero. But if you haven't seen that, then here is a short a very short introduction to mu zero to the algorithm. So in a classic reinforcement learning setting, you have your your basic setup of you have the environment, and you have the actor and the environment gives the actor some sort of an observation at time step, let's call it T. The actor uses that observation to come up with some sort of an action at time step T. And then the environment gives the actor back a reward for that time step. And the next observation T plus one. And that goes on and on and on. So the question is, how is the actor supposed to come up with this action right here, given the past observations that it has seen from the environment in order to maximize all of the reward that it gets. Now, in a regular reinforcement learning algorithm, or regular, let's say in the simpler reinforcement learning algorithm, what people are doing is they're doing model free reinforcement learning, which essentially means that they take the series of observation, observation one, observation two, and so on that they've seen so far, they take that they stick it in a big neural network, and they train it to output some sort of an action. And they train the neural network in order to maximize this reward right here, usually using some sort of policy gradient or something like this. So this is a rather, rather direct way we call that model free reinforcement learning, because you directly predict the action without without an explicit model of the world. Now, when you have a model of the world, so when this environment here is well described, for example, a chessboard, in a chessboard, you know the rules, you know, everything that's going to happen in a chessboard, you can use a model of the chessboard. So what you can do is this, you can take these observations, and these observations would correspond to some defined states or let's let's say tic tac toe, tic tac toe is a better example. So, you know, with the observation, I can actually construct the board of tic tac toe that I'm in. And then what I can do is I can actually search, I can try out, I can say, okay, what if I put, you know, something here, then my opponent's certainly going to do that right here. And then what if I put something here, and then my opponent is going to do that, and then they win, right. So that is one, that is one way to do it. And usually you visualize this as a tree. So you are here at a root note, that's your state. And you have several options to do things. And in the several options, your opponent has several options, or if it's a one player game, you have several options again, and so on. So what you want to do is you want to search this tree for the best possible path. And this is what things like alpha go alpha zero, and so on did. They have these explicit model, and they search through it. And now the neural networks no longer predict actions directly, the neural network help you search through that tree, which means they, they vote essentially on which path paths of the tree to explore, because the tree quickly becomes too large to explore as a whole, right, you can't, like if it's more than three moves ahead, the possibilities just get giant even like especially in a game like go. So the neural networks are here to guide the tree search. And that was, in general, the techniques of that center around the Monte Carlo tree search, because at some point, you abort the search, and you simply play one game to the end, as sort of an approximation of what happens. And so on, I'm not going to go into that super duper right here. But what mu zero does is mu zero says, well, this this whole tree search stuff essentially only works if I have an explicit model of the world, such as the tic tac toe board is clearly defined how it works, right? Also, I can I can I can have a simulator for it, I can rewind, I can try again, this doesn't happen when you're interacting with any sort of real world thing, let's say, or even the Atari benchmark. So in Atari, I know there is there's hacks where you can save the ROM and so on. But essentially, you're not supposed to go back in time or go forward in time, you're not supposed to be able to try something out and then say, well, now that didn't work, I'm going to search for a different path in the tree instead. So what people do is they, they try to learn a model of the environment. So in absence of the model of the environment, they try to learn one and there are many, many different ways of doing this. And what mu zero does is it learns a latent model of the environment. So how does that look? So here you have the current observation observation T, what mu zero does is it uses a neural network, I think they call this H or something to get this into a hidden state. So they map the current observation into a hidden state. And then they plan using the hidden state. So they plan, they say, okay, I'm not going to predict what the next observation is going to be like in the tic tac toe board. I'm only going to predict what is the next hidden state going to be t plus one, t plus one, like this is one, this is two, this is three. So you know, depending on which action I do, which which is going what is going to be the next hidden state of the environment? Sorry, of Yeah, of the environment, what's going to be the next hidden state. And from that hidden state, I always going to predict what's going what's going to be the reward for transitioning there, what's going to be my own policy, which is a bit weird that you have to do this, but you have to, and which is going which what's going to be sort of the value and the value is what is going to be my future reward when I go from here. So these are the sort of things that mu zero predicts. And with that, it is able to search this latent tree. Note the addition to mu zero, sorry. Yeah, the addition, sorry to alpha zero, which is this run right here. So we might we might label this, this is something like reinforce. This is alpha zero. And this is mu zero. So the difference to alpha zero being that we no longer have an explicit model. So in order to do tree search, we have to learn a model. And the model that mu zero learns is in the latent space purely right there is it doesn't predict future observations. And it only learns all of this from the signal that it so it predicts the reward, it predicts its own policy, and it predicts the future value. And those are the only learning signals for the world model. That is good because it focuses the algorithm on what's essential, it is essential to get the maximum reward possible. And therefore, the learning the more the learning signals center around those concepts, the better. But that also means learning the entire world model just from signals like the reward is extremely sparse. So it uses a lot of data. And that is that's essentially the catch right here. So we're not going to go into you know, how exactly mu zero does Monte Carlo tree search, they have a way of balancing exploration and exploitation right here by essentially using an upper confidence bound formula that you can see right here. But so efficient zero goes and says there are three main weaknesses with mu zero. First of all, they say lack of supervision on the environment model. That's what I just said, all the the model, the latent model of the environment is learned purely from the signals of the end from the reward signal, the value signal, these are simple single numbers. And to ask the model to learn a transition function for the environment model is a big ask and of course needs a lot of data just from that. The second one is hardness to deal with aleatoric uncertainty. I like I've given up on trying to remember which one is aleatoric and which one is what's the other one epistemic, I have no idea. Okay, let's just read the paragraph. The predicted rewards have large prediction errors. So if there is uncertainty in the environment, for example, the environment is hard to model, the reward prediction errors will accumulate when expanding the Monte Carlo tree search tree to a large depth, resulting in suboptimal performance in exploration and evaluation. So what they mean is that if I predict if I'm if this reward right here has a bit of an error, and then I go on searching right these branches right here, and then the reward I predict right here also has a bit of an error, and so on. And we go down the tree. And every reward has a bit of an error. What I'll do in order to add you know, at the end, at the end right here, I have a path. And I don't go to the end, I stop after a while and I add up the rewards that led me here. And that's sort of, you know, how valuable this notice plus the value that I predict right here, that's going to be the, the value of this path is going to be the sum of the rewards until I'm here plus the value from here on out. And if all of these little rewards have little errors on them, that quickly adds up to a big error. So that's their second criticism right here. That's something we're going to have to solve. And thirdly, off policy issues with multi step value. And that is a general, that is a general thing in these reinforcement learning algorithms, the more distributed you make them, the more sort of what people usually do is they have like a learner box in the middle, learn. So there's a neural network there. But then they have a lot of actors, actor machines, so they distribute training and interacting with the environment and these send back data, there's usually a replay buffer right here somewhere. And that means just that the neural network that is here at the learner is not the same that generated the data, because the data is kind of old. And until you use the data to practice the neural network will have already learned from other data, and therefore you get an off policy issue, even though it's an on policy algorithm. Now, mu zero does a little bit to correct this. But they say this has to be done more. So how are they? Now, now we tackle these these three things. So the first thing they tackle is this lack of supervision on the environment model. So what they do is they add a self supervised consistency loss, you remember that we map the observation at time t to a state a hidden state at time t. And then we use our latent model to predict for a given action, what's the state going to be at time t plus one. And that's an estimate, right? Now, what this paper says is that wait a minute, if we simply look at what happens in the real world, right, observation t plus one, and we send it through the same, so through this, through this same encoding function, then that gives us the hidden state at time t plus one. So technically, these two things here should be equal. So the hidden state at time t plus one, and the estimated hidden state at time t plus one, they should be kind of the same. So what they do is they use a self supervised consistency loss that they they nap from symposium. So symposium is a contrastive learning framework, or self supervised learning framework. And it's usually used to have two images which have been differently augmented. So to make their representation equal. So till the model learns to sort of ignore the data augmentation, that's how you train self supervised image models. But here, we don't augment differently, what we do is we take an observation, and we take the observation at time t plus one. And the first observation, we actually map it through that function that is supposed to give us this estimation of the next state. And then we use a similarity loss in order to pull those two things together. So this function that gives us the next state, and the representation functions, they're now going to be trained in order to make those two things, the next hidden state, and the estimation of the next hidden state, similar to each other. In fact, the the left branch right here is the one that's trained. But that includes the representation function and the next state function. So you might you might ask, you know, this is kind of the first question that everyone in mu zero has is like, why is this not done? Because this is, if you look at the loss of mu zero, you can pretty easily see that that is possible. And I think the mu zero authors have deliberately not introduced a loss like this, because they say no, if we learn from just the reward signals, that is going to be a better algorithm, even though, you know, it might use more data. But at the end, it really trains for what is important for what is the end goal. And that's why they didn't introduce a loss like this, introducing a loss like this clearly trades off the what's the actual target is. And so it trades that off for sample efficiency, because now the supervision signal here is much, much larger, because now we work with different hidden states, which are entire vectors. So that's going to be a much better signal. So that's the first improvement. The second improvement is what they say, the second improvement is the second improvement is the second improvement is what they say, end to end prediction of the value prefix. So they make an example right here of saying, okay, what's what's the value, you know, if you if you look at this, you have to predict sort of the future value, can you really predict what's it going to be like either the green player, let's say the ball flies in this direction, the green player is going to catch the ball or not, right. And that makes a huge difference. Now, you as a human, at this point, you know that it's not going to the green player is not going to catch that ball. And at this time, you're you're kind of sure. But it's quite hard to predict at this time right here. And it's even harder to predict when you know, at which step in time that player is going to miss the ball. And that's an argument they make for for essentially saying, if we add up the rewards of our own predictions, they can introduce a lot of mistakes. And but that's exactly what we do. If we look at the Q value that we use in this tree search, what we do is we add up the Q value of the tree search, we add up the rewards that we got in the path so far, and we add the value at that particular path. And that is very error prone, because this sum right here accumulates all the little errors that that that happened in in prediction. And, you know, as I said, if if we're not exactly sure at which point that is just one of the examples to show you how hard this problem is of predicting rewards step by step, if you look into the future. So what they do is is pretty simple. They say instead of adding up all the rewards, k steps into the future, what if we simply take the hidden states that we predict k steps into the future, and just shove them into a neural network. And then that neural network will output the sum of the rewards. So instead of summing the rewards directly, we have a neural network output the total sum, much like we have a neural network that outputs the value function at that looks ahead, this neural network right here, it will look sort of back, it will look into the past from the current state to the state, the end state that we rolled out in imagination, it will predict the entire value, they're using LSTM for that, because it can take in arbitrary number of states. And the LSTM has a per step rich supervision, because we have a reward at each step. And therefore, they say that works quite well. So that's the second thing. The third thing is the model based off policy correction. So yeah, this one is a little bit more tricky. But essentially, we can see where is it, we can read a bit through it to see what it does. This is an off policy correction mechanism. And they have two different mechanisms to do off policy correction already said off policy correction, you have to do it because the data that you get to learn from comes from your replay buffer comes from delay from the network and so on, and is a little bit older than the network that you're learning. And that turns out to be quite a big problem. So what we usually do is we sample a trajectory from the replay buffer and we compute, and we compute this target value z right here for the value function. The value target sums from off, sorry, suffers from off policy issues since the trajectory is rolled out using an older policy, and thus the value target is no longer accurate. Now, mu zero reanalyzed, this is a particular version of mu zero already handles that a little bit in that it actually recomputes the values, the scalar values with the current network before it learns from them. But still the policy used to generate that data is from an old policy. And so they say, when data is limited, we have to reuse the data sample from a much older policy, thus exaggerating the inaccurate value target issue. So what they do is they say, well, instead of using instead of using sort of the path, so we're, this is the state, right? And here is what actually happened, right? We took some actions, that's what actually happened. And now, what we would like to do is we would like to take this and learn from it. But the policy used to generate that path is an old policy. So we have to take this and learn from it. And so what they say is that the policy used to generate that path is an old policy. So the current network might have done something entirely different, it might have done a different action right here and got to a different point. And that is a problem because in an own policy method, we'd largely like to learn from actions that have been generated with the policy. So we're simply going to not use the entire trajectory for learning. But we're going to cut off at some point, because of course, the further out the more uncertain we get. And that cutoff point is going to be closer, the older the trajectory is. So for a very recent trajectory, my cutoff towards the end, but for a very old trajectory, we might cut off like all the way to the end. And then what we do after the cutoff point is, so we take this, we cut it off at some point, we say, well, it's old, but you know, this part right here is still sort of the uncertainty is, is not large enough for us to worry so much. And then what they do is they use because they have a latent model for for the environment for the world, they use that model to imagine a rollout. So much like something like dreamer or so they now train using imaginary rollouts from the point where they cut off. So the the trajectories in the replay buffer are more like seed values. And after that, they imagine rollouts using their latent model of the world. All right, so yeah, so I think that's it. We redo an MCTS search with the current policy on the last state and compute the empirical mean value through. Oh, yeah, so at the last, so at the last node right here, they redo an MCTS search they in order to get a really good target value there with the current policy. Yep, that's that's it. Okay, so these are the three improvements. Again, they introduce a consistency loss on the hidden states to make their transition model better. Second, they directly predict the value what they call value prefix this thing right here instead of summing up the rewards as they go along the tree search. And thirdly, they seed they use the collective trajectories as seed values and then train essentially in half imagined, half imagined rollouts with the current policy. So that's it. So what does that give them? It gives them very good performance on this Atari 100k benchmark, they do some additional they do some additional things right here, additional ablation studies, for example, they try to reconstruct the observation from the hidden state, and they see that for example, if you don't have a consistency loss, this quickly fails. So this will be the original mu zero, whereas with the consistency loss, you can see that kind of sort of there is an, you know, there is something right there that looks like the observation. Now here, I don't know if that is after the 100k steps, because of course, mu zero after 100k steps also doesn't perform super duper well. And therefore, you wouldn't be surprised like that this is, or it could be because their reconstruction method is just kind of poor as well. But the difference is noticeable between the two models, the one that has the consistency loss and the one that doesn't. They also analyze, for example, the validation loss, if you have if you directly predict the rewards, or if you use this value prefix prediction method, you can see during training, it's approximately the same. However, at validation time, this loss is much, much lower. And lastly, lastly, but they do a lot of ablations that that is it, what I was surprised or not surprised what I noticed in the ablations, and this is pretty much in all the ablations, there is no consistent ranking. So they have three improvements right here. And sometimes this improvement right here, for example, will be the most valuable. So you can see that without the value prefix, alien drops quite a bit. And in other times, you can see right here, this one will be the most valuable. And yet in other times, some some other one, like the last one will be the most valuable, don't see one right now. But I have, I've looked at it and that there is no consistent thing. So that it means that there's not a single recipe to make this thing better. It's a conglomeration. And for different Atari games, different things are important. And that sort of leads you to think, you know, is this, this isn't a this isn't a method from let's say principle, this is they have looked at what fails, and they fixed essentially one by one, the major mistakes that they found. And that is that is a way to go about it. But it is also a danger that we sort of over engineer to make this sort of over engineer to the benchmarks that we have, because, you know, clearly, if I just put one of these improvements, and some of the Atari games will improve by a lot, but others won't. And that, to me is a little bit of the danger right here. And this is why I'm not, you know, like, I can't I can't tell you if this algorithm is going to be a staple algorithm for sample efficient RL, or if it just works particularly well on this benchmark, they do do another benchmark, they do do the deep mind control benchmark. But I think there's going to be more evaluation needed. But I am excited, it really has the potential to be something something cool. All right, that was it from me. Thank you so much for listening, watching. Let me know what you think in the comments. And bye bye.
 Hi there, today we're going to look at Mastering Atari Games with Limited Data by Waziru Yeh,
 Shahuwa Liu, Tanahar Kuretach, Pietra Biel and Yang Gao. This paper presents the Efficient Zero
 model, which is a model that can do reinforcement learning with severely limited data. So the paper
 tackles the Atari 100k benchmark, which means to learn Atari, the Atari benchmark as a reinforcement
 learning task, as for example, deep Q networks did, but you only get 100k transitions. This is
 about it's about two days worth of real time data to work with. And after that, the model supposedly
 be able to play Atari. So this is a variant on mu zero mu zero, which is an insanely data intensive
 reinforcement learning algorithm. And it introduces various tricks and various amendments to
 mu zero to make it more sample efficient. So when we look at this paper, you can see the gist of it
 right here. If you do this Atari 100k benchmark, you can see that a lot of the other reinforcement
 learning algorithm, they fail to even reach human level performance. Whereas this new algorithm
 out competes not only the other RL algorithms on in this low data regime, but also the humans.
 Here they say, efficient zeros performance is close to DQN performance at 200 million frames
 while we consume 500 times less data. Efficient zeros, low sample complexity, and high performance
 can bring RL closer to real world applicability. They even say we implement their algorithm in an
 easy to understand manner. And it is available at this GitHub address. So this code is out there,
 especially if you want to do reinforcement learning, but you don't have as much computer time
 or money. This might be for you. So we'll go through the paper, we'll see what the improvements
 are. There's not a single improvement. There are many improvements, three big ones to be exact.
 And yeah, if you like content like this, don't hesitate to subscribe and tell your friends,
 and family and professors, I guess. Alright, so we'll first take a small look at what mu zero does.
 Just as a recap, I have done a video on mu zero. But if you haven't seen that, then here is a short
 a very short introduction to mu zero to the algorithm. So in a classic reinforcement
 learning setting, you have your your basic setup of you have the environment, and you have the
 actor and the environment gives the actor some sort of an observation at time step, let's call it T.
 The actor uses that observation to come up with some sort of an action at time step T. And then
 the environment gives the actor back a reward for that time step. And the next observation T plus one.
 And that goes on and on and on. So the question is, how is the actor supposed to come up with
 this action right here, given the past observations that it has seen from the environment
 in order to maximize all of the reward that it gets. Now, in a regular reinforcement learning
 algorithm, or regular, let's say in the simpler reinforcement learning algorithm, what people are
 doing is they're doing model free reinforcement learning, which essentially means that they take
 the series of observation, observation one, observation two, and so on that they've seen so
 far, they take that they stick it in a big neural network, and they train it to output some sort of
 an action. And they train the neural network in order to maximize this reward right here, usually
 using some sort of policy gradient or something like this. So this is a rather, rather direct way
 we call that model free reinforcement learning, because you directly predict the action without
 without an explicit model of the world. Now, when you have a model of the world, so when this
 environment here is well described, for example, a chessboard, in a chessboard, you know the rules,
 you know, everything that's going to happen in a chessboard, you can use a model of the chessboard.
 So what you can do is this, you can take these observations, and these observations would
 correspond to some defined states or let's let's say tic tac toe, tic tac toe is a better example.
 So, you know, with the observation, I can actually construct the board of tic tac toe that I'm in.
 And then what I can do is I can actually search, I can try out, I can say, okay, what if I put,
 you know, something here, then my opponent's certainly going to do that right here. And then
 what if I put something here, and then my opponent is going to do that, and then they win, right.
 So that is one, that is one way to do it. And usually you visualize this as a tree. So you are
 here at a root note, that's your state. And you have several options to do things. And in the
 several options, your opponent has several options, or if it's a one player game, you have several
 options again, and so on. So what you want to do is you want to search this tree for the best possible
 path. And this is what things like alpha go alpha zero, and so on did. They have these explicit model,
 and they search through it. And now the neural networks no longer predict actions directly,
 the neural network help you search through that tree, which means they, they vote essentially on
 which path paths of the tree to explore, because the tree quickly becomes too large to explore
 as a whole, right, you can't, like if it's more than three moves ahead, the possibilities just
 get giant even like especially in a game like go. So the neural networks are here to guide
 the tree search. And that was, in general, the techniques of that center around the Monte Carlo
 tree search, because at some point, you abort the search, and you simply play one game to the end,
 as sort of an approximation of what happens. And so on, I'm not going to go into that super duper
 right here. But what mu zero does is mu zero says, well, this this whole tree search stuff
 essentially only works if I have an explicit model of the world, such as the tic tac toe board is
 clearly defined how it works, right? Also, I can I can I can have a simulator for it, I can rewind,
 I can try again, this doesn't happen when you're interacting with any sort of real world thing,
 let's say, or even the Atari benchmark. So in Atari, I know there is there's hacks where you
 can save the ROM and so on. But essentially, you're not supposed to go back in time or go
 forward in time, you're not supposed to be able to try something out and then say, well,
 now that didn't work, I'm going to search for a different path in the tree instead. So what people
 do is they, they try to learn a model of the environment. So in absence of the model of the
 environment, they try to learn one and there are many, many different ways of doing this. And what
 mu zero does is it learns a latent model of the environment. So how does that look? So here you
 have the current observation observation T, what mu zero does is it uses a neural network, I think
 they call this H or something to get this into a hidden state. So they map the current observation
 into a hidden state. And then they plan using the hidden state. So they plan, they say, okay,
 I'm not going to predict what the next observation is going to be like in the tic tac toe board.
 I'm only going to predict what is the next hidden state going to be t plus one, t plus one, like
 this is one, this is two, this is three. So you know, depending on which action I do, which which
 is going what is going to be the next hidden state of the environment? Sorry, of Yeah, of the
 environment, what's going to be the next hidden state. And from that hidden state, I always going
 to predict what's going what's going to be the reward for transitioning there, what's going to
 be my own policy, which is a bit weird that you have to do this, but you have to, and which is
 going which what's going to be sort of the value and the value is what is going to be my future
 reward when I go from here. So these are the sort of things that mu zero predicts. And with that,
 it is able to search this latent tree. Note the addition to mu zero, sorry. Yeah, the addition,
 sorry to alpha zero, which is this run right here. So we might we might label this, this is something
 like reinforce. This is alpha zero. And this is mu zero. So the difference to alpha zero being that
 we no longer have an explicit model. So in order to do tree search, we have to learn a model. And
 the model that mu zero learns is in the latent space purely right there is it doesn't predict
 future observations. And it only learns all of this from the signal that it so it predicts the
 reward, it predicts its own policy, and it predicts the future value. And those are the only learning
 signals for the world model. That is good because it focuses the algorithm on what's essential,
 it is essential to get the maximum reward possible. And therefore, the learning the more the learning
 signals center around those concepts, the better. But that also means learning the entire world model
 just from signals like the reward is extremely sparse. So it uses a lot of data. And that is
 that's essentially the catch right here. So we're not going to go into you know, how exactly mu zero
 does Monte Carlo tree search, they have a way of balancing exploration and exploitation right here
 by essentially using an upper confidence bound formula that you can see right here.
 But so efficient zero goes and says there are three main weaknesses with mu zero. First of all,
 they say lack of supervision on the environment model. That's what I just said, all the the model,
 the latent model of the environment is learned purely from the signals of the end from the reward
 signal, the value signal, these are simple single numbers. And to ask the model to learn a transition
 function for the environment model is a big ask and of course needs a lot of data just from that.
 The second one is hardness to deal with aleatoric uncertainty. I like I've given up on trying to
 remember which one is aleatoric and which one is what's the other one epistemic, I have no idea.
 Okay, let's just read the paragraph. The predicted rewards have large prediction errors. So if there
 is uncertainty in the environment, for example, the environment is hard to model, the reward
 prediction errors will accumulate when expanding the Monte Carlo tree search tree to a large depth,
 resulting in suboptimal performance in exploration and evaluation. So what they mean is that if I
 predict if I'm if this reward right here has a bit of an error, and then I go on searching right
 these branches right here, and then the reward I predict right here also has a bit of an error,
 and so on. And we go down the tree. And every reward has a bit of an error. What I'll do in
 order to add you know, at the end, at the end right here, I have a path. And I don't go to the end,
 I stop after a while and I add up the rewards that led me here. And that's sort of, you know,
 how valuable this notice plus the value that I predict right here, that's going to be the,
 the value of this path is going to be the sum of the rewards until I'm here plus the value from
 here on out. And if all of these little rewards have little errors on them, that quickly adds up
 to a big error. So that's their second criticism right here. That's something we're going to have
 to solve. And thirdly, off policy issues with multi step value. And that is a general, that is
 a general thing in these reinforcement learning algorithms, the more distributed you make them,
 the more sort of what people usually do is they have like a learner box in the middle, learn.
 So there's a neural network there. But then they have a lot of actors, actor machines, so they
 distribute training and interacting with the environment and these send back data, there's
 usually a replay buffer right here somewhere. And that means just that the neural network that is
 here at the learner is not the same that generated the data, because the data is kind of old. And
 until you use the data to practice the neural network will have already learned from other data,
 and therefore you get an off policy issue, even though it's an on policy algorithm. Now,
 mu zero does a little bit to correct this. But they say this has to be done more.
 So how are they? Now, now we tackle these these three things. So the first thing they tackle
 is this lack of supervision on the environment model. So what they do is they add a self supervised
 consistency loss, you remember that we map the observation at time t to a state a hidden state
 at time t. And then we use our latent model to predict for a given action, what's the state
 going to be at time t plus one. And that's an estimate, right? Now, what this paper says is
 that wait a minute, if we simply look at what happens in the real world, right, observation t
 plus one, and we send it through the same, so through this, through this same encoding function,
 then that gives us the hidden state at time t plus one. So technically, these two things here
 should be equal. So the hidden state at time t plus one, and the estimated hidden state at time t
 plus one, they should be kind of the same. So what they do is they use a self supervised
 consistency loss that they they nap from symposium. So symposium is a contrastive learning framework,
 or self supervised learning framework. And it's usually used to have two images which have been
 differently augmented. So to make their representation equal. So till the model learns to sort of
 ignore the data augmentation, that's how you train self supervised image models. But here,
 we don't augment differently, what we do is we take an observation, and we take the observation
 at time t plus one. And the first observation, we actually map it through that function that is
 supposed to give us this estimation of the next state. And then we use a similarity loss
 in order to pull those two things together. So this function that gives us the next state,
 and the representation functions, they're now going to be trained in order to make those two
 things, the next hidden state, and the estimation of the next hidden state, similar to each other.
 In fact, the the left branch right here is the one that's trained. But that includes the
 representation function and the next state function. So you might you might ask, you know,
 this is kind of the first question that everyone in mu zero has is like, why is this not done?
 Because this is, if you look at the loss of mu zero, you can pretty easily see that that is
 possible. And I think the mu zero authors have deliberately not introduced a loss like this,
 because they say no, if we learn from just the reward signals, that is going to be a better
 algorithm, even though, you know, it might use more data. But at the end, it really trains for
 what is important for what is the end goal. And that's why they didn't introduce a loss like this,
 introducing a loss like this clearly trades off the what's the actual target is. And so
 it trades that off for sample efficiency, because now the supervision signal here is much,
 much larger, because now we work with different hidden states, which are entire vectors. So
 that's going to be a much better signal. So that's the first improvement. The second
 improvement is what they say, the second improvement is the second improvement is the
 second improvement is what they say, end to end prediction of the value prefix. So they make an
 example right here of saying, okay, what's what's the value, you know, if you if you look at this,
 you have to predict sort of the future value, can you really predict what's it going to be like
 either the green player, let's say the ball flies in this direction, the green player is going to
 catch the ball or not, right. And that makes a huge difference. Now, you as a human, at this point,
 you know that it's not going to the green player is not going to catch that ball. And at this time,
 you're you're kind of sure. But it's quite hard to predict at this time right here. And it's even
 harder to predict when you know, at which step in time that player is going to miss the ball. And
 that's an argument they make for for essentially saying, if we add up the rewards of our own
 predictions, they can introduce a lot of mistakes. And but that's exactly what we do. If we look at
 the Q value that we use in this tree search, what we do is we add up the Q value of the tree search,
 we add up the rewards that we got in the path so far, and we add the value at that particular path.
 And that is very error prone, because this sum right here accumulates all the little errors that
 that that happened in in prediction. And, you know, as I said, if if we're not exactly sure
 at which point that is just one of the examples to show you how hard this problem is of predicting
 rewards step by step, if you look into the future. So what they do is is pretty simple. They say
 instead of adding up all the rewards, k steps into the future, what if we simply take the hidden
 states that we predict k steps into the future, and just shove them into a neural network.
 And then that neural network will output the sum of the rewards. So instead of summing the
 rewards directly, we have a neural network output the total sum, much like we have a neural network
 that outputs the value function at that looks ahead, this neural network right here, it will look
 sort of back, it will look into the past from the current state to the state, the end state that we
 rolled out in imagination, it will predict the entire value, they're using LSTM for that,
 because it can take in arbitrary number of states. And the LSTM has a per step rich supervision,
 because we have a reward at each step. And therefore, they say that works quite well.
 So that's the second thing. The third thing is the model based off policy correction. So
 yeah, this one is a little bit more tricky. But essentially, we can see where is it,
 we can read a bit through it to see what it does. This is an off policy correction mechanism. And
 they have two different mechanisms to do off policy correction already said off policy
 correction, you have to do it because the data that you get to learn from comes from your replay
 buffer comes from delay from the network and so on, and is a little bit older than the network
 that you're learning. And that turns out to be quite a big problem. So
 what we usually do is we sample a trajectory from the replay buffer and we compute, and we compute
 this target value z right here for the value function. The value target sums from off, sorry,
 suffers from off policy issues since the trajectory is rolled out using an older policy,
 and thus the value target is no longer accurate. Now, mu zero reanalyzed, this is a particular
 version of mu zero already handles that a little bit in that it actually recomputes the values,
 the scalar values with the current network before it learns from them. But still the policy used to
 generate that data is from an old policy. And so they say, when data is limited, we have to reuse
 the data sample from a much older policy, thus exaggerating the inaccurate value target issue.
 So what they do is they say, well, instead of using instead of using sort of the path, so we're,
 this is the state, right? And here is what actually happened, right? We took some actions,
 that's what actually happened. And now, what we would like to do is we would like to take this
 and learn from it. But the policy used to generate that path is an old policy. So we have to
 take this and learn from it. And so what they say is that the policy used to generate that path is
 an old policy. So the current network might have done something entirely different, it might have
 done a different action right here and got to a different point. And that is a problem because
 in an own policy method, we'd largely like to learn from actions that have been generated with
 the policy. So we're simply going to not use the entire trajectory for learning. But we're going to
 cut off at some point, because of course, the further out the more uncertain we get. And that
 cutoff point is going to be closer, the older the trajectory is. So for a very recent trajectory,
 my cutoff towards the end, but for a very old trajectory, we might cut off like all the way
 to the end. And then what we do after the cutoff point is, so we take this, we cut it off at some
 point, we say, well, it's old, but you know, this part right here is still sort of the uncertainty
 is, is not large enough for us to worry so much. And then what they do is they use because they
 have a latent model for for the environment for the world, they use that model to imagine a rollout.
 So much like something like dreamer or so they now train using imaginary rollouts from the point
 where they cut off. So the the trajectories in the replay buffer are more like seed values.
 And after that, they imagine rollouts using their latent model of the world. All right, so
 yeah, so I think that's it. We redo an MCTS search with the current policy on the last state and
 compute the empirical mean value through. Oh, yeah, so at the last, so at the last node right here,
 they redo an MCTS search they in order to get a really good target value there with the current
 policy. Yep, that's that's it. Okay, so these are the three improvements. Again, they introduce a
 consistency loss on the hidden states to make their transition model better. Second, they directly
 predict the value what they call value prefix this thing right here instead of summing up the rewards
 as they go along the tree search. And thirdly, they seed they use the collective trajectories as
 seed values and then train essentially in half imagined, half imagined rollouts with the current
 policy. So that's it. So what does that give them? It gives them very good performance on this Atari
 100k benchmark, they do some additional they do some additional things right here, additional
 ablation studies, for example, they try to reconstruct the observation from the hidden state,
 and they see that for example, if you don't have a consistency loss, this quickly fails. So this
 will be the original mu zero, whereas with the consistency loss, you can see that kind of sort
 of there is an, you know, there is something right there that looks like the observation. Now here,
 I don't know if that is after the 100k steps, because of course, mu zero after 100k steps also
 doesn't perform super duper well. And therefore, you wouldn't be surprised like that this is, or
 it could be because their reconstruction method is just kind of poor as well. But the difference is
 noticeable between the two models, the one that has the consistency loss and the one that doesn't.
 They also analyze, for example, the validation loss, if you have if you directly predict the
 rewards, or if you use this value prefix prediction method, you can see during training,
 it's approximately the same. However, at validation time, this loss is much, much lower. And lastly,
 lastly, but they do a lot of ablations that that is it, what I was surprised or not surprised what
 I noticed in the ablations, and this is pretty much in all the ablations, there is no consistent
 ranking. So they have three improvements right here. And sometimes this improvement right here,
 for example, will be the most valuable. So you can see that without the value prefix, alien drops
 quite a bit. And in other times, you can see right here, this one will be the most valuable.
 And yet in other times, some some other one, like the last one will be the most valuable,
 don't see one right now. But I have, I've looked at it and that there is no consistent thing. So
 that it means that there's not a single recipe to make this thing better. It's a conglomeration.
 And for different Atari games, different things are important. And that sort of leads you to think,
 you know, is this, this isn't a this isn't a method from let's say principle, this is they have looked
 at what fails, and they fixed essentially one by one, the major mistakes that they found. And that
 is that is a way to go about it. But it is also a danger that we sort of over engineer to make this
 sort of over engineer to the benchmarks that we have, because, you know, clearly, if I just put
 one of these improvements, and some of the Atari games will improve by a lot, but others won't.
 And that, to me is a little bit of the danger right here. And this is why I'm not, you know,
 like, I can't I can't tell you if this algorithm is going to be a staple algorithm for sample
 efficient RL, or if it just works particularly well on this benchmark, they do do another
 benchmark, they do do the deep mind control benchmark. But I think there's going to be more
 evaluation needed. But I am excited, it really has the potential to be something something cool.
 All right, that was it from me. Thank you so much for listening, watching. Let me know what you
 think in the comments. And bye bye.