Agent57: Outperforming the Atari Human Benchmark
deep learning machine learning arxiv google rnn recurrent deepmind r2d2 ngu reinforcement learning deep q learning replay buffer exploration exploitation tradeoff policy lstm atari
DeepMind's Agent57 is the first RL agent to outperform humans in all 57 Atari benchmark games. It extends previous algorithms like Never Give Up and R2D2 by meta-learning the exploration-exploitation tradeoff controls.

https://arxiv.org/abs/2003.13350
https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark

Abstract:
Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

Authors: Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Charles Blundell

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, you're looking at Solaris, which is a game in the Atari benchmark, and it has been one of the hardest games for reinforcement learning agents to solve. What you're seeing is Agent 57, which is a new agent by DeepMind that is the first one to beat all of the 57 games in the Atari suite to a human or superhuman performance. So some of these games have been pretty easy for RL agents, but some of them, look at this one here, have been pretty hard, mainly because of the reward structure. Now you see on the top edge, the reward, it's not going up for a long time, and this kind of games where the reward doesn't go up for a long time is very hard for RL agents. So Agent 57 builds on a number of previous improvements to the original DeepQ networks of DeepMind, and today we'll look into this. So it's called Agent 57, as I said, because it beats all of these 57 games. They're quite diverse, and it's a cool thing that a single system can beat it. So they go into this. This is a printout of the website, so I can scribble on it. And this here, it's been cut off, but it should say DQN here. DQN from 2015. All right, so this DQN paper of 2015 was kind of the original paper that popularized this Atari benchmark and introduced neural networks to reinforcement learning, basically, that made it work. Since then, there have been a number of improvements. So maybe we'll just go into what DeepQ learning is. So in reinforcement learning, usually, you have an agent here, and you have an environment over here, right? And the environment will give you an observation. Now, the observation in our case would be something like the frame of a game, right? And you're here, you're a little rocket, and there is a bunch of meteors, right? And then the agent needs to somehow give back an action. So an action, and the actions in the Atari benchmark are always defined. So you can, in Atari, you used to have this kind of joystick thing. You can put it up, down, left, right, or you can put it upright, up, left, and so on. And also you have a button, I think, one or two buttons. I don't actually remember, but you can press at least one button. So these are the actions. Let's say there is something like 20 different actions. So all of the directions here, and then you can always press or not press a button with it. So you have to give, you send back this action here. You say, I want to put the joystick up, and I want to press the button at the same time. And then the environment will give you back a, it will say, okay, we'll give you back a new observation, which would be the next frame of the game. You've pressed up, so your little rocket is a bit more forward. You've pressed the button, so you fired a shot, and the meteors are still here. And it will also give you back a reward. So the reward, different games give different rewards. For example, in Pac-Man, every time your Pac-Man eats a little one of these dots, you get a reward. But in other games, most famously games like Montezuma's Revenge, you're in this room, and there are these platforms and ladders and stuff, and you're here, and there are opponents rolling around, and there's a door over here. You need to go down, jump over here, get up, get some key, and then go to the door. And only then will you get a reward. So games vary in many ways in how you get this reward. So that's kind of the intrinsic problem here. So deep queue learning is the following. We have a neural network taking in the observation. So we have a neural network, let's designate it as this. And the observation goes in here, right? This is O, the observation goes in here. And also, the action goes in here. Now let's call this AI, because we have different actions. And O, observation at step T. And it will give you a queue value. So the queue value for the observation at time T and action I. Now you do this for every single action. So you put observation with action A, J in the same network, right? You get an output that is the queue value for observation, the same observation with this different action. You do this for every action. And wherever the queue value is the highest, right? Wherever that's the highest, that's the action you go with. So what you have to do is you have to train this neural network to predict the queue value as accurate as possible. And the queue value is basically the reward that you expect from now until the end of the episode by performing this action in this situation, right? That's queue learning. Simply predicting if I do action I right now, how much reward am I going to get from now until the end of the episode, right? That's basically it. That's deep queue and deep queue learning simply because you have a neural network doing the learning. So that was deep queue networks and they work pretty well, but they don't work for these long time horizons because you always just learn. You just see one observation, right? And you kind of learn one step at a time and you rely on these queue values propagating through from your experience. It doesn't work very well for these long credit assignments. Now a significant improvement upon that is this R2D2 algorithm that incorporated LSTMs or GRUs, which are recurrent neural networks. So not only does your observation go into the neural network, right? Now your history of observations, so what happened before, so not only the current game state, right? But here you have the observation from step one, the action you did at that step, then the observation time two, the action you did at time two and so on. They now all, so this is encoded and this is encoded and then you have a recurrent neural network that incorporates all of these things that happened previously, right? To your current representation. So now not only does the agent see what is happening right now, it also gets the information of what happened previously, what did it do previously and it can also now back propagate through these things and kind of learn a longer range credit assignment. Credit assignment means it gets to figure out which actions had actually an influence on the final reward. If you incorporate the history, right, you can have a direct gradient flow across that history. So notably these LSTMs or GRUs, you can let them, you know, compute over maybe 10 or 100 steps, right? And then you get a pretty good idea of which of these actions within those 100 steps led to which rewards. And the other thing on R2D2 is of course it is now more distributed. So this was already here improvements to DQN, but the R2D2 agent is also distributed, meaning that you have like a central instance. So this is now engineering, right? You have a central instance that is called the learner. And the learner has the main weights, which I'm going to designate with theta here. And it just takes in experience from all of these workers. So there's worker one, worker two, worker three, four, and so on. And these, they will all just run episodes. They will all do work, work, work, work, work, work, work, work, work independently of each other and then send back their experience to the learner. And every now and then the learner sinks out the weights of the neural networks to the workers. So that's kind of distributed RL in this sense. You have a central learner, then you have many, many workers doing the actual interaction with the environment. So one of the main pitfalls of R2D2 is still it has a poor exploration, exploitation strategy, which I believe it is still just kind of epsilon greedy. What does it mean? So in order to understand this, maybe consider again our screen here, right? So let's say you're here with your space ship, right? And there are, there's a meteor coming right here and one right here. And there is a gold coin right here. Let's make this gold, right? So let's say you get a reward for collecting the coin, but you also get a reward for shooting the meteors, right? So what happens if you shoot right now? So if you shoot, then let's say you shoot and this meteor explodes, right? So you get a reward. Yeah. So you get one reward, but then the meteor right behind it will hit you, right? It's coming toward you. You'll have no way, no time to get out of the way. So one reward and then death, right? Let's make a little arrow here, right? So in total you get one reward. Now what happens instead if you move to the right? So move, right? So the next, in the next frame, the meteor will fly past you. You are over here, right? But the gold coin is here. Now this has given you so far zero reward, right? Oops. This has given you zero reward, but then in the next frame, you know, these meteors have passed now and you are going to get that gold coin. And that gives you one reward and no death, right? So you can technically go on here and maybe you'll get five more rewards down the line. So the, the, this is, here's the exploration exploitation dilemma. If an agent has for some reason learned that the shooting action in this situation will give it a one reward and the move action will give it zero reward, but has not learned to look past this. So this is kind of nebulous here. It has only experienced, it has only experienced one frame of here. Yeah. It has only experienced one frame of experience. It will say, wait a minute, shoot here appears to be like really good. It gives me one reward and move gives me zero reward. So from now on I'll just always do shoot, right? Shoot, shoot, shoot. Now what you would like to do. So this is called exploitation, right? Exploitation. It has learned something that gives it a reward. So it will just do that over and over again. Whereas here you could say, ah, I, I might go this way, even though it's zero word, because I can hope, right? I don't know yet, but I can hope that I will get a more reward down here. This is exploration. And the question in, in reinforcement learning is always how to trade off these two, right? And ideally you would want your agent to collect maximum reward that speaks for exploitation of what it has already learned. But also you never want to discard the possibility that, um, down the line of things that you don't yet know, there might be even more reward. And that speaks for exploration. I'm just, this both are abbreviated, same exploit, explore. This was dumb. Um, so in the original deep QN formulation, and I believe also in R2D2, this is done with Epsilon greedy, um, which is surprisingly performing well. Uh, so in Epsilon greedy, you simply say, I'm going to have a constant Epsilon. This is E Epsilon. Um, this is maybe 5% or something. I'm going to simply do something at random and the other one minus Epsilon. I'm just going to go with the, um, with the thing I have already learned. And this performs pretty well, but you might imagine that there is something smarter to do. So never give up. Um, these, this algorithm, it kind of goes into this, um, exploration, uh, mode where it tries to get, get to smarter ways to do exploration. And the keywords here are things like intrinsic motivation. So intrinsic motivation and curiosity refer to the fact that, um, it is so in addition to the reward you get from the environment here, right? This, this reward right here, you can also interject at this point and say, ah, I'm going to give some R prime, some reward of myself, right? To to kind of encourage some behavior in the agent. And this here we call intrinsic intrinsic. Um, so that means you add to the reward of the environment, you add some reward of your own that has nothing to do with the environment or not much, um, but just encourages certain behavior in the agent that is now also trying to maximize this intrinsic reward. Um, and in curiosity and intrinsic motivation formulations, usually you are rewarded for novelty. Novelty, which means the agent is rewarded for finding things that it has not yet seen. Um, so you, in this situation over here, you might see why this encourages the agent to go this route here because it says, wait a minute, there's a bunch of stuff like here. I just die, right? But there is a bunch of stuff I haven't seen yet down here. So I might want to go explore that and we give it extra intrinsic reward or prime for seeing things it hasn't seen yet. So it will learn if I do things that I have never done, I will get this sweet intrinsic reward and then it will go explore. Now, of course it's a, it's a big engineering question of how exactly to set this intrinsic reward. And there are many, many different formulations of that, um, that fall under this term of, let's say curiosity or something like this. Um, nevertheless, this never give up has, has, um, improved over R2D2, uh, using ideas like that. And now agent 57 improves again. Now how does agent 57 improve again? And it is mainly, um, it is mainly in, in the, in the, in this, what I just said. So how exactly do you apply this intrinsic reward? How exactly do you navigate the exploration, exploitation trade off? That's where agent 57 comes in because what they've realized this, that for these different Atari games right here, uh, some are very easy. Some you don't need much exploration. Some you need a lot. Some you need it over a large time scale and simply one agent, um, one never give up agent with the same settings of this curiosity of how long it looks into the future is not going to solve all the games. So agent 57 learns, um, how to, to modulate this exploration, exploitation trade off. So let's jump into the paper a bit more. I encourage you to read the blog post that is quite thorough and, um, the paper is a bit more technical. Sorry. Let me switch over. This is the paper agent 57 up forming the Atari human benchmark by Google deep mind. And um, here they say improvements to end you to never give up. So the first improvement they do is, um, so we've, we've already talked about how this is classic Q learning, right? So you're trying to learn this function, uh, that gives you the Q value of an action and the state. Um, now since we're going to deal with intrinsic reward in addition to extrinsic reward, uh, it makes sense. That's what they argue to split the Q learning function into two different parts. One part that learns the extrinsic reward and one part that learns the intrinsic reward. Right. And then you have a parameter beta, um, in front of it. Now beta in this case is the trade off. How much do you want to value this intrinsic reward? Right. Um, and here we see our first lever on the exploitation, exploration trade off. If an agent gets lots of reward for, uh, for exploring, right, it might never exploit and exploiting might actually be a good, a good option in the game that you're in. So you might want to set beta small, but in other games you might want to encourage exploration to the max and therefore set beta very high. Um, all right, another, uh, constant along with that, that they modulate is the, is the, um, the discount factor. So which is called this gamma here. So you already see here this beta we've already seen and they also modulate this gamma. Now what does gamma do, um, if I have my state and action, we already said, so here is an observation one and I do action one and that gives me observation two and I do action two and that gives me observation three and I do action three and each time I get a reward, right? An extrinsic reward and an intrinsic reward. So reward one, reward two, reward three and so on. Now usually, um, an RL agent will look at these rewards and let's say you are here, you are at observation one and you're trying to estimate your future rewards. Um, what will be most important will be the reward that you're getting right now, right? Because that's the most sure because, um, this reward here that you might get two steps from now, you know, a lot of things could happen, right? You are pretty sure that if you do action one, you're going to get to this state, but you're not entirely sure. You could also get to another state and therefore you had to do another action and therefore this reward here could be something different. Um, so these algorithms are, are having what's known as a discount factor. That means the value of a state, uh, of a state S is going to be the sum from time, uh, zero, let's say K equals T that's stated time T up until some horizon. I think they call it H in the paper. You could also think of this as infinity of the reward at step K, but discounted by this factor. Um, and you raise it to the, to the power of K usually or T T minus, uh, yeah, K minus T. So basically means that you, this is if T is one, so it's the reward at the at this time step plus let's say gamma here is 0.99, right? Plus 0.99 the reward at the next time step plus 0.99 squared, uh, the reward of that after that. And you see that the more, the more into the future you look, the less, um, value these rewards have. So little bars here indicate that you're going to value future rewards less and less. This is called a discount factor right here. And it's, um, how to set it is very important because if you set it very low, let's say you set it to 0.1, that means all that you want to do is maximize the rewards that you're getting in the likely the next and next, next step. Uh, you're not really looking into the future. Um, this is very good for games that give you immediate reward for good actions. But if you, uh, if you set it very high, let's say 0.999, right? That means a reward a hundred steps from now doesn't, you know, is, is almost the same to you as a reward one step from now. And this is very valuable for games that don't give you a reward immediately or that kind of trying to trick you as we saw before. Like if you shoot the meteor now, then you get one reward, but if you don't and pass on the opportunity, you might get much more later. So the modulation of the discount factor is also very important, uh, to set and really depends on the game. So we have two quantities here that really depend on what kind of game it is. And also they argue, um, it, it also depends where in the learning process you are. So if you're at the very beginning of the learning process, you might want to have a very high goal, the high intrinsic reward to go explore. And you want, might want to get, have a very low discount factor in order to learn a good immediate value function. But then as time goes on, you might want to bring down the intrinsic reward because now you really want actually, because your end goal is to maximize the extrinsic reward and you want to up this discount factor to look more into the future. Now that you have already learned the immediate values very well. So if I had to summarize and simplify what agent 57 does is it builds a neural network that adjusts these two quantities across the training, right? Um, so, so it adjusts the beta and gamma across the training and it does this in a so-called bandit setting. Now there is no real good picture in this paper that I can show you. So I'm just going to have to, to draw. So you have an agent, right? It interacts with this environment here and it always gets these rewards. Now what you have here is a meta controller, right? So the agents, it has two parameters. It has this beta and this gamma and the meta controller now observes this. It observes this interaction and it outputs values for these two constants and the does this dynamically as the training progresses, right? So the agent, the agent will, will kind of learn, the agent will change its behavior over time. Now this is actually implemented in a slightly different way in that the meta controller doesn't control the values directly, but it, it has kind of options. So what you do is you define a bunch of possibilities for beta and gamma. So you say I have strategy one, strategy one has beta at 0.1 and gamma at 0.9. Strategy two has beta at 0.2 and gamma at 0.8 and so on. Right? And now the meta controller has to choose between one of these, in this case, six different strategies across training. So it might start off, as we said, with a high beta, which might be over here, 0.9, 0.1. It might start off with a high beta and then transition to the lower ends. And it can do so depending on the game and depending on the progress in the game. So this is, this is dynamic and this is the improvement over never give up over this other agent, because this other agent simply had these strategies and trained them at the same time. And now this meta controller here controls which strategy is currently trained and which one is used to generate the experience. So this is, this is basically, I mean, there's a, they also, of course, they also say, well, we also increase the window of, let me go back. So this LSTM, these, I've shown you these things here that incorporate experience over time. They also say, well, we increase the window of how long the LSTM, the time window of how much experience is incorporated. And they do a bunch of other things, which I always find kind of annoying because it's always really, really hard to see where the improvements come from that they claim they made. So, but, you know, barring that, basically they built this meta controller to choose the strategies for the agent over time. Now of course, this meta controller again is trained by the rewards that you get back from the environment. So the meta controller as an action has the choice of strategy, right? And the reward, it gets back from the agent environment interaction, right? So in itself, it is a reinforcement learning problem. Now why, like, to me it seems just shifts the, it just shifts the problem of exploration exploitation one level higher. They use a sliding window bandit algorithm to do this. But again, you have hyper parameters there, like how long is the sliding window and how does the bandit algorithm do the exploration exploitation tradeoff. So it seems to me you're just shifting it one level higher. And it also seems like we're getting into the region of where we are meta over engineering our approaches to the specifics of this Atari benchmark. Because we're kind of observing, oh, okay, these agents do this wrong, these agents do this wrong. So let's just build an agent that can do both sort of. And then the kind of audastic thing I find that they open with how to measure artificial general intelligence, which, I mean, come on, you're just it's kind of amnest right now you're just kind of over and over and overfitting on this one benchmark, there's not really a need to, to make this into a story on artificial general intelligence. Alright, so this was my two cents to this. I hope you enjoyed this and bye bye.
 Hi there, you're looking at Solaris, which is a game in the Atari benchmark, and it has
 been one of the hardest games for reinforcement learning agents to solve.
 What you're seeing is Agent 57, which is a new agent by DeepMind that is the first one
 to beat all of the 57 games in the Atari suite to a human or superhuman performance.
 So some of these games have been pretty easy for RL agents, but some of them, look at this
 one here, have been pretty hard, mainly because of the reward structure.
 Now you see on the top edge, the reward, it's not going up for a long time, and this kind
 of games where the reward doesn't go up for a long time is very hard for RL agents.
 So Agent 57 builds on a number of previous improvements to the original DeepQ networks
 of DeepMind, and today we'll look into this.
 So it's called Agent 57, as I said, because it beats all of these 57 games.
 They're quite diverse, and it's a cool thing that a single system can beat it.
 So they go into this.
 This is a printout of the website, so I can scribble on it.
 And this here, it's been cut off, but it should say DQN here.
 DQN from 2015.
 All right, so this DQN paper of 2015 was kind of the original paper that popularized this
 Atari benchmark and introduced neural networks to reinforcement learning, basically, that
 made it work.
 Since then, there have been a number of improvements.
 So maybe we'll just go into what DeepQ learning is.
 So in reinforcement learning, usually, you have an agent here, and you have an environment
 over here, right?
 And the environment will give you an observation.
 Now, the observation in our case would be something like the frame of a game, right?
 And you're here, you're a little rocket, and there is a bunch of meteors, right?
 And then the agent needs to somehow give back an action.
 So an action, and the actions in the Atari benchmark are always defined.
 So you can, in Atari, you used to have this kind of joystick thing.
 You can put it up, down, left, right, or you can put it upright, up, left, and so on.
 And also you have a button, I think, one or two buttons.
 I don't actually remember, but you can press at least one button.
 So these are the actions.
 Let's say there is something like 20 different actions.
 So all of the directions here, and then you can always press or not press a button with
 it.
 So you have to give, you send back this action here.
 You say, I want to put the joystick up, and I want to press the button at the same time.
 And then the environment will give you back a, it will say, okay, we'll give you back
 a new observation, which would be the next frame of the game.
 You've pressed up, so your little rocket is a bit more forward.
 You've pressed the button, so you fired a shot, and the meteors are still here.
 And it will also give you back a reward.
 So the reward, different games give different rewards.
 For example, in Pac-Man, every time your Pac-Man eats a little one of these dots, you get a
 reward.
 But in other games, most famously games like Montezuma's Revenge, you're in this room,
 and there are these platforms and ladders and stuff, and you're here, and there are
 opponents rolling around, and there's a door over here.
 You need to go down, jump over here, get up, get some key, and then go to the door.
 And only then will you get a reward.
 So games vary in many ways in how you get this reward.
 So that's kind of the intrinsic problem here.
 So deep queue learning is the following.
 We have a neural network taking in the observation.
 So we have a neural network, let's designate it as this.
 And the observation goes in here, right?
 This is O, the observation goes in here.
 And also, the action goes in here.
 Now let's call this AI, because we have different actions.
 And O, observation at step T.
 And it will give you a queue value.
 So the queue value for the observation at time T and action I.
 Now you do this for every single action.
 So you put observation with action A, J in the same network, right?
 You get an output that is the queue value for observation, the same observation with
 this different action.
 You do this for every action.
 And wherever the queue value is the highest, right?
 Wherever that's the highest, that's the action you go with.
 So what you have to do is you have to train this neural network to predict the queue value
 as accurate as possible.
 And the queue value is basically the reward that you expect from now until the end of
 the episode by performing this action in this situation, right?
 That's queue learning.
 Simply predicting if I do action I right now, how much reward am I going to get from now
 until the end of the episode, right?
 That's basically it.
 That's deep queue and deep queue learning simply because you have a neural network doing
 the learning.
 So that was deep queue networks and they work pretty well, but they don't work for these
 long time horizons because you always just learn.
 You just see one observation, right?
 And you kind of learn one step at a time and you rely on these queue values propagating
 through from your experience.
 It doesn't work very well for these long credit assignments.
 Now a significant improvement upon that is this R2D2 algorithm that incorporated LSTMs
 or GRUs, which are recurrent neural networks.
 So not only does your observation go into the neural network, right?
 Now your history of observations, so what happened before, so not only the current game
 state, right?
 But here you have the observation from step one, the action you did at that step, then
 the observation time two, the action you did at time two and so on.
 They now all, so this is encoded and this is encoded and then you have a recurrent neural
 network that incorporates all of these things that happened previously, right?
 To your current representation.
 So now not only does the agent see what is happening right now, it also gets the information
 of what happened previously, what did it do previously and it can also now back propagate
 through these things and kind of learn a longer range credit assignment.
 Credit assignment means it gets to figure out which actions had actually an influence
 on the final reward.
 If you incorporate the history, right, you can have a direct gradient flow across that
 history.
 So notably these LSTMs or GRUs, you can let them, you know, compute over maybe 10 or 100
 steps, right?
 And then you get a pretty good idea of which of these actions within those 100 steps led
 to which rewards.
 And the other thing on R2D2 is of course it is now more distributed.
 So this was already here improvements to DQN, but the R2D2 agent is also distributed, meaning
 that you have like a central instance.
 So this is now engineering, right?
 You have a central instance that is called the learner.
 And the learner has the main weights, which I'm going to designate with theta here.
 And it just takes in experience from all of these workers.
 So there's worker one, worker two, worker three, four, and so on.
 And these, they will all just run episodes.
 They will all do work, work, work, work, work, work, work, work, work independently
 of each other and then send back their experience to the learner.
 And every now and then the learner sinks out the weights of the neural networks to the
 workers.
 So that's kind of distributed RL in this sense.
 You have a central learner, then you have many, many workers doing the actual interaction
 with the environment.
 So one of the main pitfalls of R2D2 is still it has a poor exploration, exploitation strategy,
 which I believe it is still just kind of epsilon greedy.
 What does it mean?
 So in order to understand this, maybe consider again our screen here, right?
 So let's say you're here with your space ship, right?
 And there are, there's a meteor coming right here and one right here.
 And there is a gold coin right here.
 Let's make this gold, right?
 So let's say you get a reward for collecting the coin, but you also get a reward for shooting
 the meteors, right?
 So what happens if you shoot right now?
 So if you shoot, then let's say you shoot and this meteor explodes, right?
 So you get a reward.
 Yeah.
 So you get one reward, but then the meteor right behind it will hit you, right?
 It's coming toward you.
 You'll have no way, no time to get out of the way.
 So one reward and then death, right?
 Let's make a little arrow here, right?
 So in total you get one reward.
 Now what happens instead if you move to the right?
 So move, right?
 So the next, in the next frame, the meteor will fly past you.
 You are over here, right?
 But the gold coin is here.
 Now this has given you so far zero reward, right?
 Oops.
 This has given you zero reward, but then in the next frame, you know, these meteors have
 passed now and you are going to get that gold coin.
 And that gives you one reward and no death, right?
 So you can technically go on here and maybe you'll get five more rewards down the line.
 So the, the, this is, here's the exploration exploitation dilemma.
 If an agent has for some reason learned that the shooting action in this situation will
 give it a one reward and the move action will give it zero reward, but has not learned to
 look past this.
 So this is kind of nebulous here.
 It has only experienced, it has only experienced one frame of here.
 Yeah.
 It has only experienced one frame of experience.
 It will say, wait a minute, shoot here appears to be like really good.
 It gives me one reward and move gives me zero reward.
 So from now on I'll just always do shoot, right?
 Shoot, shoot, shoot.
 Now what you would like to do.
 So this is called exploitation, right?
 Exploitation.
 It has learned something that gives it a reward.
 So it will just do that over and over again.
 Whereas here you could say, ah, I, I might go this way, even though it's zero word, because
 I can hope, right?
 I don't know yet, but I can hope that I will get a more reward down here.
 This is exploration.
 And the question in, in reinforcement learning is always how to trade off these two, right?
 And ideally you would want your agent to collect maximum reward that speaks for exploitation
 of what it has already learned.
 But also you never want to discard the possibility that, um, down the line of things that you
 don't yet know, there might be even more reward.
 And that speaks for exploration.
 I'm just, this both are abbreviated, same exploit, explore.
 This was dumb.
 Um, so in the original deep QN formulation, and I believe also in R2D2, this is done with
 Epsilon greedy, um, which is surprisingly performing well.
 Uh, so in Epsilon greedy, you simply say, I'm going to have a constant Epsilon.
 This is E Epsilon.
 Um, this is maybe 5% or something.
 I'm going to simply do something at random and the other one minus Epsilon.
 I'm just going to go with the, um, with the thing I have already learned.
 And this performs pretty well, but you might imagine that there is something smarter to
 do.
 So never give up.
 Um, these, this algorithm, it kind of goes into this, um, exploration, uh, mode where
 it tries to get, get to smarter ways to do exploration.
 And the keywords here are things like intrinsic motivation.
 So intrinsic motivation and curiosity refer to the fact that, um, it is so in addition
 to the reward you get from the environment here, right?
 This, this reward right here, you can also interject at this point and say, ah, I'm going
 to give some R prime, some reward of myself, right?
 To to kind of encourage some behavior in the agent.
 And this here we call intrinsic intrinsic.
 Um, so that means you add to the reward of the environment, you add some reward of your
 own that has nothing to do with the environment or not much, um, but just encourages certain
 behavior in the agent that is now also trying to maximize this intrinsic reward.
 Um, and in curiosity and intrinsic motivation formulations, usually you are rewarded for
 novelty.
 Novelty, which means the agent is rewarded for finding things that it has not yet seen.
 Um, so you, in this situation over here, you might see why this encourages the agent to
 go this route here because it says, wait a minute, there's a bunch of stuff like here.
 I just die, right?
 But there is a bunch of stuff I haven't seen yet down here.
 So I might want to go explore that and we give it extra intrinsic reward or prime for
 seeing things it hasn't seen yet.
 So it will learn if I do things that I have never done, I will get this sweet intrinsic
 reward and then it will go explore.
 Now, of course it's a, it's a big engineering question of how exactly to set this intrinsic
 reward.
 And there are many, many different formulations of that, um, that fall under this term of,
 let's say curiosity or something like this.
 Um, nevertheless, this never give up has, has, um, improved over R2D2, uh, using ideas
 like that.
 And now agent 57 improves again.
 Now how does agent 57 improve again?
 And it is mainly, um, it is mainly in, in the, in the, in this, what I just said.
 So how exactly do you apply this intrinsic reward?
 How exactly do you navigate the exploration, exploitation trade off?
 That's where agent 57 comes in because what they've realized this, that for these different
 Atari games right here, uh, some are very easy.
 Some you don't need much exploration.
 Some you need a lot.
 Some you need it over a large time scale and simply one agent, um, one never give up agent
 with the same settings of this curiosity of how long it looks into the future is not going
 to solve all the games.
 So agent 57 learns, um, how to, to modulate this exploration, exploitation trade off.
 So let's jump into the paper a bit more.
 I encourage you to read the blog post that is quite thorough and, um, the paper is a
 bit more technical.
 Sorry.
 Let me switch over.
 This is the paper agent 57 up forming the Atari human benchmark by Google deep mind.
 And um, here they say improvements to end you to never give up.
 So the first improvement they do is, um, so we've, we've already talked about how this
 is classic Q learning, right?
 So you're trying to learn this function, uh, that gives you the Q value of an action and
 the state.
 Um, now since we're going to deal with intrinsic reward in addition to extrinsic reward, uh,
 it makes sense.
 That's what they argue to split the Q learning function into two different parts.
 One part that learns the extrinsic reward and one part that learns the intrinsic reward.
 Right.
 And then you have a parameter beta, um, in front of it.
 Now beta in this case is the trade off.
 How much do you want to value this intrinsic reward?
 Right.
 Um, and here we see our first lever on the exploitation, exploration trade off.
 If an agent gets lots of reward for, uh, for exploring, right, it might never exploit and
 exploiting might actually be a good, a good option in the game that you're in.
 So you might want to set beta small, but in other games you might want to encourage exploration
 to the max and therefore set beta very high.
 Um, all right, another, uh, constant along with that, that they modulate is the, is the,
 um, the discount factor.
 So which is called this gamma here.
 So you already see here this beta we've already seen and they also modulate this gamma.
 Now what does gamma do, um, if I have my state and action, we already said, so here is an
 observation one and I do action one and that gives me observation two and I do action two
 and that gives me observation three and I do action three and each time I get a reward,
 right?
 An extrinsic reward and an intrinsic reward.
 So reward one, reward two, reward three and so on.
 Now usually, um, an RL agent will look at these rewards and let's say you are here,
 you are at observation one and you're trying to estimate your future rewards.
 Um, what will be most important will be the reward that you're getting right now, right?
 Because that's the most sure because, um, this reward here that you might get two steps
 from now, you know, a lot of things could happen, right?
 You are pretty sure that if you do action one, you're going to get to this state, but
 you're not entirely sure.
 You could also get to another state and therefore you had to do another action and therefore
 this reward here could be something different.
 Um, so these algorithms are, are having what's known as a discount factor.
 That means the value of a state, uh, of a state S is going to be the sum from time,
 uh, zero, let's say K equals T that's stated time T up until some horizon.
 I think they call it H in the paper.
 You could also think of this as infinity of the reward at step K, but discounted by this
 factor.
 Um, and you raise it to the, to the power of K usually or T T minus, uh, yeah, K minus
 T. So basically means that you, this is if T is one, so it's the reward at the
 at this time step plus let's say gamma here is 0.99, right?
 Plus 0.99 the reward at the next time step plus 0.99 squared, uh, the reward of that
 after that.
 And you see that the more, the more into the future you look, the less, um, value these
 rewards have. So little bars here indicate that you're going to value future rewards
 less and less.
 This is called a discount factor right here.
 And it's, um, how to set it is very important because if you set it very low, let's say
 you set it to 0.1, that means all that you want to do is maximize the rewards that you're
 getting in the likely the next and next, next step.
 Uh, you're not really looking into the future.
 Um, this is very good for games that give you immediate reward for good actions.
 But if you, uh, if you set it very high, let's say 0.999, right?
 That means a reward a hundred steps from now doesn't, you know, is, is almost the same
 to you as a reward one step from now.
 And this is very valuable for games that don't give you a reward immediately or that kind
 of trying to trick you as we saw before.
 Like if you shoot the meteor now, then you get one reward, but if you don't and pass
 on the opportunity, you might get much more later.
 So the modulation of the discount factor is also very important, uh, to set and really
 depends on the game.
 So we have two quantities here that really depend on what kind of game it is.
 And also they argue, um, it, it also depends where in the learning process you are.
 So if you're at the very beginning of the learning process, you might want to have a
 very high goal, the high intrinsic reward to go explore.
 And you want, might want to get, have a very low discount factor in order to learn a good
 immediate value function.
 But then as time goes on, you might want to bring down the intrinsic reward because now
 you really want actually, because your end goal is to maximize the extrinsic reward and
 you want to up this discount factor to look more into the future.
 Now that you have already learned the immediate values very well.
 So if I had to summarize and simplify what agent 57 does is it builds a neural network
 that adjusts these two quantities across the training, right?
 Um, so, so it adjusts the beta and gamma across the training and it does this in a so-called
 bandit setting.
 Now there is no real good picture in this paper that I can show you.
 So I'm just going to have to, to draw.
 So you have an agent, right?
 It interacts with this environment here and it always gets these rewards.
 Now what you have here is a meta controller, right?
 So the agents, it has two parameters.
 It has this beta and this gamma and the meta controller now observes this.
 It observes this interaction and it outputs values for these two constants and the does
 this dynamically as the training progresses, right?
 So the agent, the agent will, will kind of learn, the agent will change its behavior
 over time.
 Now this is actually implemented in a slightly different way in that the meta controller
 doesn't control the values directly, but it, it has kind of options.
 So what you do is you define a bunch of possibilities for beta and gamma.
 So you say I have strategy one, strategy one has beta at 0.1 and gamma at 0.9.
 Strategy two has beta at 0.2 and gamma at 0.8 and so on.
 Right?
 And now the meta controller has to choose between one of these, in this case, six different
 strategies across training.
 So it might start off, as we said, with a high beta, which might be over here, 0.9,
 0.1.
 It might start off with a high beta and then transition to the lower ends.
 And it can do so depending on the game and depending on the progress in the game.
 So this is, this is dynamic and this is the improvement over never give up over this other
 agent, because this other agent simply had these strategies and trained them at the same
 time.
 And now this meta controller here controls which strategy is currently trained and which
 one is used to generate the experience.
 So this is, this is basically, I mean, there's a, they also, of course, they also say, well,
 we also increase the window of, let me go back.
 So this LSTM, these, I've shown you these things here that incorporate experience over
 time.
 They also say, well, we increase the window of how long the LSTM, the time window of how
 much experience is incorporated.
 And they do a bunch of other things, which I always find kind of annoying because it's
 always really, really hard to see where the improvements come from that they claim they
 made.
 So, but, you know, barring that, basically they built this meta controller to choose
 the strategies for the agent over time.
 Now of course, this meta controller again is trained by the rewards that you get back
 from the environment.
 So the meta controller as an action has the choice of strategy, right?
 And the reward, it gets back from the agent environment interaction, right?
 So in itself, it is a reinforcement learning problem.
 Now why, like, to me it seems just shifts the, it just shifts the problem of exploration
 exploitation one level higher.
 They use a sliding window bandit algorithm to do this.
 But again, you have hyper parameters there, like how long is the sliding window and how
 does the bandit algorithm do the exploration exploitation tradeoff.
 So it seems to me you're just shifting it one level higher.
 And it also seems like we're getting into the region of where we are meta over engineering
 our approaches to the specifics of this Atari benchmark.
 Because we're kind of observing, oh, okay, these agents do this wrong, these agents do
 this wrong.
 So let's just build an agent that can do both sort of.
 And then the kind of audastic thing I find that they open with how to measure artificial
 general intelligence, which, I mean, come on, you're just it's kind of amnest right
 now you're just kind of over and over and overfitting on this one benchmark, there's
 not really a need to, to make this into a story on artificial general intelligence.
 Alright, so this was my two cents to this.
 I hope you enjoyed this and bye bye.