Go-Explore: a New Approach for Hard-Exploration Problems
machine learning ml reinforcement learning rl ai artificial intelligence uber exploration hard exploration research novelty graph robustify explore montezuma montezuma's revenge pitfall atari
This algorithm solves the hardest games in the Atari suite and makes it look so easy! This modern version of Dijkstra's shortest path algorithm is outperforming everything else by orders of magnitude, and all based on random exploration.

https://arxiv.org/abs/1901.10995
https://eng.uber.com/go-explore/
https://github.com/uber-research/go-explore

Abstract:
A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics).

Authors: Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune
 Hi there, what you're seeing here is the game Montezuma's Revenge and it has been a problem for a long time for reinforcement learning algorithms. What you can see is this little person that has to kind of jump around, collect keys, collect these coins, kind of get over enemies and so on, and all of this is super hard because the reward is so sparse, so sometimes you have to do hundreds of actions until you get the next improvement in score. You can see on the top how your score is increasing and it seems like this algorithm is pretty efficient on this, but keep in mind this algorithm has to learn from just the pixel input. It has to learn every single move of the agent. So if you see here for example jumping over the enemies, stopping when these blue bars come and going down the ladders without hitting the spider, this is a really really hard problem. So far reinforcement learning algorithms have had a very hard time doing this until this algorithm showed up. GoExplore, which was the first one that actually surpassed I believe human experts or widely surpassed human experts at this game, in fact the first reinforcement learning algorithm that without human demonstration could do anything at all at this game. So let's dive in and see how this algorithm does what it does. And the paper to this is called GoExplore, a new approach for hard exploration problems by Adria Ecofe, Joost Huizinga, Joel Lehmann, Kenneth O. Stanley and Jeff Klun from Uber AI Labs. So they break down the problem into what they call two problems. So these hard exploration problems, they say they suffer from two things, detachment and derailment. You can see here detachment and derailment. So they explain those in detail. Detachment and derailment are related to each other. Detachment is when an exploration algorithm that has some sort of intrinsic motivation, right? This is how you usually do these hard exploration problems. You give intrinsic motivation to the agent to explore new things, like in absence of a reward, if there's no reward around, it should just reach some kind of new state. And you give the algorithm points for reaching states that it has never seen before. But this can come to this sort of detachment problem. They illustrate this here. So let's say your algorithm starts actually here in the middle, right? And everything that's green here is intrinsic reward. So you collect the green stuff that gives you points, right? So the goal might actually be in here or in here. But you have to teach the algorithm to go all this way around. And you do that by simply motivating it to go to new states by giving it a reward for every state it hasn't been. So it starts exploring, goes here, and maybe the first episode reaches here right before it is reset, usually reset after, well, like it bounces kind of around, it's like, ah, there's new stuff. And then it goes here and it will explore kind of it. And it will be motivated to explore because there's always this green stuff here. So after a while here, whatever is purple has been explored, right? Recently. So with purple, they mark what has been recently explored. All of this has been recently explored, right? So it is gone until here. But usually you also have like a component that isn't purely seeking this green stuff, but is also doing some kind of random exploration. And so what happens, what can happen in these algorithms is that if you at one of these times you start the episode here, by chance, it actually goes into the other direction. All right. And then it's like, wow, there's all this green stuff over here, right? And then it's like, woo, so much green stuff. Right. And then what usually happens is it kind of forgets that there's green stuff over here. So it explores all of this stuff around here. It explores, explores, explores, but there's no more stuff. And then it's stuck, right? It's stuck here. And it says, where, where am I going to go? Like I know over here, there's no more green stuff. And over here, there doesn't appear to be any green stuff because it's forgotten about this. So this, they claim these intrinsic motivation algorithms, what they can lead to is you can detach from your frontier of new knowledge, right? Like they can forget that there is, that here at one point they were here and the algorithm, what the algorithm did, it was it explored here until here, and then it explored over here. So it thinks that this thing over here is its most recent frontier of knowledge, right? This is, this is my state here. This is where I go explore from, but there is nowhere to explore from, right? What it should remember is that here it actually kind of jumped over by random chance. I hope this makes sense. This is called detachment of intrinsic motivation algorithms. And it happens when you, when you kind of give these points according to simply reaching new states. And then another thing is what they call derailment. And derailment is a bit of a more subtle problem. So in derailment, what happens is maybe you, maybe you've actually, let's say this same situation. You've discovered a promising state, right, by some miracle. Here is the goal, right? You've reached the goal. You've done this by exploration. You've explored a bunch and you've reached the goal. Now the problem is, can you do it again? Right? Especially if the environment is a bit stochastic, right? If there is noise, if the environment isn't always the same, can you actually learn how to do this robustly, like such that you can repeat your success? And in derailment is the problem that often these algorithms, while they find promising things, they kind of struggle to robustly reach those promising states. Go Explorer solves these problems in two separate phases, one for each, basically. So what it does is in a phase one, it explores, right? Explore and this is a crucial part, until solved. So this is an explorer, a method that explores until the problem is solved with the focus on explore, right? And then in stage two, robustify. And by robustify means that if stage one has resulted in trajectories that have solved the game or the environment, then phase two is simply tasked with robustly finding those. So let's look at phase one. Phase one is kind of like, think of Dijkstra's algorithm. So in Dijkstra's algorithm, this is a shortest path algorithm in graphs. So in Dijkstra's algorithm, you have a graph and you want to reach this from the start, let's call this the start. And this is the end or the goal. And the graph is connected with edges. And these edges have usually sometimes they have weights. We can simply, the goal is how to go the shortest path from start to the end. And what Dijkstra's algorithm does, it starts exploring. So it's like it goes here. All right, and then it says, ah, this is a new state. I reached the state in one step. All right, explore some more. I reached this state in two steps. And then it's like, I reached a state in three steps. Okay, but I can also go here, I reached this state in one step, in two steps. I've already been here. Okay. But then it can, it can say, okay, from here, I reached this state into this is a bad example. Let's say we actually have to make a shortest path. This is the graph, right? So it reaches this state in two steps, but then it explores this thing. It's like, ah, wait a minute, I've seen this state. But before I've reached it in two steps. Now I'm reaching it in one step. This is better. So this path here is better than this path here. And then it goes on from here. It goes on it says, okay, I'm reaching this goal in two steps. I've reached it in three steps before. So clearly, this bottom path here is better than what I've done before this top or this path. So this is this is what Go Explorer does. In a nutshell, what it does is has an archive of states, right? An archive of states that it has visited previously. And the crucial thing here is, and this is kind of necessary to their algorithm, that this is completely deterministic. So what they actually do is they will save the state of the game emulator, right? They are here, right? And they do some exploration, jumping some until their person is here, their game is in some state, and they will save the emulator to a buffer. This is kind of crucial, such that at a later point, they can select this, this exactly this state that they were in, and from here, run a bunch of explorations again, right? So if they say select state from archive, and then go to that state, this is simply restoring the emulator state. But you could also what you could also do if if this is a purely deterministic environment, you could simply save the sequence of actions that you've done to come here, and simply buy so maybe you gone right, right, and here you jump, and you go right, you can simply replay those to get to the exact same state, they discuss that this can be expanded to also handle a kind of stochastic environments. But in their case, at the phase one, the environment is completely deterministic. So they can do this, they can go, sorry, they can go to a state deterministically. So they'll select a state from an archive, they have an algorithm for selecting kind of promising states. They go to that state, and then they explore from that state and they simply do this random. So this is random. And then they update the archive. So what do they do? Right? So we saw so here, maybe a new graph, so they go to a state, this is their state, and then they explore. Now there, there are multiple things that can happen. One they can encounter a new state, right? New state never seen before. All right, what they do is they save it to the buffer. They say, okay, this new state, let's call it n, this new state, I've reached it in. And here we have done s steps, I've reached an s plus one step. And whatever here is the emulator state that we had before, right? So I can at any point, I can go back. If, however, the state has already been seen, let's call this m, they retrieve m, m prime from the buffer because they've already seen it, it's in the buffer, right? They compare, hey, these steps, so is s prime, is this smaller or larger than s plus one? So basically, I've seen this state before, but using this path, can I reach it in fewer steps than I've reached it before? If yes, then I'm going to replace this, replace this s by s plus one, and then save it again in the buffer. All right, so I can, I now have a better path to reach this state than before. So it's almost exactly like Dijkstra's algorithm in that you simply explore and every new state you find you've either already seen, so you just simply have a new way of getting to that state. If you haven't seen it, you simply remember it, and then you do it all again. So you can imagine with time, these number of states in this buffer will explode. And it's not feasible for Montezuma's revenge. Like imagine this game, right? You have to, you have to go everywhere and explore everything, right? This, I mean, every single action here could be a state. That's why, let me pause this. That's why what they do is they, they have to come up with a notion of state that is, doesn't simply include every single game state there is. And what they do is, this is sampled here, they down sample the image. And then this, sorry, I've tried drawing over a blog post, they down sample the image, and then they simply say, all right, so this, this thing would become this thing. And they simply say, okay, if two of these images have the same representation, so grayscale, down sampled, quantized, then they are the same state. And that's kind of the crux of the algorithm I find. So if two things have the same state, then the algorithm is prone to kind of confusing them for each other. It thinks one is the other, not exactly, but it does kind of assume that they are close actually here. But there is a crucial difference between the two. The algorithm will have a very hard time in some situations. I don't want to, like, you can think of, it needs to be kind of convoluted situations, but it can be the kind of crux of the algorithm very much if the state representation isn't done well. And they actually have two methods. One simply relies on this down sampling and the other one, they provide domain knowledge, which means kind of which level you're in, where the player is, and so on. But this is, this is pretty cool. So if you are able, so if, if your reinforcement learning problem, first of all, is deterministic. At least in a simulator. And second, allows for good state representations, kind of for, for low dimensional state representations. If those two things are given, you can use GoExplore. And as I said, this, this representation here is key. So now you know how they do it. They simply explore these states. And if they come on a new state, and every state is, is, is, so we don't mean this here, we actually mean this representation of it, they store it and they remember how to get to it. And simply by exploring like this and having a smart algorithm that picks which state to explore from, which of course is also a lot of domain knowledge, they are able to solve the game, right? So you see, goes way past human expert, and they're, they're able to, to actually perform really well simply by exploring. This is the exploration phase. This is simply random exploration from promising states. And then in the second part, in the second phase, they now robustify it. So now they introduce noise into their environment, right? Because usually environments have noise or some sort of stochasticity, and they run imitation learning on the best trajectories they found. And what that does is, what they do is they have a trajectory, let's say, let's say this is a trajectory, right? These are actions you need to reach this goal state. This imitation learning algorithm, what they do is they take a few steps back, say here, and they just use imitation learning, which is basically a form of reinforcement learning to reach the goal state from here, simply reach the goal state, right? Once in under noise, right? So you can't just take the exact same actions. Once this has been learned, back up a few more steps, maybe here, and then try to reach the goal state. Now you've already learned how to do this part. So this this bigger part should become should be easier than simply starting from here. And you do that until you've kind of backed up your entire trajectory. This is a well known method from imitation learning. But usually you have usually this red thing is a human demonstration. But now this red trajectory has been found by go explore. It turns out if you have a bunch of these trajectories from go explore, you can do a pretty good job at that. All right, that's basically all that I wanted to say about go explore. It's basically Dijkstra's algorithm. It works under very specific circumstances, but I think it's super promising. And it's kind of a new way of thinking about it. So the video I've shown is actually go explore solving Montezuma's revenge getting like a new high score. And you can see how like skilled this this algorithm becomes. All right, with that, I say goodbye and hope to see you next time.
 Hi there, what you're seeing here is the game Montezuma's Revenge and it has been a problem
 for a long time for reinforcement learning algorithms.
 What you can see is this little person that has to kind of jump around, collect keys,
 collect these coins, kind of get over enemies and so on, and all of this is super hard because
 the reward is so sparse, so sometimes you have to do hundreds of actions until you get
 the next improvement in score.
 You can see on the top how your score is increasing and it seems like this algorithm is pretty
 efficient on this, but keep in mind this algorithm has to learn from just the pixel input.
 It has to learn every single move of the agent.
 So if you see here for example jumping over the enemies, stopping when these blue bars
 come and going down the ladders without hitting the spider, this is a really really hard problem.
 So far reinforcement learning algorithms have had a very hard time doing this until this
 algorithm showed up.
 GoExplore, which was the first one that actually surpassed I believe human experts or widely
 surpassed human experts at this game, in fact the first reinforcement learning algorithm
 that without human demonstration could do anything at all at this game.
 So let's dive in and see how this algorithm does what it does.
 And the paper to this is called GoExplore, a new approach for hard exploration problems
 by Adria Ecofe, Joost Huizinga, Joel Lehmann, Kenneth O. Stanley and Jeff Klun from Uber
 AI Labs.
 So they break down the problem into what they call two problems.
 So these hard exploration problems, they say they suffer from two things, detachment and
 derailment.
 You can see here detachment and derailment.
 So they explain those in detail.
 Detachment and derailment are related to each other.
 Detachment is when an exploration algorithm that has some sort of intrinsic motivation,
 right?
 This is how you usually do these hard exploration problems.
 You give intrinsic motivation to the agent to explore new things, like in absence of
 a reward, if there's no reward around, it should just reach some kind of new state.
 And you give the algorithm points for reaching states that it has never seen before.
 But this can come to this sort of detachment problem.
 They illustrate this here.
 So let's say your algorithm starts actually here in the middle, right?
 And everything that's green here is intrinsic reward.
 So you collect the green stuff that gives you points, right?
 So the goal might actually be in here or in here.
 But you have to teach the algorithm to go all this way around.
 And you do that by simply motivating it to go to new states by giving it a reward for
 every state it hasn't been.
 So it starts exploring, goes here, and maybe the first episode reaches here right before
 it is reset, usually reset after, well, like it bounces kind of around, it's like, ah,
 there's new stuff.
 And then it goes here and it will explore kind of it.
 And it will be motivated to explore because there's always this green stuff here.
 So after a while here, whatever is purple has been explored, right?
 Recently.
 So with purple, they mark what has been recently explored.
 All of this has been recently explored, right?
 So it is gone until here.
 But usually you also have like a component that isn't purely seeking this green stuff,
 but is also doing some kind of random exploration.
 And so what happens, what can happen in these algorithms is that if you at one of these
 times you start the episode here, by chance, it actually goes into the other direction.
 All right.
 And then it's like, wow, there's all this green stuff over here, right?
 And then it's like, woo, so much green stuff.
 Right.
 And then what usually happens is it kind of forgets that there's green stuff over here.
 So it explores all of this stuff around here.
 It explores, explores, explores, but there's no more stuff.
 And then it's stuck, right?
 It's stuck here.
 And it says, where, where am I going to go?
 Like I know over here, there's no more green stuff.
 And over here, there doesn't appear to be any green stuff because it's forgotten about
 this.
 So this, they claim these intrinsic motivation algorithms, what they can lead to is you can
 detach from your frontier of new knowledge, right?
 Like they can forget that there is, that here at one point they were here and the algorithm,
 what the algorithm did, it was it explored here until here, and then it explored over
 here.
 So it thinks that this thing over here is its most recent frontier of knowledge, right?
 This is, this is my state here.
 This is where I go explore from, but there is nowhere to explore from, right?
 What it should remember is that here it actually kind of jumped over by random chance.
 I hope this makes sense.
 This is called detachment of intrinsic motivation algorithms.
 And it happens when you, when you kind of give these points according to simply reaching
 new states.
 And then another thing is what they call derailment.
 And derailment is a bit of a more subtle problem.
 So in derailment, what happens is maybe you, maybe you've actually, let's say this same
 situation.
 You've discovered a promising state, right, by some miracle.
 Here is the goal, right?
 You've reached the goal.
 You've done this by exploration.
 You've explored a bunch and you've reached the goal.
 Now the problem is, can you do it again?
 Right?
 Especially if the environment is a bit stochastic, right?
 If there is noise, if the environment isn't always the same, can you actually learn how
 to do this robustly, like such that you can repeat your success?
 And in derailment is the problem that often these algorithms, while they find promising
 things, they kind of struggle to robustly reach those promising states.
 Go Explorer solves these problems in two separate phases, one for each, basically.
 So what it does is in a phase one, it explores, right?
 Explore and this is a crucial part, until solved.
 So this is an explorer, a method that explores until the problem is solved with the focus
 on explore, right?
 And then in stage two, robustify.
 And by robustify means that if stage one has resulted in trajectories that have solved
 the game or the environment, then phase two is simply tasked with robustly finding those.
 So let's look at phase one.
 Phase one is kind of like, think of Dijkstra's algorithm.
 So in Dijkstra's algorithm, this is a shortest path algorithm in graphs.
 So in Dijkstra's algorithm, you have a graph and you want to reach this from the start,
 let's call this the start.
 And this is the end or the goal.
 And the graph is connected with edges.
 And these edges have usually sometimes they have weights.
 We can simply, the goal is how to go the shortest path from start to the end.
 And what Dijkstra's algorithm does, it starts exploring.
 So it's like it goes here.
 All right, and then it says, ah, this is a new state.
 I reached the state in one step.
 All right, explore some more.
 I reached this state in two steps.
 And then it's like, I reached a state in three steps.
 Okay, but I can also go here, I reached this state in one step, in two steps.
 I've already been here.
 Okay.
 But then it can, it can say, okay, from here, I reached this state into this is a bad example.
 Let's say we actually have to make a shortest path.
 This is the graph, right?
 So it reaches this state in two steps, but then it explores this thing.
 It's like, ah, wait a minute, I've seen this state.
 But before I've reached it in two steps.
 Now I'm reaching it in one step.
 This is better.
 So this path here is better than this path here.
 And then it goes on from here.
 It goes on it says, okay, I'm reaching this goal in two steps.
 I've reached it in three steps before.
 So clearly, this bottom path here is better than what I've done before this top or this
 path.
 So this is this is what Go Explorer does.
 In a nutshell, what it does is has an archive of states, right?
 An archive of states that it has visited previously.
 And the crucial thing here is, and this is kind of necessary to their algorithm, that
 this is completely deterministic.
 So what they actually do is they will save the state of the game emulator, right?
 They are here, right?
 And they do some exploration, jumping some until their person is here, their game is
 in some state, and they will save the emulator to a buffer.
 This is kind of crucial, such that at a later point, they can select this, this exactly
 this state that they were in, and from here, run a bunch of explorations again, right?
 So if they say select state from archive, and then go to that state, this is simply
 restoring the emulator state.
 But you could also what you could also do if if this is a purely deterministic environment,
 you could simply save the sequence of actions that you've done to come here, and simply
 buy so maybe you gone right, right, and here you jump, and you go right, you can simply
 replay those to get to the exact same state, they discuss that this can be expanded to
 also handle a kind of stochastic environments.
 But in their case, at the phase one, the environment is completely deterministic.
 So they can do this, they can go, sorry, they can go to a state deterministically.
 So they'll select a state from an archive, they have an algorithm for selecting kind
 of promising states.
 They go to that state, and then they explore from that state and they simply do this random.
 So this is random.
 And then they update the archive.
 So what do they do?
 Right?
 So we saw so here, maybe a new graph, so they go to a state, this is their state, and then
 they explore.
 Now there, there are multiple things that can happen.
 One they can encounter a new state, right?
 New state never seen before.
 All right, what they do is they save it to the buffer.
 They say, okay, this new state, let's call it n, this new state, I've reached it in.
 And here we have done s steps, I've reached an s plus one step.
 And whatever here is the emulator state that we had before, right?
 So I can at any point, I can go back.
 If, however, the state has already been seen, let's call this m, they retrieve m, m prime
 from the buffer because they've already seen it, it's in the buffer, right?
 They compare, hey, these steps, so is s prime, is this smaller or larger than s plus one?
 So basically, I've seen this state before, but using this path, can I reach it in fewer
 steps than I've reached it before?
 If yes, then I'm going to replace this, replace this s by s plus one, and then save it again
 in the buffer.
 All right, so I can, I now have a better path to reach this state than before.
 So it's almost exactly like Dijkstra's algorithm in that you simply explore and every new state
 you find you've either already seen, so you just simply have a new way of getting to that
 state.
 If you haven't seen it, you simply remember it, and then you do it all again.
 So you can imagine with time, these number of states in this buffer will explode.
 And it's not feasible for Montezuma's revenge.
 Like imagine this game, right?
 You have to, you have to go everywhere and explore everything, right?
 This, I mean, every single action here could be a state.
 That's why, let me pause this.
 That's why what they do is they, they have to come up with a notion of state that is,
 doesn't simply include every single game state there is.
 And what they do is, this is sampled here, they down sample the image.
 And then this, sorry, I've tried drawing over a blog post, they down sample the image, and
 then they simply say, all right, so this, this thing would become this thing.
 And they simply say, okay, if two of these images have the same representation, so grayscale,
 down sampled, quantized, then they are the same state.
 And that's kind of the crux of the algorithm I find.
 So if two things have the same state, then the algorithm is prone to kind of confusing
 them for each other.
 It thinks one is the other, not exactly, but it does kind of assume that they are close
 actually here.
 But there is a crucial difference between the two.
 The algorithm will have a very hard time in some situations.
 I don't want to, like, you can think of, it needs to be kind of convoluted situations,
 but it can be the kind of crux of the algorithm very much if the state representation isn't
 done well.
 And they actually have two methods.
 One simply relies on this down sampling and the other one, they provide domain knowledge,
 which means kind of which level you're in, where the player is, and so on.
 But this is, this is pretty cool.
 So if you are able, so if, if your reinforcement learning problem, first of all, is deterministic.
 At least in a simulator.
 And second, allows for good state representations, kind of for, for low dimensional state representations.
 If those two things are given, you can use GoExplore.
 And as I said, this, this representation here is key.
 So now you know how they do it.
 They simply explore these states.
 And if they come on a new state, and every state is, is, is, so we don't mean this here,
 we actually mean this representation of it, they store it and they remember how to get
 to it.
 And simply by exploring like this and having a smart algorithm that picks which state to
 explore from, which of course is also a lot of domain knowledge, they are able to solve
 the game, right?
 So you see, goes way past human expert, and they're, they're able to, to actually perform
 really well simply by exploring.
 This is the exploration phase.
 This is simply random exploration from promising states.
 And then in the second part, in the second phase, they now robustify it.
 So now they introduce noise into their environment, right?
 Because usually environments have noise or some sort of stochasticity, and they run imitation
 learning on the best trajectories they found.
 And what that does is, what they do is they have a trajectory, let's say, let's say this
 is a trajectory, right?
 These are actions you need to reach this goal state.
 This imitation learning algorithm, what they do is they take a few steps back, say here,
 and they just use imitation learning, which is basically a form of reinforcement learning
 to reach the goal state from here, simply reach the goal state, right?
 Once in under noise, right?
 So you can't just take the exact same actions.
 Once this has been learned, back up a few more steps, maybe here, and then try to reach
 the goal state.
 Now you've already learned how to do this part.
 So this this bigger part should become should be easier than simply starting from here.
 And you do that until you've kind of backed up your entire trajectory.
 This is a well known method from imitation learning.
 But usually you have usually this red thing is a human demonstration.
 But now this red trajectory has been found by go explore.
 It turns out if you have a bunch of these trajectories from go explore, you can do a
 pretty good job at that.
 All right, that's basically all that I wanted to say about go explore.
 It's basically Dijkstra's algorithm.
 It works under very specific circumstances, but I think it's super promising.
 And it's kind of a new way of thinking about it.
 So the video I've shown is actually go explore solving Montezuma's revenge getting like a
 new high score.
 And you can see how like skilled this this algorithm becomes.
 All right, with that, I say goodbye and hope to see you next time.