POET: Endlessly Generating Increasingly Complex and Diverse Learning Environments and Solutions
deep learning machine learning arxiv evolution reinforcement learning neat open-ended never ending population bipedal walker
From the makers of Go-Explore, POET is a mixture of ideas from novelty search, evolutionary methods, open-ended learning and curriculum learning.

https://arxiv.org/abs/1901.01753

Abstract:
While the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula, and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper does just that: it pairs the generation of environmental challenges and the optimization of agents to solve those challenges. It simultaneously explores many different paths through the space of possible problems and solutions and, critically, allows these stepping-stone solutions to transfer between problems if better, catalyzing innovation. The term open-ended signifies the intriguing potential for algorithms like POET to continue to create novel and increasingly complex capabilities without bound. Our results show that POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved by direct optimization alone, or even through a direct-path curriculum-building control algorithm introduced to highlight the critical role of open-endedness in solving ambitious challenges. The ability to transfer solutions from one environment to another proves essential to unlocking the full potential of the system as a whole, demonstrating the unpredictable nature of fortuitous stepping stones. We hope that POET will inspire a new push towards open-ended discovery across many domains, where algorithms like POET can blaze a trail through their interesting possible manifestations and solutions.

Authors: Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Alright, so what you're seeing here are solutions found to this bipedal walker problem by a new algorithm called PoET. So as you might guess, the challenge is to keep this little thing here walking to the right as far as you can while it encounters various obstacles. And it is and remains a challenging reinforcement learning problem to have an agent learn to overcome various obstacles and walk well in different environments. So the paper we're going to look at is called PoET. It's by Uber Engineering. And the full pronunciation is the Paired Open-Ended Trail Blazer, endlessly generating increasingly complex and diverse learning environments and their solutions by Roy Wang, Joel Lehmann, Jeff Klun and Kenneth O. Stanley, as I said from Uber AI Labs. So as you already saw, the challenge they take on is this bipedal walker problem. Now their method is very general and not limited to this problem, but this is the problem that they focus on. I'm going to jump some of the explanations here and dig right into the problem. As you can see, the problem is the following. You have this thing here, which is the walker, and it has two legs and specifically it has four joints. So the four joints are here too, and here too. And you can give torque on all of the four joints. So it's basically a four output problem. And you do have sensors as input. So the inputs, I believe, is a LIDAR. So the LIDAR is this red line you see here. I think it has 16 of those in various angles. And also it has pressure detection on the feet, I believe, to see whether or not they are in contact with the ground. And it might also have a gyroscope in that tells you which angle with respect to the ground the head is. So you have various sensors on these things, and you're able to basically control what the legs are doing. And your goal is to make this go as far to the right and as fast as possible. You see the reward down here is negative 100 if the robot falls over. That means if the head hits the ground. And then it is 130 times delta x. That's how far you go to the right minus the whole angle. And the whole angle, as I said, is this angle here. So you want to keep it as stable as possible. Because if there's a difference in the angle per step, then you get penalized. And also you get penalized for each torque you apply. So you want to kind of apply minimal force on the joints in order to go very far. But by far the most important point is to go to the right as far and as fast as you can. There is an end here somewhere. And if you reach it, you get a score that is above 230. They choose the limit of 230 here to determine. So if the agent gets 230 or more, then it has solved the environment. That's what they claim. That's from experience. So as you see, the environment has various obstacles here. There are holes that you can fall into that you need to jump or step over. There are these kind of stumps here. They can be of various height. So this is a bit shorter and this is a bit longer. And the general terrain has a roughness. And this can go to very rough from very smooth. So this is a parameterized environment. And obviously they are able to generate these environments from parameters. And the goal now is to have an agent that walks well in any environment that you can think of. Right, so here on the left you see this is very challenging down the stairs. This also isn't too easy because there is a gap here. And there are five parameters of these environments. So there is the general roughness of the terrain. That means how many hills it has and how fast they are coming. There is the stump lower bound and stump upper bound, I believe. So how high the stumps are. And also how long the gaps are. And with these parameters you control how difficult an environment is. So the straightforward thing to do is simply to sample environments and have a reinforcement learn approach to this. And that usually doesn't work. I already want to see this without having talked about what the algorithm is. This is the approach where you try this thing. It's called evolution strategies. But you can think of it as just a straightforward optimization procedure. So there is an agent and there is an environment and you are trying to solve the environment using just straightforward optimization. Now the evolution strategies are not your classic algorithm but you can compare it to it. It's just that these people, they like the more, I have a feeling they like the more esoteric learning algorithms. In any case, you see in these environments large gap, rough surface and so on. These are supposed to be the platinum figures. So these two environments and also these environments here. The evolution strategy, so the classic approach if you just straight forward optimize, they get very low scores on average, whereas poet gets here very high scores above the 230 threshold. So what's happening? If you're trying to just solve these environments from scratch, you basically don't really have a big chance of solving them. Because let's say you're here and you're trying to move to the right, you know, you might learn how to do this and you see this from scratch solution actually manages to get to the right. But then as soon as you reach this, you're in this gap and you just fall down the gap because all you've learned so far is how to move right. So what you would need to do is you would need to plan ahead like what poet does. You need to see that there is a gap. You need to plan ahead and already lift up a leg in order to then step over the gap here and then do a little jump right here. And this sequence of action, this kind of planning ahead, it is very difficult to learn this for a classic RL algorithm because you basically get reward for everything you do. So initially you get reward for moving to the right. So that's 10 if you reach here, another 10 if you reach here. And so there is another 10 if you reach here and another 10 if you reach here. Whereas if you lift up your leg, that's like minus five because now this you've changed this angle and we saw this is negative reward, right? So a classic optimization algorithm will always fall into the hole because that is where you get the immediate reward. Whereas you'd have to you'd have to do a sequence of action that doesn't give you a reward right now, but it gives you more reward later. And in order to learn this, we need a kind of a better algorithm that just straightforward optimization. So maybe I can explain this if you have a maze, here is the start and here is the goal and there is like walls and the walls are something like this. What you need to do is go around here. But what a classic optimization algorithm does is always like goes here because that's ever so closer to the goal. And then it just gets stuck because it can't fathom that it needs to go around here. So it needs to go farther away before it gets closer. So these people we've talked about this before in like open ended learning novelty search. What you would want to do is you would want to gradually build up solutions that can explore the space like to go here, go here, go here and basically build up these solutions. And there are two components to what this poet algorithm does. So the first component is curriculum learning. Curriculum learning. What does curriculum learning mean? Curriculum learning means that you start off with easy tasks and you increasingly build up more and more and more complex tasks. So let's say I have an environment here and I'm going to draw and at the beginning we just kind of start off with this flat surface right and here is our little walker right here. And we'll just train it to move right on that and that should be doable with kind of a classic approach. And then we gradually move to more difficult environments. So maybe we'll make it a bit more rough right. And an agent that can already walk to the right already kind of has think of it as a pre-training in like NLP. You can then get more and more challenging and then maybe at some point you can build in a gap right. So you build in one of these gaps and now it already knows how to move to the right and now it might actually learn to jump a small gap right if you make it small at the beginning not like this one down here. There's a very large gap. But if you make it small by accident it might stumble over it and then learn and continuously how to master the gap. So this is the curriculum learning approach. It means that from environment to environment you get harder, harder and harder challenges. So first flat then more rough then more rough with a gap and so on. The second approach, the second ingredient to POET is what they call stepping stone learning or transfer learning or things like this. And that's where you kind of have to think of this not as a single agent optimizing but as a population of agents. So let's say you do this curriculum learning right. And you're getting fairly well here at rough terrains right. More and more rough terrains. But in parallel you also have a second optimization procedure. You also start out kind of flat but with this thing you go as we said before small gap you keep it flat but you just increase the number of gaps here right. Whereas over here you just keep making the terrain rougher and rougher. So what the philosophy is that an agent that might be able to master this rougher terrain it might actually that skill because here you this kind of this kind of looks like a gap here. The skill of hopping over this gap here might actually transfer to the environment over here where you do have a proper you know a gap in the environment or the skill that you learn from an environment where you have one of these stumps right. So here let's draw in one of these stumps where you have to go over and if you have a walker that can successfully walk over this that skill now might transfer over here in order to get over this over this peaky terrain here. So the idea of poet is to start off with a generic flat very easy environment and then spawn new ones so you want to spawn new environments in kind of a hereditary way. So this one might get a bit rougher this one might include this and this one might include a gap or something like this and then again you want to spawn new environments and more rough more rough more rough with a stump here and this one retains the gap sorry and um this one now gets two gaps and so on and you want to continuously train these and then always you want to check whether or not the skill that you learn over here might actually transfer to anyone over here. So you get this tree of this continuous tree of solutions and once you improve on one branch this might actually be good on another branch right they always make the comparison to let's say biological evolution where a strategy that works over here for birds is all of a sudden can be cross adopted by mammals for an entirely different problem but the same skill might be valuable. Yeah so this this is basically the two ingredients of poet and now I want to show you the complete poet algorithm. So what does it do you start off with an initial environment right and in poet every environment is paired with an agent so there is one agent per environment right so for the time steps what you do is first of all you go through your environments and you mutate them and we already seen these environments they can be generated from a parameter vector so we have five numbers right how rough how stumpy and how wide the gaps are let's say we have three numbers to two and this might be one this might be two this might be five right so what you want to do is you want to mutate them right you want to spawn children and each of these parameters has a chance of mutating this might be one three five and this environment might be one four six and this one might be two two five right you spawn new ones you already see that the requirement here is that you can actually have environments that are procedurally generated and mutated like this where a small mutation probably is going to lead to a small change in the environment in any case you mutate them and then you you want to let's you want to optimize your eight each agent so each of these environments is paired with a new agent that always tries to solve that particular environment so now within one environment you simply do your classic optimization we already saw here the evolution strategy is akin to a classic optimization algorithm from reinforcement learning all right so each agent you optimize for a couple of steps right not fully every time but for a couple of steps so each agent including the one in the original environment each agent is continuously trained on its environment throughout the process of course you like you have to be you have bounded computation so you need to drop out the very old ones but in principle continuously as all of this goes on all the agents are always trained on their environments so the agent here this Walker will always try to solve this particular environment and the Walker here that is now newly generated when the environment is generated will only try to solve this particular environment throughout the whole algorithm right and then all right so you do mutations you spawn new ones and then you do a couple of steps in optimization right and yes step and then you do this transfer attempt right what you want to do is you want to evaluate all the candidates on all the environments in principle you can you can cut this down but in principle you want to go through the environments and say okay this environment right here I'm going to evaluate all of the other agents in this environment you can do this in a couple of different ways where you just straight up try them or try to optimize them for a few steps to see whether they can be adapted easily to that environment but ultimately you have to come up with a criterion to say for each agent is the agent better or worse than the agent that is continuously trained on this environment if it's worse then you keep this one if if anyone is better then you transfer that better one to replace this one right and you basically copy it over to this new environment and that's where this transfer learning comes in so you're continuously trying all the agents on all the environments and if they are better you transfer them right so here you say if the environment score is better than the one that you have you transfer it all right now there is a lot hidden here for example in this mutate environment step they do check whether or not the new mutated environments are not too hard and not too easy and that basically means whether or not the agents can solve them but not solve them too easily they also check whether the environments are enough novel so you need a couple of checks here you solvable and that that means not too easy and not too hard right so they need to pass like a certain score but they need to be kind of solvable to a to an okay score so there's a score range and also novel they check whether or not the out the mutated environments are novel enough and I believe they just do this by calculating the the distance between two environments in terms of their parameter vectors so to determine whether or not these are novel and sorry I don't mean the distance just between two but the distance of all of the ones you've seen so far so if we go to original very beautiful drawing here where is my tree if you create a new environment let's say you create a new environment right here then you want to check it against all environments you've seen so far to determine whether or not it is new or not so you want to create the distance to all of these and if you have enough distance to your nearest neighbors then you are novel and that's kind of how they they determine whether environment is new all right so that's basically the poet algorithm you continuously create new environments by mutation you ensure that they are solvable not hard enough sorry not too hard but hard enough ensure that they are novel and then you optimize each agent for its own environment continuously as the process goes on and so it's not I want to stress this it's not only the frontier so you're not only looking at the newest generation but you're always looking at all of the generation of the because the older ones while the environments are easier they have been optimized for longer on this environment so the skills might be very handy so you always want to look at your entire population and then you do crucially you do this these transfer attempts so that's the poet algorithm there is a lot hidden here and I kind of want to stress that just if you just look at the amount of hyper parameters there is so many hyper parameters in this how much you transfer how much you mutate how many steps you do each of these subroutines here has a billion hyper parameters and learning rates and and so on so to me that's a that is kind of if I look at this algorithm I am very scared if I attempted to do something like this myself it's it's going to be a long and hard thing to evaluate all of these different hyper parameters that you have to do shortly want to dip into what the evolution strategy does just so you know because you just might be familiar with your classic your classic reinforce algorithm so in policy gradient methods what you do is you scale your parameters of your neural network which is you can if this is your policy then your policy network here you want to scale the gradient according to your reward so in classic reinforcement learning this here would be the reward you got which basically means if you did an action and you got higher reward you want to make your network do that action more right here in evolution strategies what you do is you spawn it's a different way of doing the same thing basically you spawn different environments and sorry you spawn you spawn different agents so you have your current parameters and you want to spawn a number of noisy versions of those parameters and then you want to evaluate each one right and now you want to adjust your parameters into the direction of that particular so basically you are here with your parameters you create a bunch of noisy versions of it right and let's say these two performed really well you want to adjust your parameters into the direction of those two right that's basically what this says so this is the noisy version and then this is the noise that produced the noisy version so if this is high if this number here is high then you will adjust your parameters into that direction it's a fairly cool way if you especially if you can't back prop through your policy as it's pretty neat thing so this is the ES step algorithm but you can think of it just as a RL algorithm all right so they do various experiments to show that this actually has merits I've already shown you if you're trying if you take the same environments and try to solve them directly by this evolution step then it will not succeed because of the problems we've discussed before now the comparison is a bit unfair because um of course these environments for poet poet the problem here is you can't have it solve a particular environments because the environments they constantly change right you constantly mutate the environments you never know where it's going it's not directed so if your goal is to solve a particular environment you cannot do it with poet you can hope that the agent that comes out will perform well right you can do something like this but I believe I believe that these environments that they test on here are ones that appeared during the poet run right so it's kind of an unfair comparison I feel to to do this on an environment that you know this environment this poet agent actually comes from an environment that poet has generated in its all mutation tree curriculum while building it up and then the poor ES algorithm is simply tasked with solving that particular environment from scratch so yes always keep in mind this is this can have a goal this doesn't have a goal right that's kind of the drawback but as you can see poet does get super high scores whereas es the classic algorithm completely fails and they also investigate the importance of transfer learning so they compare to like a classic classic curriculum learning algorithms there are curriculum learning algorithms where you can continuously try to build up the difficulties of these environments but you also do it in a goal-directed way so as I said if you have an environment that has like a gap and then a stump a high stump or two high stumps you want to start out flat and then maybe build in a small gap and a small stump and so on until you're here it's very much goal-directed but it doesn't have this kind of population with transfer learning aspect of poet so if they compare this you can see here the red the red the red one sorry colored it blue stupidly the red one is whatever poet was able to solve now these are the five dimensions of the parameters and the more on the outside it is the harder the environment and for the same for the same environment the blue one is what the curriculum learning algorithm has managed so it's the best environment the curriculum learning algorithm has been able to solve while trying to build up to the so if we take this here is the environment that poet solved again the comparison is kind of unfair because we're starting out from an environment that poet has already solved and then we're trying to build our way up to it with the classic algorithm by basically again this is it's comparing a non goal-directed thing something that just happened to a goal-directed process that needs to get this particular environment to work in any case at some point this curriculum learning algorithm will fail like let's say that's here that's the environment that has somewhat of a gap but no stump right and that would be the the blue line here they do like five runs and they plot them here and you can see every time the classic curriculum learning algorithm manages to only solve a much much less challenging environment than the poet algorithm achieved even though it's it's trying to reach exactly that right and so here they show the difference so if you just the classified environment if it's just challenging then the classic algorithm the curriculum learning algorithm can solve it somewhat so the distance is close to zero but as you go more and more challenging the distance between poet and the classic becomes larger and larger they do give some examples of what this transfer learning does so they have this parent environment that just kind of slouches forward on the ground and then the child environment has a mutation that has now little stumps in it right so you can't get over it right now but the child environment because it's it's a small stump so it might stumble across learns to lift its leg here and it transfers this back to the parent right at a later iteration which is pretty cool and then the parent gets even better as a result of that transfer so we have two transfer learning events here that mutually help these agents remember both the parent and the child are continuously trained as the process goes on all right and they do some more things where they do actual poet not a classic algorithm but poet without transfer learning and they see that okay the poet without transfer is able to solve some of the very challenging problems but never reaches the extremely challenging stage and that's kind of their argument why the transfer learning is necessary so in total I would say this is a cool algorithm it has many many many many many many hyper parameters and these experimental results with that many hyper parameters you need to take it with a grain of salt because it's always possible that they just haven't put as much effort into their comparisons as they have into their own thing to get it to work all right with that I wish you a nice day and check out the paper they have lots of descriptions check out the blog post where they have animations and the YouTube video and with that bye bye
 Alright, so what you're seeing here are solutions found to this bipedal walker problem by a
 new algorithm called PoET.
 So as you might guess, the challenge is to keep this little thing here walking to the
 right as far as you can while it encounters various obstacles.
 And it is and remains a challenging reinforcement learning problem to have an agent learn to
 overcome various obstacles and walk well in different environments.
 So the paper we're going to look at is called PoET.
 It's by Uber Engineering.
 And the full pronunciation is the Paired Open-Ended Trail Blazer, endlessly generating increasingly
 complex and diverse learning environments and their solutions by Roy Wang, Joel Lehmann,
 Jeff Klun and Kenneth O. Stanley, as I said from Uber AI Labs.
 So as you already saw, the challenge they take on is this bipedal walker problem.
 Now their method is very general and not limited to this problem, but this is the problem that
 they focus on.
 I'm going to jump some of the explanations here and dig right into the problem.
 As you can see, the problem is the following.
 You have this thing here, which is the walker, and it has two legs and specifically it has
 four joints.
 So the four joints are here too, and here too.
 And you can give torque on all of the four joints.
 So it's basically a four output problem.
 And you do have sensors as input.
 So the inputs, I believe, is a LIDAR.
 So the LIDAR is this red line you see here.
 I think it has 16 of those in various angles.
 And also it has pressure detection on the feet, I believe, to see whether or not they
 are in contact with the ground.
 And it might also have a gyroscope in that tells you which angle with respect to the
 ground the head is.
 So you have various sensors on these things, and you're able to basically control what
 the legs are doing.
 And your goal is to make this go as far to the right and as fast as possible.
 You see the reward down here is negative 100 if the robot falls over.
 That means if the head hits the ground.
 And then it is 130 times delta x.
 That's how far you go to the right minus the whole angle.
 And the whole angle, as I said, is this angle here.
 So you want to keep it as stable as possible.
 Because if there's a difference in the angle per step, then you get penalized.
 And also you get penalized for each torque you apply.
 So you want to kind of apply minimal force on the joints in order to go very far.
 But by far the most important point is to go to the right as far and as fast as you
 can.
 There is an end here somewhere.
 And if you reach it, you get a score that is above 230.
 They choose the limit of 230 here to determine.
 So if the agent gets 230 or more, then it has solved the environment.
 That's what they claim.
 That's from experience.
 So as you see, the environment has various obstacles here.
 There are holes that you can fall into that you need to jump or step over.
 There are these kind of stumps here.
 They can be of various height.
 So this is a bit shorter and this is a bit longer.
 And the general terrain has a roughness.
 And this can go to very rough from very smooth.
 So this is a parameterized environment.
 And obviously they are able to generate these environments from parameters.
 And the goal now is to have an agent that walks well in any environment that you can
 think of.
 Right, so here on the left you see this is very challenging down the stairs.
 This also isn't too easy because there is a gap here.
 And there are five parameters of these environments.
 So there is the general roughness of the terrain.
 That means how many hills it has and how fast they are coming.
 There is the stump lower bound and stump upper bound, I believe.
 So how high the stumps are.
 And also how long the gaps are.
 And with these parameters you control how difficult an environment is.
 So the straightforward thing to do is simply to sample environments and have a reinforcement
 learn approach to this.
 And that usually doesn't work.
 I already want to see this without having talked about what the algorithm is.
 This is the approach where you try this thing.
 It's called evolution strategies.
 But you can think of it as just a straightforward optimization procedure.
 So there is an agent and there is an environment and you are trying to solve the environment
 using just straightforward optimization.
 Now the evolution strategies are not your classic algorithm but you can compare it to
 it.
 It's just that these people, they like the more, I have a feeling they like the more
 esoteric learning algorithms.
 In any case, you see in these environments large gap, rough surface and so on.
 These are supposed to be the platinum figures.
 So these two environments and also these environments here.
 The evolution strategy, so the classic approach if you just straight forward optimize, they
 get very low scores on average, whereas poet gets here very high scores above the 230 threshold.
 So what's happening?
 If you're trying to just solve these environments from scratch, you basically don't really have
 a big chance of solving them.
 Because let's say you're here and you're trying to move to the right, you know, you might
 learn how to do this and you see this from scratch solution actually manages to get to
 the right.
 But then as soon as you reach this, you're in this gap and you just fall down the gap
 because all you've learned so far is how to move right.
 So what you would need to do is you would need to plan ahead like what poet does.
 You need to see that there is a gap.
 You need to plan ahead and already lift up a leg in order to then step over the gap here
 and then do a little jump right here.
 And this sequence of action, this kind of planning ahead, it is very difficult to learn
 this for a classic RL algorithm because you basically get reward for everything you do.
 So initially you get reward for moving to the right.
 So that's 10 if you reach here, another 10 if you reach here.
 And so there is another 10 if you reach here and another 10 if you reach here.
 Whereas if you lift up your leg, that's like minus five because now this you've changed
 this angle and we saw this is negative reward, right?
 So a classic optimization algorithm will always fall into the hole because that is where you
 get the immediate reward.
 Whereas you'd have to you'd have to do a sequence of action that doesn't give you a reward right
 now, but it gives you more reward later.
 And in order to learn this, we need a kind of a better algorithm that just straightforward
 optimization.
 So maybe I can explain this if you have a maze, here is the start and here is the goal
 and there is like walls and the walls are something like this.
 What you need to do is go around here.
 But what a classic optimization algorithm does is always like goes here because that's
 ever so closer to the goal.
 And then it just gets stuck because it can't fathom that it needs to go around here.
 So it needs to go farther away before it gets closer.
 So these people we've talked about this before in like open ended learning novelty search.
 What you would want to do is you would want to gradually build up solutions that can explore
 the space like to go here, go here, go here and basically build up these solutions.
 And there are two components to what this poet algorithm does.
 So the first component is curriculum learning.
 Curriculum learning.
 What does curriculum learning mean?
 Curriculum learning means that you start off with easy tasks and you increasingly build
 up more and more and more complex tasks.
 So let's say I have an environment here and I'm going to draw and at the beginning we
 just kind of start off with this flat surface right and here is our little walker right
 here.
 And we'll just train it to move right on that and that should be doable with kind of a classic
 approach.
 And then we gradually move to more difficult environments.
 So maybe we'll make it a bit more rough right.
 And an agent that can already walk to the right already kind of has think of it as a
 pre-training in like NLP.
 You can then get more and more challenging and then maybe at some point you can build
 in a gap right.
 So you build in one of these gaps and now it already knows how to move to the right
 and now it might actually learn to jump a small gap right if you make it small at the
 beginning not like this one down here.
 There's a very large gap.
 But if you make it small by accident it might stumble over it and then learn and continuously
 how to master the gap.
 So this is the curriculum learning approach.
 It means that from environment to environment you get harder, harder and harder challenges.
 So first flat then more rough then more rough with a gap and so on.
 The second approach, the second ingredient to POET is what they call stepping stone learning
 or transfer learning or things like this.
 And that's where you kind of have to think of this not as a single agent optimizing but
 as a population of agents.
 So let's say you do this curriculum learning right.
 And you're getting fairly well here at rough terrains right.
 More and more rough terrains.
 But in parallel you also have a second optimization procedure.
 You also start out kind of flat but with this thing you go as we said before small gap you
 keep it flat but you just increase the number of gaps here right.
 Whereas over here you just keep making the terrain rougher and rougher.
 So what the philosophy is that an agent that might be able to master this rougher terrain
 it might actually that skill because here you this kind of this kind of looks like a
 gap here.
 The skill of hopping over this gap here might actually transfer to the environment over
 here where you do have a proper you know a gap in the environment or the skill that you
 learn from an environment where you have one of these stumps right.
 So here let's draw in one of these stumps where you have to go over and if you have
 a walker that can successfully walk over this that skill now might transfer over here in
 order to get over this over this peaky terrain here.
 So the idea of poet is to start off with a generic flat very easy environment and then
 spawn new ones so you want to spawn new environments in kind of a hereditary way.
 So this one might get a bit rougher this one might include this and this one might include
 a gap or something like this and then again you want to spawn new environments and more
 rough more rough more rough with a stump here and this one retains the gap sorry and um
 this one now gets two gaps and so on and you want to continuously train these and then
 always you want to check whether or not the skill that you learn over here might actually
 transfer to anyone over here.
 So you get this tree of this continuous tree of solutions and once you improve on one branch
 this might actually be good on another branch right they always make the comparison to let's
 say biological evolution where a strategy that works over here for birds is all of a
 sudden can be cross adopted by mammals for an entirely different problem but the same
 skill might be valuable.
 Yeah so this this is basically the two ingredients of poet and now I want to show you the complete
 poet algorithm.
 So what does it do you start off with an initial environment right and in poet every environment
 is paired with an agent so there is one agent per environment right so for the time steps
 what you do is first of all you go through your environments and you mutate them and
 we already seen these environments they can be generated from a parameter vector so we
 have five numbers right how rough how stumpy and how wide the gaps are let's say we have
 three numbers to two and this might be one this might be two this might be five right
 so what you want to do is you want to mutate them right you want to spawn children and
 each of these parameters has a chance of mutating this might be one three five and this environment
 might be one four six and this one might be two two five right you spawn new ones you
 already see that the requirement here is that you can actually have environments that are
 procedurally generated and mutated like this where a small mutation probably is going to
 lead to a small change in the environment in any case you mutate them and then you you
 want to let's you want to optimize your eight each agent so each of these environments is
 paired with a new agent that always tries to solve that particular environment so now
 within one environment you simply do your classic optimization we already saw here the
 evolution strategy is akin to a classic optimization algorithm from reinforcement learning all
 right so each agent you optimize for a couple of steps right not fully every time but for
 a couple of steps so each agent including the one in the original environment each agent
 is continuously trained on its environment throughout the process of course you like
 you have to be you have bounded computation so you need to drop out the very old ones
 but in principle continuously as all of this goes on all the agents are always trained
 on their environments so the agent here this Walker will always try to solve this particular
 environment and the Walker here that is now newly generated when the environment is generated
 will only try to solve this particular environment throughout the whole algorithm right and then
 all right so you do mutations you spawn new ones and then you do a couple of steps in
 optimization right and yes step and then you do this transfer attempt right what you want
 to do is you want to evaluate all the candidates on all the environments in principle you can
 you can cut this down but in principle you want to go through the environments and say
 okay this environment right here I'm going to evaluate all of the other agents in this
 environment you can do this in a couple of different ways where you just straight up
 try them or try to optimize them for a few steps to see whether they can be adapted easily
 to that environment but ultimately you have to come up with a criterion to say for each
 agent is the agent better or worse than the agent that is continuously trained on this
 environment if it's worse then you keep this one if if anyone is better then you transfer
 that better one to replace this one right and you basically copy it over to this new
 environment and that's where this transfer learning comes in so you're continuously trying
 all the agents on all the environments and if they are better you transfer them right
 so here you say if the environment score is better than the one that you have you transfer
 it all right now there is a lot hidden here for example in this mutate environment step
 they do check whether or not the new mutated environments are not too hard and not too
 easy and that basically means whether or not the agents can solve them but not solve them
 too easily they also check whether the environments are enough novel so you need a couple of checks
 here you solvable and that that means not too easy and not too hard right so they need
 to pass like a certain score but they need to be kind of solvable to a to an okay score
 so there's a score range and also novel they check whether or not the out the mutated environments
 are novel enough and I believe they just do this by calculating the the distance between
 two environments in terms of their parameter vectors so to determine whether or not these
 are novel and sorry I don't mean the distance just between two but the distance of all of
 the ones you've seen so far so if we go to original very beautiful drawing here where
 is my tree if you create a new environment let's say you create a new environment right
 here then you want to check it against all environments you've seen so far to determine
 whether or not it is new or not so you want to create the distance to all of these and
 if you have enough distance to your nearest neighbors then you are novel and that's kind
 of how they they determine whether environment is new all right so that's basically the poet
 algorithm you continuously create new environments by mutation you ensure that they are solvable
 not hard enough sorry not too hard but hard enough ensure that they are novel and then
 you optimize each agent for its own environment continuously as the process goes on and so
 it's not I want to stress this it's not only the frontier so you're not only looking at
 the newest generation but you're always looking at all of the generation of the because the
 older ones while the environments are easier they have been optimized for longer on this
 environment so the skills might be very handy so you always want to look at your entire
 population and then you do crucially you do this these transfer attempts so that's the
 poet algorithm there is a lot hidden here and I kind of want to stress that just if
 you just look at the amount of hyper parameters there is so many hyper parameters in this
 how much you transfer how much you mutate how many steps you do each of these subroutines
 here has a billion hyper parameters and learning rates and and so on so to me that's a that
 is kind of if I look at this algorithm I am very scared if I attempted to do something
 like this myself it's it's going to be a long and hard thing to evaluate all of these different
 hyper parameters that you have to do shortly want to dip into what the evolution strategy
 does just so you know because you just might be familiar with your classic your classic
 reinforce algorithm so in policy gradient methods what you do is you scale your parameters
 of your neural network which is you can if this is your policy then your policy network
 here you want to scale the gradient according to your reward so in classic reinforcement
 learning this here would be the reward you got which basically means if you did an action
 and you got higher reward you want to make your network do that action more right here
 in evolution strategies what you do is you spawn it's a different way of doing the same
 thing basically you spawn different environments and sorry you spawn you spawn different agents
 so you have your current parameters and you want to spawn a number of noisy versions of
 those parameters and then you want to evaluate each one right and now you want to adjust
 your parameters into the direction of that particular so basically you are here with
 your parameters you create a bunch of noisy versions of it right and let's say these two
 performed really well you want to adjust your parameters into the direction of those two
 right that's basically what this says so this is the noisy version and then this is the
 noise that produced the noisy version so if this is high if this number here is high
 then you will adjust your parameters into that direction it's a fairly cool way if you
 especially if you can't back prop through your policy as it's pretty neat thing so this
 is the ES step algorithm but you can think of it just as a RL algorithm all right so
 they do various experiments to show that this actually has merits I've already shown you
 if you're trying if you take the same environments and try to solve them directly by this evolution
 step then it will not succeed because of the problems we've discussed before now the comparison
 is a bit unfair because um of course these environments for poet poet the problem here
 is you can't have it solve a particular environments because the environments they constantly change
 right you constantly mutate the environments you never know where it's going it's not directed
 so if your goal is to solve a particular environment you cannot do it with poet you can hope that
 the agent that comes out will perform well right you can do something like this but I
 believe I believe that these environments that they test on here are ones that appeared
 during the poet run right so it's kind of an unfair comparison I feel to to do this
 on an environment that you know this environment this poet agent actually comes from an environment
 that poet has generated in its all mutation tree curriculum while building it up and then
 the poor ES algorithm is simply tasked with solving that particular environment from scratch
 so yes always keep in mind this is this can have a goal this doesn't have a goal right
 that's kind of the drawback but as you can see poet does get super high scores whereas
 es the classic algorithm completely fails and they also investigate the importance of transfer
 learning so they compare to like a classic classic curriculum learning algorithms there
 are curriculum learning algorithms where you can continuously try to build up the difficulties
 of these environments but you also do it in a goal-directed way so as I said if you have
 an environment that has like a gap and then a stump a high stump or two high stumps you
 want to start out flat and then maybe build in a small gap and a small stump and so on
 until you're here it's very much goal-directed but it doesn't have this kind of population
 with transfer learning aspect of poet so if they compare this you can see here the red
 the red the red one sorry colored it blue stupidly the red one is whatever poet was
 able to solve now these are the five dimensions of the parameters and the more on the outside
 it is the harder the environment and for the same for the same environment the blue one
 is what the curriculum learning algorithm has managed so it's the best environment the
 curriculum learning algorithm has been able to solve while trying to build up to the so
 if we take this here is the environment that poet solved again the comparison is kind of
 unfair because we're starting out from an environment that poet has already solved and
 then we're trying to build our way up to it with the classic algorithm by basically again
 this is it's comparing a non goal-directed thing something that just happened to a goal-directed
 process that needs to get this particular environment to work in any case at some point
 this curriculum learning algorithm will fail like let's say that's here that's the environment
 that has somewhat of a gap but no stump right and that would be the the blue line here they
 do like five runs and they plot them here and you can see every time the classic curriculum
 learning algorithm manages to only solve a much much less challenging environment than
 the poet algorithm achieved even though it's it's trying to reach exactly that right and
 so here they show the difference so if you just the classified environment if it's just
 challenging then the classic algorithm the curriculum learning algorithm can solve it
 somewhat so the distance is close to zero but as you go more and more challenging the
 distance between poet and the classic becomes larger and larger they do give some examples
 of what this transfer learning does so they have this parent environment that just kind
 of slouches forward on the ground and then the child environment has a mutation that
 has now little stumps in it right so you can't get over it right now but the child environment
 because it's it's a small stump so it might stumble across learns to lift its leg here
 and it transfers this back to the parent right at a later iteration which is pretty cool
 and then the parent gets even better as a result of that transfer so we have two transfer
 learning events here that mutually help these agents remember both the parent and the child
 are continuously trained as the process goes on all right and they do some more things
 where they do actual poet not a classic algorithm but poet without transfer learning and they
 see that okay the poet without transfer is able to solve some of the very challenging
 problems but never reaches the extremely challenging stage and that's kind of their argument why
 the transfer learning is necessary so in total I would say this is a cool algorithm it has
 many many many many many many hyper parameters and these experimental results with that many
 hyper parameters you need to take it with a grain of salt because it's always possible
 that they just haven't put as much effort into their comparisons as they have into their
 own thing to get it to work all right with that I wish you a nice day and check out the
 paper they have lots of descriptions check out the blog post where they have animations
 and the YouTube video and with that bye bye