Enhanced POET: Open-Ended RL through Unbounded Invention of Learning Challenges and their Solutions
deep learning machine learning unbounded open-ended evolution evolutionary uber uber ai distributed reinforcement learning rl generative
The enhanced POET makes some substantial and well-crafted improvements over the original POET algorithm and excels at open-ended learning like no system before.

https://arxiv.org/abs/2003.08536
https://youtu.be/RX0sKDRq400

Abstract:
Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.

Authors: Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, Kenneth O. Stanley

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 There, before we jump into today's paper, I just want to give a shout out to Machine Learning Street Talk, where every week we talk about current or big trends or topics in machine learning. The first discussion that we launched is actually on today's paper, The Enhanced Poet. So if you like the following video, you might want to jump over to Machine Learning Street Talk and check out our discussion about it. It's very interesting. Alright, have fun. Hi there. What you're seeing here are many different environments from a single run of a system that's called The Enhanced Poet. Last time we've taken a look at a system called Poet, and The Enhanced Poet is kind of an improvement over the original Poet, fixing some of its shortcomings. And you see here that the agent is able to solve this very, very diverse set of environments. And the notable thing is, this is from a single run of this algorithm. So one run will produce all these different environments and will produce agents that are able to solve all the different environments at the same time in parallel. So it's a population-based method. If you haven't seen the video I did on Poet, I suggest you go and see that now. This is simply an enhancement to it, and I expect people to know kind of what I'm talking about. Alright, it's going to be a short video, but I think it is a good addendum to Poet. So it's The Enhanced Poet, Open-ended Reinforcement Learning Through Unbounded Invention of Learning Challenges and Their Solutions by Rui Wangchou, Leymann Adhitar Wahl, Jial Qi, Julun Li, Jeff Klun, and Kenneth O. Stanley. So we'll jump right in. They make a number of improvements to the original Poet, and I simply want to discuss the most important ones. So you know, they have a nice graphic down here of what happens in Poet. Poet builds up this tree of environments, and to each environment it has an agent that it trains to solve that environment at the same time. So at the same time it will kind of start out here. It will generate offspring. It will continuously generate offspring, and then it will also continuously train agents in each environment that it produced in order to solve that environment. And it keeps doing that while producing more and more offspring. And then once in a while it does what is called a transfer. So that means that, for example, you see here the offspring produced here from this environment. You kind of see that the lineage here kind of focuses on squiggly environments, right? You see that there's a bit of a squiggle here and a bit of a squiggle here. And then the offspring all of a sudden is a bit more smooth, but has this little step here. And then this offspring of this environment has this large step here. Now the agents that come from here have kind of been optimized to solve the squiggliness problem. But here, over here, this lineage has specified or specialized more and more in kind of like these kind of large drops or steep hills. So the agent that was trained over here was found to be very effective in this environment and therefore can be transferred. So this kind of population branching out into the different trees and then transferring solutions between the parts of the trees, that's what makes Poet very very powerful mechanism to solve these kind of tasks. All right, so how does this improve? Now the first thing that Poet does is it generates these environments and it always wants to generate new environments. So it always generates offspring to each environment. So let's say we are here, it will generate offspring to each environment here, each that we have. Let's see, we have only seen so far. And then it only picks the most novel ones, the ones that are most novel, which is this, probably this. Then there are other criteria, namely that it can be solved by some agents, but it cannot be solved by others. It's not too difficult, but also not too hard. But one of the aspects is it must be novel, right? So we're not seeing any here, which means that those weren't novel enough. How does it measure novel? In the original implementation of Poet, you had this environment generator, which was like a level generator, which made these gaps here and the stumps here. And you could specify, I believe, five numbers. So there was a five-point scale in which you could specify how high the stumps were. You get this kind of pentagon here, how high the stumps were and how deep the gaps were and how rough the terrain was. And the level generator would generate this level. And so basically your distance metric between environments was a vector of size five, right? This is environment one. And you had environment two, which if it's more, it has higher stumps, right? Than this particular number here, maybe would be higher than this number here. So it was kind of limited to taking the Euclidean distance between two environment encodings in order to measure the distance between environments. This is very, very domain specific. And the authors here argue what we should rather do is have a general environment agnostic distance metric, right? So here is what they propose. They propose the following. Why don't we, if we have a new environment, right? Let's say we have a new environment. We measure all of the agents, the current agents and the ones we've already seen, right? We measure all the agents in our database on this new environment. That's this. And they come up with scores, right? Each of them gets a score. And then we, you know, clip and bound the score. So the max here is 300 and the minimum is 50. But in any case, we then rank them, right? So we evaluate them and then we rank them from best to worst. And then we normalize, which simply means that the best one gets a score of 0.5 and the worst one gets a score of negative 0.5. And now this vector here, this is now used to compare environments. So if we have another environment, right? Right here, we have E2 and that gets a different ordering, right? So maybe agent one is now the best agent two is really bad and so on, right? That gets a different ordering. Then the resulting vector here will be very, very different from from this vector right here. And this is very agnostic. So no matter which environment it is, if the ordering of agents in it, the score they get, the order of it is the same, the environments aren't really different from each other, the authors argue. But if the scores are very differently ranked, right? So imagine the environment is harder but essentially the same, then the scores will be lower, but still the agents would be ranked the same. So you can argue, well, that's just kind of the same environment, except a step like this now has a super steep step, right? It's not very different. But if instead of that, you get an environment that is like this, like you say, wow, that's qualitatively different. And you would expect from this one to this one that the agents would be ranked the same, the agents performance would roughly stay the same, but you would expect from the middle one to this one that an entirely different set of agents might perform well right in this one. So that's how novelty is measured and I think it's a pretty cool way. I don't have coronavirus, by the way, maybe, who knows? No, I just have a dry throat. All right, so this is the first enhancement they make is that they now measure novelty in this domain agnostic way. Pretty cool so far. And what this allows them to do, this allows them to actually not rely on this level generator with the five parameters in order to generate these levels. But these levels can now be produced however they want with different generators and that's exactly what they do. They now employ neural networks. Well, it's kind of a prototypical, it's called a CP&N that generates these things. You might have seen in the examples the enhanced poet doesn't have these gaps and stumps anymore. It simply has these landscapes that are super diverse, but they're still just their landscapes. And what it does is it evolves neural networks at the same time as it evolves the population. It evolves these, so the architecture of these networks isn't fixed. It's actually evolving along with the agent to make the challenges harder and harder. So you see there are like cosines and sines in here and you can add them and subtract and so on. And that will give you a mapping from x, which is the x coordinate here, to y, which is the y coordinate. And that will give you kind of a continuous landscape depending on the architecture here and on the internal parameters of course. I guess there would also be a node, some here like times a lambda factor and then the lambda would also be a thing that is evolved. So pretty cool. Of course the internals of this now aren't just described by a fixed vector anymore, but you don't need that anymore because we have a method to compare environments even if they come from completely different architectures of generators. So it's pretty cool that the agnostic comparison of environments allows you to now have a much more general level generator and of course now produce much more diverse environments. And that's exactly what they see. Of course you see here the environments get super crazy. So they also propose kind of a novel metric to measure novelty, sorry to measure progress. So the question is how do we measure progress in these algorithms, in these open-ended algorithms? And what they propose is this ANNX score, which is, I have to go and look it up, the ANNX score I think is the number of new environments that are solved. Yes, so exactly. The question is whether a system continues to generate interesting new things. And the way they measure it is by the accumulated number of novel environments created and solved. So the question here is accumulated, that means over the entire run they count up how many environments that they've seen that are novel, and we've already had the definition of novel. And in this case it basically means that it must pass the minimal criterion. It's neither too hard nor too easy. We've already seen this in how the offspring of environments is generated. There's a minimal criterion and it must be eventually solved. So that means the novel environments created and solved. So how many new environments are created and solved? And then at a later point solved. You can see the difference to the original poet in this graph. So the original poet eventually runs out of new environments because its generator is just not powerful enough. It can only modify these five variables and eventually the environments aren't substantially novel from the old environments. Whereas the enhanced poet you can see even after this run, and I'm sure they have large infrastructure to do these experiments, it just continues to innovate new more elaborate environments continuously. So this I think are the main things. They also do some improvement to the transfers and so on. I don't want to go into that. I just wanted to show these improvements so that you have the complete picture of how such an algorithm runs. My criticism to this is that if you just look at their thing is that with the leaving out of the gaps and the stumps and so on, in a weird way, of course the levels are diverse, but they have become even more similar it seems. Like you're really relying on your ability to kind of continuously create these levels. Kind of like a GAN for levels, right? And you're relying on your ability to smoothly increase the difficulty of the levels, right? To actually have a diversity in your level generator, but also a kind of a smoothness with regard to the difficulty in order to build this whole curriculum. And I think even though the environments look more diverse, it might be going into a direction where you kind of engineer yourself into a corner where you are now even more and more relying on these evolving and parameterizable generators. Nonetheless, the ideas I think are pretty cool and that's all I have to say about it. Bye bye!
 There, before we jump into today's paper, I just want to give a shout out to Machine Learning Street
 Talk, where every week we talk about current or big trends or topics in machine learning.
 The first discussion that we launched is actually on today's paper, The Enhanced Poet. So if you like
 the following video, you might want to jump over to Machine Learning Street Talk and check out our
 discussion about it. It's very interesting. Alright, have fun. Hi there. What you're seeing here are
 many different environments from a single run of a system that's called The Enhanced Poet.
 Last time we've taken a look at a system called Poet, and The Enhanced Poet is kind of an improvement
 over the original Poet, fixing some of its shortcomings. And you see here that the
 agent is able to solve this very, very diverse set of environments. And the notable thing is,
 this is from a single run of this algorithm. So one run will produce all these different
 environments and will produce agents that are able to solve all the different environments
 at the same time in parallel. So it's a population-based method. If you haven't
 seen the video I did on Poet, I suggest you go and see that now. This is simply an enhancement to it,
 and I expect people to know kind of what I'm talking about. Alright, it's going to be a short
 video, but I think it is a good addendum to Poet. So it's The Enhanced Poet, Open-ended
 Reinforcement Learning Through Unbounded Invention of Learning Challenges and Their Solutions
 by Rui Wangchou, Leymann Adhitar Wahl, Jial Qi, Julun Li, Jeff Klun, and Kenneth O. Stanley.
 So we'll jump right in. They make a number of improvements to the original Poet, and I simply
 want to discuss the most important ones. So you know, they have a nice graphic down here of what
 happens in Poet. Poet builds up this tree of environments, and to each environment it has an
 agent that it trains to solve that environment at the same time. So at the same time it will kind of
 start out here. It will generate offspring. It will continuously generate offspring, and then it will
 also continuously train agents in each environment that it produced in order to solve that environment.
 And it keeps doing that while producing more and more offspring. And then once in a while
 it does what is called a transfer. So that means that, for example, you see here the offspring
 produced here from this environment. You kind of see that the lineage here kind of focuses on
 squiggly environments, right? You see that there's a bit of a squiggle here and a bit of a squiggle
 here. And then the offspring all of a sudden is a bit more smooth, but has this little step here.
 And then this offspring of this environment has this large step here. Now the agents that come
 from here have kind of been optimized to solve the squiggliness problem. But here, over here,
 this lineage has specified or specialized more and more in kind of like these kind of large
 drops or steep hills. So the agent that was trained over here was found to be very effective
 in this environment and therefore can be transferred. So this kind of population branching out into the
 different trees and then transferring solutions between the parts of the trees, that's what makes
 Poet very very powerful mechanism to solve these kind of tasks. All right, so how does this improve?
 Now the first thing that Poet does is it generates these environments and it always wants to generate
 new environments. So it always generates offspring to each environment. So let's say we are here,
 it will generate offspring to each environment here, each that we have. Let's see, we have only seen
 so far. And then it only picks the most novel ones, the ones that are most novel, which is this,
 probably this. Then there are other criteria, namely that it can be solved by some agents,
 but it cannot be solved by others. It's not too difficult, but also not too hard. But one of the
 aspects is it must be novel, right? So we're not seeing any here, which means that those weren't
 novel enough. How does it measure novel? In the original implementation of Poet, you had this
 environment generator, which was like a level generator, which made these gaps here and the
 stumps here. And you could specify, I believe, five numbers. So there was a five-point scale in
 which you could specify how high the stumps were. You get this kind of pentagon here, how high the
 stumps were and how deep the gaps were and how rough the terrain was. And the level generator
 would generate this level. And so basically your distance metric between environments was
 a vector of size five, right? This is environment one. And you had environment two, which if it's
 more, it has higher stumps, right? Than this particular number here, maybe would be higher
 than this number here. So it was kind of limited to taking the Euclidean distance between two
 environment encodings in order to measure the distance between environments. This is very,
 very domain specific. And the authors here argue what we should rather do is have a general
 environment agnostic distance metric, right? So here is what they propose. They propose the
 following. Why don't we, if we have a new environment, right? Let's say we have a new
 environment. We measure all of the agents, the current agents and the ones we've already seen,
 right? We measure all the agents in our database on this new environment. That's this. And they
 come up with scores, right? Each of them gets a score. And then we, you know, clip and bound the
 score. So the max here is 300 and the minimum is 50. But in any case, we then rank them, right? So
 we evaluate them and then we rank them from best to worst. And then we normalize, which simply
 means that the best one gets a score of 0.5 and the worst one gets a score of negative 0.5. And
 now this vector here, this is now used to compare environments. So if we have another environment,
 right? Right here, we have E2 and that gets a different ordering, right? So maybe agent one is
 now the best agent two is really bad and so on, right? That gets a different ordering. Then the
 resulting vector here will be very, very different from from this vector right here. And this is very
 agnostic. So no matter which environment it is, if the ordering of agents in it, the score they get,
 the order of it is the same, the environments aren't really different from each other, the
 authors argue. But if the scores are very differently ranked, right? So imagine the
 environment is harder but essentially the same, then the scores will be lower, but still the
 agents would be ranked the same. So you can argue, well, that's just kind of the same environment,
 except a step like this now has a super steep step, right? It's not very different. But if
 instead of that, you get an environment that is like this, like you say, wow, that's qualitatively
 different. And you would expect from this one to this one that the agents would be ranked
 the same, the agents performance would roughly stay the same, but you would expect from the middle
 one to this one that an entirely different set of agents might perform well right in this one.
 So that's how novelty is measured and I think it's a pretty cool way. I don't have coronavirus,
 by the way, maybe, who knows? No, I just have a dry throat. All right, so this is the first
 enhancement they make is that they now measure novelty in this domain agnostic way. Pretty cool
 so far. And what this allows them to do, this allows them to actually not rely on this level
 generator with the five parameters in order to generate these levels. But these levels can now
 be produced however they want with different generators and that's exactly what they do.
 They now employ neural networks. Well, it's kind of a prototypical, it's called a CP&N that generates
 these things. You might have seen in the examples the enhanced poet doesn't have these gaps and
 stumps anymore. It simply has these landscapes that are super diverse, but they're still just
 their landscapes. And what it does is it evolves neural networks at the same time as it evolves
 the population. It evolves these, so the architecture of these networks isn't fixed. It's actually
 evolving along with the agent to make the challenges harder and harder. So you see there
 are like cosines and sines in here and you can add them and subtract and so on. And that will give
 you a mapping from x, which is the x coordinate here, to y, which is the y coordinate. And that
 will give you kind of a continuous landscape depending on the architecture here and on the
 internal parameters of course. I guess there would also be a node, some here like times a lambda
 factor and then the lambda would also be a thing that is evolved. So pretty cool. Of course the
 internals of this now aren't just described by a fixed vector anymore, but you don't need that
 anymore because we have a method to compare environments even if they come from completely
 different architectures of generators. So it's pretty cool that the agnostic
 comparison of environments allows you to now have a much more general level generator and of course
 now produce much more diverse environments. And that's exactly what they see. Of course you see
 here the environments get super crazy. So they also propose kind of a novel metric to measure
 novelty, sorry to measure progress. So the question is how do we measure progress in these
 algorithms, in these open-ended algorithms? And what they propose is this ANNX score, which is,
 I have to go and look it up, the ANNX score I think is the number of new environments that are solved.
 Yes, so exactly. The question is whether a system continues to generate interesting new things.
 And the way they measure it is by the accumulated number of novel environments created and solved.
 So the question here is accumulated, that means over the entire run they count up how many
 environments that they've seen that are novel, and we've already had the definition of novel.
 And in this case it basically means that it must pass the minimal criterion. It's neither too hard
 nor too easy. We've already seen this in how the offspring of environments is generated.
 There's a minimal criterion and it must be eventually solved. So that means the novel
 environments created and solved. So how many new environments are created and solved?
 And then at a later point solved. You can see the difference to the original poet in this graph.
 So the original poet eventually runs out of new environments because its generator is just not
 powerful enough. It can only modify these five variables and eventually the environments aren't
 substantially novel from the old environments. Whereas the enhanced poet you can see even after
 this run, and I'm sure they have large infrastructure to do these experiments,
 it just continues to innovate new more elaborate environments continuously.
 So this I think are the main things. They also do some improvement to the transfers and so on.
 I don't want to go into that. I just wanted to show these improvements so that you have
 the complete picture of how such an algorithm runs. My criticism to this is that if you just
 look at their thing is that with the leaving out of the gaps and the stumps and so on,
 in a weird way, of course the levels are diverse, but they have become even more similar it seems.
 Like you're really relying on your ability to kind of continuously create these levels. Kind of like
 a GAN for levels, right? And you're relying on your ability to smoothly increase the difficulty
 of the levels, right? To actually have a diversity in your level generator, but also a kind of a
 smoothness with regard to the difficulty in order to build this whole curriculum. And I think
 even though the environments look more diverse, it might be going into a direction where you kind of
 engineer yourself into a corner where you are now even more and more relying on these evolving
 and parameterizable generators. Nonetheless, the ideas I think are pretty cool and that's
 all I have to say about it. Bye bye!