ACCEL: Evolving Curricula with Regret-Based Environment Design (Paper Review)
#ai #accel #evolution

Automatic curriculum generation is one of the most promising avenues for Reinforcement Learning today. Multiple approaches have been proposed, each with their own set of advantages and drawbacks. This paper presents ACCEL, which takes the next step into the direction of constructing curricula for multi-capable agents. ACCEL combines the adversarial adaptiveness of regret-based sampling methods with the capabilities of level-editing, usually found in Evolutionary Methods.

OUTLINE:
0:00 - Intro & Demonstration
3:50 - Paper overview
5:20 - The ACCEL algorithm
15:25 - Looking at the pseudocode
23:10 - Approximating regret
33:45 - Experimental results
40:00 - Discussion & Comments

Website: https://accelagent.github.io
Paper: https://arxiv.org/abs/2203.01302

Abstract:
It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at this http URL.

Authors: Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Check this out. What you're seeing here is a bunch of agents that have all never seen this level before. This level is in fact procedurally generated and the agents must somehow overcome the obstacles right here. You can see there's stumps, there's gaps. The green one is performing pretty well right here. Coincidentally, the green one is also what we're going to look at in today's paper. The idea here is, as I said, these agents have never seen these environments and the environments are procedurally generated. Every time I hit reset here, a different environment is created. Also notably on the right side right here, I have these sliders with which I can control the different properties of the procedurally generated environments, such as how wide the gaps are, how many steps to the stairs there are. As I modify these, you can see the environments get more and more challenging as I slide these things to the right hand side. Now, they get super challenging at some point and the question is, how do we train an agent using reinforcement learning in order to be able to solve these challenging environments? Because it's pretty clear that if I want an agent to solve an environment like this, and remember it's a procedurally generated environment, so I can't just train it on the same environment over and over and over again until it gets it. If I want to train an agent to solve the family of environments that are very hard here, it's almost impossible to do so using from scratch reinforcement learning because there's just never any success of any of the agents. They never finish an episode, they never get good reward, they always stumble at the first obstacle. So what's the way we... I still want the green one to actually make this. Come on green one, come on! It's not gonna make it right. So the idea is that what we want to do is we want to develop a curriculum. So a curriculum means that we're going to use this ability to create levels of different difficulties to guide the agent to learn more... No... to learn more and more difficult environments. So we're going to start with very easy environments, very flat environments, not many gaps in them, not many stairs in them. So fairly easy environments like this. And we use reinforcement learning and try to teach the agent just to solve this level. Now most of them will do a fairly good job at that level. As you can see, not too much of a problem. Some stumble, some don't, but you know this is solvable. And then we will progressively, as the agent gets better and better, increase the difficulties of the level. And using that, using that difficulty increase over time, there is a chance that the agents, they learn more and more to go and solve these levels. So from scratch learning of the difficult environment might not be possible. However, there is a chance if we design a curriculum in the correct sequence of difficulties for the agents to learn. This is not unlike humans learn in... You may have heard of this... What you want to do is train in the zone of proximal development or something like this, which essentially means that you want to always challenge yourself just outside of your current abilities. And that's how you maximize your progress in learning. That's the same idea that we have here with these evolving curricula over time. So the paper we're going to look at is called Evolving Curricula with Regret-Based Environment Design by Jack Parker Holder and Minki Jiang and others, mainly by Minki Jiang and others, mainly by Minki Jiang. And others, mainly by Meta AI, but there's a bunch of collaborations with UC Berkeley, University of Oxford, and yeah, I guess that's it. So this paper combines the recent developments in regret-based algorithms that go about making a curriculum and evolution, which is another way that people go about this. So the paper proposes to train a single agent, not a family of agents, a single agent that is generally capable of solving all kinds of difficulties and levels. And to do that via an automated curriculum that is given by a teacher algorithm. The teacher algorithm itself is not learned. The teacher algorithm is actually defined by this schematic right here. And all of this is regret-based, which makes it independent of kind of domain-specific heuristics. So the goal of this algorithm right here is to have a general algorithm to design these curricula without being reliant on essentially creating a new heuristics for all of the different tasks it needs to solve. So we're going to look at it. Here's a brief overview over the algorithm itself. How does it do it? How does it get an agent to learn step by step? And the most difficult question is, you know, how fast do you increase with the difficulties of your levels? Because if you increase not fast enough that you're essentially stuck in learning, if you increase the difficulty too fast, you have the same problem again, in that the agent will not be capable of keeping up. So what you want to do is you want to have some sort of a level generator. And that is what we just saw before in this web demo. By the way, you can go look, try out this web demo for yourself at accelagent.github.io. I'll obviously, I'll link it in the description to this video. But you want to have some sort of a level generator, which is essentially the thing that I have here on the right. I want to have the ability to create different levels. This doesn't need to be parameterized like it is here. For example, in this maze world that they portray right here, all I have is an empty room, and then I have the ability to place blocks in it. So every pixel can either be a wall or not a wall. And that's it. That's a generator. The generator can just place blocks and that's it. There's no need for some sort of a slider here that controls the difficulty. That's going to be done completely automatically as you'll see. So once we have the generator, we could already build some sort of a curriculum algorithm, right? We could just sample different levels from the generator and then just train the agent on all of them. However, that wouldn't amount to much of a curriculum as it would probably generate easy and hard levels all throughout each other. And the agent would be able to solve the easy levels maybe a little bit, and then maybe a bit of the harder levels. But if you don't sequence this correctly, there's a big chance that you're going to fail, mostly because as the level design space gets higher and higher, most levels are either going to fall in the too easy or way too hard section. And not a lot are going to be in that zone of proximal development. And therefore you don't have much of a learning signal. So we need to somehow filter and curate these levels that we generate. So we have a generator and the generator simply gives us the starting bunch of levels. And I believe you can also go to the generator within the algorithm and so on. But imagine the generator gives us just a bunch of starting levels. This is one of these starting levels. I'm going to take a different color right here. Otherwise, you won't see. That's even worse. Thank you. So the generator gives us a bunch of starting levels. And these go to the student, again, the student here, that's a single agent, that is not a family of agents. The evolutionary methods here are not in with regard to the student, but to the levels themselves. So there's one student that trains on all the different levels. So what we do is we simply evaluate, we ask, we let the student run on this level and we see how well it does. And we're going to measure its regret. So the regret of a student, we're going to get to that measure. It's essentially an estimate of how far the student is away from the optimal policy on that particular level. And what we want to do is we want to strictly select for levels that have high regret. So levels where the student is far away from the optimal policy, because those are the levels where the student can still learn something. And if we do that correctly, then this automatically sequences these levels in the sequence of difficulty such that they're always just at the edge of what the student can do. And you'll see how that works in a bit. So we want to measure their regret. And we have this, we have the buffer right here. The buffer is where all the levels that we currently think are interesting for the student to learn at reside. This buffer is managed by the curator. The curator is essentially just a bucket of levels that we think are interesting. What we then do is we can replay those levels. So we can actually train the student on the levels. But if we just train the students on these levels, that's not much of an interesting thing. So we also need a way to update that buffer. And the way we update the buffer is we select some of the levels for editing. So some of the levels we think, okay, these are good levels, but could we make them like just a bit more difficult because the student can solve them now. So what's a way to make them more difficult, then we send them through an editor. And the editor again, this can be pretty much anything. So in our example up here, the editor could simply either place another block right here, or remove a block. What is important is that it's different from the generator. The generator just generates a new thing while the editor modifies the existing things. And the assumption is that if I modify something that has a difficulty x, then if I modify it to x hat, then the difficulty of x hat will not be too much different. So what I'm going to do is I'm at, let's say here is the student's starting point, and the student increases its ability round by round. So maybe this is the zone that the student can solve right now. And I select a level that is here, so the student can just about solve it. And then I modify that with the editor a little bit. And I maybe produce a produce different offspring, like here, here, here, and here. So what I want to do is I want to select for the offspring. And here's where that's where the evolutionary method comes in. I want to select for the offspring that will make progress for the student so that the student just can't solve right now. And add that to the buffer of things where I do reinforcement learning on. So with the editor, I create a bunch of different offspring for this level, as we see right here. And I evaluate the student on them, I measure the students regret. And if the regret is high, I put that back into the buffer. So in this way, I always keep the buffer filled with levels that the student just can't like it's just at the zone of where the student can solve them. So if I now add the blue circled levels, obviously the next you know, I'm going to increase my ability to out here a little bit in this direction, right. And then maybe here is another level that I modify with these two, and that increases the student's ability to hear. And then from these levels, I will again create offspring maybe to hear and hear again, I will filter out the ones that become easier. And so, as you can see the students abilities, they will continually increase guided by this metric of this regret. So that's the entire algorithm. Essentially, you'll have one student that is generally capable. And the buffer right here will always contain levels that the student just can't, or just about can solve by measure of these regret and continuously editing. Obviously, this doesn't work everywhere. Like there needs, there's a lot of preconditions for this to work. For example, you need to be able to have this level generator and level editor. You need to be able to create levels of various difficulties, not out of the box, but it like should be possible in principle. There should be the possibility of creating a curriculum in the first place, which is not possible for all the tasks, especially with the condition that if I modify the problem a little bit, like this thing right here, if I modify the problem a little bit, then the difficulty should only be modified by a little bit. Like that is not a given for many, many tasks. However, if this is all given, then it becomes suddenly possible. And of course, we run into all the problems of having a single student, like there's catastrophic forgetting and so on. But we don't we don't worry about this right here. As you might have seen previously, that the Excel agent right here, this the green agent, no matter kind of what the terrain is, its strategy is always sort of the same. So its strategy is always to kind of hold one leg out and bounce on the hind leg. And okay, that that might not have been so it will always it's not going to make that it was bounce on the hind leg, actually, most of them will do it bounce on the hind leg and kind of wiggle the front leg. And that's how it bridges gaps and stairs and ladders and so on. Okay, most of them do that. But you'll see that this is a problem of I think having a single single agent solve these things. If you want a single agent to be solved to solve all the environments, that means that implicitly kind of one one strategy or one set of strategies must be enough to solve all the environments, which is also not a given for much of the world of reinforcement learning. However, this can all be fixed. So this was the overview. Now let's dive into a little bit more into the algorithm itself. Again, we have not yet we there's still a crucial element. And that is this regret that we haven't talked about yet. But the algorithm in code looks like this. I want to I want to initialize a policy that this is the student policy pi and this level buffer. So the buffer is is lambda, I guess lambda. Okay, so I'm gonna sample some initial levels. And I'll just assume that the initial levels here, they're going to be mixed in difficulty. So they're going to be some some easy levels, and some hard levels, and some levels that the student might just be able to solve out of the box or not. So what then we're going into a while loop, the big like while not converged, we're going to sample a replay decision. And the replay decision is essentially it's a binary variable that tells me, do I want to take a level from the buffer? Or do I want to take a level from the new level from the generator? Because if you only have initial levels in your buffer, right, then you're kind of limited by the evolution of these levels. So much unlike we have non convex optimization problems in deep learning, these these landscapes of levels might be super duper non convex. And that's why if you just evolve a bunch of levels, there is obviously the danger that you get that you sort of narrow like you, you never. So if you if you go down a bunch, if you teach the agent to go like down a bunch of stairs, and you go ever more and more stairs, more and more stairs, but the initial levels never had like a big cliff like this, your agent will not be able to solve it even with this method, because no amount of adding stair steps will get you to the big cliff. And that's why it's important to every now and then actually sample a level from the level generator to bring some diversity in there. Because that's what I see with this method is probably pretty easy to teach yourself into a corner. So if we have something from the level generator, we collect the trajectory. And it's important that we have two different modes right here, we have the student in evaluation mode. So every time that we have some level, some new level, we first evaluate the student on it, we want to know whether the student can actually solve it or not on how well it can solve it. So what do we do? We compute the approximate regret, we don't actually train on this level, we just evaluate it. And that is a property, I think that reduces the signal to noise ratio tremendously, we want to pre filter what levels we train on, we don't just want to train on all of them. So this is a this is interestingly enough, a method where even though we have the training data available, it seems to be better if we filter the training data, it's still good training data, right? Any of these levels is good training data for reinforcement learning. It's not like there's noisy data or the label is wrong or something. But it seems to be quite important to accurately select the levels we want to train on. So that is that is an interesting thing by itself. But you what you'll see in this algorithm is that they always will first evaluate a level, determine whether the regret is high or whether it is in the zone of proximal development and only then use this that level to actually train the agent on. That is interesting. So we compute this regret, and we add the level to the buffer. So the level here is this theta. So these are the parameters again here that we evolve, we evolve two sets of parameters, the parameters of pi, which is the student's policy. But that is just a very simple proximal policy optimization, reinforcement learning algorithm right here, we don't actually care what kind of RL algorithm it is as long as it can learn. The interesting parameters here are the parameters of the levels. And this could be the level itself in case of this maze, or it could be the parameters. No, actually, it would be the level the level itself. Right, it needs to be an actual instantiation of the level, not just the parameters that you enter into the generator, unless the generator is deterministic. And we only add it to the buffer if the score meets a threshold. So that is where we filter out things where the regret is either where the regret is too low. So only if it is a hard level for the student to solve, we put it into the buffer and we'll get to how we actually filter out the levels that are too hard in a second. So that's just if we decide we need a new level. If we decide actually that we want to go into the buffer, we're going to sample a level that we've previously added into the buffer. And remember, we've determined that all of these are in the zone of proximal development. We train, we collect the policy, and we actually train. So this is where we train. We train on a level that we sampled from the buffer in the first place. It's the only time we train the agent at all. And then we are not done with this level yet. What we do is we take the same level that we just sampled and we actually edit it. So here, edit to produce theta prime. And the editing can be, as I said, anything as long as you can reasonably assume that any edit will not distort the difficulty too much. So it needs to distort the difficulty somewhat, but not too much. Again, we collect the trajectory, we do not train it, we simply run the student on the new levels, exact same way we did before, we compute the regret, and we add it to the buffer if the score meets a threshold. Optionally update the editor using the score. So that can be the editor itself could be some sort of dynamic algorithm or not. So that is the algorithm in a nutshell. It's pretty simple. There is a buffer. I train on levels inside the buffer and only on levels that are inside the buffer. How do levels get into the buffer? Two ways. They can be sampled either from the level generator, or they can be edited from levels that are already in the buffer. However, both of them will only get into the buffer if we evaluate the agent on them first, compute its regret, and if the regret is higher than some threshold. That's how we curate the buffer. And that's it. That's the entire algorithm. So they have a bunch of experiments right here. And that's, it's probably better to go back to the website to look at the experiments. So, oh no, we need to look at what the regret is, obviously. So regret is just the way it's formulated right here. The regret is the difference between the expected rewards of two policy. So if I have a, this here is regret. So the regret of theta, and now you know theta is a level, right? So the regret specific to a level would be, and here is policy one and policy two. Now in this case, it's the current policy and the optimal policy. But you can see down here, the regret can be defined over any two arbitrary policies. It is simply the difference in the values of the two policies. And what's the value? The value is the expected future reward. And if I pose it like this, it's probably just the expected reward. So the formulation right here, where I plug in the optimal policy would simply be, you know, what, I have some sort of level, right? And I have my current agent right here. And the agent expects to get some sort of reward, like maybe it gets onto here and then it crashes. So that's a reward of, I don't know, 50. And the optimal policy, if the level is solvable at all, it could actually go to the end and solve it and get a reward of 100. So my regret in this case would be 50. And that is a good measure of how difficult a level is, or let's say how much you can still learn from that level. Because if a level is too difficult, and that's the catch, if a level is too difficult, then not even the optimal policy will be able to achieve much in that level. And therefore, you know, why are you like, what point is it to go to that level and actually solve it? Or if there is any stochasticity, if a level needs a lot of luck, right, then as well, the expected the expected reward, the expected future reward of the optimal policy will also be not super high. So by selecting things that have high regret, meaning that have a high difference between the optimal policy and the current policy, we select for levels that where the current student can still learn a lot of things. So it's still there's still headroom to learn. Now, the optimal policy is obviously hard to compute, because if we had it, we wouldn't have to solve the problem. So that there is a an approximation we need to do, because we don't have access to the optimal policy. And the approximation is this thing right here, which is called the positive value loss. This is from previous work. By the way, this this work is essentially a combination of two previous works. This, this PLR, I don't it's okay, I don't know, exactly right now what it stands for. But what PLR does is it also uses this regret objective, but it simply applies it to randomly generated levels. So it randomly generates, and it just curates that random random those randomly generated levels. And the other thing that it borrows from is evolutionary methods, which maintain the evolutionary methods always maintain a population, and they do this sort of editing the population, and then evaluating their fitness. However, most of the evolutionary methods, they are very hand tailored things of what it means to be fit. So the fitness function could be quite, quite specific to a given environment. And remember, we're not, we're not evolving the, the agents here, which of which fitness would obviously just be like, how well can you solve a level, we're evolving the levels themselves. So the idea of this paper right here is to simply use the regret and as a fitness function, and then curate the levels according to the according to the regret. So it brings in evolution into the PLR algorithm with regret being the fitness that's, I guess, the formulated in two different ways. So the positive value loss, let's unpack that real quick. It stems from this thing right here, a delta K, delta K is the TD error at time step T. So if I'm in a level, and I'm at some time, time, these are the time steps and the observation that I make through the time steps, the TD error is I can compute after I've completed the episode. So at each step, I've gotten some sort of reward, maybe my reward here is R1, my reward here is R2, R3, R4, and so on. So in temporal difference learning, what I do is I always at the beginning of the episode, let's say I'm here, I want to estimate my future reward that I'm going to make, and that would be my value function, right? So my value function tells me what the future reward will hold. Now I can estimate the reward one step into the future or two steps into the future, or three steps and so on. My temporal difference error is simply and if it's written in the same way, I think that's, I'm not entirely sure if that's like a TD lambda or a TD1 error. But in general, what I can do is I can just predict all of my future rewards and the difference between what I predict my future rewards to be and what they actually are, which I know after I've completed the episode, is that I can predict the future rewards. So after I've completed the episode, that's my TD error, that's my temporal difference error. I can use the temporal difference error to learn a value function, because otherwise I'd have to learn the value function just from the rewards that I get. And the TD error is a bit more of a smooth objective. And I believe it converges to the same thing ultimately. But you can reduce the variance a little bit under certain assumptions. The TD error that we're interested in right here, it doesn't matter if the agent uses it to learn or not, but the agent simply predicts the future rewards along the way as it solves the level. After the level is completed, we compare that to the actual rewards that it got, we calculate the difference of that, and that becomes the TD error. Then we sum up the TD error across from each time step. I can calculate a TD error, right? So I can do that from each time step. If I'm at time step t, I can look ahead. Okay, yeah, I can look ahead from each time step until the end. And probably, possibly, the TD error could be looking from either from or to that particular time step. That is not exactly specified. I would have to go and read this paper, possibly, or the PLR paper. It's not super important. We can add that up. Here are some discount factors that we use for that. But you can disregard these for now. Essentially, it simply means okay, from time step t on, you know, how wrong am I about the future? And what we're going to do is we're going to apply a relu to that. So essentially, we're going to cap it at zero, which means that I'm only going to be interested in wherever I under or overestimate. Now let's think about this, wherever I overestimate. So the TD error, as far as I know, is the value minus the reward. Correct me if that's a different way around. But it's what I estimate minus what it truly is. Now, if this is high, it means that I completely overestimated my ability to achieve reward in this level. And that could be, you know, a good level to train on. If I underestimated my ability to achieve reward, then I'm going to guess that that level might be easier than I had anticipated. So but if I overestimated that level might be harder than I anticipated. And that's exactly the levels that I want to train at. So I'm going to cap that at zero, I'm going to sum that up across all the time steps. And if this number is very high, it means that throughout the level, I consistently overestimated my ability to make progress in this level to get reward. And therefore, that level should go into the buffer. So this is the approximation to regret that we're going to use right here. And now you have the entire algorithm. Okay. Generate levels, give them to the student, evaluate them, evaluate this measure, does the student under or overestimate its ability, if it overestimates its ability, put it into the buffer, then take stuff from the buffer, train the student on it, give it to the editor, modify it, and evaluate the student again on it. If the student overestimates its ability on the edited levels, put them back into the buffer and train on them. That's it. You can also see a little bit why this doesn't necessarily suggest levels that are way too hard. Because if you had a level that was way too hard, the student might even correctly estimate that it's not going to make a lot of progress there. Because it's pretty easy to recognize that you're not going to make a lot of progress if the level is super duper hard. So the levels that this is going to select, again, is exactly the levels where the student thinks it should do well, but it doesn't really do well. So let's look a bit into the experiments. The experiments, as I said, are best probably viewed on this website because they're a bit interactive. So what they first do is they come up with these lava grid levels and has the website crashed again. So the lava grid levels are procedurally generated. The agent must get to the goal while avoiding the lava grids. And as the experiments show, these get progressively harder and harder. They next go to these mazes, and Excel starts from just empty rooms. So they start from empty rooms. And up here, I believe, you can see some of the generated levels by this algorithm. And the website has indeed crashed. Let's refresh. So if we look at what levels it generates, you can see that the levels are, they're fairly difficult, right? But they're also kind of random. They don't really look like human levels. So you might be a bit doubtful of whether that's going to help in mazes that we typically know. But you can clearly see the progress from the initially empty rooms to it filling up and to actually becoming harder and harder and harder. And if you then evaluate these things on levels that humans have designed, there's this benchmark right here, it will do pretty well, especially against these other methods that also do curriculum evolution of levels. So especially things here like large corridors. So these are very difficult. The agent only gets a little window around itself to view. It doesn't get an overview over the entire level. So it's very difficult to get an overview over the entire level. And therefore, it needs to sort of keep in mind things that it did previously. And that is a hard task. And they even this is really cool. What they do is they have the agent generalize, I believe from 16 by 16 grids, which they train on to this grid. And you can see that the agent it kind of follows it goes like left, always left. And that works because this maze does has no loops. At least I believe it has no loops. So it in the end, it actually finds the goal. Why this is exactly 51 by 51, I don't know. Maybe because the inside then is 50 by 50, or because that was just the largest maze that it worked on. But it is astounding that it can sort of generalize to much, much larger things. Because in the small mazes, it is conceivable that it could kind of keep all of its all of its history and memory. But here you can really see that it has learned to develop an actual algorithm for what it does. Right. So there is an algorithm like always go left. Yeah, pretty I could, you know, watch forever. Then they go on to these terrains. And again, the thing here is that without hand crafting fitness functions or anything like this, just purely based on these regret measures, this these levels, they continuously evolve, which you can see right here, in what directions the levels evolve. So first, steps are increased, then stair heights, and so on. And at the end, you'll have a generally capable agent. They, they compare this. So they do some ablations. But interestingly, they compare this to poet. And poet is an interesting algorithm because poet trains a population of agents. So poet will always pair environments and agents and try to get the best achieving population of agents, which leads to very specialized agents for very specialized types of environments. So the comparison is not exactly accurate. But they do, they do, they do, they do do, I believe they do show that their algorithm takes a lot less interactions, obviously, because it's only one student, and poet has an entire population of students. And they also analyze over the course of training, how their levels would fall into poets, because poet has a categorization of levels of which ones are easy and hard and so on. And as you can see right here, it starts off with a lot of easy levels on the left and quite a bit of challenging levels, but not very many very challenging or extremely challenging levels. And as time progresses, you can see that at least a little bit the proportion of easy levels, it sort of takes a backseat. And then the proportion of extremely challenging levels increases. What is also interesting, at least for me, is that there's not a monotone, monotonic development into the direction of challenging levels. And that is what, you know, I believe maybe this might be a little bit of a sign of this catastrophic forgetting, because this is only a single agent. Essentially, if you train it into one direction, it might forget the other directions that exist. And specifically, it might forget how to do easy levels, because there's always a hill in the challenging levels, it might fall over once it just encounters a flat plane. I've actually seen this a bunch of times in the trial runs that I did on the website. So it's pretty interesting to see that even though extremely challenging levels get added, and there's certainly more very challenging level than at the beginning, and less easy levels, it does not converge to only having extremely challenging levels. So that is also interesting. Here you can see a little bit of a comparison, notably the top row, a poet is a population based algorithm, as you can see here, which is what makes it different here and not super duper comparable. Then the other ones are so the PLR, as you can see, it also uses the minimax regret strategy to curate levels. However, there is no, it simply relies on random sampling from the generator, whereas Excel uses the random sampling plus evolution, which essentially means that it pairs the PLR algorithm with the poet algorithm. And that appears to work quite well. So that is all that I wanted to say on this work. There's a lot more to say, but I hope that is being clarified in the interview with the authors. What is a bit worrisome to me about this paper is just the fact that while they frame it as, oh, this is very general, this needs essentially no heuristics and so on. I believe that is not entirely the case. I believe there's a lot of domain knowledge that kind of gets sneaked in side. For example, we need this threshold, right? We need the threshold on the regret. So there is a threshold. Only if it hits the threshold, we put it into the buffer. Like they criticize poet for filtering levels where the agent gets between 50 and 300 reward. And they kind of say, well, that's kind of really arbitrary and is really made for that level. And I agree. But then there is kind of a regret threshold, which is, again, that is kind of a hyper parameter that I'm gonna guess that you have to tune. And the same thing goes for, you know, how do I edit these levels and so on? I believe them that it can be an arbitrary editor. But again, this is it's, it's very specific. And I believe what is most specific here is just the choice of the choice of tasks that you go about, not every task. And I would argue that few very few tasks are actually lend themselves to this kind of evolution, because again, you need a very, you need to be able to create a very smooth trajectory from easy to hard, where the same or similar strategies will solve all the different difficulties. And in addition, you need also to to be able for the editor to edit levels in such a way that such a path can be created, right. And you need to avoid the catastrophic forgetting, you can't evolve into too many different things at the same time, and so on. But I do think it's a cool method. And there's certainly certainly applications and curriculum learning, I think is one of the most interesting things that we can currently do. Because gone are the days of, like you essentially shift some responsibility from the agent algorithm to the environment creation algorithm, which I like, right, because we've seen, we've seen scaling up of agents dramatically, drastically. And maybe we can end up with a leaner agent if we shift some of that learning difficulty to the environment. All right, that's what I had to say. Thank you very much for listening. Bye bye.
 Check this out. What you're seeing here is a bunch of agents that have all never seen this level
 before. This level is in fact procedurally generated and the agents must somehow overcome
 the obstacles right here. You can see there's stumps, there's gaps. The green one is performing
 pretty well right here. Coincidentally, the green one is also what we're going to look at in today's
 paper. The idea here is, as I said, these agents have never seen these environments and the
 environments are procedurally generated. Every time I hit reset here, a different environment is
 created. Also notably on the right side right here, I have these sliders with which I can
 control the different properties of the procedurally generated environments, such as how wide the gaps
 are, how many steps to the stairs there are. As I modify these, you can see the environments get
 more and more challenging as I slide these things to the right hand side. Now, they get super
 challenging at some point and the question is, how do we train an agent using reinforcement learning
 in order to be able to solve these challenging environments? Because it's pretty clear that
 if I want an agent to solve an environment like this, and remember it's a procedurally generated
 environment, so I can't just train it on the same environment over and over and over again until it
 gets it. If I want to train an agent to solve the family of environments that are very hard here,
 it's almost impossible to do so using from scratch reinforcement learning because there's just never
 any success of any of the agents. They never finish an episode, they never get good reward,
 they always stumble at the first obstacle. So what's the way we... I still want the green one
 to actually make this. Come on green one, come on! It's not gonna make it right. So the idea is that
 what we want to do is we want to develop a curriculum. So a curriculum means that we're
 going to use this ability to create levels of different difficulties to guide the agent to
 learn more... No... to learn more and more difficult environments. So we're going to start with very
 easy environments, very flat environments, not many gaps in them, not many stairs in them. So fairly
 easy environments like this. And we use reinforcement learning and try to teach the agent just to solve
 this level. Now most of them will do a fairly good job at that level. As you can see, not too much of
 a problem. Some stumble, some don't, but you know this is solvable. And then we will progressively,
 as the agent gets better and better, increase the difficulties of the level. And using that,
 using that difficulty increase over time, there is a chance that the agents, they learn more and more
 to go and solve these levels. So from scratch learning of the difficult environment
 might not be possible. However, there is a chance if we design a curriculum in the correct
 sequence of difficulties for the agents to learn. This is not unlike humans learn in...
 You may have heard of this... What you want to do is train in the zone of proximal development
 or something like this, which essentially means that you want to always challenge yourself
 just outside of your current abilities. And that's how you maximize your progress in learning.
 That's the same idea that we have here with these evolving curricula over time. So the paper we're
 going to look at is called Evolving Curricula with Regret-Based Environment Design by Jack Parker
 Holder and Minki Jiang and others, mainly by Minki Jiang and others, mainly by Minki Jiang.
 And others, mainly by Meta AI, but there's a bunch of collaborations with UC Berkeley,
 University of Oxford, and yeah, I guess that's it. So this paper combines the recent developments
 in regret-based algorithms that go about making a curriculum and evolution, which is another way
 that people go about this. So the paper proposes to train a single agent, not a family of agents,
 a single agent that is generally capable of solving all kinds of difficulties and levels.
 And to do that via an automated curriculum that is given by a teacher algorithm. The teacher
 algorithm itself is not learned. The teacher algorithm is actually defined by this schematic
 right here. And all of this is regret-based, which makes it independent of kind of domain-specific
 heuristics. So the goal of this algorithm right here is to have a general algorithm to design
 these curricula without being reliant on essentially creating a new heuristics for all of the
 different tasks it needs to solve. So we're going to look at it. Here's a brief overview
 over the algorithm itself. How does it do it? How does it get an agent to learn step by step?
 And the most difficult question is, you know, how fast do you increase with the difficulties of your
 levels? Because if you increase not fast enough that you're essentially stuck in learning,
 if you increase the difficulty too fast, you have the same problem again, in that the agent will not
 be capable of keeping up. So what you want to do is you want to have some sort of a level generator.
 And that is what we just saw before in this web demo. By the way, you can go look, try out this
 web demo for yourself at accelagent.github.io. I'll obviously, I'll link it in the description to this
 video. But you want to have some sort of a level generator, which is essentially the thing that I
 have here on the right. I want to have the ability to create different levels. This doesn't need to
 be parameterized like it is here. For example, in this maze world that they portray right here,
 all I have is an empty room, and then I have the ability to place blocks in it. So every pixel can
 either be a wall or not a wall. And that's it. That's a generator. The generator can just
 place blocks and that's it. There's no need for some sort of a slider here that controls the
 difficulty. That's going to be done completely automatically as you'll see. So once we have the
 generator, we could already build some sort of a curriculum algorithm, right? We could just sample
 different levels from the generator and then just train the agent on all of them. However,
 that wouldn't amount to much of a curriculum as it would probably generate easy and hard levels
 all throughout each other. And the agent would be able to solve the easy levels maybe a little bit,
 and then maybe a bit of the harder levels. But if you don't sequence this correctly,
 there's a big chance that you're going to fail, mostly because as the level design space gets
 higher and higher, most levels are either going to fall in the too easy or way too hard section.
 And not a lot are going to be in that zone of proximal development. And therefore you don't
 have much of a learning signal. So we need to somehow filter and curate these levels that we
 generate. So we have a generator and the generator simply gives us the starting bunch of levels.
 And I believe you can also go to the generator within the algorithm and so on. But imagine the
 generator gives us just a bunch of starting levels. This is one of these starting levels.
 I'm going to take a different color right here. Otherwise, you won't see. That's even worse.
 Thank you. So the generator gives us a bunch of starting levels. And these go to the student,
 again, the student here, that's a single agent, that is not a family of agents. The evolutionary
 methods here are not in with regard to the student, but to the levels themselves. So there's one
 student that trains on all the different levels. So what we do is we simply evaluate, we ask,
 we let the student run on this level and we see how well it does. And we're going to measure its
 regret. So the regret of a student, we're going to get to that measure. It's essentially an estimate
 of how far the student is away from the optimal policy on that particular level. And what we want
 to do is we want to strictly select for levels that have high regret. So levels where the student
 is far away from the optimal policy, because those are the levels where the student can still
 learn something. And if we do that correctly, then this automatically sequences these levels
 in the sequence of difficulty such that they're always just at the edge of what the student can
 do. And you'll see how that works in a bit. So we want to measure their regret. And we have this,
 we have the buffer right here. The buffer is where all the levels that we currently think are
 interesting for the student to learn at reside. This buffer is managed by the curator. The curator
 is essentially just a bucket of levels that we think are interesting. What we then do is we can
 replay those levels. So we can actually train the student on the levels. But if we just train the
 students on these levels, that's not much of an interesting thing. So we also need a way to update
 that buffer. And the way we update the buffer is we select some of the levels for editing.
 So some of the levels we think, okay, these are good levels, but could we make them like just a
 bit more difficult because the student can solve them now. So what's a way to make them more
 difficult, then we send them through an editor. And the editor again, this can be pretty much
 anything. So in our example up here, the editor could simply either place another block right here,
 or remove a block. What is important is that it's different from the generator. The generator just
 generates a new thing while the editor modifies the existing things. And the assumption is that
 if I modify something that has a difficulty x, then if I modify it to x hat, then the difficulty
 of x hat will not be too much different. So what I'm going to do is I'm at, let's say here is the
 student's starting point, and the student increases its ability round by round. So maybe this is the
 zone that the student can solve right now. And I select a level that is here, so the student can
 just about solve it. And then I modify that with the editor a little bit. And I maybe produce a
 produce different offspring, like here, here, here, and here. So what I want to do is I want to select
 for the offspring. And here's where that's where the evolutionary method comes in. I want to select
 for the offspring that will make progress for the student so that the student just can't solve right
 now. And add that to the buffer of things where I do reinforcement learning on. So with the editor,
 I create a bunch of different offspring for this level, as we see right here. And I evaluate the
 student on them, I measure the students regret. And if the regret is high, I put that back into
 the buffer. So in this way, I always keep the buffer filled with levels that the student just can't
 like it's just at the zone of where the student can solve them. So if I now add the blue circled
 levels, obviously the next you know, I'm going to increase my ability to out here a little bit in
 this direction, right. And then maybe here is another level that I modify with these two,
 and that increases the student's ability to hear. And then from these levels, I will again create
 offspring maybe to hear and hear again, I will filter out the ones that become easier. And so,
 as you can see the students abilities, they will continually increase guided by this metric of this
 regret. So that's the entire algorithm. Essentially, you'll have one student that is generally capable.
 And the buffer right here will always contain levels that the student just can't, or just about
 can solve by measure of these regret and continuously editing. Obviously, this doesn't work everywhere.
 Like there needs, there's a lot of preconditions for this to work. For example, you need to be able
 to have this level generator and level editor. You need to be able to create levels of various
 difficulties, not out of the box, but it like should be possible in principle. There should be
 the possibility of creating a curriculum in the first place, which is not possible for all the
 tasks, especially with the condition that if I modify the problem a little bit, like this thing
 right here, if I modify the problem a little bit, then the difficulty should only be modified by a
 little bit. Like that is not a given for many, many tasks. However, if this is all given, then
 it becomes suddenly possible. And of course, we run into all the problems of having a single student,
 like there's catastrophic forgetting and so on. But we don't we don't worry about this right here.
 As you might have seen previously, that the Excel agent right here, this the green agent,
 no matter kind of what the terrain is, its strategy is always sort of the same. So its
 strategy is always to kind of hold one leg out and bounce on the hind leg. And okay, that that
 might not have been so it will always it's not going to make that it was bounce on the hind leg,
 actually, most of them will do it bounce on the hind leg and kind of wiggle the front leg. And
 that's how it bridges gaps and stairs and ladders and so on. Okay, most of them do that. But you'll
 see that this is a problem of I think having a single single agent solve these things. If you
 want a single agent to be solved to solve all the environments, that means that implicitly kind of
 one one strategy or one set of strategies must be enough to solve all the environments, which is also
 not a given for much of the world of reinforcement learning. However, this can all be fixed.
 So this was the overview. Now let's dive into a little bit more into the algorithm itself. Again,
 we have not yet we there's still a crucial element. And that is this regret that we haven't talked
 about yet. But the algorithm in code looks like this. I want to I want to initialize a policy that
 this is the student policy pi and this level buffer. So the buffer is is lambda, I guess lambda.
 Okay, so I'm gonna sample some initial levels. And I'll just assume that the initial levels here,
 they're going to be mixed in difficulty. So they're going to be some some easy levels,
 and some hard levels, and some levels that the student might just be able to solve out of the
 box or not. So what then we're going into a while loop, the big like while not converged,
 we're going to sample a replay decision. And the replay decision is essentially it's a binary
 variable that tells me, do I want to take a level from the buffer? Or do I want to take a level from
 the new level from the generator? Because if you only have initial levels in your buffer, right,
 then you're kind of limited by the evolution of these levels. So much unlike we have non convex
 optimization problems in deep learning, these these landscapes of levels might be super duper
 non convex. And that's why if you just evolve a bunch of levels, there is obviously the danger
 that you get that you sort of narrow like you, you never. So if you if you go down a bunch,
 if you teach the agent to go like down a bunch of stairs, and you go ever more and more stairs,
 more and more stairs, but the initial levels never had like a big cliff like this, your agent
 will not be able to solve it even with this method, because no amount of adding stair steps will get
 you to the big cliff. And that's why it's important to every now and then actually sample a level from
 the level generator to bring some diversity in there. Because that's what I see with this method
 is probably pretty easy to teach yourself into a corner. So if we have something from the level
 generator, we collect the trajectory. And it's important that we have two different modes right
 here, we have the student in evaluation mode. So every time that we have some level, some new level,
 we first evaluate the student on it, we want to know whether the student can actually solve it or
 not on how well it can solve it. So what do we do? We compute the approximate regret, we don't
 actually train on this level, we just evaluate it. And that is a property, I think that reduces the
 signal to noise ratio tremendously, we want to pre filter what levels we train on, we don't just
 want to train on all of them. So this is a this is interestingly enough, a method where even though
 we have the training data available, it seems to be better if we filter the training data, it's still
 good training data, right? Any of these levels is good training data for reinforcement learning. It's
 not like there's noisy data or the label is wrong or something. But it seems to be quite important to
 accurately select the levels we want to train on. So that is that is an interesting thing by itself.
 But you what you'll see in this algorithm is that they always will first evaluate a level,
 determine whether the regret is high or whether it is in the zone of proximal development and only
 then use this that level to actually train the agent on. That is interesting. So we compute this
 regret, and we add the level to the buffer. So the level here is this theta. So these are the
 parameters again here that we evolve, we evolve two sets of parameters, the parameters of pi,
 which is the student's policy. But that is just a very simple proximal policy optimization,
 reinforcement learning algorithm right here, we don't actually care what kind of RL algorithm it
 is as long as it can learn. The interesting parameters here are the parameters of the levels.
 And this could be the level itself in case of this maze, or it could be the parameters.
 No, actually, it would be the level the level itself. Right, it needs to be an actual
 instantiation of the level, not just the parameters that you enter into the generator, unless the
 generator is deterministic. And we only add it to the buffer if the score meets a threshold. So
 that is where we filter out things where the regret is either where the regret is too low.
 So only if it is a hard level for the student to solve, we put it into the buffer and we'll
 get to how we actually filter out the levels that are too hard in a second. So that's just if we
 decide we need a new level. If we decide actually that we want to go into the buffer, we're going to
 sample a level that we've previously added into the buffer. And remember, we've determined that
 all of these are in the zone of proximal development. We train, we collect the policy, and we actually
 train. So this is where we train. We train on a level that we sampled from the buffer in the
 first place. It's the only time we train the agent at all. And then we are not done with this level
 yet. What we do is we take the same level that we just sampled and we actually edit it. So here,
 edit to produce theta prime. And the editing can be, as I said, anything as long as you can
 reasonably assume that any edit will not distort the difficulty too much. So it needs to distort
 the difficulty somewhat, but not too much. Again, we collect the trajectory, we do not train it,
 we simply run the student on the new levels, exact same way we did before, we compute the regret,
 and we add it to the buffer if the score meets a threshold. Optionally update the editor using
 the score. So that can be the editor itself could be some sort of dynamic algorithm or not.
 So that is the algorithm in a nutshell. It's pretty simple. There is a buffer. I train on levels
 inside the buffer and only on levels that are inside the buffer. How do levels get into the
 buffer? Two ways. They can be sampled either from the level generator, or they can be edited
 from levels that are already in the buffer. However, both of them will only get into the buffer
 if we evaluate the agent on them first, compute its regret, and if the regret is higher than some
 threshold. That's how we curate the buffer. And that's it. That's the entire algorithm.
 So they have a bunch of experiments right here. And that's, it's probably better to go back to the
 website to look at the experiments. So, oh no, we need to look at what the regret is, obviously.
 So regret is just the way it's formulated right here. The regret is the difference between the
 expected rewards of two policy. So if I have a, this here is regret. So the regret of theta,
 and now you know theta is a level, right? So the regret specific to a level would be, and here is
 policy one and policy two. Now in this case, it's the current policy and the optimal policy.
 But you can see down here, the regret can be defined over any two arbitrary policies. It is
 simply the difference in the values of the two policies. And what's the value? The value is the
 expected future reward. And if I pose it like this, it's probably just the expected reward.
 So the formulation right here, where I plug in the optimal policy would simply be, you know, what,
 I have some sort of level, right? And I have my current agent right here. And the agent expects
 to get some sort of reward, like maybe it gets onto here and then it crashes. So that's a reward
 of, I don't know, 50. And the optimal policy, if the level is solvable at all, it could actually
 go to the end and solve it and get a reward of 100. So my regret in this case would be 50.
 And that is a good measure of how difficult a level is, or let's say how much you can still
 learn from that level. Because if a level is too difficult, and that's the catch, if a level is
 too difficult, then not even the optimal policy will be able to achieve much in that level. And
 therefore, you know, why are you like, what point is it to go to that level and actually solve it?
 Or if there is any stochasticity, if a level needs a lot of luck, right, then as well, the expected
 the expected reward, the expected future reward of the optimal policy will also be not super
 high. So by selecting things that have high regret, meaning that have a high difference
 between the optimal policy and the current policy, we select for levels that where the
 current student can still learn a lot of things. So it's still there's still headroom to learn.
 Now, the optimal policy is obviously hard to compute, because if we had it, we wouldn't have
 to solve the problem. So that there is a an approximation we need to do, because we don't
 have access to the optimal policy. And the approximation is this thing right here, which
 is called the positive value loss. This is from previous work. By the way, this this work is
 essentially a combination of two previous works. This, this PLR, I don't it's okay, I don't know,
 exactly right now what it stands for. But what PLR does is it also uses this regret objective,
 but it simply applies it to randomly generated levels. So it randomly generates, and it just
 curates that random random those randomly generated levels. And the other thing that it
 borrows from is evolutionary methods, which maintain the evolutionary methods always maintain
 a population, and they do this sort of editing the population, and then evaluating their fitness.
 However, most of the evolutionary methods, they are very hand tailored things of what it means
 to be fit. So the fitness function could be quite, quite specific to a given environment. And
 remember, we're not, we're not evolving the, the agents here, which of which fitness would
 obviously just be like, how well can you solve a level, we're evolving the levels themselves.
 So the idea of this paper right here is to simply use the regret and as a fitness function,
 and then curate the levels according to the according to the regret. So it brings in evolution
 into the PLR algorithm with regret being the fitness that's, I guess, the
 formulated in two different ways. So the positive value loss, let's unpack that real quick.
 It stems from this thing right here, a delta K, delta K is the TD error at time step T.
 So if I'm in a level, and I'm at some time, time, these are the time steps and the observation
 that I make through the time steps, the TD error is I can compute after I've completed the episode.
 So at each step, I've gotten some sort of reward, maybe my reward here is R1, my reward here is R2,
 R3, R4, and so on. So in temporal difference learning, what I do is I always at the beginning
 of the episode, let's say I'm here, I want to estimate my future reward that I'm going to make,
 and that would be my value function, right? So my value function tells me what the future reward
 will hold. Now I can estimate the reward one step into the future or two steps into the future,
 or three steps and so on. My temporal difference error is simply and if it's written in the
 same way, I think that's, I'm not entirely sure if that's like a TD lambda or a TD1 error.
 But in general, what I can do is I can just predict all of my future rewards and
 the difference between what I predict my future rewards to be and what they actually are,
 which I know after I've completed the episode, is that I can predict the future rewards.
 So after I've completed the episode, that's my TD error, that's my temporal difference error.
 I can use the temporal difference error to learn a value function, because otherwise I'd have to
 learn the value function just from the rewards that I get. And the TD error is a bit more of a
 smooth objective. And I believe it converges to the same thing ultimately. But you can reduce
 the variance a little bit under certain assumptions. The TD error that we're interested
 in right here, it doesn't matter if the agent uses it to learn or not, but the agent simply
 predicts the future rewards along the way as it solves the level. After the level is completed,
 we compare that to the actual rewards that it got, we calculate the difference of that,
 and that becomes the TD error. Then we sum up the TD error across from each time step. I can calculate
 a TD error, right? So I can do that from each time step. If I'm at time step t, I can look ahead.
 Okay, yeah, I can look ahead from each time step until the end. And probably, possibly, the TD
 error could be looking from either from or to that particular time step. That is not exactly
 specified. I would have to go and read this paper, possibly, or the PLR paper. It's not super
 important. We can add that up. Here are some discount factors that we use for that. But you
 can disregard these for now. Essentially, it simply means okay, from time step t on, you know,
 how wrong am I about the future? And what we're going to do is we're going to apply a relu to
 that. So essentially, we're going to cap it at zero, which means that I'm only going to be
 interested in wherever I under or overestimate. Now let's think about this, wherever I overestimate.
 So the TD error, as far as I know, is the value minus the reward. Correct me if that's a different
 way around. But it's what I estimate minus what it truly is. Now, if this is high, it means that I
 completely overestimated my ability to achieve reward in this level. And that could be, you know,
 a good level to train on. If I underestimated my ability to achieve reward, then I'm going to guess
 that that level might be easier than I had anticipated. So but if I overestimated that
 level might be harder than I anticipated. And that's exactly the levels that I want to train at.
 So I'm going to cap that at zero, I'm going to sum that up across all the time steps. And if this
 number is very high, it means that throughout the level, I consistently overestimated my ability
 to make progress in this level to get reward. And therefore, that level should go into the buffer.
 So this is the approximation to regret that we're going to use right here. And now you have the
 entire algorithm. Okay. Generate levels, give them to the student, evaluate them, evaluate this
 measure, does the student under or overestimate its ability, if it overestimates its ability,
 put it into the buffer, then take stuff from the buffer, train the student on it, give it to the
 editor, modify it, and evaluate the student again on it. If the student overestimates its ability on
 the edited levels, put them back into the buffer and train on them. That's it. You can also see a
 little bit why this doesn't necessarily suggest levels that are way too hard. Because if you had
 a level that was way too hard, the student might even correctly estimate that it's not going to
 make a lot of progress there. Because it's pretty easy to recognize that you're not going to make a
 lot of progress if the level is super duper hard. So the levels that this is going to select, again,
 is exactly the levels where the student thinks it should do well, but it doesn't really do well.
 So let's look a bit into the experiments. The experiments, as I said, are best probably
 viewed on this website because they're a bit interactive. So what they first do is they come
 up with these lava grid levels and has the website crashed again. So the lava grid levels
 are procedurally generated. The agent must get to the goal while avoiding the lava grids. And as
 the experiments show, these get progressively harder and harder. They next go to these mazes,
 and Excel starts from just empty rooms. So they start from empty rooms. And up here, I believe,
 you can see some of the generated levels by this algorithm. And the website has indeed crashed.
 Let's refresh. So if we look at what levels it generates, you can see that the levels are,
 they're fairly difficult, right? But they're also kind of random. They don't really look like human
 levels. So you might be a bit doubtful of whether that's going to help in mazes that we typically
 know. But you can clearly see the progress from the initially empty rooms to it filling up and to
 actually becoming harder and harder and harder. And if you then evaluate these things on levels
 that humans have designed, there's this benchmark right here, it will do pretty well, especially
 against these other methods that also do curriculum evolution of levels. So especially
 things here like large corridors. So these are very difficult. The agent only gets a little
 window around itself to view. It doesn't get an overview over the entire level. So it's very
 difficult to get an overview over the entire level. And therefore, it needs to sort of keep in mind
 things that it did previously. And that is a hard task. And they even this is really cool. What they
 do is they have the agent generalize, I believe from 16 by 16 grids, which they train on to this
 grid. And you can see that the agent it kind of follows it goes like left, always left. And that
 works because this maze does has no loops. At least I believe it has no loops. So it in the end,
 it actually finds the goal. Why this is exactly 51 by 51, I don't know. Maybe because the inside
 then is 50 by 50, or because that was just the largest maze that it worked on. But it is
 astounding that it can sort of generalize to much, much larger things. Because in the small mazes,
 it is conceivable that it could kind of keep all of its all of its history and memory. But here you
 can really see that it has learned to develop an actual algorithm for what it does. Right. So there
 is an algorithm like always go left. Yeah, pretty I could, you know, watch forever. Then they go on
 to these terrains. And again, the thing here is that without hand crafting fitness functions or
 anything like this, just purely based on these regret measures, this these levels, they continuously
 evolve, which you can see right here, in what directions the levels evolve. So first, steps
 are increased, then stair heights, and so on. And at the end, you'll have a generally capable agent.
 They, they compare this. So they do some ablations. But interestingly,
 they compare this to poet. And poet is an interesting algorithm because poet trains a
 population of agents. So poet will always pair environments and agents and try to get the best
 achieving population of agents, which leads to very specialized agents for very specialized types
 of environments. So the comparison is not exactly accurate. But they do, they do, they do, they do
 do, I believe they do show that their algorithm takes a lot less interactions, obviously, because
 it's only one student, and poet has an entire population of students. And they also analyze
 over the course of training, how their levels would fall into poets, because poet has a
 categorization of levels of which ones are easy and hard and so on. And as you can see right here,
 it starts off with a lot of easy levels on the left and quite a bit of challenging levels,
 but not very many very challenging or extremely challenging levels. And as time progresses,
 you can see that at least a little bit the proportion of easy levels, it sort of takes
 a backseat. And then the proportion of extremely challenging levels increases. What is also
 interesting, at least for me, is that there's not a monotone, monotonic development into the direction
 of challenging levels. And that is what, you know, I believe maybe this might be a little bit of a
 sign of this catastrophic forgetting, because this is only a single agent. Essentially, if you train
 it into one direction, it might forget the other directions that exist. And specifically, it might
 forget how to do easy levels, because there's always a hill in the challenging levels, it might
 fall over once it just encounters a flat plane. I've actually seen this a bunch of times in the
 trial runs that I did on the website. So it's pretty interesting to see that even though extremely
 challenging levels get added, and there's certainly more very challenging level than at the beginning,
 and less easy levels, it does not converge to only having extremely challenging levels.
 So that is also interesting. Here you can see a little bit of a comparison,
 notably the top row, a poet is a population based algorithm, as you can see here, which is what makes
 it different here and not super duper comparable. Then the other ones are so the PLR, as you can see,
 it also uses the minimax regret strategy to curate levels. However, there is no, it simply relies on
 random sampling from the generator, whereas Excel uses the random sampling plus evolution,
 which essentially means that it pairs the PLR algorithm with the poet algorithm.
 And that appears to work quite well. So that is all that I wanted to say on this work. There's
 a lot more to say, but I hope that is being clarified in the interview with the authors.
 What is a bit worrisome to me about this paper is just the fact that while they frame it as,
 oh, this is very general, this needs essentially no heuristics and so on. I believe that is not
 entirely the case. I believe there's a lot of domain knowledge that kind of gets sneaked in
 side. For example, we need this threshold, right? We need the threshold on the regret.
 So there is a threshold. Only if it hits the threshold, we put it into the buffer.
 Like they criticize poet for filtering levels where the agent gets between 50 and 300 reward.
 And they kind of say, well, that's kind of really arbitrary and is really made for that level.
 And I agree. But then there is kind of a regret threshold, which is, again, that is kind of a
 hyper parameter that I'm gonna guess that you have to tune. And the same thing goes for, you know,
 how do I edit these levels and so on? I believe them that it can be an arbitrary editor. But
 again, this is it's, it's very specific. And I believe what is most specific here is just the
 choice of the choice of tasks that you go about, not every task. And I would argue that few very
 few tasks are actually lend themselves to this kind of evolution, because again, you need a very,
 you need to be able to create a very smooth trajectory from easy to hard, where the same or
 similar strategies will solve all the different difficulties. And in addition, you need also to
 to be able for the editor to edit levels in such a way that such a path can be created, right.
 And you need to avoid the catastrophic forgetting, you can't evolve into too many
 different things at the same time, and so on. But I do think it's a cool method. And there's
 certainly certainly applications and curriculum learning, I think is one of the most interesting
 things that we can currently do. Because gone are the days of, like you essentially shift
 some responsibility from the agent algorithm to the environment creation algorithm, which I like,
 right, because we've seen, we've seen scaling up of agents dramatically, drastically. And maybe
 we can end up with a leaner agent if we shift some of that learning difficulty to the environment.
 All right, that's what I had to say. Thank you very much for listening. Bye bye.