Author Interview - ACCEL: Evolving Curricula with Regret-Based Environment Design
#ai #accel #evolution

This is an interview with the authors Jack Parker-Holder and Minqi Jiang.
Original Paper Review Video: https://www.youtube.com/watch?v=povBDxUn1VQ

Automatic curriculum generation is one of the most promising avenues for Reinforcement Learning today. Multiple approaches have been proposed, each with their own set of advantages and drawbacks. This paper presents ACCEL, which takes the next step into the direction of constructing curricula for multi-capable agents. ACCEL combines the adversarial adaptiveness of regret-based sampling methods with the capabilities of level-editing, usually found in Evolutionary Methods.

OUTLINE:
0:00 - Intro
1:00 - Start of interview
4:45 - How did you get into this field?
8:10 - What is minimax regret?
11:45 - What levels does the regret objective select?
14:20 - Positive value loss (correcting my mistakes)
21:05 - Why is the teacher not learned?
24:45 - How much domain-specific knowledge is needed?
29:30 - What problems is this applicable to?
33:15 - Single agent vs population of agents
37:25 - Measuring and balancing level difficulty
40:35 - How does generalization emerge?
42:50 - Diving deeper into the experimental results
47:00 - What are the unsolved challenges in the field?
50:00 - Where do we go from here?

Website: https://accelagent.github.io
Paper: https://arxiv.org/abs/2203.01302
ICLR Workshop: https://sites.google.com/view/aloe2022
Book on topic: https://www.oreilly.com/radar/open-endedness-the-last-grand-challenge-youve-never-heard-of/

Abstract:
It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at this http URL.

Authors: Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi, this is an interview with the authors of the paper evolving curricula with regret based environment design. If you haven't seen it, I've made a review of this paper yesterday, the day before this video is released. And I went over the paper in detail and explained what's inside of it. So if you haven't seen that, it would be a good place to start today I'm interviewing the authors of this paper, Jack and Minchi, who are real experts in this domain. Now during the interview, we go a lot deeper than I could do myself in the paper review. And you learn a lot more about how things work in this paper, but also in the entire field. It's a very exciting field. And it's a real privilege to be able to interview all of these people. I hope you're having fun. Please let me know in the comments how I can make these videos better for you. And thank you to everyone who does watch who does comment who does share. Thank you to all the supporters on Patreon to all the discord members and to everyone else who is excited by machine learning. I hope you're doing well. Stay hydrated. Now let's get into the interview. Parker Holder and Minchi Chang. Did I get this right? Yeah. Thank you. Welcome very much to the show. Thanks for having us. I think your paper here, it was of one one sort of an example of a very cool paper, because it's not on a state's a bit out of the mainstream, usually reinforcement learning tackles improving the agent as much as possible, where you you go much into this road of poet and work before it improving the environment. But also I think it's a good lesson in how to kind of put a bit of publicity behind the paper because you made this this very cool website right here when this with the interactive demo where I can play around with the terrain, right? Okay, if it only works. And you have these these kind of nice animations of how things develop during training and so on. And I think, like, how much do you think something like this helps a paper after it's released? Like, what was your impression of just just kind of, or maybe you can tell me a little bit. How did you how did you even decide paper aside to make a website like this and present it in a form that's interactive? I think with RL research, especially when you look at curriculum design, you're modifying the environments, there's always really interesting visualizations that you can share. But I think having just like the standard PDF format that everyone publishes on archive, then is really, really limiting. And there's just so much, there's so much amazing like assets you can actually share in terms of your agent behavior, in terms of the emergent complexity that these algorithms generate. So we really wanted to share that with readers. And we thought that would definitely capture more of people's imaginations when they engage with our work. And there's like also just a huge sort of lineage of work that tries to do a similar thing, like our template for this website is actually taken from distil. So distil pub has so many great works, and they put so much effort into making such beautiful interactive publications. And we definitely took a lot of inspiration from that. David Ha, Google Brain has a bunch of publications like with world models and tension agent that did similar things. Yeah. And then also we use the teach my agent work from the flowers lab as well, which had some of the like building blocks for this. And that was really cool. But I think the other thing is like, there's always this question with these type of methods, if you picked the test environments by your method works, and as reviewers ourselves, we're always very cynical of this. And so we kind of thought, what if we just let people try and break it into what happens. And of course, you can break it pretty easily. And that actually leads to kind of exciting questions of how you can make it better in future work. But at the same time, it's kind of nice to see how it does and doesn't work. Because then the day I think we should be more honest about the robustness of our agents. And this is quite a nice tool to not only make it fun, but also kind of demonstrate it. I think more also for not just for readers, but I think just for ourselves as researchers, like in the process of making this tool, and starting to actually run the agent and tons of visualized environments, we actually started to discover certain shortcomings of the agent. Like you can look at all these plots all day long, and you see all the metrics go up into the right. But then you don't actually see sort of the blind spots that come up during training until you actually visualize it. And we discovered a few interesting motifs that that consistently challenged the agent, even though it's overall quite robust. Yeah, because we're actually going to talk we're talking about maybe like making it so that it defaulted to levels that we know it can do well on but then we just thought I kind of removed the fun. And at the end of the day, if it breaks and someone's inspired to improve it, that's ultimately a good thing. Yeah, I mean, you do have the metrics to prove that it does something well, right? And anything after that is a bonus, essentially. How did you get even into this? How did you get even into this field? Do you maybe want to like give a 30 second bio of yourself? Like how did you arrive at this point? Sure. So I mean, from my perspective, I came out before my PhD, and I thought it was really inspirational, really cool work. But I didn't really know if I'd ever get to work on something like that. And then obviously, interning last summer at a matter with Tim and Ed and Munchi, who are on paper and Mika as well. The group was working on generalization and starting to improve on idea and build on ideas such as like paired and these algorithms. And so then, so when I came in, we were talking a little bit about like shortcomings of our methods. And then Poet obviously comes up as another example. And we were kind of thinking, how do we take some of the ideas from Poet and really incorporate into our existing, like regret based curriculum methods. And so then it became kind of obvious that we want to try this environment and this type of work. I guess it was kind of a fusion of different things. So it was like top down initially, and then also ended up being bottom up. Yeah. And I guess curriculum learning was something I kind of stumbled on in the first year of my PhD. And basically, I was originally trying a bunch of sort of random ideas of, I always had this notion that maybe RL could be made more efficient if you train agents on levels that were just within reach. And then you basically progressively increased the level complexity in terms of a curriculum. And so we worked on a prior method as well called Prior Test Level Replay, which is this pink PLR baseline here. And that one ended up doing quite well, especially when combined with data augmentation on the OpenAI ProcGem benchmark. And so right after that, I got in touch with another researcher at UC Berkeley, a fellow named Michael Dennis, and he was one of the first authors on the Emerging Complexity for Zero Shot Robustness paper that introduced the paired algorithm. And so this is the paper that kind of introduced a lot of the formal theory, decision theory around minimax regret policies in their application within Deep RL. And it kind of was the first paper that showed that if you optimize for minimax regret in using Deep RL, it makes sense and you get nice experimental results that show robustness in zero shot transfer. And so we started discussing and we realized that actually a lot of the theory could be applied to PLR. And that PLR was actually another instantiation of this minimax regret game, which is at the heart of this theory. And Excel is sort of like the latest version. It's sort of the culmination of the ideas we've explored so far in this direction. Yeah, I guess it's worth noting that we published the robust PLR paper in Europe last year. So that was really that what was finishing just around June, July time when I joined that meta. And so really we were looking, we kind of knew that method was very empirically strong and theoretically nice, but it still maybe lacked something in that it couldn't really have some creative process to design its own levels because it could only sample, I think, as you as you pointed out in your review. So ultimately, if the space is very high dimensional, and you only sample one high regret level, once you've mastered it, you have to then go back to the drawing board. Whereas the nice thing about Excel is that it's by a poet, it can really kind of build its own complexity over time. And so it really is kind of like a progression through to really sequence of papers, I guess. And, and to be fair, Michael's been on now three of them in a row because he was on paired and then robust PLR and Excel. Can you give like a layman's layman's explanation for optimizing for mini max regret? Because there are a bunch of like, it's regret, and then max and then min. What's what what does it ultimately boil down to? So, so, so, this largely comes from this emerging complexity paper from Michael Dennis and Natasha Jax. Essentially, the theory there is essentially framing, framing a concept called unsupervised environment design, as essentially this problem where you want to design environments that maximize for some metric, and that metric is usually some behavioral metric that's associated with the student agent. And so in this game, in this mini max regret game, we care about maximizing the regret of the agent. And so if you frame the game as a game where it's a two player game, it's zero sum, the payoff for the student is the negative regret, and the payoff for the teacher is the positive regret. Essentially, you have a game where the teacher tries to increase the regret of the student and students trying to minimize its regret. So if you think about two players, you're some games, they always have a Nash equilibrium. And at the Nash equilibrium of this game, it's got to be the policy that the student plays that essentially is a mini max regret policy, it's minimizing its worst case regret. Because if it's not doing this, the teacher must be able to change its policy and play more of a certain level that further increases the regret. And so by definition at a Nash equilibrium, neither player has an improving response. So it must be that the student has a mini max regret policy. So what does that mean in layman's terms? It basically means that the student behaves in a way that essentially it's able to do well in any level that's solvable inside of the parameterized space of tasks that the teacher can use to propose the next level. So the yes, it should always be sorry, the teacher would have the teacher's moves was essentially be the levels like the actions of the teacher would be I play this level. Yeah. So it's within the subtraction, called a U-POM DP, which is just like a partially observable Markov decision process. But you add an additional set of variables called the free parameters. In the papers, we usually use the term theta to denote them. And so then those are like the positions of where the obstacles are in the maze, in the maze domain, might be like starting position of the agent goal position. Inside of the car racing environment, it might be like the position of where the tracks are. And so these are the design parameters. And so a strategy of the teacher is essentially like choose some distribution over choices of the possible free parameters that it can sample as the next level. Sorry, Jack, you go. All right. I was gonna say like the nice intuitive property of this is that it makes the agent has to learn to solve all of the simplest solvable environments as well. So in some other methods like poet, they're trying to achieve the maximum complexity, which is like, it's very cool as well motivated. But this is quite different in that we're actually happy if even later in training our agents training on simple levels, if it means that it can solve all of the simple levels, because we don't really care as much about solving like crazy complex things if it can break some simple thing, which I think is seems to make sense, at least to me. Yeah, that was one of my let's say worries right here is that if you if you and I framed this a little bit as you are at this zone of proximal development with your agent in that somehow made it wrong, like you try to reach levels that are just outside of where the agent can handle it. And then you you try to edit those a little bit or maybe just where the agent can handle them. And then you try to edit them a little bit. And you try to filter by the ones that pass some threshold in this estimated regret. So my first question would be coming back to this regret, you you formulated as the so it's it's formulated as the difference to the optimal policy, right? The difference to to the optimal policy, I'm going to guess on this particular level that you're at. Why doesn't this like disregard the approximation that you do? If I could calculate this very accurately, wouldn't this select for super duper difficult levels that could be solved with the optimal policy, right? Not impossible, but just super difficult ones? That's a great question. And I think part of the part of the nuanced detail here is that so one reason that makes this all work is the discount factor. So basically, the so in the original paper that introduced paired and this idea of the mini master game, the reward function for that environment actually, it actually your reward, your final return decreases with the length of your trajectory. And so there's a natural discounting in terms of the return. And so essentially, by doing mini max regret, it ends up prioritizing for those levels where the solutions within reach in the fewest number of steps. And you get this nice curriculum. But because here in all of our approximate single agent regret estimators, we're using a value function, which is bootstrapped off of a generalized advantage estimator, which itself is discounted, you essentially have discounting built into your value function. And so you end up with discounting even if they're even if your environments are final, you know, sparse reward, no discounting naturally in the external reward, you still get discounting because your value function is going to be discounting using gamma. And if you use GAE, you have further discounting with lambda. Cool. Yeah, that was one of my one of the things that I didn't exactly understand here in this. Okay, I was like, disregard the discount factors. They're not important. Turns out they're actually one of the most important parts right here to actually make it work. Although you use this this positive value loss. Now, I think you wrote me in an email before that I got this wrong in the in the paper review. Do you want to maybe quickly discuss what the individual parts of this formula mean and what they do? I mean, so essentially, the I guess we can start from sort of the outside in, I guess, or maybe it makes sense to do the inside out. So basically, the innermost term is essentially just a TD error. It's a one step TD error. And it's future facing. So it's from your current time step t until the horizon t, capital T. And essentially, the inner term except for within the max, that term is basically, if you look at the sum from t to capital T, that's basically the generalized advantage estimator from Schumann et al. And so that one is the most common, that's the advantage estimator used in PPO. It's used in other policy gradient algorithms as well. But essentially, that is essentially estimating your advantage while trying to do a trade off between one step TD errors being more biased because it's bootstrapping off of fewer steps, and longer TD errors being less biased, but having more variance. And so lambda is a discount factor that controls for that. And so in a nutshell, though, this is estimating advantage, which is basically, this is my actual return minus my typical return, which you can think of as what the value function outputs. And so the zero... So this is, sorry, this is return minus value. Yeah, you can think of it as the return you achieved minus your value prediction at each step in your trajectory. And we average it over the trajectory. And essentially, that's telling us, if that's really high, it means that I'm doing better than what I typically do. And so directionally, this is like in the direction of regret, because it means that in terms of external regret, I can actually get a higher return than I typically do, which means that this is a level where I experience regret. And then we max this with zero, which just means that we are only looking at the positive time steps where, at the time steps at which this term is positive. So we're only looking when the agent does better than it typically does. And if on average, when it does better than it typically does is quite high, it means that it's a level where the agent can experience a lot of regret in its decision making. How so though? Like my logic was a little bit, if I'm worse than I estimated, that means kind of it's a difficult level. Like where's my thinking wrong here? So if you do worse than you estimated, I think in terms of just the mini match regret framework, it's just a little bit sideways in terms of measuring the direction of regret. I think if you think of it as looking for cases where you do better than you typically do, that's really just you discovering regret. It's like you discovered a case where you achieve regret relative to your typical self, as sort of amortized by this value function that predicts how well you typically do over time. So with respect to sort of this average prediction of yourself, you're doing better. And so you're essentially discovering sources of new regret in this level. And that's basically directionally aligned with maximizing regret. While if you were to do the opposite, if you were to say, I want to look for the steps where I do worse than I think I do, I think that's an interesting thing to try actually. But at least theoretically, it doesn't seem to align with mini match regret as well. Yeah, okay. I can see the logic in that you say, I want to find levels where there's something unexpected positive thing happening. Yeah, it's worth noting as well that impaired, which was the first UD algorithm to use regret, they had a very different approaches, which had a second agent called an antagonist. And the regret was just the difference in performance between those two. And so maybe that's like, a bit more intuitive, because if the antagonist can solve a level and the protagonist, the student agent, can't, then I guess that's more intuitive in terms of what you expect from regret. But the nice thing about this is it's kind of a cheap approximate for single agent regret. And we definitely feel like maybe coming up with better metrics for single agent regret is exciting future work that could be improved upon here. But this was taken just from the robust PLR paper, and we were surprised how well it worked in quite different environments. And another detail is in the robust PLR work, another regress meter we use is the another regress meter we used that we explored was what we call maximum Monte Carlo regret estimator. And essentially, it's the same, it's almost the same expression, except the regret target is no longer what you just received inside of a recent episodic rollout. It's for every level, we keep track of the highest return you ever achieved throughout training on that level. And we use that as an estimate for the maximum performance on that level. And then we use that as the target to subtract your value prediction on. And so that's like a more off policy regret, which I think, in some cases might be better because it's less coupled to your current policy. While the positive value loss, it's always what you recently received in a rollout in terms of your target, minus your value function prediction. Yeah. Is that worth because you would introduce some extra variance, because you're not essentially subtracting your own bait, like use this as a baseline in the advantage estimate? Or am I seeing this wrong? So this would introduce extra variance. It's not using the policy update, it's used just to score the levels. So essentially, you're saying the best you've ever done, which might be more, it's going to upper bound your current performance, right? The best you've ever done, including your current performance, versus your value function. So it's slightly nicer in a sense that if you've experienced a level many times, maybe you've had some forgetting, then the regret should be higher because you've done well in the past. But the negative is you have to then store all of your previous episodes for every level. And then oftentimes you don't actually have any previous experience. So it's not even that applicable, but there's a trade-off. And I think, again, I think this is something that could be improved in future work. Especially with procedurally generated content, it's probably hard. You'd have to build some sort of a, even a model to estimate the best possible regret given past procedurally generated levels to sort of predict for any new one. And those two models will probably make similar sorts of mistakes, like the mistakes might even be correlated between the... Okay. So with respect to your method here, which is decently simple, what I was surprised by is that you deliberately go away from the teacher being its own agent, right? The teacher here is, let's say, a fixed algorithm. It has some randomized components with the level editing and so on. But I think this differs from a lot of these kind of curriculum approaches where people try to make the teacher deliberately into its own agent and try to sort of frame the adversarial setting in terms of two learning things, doing self-play. What kept you from doing... Are you still convinced that this might be a good way or are you also looking into the direction of making the teacher kind of a learnable component? Yes. So I guess the first thing to say is that when we started this project, we actually did envisage ourselves using a learned editor. And that was like what, personally, what I was really excited about at the beginning was having maybe even a population of editors that make different edits learned somehow, maybe to compete with each other. But the first thing we tried was the simplest thing. And often you hear this in research that the simple thing worked surprisingly well. And so we didn't really feel the need to really go beyond when we got results in mini-grid initially that were better than anything we'd seen before. We felt that it was actually better to go with a simpler approach. And maybe in the future we could consider ways to improve this by adding more learned components because that has been the trend elsewhere. But I think going from random sampling to evolution was enough to significantly improve based on the previous work. So we didn't need to go all the way to learn edits as well. But I mean, she has some additional thoughts on this. Yeah, I totally agree. I think the simplicity of it was... It was pleasantly surprising that such a simple method could unlock such a big gain in performance. In terms of treating the teacher as an agent, I guess a lot of where this work derives from is this paired method, which did treat the teacher as an agent. And actually the teacher was trained using reinforcement learning. And from based on all the empirical results that we've so far collected in the process of writing these papers, one thing that we have seen is that it seems that RL is not a very efficient way to train an agent to solve this problem of presenting always the most challenging task for a student. And I think the reason is because it's such a highly non-stationary problem. Basically, throughout training, your student's going to get better at certain things, maybe get worse at others. And the policy is always evolving. It's very non-stationary. So to be able to always track where in the parameter space will correspond to the levels that maximally challenge that non-stationary policy, I think that's a very hard problem for RL to solve, especially given how sample inefficient RL can be. And so I think one of the reasons why methods like random sampling that PLR does, it works so well, is because it's really able to escape sort of the limitations of RL and just directly sample for points in the space. And you're also not locally bound to just only be able to move a small amount based on a gradient step. You can really just sample anything anywhere in the space because it's randomly searching, and then the curator just creates the best ones. So I think that at least within these types of domains we've looked at, this type of random search plus evolution strategy just definitely outperforms a learned teacher. And in your architecture, I found you mentioned a bunch of times that you are relatively independent of domain-specific heuristics and things like this. Specifically, you criticized Poet for choosing an arbitrary range of returns of... They just select levels where the agents achieve between 50 and 300, which they claim to be hard, but not too hard. And yet I find, for example, in your algorithm, you need something like, well, we only put something into the buffer if the regret is above a certain threshold. Couldn't I leverage kind of the same criticism to you and say, well, probably that threshold is going to be problem-specific, right? And it's kind of a hyperparameter that doesn't seem like it's dependent on the environment, but is it? I think you're right that this is dependent on the domain. But I'll say the specific point about the hyperparameter. That one is actually a bit more benevolent of an issue, I think, because that's actually not a hyperparameter in our method. It's just whatever is the lowest score inside the buffer is the threshold. But I think that's definitely... I think if someone like you read it that way, I think we should definitely reword that in the paper. I think that's definitely going to be an improvement to clarity on that point. But the threshold is basically whatever is the lowest score in the level buffer. And if it's better than the lowest one, we replace it. So it's kind of like a priority queue in terms of the regret. But I agree with you. I think that methods like Excel and methods that basically require you to directly modify levels to construct them, I think these types of methods are always going to be domain-specific because I think at the end of the day, you need to have a way of parameterizing the environment. And that's domain knowledge. And you need to parameterize how you're editing that level. Yeah, I guess the editing itself is also, I think it's more... There's probably more domain knowledge than one cares to admit. Because yeah, you think like, okay, in block world, I'm just modifying one block to be there or not, right? But there is a decision of, you know, do I modify one block? Do I modify a block of blocks? Do I place a wall, an entire wall or not? And things like this. And depending on how much you edit, because you have this assumption, right? Which is that if I modify, if I make... Like my modifications need to be small enough such they don't influence the hardness of the level too much, yet they need to be large enough such that they do bring some variation into the picture, right? And that balance, do you think that balance, do you think that balance, it might be easy in these kinds of levels? What, like, how do you find this balance in more challenging problems? Like, I don't know if you think further, yeah. So I guess in these problems, it's worth noting that for the block situation, the actual domain randomization process places the blocks one at a time. So all we're really doing is kind of saying you have a few more steps of that initial process. So it is fairly aligned with the whole problem there. And then in the Bipedal Walker setting, we're just making small changes to the encoding vector. And in both settings, we have these details of this in the appendix, if you dare to venture. But in both settings, we did sort of a sweep over the number of edits you can make in one go. And in both cases, we found that all the values worked well. We obviously picked the one that was the best performing on our validation sets. But it didn't, it seemed fairly robust to the number of edits you make. And the thing worth noting, again, there is that what you could do is if, for example, you don't care as much about the number of samples you use to find a high regret level, you could just try out, try all of these values in one batch. And then because with PLR based methods, you just curate the ones that high regret, you could say, okay, I'm going to do some with one edit, some with two, some with three, some with four, or whatever it might be. And you could almost scale the size of the edits. And then just from that batch, just take the high regret ones. And you're probably still going to have more new high regret levels than you would if you ran an example from the initial distribution. So I think that there is some flexibility to do something like that. And I would argue that you could frame a lot of things in this editing sort of framework. And I think we mentioned a couple of examples, like perturbing latent, latency in a generative model, for example, that may be seen as more general than specific encoding for environments. It is a good point. I want to stick on this a little bit the the types of problems where these methods are applicable, because they seem very general, yet it feels like you need a problem where you can construct such a curriculum. And that curriculum needs to be fairly smooth, let's say so that the difficulty increase needs to be manageable, and so on. And also, the regret, the way you calculate regret with the with the TD error, it means that probably an environment like the Walker, where I, you know, I get more reward, the further I go, is probably more conducive than something like a Montezuma's revenge, even though I have a TD error and so on that kind of smooths out the loss itself. Can you comment a little bit on what kind of problems would like, where would it start to struggle? Like, where would you probably have trouble applying something like this? And where would it work? Obviously, work super well on these types of things that you tried it on. But where would it struggle? Yeah, I think you're right. It's got to have it's got to be a domain where you do have some structure that progressively gets, you know, goes from simpler to more complex. And it's, I guess, one nice benefit of these methods is that you don't need to know ahead of time what exactly does it mean for a level in this domain to be hard, easy or hard, because we have this regret based heuristic to tell us that. And if you do have sort of this progressive structure within the domain, then these methods can sort of start to emerge that based on the statistic. But I think that at least with these PLR based methods, because the core is still needle in the haystack, you're looking for high regret levels by random search, and then evolution in Excel just massively augments that in terms of the amount of training data you can get from high regret levels. But the bottleneck step is still sort of like this limitation around at some point, you still have to just get that needle in the haystack. And so I think as the design space, like the dimensionality of your environment gets bigger and bigger, I would expect that these methods become less and less efficient. Do you? Yeah, a couple of... Oh, sorry. I think we have like a one second lag or so. All right, sorry. So I guess one other thing, one perspective of this is it's really just a black box optimization problem where the function returns regret. And so we've gone from random sampling to evolution. But if you look at black box optimization literature, there are plenty of methods that trade off between global and local optimization in a more elegant way. And so what you could do is have some model or approach that maybe samples points more like diversity in the space. And then you use something like Excel locally to make edits once you found that needle in the haystack that Minxie mentioned. And then the second thing is that I think one place where this might break down is because it is quite a kind of greedy local optimization process, is if you haven't got sort of a very clear, like high to low sort of environment, then maybe you need something to encourage diversity. So you need to maybe have some sort of like either buffer could be maybe like hierarchical or something, or you could try and preserve levels that you think are conducive to edits later on, even if they're not the current high regret levels. And these are all ideas we talked about future work. I think really what we need is we need to have these more challenging problems to actually break our current methods before we can really think of the hammer for these nails. But yeah. What is a bit special as well is that you train a single agent, right, because usually the evolutionary methods they are trying to get a population of agents to work, even if they want to end up with a single agent, very often. And you encode all of this into a single agent. And that's kind of a PPO really basic agent, if I want to say. And I have noticed a little bit that in these demonstrations, no matter what the level is, kind of the strategy tends to be the same, right? It tends to kind of, it tends to hop on this one leg with the other one with the other one out. And that is sort of the best strategy to overcome any and all obstacles. And then kind of rebalance itself once it's, yeah, this one, see? So, yeah, maybe we've been walking wrong our whole lives. But no, I mean, it's obvious if you instill this in a single agent, how much of a how much because I also observed some of your results here over time, which was also really cool to see when you compare it to the poet algorithm, in that you do get kind of more challenging levels later on, but they also, like, they don't dominate, it doesn't get more and more and more and more challenging, right? How much of this is a property of like catastrophic forgetting of the agent itself, where you kind of push for the more complicated levels, but all of a sudden, it can't can't solve the easy ones anymore. And therefore, the easy ones become high regret. And then there's kind of this, like how much of this is due to your algorithm? And how much of this is due to the fact that you have a single agent trained with PPO that needs to take care of all of these tasks at the same time? My guess is it's the latter part. Because I think that having this buffer that we do have, which in the robust PLR and the previous PLR paper, it does somewhat help with forgetting because you're able to sample things you haven't seen for a while. And if and if you now can't solve them as well, or or if you now have high regret in these levels, then you should retrain on them. So it should somewhat eliminate forgetting. But I do think it's worth noting that this agent is just a two hidden layer neural net policy. It's not not very flexible. It's pretty like low dimensional. And I think it really is unable to adapt to every different possible behavior. And so I think either having something where you can co evolve the architecture as well to maybe make it more flexible as the levels get harder, or even just making your agent be some sort of adaptive agent, like a meta learning algorithm, for example, that does zero shot adaptation. I think these approaches are things that we're excited about maybe for future work. But I think for this, it's sort of an inevitability that you try and have this like lofty goal of having a generally capable agent, it's going to have some brittleness to some certain components. I think we found a few cases like uphill, it's not particularly good. Yeah, when we started visualizing it in this viewer that we have in the demo, we noticed that, you know, like, when we were we're training this thing, all the complexity metrics for like roughness of the ground, it started going up very quickly. But then when we actually printed out a lot of the levels where it's successful, they tend to be levels, where it's all downhill, which means that this pogo stick strategy, it's very good at just like hopping down the hill, and it's really robust at landing, like just sticking the landing in terms of like really high clips. So it's really good for us. But when you start to get more like these rugged hills going uphill, where the slope is positive, that's where it starts to struggle. So that's like a really interesting and I think a very tangible sort of example, where there's sort of a collapse in diversity in a way in the curriculum where, because it is a limited, we do replay old levels, but again, it's a limited finite buffer. So you can get, you know, sort of like a buffer overflow in a sense of, you know, levels that collapse in terms of similar challenges. And then maybe the agent just gets too good at going downhill, jumping down really challenging hills, but then it starts to the curriculum starts to forget that also going uphill is also important. And maybe that's what happened in some of these training runs. I like the, I like the approach. I think poet or poet V2 had some sort of an approach where they do of course have different agents, but they had this metric of ranking the environments that they have in the buffer, right? And sort of ranking them with respect to different agents. And their conclusion was that if the different agents rank the environments in a different way, that kind of indicates a diversity of levels, right? Whereas if they rank them the same way, it's kind of like, well, they're not really diverse. I think much like your regret measure, I'm a big fan of these, they're not super domain independent, but they are domain independent enough, right? So that you could like, you can kind of disconnect them from the real problem at hand. That's pretty cool. That one is definitely, I think more general. I think that's quite an exciting approach. Maybe if you wanted to use population, maybe even generate experiences, then that's quite a nice way of evaluating the diversity, I think. So is it fair to say that kind of the end here, like the most, let's say you train this, let's assume this is convergence at 5,000 step, that this is kind of a representation, it's almost like a fingerprint of the agent's ability in the face of a curriculum that tries to push harder and harder, right? Because there's a trade off that the easy levels, not being in the buffer or not being, yeah, not being in the buffer means they're easy, they can be solved, right? But then also, yeah, this is, it seems like this is the curriculum that's needed for the agent to be as general as possible, not necessarily as good as possible. So yeah, I think it's worth noting as well that Minxie added a really cool feature to the website where you can actually see five seeds of each method. I don't know if you've seen that version, but you can see that the Excel agents are pretty remarkably similar. So they almost all seem to follow quite a similar gate, which makes me think that this is kind of the solution that for this network does cover the space as best as possible. And so it might be the case maybe that to get better behavior and better performance, maybe you need to have, there you go, show all seeds, maybe you need to have something that's a little bit more flexible, either something with memory, or I think some implementations like that of Walker use frame stacking, these types of things, maybe you can get more capacity into the network that way. And I think it's probably possible or likely that, there you go, it's probably quite likely that this is the best policy you can get with this network to have this Minx regret approach. Yeah, there is one survivor. Well, we'll see. Yeah, excellent. Cool. Yeah, the website is definitely pretty cool. The last interesting thing I found, at least for me here, was this generalization to the maze. And I mean, it's very cool because you train on these made up mazes starting from empty rooms, and then you test on these kind of human generated mazes right here, and then you generalize to this giant maze here. Now, you say yourself, the agent seems to follow this kind of bit of a left hand rule. How does something like this emerge? Because it doesn't seem like in the generated levels, a left hand rule would be beneficial because they're actually going to be more of a loop and stuff in that. How does a strategy like this emerge? I guess one thing that's quite worth noting in this environment is partially observable. So you only need to regenerate a small bit of structure within the grid for it to kind of generalize maybe to larger grids. But I think that's the thing that's more impressive about it. Yeah, exactly. And that actually makes this really hard, even for a human. If you imagine you didn't know where the green dot was and try and do this, as a 5,000... I think most humans would not be able to do this. I certainly lost patience with it after a couple of goes. There's like a 5,000 step limit, so it's quite long. But if you look at the Excel sort of towards the end of training as well, in the mini grid domain, a lot of the levels... So it ends up converging towards around 60 block count. And that's sort of like the threshold beyond which a lot of the levels where you randomly sample like more than 60 blocks, they tend to be unsolvable. So they tend to have a block preventing you from getting to the goal. And so 60 seems to be like the sweet spot for a 15 by 15 maze. And when you get to that set, like that amount of saturation, you're going to be able to do a lot of things. And when you get to that set, like that amount of saturation of blocks, a lot of the levels tend to actually become effectively single component mazes. And so those are unsolvable by the left hand rule. So I think that's also like just a contributing factor, like some property of the specific dimensionality that we looked at resulted in the complexity converging to lots of mazes that are single component. And it helps the agent basically learn this left hand rule. Yeah, it's pretty cool. Do you, I didn't dive too much into the experimental results in my review. Is there like, what are some of the things that you might want to highlight across your experimental results, maybe that you find more interesting than the average person would when they read the paper? I guess for me, it's two things. So the first one is that the complexity is entirely emergent. So we never encourage the agents to actually increase the block count. We never encourage it to increase the stump height and bipedal walker. It just has to do that to increase the grip. So some other papers maybe all works, maybe they have some like ways to encourage this, whereas we actually didn't. So if we were to do that, maybe in the future, that's could even increase it even further. And then the second thing is that all of the test cases are zero shot evaluations. So the agents never seen the test levels. And I think it's quite remarkable how robust it is in quite a wide range of settings. So that's probably the two takeaways for me. We also had some results in the appendix where we actually, we also test the final Excel bipedal walker agent on top of the poet levels. So in poet, actually, they publish a few of the rose plots showing the different parameter settings for bipedal walker for some of the crazier environments. And we actually tested bipedal walk, our bipedal walker with Excel on those environments. But it actually, it didn't perform very strongly. So it's what's interesting is I think what's interesting about this result is it sort of highlights this duality between like the goals of these two algorithms, where I kind of see Excel as being on one side of the spectrum, which is about robustness, general robustness to unknown environments, and poet beyond the other side of the spectrum, where it's focused on getting specialists for basically finding these agent environment specialist pairs, where this agent just always solves this environment. And so it's kind of an interesting philosophical idea, because it's kind of saying that if you're building an AI system, do you really care about being robust to things that you don't know about? Or do you want to maximize your performance as a specialist? And I think it's a really interesting open question. And the way we navigate this trade off, I think is really full of rich ideas for future research projects. Yeah, especially ideas that could combine some of these things as well. And we've obviously talked about a lot of possible things. But actually, if you go a little bit few pages down, what we did was we actually took the some of the most complex levels that poet generates, and then we produced them in our own setting. And that's also 100 by 100 maze, if you're interested. 100 by 100. Did it solve it? Yeah, it has to be odd number for the for the simulators to work. Okay, okay. That one against the thing 8% success rate on that one. It's I think a bit above this. Is it table? Yeah. Higher up, higher up. Maybe. Do you want to check? What are you looking for? The poet. Yeah, it should be a small, it's like a very small table. I think it's down below. Search in the paper itself, I guess. We should have probably had paper up on our own screen. Well, my bad for for not knowing it too well. Oh, yeah, this is actually on the next page. This is the like main experiments on the next page. Ah, this is yes. Yeah, so one eight to three B are in the paper towards the end. They have like a rose plot for some of the most extremely challenging levels that each of their seeds generated. So for all three of their seeds, they pick two different levels that they're particularly high values. And we tested our agent zero shot on those. And yeah, the scores are pretty low. But I think the fact that they're above zero is cool. But at the same time, it does make you think that if they can solve those repeatedly, then maybe you do need specialists in some cases to get the most complex things. So some hybrid of specialists and generalists might be an even more powerful algorithm than either of them combined. Excellent. So you mentioned a bunch of different and you also have a future work section and so on. What do you think are apart from the things you're going to do next? What are like the big unsolved challenges in the field? Like what's what's everyone after but no one's been able to do it so far? Well, so the big one is a theme that we we as a group have gotten very interested in recently. And we're actually holding a workshop at iClear about this. And essentially, it's about Asian environment co-evolution. But in this in the context of this much older problem called open-endedness. And basically, open-endedness is an idea that it kind of came from a group of researchers, Ken Stanley, Joe Lehman, and Jeff Klun. And I think Jeff Klun has this concept of AI generating AI. And it's related to this idea of open-endedness where can you basically create a learning system that essentially ends up evolving just an unbounded amount of novelty and complexity. And if you can kickstart a process that achieves true open-endedness, then the idea is that maybe you can replicate the emergence of some really complex intelligences like human level intelligence. Because evolution like the tree of life, this is all sort of the result of an open-ended learning process. And so a lot of where we see this work going is that when we when we see our work is sort of fitting within this bigger theme of open-endedness, and this larger theme of agent environment co-evolution to achieve this open-endedness. And so I think that that's sort of to me is one of the most interesting open problems in AI or machine learning, or maybe it goes beyond even these two subjects. Yeah, so I think that if we can actually kick off a process like this, that would be incredible. And I'd be very curious to see what kinds of things fall out of it. Yeah, and for me, the thing I'm really excited about is that, again, tying in with Minchis is this seems like the only limitation to this really being open-ended is requirement for a simulator. So I'm really excited about whether we can actually learn simulators, for example, world models. So I was obviously very inspired by the Harnsh Riddhiever work from 2018. But more modern like offline RL world models. So maybe you have some transformer world model that learns from all this crazy amount of data. And then you can use that to design environments for an RL agent and then collect more data and just keep going. And maybe that's how you really get towards this true open-endedness, because you're not bounded by just the open AI environment that you're given. And so this is maybe it's a little bit more of a medium to long term goal, because I think we're a bit away from that right now. But I think that that could be where these different fields intersect and really produce something pretty crazy. Yeah. My issue a little bit with the agent environment coevolution work is that it just seems to shift the problem away from because, okay, we're evolving the environments right here, but they're still extremely bounded in an extremely parameterized space. And there's only these many ways that the environment can vary. And the true environment is kind of like the environment generator itself. And it seems like we could go a level higher and so on. But is there a method to generally break out of this being bound to any framework? I think one way is it's related to what Jack just described, which is this. So you've heard of sim to real as the paradigm, where you train intelligence in simulation, you transfer to reality. And that's obviously bounded by the fidelity of your simulator for your target domain. There's a new paradigm emerging. And it's like sort of pushed by all these advances in computer vision, which some people have called real to sim to real. And basically the idea that you can essentially collect data in a loop where you may have some exploratory agent, maybe it's a hand coded controller, or maybe it's an RL agent, the one you're training, and you send it out into the wild, it collects lots of data about what the world is like. And then you use that data to essentially enrich your simulator to basically fit your simulator to reality, to all the new things it's learned. And then you get a better, more expansive simulator, you train your agent again in that simulator, and you get a new agent to transfer to reality. And then this loop just keeps repeating. And maybe you can do this in a population of agents doing this. And you get really huge coverage in terms of what's out there. I think that's one promising way to do it. The other though, I think it kind of just generally the strategy is, like you said, all these simulators are bounded in terms of their parameterization. Like we are looking at 15 by 15 NASES. There's a finite number of them. I think what would be really cool is if we started as RL researchers, started focusing more on environments that are unbounded in parameterization. So moving into these like more almost non-parametric settings, where the environment can just keep growing arbitrarily in its number of parameters. And I actually think the real to sim to real loop is one way to do that, just because the space of possible worlds you can represent as a world model, as a neural network, is pretty much infinite. But maybe there are other simpler ways you can do this as initial toy tests as well. And then when you have that real sim to real world model, you can then train a mini max regret policy inside it. Yeah. Because then you have like this idea of the population generating this diverse, you know, very high dimensional world model, but then a single agent maybe that could be robust to any possible variation. And so this is maybe a bit of a medium term. But I think for us, it's kind of a North Star at the moment. Do you think there will ever be sorry, last question by me, do you think there will ever be this distinction between agent and environment? Will this continue to be an important distinction? Or is that something that you see in the future vanish and kind of almost become like, let's say interchangeable because people are already like pitting them against each other, training them both with RL and so on? Like, why do we even make the distinction? Well, I guess one thing that's interesting is even in the original world models paper, because the world model itself was generative model, the policy was very low dimensional, it just trained inside the latent state, latent space of the generative model. So then when you actually interacted with the real environment, you still use the encoder from the world model to process the input so that the policy can then operate. And so in that sense, it's like the world model is the environment at training time offline. But then at test time, when you go back to the real environment, the world model is used to process the inputs for the policy. And so they're kind of taking a very like, I guess, competitive and then a cooperative mindset. So I think maybe there's something like that, where you have world models that are your environment for training time, but then you use them as knowledge bases for test time. I think that's pretty exciting. And it also kind of relates to this idea of the cherry on top, because the policy is very small, although I hate to use too many cliches. But it does seem to relate to that sort of self supervised learning large world models, and then RL just for controllers inside that, that can operate on the representations. I don't know if I mentioned you things about that. Well, I think to sort of answer the other side of that question, I think that agent environment, I guess the distinction is, in some ways, it's arbitrary, because you can imagine, you know, like what part of this learning system actually belongs to the agent? Like, is the agent really like at the activation level? Is it at the observation level? Like, where do you even draw the boundary in terms of the agent? I think that's an interesting question. But I also think that at some point, there's going to be some substrate in which the agent has to operate within. And there seems to be, like, basically, if you wanted to emerge a diverse sort of, you know, a tree of life of different RL agents and environments, it seems like there is some sort of asymmetry there in the sense that agents have to operate within an environment, and you can't have it reversed. And so in some to some extent, I think we'll still have to have this distinction between agents and environments. But it's also possible, you know, like, maybe we could also just learn, you know, joint distributions over agents and environments, where you basically just learn, you know, like, the agents parameters themselves are now part of the environment design. And so now you're just emerging agents and environments together inside of a single generative model. I think that's an exciting idea. But and maybe at some point, we'll figure out how to do that. Where can people get started with this if they want to dive into it? So there's a great for open endedness, there's a great primer to it on O'Reilly, I can actually send you the link after, but it's written by some of the original sort of pioneers within this field. And essentially, it's quite long, but it summarizes the whole field. Another, another really interesting work would be, I think, just to check out the original mini max regret paper for RL, which is this emerging complexity for zero shot generalization from Michael Dennis and Natasha Jig, Jax. And I would definitely recommend, you know, our line of work with robust PLR checking out this paper. And there's older methods like teacher student curriculum learning from Shuman Shuman's group at OpenAI. And workshop. Yeah. So we're going to have an iClear workshop called agent learning in open endedness, alo. And that's going to feature a lot of speakers and researchers actively making progress in this field. So if people are really interested, they should attend some of the talks and check out the poster session that'll be that's April 29, April 29. Yeah, Friday. Good. Also, more in a multi agent setting, there's the curriculum learning manifesto from Joel, Joel Levo, and DeepMind. And that has some really nice ideas in terms of automatic curriculum learning, emerging, emerging complexity. Cool. Minchi and Jack, thank you very much for being here. This was really cool. Thank you for having us. It was very fun.
 Hi, this is an interview with the authors of the paper evolving curricula with regret
 based environment design. If you haven't seen it, I've made a review of this paper yesterday,
 the day before this video is released. And I went over the paper in detail and explained
 what's inside of it. So if you haven't seen that, it would be a good place to start today
 I'm interviewing the authors of this paper, Jack and Minchi, who are real experts in this
 domain. Now during the interview, we go a lot deeper than I could do myself in the paper
 review. And you learn a lot more about how things work in this paper, but also in the
 entire field. It's a very exciting field. And it's a real privilege to be able to interview
 all of these people. I hope you're having fun. Please let me know in the comments how
 I can make these videos better for you. And thank you to everyone who does watch who does
 comment who does share. Thank you to all the supporters on Patreon to all the discord members
 and to everyone else who is excited by machine learning. I hope you're doing well. Stay hydrated.
 Now let's get into the interview.
 Parker Holder and Minchi Chang. Did I get this right? Yeah. Thank you. Welcome very much
 to the show. Thanks for having us. I think your paper here, it was of one one sort of
 an example of a very cool paper, because it's not on a state's a bit out of the mainstream,
 usually reinforcement learning tackles improving the agent as much as possible, where you you
 go much into this road of poet and work before it improving the environment. But also I think
 it's a good lesson in how to kind of put a bit of publicity behind the paper because
 you made this this very cool website right here when this with the interactive demo where
 I can play around with the terrain, right? Okay, if it only works. And you have these
 these kind of nice animations of how things develop during training and so on. And I think,
 like, how much do you think something like this helps a paper after it's released? Like,
 what was your impression of just just kind of, or maybe you can tell me a little bit.
 How did you how did you even decide paper aside to make a website like this and present
 it in a form that's interactive? I think with RL research, especially when you look at curriculum
 design, you're modifying the environments, there's always really interesting visualizations
 that you can share. But I think having just like the standard PDF format that everyone
 publishes on archive, then is really, really limiting. And there's just so much, there's
 so much amazing like assets you can actually share in terms of your agent behavior, in
 terms of the emergent complexity that these algorithms generate. So we really wanted to
 share that with readers. And we thought that would definitely capture more of people's
 imaginations when they engage with our work. And there's like also just a huge sort of
 lineage of work that tries to do a similar thing, like our template for this website
 is actually taken from distil. So distil pub has so many great works, and they put so much
 effort into making such beautiful interactive publications. And we definitely took a lot
 of inspiration from that. David Ha, Google Brain has a bunch of publications like with
 world models and tension agent that did similar things.
 Yeah. And then also we use the teach my agent work from the flowers lab as well, which had
 some of the like building blocks for this. And that was really cool. But I think the
 other thing is like, there's always this question with these type of methods, if you picked
 the test environments by your method works, and as reviewers ourselves, we're always very
 cynical of this. And so we kind of thought, what if we just let people try and break it
 into what happens. And of course, you can break it pretty easily. And that actually
 leads to kind of exciting questions of how you can make it better in future work. But
 at the same time, it's kind of nice to see how it does and doesn't work. Because then
 the day I think we should be more honest about the robustness of our agents. And this is
 quite a nice tool to not only make it fun, but also kind of demonstrate it.
 I think more also for not just for readers, but I think just for ourselves as researchers,
 like in the process of making this tool, and starting to actually run the agent and tons
 of visualized environments, we actually started to discover certain shortcomings of the agent.
 Like you can look at all these plots all day long, and you see all the metrics go up into
 the right. But then you don't actually see sort of the blind spots that come up during
 training until you actually visualize it. And we discovered a few interesting motifs
 that that consistently challenged the agent, even though it's overall quite robust.
 Yeah, because we're actually going to talk we're talking about maybe like making it so
 that it defaulted to levels that we know it can do well on but then we just thought I
 kind of removed the fun. And at the end of the day, if it breaks and someone's inspired
 to improve it, that's ultimately a good thing.
 Yeah, I mean, you do have the metrics to prove that it does something well, right? And anything
 after that is a bonus, essentially. How did you get even into this? How did you get even
 into this field? Do you maybe want to like give a 30 second bio of yourself? Like how
 did you arrive at this point?
 Sure. So I mean, from my perspective, I came out before my PhD, and I thought it was really
 inspirational, really cool work. But I didn't really know if I'd ever get to work on something
 like that. And then obviously, interning last summer at a matter with Tim and Ed and Munchi,
 who are on paper and Mika as well. The group was working on generalization and starting
 to improve on idea and build on ideas such as like paired and these algorithms. And so
 then, so when I came in, we were talking a little bit about like shortcomings of our
 methods. And then Poet obviously comes up as another example. And we were kind of thinking,
 how do we take some of the ideas from Poet and really incorporate into our existing,
 like regret based curriculum methods. And so then it became kind of obvious that we
 want to try this environment and this type of work. I guess it was kind of a fusion of
 different things. So it was like top down initially, and then also ended up being bottom
 up.
 Yeah. And I guess curriculum learning was something I kind of stumbled on in the first
 year of my PhD. And basically, I was originally trying a bunch of sort of random ideas of,
 I always had this notion that maybe RL could be made more efficient if you train agents
 on levels that were just within reach. And then you basically progressively increased
 the level complexity in terms of a curriculum. And so we worked on a prior method as well
 called Prior Test Level Replay, which is this pink PLR baseline here. And that one ended
 up doing quite well, especially when combined with data augmentation on the OpenAI ProcGem
 benchmark. And so right after that, I got in touch with another researcher at UC Berkeley,
 a fellow named Michael Dennis, and he was one of the first authors on the Emerging Complexity
 for Zero Shot Robustness paper that introduced the paired algorithm. And so this is the paper
 that kind of introduced a lot of the formal theory, decision theory around minimax regret
 policies in their application within Deep RL. And it kind of was the first paper that
 showed that if you optimize for minimax regret in using Deep RL, it makes sense and you get
 nice experimental results that show robustness in zero shot transfer. And so we started discussing
 and we realized that actually a lot of the theory could be applied to PLR. And that PLR
 was actually another instantiation of this minimax regret game, which is at the heart
 of this theory. And Excel is sort of like the latest version. It's sort of the culmination
 of the ideas we've explored so far in this direction.
 Yeah, I guess it's worth noting that we published the robust PLR paper in Europe last year.
 So that was really that what was finishing just around June, July time when I joined
 that meta. And so really we were looking, we kind of knew that method was very empirically
 strong and theoretically nice, but it still maybe lacked something in that it couldn't
 really have some creative process to design its own levels because it could only sample,
 I think, as you as you pointed out in your review. So ultimately, if the space is very
 high dimensional, and you only sample one high regret level, once you've mastered it,
 you have to then go back to the drawing board. Whereas the nice thing about Excel is that
 it's by a poet, it can really kind of build its own complexity over time. And so it really
 is kind of like a progression through to really sequence of papers, I guess. And, and to be
 fair, Michael's been on now three of them in a row because he was on paired and then
 robust PLR and Excel.
 Can you give like a layman's layman's explanation for optimizing for mini max regret? Because
 there are a bunch of like, it's regret, and then max and then min. What's what what does
 it ultimately boil down to?
 So, so, so, this largely comes from this emerging complexity paper from Michael Dennis and Natasha
 Jax. Essentially, the theory there is essentially framing, framing a concept called unsupervised
 environment design, as essentially this problem where you want to design environments that
 maximize for some metric, and that metric is usually some behavioral metric that's associated
 with the student agent. And so in this game, in this mini max regret game, we care about
 maximizing the regret of the agent. And so if you frame the game as a game where it's
 a two player game, it's zero sum, the payoff for the student is the negative regret, and
 the payoff for the teacher is the positive regret. Essentially, you have a game where
 the teacher tries to increase the regret of the student and students trying to minimize
 its regret. So if you think about two players, you're some games, they always have a Nash
 equilibrium. And at the Nash equilibrium of this game, it's got to be the policy that
 the student plays that essentially is a mini max regret policy, it's minimizing its worst
 case regret. Because if it's not doing this, the teacher must be able to change its policy
 and play more of a certain level that further increases the regret. And so by definition
 at a Nash equilibrium, neither player has an improving response. So it must be that
 the student has a mini max regret policy. So what does that mean in layman's terms?
 It basically means that the student behaves in a way that essentially it's able to do
 well in any level that's solvable inside of the parameterized space of tasks that the
 teacher can use to propose the next level. So the yes, it should always be sorry, the
 teacher would have the teacher's moves was essentially be the levels like the actions
 of the teacher would be I play this level. Yeah. So it's within the subtraction, called
 a U-POM DP, which is just like a partially observable Markov decision process. But you
 add an additional set of variables called the free parameters. In the papers, we usually
 use the term theta to denote them. And so then those are like the positions of where
 the obstacles are in the maze, in the maze domain, might be like starting position of
 the agent goal position. Inside of the car racing environment, it might be like the position
 of where the tracks are. And so these are the design parameters. And so a strategy of
 the teacher is essentially like choose some distribution over choices of the possible
 free parameters that it can sample as the next level. Sorry, Jack, you go.
 All right. I was gonna say like the nice intuitive property of this is that it makes the agent
 has to learn to solve all of the simplest solvable environments as well. So in some
 other methods like poet, they're trying to achieve the maximum complexity, which is like,
 it's very cool as well motivated. But this is quite different in that we're actually
 happy if even later in training our agents training on simple levels, if it means that
 it can solve all of the simple levels, because we don't really care as much about solving
 like crazy complex things if it can break some simple thing, which I think is seems
 to make sense, at least to me. Yeah, that was one of my let's say worries
 right here is that if you if you and I framed this a little bit as you are at this zone
 of proximal development with your agent in that somehow made it wrong, like you try to
 reach levels that are just outside of where the agent can handle it. And then you you
 try to edit those a little bit or maybe just where the agent can handle them. And then
 you try to edit them a little bit. And you try to filter by the ones that pass some threshold
 in this estimated regret. So my first question would be coming back to this regret, you you
 formulated as the so it's it's formulated as the difference to the optimal policy, right?
 The difference to to the optimal policy, I'm going to guess on this particular level that
 you're at. Why doesn't this like disregard the approximation that you do? If I could
 calculate this very accurately, wouldn't this select for super duper difficult levels that
 could be solved with the optimal policy, right? Not impossible, but just super difficult ones?
 That's a great question. And I think part of the part of the nuanced detail here is that
 so one reason that makes this all work is the discount factor. So basically, the so
 in the original paper that introduced paired and this idea of the mini master game, the reward
 function for that environment actually, it actually your reward, your final return decreases
 with the length of your trajectory. And so there's a natural discounting in terms of the return.
 And so essentially, by doing mini max regret, it ends up prioritizing for those levels where
 the solutions within reach in the fewest number of steps. And you get this nice curriculum. But
 because here in all of our approximate single agent regret estimators, we're using a value
 function, which is bootstrapped off of a generalized advantage estimator, which itself is discounted,
 you essentially have discounting built into your value function. And so you end up with discounting
 even if they're even if your environments are final, you know, sparse reward, no discounting
 naturally in the external reward, you still get discounting because your value function is going
 to be discounting using gamma. And if you use GAE, you have further discounting with lambda.
 Cool. Yeah, that was one of my one of the things that I didn't exactly understand here in this.
 Okay, I was like, disregard the discount factors. They're not important. Turns out they're actually
 one of the most important parts right here to actually make it work. Although you use this
 this positive value loss. Now, I think you wrote me in an email before that I got this wrong in the
 in the paper review. Do you want to maybe quickly discuss what the individual parts of this formula
 mean and what they do?
 I mean, so essentially, the I guess we can start from sort of the outside in, I guess, or maybe it
 makes sense to do the inside out. So basically, the innermost term is essentially just a TD error.
 It's a one step TD error. And it's future facing. So it's from your current time step t until the
 horizon t, capital T. And essentially, the inner term except for within the max, that term is
 basically, if you look at the sum from t to capital T, that's basically the generalized advantage
 estimator from Schumann et al. And so that one is the most common, that's the advantage estimator
 used in PPO. It's used in other policy gradient algorithms as well. But essentially, that is
 essentially estimating your advantage while trying to do a trade off between one step TD errors being
 more biased because it's bootstrapping off of fewer steps, and longer TD errors being less biased,
 but having more variance. And so lambda is a discount factor that controls for that.
 And so in a nutshell, though, this is estimating advantage, which is basically, this is my actual
 return minus my typical return, which you can think of as what the value function outputs.
 And so the zero... So this is, sorry, this is return minus value.
 Yeah, you can think of it as the return you achieved minus your value prediction at each step
 in your trajectory. And we average it over the trajectory. And essentially, that's telling us,
 if that's really high, it means that I'm doing better than what I typically do.
 And so directionally, this is like in the direction of regret, because it means that
 in terms of external regret, I can actually get a higher return than I typically do,
 which means that this is a level where I experience regret. And then we max this with zero,
 which just means that we are only looking at the positive time steps where, at the time steps at
 which this term is positive. So we're only looking when the agent does better than it typically does.
 And if on average, when it does better than it typically does is quite high, it means that
 it's a level where the agent can experience a lot of regret in its decision making.
 How so though? Like my logic was a little bit, if I'm worse than I estimated, that means kind of
 it's a difficult level. Like where's my thinking wrong here?
 So if you do worse than you estimated, I think in terms of just the mini match regret framework,
 it's just a little bit sideways in terms of measuring the direction of regret.
 I think if you think of it as looking for cases where you do better than you typically do,
 that's really just you discovering regret. It's like you discovered a case where you achieve
 regret relative to your typical self, as sort of amortized by this value function that predicts
 how well you typically do over time. So with respect to sort of this average prediction of
 yourself, you're doing better. And so you're essentially discovering sources of new regret
 in this level. And that's basically directionally aligned with maximizing regret. While if you were
 to do the opposite, if you were to say, I want to look for the steps where I do worse than I think
 I do, I think that's an interesting thing to try actually. But at least theoretically,
 it doesn't seem to align with mini match regret as well.
 Yeah, okay. I can see the logic in that you say, I want to find levels where there's something
 unexpected positive thing happening.
 Yeah, it's worth noting as well that impaired, which was the first UD algorithm to use regret,
 they had a very different approaches, which had a second agent called an antagonist. And the regret
 was just the difference in performance between those two. And so maybe that's like, a bit more
 intuitive, because if the antagonist can solve a level and the protagonist, the student agent,
 can't, then I guess that's more intuitive in terms of what you expect from regret. But the nice thing
 about this is it's kind of a cheap approximate for single agent regret. And we definitely feel like
 maybe coming up with better metrics for single agent regret is exciting future work that could
 be improved upon here. But this was taken just from the robust PLR paper, and we were surprised
 how well it worked in quite different environments.
 And another detail is in the robust PLR work, another regress meter we use is the
 another regress meter we used that we explored was what we call maximum Monte Carlo regret estimator.
 And essentially, it's the same, it's almost the same expression, except the regret target is no
 longer what you just received inside of a recent episodic rollout. It's for every level, we keep
 track of the highest return you ever achieved throughout training on that level. And we use
 that as an estimate for the maximum performance on that level. And then we use that as the target to
 subtract your value prediction on. And so that's like a more off policy regret, which I think,
 in some cases might be better because it's less coupled to your current policy. While the positive
 value loss, it's always what you recently received in a rollout in terms of your target,
 minus your value function prediction.
 Yeah. Is that worth because you would introduce some extra variance, because you're not
 essentially subtracting your own bait, like use this as a baseline in the advantage estimate?
 Or am I seeing this wrong? So this would introduce extra variance.
 It's not using the policy update, it's used just to score the levels. So essentially,
 you're saying the best you've ever done, which might be more, it's going to upper bound your
 current performance, right? The best you've ever done, including your current performance,
 versus your value function. So it's slightly nicer in a sense that if you've experienced
 a level many times, maybe you've had some forgetting, then the regret should be higher
 because you've done well in the past. But the negative is you have to then store all of your
 previous episodes for every level. And then oftentimes you don't actually have any previous
 experience. So it's not even that applicable, but there's a trade-off. And I think, again,
 I think this is something that could be improved in future work.
 Especially with procedurally generated content, it's probably hard. You'd have to build some sort
 of a, even a model to estimate the best possible regret given past procedurally generated levels
 to sort of predict for any new one. And those two models will probably make similar sorts of
 mistakes, like the mistakes might even be correlated between the... Okay. So with respect to your
 method here, which is decently simple, what I was surprised by is that you deliberately go away
 from the teacher being its own agent, right? The teacher here is, let's say, a fixed algorithm.
 It has some randomized components with the level editing and so on. But I think this differs from
 a lot of these kind of curriculum approaches where people try to make the teacher deliberately
 into its own agent and try to sort of frame the adversarial setting in terms of two learning
 things, doing self-play. What kept you from doing... Are you still convinced that
 this might be a good way or are you also looking into the direction of making the teacher kind of
 a learnable component? Yes. So I guess the first thing to say is that when we started this project,
 we actually did envisage ourselves using a learned editor. And that was like what, personally,
 what I was really excited about at the beginning was having maybe even a population of editors
 that make different edits learned somehow, maybe to compete with each other. But the first thing
 we tried was the simplest thing. And often you hear this in research that the simple thing worked
 surprisingly well. And so we didn't really feel the need to really go beyond when we got results in
 mini-grid initially that were better than anything we'd seen before. We felt that it was actually
 better to go with a simpler approach. And maybe in the future we could consider ways to
 improve this by adding more learned components because that has been the trend elsewhere. But
 I think going from random sampling to evolution was enough to significantly improve based on the
 previous work. So we didn't need to go all the way to learn edits as well. But I mean, she has
 some additional thoughts on this. Yeah, I totally agree. I think the simplicity of it was... It was
 pleasantly surprising that such a simple method could unlock such a big gain in performance.
 In terms of treating the teacher as an agent, I guess a lot of where this work derives from
 is this paired method, which did treat the teacher as an agent. And actually the teacher was
 trained using reinforcement learning. And from based on all the empirical results that we've
 so far collected in the process of writing these papers, one thing that we have seen is that it
 seems that RL is not a very efficient way to train an agent to solve this problem of presenting
 always the most challenging task for a student. And I think the reason is because it's such a
 highly non-stationary problem. Basically, throughout training, your student's going to get
 better at certain things, maybe get worse at others. And the policy is always evolving. It's
 very non-stationary. So to be able to always track where in the parameter space will correspond to
 the levels that maximally challenge that non-stationary policy, I think that's a very
 hard problem for RL to solve, especially given how sample inefficient RL can be. And so I think one
 of the reasons why methods like random sampling that PLR does, it works so well, is because it's
 really able to escape sort of the limitations of RL and just directly sample for points in the space.
 And you're also not locally bound to just only be able to move a small amount based on a gradient
 step. You can really just sample anything anywhere in the space because it's randomly searching,
 and then the curator just creates the best ones. So I think that at least within these types of
 domains we've looked at, this type of random search plus evolution strategy just definitely
 outperforms a learned teacher. And in your architecture, I found you mentioned a bunch
 of times that you are relatively independent of domain-specific heuristics and things like this.
 Specifically, you criticized Poet for choosing an arbitrary range of returns of... They just select
 levels where the agents achieve between 50 and 300, which they claim to be hard, but not too hard.
 And yet I find, for example, in your algorithm, you need something like,
 well, we only put something into the buffer if the regret is above a certain threshold. Couldn't I
 leverage kind of the same criticism to you and say, well, probably that threshold is going to
 be problem-specific, right? And it's kind of a hyperparameter that doesn't seem like it's
 dependent on the environment, but is it? I think you're right that this is dependent
 on the domain. But I'll say the specific point about the hyperparameter. That one is actually a
 bit more benevolent of an issue, I think, because that's actually not a hyperparameter in our
 method. It's just whatever is the lowest score inside the buffer is the threshold. But I think
 that's definitely... I think if someone like you read it that way, I think we should definitely
 reword that in the paper. I think that's definitely going to be an improvement to clarity on that
 point. But the threshold is basically whatever is the lowest score in the level buffer. And if it's
 better than the lowest one, we replace it. So it's kind of like a priority queue in terms of the
 regret. But I agree with you. I think that methods like Excel and methods that basically require you
 to directly modify levels to construct them, I think these types of methods are always going to
 be domain-specific because I think at the end of the day, you need to have a way of parameterizing
 the environment. And that's domain knowledge. And you need to parameterize how you're editing
 that level. Yeah, I guess the editing itself is also, I think it's more... There's probably more
 domain knowledge than one cares to admit. Because yeah, you think like, okay, in block world, I'm
 just modifying one block to be there or not, right? But there is a decision of, you know, do I modify
 one block? Do I modify a block of blocks? Do I place a wall, an entire wall or not? And things
 like this. And depending on how much you edit, because you have this assumption, right? Which
 is that if I modify, if I make... Like my modifications need to be small enough such they
 don't influence the hardness of the level too much, yet they need to be large enough such that
 they do bring some variation into the picture, right? And that balance, do you think that balance,
 do you think that balance, it might be easy in these kinds of levels? What, like, how do you
 find this balance in more challenging problems? Like, I don't know if you think further, yeah.
 So I guess in these problems, it's worth noting that for the block situation, the actual domain
 randomization process places the blocks one at a time. So all we're really doing is kind of saying
 you have a few more steps of that initial process. So it is fairly aligned with the whole problem
 there. And then in the Bipedal Walker setting, we're just making small changes to the encoding
 vector. And in both settings, we have these details of this in the appendix, if you dare to
 venture. But in both settings, we did sort of a sweep over the number of edits you can make in
 one go. And in both cases, we found that all the values worked well. We obviously picked the one
 that was the best performing on our validation sets. But it didn't, it seemed fairly robust to
 the number of edits you make. And the thing worth noting, again, there is that what you could do
 is if, for example, you don't care as much about the number of samples you use to find a high regret
 level, you could just try out, try all of these values in one batch. And then because with PLR
 based methods, you just curate the ones that high regret, you could say, okay, I'm going to do some
 with one edit, some with two, some with three, some with four, or whatever it might be. And you
 could almost scale the size of the edits. And then just from that batch, just take the high regret
 ones. And you're probably still going to have more new high regret levels than you would if you ran
 an example from the initial distribution. So I think that there is some flexibility to do something
 like that. And I would argue that you could frame a lot of things in this editing sort of framework.
 And I think we mentioned a couple of examples, like perturbing latent,
 latency in a generative model, for example, that may be seen as more general than specific
 encoding for environments. It is a good point. I want to stick on this a little bit the
 the types of problems where these methods are applicable, because they seem very general,
 yet it feels like you need a problem where you can construct such a curriculum. And that curriculum
 needs to be fairly smooth, let's say so that the difficulty increase needs to be manageable,
 and so on. And also, the regret, the way you calculate regret with the with the TD error,
 it means that probably an environment like the Walker, where I, you know, I get more reward,
 the further I go, is probably more conducive than something like a Montezuma's revenge,
 even though I have a TD error and so on that kind of smooths out the loss itself. Can you comment a
 little bit on what kind of problems would like, where would it start to struggle? Like, where
 would you probably have trouble applying something like this? And where would it work? Obviously,
 work super well on these types of things that you tried it on. But where would it struggle?
 Yeah, I think you're right. It's got to have it's got to be a domain where you do have some
 structure that progressively gets, you know, goes from simpler to more complex. And it's,
 I guess, one nice benefit of these methods is that you don't need to know ahead of time what
 exactly does it mean for a level in this domain to be hard, easy or hard, because we have this
 regret based heuristic to tell us that. And if you do have sort of this progressive structure
 within the domain, then these methods can sort of start to emerge that based on the statistic.
 But I think that at least with these PLR based methods, because the core is still
 needle in the haystack, you're looking for high regret levels by random search, and then evolution
 in Excel just massively augments that in terms of the amount of training data you can get from high
 regret levels. But the bottleneck step is still sort of like this limitation around at some point,
 you still have to just get that needle in the haystack. And so I think as the design space,
 like the dimensionality of your environment gets bigger and bigger, I would expect that
 these methods become less and less efficient. Do you?
 Yeah, a couple of... Oh, sorry.
 I think we have like a one second lag or so.
 All right, sorry. So I guess one other thing, one perspective of this is it's really just a black
 box optimization problem where the function returns regret. And so we've gone from random
 sampling to evolution. But if you look at black box optimization literature, there are plenty
 of methods that trade off between global and local optimization in a more elegant way. And so what
 you could do is have some model or approach that maybe samples points more like diversity in the
 space. And then you use something like Excel locally to make edits once you found that needle
 in the haystack that Minxie mentioned. And then the second thing is that I think one place where
 this might break down is because it is quite a kind of greedy local optimization process,
 is if you haven't got sort of a very clear, like high to low sort of environment, then maybe you
 need something to encourage diversity. So you need to maybe have some sort of like either buffer
 could be maybe like hierarchical or something, or you could try and preserve levels that you think
 are conducive to edits later on, even if they're not the current high regret levels. And these are
 all ideas we talked about future work. I think really what we need is we need to have these more
 challenging problems to actually break our current methods before we can really think of the hammer
 for these nails. But yeah. What is a bit special as well is that you train a single agent, right,
 because usually the evolutionary methods they are trying to get a population of agents to work,
 even if they want to end up with a single agent, very often. And you encode all of this into a
 single agent. And that's kind of a PPO really basic agent, if I want to say. And I have noticed a
 little bit that in these demonstrations, no matter what the level is, kind of the strategy
 tends to be the same, right? It tends to kind of, it tends to hop on this one leg with the other one
 with the other one out. And that is sort of the best strategy to overcome any and all obstacles.
 And then kind of rebalance itself once it's, yeah, this one, see? So, yeah, maybe we've been
 walking wrong our whole lives. But no, I mean, it's obvious if you instill this in a single agent,
 how much of a how much because I also observed some of your results here over time, which was
 also really cool to see when you compare it to the poet algorithm, in that you do get kind of
 more challenging levels later on, but they also, like, they don't dominate, it doesn't get more and
 more and more and more challenging, right? How much of this is a property of like catastrophic
 forgetting of the agent itself, where you kind of push for the more complicated levels, but all of
 a sudden, it can't can't solve the easy ones anymore. And therefore, the easy ones become high
 regret. And then there's kind of this, like how much of this is due to your algorithm? And how
 much of this is due to the fact that you have a single agent trained with PPO that needs to take
 care of all of these tasks at the same time?
 My guess is it's the latter part. Because I think that having this buffer that we do have, which
 in the robust PLR and the previous PLR paper, it does somewhat help with forgetting because
 you're able to sample things you haven't seen for a while. And if and if you now can't solve them as
 well, or or if you now have high regret in these levels, then you should retrain on them. So it
 should somewhat eliminate forgetting. But I do think it's worth noting that this agent is just a
 two hidden layer neural net policy. It's not not very flexible. It's pretty like low dimensional.
 And I think it really is unable to adapt to every different possible behavior. And so I think either
 having something where you can co evolve the architecture as well to maybe make it more
 flexible as the levels get harder, or even just making your agent be some sort of adaptive agent,
 like a meta learning algorithm, for example, that does zero shot adaptation. I think these
 approaches are things that we're excited about maybe for future work. But I think for this,
 it's sort of an inevitability that you try and have this like lofty goal of having a generally
 capable agent, it's going to have some brittleness to some certain components. I think we found a
 few cases like uphill, it's not particularly good.
 Yeah, when we started visualizing it in this viewer that we have in the demo, we noticed that,
 you know, like, when we were we're training this thing, all the complexity metrics for like
 roughness of the ground, it started going up very quickly. But then when we actually printed out a
 lot of the levels where it's successful, they tend to be levels, where it's all downhill, which means
 that this pogo stick strategy, it's very good at just like hopping down the hill, and it's really
 robust at landing, like just sticking the landing in terms of like really high clips. So it's
 really good for us. But when you start to get more like these rugged hills going uphill, where the
 slope is positive, that's where it starts to struggle. So that's like a really interesting and
 I think a very tangible sort of example, where there's sort of a collapse in diversity in a way
 in the curriculum where, because it is a limited, we do replay old levels, but again, it's a limited
 finite buffer. So you can get, you know, sort of like a buffer overflow in a sense of, you know,
 levels that collapse in terms of similar challenges. And then maybe the agent just gets too good at
 going downhill, jumping down really challenging hills, but then it starts to the curriculum
 starts to forget that also going uphill is also important. And maybe that's what happened in some
 of these training runs. I like the, I like the approach. I think poet or poet V2 had some sort
 of an approach where they do of course have different agents, but they had this metric of
 ranking the environments that they have in the buffer, right? And sort of ranking them with
 respect to different agents. And their conclusion was that if the different agents rank the
 environments in a different way, that kind of indicates a diversity of levels, right? Whereas
 if they rank them the same way, it's kind of like, well, they're not really diverse. I think much
 like your regret measure, I'm a big fan of these, they're not super domain independent, but they are
 domain independent enough, right? So that you could like, you can kind of disconnect them from
 the real problem at hand. That's pretty cool. That one is definitely, I think more general.
 I think that's quite an exciting approach. Maybe if you wanted to use population, maybe even generate
 experiences, then that's quite a nice way of evaluating the diversity, I think. So is it fair
 to say that kind of the end here, like the most, let's say you train this, let's assume this is
 convergence at 5,000 step, that this is kind of a representation, it's almost like a fingerprint
 of the agent's ability in the face of a curriculum that tries to push harder and harder, right?
 Because there's a trade off that the easy levels, not being in the buffer or not being,
 yeah, not being in the buffer means they're easy, they can be solved, right? But then also,
 yeah, this is, it seems like this is the curriculum that's needed for the agent to be as general as
 possible, not necessarily as good as possible. So yeah, I think it's worth noting as well that
 Minxie added a really cool feature to the website where you can actually see five seeds of each
 method. I don't know if you've seen that version, but you can see that the Excel agents are pretty
 remarkably similar. So they almost all seem to follow quite a similar gate, which makes me think
 that this is kind of the solution that for this network does cover the space as best as possible.
 And so it might be the case maybe that to get better behavior and better performance, maybe you
 need to have, there you go, show all seeds, maybe you need to have something that's a little bit
 more flexible, either something with memory, or I think some implementations like that of Walker
 use frame stacking, these types of things, maybe you can get more capacity into the network
 that way. And I think it's probably possible or likely that, there you go,
 it's probably quite likely that this is the best policy you can get with this network to have this
 Minx regret approach. Yeah, there is one survivor. Well, we'll see.
 Yeah, excellent. Cool. Yeah, the website is definitely pretty cool. The last interesting
 thing I found, at least for me here, was this generalization to the maze. And I mean, it's
 very cool because you train on these made up mazes starting from empty rooms, and then you test on
 these kind of human generated mazes right here, and then you generalize to this giant maze here.
 Now, you say yourself, the agent seems to follow this kind of bit of a left hand rule. How does
 something like this emerge? Because it doesn't seem like in the generated levels, a left hand rule
 would be beneficial because they're actually going to be more of a
 loop and stuff in that. How does a strategy like this emerge?
 I guess one thing that's quite worth noting in this environment is partially observable.
 So you only need to regenerate a small bit of structure within the grid for it to kind of
 generalize maybe to larger grids. But I think that's the thing that's more impressive about it.
 Yeah, exactly. And that actually makes this really hard, even for a human. If you imagine you didn't
 know where the green dot was and try and do this, as a 5,000...
 I think most humans would not be able to do this.
 I certainly lost patience with it after a couple of goes. There's like a 5,000 step limit, so it's
 quite long. But if you look at the Excel sort of towards the end of training as well, in the mini
 grid domain, a lot of the levels... So it ends up converging towards around 60 block count.
 And that's sort of like the threshold beyond which a lot of the levels where you randomly sample
 like more than 60 blocks, they tend to be unsolvable. So they tend to have a block preventing you from
 getting to the goal. And so 60 seems to be like the sweet spot for a 15 by 15 maze. And when you
 get to that set, like that amount of saturation, you're going to be able to do a lot of things.
 And when you get to that set, like that amount of saturation of blocks, a lot of the levels tend
 to actually become effectively single component mazes. And so those are unsolvable by the left
 hand rule. So I think that's also like just a contributing factor, like some property of
 the specific dimensionality that we looked at resulted in the complexity converging to
 lots of mazes that are single component. And it helps the agent basically learn this left hand rule.
 Yeah, it's pretty cool. Do you, I didn't dive too much into the experimental results in my review.
 Is there like, what are some of the things that you might want to highlight across your
 experimental results, maybe that you find more interesting than the average person would when
 they read the paper? I guess for me, it's two things. So the first one is that the complexity
 is entirely emergent. So we never encourage the agents to actually increase the block count.
 We never encourage it to increase the stump height and bipedal walker. It just has to do that to
 increase the grip. So some other papers maybe all works, maybe they have some like ways to encourage
 this, whereas we actually didn't. So if we were to do that, maybe in the future, that's could even
 increase it even further. And then the second thing is that all of the test cases are zero
 shot evaluations. So the agents never seen the test levels. And I think it's quite remarkable how
 robust it is in quite a wide range of settings. So that's probably the two takeaways for me.
 We also had some results in the appendix where we actually, we also test the final Excel bipedal
 walker agent on top of the poet levels. So in poet, actually, they publish a few of the rose plots
 showing the different parameter settings for bipedal walker for some of the crazier environments.
 And we actually tested bipedal walk, our bipedal walker with Excel on those environments. But it
 actually, it didn't perform very strongly. So it's what's interesting is I think what's interesting
 about this result is it sort of highlights this duality between like the goals of these two
 algorithms, where I kind of see Excel as being on one side of the spectrum, which is about robustness,
 general robustness to unknown environments, and poet beyond the other side of the spectrum, where
 it's focused on getting specialists for basically finding these agent environment specialist pairs,
 where this agent just always solves this environment. And so it's kind of an interesting
 philosophical idea, because it's kind of saying that if you're building an AI system, do you really
 care about being robust to things that you don't know about? Or do you want to maximize your
 performance as a specialist? And I think it's a really interesting open question. And the way
 we navigate this trade off, I think is really full of rich ideas for future research projects.
 Yeah, especially ideas that could combine some of these things as well. And we've obviously
 talked about a lot of possible things. But actually, if you go a little bit few pages down,
 what we did was we actually took the some of the most complex levels that poet generates,
 and then we produced them in our own setting. And that's also 100 by 100 maze, if you're interested.
 100 by 100. Did it solve it?
 Yeah, it has to be odd number for the for the simulators to work.
 Okay, okay.
 That one against the thing 8% success rate on that one. It's I think a bit above this.
 Is it table?
 Yeah. Higher up, higher up. Maybe.
 Do you want to check?
 What are you looking for?
 The poet.
 Yeah, it should be a small, it's like a very small table. I think it's down below.
 Search in the paper itself, I guess.
 We should have probably had paper up on our own screen.
 Well, my bad for for not knowing it too well.
 Oh, yeah, this is actually on the next page.
 This is the like main experiments on the next page.
 Ah, this is yes.
 Yeah, so one eight to three B are in the paper towards the end. They have like a rose plot for
 some of the most extremely challenging levels that each of their seeds generated. So for all three of
 their seeds, they pick two different levels that they're particularly high values. And we tested
 our agent zero shot on those. And yeah, the scores are pretty low. But I think the fact that they're
 above zero is cool. But at the same time, it does make you think that if they can solve those
 repeatedly, then maybe you do need specialists in some cases to get the most complex things.
 So some hybrid of specialists and generalists might be an even more powerful algorithm than either of
 them combined.
 Excellent. So you mentioned a bunch of different and you also have a future work section and so on.
 What do you think are apart from the things you're going to do next? What are like the big unsolved
 challenges in the field? Like what's what's everyone after but no one's been able to do it so far?
 Well, so the big one is a theme that we we as a group have gotten very interested in recently.
 And we're actually holding a workshop at iClear about this. And essentially, it's about
 Asian environment co-evolution. But in this in the context of this much older problem called
 open-endedness. And basically, open-endedness is an idea that it kind of came from a group of
 researchers, Ken Stanley, Joe Lehman, and Jeff Klun. And I think Jeff Klun has this concept of
 AI generating AI. And it's related to this idea of open-endedness where can you basically create
 a learning system that essentially ends up evolving just an unbounded amount of novelty and
 complexity. And if you can kickstart a process that achieves true open-endedness, then the idea
 is that maybe you can replicate the emergence of some really complex intelligences like human level
 intelligence. Because evolution like the tree of life, this is all sort of the result of an
 open-ended learning process. And so a lot of where we see this work going is that when we
 when we see our work is sort of fitting within this bigger theme of open-endedness, and this
 larger theme of agent environment co-evolution to achieve this open-endedness. And so I think that
 that's sort of to me is one of the most interesting open problems in AI or machine learning, or maybe
 it goes beyond even these two subjects. Yeah, so I think that if we can actually kick off a process
 like this, that would be incredible. And I'd be very curious to see what kinds of things fall out
 of it. Yeah, and for me, the thing I'm really excited about is that, again, tying in with
 Minchis is this seems like the only limitation to this really being open-ended is requirement
 for a simulator. So I'm really excited about whether we can actually learn simulators, for example,
 world models. So I was obviously very inspired by the Harnsh Riddhiever work from 2018. But more
 modern like offline RL world models. So maybe you have some transformer world model that learns from
 all this crazy amount of data. And then you can use that to design environments for an RL agent and
 then collect more data and just keep going. And maybe that's how you really get towards this true
 open-endedness, because you're not bounded by just the open AI environment that you're given.
 And so this is maybe it's a little bit more of a medium to long term goal, because I think we're
 a bit away from that right now. But I think that that could be where these different fields
 intersect and really produce something pretty crazy. Yeah. My issue a little bit with the
 agent environment coevolution work is that it just seems to shift the problem away from because,
 okay, we're evolving the environments right here, but they're still extremely bounded in an extremely
 parameterized space. And there's only these many ways that the environment can vary. And the true
 environment is kind of like the environment generator itself. And it seems like we could
 go a level higher and so on. But is there a method to generally break out of this being bound to any
 framework? I think one way is it's related to what Jack just described, which is this. So you've
 heard of sim to real as the paradigm, where you train intelligence in simulation, you transfer to
 reality. And that's obviously bounded by the fidelity of your simulator for your target domain.
 There's a new paradigm emerging. And it's like sort of pushed by all these advances in computer
 vision, which some people have called real to sim to real. And basically the idea that you can
 essentially collect data in a loop where you may have some exploratory agent, maybe it's a hand
 coded controller, or maybe it's an RL agent, the one you're training, and you send it out into the
 wild, it collects lots of data about what the world is like. And then you use that data to
 essentially enrich your simulator to basically fit your simulator to reality, to all the new things
 it's learned. And then you get a better, more expansive simulator, you train your agent again
 in that simulator, and you get a new agent to transfer to reality. And then this loop just
 keeps repeating. And maybe you can do this in a population of agents doing this. And you get
 really huge coverage in terms of what's out there. I think that's one promising way to do it. The
 other though, I think it kind of just generally the strategy is, like you said, all these simulators
 are bounded in terms of their parameterization. Like we are looking at 15 by 15 NASES. There's a
 finite number of them. I think what would be really cool is if we started as RL researchers,
 started focusing more on environments that are unbounded in parameterization. So moving into
 these like more almost non-parametric settings, where the environment can just keep growing
 arbitrarily in its number of parameters. And I actually think the real to sim to real loop is
 one way to do that, just because the space of possible worlds you can represent as a world
 model, as a neural network, is pretty much infinite. But maybe there are other simpler ways you can do
 this as initial toy tests as well. And then when you have that real sim to real world model,
 you can then train a mini max regret policy inside it. Yeah. Because then you have like this idea of
 the population generating this diverse, you know, very high dimensional world model, but then a
 single agent maybe that could be robust to any possible variation. And so this is maybe a bit of
 a medium term. But I think for us, it's kind of a North Star at the moment. Do you think there will
 ever be sorry, last question by me, do you think there will ever be this distinction between agent
 and environment? Will this continue to be an important distinction? Or is that something that
 you see in the future vanish and kind of almost become like, let's say interchangeable because
 people are already like pitting them against each other, training them both with RL and so on?
 Like, why do we even make the distinction? Well, I guess one thing that's interesting is even in
 the original world models paper, because the world model itself was generative model, the policy was
 very low dimensional, it just trained inside the latent state, latent space of the generative model.
 So then when you actually interacted with the real environment, you still use the encoder from the
 world model to process the input so that the policy can then operate. And so in that sense,
 it's like the world model is the environment at training time offline. But then at test time,
 when you go back to the real environment, the world model is used to process the inputs for
 the policy. And so they're kind of taking a very like, I guess, competitive and then a cooperative
 mindset. So I think maybe there's something like that, where you have world models that
 are your environment for training time, but then you use them as knowledge bases for test time.
 I think that's pretty exciting. And it also kind of relates to this idea of the cherry on top,
 because the policy is very small, although I hate to use too many cliches. But it does seem to relate
 to that sort of self supervised learning large world models, and then RL just for controllers
 inside that, that can operate on the representations. I don't know if I mentioned you things about that.
 Well, I think to sort of answer the other side of that question, I think that agent environment,
 I guess the distinction is, in some ways, it's arbitrary, because you can imagine, you know,
 like what part of this learning system actually belongs to the agent? Like, is the agent really
 like at the activation level? Is it at the observation level? Like, where do you even
 draw the boundary in terms of the agent? I think that's an interesting question. But I also think
 that at some point, there's going to be some substrate in which the agent has to operate within.
 And there seems to be, like, basically, if you wanted to emerge a diverse sort of, you know,
 a tree of life of different RL agents and environments, it seems like there is some
 sort of asymmetry there in the sense that agents have to operate within an environment, and you
 can't have it reversed. And so in some to some extent, I think we'll still have to have this
 distinction between agents and environments. But it's also possible, you know, like, maybe we could
 also just learn, you know, joint distributions over agents and environments, where you basically
 just learn, you know, like, the agents parameters themselves are now part of the environment design.
 And so now you're just emerging agents and environments together inside of a single
 generative model. I think that's an exciting idea. But and maybe at some point, we'll figure
 out how to do that. Where can people get started with this if they want to dive into it?
 So there's a great for open endedness, there's a great primer to it on O'Reilly, I can actually
 send you the link after, but it's written by some of the original sort of pioneers within this field.
 And essentially, it's quite long, but it summarizes the whole field. Another, another really
 interesting work would be, I think, just to check out the original mini max regret paper for RL,
 which is this emerging complexity for zero shot generalization from Michael Dennis and Natasha
 Jig, Jax. And I would definitely recommend, you know, our line of work with robust PLR checking
 out this paper. And there's older methods like teacher student curriculum learning from
 Shuman Shuman's group at OpenAI. And workshop. Yeah. So we're going to have an iClear workshop
 called agent learning in open endedness, alo. And that's going to feature a lot of speakers
 and researchers actively making progress in this field. So if people are really interested, they
 should attend some of the talks and check out the poster session that'll be that's April 29,
 April 29. Yeah, Friday. Good. Also, more in a multi agent setting, there's the curriculum
 learning manifesto from Joel, Joel Levo, and DeepMind. And that has some really nice ideas
 in terms of automatic curriculum learning, emerging, emerging complexity.
 Cool. Minchi and Jack, thank you very much for being here. This was really cool.
 Thank you for having us. It was very fun.