Author Interview - Improving Intrinsic Exploration with Language Abstractions
#reinforcementlearning #ai #explained

This is an interview with Jesse Mu, first author of the paper.
Original Paper Review: https://youtu.be/NeGJAUSQEJI

Exploration is one of the oldest challenges for Reinforcement Learning algorithms, with no clear solution to date. Especially in environments with sparse rewards, agents face significant challenges in deciding which parts of the environment to explore further. Providing intrinsic motivation in form of a pseudo-reward is sometimes used to overcome this challenge, but often relies on hand-crafted heuristics, and can lead to deceptive dead-ends. This paper proposes to use language descriptions of encountered states as a method of assessing novelty. In two procedurally generated environments, they demonstrate the usefulness of language, which is in itself highly concise and abstractive, which lends itself well for this task.

OUTLINE:
0:00 - Intro
0:55 - Paper Overview
4:30 - Aren't you just adding extra data?
9:35 - Why are you splitting up the AMIGo teacher?
13:10 - How do you train the grounding network?
16:05 - What about causally structured environments?
17:30 - Highlights of the experimental results
20:40 - Why is there so much variance?
22:55 - How much does it matter that we are testing in a video game?
27:00 - How does novelty interface with the goal specification?
30:20 - The fundamental problems of exploration
32:15 - Are these algorithms subject to catastrophic forgetting?
34:45 - What current models could bring language to other environments?
40:30 - What does it take in terms of hardware?
43:00 - What problems did you encounter during the project?
46:40 - Where do we go from here?

Paper: https://arxiv.org/abs/2202.08938

Abstract:
Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 45-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Authors: Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, Edward Grefenstette


Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is an interview with Jesse Mu, who is the first author of the paper improving intrinsic exploration with language abstractions. This paper is really cool because it combines the knowledge that is inherent in language with the problem of exploration in reinforcement learning. I've made a comprehensive review of this paper in the last video, so be sure to check that out. Today, Jesse has seen the video and we're able to dive right into the questions, criticisms and anything that came up during the video. The interview was super valuable to me. I learned a lot. I hope you do too. If you like, then please leave a like on the video. Tell me what you think in the comments. Tell me how I can make these videos better above all else. And I'll see you around. Bye bye. Hi, everyone. Today, I'm here with Jesse Mu, who is the first author of the paper improving intrinsic exploration with language abstractions, which is a really cool paper. I've enjoyed reading it. I like the bringing language into the reinforcement learning domain. I think it makes a lot of sense and I was very happy to see this paper. Yeah, Jesse, welcome to the channel. Yeah, thanks for having me. So I've presumably the viewers here have already seen my little review of the paper. What would be your maybe for people who haven't seen that or just in your words, your like short elevator pitch of the paper itself? What would that be? Yeah. So the way that I would pitch the paper is that reinforcement learning for a while now has wrestled with perhaps the central problem, which is how do we encourage exploration in these environments with more complex tasks and longer time horizons where the extrinsic reward that you get from the environment is very sparse. So in the absence of extrinsic rewards, how do we encourage agents to explore? And typically the way we do so is we assume and this is a very cognitively appealing intuition that we should motivate an agent to achieve novelty in the environment. We should make it do things that it hasn't done before, encounter states that it hasn't seen before, et cetera. And then hopefully we'll enable the agent to acquire the skills that we actually want the agent to acquire in the environment. But the problem with this, of course, is how we define novelty. In a lot of scenarios, there are environments that can look very different, but they have the same underlying semantics. So the example I have in the paper is like a kitchen and the appliances might be differently branded and differently colored, but ultimately every kitchen is a kitchen. And the way that you approach kitchens and the way that you operate in them is the same. And so the idea of this paper is we should be using natural language as the measure for how we describe states and how we describe actions within states and use kind of traditional approaches to exploration, reinforcement learning, but simply parameterize them with language rather than with state abstractions, which is usually the way in which exploration is done in these kinds of environments. And so what we do is we take existing state of the art exploration methods and then kind of see what happens when you swap in language as a component. And do you get better performance? And we showed that in a variety of settings, at least in the kinds of RL environments that people have been looking at in recent work, we do see again in using language to parameterize exploration rather than states. Yeah. I think it's very apt to describe it as you, it's not suggesting like a new exploration algorithm, but it's simply the re-parameterization in terms of language. And coincidentally, these environments, they do come with this kind of language annotations, which we do focus on. I like that. So I think what I really liked about this paper is just the research mindset in that any other paper or a lot of other papers, they would have done, they would have tried doing like three things at the same time. Like you know, we have a language generator and we do this and we do that. And what you're I think doing correctly from a standpoint of research is you keep pretty much everything constant, the algorithms constant, right? Even the environments, you assume that you have a perfect language oracle and you just add the language, which I really appreciate as like a reviewer, let's say. So I think this gets us right into our or my biggest, essentially criticism of the paper or what I called in that you add language to these algorithms, but you just said we swap in language. And to me, it felt more like it's not really a swapping in. It's more like you add language on top of what these algorithms are doing. And therefore, can't I just see your method as adding more data? Essentially, there is features that are available from the simulator, right, which the other methods just don't use, they just discard this part and you just add this part. Do you have an indication in how much of your effect is really due to language and how much of the effect is just due to the fact that you have more data available? Yeah, that's a great question. And it's definitely a point that I think a lot of people will fairly make against the paper is, yeah, we're using extra data, right? And yeah, I think my verb swap was maybe only accurate in half of this paper, which is that in Amigo, which is the first method that we look at, it really is a swap, right? So if you read the paper, the traditional kind of Amigo teacher network proposes coordinates X, Y positions as goals. And here we're just completely eliminating that kind of goal specification and we're moving towards language. So that can be seen as more of a swap. Although of course, in novelty, which is the second method that we look at, that is definitely more of kind of an addition, as you say, because we keep the extrinsic bonus and we do have experiments that measure what happens if you don't have novelty by itself. You only have the kind of language novelty bonus and it doesn't do as well. So you're right that I would say that we explore this idea of swapping in language in a bit of the paper, but there are points where it's more of kind of a bolt on and we're not like super clearly looking at or distinguishing when is it okay to have language just be a complete drop in replacement versus just some additional information. So yeah, I think we're showing that in general, if you're trying to add language into these environments, you're seeing a gain, but how precisely that gain manifests is still a little requires some more exploration for sure. So I guess more generally to your comment on using extra data. Yeah, I mean, I think we have some intuition that this data should help, right? It's a fairly clean linguistic signal, but how to use this data concretely is an open question, right? And so that's kind of where I view the contribution of this paper as even though we have some intuition that adding extra data will help, we actually need the equations written down, right? And here are two concrete ways in which we can operationalize this data for the purposes of actually getting better performance in your environment. And there are a lot of examples of this in machine learning, right? So like you have some large language model, for example, and then you want to fine tune it for some domain or you want to fine tune it on human preferences. I mean, that's fundamentally, you're adding extra data for the purposes of getting something that works well on a task that you care about, right? And how to use that data is the open question. The other point that I would say is that we have some deep seated intuition that this language should help. As you say, it's really high quality. It comes from an Oracle. It comes from the game engine. But we actually still need to get that kind of empirical verification that it works, right? And there's actually a lot of reasons why maybe these experiments might not have worked out. For example, the language is Oracle generated, as I mentioned, but it is also very noisy. So as I described in kind of the method section of the paper, most of the messages that you see in the environments are actually not necessary to complete the extrinsic task. And I kind of exhaustively show which of the messages do matter. And so it could be the case that, well, the language signal, at least in these environments, is too noisy. The state abstraction captures all of the factors of variation that you might care about in an environment. And so you don't ultimately need language, right? And that's an imperial question that we have to measure. And so I view this paper as providing that empirical verification, which in hindsight, I think, is a fairly straightforward intuition. It's something that I definitely thought would happen. But yeah, it's nice to see those results kind of in writing. Yes, it's easy. I think you're right. It's easy to look back and say, of course, like, well, all you do is you do this. But exploration has been since since, you know, people have thought about reinforcement learning, they've obviously thought about exploration methods and intrinsic rewards are like as old as Schmidhuber himself. And we you know, the fact is that, you know, new things are developed. And this is at least one of the first things into into really the direction of incorporating. There have been incorporation of languages before, but a systematic adding it to the state of the art methods. And it seems like I am I am convinced the method at least the El Amigo method is quite well outlined, I think, in these diagrams, the contrast of the left being the original Amigo and the right side being the language Amigo. A question I had right here is that on the left side, you have this teacher network, and it simply outputs a coordinate to reach and it has to pay attention to the fact that the coordinate is not too hard and not too easy, right? Therefore, it has to learn that too easy coordinate. Yes, one that is, you know, close, but also it has to learn maybe unreachable coordinates or coordinates that are inside the walls, right? They can't be reached or something like this. However, on the right side in the language, I mean, you seem to split these two tasks out into one network that that determines which goals can even be reached and one that then orders them essentially, why? Why are you doing this? Like what's the is there a particular reason behind why one network couldn't do both at the same time? Yeah, so the reason why we split the Amigo network up into two parts, and as you say, we don't have to do this. And there are ablation studies in the appendix that shows what happens if you get rid of the grounding and you just have a single network predicting both goal achievability and, you know, actual the actual goal that's seen by the students. So it kind of a goal difficulty network. It does find in some environments, especially in mini hack, but it doesn't do as well in other environments such as mini grid. And part of the reason, as you've described, is that at least in these environments, the coordinate space stays consistent across episodes. And so you're right that there are some coordinates that are perhaps unreachable in certain environments and not in others, but there's much less variation than the set of language goals that are achievable in an environment because the environment will have different colored doors, for example. And so the goal go to the red door only makes sense in, let's say, half of your environments. So it's possible for the teacher to the Alamigo teacher to hopefully learn this distinction kind of just through, you know, the policy gradient method. So basically just like Amigo, but this is relatively sample inefficient because the problem is that when you propose a goal that's simply impossible in the environment and you get negative reward, that negative reward only comes after the student has tried to complete the goal for, let's say, a few hundred steps. Right. And so it's a relatively sample of inefficient way of telling the teacher, hey, the student did not achieve this goal in the environment. And moreover, that negative reward, you know, there's two possible sources of that reward. Right. So if the student never completed the goal, is it the case that it was just too difficult for the student, but it is achievable in practice? Or is it that the goal is simply never achievable in the first place in the environment? Right. And those kind of two failure cases are a little bit hard to distinguish. Whereas we have kind of this more frequent source of supervision, which is simply, you know, as the student is randomly exploring in the environment, it's encountering a lot of goals, a lot of messages because we have a language annotator and we're kind of, you know, if we if we kind of ignore that signal, that seems like something that we should be using. And so we have kind of this dual thing where we have a grounding number, which is updated more frequently in the environment, which is updated from the messages that are seen by the students. And then finally, the policy network, which is actually trained to satisfy the kind of difficulty objective and actually get the student to complete goals in the environment. Can you go a little bit more into because that was, I think, the only part that confused me a little bit, which is the how exactly you train this grounding network. There is a there is this this notion of whatever the first language description encountered along a trajectory being sort of the positive sample and then the rest being the negative samples. And that kind of confused me because it means the negative samples would also include goals that were encountered just not as the first message. Could you maybe clarify maybe I didn't understand something right? Or maybe I don't, you know, see the reasoning behind this exact choice. Yeah. So I think your intuition is correct. I think you've described it correctly. It is kind of a weird thing to do, which is that we are treating negative samples as basically all of the goals besides the first one that was achieved. Right. And of course, that is incorrectly treating negative samples of goals that were achieved later. Right. So negative samples are noisily generated, as I as I say, in the limit, this noise should even out, though. So you can compare, you know, like we're just kind of noisy, noisily generating negative samples here. We can compare that to maybe a setting where we had a more oracle sense of when a goal is truly infeasible in an environment. Right. And so what happens is, you know, just in general, a goal is going to appear in this negative sample term more and more often as we train the network. But because it's we're kind of, you know, downweighing all possible goals in the space, the idea is that hopefully, you know, this noise of of class of incorrectly classifying a goal is unachievable in an environment kind of evens out over time. Right. And so, yeah, it's a little bit tricky because we don't have the oracle saying, oh, you can't achieve this goal in an environment. Right. We only know that. Well, you know, the student just didn't happen to achieve the goal in this environment. So I could imagine other ways in which you try to come up with some heuristic that better captures this idea of kind of unachievability. But this is what we came up with, which seems to work reasonably well in practice. And alternative way that you can interpret this is we're not really measuring true achievability. Like, you know, is this at all possible in an environment? What we're really trying to have the grounding network capture here is what are the goals that the student tends to reach? So like are feasible at the current state of training, right? The current policy, what goals can it reach? And that's really what we need, right, is we need like to propose goals that at least for now are eventually reachable by a student. And that doesn't mean that it's, you know, unachievable in all possible students under all possible environments, but at least just for current, you know, in the current stage of the training process, it's a reasonable target. I can imagine that this gets very, that this may require an adjustment or that this breaks down in environments that are more causally structured. For example, if I always have to go through the green door before I reach the red door, right, then the goal would always be in any trajectory that I do, the green door would always be the first goal. And therefore my grounding network would never recognize the red door as a reachable goal, because that's always going to be at least the second goal, right? So I guess depending on the environment, it's not hard to make a change to this, obviously, in that case, but I guess that's one thing that might have to adjust a little bit to the environment at hand. Yeah, that's a that's a great point is that we do not. There are settings where you might just, you know, want to run it without the grounding network. And obviously, that's actually a simpler version. So it should be fairly easy to experiment with that. And also, in the setting that you described, what will happen is, like you say, you know, the green the go to the green door goal will get a lot of weight, but hopefully can be counteracted to some degree by the policy network, which will, you know, learn to not put any weight on that once it realizes that it's getting absolutely zero reward for that setting. But I agree that this kind of introduces some weird training dynamics that we don't really want might be cleaner just to remove the grounding network entirely. If you as as you say, you've looked at my paper review a little bit, I didn't go too much into the experimental results as such. Is there also I didn't go into the appendix at all, because honestly, I haven't read the appendix because I sometimes I don't I think I should probably. But is there anything that you want to highlight specifically about the experimental results or or maybe something that you did in the expand appendix, which is also has a lot of experiments in it? Things that you think people should take away from the paper from the experiment section? Yeah, so broad takeaways are and I think that you mentioned this in the review is, you know, we're in these kind of DRL environments and and the individual training runs are just incredibly noisy, you know, and that can be sometimes like rather difficult to get a sense of, oh, is my method actually working better than others? Right. But there has been some great recent work from I think a team at Miele, which won an outstanding paper award at New York's last year, which was called deep reinforcement learning on the edge of the statistical precipice. And the basic idea is, you know, we're compute constrained. We have these environments, they're very high variance. But even despite all of this, you know, what are the kind of statistical best principles that we can follow to really see whether or not our methods are actually making a measurable and replicable difference in the environments that we're testing? And so they have a lot of good recommendations, which we try to subscribe to as close as possible in this setting. Right. So these training curves here give you kind of a qualitative sense about not only kind of the ultimate performance attained by any of the models, but also of the differences in sample efficiency that we see. Right. So it could be the case that, well, ultimately, both Amigo and El Amigo reach the same asymptotic performance, but Amigo just gets there faster or more reliably. And that's something that you can, sorry, El Amigo gets there faster and more reliably. And that's something that you can look at in these graphs. But I think the more kind of statistically rigorous way of verifying that language is giving a gain in the environments is in the subsequent figure, which is figure four, which should be right below this one, I think. And this is really, you know, us trying to statistically verify, you know, is there an effect happening here? And so these here are bootstrap confidence intervals, five runs in each experimental condition. And we're plotting the 95 percent confidence intervals for the interquartile mean of models across tasks. So this is kind of like the mean performance, assuming that you drop some of the outliers, because again, these runs are very high variance. Right. And so this is kind of a statistical recommendation from the authors of that deep RL paper. And we show that, yes, the individual runs here have really high variance naturally. But as you begin to look at the runs in aggregate across both the mini grid and mini hack environment suites, we begin to see a trend that it's clear that, you know, overall we're seeing a good effect of language in these environments. And so this is obviously these are aggregate metrics, overall metrics and so on. When we look at the plots themselves, there is quite considerable variance, even in the ranks of the method. Do you have an intuition of between the language methods, which works better in what kind of environments and in what kind of environments does language even maybe hurt? And why do you have an idea? Yeah. So the trend that I try to highlight in the paper is that in larger environments, language exploration does better. And the reason why you might expect this is that in larger environments, Amigo and Novelty kind of suffer from this problem of increased noise. Right. There's a lot more coordinates, for example, that you can propose, which essentially describe kind of the same semantic action. Right. You have like you want to get the agent into one room of this maze. And you know, because the environment is larger, now there are four or five different coordinates that all kind of mean the same thing. Whereas as you increase the size of the environment, the language set, the set of language goals is relatively more consistent. Right. It's kind of one of those complexity analyses. Right. It's like kind of space complexity, almost of the goal space. And so you can see this trend happen a bit. For example, in the Wand of Death task, so WOD, this is in the top right corner here. We have WOD medium and WOD hard, where in WOD medium, Amigo actually outperforms El Amigo. So it gets you to higher performance quicker. Whereas in WOD Wand of Death hard, Amigo is actually not able to learn at all. And the only difference between these environments, it's fundamentally the same task. But the only difference is that in WOD hard, the room is a lot bigger. So instead of a narrow corridor, you actually have to search for the Wand of Death, that's the task, in some in some room beforehand. And you can see that just simply increasing the size of the possible coordinate spaces results in both traditional novelty and traditional Amigo doing much worse in this environment. And I think that kind of shows that these kind of state based exploration methods are very brittle to the size of your state base. Right. So you can kind of increase your state space infinitely and it'll make these methods perform worse, even if the underlying semantics of your environment haven't changed yet. Do you have an idea, do you have a feeling maybe, if this is a property of the world in general, like let's say I as a human, right? I'm put into a small whatever environment or a big environment, would my descriptions of language also not grow very much? Or is it a property of just game developers? You know, I add a few extra rooms, I can reuse these languages, you know, I just kind of tile, you know, the other the big games, I mean, the biggest games are procedurally generated like Minecraft there, it's really, it's just the same thing over and over. But even in like the like, these big open world games, like Grand Theft Auto or so, the same textures are reused and the same cars and the same NPC characters, right? Is this a property of the world or of the video game developers? Yeah, so this is a really deep and almost philosophical question. Yeah, is something that I think about a lot is you can certainly and this is a totally valid statement, right, you can say, well, there are a lot of language actions that you can describe in our world and even in the video game world, which just described these like kind of infinitely complex and nested sequences of actions, which have absolutely nothing to do with the extrinsic task, right? I could tell you to, you know, oh, you know, run at the wall six times do a 360. And then, you know, continue hitting the wall eight times, right. And that's like an incredibly difficult goal, which you can imagine a very structured curriculum to get to that point, right, of just like infinitely kind of bumping your head against the wall, which satisfies, you know, maybe the difficulty threshold of El Amigo, but is absolutely orthogonal to the task that we care about. And I can imagine that there are settings where the language is kind of useless and doesn't end up, you know, giving you any gains in this setting. And so there's kind of this open question that we haven't really touched on sufficiently in this paper, which is how good does the language have to be in order to get this to work? So as I say, you know, the language is Oracle, it's game developers, but it also is noisy. There's a lot of actions like running into walls or trying to throw stones at a minotaur that are ultimately useless in the environment. The argument we're making here is that hopefully, you know, the noisiness of language scales a little bit less than the noisiness of your state environment, right. But there's still a lot of kind of edge cases and kind of unexplored territory here. I think more philosophically, if you think about our world and our environment, right, there are a lot of ways that we can describe actions that are not particularly useful in the world that you and I inhabit, right. I mean, I can again tell you to do handstands and hit a wall and, you know, walk around and write endless, you know, trivial things in the dust. But at the same time, there's a lot of our action space in the real world that we simply don't have language descriptions for, right. So like every single precise movement on my hand and my arm, you know, I could presumably come up with some language to describe, oh, I'm actuating this joint, you know, by 0.03 degrees. And there's like, you know, how many joints in my hand, right. I mean, there's like endless complexity in terms of the possible action space just by moving a hand that in language we have absolutely no words for, right. And so it's really it's a really tough question, right. Like we have a lot of kind of ways of describing useless actions in the world. But at the same time, it's very clear that the language that we do use to describe the world is operating at a higher level abstraction than perhaps the kinds of actions that RL agents have access to, right. And for example, actuating some sort of limb or something. You make a you make a good point that in the paper that language is a strong prior over what is essentially important to humans, right. If I can describe something with a short piece of language, like, of course, I can say do three backflips and then, you know, do eight of that and so on. But it's a fairly complex sentence in itself. If I can describe something with a short piece of language, usually that is something that matters to some human somewhere, right. Otherwise that wouldn't be mapped to a short string. But that brings me a bit to a different question. And that is the question of isn't isn't the I think in these environments, there's always a goal, right. There is one reward at the end that you need to reach. I can imagine, though, that novelty or not novelty in general or how how important a state is, is really dependent on your goal. Whether I circumvent the minotaur at the, you know, below or above that might not be important if I want to reach whatever the goal behind it. But it is really important maybe for a different task. It's likewise I as a human, whether I move from here to there by walking forward or backward doesn't matter if I want to get to the fridge. But it matters really if I'm if I'm dancing, right. So is that something that like how does that interplay here with these with these language things? What do you do when a language it almost like needs to incorporate a piece of the goal that you want to reach in order to be useful or not? Yeah, so I think thinking about or trying to filter the language descriptions that you have to language that is relevant for your task is going to be important if we scale this up to environments where it's clear that using unfiltered language is not helping. Right. And again, as I mentioned, the robustness of these kinds of exploration methods to the noisiness or relevance of your language signal is still an open question. If we do have task descriptions, so we have extrinsic task descriptions like your job is to defeat the Minotaur, then it's really intuitive that we should be able to use that as a signal for kind of waiting how relevant a sub goal or language description that we encounter waiting how useful that is for the extrinsic task. Right. So if the extrinsic goal is combat, then we should be prioritizing combat related messages. If the extrinsic goal is buying something, then we should promote acquiring money and things like that. And so that's something that I think is a kind of natural extension of this is you extend this to a multitask setting where you have task descriptions and the task descriptions ought to kind of heavily filter what sub goals should be relevant for the task. I think when you include task descriptions, there are some more comparisons to related work. There's some related work, which you mentioned the paper where let's imagine you're doing basically hierarchical reinforcement learning. So you have some extrinsic goal and then you want to explicitly decompose the extrinsic goal into sub goals that you want to complete in order. Right. And that's those are certainly kind of relevant methods to look at when you start thinking about multitask or goal condition settings. But this is kind of a slightly different focus where we're not trying to identify sub goals that need to be completed on the way to some extrinsic goal. There's still kind of this exploration component, which is a bit of a different use of language than this kind of hierarchical stuff. But certainly I would say that there are people who have looked at kind of language conditioned RL and hierarchical RL that think a lot and very deeply about this problem of proposing sub goals that are relevant for the extrinsic goal, assuming you have some structured description of what the extrinsic goal is. Although I can imagine you run into sort of the, let's say the more abstract problem of the exploration problem is that, you know, without an outside signal, I don't really know what to do. And there is no clear, let's say gradient towards the goal. Right. Otherwise, the exploration problem in RL would be relatively easy. Now when we say, well, we'll just filter out all the messages that don't have anything to do with our combat goal. Right. So it is like we could run into the exact same thing again, where, you know, maybe in order to acquire a weapon, I first need money, right? That doesn't, that's not directly related to my combat goal. So there is like another exploration problem again, on top of the thing we introduce. I guess maybe we can hope that if we introduce enough levels, the highest level of abstraction will have a small number of states so that, you know, random exploration works. But it's kind of funny that the problems repeat or replicate. Yeah. Yeah. It's really tricky. And that's essentially just kind of a deeper or more nested failure case of not knowing what's novel and not knowing what's relevant for your goal. Right. So if you're prioritizing words that have combat in them because your extrinsic goal is combat, but you first need to buy something, then your, your, your semantics, you know, your measure of novelty or relevance is just not good enough. Right. So that's going to just be a fundamental problem in exploration is how do we know whether it's states or language, you know, how do we know when a state is relevant for the ultimate task? Yeah. And I guess humans aren't very much different, right? I mean, science is a really hard process. It's not, you know, that exploration takes millions of humans and hundreds of years. So we can't fault our RL agents here for not, not doing that great of a job. Here, I found these plots to be really cool, like the analysis, sort of the evolution of what the teachers propose. And of course, these being language, it's quite insightful and understandable what's happening in the algorithm. My, my surprise was a little bit, aren't these things kind of subject to like catastrophic forgetting or things like this? I can imagine, right? If I train these things online and they're at some difficulty level, all of a sudden they forget that reaching the red door is kind of really easy. Or so is that have you ever thought is that a problem? Or was that ever a problem? Did you encounter that? Or why don't we encounter that? Yeah. So I expect that that is a problem that happens in these agents. I don't think we really precisely tried to measure whether or not catastrophic forgetting is a problem. I think the fact is that we evaluate in environments where we are not testing the agents kind of continuously for mastery of all of the skills that it has learned in its curriculum proposed by the teacher. And so this problem of, oh, you know, you forgot how to specifically open a specific color door is not an issue as long as the student is still quite good at completing whatever goals it needs to complete to try to achieve the extrinsic goal that is currently being set by the teacher. Right. So if you forget things that are at the very beginning of training, that's not a big deal. So long as whatever path that the teacher is leading you on is something that will eventually get you to the extrinsic goal that we care about. And I think that happens to be the case in these environments because there was only one extrinsic goal and because we're not testing it to master every single skill from kind of low level to high level abstractions. But if we were in a setting where being able to complete those lower level goals kind of, you know, on a dime and kind of, you know, switch kind of do context switching like that, if that were more important, then we would have to deal with this problem of catastrophic forgetting. Right. An important point here is that we really don't care about how well the student is able to follow instructions proposed by the teacher. That's, I mean, we hope the goal is that that property emerges such that we can complete the extrinsic goal. Right. But we're never actually trying to learn a student that can follow instructions. We never really evaluated exclusively in an instruction following setting. Because if we think ahead a little bit, and I'm going to want to just scroll down to the environments just because, yeah, maybe this this will inspire us a little bit. If we think ahead a little bit beyond this work, here you have this very, this Oracle language descriptor. And you say also in the outlook of future work that that is something obviously that we're trying to get rid of because not every environment, like the fewest of environments actually have such a built in language description or easily accessible one. So we might have to regress to something else. So I want to I want to think about three different external models that we could bring in. And I wonder what you think of each of them, like how these could fit in. The first would be something like GPT-3, like just a pure language model. How could that help us? Maybe in combination with these things, because we need some starting point, right? But how could a pre-trained language model that knows something about the world help us? Then something like CLIP, maybe something that can take an image and language and say whether they're good together or not. And then maybe even something like or maybe a captioning model. Right. And maybe something like DALEE, like something that takes language and generates. Is there in this cloud of models, what possibilities do we have to bring in sort of to replace this Oracle thing with with learned systems? It doesn't even need to be learned online, right? It can be pre-trained. I'm probably much more excited about that. Yeah. Yeah, these are, I think, going to be the most fun questions to look at in kind of language conditions are all going forward is taking the boom in pre-trained models in large language models and resulting, you know, bringing these into concrete and actionable gains in reinforcement learning. It's funny that you mentioned this kind of what I described as almost a gradation from ungrounded language models like GPT-3, right, which are trained on text only corpora and whether those can actually help in these environments, which I would call are fundamentally grounded, right? They're grounded in some some visual or perceptual world. And ungrounded language models still result in gains in these settings. And my intuition is, yeah, they probably still can because, you know, even if you don't exactly know what it means to acquire a wand or kill a minotaur in some environment because you don't know what a minotaur looks like or what a wand looks like, GPT, as I mentioned, you know, this idea of priors, right? GPT has strong priors on sensible sequences of actions, right? So insofar as these environments are testing kind of sequences of actions that humans kind of have an intuition for, you know, it's some fantasy world, but we have some intuition, oh, in order to defeat the minotaur, we need to get a weapon first. We probably look around for a weapon. Maybe there's a shop. Maybe we can buy a weapon from the shop, right? Video games are testing knowledge that we have very like deep seated commonsense knowledge that we have that hopefully generalizes to these fantasy worlds. And GPT certainly contains a lot of that information, right? So you might imagine we should reward or filter the kinds of descriptions that we see to those that seem sensible narratives that GPT-3 would generate, right? So a sensible sequence of actions along the way to defeating the minotaur is collecting a wand and buying it and things like that. And I think you actually already see some examples of this happening in more goal conditioned or instruction following RL. So there's been some recent work from, I know, teams at Berkeley, maybe Google as well, that are looking at using pre-trained language models, which are not necessarily even grounded. They're just, you know, GPT-3, using them to construct sensible plans, action plans or sub goals for completing certain actions. So in some home environment, for example, maybe my action is get a cup of coffee. And then the goal of GPT is even though I don't really know what my environment looks like, I don't know what kitchen you're in, I know that sensibly this should include finding a mug and then heating up the kettle and things like that. And so we already see some promising use of kind of ungrounded models for improving grounded decision making settings. Yeah, did you want to comment on that? Or I can also- No, no, that's cool. I think, yeah, I think I've even had at least one of these works here on the channel in this home environment. That's exactly, I was also really cool to see. Obviously, these models know a lot about the world, right? And I think people overestimate how or underestimate maybe, well, whatever. That the thing, if we humans look at a board like this, like at a mini hack board, we see a map, right? We see paths to walk on and stuff like this, even if we've never played a video game. But this is, these are such strong priors built into us. And we sometimes think like, why can't that dumb computer just like walk around the wall, right? And we're like, what's up? And I think these large models are a way we can really get that knowledge from the human world into this world. So yeah, I think that's, it's a great outlook. Also with the models that combine images and text, I feel that could be really like adding a lot of value to the RL world. At least the RL environments that are like human environments. Of course, there's reinforcement learning for computer chip design, and things like this. I don't think those are necessarily going to be profiting that much from it. But yeah, yeah, really cool is so you're you're at Stanford? Or did you do the work at Stanford? Or were you at some internship? Yeah, I did it while I had an internship last fall. So this is fall 2021. Okay, continue to work a little bit while at Stanford. But it was mostly in collaboration with some people at fair or meta, I guess now in London. Reinforcement learning is notoriously also kind of hardware intensive. Although this work right here seems like maybe not that much because you describe a little bit sort of what what it takes to investigate a project like this. Yeah, unfortunately, I think even for these environments, it's fairly hardware intensive, certainly still feasible, I think, on let's say, a more academically sized compute budget. But for being able to run the experimentation needed to iterate quickly, you know, you do really definitely benefit from kind of industry level scale, which is one of the unfortunate things about this kind of research is that it is a little bit less accessible to people in smaller compute settings. So maybe the typical kind of RL environments you think of our compute heavy are the ones that are in 3D simulation, you know, very, you know, need physics, need soft joint contact and all of these things to model. And those are really expensive. I think compared to that, these are kind of more symbolic grid worlds. You know, the whole point as to why mini hack or net hack was chosen as a reinforcement learning test bed was because the code base is, you know, written entirely in C and is very optimized, and so you can run simulations very quickly on modern hardware. But that being said, it's still relatively compute expensive. Again, the just amount of experience needed by state of the art, deep RL methods, even with extrinsic or intrinsic exploration bonuses is still very expensive, right? So for example, one of these runs, we would typically have, let's say, 40 CPU actors collecting experience at the same time in parallel, and then kind of one or two GPU learner threads in the background kind of updating from this experience. So even just a single, you know, computational experiment here needs non trivial hardware for sure. Yeah. And, and you ideally you want to do that in parallel, right? Because you want to try out a bunch of things are repeated a bunch of times because one experiment really tells you almost nothing, right? Unless it succeeds, right? If it succeeds, it's good. But if it fails, you never know if you repeat it a bunch of times. Yeah, but I mean, it's still it's not it's not the most extreme thing, right? Like two GPUs or so and a bunch of CPUs. As you say, that can that's still academically doable, which I find cool. Could you maybe tell us a bit about the process of researching of researching this? Like, did everything work out as planned from the beginning? Or where was your starting point? And what changed about your plan during the research, like maybe something didn't work out or so? Yeah. Yeah, I feel I don't I feel it's always good for people to hear that other people encounter problems and how they get around problems. Yeah. Yeah. So yeah, it's a great question. The intuition that I think me and my collaborators started with was, you know, fairly sensible. It's language is clearly going to help in these environments. You know, it has some nice parallels to human exploration. And so let's just see whether or not language will work in these environments. What's funny, though, is that we actually started out the project less about the more abstract question of like, does language help exploration and more a very concrete question of how do we improve upon Amigo? So how do we improve upon an existing state of the art algorithm for exploration? Let's propose something that we argue is better than everything. It's like we're going to propose a state of the art exploration method called El Amigo, which will get 100 percent accuracy in all these environments. And none of the existing methods will work. Right. That's that's kind of the narrative that you set up for yourself when you're starting research is I'm going to build something that's new and that's the best. Right. However, I think the focus of this paper and the story has shifted considerably. I think it's shifted for the better, actually. And part of this shift happened because we implemented El Amigo and it was working fine and it worked better than Amigo. So we were quite excited. But at the same time, the field is moving so fast. And at NeurIPS last year, some researchers came out with this method called novelty and we ran novelty and novelty also did really well. And you know, in some environments, it totally like blew Amigo out of the water. Right. And El Amigo. And part of our thinking was, well, OK, now we can't really say, oh, we have El Amigo and it's the best model. It's the best environment. And you should only use this. And at first I thought, you know, this is derailing our narrative. Right. We're not proposing anything new. We're not proposing anything state of the art. So what's the point? But I think after some kind of juggling and shuffling, we realized that what we're really interested in is the scientific question of does language help exploration? So take existing method X and then do X plus language. Right. And so this question can be answered kind of agnostic to the specific method that we actually use. Right. And so it was that juncture where we actually decided, OK, let's actually look at novelty closely and let's imagine adding language to novelty as well. And do we see the same kind of results? Right. And so I think this is kind of an outcome of the paper that was kind of on the fly changed. But I'm very happy with which is that we're not trying to claim that we have a method that is state of the art or that is best or that anyone should be using our method. We are very agnostic to the particular choice of method. Right. We're trying to answer kind of a more abstract question, which is when does language help exploration? And I think this is a little bit more egalitarian. We're not saying that our method is better than anyone else's. And we also don't have to exhaustively compare to like a lot of existing work. We're just saying that if you take whatever method that we have and you add language, you do better and here are two examples where that happens. Cool. And it is a good way to preempt some reviewers from saying that you didn't train on ImageNet and that's bad. Yeah. Is there anything else that you want to get out to viewers? Maybe a way they can get started if that's possible or anything that you'd like them to know? Yeah, I think that we've discussed a lot about these kind of higher level ideas of one holy grail is that we have clip generating descriptions or open GPT-3 and then we're evaluating in these really high dimensional spaces with actual motor joints and we're going to show how language helps in these like mojoco style, like really deep RL, realistic environments and maybe you can transfer to the real world. I think that's the broad vision but I think it is still very far away. I think we even in this paper abstracted away a lot of difficulty of the problem. We're assuming that we have Oracle language annotations. We're only looking at these kind of symbolic grid worlds and although it's tempting to dive in and say, okay, now let's kind of straightforwardly let's extend this to a real world environment where I have to actually move my coffee mug to make coffee and tea, I think we're still quite far away from that broad vision of kind of household enabled robots in RL and is probably not the most I think like beginner friendly way of starting. There's just so many deep problems that need to be solved jointly from perception to action to planning and before we even consider how we better incorporate language into the mix. And so I think the way to build upon this work is just these kind of very small progressive relaxations of the assumptions that I and many of the other people who have worked in this space have. Right. So again, let's imagine let's just imagine we get rid of the Oracle language annotator and we train a model to emit states for these simple environments. You know, we didn't really explore that, but that's a very sensible way to extend this kind of work while keeping the environment and the models fixed. Right. So this goes back to the very beginning when you mentioned the kind of way in which we approach this paper was to keep everything fixed and then just look at this kind of very small change and see how that results in different performance in our environment. I think that's really just kind of the way to go. It's very slow. It's very incremental work, but hopefully it's getting us more towards that kind of guiding star of eventually having these models that operate in these realistic environments and use pre-trained model language to help exploration. Cool. Jesse, thank you very much for being here. This was awesome. Thanks. Have a lot of fun.
 Hello, this is an interview with Jesse Mu, who is the first author of the paper improving
 intrinsic exploration with language abstractions.
 This paper is really cool because it combines the knowledge that is inherent in language
 with the problem of exploration in reinforcement learning.
 I've made a comprehensive review of this paper in the last video, so be sure to check that
 out.
 Today, Jesse has seen the video and we're able to dive right into the questions, criticisms
 and anything that came up during the video.
 The interview was super valuable to me.
 I learned a lot.
 I hope you do too.
 If you like, then please leave a like on the video.
 Tell me what you think in the comments.
 Tell me how I can make these videos better above all else.
 And I'll see you around.
 Bye bye.
 Hi, everyone.
 Today, I'm here with Jesse Mu, who is the first author of the paper improving intrinsic
 exploration with language abstractions, which is a really cool paper.
 I've enjoyed reading it.
 I like the bringing language into the reinforcement learning domain.
 I think it makes a lot of sense and I was very happy to see this paper.
 Yeah, Jesse, welcome to the channel.
 Yeah, thanks for having me.
 So I've presumably the viewers here have already seen my little review of the paper.
 What would be your maybe for people who haven't seen that or just in your words, your like
 short elevator pitch of the paper itself?
 What would that be?
 Yeah.
 So the way that I would pitch the paper is that reinforcement learning for a while now
 has wrestled with perhaps the central problem, which is how do we encourage exploration in
 these environments with more complex tasks and longer time horizons where the extrinsic
 reward that you get from the environment is very sparse.
 So in the absence of extrinsic rewards, how do we encourage agents to explore?
 And typically the way we do so is we assume and this is a very cognitively appealing intuition
 that we should motivate an agent to achieve novelty in the environment.
 We should make it do things that it hasn't done before, encounter states that it hasn't
 seen before, et cetera.
 And then hopefully we'll enable the agent to acquire the skills that we actually want
 the agent to acquire in the environment.
 But the problem with this, of course, is how we define novelty.
 In a lot of scenarios, there are environments that can look very different, but they have
 the same underlying semantics.
 So the example I have in the paper is like a kitchen and the appliances might be differently
 branded and differently colored, but ultimately every kitchen is a kitchen.
 And the way that you approach kitchens and the way that you operate in them is the same.
 And so the idea of this paper is we should be using natural language as the measure for
 how we describe states and how we describe actions within states and use kind of traditional
 approaches to exploration, reinforcement learning, but simply parameterize them with language
 rather than with state abstractions, which is usually the way in which exploration is
 done in these kinds of environments.
 And so what we do is we take existing state of the art exploration methods and then kind
 of see what happens when you swap in language as a component.
 And do you get better performance?
 And we showed that in a variety of settings, at least in the kinds of RL environments that
 people have been looking at in recent work, we do see again in using language to parameterize
 exploration rather than states.
 Yeah.
 I think it's very apt to describe it as you, it's not suggesting like a new exploration
 algorithm, but it's simply the re-parameterization in terms of language.
 And coincidentally, these environments, they do come with this kind of language annotations,
 which we do focus on.
 I like that.
 So I think what I really liked about this paper is just the research mindset in that
 any other paper or a lot of other papers, they would have done, they would have tried
 doing like three things at the same time.
 Like you know, we have a language generator and we do this and we do that.
 And what you're I think doing correctly from a standpoint of research is you keep pretty
 much everything constant, the algorithms constant, right?
 Even the environments, you assume that you have a perfect language oracle and you just
 add the language, which I really appreciate as like a reviewer, let's say.
 So I think this gets us right into our or my biggest, essentially criticism of the paper
 or what I called in that you add language to these algorithms, but you just said we
 swap in language.
 And to me, it felt more like it's not really a swapping in.
 It's more like you add language on top of what these algorithms are doing.
 And therefore, can't I just see your method as adding more data?
 Essentially, there is features that are available from the simulator, right, which the other
 methods just don't use, they just discard this part and you just add this part.
 Do you have an indication in how much of your effect is really due to language and how much
 of the effect is just due to the fact that you have more data available?
 Yeah, that's a great question.
 And it's definitely a point that I think a lot of people will fairly make against the
 paper is, yeah, we're using extra data, right?
 And yeah, I think my verb swap was maybe only accurate in half of this paper, which is that
 in Amigo, which is the first method that we look at, it really is a swap, right?
 So if you read the paper, the traditional kind of Amigo teacher network proposes coordinates
 X, Y positions as goals.
 And here we're just completely eliminating that kind of goal specification and we're
 moving towards language.
 So that can be seen as more of a swap.
 Although of course, in novelty, which is the second method that we look at, that is definitely
 more of kind of an addition, as you say, because we keep the extrinsic bonus and we do have
 experiments that measure what happens if you don't have novelty by itself.
 You only have the kind of language novelty bonus and it doesn't do as well.
 So you're right that I would say that we explore this idea of swapping in language in a bit
 of the paper, but there are points where it's more of kind of a bolt on and we're not like
 super clearly looking at or distinguishing when is it okay to have language just be a
 complete drop in replacement versus just some additional information.
 So yeah, I think we're showing that in general, if you're trying to add language into these
 environments, you're seeing a gain, but how precisely that gain manifests is still a
 little requires some more exploration for sure.
 So I guess more generally to your comment on using extra data.
 Yeah, I mean, I think we have some intuition that this data should help, right?
 It's a fairly clean linguistic signal, but how to use this data concretely is an open
 question, right?
 And so that's kind of where I view the contribution of this paper as even though we have some
 intuition that adding extra data will help, we actually need the equations written down,
 right?
 And here are two concrete ways in which we can operationalize this data for the purposes
 of actually getting better performance in your environment.
 And there are a lot of examples of this in machine learning, right?
 So like you have some large language model, for example, and then you want to fine tune
 it for some domain or you want to fine tune it on human preferences.
 I mean, that's fundamentally, you're adding extra data for the purposes of getting something
 that works well on a task that you care about, right?
 And how to use that data is the open question.
 The other point that I would say is that we have some deep seated intuition that this language
 should help.
 As you say, it's really high quality.
 It comes from an Oracle.
 It comes from the game engine.
 But we actually still need to get that kind of empirical verification that it works, right?
 And there's actually a lot of reasons why maybe these experiments might not have worked
 out.
 For example, the language is Oracle generated, as I mentioned, but it is also very noisy.
 So as I described in kind of the method section of the paper, most of the messages that you
 see in the environments are actually not necessary to complete the extrinsic task.
 And I kind of exhaustively show which of the messages do matter.
 And so it could be the case that, well, the language signal, at least in these environments,
 is too noisy.
 The state abstraction captures all of the factors of variation that you might care about
 in an environment.
 And so you don't ultimately need language, right?
 And that's an imperial question that we have to measure.
 And so I view this paper as providing that empirical verification, which in hindsight,
 I think, is a fairly straightforward intuition.
 It's something that I definitely thought would happen.
 But yeah, it's nice to see those results kind of in writing.
 Yes, it's easy.
 I think you're right.
 It's easy to look back and say, of course, like, well, all you do is you do this.
 But exploration has been since since, you know, people have thought about reinforcement learning,
 they've obviously thought about exploration methods and intrinsic rewards are like as
 old as Schmidhuber himself.
 And we you know, the fact is that, you know, new things are developed.
 And this is at least one of the first things into into really the direction of incorporating.
 There have been incorporation of languages before, but a systematic adding it to the
 state of the art methods.
 And it seems like I am I am convinced the method at least the El Amigo method is quite
 well outlined, I think, in these diagrams, the contrast of the left being the original
 Amigo and the right side being the language Amigo.
 A question I had right here is that on the left side, you have this teacher network,
 and it simply outputs a coordinate to reach and it has to pay attention to the fact that
 the coordinate is not too hard and not too easy, right?
 Therefore, it has to learn that too easy coordinate.
 Yes, one that is, you know, close, but also it has to learn maybe unreachable coordinates
 or coordinates that are inside the walls, right?
 They can't be reached or something like this.
 However, on the right side in the language, I mean, you seem to split these two tasks
 out into one network that that determines which goals can even be reached and one that
 then orders them essentially, why?
 Why are you doing this?
 Like what's the is there a particular reason behind why one network couldn't do both at
 the same time?
 Yeah, so the reason why we split the Amigo network up into two parts, and as you say,
 we don't have to do this.
 And there are ablation studies in the appendix that shows what happens if you get rid of
 the grounding and you just have a single network predicting both goal achievability and, you
 know, actual the actual goal that's seen by the students.
 So it kind of a goal difficulty network.
 It does find in some environments, especially in mini hack, but it doesn't do as well in
 other environments such as mini grid.
 And part of the reason, as you've described, is that at least in these environments, the
 coordinate space stays consistent across episodes.
 And so you're right that there are some coordinates that are perhaps unreachable in certain environments
 and not in others, but there's much less variation than the set of language goals that are achievable
 in an environment because the environment will have different colored doors, for example.
 And so the goal go to the red door only makes sense in, let's say, half of your environments.
 So it's possible for the teacher to the Alamigo teacher to hopefully learn this distinction
 kind of just through, you know, the policy gradient method.
 So basically just like Amigo, but this is relatively sample inefficient because the
 problem is that when you propose a goal that's simply impossible in the environment and you
 get negative reward, that negative reward only comes after the student has tried to
 complete the goal for, let's say, a few hundred steps.
 Right.
 And so it's a relatively sample of inefficient way of telling the teacher, hey, the student
 did not achieve this goal in the environment.
 And moreover, that negative reward, you know, there's two possible sources of that reward.
 Right.
 So if the student never completed the goal, is it the case that it was just too difficult
 for the student, but it is achievable in practice?
 Or is it that the goal is simply never achievable in the first place in the environment?
 Right.
 And those kind of two failure cases are a little bit hard to distinguish.
 Whereas we have kind of this more frequent source of supervision, which is simply, you
 know, as the student is randomly exploring in the environment, it's encountering a lot
 of goals, a lot of messages because we have a language annotator and we're kind of, you
 know, if we if we kind of ignore that signal, that seems like something that we should be
 using.
 And so we have kind of this dual thing where we have a grounding number, which is updated
 more frequently in the environment, which is updated from the messages that are seen
 by the students.
 And then finally, the policy network, which is actually trained to satisfy the kind of
 difficulty objective and actually get the student to complete goals in the environment.
 Can you go a little bit more into because that was, I think, the only part that confused
 me a little bit, which is the how exactly you train this grounding network.
 There is a there is this this notion of whatever the first language description encountered
 along a trajectory being sort of the positive sample and then the rest being the negative
 samples.
 And that kind of confused me because it means the negative samples would also include goals
 that were encountered just not as the first message.
 Could you maybe clarify maybe I didn't understand something right?
 Or maybe I don't, you know, see the reasoning behind this exact choice.
 Yeah.
 So I think your intuition is correct.
 I think you've described it correctly.
 It is kind of a weird thing to do, which is that we are treating negative samples as basically
 all of the goals besides the first one that was achieved.
 Right.
 And of course, that is incorrectly treating negative samples of goals that were achieved
 later.
 Right.
 So negative samples are noisily generated, as I as I say, in the limit, this noise should
 even out, though.
 So you can compare, you know, like we're just kind of noisy, noisily generating negative
 samples here.
 We can compare that to maybe a setting where we had a more oracle sense of when a goal
 is truly infeasible in an environment.
 Right.
 And so what happens is, you know, just in general, a goal is going to appear in this
 negative sample term more and more often as we train the network.
 But because it's we're kind of, you know, downweighing all possible goals in the space,
 the idea is that hopefully, you know, this noise of of class of incorrectly classifying
 a goal is unachievable in an environment kind of evens out over time.
 Right.
 And so, yeah, it's a little bit tricky because we don't have the oracle saying, oh, you
 can't achieve this goal in an environment.
 Right.
 We only know that.
 Well, you know, the student just didn't happen to achieve the goal in this environment.
 So I could imagine other ways in which you try to come up with some heuristic that better
 captures this idea of kind of unachievability.
 But this is what we came up with, which seems to work reasonably well in practice.
 And alternative way that you can interpret this is we're not really measuring true achievability.
 Like, you know, is this at all possible in an environment?
 What we're really trying to have the grounding network capture here is what are the goals
 that the student tends to reach?
 So like are feasible at the current state of training, right?
 The current policy, what goals can it reach?
 And that's really what we need, right, is we need like to propose goals that at least
 for now are eventually reachable by a student.
 And that doesn't mean that it's, you know, unachievable in all possible students under
 all possible environments, but at least just for current, you know, in the current stage
 of the training process, it's a reasonable target.
 I can imagine that this gets very, that this may require an adjustment or that this breaks
 down in environments that are more causally structured.
 For example, if I always have to go through the green door before I reach the red door,
 right, then the goal would always be in any trajectory that I do, the green door would
 always be the first goal.
 And therefore my grounding network would never recognize the red door as a reachable goal,
 because that's always going to be at least the second goal, right?
 So I guess depending on the environment, it's not hard to make a change to this, obviously,
 in that case, but I guess that's one thing that might have to adjust a little bit to
 the environment at hand.
 Yeah, that's a that's a great point is that we do not.
 There are settings where you might just, you know, want to run it without the grounding
 network.
 And obviously, that's actually a simpler version.
 So it should be fairly easy to experiment with that.
 And also, in the setting that you described, what will happen is, like you say, you know,
 the green the go to the green door goal will get a lot of weight, but hopefully can be
 counteracted to some degree by the policy network, which will, you know, learn to not
 put any weight on that once it realizes that it's getting absolutely zero reward for that
 setting.
 But I agree that this kind of introduces some weird training dynamics that we don't really
 want might be cleaner just to remove the grounding network entirely.
 If you as as you say, you've looked at my paper review a little bit, I didn't go too
 much into the experimental results as such.
 Is there also I didn't go into the appendix at all, because honestly, I haven't read the
 appendix because I sometimes I don't I think I should probably.
 But is there anything that you want to highlight specifically about the experimental results
 or or maybe something that you did in the expand appendix, which is also has a lot of
 experiments in it?
 Things that you think people should take away from the paper from the experiment section?
 Yeah, so broad takeaways are and I think that you mentioned this in the review is, you know,
 we're in these kind of DRL environments and and the individual training runs are just
 incredibly noisy, you know, and that can be sometimes like rather difficult to get a sense
 of, oh, is my method actually working better than others?
 Right.
 But there has been some great recent work from I think a team at Miele, which won an
 outstanding paper award at New York's last year, which was called deep reinforcement
 learning on the edge of the statistical precipice.
 And the basic idea is, you know, we're compute constrained.
 We have these environments, they're very high variance.
 But even despite all of this, you know, what are the kind of statistical best principles
 that we can follow to really see whether or not our methods are actually making a measurable
 and replicable difference in the environments that we're testing?
 And so they have a lot of good recommendations, which we try to subscribe to as close as possible
 in this setting.
 Right.
 So these training curves here give you kind of a qualitative sense about not only kind
 of the ultimate performance attained by any of the models, but also of the differences
 in sample efficiency that we see.
 Right.
 So it could be the case that, well, ultimately, both Amigo and El Amigo reach the same asymptotic
 performance, but Amigo just gets there faster or more reliably.
 And that's something that you can, sorry, El Amigo gets there faster and more reliably.
 And that's something that you can look at in these graphs.
 But I think the more kind of statistically rigorous way of verifying that language is
 giving a gain in the environments is in the subsequent figure, which is figure four, which
 should be right below this one, I think.
 And this is really, you know, us trying to statistically verify, you know, is there an
 effect happening here?
 And so these here are bootstrap confidence intervals, five runs in each experimental
 condition.
 And we're plotting the 95 percent confidence intervals for the interquartile mean of models
 across tasks.
 So this is kind of like the mean performance, assuming that you drop some of the outliers,
 because again, these runs are very high variance.
 Right.
 And so this is kind of a statistical recommendation from the authors of that deep RL paper.
 And we show that, yes, the individual runs here have really high variance naturally.
 But as you begin to look at the runs in aggregate across both the mini grid and mini hack environment
 suites, we begin to see a trend that it's clear that, you know, overall we're seeing
 a good effect of language in these environments.
 And so this is obviously these are aggregate metrics, overall metrics and so on.
 When we look at the plots themselves, there is quite considerable variance, even in the
 ranks of the method.
 Do you have an intuition of between the language methods, which works better in what kind of
 environments and in what kind of environments does language even maybe hurt?
 And why do you have an idea?
 Yeah.
 So the trend that I try to highlight in the paper is that in larger environments, language
 exploration does better.
 And the reason why you might expect this is that in larger environments, Amigo and Novelty
 kind of suffer from this problem of increased noise.
 Right.
 There's a lot more coordinates, for example, that you can propose, which essentially describe
 kind of the same semantic action.
 Right.
 You have like you want to get the agent into one room of this maze.
 And you know, because the environment is larger, now there are four or five different coordinates
 that all kind of mean the same thing.
 Whereas as you increase the size of the environment, the language set, the set of language goals
 is relatively more consistent.
 Right.
 It's kind of one of those complexity analyses.
 Right.
 It's like kind of space complexity, almost of the goal space.
 And so you can see this trend happen a bit.
 For example, in the Wand of Death task, so WOD, this is in the top right corner here.
 We have WOD medium and WOD hard, where in WOD medium, Amigo actually outperforms El
 Amigo.
 So it gets you to higher performance quicker.
 Whereas in WOD Wand of Death hard, Amigo is actually not able to learn at all.
 And the only difference between these environments, it's fundamentally the same task.
 But the only difference is that in WOD hard, the room is a lot bigger.
 So instead of a narrow corridor, you actually have to search for the Wand of Death, that's
 the task, in some in some room beforehand.
 And you can see that just simply increasing the size of the possible coordinate spaces
 results in both traditional novelty and traditional Amigo doing much worse in this environment.
 And I think that kind of shows that these kind of state based exploration methods are
 very brittle to the size of your state base.
 Right.
 So you can kind of increase your state space infinitely and it'll make these methods perform
 worse, even if the underlying semantics of your environment haven't changed yet.
 Do you have an idea, do you have a feeling maybe, if this is a property of the world
 in general, like let's say I as a human, right?
 I'm put into a small whatever environment or a big environment, would my descriptions
 of language also not grow very much?
 Or is it a property of just game developers?
 You know, I add a few extra rooms, I can reuse these languages, you know, I just kind of
 tile, you know, the other the big games, I mean, the biggest games are procedurally generated
 like Minecraft there, it's really, it's just the same thing over and over.
 But even in like the like, these big open world games, like Grand Theft Auto or so,
 the same textures are reused and the same cars and the same NPC characters, right?
 Is this a property of the world or of the video game developers?
 Yeah, so this is a really deep and almost philosophical question.
 Yeah, is something that I think about a lot is you can certainly and this is a totally
 valid statement, right, you can say, well, there are a lot of language actions that you
 can describe in our world and even in the video game world, which just described these
 like kind of infinitely complex and nested sequences of actions, which have absolutely
 nothing to do with the extrinsic task, right?
 I could tell you to, you know, oh, you know, run at the wall six times do a 360.
 And then, you know, continue hitting the wall eight times, right.
 And that's like an incredibly difficult goal, which you can imagine a very structured curriculum
 to get to that point, right, of just like infinitely kind of bumping your head against
 the wall, which satisfies, you know, maybe the difficulty threshold of El Amigo, but
 is absolutely orthogonal to the task that we care about.
 And I can imagine that there are settings where the language is kind of useless and
 doesn't end up, you know, giving you any gains in this setting.
 And so there's kind of this open question that we haven't really touched on sufficiently
 in this paper, which is how good does the language have to be in order to get this to
 work?
 So as I say, you know, the language is Oracle, it's game developers, but it also is noisy.
 There's a lot of actions like running into walls or trying to throw stones at a minotaur
 that are ultimately useless in the environment.
 The argument we're making here is that hopefully, you know, the noisiness of language scales
 a little bit less than the noisiness of your state environment, right.
 But there's still a lot of kind of edge cases and kind of unexplored territory here.
 I think more philosophically, if you think about our world and our environment, right,
 there are a lot of ways that we can describe actions that are not particularly useful in
 the world that you and I inhabit, right.
 I mean, I can again tell you to do handstands and hit a wall and, you know, walk around
 and write endless, you know, trivial things in the dust.
 But at the same time, there's a lot of our action space in the real world that we simply
 don't have language descriptions for, right.
 So like every single precise movement on my hand and my arm, you know, I could presumably
 come up with some language to describe, oh, I'm actuating this joint, you know, by 0.03
 degrees.
 And there's like, you know, how many joints in my hand, right.
 I mean, there's like endless complexity in terms of the possible action space just by
 moving a hand that in language we have absolutely no words for, right.
 And so it's really it's a really tough question, right.
 Like we have a lot of kind of ways of describing useless actions in the world.
 But at the same time, it's very clear that the language that we do use to describe the
 world is operating at a higher level abstraction than perhaps the kinds of actions that RL
 agents have access to, right.
 And for example, actuating some sort of limb or something.
 You make a you make a good point that in the paper that language is a strong prior over
 what is essentially important to humans, right.
 If I can describe something with a short piece of language, like, of course, I can say do
 three backflips and then, you know, do eight of that and so on.
 But it's a fairly complex sentence in itself.
 If I can describe something with a short piece of language, usually that is something that
 matters to some human somewhere, right.
 Otherwise that wouldn't be mapped to a short string.
 But that brings me a bit to a different question.
 And that is the question of isn't isn't the I think in these environments, there's always
 a goal, right.
 There is one reward at the end that you need to reach.
 I can imagine, though, that novelty or not novelty in general or how how important a
 state is, is really dependent on your goal.
 Whether I circumvent the minotaur at the, you know, below or above that might not be
 important if I want to reach whatever the goal behind it.
 But it is really important maybe for a different task.
 It's likewise I as a human, whether I move from here to there by walking forward or backward
 doesn't matter if I want to get to the fridge.
 But it matters really if I'm if I'm dancing, right.
 So is that something that like how does that interplay here with these with these language
 things?
 What do you do when a language it almost like needs to incorporate a piece of the goal that
 you want to reach in order to be useful or not?
 Yeah, so I think thinking about or trying to filter the language descriptions that you
 have to language that is relevant for your task is going to be important if we scale
 this up to environments where it's clear that using unfiltered language is not helping.
 Right.
 And again, as I mentioned, the robustness of these kinds of exploration methods to the
 noisiness or relevance of your language signal is still an open question.
 If we do have task descriptions, so we have extrinsic task descriptions like your job
 is to defeat the Minotaur, then it's really intuitive that we should be able to use that
 as a signal for kind of waiting how relevant a sub goal or language description that we
 encounter waiting how useful that is for the extrinsic task.
 Right.
 So if the extrinsic goal is combat, then we should be prioritizing combat related messages.
 If the extrinsic goal is buying something, then we should promote acquiring money and
 things like that.
 And so that's something that I think is a kind of natural extension of this is you extend
 this to a multitask setting where you have task descriptions and the task descriptions
 ought to kind of heavily filter what sub goals should be relevant for the task.
 I think when you include task descriptions, there are some more comparisons to related
 work.
 There's some related work, which you mentioned the paper where let's imagine you're doing
 basically hierarchical reinforcement learning.
 So you have some extrinsic goal and then you want to explicitly decompose the extrinsic
 goal into sub goals that you want to complete in order.
 Right.
 And that's those are certainly kind of relevant methods to look at when you start thinking
 about multitask or goal condition settings.
 But this is kind of a slightly different focus where we're not trying to identify sub goals
 that need to be completed on the way to some extrinsic goal.
 There's still kind of this exploration component, which is a bit of a different use of language
 than this kind of hierarchical stuff.
 But certainly I would say that there are people who have looked at kind of language conditioned
 RL and hierarchical RL that think a lot and very deeply about this problem of proposing
 sub goals that are relevant for the extrinsic goal, assuming you have some structured description
 of what the extrinsic goal is.
 Although I can imagine you run into sort of the, let's say the more abstract problem of
 the exploration problem is that, you know, without an outside signal, I don't really
 know what to do.
 And there is no clear, let's say gradient towards the goal.
 Right.
 Otherwise, the exploration problem in RL would be relatively easy.
 Now when we say, well, we'll just filter out all the messages that don't have anything
 to do with our combat goal.
 Right.
 So it is like we could run into the exact same thing again, where, you know, maybe in
 order to acquire a weapon, I first need money, right?
 That doesn't, that's not directly related to my combat goal.
 So there is like another exploration problem again, on top of the thing we introduce.
 I guess maybe we can hope that if we introduce enough levels, the highest level of abstraction
 will have a small number of states so that, you know, random exploration works.
 But it's kind of funny that the problems repeat or replicate.
 Yeah.
 Yeah.
 It's really tricky.
 And that's essentially just kind of a deeper or more nested failure case of not knowing
 what's novel and not knowing what's relevant for your goal.
 Right.
 So if you're prioritizing words that have combat in them because your extrinsic goal
 is combat, but you first need to buy something, then your, your, your semantics, you know,
 your measure of novelty or relevance is just not good enough.
 Right.
 So that's going to just be a fundamental problem in exploration is how do we know whether it's
 states or language, you know, how do we know when a state is relevant for the ultimate
 task?
 Yeah.
 And I guess humans aren't very much different, right?
 I mean, science is a really hard process.
 It's not, you know, that exploration takes millions of humans and hundreds of years.
 So we can't fault our RL agents here for not, not doing that great of a job.
 Here, I found these plots to be really cool, like the analysis, sort of the evolution of
 what the teachers propose.
 And of course, these being language, it's quite insightful and understandable what's
 happening in the algorithm.
 My, my surprise was a little bit, aren't these things kind of subject to like catastrophic
 forgetting or things like this?
 I can imagine, right?
 If I train these things online and they're at some difficulty level, all of a sudden
 they forget that reaching the red door is kind of really easy.
 Or so is that have you ever thought is that a problem?
 Or was that ever a problem?
 Did you encounter that?
 Or why don't we encounter that?
 Yeah.
 So I expect that that is a problem that happens in these agents.
 I don't think we really precisely tried to measure whether or not catastrophic forgetting
 is a problem.
 I think the fact is that we evaluate in environments where we are not testing the agents kind of
 continuously for mastery of all of the skills that it has learned in its curriculum proposed
 by the teacher.
 And so this problem of, oh, you know, you forgot how to specifically open a specific
 color door is not an issue as long as the student is still quite good at completing
 whatever goals it needs to complete to try to achieve the extrinsic goal that is currently
 being set by the teacher.
 Right.
 So if you forget things that are at the very beginning of training, that's not a big deal.
 So long as whatever path that the teacher is leading you on is something that will eventually
 get you to the extrinsic goal that we care about.
 And I think that happens to be the case in these environments because there was only
 one extrinsic goal and because we're not testing it to master every single skill from kind
 of low level to high level abstractions.
 But if we were in a setting where being able to complete those lower level goals kind of,
 you know, on a dime and kind of, you know, switch kind of do context switching like that,
 if that were more important, then we would have to deal with this problem of catastrophic
 forgetting.
 Right.
 An important point here is that we really don't care about how well the student is able
 to follow instructions proposed by the teacher.
 That's, I mean, we hope the goal is that that property emerges such that we can complete
 the extrinsic goal.
 Right.
 But we're never actually trying to learn a student that can follow instructions.
 We never really evaluated exclusively in an instruction following setting.
 Because if we think ahead a little bit, and I'm going to want to just scroll down to the
 environments just because, yeah, maybe this this will inspire us a little bit.
 If we think ahead a little bit beyond this work, here you have this very, this Oracle
 language descriptor.
 And you say also in the outlook of future work that that is something obviously that
 we're trying to get rid of because not every environment, like the fewest of environments
 actually have such a built in language description or easily accessible one.
 So we might have to regress to something else.
 So I want to I want to think about three different external models that we could bring in.
 And I wonder what you think of each of them, like how these could fit in.
 The first would be something like GPT-3, like just a pure language model.
 How could that help us?
 Maybe in combination with these things, because we need some starting point, right?
 But how could a pre-trained language model that knows something about the world help
 us?
 Then something like CLIP, maybe something that can take an image and language and say
 whether they're good together or not.
 And then maybe even something like or maybe a captioning model.
 Right.
 And maybe something like DALEE, like something that takes language and generates.
 Is there in this cloud of models, what possibilities do we have to bring in sort of to replace
 this Oracle thing with with learned systems?
 It doesn't even need to be learned online, right?
 It can be pre-trained.
 I'm probably much more excited about that.
 Yeah.
 Yeah, these are, I think, going to be the most fun questions to look at in kind of language
 conditions are all going forward is taking the boom in pre-trained models in large language
 models and resulting, you know, bringing these into concrete and actionable gains in reinforcement
 learning.
 It's funny that you mentioned this kind of what I described as almost a gradation from
 ungrounded language models like GPT-3, right, which are trained on text only corpora and
 whether those can actually help in these environments, which I would call are fundamentally grounded,
 right?
 They're grounded in some some visual or perceptual world.
 And ungrounded language models still result in gains in these settings.
 And my intuition is, yeah, they probably still can because, you know, even if you don't exactly
 know what it means to acquire a wand or kill a minotaur in some environment because you
 don't know what a minotaur looks like or what a wand looks like, GPT, as I mentioned, you
 know, this idea of priors, right?
 GPT has strong priors on sensible sequences of actions, right?
 So insofar as these environments are testing kind of sequences of actions that humans kind
 of have an intuition for, you know, it's some fantasy world, but we have some intuition,
 oh, in order to defeat the minotaur, we need to get a weapon first.
 We probably look around for a weapon.
 Maybe there's a shop.
 Maybe we can buy a weapon from the shop, right?
 Video games are testing knowledge that we have very like deep seated commonsense knowledge
 that we have that hopefully generalizes to these fantasy worlds.
 And GPT certainly contains a lot of that information, right?
 So you might imagine we should reward or filter the kinds of descriptions that we see to those
 that seem sensible narratives that GPT-3 would generate, right?
 So a sensible sequence of actions along the way to defeating the minotaur is collecting
 a wand and buying it and things like that.
 And I think you actually already see some examples of this happening in more goal conditioned
 or instruction following RL.
 So there's been some recent work from, I know, teams at Berkeley, maybe Google as well, that
 are looking at using pre-trained language models, which are not necessarily even grounded.
 They're just, you know, GPT-3, using them to construct sensible plans, action plans
 or sub goals for completing certain actions.
 So in some home environment, for example, maybe my action is get a cup of coffee.
 And then the goal of GPT is even though I don't really know what my environment looks
 like, I don't know what kitchen you're in, I know that sensibly this should include finding
 a mug and then heating up the kettle and things like that.
 And so we already see some promising use of kind of ungrounded models for improving grounded
 decision making settings.
 Yeah, did you want to comment on that?
 Or I can also-
 No, no, that's cool.
 I think, yeah, I think I've even had at least one of these works here on the channel in
 this home environment.
 That's exactly, I was also really cool to see.
 Obviously, these models know a lot about the world, right?
 And I think people overestimate how or underestimate maybe, well, whatever.
 That the thing, if we humans look at a board like this, like at a mini hack board, we see
 a map, right?
 We see paths to walk on and stuff like this, even if we've never played a video game.
 But this is, these are such strong priors built into us.
 And we sometimes think like, why can't that dumb computer just like walk around the wall,
 right?
 And we're like, what's up?
 And I think these large models are a way we can really get that knowledge from the human
 world into this world.
 So yeah, I think that's, it's a great outlook.
 Also with the models that combine images and text, I feel that could be really like adding
 a lot of value to the RL world.
 At least the RL environments that are like human environments.
 Of course, there's reinforcement learning for computer chip design, and things like
 this.
 I don't think those are necessarily going to be profiting that much from it.
 But yeah, yeah, really cool is so you're you're at Stanford?
 Or did you do the work at Stanford?
 Or were you at some internship?
 Yeah, I did it while I had an internship last fall.
 So this is fall 2021.
 Okay, continue to work a little bit while at Stanford.
 But it was mostly in collaboration with some people at fair or meta, I guess now in London.
 Reinforcement learning is notoriously also kind of hardware intensive.
 Although this work right here seems like maybe not that much because you describe a little
 bit sort of what what it takes to investigate a project like this.
 Yeah, unfortunately, I think even for these environments, it's fairly hardware intensive,
 certainly still feasible, I think, on let's say, a more academically sized compute budget.
 But for being able to run the experimentation needed to iterate quickly, you know, you do
 really definitely benefit from kind of industry level scale, which is one of the unfortunate
 things about this kind of research is that it is a little bit less accessible to people
 in smaller compute settings.
 So maybe the typical kind of RL environments you think of our compute heavy are the ones
 that are in 3D simulation, you know, very, you know, need physics, need soft joint contact
 and all of these things to model.
 And those are really expensive.
 I think compared to that, these are kind of more symbolic grid worlds.
 You know, the whole point as to why mini hack or net hack was chosen as a reinforcement
 learning test bed was because the code base is, you know, written entirely in C and is
 very optimized, and so you can run simulations very quickly on modern hardware.
 But that being said, it's still relatively compute expensive.
 Again, the just amount of experience needed by state of the art, deep RL methods, even
 with extrinsic or intrinsic exploration bonuses is still very expensive, right?
 So for example, one of these runs, we would typically have, let's say, 40 CPU actors collecting
 experience at the same time in parallel, and then kind of one or two GPU learner threads
 in the background kind of updating from this experience.
 So even just a single, you know, computational experiment here needs non trivial hardware
 for sure.
 Yeah.
 And, and you ideally you want to do that in parallel, right?
 Because you want to try out a bunch of things are repeated a bunch of times because one
 experiment really tells you almost nothing, right?
 Unless it succeeds, right?
 If it succeeds, it's good.
 But if it fails, you never know if you repeat it a bunch of times.
 Yeah, but I mean, it's still it's not it's not the most extreme thing, right?
 Like two GPUs or so and a bunch of CPUs.
 As you say, that can that's still academically doable, which I find cool.
 Could you maybe tell us a bit about the process of researching of researching this?
 Like, did everything work out as planned from the beginning?
 Or where was your starting point?
 And what changed about your plan during the research, like maybe something didn't work
 out or so?
 Yeah.
 Yeah, I feel I don't I feel it's always good for people to hear that other people encounter
 problems and how they get around problems.
 Yeah.
 Yeah.
 So yeah, it's a great question.
 The intuition that I think me and my collaborators started with was, you know, fairly sensible.
 It's language is clearly going to help in these environments.
 You know, it has some nice parallels to human exploration.
 And so let's just see whether or not language will work in these environments.
 What's funny, though, is that we actually started out the project less about the more
 abstract question of like, does language help exploration and more a very concrete question
 of how do we improve upon Amigo?
 So how do we improve upon an existing state of the art algorithm for exploration?
 Let's propose something that we argue is better than everything.
 It's like we're going to propose a state of the art exploration method called El Amigo,
 which will get 100 percent accuracy in all these environments.
 And none of the existing methods will work.
 Right.
 That's that's kind of the narrative that you set up for yourself when you're starting research
 is I'm going to build something that's new and that's the best.
 Right.
 However, I think the focus of this paper and the story has shifted considerably.
 I think it's shifted for the better, actually.
 And part of this shift happened because we implemented El Amigo and it was working fine
 and it worked better than Amigo.
 So we were quite excited.
 But at the same time, the field is moving so fast.
 And at NeurIPS last year, some researchers came out with this method called novelty and
 we ran novelty and novelty also did really well.
 And you know, in some environments, it totally like blew Amigo out of the water.
 Right.
 And El Amigo.
 And part of our thinking was, well, OK, now we can't really say, oh, we have El Amigo
 and it's the best model.
 It's the best environment.
 And you should only use this.
 And at first I thought, you know, this is derailing our narrative.
 Right.
 We're not proposing anything new.
 We're not proposing anything state of the art.
 So what's the point?
 But I think after some kind of juggling and shuffling, we realized that what we're really
 interested in is the scientific question of does language help exploration?
 So take existing method X and then do X plus language.
 Right.
 And so this question can be answered kind of agnostic to the specific method that we
 actually use.
 Right.
 And so it was that juncture where we actually decided, OK, let's actually look at novelty
 closely and let's imagine adding language to novelty as well.
 And do we see the same kind of results?
 Right.
 And so I think this is kind of an outcome of the paper that was kind of on the fly changed.
 But I'm very happy with which is that we're not trying to claim that we have a method
 that is state of the art or that is best or that anyone should be using our method.
 We are very agnostic to the particular choice of method.
 Right.
 We're trying to answer kind of a more abstract question, which is when does language help
 exploration?
 And I think this is a little bit more egalitarian.
 We're not saying that our method is better than anyone else's.
 And we also don't have to exhaustively compare to like a lot of existing work.
 We're just saying that if you take whatever method that we have and you add language,
 you do better and here are two examples where that happens.
 Cool.
 And it is a good way to preempt some reviewers from saying that you didn't train on ImageNet
 and that's bad.
 Yeah.
 Is there anything else that you want to get out to viewers?
 Maybe a way they can get started if that's possible or anything that you'd like them
 to know?
 Yeah, I think that we've discussed a lot about these kind of higher level ideas of one holy
 grail is that we have clip generating descriptions or open GPT-3 and then we're evaluating in
 these really high dimensional spaces with actual motor joints and we're going to show
 how language helps in these like mojoco style, like really deep RL, realistic environments
 and maybe you can transfer to the real world.
 I think that's the broad vision but I think it is still very far away.
 I think we even in this paper abstracted away a lot of difficulty of the problem.
 We're assuming that we have Oracle language annotations.
 We're only looking at these kind of symbolic grid worlds and although it's tempting to
 dive in and say, okay, now let's kind of straightforwardly let's extend this to a real world environment
 where I have to actually move my coffee mug to make coffee and tea, I think we're still
 quite far away from that broad vision of kind of household enabled robots in RL and is probably
 not the most I think like beginner friendly way of starting.
 There's just so many deep problems that need to be solved jointly from perception to action
 to planning and before we even consider how we better incorporate language into the mix.
 And so I think the way to build upon this work is just these kind of very small progressive
 relaxations of the assumptions that I and many of the other people who have worked in
 this space have.
 Right. So again, let's imagine let's just imagine we get rid of the Oracle language
 annotator and we train a model to emit states for these simple environments.
 You know, we didn't really explore that, but that's a very sensible way to extend this
 kind of work while keeping the environment and the models fixed.
 Right.
 So this goes back to the very beginning when you mentioned the kind of way in which we
 approach this paper was to keep everything fixed and then just look at this kind of very
 small change and see how that results in different performance in our environment.
 I think that's really just kind of the way to go.
 It's very slow.
 It's very incremental work, but hopefully it's getting us more towards that kind of
 guiding star of eventually having these models that operate in these realistic environments
 and use pre-trained model language to help exploration.
 Cool.
 Jesse, thank you very much for being here.
 This was awesome.
 Thanks.
 Have a lot of fun.