Improving Intrinsic Exploration with Language Abstractions (Machine Learning Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper machine learning news ml paper machine learning paper language nlp natural language processing stanford reinforcement learning data science deep learning tutorial deep learning paper language in reinforcement learning rl nlp nlp rl nlp reinforcement learning exploration exploitation rl exploration
#reinforcementlearning #ai #explained

Exploration is one of the oldest challenges for Reinforcement Learning algorithms, with no clear solution to date. Especially in environments with sparse rewards, agents face significant challenges in deciding which parts of the environment to explore further. Providing intrinsic motivation in form of a pseudo-reward is sometimes used to overcome this challenge, but often relies on hand-crafted heuristics, and can lead to deceptive dead-ends. This paper proposes to use language descriptions of encountered states as a method of assessing novelty. In two procedurally generated environments, they demonstrate the usefulness of language, which is in itself highly concise and abstractive, which lends itself well for this task.

OUTLINE:
0:00 - Intro
1:10 - Paper Overview: Language for exploration
5:40 - The MiniGrid & MiniHack environments
7:00 - Annotating states with language
9:05 - Baseline algorithm: AMIGo
12:20 - Adding language to AMIGo
22:55 - Baseline algorithm: NovelD and Random Network Distillation
29:45 - Adding language to NovelD
31:50 - Aren't we just using extra data?
34:55 - Investigating the experimental results
40:45 - Final comments

Paper: https://arxiv.org/abs/2202.08938

Abstract:
Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 45-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Authors: Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, Edward Grefenstette

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, this is a comprehensive paper review on the paper Improving Intrinsic Exploration with Language Abstractions. This is a very cool paper because it combines a language and the information that is in language with reinforcement learning, specifically the problem of exploration. I don't want to tell you too much more right now because we're going to dive into the paper in just a bit. So this video will explain in detail what is in the paper, how the method works, what they're doing. So by the end of this video, you should have a really good idea of what's in the paper. In the next video published tomorrow, there's going to be an interview with the authors of the paper, which is very, very cool. It's super valuable, and I was very happy to host this interview. So I hope you draw some value out of either one of these videos. Hopefully both. As always, thank you very much for watching. Thanks to everyone who likes and comments and supports in any way. It's really cool to be able to do these things. And I'll see you around. Bye bye. Hi there. Today, we're looking at Improving Intrinsic Exploration with Language Abstractions by researchers of Stanford University, University of Washington, Meta AI and University College, London. This paper on a high level uses language to facilitate intrinsic exploration. That is when in the face of a very sparse environment, a reinforcement learning agent has to come up with its own goals in order to make progress. So the intrinsic exploration or intrinsic motivation refers to the fact that the there's an additional reward that we give to the agent just for attaining, let's say new states novel things in the environment. Now it turns out that that's not super duper easy, because not all new things are equal. And especially, let's say there is a random component in the environment, then, you know, that's going to be new every time, yet it might not be interesting. So how you go about this is quite a challenge. It's clear that we need something like this in sparse, sparse rewards environment. But how exactly to do it is still still challenging. This paper adds language to the mix and argues that language descriptions could be one such source of novel of indicators of novel states. So we're going to go through the paper, let me know what you think in the comments, definitely. And yeah, let's dive in. So they say they want to solve these these complex long horizon tasks with sparse rewards. And as I already said, that is not really a picnic for reinforcement learning agents. Usually those need very tight, very dense rewards in order to work. And that's why we give these intrinsic rewards for exploration. And that is encouraging the agent even in the absence of rewards to go out and explore things and do new things. And we hope that through the exploration, at some point, it will learn the skills or it would encounter something that will that will actually give true reward. So they correctly claim there is a design choice on how to measure exploration and a an implicit like a common answer that the agent should be rewarded for attaining novel states in the environment. But that is, as we already said, quite difficult to actually implement. For example, states can look cosmetically different, but have the same underlying semantics and thus not be truly novel. So the two fundamental challenges for intrinsic exploration they they they list here is first, how can we reward true progress in the environment over meaningless exploration? Second, how can we tell when a state is not just superficially but semantically novel? And that's where they add in language. They say, well, if we had language describing the states, then certainly, for example, here, we have language that describes the state. Here the the language description says in what direction, indicating that you can go in a couple of directions or do something in a couple of directions, you see here a crystal wand, that means there's something to pick up. So when you don't have this message, that might be an indication that the state is meaningfully different, namely, it doesn't have the crystal wand. So as you can see, these authors imagine that if we had a language description of the environment, that could give us an indication of when something is novel, and when something is just the same but looks a little bit different. They say language obviously has strong priors over the features and behaviors needed for meaningful interaction and skill acquisition. That's just a matter of fact that language has been developed to communicate things that are useful to humans. And they also say correctly that you can describe with language very particular things such as move left or very abstract things like acquire the amulet and defeat the wizard. Although one of the abstraction here comes from the end, but still defeat the wizard is a very, very abstract thing. Now, as we already said, what they're going to do here is they're going to look at these environments, at these reinforcement learning environments. So there's mini grid on the left. And in mini grid, I believe the agent here, you're that that's the the red triangle. And the agent is supposed to I think, go to the keys, get the keys, open the doors and eventually get the final reward that is somewhere on the map. These are procedurally generated. So it always kind of looks different. And that's one challenge because if you have to make sequences of actions like go over here, get that key, go to the door, and then go further and get the reward, that is a sequence of actions that is unlikely to happen by by chance, right to stumble over the key and to stumble over the door and to stumble over the reward. You know, the amount of times you're going to try randomly until that's the case is is staggering. And therefore something like Q learning, which just requires on random exploration is going to almost certainly fail right here. But this is one of the environments, which is a challenging environment that they pick up. And that has these language descriptions, or I think in this one, they add the language descriptions. But in any case, this is not about language models or anything like this. They assume that they have a function which they call L, the language annotator that takes in a state takes in and gives you the description. And they just assume they have an oracle that does that. So for the environments they do test, they actually have that. And so in minihack here, this is even part of the game, right? In minihack, you will always get a message like this to every step that you do in almost most of them, most of these states have such a description available. So again, there's this function L, which in this case is just the game engine, it takes in a state and it gives you back the description. So if you, you could guess here that we might learn this language descriptor, right? We might even initialize it with a language model, we can use something like clip or something like this. This is certainly in the future work, they list this, but not here. Here we assume we have these oracle. Now what can we do once we have such a language description? Well, we can use it for exploration. So there is a little bit of mathy math right here, which we're going to skip. Essentially this just discusses that yeah, they have this annotator L that produces these natural language descriptions and they add an intrinsic reward to this. And now we're going to look at what the intrinsic reward is. So they're going to take two different in like two different algorithms that are already made for intrinsic motivation, and they're going to augment them with language. The reasoning behind it is that those two algorithms, the one is called Amigo, the other one we'll get to in a second, they're already kind of state of the art in this domain. So what they say is if we add language to those, and we can get a better result, then that kind of shows the usefulness of language of the language descriptions. So we're going to look at these algorithms briefly remember these algorithms aren't by this paper, this paper is how to add language to them. So Amigo, the adversarially motivated intrinsic goals trains a student and a teacher. So there is a teacher that generates goals. And then the student is just a goal conditioned policy. The goal is, as we said, provided by the teacher. So the student is the real reinforcement learner, but the student is simply conditioned on some goal that's provided by the teacher. It is not it doesn't try to solve the actual problem. It solves the goal that the teacher gives it. I mean, it probably gets reward when it accidentally also fulfills the true reward goal, but it does get intrinsic reward when it fulfills the goal set by the teacher. Now the goal set by the teacher. That's the trick obviously right here, the teacher policy is quite smart. The teacher policy takes in the state of the student. So it looks at you know, where is the student. And it needs to now decide what do I do? What kind of goal do I give the student on the top left here you see this in in in this mini grid environment. The teacher is this network or this function right here. It gives a coordinates that the student has to get to. And then these coordinates as you can see there. I'm not sure if those are the actual coordinates. But whenever the student actually reaches them, so it provides the goal to the student when the student reaches it, it gets reward. So that's it. There is also a notion of a difficulty threshold. That difficulty threshold is it increases during training. So the idea is that at the beginning, the teacher wants to suggest kind of easy goals. And then as time progresses, the teacher has to learn essentially how to make the goals harder and harder. And by making the goals harder, the student essentially has a curriculum of harder to reach skills. So the teacher should kind of learn to propose more hard goals. So I think that the work here is definitely done mostly by this teacher network and the challenges. In any case, there is this difficulty threshold. This difficulty threshold is increased linearly during training. And the student, no, sorry, the teacher, the teacher is given a positive reward if it proposes goals that take the student more than T star time steps to complete and a negative reward for goals that are completed sooner or never completed within the finite time horizon. So you also can't go impossible or it can't go too hard. It needs to go exactly as hard that the student reaches the goal, which means even even if it's a possible goal, it can't go too hard for the current student. It needs to essentially propose goals that are just outside the outside the abilities of the current student. So that that zone of proximal development is kind of formalized in this teacher. That's Amigo. The other so how do we add? How do we add language to that? We saw that usually the teacher supposes or proposes coordinates for the student to get to. Now if we have language descriptions for every state, so every state the student finds itself in, there is a language description. The teacher can simply output a language description of a state. In this case, these are formulated as as kind of instructions. But remember, they are just descriptions as far as I can tell of of the state. It is more evident in the mini hack environment. So these these are just descriptions of the state, whatever the game would output if you're in this state. And the teacher simply proposes these. So it just says, well, here is a goal for you. Try to get to a state where the language descriptor outputs that. So that those are the goals that the teacher can choose. Where are we? Yeah. So we don't have x y goals, but we have natural language goals. The student is rewarded if it reaches a state with a natural language description that the teacher outputs. Easy enough. So how does the teacher do this? It selects goals from the set of possible language descriptions in the environment. Now, initially, these are unknown. So the teacher doesn't know yet what the environment has in store. Because again, we don't assume that say extra information, we need to get out everything of the environment. Therefore, as we go through the environment, we collect more and more of these goals. And these are the goals that the teacher can choose. The teacher maintains a running set of goals that is updated as the student encounters new state descriptions. The teacher has this move to language, they say creates a challenge. Not only must the teacher choose which goal to give to the student, it must also determine which goals are achievable. And that's why they train two different networks. There is a policy network, which produces the distribution over goals given a student state and a grounding network, which predicts the probability that a goal is likely to be achieved in the first place. So remember, these environments, they're procedurally generated. So every time the student is every new episode, I believe that's how it works. The student is placed in some environment that it has essentially never seen before. So now the teacher takes that in, and it produces two things, it looks it looks at this environment, produces two things from the set of goals that it has, it picks one that it wants to propose. That needs to be right so for it cannot always do the same. That's the interesting part right here. So if the green door is over here, go to the green door might be very easy in one environment, but very hard in the other environment. When I first read this, I thought, well, if you know, if the teacher knows no goals at the beginning, and it only collects these goals that the students student encounters over the course of the episode, we're still kind of relying on random exploration of the student right because any goal it hasn't achieved yet cannot be proposed. Whereas in the original x y coordinate, I can, I believe at least I can just propose any x y coordinate like get to that. However, since this is procedurally generated, you might imagine that a student encounters like the green door in one environment where it's very easy, it essentially just stumbles upon it. And then the in the next one, that's kind of a bit more challenging to reach. So we are still good on collecting goals. The other network it does is this grounding network. So the grounds, let's call that GD, the grounding network, it, it gets the initial state, and it proposes it checks which of the goals are even possible to reach. So these are two slightly different targets. The policy or let's call that Paul, well, okay, the policy network wants to propose goals which it finds challenging enough, right? For the student to fulfill the grounding network wants to check which of the goals are even reachable in the first place. And the the grounding network specifically is trained as this multi class, they say a multi label binary cross entropy loss, which I find to be a weird term, but okay, but essentially, it's given the initial state of an episode, we ask the grounding network to predict the first language description encountered along this trajectory, where t is the minimum t such that there is a description at all. So we're training, we're training the grounding network to predict the first language description term against all the other term in its encountered goals. This is kind of like a contrastive loss. So the that first goal is certainly reachable from the initial state. And we simply take all the other ones as kind of a negatives for that for that first one. And exactly. So the second one can be seen as noisily generating negative samples of start state and unachieved description. Now now, yeah, based on the set of descriptions known to the teacher. Now this seems a bit weird, right to train the grounding network like this. Like what about the second text description that was encountered? That's certainly reachable to know, at least I would, at least I would, I would guess so. Is this really necessary? Or maybe this here, maybe this here should be over goals that weren't encountered in the episode at all. Right. But this seems quite weird to only take the first encountered language description as a positive example of this grounding network. Further, and let's go into criticism right after we conclude here. They say to summarize the teacher training, training the teacher involves three steps, updating the running set of descriptions seen in the environment. That's collecting the goals essentially, learning the policy network based on whether the student achieved the goals proposed by the teacher. Okay, that's the same as the original Amigo. And third, learning the grounding network by predicting descriptions encountered from initial states. Okay, well, the this description here I can agree with. I don't I just don't see why only the first is taken as the as the positive sample. So what what are we doing right here? And why? What I find weird is that this grounding network has to exist at all. In the original description, I don't know if these things are generated. If these certainly all the coordinates exist right somewhere, but they're not necessarily reachable either. For the original Amigo, it seems weird that the policy network itself with whose goal it is to propose a goal that is just outside of the reach essentially of the student couldn't itself make the determination of whether a state is reachable at all, because the original Amigo network seems to be perfectly capable of making that determination for a set of coordinates, right? So it might you know, there is a difference in that the something that go to the green door, there might be not a green door at all in the environment. But it seems it seems a bit weird to split this stuff up into different into different networks. And it tells me maybe they tried it first, and that didn't work. So they had to throw in kind of another loss, which is is kind of a bit just a bit annoying. But you know, if it works with the extra loss, then okay. Here you can see again, we have the Amigo teacher first that's the grounding network, what is even possible in this environment, then it that is related to the policy network or multiplied by the output of the policy network. Policy network predicts goals that the student in its current state could reach but not under the threshold. All the while we add new goals, we train the grounding network on states that were actually reached during what language was achieved during the episodes, we take the other ones as negatives. And then lastly, the policy network is trained like Amigo. Now there is a typo here, I believe, I believe, because here it says the reward is given if the goal is achieved in less than t star steps. But I believe it should be more. I believe this should be more. Because that's what it says in the text. Yeah, so that's that. Yeah, I don't know why by the split. So the important difference as well is that the policy network is trained essentially with reinforcement learning, right? It's a it's a I guess an actor critic framework. And it's trained on the action that it actually output like in classic reinforcement learning fashion. Yet, the grounding network seems to be more achieved in a classic supervised sense, just as an online classifier. I'm not sure if they have done ablations. I haven't seen the ablation of what the El Amigo does without the grounding network. But it would be interesting to see the second. So here you can see how they add language, right? They add language by essentially replacing that teacher student relationship where the teacher proposes goals in coordinate. Now the teacher proposes goals in language. So that's the novelty here. The other one, the other algorithm is this novelty algorithm. So the novelty algorithm is a little bit different. It defines intrinsic reward to be the difference in novelty between a state and the previous state. So there's this notion of novelty. And we're not going to take that as as itself. Like we're not going to take the novelty and and and give the agent reward simply for achieving whatever we call novelty, right? And we can define novelty in whatever way we choose. What we do is we we give the reward if the agent transitions from a state of low novelty to a state of high novelty. And so that's the that's this thing right here. The max with zero is so that this cannot be negative. So we don't penalize going from high novelty states to low novelty states, because, you know, sometimes that is necessary. And we also only give that reward if a state is encountered for the first time. So here the agent is encouraged to find new states because it only gets rewards when it encounters new states. And it is especially encountered to find to find new states that are a significant increase in novelty from the previous states. This is this is one, I guess one way. What this avoids, I guess, is to get stuck in this loop. Yeah, let's say it's like you're in you're in an environment, right? And you're in an environment. And then here is like a random, just some random thing. People usually they they say there is a TV with static on like just kind of like or there's a bunch of leaves flowing around or something like this. And the agent that is just going for novelty would just indefinitely stare at it. And this prevents it because whatever you call novelty, if you call this novel, like a TV with static, because it's essentially a random signal, so it's super duper novel. However, you wouldn't get a reward for consecutively looking at the TV because you would already be in an equally novel state going to a new novel state. And that will give you no reward at all. So you're encouraged actually to go away from the TV go somewhere else where you can transition from a low novelty to a single high novelty state. All right, so yeah, what they say is in the first term, the n is the novelty that this quantity describes the difference in novelty between successive stage which is clicked larger than zero, this written a little bit weird. This quantity here refers to the first term, not to this thing right here. This thing is just a an explanation of what's in the term. So n is the novelty, and the reward is the difference in novelty. The second term, right only if we encounter it for the first time. And how does this thing, how does this thing track novelty? This is an interesting concept. How do we do know like how do we know if a state is novel? Because it is sufficient, they say to track exact state visitation counts. But obviously, as soon as the environment gets larger and a bit more complex, this is not possible anymore. So what do we do? We use this random network distillation. And I have to say I have never heard of this. And that seems quite smart. So what we do is we have a state again, so your agent is here, there is a bunch of walls and so on. What we do is we, we have a random neural network. Now that's always the same, but it is essentially essentially random. So we take the state, we feed it through the random neural network, we get out some vector, just some vector, because it's randomly initialized fixed neural network, it's going to be some kind of embedding of that, not a useful one, but just some sort of an embedding. And then what we do is we train a what what do they call it, we train an estate embedding network. So let's call that E, we train embedding. Again, this one takes this in, and it tries to predict this vector, right, tries to predict it. Now, obviously, it doesn't it can't see the weights of this neural network. Otherwise, this would be quite useless. But it tries to predict this vector. And it is trained. So the E is trained with back propagation, while the blue one is fixed. Now the logic here is that if I encounter a new state, right, so here's my new state, agent is here, there's just one wall here, there's like a door here. I put it through both loops, I put it through both of these new color, I put it through Hey, yo, I put it through this one, and I put it through this one. And then I get a vector here, and I get a vector here, I look at the error between the two, right? So what's what's the difference? If the error is small, I can safely assume that I have seen states like this before. Because if the error is small, it means that this thing has learned to match this thing for some kind of similar state, right? We know that neural networks generalize well if they have training data in the same vicinity of the data that you want to test on. Therefore, if the states are quite close, that means the outputs are quite close, that's a property of random neural networks. If you don't change the states much, it depends a little bit on parameterization. But essentially, if you change the input a little bit, the neural networks output will change a little bit. And therefore, if you've encountered states like this before, this E would be trained on those states would actually learn to match the blue fixed networks output. And therefore, the distance here would be small. However, if the state is super novel, that would not have been like anything in the training data. And therefore, this E network would make a large mistake when trying to predict the vector and from that mistake right here, because that's you have that at inference time, right? You can determine whether something is novel, there's a bunch of caveats. But since this paper isn't about novelty itself, I'm not gonna I'm going to reserve that for another time. So what do we do it to add language? That's this paper now, we add an additional exploration bonus based on novelty defined according to the natural language description of states. So again, we it is simply a repetition of the formula, we have some sort of a notion of novelty of a linguistic description. And we give the reward if the novelty of the new state is higher than novelty of the old state for whatever definition, and only the first time we encounter it. So they say nl is the novelty of the description l, as measured by a separately parameterized random network distillation network encoding the description. So presumably, other than inputting states, now every state also has a language description. So language description here, language description here, we have a separate network that a separate random network that we can put them through. And we can, we also have a separate embedding network, let's call that EL, the language embedding network. And we do the exact same thing with the language as we did with the states themselves. We try to train this EL in order to predict to match the predictions of the random network. If at inference time, the two match closely, we assume that this is like something we've seen in the training data, and otherwise, it's novel. So here you can see, they say we keep the original exploration bonus as language rewards may be sparse. They, they add both the intrinsic reward is the original one, that is just about the state, and the new one with a hyper parameter. And here, I think it becomes clear what, for me, the biggest criticism of this paper is. And that, I think, so they make the point that well, you know, language helps. And if you if you look at the experiments, they say, linguistic exploration outperforms non linguistic exploration. That's one of their experimental findings. You can look at the results, although the confidence intervals like this is just reinforcement learning. But yo, you had to work hard to make those, you know, to make these overall intervals not not overlap. That that is, you know, good job. But still, the noise in these environments is quite significant. And linguistic exploration excels in larger environments, which you can imagine, right, because in larger environments, they might be also more complex environments. And therefore, just state abstractions themselves might not be the best one. But my criticism here is that essentially, they add extra data, right? So it's not like linguistic exploration outperforms non linguistic exploration. It's Hey, the environment actually has this data right here. And no one without this one, no one's used that. So people just have used the image or whatnot, and the actions and the rewards. And there's this extra data. What if we use this extra data? Oh, we get better. Wow. And the data is obviously very good because it's made by humans and the game creators have essentially so the game creators know which states are equal, right? They code the game, and in the same vein, they produce these language descriptions. So the language descriptions are almost like a little bit of a view into the internal state of the game code itself. Even if that weren't the case, language obviously is quite powerful. But I get their argument that, you know, language gives you abstraction, yada, yada, yada, and so on. However, I think the gains here aren't language is better than, you know, not language, because I don't think it's necessarily a fair comparison. It is, you know, adding more stuff, adding more information, especially really good, really high quality information like they have is better than non not adding that information. Now obviously, it matters what they do with the information. But yeah, I think a lot of the gains simply come from the fact that they add something on top. So not to say like they, for example, in El Amigo, they drop the original teacher, right? But in this, in this in this novel D, they don't even drop the original intrinsic exploration. Yeah, so, you know, it's essentially really extra data that they add. What is interesting is that they analyze the curricula that emerge, right? It's given that its language you can you have a pretty good idea of what's happening over time. And they have these nice analyses right here, where for example, first, the teacher proposes open the door before it proposes open the color door. So see here is a variable that holds the color. So you can see that the teacher first proposes the easier goal of opening any door, and then it proposes a lot of opening the opening color doors, it then discovers keys going to the keys picking up keys, then going next to the door with the key. And after it goes through the door, it picks up the ball, which is the final the final goal. So you can see clearly that as the training progresses, the teacher gives more and more complex goals. And that is is kind of true is true for El Amigo. And this novel D, it is not that true in all the environments for the for the net hack environment, I believe. It's a little bit more they call it a little bit more exploratory in that it it just tries to explore a lot of stuff, which is also good, right? That does, it doesn't need to be progressive, right? As long as the teacher encourages the student to, you know, do this. And now okay, now you're really good at that. So I can't essentially propose that anymore, because you'll you'll fulfill it in less than the threshold time steps. Now, you know, do something else. Now do something else. And do something else. And these aren't the descriptions, right? It's this these are these are meant to be descriptions, not instructions. So this here, I guess is a is a better again a better example. So you want to reach a state that has the description of there is a staircase up here, right? So you just tell the student please reach any state with that description. And you can see how this develops, which is pretty cool. The last thing they do is something that I also find very, very interesting in that even though right, even though as far as I understand, and I think they say this somewhere, they don't use pre trained language models or anything like this in here. They do obviously output language and so on. So they need some sort of language model, but they don't use they don't make use of any pre training on any external data or anything like this. Yet still, the semantics of the language seem to be captured a little bit. For example, they do this experiment where they replace all the language goals with unique identifiers. So go to the red door would just become token one, go to the blue door would become token two. So now there is no shared substrings. So the model cannot generalize from this go to the door construction and sort of generalize the skills or generalize the reachability estimate of the goal. The result is one whole course performed quite competitively, which is good, right? So that lends more credence to what I say, like this is just this is extra data. Then the second thing is the l Amigo is better able to exploit semantics with a more significant improvement in aggregate performance over the one hot goals in contrast to l novel D, which shows less of a difference. So at least one of the methods is actually able to exploit these semantics in the language. And that is a promising outlook. If we now want to go ahead and you know, use something like pre trained language models in these, or something like clip to even to even get the description out of the state itself, that would be that would be really cool or some sort of a some sort of a clip modified for reinforcement learning. So we don't need to rely on environments, which are which have this language description already built in, because very, very few do right. And it seems to be it seems to be quite hard to get, honestly, right, if we want to train a good model for that, that is that is challenging, right? If let's say Atari or so very challenging, you either need to collect labeled data for you know, describing Atari states, which itself is really hard. And if you let three humans do it, you're going to get three completely different descriptions. And at that point, we're going to need these large language models, because the large language models need to be able to tell, well, these two wildly different descriptions are actually meaning the same thing, right? And how much of a gain at that point is still left? When all this noise comes on top of the learned description models and of the inferring whether two language descriptions are the same or not, whether or not there's still an actual difference there to to like l Amigo and Amigo remains to be seen, right? This paper here uses a lot of oracles, right? To to get its data, which is which is fine for research, but it's not necessarily means that this is going to be a practical thing in the future. So yeah, they say this, though, they criticize themselves. I fairly well, I think, say they want to alleviate the restriction on Oracle language annotations, perhaps by using learned state description models. Yeah, exciting extension would be to propose abstract goals, which is also pretty cool. And again, something where large language models can come in and help pre trained ones even write, you don't even have to train them. And yeah, using pre trained. Well, okay, that's it's stuck in my mind from reading it the last time pre trained models to imbue semantics into the model beforehand, they say would also be pretty interesting among a lot of other things. They also criticize the noisiness and and so on. So that was it for the paper overview. Let me know what you think about this paper. I find it to be pretty interesting, and I think it's a really cool, cool idea. And if we can extend this to not use oracles, I would be super happy. And I think this essentially is how humans also learn a lot of times by talking about things, by talking about goals and so on. Language does provide a really good abstraction for these types of stuff. Yeah, let me know what you think in the comments. Leave a like if you do, and I'll see you around. Bye bye.
 Hi there, this is a comprehensive paper review on the paper Improving Intrinsic Exploration
 with Language Abstractions.
 This is a very cool paper because it combines a language and the information that is in
 language with reinforcement learning, specifically the problem of exploration.
 I don't want to tell you too much more right now because we're going to dive into the paper
 in just a bit.
 So this video will explain in detail what is in the paper, how the method works, what
 they're doing.
 So by the end of this video, you should have a really good idea of what's in the paper.
 In the next video published tomorrow, there's going to be an interview with the authors
 of the paper, which is very, very cool.
 It's super valuable, and I was very happy to host this interview.
 So I hope you draw some value out of either one of these videos.
 Hopefully both.
 As always, thank you very much for watching.
 Thanks to everyone who likes and comments and supports in any way.
 It's really cool to be able to do these things.
 And I'll see you around.
 Bye bye.
 Hi there.
 Today, we're looking at Improving Intrinsic Exploration with Language Abstractions by
 researchers of Stanford University, University of Washington, Meta AI and University College,
 London.
 This paper on a high level uses language to facilitate intrinsic exploration.
 That is when in the face of a very sparse environment, a reinforcement learning agent
 has to come up with its own goals in order to make progress.
 So the intrinsic exploration or intrinsic motivation refers to the fact that the there's
 an additional reward that we give to the agent just for attaining, let's say new states novel
 things in the environment.
 Now it turns out that that's not super duper easy, because not all new things are equal.
 And especially, let's say there is a random component in the environment, then, you know,
 that's going to be new every time, yet it might not be interesting.
 So how you go about this is quite a challenge.
 It's clear that we need something like this in sparse, sparse rewards environment.
 But how exactly to do it is still still challenging.
 This paper adds language to the mix and argues that language descriptions could be one such
 source of novel of indicators of novel states.
 So we're going to go through the paper, let me know what you think in the comments, definitely.
 And yeah, let's dive in.
 So they say they want to solve these these complex long horizon tasks with sparse rewards.
 And as I already said, that is not really a picnic for reinforcement learning agents.
 Usually those need very tight, very dense rewards in order to work.
 And that's why we give these intrinsic rewards for exploration.
 And that is encouraging the agent even in the absence of rewards to go out and explore
 things and do new things.
 And we hope that through the exploration, at some point, it will learn the skills or
 it would encounter something that will that will actually give true reward.
 So they correctly claim there is a design choice on how to measure exploration and a
 an implicit like a common answer that the agent should be rewarded for attaining novel
 states in the environment.
 But that is, as we already said, quite difficult to actually implement.
 For example, states can look cosmetically different, but have the same underlying semantics
 and thus not be truly novel.
 So the two fundamental challenges for intrinsic exploration they they they list here is first,
 how can we reward true progress in the environment over meaningless exploration?
 Second, how can we tell when a state is not just superficially but semantically novel?
 And that's where they add in language.
 They say, well, if we had language describing the states, then certainly, for example, here,
 we have language that describes the state.
 Here the the language description says in what direction, indicating that you can go
 in a couple of directions or do something in a couple of directions, you see here a
 crystal wand, that means there's something to pick up.
 So when you don't have this message, that might be an indication that the state is meaningfully
 different, namely, it doesn't have the crystal wand.
 So as you can see, these authors imagine that if we had a language description of the environment,
 that could give us an indication of when something is novel, and when something is just the same
 but looks a little bit different.
 They say language obviously has strong priors over the features and behaviors needed for
 meaningful interaction and skill acquisition.
 That's just a matter of fact that language has been developed to communicate things that
 are useful to humans.
 And they also say correctly that you can describe with language very particular things such
 as move left or very abstract things like acquire the amulet and defeat the wizard.
 Although one of the abstraction here comes from the end, but still defeat the wizard
 is a very, very abstract thing.
 Now, as we already said, what they're going to do here is they're going to look at these
 environments, at these reinforcement learning environments.
 So there's mini grid on the left.
 And in mini grid, I believe the agent here, you're that that's the the red triangle.
 And the agent is supposed to I think, go to the keys, get the keys, open the doors and
 eventually get the final reward that is somewhere on the map.
 These are procedurally generated.
 So it always kind of looks different.
 And that's one challenge because if you have to make sequences of actions like go over
 here, get that key, go to the door, and then go further and get the reward, that is a sequence
 of actions that is unlikely to happen by by chance, right to stumble over the key and
 to stumble over the door and to stumble over the reward.
 You know, the amount of times you're going to try randomly until that's the case is is
 staggering.
 And therefore something like Q learning, which just requires on random exploration is going
 to almost certainly fail right here.
 But this is one of the environments, which is a challenging environment that they pick
 up.
 And that has these language descriptions, or I think in this one, they add the language
 descriptions.
 But in any case, this is not about language models or anything like this.
 They assume that they have a function which they call L, the language annotator that takes
 in a state takes in and gives you the description.
 And they just assume they have an oracle that does that.
 So for the environments they do test, they actually have that.
 And so in minihack here, this is even part of the game, right?
 In minihack, you will always get a message like this to every step that you do in almost
 most of them, most of these states have such a description available.
 So again, there's this function L, which in this case is just the game engine, it takes
 in a state and it gives you back the description.
 So if you, you could guess here that we might learn this language descriptor, right?
 We might even initialize it with a language model, we can use something like clip or something
 like this.
 This is certainly in the future work, they list this, but not here.
 Here we assume we have these oracle.
 Now what can we do once we have such a language description?
 Well, we can use it for exploration.
 So there is a little bit of mathy math right here, which we're going to skip.
 Essentially this just discusses that yeah, they have this annotator L that produces these
 natural language descriptions and they add an intrinsic reward to this.
 And now we're going to look at what the intrinsic reward is.
 So they're going to take two different in like two different algorithms that are already
 made for intrinsic motivation, and they're going to augment them with language.
 The reasoning behind it is that those two algorithms, the one is called Amigo, the other
 one we'll get to in a second, they're already kind of state of the art in this domain.
 So what they say is if we add language to those, and we can get a better result, then
 that kind of shows the usefulness of language of the language descriptions.
 So we're going to look at these algorithms briefly remember these algorithms aren't by
 this paper, this paper is how to add language to them.
 So Amigo, the adversarially motivated intrinsic goals trains a student and a teacher.
 So there is a teacher that generates goals.
 And then the student is just a goal conditioned policy.
 The goal is, as we said, provided by the teacher.
 So the student is the real reinforcement learner, but the student is simply conditioned on some
 goal that's provided by the teacher.
 It is not it doesn't try to solve the actual problem.
 It solves the goal that the teacher gives it.
 I mean, it probably gets reward when it accidentally also fulfills the true reward goal, but it
 does get intrinsic reward when it fulfills the goal set by the teacher.
 Now the goal set by the teacher.
 That's the trick obviously right here, the teacher policy is quite smart.
 The teacher policy takes in the state of the student.
 So it looks at you know, where is the student.
 And it needs to now decide what do I do?
 What kind of goal do I give the student on the top left here you see this in in in this
 mini grid environment.
 The teacher is this network or this function right here.
 It gives a coordinates that the student has to get to.
 And then these coordinates as you can see there.
 I'm not sure if those are the actual coordinates.
 But whenever the student actually reaches them, so it provides the goal to the student
 when the student reaches it, it gets reward.
 So that's it.
 There is also a notion of a difficulty threshold.
 That difficulty threshold is it increases during training.
 So the idea is that at the beginning, the teacher wants to suggest kind of easy goals.
 And then as time progresses, the teacher has to learn essentially how to make the goals
 harder and harder.
 And by making the goals harder, the student essentially has a curriculum of harder to
 reach skills.
 So the teacher should kind of learn to propose more hard goals.
 So I think that the work here is definitely done mostly by this teacher network and the
 challenges.
 In any case, there is this difficulty threshold.
 This difficulty threshold is increased linearly during training.
 And the student, no, sorry, the teacher, the teacher is given a positive reward if it proposes
 goals that take the student more than T star time steps to complete and a negative reward
 for goals that are completed sooner or never completed within the finite time horizon.
 So you also can't go impossible or it can't go too hard.
 It needs to go exactly as hard that the student reaches the goal, which means even even if
 it's a possible goal, it can't go too hard for the current student.
 It needs to essentially propose goals that are just outside the outside the abilities
 of the current student.
 So that that zone of proximal development is kind of formalized in this teacher.
 That's Amigo.
 The other so how do we add?
 How do we add language to that?
 We saw that usually the teacher supposes or proposes coordinates for the student to get
 to.
 Now if we have language descriptions for every state, so every state the student finds itself
 in, there is a language description.
 The teacher can simply output a language description of a state.
 In this case, these are formulated as as kind of instructions.
 But remember, they are just descriptions as far as I can tell of of the state.
 It is more evident in the mini hack environment.
 So these these are just descriptions of the state, whatever the game would output if you're
 in this state.
 And the teacher simply proposes these.
 So it just says, well, here is a goal for you.
 Try to get to a state where the language descriptor outputs that.
 So that those are the goals that the teacher can choose.
 Where are we?
 Yeah.
 So we don't have x y goals, but we have natural language goals.
 The student is rewarded if it reaches a state with a natural language description that the
 teacher outputs.
 Easy enough.
 So how does the teacher do this?
 It selects goals from the set of possible language descriptions in the environment.
 Now, initially, these are unknown.
 So the teacher doesn't know yet what the environment has in store.
 Because again, we don't assume that say extra information, we need to get out everything
 of the environment.
 Therefore, as we go through the environment, we collect more and more of these goals.
 And these are the goals that the teacher can choose.
 The teacher maintains a running set of goals that is updated as the student encounters
 new state descriptions.
 The teacher has this move to language, they say creates a challenge.
 Not only must the teacher choose which goal to give to the student, it must also determine
 which goals are achievable.
 And that's why they train two different networks.
 There is a policy network, which produces the distribution over goals given a student
 state and a grounding network, which predicts the probability that a goal is likely to be
 achieved in the first place.
 So remember, these environments, they're procedurally generated.
 So every time the student is every new episode, I believe that's how it works.
 The student is placed in some environment that it has essentially never seen before.
 So now the teacher takes that in, and it produces two things, it looks it looks at this environment,
 produces two things from the set of goals that it has, it picks one that it wants to
 propose.
 That needs to be right so for it cannot always do the same.
 That's the interesting part right here.
 So if the green door is over here, go to the green door might be very easy in one environment,
 but very hard in the other environment.
 When I first read this, I thought, well, if you know, if the teacher knows no goals at
 the beginning, and it only collects these goals that the students student encounters
 over the course of the episode, we're still kind of relying on random exploration of the
 student right because any goal it hasn't achieved yet cannot be proposed.
 Whereas in the original x y coordinate, I can, I believe at least I can just propose
 any x y coordinate like get to that.
 However, since this is procedurally generated, you might imagine that a student encounters
 like the green door in one environment where it's very easy, it essentially just stumbles
 upon it.
 And then the in the next one, that's kind of a bit more challenging to reach.
 So we are still good on collecting goals.
 The other network it does is this grounding network.
 So the grounds, let's call that GD, the grounding network, it, it gets the initial state, and
 it proposes it checks which of the goals are even possible to reach.
 So these are two slightly different targets.
 The policy or let's call that Paul, well, okay, the policy network wants to propose
 goals which it finds challenging enough, right?
 For the student to fulfill the grounding network wants to check which of the goals are even
 reachable in the first place.
 And the the grounding network specifically is trained as this multi class, they say a
 multi label binary cross entropy loss, which I find to be a weird term, but okay, but essentially,
 it's given the initial state of an episode, we ask the grounding network to predict the
 first language description encountered along this trajectory, where t is the minimum t
 such that there is a description at all.
 So we're training, we're training the grounding network to predict the first language description
 term against all the other term in its encountered goals.
 This is kind of like a contrastive loss.
 So the that first goal is certainly reachable from the initial state.
 And we simply take all the other ones as kind of a negatives for that for that first one.
 And exactly.
 So the second one can be seen as noisily generating negative samples of start state and unachieved
 description.
 Now now, yeah, based on the set of descriptions known to the teacher.
 Now this seems a bit weird, right to train the grounding network like this.
 Like what about the second text description that was encountered?
 That's certainly reachable to know, at least I would, at least I would, I would guess so.
 Is this really necessary?
 Or maybe this here, maybe this here should be over goals that weren't encountered in
 the episode at all.
 Right.
 But this seems quite weird to only take the first encountered language description as
 a positive example of this grounding network.
 Further, and let's go into criticism right after we conclude here.
 They say to summarize the teacher training, training the teacher involves three steps,
 updating the running set of descriptions seen in the environment.
 That's collecting the goals essentially, learning the policy network based on whether the student
 achieved the goals proposed by the teacher.
 Okay, that's the same as the original Amigo.
 And third, learning the grounding network by predicting descriptions encountered from
 initial states.
 Okay, well, the this description here I can agree with.
 I don't I just don't see why only the first is taken as the as the positive sample.
 So what what are we doing right here?
 And why?
 What I find weird is that this grounding network has to exist at all.
 In the original description, I don't know if these things are generated.
 If these certainly all the coordinates exist right somewhere, but they're not necessarily
 reachable either.
 For the original Amigo, it seems weird that the policy network itself with whose goal
 it is to propose a goal that is just outside of the reach essentially of the student couldn't
 itself make the determination of whether a state is reachable at all, because the original
 Amigo network seems to be perfectly capable of making that determination for a set of
 coordinates, right?
 So it might you know, there is a difference in that the something that go to the green
 door, there might be not a green door at all in the environment.
 But it seems it seems a bit weird to split this stuff up into different into different
 networks.
 And it tells me maybe they tried it first, and that didn't work.
 So they had to throw in kind of another loss, which is is kind of a bit just a bit annoying.
 But you know, if it works with the extra loss, then okay.
 Here you can see again, we have the Amigo teacher first that's the grounding network,
 what is even possible in this environment, then it that is related to the policy network
 or multiplied by the output of the policy network.
 Policy network predicts goals that the student in its current state could reach but not under
 the threshold.
 All the while we add new goals, we train the grounding network on states that were actually
 reached during what language was achieved during the episodes, we take the other ones
 as negatives.
 And then lastly, the policy network is trained like Amigo.
 Now there is a typo here, I believe, I believe, because here it says the reward is given if
 the goal is achieved in less than t star steps.
 But I believe it should be more.
 I believe this should be more.
 Because that's what it says in the text.
 Yeah, so that's that.
 Yeah, I don't know why by the split.
 So the important difference as well is that the policy network is trained essentially
 with reinforcement learning, right?
 It's a it's a I guess an actor critic framework.
 And it's trained on the action that it actually output like in classic reinforcement learning
 fashion.
 Yet, the grounding network seems to be more achieved in a classic supervised sense, just
 as an online classifier.
 I'm not sure if they have done ablations.
 I haven't seen the ablation of what the El Amigo does without the grounding network.
 But it would be interesting to see the second.
 So here you can see how they add language, right?
 They add language by essentially replacing that teacher student relationship where the
 teacher proposes goals in coordinate.
 Now the teacher proposes goals in language.
 So that's the novelty here.
 The other one, the other algorithm is this novelty algorithm.
 So the novelty algorithm is a little bit different.
 It defines intrinsic reward to be the difference in novelty between a state and the previous
 state.
 So there's this notion of novelty.
 And we're not going to take that as as itself.
 Like we're not going to take the novelty and and and give the agent reward simply for achieving
 whatever we call novelty, right?
 And we can define novelty in whatever way we choose.
 What we do is we we give the reward if the agent transitions from a state of low novelty
 to a state of high novelty.
 And so that's the that's this thing right here.
 The max with zero is so that this cannot be negative.
 So we don't penalize going from high novelty states to low novelty states, because, you
 know, sometimes that is necessary.
 And we also only give that reward if a state is encountered for the first time.
 So here the agent is encouraged to find new states because it only gets rewards when it
 encounters new states.
 And it is especially encountered to find to find new states that are a significant increase
 in novelty from the previous states.
 This is this is one, I guess one way.
 What this avoids, I guess, is to get stuck in this loop.
 Yeah, let's say it's like you're in you're in an environment, right?
 And you're in an environment.
 And then here is like a random, just some random thing.
 People usually they they say there is a TV with static on like just kind of like or there's
 a bunch of leaves flowing around or something like this.
 And the agent that is just going for novelty would just indefinitely stare at it.
 And this prevents it because whatever you call novelty, if you call this novel, like
 a TV with static, because it's essentially a random signal, so it's super duper novel.
 However, you wouldn't get a reward for consecutively looking at the TV because you would already
 be in an equally novel state going to a new novel state.
 And that will give you no reward at all.
 So you're encouraged actually to go away from the TV go somewhere else where you can transition
 from a low novelty to a single high novelty state.
 All right, so yeah, what they say is in the first term, the n is the novelty that this
 quantity describes the difference in novelty between successive stage which is clicked
 larger than zero, this written a little bit weird.
 This quantity here refers to the first term, not to this thing right here.
 This thing is just a an explanation of what's in the term.
 So n is the novelty, and the reward is the difference in novelty.
 The second term, right only if we encounter it for the first time.
 And how does this thing, how does this thing track novelty?
 This is an interesting concept.
 How do we do know like how do we know if a state is novel?
 Because it is sufficient, they say to track exact state visitation counts.
 But obviously, as soon as the environment gets larger and a bit more complex, this is
 not possible anymore.
 So what do we do?
 We use this random network distillation.
 And I have to say I have never heard of this.
 And that seems quite smart.
 So what we do is we have a state again, so your agent is here, there is a bunch of walls
 and so on.
 What we do is we, we have a random neural network.
 Now that's always the same, but it is essentially essentially random.
 So we take the state, we feed it through the random neural network, we get out some vector,
 just some vector, because it's randomly initialized fixed neural network, it's going to be some
 kind of embedding of that, not a useful one, but just some sort of an embedding.
 And then what we do is we train a what what do they call it, we train an estate embedding
 network.
 So let's call that E, we train embedding.
 Again, this one takes this in, and it tries to predict this vector, right, tries to predict
 it.
 Now, obviously, it doesn't it can't see the weights of this neural network.
 Otherwise, this would be quite useless.
 But it tries to predict this vector.
 And it is trained.
 So the E is trained with back propagation, while the blue one is fixed.
 Now the logic here is that if I encounter a new state, right, so here's my new state,
 agent is here, there's just one wall here, there's like a door here.
 I put it through both loops, I put it through both of these new color, I put it through
 Hey, yo, I put it through this one, and I put it through this one.
 And then I get a vector here, and I get a vector here, I look at the error between the
 two, right?
 So what's what's the difference?
 If the error is small, I can safely assume that I have seen states like this before.
 Because if the error is small, it means that this thing has learned to match this thing
 for some kind of similar state, right?
 We know that neural networks generalize well if they have training data in the same vicinity
 of the data that you want to test on.
 Therefore, if the states are quite close, that means the outputs are quite close, that's
 a property of random neural networks.
 If you don't change the states much, it depends a little bit on parameterization.
 But essentially, if you change the input a little bit, the neural networks output will
 change a little bit.
 And therefore, if you've encountered states like this before, this E would be trained
 on those states would actually learn to match the blue fixed networks output.
 And therefore, the distance here would be small.
 However, if the state is super novel, that would not have been like anything in the training
 data.
 And therefore, this E network would make a large mistake when trying to predict the vector
 and from that mistake right here, because that's you have that at inference time, right?
 You can determine whether something is novel, there's a bunch of caveats.
 But since this paper isn't about novelty itself, I'm not gonna I'm going to reserve that for
 another time.
 So what do we do it to add language?
 That's this paper now, we add an additional exploration bonus based on novelty defined
 according to the natural language description of states.
 So again, we it is simply a repetition of the formula, we have some sort of a notion
 of novelty of a linguistic description.
 And we give the reward if the novelty of the new state is higher than novelty of the old
 state for whatever definition, and only the first time we encounter it.
 So they say nl is the novelty of the description l, as measured by a separately parameterized
 random network distillation network encoding the description.
 So presumably, other than inputting states, now every state also has a language description.
 So language description here, language description here, we have a separate network that a separate
 random network that we can put them through.
 And we can, we also have a separate embedding network, let's call that EL, the language embedding
 network.
 And we do the exact same thing with the language as we did with the states themselves.
 We try to train this EL in order to predict to match the predictions of the random network.
 If at inference time, the two match closely, we assume that this is like something we've
 seen in the training data, and otherwise, it's novel.
 So here you can see, they say we keep the original exploration bonus as language rewards
 may be sparse.
 They, they add both the intrinsic reward is the original one, that is just about the state,
 and the new one with a hyper parameter.
 And here, I think it becomes clear what, for me, the biggest criticism of this paper is.
 And that, I think, so they make the point that well, you know, language helps.
 And if you if you look at the experiments, they say, linguistic exploration outperforms
 non linguistic exploration.
 That's one of their experimental findings.
 You can look at the results, although the confidence intervals like this is just reinforcement
 learning.
 But yo, you had to work hard to make those, you know, to make these overall intervals
 not not overlap.
 That that is, you know, good job.
 But still, the noise in these environments is quite significant.
 And linguistic exploration excels in larger environments, which you can imagine, right,
 because in larger environments, they might be also more complex environments.
 And therefore, just state abstractions themselves might not be the best one.
 But my criticism here is that essentially, they add extra data, right?
 So it's not like linguistic exploration outperforms non linguistic exploration.
 It's Hey, the environment actually has this data right here.
 And no one without this one, no one's used that.
 So people just have used the image or whatnot, and the actions and the rewards.
 And there's this extra data.
 What if we use this extra data?
 Oh, we get better.
 Wow.
 And the data is obviously very good because it's made by humans and the game creators
 have essentially so the game creators know which states are equal, right?
 They code the game, and in the same vein, they produce these language descriptions.
 So the language descriptions are almost like a little bit of a view into the internal state
 of the game code itself.
 Even if that weren't the case, language obviously is quite powerful.
 But I get their argument that, you know, language gives you abstraction, yada, yada, yada, and
 so on.
 However, I think the gains here aren't language is better than, you know, not language, because
 I don't think it's necessarily a fair comparison.
 It is, you know, adding more stuff, adding more information, especially really good,
 really high quality information like they have is better than non not adding that information.
 Now obviously, it matters what they do with the information.
 But yeah, I think a lot of the gains simply come from the fact that they add something
 on top.
 So not to say like they, for example, in El Amigo, they drop the original teacher, right?
 But in this, in this in this novel D, they don't even drop the original intrinsic exploration.
 Yeah, so, you know, it's essentially really extra data that they add.
 What is interesting is that they analyze the curricula that emerge, right?
 It's given that its language you can you have a pretty good idea of what's happening over
 time.
 And they have these nice analyses right here, where for example, first, the teacher proposes
 open the door before it proposes open the color door.
 So see here is a variable that holds the color.
 So you can see that the teacher first proposes the easier goal of opening any door, and then
 it proposes a lot of opening the opening color doors, it then discovers keys going to the
 keys picking up keys, then going next to the door with the key.
 And after it goes through the door, it picks up the ball, which is the final the final
 goal.
 So you can see clearly that as the training progresses, the teacher gives more and more
 complex goals.
 And that is is kind of true is true for El Amigo.
 And this novel D, it is not that true in all the environments for the for the net hack
 environment, I believe.
 It's a little bit more they call it a little bit more exploratory in that it it just tries
 to explore a lot of stuff, which is also good, right?
 That does, it doesn't need to be progressive, right?
 As long as the teacher encourages the student to, you know, do this.
 And now okay, now you're really good at that.
 So I can't essentially propose that anymore, because you'll you'll fulfill it in less than
 the threshold time steps.
 Now, you know, do something else.
 Now do something else.
 And do something else.
 And these aren't the descriptions, right?
 It's this these are these are meant to be descriptions, not instructions.
 So this here, I guess is a is a better again a better example.
 So you want to reach a state that has the description of there is a staircase up here,
 right?
 So you just tell the student please reach any state with that description.
 And you can see how this develops, which is pretty cool.
 The last thing they do is something that I also find very, very interesting in that even
 though right, even though as far as I understand, and I think they say this somewhere, they
 don't use pre trained language models or anything like this in here.
 They do obviously output language and so on.
 So they need some sort of language model, but they don't use they don't make use of
 any pre training on any external data or anything like this.
 Yet still, the semantics of the language seem to be captured a little bit.
 For example, they do this experiment where they replace all the language goals with unique
 identifiers.
 So go to the red door would just become token one, go to the blue door would become token
 two.
 So now there is no shared substrings.
 So the model cannot generalize from this go to the door construction and sort of generalize
 the skills or generalize the reachability estimate of the goal.
 The result is one whole course performed quite competitively, which is good, right?
 So that lends more credence to what I say, like this is just this is extra data.
 Then the second thing is the l Amigo is better able to exploit semantics with a more significant
 improvement in aggregate performance over the one hot goals in contrast to l novel D,
 which shows less of a difference.
 So at least one of the methods is actually able to exploit these semantics in the language.
 And that is a promising outlook.
 If we now want to go ahead and you know, use something like pre trained language models
 in these, or something like clip to even to even get the description out of the state
 itself, that would be that would be really cool or some sort of a some sort of a clip
 modified for reinforcement learning.
 So we don't need to rely on environments, which are which have this language description
 already built in, because very, very few do right.
 And it seems to be it seems to be quite hard to get, honestly, right, if we want to train
 a good model for that, that is that is challenging, right?
 If let's say Atari or so very challenging, you either need to collect labeled data for
 you know, describing Atari states, which itself is really hard.
 And if you let three humans do it, you're going to get three completely different descriptions.
 And at that point, we're going to need these large language models, because the large language
 models need to be able to tell, well, these two wildly different descriptions are actually
 meaning the same thing, right?
 And how much of a gain at that point is still left?
 When all this noise comes on top of the learned description models and of the inferring whether
 two language descriptions are the same or not, whether or not there's still an actual
 difference there to to like l Amigo and Amigo remains to be seen, right?
 This paper here uses a lot of oracles, right?
 To to get its data, which is which is fine for research, but it's not necessarily means
 that this is going to be a practical thing in the future.
 So yeah, they say this, though, they criticize themselves.
 I fairly well, I think, say they want to alleviate the restriction on Oracle language annotations,
 perhaps by using learned state description models.
 Yeah, exciting extension would be to propose abstract goals, which is also pretty cool.
 And again, something where large language models can come in and help pre trained ones
 even write, you don't even have to train them.
 And yeah, using pre trained.
 Well, okay, that's it's stuck in my mind from reading it the last time pre trained models
 to imbue semantics into the model beforehand, they say would also be pretty interesting
 among a lot of other things.
 They also criticize the noisiness and and so on.
 So that was it for the paper overview.
 Let me know what you think about this paper.
 I find it to be pretty interesting, and I think it's a really cool, cool idea.
 And if we can extend this to not use oracles, I would be super happy.
 And I think this essentially is how humans also learn a lot of times by talking about
 things, by talking about goals and so on.
 Language does provide a really good abstraction for these types of stuff.
 Yeah, let me know what you think in the comments.
 Leave a like if you do, and I'll see you around.
 Bye bye.