Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (+Author)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper natural language processing training data deep learning tutorial nlp gpt3 gpt 3 codex openai codex large language models gpt 3 planning zero-shot planning zero shot learning virtualhome virtual home bert bert model bert translation bert embedding pieter abbeel reinforcement learning human language learning
#gpt3 #embodied #planning

In this video: Paper explanation, followed by first author interview with Wenlong Huang.
Large language models contain extraordinary amounts of world knowledge that can be queried in various ways. But their output format is largely uncontrollable. This paper investigates the VirtualHome environment, which expects a particular set of actions, objects, and verbs to be used. Turns out, with proper techniques and only using pre-trained models (no fine-tuning), one can translate unstructured language model outputs into the structured grammar of the environment. This is potentially very useful anywhere where the models' world knowledge needs to be provided in a particular structured format.

OUTLINE:
0:00 - Intro & Overview
2:45 - The VirtualHome environment
6:25 - The problem of plan evaluation
8:40 - Contributions of this paper
16:40 - Start of interview
24:00 - How to use language models with environments?
34:00 - What does model size matter?
40:00 - How to fix the large models' outputs?
55:00 - Possible improvements to the translation procedure
59:00 - Why does Codex perform so well?
1:02:15 - Diving into experimental results
1:14:15 - Future outlook

Paper: https://arxiv.org/abs/2201.07207
Website: https://wenlong.page/language-planner/
Code: https://github.com/huangwl18/language-planner
Wenlong's Twitter: https://twitter.com/wenlong_huang

Abstract:
Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (e.g. "make breakfast"), to a chosen set of actionable steps (e.g. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards extracting actionable knowledge from language models. Website at this https URL

Authors: Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch


Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we're looking at language models as zero-shot planners, extracting actionable knowledge for embodied agents. And I'm going to interview the first author Wenlong Huang in a few minutes. So first there's an explanation of the paper, 10-15 minutes or so, I'm going to try to keep to it. And then we jump into the interview where we can discuss this paper at length. On a high level, this paper asks, can we use the knowledge that is inherent in large language models like GPT-3, or surprisingly, OpenAI's codecs in order to do planning in what they call embodied agents. Ultimately, it's going to be this environment right here. The, I don't even know what it's the virtual home environment. And it's about a virtual home, you have to fulfill some tasks like brushed your teeth, then the model has to come up with a sequence of steps that are admissible by the environment. So there's a level of admissibility of action, predefined actions that are admissible, the model has to come up with these actions in order to fulfill the task. The model is then rated based on executability and correctness of their plans. And it turns out that the larger the models get, as you can see right here, the less executable the plans become, which means that the actions they generate aren't admissible by the environment, probably because the models are more, let's say powerful, they can express themselves in more ways, they have different ideas of how to reach goals. However, the correctness, this is human evaluated of these models rise as they grow larger. So this gives you an indication that the large models seem to have quite a lot of knowledge. And we have to say these are not trained, the entire paper just works except for one baseline evaluation, just works with pre-trained models, they're not fine tuned at all on this environment right here. So what this paper does is it says, well, given that the larger the models get, the more correct their plans are, can we do something to fix the issue with the executability? To that, they develop this translation procedure right here. These are three specific improvements they do to the models. In order to get their executability up, you can see they sacrifice like a little bit of the correctness, but they do make the plans largely executable in the environment. And therefore, procedures like this could be applied in many different ways. It's not only about the virtual home environment and so on. It's essentially anywhere where you bring together the knowledge that is inherent in large language models with some sort of a domain specific language or a grammar or any anything like this, like where you have to transfer that knowledge into a new domain, but you don't want to train a model to do so. So we're going to see how they do it really briefly. First of all, the environment itself, as I already said, is this now this is visualized, although they never work, you know, actually in 3D, just a small correction here, because I messed this up. There are actually two versions of the virtual home environment. One is a Python version that focuses on the textual interaction with the environment. The other one is implemented in Unity and actually does work in 3D. The developers of the environment mostly focus on the Unity environment because it's more real. But as of yet, that has a subset of the actions available that the Python environment has. And the authors of the paper use the Python environment and the data set that comes along with that. We're going to go into this more in the interview. Stay tuned. They simply grab the data set of possible tasks, some tasks you can see right here, a task could be throw away paper, another task could be brush teeth, and there there'd be a sequence of steps. This environment is made by humans. So the tasks are made by humans. And then other humans have to come up with the steps that are admissible, admissible actions in this environment. There are, I believe, a number of objects that are defined, they're predefined. Yeah, so there are a number of objects, for example, living room, television, sofa, and so on. And there are a number of verbs. So walk, find, switch on, and so on. And not every verb object combination is possible. Some verbs have two objects and so on. But essentially, you combine the predefined verbs and the predefined objects, and then the state of the world changes. So the world keeps track of states, there are certain preconditions. For example, you can probably only sit on the sofa if you are in the vicinity of it. So you need to first find the sofa, you can only switch on the television. Similarly, if you have first found the television or walked to the television or something like this, if the television is in the living room, you first need to go to the living room, and so on. So there's a hidden kind of a state. But all of this is constructed. And we talked about this in the interview, like, what's the appropriate granularity of actions like this? And isn't this a major issue? But it is made all with the humans in the loop. So the data set is supposed to be kind of the most natural expression of these tasks, as split into steps that a human would come up with. So this is the grammar of the environment. And the language models, they don't know about this grammar. They're just language models. So what they do is they take something like GPT-3, and they make a prompt. Now the prompt, as you might know, in GPT-3, you have to give a prompt. So the prompt could just be like, here's the task, you know, blah, blah, blah, brush your teeth, then what's step one, right? And then GPT-3 will probably it will probably even generate step two and three and four. But it will probably not be according to the these actions in these templates, you can help this a little bit by putting a prompt up here. So the prompt they use is one, I believe one specific plan. So they have already like task up here, some task, and then some number of steps, so that the model kind of knows what is expected. We also talked about this in the interview, and this could potentially be improved by multiple, multiple prompts and so on. But in the baseline, they have one particular prompt. And then one of the improvements is actually to select a more optimal prompt. This is the basic setup. You have a goal in this environment with a fixed grammar, and you task, you input this right here to your language model, and the language model will spit out the plan. Now what do you do with the plan? The plan, you score, like how good is the plan? And they have two different scoring available. One is executability. And executability is just like, it's essentially parsability by the environment. So in executability, you ask yourself, can it be correctly parsed, which means that is the syntax according to the syntax of the environment. And they do have a little translation procedure, like a little heuristic translation procedure for the baseline in place, so that the language model probably can't get it exactly right. But they do sort of translate to the closest action there. But also one of the improvements is related to this. And then also does it satisfy the common sense constraints of the environment. And these would be programmed in like, for example, you can only pour yourself a glass of milk if you first open the fridge and grab the milk, this can be measured directly, what cannot be measured that well is correctness. So these models, they would come up with plans and independent of whether they're executable or not, they could be correct, right. And that's where they ask humans. So they use human evaluations, they conduct human evaluations in order to score the correctness of whatever these models output. So they give it to a human, ask the human, does this look like a sensible plan in order to brush your teeth, and the human would either say yes or no, when they do like ablations, and so on. They also use like longest common sub sequences between two programs and so on in order to not spend ginormous amounts of money on humans. But essentially, the correctness metric is a human metric. It's also interesting because you thought you could just execute like the plan in the environment and that give you like, does it succeed or not, but they say correctly that for a task like make breakfast, there's not really a defined end condition that you could program into the environment to give a reward. So it's more accurate to ask humans whether a plan is correct. As you might have guessed, this environment is very human centric, it's made by humans with humans in the loop and so on. It's supposed to really be sort of a representation of human tasks and human plans to human tasks. All right, so now we're going into the improvements. There are three distinct improvements they make. So if they just do this, if they just do what we've described so far, then the graph up here results, excluding the two models on the right, you can see the larger the models get, the higher their correctness, but the worse their executability. So now the thought is, can we change that? Can we raise the executability? And so this is the baseline right here, zero-shot planning via causal large language model, you put in a task as a prompt, and along with like the format you expect, which is this one right here, which is some other task from the data set, then you use the pre-trained language model like GPT-3 or something, and that will give you a plan. And that's it. So the next thing they do is they do what they call a translation model. So they introduce a second model, which is also pre-trained. And this is it's not trained on translation. It's just trained on masked large language modeling. So think of this like, this is just BERT. In fact, I believe they use sentence BERT, just pre-trained on English language. And what they do is they make a big vocabulary of all the admissible actions. So all the admissible actions would just be like any combination between any verb and any object that would actually go with that, that is admissible to this verb. So from this, they make like a giant list of all of the admissible actions. And then they embed that giant list. So they put this into some embedding space using the sentence BERT model pre-trained, right. And then whenever the large language model outputs something, they don't implement it into the plan directly. They first embed whatever the model outputs. Let's put this over here, they embed it, let's say that becomes this right here. Then they see what's the nearest neighbor of my admissible actions to this thing. And then they simply replace whatever the model output with the nearest neighbor. And they call that translation. So essentially, it translates from general natural language space into the space of the admissible actions or the grammar of the model. Now this has some problems on its own. For example, if the model outputs the compound actions. So if it says, for example, squeeze out the glob of lotion and put it in your mouth or so or on your face, I guess, then well, it's apply lotion, it's anywhere. Squeeze out the glob of lotion and put it on your skin. That would be still one action. Now which one would be the closest right here, there's going to be somewhere like squeeze out a bit of lotion and the other one is going to be like, put the lotion on your skin. Yet you only have one action like it's it's it's one line. So one action, it just contains like an and now the end might be easy to recognize, but there are other there are going to be other like compound actions. And this is going to be a problem here, because you just map one action to one admissible action. But in any case, doing this already helps a lot, even though there are still some problems to alleviate the rest of the problems. They have two more improvements. The first improvement they do is they say, well, if there is a compound action, we can still kind of alleviate that a little bit. So in the original method, what they did is they simply took this through the through the language model, and they got out just a list of steps, right? Here is step one, here is step two, here is step three, and so on. That is just a list of steps. And they would translate even when they use the translation model, they would translate each of them to a admissible action translate this one to an admissible action. Well, now you have no idea of whether that sequence of admissible actions even makes sense, right? For example, one could be a compound action, and it just gets translated to one of the two actions. And then the next action doesn't have a precondition. So what they do is they interleave the two steps, right? They interleave this translation with the generation. So they would only generate one step at a time, like step one, then they would translate it, and then they would use the translated version and put it back into the language model to get step two. That way, the language model always is conditioned on admissible actions instead of just being free form and then translating after the fact. So this is autoregressive generation. The last improvement they make, which is, I guess, more of a minor improvement. That's why it's not in this diagram. However, what they do is instead of having a generic prompt, what they do is they take the task, they embed it using the same sentence verb embedding, and they compare it to embeddings of all of the tasks that they have in the data set. And they just pick the closest task in the data set to act as a prompt, which could still transfer some in-context knowledge for the current task. So that is essentially the method. They investigate this, they have an algorithm right here. I formulated it in a rather easy way, but they do not only consider the closest action, they consider actually a waiting of, so in the translation, they consider a waiting between how close is it to an admissible action and how likely is that action that they output. So they would generate not only one action and then translate it, they would actually generate a bunch of variants and they consider each one of them, like how close is it to an admissible action and also how likely is it. And then they take the best combination of the two. That is obviously modulated by a hyperparameter. They have early stopping and all of this kind of stuff. And this results in a neat algorithm. And we're going to talk about these things in a bit and also the results right here. I want to highlight that if you look at, for example, vanilla GPT-3 has a really low executability, it does have a high correctness. However, if you look at the translated version, which is after their improvements, you can see the executability has risen dramatically while the correctness is a bit lower. Like you get a bit lower in correctness because of the whole translation procedure and so on. You're mocking with the outputs, humans may not like it as much. This is all stuff we're going to touch on in the interview. Just interestingly highlighting that codecs, like the codecs model seems to be scoring quite well on these tasks. So also the translated codecs is much smaller. However, it scores high, really high. So parameter for parameter, the codecs model is actually pretty, pretty good at this, which was a surprise to me. So I think this is an exciting paper. It except as I said, for a fine tuning baseline, it turns out to work completely without any training. It's just evaluation, so to say. And I liked it. And I think this does have applications like getting the knowledge out of these large language models is something we should, you know, be getting better at doing. Otherwise, I don't think we make full use of them. All right, so now I want to jump into the interview with Wenlong. I hope you enjoy that as well. Tell me how you like these, these videos with the interviews without the interviews, anything you want in the comments. I'll see you. Bye bye. Welcome everyone. Today with me here is Wenlong Huang, who is the first author of the paper about language models as zero shop planners and very, very happy to have you here. Welcome Wenlong. Thank you, Yaning. Yeah, super, super happy to be here. This is, I've already told you about this paper is different and I like different papers. And it's, it's different in a way that maybe wasn't expected every, it seems like every day, we find a new applications for these large language models and yet another thing that they can do here. And when I, when I saw this, I was reminded of a friend of mine who had like similar ideas, but it never really materialized. I tried some of this stuff as well, combining large language models with planning with telling me what to do in the real world. I even made a video where GPT-3 told me a recipe and then I cooked the rest, like me and my friend, we cooked the recipe and so on. But it seemed like always a bit, a bit out of place, a bit, a bit off just to give you detailed instructions. And when I saw a paper that was really trying to make this work in a real environment, I was, I was very happy to see that. And yeah, that's, that is, that is this paper. And also, to be said, you have a, you have a stellar board of, of co-collaborators right here. How, how did this come about? Like, how did you even get to the idea, hey, I could use these language models to do planning. Was it like, did it immediately come to you? Did it sort of build up from some basic idea or what was the process? So yeah, thanks for the briefing. So I think that's actually came out to be really surprising to us as well. So first we were just having, when we just playing around with the largest language models on the, on many of the web interface, we found that like, actually there is something there, like you said, if you ask it for a recipe or we actually originally study, like whether it can output the steps for making coffee, et cetera. So we found that like, when the models get large enough, there's actually something there. And this is the sign of life, I think for us to kind of go on and investigate how we can make that actually useful for, for agents. So we kind of just started from there and actually it came out to be pretty surprising originally without like, maybe we need some training data sets to maybe like train something, a translator or something to actually make it useful. But it turns out like, but we really trying to constrain ourselves in the meantime, because we don't want it to be tailored to a specific environment. So we would just want to see like just the language model itself, like how well it can do, how far it can go. So this is what got us in the end. We just like explored for like two months and then found like you can actually do this without any any training. And yeah, it's actually truly surprising and actually a really fun project for me as well. It sounds like fun. Yeah, just trying to see whether you can output something like really realistic and really fun. Yeah. So you came across this environment right here, this virtual home environment. Was this always the plan or why did you choose like there are a million environments, OpenAI, Jim and these Mojoco kind of robot simulations. Why was this one particularly useful? Did you immediately think of this one or how did this came about? Thanks. Yeah. So actually I wasn't doing too much research in this in body agents area, especially for this like really high level tasks. And then I actually went to the like Google Scholar and then search for appropriate environments for this. And we found this virtual home environment and we really liked it because it actually can model any any tasks that we can express in terms of this like textual language plan. Like just like textual plan. So and actually there are many other environments as well, but some of them are limited by, I think a lot of people also use Alfred environment. That's a really good environment too. And I think it's a bit more structured there, but the tasks are often come from like a template. So it's usually like pick something, pull something. But actually there are a lot of challenges there. I think it's a different set of challenges. And we found like what the virtual home tackles is exactly what we look for because it can model like any task expressed in free form language, especially those like really challenging tasks like people do actually every day, like make breakfast, make tea, make coffee. And then it particularly cares about the common sense constraints in them. So specifically this environment has a set of like preconditions and post conditions for each action. So for example, if you want to grab a glass of milk from the fridge, you can't just like say go to the fridge and grab glass of milk because you first got to open the fridge first and then like preferably you want to close the fridge afterwards. So it's really this like these constraints I think are really useful and really interesting to study whether the language models can handle this. And you've investigated several different language models. And just to be clear, this environment, it has this kind of syntax, it has very defined things you can do. And somewhere I think you say it's about 50,000 actions that are ultimately possible. It's kind of a combination of a bunch of verbs, which are grab, open, go to, and lift or things like this, and a bunch of objects like kitchen, fridge, and so on. So any plan would consist of a sequence of verb object, verb object, like here, walk to kitchen, open fridge, grab milk. So any planner in this environment would have to output this syntax directly. Now you had a plan of not training anything, right? You didn't want to train anything, you simply wanted to investigate what knowledge is already there in the language models. And you came up with kind of a way to translate that. You want to maybe elaborate how do you query these language models and how do you make them actually conform to the syntax here? Of course. Yeah. So the way that Virtual Home expresses these actions are via this specific format where you put a square bracket for the action, atomic action, like grab, put open, and then you put, I think it's a parenthesis or something for the arguments. But the problem is we can't just expect language models to handle this because even if we put an example in front, maybe they can do it, but it's definitely not the way that usually humans produce language. And after all, these language models are trained on human text. So we decide maybe it's not the right way to query these models. Have you ever tried letting them output directly the syntax, or was it just like, yeah, it's not going to work anyway? I tried briefly, but it's definitely not thoroughly investigated. And intuition-wise, I think it's definitely to use natural language. But we did adopt for the most basic approach that we can think of, which is just define a straight up template for each atomic action. And actually, because these atomic actions are simple enough, just walk, grab, and those things. So this atomic action, I mean, the templates we actually came up with are, I think, actually, just in a natural way, people say things. So turn off something, turn off something, and then add some words in between, like in, on, on top of, et cetera. Yeah. And then you just query these models, and you have multiple ways of evaluating this, right? You care about two things, you care about correctness, and you care about executability. And in at least, so you also make use of humans. How did you design, like what was your thinking behind designing the evaluation? Yeah. So actually, it came out to be really challenging to evaluate these things. Like I said, so like this task art, because they're expressed in free form language. So that means they're really open-ended. So it might be deterministic, whether like if you want to grab a glass of milk, you just want to look in the end, whether you have a glass of milk. But if you really think about it, if we don't want to constrain anything in the task that we want to do, like making breakfast, like what is the correct way to make breakfast? Everyone has different preferences. So it's hard for us. Actually, I think it's still a challenge in this sort of task is like really determine the correctness. I'm sorry. It's the success rate for each task. So you can't really tell if a task is really successful depending on how open-ended it is. So we decided that, okay, so if it's hard to computationally produce a metric for a success rate, but as humans, we can definitely tell if it's making something semantically meaningful. So we'll just use part of human evaluations to do this. But we don't want to entirely rely on humans because as you can tell for the tasks that like, for the action plan that real language models generate, they're so realistic that they can even fool many humans that are too realistic. So you can't just entirely rely on humans to say if it's successful. So we also use this metric executability, which is also used in past papers that uses virtual home. So we just use this metric as well to basically determine whether the plan satisfy the common sense constraints in this environment, namely just whether you make sure to open the fridge before grabbing something from it. It's interesting because when the humans raid it, the humans would also skip a bunch of steps. If you tell a human, go to the fridge and grab a glass of milk, the human will go like, oh yeah, of course. Which is one of my, maybe this is jumping ahead a little bit, but one of the questions I had most when I read this was just there is a level of specificity that is required right here, which is kind of ambiguous. You have a high level description, which is like make breakfast, and then you have a bunch of steps which you need to follow. And sure these steps correspond to actions in the environment, so they're kind of given by that, but the language model doesn't know that. The language model just knows I need to produce a plan. So how is the language model, why do we expect the language model to figure out that it needs to say open the fridge before you get a glass, but for example it doesn't need to say put one foot in front of the other foot in order to walk. So did you have any insights or concerns with like, there seems to be like a very specific level of specificity of these plans? Yeah, so that's a really good question. Actually this granularity actually comes from the dataset or the virtual whole environment itself, because we essentially follow the format of virtual whole environment, and also this dataset they collected from humans of how to do this really human activity task. So the way they collect, they build this environment is they first ask many humans to come up with a set of tasks that they do in everyday household, and then they ask a different group of human to come up with a detailed plan that can drive a robot to perform these tasks. And it's after that they build this environment based on the verbs used by those humans. So you can think of like this environment is really built on top of what humans say. Now the developers who just say like, okay, we want this granularity, we want this like walk, grab, and those etc. So they actually ask these humans to give those verbs and then build those actions according to those verbs. And they did make sure for each of the verb to develop a set of common sense constraints, which completely makes sense. And I think they're actually like reasonably exhaustive for those actions. So if you want to grab something, you definitely need to make sure the things you grab is not within a closed container, for example. So in this case, the fridge is a container and it has this attribute of being open or being closed. So they internally keep track of the attributes for each of the object. And then to make sure that if you do something like this, you don't violate the common sense constraints. So to answer your question, this granularity really depends on the humans. And I think this is where language models really shine because essentially language models are trained on human produced text. So my hypothesis, although this is definitely not something they're only tested by, my hypothesis is that because it's trained on human produced text, and humans after all produce these actions. So if you do it careful enough, and then use some techniques to properly translate them or doing something else, you can essentially get back something similar to what humans produced in the beginning. Yeah, I mean, you would imagine that sort of the human-ness of how the environment was built would also be present a little bit in these language models, which makes sense. I don't have a better idea of how to build an environment like this. So it seems pretty reasonable. Yeah, it's actually not to be really interesting to me because it's super hard for me if I were to develop this environment, how would you even animate all of these really human tasks even just in a household setting? It's super difficult. And I think they did a really good job here. And then I think this is also what makes language models particularly useful for this task because these are basically just human tasks and language models are really good at mimicking humans. Yeah. Yeah. So on the left here, we see a bunch of models that you've evaluated right here. So again, executability is sort of how, if it matches the syntax of the environment, if I can map it to that, and also, I guess, if it violates any of these common sense constraints. So just like how executable is the plan in the environment, no matter whether it's the wrong thing, right? And that comes in a second. And correctness is a thing that is rated by human annotators. They look at the plan that was produced and they just, from their own intuition, are like, well, is this a good plan to make breakfast? Yes or no. And we clearly see there's this downward trend. If we exclude the models on the right, there is this trend line here where the larger models, they seem to produce more correct plans, which means plans that the humans like more, but they are less executable. Whereas the smaller models, they are less correct, which we can, that's correct. I would have expected that, but they're more executable. And you've noticed in the paper that very often they just produce plans that have nothing to do with the task description. They would just produce like a plan that is according to the syntax of the examples that you give in the prompt, right? But how can you explain that? Like even on the top here, like the large models, it's even better than humans at correctness. So humans rating other humans think that GPT-3 produces more correct plans. Why is it so bad at executability? Yeah. So there are actually two questions that I think you raised. One is why this smaller models, like when I say smaller, it's actually still pretty large, the largest GPT-2 model. So why do they produce more executable plans? And the second question is why the GPT-3, the largest GPT-3 model is actually better than human. So to answer the first question, I think that's because we did find some failure modes here for smaller models. I think the two most prominent ones are first, it frequently tries to like repeat the given example. For example, you give it like how to browse internet. You said like go out to the computer and type on the keyboard, et cetera. And then you ask it to brush teeth. It still goes to the computer and then type out on the keyboard. So it's totally nothing like sensible here. And then the second source of error is sometimes it just outputs really short plans. If you say like sleep task, go to sleep, it's just like go to the bathroom and just stop. So that's this right here, brush teeth. It's just like go to bathroom. Yeah, exactly. So when these plans are short enough, even though it can be executed, if you just say like walk to bathroom, walk to the bathroom, just one single action, for walk, there is not much common sense constraints there. So you can totally imagine it's super executable. But if you present them to humans, of course, humans will spot this and then say, okay, this is not correct. Because when we do human evaluations, we're trying to make it simple so that the error here is not too big, because we don't ask hundreds of humans to evaluate this. We only got to ask 10 evaluators in this case. So that's why this smaller models are now really good at escalability. And the second question that you ask is why these larger models are actually better than humans. So actually, this is now the completely fair comparison if you just look at one axis. So all the results here, we look at from two axes that we care about. So one is the semantic correctness, which is evaluated by humans. And the second is the executability. So this human plans that we use are from this data set that virtual home developers cross source from Amazon Turkers. So these plans, they make sure that these are executable plans. So which means that they have one hand here. They'd be over here. Yeah, but we don't want to put a spot right there on the right, because it's hard to see, because humans are a big baseline and reference here. It's not a baseline that we're trying to beat. Of course, GPT-3 is not there yet in terms of at the same time outputting correct action plans and semantically correct action plans, and also being able to really ground them in the environment. But using these two axes, we can really see, for example, which axis is the place that, as a community, that we may want to work more on to get it better to get the human levels. And with this paper, we find this result actually a bit interesting to us. Is that for these larger models, in terms of semantic correctness, you don't need to worry too much about it. It's kind of already there if you do it, extract them. But the real question is, how do we make them executable for agents that we care about? And that's exactly what you do in the meat of the paper. And the result are these translated models right here that, notably, they do drop a little bit in terms of their correctness as rated by humans, but they gain massively in executability. And this is the result of a bunch of different ingredients, like three main ingredients, as far as I could tell. You quickly want to go tell what the ingredients are to make whatever these models output into something that... I mean, the virtual home is maybe a test bed, right? I don't see this paper being about virtual home. It's more like, here is a model that outputs something, yet I need the output in some other form, right? And this is a very general problem, as many applications. And if we could solve that bridge, that technically is a big gain. That's exactly what you do. So how did you go about this? Yeah. So actually, I just want to make sure that actually this paper just presents a really preliminary step. I don't think it solves anything particularly. I mean, it does, like if this problem... Sure, but it's a big step, I believe. I mean, the executability I have raises pretty high. I didn't want to oversell you, but also not undersell you, certainly. Yeah. But to answer the question, so we actually found there are three ingredients, but central to this is one really simple technique that we found that's the most useful, which is action translation. So because in this virtual home environment, the actions that it supports are a limited set. I mean, it's not small, but it's something that we can definitely enumerate with our computational hardware and in a really quick manner. So like just one-tenth of a second or something like that. So let's say if we can enumerate all the actions that are supported by the environment, then the question now becomes, how do we translate this really sensible action plans generated by language models, but not really executable plans? How can we translate that into those actions supported by environment? Or if you want to deploy something in the real world, let's say your robot only supports 10 actions. How do you map those tasks into the 10 actions that the robot supports? So what we found is that you first need to enumerate all the actions. And then we found that you can again leverage the world knowledge in this language models by using another language model, namely here we use Roberta, which is a language model really similar to BERT. And it's a different language model because it essentially is a mass language model. So it's really good at outputting a useful embedding. It's really good in terms of about the semantic meaning for that sentence. So what we do is that we take the sentence output by GPT-3 or codecs, and then we just compare that against all the possible admissible actions, allowed actions by the environments. And then we found the most similar one in terms of this distance in the embedding space. We actually use just cosine distance and found that to work decently well. Yeah, there's an entire space somewhere, and you just place all the actions. I guess you can even pre-compute those. You can pre-compute the embedding of all possible actions there. And once my language model outputs anything at all, all I need to do is ship it through the Roberta model, get its embedding, put it somewhere, get the nearest neighbor. And that's my translated action. So here we have an example that would translate like squeeze out a glob of lotion into pour lotion into right hand. So it would map action into and pour, it would be the verb lotion, the object and right hand also one of the objects. So maybe there's two arguments to pour. It seems very simple, but I was at a talk by the people who made the first version of the... In Gmail, you have these always three options to respond to, like the quick options to respond. And I think the first, I'm not sure how it is done now, but the first version of this, we were like, wow, this is cool. It actually takes into account the email message that was there. We always thought it was kind of like a language model, generative model somewhere. So I went to a talk and they were just like, no, we just have a big list of responses. We just classify, right? Whatever. We just take your message, right? And we just put it through a model and then we just classify into this big, big bucket of possible answers. So I mean, this is even though it is simple, it's a very powerful method. And that being said, you don't even train this. You take an off the shelf embedding model and you compute nearest neighbors and it does turn out quite well. You do, however, you talk about this in the paper, there is a bunch of problems. And one of the problems I see is whenever a step contains like multiple steps, right? Is that like, is that a big, have you found this to be a big problem? Because this just maps one action to one other action. But if it's like, you know, open the fridge and take a glass of milk, then I have essentially no way of translating that into an admissible sequence. Yeah, that's a, that's a good question. And I think that's one of the main errors that like this, this Roberta model that we use, it's actually a sentence Roberta model because it's trained with a different objective such that it can really, you can actually calculate cosine distance between the embeddings they generate. So it's a, like we found like it's pretty difficult to map a compounded action. Like you said, like two actions in one sentence into one admissible action. But this is partly mitigated by how you tune the temperature, the sampling parameter, just the temperature for the GPT-3 or codex models. Because we found that if you do increase the temperature, then it tends to output something more verbally expressive answers for each step. So that means it's harder to translate. And we, if you, if you try like all this, like different settings, we did, in the end, we found like, usually you want to use like a lower temperature than what people mostly use for language generation, for example. So that like each action is like small enough and succinct enough. And then, and then after we translate this action, so that it's easier for this bird model, Roberta model to translate. And yeah, something I forgot to mention, like after we got this translated action, we found that it's still useful to put that back to the original prompt, put the translated action back instead of like the original action so that you can add the GPT-3 and codex model to reason, like how am I going to do based on this like action already performed? So yeah, like you said, like you pointed, this is the third sub figure here. So we would take instead of instead of generating the entire plan at once, we just generate one action, then we translate it. And then we substitute essentially whatever GPT-3 output with whatever the translated thing is. And then based on that, create the next action. It makes sense because you it's like almost like a guiding, like a bit of a guardrail for, for the language model. Instead, if you were to let it generate all at once, and then you translate each action individually, they almost like lose connection to each other, right? So this, this here might mitigate some of this, this stuff ready, if I have a compound action, like go to the fridge and grab a glass, and the closest, I hope that the closest sentence is to go to fridge, right? The language model might still recover and recognize, aha, I haven't, you know, grabbed, haven't grabbed a glass yet. So that is, so these are improvements one and two. And then the third, the third thing you found that really helps is the prompt up here. So the priming, which I think in GPT-3, it's very common to have these priming prompts to tell the model what kind of stuff you, you expect as an output. I was surprised to see that you only have one priming prompt. Whereas in general, people put more than one, usually people put like three or something like this. Is there a particular reason why you used just one? There is actually not a particular reason. I actually found like, I mean, in the beginning, we were, we know that we have this data set, right? And then we, we found, originally, we actually tried to train something to achieve this, but in the end, we found that like, we don't even need to train something. And like, now the question becomes like, like, can you even leverage this data set to some extent to make it useful? Of course, this is something like additional, I mean, it would definitely be better without any, any of this. But if you have this data set, you can actually found like this most similar example to the query task here. For example, like this is apply lotion. So like, shape, task shape is determined to be most similar. Again, judged by this Roberto model using the same technique. Yeah. So I think that that's the, that's the main motivation for using this, but we didn't thoroughly investigate it, like how you structure the prompts, whether you add like multiple things there and then, or you change the template here, because I just defined this template from day one, like task something, step one, something, two something, maybe there is a better template. Maybe you want to add some instruction there to make it better. And so I like, I mean, this is definitely possible and we don't investigate them here because we don't just want to get the best performance out of this. We want to get the best performance out of this. We want to show people like, this is something possible and it's really interesting to us. So that's why we ended up like, like just using the most simple technique here. Yeah. And to answer your question, why we don't put multiple things there, I think one important reason is like, because this example plans that we put in front are produced by humans. And this is because due to space constraint, I'm using an oversimplified version in this figure specifically, but in practice, these plans are actually pretty long. So, and they actually already take up a lot of space in the prompt. So if you put more than one, sometimes it gets too long. And I mean, it's maybe something handleable by larger models, but we just opt for the most similar, most simple case. And I actually read this, like there's a recent paper investigating why in context learning works, they frame this as a implicit Bayesian inference problem. And they did come to a conclusion that the longer the prompt, if I remember correctly, like it helps the model. So, in this way, you kind of like trade off the number of examples you put and the length of each example. So in those cases, I think you mentioned many people put many examples before the query. Those are usually the cases where the tasks they care about are like smaller. So for example, like you want to ask Einstein was born somewhere, then like this is just a sentence. So you probably want to put like more than one sentence there. But this case, our case is like, it's an extensive action plan. So it's already pretty lengthy and we don't want to go too crazy over here. I mean, it's, yeah. Sorry, the recording has stopped on the screen side, but we can still see it. Okay. Yeah. So yeah, I was quite interested in the sense of the prompt structuring, because I know that can also make a big difference. But I also like the sort of approach of not having too many moving parts in one single thing, because it makes things complicated. And for many papers, it makes you wonder like what was exactly the thing that gave the improvement here. Now you do very good ablations of all of these different improvements, which I really liked. And you showed that kind of the translation is the main part right here, although the other things certainly also help. Have you ever, so it reminds me a bit of this, you know, this retro model, these language models that retrieve from the internet as they produce text, it reminds a little bit of this, right, in that you produce, you go and retrieve the closest samples in the data set as you produce the text. Yeah, I think this combination of retrieval and generation is picking up steam. And it looks pretty interesting. My question is a little bit, have you tried also, because essentially, you now rely on this translation procedure to produce the correct actions. Have you tried any way to like let the model know what the possible actions are? Like something like, you know, I can imagine maybe I, you know, I ask the model first, and then I get maybe the five closest actions or the 10 closest actions in embedding space. And then I somehow put these in the prompt here, like, you know, in between, you know, what am I going to do next? Is it this or this or this or this, right? And then the model could, maybe I could prime the model to output one of them. And, you know, is there, did you try any way of telling the model more what's even possible in the environment? Because right now you're essentially relying on on just the language model itself. Yeah, that's a really good question, too. So like, we actually didn't try the specific thing that you talk about, like generate a bunch of possible actions and then ask the model again, which of these are the best. But we did try something similar, which is like Beam search. So essentially in Beam search, you look ahead to see like what the outcomes are, are like having in the end get the highest likelihood. So we did try to constrain the strain the vocabulary that can be used in the Beam search. But this is only conducted on smaller models, because obviously the GBT-3 and codex models are now open to fully open to public. So we can't, we don't really have full access to different features. Like, you can't restrict the vocabulary dynamically. Yes. So I've only done this on smaller mode, relatively smaller models like the GBT-Neo. And then I think I might have tried on GBT-J as well, which is a 6 billion parameter model. And it actually turns out that they don't do really well with if you really just constrain the vocabulary that way. And yeah, specifically just the Beam search constraining the vocabulary can generate. But so my hypothesis, this is now thoroughly tested because it's now invested on larger models as well. But my intuition why it doesn't work so well is that this language models are really trained on human text. So it really, they're really used to how humans speak a certain language in this case English. So like people don't speak things in this way, step one, something, two, something, step three, something. So that's why if you really constrain the models this way, a lot of the world knowledge encoded in these models are lost. So basically, and personally, just a personal opinion, I don't think these models are doing super intelligent reasoning here. It's basically just doing kind of retrieving what's what is trained on. So, retrieving this large scale text. So if you want to retrieve better, you better adopt the same way that humans speak a language. So like if you don't constrain the vocabulary, you can get the most out of a language model. And you can really tell if you adjust the temperature. Like if you go different temperature, they can tell you like different levels of things and they can be really realistic. But if you really constrain it, a lot of this knowledge is lost. And it can really do too much like common sense reasoning here. I was, you mentioned this a bunch of times, I was surprised to find codecs as a model. And so you have, these are sort of vanilla models. And then you have the translated ones where all your all your improvements are in there. So there is the action translation, there is the sampling, even according to the probability and executability, there is the retrieval of the closest prompt and so on. And these translated models, they perform really well. What I was surprised by and also by the results is that codecs, I mean, that it's even in here, it's like a code model, but also that comparably, it holds up, right? It's not as good as the GPT-3 model, but it's also very, very much smaller. So parameter by parameter codecs is outshining GPT on this task very well. How did you even consider using codecs? And how can you explain that this model is doing so well? Yeah. So one intuition why, this actually came out to be pretty surprising to us as well. So we did find like this codecs models are really good at generating these plans. And actually from my own experience playing with this models, I did find like codecs thinks that this is part of some doc stream. So it's actually imagining like people just like asking the doc stream here, but instead of letting keep generating the code, we kind of just stop here. So, okay. Yeah. When it's the doc stream for us, that's enough. So yeah, so it's actually doing some of this kind of doc stream. It generates this doc stream thing. And the reason I think the smaller codecs model are actually better than the same size GPT-3 model is that because it's trained on a more structured data. So like code and specifically many of this code examples in the training data set consists of doc stream and the code. So it not only can handle code really well, it can also generate really realistic doc streams. So, and people in doc stream, they don't write in like... Yeah, they don't write a novel. Yeah. So they write something really step by step and have more structure in it. So that's my intuition why it actually does really well with this task. So you can really process this sequential like logical reasoning better than the same size GPT-3 model. But of course, if you use a larger model, that potentially be more helpful. Yeah. Or I mean, there is, as you said, there is still a lot of open questions about how exactly you structure the prompts. Like maybe this step one, step two, step three isn't ideal for these language models. Maybe you need to more let them write like a Reddit post or something about how they went and got a glass of milk yesterday and then translate that somehow. But yeah, it's pretty cool. So one thing that just came to my attention right here is this top row right here, which I found hilarious. So the task is complete Amazon Turk surveys. So the four steps apparently that you need to do is walk to home office, sit on chair, switch on computer, look at computer. Like, is this the description of complete Amazon Turk? It's a pretty accurate description maybe of what Amazon Turk workers do. So like I said, these tasks are generated by crowdsource from humans. And the humans here happen to be Amazon Turkers. So one of them decided that, okay, if you want me to generate some tasks, I would say like just complete surveys on Amazon Turkers. Yeah, so they decided to put one of this here and we found this here, there is two. So like I said, so this language model, so they can't really handle anything that you wanted to generate. So because we did put the example in the front. So I think in this case, the example happens to be something related to computer and the example is that you can't really see the models actually happen to reason or potentially you could just repeat the example. But depending on other tasks, it doesn't seem like that's the case, but it does come to the reasoning that like this might be something related to computer too. And I'm going to put like this steps here. Yeah, yeah. I mean, this is, I mean, it has something like melancholic and it also has something a bit, as you said, rebellious of like, you know, I'm here doing my Amazon Turk work, I'm gonna, you know, I'm just gonna put my Easter egg in there in this data set or like show you, but it also shows something I think about the interaction with this environment because, you know, if you ask me, you know, what did you do today, I could tell you, you know, I programmed this, I viewed a poll request, I sent some email and so on. But in the action space of this environment, this would all just be characterized as go to desk, sit on chair, switch on computer, look at computer. And yeah, so it is really, maybe also a constraint of the environment itself. And as I said, I think the challenge is going to be there's so much knowledge in these language models, and we somehow need to get it out into the domain that we care about. And yeah, I guess, I guess many opportunities are still there. And in this particular environment, is it so the way I see it, we have this environment, it's a 3d environment, but you never actually for your studies, you never actually had to actually execute anything in the environment. Is that correct? Or do I see something wrong here? I think those when you say execute do you mean like, like run in the environment? Yeah, like run the 3d environment, like actually give it to the environment, because you evaluate executability, you can do with a parser, right, to see whether it matches the actions and constraints. And the correctness you evaluate with the humans, because my question was also a little bit like, why can't I just run it and see if, you know, at the end, there's breakfast, but you already, you already said that the tasks are so, so open, like, how would you how would you detect there's breakfast, right? So, so, in terms of so a bit background here for the virtual environment. So it comes in two versions. One is the, I think that they call the evolving graph version, which is a pure, like you said, a state machine, a Python, like reading in Python. So it just goes in and then checks which whether the actions can be parsed, and then we satisfy the common sense constraint. And the other version they implement is this, is this visualized version, where they actually only implement a subset of the act the total action supported in the environment. So I think they, so in the evolving graph version, the Python version, there are 42 actions. And in the visualized version, there are only 10 actions. So it's limited. Like the plans we can generate, we can really visualize are limited. So that's also part of the reason we don't show the visualized version to humans. Like, can you tell us whether this is successful or not? So, yeah, that's, that's a, that's indeed something we can do right now. And I think that's like as a community, as we go, go on, like, to this next step with more complex tasks that humans do every day, instead of just like, lower level tasks. As a community, I think more efforts can be can be put here and to develop better simulator and also maybe beyond even household environment. So just as a, as a story here, I did play around with the codecs and then GPT-3 models to have it generate something out of the household domain. And seems like they do have some, a lot of knowledge for those as well. So if you can ask it, how do, how do I pay bills at a restaurant? And how do I work out at the gym? And I think in, on Twitter, there's also someone tries to, after the posting of this paper, they try to ask the GPT-3 model, how do I start a company? So yeah, they do have a lot of knowledge for this. And as long as you can provide a set of actions that are necessary to complete these tasks, I think no matter what, what the granularity is, ideally it should be at the same granularity as of humans. So ideally it should be, this model should be able to generate something, something sensible and reasonable. But yeah, right now is something that you definitely can't trust to put on a robot, of course. Yeah. Yeah. I mean, it's, I've always, I've always seen people thinking when they think GPT-3 or so they, they, and they think, for example, of video games, they always imagine, you know, we can have our NPC, our characters, the dialogue be generated by GPT-3. So it, the dialogue is more realistic, but I think this shows that it can go further if we are able to map sort of GPT-3's knowledge into a sort of structured domain that we choose, we could potentially also let these models generate the action sequences of like, of characters, for example, let's say in video games, because that's like a common complaint that, you know, the guards, they always walk up and then down and then left and then right and then up and then down and right. They have these, even if the dialogue gets really good, their behavior is still kind of lame, either that or they cheat, they know where you are at all times. But with, I feel with models like this, we can almost like take this common sense knowledge and maybe have the hopes of transferring that to various domains and infuse a lot of areas with common sense. And that I find that to be, I find that to be pretty cool in itself. That would be really exciting and interesting application. Yeah. Yeah. Yeah. So I mean, there's a lot of things to be gained. So what I did, I was specifically intrigued about clip. I don't know if you are thinking about this or not. But what I tried to do is I tried to take like a frame of Pac-Man, like, and you know, there's like walls here and here and here. And I had Pac-Man be like, you know, here facing a wall. And then there's like a ghost behind Pac-Man, right? And then there's like these little dots over here to eat. And so it was like super clear what you have to do. So I tried to feed that to clip. And you know, you can make clip classify things by just evaluating a bunch of different strings with it. So I like try to, I try to evaluate the strings, go left, go up, go right, go down, or like Pac-Man should go left, Pac-Man should go up, but it never worked out. So if you can, if you could get something like this running, this would be amazing. Maybe with your knowledge, maybe Pac-Man isn't the right environment because clip was trained on whatever picture scraped from Instagram. But I think just this this type of, you know, thinking beyond just the strings in terms of language, but thinking in terms of I have some structured environment and I want to leverage this, this knowledge of these models is super cool. Yeah, that would be a super interesting application. I think using clip here, like, because it feels in another modality, which is image could be really interesting. So I think it kind of solves one of the major limitations of this paper, namely just the, because currently we generate plans regardless of the environment state. So it doesn't condition on environment state and potentially using clip, you can encode something there because you can also take image as input to, to an image can serve, can serve as state for, for, for the environment. I think that would be really cool. And yeah, so yeah. So just to be, to be clear to the listeners, the basic idea for this I have from, from a PhD student that was partially in our lab called John Battista Parascandolo. So the, the credit fully goes to him of, of this whole idea. I didn't want to, but I just, it got me thinking so much about, you know, we can extract this knowledge into, into other modalities. And that's, that's pretty cool. Is there anything you want to maybe say about the experiments? Is there anything that was very surprising to you or, you know, something you didn't expect or something you particularly want to highlight? Actually, I think we covered most things, but I think I might say something about the, the, the baseline here. I see, you can probably see, except for the human references, we also got to got to fine tune a GPT-3 version. And we did find that fine tuning can, can be a really strong baseline here, because as you can probably tell the, one of the measures here, LCS, which is the longest common subsequence. This measure here is much higher than the others. So this measure basically calculates how much overlapping there is in your generative plants against the those plants written by humans. So it's kind of calculating this IOU score. So we did find that, find this to be a strong baseline. And I think it still actually makes sense to, to be a strong baseline because this is trained on such data. And so this is kind of to illustrate that, like if you do have domain data, it's still really helpful to, to train your models, fine tune your models this way. But if you don't have something like this, you can potentially just leverage the knowledge already in this language models. Cool. Yeah. So where, where does your future lie? What are you, I, I, are you going to, are you going more into this direction? Or was this sort of like a one-off thing? Or do you have, I mean, what are the interesting questions that, that you are asking now maybe as a follow-up to this? Yeah. So I personally, I haven't decided because I, I'm in a stage where like I'm applying to PhD programs and, and, and also other positions. So like, but, but as a follow-up, I think it would be really interesting. As I mentioned, one limitation, major limitation of, of this work is that we haven't found a clear way to condition on the environment state. So that like, if you really place an agent in, in the household, for example, there is no, if you want to make coffee, but there is no cough, but there, there's no, there isn't a automatic coffee machine. How would you make a coffee with some, maybe a similar devices. So the agent can really reason if you just put it this way, because it doesn't condition on the environment state. So I think it would be really interesting to like investigate how you can also condition on the current environments and then, and then reason from there. But this might require some training data. And I think that's part of the reason why we don't like go full length here to investigate this, because this is something just for us to tell people, like this is an interesting finding and we may be able to leverage something here. But I think this will be really exciting and like interesting future work. Cool. Excellent. Wenlong, thank you very much for being here. This was awesome. So great to hear from, you know, from always from the people who made the stuff. So yeah, thanks a lot. Yeah, thank you so much. Yeah. And yeah, I think I also want to also want to like point that like, this is a group effort and really a lot of thanks goes to three of my advisors, Peter Bill, Deepak Pathak and Igor Mordac. Excellent. All right. Thank you. And I hope to see you again. Yeah, I'm like, it would be an honor to always to be here. Yeah. Excellent. All right. Bye bye. Yeah. See you.
 Hello there, today we're looking at language models as zero-shot planners, extracting actionable
 knowledge for embodied agents. And I'm going to interview the first author Wenlong Huang
 in a few minutes. So first there's an explanation of the paper, 10-15 minutes or so, I'm going to
 try to keep to it. And then we jump into the interview where we can discuss this paper at
 length. On a high level, this paper asks, can we use the knowledge that is inherent in large
 language models like GPT-3, or surprisingly, OpenAI's codecs in order to do planning in what
 they call embodied agents. Ultimately, it's going to be this environment right here. The,
 I don't even know what it's the virtual home environment. And it's about a virtual home,
 you have to fulfill some tasks like brushed your teeth, then the model has to come up with a
 sequence of steps that are admissible by the environment. So there's a level of admissibility
 of action, predefined actions that are admissible, the model has to come up with these actions in
 order to fulfill the task. The model is then rated based on executability and correctness
 of their plans. And it turns out that the larger the models get, as you can see right here, the
 less executable the plans become, which means that the actions they generate aren't admissible
 by the environment, probably because the models are more, let's say powerful, they can express
 themselves in more ways, they have different ideas of how to reach goals. However, the correctness,
 this is human evaluated of these models rise as they grow larger. So this gives you an indication
 that the large models seem to have quite a lot of knowledge. And we have to say these are not
 trained, the entire paper just works except for one baseline evaluation, just works with pre-trained
 models, they're not fine tuned at all on this environment right here. So what this paper does
 is it says, well, given that the larger the models get, the more correct their plans are,
 can we do something to fix the issue with the executability? To that, they develop this
 translation procedure right here. These are three specific improvements they do to the models.
 In order to get their executability up, you can see they sacrifice like a little bit of the
 correctness, but they do make the plans largely executable in the environment. And therefore,
 procedures like this could be applied in many different ways. It's not only about the virtual
 home environment and so on. It's essentially anywhere where you bring together the knowledge
 that is inherent in large language models with some sort of a domain specific language or a
 grammar or any anything like this, like where you have to transfer that knowledge into a new domain,
 but you don't want to train a model to do so. So we're going to see how they do it really briefly.
 First of all, the environment itself, as I already said, is this now this is visualized, although
 they never work, you know, actually in 3D, just a small correction here, because I messed this up.
 There are actually two versions of the virtual home environment. One is a Python version that
 focuses on the textual interaction with the environment. The other one is implemented in
 Unity and actually does work in 3D. The developers of the environment mostly focus on the Unity
 environment because it's more real. But as of yet, that has a subset of the actions available that
 the Python environment has. And the authors of the paper use the Python environment and the data set
 that comes along with that. We're going to go into this more in the interview. Stay tuned.
 They simply grab the data set of possible tasks, some tasks you can see right here, a task could be
 throw away paper, another task could be brush teeth, and there there'd be a sequence of steps.
 This environment is made by humans. So the tasks are made by humans. And then other humans have
 to come up with the steps that are admissible, admissible actions in this environment. There are,
 I believe, a number of objects that are defined, they're predefined. Yeah, so there are a number of
 objects, for example, living room, television, sofa, and so on. And there are a number of verbs.
 So walk, find, switch on, and so on. And not every verb object combination is possible. Some verbs
 have two objects and so on. But essentially, you combine the predefined verbs and the predefined
 objects, and then the state of the world changes. So the world keeps track of states, there are
 certain preconditions. For example, you can probably only sit on the sofa if you are in the
 vicinity of it. So you need to first find the sofa, you can only switch on the television.
 Similarly, if you have first found the television or walked to the television or something like
 this, if the television is in the living room, you first need to go to the living room, and so on.
 So there's a hidden kind of a state. But all of this is constructed. And we talked about this in
 the interview, like, what's the appropriate granularity of actions like this? And isn't
 this a major issue? But it is made all with the humans in the loop. So the data set is supposed
 to be kind of the most natural expression of these tasks, as split into steps that a human would come
 up with. So this is the grammar of the environment. And the language models, they don't know about
 this grammar. They're just language models. So what they do is they take something like GPT-3,
 and they make a prompt. Now the prompt, as you might know, in GPT-3, you have to give a prompt.
 So the prompt could just be like, here's the task, you know, blah, blah, blah, brush your teeth,
 then what's step one, right? And then GPT-3 will probably it will probably even generate step two
 and three and four. But it will probably not be according to the these actions in these templates,
 you can help this a little bit by putting a prompt up here. So the prompt they use is one,
 I believe one specific plan. So they have already like task up here, some task, and then some number
 of steps, so that the model kind of knows what is expected. We also talked about this in the interview,
 and this could potentially be improved by multiple, multiple prompts and so on. But in the baseline,
 they have one particular prompt. And then one of the improvements is actually to select a more
 optimal prompt. This is the basic setup. You have a goal in this environment with a fixed grammar,
 and you task, you input this right here to your language model, and the language model will spit
 out the plan. Now what do you do with the plan? The plan, you score, like how good is the plan?
 And they have two different scoring available. One is executability. And executability is just like,
 it's essentially parsability by the environment. So in executability, you ask yourself, can it be
 correctly parsed, which means that is the syntax according to the syntax of the environment. And
 they do have a little translation procedure, like a little heuristic translation procedure for the
 baseline in place, so that the language model probably can't get it exactly right. But they do
 sort of translate to the closest action there. But also one of the improvements is related to this.
 And then also does it satisfy the common sense constraints of the environment. And these would
 be programmed in like, for example, you can only pour yourself a glass of milk if you first open
 the fridge and grab the milk, this can be measured directly, what cannot be measured that well is
 correctness. So these models, they would come up with plans and independent of whether they're
 executable or not, they could be correct, right. And that's where they ask humans. So they use
 human evaluations, they conduct human evaluations in order to score the correctness of whatever
 these models output. So they give it to a human, ask the human, does this look like a sensible plan
 in order to brush your teeth, and the human would either say yes or no, when they do like ablations,
 and so on. They also use like longest common sub sequences between two programs and so on in
 order to not spend ginormous amounts of money on humans. But essentially, the correctness metric
 is a human metric. It's also interesting because you thought you could just execute like the plan
 in the environment and that give you like, does it succeed or not, but they say correctly that for a
 task like make breakfast, there's not really a defined end condition that you could program
 into the environment to give a reward. So it's more accurate to ask humans whether a plan is correct.
 As you might have guessed, this environment is very human centric, it's made by humans with humans in
 the loop and so on. It's supposed to really be sort of a representation of human tasks and human
 plans to human tasks. All right, so now we're going into the improvements. There are three
 distinct improvements they make. So if they just do this, if they just do what we've described so
 far, then the graph up here results, excluding the two models on the right, you can see the larger
 the models get, the higher their correctness, but the worse their executability. So now the thought
 is, can we change that? Can we raise the executability? And so this is the baseline right
 here, zero-shot planning via causal large language model, you put in a task as a prompt, and along
 with like the format you expect, which is this one right here, which is some other task from the
 data set, then you use the pre-trained language model like GPT-3 or something, and that will give
 you a plan. And that's it. So the next thing they do is they do what they call a translation model.
 So they introduce a second model, which is also pre-trained. And this is it's not trained on
 translation. It's just trained on masked large language modeling. So think of this like,
 this is just BERT. In fact, I believe they use sentence BERT, just pre-trained on English
 language. And what they do is they make a big vocabulary of all the admissible actions. So all
 the admissible actions would just be like any combination between any verb and any object that
 would actually go with that, that is admissible to this verb. So from this, they make like a giant
 list of all of the admissible actions. And then they embed that giant list. So they put this into
 some embedding space using the sentence BERT model pre-trained, right. And then whenever the large
 language model outputs something, they don't implement it into the plan directly. They first
 embed whatever the model outputs. Let's put this over here, they embed it, let's say that becomes
 this right here. Then they see what's the nearest neighbor of my admissible actions to this thing.
 And then they simply replace whatever the model output with the nearest neighbor. And they call
 that translation. So essentially, it translates from general natural language space into the
 space of the admissible actions or the grammar of the model. Now this has some problems on its own.
 For example, if the model outputs the compound actions. So if it says, for example, squeeze out
 the glob of lotion and put it in your mouth or so or on your face, I guess, then well, it's apply
 lotion, it's anywhere. Squeeze out the glob of lotion and put it on your skin. That would be
 still one action. Now which one would be the closest right here, there's going to be somewhere
 like squeeze out a bit of lotion and the other one is going to be like, put the lotion on your skin.
 Yet you only have one action like it's it's it's one line. So one action, it just contains like an
 and now the end might be easy to recognize, but there are other there are going to be other like
 compound actions. And this is going to be a problem here, because you just map one action to one
 admissible action. But in any case, doing this already helps a lot, even though there are still
 some problems to alleviate the rest of the problems. They have two more improvements.
 The first improvement they do is they say, well, if there is a compound action, we can still kind of
 alleviate that a little bit. So in the original method, what they did is they simply took this
 through the through the language model, and they got out just a list of steps, right? Here is step
 one, here is step two, here is step three, and so on. That is just a list of steps. And they would
 translate even when they use the translation model, they would translate each of them to
 a admissible action translate this one to an admissible action. Well, now you have no idea
 of whether that sequence of admissible actions even makes sense, right? For example, one could
 be a compound action, and it just gets translated to one of the two actions. And then the next action
 doesn't have a precondition. So what they do is they interleave the two steps, right? They interleave
 this translation with the generation. So they would only generate one step at a time, like step one,
 then they would translate it, and then they would use the translated version and put it back into
 the language model to get step two. That way, the language model always is conditioned on admissible
 actions instead of just being free form and then translating after the fact. So this is
 autoregressive generation. The last improvement they make, which is, I guess, more of a minor
 improvement. That's why it's not in this diagram. However, what they do is instead of having a
 generic prompt, what they do is they take the task, they embed it using the same sentence
 verb embedding, and they compare it to embeddings of all of the tasks that they have in the data set.
 And they just pick the closest task in the data set to act as a prompt, which could still transfer
 some in-context knowledge for the current task. So that is essentially the method. They investigate
 this, they have an algorithm right here. I formulated it in a rather easy way, but they
 do not only consider the closest action, they consider actually a waiting of, so in the
 translation, they consider a waiting between how close is it to an admissible action and how
 likely is that action that they output. So they would generate not only one action and then
 translate it, they would actually generate a bunch of variants and they consider each one of them,
 like how close is it to an admissible action and also how likely is it. And then they take
 the best combination of the two. That is obviously modulated by a hyperparameter.
 They have early stopping and all of this kind of stuff. And this results in a neat algorithm.
 And we're going to talk about these things in a bit and also the results right here. I want to
 highlight that if you look at, for example, vanilla GPT-3 has a really low executability,
 it does have a high correctness. However, if you look at the translated version, which is
 after their improvements, you can see the executability has risen dramatically while
 the correctness is a bit lower. Like you get a bit lower in correctness because of the whole
 translation procedure and so on. You're mocking with the outputs, humans may not like it as much.
 This is all stuff we're going to touch on in the interview. Just interestingly highlighting that
 codecs, like the codecs model seems to be scoring quite well on these tasks. So also the translated
 codecs is much smaller. However, it scores high, really high. So parameter for parameter,
 the codecs model is actually pretty, pretty good at this, which was a surprise to me. So I think
 this is an exciting paper. It except as I said, for a fine tuning baseline, it turns out to work
 completely without any training. It's just evaluation, so to say. And I liked it. And I
 think this does have applications like getting the knowledge out of these large language models is
 something we should, you know, be getting better at doing. Otherwise, I don't think we make full
 use of them. All right, so now I want to jump into the interview with Wenlong. I hope you enjoy that
 as well. Tell me how you like these, these videos with the interviews without the interviews,
 anything you want in the comments. I'll see you. Bye bye.
 Welcome everyone. Today with me here is Wenlong Huang, who is the first author of the paper about
 language models as zero shop planners and very, very happy to have you here. Welcome Wenlong.
 Thank you, Yaning. Yeah, super, super happy to be here.
 This is, I've already told you about this paper is different and I like different papers. And
 it's, it's different in a way that maybe wasn't expected every, it seems like every day,
 we find a new applications for these large language models and yet another thing that they
 can do here. And when I, when I saw this, I was reminded of a friend of mine who had like
 similar ideas, but it never really materialized. I tried some of this stuff as well, combining
 large language models with planning with telling me what to do in the real world. I even made a
 video where GPT-3 told me a recipe and then I cooked the rest, like me and my friend, we cooked
 the recipe and so on. But it seemed like always a bit, a bit out of place, a bit, a bit off just
 to give you detailed instructions. And when I saw a paper that was really trying to make this work
 in a real environment, I was, I was very happy to see that. And yeah, that's, that is, that is this
 paper. And also, to be said, you have a, you have a stellar board of, of co-collaborators right here.
 How, how did this come about? Like, how did you even get to the idea, hey, I could use
 these language models to do planning. Was it like, did it immediately come to you? Did it sort of
 build up from some basic idea or what was the process? So yeah, thanks for the briefing. So I
 think that's actually came out to be really surprising to us as well. So first we were just
 having, when we just playing around with the largest language models on the, on many of the web
 interface, we found that like, actually there is something there, like you said, if you ask it for
 a recipe or we actually originally study, like whether it can output the steps for making coffee,
 et cetera. So we found that like, when the models get large enough, there's actually something there.
 And this is the sign of life, I think for us to kind of go on and investigate how we can make that
 actually useful for, for agents. So we kind of just started from there and actually it came out to be
 pretty surprising originally without like, maybe we need some training data sets to maybe like
 train something, a translator or something to actually make it useful. But it turns out like,
 but we really trying to constrain ourselves in the meantime, because we don't want it to be
 tailored to a specific environment. So we would just want to see like just the language model
 itself, like how well it can do, how far it can go. So this is what got us in the end.
 We just like explored for like two months and then found like you can actually do this without any
 any training. And yeah, it's actually truly surprising and actually a really fun project for me as well.
 It sounds like fun.
 Yeah, just trying to see whether you can output something like really realistic and really fun.
 Yeah. So you came across this environment right here, this virtual home environment. Was this
 always the plan or why did you choose like there are a million environments, OpenAI,
 Jim and these Mojoco kind of robot simulations. Why was this one particularly useful? Did you
 immediately think of this one or how did this came about?
 Thanks. Yeah. So actually I wasn't doing too much research in this in body agents area,
 especially for this like really high level tasks. And then I actually went to the like Google
 Scholar and then search for appropriate environments for this. And we found this virtual
 home environment and we really liked it because it actually can model any any tasks that we
 can express in terms of this like textual language plan. Like just like textual plan.
 So and actually there are many other environments as well, but some of them are limited by,
 I think a lot of people also use Alfred environment. That's a really good environment
 too. And I think it's a bit more structured there, but the tasks are often come from
 like a template. So it's usually like pick something, pull something. But actually there
 are a lot of challenges there. I think it's a different set of challenges. And we found like
 what the virtual home tackles is exactly what we look for because it can model like any task
 expressed in free form language, especially those like really challenging tasks like people do
 actually every day, like make breakfast, make tea, make coffee. And then it particularly cares about
 the common sense constraints in them. So specifically this environment has a set of like
 preconditions and post conditions for each action. So for example, if you want to grab a glass of
 milk from the fridge, you can't just like say go to the fridge and grab glass of milk because you
 first got to open the fridge first and then like preferably you want to close the fridge afterwards.
 So it's really this like these constraints I think are really useful and really interesting
 to study whether the language models can handle this. And you've investigated several different
 language models. And just to be clear, this environment, it has this kind of syntax, it has
 very defined things you can do. And somewhere I think you say it's about 50,000 actions that
 are ultimately possible. It's kind of a combination of a bunch of verbs, which are grab, open, go to,
 and lift or things like this, and a bunch of objects like kitchen, fridge, and so on. So
 any plan would consist of a sequence of verb object, verb object, like here, walk to kitchen,
 open fridge, grab milk. So any planner in this environment would have to output this syntax
 directly. Now you had a plan of not training anything, right? You didn't want to train anything,
 you simply wanted to investigate what knowledge is already there in the language models. And you
 came up with kind of a way to translate that. You want to maybe elaborate how do you query these
 language models and how do you make them actually conform to the syntax here?
 Of course. Yeah. So the way that Virtual Home expresses these actions are via this
 specific format where you put a square bracket for the action, atomic action, like grab, put open,
 and then you put, I think it's a parenthesis or something for the arguments. But the problem is
 we can't just expect language models to handle this because even if we put an example in front,
 maybe they can do it, but it's definitely not the way that usually humans produce language.
 And after all, these language models are trained on human text. So we decide maybe it's not the
 right way to query these models. Have you ever tried letting them output directly the syntax,
 or was it just like, yeah, it's not going to work anyway?
 I tried briefly, but it's definitely not thoroughly investigated. And intuition-wise,
 I think it's definitely to use natural language. But we did adopt for the most basic approach that
 we can think of, which is just define a straight up template for each atomic action. And actually,
 because these atomic actions are simple enough, just walk, grab, and those things. So
 this atomic action, I mean, the templates we actually came up with are, I think, actually,
 just in a natural way, people say things. So turn off something, turn off something,
 and then add some words in between, like in, on, on top of, et cetera.
 Yeah. And then you just query these models, and you have multiple ways of evaluating this,
 right? You care about two things, you care about correctness, and you care about executability.
 And in at least, so you also make use of humans. How did you design, like what was your thinking
 behind designing the evaluation? Yeah. So actually, it came out to be really
 challenging to evaluate these things. Like I said, so like this task art, because they're
 expressed in free form language. So that means they're really open-ended. So it might be
 deterministic, whether like if you want to grab a glass of milk, you just want to look in the end,
 whether you have a glass of milk. But if you really think about it, if we don't want to constrain
 anything in the task that we want to do, like making breakfast, like what is the correct way
 to make breakfast? Everyone has different preferences. So it's hard for us. Actually,
 I think it's still a challenge in this sort of task is like really determine the correctness.
 I'm sorry. It's the success rate for each task. So you can't really tell if a task is really
 successful depending on how open-ended it is. So we decided that, okay, so if it's hard to
 computationally produce a metric for a success rate, but as humans, we can definitely tell
 if it's making something semantically meaningful. So we'll just use part of human evaluations
 to do this. But we don't want to entirely rely on humans because as you can tell for the
 tasks that like, for the action plan that real language models generate, they're so realistic
 that they can even fool many humans that are too realistic. So you can't just entirely rely on
 humans to say if it's successful. So we also use this metric executability, which is also used in
 past papers that uses virtual home. So we just use this metric as well to basically determine
 whether the plan satisfy the common sense constraints in this environment, namely just
 whether you make sure to open the fridge before grabbing something from it.
 It's interesting because when the humans raid it, the humans would also skip a bunch of steps.
 If you tell a human, go to the fridge and grab a glass of milk, the human will go like, oh yeah,
 of course. Which is one of my, maybe this is jumping ahead a little bit, but one of the
 questions I had most when I read this was just there is a level of specificity that is required
 right here, which is kind of ambiguous. You have a high level description, which is like make
 breakfast, and then you have a bunch of steps which you need to follow. And sure these steps
 correspond to actions in the environment, so they're kind of given by that, but the language
 model doesn't know that. The language model just knows I need to produce a plan. So how is the
 language model, why do we expect the language model to figure out that it needs to say open
 the fridge before you get a glass, but for example it doesn't need to say put one foot in front of
 the other foot in order to walk. So did you have any insights or concerns with like, there seems
 to be like a very specific level of specificity of these plans? Yeah, so that's a really good
 question. Actually this granularity actually comes from the dataset or the virtual whole
 environment itself, because we essentially follow the format of virtual whole environment,
 and also this dataset they collected from humans of how to do this really human activity task.
 So the way they collect, they build this environment is they first ask many humans to come up with a
 set of tasks that they do in everyday household, and then they ask a different group of human
 to come up with a detailed plan that can drive a robot to perform these tasks. And it's after that
 they build this environment based on the verbs used by those humans. So you can think of like
 this environment is really built on top of what humans say. Now the developers who just say like,
 okay, we want this granularity, we want this like walk, grab, and those etc. So they actually ask
 these humans to give those verbs and then build those actions according to those verbs. And
 they did make sure for each of the verb to develop a set of common sense constraints, which
 completely makes sense. And I think they're actually like reasonably exhaustive for those
 actions. So if you want to grab something, you definitely need to make sure the things you grab
 is not within a closed container, for example. So in this case, the fridge is a container and
 it has this attribute of being open or being closed. So they internally keep track of the
 attributes for each of the object. And then to make sure that if you do something like this,
 you don't violate the common sense constraints. So to answer your question, this granularity
 really depends on the humans. And I think this is where language models really shine because
 essentially language models are trained on human produced text. So my hypothesis, although this
 is definitely not something they're only tested by, my hypothesis is that because it's trained on
 human produced text, and humans after all produce these actions. So if you do it careful enough,
 and then use some techniques to properly translate them or doing something else, you can essentially
 get back something similar to what humans produced in the beginning.
 Yeah, I mean, you would imagine that sort of the human-ness of how the environment was built
 would also be present a little bit in these language models, which makes sense. I don't have
 a better idea of how to build an environment like this. So it seems pretty reasonable.
 Yeah, it's actually not to be really interesting to me because it's super hard for me if I were
 to develop this environment, how would you even animate all of these really human tasks
 even just in a household setting? It's super difficult. And I think they did a really good job
 here. And then I think this is also what makes language models particularly useful for this
 task because these are basically just human tasks and language models are really good at
 mimicking humans. Yeah. Yeah. So on the left here, we see a bunch of models that you've evaluated
 right here. So again, executability is sort of how, if it matches the syntax of the environment,
 if I can map it to that, and also, I guess, if it violates any of these common sense constraints.
 So just like how executable is the plan in the environment, no matter whether it's the wrong
 thing, right? And that comes in a second. And correctness is a thing that is rated by human
 annotators. They look at the plan that was produced and they just, from their own intuition, are like,
 well, is this a good plan to make breakfast? Yes or no. And we clearly see there's this downward
 trend. If we exclude the models on the right, there is this trend line here where the larger
 models, they seem to produce more correct plans, which means plans that the humans like more,
 but they are less executable. Whereas the smaller models, they are less correct, which we can,
 that's correct. I would have expected that, but they're more executable. And you've noticed in
 the paper that very often they just produce plans that have nothing to do with the task description.
 They would just produce like a plan that is according to the syntax of the examples that
 you give in the prompt, right? But how can you explain that? Like even on the top here, like
 the large models, it's even better than humans at correctness. So humans rating other humans
 think that GPT-3 produces more correct plans. Why is it so bad at executability?
 Yeah. So there are actually two questions that I think you raised. One is why this smaller models,
 like when I say smaller, it's actually still pretty large, the largest GPT-2 model. So why
 do they produce more executable plans? And the second question is why the GPT-3,
 the largest GPT-3 model is actually better than human. So to answer the first question,
 I think that's because we did find some failure modes here for smaller models. I think the two
 most prominent ones are first, it frequently tries to like repeat the given example. For example,
 you give it like how to browse internet. You said like go out to the computer and type on the
 keyboard, et cetera. And then you ask it to brush teeth. It still goes to the computer and then type
 out on the keyboard. So it's totally nothing like sensible here. And then the second source of error
 is sometimes it just outputs really short plans. If you say like sleep task, go to sleep, it's just
 like go to the bathroom and just stop. So that's this right here, brush teeth. It's just like
 go to bathroom. Yeah, exactly. So when these plans are short enough, even though it can be
 executed, if you just say like walk to bathroom, walk to the bathroom, just one single action,
 for walk, there is not much common sense constraints there. So you can totally imagine
 it's super executable. But if you present them to humans, of course, humans will spot this and then
 say, okay, this is not correct. Because when we do human evaluations, we're trying to make it simple
 so that the error here is not too big, because we don't ask hundreds of humans to evaluate this.
 We only got to ask 10 evaluators in this case. So that's why this smaller models are now really
 good at escalability. And the second question that you ask is why these larger models are actually
 better than humans. So actually, this is now the completely fair comparison if you just look at
 one axis. So all the results here, we look at from two axes that we care about. So one is the
 semantic correctness, which is evaluated by humans. And the second is the executability.
 So this human plans that we use are from this data set that virtual home developers
 cross source from Amazon Turkers. So these plans, they make sure that these are executable plans.
 So which means that they have one hand here. They'd be over here.
 Yeah, but we don't want to put a spot right there on the right, because it's hard to see,
 because humans are a big baseline and reference here. It's not a baseline that we're trying to
 beat. Of course, GPT-3 is not there yet in terms of at the same time outputting correct action plans
 and semantically correct action plans, and also being able to really ground them in the environment.
 But using these two axes, we can really see, for example, which axis is the place that,
 as a community, that we may want to work more on to get it better to get the human levels.
 And with this paper, we find this result actually a bit interesting to us.
 Is that for these larger models, in terms of semantic correctness, you don't need to worry
 too much about it. It's kind of already there if you do it, extract them. But the real question is,
 how do we make them executable for agents that we care about?
 And that's exactly what you do in the meat of the paper. And the result are these translated
 models right here that, notably, they do drop a little bit in terms of their correctness as
 rated by humans, but they gain massively in executability. And this is the result of a bunch
 of different ingredients, like three main ingredients, as far as I could tell. You quickly
 want to go tell what the ingredients are to make whatever these models output into something that...
 I mean, the virtual home is maybe a test bed, right? I don't see this paper being about
 virtual home. It's more like, here is a model that outputs something, yet I need the output in some
 other form, right? And this is a very general problem, as many applications. And if we could
 solve that bridge, that technically is a big gain. That's exactly what you do. So how did you go
 about this? Yeah. So actually, I just want to make sure that actually this paper just presents
 a really preliminary step. I don't think it solves anything particularly. I mean, it does,
 like if this problem... Sure, but it's a big step, I believe. I mean, the executability I have raises
 pretty high. I didn't want to oversell you, but also not undersell you, certainly.
 Yeah. But to answer the question, so we actually found there are three ingredients, but
 central to this is one really simple technique that we found that's the most useful, which is
 action translation. So because in this virtual home environment, the actions that it supports are
 a limited set. I mean, it's not small, but it's something that we can definitely enumerate with
 our computational hardware and in a really quick manner. So like just one-tenth of a second or
 something like that. So let's say if we can enumerate all the actions that are supported
 by the environment, then the question now becomes, how do we translate this really
 sensible action plans generated by language models, but not really executable plans?
 How can we translate that into those actions supported by environment? Or if you want to
 deploy something in the real world, let's say your robot only supports 10 actions. How do you
 map those tasks into the 10 actions that the robot supports? So what we found is that you first need
 to enumerate all the actions. And then we found that you can again leverage the world knowledge
 in this language models by using another language model, namely here we use Roberta, which is a
 language model really similar to BERT. And it's a different language model because it essentially
 is a mass language model. So it's really good at outputting a useful embedding. It's
 really good in terms of about the semantic meaning for that sentence. So what we do is that we
 take the sentence output by GPT-3 or codecs, and then we just compare that against all the possible
 admissible actions, allowed actions by the environments. And then we found the
 most similar one in terms of this distance in the embedding space. We actually use just
 cosine distance and found that to work decently well. Yeah, there's an entire space somewhere,
 and you just place all the actions. I guess you can even pre-compute those. You can pre-compute
 the embedding of all possible actions there. And once my language model outputs anything at all,
 all I need to do is ship it through the Roberta model, get its embedding, put it somewhere,
 get the nearest neighbor. And that's my translated action. So here we have an example that would
 translate like squeeze out a glob of lotion into pour lotion into right hand. So it would map
 action into and pour, it would be the verb lotion, the object and right hand also one of the objects.
 So maybe there's two arguments to pour. It seems very simple, but I was at a talk
 by the people who made the first version of the... In Gmail, you have these always three options to
 respond to, like the quick options to respond. And I think the first, I'm not sure how it is done now,
 but the first version of this, we were like, wow, this is cool. It actually takes into account the
 email message that was there. We always thought it was kind of like a language model, generative
 model somewhere. So I went to a talk and they were just like, no, we just have a big list of responses.
 We just classify, right? Whatever. We just take your message, right? And we just put it through
 a model and then we just classify into this big, big bucket of possible answers. So I mean, this is
 even though it is simple, it's a very powerful method. And that being said, you don't even
 train this. You take an off the shelf embedding model and you compute nearest neighbors and it
 does turn out quite well. You do, however, you talk about this in the paper, there is a bunch of
 problems. And one of the problems I see is whenever a step contains like multiple steps, right? Is that
 like, is that a big, have you found this to be a big problem? Because this just maps one action to
 one other action. But if it's like, you know, open the fridge and take a glass of milk, then I have
 essentially no way of translating that into an admissible sequence. Yeah, that's a, that's a good
 question. And I think that's one of the main errors that like this, this Roberta model that we use,
 it's actually a sentence Roberta model because it's trained with a different objective such that
 it can really, you can actually calculate cosine distance between the embeddings they generate.
 So it's a, like we found like it's pretty difficult to map a compounded action. Like you said, like
 two actions in one sentence into one admissible action. But this is partly mitigated by how you
 tune the temperature, the sampling parameter, just the temperature for the GPT-3 or codex models.
 Because we found that if you do increase the temperature, then it tends to output something
 more verbally expressive answers for each step. So that means it's harder to translate. And we,
 if you, if you try like all this, like different settings, we did, in the end, we found like,
 usually you want to use like a lower temperature than what people mostly use for language generation,
 for example. So that like each action is like small enough and succinct enough. And then,
 and then after we translate this action, so that it's easier for this bird model,
 Roberta model to translate. And yeah, something I forgot to mention, like after we got this
 translated action, we found that it's still useful to put that back to the original prompt,
 put the translated action back instead of like the original action so that you can add the GPT-3 and
 codex model to reason, like how am I going to do based on this like action already performed?
 So yeah, like you said, like you pointed, this is the third sub figure here.
 So we would take instead of instead of generating the entire plan at once, we just generate
 one action, then we translate it. And then we substitute essentially whatever GPT-3 output
 with whatever the translated thing is. And then based on that, create the next action. It makes
 sense because you it's like almost like a guiding, like a bit of a guardrail for, for the language
 model. Instead, if you were to let it generate all at once, and then you translate each action
 individually, they almost like lose connection to each other, right? So this, this here might mitigate
 some of this, this stuff ready, if I have a compound action, like go to the fridge and grab a glass,
 and the closest, I hope that the closest sentence is to go to fridge, right? The language model might
 still recover and recognize, aha, I haven't, you know, grabbed, haven't grabbed a glass yet. So that
 is, so these are improvements one and two. And then the third, the third thing you found that really
 helps is the prompt up here. So the priming, which I think in GPT-3, it's very common to have these
 priming prompts to tell the model what kind of stuff you, you expect as an output. I was surprised
 to see that you only have one priming prompt. Whereas in general, people put more than one,
 usually people put like three or something like this. Is there a particular reason why you used
 just one? There is actually not a particular reason. I actually found like, I mean, in the beginning,
 we were, we know that we have this data set, right? And then we, we found, originally, we actually
 tried to train something to achieve this, but in the end, we found that like, we don't even need
 to train something. And like, now the question becomes like, like, can you even leverage this
 data set to some extent to make it useful? Of course, this is something like additional, I mean,
 it would definitely be better without any, any of this. But if you have this data set, you can
 actually found like this most similar example to the query task here. For example, like this is
 apply lotion. So like, shape, task shape is determined to be most similar. Again, judged by
 this Roberto model using the same technique. Yeah. So I think that that's the, that's the main
 motivation for using this, but we didn't thoroughly investigate it, like how you structure the
 prompts, whether you add like multiple things there and then, or you change the template here,
 because I just defined this template from day one, like task something, step one, something,
 two something, maybe there is a better template. Maybe you want to add some instruction there to
 make it better. And so I like, I mean, this is definitely possible and we don't investigate them
 here because we don't just want to get the best performance out of this. We want to get the best
 performance out of this. We want to show people like, this is something possible and it's really
 interesting to us. So that's why we ended up like, like just using the most simple technique here.
 Yeah. And to answer your question, why we don't put multiple things there, I think one important
 reason is like, because this example plans that we put in front are produced by humans. And this is
 because due to space constraint, I'm using an oversimplified version in this figure specifically,
 but in practice, these plans are actually pretty long. So, and they actually already take up a lot
 of space in the prompt. So if you put more than one, sometimes it gets too long. And I mean,
 it's maybe something handleable by larger models, but we just opt for the most similar,
 most simple case. And I actually read this, like there's a recent paper investigating why
 in context learning works, they frame this as a implicit Bayesian inference problem. And they did
 come to a conclusion that the longer the prompt, if I remember correctly, like it helps the model.
 So, in this way, you kind of like trade off the number of examples you put and the length of each
 example. So in those cases, I think you mentioned many people put many examples before the query.
 Those are usually the cases where the tasks they care about are like smaller. So for example, like
 you want to ask Einstein was born somewhere, then like this is just a sentence. So you probably want
 to put like more than one sentence there. But this case, our case is like, it's an extensive
 action plan. So it's already pretty lengthy and we don't want to go too crazy over here.
 I mean, it's, yeah. Sorry, the recording has stopped on the screen side, but we can still see it.
 Okay. Yeah. So yeah, I was quite interested in the sense of the prompt structuring,
 because I know that can also make a big difference. But I also like the sort of approach of not having
 too many moving parts in one single thing, because it makes things complicated. And for many papers,
 it makes you wonder like what was exactly the thing that gave the improvement here. Now you
 do very good ablations of all of these different improvements, which I really liked. And you showed
 that kind of the translation is the main part right here, although the other things certainly
 also help. Have you ever, so it reminds me a bit of this, you know, this retro model,
 these language models that retrieve from the internet as they produce text, it reminds a
 little bit of this, right, in that you produce, you go and retrieve the closest samples in the
 data set as you produce the text. Yeah, I think this combination of retrieval and generation
 is picking up steam. And it looks pretty interesting. My question is a little bit,
 have you tried also, because essentially, you now rely on this translation procedure to produce
 the correct actions. Have you tried any way to like let the model know what the possible actions
 are? Like something like, you know, I can imagine maybe I, you know, I ask the model first, and then
 I get maybe the five closest actions or the 10 closest actions in embedding space. And then I
 somehow put these in the prompt here, like, you know, in between, you know, what am I going to
 do next? Is it this or this or this or this, right? And then the model could, maybe I could prime the
 model to output one of them. And, you know, is there, did you try any way of telling the model
 more what's even possible in the environment? Because right now you're essentially relying on
 on just the language model itself. Yeah, that's a really good question, too. So like, we actually
 didn't try the specific thing that you talk about, like generate a bunch of possible actions and then
 ask the model again, which of these are the best. But we did try something similar, which is
 like Beam search. So essentially in Beam search, you look ahead to see like what the outcomes are,
 are like having in the end get the highest likelihood. So we did try to constrain the
 strain the vocabulary that can be used in the Beam search. But this is only conducted on smaller
 models, because obviously the GBT-3 and codex models are now open to fully open to public. So
 we can't, we don't really have full access to different features. Like,
 you can't restrict the vocabulary dynamically. Yes. So I've only done this on smaller mode,
 relatively smaller models like the GBT-Neo. And then I think I might have tried on GBT-J as well,
 which is a 6 billion parameter model. And it actually turns out that they don't do really
 well with if you really just constrain the vocabulary that way. And yeah, specifically
 just the Beam search constraining the vocabulary can generate. But so my hypothesis, this is now
 thoroughly tested because it's now invested on larger models as well. But my intuition why it
 doesn't work so well is that this language models are really trained on human text. So it really,
 they're really used to how humans speak a certain language in this case English. So like people
 don't speak things in this way, step one, something, two, something, step three, something. So that's why
 if you really constrain the models this way, a lot of the world knowledge encoded in these models are
 lost. So basically, and personally, just a personal opinion, I don't think these models are doing
 super intelligent reasoning here. It's basically just doing kind of retrieving what's
 what is trained on. So, retrieving this large scale text. So if you want to retrieve better,
 you better adopt the same way that humans speak a language. So like if you don't constrain the
 vocabulary, you can get the most out of a language model. And you can really tell if you adjust the
 temperature. Like if you go different temperature, they can tell you like different levels of things
 and they can be really realistic. But if you really constrain it, a lot of this knowledge is lost. And
 it can really do too much like common sense reasoning here.
 I was, you mentioned this a bunch of times, I was surprised to find codecs as a model. And so you
 have, these are sort of vanilla models. And then you have the translated ones where all your
 all your improvements are in there. So there is the action translation, there is the sampling,
 even according to the probability and executability, there is the retrieval of the
 closest prompt and so on. And these translated models, they perform really well. What I was
 surprised by and also by the results is that codecs, I mean, that it's even in here, it's like a code
 model, but also that comparably, it holds up, right? It's not as good as the GPT-3 model, but
 it's also very, very much smaller. So parameter by parameter codecs is outshining GPT on this task
 very well. How did you even consider using codecs? And how can you explain that this model is
 doing so well? Yeah. So one intuition why, this actually came out to be pretty surprising to us
 as well. So we did find like this codecs models are really good at generating these plans. And
 actually from my own experience playing with this models, I did find like codecs thinks that this is
 part of some doc stream. So it's actually imagining like people just like asking the doc stream here,
 but instead of letting keep generating the code, we kind of just stop here. So, okay.
 Yeah. When it's the doc stream for us, that's enough. So yeah, so it's actually doing some of
 this kind of doc stream. It generates this doc stream thing. And the reason I think the smaller
 codecs model are actually better than the same size GPT-3 model is that because it's trained on
 a more structured data. So like code and specifically many of this code examples
 in the training data set consists of doc stream and the code. So it not only can handle code really
 well, it can also generate really realistic doc streams. So, and people in doc stream, they don't
 write in like... Yeah, they don't write a novel. Yeah. So they write something really step by step
 and have more structure in it. So that's my intuition why it actually does really well with
 this task. So you can really process this sequential like logical reasoning better than the same
 size GPT-3 model. But of course, if you use a larger model, that potentially be more helpful.
 Yeah. Or I mean, there is, as you said, there is still a lot of open questions about how exactly
 you structure the prompts. Like maybe this step one, step two, step three isn't ideal for these
 language models. Maybe you need to more let them write like a Reddit post or something about
 how they went and got a glass of milk yesterday and then translate that somehow. But yeah,
 it's pretty cool. So one thing that just came to my attention right here is this top row right here,
 which I found hilarious. So the task is complete Amazon Turk surveys. So the four steps apparently
 that you need to do is walk to home office, sit on chair, switch on computer, look at computer.
 Like, is this the description of complete Amazon Turk? It's a pretty accurate description maybe of
 what Amazon Turk workers do. So like I said, these tasks are generated by crowdsource from humans.
 And the humans here happen to be Amazon Turkers. So one of them decided that, okay, if you want me
 to generate some tasks, I would say like just complete surveys on Amazon Turkers. Yeah,
 so they decided to put one of this here and we found this here, there is two. So like I said,
 so this language model, so they can't really handle anything that you wanted to
 generate. So because we did put the example in the front. So I think in this case, the example
 happens to be something related to computer and the example is that you can't really see
 the models actually happen to reason or potentially you could just repeat the example.
 But depending on other tasks, it doesn't seem like that's the case, but it does come to the
 reasoning that like this might be something related to computer too. And I'm going to put
 like this steps here. Yeah, yeah. I mean, this is, I mean, it has something like melancholic
 and it also has something a bit, as you said, rebellious of like, you know, I'm here doing my
 Amazon Turk work, I'm gonna, you know, I'm just gonna put my Easter egg in there in this data
 set or like show you, but it also shows something I think about the interaction with this environment
 because, you know, if you ask me, you know, what did you do today, I could tell you, you know,
 I programmed this, I viewed a poll request, I sent some email and so on. But in the action space of
 this environment, this would all just be characterized as go to desk, sit on chair, switch on computer,
 look at computer. And yeah, so it is really, maybe also a constraint of the environment itself. And
 as I said, I think the challenge is going to be there's so much knowledge in these language
 models, and we somehow need to get it out into the domain that we care about. And yeah, I guess,
 I guess many opportunities are still there. And in this particular environment, is it so the way I
 see it, we have this environment, it's a 3d environment, but you never actually for your
 studies, you never actually had to actually execute anything in the environment. Is that
 correct? Or do I see something wrong here? I think those when you say execute do you mean like,
 like run in the environment? Yeah, like run the 3d environment, like actually give it to the
 environment, because you evaluate executability, you can do with a parser, right, to see whether
 it matches the actions and constraints. And the correctness you evaluate with the humans,
 because my question was also a little bit like, why can't I just run it and see if, you know,
 at the end, there's breakfast, but you already, you already said that the tasks are so, so open,
 like, how would you how would you detect there's breakfast, right? So, so, in terms of so a bit
 background here for the virtual environment. So it comes in two versions. One is the, I think
 that they call the evolving graph version, which is a pure, like you said, a state machine, a Python,
 like reading in Python. So it just goes in and then checks which whether the actions can be parsed,
 and then we satisfy the common sense constraint. And the other version they implement is this,
 is this visualized version, where they actually only implement a subset of
 the act the total action supported in the environment. So I think they, so in the
 evolving graph version, the Python version, there are 42 actions. And in the visualized version,
 there are only 10 actions. So it's limited. Like the plans we can generate, we can really
 visualize are limited. So that's also part of the reason we don't show the visualized version to
 humans. Like, can you tell us whether this is successful or not? So, yeah, that's, that's a,
 that's indeed something we can do right now. And I think that's like as a community, as we go,
 go on, like, to this next step with more complex tasks that humans do every day, instead of just
 like, lower level tasks. As a community, I think more efforts can be can be put here and
 to develop better simulator and also maybe beyond even household environment. So just as a,
 as a story here, I did play around with the codecs and then GPT-3 models to have it generate
 something out of the household domain. And seems like they do have some, a lot of knowledge for
 those as well. So if you can ask it, how do, how do I pay bills at a restaurant? And how do I
 work out at the gym? And I think in, on Twitter, there's also someone tries to, after the posting
 of this paper, they try to ask the GPT-3 model, how do I start a company? So yeah, they do have
 a lot of knowledge for this. And as long as you can provide a set of actions that are necessary
 to complete these tasks, I think no matter what, what the granularity is, ideally it should be
 at the same granularity as of humans. So ideally it should be, this model should be able to
 generate something, something sensible and reasonable. But yeah, right now is something
 that you definitely can't trust to put on a robot, of course. Yeah. Yeah. I mean, it's,
 I've always, I've always seen people thinking when they think GPT-3 or so they, they, and they think,
 for example, of video games, they always imagine, you know, we can have our NPC, our characters,
 the dialogue be generated by GPT-3. So it, the dialogue is more realistic, but I think
 this shows that it can go further if we are able to map sort of GPT-3's knowledge into a sort of
 structured domain that we choose, we could potentially also let these models generate the
 action sequences of like, of characters, for example, let's say in video games, because that's
 like a common complaint that, you know, the guards, they always walk up and then down and then left
 and then right and then up and then down and right. They have these, even if the dialogue
 gets really good, their behavior is still kind of lame, either that or they cheat, they know where
 you are at all times. But with, I feel with models like this, we can almost like take this common sense
 knowledge and maybe have the hopes of transferring that to various domains and infuse a lot of areas
 with common sense. And that I find that to be, I find that to be pretty cool in itself.
 That would be really exciting and interesting application. Yeah.
 Yeah. Yeah. So I mean, there's a lot of things to be gained. So what I did, I was specifically
 intrigued about clip. I don't know if you are thinking about this or not. But what I tried to
 do is I tried to take like a frame of Pac-Man, like, and you know, there's like walls here and
 here and here. And I had Pac-Man be like, you know, here facing a wall. And then there's like
 a ghost behind Pac-Man, right? And then there's like these little dots over here to eat. And so
 it was like super clear what you have to do. So I tried to feed that to clip. And you know, you can
 make clip classify things by just evaluating a bunch of different strings with it. So I like try
 to, I try to evaluate the strings, go left, go up, go right, go down, or like Pac-Man should go left,
 Pac-Man should go up, but it never worked out. So if you can, if you could get something like
 this running, this would be amazing. Maybe with your knowledge, maybe Pac-Man isn't the right
 environment because clip was trained on whatever picture scraped from Instagram. But I think just
 this this type of, you know, thinking beyond just the strings in terms of language, but thinking in
 terms of I have some structured environment and I want to leverage this, this knowledge of these
 models is super cool. Yeah, that would be a super interesting application. I think using clip here,
 like, because it feels in another modality, which is image could be really interesting. So I think
 it kind of solves one of the major limitations of this paper, namely just the, because currently
 we generate plans regardless of the environment state. So it doesn't condition on environment
 state and potentially using clip, you can encode something there because you can also take image
 as input to, to an image can serve, can serve as state for, for, for the environment. I think
 that would be really cool. And yeah, so yeah. So just to be, to be clear to the listeners,
 the basic idea for this I have from, from a PhD student that was partially in our lab called
 John Battista Parascandolo. So the, the credit fully goes to him of, of this whole idea. I didn't
 want to, but I just, it got me thinking so much about, you know, we can extract this knowledge
 into, into other modalities. And that's, that's pretty cool. Is there anything you want to maybe
 say about the experiments? Is there anything that was very surprising to you or, you know,
 something you didn't expect or something you particularly want to highlight?
 Actually, I think we covered most things, but I think I might say something about the, the,
 the baseline here. I see, you can probably see, except for the human references, we also got to
 got to fine tune a GPT-3 version. And we did find that fine tuning can, can be a really strong
 baseline here, because as you can probably tell the, one of the measures here, LCS, which is the
 longest common subsequence. This measure here is much higher than the others. So this measure
 basically calculates how much overlapping there is in your generative plants against the
 those plants written by humans. So it's kind of calculating this IOU score. So we did find that,
 find this to be a strong baseline. And I think it still actually makes sense to, to be a strong
 baseline because this is trained on such data. And so this is kind of to illustrate that, like
 if you do have domain data, it's still really helpful to, to train your models, fine tune your
 models this way. But if you don't have something like this, you can potentially just leverage the
 knowledge already in this language models. Cool. Yeah. So where, where does your future lie? What
 are you, I, I, are you going to, are you going more into this direction? Or was this sort of like a
 one-off thing? Or do you have, I mean, what are the interesting questions that, that you are asking
 now maybe as a follow-up to this? Yeah. So I personally, I haven't decided because I,
 I'm in a stage where like I'm applying to PhD programs and, and, and also other positions.
 So like, but, but as a follow-up, I think it would be really interesting. As I mentioned,
 one limitation, major limitation of, of this work is that we haven't found a clear way to
 condition on the environment state. So that like, if you really place an agent in, in the household,
 for example, there is no, if you want to make coffee, but there is no cough, but there, there's no,
 there isn't a automatic coffee machine. How would you make a coffee with some, maybe a similar
 devices. So the agent can really reason if you just put it this way, because it doesn't condition
 on the environment state. So I think it would be really interesting to like investigate how you can
 also condition on the current environments and then, and then reason from there. But this might
 require some training data. And I think that's part of the reason why we don't like go full length
 here to investigate this, because this is something just for us to tell people, like this is an
 interesting finding and we may be able to leverage something here. But I think this will be really
 exciting and like interesting future work. Cool. Excellent. Wenlong, thank you very much for being
 here. This was awesome. So great to hear from, you know, from always from the people who made the
 stuff. So yeah, thanks a lot. Yeah, thank you so much. Yeah. And yeah, I think I also want to
 also want to like point that like, this is a group effort and really a lot of thanks goes to
 three of my advisors, Peter Bill, Deepak Pathak and Igor Mordac. Excellent. All right. Thank you.
 And I hope to see you again. Yeah, I'm like, it would be an honor to always to be here. Yeah.
 Excellent. All right. Bye bye. Yeah. See you.