Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan - Paper Explained)
#saycan #robots #ai

Large Language Models are excellent at generating plausible plans in response to real-world problems, but without interacting with the environment, they have no abilities to estimate which of these plans are feasible or appropriate. SayCan combines the semantic capabilities of language models with a bank of low-level skills, which are available to the agent as individual policies to execute. SayCan automatically finds the best policy to execute by considering a trade-off between the policy's ability to progress towards the goal, given by the language model, and the policy's probability of executing successfully, given by the respective value function. The result is a system that can generate and execute long-horizon action sequences in the real world to fulfil complex tasks.

Sponsor: Zeta Alpha
https://zeta-alpha.com
Use code YANNIC for 20% off!

OUTLINE:
0:00 - Introduction & Overview
3:20 - Sponsor: Zeta Alpha
5:00 - Using language models for action planning
8:00 - Combining LLMs with learned atomic skills
16:50 - The full SayCan system
20:30 - Experimental setup and data collection
21:25 - Some weaknesses & strengths of the system
27:00 - Experimental results

Paper: https://arxiv.org/abs/2204.01691
Website: https://say-can.github.io/

Abstract:
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at this https URL

Authors: Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, check out this video. So there's a Coke can and there's a spill, a Coke spill. So the instructor here says, I spilled my Coke on the table. How would you throw it away and bring me something to help clean? So the robot here forms a plan as it goes about it. First, it says I would find a Coke can. Then second, I would pick up the Coke can. You can see it has done it. Third, I would go to the trash can. Fourth, I would put down the Coke can. Note that he puts down the Coke can next to the trash can, not in the trash can, because the robot is environmentally friendly and wants to preserve the can for the recycling bin for cans. And, you know, it doesn't belong in the trash. Good little robot. So next it says I will find the sponge. I will pick up the sponge and then will it clean the Coke? No, it will not clean up the spill. It will actually give the sponge to the human to clean up the spill, because that's how the future is going to be. The robots, they're not going to take our, you know, people always think the robots will take our dirty jobs. They'll take all the like these tasks like cleaning and doing things. No, no, no, no, no, no. They'll abuse us, the humans to do that. They'll just throw down our stuff. They'll throw down the sponge and be like, here human, clean up your own mess. Well, if that's a future that you look forward to, too, then join me in today's paper. We're going to look at do as I can, not as I say, grounding language in robotic affordances by researchers at robotics at Google and everyday robots. So as you saw in this video, what happened here is that from a simple instruction that the instructor gave this essentially this I spilled a Coke can, you know, please help me find something to clean and throw it away. The robot formed a plan, the plan you can see at the very end here. You can see it developing in the bottom. At the very end, you can see the full plan. I'm not sure if I can make this bottom thing go away. In essence, it makes a plan like it always plans the next step or at least it determines what the next step should be. And then it actually also does it. So this is a good example of grounded, a grounded language model, or also an example of embodied intelligence. This work connects large language models and the knowledge that are that is inherent to large language models with the skills of robots that act in the real world, which really cool. And usually these two things are quite disjoint, but this could be really powerful. So we're going to look at this paper. I also have already recorded an interview with the authors this for time reasons. We did it the other way around this time. So I don't want to take away too much on the paper review right here. I'll tell you what the method is about how it works. And I'll leave the rest to the authors who are extremely competent and I learned I learned like I learned a lot in the interview. I hope you will too. In any case, the interview will be out tomorrow. If you're watching this the day it comes out, which obviously you do. How do you find new papers? Frankly, machine learning has become unbearable. There are thousands of new papers each month. And to keep the overview, we need good tools. Today's sponsor is Zeta Alpha, which is a search and recommendation engines for papers. This is really powerful. For example, here I've searched for today's paper, say can you can immediately see that not only do I get the paper itself, but I also get an aggregation of all the social media mentions of this paper. That doesn't stop there with one click, I can find related papers. These are not only papers that are cited, but these are semantically similar papers. This is powered by neural search, which is really cool. I can further now add this paper to my tags. And what that will do is it will build categories of papers and then serve me recommendations that semantically fit those categories. This is really powerful. Essentially, this is a newsfeed of papers that is personalized to you specifically. Just recently Zeta Alpha has released their own PDF reader. This is really strong right out of the gate. Not only does it let you read the paper, you know, but also it shows the important information about a paper and it lets you take notes. Now what I find by far the coolest thing is that you can actually use one of these notes to search for other papers. So whenever you find something within a paper that seems interesting, you can use that particular piece of text and go search for other papers that might deal with the same topic. Sign up now to Zeta Alpha, there is a free tier and the pro tier is actually free for students for academics. But in case you are not one of these, the promo code Yannick will get you 20% off the pro subscription. The authors here state that if you try to ask a language model to clean a spill, as they just did in the video. So if you ask a language model to clean a spill, it might result in a reasonable narrative as we've all come to know the large language models like GPT-3 or so they give very convincing outputs. So when you ask them, how would you clean up a spill, they'll give you a reasonable plan to clean up a skill. But the authors say may not be applicable to a particular agent such as a robot that needs to perform this task in a particular environment. They have a bunch of examples right here. So I spilled my drink, how can you help is up here. GPT-3 would say something like you could try using a vacuum cleaner. Well, GPT-3 has no idea of A, whether there is a vacuum cleaner in this environment, or B, whether the robot or whatever agent is capable of executing that action. So is capable of handling a vacuum cleaner, because it's not the easiest thing to use. You have to go get it, plug it in and so on, there's moving parts. Similarly, models like Lambda and Flan, of course, they're made for different things, but still, they will pay no attention to what is actually possible in the environment. Now you can get around this a little bit by prompting, by prompt engineering, telling the model what's possible in the current world, but it will only get you so far. So we need something else, we need something better than this. And that's where this system comes in. So they say what they want to do, they want to provide a real world grounding by means of pre-trained skills. And this leads to a situation where you only consider actions that are both feasible and contextually appropriate. So these two things need to be brought together. The language model supplies the high level semantic knowledge about the task, and the language model provides and the robot itself or the policy in the robot provides the feasibility of the tasks to be executed. So the two things are brought together, contextually appropriate from the language model side and feasibility from the robot side. So how are they going to do this? They're going to combine, as I said, large language models with policy or value functions, let's say value functions, and then they execute a policy. There's a bit more explanation right here, but I think I've said many things already. We'll get to the meat right here. They say, let's say we have a robot. The robot might be, or in this case is, equipped with a repertoire of learned skills for basic atomic behaviors. These skills are capable of low level perception and control. So one of these atomic behaviors is, for example, if you remember from the video, pick up something, pick up the Coke can. That's an atomic behavior. You can teach the robot to pick up a Coke can in isolation. It can do it many times. You can train some imitation learning or reinforcement learning, or you can even hard code that particular policy. It doesn't matter. What matters is that you can train it in isolation. It is an atomic action, and these atomic actions can then be chained together to form a sequence of actions and execute a plan. So the atomic actions are going to be supplied by the robot, and then the sequencing of the atomic actions is going to be determined by the language model. They say, if we can simply make the large language model aware of the available and feasible repertoire of skills, this can provide it with an awareness of both the agent's capabilities and the current state of the environment. So if they have a large language model, many people use large language models to sample, which means that they input, they would input something like, you know, I, I is capitalized, I spilled a drink, dot dot dot, and then they would let the language model generate stuff right here, and then they would try to interpret this stuff. We've seen this in other paper, and there are situations where it can work, especially if you put like some reasonable prompt in front of it. But the approaches have been largely to just let the model generate some stuff and then try to map that stuff, whatever comes here, into the action space of the robot. But that is not always possible. Instead, what this paper does is it says, well, we can also use the language model not to generate, but simply to compute the likelihood of certain inputs. So I spilled a drink, and then let's say I just have five actions at my disposal. All the robot can do is these five actions. So I would, or let's, let's say it says, I spilled a drink, I will, and then, clean up, I will go away, I will eat pizza, I will, and so on, right? So there are these different actions that the robot has available to do, and these correspond obviously directly to these atomic actions. So cleaning up something would be an atomic action that you could train in isolation. Going away would be an atomic action. You can hard code or you can path find your way out the door. Eat pizza. Maybe these are even too high level that the way that I describe right now, but just imagine these are low level actions. And all we have to do with the language model is we simply have to compute the likelihood of each. So what's the likelihood of the sentence, I spilled a drink, I will clean up, right? And then I compare that to the likelihood of the sentence, I spilled a drink, I will go away. And then I compare that to the likelihood of the sentence, I spilled a drink, I will eat pizza. So for every continuation here in my repertoire, I will get a likelihood number. And that represents how contextually appropriate is that particular skill in this case. So how much does the language model think this skill would be useful right here? Now there's obviously an issue in how you formulate these things right here. Depending on how you formulate them, they might become more or less likely. However, I think the authors here work around this simply by the fact that these skills that they have, they are so separated from each other, there is not really too much of an issue with that. But that's kind of what my concern was when I read this. But in essence, it's a good idea, I think. So you simply for every single, wow, this all became orange, for every single continuation, you get a number, which is the likelihood of that thing. That's what they say right here. No, instead of using the large language model to integrate an instruction, we can use it to score the likelihood that an individual skill makes progress towards completing the high level instruction. Furthermore, and that's where the second part comes in. If each skill has an accompanying affordance function that quantifies how likely it is to succeed from the current state, such as a learned value function, its value can be used to weigh the skill's likelihood. It's best if we go down here and say that the skill is the best. It's best if we go down here to the diagrams of how this works so you can see how this fits together. This part here is the part we just described. Let's say I'm in a situation, this is the prompt that I put in. How would you put an apple on the table? You prompt, well, you prompt the language model with this thing right here, which has a prompt engineering part. You can see there are a bunch of examples of instruction and then a sequence of steps. Again, instruction, a sequence of steps. Here it comes again, instruction, and then here you'd get a sequence of steps. However, instead of generating, you'd simply score the likelihood of each of the skills that you have available. Find an apple, find a Coke, find a sponge, pick up an apple, pick up a Coke, yada, yada, yada until go to the counter. Each one of these skills gets a likelihood number assigned to it. That's part one. Part two is where you train the robot for these basic skills, these atomic skills. Here you can see one of these training stations where you can simply teach the robot to do things such as picking up an apple or picking up a Red Bull can. Now, what you have to do is not only teach the robot a skill, but also train a value function for it. If you do something like A2C reinforcement learning, you get the value function directly out of that algorithm. If not, you have to somehow come up with a value function that makes sense. In any case, what you want to do is train a policy and a value function. The value function is important because it tells you from a given input, by the way, the low level policy has the picture here and input. Well, obviously the language model doesn't. Now, I believe with Flamingo by DeepMind, that just came out today, that might actually change. But the low level policy has the image available. So the value function, given this picture right here, can tell you pretty quickly. My skill that's called pick up the Red Bull can, I can execute that policy and I can probably make it happen. That's why the value is relatively large here. Also for the pick up the apple action, the value function tells you, you know, given this picture right here, I can probably make that happen. However, when it's pick up the water bottle, pick up the bag of chips and so on, there is no water bottle. So the value function very accurately says, no, I cannot make that happen if I execute that policy. So the value function gives you inherently a score of given the current observation, how likely am I to succeed at a particular skill, which is exactly what we want, because that's the second part of our puzzle. So on the right here, you see another example where none of these pick up skills, picking up, sorry, not pick up, picking up skills have any value because there are no objects. But in this case, maybe other actions would score very highly in the value function. For example, go and find a sponge. Like I can always go and find something, right? And if I know there's a sponge somewhere, I'll probably succeed. So in any case, this value function now we can combine, you can see we got a number for each action from the language model, how likely is that action to progress towards the goal. We got a number for each action from the value function, which is how likely is this action to succeed given the current observation. And all we do now is essentially multiply the two things together. If they are log likelihoods, we obviously want to add them. But in any case, we combine the two numbers, and then we choose the skill that is the best trade off between what makes progress towards a goal and what is feasible currently. Here is an example. The input is how would you put an apple on the table like an apple? So we query the language model with this prompt here and the prompt engineering we've seen before. This is not displayed here, but it is the case. And the top actions that the language model gives are pick up an apple, you see that's the highest action that we have, place the apple, and only at third instance, find an apple. However, the language model has no clue about the current state, right? And that's where the value function come in. So this is the current observation. We ask the value function which skills are doable in the current environment, in the current observation. So the value function say, well, finding an apple, finding a coke, finding a sponge, these are pretty high. I could do these. I could also go to the table. I could also go to the counter, right? These are fairly doable. However, I cannot place an apple or place a coke because I don't have a coke in my gripper. I can also not pick up an apple or pick up a coke because I don't see them anywhere in the picture right here. So even though pick up the apple was scored highest by the language model, it is now severely down ranked because the value function for this policy doesn't, isn't very confident that it will succeed if you execute that right now. And therefore, the action that is chosen is the best trade off, which is find an apple. Then you can see or not see, but this is represented here that after this is done, the policy is executed. So the find an apple policy is executed. The find an apple action is added to the prompt and then the whole process repeats. But instead of asking for the first step, this whole thing is now the prompt, including the instruction. And we simply ask the language model for the second step and the input to the value function is now the current updated picture. So here you see it succeeded in finding an apple and now hopefully the second step, if we go through the same process again, is going to be the pick up an apple action. Because, well, that might already be high by the language model, but also the value function, given that there's an apple in the picture should now say, yes, I can probably succeed at that. So that's the whole issue or the whole process here. This is repeated until the end. This is the say can method. What is really impressive is just the amount of effort and work that went into designing these systems, training these systems, evaluating these systems. They have different areas here on the left. This is like a kitchen. On the right is a different environment. They have these training stations. They collect so much data from human operators and so on. This is, if you saw that there are a lot of authors, this is because this was or seems like a quite big project. But, yeah, it's definitely worth it. It's cool to have something in the real world. There are definitely a bunch of criticisms I have right here, which I also brought up to the authors, and I thought they responded quite admirably and quite well. The one criticism I already raised was that if, you know, it obviously depends on how you spell. So what you have is this bank of skills on the right-hand side here. Now, in order for the language model to score them, they need to actually be formulated as a piece of language. And now it all of a sudden depends on how you formulate that. For example, we know that longer queries always have kind of lower likelihood because they have more tokens. Also how you phrase things is differently and so on. So it is quite tricky. And I believe if you go into more actions, maybe actions, maybe the robot has two actions that are very close together in terms of semantics or in terms of wording, the model might get confused more easily. Second of all, currently, there is no consideration as to whether an action succeeds or not. So you simply assume that once you execute a low-level policy, that the robot is going to succeed at executing that low-level policy. That is why, if it does not succeed, and a lot of these things are still pretty hard, then there is very little recovery. The value functions might still give you, like, let us say you find an apple, you try to pick up the apple, but you do not manage to do it. The pick up an apple instruction will be pick up an apple, will be in your prompt. So now the value function will probably say, well, I could pick up the apple again because it again sees an apple because you failed to pick it up. But the likelihood that the language model is going to say pick up an apple again after it just did is quite lower. Now, in coincidence, as we know language models, if you go on here repeating the sentence pick up an apple, at some point it actually becomes pretty likely, given the language model. But hopefully, we will not get there. So there are quite a number of weaknesses yet in this setup. The other weakness is just the limitations of hardware. These robots, they are, this video was 10x speed. So this was 10 times speed. And still it's quite slow. It, as you can see, it can't do many things like it cannot wipe itself with the sponge and so on. It needs to navigate around slowly. Yeah, but still these are, I think, limitations that can be overcome because it like carefully grabs. And yeah, in any case, there are also a lot of good things right here. And I want to highlight that because what I really like about this is that these two things are disjoint. So the language model side on the left hand side and these value functions, this policy bank, these atomic actions, they are disjoint. So they are disjoint. So they are not actions. They are disjoint. The language model can, is not trained. It is a frozen language model. It can be trained completely in isolation to the system. All you have to do is get it to score the likelihoods of some actions. Likewise, the bank on the right here, it is completely, in fact, not the bank itself, but each individual skill, each individual entry is trained completely isolated from all the others. All you need to add a new skill right here is a policy that can execute that skill at any given moment and a value function that estimates, given some state input, that estimates how likely the policy is to succeed if this action, if this policy were to be executed at this particular moment. That's all you need. You can add this to your bank of actions and you have to, you don't have to retrain anything in this system. It is directly useful. So you could think of shipping out these robots essentially and then upgrading the language model so they are better at planning stuff. Or you could just ship new skills, right? It's like, well, our coders have developed some new skill for the robot, right? You just amend, you mend it. You just put it in. There's no, you don't need to update the full system. This is not an end-to-end system. And usually in deep learning, we're quite end-to-end happy. But in this case, I think this is a really good case where modularity is really the key. I think this goes so much beyond just robots and grounding in the real world. But to have a model like on the left that has knowledge about, you know, semantic knowledge, high level knowledge, and so on, sequential knowledge, essentially, to provide that with a set of modular pieces of external things that it can use. I think that idea is powerful way beyond just the robotics use case. But obviously, the robotics use case is quite a cool one. So I don't want to discourage that. Yeah, we in the interview, we go into all of this, we go into the experimental results as well. The experimental results, they're not perfect. However, they are quite impressive in that the robots they are able to plan across many, many time steps. They're able to chain these actions. You can see on the right here, that's maybe two pixels. But these are like 17 of these atomic actions that are done in sequence. And, you know, that's quite impressive. These episodes are very, very long. And if you think you can get to that in the real world with sort of a reinforcement learning approach, then good luck. Yeah, so the success rates are among the 70% ish of plan success rate, 61% execution success rate, which the plan success rate, I believe is if the plan itself makes sense, and the execution success rate is if also the policies all execute correctly. And you can see this is very different for the different test sets. But all in all, it's very impressive. Here are a bunch of more examples of these low level atomic skills being practiced and the value functions being evaluated and the language, the language model likelihoods in blue as well. So I don't want to make this artificially too long. As I said, interviews coming up. I hope you like explanations like these, even if they are a bit shorter. And I'll see you around. Check out the paper, subscribe, stay hydrated. Bye bye.
 Hi there, check out this video. So there's a Coke can and there's a spill, a Coke spill.
 So the instructor here says, I spilled my Coke on the table. How would you throw it away and
 bring me something to help clean? So the robot here forms a plan as it goes about it. First,
 it says I would find a Coke can. Then second, I would pick up the Coke can. You can see it has
 done it. Third, I would go to the trash can. Fourth, I would put down the Coke can. Note that
 he puts down the Coke can next to the trash can, not in the trash can, because the robot is
 environmentally friendly and wants to preserve the can for the recycling bin for cans. And,
 you know, it doesn't belong in the trash. Good little robot. So next it says I will find the
 sponge. I will pick up the sponge and then will it clean the Coke? No, it will not clean up the
 spill. It will actually give the sponge to the human to clean up the spill, because that's how
 the future is going to be. The robots, they're not going to take our, you know, people always
 think the robots will take our dirty jobs. They'll take all the like these tasks like cleaning and
 doing things. No, no, no, no, no, no. They'll abuse us, the humans to do that. They'll just throw down
 our stuff. They'll throw down the sponge and be like, here human, clean up your own mess.
 Well, if that's a future that you look forward to, too, then join me in today's paper. We're going
 to look at do as I can, not as I say, grounding language in robotic affordances by researchers
 at robotics at Google and everyday robots. So as you saw in this video, what happened here is that
 from a simple instruction that the instructor gave this essentially this I spilled a Coke can,
 you know, please help me find something to clean and throw it away. The robot formed a plan,
 the plan you can see at the very end here. You can see it developing in the bottom. At the very end,
 you can see the full plan. I'm not sure if I can make this bottom thing go away. In essence,
 it makes a plan like it always plans the next step or at least it determines what the next step
 should be. And then it actually also does it. So this is a good example of grounded, a grounded
 language model, or also an example of embodied intelligence. This work connects large language
 models and the knowledge that are that is inherent to large language models with the skills of robots
 that act in the real world, which really cool. And usually these two things are quite disjoint,
 but this could be really powerful. So we're going to look at this paper. I also have already recorded
 an interview with the authors this for time reasons. We did it the other way around this time. So I
 don't want to take away too much on the paper review right here. I'll tell you what the method
 is about how it works. And I'll leave the rest to the authors who are extremely competent and I
 learned I learned like I learned a lot in the interview. I hope you will too. In any case,
 the interview will be out tomorrow. If you're watching this the day it comes out, which obviously
 you do. How do you find new papers? Frankly, machine learning has become unbearable. There
 are thousands of new papers each month. And to keep the overview, we need good tools. Today's
 sponsor is Zeta Alpha, which is a search and recommendation engines for papers. This is really
 powerful. For example, here I've searched for today's paper, say can you can immediately see
 that not only do I get the paper itself, but I also get an aggregation of all the social media
 mentions of this paper. That doesn't stop there with one click, I can find related papers. These
 are not only papers that are cited, but these are semantically similar papers. This is powered by
 neural search, which is really cool. I can further now add this paper to my tags. And what that will
 do is it will build categories of papers and then serve me recommendations that semantically
 fit those categories. This is really powerful. Essentially, this is a newsfeed of papers that
 is personalized to you specifically. Just recently Zeta Alpha has released their own PDF reader.
 This is really strong right out of the gate. Not only does it let you read the paper, you know,
 but also it shows the important information about a paper and it lets you take notes. Now what I
 find by far the coolest thing is that you can actually use one of these notes to search for
 other papers. So whenever you find something within a paper that seems interesting, you can use that
 particular piece of text and go search for other papers that might deal with the same topic. Sign
 up now to Zeta Alpha, there is a free tier and the pro tier is actually free for students for
 academics. But in case you are not one of these, the promo code Yannick will get you 20% off the
 pro subscription. The authors here state that if you try to ask a language model to clean a spill,
 as they just did in the video. So if you ask a language model to clean a spill, it might result
 in a reasonable narrative as we've all come to know the large language models like GPT-3 or so
 they give very convincing outputs. So when you ask them, how would you clean up a spill,
 they'll give you a reasonable plan to clean up a skill. But the authors say may not be applicable
 to a particular agent such as a robot that needs to perform this task in a particular environment.
 They have a bunch of examples right here. So I spilled my drink, how can you help is up here.
 GPT-3 would say something like you could try using a vacuum cleaner. Well, GPT-3 has no idea of
 A, whether there is a vacuum cleaner in this environment, or B, whether the robot or whatever
 agent is capable of executing that action. So is capable of handling a vacuum cleaner, because
 it's not the easiest thing to use. You have to go get it, plug it in and so on, there's moving parts.
 Similarly, models like Lambda and Flan, of course, they're made for different things,
 but still, they will pay no attention to what is actually possible in the environment. Now you can
 get around this a little bit by prompting, by prompt engineering, telling the model what's
 possible in the current world, but it will only get you so far. So we need something else,
 we need something better than this. And that's where this system comes in.
 So they say what they want to do, they want to provide a real world grounding by means of
 pre-trained skills. And this leads to a situation where you only consider actions that are both
 feasible and contextually appropriate. So these two things need to be brought together. The language
 model supplies the high level semantic knowledge about the task, and the language model provides
 and the robot itself or the policy in the robot provides the feasibility of the tasks to be
 executed. So the two things are brought together, contextually appropriate from the language model
 side and feasibility from the robot side. So how are they going to do this? They're going to combine,
 as I said, large language models with policy or value functions, let's say value functions,
 and then they execute a policy. There's a bit more explanation right here, but I think I've said
 many things already. We'll get to the meat right here. They say, let's say we have a robot. The
 robot might be, or in this case is, equipped with a repertoire of learned skills for basic atomic
 behaviors. These skills are capable of low level perception and control. So one of these atomic
 behaviors is, for example, if you remember from the video, pick up something, pick up the Coke can.
 That's an atomic behavior. You can teach the robot to pick up a Coke can in isolation. It can do it
 many times. You can train some imitation learning or reinforcement learning, or you can even hard
 code that particular policy. It doesn't matter. What matters is that you can train it in isolation.
 It is an atomic action, and these atomic actions can then be chained together to form a sequence of
 actions and execute a plan. So the atomic actions are going to be supplied by the robot, and then
 the sequencing of the atomic actions is going to be determined by the language model. They say,
 if we can simply make the large language model aware of the available and feasible repertoire of
 skills, this can provide it with an awareness of both the agent's capabilities and the current
 state of the environment. So if they have a large language model, many people use large language
 models to sample, which means that they input, they would input something like, you know, I,
 I is capitalized, I spilled a drink, dot dot dot, and then they would let the language model
 generate stuff right here, and then they would try to interpret this stuff. We've seen this in
 other paper, and there are situations where it can work, especially if you put like some reasonable
 prompt in front of it. But the approaches have been largely to just let the model generate some
 stuff and then try to map that stuff, whatever comes here, into the action space of the robot.
 But that is not always possible. Instead, what this paper does is it says, well, we can also use the
 language model not to generate, but simply to compute the likelihood of certain inputs. So I
 spilled a drink, and then let's say I just have five actions at my disposal. All the robot can do
 is these five actions. So I would, or let's, let's say it says, I spilled a drink, I will, and then,
 clean up, I will go away, I will eat pizza, I will, and so on, right? So there are these different
 actions that the robot has available to do, and these correspond obviously directly to these atomic
 actions. So cleaning up something would be an atomic action that you could train in isolation.
 Going away would be an atomic action. You can hard code or you can path find your way out the door.
 Eat pizza. Maybe these are even too high level that the way that I describe right now, but just
 imagine these are low level actions. And all we have to do with the language model is we simply
 have to compute the likelihood of each. So what's the likelihood of the sentence, I spilled a drink,
 I will clean up, right? And then I compare that to the likelihood of the sentence, I spilled a drink,
 I will go away. And then I compare that to the likelihood of the sentence, I spilled a drink,
 I will eat pizza. So for every continuation here in my repertoire, I will get a likelihood number.
 And that represents how contextually appropriate is that particular skill in this case. So
 how much does the language model think this skill would be useful right here? Now there's obviously
 an issue in how you formulate these things right here. Depending on how you formulate them, they
 might become more or less likely. However, I think the authors here work around this simply by the
 fact that these skills that they have, they are so separated from each other, there is not really
 too much of an issue with that. But that's kind of what my concern was when I read this. But in
 essence, it's a good idea, I think. So you simply for every single, wow, this all became orange,
 for every single continuation, you get a number, which is the likelihood of that thing.
 That's what they say right here. No, instead of using the large language model to integrate
 an instruction, we can use it to score the likelihood that an individual skill makes
 progress towards completing the high level instruction. Furthermore, and that's where
 the second part comes in. If each skill has an accompanying affordance function that quantifies
 how likely it is to succeed from the current state, such as a learned value function, its value can
 be used to weigh the skill's likelihood. It's best if we go down here and say that the skill
 is the best. It's best if we go down here to the diagrams of how this works so you can see how this
 fits together. This part here is the part we just described. Let's say I'm in a situation,
 this is the prompt that I put in. How would you put an apple on the table? You prompt, well,
 you prompt the language model with this thing right here, which has a prompt engineering part.
 You can see there are a bunch of examples of instruction and then a sequence of steps.
 Again, instruction, a sequence of steps. Here it comes again, instruction, and then here you'd get
 a sequence of steps. However, instead of generating, you'd simply score the likelihood of each of the
 skills that you have available. Find an apple, find a Coke, find a sponge, pick up an apple,
 pick up a Coke, yada, yada, yada until go to the counter. Each one of these skills gets a likelihood
 number assigned to it. That's part one. Part two is where you train the robot for these basic skills,
 these atomic skills. Here you can see one of these training stations where you can simply teach the
 robot to do things such as picking up an apple or picking up a Red Bull can. Now, what you have to
 do is not only teach the robot a skill, but also train a value function for it. If you do something
 like A2C reinforcement learning, you get the value function directly out of that algorithm.
 If not, you have to somehow come up with a value function that makes sense. In any case,
 what you want to do is train a policy and a value function. The value function is important
 because it tells you from a given input, by the way, the low level policy has the picture here
 and input. Well, obviously the language model doesn't. Now, I believe with Flamingo by DeepMind,
 that just came out today, that might actually change. But the low level policy has the image
 available. So the value function, given this picture right here, can tell you pretty quickly.
 My skill that's called pick up the Red Bull can, I can execute that policy and I can probably
 make it happen. That's why the value is relatively large here. Also for the pick up the apple action,
 the value function tells you, you know, given this picture right here, I can probably make that
 happen. However, when it's pick up the water bottle, pick up the bag of chips and so on,
 there is no water bottle. So the value function very accurately says, no, I cannot make that happen
 if I execute that policy. So the value function gives you inherently a score of given the current
 observation, how likely am I to succeed at a particular skill, which is exactly what we want,
 because that's the second part of our puzzle. So on the right here, you see another example where
 none of these pick up skills, picking up, sorry, not pick up, picking up skills have any value
 because there are no objects. But in this case, maybe other actions would score very highly in
 the value function. For example, go and find a sponge. Like I can always go and find something,
 right? And if I know there's a sponge somewhere, I'll probably succeed. So in any case, this value
 function now we can combine, you can see we got a number for each action from the language model,
 how likely is that action to progress towards the goal. We got a number for each action from
 the value function, which is how likely is this action to succeed given the current observation.
 And all we do now is essentially multiply the two things together. If they are log likelihoods, we
 obviously want to add them. But in any case, we combine the two numbers, and then we choose
 the skill that is the best trade off between what makes progress towards a goal and what is
 feasible currently. Here is an example. The input is how would you put an apple on the table like
 an apple? So we query the language model with this prompt here and the prompt engineering we've seen
 before. This is not displayed here, but it is the case. And the top actions that the language model
 gives are pick up an apple, you see that's the highest action that we have, place the apple,
 and only at third instance, find an apple. However, the language model has no clue about
 the current state, right? And that's where the value function come in. So this is the current
 observation. We ask the value function which skills are doable in the current environment,
 in the current observation. So the value function say, well, finding an apple, finding a coke,
 finding a sponge, these are pretty high. I could do these. I could also go to the table. I could
 also go to the counter, right? These are fairly doable. However, I cannot place an apple or place
 a coke because I don't have a coke in my gripper. I can also not pick up an apple or pick up a coke
 because I don't see them anywhere in the picture right here. So even though pick up the apple was
 scored highest by the language model, it is now severely down ranked because the value function
 for this policy doesn't, isn't very confident that it will succeed if you execute that right now.
 And therefore, the action that is chosen is the best trade off, which is find an apple.
 Then you can see or not see, but this is represented here that after this is done,
 the policy is executed. So the find an apple policy is executed. The find an apple action is
 added to the prompt and then the whole process repeats. But instead of asking for the first step,
 this whole thing is now the prompt, including the instruction. And we simply ask the language model
 for the second step and the input to the value function is now the current updated picture.
 So here you see it succeeded in finding an apple and now hopefully the second step,
 if we go through the same process again, is going to be the pick up an apple action. Because, well,
 that might already be high by the language model, but also the value function, given that there's
 an apple in the picture should now say, yes, I can probably succeed at that. So that's the whole
 issue or the whole process here. This is repeated until the end. This is the say can method.
 What is really impressive is just the amount of effort and work that went into designing these
 systems, training these systems, evaluating these systems. They have different areas here on the
 left. This is like a kitchen. On the right is a different environment. They have these training
 stations. They collect so much data from human operators and so on. This is, if you saw that
 there are a lot of authors, this is because this was or seems like a quite big project. But, yeah,
 it's definitely worth it. It's cool to have something in the real world. There are definitely a
 bunch of criticisms I have right here, which I also brought up to the authors, and I thought they
 responded quite admirably and quite well. The one criticism I already raised was that if, you know,
 it obviously depends on how you spell. So what you have is this bank of skills on the right-hand
 side here. Now, in order for the language model to score them, they need to actually be formulated
 as a piece of language. And now it all of a sudden depends on how you formulate that. For example,
 we know that longer queries always have kind of lower likelihood because they have more tokens.
 Also how you phrase things is differently and so on. So it is quite tricky. And I believe if you
 go into more actions, maybe actions, maybe the robot has two actions that are very close together
 in terms of semantics or in terms of wording, the model might get confused more easily. Second of
 all, currently, there is no consideration as to whether an action succeeds or not. So you simply
 assume that once you execute a low-level policy, that the robot is going to succeed at executing
 that low-level policy. That is why, if it does not succeed, and a lot of these things are still
 pretty hard, then there is very little recovery. The value functions might still give you, like,
 let us say you find an apple, you try to pick up the apple, but you do not manage to do it.
 The pick up an apple instruction will be pick up an apple, will be in your prompt. So
 now the value function will probably say, well, I could pick up the apple again because it again
 sees an apple because you failed to pick it up. But the likelihood that the language model is
 going to say pick up an apple again after it just did is quite lower. Now, in coincidence,
 as we know language models, if you go on here repeating the sentence pick up an apple,
 at some point it actually becomes pretty likely, given the language model. But hopefully,
 we will not get there. So there are quite a number of weaknesses yet in this setup. The other
 weakness is just the limitations of hardware. These robots, they are, this video was 10x speed.
 So this was 10 times speed. And still it's quite slow. It, as you can see, it can't do many things
 like it cannot wipe itself with the sponge and so on. It needs to navigate around slowly.
 Yeah, but still these are, I think, limitations that can be overcome because
 it like carefully grabs. And yeah, in any case, there are also a lot of good things right here.
 And I want to highlight that because what I really like about this is that these two things
 are disjoint. So the language model side on the left hand side and these value functions,
 this policy bank, these atomic actions, they are disjoint. So they are disjoint. So they are
 not actions. They are disjoint. The language model can, is not trained. It is a frozen language
 model. It can be trained completely in isolation to the system. All you have to do is get it to
 score the likelihoods of some actions. Likewise, the bank on the right here, it is completely,
 in fact, not the bank itself, but each individual skill, each individual entry is trained
 completely isolated from all the others. All you need to add a new skill right here is a policy
 that can execute that skill at any given moment and a value function that estimates, given some
 state input, that estimates how likely the policy is to succeed if this action, if this policy were
 to be executed at this particular moment. That's all you need. You can add this to your bank of
 actions and you have to, you don't have to retrain anything in this system. It is directly useful.
 So you could think of shipping out these robots essentially and then upgrading the language model
 so they are better at planning stuff. Or you could just ship new skills, right? It's like, well,
 our coders have developed some new skill for the robot, right? You just amend, you mend it.
 You just put it in. There's no, you don't need to update the full system. This is not an end-to-end
 system. And usually in deep learning, we're quite end-to-end happy. But in this case, I think this
 is a really good case where modularity is really the key. I think this goes so much beyond just
 robots and grounding in the real world. But to have a model like on the left that has knowledge
 about, you know, semantic knowledge, high level knowledge, and so on, sequential knowledge,
 essentially, to provide that with a set of modular pieces of external things that it can use.
 I think that idea is powerful way beyond just the robotics use case. But obviously, the robotics use
 case is quite a cool one. So I don't want to discourage that. Yeah, we in the interview,
 we go into all of this, we go into the experimental results as well. The experimental results,
 they're not perfect. However, they are quite impressive in that the robots they are able
 to plan across many, many time steps. They're able to chain these actions. You can see on the right
 here, that's maybe two pixels. But these are like 17 of these atomic actions that are done in sequence.
 And, you know, that's quite impressive. These episodes are very, very long. And if you think
 you can get to that in the real world with sort of a reinforcement learning approach, then good luck. Yeah,
 so the success rates are among the 70% ish of plan success rate, 61% execution success rate, which
 the plan success rate, I believe is if the plan itself makes sense, and the execution success rate
 is if also the policies all execute correctly. And you can see this is very different for the
 different test sets. But all in all, it's very impressive. Here are a bunch of more examples of
 these low level atomic skills being practiced and the value functions being evaluated and the language,
 the language model likelihoods in blue as well. So I don't want to make this artificially too long.
 As I said, interviews coming up. I hope you like explanations like these, even if they are a bit
 shorter. And I'll see you around. Check out the paper, subscribe, stay hydrated. Bye bye.