Author Interview: SayCan - Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
#saycan #robots #ai

This is an interview with the authors Brian Ichter, Karol Hausman, and Fei Xia.
Original Paper Review Video: https://youtu.be/Ru23eWAQ6_E
Large Language Models are excellent at generating plausible plans in response to real-world problems, but without interacting with the environment, they have no abilities to estimate which of these plans are feasible or appropriate. SayCan combines the semantic capabilities of language models with a bank of low-level skills, which are available to the agent as individual policies to execute. SayCan automatically finds the best policy to execute by considering a trade-off between the policy's ability to progress towards the goal, given by the language model, and the policy's probability of executing successfully, given by the respective value function. The result is a system that can generate and execute long-horizon action sequences in the real world to fulfil complex tasks.

OUTLINE:
0:00 - Introduction & Setup
3:40 - Acquiring atomic low-level skills
7:45 - How does the language model come in?
11:45 - Why are you scoring instead of generating?
15:20 - How do you deal with ambiguity in language?
20:00 - The whole system is modular
22:15 - Going over the full algorithm
23:20 - What if an action fails?
24:30 - Debunking a marketing video :)
27:25 - Experimental Results
32:50 - The insane scale of data collection
40:15 - How do you go about large-scale projects?
43:20 - Where did things go wrong?
45:15 - Where do we go from here?
52:00 - What is the largest unsolved problem in this?
53:35 - Thoughts on the Tesla Bot
55:00 - Final thoughts

Paper: https://arxiv.org/abs/2204.01691
Website: https://say-can.github.io/

Abstract:
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at this https URL

Authors: Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 So today we're here with three of the authors of this paper with I have to say a lot of authors It seems like a giant work just from what I could gather From the from the paper itself and the data collection and the evaluation and so on So this was a huge thing, but the results are pretty cool. So here with me today are Faye Xia Brian Ictor and Karol Hausmann who are three of the authors of this work. Welcome to the channel everyone Thanks. Thank you for having us. It's great to have you here The I like I love the title Because it's a bit of a mantra on the do as I do as I say not as I do which is kind of the other Way around right here and this idea of Connecting robots and language. It seems pretty natural I have to say I've I've seen a lot number of paper attempt to do something like this Like can we maybe translate what the language model says into the space of what the robot understands and things like this? But this here it seems like a bit of a new approach Why why did you try? Why did you attempt to do this? Like why does this seem promising and why did no one else do this thing yet? Yeah, I think to start like the To I guess like prior work on like using a language model to kind of translate it down I think we first started out with sort of like playing around with that and and realized I guess how much information is Embued in these language models and how well they're able to reason over sequences and remember what they've done But when we really like started thinking about applying it to the world It was sort of like odd that there's no way to basically Make sure that whatever it's saying actually makes sense for the environment that was in And so I think like after playing around that for a while we were sort of like stuck there like okay We have these like interesting plans But they don't actually make sense for everything that the the robot can do and so we started kind of like shifting towards towards that problem Yeah, I think also separately we've been trying to get robots to do many things and learn multiple skills and This is a very difficult problem and we were debating kind of the the best way to do this whether we should predefined the skills up front or whether we should just demonstrate kind of anything that comes to mind and label it afterwards and Just connecting these two dots the language models with the skills that we already have on the robots Seems like a nice way of factorizing this problem Did you always could you so you have this robot in this environment and is if I understood correctly? Maybe here is a good demonstration of that. So you have the robot in these two environments and These are the environments that exist to understand this correctly. So it's only these two environments. There's no generalization across environments Yeah, so we've been collecting data in beautiful environments. These are the two environments that we use for evaluation We also have a separate environment that is right next to the environment that it's a Mark this be here where robots are practicing But it looks fairly similar to to at least the stations that the robots practice on are fairly similar to the stations that you see here The backgrounds are changing the the objects are changing that we practice with and things like that. We also use Simulation as an additional environment that we then try to make look similar to the real world But we don't really focus in this paper on generalization to Completely new environment. We rather try to focus on kind of having a robot do as many things In a single environment when we talk about robot practicing things, I guess that's where your methods starts with robots Practicing things and by things I guess we mean a bunch of very Low-level let's call them unit unitary Skills like here. For example find a coke can pick up the coke can bring it to you something like this So these these could be things that Conceivably we could learn with something like behavior cloning or something like this. How did you? Decide on what actions are possible for these robots to do on their own like as a unit Some of it is based on what the robots capable of some of it's like what? Gives us a like a easy reward function And some of it was sort of motivated by what? Composes well into long horizon behaviors that you really want to do in the world like if we have a robot operating in a kitchen What would I ask it to do what what's required of it to do that? And how would I break down the task? I think was like part of the motivation like really how this robot is gonna operate in the world Yeah, and also it's interesting to see how this picture came out So initially we kind of have to come up with these and we kind of have to think up front What would that person ask a robot to do? But now that we have something running we can actually ask people and see how they interact with the robot and Decide on which skills we should be learning next based on that Sorry, I want to add that at the beginning we choose pick and place because these are Two fundamental skills that can unlock a large number of instructions that we are able to solve But it is also very easy to add new skills into the picture like we only need to Have a have a language language description for the skill and we also need a policy and value function So these are all the three three things you need to import a new skill into the second framework What I like here is that you said you need a policy and a value function that policy doesn't even have to be like Neural network based policy conceivably one skill can be a very classic control problem I believe when you pick up things You is is that correct that you classically control where the actuator should go and when you move the robot you kind of plan in space So not everything is like reinforcement learned or behavior cloned Yeah, so different skills are learned differently in this case pick was learned through behavior cloning on real data But yeah, for instance for instance moving around this is not Trained with reinforcement learning or behavior cloning. So yeah, you can compose you can have different algorithms Train different skills and these skills just to to round out the picture right here the input is Whatever the camera sees plus, you know kind of all the states of the actuators So that conceivably there's an apple in front of you and the task is pick up an apple And that that would be kind of the state from from where you operate. That's right Yeah, we are in the state of the actuators. So that's the input From where you operate. That's right. Yeah, we are going the value function. The value function describes kind of how likely you are to fulfill that task That's right. Yeah, so the input to the policy is the image that the robot sees that you get at every after every action We actuate the arm by doing end effector position control Yeah, these are the inputs and outputs And also there's a terminate action, right? Sorry that so the robot can say it itself when it's done Yes, so one of the actions that the robot can command is terminate which basically means i'm done now we can move on to the next one And okay, so now I guess that this is one part of the puzzle You have robots you have all these policies for all the little things that the robots could do these little things Were developed by you by the community conceivably you could also use the large language models itself To suggest new things to train right on the basic level. You could ask gpd3 What would you do right here and then the little steps you could conceivably Make into like train into little actions, but you have this library of things And now the question is how do you how do you compose them and that's where the large language models comes in Do you want to comment maybe a little bit on like how does that look in a base in a basic way? How do we combine the knowledge of language models with these skills that the robots can do? Yeah, I guess uh at a high level so the language model already has So much knowledge about the world And how to do things in order and memory and things like that And the way to get it to like really speak in the way that is amenable to the robot first We show it a few like prompt examples. So we show it solving, you know, like about 10 problems And breaking it down from the query into the sequence of steps that it would take to solve that It turns out you can actually not use that and you still actually get like Some level of performance maybe like half the performance. So the language model just like comes out of the box With pretty good understanding of these tasks We then show with these examples this kind of brings it into the right frame of thought But if you do that and you ask for something new it doesn't like fully constrain it in a way that the robot will be able to understand it So our tasks along with the image and the states that we uh mentioned before also takes in like a task id So it says like um like pick up the apple So really what we need it to do is like output pick up the apple It can't say like pick up the fruit because the low level policies are not generalizing to that So to make sure that every time we actually output things you can do we instead of like taking the generative output of the language model We use what's called a scoring model So when a language model outputs, uh some text it also comes with a probability that it would output that text And so instead we can just like force it to only respond in these ways and say basically how likely it is to respond in that way So in this case we get like a score of if I were going to pick up the apple or put the apple somewhere These are the things i'd likely to respond These are the things there's no way I would respond And this gives us some like probability that the language model thinks this is really useful to the downstream task On the other side we have these value functions and policies that we've talked about There are actually the value functions outputting how likely it is to achieve a task I think actually there's another uh slide one like one more down Um, but this is basically or yeah, this yeah is saying basically these are possible from this state And so on one hand we have a language model saying this seems really useful for the task And on the other hand we have the value function saying this seems possible And together they give some probability that this is what you want to do to basically accomplish the high level instruction I have a number of okay. Let's just start at at the beginning at the at the language model level I see the high level picture you you ask both the language model and the value functions What's what you know what they think should happen next and uh the combination of the two is what then you really do Which makes a lot of sense when you do ask these language models what to do um right here You said you use the you use the essentially you you ask the language model for the likelihood of of an output instead of letting it generate the output Was this your first try because one could also imagine uh you know saying something like of the following options Which one would you pick right and then you list all the options which would conceivably be more general because you could option you could add options over time And stuff like I guess you could do that here as well but um was this your first attempt or did you did you have some prompt engineering attempts before that Yeah I think at first we tried just like prompt engineering to see like basically what the generative model would output I think like our our initial thinking was we just want the generative model to basically plan as much as we can But that runs into two problems one is that it doesn't constrain the output fully so if I give it all these examples and then I said How would you put a fruit on the table instead of an apple on the table the generative model will actually respond with like number one find a fruit number two pick up the fruit And then you need to figure out how to like take that and project it into the final like thing that the robot can actually handle You can project this in some sort of like embedding space and that works sort of well but you actually lose some context on the overall query so I guess the way that we do it is a little bit more like well founded so to speak But the other really nice benefit of this is it gives us scores for everything which is really interpretable it lets us like see the trade off between these two options So in your example you said you know what if I just said here are your options pick one and the language model would probably pick one but now you only know that this is its favorite option You don't know the probability that it would have done maybe maybe it's actually okay with the next three options so this gives us this like interpretable score that we can then combine with the value functions Yeah there are some caveats to this I feel in that for example we know that by definition longer outputs are less likely right so I guess it's not too much of a problem for you because most of yours are like three or four words but have you noticed any of kind of these effects of just how these probabilities are constructed as kind of multiplications of softmax outputs like that's got to bring its own bias into the into the picture Have you observed any of that have you had problems with any of that or was it was it generally okay Yeah it's it's definitely a little bit of an issue I mean I think in general it's also very particular to the to these like if you were to misspell a word in there or like have an A versus an M it's not particularly robust to those in the options it is in the query like to what the user might say but not when you're scoring these options because if one word is off then this like multiplication of each word just kind of tanks the entire score so we did have to be somewhat careful with what we have one way to kind of like get around this a little bit is if you have some like end of statement token and if it if it adds extra words on the end then it's saying if if there's like more to come that end of token will basically kind of normalize the rest of it like you can't end a statement or a word and a statement early the yeah I think in the other thing that we did try to do is like potentially normalize them so knowing that this query is longer perhaps we need to upweight it or have some normalization on the language output but we found that it wasn't particularly consistent and there wasn't like just a constant effect across one or the other and it depends on the way you like referred to the query and so at the end of the day we just took the outputs as they were so it was an issue but it wasn't like a huge one I imagine that there's another factor here for example if you if you say you said before pick up a fruit or please bring me a fruit or something of this you're essentially relying on the ability of the large language model to sort of recognize that apple is a fruit and and and kind of interpret that in the same way and and so on so the kind of close as the language model estimates how close the things are did you find this generally in agreement of in how how humans find how close the things are and maybe yeah I'm just I'm just wondering about this notion of how how close things in language are together also what happens if you for example have an apple and an orange in your scene these two things would be quite close together so even if you said you know please pick up an apple the pickup an orange thing would conceivably score quite high in the language model which might perturb your things so I can kind of I can sort of make out that you have an ideal environment right here in that you probably picked objects that are distinct from each other locations that are fairly distinct from each other right such that there's a nice semantic gap between the things what like do you think this is well applicable to a real world setting or what kind of hurdles could there be with connecting language models and the set of actions in this way so I guess the first question was about do these families kind of align with what you would expect and that was actually that was one of the first things that I was looking at was like how well do these scores sort of match up to what you think it's going to be so yeah it turns out that like apples are apples and orange and banana are all going to score quite highly when you're asking for a fruit if you ask for a snack all the food options are going to score highly similarly drink soda any category like that it performs about yes you would expect as a human which is good but then yeah it comes to this problem of what if there's an apple and orange or what if there's an orange but not an apple and that's where these value functions come in this is actually like one of the key reasons why we have to do this in volume and grounding. Because if you just asked a regular language model that doesn't know what's there then how does it make that decision maybe it uses the wrong one then your plan isn't really correct and our policies may not actually work. But the value function tells you if there is an apple in the scene and no orange then you're going to see a high value function on the apple because the pick apple command could work versus the orange command is going to be quite low and so that actually lets you sort of like disambiguate this so. In the in figure B if it had a pick up the Red Bull if you said bring me a drink and there's a Red Bull but no water it's going to pick up the Red Bull because that's actually what's there. And if not then then the instruction itself is ambiguous right if you say pick up a drink and there's two drinks and both are affordable according to the value function yeah. Yeah then we think like either is completely fine I think it's also interesting because then the robot is making the trade off itself dependent maybe on the value function so for instance if you ask for a fruit and there's an orange and an apple but it's much better at picking up apples. Maybe it will pick up the apple because the value function will just tip the scale. So it will make some errors in that sense but since this is interpretable and you can kind of look back and see why I decided for that it can also inform us as to what skill we should train a little bit more or which value functions are a little under fitted and things like that so it will make some sort of mistake. But maybe that's that's okay maybe that's acceptable. I think one like really nice feature of that too is it's not necessarily always like it's better at picking up oranges or apples but you can see like these objects are in different locations one may be better for the policy than the other. So we're going to end up doing the one that's a little more robust and a little more likely to succeed as long as it still fulfills the high level query. Yeah I like the fact that you have success probability as sort of the ultimate score because I was I also thought one failure mode here is that some tasks are inherently harder than others right and so naturally your value function would be lower and therefore you can misinterpret just by the fact like well like this this this is me the procrastinator like this thing seems really hard and we'll do this other thing that I'm not sure how to do. But it's really easy so it's almost it's almost too human how the robot would act in this way. So yeah you have these what I like here as well is that you have to bank of value functions on one hand the language model on the other hand and the language model on the other hand is the one that's the most difficult to do. So yeah you have these what I like here as well is that you have to bank of value functions on one hand the language model on the other hand and they are never if I understand correctly trained together right there never in fact the language model is probably just frozen. So they're never trained together which means that you could conceivably just add a skill to the robot train its value function for it and just plug it in and and go. Yeah we can scale this fairly easily so we can continue adding skills we can also change the underlying algorithm how we train the skills. Or how we train the particular skill that we want to add if we if suddenly there is a really good script that allows to I don't know swipe the floor or something like that. We can we can also add that as long as we have a value function for it and also at the same time if the language model becomes better we can also swap out the language model and get improvements through that. I want to add that so our current value function is one way that we instantiate affordance but there are many other ways that we can instantiate affordance like for example we can directly do prediction. We can also use classical motion planning like to calculate for example length of the trajectory is also or the probability of success if you do like sampling based motion planning. So there are many ways that we can come to the affordance and the method is really flexible to plug in any type of affordance. I guess a big topic in maybe maybe it's more the space of blockchains and things like this is agents that do an action for you but also optimize for example for cost or for resources or something like this. This could directly flow into that where you can tell the robot you know do whatever fulfills my task but also costs very little and this could if this directly flows into affordance there might be a normalization issue but if this directly flows in you'd have you could tune the knobs on these on these functions fairly easily. So this is the full algorithm I guess we haven't talked yet about how you extend this to multiple steps but it is as far as I can tell fairly easy in that you do this in sort of a stepwise fashion so first you ask your language model your value functions at the current state and the current camera position where what should be done. Then you try to whatever should be done according to both scores combined you execute that and after you execute it you ask the same thing again but now the prompt changes and it's simply that you so here the prompt is essentially I would first and then first action is decided and once you go on the prompt now says I would first. The prompt now says I would first and then whatever was decided on and then second and then it's simply the same thing with the next action did I get this approximately correct. Do you pay any attention to whether or not the task was fulfilled successfully. So right now we don't we assume it will successfully execute I think some things could happen like if it fails at a navigation task say it was trying to navigate to an apple and the and it doesn't get there then the value functions at that next state are going to be quite low so you're not going to be able to basically pick something up or whatever so maybe then you end up selecting navigate to the apple again or navigate to a table instead but we don't have any like explicit success. Detection I think this is like one area that we're like pretty interested in going basically like finishing the job closing the loop entirely on when you try to do something did you succeed telling the language model and then having a language model adapt accordingly. I want to show one video from from your website which in this case if I got this right it confused me I guess a little bit because this thing right here if you see it kind of looks around sees all the things right like looks and sees and then it kind of scores the actions. And like this so pick apple I can't do that pick sponge okay. Bring you a sponge no not go to trash can place the sponge place the sponge is good and that's the place the sponge kind of up ways to bring you a sponge or like what's going on right here because in my in my estimation the robot shouldn't even look around initially the robot should just have its camera position fixed and then it in first instance. It should probably figure out like find a sponge or something like this and then it would move and then it would see consider these next actions like what is what is this video supposed to to show. Yeah I think you're understanding is completely correct so this is more like a conceptual video where we wanted to kind of across that it can accomplish longer tasks but you're right that the way it would happen is that it would look at the current image then it would decide that at first needs to find a sponge or maybe pick up the sponge if the sponge is already available then append that to prompt and continue. So we just wanted to make it short so that you can still get to get that idea across but only by having a single image. Yeah so it might be a little bit confusing that doesn't I think depict fully how the method works. Yeah I think we just got excited by the visual of a language model sort of seeing nothing and then waking up and saying oh I'm a robot. Okay here's my history of what I've done before. Okay depending on that what I thought I made a lot of sense doesn't make any sense anymore so it's more like excitement than anything else. It does look pretty sweet like it looks pretty cool especially like the effects on like the zoom seeing what's around. You use by the way we've not shown this yet you use these everyday robots constructions which look semi creepy but also quite cool especially when they pick up stuff they like hold it behind their back like it's like a mixture of a butler and someone who just has a knife and wants to stab you. But pretty sweet and it works surprisingly well. So maybe we can talk about the results of a little bit next because my next question would sort of be okay how well does it actually work in the environments where you tested on. Do you maybe want to comment a little bit on what was what were the general results and then you have some ablations. If a do you want to take this or do you. Yeah I think I can take this. So we tested this on two environments. One is the real office kitchen and another one is a kind of a mock office kitchen showing in figure five I think and we tested on a hundred and one in standard. From like six categories. Yeah so here here are the test environment that the A is a real kitchen and B is a mock kitchen. There are 15 objects that we focus on and also five semantic semantic locations. Like these locations are semantically meaningful like table trash can close counter far counter and a robot operator location where we define like bring back to you. That's where it is supposed to bring it back to. We test on a hundred and one instructions from six or seven categories if you scroll down a little bit. It's mainly to test different capabilities of the robot for example can it understand synonyms like non synonyms or verb synonyms like what does that mean. Throw away means bring something to the trash can like recycle means bring something to the trash can and also structure language which is just like verb non compositions. And also we test embodiment which means we test if the robot is not in the trash can. And also we test embodiment which means we test if the robot understands what its current embodiment is. For example if I already pick up something I shouldn't try to find it again because I already have it. Also we test on crowdsourced basically it's unstructured human queries from like coworkers for example and long source. And also we test embodiment which means we test if the robot understands what its current embodiment is. For example if I already pick up something I shouldn't try to find it again because I already have it. Also we test on crowdsourced basically it's unstructured human queries from like coworkers for example and long horizon tasks which are some of the really really challenging instructions such as I spilled my coke on the table how would you throw it away and then bring me something to clean. So that's a really challenging task the robot need to understand what does spill mean like what tools you can use to clean up a spill. So these are the instructions that we tested and overall I think we achieved 71 percent planning success rate and 66 percent execution success rate. And it's the hardest question is do the longer horizon tasks. So I think we only have about like 30 or 40 percent success rate. And yeah we are working on improving those like other success rate on those other questions. Ryan if you have anything to add. Yeah the only thing I was going to say is that the long horizon ones are particularly challenging both from like reasoning and language side. But a lot of the issue comes with if you have like a 90 percent success rate manipulation policy which is still quite high. Every time you do this you reduce the probability that your overall plan is going to succeed. And so that starts to like both it's a big challenge and we want to get our manipulation policies better and better and each of our low level skills better and better. But also having some sort of like closed loop that so the language model knows to retry would be really helpful here. And you I saw I saw in the results that it was pretty interesting in that you did ablate a lot of these things. For example you did ablate what for example if we don't have the language model and these are the overall success rate. You ablate what if we don't have the language model and what if we don't have the scoring model and generally they were worse much worse in both cases which was pretty cool to see and not always the same. Except in this one it is one thing to understand this correctly if you drop the generative model on a generative uses it uses a large language on a projects the nearest to the nearest skill via an embedding. That is actually better than your original policy. Is that just noise or is there something behind it if you use this verbs category. My guess is I think it's more noise than than anything else. But there were definitely times where so we see it like really fail in certain circumstances. So embodiment because there's no value function there to tell it that it can't do something. There's a real issue for it. And so there are a lot of failures for anything that didn't have a value function there. I think we saw some like some pretty interesting differences between the no value function. So this is the scoring model only without a value function and the generative model. And so some of the issues with the general model came around with like nouns for instance. And this is because when you do this projection. So the say I said I just worked out I want a snack it then projects to or then these the plan will say bring me a snack. But really what I want is a snack to help me recover from my workout. And so that like a little bit of information is enough to say it's probably not like potato chips but maybe something like healthier. Similarly like a drink there would lose a lot of its information. And so on the noun ones we saw that it ended up like losing this information and that cost a lot of the success rate. Whereas the scoring model did OK across the board but maybe not as like smoothly in the verb category. Another really fascinating thing here is at least in my opinion just the scale of data collection in this project. I have I have made a few notes and at one point it says something like you use a lot of human labelers for for example the success rate of these little policies. So even when when you train these little or small unit let's call them unit policies you use humans to see whether they're correct or not. And you use three human raters per execution and you get it you get give it one single sparse reward if two out of three agree. So like this scale seems immense. Is this really like how did you determine this was the best way to spend the human time and not maybe together more noisy but three times more like. Noisy but three times more labels or something like this. How did this come to be. Yeah this is a good question. I think we are still figuring this out. A lot of these questions and how to spend how to spend human time in the most efficient way that that helps the policies the most. And I think there is a question of crowd labeling as you as you mentioned. So how much noise can you tolerate in the reward function compared to like the throughput of that. Also how much time you should spend collecting human demonstrations versus how much time humans maybe should be just supervising robots collecting data autonomously. How much should we be spending time developing assets and policies in simulation and transferring them to the real world. So we are still kind of trying to find the trade-offs between all of these. I don't think we have any any very good answers right now. As for labeling itself we noticed in previous projects that the noise on the on the reward signal is going to be really can have a big influence on performance. So that's why we decided to have three labor laborers to to agree on the two of which we have to agree to to market the reward. And we also had additional questions such as was the behavior undesirable or unsafe. And these are sometimes quite ambiguous. So it's actually it helps quite a lot to have multiple people look at the video and and tell us what they think. Did you always have these additional things in. So you have as you say and also wrote this down somewhere a unsafe undesirable or infeasible. Did you always have this in or was this kind of a development that happened over time that you realized oh crap we're asking people how likely is the robot to pick up an apple but there is no apple in sight and things like this. Yeah so some of them we added. So initially we knew that safety is a is a big problem. So we started with with that question and we noticed that sometimes the robot would do something that isn't necessarily unsafe but we still don't want it to do it. For instance it will touch the object that it wasn't supposed to touch or it will just poke something and it will fall off the table. So then then we added the undesirable which is like has a slightly different definition and we can also optimize for it differently in the reward function. And then regarding the the last one the infeasibility this is something that we noticed with reinforcement learning algorithms that if you add a lot of data where the task wasn't feasible even though the data is technically correct. The robot didn't accomplish the task it got reward zero but it seems to be influencing the real algorithms in a bad way. So we added this in addition to prevent that and potentially filter for this data or see how we can change the real algorithms to handle that kind of data better. And why do you only give a single reward. I mean presumably a human watching a video like this could be you know every couple of frames could be like yeah good job robot yeah that's the right way yeah oh no don't do that. Like essentially like Peter Pan or like you know warmer warmer warmer colder colder which would give sort of a much more dense label space. Was this is like a technical limitation or did you also consciously choose to say no we got it's one single reward and that's only it's one when you fulfill the task and zero everywhere else. Yeah so there's I think a few reasons for this first I think the ambiguity that comes with it. You know it's already sometimes difficult to decide whether the task was accomplished correctly or whether it was undesirable or not. If in addition to this you have to add this continuous signal whether the robot is going in the right direction I think it can be fairly ambiguous depending on what the robot is doing. Secondly we made a decision some time ago that optimizing for sparse reward tasks would be just more scalable for the future. There are some tasks where it's quite difficult to say whether the robot is actually going in the in the right direction and sometimes that accomplishes a task in a surprising way and we don't necessarily want to eliminate that and introduce human bias of like well I think it should go that way. So our real algorithm is that we've been developing have also been optimized for the sparse reward setting. So that was kind of another factor that we that we thought about when when considering the reward function. So speaking about doing it like humans there's a yet another set of data collection in this project and that is that not only do you collect the labels but you also do quite a considerable amount of behavior cloning. From essentially learning from demonstrations from humans with another set of data gathered from you call it teleoperated teleoperator sessions. How can we how can we imagine such a teleoperator session like how many of these kitchens and robots do you have and how long does this take to gather a data set that you could conceivably use to collect data from humans. Gather a data set that you could conceivably do behavior cloning from. Yeah so I think we specified in the paper that we gathered at that point around 70,000 demonstrations for all these different tasks. This is across 11 robots I believe we built a little we built little stations for the robots like the stations that you can see in the picture here. Where the robots can can practice these things and people can demonstrate how to how to how to do things. I think we are still kind of trying to see how much of this if we if we filter the data set for instance how much can we filter it and still get really high result. So I think we we don't have very good answers to that yet. Yeah but this is something we're looking into kind of the trade-offs between how much demonstration how many demonstrations you're collecting how much autonomous data and so on. Where is this just because this is at Google which is a company and sure there's like a cash cow that generates infinite money but there's got to be some kind of constraint on you just or how do you how do you how does this work maybe. What robotics at Google what is your mission there and how do you pitch such a thing to to management like yeah essentially we want to collect 70,000 sessions of tele operated things every time a human presumably not a random human because they would just crash the robot out of spite. But like a trained trusted human needs to sit down and spend their time and there's robots are quite slow as of now. There's got to be a considerable budget behind all of this data collection and labeling and so on. How do you do you have to make a case for that or are you relatively free in doing this. How does how does your work in the same in the business perspective look like. Yeah I think in any company you kind of have to make a case or even in in academia you have to make a case for your project why you think this is how the money should be spent and where the resources should go. So usually the way we we kind of justify it as by showing kind of step by step results and showing if we extrapolate this where this is going to go. So we we we've done some projects previously where we showed reinforcement learning at scale with six robots or behavior cloning at scale with just two or three robots. And then we start seeing that with the amount of data that we collected there we already can see some interesting results and now if we want to get these robots to do many more things we need more robots we need more data. And this is kind of one big bet that we that we have in robotics at Google is that this large scale machine learning could be a way to really help robotics. So we want to we want to be able to be risk some of those questions for the for the community right. Like if we can actually buy a lot of robots and provide a lot of demonstrations how does it scale. How does it work. I think one of the sides or one of the figures in the appendix actually has somewhat like the way that we built up these skills one by one it's maybe I don't know what page it's on but it's a little higher than that. Yeah this one sort of shows like how these were built up over time and and how more one more and more skills were added more and more data was collected each time seeing signs of life for the algorithms and performance and improving upon that. And you can see that from time to time there's a new skill being added so that kind of goes from zero up in the meantime there's also the underlying code is changing. So it's kind of like improvements over time. So this goes it goes up and to the right which is what we all love. And was there was there major downturns in this project like times where you know things didn't seem to work out or you didn't exactly know what the problem was things like this. Could you get us a bit behind the scenes into when when things go wrong. No problem. There's quite a lot I'm just trying to think which one to tell you. There's quite a lot also from previous projects but I think one thing that was quite surprising to me personally and I think we are still kind of working on that is that if you spend in if you classify approaches into let's say imitation learning and reinforcement learning. If you spend enough time and data on either of them you can get them to work. So we some of the results that you see here most of them are from behavioral calling but we can achieve very comparable results with reinforcement learning either by transferring policies from simulation and then continue collecting with that policy and kind of fine tuning it to a high performance. Or by just bootstrapping from real data and improving upon that. But what is quite surprising is that combining these these two have have has been quite tricky. So kind of having a single algorithm that can digest all of that data that can digest all of the demonstrations as well as the autonomous data that was collected data that we collect in simulation and so on and have it have all the properties that fit into the data. So it performs at least as good as behavioral cloning but it can also improve autonomously and so on. This has been this has been quite surprising and tricky. I want to maybe have a bit of an or make a bit of an outlook right here because it seems we have a pretty cool way to go from skills that are described by language. But you have to define them. Let's just scroll to one of them. You have to define them ahead of time. Right. You have to define pick up the Coke can bring it to you find the Coke can and so on. You have to just you have to design these even though they're described by language. They're pretty fixed set. Now the first thing that maybe one can think about is how to extend that set and not necessarily extend the data. Just linearly. But I'm thinking of something when I say please clean up the table. You might not know what's on the table. So we need this kind of a concept of like almost like a variable or an unknown. You know like so the plan could be go to the table and then kind of decide what to do next. So the language model could get even or has to get a feedback almost from either the value functions or from the picture itself. Is that anything that's on your your radar sort of what if I don't what if I have to adjust my plan on the fly to the state that I'm going to encounter. How could this model be extended to to handle that. Let's say all the actions are in your action space. But you just don't know at the beginning which ones you're going to take. Yeah I guess right now we kind of like count on the value functions to sort of like collapse whatever your plan is into the thing that is actually possible in the world. I think like one of the most I guess straightforward ways to do it though maybe not straightforward in practice is to use things like visual transformers or like structured scene representations that actually tell the language model what's possible so that they can start like reasoning over it earlier on. The other thing is to add in something like these success rates success detectors that say OK you tried to do this and it wasn't possible. So maybe you tried to find an apple that was impossible. Perhaps the next thing to do is try to find an orange that may actually be in the scene. So there's some like combination of value functions giving it feedback about the scene. But right now we don't have anything that like has the language model really really reasoning over the steps because the value functions takes it take care of that like interaction. But one could fine tune it on some data that allows it to do that is probably the most straightforward way to do it. But whether that works is open question. I guess the other thing is and this would really also close the loop or close one of the loops is if I imagine that I also had a model that could take any visual input and then kind of describe that describe what's happening in the visual input. So I'm going to give it a video of pick up the of something picking up the Coke can and the thing would come up with like a label for it like this video shows pick up a Coke can. Then I'd have almost limitless possibilities. I could just let a robot move at random essentially let the language model or let this model describe what it's doing then kind of feed that to the language model and so on. So instead of you designing the actions that it should train I could just let it do stuff and then have a model describe that stuff and then use that. Is is that a plan or is there like a major hurdle on the way there because that would kind of result in a almost autonomously learning system. If you give it a good language model the language model could even also prompted what to try next right. But the language model could be like OK what should I learn next. I should probably learn to pick up an orange and then you just ran them around until the thing the description model says this looks like picking up an orange. I guess I can say something first and then I will ask like Carol because he has previously worked current Brian worked a little bit on like learning from play data. So what you describe kind of similar to that. What I want to mention is that we find language is a great kind of state obstruction because people invent language because they obstruct some states right. Like every every every word every sentence is meaningful. So there are some work in language showing that using language obstruction can improve exploration. For example you can use that to guide your exploration and summarize current states. So that's one potential direction that we can go. Yeah I think there is kind of multiple ways you can see pushing this to an extreme. I think like one small step in the direction would be rather than having these predefined skills label everything in hindsight as I think you're describing as well. And and train policies based on the hindsight labels. So it's not just pick up an apple but you know kind of however the person that looked at that video described it. That's the skill that the robot was performing. And then you maybe don't have to constrain the language model to pick across the skills that you train. But maybe you can just take the generative output and see how that works. I think there is also a potential potential research to be done in how much can language actually take from the robotics problem and how much can it help solving it. So right now we are operating at a certain level of abstraction like you command things like pick up the coke can and then the language model can operate on that. But you can also imagine operating on much lower level which is just like you know move this direction or that direction or something like that. And the language model commands all of that. And you kind of you can choose where in that abstraction you want to be. And I think it's quite interesting that we at least can contrive things like this because of how good language models are today. Yeah and I think I guess to that there's also works on using language basically to predict rewards like over states. And so that's like one way to kind of like hook it all together. We have this like general framework. What's the biggest hurdle like what's the biggest let's say unsolved problem to push push these sort of everyday robots not the company but like the the expression the robots that help us doing our tasks. What where's the like the biggest roadblock in getting these to a point where they could actually be usable. I think right now given kind of how much time we spend on different parts of the system. It's the skills themselves. The ball neck is still the robot actually doing the thing that you ask it to do. Even though these skills are simple to get them to the place where they generalize to any environment can kind of pick up any object even the object that wasn't trained on and do these tasks. And with large diversity of objects environments and so on to very high performance this is still really really hard. So I think if if we get much better skills underlying skills then well would have made a big step towards this actually being very useful. I was going to say the along with those skills like the way that we use the value functions is that as the skill improves so does the like value functions estimate of what it can do. So it's kind of nice where like position both to use these skills but it also improve the overall algorithm by having a better estimate of a success probability. So I think we're like I think sake and itself is at least set up in a good way to sort of like scale along with as this bottleneck is relieved. Last question from from my side what do you think of the Tesla bought. And when I give you the short pro in in in briefly in that it is the ultimate platform because the world is designed for designed for humans right. So if you have the humanoid robot conceivably it could do anything the human can at least mechanically. Do you does this sound good to you or is there like major skepticism. No comments. You can you can wager wager bets right now. I think one one thing that is maybe that I'm excited to see is I think Tesla has the ability to scale things up quite well. They seem to be a really good hardware company. And so it would be interesting to see how some of the problems change. This is also things that we are researching as well how problems change and how solutions change when you have many many of these robots. So I would be I would be excited to see they have any any good insights there. Is there last things that we maybe haven't touched on yet that you would like people to know here just for visuals. I'm showing what some of the successful episodes at the end which are quite impressive like very multi. So there's just one robot. This is this is a collage but very multi-step things. And I think that's just really impressive very long horizon planning things down to these individual actions. Yeah that's that's pretty cool. Anything any last thing you want to want to let people know how can they get started. Where can they find out more information. I just want to mention that we have the website on the website we have a couple of videos demo demonstrating how the robot works and how the inference process works along with the decision process. All the scores we have calculated along with the robot execution. So if there are anyone interested in like how our algorithm works check definitely check that out. I think like I guess what I'm most excited about with it is like how interpretable it is that you can actually see how the decision is being reached by the robot that you can see that the language model likes these things and that the affordance model understands that these tasks make sense or do not make sense in a given world embodied environment. I think it's like nice that it scales really well to adding in new tasks as we go. And then I guess towards how people would use it I think to start. Yeah I mean the paper and the website is a good place to go. I think we're planning to open source a version of it on a more kind of toy environment in the coming months. So hopefully that'll be like an exciting like easy way to sort of like get in the mix with both this and language models. I think there's a lot of power in in leveraging language models and kind of giving them these like hands and eyes to execute real world tasks. I also think you had a point earlier about basically like we use affordances but really it's just a value function. It's this value function doesn't necessarily have to map to an affordance. And I think that's a really powerful idea that we're basically taking all the knowledge in a language model and then hopefully applying it with a value function that isn't even necessarily normalized to can you do this or not. It's sort of what's helpful what's possible for whatever the RL train policy is doing. I think that's like a really I don't know open space. Yeah I'm also quite excited about how language can kind of chip away a little bit from the robotics problem. I think that's something that we haven't really thought about that much before. And we see that we can handle much more much longer horizon commands abstract commands and so on while keeping the policies fairly simple. So it's I think it's quite exciting to see how much further we can we can push that direction. Yeah I think representations have always been such a challenge for especially like task representations are such a challenge for robotics. And I think language has provided this like really nice interface to interact with the robot and then have the robot interact with the world. Excellent. Well Carl, Brian, Faye thank you very much for being here. This was a lot of fun and I hope to see you again soon. Thank you. Thank you for having us.
 So today we're here with three of the authors of this paper with I have to say a lot of authors
 It seems like a giant work just from what I could gather
 From the from the paper itself and the data collection and the evaluation and so on
 So this was a huge thing, but the results are pretty cool. So here with me today are Faye Xia
 Brian Ictor and Karol Hausmann who are three of the authors of this work. Welcome to the channel everyone
 Thanks. Thank you for having us. It's great to have you here
 The I like I love the title
 Because it's a bit of a mantra on the do as I do as I say not as I do which is kind of the other
 Way around right here and this idea of
 Connecting robots and language. It seems pretty natural
 I have to say I've I've seen a lot number of paper attempt to do something like this
 Like can we maybe translate what the language model says into the space of what the robot understands and things like this?
 But this here it seems like a bit of a new approach
 Why why did you try? Why did you attempt to do this?
 Like why does this seem promising and why did no one else do this thing yet?
 Yeah, I think to start like the
 To I guess like prior work on like using a language model to kind of translate it down
 I think we first started out with sort of like playing around with that and and realized I guess how much information is
 Embued in these language models and how well they're able to reason over sequences and remember what they've done
 But when we really like started thinking about applying it to the world
 It was sort of like odd that there's no way to basically
 Make sure that whatever it's saying actually makes sense for the environment that was in
 And so I think like after playing around that for a while we were sort of like stuck there like okay
 We have these like interesting plans
 But they don't actually make sense for everything that the the robot can do and so we started kind of like shifting towards towards that problem
 Yeah, I think also separately we've been trying to get robots to do many things and learn multiple skills and
 This is a very difficult problem
 and we were debating kind of the the best way to do this whether we should predefined the skills up front or whether we should just
 demonstrate kind of anything that comes to mind and label it afterwards and
 Just connecting these two dots the language models with the skills that we already have on the robots
 Seems like a nice way of factorizing this problem
 Did you always could you so you have this robot in this environment and is if I understood correctly?
 Maybe here is a good demonstration of that. So you have the robot in these two environments and
 These are the environments that exist to understand this correctly. So it's only these two environments. There's no
 generalization across environments
 Yeah, so we've been collecting data in beautiful environments. These are the two environments that we use for evaluation
 We also have a separate environment that is right next to the environment that it's a
 Mark this be here where robots are practicing
 But it looks fairly similar to to at least the stations that the robots practice on are fairly similar to the stations that you see here
 The backgrounds are changing the the objects are changing that we practice with and things like that. We also use
 Simulation as an additional environment that we then try to make look similar to the real world
 But we don't really focus in this paper on generalization to
 Completely new environment. We rather try to focus on kind of having a robot do as many things
 In a single environment when we talk about robot practicing things, I guess that's where your methods starts with robots
 Practicing things and by things I guess we mean a bunch of very
 Low-level let's call them unit unitary
 Skills like here. For example find a coke can pick up the coke can bring it to you something like this
 So these these could be things that
 Conceivably we could learn with something like behavior cloning or something like this. How did you?
 Decide on what actions are possible for these robots to do on their own like as a unit
 Some of it is based on what the robots capable of some of it's like what?
 Gives us a like a easy reward function
 And some of it was sort of motivated by what?
 Composes well into long horizon behaviors that you really want to do in the world like if we have a robot operating in a kitchen
 What would I ask it to do what what's required of it to do that?
 And how would I break down the task? I think was like part of the motivation like really how this robot is gonna operate in the world
 Yeah, and also it's interesting to see how this picture came out
 So initially we kind of have to come up with these and we kind of have to think up front
 What would that person ask a robot to do?
 But now that we have something running we can actually ask people and see how they interact with the robot and
 Decide on which skills we should be learning next based on that
 Sorry, I want to add that at the beginning we choose pick and place because these are
 Two fundamental skills that can unlock a large number of instructions that we are able to solve
 But it is also very easy to add new skills into the picture like we only need to
 Have a have a language language description for the skill and we also need a policy and value function
 So these are all the three three things you need to import a new skill into the second framework
 What I like here is that you said you need a policy and a value function that policy doesn't even have to be like
 Neural network based policy conceivably one skill can be a very classic control problem
 I believe when you pick up things
 You is is that correct that you classically control where the actuator should go and when you move the robot you kind of plan in space
 So not everything is like reinforcement learned or behavior cloned
 Yeah, so different skills are learned differently in this case pick was learned through behavior cloning on real data
 But yeah, for instance for instance moving around this is not
 Trained with reinforcement learning or behavior cloning. So yeah, you can compose you can have different algorithms
 Train different skills and these skills just to to round out the picture right here the input is
 Whatever the camera sees plus, you know kind of all the states of the actuators
 So that conceivably there's an apple in front of you and the task is pick up an apple
 And that that would be kind of the state from from where you operate. That's right
 Yeah, we are in the state of the actuators. So that's the input
 From where you operate. That's right. Yeah, we are going the value function. The value function describes kind of how likely you are to fulfill that task
 That's right. Yeah, so the input to the policy is the image that the robot sees that you get at every
 after every action
 We actuate the arm by doing end effector position control
 Yeah, these are the inputs and outputs
 And also there's a terminate action, right?
 Sorry that so the robot can say it itself when it's done
 Yes, so one of the actions that the robot can command is terminate which basically means i'm done now we can move on to the next one
 And okay, so now I guess that this is one part of the puzzle
 You have robots you have all these policies for all the little things that the robots could do these little things
 Were developed by you by the community conceivably you could also use the large language models itself
 To suggest new things to train right on the basic level. You could ask gpd3
 What would you do right here and then the little steps you could conceivably
 Make into like train into little actions, but you have this library of things
 And now the question is how do you how do you compose them and that's where the large language models comes in
 Do you want to comment maybe a little bit on like how does that look in a base in a basic way?
 How do we combine the knowledge of language models with these skills that the robots can do?
 Yeah, I guess uh at a high level so the language model already has
 So much knowledge about the world
 And how to do things in order and memory and things like that
 And the way to get it to like really speak in the way that is amenable to the robot first
 We show it a few like prompt examples. So we show it solving, you know, like about 10 problems
 And breaking it down from the query into the sequence of steps that it would take to solve that
 It turns out you can actually not use that and you still actually get like
 Some level of performance maybe like half the performance. So the language model just like comes out of the box
 With pretty good understanding of these tasks
 We then show with these examples this kind of brings it into the right frame of thought
 But if you do that and you ask for something new it doesn't like fully constrain it in a way that the robot will be able to understand it
 So our tasks along with the image and the states that we uh mentioned before also takes in like a task id
 So it says like um like pick up the apple
 So really what we need it to do is like output pick up the apple
 It can't say like pick up the fruit because the low level policies are not generalizing to that
 So to make sure that every time we actually output things you can do we instead of like taking the generative output of the language model
 We use what's called a scoring model
 So when a language model outputs, uh some text it also comes with a probability that it would output that text
 And so instead we can just like force it to only respond in these ways and say basically how likely it is to respond in that way
 So in this case we get like a score of if I were going to pick up the apple or put the apple somewhere
 These are the things i'd likely to respond
 These are the things there's no way I would respond
 And this gives us some like probability that the language model thinks this is really useful to the downstream task
 On the other side we have these value functions and policies that we've talked about
 There are actually the value functions outputting how likely it is to achieve a task
 I think actually there's another uh slide one like one more down
 Um, but this is basically or yeah, this yeah is saying basically these are possible from this state
 And so on one hand we have a language model saying this seems really useful for the task
 And on the other hand we have the value function saying this seems possible
 And together they give some probability that this is what you want to do to basically accomplish the high level instruction
 I have a number of okay. Let's just start at at the beginning at the at the language model level
 I see the high level picture you you ask both the language model and the value functions
 What's what you know what they think should happen next and uh the combination of the two is what then you really do
 Which makes a lot of sense when you do ask these language models what to do um right here
 You said you use the you use the essentially you you ask the language model for the likelihood of of an output instead of letting it generate the output
 Was this your first try because one could also imagine uh you know saying something like of the following options
 Which one would you pick right and then you list all the options which would conceivably be more general because you could option you could add options over time
 And stuff like I guess you could do that here as well but um was this your first attempt or did you did you have some prompt engineering attempts before that
 Yeah I think at first we tried just like prompt engineering to see like basically what the generative model would output I think like our our initial thinking was we just want the generative model to basically plan as much as we can
 But that runs into two problems one is that it doesn't constrain the output fully so if I give it all these examples and then I said
 How would you put a fruit on the table instead of an apple on the table the generative model will actually respond with like number one find a fruit number two pick up the fruit
 And then you need to figure out how to like take that and project it into the final like thing that the robot can actually handle
 You can project this in some sort of like embedding space and that works sort of well but you actually lose some context on the overall query so I guess the way that we do it is a little bit more like well founded so to speak
 But the other really nice benefit of this is it gives us scores for everything which is really interpretable it lets us like see the trade off between these two options
 So in your example you said you know what if I just said here are your options pick one and the language model would probably pick one but now you only know that this is its favorite option
 You don't know the probability that it would have done maybe maybe it's actually okay with the next three options so this gives us this like interpretable score that we can then combine with the value functions
 Yeah there are some caveats to this I feel in that for example we know that by definition longer outputs are less likely right so I guess it's not too much of a problem for you because most of yours are like three or four words but have you noticed any of kind of these effects of just how these probabilities are constructed as kind of multiplications of softmax outputs like that's got to bring its own bias into the into the picture
 Have you observed any of that have you had problems with any of that or was it was it generally okay
 Yeah it's it's definitely a little bit of an issue I mean I think in general it's also very particular to the to these like if you were to misspell a word in there or like have an A versus an M it's not particularly robust to those in the options it is in the query like to what the user might say
 but not when you're scoring these options because if one word is off then this like multiplication of each word just kind of tanks the entire score so we did have to be somewhat careful with what we have one way to kind of like get around this a little bit is if you have some like end of statement token and if it if it adds extra words on the end then it's saying if if there's like more to come that end of token will basically kind of normalize the rest of it like you can't end a statement or a word
 and a statement early
 the yeah I think in the other thing that we did try to do is like potentially normalize them so knowing that this query is longer perhaps we need to upweight it or have some normalization on the language output
 but we found that it wasn't particularly consistent and there wasn't like just a constant effect across one or the other and it depends on the way you like referred to the query and so at the end of the day we just took the outputs as they were so it was an issue but it wasn't like a huge one
 I imagine that there's another factor here for example if you if you say you said before pick up a fruit or please bring me a fruit or something of this you're essentially relying on the ability of the large language model to sort of recognize that apple is a fruit and and and kind of interpret that in the same way and and so on
 so the kind of close as the language model estimates how close the things are did you find this generally in agreement of in how how humans find how close the things are and maybe yeah I'm just I'm just wondering about this notion of how how close things in language are together also what happens if you for example have an apple and an orange in your scene these two things would be quite close together so even if you said you know
 please pick up an apple the pickup an orange thing would conceivably score quite high in the language model which might perturb your things so I can kind of I can sort of make out that you have an ideal environment right here in that you probably picked objects that are distinct from each other locations that are fairly distinct from each other right such that there's a nice semantic gap between the things
 what like do you think this is well applicable to a real world setting or what kind of hurdles could there be with connecting language models and the set of actions in this way so I guess the first question was about do these families kind of align with what you would expect and that was actually that was one of the first things that I was looking at was like how well do these scores sort of match up to what you think it's going to be so yeah it turns out that like apples are
 apples and orange and banana are all going to score quite highly when you're asking for a fruit if you ask for a snack all the food options are going to score highly similarly drink soda any category like that it performs about yes you would expect as a human which is good but then yeah it comes to this problem of what if there's an apple and orange or what if there's an orange but not an apple and that's where these value functions come in this is actually like one of the key reasons why we have to do this in volume and grounding.
 Because if you just asked a regular language model that doesn't know what's there then how does it make that decision maybe it uses the wrong one then your plan isn't really correct and our policies may not actually work.
 But the value function tells you if there is an apple in the scene and no orange then you're going to see a high value function on the apple because the pick apple command could work versus the orange command is going to be quite low and so that actually lets you sort of like disambiguate this so.
 In the in figure B if it had a pick up the Red Bull if you said bring me a drink and there's a Red Bull but no water it's going to pick up the Red Bull because that's actually what's there.
 And if not then then the instruction itself is ambiguous right if you say pick up a drink and there's two drinks and both are affordable according to the value function yeah.
 Yeah then we think like either is completely fine I think it's also interesting because then the robot is making the trade off itself dependent maybe on the value function so for instance if you ask for a fruit and there's an orange and an apple but it's much better at picking up apples.
 Maybe it will pick up the apple because the value function will just tip the scale.
 So it will make some errors in that sense but since this is interpretable and you can kind of look back and see why I decided for that it can also inform us as to what skill we should train a little bit more or which value functions are a little under fitted and things like that so it will make some sort of mistake.
 But maybe that's that's okay maybe that's acceptable.
 I think one like really nice feature of that too is it's not necessarily always like it's better at picking up oranges or apples but you can see like these objects are in different locations one may be better for the policy than the other.
 So we're going to end up doing the one that's a little more robust and a little more likely to succeed as long as it still fulfills the high level query.
 Yeah I like the fact that you have success probability as sort of the ultimate score because I was I also thought one failure mode here is that some tasks are inherently harder than others right and so naturally your value function would be lower and therefore you can misinterpret just by the fact like well like this this this is me the procrastinator like this thing seems really hard and we'll do this other thing that I'm not sure how to do.
 But it's really easy so it's almost it's almost too human how the robot would act in this way.
 So yeah you have these what I like here as well is that you have to bank of value functions on one hand the language model on the other hand and the language model on the other hand is the one that's the most difficult to do.
 So yeah you have these what I like here as well is that you have to bank of value functions on one hand the language model on the other hand and they are never if I understand correctly trained together right there never in fact the language model is probably just frozen.
 So they're never trained together which means that you could conceivably just add a skill to the robot train its value function for it and just plug it in and and go.
 Yeah we can scale this fairly easily so we can continue adding skills we can also change the underlying algorithm how we train the skills.
 Or how we train the particular skill that we want to add if we if suddenly there is a really good script that allows to I don't know swipe the floor or something like that.
 We can we can also add that as long as we have a value function for it and also at the same time if the language model becomes better we can also swap out the language model and get improvements through that.
 I want to add that so our current value function is one way that we instantiate affordance but there are many other ways that we can instantiate affordance like for example we can directly do prediction.
 We can also use classical motion planning like to calculate for example length of the trajectory is also or the probability of success if you do like sampling based motion planning.
 So there are many ways that we can come to the affordance and the method is really flexible to plug in any type of affordance.
 I guess a big topic in maybe maybe it's more the space of blockchains and things like this is agents that do an action for you but also optimize for example for cost or for resources or something like this.
 This could directly flow into that where you can tell the robot you know do whatever fulfills my task but also costs very little and this could if this directly flows into affordance there might be a normalization issue but if this directly flows in you'd have you could tune the knobs on these on these functions fairly easily.
 So this is the full algorithm I guess we haven't talked yet about how you extend this to multiple steps but it is as far as I can tell fairly easy in that you do this in sort of a stepwise fashion so first you ask your language model your value functions at the current state and the current camera position where what should be done.
 Then you try to whatever should be done according to both scores combined you execute that and after you execute it you ask the same thing again but now the prompt changes and it's simply that you so here the prompt is essentially I would first and then first action is decided and once you go on the prompt now says I would first.
 The prompt now says I would first and then whatever was decided on and then second and then it's simply the same thing with the next action did I get this approximately correct.
 Do you pay any attention to whether or not the task was fulfilled successfully.
 So right now we don't we assume it will successfully execute I think some things could happen like if it fails at a navigation task say it was trying to navigate to an apple and the and it doesn't get there then the value functions at that next state are going to be quite low so you're not going to be able to basically pick something up or whatever so maybe then you end up selecting navigate to the apple again or navigate to a table instead but we don't have any like explicit success.
 Detection I think this is like one area that we're like pretty interested in going basically like finishing the job closing the loop entirely on when you try to do something did you succeed telling the language model and then having a language model adapt accordingly.
 I want to show one video from from your website which in this case if I got this right it confused me I guess a little bit because this thing right here if you see it kind of looks around sees all the things right like looks and sees and then it kind of scores the actions.
 And like this so pick apple I can't do that pick sponge okay.
 Bring you a sponge no not go to trash can place the sponge place the sponge is good and that's the place the sponge kind of up ways to bring you a sponge or like what's going on right here because in my in my estimation the robot shouldn't even look around initially the robot should just have its camera position fixed and then it in first instance.
 It should probably figure out like find a sponge or something like this and then it would move and then it would see consider these next actions like what is what is this video supposed to to show.
 Yeah I think you're understanding is completely correct so this is more like a conceptual video where we wanted to kind of across that it can accomplish longer tasks but you're right that the way it would happen is that it would look at the current image then it would decide that at first needs to find a sponge or maybe pick up the sponge if the sponge is already available then append that to prompt and continue.
 So we just wanted to make it short so that you can still get to get that idea across but only by having a single image.
 Yeah so it might be a little bit confusing that doesn't I think depict fully how the method works.
 Yeah I think we just got excited by the visual of a language model sort of seeing nothing and then waking up and saying oh I'm a robot. Okay here's my history of what I've done before. Okay depending on that what I thought I made a lot of sense doesn't make any sense anymore so it's more like excitement than anything else.
 It does look pretty sweet like it looks pretty cool especially like the effects on like the zoom seeing what's around.
 You use by the way we've not shown this yet you use these everyday robots constructions which look semi creepy but also quite cool especially when they pick up stuff they like hold it behind their back like it's like a mixture of a butler and someone who just has a knife and wants to stab you.
 But pretty sweet and it works surprisingly well. So maybe we can talk about the results of a little bit next because my next question would sort of be okay how well does it actually work in the environments where you tested on.
 Do you maybe want to comment a little bit on what was what were the general results and then you have some ablations.
 If a do you want to take this or do you. Yeah I think I can take this. So we tested this on two environments. One is the real office kitchen and another one is a kind of a mock office kitchen showing in figure five I think and we tested on a hundred and one in standard.
 From like six categories. Yeah so here here are the test environment that the A is a real kitchen and B is a mock kitchen. There are 15 objects that we focus on and also five semantic semantic locations.
 Like these locations are semantically meaningful like table trash can close counter far counter and a robot operator location where we define like bring back to you. That's where it is supposed to bring it back to.
 We test on a hundred and one instructions from six or seven categories if you scroll down a little bit. It's mainly to test different capabilities of the robot for example can it understand synonyms like non synonyms or verb synonyms like what does that mean.
 Throw away means bring something to the trash can like recycle means bring something to the trash can and also structure language which is just like verb non compositions. And also we test embodiment which means we test if the robot is not in the trash can.
 And also we test embodiment which means we test if the robot understands what its current embodiment is. For example if I already pick up something I shouldn't try to find it again because I already have it. Also we test on crowdsourced basically it's unstructured human queries from like coworkers for example and long source.
 And also we test embodiment which means we test if the robot understands what its current embodiment is. For example if I already pick up something I shouldn't try to find it again because I already have it. Also we test on crowdsourced basically it's unstructured human queries from like coworkers for example and long horizon tasks which are some of the really really challenging instructions such as I spilled my coke on the table how would you throw it away and then bring me something to clean.
 So that's a really challenging task the robot need to understand what does spill mean like what tools you can use to clean up a spill. So these are the instructions that we tested and overall I think we achieved 71 percent planning success rate and 66 percent execution success rate.
 And it's the hardest question is do the longer horizon tasks. So I think we only have about like 30 or 40 percent success rate.
 And yeah we are working on improving those like other success rate on those other questions. Ryan if you have anything to add.
 Yeah the only thing I was going to say is that the long horizon ones are particularly challenging both from like reasoning and language side.
 But a lot of the issue comes with if you have like a 90 percent success rate manipulation policy which is still quite high. Every time you do this you reduce the probability that your overall plan is going to succeed.
 And so that starts to like both it's a big challenge and we want to get our manipulation policies better and better and each of our low level skills better and better.
 But also having some sort of like closed loop that so the language model knows to retry would be really helpful here.
 And you I saw I saw in the results that it was pretty interesting in that you did ablate a lot of these things.
 For example you did ablate what for example if we don't have the language model and these are the overall success rate.
 You ablate what if we don't have the language model and what if we don't have the scoring model and generally they were worse much worse in both cases which was pretty cool to see and not always the same.
 Except in this one it is one thing to understand this correctly if you drop the generative model on a generative uses it uses a large language on a projects the nearest to the nearest skill via an embedding.
 That is actually better than your original policy. Is that just noise or is there something behind it if you use this verbs category.
 My guess is I think it's more noise than than anything else.
 But there were definitely times where so we see it like really fail in certain circumstances.
 So embodiment because there's no value function there to tell it that it can't do something.
 There's a real issue for it. And so there are a lot of failures for anything that didn't have a value function there.
 I think we saw some like some pretty interesting differences between the no value function.
 So this is the scoring model only without a value function and the generative model.
 And so some of the issues with the general model came around with like nouns for instance.
 And this is because when you do this projection.
 So the say I said I just worked out I want a snack it then projects to or then these the plan will say bring me a snack.
 But really what I want is a snack to help me recover from my workout.
 And so that like a little bit of information is enough to say it's probably not like potato chips but maybe something like healthier.
 Similarly like a drink there would lose a lot of its information.
 And so on the noun ones we saw that it ended up like losing this information and that cost a lot of the success rate.
 Whereas the scoring model did OK across the board but maybe not as like smoothly in the verb category.
 Another really fascinating thing here is at least in my opinion just the scale of data collection in this project.
 I have I have made a few notes and at one point it says something like you use a lot of human labelers for for example the success rate of these little policies.
 So even when when you train these little or small unit let's call them unit policies you use humans to see whether they're correct or not.
 And you use three human raters per execution and you get it you get give it one single sparse reward if two out of three agree.
 So like this scale seems immense.
 Is this really like how did you determine this was the best way to spend the human time and not maybe together more noisy but three times more like.
 Noisy but three times more labels or something like this.
 How did this come to be.
 Yeah this is a good question.
 I think we are still figuring this out.
 A lot of these questions and how to spend how to spend human time in the most efficient way that that helps the policies the most.
 And I think there is a question of crowd labeling as you as you mentioned.
 So how much noise can you tolerate in the reward function compared to like the throughput of that.
 Also how much time you should spend collecting human demonstrations versus how much time humans maybe should be just supervising robots collecting data autonomously.
 How much should we be spending time developing assets and policies in simulation and transferring them to the real world.
 So we are still kind of trying to find the trade-offs between all of these.
 I don't think we have any any very good answers right now.
 As for labeling itself we noticed in previous projects that the noise on the on the reward signal is going to be really can have a big influence on performance.
 So that's why we decided to have three labor laborers to to agree on the two of which we have to agree to to market the reward.
 And we also had additional questions such as was the behavior undesirable or unsafe.
 And these are sometimes quite ambiguous.
 So it's actually it helps quite a lot to have multiple people look at the video and and tell us what they think.
 Did you always have these additional things in.
 So you have as you say and also wrote this down somewhere a unsafe undesirable or infeasible.
 Did you always have this in or was this kind of a development that happened over time that you realized oh crap we're asking people how likely is the robot to pick up an apple but there is no apple in sight and things like this.
 Yeah so some of them we added.
 So initially we knew that safety is a is a big problem.
 So we started with with that question and we noticed that sometimes the robot would do something that isn't necessarily unsafe but we still don't want it to do it.
 For instance it will touch the object that it wasn't supposed to touch or it will just poke something and it will fall off the table.
 So then then we added the undesirable which is like has a slightly different definition and we can also optimize for it differently in the reward function.
 And then regarding the the last one the infeasibility this is something that we noticed with reinforcement learning algorithms that if you add a lot of data where the task wasn't feasible even though the data is technically correct.
 The robot didn't accomplish the task it got reward zero but it seems to be influencing the real algorithms in a bad way.
 So we added this in addition to prevent that and potentially filter for this data or see how we can change the real algorithms to handle that kind of data better.
 And why do you only give a single reward.
 I mean presumably a human watching a video like this could be you know every couple of frames could be like yeah good job robot yeah that's the right way yeah oh no don't do that.
 Like essentially like Peter Pan or like you know warmer warmer warmer colder colder which would give sort of a much more dense label space.
 Was this is like a technical limitation or did you also consciously choose to say no we got it's one single reward and that's only it's one when you fulfill the task and zero everywhere else.
 Yeah so there's I think a few reasons for this first I think the ambiguity that comes with it.
 You know it's already sometimes difficult to decide whether the task was accomplished correctly or whether it was undesirable or not.
 If in addition to this you have to add this continuous signal whether the robot is going in the right direction I think it can be fairly ambiguous depending on what the robot is doing.
 Secondly we made a decision some time ago that optimizing for sparse reward tasks would be just more scalable for the future.
 There are some tasks where it's quite difficult to say whether the robot is actually going in the in the right direction and sometimes that accomplishes a task in a surprising way and we don't necessarily want to eliminate that and introduce human bias of like well I think it should go that way.
 So our real algorithm is that we've been developing have also been optimized for the sparse reward setting.
 So that was kind of another factor that we that we thought about when when considering the reward function.
 So speaking about doing it like humans there's a yet another set of data collection in this project and that is that not only do you collect the labels but you also do quite a considerable amount of behavior cloning.
 From essentially learning from demonstrations from humans with another set of data gathered from you call it teleoperated teleoperator sessions.
 How can we how can we imagine such a teleoperator session like how many of these kitchens and robots do you have and how long does this take to gather a data set that you could conceivably use to collect data from humans.
 Gather a data set that you could conceivably do behavior cloning from.
 Yeah so I think we specified in the paper that we gathered at that point around 70,000 demonstrations for all these different tasks.
 This is across 11 robots I believe we built a little we built little stations for the robots like the stations that you can see in the picture here.
 Where the robots can can practice these things and people can demonstrate how to how to how to do things.
 I think we are still kind of trying to see how much of this if we if we filter the data set for instance how much can we filter it and still get really high result.
 So I think we we don't have very good answers to that yet.
 Yeah but this is something we're looking into kind of the trade-offs between how much demonstration how many demonstrations you're collecting how much autonomous data and so on.
 Where is this just because this is at Google which is a company and sure there's like a cash cow that generates infinite money but there's got to be some kind of constraint on you just or how do you how do you how does this work maybe.
 What robotics at Google what is your mission there and how do you pitch such a thing to to management like yeah essentially we want to collect 70,000 sessions of tele operated things every time a human presumably not a random human because they would just crash the robot out of spite.
 But like a trained trusted human needs to sit down and spend their time and there's robots are quite slow as of now.
 There's got to be a considerable budget behind all of this data collection and labeling and so on.
 How do you do you have to make a case for that or are you relatively free in doing this.
 How does how does your work in the same in the business perspective look like.
 Yeah I think in any company you kind of have to make a case or even in in academia you have to make a case for your project why you think this is how the money should be spent and where the resources should go.
 So usually the way we we kind of justify it as by showing kind of step by step results and showing if we extrapolate this where this is going to go.
 So we we we've done some projects previously where we showed reinforcement learning at scale with six robots or behavior cloning at scale with just two or three robots.
 And then we start seeing that with the amount of data that we collected there we already can see some interesting results and now if we want to get these robots to do many more things we need more robots we need more data.
 And this is kind of one big bet that we that we have in robotics at Google is that this large scale machine learning could be a way to really help robotics.
 So we want to we want to be able to be risk some of those questions for the for the community right.
 Like if we can actually buy a lot of robots and provide a lot of demonstrations how does it scale.
 How does it work.
 I think one of the sides or one of the figures in the appendix actually has somewhat like the way that we built up these skills one by one it's maybe I don't know what page it's on but it's a little higher than that.
 Yeah this one sort of shows like how these were built up over time and and how more one more and more skills were added more and more data was collected each time seeing signs of life for the algorithms and performance and improving upon that.
 And you can see that from time to time there's a new skill being added so that kind of goes from zero up in the meantime there's also the underlying code is changing.
 So it's kind of like improvements over time.
 So this goes it goes up and to the right which is what we all love.
 And was there was there major downturns in this project like times where you know things didn't seem to work out or you didn't exactly know what the problem was things like this.
 Could you get us a bit behind the scenes into when when things go wrong.
 No problem.
 There's quite a lot I'm just trying to think which one to tell you.
 There's quite a lot also from previous projects but I think one thing that was quite surprising to me personally and I think we are still kind of working on that is that if you spend in if you classify approaches into let's say imitation learning and reinforcement learning.
 If you spend enough time and data on either of them you can get them to work.
 So we some of the results that you see here most of them are from behavioral calling but we can achieve very comparable results with reinforcement learning either by transferring policies from simulation and then continue collecting with that policy and kind of fine tuning it to a high performance.
 Or by just bootstrapping from real data and improving upon that.
 But what is quite surprising is that combining these these two have have has been quite tricky.
 So kind of having a single algorithm that can digest all of that data that can digest all of the demonstrations as well as the autonomous data that was collected data that we collect in simulation and so on and have it have all the properties that fit into the data.
 So it performs at least as good as behavioral cloning but it can also improve autonomously and so on.
 This has been this has been quite surprising and tricky.
 I want to maybe have a bit of an or make a bit of an outlook right here because it seems we have a pretty cool way to go from skills that are described by language.
 But you have to define them.
 Let's just scroll to one of them.
 You have to define them ahead of time.
 Right.
 You have to define pick up the Coke can bring it to you find the Coke can and so on.
 You have to just you have to design these even though they're described by language.
 They're pretty fixed set.
 Now the first thing that maybe one can think about is how to extend that set and not necessarily extend the data.
 Just linearly.
 But I'm thinking of something when I say please clean up the table.
 You might not know what's on the table.
 So we need this kind of a concept of like almost like a variable or an unknown.
 You know like so the plan could be go to the table and then kind of decide what to do next.
 So the language model could get even or has to get a feedback almost from either the value functions or from the picture itself.
 Is that anything that's on your your radar sort of what if I don't what if I have to adjust my plan on the fly to the state that I'm going to encounter.
 How could this model be extended to to handle that.
 Let's say all the actions are in your action space.
 But you just don't know at the beginning which ones you're going to take.
 Yeah I guess right now we kind of like count on the value functions to sort of like collapse whatever your plan is into the thing that is actually possible in the world.
 I think like one of the most I guess straightforward ways to do it though maybe not straightforward in practice is to use things like visual transformers or like structured scene representations that actually tell the language model what's possible so that they can start like reasoning over it earlier on.
 The other thing is to add in something like these success rates success detectors that say OK you tried to do this and it wasn't possible. So maybe you tried to find an apple that was impossible.
 Perhaps the next thing to do is try to find an orange that may actually be in the scene.
 So there's some like combination of value functions giving it feedback about the scene.
 But right now we don't have anything that like has the language model really really reasoning over the steps because the value functions takes it take care of that like interaction.
 But one could fine tune it on some data that allows it to do that is probably the most straightforward way to do it.
 But whether that works is open question.
 I guess the other thing is and this would really also close the loop or close one of the loops is if I imagine that I also had a model that could take any visual input and then kind of describe that describe what's happening in the visual input.
 So I'm going to give it a video of pick up the of something picking up the Coke can and the thing would come up with like a label for it like this video shows pick up a Coke can.
 Then I'd have almost limitless possibilities.
 I could just let a robot move at random essentially let the language model or let this model describe what it's doing then kind of feed that to the language model and so on.
 So instead of you designing the actions that it should train I could just let it do stuff and then have a model describe that stuff and then use that.
 Is is that a plan or is there like a major hurdle on the way there because that would kind of result in a almost autonomously learning system.
 If you give it a good language model the language model could even also prompted what to try next right.
 But the language model could be like OK what should I learn next.
 I should probably learn to pick up an orange and then you just ran them around until the thing the description model says this looks like picking up an orange.
 I guess I can say something first and then I will ask like Carol because he has previously worked current Brian worked a little bit on like learning from play data.
 So what you describe kind of similar to that.
 What I want to mention is that we find language is a great kind of state obstruction because people invent language because they obstruct some states right.
 Like every every every word every sentence is meaningful.
 So there are some work in language showing that using language obstruction can improve exploration.
 For example you can use that to guide your exploration and summarize current states. So that's one potential direction that we can go.
 Yeah I think there is kind of multiple ways you can see pushing this to an extreme.
 I think like one small step in the direction would be rather than having these predefined skills label everything in hindsight as I think you're describing as well.
 And and train policies based on the hindsight labels.
 So it's not just pick up an apple but you know kind of however the person that looked at that video described it.
 That's the skill that the robot was performing.
 And then you maybe don't have to constrain the language model to pick across the skills that you train.
 But maybe you can just take the generative output and see how that works.
 I think there is also a potential potential research to be done in how much can language actually take from the robotics problem and how much can it help solving it.
 So right now we are operating at a certain level of abstraction like you command things like pick up the coke can and then the language model can operate on that.
 But you can also imagine operating on much lower level which is just like you know move this direction or that direction or something like that.
 And the language model commands all of that.
 And you kind of you can choose where in that abstraction you want to be.
 And I think it's quite interesting that we at least can contrive things like this because of how good language models are today.
 Yeah and I think I guess to that there's also works on using language basically to predict rewards like over states.
 And so that's like one way to kind of like hook it all together.
 We have this like general framework.
 What's the biggest hurdle like what's the biggest let's say unsolved problem to push push these sort of everyday robots not the company but like the the expression the robots that help us doing our tasks.
 What where's the like the biggest roadblock in getting these to a point where they could actually be usable.
 I think right now given kind of how much time we spend on different parts of the system.
 It's the skills themselves.
 The ball neck is still the robot actually doing the thing that you ask it to do.
 Even though these skills are simple to get them to the place where they generalize to any environment can kind of pick up any object even the object that wasn't trained on and do these tasks.
 And with large diversity of objects environments and so on to very high performance this is still really really hard.
 So I think if if we get much better skills underlying skills then well would have made a big step towards this actually being very useful.
 I was going to say the along with those skills like the way that we use the value functions is that as the skill improves so does the like value functions estimate of what it can do.
 So it's kind of nice where like position both to use these skills but it also improve the overall algorithm by having a better estimate of a success probability.
 So I think we're like I think sake and itself is at least set up in a good way to sort of like scale along with as this bottleneck is relieved.
 Last question from from my side what do you think of the Tesla bought.
 And when I give you the short pro in in in briefly in that it is the ultimate platform because the world is designed for designed for humans right.
 So if you have the humanoid robot conceivably it could do anything the human can at least mechanically.
 Do you does this sound good to you or is there like major skepticism.
 No comments.
 You can you can wager wager bets right now.
 I think one one thing that is maybe that I'm excited to see is I think Tesla has the ability to scale things up quite well.
 They seem to be a really good hardware company.
 And so it would be interesting to see how some of the problems change.
 This is also things that we are researching as well how problems change and how solutions change when you have many many of these robots.
 So I would be I would be excited to see they have any any good insights there.
 Is there last things that we maybe haven't touched on yet that you would like people to know here just for visuals.
 I'm showing what some of the successful episodes at the end which are quite impressive like very multi.
 So there's just one robot.
 This is this is a collage but very multi-step things.
 And I think that's just really impressive very long horizon planning things down to these individual actions.
 Yeah that's that's pretty cool.
 Anything any last thing you want to want to let people know how can they get started.
 Where can they find out more information.
 I just want to mention that we have the website on the website we have a couple of videos demo demonstrating how the robot works and how the inference process works along with the decision process.
 All the scores we have calculated along with the robot execution.
 So if there are anyone interested in like how our algorithm works check definitely check that out.
 I think like I guess what I'm most excited about with it is like how interpretable it is that you can actually see how the decision is being reached by the robot that you can see that the language model likes these things and that the affordance model understands that these tasks make sense or do not make sense in a given world embodied environment.
 I think it's like nice that it scales really well to adding in new tasks as we go.
 And then I guess towards how people would use it I think to start.
 Yeah I mean the paper and the website is a good place to go.
 I think we're planning to open source a version of it on a more kind of toy environment in the coming months.
 So hopefully that'll be like an exciting like easy way to sort of like get in the mix with both this and language models.
 I think there's a lot of power in in leveraging language models and kind of giving them these like hands and eyes to execute real world tasks.
 I also think you had a point earlier about basically like we use affordances but really it's just a value function.
 It's this value function doesn't necessarily have to map to an affordance.
 And I think that's a really powerful idea that we're basically taking all the knowledge in a language model and then hopefully applying it with a value function that isn't even necessarily normalized to can you do this or not.
 It's sort of what's helpful what's possible for whatever the RL train policy is doing.
 I think that's like a really I don't know open space.
 Yeah I'm also quite excited about how language can kind of chip away a little bit from the robotics problem.
 I think that's something that we haven't really thought about that much before.
 And we see that we can handle much more much longer horizon commands abstract commands and so on while keeping the policies fairly simple.
 So it's I think it's quite exciting to see how much further we can we can push that direction.
 Yeah I think representations have always been such a challenge for especially like task representations are such a challenge for robotics.
 And I think language has provided this like really nice interface to interact with the robot and then have the robot interact with the world.
 Excellent.
 Well Carl, Brian, Faye thank you very much for being here.
 This was a lot of fun and I hope to see you again soon.
 Thank you. Thank you for having us.