This Team won the Minecraft RL BASALT Challenge! (Paper Explanation & Interview with the authors)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper minecraft minerl minerl basalt minecraft machine learning minecraft ai human-like ai minecraft bot minecraft ai challenge minecraft reinforcement learning behavior cloning kairos minecraft kairos minerl kairos minerl winners interview with the authors minecraft deep learning minecraft behavior cloning gail generative adversarial imitation learning state machine
#minerl #minecraft #deeplearning

The MineRL BASALT challenge has no reward functions or technical descriptions of what's to be achieved. Instead, the goal of each task is given as a short natural language string, and the agent is evaluated by a team of human judges who rate both how well the goal has been fulfilled, as well as how human-like the agent behaved. In this video, I interview KAIROS, the winning team of the 2021 challenge, and discuss how they used a combination of machine learning, efficient data collection, hand engineering, and a bit of knowledge about Minecraft to beat all other teams.

OUTLINE:
0:00 - Introduction
4:10 - Paper Overview
11:15 - Start of Interview
17:05 - First Approach
20:30 - State Machine
26:45 - Efficient Label Collection
30:00 - Navigation Policy
38:15 - Odometry Estimation
46:00 - Pain Points & Learnings
50:40 - Live Run Commentary
58:50 - What other tasks can be solved?
1:01:55 - What made the difference?
1:07:30 - Recommendations & Conclusion
1:11:10 - Full Runs: Waterfall
1:12:40 - Full Runs: Build House
1:17:45 - Full Runs: Animal Pen
1:20:50 - Full Runs: Find Cave

Paper: https://arxiv.org/abs/2112.03482
Code: https://github.com/viniciusguigo/kairos_minerl_basalt
Challenge Website: https://minerl.io/basalt/

Paper Title: Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft

Abstract:
Real-world tasks of interest are generally poorly defined by human-readable descriptions and have no pre-defined reward signals unless it is defined by a human designer. Conversely, data-driven algorithms are often designed to solve a specific, narrowly defined, task with performance metrics that drives the agent's learning. In this work, we present the solution that won first place and was awarded the most human-like agent in the 2021 NeurIPS Competition MineRL BASALT Challenge: Learning from Human Feedback in Minecraft, which challenged participants to use human data to solve four tasks defined only by a natural language description and no reward function. Our approach uses the available human demonstration data to train an imitation learning policy for navigation and additional human feedback to train an image classifier. These modules, together with an estimated odometry map, are then combined into a state-machine designed based on human knowledge of the tasks that breaks them down in a natural hierarchy and controls which macro behavior the learning agent should follow at any instant. We compare this hybrid intelligence approach to both end-to-end machine learning and pure engineered solutions, which are then judged by human evaluators. Codebase is available at this https URL.

Authors: Vinicius G. Goecks, Nicholas Waytowich, David Watkins, Bharat Prakash

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 If we just do a behavior cloning using this data, it won't cut it. Like, we don't have enough data. Hello there! Today we're going to look at this right here. This is an agent in Minecraft that's trying to build a waterfall. So the goal is to go up a mountain, find a good spot, put down some water, turn around and then take a beautiful picture of the waterfall. That is one of the four tasks of the Mine RL Basalt Competition. This is what we're going to talk about today. And not only are we going to talk about the challenge, the competition, as you can see, make waterfall is one of the four sub tasks. We're actually going to talk to the winning team, to the Kairos team, in just a second. This is just the intro. I want to tell you a little bit about what's going on so that later in the interview with the authors you can follow. If you don't know what Minecraft is or the basics of these competitions. If you do, feel free to skip ahead. This is just going to take 5 to 10 minutes. I'm going to show you another one to give you a little bit of the impression of what these agents can do. I haven't actually looked at many of them. I don't know what's going to happen right here, whether that's successful or not. These are the actual videos that the judges saw that were part of these competitions. The competition is human judged. There's no reward function. It's literally, you just give 10 videos to a human and they're supposed to rate how good these things are, how human-like they are, and so on. Ah, it missed the waterfall a little bit right there. Let's see whether I can turn around. Yeah, it can. Not spot on as you can imagine. And not spot on in any of the 10 things. But good enough to win this competition. So how did this team go about this? If you don't know what Minecraft is, Minecraft is this game that looks like it's from 1990 or so. Everything is made of blocks, but it is a really cool game. It's a completely open world game. You can do anything and everything. You can craft items. All of these blocks you can destroy and build up somewhere else. You can collect items and craft new, better items from it. For example, you can craft a pickaxe with which you can mine things, mine stone. From that you can build like an oven, a smelter, and smelt iron ore. From that you can build iron tools and so on. This world is completely procedurally generated. The level is never the same. That's one of the things that makes these challenges so hard. The other thing is the sheer amount of freedom that you have right here. The agent now has spent quite a bit of time looking for a good place to build the waterfall. It looks like it got stuck right here. That's one of the failure cases I imagine. It's going to get out. It's going to get out. What a clinch play there. It looks like here it's a good spot for waterfall. Yes, put it down. Walk away from it. Turn around. Snap picture with the sheep in it. Beautiful. This has actually led to a paper as well by the winning team called Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft along with open source code that you can check out. You can retrain their agent. You can look at their code and you can improve it. It's MIT licensed. Therefore, all good to go for you. What did this team do that gave them the winning submission? The challenge in itself is you're given the tasks in just a short string. There's not a reward function or anything like this. The short string literally is, for example, the find cave. The agent should search for a cave and terminate the episode when it is inside one. That is the entire description of the task. As I said, no reward functions. You do get 40 to 80 playthroughs, 40 to 80 human demonstrations for each task. Not all of them completing the task though. And a bit of a code base. And that's it. This team came up with the following solution. They built at the core, they built what they call a state machine. But I want to start somewhere else. I want to start from how they used the human demonstrations. They had human demonstrations of humans solving this task. And then they trained a navigation policy. This is trained via behavior cloning. You try to make an agent that just kind of clones the human movements. They did cut out all of the interacting with the environment things from the human demonstrations. Such that it was just only navigation going from point A to point B. This is a policy that they can activate at any time. So as you can see right here, this gives rise to one of what they call learned or engineered subtasks. They have a stack of these subtasks. One of them is this navigation subtask that is obviously learned. They have other ones that are just hard coded. For example, when it's time to actually place the waterfall at a point, when you think you're at a good point to build a waterfall, this movement of stacking up the blocks and then putting the waterfall on top, that is a hard coded policy. So these subtasks are hard coded, partially and partially learned, and they're controlled by this state machine. On top of that state machine, which we're going to get to in a minute, the state machine itself is controlled by this state classifier. So the state classifier is a thing that they came up with. They take pictures from the game, frames from the game, and they collect additional human labeled data. Where for each picture, they let the humans label, for example, is this inside a cave? Which you can see right here, that's inside a cave. If you play Minecraft, you know. Is there danger ahead, which means kind of a large body of water that you should avoid or something like this? Do you have animals, which is relevant for some of the tasks? So they build up this state classifier, which is also learned. And that state classifier is now going to control this state machine. I'm not sure if they actually have it somewhere for one of the tasks in the paper. They do have it in the accompanying presentation. The state machine controls what the age or which sub policy is active at any given point. Let's see. It's not here. Well, I can maybe maybe I can I can draw it a little bit. You're going to see in the presentation. So you start and then you, for example, if it's the make waterfall task, you go, you get to a point where you want to ask, is there a good spot to place the waterfall? Is a good spot in sort of the view of the agent? If no, then you go to the explore sub policy. And if yes, then you go to the go there. The go there sub policy is activated. These are these sub policies that we saw are either learned or hard coded. For example, the Explorer one, you can imagine maybe it's just sort of walking around until the state class classifier tells you that there is actually a good spot. So what makes the decision between no and yes, that is exactly this state classifier, this trained state classifier. At some point, it will tell you, ah, now you found a good spot and then you can switch policy. So from there, if after the go there, you get to another decision point and the decision point might be like, are you in front of a big wall? If yes, use the jump policy. If no, use the walk policy or something like this. So as you can see, the state machine itself is hard coded. So the humans came up with what do we need to do to complete the tasks? But the individual steps, they can be either learned or hard coded policies. And that's how they go through fulfilling these tasks. They use the state classifier to always tell them what specific subtask here should be activated at any given point controlled by the state machine. And, you know, with that, they finish the task. One additional thing that they sometimes need is this estimated odometry. This is where they just look at the actions they've performed so far. And they build this overhead map of the agent as the agent walks through the environment. They're able to sort of remember things. For example, this here is has animals. So they're going to remember locations of animals, of bodies of water and so on. And that allows them later if in the later stages, if they need to go back to something, they can efficiently find it again. For example, in the waterfall subtask, they have to go away from the waterfall, turn around to put the waterfall inside of their field of view, and then take a picture or finish the episode. That could be controlled by this overhead map that they build up. It's pretty interesting. All the while, they only have access to the image of the simulator. They do not have access to like the F3 menu or anything like this. All they have is the image. They do have some information on their inventory and their current item, but not much more than that. All right. That was it from me. If you're interested, read this paper. It's a pretty good write up. And also it has a lot of evaluation. They did a lot of human evaluation as well, computing these true skill ranking scores and so on to compare their system and do various ablations. It's really interesting. But now I want to give over to the interview part of this. Let me know how you like these more interviewee style of ways of presenting papers. This one is obviously a very, very applied paper, very visual paper. But yeah, let me know what you think and now enjoy. Hi, everyone. Welcome. Welcome. This is a really, really awesome opportunity right here. I'm joined by the winning team of the Mayan RL Basalt Challenge 2021 by David Watkins, Nick Waitowicz and Vinicius Goeks, who managed to somehow lock their way into winning this competition. No, I'm kidding. I'm kidding. It's really awesome. I've seen the videos of your agent and congratulations, first of all, on winning. And welcome to the channel. Thanks for having us. Yeah, thank you very much for having us. We're excited to talk about the work. So if you could describe in your words the challenge itself, the challenge is about just sort of a bunch of tasks and then humans rate these tasks. What made you decide to take part in this challenge even? How did you find it? Did you just stumble across each other? How did you form your team? What was your interest in this? Well, I can say that we all work together. So it wasn't like we kind of find each other. We've had prior experience working together at the Army Research Lab. And I think Vinicius was actually the one that stumbled upon this challenge. And what we liked about this challenge was that it's different from most other machine learning challenges out there, different from other AI competitions. And the fact that you don't have an objective function to optimize over, right? So it immediately makes it harder. The challenge, again, is in Minecraft with these very free-form, almost lifelike tasks, where really you just have a description, a human readable description of what that task is. There's no reward function, no objective function. So automatically means you can't just apply standard reinforcement learning techniques. And you have to employ some sort of clever measures and potentially learning from humans, which is really what the core of the challenge is about, learning from humans. And that's actually, you know, each of us have machine learning backgrounds. And the research that we do is kind of human guided machine learning. So this challenge is almost like perfect for us. Like, oh, this is a great challenge. We knew it was going to be hard. But yeah, that was kind of the calling for us. And just so far, I will have introduced this, but the challenge was there were four tasks and every task was just given, if I understand correctly, like a very short description of what to do. So, for example, find cave is the agent should search for a cave and terminate the episode when it is inside one. That is all. And all you have as an input, if I understand this correctly, is the screen, right? Not nothing more. Well, you do have the screen and you do have your inventory and the item that you have currently equipped and the screen 64 by 64 RGB. That is a horrible resolution. But you do not have, because in Minecraft for people who play, there's F3, right? You can press it, you see your coordinates, you see sort of your biome and so on. You have none of that. You have to sort of do everything from the screen alone. And you're given 40 to 80 human demonstrations, if I know this correctly, but not all of them successful, right? That was a surprise for us as well when we were using those demonstrations in our agent. And we realized, like, look at this guy. He just walked around and threw the snowball to end the episode. How is that even useful? It was a surprise for us as well. And sometimes you get some items. So one of the challenges, for example, is to, it's called create village animal pen, where it is after spawning in a village, build an animal pen next to one of the houses in a village. Animal pens must contain two of a single kind of animal. You're only allowed to pen chickens, cows, pigs or sheep. Don't harm the village. And in this case, you'd be given also some sort of fence and fence gates in order to build the pen. So it's not like you would have to go collect resources, but the task is still quite challenging. Exactly. Yeah. You don't have to collect any resource or build anything. You were given everything on your inventory, but like completing all those tasks was already a huge challenge. Yeah. And especially given that, again, to remind people, the reward here is not some function you can compute. The reward is at the end, it's given to human raters. The human reads the description and then the human decides how well did your agent perform it. And most striking, I find this in a third task that is build waterfall, where the goal is that you have to, I can maybe read the description, after spawning in a mountainous area, the agent should build a beautiful waterfall. That's part of the description, a beautiful waterfall, and then reposition itself to take a scenic picture of the same waterfall. The picture of the waterfall can be taken by orienting the camera and then throwing a snowball when facing the waterfall at a good angle. So there is even an essence of sort of subjectivity, judgment, beauty, and so on in it. So that is the challenging part, I think, here. You saw this, you thought, I want to do this challenge, we want to do this challenge. What was your first try? What was the first thing you threw at the problem? Well, I can speak a little bit about it. At least me, myself, when I read the challenge, I had no idea how to approach it. Because I was thinking, okay, we have a few demonstrations, but from my experience researching everything, I thought if we just do a behavior cloning using this data, it won't cut it, we don't have enough data. And then it took us like a month to solidify an approach. We talked about behavior cloning, we talked about GAO, we thought about, okay, let's hard call this whole thing. We definitely thought about different approaches, and then I guess in the end it was a mix of everything. And that's what you make clear. So there is a paper about, you wrote a paper about your approach as well, and the paper's title is Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks. And then you have Minecraft pointing out that the best approach will be one where learned elements are mixed with hand engineered elements. So my question is, how did you come about this? Was this an iterative process? Or you said you scrambled with a bunch of things at the beginning. Did you add and add and add? What was your process? What was the first thing that maybe you realized, ah, this works now a little, right? And then how did you build up your end solution? Well, so I can add a little bit to that. So, you know, we were motivated, like the nice thing about the competitions, we were motivated to try to do well. And so we knew from the beginning that we didn't want, we wanted to take a different approach. Probably a lot of people would just try to apply end to end machine learning, you know, throw a lot of compute at it. And, you know, we kind of realized that really if we want a solution that is a little less just academic and more that works for this particular application, we're going to need to really use everything, right? Including, you know, try to inject our own domain bias about the problem into the framework, into the solution. So that really led us to these, you know, OK, well, we could have a hierarchy of different modules. Some of those are hand engineered. Some of those are learned, you know, the things that we can't engineer. And then we can have, like, you know, a state machine where we know the agent should be doing this. So, you know, let's not have the, you know, RL or machine learning component learn the things that we already know how to do from scratch, right? And just make this job harder, right? Let's add that information to the agent and let's, you know, save the learning for the things that we can't easily do, right? And then have them work together. Yeah, I think you make this clear and I'm just going to share a screen for a bit right here. You make this clear in sort of this diagram, which is an overview over your system. And at the core here is this state machine. You want to maybe talk a little bit about why a state machine might make sense right here. For example, this here is the state machine for the waterfall task. I can talk a little bit about it. So if you saw like those tasks, so, for example, let's talk about the beautiful waterfall task since we have the diagram open. There's really like a hierarchy of subtasks that needs to be complete in order, you know, to finish this whole task. For example, for the make waterfall, right? First you need to find a good spot to build your waterfall, right? And that means you need to climb up somewhere. You need to be like at the edge of a cliff, right? And then you have to actually build the waterfall, you know, you got to equip your water bucket and, you know, point it down, throw the water bucket, right? And then hopefully this waterfall will be beautiful, right? Assuming you got like a good spot. Then you have to go really far away from this waterfall and then position your camera just right to get like the best, you know, the best view of this waterfall and throw a snowball to finish it, right? So there's this whole hierarchy of tasks. It needs to be completed like one step at a time and there's like this logical order. So the state machine was our approach to make sure that the agent would actually follow this order, you know, without coming back and forth. Like if you do like, for example, some just an end-to-end machine learning approach, the agent might, you know, let's say go find a spot and then we'll go back, take a picture, you know, come back again, try to build, equip the water bucket to build the waterfall. So the state machine was our solution to make sure the agent would follow kind of this logic for each task. And I think you profit from the fact that all of these tasks can be sort of described quite well in this state machine fashion, as I think a lot of, you know, if you play Minecraft as a human, that's sort of the same thing you do, right? You if you want to beat the ender dragon, you okay, first I need to do this, then this, then this. And it's quite the same thing with a few decision nodes in between. And these decision nodes here in the in the green, those are now decided by classifier, if I understand this correctly. So you build this this little interface here where humans could rate, you were allowed in the competition to collect a little bit like a limited amount of different human feedback. And you chose among other things, you chose to have humans label different images from the game with such a with them with such maybe you can describe it a little bit. What were you interested in? And why did you choose to put the additional human labeling into this task and not any other task? What like, why did you prefer this? Something important to keep in mind is that you're allowed to include 30 megabytes of additional data in this competition. And the Minecraft simulator is such that if you were to record a bunch of actions or steps that the player took and try to replay them, it's not currently designed to handle RNG the same way every time. So if I go break a block, that block is going to fly differently depending on the state, the internal state of the random number generator. And we have no control over that. So you can't seed it necessarily. We can't seeding it just doesn't work. So we couldn't just collect more demonstration data other than videos. And that would eat into 30 megabytes very quickly, as I'm sure you could imagine. So dividing up each of the tasks into a bunch of shared states made the most sense to us. It's something we've used in previous research to handle navigation tasks before. And it works reliably and I think there's a lot of research in making state classifiers work really well. So it was more just us as a team, you know, while we're watching TV, labeling a bunch of Minecraft screens. The most difficult part, of course, though, is it's 64 by 64. And there are many situations where maybe you want to recognize that there's an animal in the frame and it's a chicken and it's the small white blob. But it could be confused with a flower and you're kind of fighting yourself to make sure that this actually works. And so there were some different strategies we were looking to employ to make sure that the state was classified correctly. But it worked pretty well. Cool. And I think people can see here maybe at this graphic, but you have such things like, for example, good waterfall view, which makes sense, right? This is a subjective thing of the reward function. So it makes total sense to include that in the human annotated data and not code or heuristic. But you also have things like a danger ahead, which you then use. So I think once you know which node you're in, right, in this state machine, very often the blue blocks right here, which are the actions, the blue blocks involve going somewhere. For example, if has mountain, then, you know, if you don't have a mountain, find the mountain. If you do have a mountain, go to the mountain. And that part means that your Minecraft agent has to go from point A to point B. And that's where you build a specialized navigation, navigation subroutine. And you said right now you've already done this in the past. Can you tell maybe a little bit in general, what does it take to make agents navigate around? So can I just mention one more thing about the state classifier? Sure. So with the state classifier, like David and Venetia were saying, it's really the core of the state machine, right? So we knew we wanted, you know, it's the thing that makes the drives our entire solution. So it has to be, you know, more or less somewhat accurate. And we needed a lot of data. So we actually collected around, I think, eighty eight thousand labels, which sounds like a lot. But of course, you know, that type of manual annotating, no one really wants to do. You know, as machine learning scientists, we'd rather spend that time trying to, you know, code up a solution to do that instead of doing it ourselves. But what we did, we tried to make it as easy as possible by, you know, we're not HCI experts, but, you know, we tried to come up with a kind of intuitive labeling interface to make it as quick as possible to kind of, you know, like one demonstration that's three minutes long at a, you know, a FPS of 20 frames per second. You know, that's a lot of images. And we try to take advantage of the fact that the images are somewhat correlated to time. Right. So the way we designed our labeling interface is kind of just a step through each image through the trajectory. And if you hold down a button, let's say one of the buttons is, you know, there's there's nothing ahead. It's just open fields. So you can just hold down that button and it's going to traverse, you know, through the demonstration until something else comes up and then you can just move a different button. So very quickly, you know, you can, you know, label 5000 images in one trajectory in like less than a minute because you're just holding down these buttons instead of like, you know, showing an individual image and then selecting the label and then the next image and select the label. I think that really allowed us to get it sacrifices a little bit of accuracy. Maybe when you're transitioning, you might miss, you know, get a few misclassifications, but you're able to get a lot more more labeled images. I think this is a recurring theme sort of in real world tasks, the efficiency of data labeling when you include humans. I've just recently watched sort of Elon Musk's appearance on Lex Friedman. And before that, I've commented on Karpati's talk about the autopilot there. It's a thing that you see again and again that the easier you make it for humans to annotate data, the more benefit you have later. Like it's almost an unfair multiplier that you have on your system. I think it's neglected currently by academia. So it's pretty cool that you thought about this as well. Yeah, I think it is neglected because it is not easy and takes a lot of time. Like manual labor, nobody wants to do manual labor, but definitely having like high quality labeled data labeled by humans makes totally the difference. So and now we'll let's let's go to the to the navigation subroutine. How do you how do you navigate? Wait, that is here. So you have a navigation policy which essentially says the agent needs to go from A to B and what does it take to build that? Like it seems very complicated in a game so complicated as Minecraft. So well, so the behavioral cloning part, right? So that part is, you know, unfortunately, just very simple. It's not any secret sauce or anything complicated. You know, we again, just prefacing by this, you know, was a competition and we had a deadline. We had so much more that we wanted to do with this particular part, right? For the solar navigation part, we wanted to do something, you know, way more than just standard behavioral cloning. You know, things like generative adversarial imitation learning, you know, trying to have better architectures. In the end, we didn't have enough time. We were scrambling and for this component, we just did behavioral cloning. The way that we did that is, you know, as you can see in this model, it's like, OK, the agent only has the image as input and its output, you know, are more or less just the direction key. So it can go forward, it can turn left, it can turn right, it can strafe left, strafe right, and then it can move its camera. And really the way that we did that is we just we had all these demonstrations for each of these tasks. We kind of the only kind of trick that we applied was that we realized this is just a navigation component. So we only want to learn to imitate the part of the demonstrations that we're navigating. Right. So let's just chop off that demonstration just to that navigation part and then feed that into our navigation policy. And so that's that's basically what we did was, you know, any any time where the agent was building, like building the pen or the village or the waterfall, we cut those segments out. The remaining segments are where the agent is just trying to go from one point to the next. We kept those in and use that as our training data for the behavioral cloning module. And in this in this model here, it says image input. Do you also give the model access to, let's say, the the results of your state classifier and maybe the current state machine state or something like this? So the agent knows where to go or do you rely on behavior cloning for the entirety of navigation? Yeah, that's a really good point. So again, it's our this particular navigation policy is just terribly simple. It's really just the the image input being driven by the state classifier in the sense that it allow, you know, the state classifier decides when to start and stop the navigation policy. But we're not feeding in any information directly from the state classifier or other other more interesting information that that certainly would help. If we had more time, we could probably do that. It would make sense to do that. But right now, the state classifier just decides when to start that navigation policy and when to terminate the. I think so. No, I just just want to add a little bit on top of that. The main reason we didn't add anything else on this is because we didn't have. So like the so this navigation sub task policy was trained from the demonstrations provided by the competition. So that data didn't have any like state machine. So the state machine was everything on our side. So we really only had access to the actions that the agent took right and the camera data. And and again, like I think the using that demonstration data provided by the competition to train only the navigation sub task made sense because let's say think about it. Let's say we want to do end to end behavior cloning, right? And then you were doing the fine cave task and the fine cave task. At some point, the human will throw a snowball when the agent is inside the cave. And that's only one data sample. And the whole episode has about two to three thousand. So you have one sample to throw in the snowball on over three thousand samples. And to find the cave, it took a lot of steps and this is all really useful for navigation. So we did this like Nick said, this preprocess to remove all those actions, leave only the navigation part and use that to train this navigation sub task. And I think that was pretty helpful to in our approach. So is it fair to say that, for example, you're here and you you are your house mountain classifier says yes, then the state machine would simply activate the navigation. Does it? But it doesn't it doesn't necessarily tell it where to go. You just rely on the fact that your demonstration in your demonstration, people have generally gone towards the mountain and therefore the navigation policy would have learned that implicitly. Exactly. Let me I guess let me explain this diagram a little bit. So what you said is correct. So the green diamonds are decision notes, right? And that's that's based on the output of the state classifier. Right. So like has my mountains, you know, if it's over, let's say 90 percent confidence, we'll take that as a yes. Right. And then we go to those blue rectangles and each blue rectangle is a sub task and those sub tasks can be either learned or coded or like hard coded. So, for example, go to go or find go actually find go was learned from the human demonstration. So we would not say like something like, oh, go to this coordinate like we didn't have. Right. We would just use the human the policy that was trained from human demonstrations to navigate, let's say, going up the mountain. Right. And then let's say on that part of the diagram where you have the dashed line, you know, there's a green diamond there written at the top. So let's say if the state classifier detect that we're on top of the mountain, right, then we would switch to this place waterfall sub task and this place waterfall sub task was hard coded. So that was not learned from the human demonstrations. And what the sub task does is basically point your camera down, keep the water bucket and throw it. You know, that's kind of placing the waterfall. So those blows are our mix of learned sub tasks and hard coded. Yeah. What my question is a little bit. You have, for example, this danger ahead state. Right. But you don't feed any state to the navigation policy. Where is the danger ahead used inside the state classifier somewhere? Like you say, if there's danger ahead, then we don't even want to activate navigation. Exactly. So that's something that it's like a safe critical sub task that takes priority over everything. So it doesn't matter if you're looking at the mounting, whatever you need to do. If there's danger ahead, just avoid it. Right. So it's like a sort of a safe override that's always on no matter which sub task we're doing, if you're following the human or not. Because, you know, just avoid danger because our first iterations of Asian and even the final one is still there sometimes. When you fall on one of those lakes, you just can't escape. It's just too hard. Like sometimes there are like two blocks tall, then it's hard to like teach the Asian to break the blocks and jump. Like do all those things that us humans do pretty well for the Asian is pretty hard. So our Asian got stuck a bunch of times. Then we had to add like some safety sub tasks to help a little bit the Asian to escape those things. And at some point you also built in this odometry estimation because you only had the image and you thought it would be... Maybe you can explain this. What led you... Because it's not a straightforward thing to include, right? If I think about how would I solve this task, what is the odometry estimation? What is it for? And why did you include it? I can talk about it. So like you mentioned at the beginning of the video, we could not... Like in Minecraft we do know where the Asian is. Like when you're playing the game, you can press F3, you can see everything, right? But in the competition we were not allowed to use that. So we had some ideas, okay let's use the simulator, but we were not allowed to do that. But we're thinking like what do we know about this problem? So we do have access to the actions that the Asian took. And we do have access to the image. Not only that, we know a little bit of Minecraft. So we know that the simulator runs at 20 frames per second. So each frame is 1 over 20, 0.05 seconds. So we know this time interval between each frame, right? And from Minecraft we know that for example the walking distance is actually I think 4.32 meters per second. So we had this information from the wiki. So let's say if the Asian sent the command to move forward, right? And not considering inertia or anything, right? We could assume that in one frame the Asian walked 4.32 times 0.05. So like this velocity times this dt, this time interval. So we know how much the Asian walked in the X direction, right? And then we had the actions, we had access to the actions for the camera control. So we could estimate the heading. So just based on the actions that the Asian took and knowledge of the simulator, right? We were able to sort of estimate velocity X, Y and heading. And then you integrate that over time because you know your time interval. So you can come up with estimates of X, Y and heading for the agent. And that's what you see on this kind of this black diagram on the right, which I can explain everything in more details too. So I mean you build this sort of map almost. Like this is an overhead map of the agent in its environment. And annotated with first of all what you've done so far, right? Your position that's been going on. Maybe if this here loads, this here is different trajectories. But you also annotate this map with various things that you find. Like whenever your state classifier says something. Where is this information used? I guess it's you said it's not in the navigation because that it doesn't get any additional features. Where is the information that you estimate from this overhead map? Where is it used? The best example for this is to make waterfall task. So when the agent places a waterfall, you know something we're thinking is maybe we'll try the behavioral cloning, but often, you know, the behavioral cloning doesn't really stay still very often because it really learned the navigation sub policy. So instead we sort of use that heading estimation to move the agent away a fixed amount and then rotate around to look at it. So there are just certain tasks that it's really important that whatever the final view is, the line with some landmark in the environment that we don't have a ground truth information for. Yeah, so it's really the odometry is mainly used in various places in the state classifier. I mean, the state machine in some of the subtasks like David was saying. Another example is the animal pen, right? The challenging part of that task is you really have to build. You first got to find an open location, then build the pen. And then you have to leave that pen and go find the animal somewhere, right? They could be anywhere. And then lure them back to the pen. So you have to remember where you built that pen. And so that's, you know, the odometry comes into play for that place. So we were using the state classifier to kind of classify. OK, here's an open location. Now we switch to pen building mode. OK, the pen is built. Let's go find some animals. We remember the location of that pen, you know, based on our estimated odometry. And then once we find some animals, then we try to go back to that location. And just to say that the try to go back will be a hard coded policy that takes as an input the remembered location of the pen and your guess of where you are in relation to that pen. Exactly. Yeah. So at that stage you have a XY coordinate of the pen and you have an XY and headings estimates of your position, right? So you can basically compute the angle between where you're looking and where the pen is. You can compute this angle, right? And the policy was literally kind of close this angle and then keep moving to kind of reduce this distance over time and go back to that location. So it's a simple policy. There are a few limitations though on the odometry side, which I just want to comment just to don't say this was like a god-tier approach for that. So, for example, since we only use the actions, right? If you think about it, the odometry is just seeing the actions, right? And then, OK, the agent is moving forward. So we're seeing this moving forward action, right? So we're integrating that over time, increasing the distance and everything, right? But what if the agent gets stuck, like behind the rock, behind the tree, and it is still moving forward? Like in Minecraft you can still kind of walk forward sort of sliding, right? But you're still stuck in place. But the odometry does not know that. We had some ideas to integrate differently in the pixels, right? Using this camera data to know when the agent is stuck. So we ignore that. But we didn't have time to do that at the end. But this approach, our current approach, still works for short distance, right? So, of course, the longer you walk, you know, like the drift will be just higher on this estimation. But for short distances, it actually works pretty well. And I guess it's sorry. I was going to say that a slam approach in a 64 by 64 image that's only RGB is incredibly challenging and probably not the right approach for this particular challenge. And it might also be fair to say that you said you had a lot of ideas. I guess if you were to go further, you'd probably let's say, try to come up with a navigation policy that's both learned but also controllable in some way. Try to come up with an odometry estimation that takes into account the picture, which could recognize when you're stuck and so on. I think there's there's a lot of stuff to improve. But I'm very impressed by sort of your your pragmatism of okay, this works well enough. Let's go on. Was there was there moments when I guess there's moments in every project when when you're or what was the moment when you most thought, ah, this is not going to work. Let's give up. Like, did you have a moment like this? And and what did you do? You guys want to comment on that? Well, there's there were, I guess, a lot of those moments. We, if you go back to the main overall diagram, we definitely like had, you know, went back and forth on, you know, what should the solution be? You know, we were still toying around at some points with with, you know, a more, you know, end to end approach in some places and whether we should put our eggs in that basket or whether we should do this current approach. Ultimately, you know, this is the one that we landed on. And we we designed this. The next thing about this approach is it's it's hierarchical, but it's very modular. Right. And the idea is that each of these sub tasks, you know, their individual models that we can improve upon or replace. And so, like, you know, if we had more time, some of the things that we would do is start to try to replace some of these hand engineered sub tasks with more learning based sub tasks and or, you know, replace the navigation module with a more advanced learning module that uses more information. One of the things we spent a lot of time on that never made into or was was kind of using generative adversarial limitation learning as our core algorithm for learning the navigation module. And, you know, with Gale, it's it's basically using a GAN. And as we found out, like everybody knows, GANs are notoriously difficult to stabilize, including GANs for Minecraft. And it ultimately didn't end up making it. So we had to revert back. So that was one of our centers. We're like, oh, this is this is definitely not going to work. You know, we spent a ton of time doing that and we had to kind of, you know, replace with our with our backup, which is just, you know, standard behavior. So go ahead. Also, the one point my brothers are very good at Minecraft and the Minecraft speed running community is a pretty big thing. So at one point we were considering, why don't we just get somebody to play Minecraft really well? But that stupid Minecraft simulator limitation and also, you know, it's it's one thing to get a bunch of people to play the game better than maybe the demonstrators were playing. But that also means that, you know, that data won't necessarily be very rich because they can't play the game well and label the data at the same time. And I think it comes back to this problem of labeling data really conveniently is difficult, especially when you're driving the agent simultaneously. So it becomes a very difficult challenge to use human data when the the amount of data you can actually collect is small. And this being Minecraft, I think I like I'm fascinated by this because I wonder how much world knowledge is in inside a human when they play Minecraft and how much is sort of learned because the world is different, like literally different every time. And I can learn Minecraft by just watching someone do it a few times, right? I can perfectly not perfectly, but I can well generalize to other worlds. Is that because I've watched someone I've done it a bunch of times or is that because I know from my life what sand is and what water is and how it behaves? And I think that I don't know. Yeah, I think I guess the main advantage of like, you know, humans is that, you know, we've lived, you know, 20, 30, 70 years already right in the real world. And then Minecraft tries to mimic that. So we humans have a huge kind of baggage that we can use. But we have to always remember like those Asians, they start from scratch. They literally start from nothing. Right. We had to collect data to teach what danger was for those agents like, like to teach, oh, don't jump in the water, you know, don't don't drown there, you know, things like that. So that's very challenging as well. And I have I have your sort of so for videos that you uploaded, and they have side by side the agent view, the classifier, but also the odometry estimation. Do you want to maybe so this is for example, do you have one that is your favorite of these four? Yeah, probably the waterfall I think will look pretty nice. So this is build build house was pretty challenging. This is 30 seconds I'm gonna I'm gonna slow it down to like point to five right here. Do you maybe. Oh, yeah, I can. Oh yeah, I can like comment like a comment a little bit on what's happening right here. So which state is it in what's happening. Yeah, so so this is a video of the agent solving the make waterfall task right and then you mainly see in the screen in the screen two panels. So on the left side, that's the RGB. So this is like a camera view of the agent right and on the right side, this black panel is the estimated odometry. So if we start there on top left, you see like action and then use tensor right. So that's the I think 12 or 13 actions that the agent was performing. So they're mostly binaries. So like move forward or not move back or not, you know, things like that. And below that you see the raw output of the state classifier. So we had 12 classes or I guess 13 with, you know, the known class. And you see like the confidence of the classifier, you know, for classifying the state like this camera, this camera image. So you see like right now, you know, facing wall is pretty much almost 100 percent. I think it is from all the stone that the agent is seeing. So it thinks it is a wall. Right. And on the right side, the odometry. So we can start there on the on the top part there. You see a X, a Y and a heading. So X, Y. So that's the estimated position of the agent. So that's not the ground truth. So again, we didn't have the ground truth. Same with the heading. So that's estimated and that camera angle there is like a vertical angle. Right. And then on the right side, you have like some time. So we kind of just have a track, keep track of time. And then you have a legend. So the legend there is for all the colors you see in the odometry. So the red one, the red dot is the agent. So right now it is down at the bottom of the screen. Whenever the way the agent walks around, it leaves like this trace. So that's the Y dashed line that you see on the screen. And then like right now you see, for example, it just saw that cyan, I think, blob at the bottom there. That's when the state classifier detect that we were on the top of the waterfall. So you see that that's the last thing on the legend there. So basically, yeah, the agent walks around and some of the relevant states that we classify, we sort of drop a pin in the map kind of just to keep track of it. In the video, the first like 25 seconds or so, what you know, this is the map. You know, it starts off basically with the navigation policy, right? The go to goal. So the behavioral cloning module that we trained is in control and it's driving. And it's basically, you know, trying to mimic all of the other human demonstrators that did this task, you know, which is more or less kind of walk around and look for a good spot. And then when the state classifier detects like, OK, this is a decent spot, that's when you saw it switch to the, all right, let's build the waterfall. And then after build the waterfall, the state classifier switch to the now go take a picture sub task. And so that's basically what you see in this video. And one thing I'll say with this, the interesting thing with the navigation policy is, you know, this is something we kind of noticed and it's just a theory. We don't have any proof on it. But like, you know, the, you know, the agent jumps around a lot. But we think that's because the agent is mimicking the human demonstrators. So like, so jumping for the sake of jumping, not necessarily to jump over stuff like, you know, there's some players. You're faster if you jump. Yeah, yeah, exactly. And that's seen in the demonstrations or some players like me, I just jump idly, you know, just a fixation. So I'm just like randomly jumping that not to particularly jump over anything. You kind of see that in the agents behavior. So it's almost, you know, makes it more human like, at least in our opinion, versus, you know, a hard coded navigation policy, which mainly, you know, you might expect it to just walk without jumping unless it needs to jump right over something here. You know, the agent is kind of just more pseudo randomly jumping like a human would. And we thought that was pretty cool because, you know, another part of this competition that we haven't talked about yet, it's not just, you know, developing agents that can do the task the best, but also there was a sub thread to the competition of who can build the most human like agent, which we also won that prize. So, you know, this potentially, I mean, really our whole system, you know, is sort of aimed at the human like because we added a lot of human knowledge to it. But like the behavioral cloning part, you know, that might also add to that because it kind of moves around more or less like it, like a human would move around. And it looks a little less robotic, like if it were kind of a more hand engineered. Except like here when it's like a good spot for a waterfall, you immediately point down and start like, I guess this is the hard coded part, like you see right now, immediately point down, build a bunch of blocks, place the bucket. And then it's interesting. So this part here is hard coded as well. It's just like move the agent away. And we see the agent kind of slide on the left a little bit because I've noticed that later when it turns around, it sort of almost misses a little bit the angle. Right. So this could be this drift that you have in the odometry estimation. So it's trying to make a picture of the waterfall directly, misses like a little bit. So I guess that would be that would sort of be the problems that you get in just having the just having the estimation from the action, which you mentioned. Yeah. So for example, when you throw the water down, right, sometimes the agent will float in the water and that will turn the agent a little bit left and right. But the odometry doesn't see that because the agent didn't command the camera movement. So it doesn't update your headings. So that can also cause problems later. But yeah, like you said, that part was hard coded like the place waterfall subtask was hard coded. But all the way up to that thing, up to that part was learned from human demonstrations, which is the navigation subtask. What I think what you what you need to do is you just need to train the navigation thing on, you know, dream. So you just want to you just want to train it on like a bunch of videos of dream and then just see what happens. I would be so curious to see what happens. Well, that's what we wanted to do that initially is we thought, oh, look at all of this awesome data on YouTube that we could maybe try to learn from. But there's no actions associated with it. Yes. OK, true. You'd sort of have to estimate the actions almost a little bit. And you'd also have to like there's a lot of things you'd have to guess at what's actually going on, which where do we crop the video? Right. There's all this stuff they have overlaid and it becomes more challenging to use YouTube data. But I see. OK, you you wait. What was I was I gonna? One thing that yeah, one thing that I was a little bit like a tiny bit dissatisfied with with this competition. Obviously, it's already super duper challenging. Right. And Minecraft is so much more complicated than this thing. But there were these four tasks and you knew them ahead of time. Right. That's why you were able to sort of build the state machine. The descriptions were very clear ahead of time. Let's say that I come and I'm the organizer and I change the challenge for next year and next year. It's still the same thing. It's human rated. It's described in just like a simple string, but I won't tell you what the string is. Right. I won't tell you ahead ahead of time. How would you how would you go about designing a system like this? Like what would you would you do? Would you try to go the same route? Or let's say you also had very limited resources like you had now. You can't train like a giant or else system. I think we would definitely be forced to go a different route, which I think would be good. You know, one of the things I like about this competition again is that it's you know, I think it's important for the field because you know, it's these tasks again that you can't just, you know, do this black box optimization over because there's no objective function. So you're forced to really try to learn from a human. Right. Or do something. Right. And and and you know, we really took that to heart. We knew like, OK, in order to do wellness competition, we cannot just use the human provided demonstrations like the majority of the other teams. We had to add our own additional human input and feedback. And we did that with the design of our state machine and in the labeling, the human exhaustive human labeling that we added. But, you know, to take it a step further, really, I think the interesting thing would be to have a system where you have you learn from real time human feedback, which our system didn't do. Because, you know, well, one is that's more challenging and we didn't have time. And because all the the tasks are known ahead of time, you don't have to have real time human feedback. You can, you know, collect your human feedback or human labeling beforehand and then use it. But if you have now a new iteration of this competition where you do not know the the tasks ahead of time, then you now might need a system where your agent needs to learn from human feedback in real time and kind of interact with the human to kind of get that learning. Because, you know, you're just seeing what you need to do the task at competition time. So I think that would be really interesting. And that would force more solutions to use something that that uses real time human feedback. What set you apart? If you you've probably seen sort of the other teams that competed and so on, and I'm sure they were also they were also engaged and motivated and tried a bunch of things. What do you think was sort of the or maybe the most defining factor that let you win? Was it I'm sure there was a level of stochasticity in the evaluation, but you know, you won, I think not one but two of the three subcategories even. So it must mean that you had a considerable, let's say edge over most of the competition. What in your estimation was that? I have a guess you guys can comment on that. I think in my opinion, I think our edge was actually using human feedback data. So like the other teams, if I remember correctly, I think number two used sort of improved algorithm that would improve on Gale. So that was kind of sort of full RL approach. The third team tried to use some of kind of learning from human preference, if you remember that paper, but they didn't use a human to rate the trajectories. They used like heuristic, right? And we were the only team that actually use human data. So we, you know, we label a bunch of data, you know, we added kind of our knowledge, our bias on the task and everything. So I think really using the human, I think was the key factor that allowed us to win two or three of the awards. 100%. Like, you know, yeah, we had a state machine approach with, you know, these modular hierarchical design, but really we wouldn't have been able to do that if we didn't have, you know, this classifier that was generated with additional, you know, human feedback and human labeling. And so it's really the thing that Sturzauper and like we said, it was, you know, the other teams, they just use the human demonstrations and even the third place team, they used a simulated human, right? Instead of, you know, doing the hard work of actually getting that human feedback, they just defined this simple heuristic. And I think that right there is like, you know, the important thing, like the field, you know, sometimes can just like, oh, well, let's just, it's easier to kind of simulate out the human. Let's, you know, come up with a better algorithm, but it really just shows like we should do a better job trying to incorporate human feedback because it's definitely, you know, valuable information and can really improve the way we develop our AI algorithms. I think it's important as well to, you know, when you look at Minecraft, it's very much feels like an open world sandbox problem, very similar to using a robot in the real world. And collecting real world data is about as difficult as I would say, well, it's a little more challenging in some ways, but challenging to collect lots of good rich human demonstrations in this particular environment. And so, if we were looking at this as a generalized approach to solving this kind of navigation problem, I think we would have used a similar approach for handling this on a robot, where, you know, robot going to go pick something up somewhere can be broken down into a bunch of discrete steps, and we solve each of those steps really well. Whereas an end to end approach, we risk having situations where the neural network is doing something that we can't debug at all. And I think that hierarchical approach really let us debug each step really well, as opposed to the monolithic approach. Now, just to say in on the on the leaderboard website, there is a team that has like a better score than you was that is that an artifact of the one leaderboard or is it late entry after the competition. So that's the that's the public leaderboard, right. And it's not officially award. This is, yeah, this highlights the other difficulty of this competition is like, again, there's nothing to just automatically grade everything that you have to just get volunteers. To literally just sit down and look at pairs of videos of different agents and see which one is better. Very, very arduous task, right. And the public leaderboard is just any random person with a web browser can go on and start rating all the people you know we we provided some ratings. It's completely unofficial, but it was just used to kind of determine who would go to the next round. So the top 10 teams and then the competition organizers actually hired professional contractors, you know, you know, but actually had, you know, not just random people, but like contractors go and do official valuations to determine the winners. And on that one, that's that's where we won first place. But on the public leaderboard, we're not showing us first place because of the stochasticity of all the human raiders. I love that the professional contractors were probably like they had to know Minecraft, right. So they're like the most competent people in it were probably like some 13 year olds. Kids to watch some videos, give some ratings. Excellent. Yeah, is there is there anything you you'd like to that was my exhaustive list of of questions that I had about this. Is there anything you feel is important to to add for people to know if they want to do something like this themselves or I think I think during the presentation we had this slide about that. So so this competition might happen again next year or I guess this year already 2022. So if you're really interested on that, make sure to go ahead and start playing with the mine RL package now because it took us a long time to to figure that out. I think I think I can speak for all all three here. I think that was our first time working with the Minecraft package like the reinforcement learning package. So it took us some time to to learn all the you know how to work with that their action space observation space and everything. So if you want to like an extra edge this next year you can maybe start playing with the package now. And I think I think that's it. Maybe play a lot of Minecraft. I think that that helped. Yeah, I mean you mentioned the paper that we have but we also made our code available for anybody that wants to try it themselves or improve upon our solution. Awesome. I think the paper got the link to the code. Yeah, I'm pretty sure. Yeah, it's there. So yeah, go ahead to play with our code. Maybe make it better. Let us know. Maybe make some pull requests. Cool. Awesome. Well, in this case, thank you so much for being here and sharing this. It's really I love I like it. I think it's really cool when when things like this get get out into the well not real world but Minecraft world which is close enough. It's incredibly hard task and for just from the videos I saw it I was surprised by you know just how far you can get with how little sort of resources and data. And just one last thing like the definitely, you know, for this first year's competition, the, you know, this is far from solved, and I think the competition organizers realize that too. So out of the four tasks which are, you know that you already mentioned, you know, basically advancing the fine cave in the make waterfall the easiest. Those are pretty much solved. The create animal pen and especially the build the village. None of those solutions came even close to really solving that, you know, I'm sure the human raiders are just looking at to really junk agents doing random stuff and trying to pick which one's better. Right. But, you know, it's still like on that build village tasks but still very simple tasks out of the range of tasks that you can conceive in Minecraft is still far from from salt. And, I mean, yeah, there's, there's no crafting yet there is no fighting there is no exploring. And this isn't even like this, this is where Minecraft starts the actual game of Minecraft is where you sort of set your own goals right and you try to achieve something new. Yeah, it's, it's cool to see that there's still a lot of a lot of stuff to do. Awesome. Thank you so much for being here. And, yeah, I hope to see you next year again. Thank you very much for having us Yannick. Like I said, I watched a bunch of your videos I really like your channel I'm excited to see. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. Let me know if you like this video, leave a like if you did and leave a comment if you have comments, suggestions, anything at all. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time. Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 If we just do a behavior cloning using this data, it won't cut it.
 Like, we don't have enough data.
 Hello there! Today we're going to look at this right here.
 This is an agent in Minecraft that's trying to build a waterfall.
 So the goal is to go up a mountain, find a good spot, put down some water,
 turn around and then take a beautiful picture of the waterfall.
 That is one of the four tasks of the Mine RL Basalt Competition.
 This is what we're going to talk about today.
 And not only are we going to talk about the challenge, the competition,
 as you can see, make waterfall is one of the four sub tasks.
 We're actually going to talk to the winning team, to the Kairos team, in just a second.
 This is just the intro. I want to tell you a little bit about what's going on
 so that later in the interview with the authors you can follow.
 If you don't know what Minecraft is or the basics of these competitions.
 If you do, feel free to skip ahead. This is just going to take 5 to 10 minutes.
 I'm going to show you another one to give you a little bit of the impression
 of what these agents can do. I haven't actually looked at many of them.
 I don't know what's going to happen right here, whether that's successful or not.
 These are the actual videos that the judges saw that were part of these competitions.
 The competition is human judged. There's no reward function.
 It's literally, you just give 10 videos to a human and they're supposed to rate
 how good these things are, how human-like they are, and so on.
 Ah, it missed the waterfall a little bit right there. Let's see whether I can turn around.
 Yeah, it can. Not spot on as you can imagine.
 And not spot on in any of the 10 things. But good enough to win this competition.
 So how did this team go about this? If you don't know what Minecraft is,
 Minecraft is this game that looks like it's from 1990 or so.
 Everything is made of blocks, but it is a really cool game.
 It's a completely open world game. You can do anything and everything.
 You can craft items. All of these blocks you can destroy and build up somewhere else.
 You can collect items and craft new, better items from it.
 For example, you can craft a pickaxe with which you can mine things, mine stone.
 From that you can build like an oven, a smelter, and smelt iron ore.
 From that you can build iron tools and so on.
 This world is completely procedurally generated.
 The level is never the same. That's one of the things that makes these challenges so hard.
 The other thing is the sheer amount of freedom that you have right here.
 The agent now has spent quite a bit of time looking for a good place to build the waterfall.
 It looks like it got stuck right here. That's one of the failure cases I imagine.
 It's going to get out.
 It's going to get out. What a clinch play there.
 It looks like here it's a good spot for waterfall. Yes, put it down.
 Walk away from it. Turn around. Snap picture with the sheep in it. Beautiful.
 This has actually led to a paper as well by the winning team called
 Combining Learning from Human Feedback and Knowledge Engineering to Solve
 Hierarchical Tasks in Minecraft along with open source code that you can check out.
 You can retrain their agent. You can look at their code and you can improve it.
 It's MIT licensed. Therefore, all good to go for you.
 What did this team do that gave them the winning submission?
 The challenge in itself is you're given the tasks in just a short string.
 There's not a reward function or anything like this.
 The short string literally is, for example, the find cave.
 The agent should search for a cave and terminate the episode when it is inside one.
 That is the entire description of the task. As I said, no reward functions.
 You do get 40 to 80 playthroughs, 40 to 80 human demonstrations for each task.
 Not all of them completing the task though. And a bit of a code base.
 And that's it. This team came up with the following solution.
 They built at the core, they built what they call a state machine.
 But I want to start somewhere else. I want to start from how they used the human demonstrations.
 They had human demonstrations of humans solving this task.
 And then they trained a navigation policy. This is trained via behavior cloning.
 You try to make an agent that just kind of clones the human movements.
 They did cut out all of the interacting with the environment things from the human demonstrations.
 Such that it was just only navigation going from point A to point B.
 This is a policy that they can activate at any time.
 So as you can see right here, this gives rise to one of what they call learned or engineered subtasks.
 They have a stack of these subtasks. One of them is this navigation subtask that is obviously learned.
 They have other ones that are just hard coded.
 For example, when it's time to actually place the waterfall at a point,
 when you think you're at a good point to build a waterfall, this movement of stacking up the blocks
 and then putting the waterfall on top, that is a hard coded policy.
 So these subtasks are hard coded, partially and partially learned, and they're controlled by this state machine.
 On top of that state machine, which we're going to get to in a minute,
 the state machine itself is controlled by this state classifier.
 So the state classifier is a thing that they came up with.
 They take pictures from the game, frames from the game, and they collect additional human labeled data.
 Where for each picture, they let the humans label, for example, is this inside a cave?
 Which you can see right here, that's inside a cave. If you play Minecraft, you know.
 Is there danger ahead, which means kind of a large body of water that you should avoid or something like this?
 Do you have animals, which is relevant for some of the tasks?
 So they build up this state classifier, which is also learned.
 And that state classifier is now going to control this state machine.
 I'm not sure if they actually have it somewhere for one of the tasks in the paper.
 They do have it in the accompanying presentation.
 The state machine controls what the age or which sub policy is active at any given point.
 Let's see. It's not here.
 Well, I can maybe maybe I can I can draw it a little bit.
 You're going to see in the presentation. So you start and then you, for example, if it's the make waterfall task,
 you go, you get to a point where you want to ask, is there a good spot to place the waterfall?
 Is a good spot in sort of the view of the agent?
 If no, then you go to the explore sub policy.
 And if yes, then you go to the go there.
 The go there sub policy is activated.
 These are these sub policies that we saw are either learned or hard coded.
 For example, the Explorer one, you can imagine maybe it's just sort of walking around
 until the state class classifier tells you that there is actually a good spot.
 So what makes the decision between no and yes, that is exactly this state classifier,
 this trained state classifier.
 At some point, it will tell you, ah, now you found a good spot and then you can switch policy.
 So from there, if after the go there, you get to another decision point
 and the decision point might be like, are you in front of a big wall?
 If yes, use the jump policy.
 If no, use the walk policy or something like this.
 So as you can see, the state machine itself is hard coded.
 So the humans came up with what do we need to do to complete the tasks?
 But the individual steps, they can be either learned or hard coded policies.
 And that's how they go through fulfilling these tasks.
 They use the state classifier to always tell them what specific subtask here should be activated
 at any given point controlled by the state machine.
 And, you know, with that, they finish the task.
 One additional thing that they sometimes need is this estimated odometry.
 This is where they just look at the actions they've performed so far.
 And they build this overhead map of the agent as the agent walks through the environment.
 They're able to sort of remember things.
 For example, this here is has animals.
 So they're going to remember locations of animals, of bodies of water and so on.
 And that allows them later if in the later stages, if they need to go back to something,
 they can efficiently find it again.
 For example, in the waterfall subtask, they have to go away from the waterfall,
 turn around to put the waterfall inside of their field of view,
 and then take a picture or finish the episode.
 That could be controlled by this overhead map that they build up.
 It's pretty interesting.
 All the while, they only have access to the image of the simulator.
 They do not have access to like the F3 menu or anything like this.
 All they have is the image.
 They do have some information on their inventory and their current item,
 but not much more than that.
 All right. That was it from me.
 If you're interested, read this paper.
 It's a pretty good write up.
 And also it has a lot of evaluation.
 They did a lot of human evaluation as well,
 computing these true skill ranking scores and so on to compare their system
 and do various ablations.
 It's really interesting.
 But now I want to give over to the interview part of this.
 Let me know how you like these more interviewee style of ways of presenting papers.
 This one is obviously a very, very applied paper, very visual paper.
 But yeah, let me know what you think and now enjoy.
 Hi, everyone. Welcome.
 Welcome. This is a really, really awesome opportunity right here.
 I'm joined by the winning team of the Mayan RL Basalt Challenge 2021
 by David Watkins, Nick Waitowicz and Vinicius Goeks,
 who managed to somehow lock their way into winning this competition.
 No, I'm kidding. I'm kidding.
 It's really awesome.
 I've seen the videos of your agent and congratulations, first of all, on winning.
 And welcome to the channel.
 Thanks for having us.
 Yeah, thank you very much for having us.
 We're excited to talk about the work.
 So if you could describe in your words the challenge itself,
 the challenge is about just sort of a bunch of tasks and then humans rate these tasks.
 What made you decide to take part in this challenge even?
 How did you find it? Did you just stumble across each other?
 How did you form your team? What was your interest in this?
 Well, I can say that we all work together.
 So it wasn't like we kind of find each other.
 We've had prior experience working together at the Army Research Lab.
 And I think Vinicius was actually the one that stumbled upon this challenge.
 And what we liked about this challenge was that it's different from most other machine learning challenges out there,
 different from other AI competitions.
 And the fact that you don't have an objective function to optimize over, right?
 So it immediately makes it harder.
 The challenge, again, is in Minecraft with these very free-form, almost lifelike tasks,
 where really you just have a description, a human readable description of what that task is.
 There's no reward function, no objective function.
 So automatically means you can't just apply standard reinforcement learning techniques.
 And you have to employ some sort of clever measures and potentially learning from humans,
 which is really what the core of the challenge is about, learning from humans.
 And that's actually, you know, each of us have machine learning backgrounds.
 And the research that we do is kind of human guided machine learning.
 So this challenge is almost like perfect for us.
 Like, oh, this is a great challenge.
 We knew it was going to be hard.
 But yeah, that was kind of the calling for us.
 And just so far, I will have introduced this,
 but the challenge was there were four tasks and every task was just given,
 if I understand correctly, like a very short description of what to do.
 So, for example, find cave is the agent should search for a cave
 and terminate the episode when it is inside one.
 That is all.
 And all you have as an input, if I understand this correctly, is the screen, right?
 Not nothing more.
 Well, you do have the screen and you do have your inventory
 and the item that you have currently equipped and the screen 64 by 64 RGB.
 That is a horrible resolution.
 But you do not have, because in Minecraft for people who play, there's F3, right?
 You can press it, you see your coordinates, you see sort of your biome and so on.
 You have none of that.
 You have to sort of do everything from the screen alone.
 And you're given 40 to 80 human demonstrations, if I know this correctly,
 but not all of them successful, right?
 That was a surprise for us as well when we were using those demonstrations in our agent.
 And we realized, like, look at this guy.
 He just walked around and threw the snowball to end the episode.
 How is that even useful?
 It was a surprise for us as well.
 And sometimes you get some items.
 So one of the challenges, for example, is to, it's called create village animal pen,
 where it is after spawning in a village, build an animal pen next to one of the houses in a village.
 Animal pens must contain two of a single kind of animal.
 You're only allowed to pen chickens, cows, pigs or sheep.
 Don't harm the village.
 And in this case, you'd be given also some sort of fence and fence gates in order to build the pen.
 So it's not like you would have to go collect resources, but the task is still quite challenging.
 Exactly. Yeah.
 You don't have to collect any resource or build anything.
 You were given everything on your inventory, but like completing all those tasks was already a huge challenge.
 Yeah. And especially given that, again, to remind people, the reward here is not some function you can compute.
 The reward is at the end, it's given to human raters.
 The human reads the description and then the human decides how well did your agent perform it.
 And most striking, I find this in a third task that is build waterfall, where the goal is that you have to,
 I can maybe read the description, after spawning in a mountainous area, the agent should build a beautiful waterfall.
 That's part of the description, a beautiful waterfall, and then reposition itself to take a scenic picture of the same waterfall.
 The picture of the waterfall can be taken by orienting the camera and then throwing a snowball when facing the waterfall at a good angle.
 So there is even an essence of sort of subjectivity, judgment, beauty, and so on in it.
 So that is the challenging part, I think, here.
 You saw this, you thought, I want to do this challenge, we want to do this challenge.
 What was your first try? What was the first thing you threw at the problem?
 Well, I can speak a little bit about it.
 At least me, myself, when I read the challenge, I had no idea how to approach it.
 Because I was thinking, okay, we have a few demonstrations, but from my experience researching everything,
 I thought if we just do a behavior cloning using this data, it won't cut it, we don't have enough data.
 And then it took us like a month to solidify an approach.
 We talked about behavior cloning, we talked about GAO, we thought about, okay, let's hard call this whole thing.
 We definitely thought about different approaches, and then I guess in the end it was a mix of everything.
 And that's what you make clear. So there is a paper about, you wrote a paper about your approach as well,
 and the paper's title is Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks.
 And then you have Minecraft pointing out that the best approach will be one where learned elements are mixed with hand engineered elements.
 So my question is, how did you come about this? Was this an iterative process?
 Or you said you scrambled with a bunch of things at the beginning. Did you add and add and add? What was your process?
 What was the first thing that maybe you realized, ah, this works now a little, right? And then how did you build up your end solution?
 Well, so I can add a little bit to that. So, you know, we were motivated, like the nice thing about the competitions,
 we were motivated to try to do well. And so we knew from the beginning that we didn't want, we wanted to take a different approach.
 Probably a lot of people would just try to apply end to end machine learning, you know, throw a lot of compute at it.
 And, you know, we kind of realized that really if we want a solution that is a little less just academic and more that works for this particular application,
 we're going to need to really use everything, right? Including, you know, try to inject our own domain bias about the problem into the framework, into the solution.
 So that really led us to these, you know, OK, well, we could have a hierarchy of different modules.
 Some of those are hand engineered. Some of those are learned, you know, the things that we can't engineer.
 And then we can have, like, you know, a state machine where we know the agent should be doing this.
 So, you know, let's not have the, you know, RL or machine learning component learn the things that we already know how to do from scratch, right?
 And just make this job harder, right? Let's add that information to the agent and let's, you know,
 save the learning for the things that we can't easily do, right? And then have them work together.
 Yeah, I think you make this clear and I'm just going to share a screen for a bit right here.
 You make this clear in sort of this diagram, which is an overview over your system.
 And at the core here is this state machine. You want to maybe talk a little bit about why a state machine might make sense right here.
 For example, this here is the state machine for the waterfall task.
 I can talk a little bit about it. So if you saw like those tasks, so, for example, let's talk about the beautiful waterfall task since we have the diagram open.
 There's really like a hierarchy of subtasks that needs to be complete in order, you know, to finish this whole task.
 For example, for the make waterfall, right? First you need to find a good spot to build your waterfall, right?
 And that means you need to climb up somewhere. You need to be like at the edge of a cliff, right?
 And then you have to actually build the waterfall, you know, you got to equip your water bucket and, you know, point it down, throw the water bucket, right?
 And then hopefully this waterfall will be beautiful, right? Assuming you got like a good spot.
 Then you have to go really far away from this waterfall and then position your camera just right to get like the best, you know, the best view of this waterfall and throw a snowball to finish it, right?
 So there's this whole hierarchy of tasks. It needs to be completed like one step at a time and there's like this logical order.
 So the state machine was our approach to make sure that the agent would actually follow this order, you know, without coming back and forth.
 Like if you do like, for example, some just an end-to-end machine learning approach, the agent might, you know, let's say go find a spot and then we'll go back, take a picture, you know,
 come back again, try to build, equip the water bucket to build the waterfall.
 So the state machine was our solution to make sure the agent would follow kind of this logic for each task.
 And I think you profit from the fact that all of these tasks can be sort of described quite well in this state machine fashion, as I think a lot of, you know, if you play Minecraft as a human, that's sort of the same thing you do, right?
 You if you want to beat the ender dragon, you okay, first I need to do this, then this, then this.
 And it's quite the same thing with a few decision nodes in between.
 And these decision nodes here in the in the green, those are now decided by classifier, if I understand this correctly.
 So you build this this little interface here where humans could rate, you were allowed in the competition to collect a little bit like a limited amount of different human feedback.
 And you chose among other things, you chose to have humans label different images from the game with such a with them with such maybe you can describe it a little bit.
 What were you interested in? And why did you choose to put the additional human labeling into this task and not any other task?
 What like, why did you prefer this?
 Something important to keep in mind is that you're allowed to include 30 megabytes of additional data in this competition.
 And the Minecraft simulator is such that if you were to record a bunch of actions or steps that the player took and try to replay them, it's not currently designed to handle RNG the same way every time.
 So if I go break a block, that block is going to fly differently depending on the state, the internal state of the random number generator.
 And we have no control over that.
 So you can't seed it necessarily.
 We can't seeding it just doesn't work.
 So we couldn't just collect more demonstration data other than videos.
 And that would eat into 30 megabytes very quickly, as I'm sure you could imagine.
 So dividing up each of the tasks into a bunch of shared states made the most sense to us.
 It's something we've used in previous research to handle navigation tasks before.
 And it works reliably and I think there's a lot of research in making state classifiers work really well.
 So it was more just us as a team, you know, while we're watching TV, labeling a bunch of Minecraft screens.
 The most difficult part, of course, though, is it's 64 by 64.
 And there are many situations where maybe you want to recognize that there's an animal in the frame and it's a chicken and it's the small white blob.
 But it could be confused with a flower and you're kind of fighting yourself to make sure that this actually works.
 And so there were some different strategies we were looking to employ to make sure that the state was classified correctly.
 But it worked pretty well.
 Cool. And I think people can see here maybe at this graphic, but you have such things like, for example, good waterfall view, which makes sense, right?
 This is a subjective thing of the reward function.
 So it makes total sense to include that in the human annotated data and not code or heuristic.
 But you also have things like a danger ahead, which you then use.
 So I think once you know which node you're in, right, in this state machine, very often the blue blocks right here, which are the actions, the blue blocks involve going somewhere.
 For example, if has mountain, then, you know, if you don't have a mountain, find the mountain.
 If you do have a mountain, go to the mountain. And that part means that your Minecraft agent has to go from point A to point B.
 And that's where you build a specialized navigation, navigation subroutine.
 And you said right now you've already done this in the past.
 Can you tell maybe a little bit in general, what does it take to make agents navigate around?
 So can I just mention one more thing about the state classifier?
 Sure.
 So with the state classifier, like David and Venetia were saying, it's really the core of the state machine, right?
 So we knew we wanted, you know, it's the thing that makes the drives our entire solution.
 So it has to be, you know, more or less somewhat accurate.
 And we needed a lot of data. So we actually collected around, I think, eighty eight thousand labels, which sounds like a lot.
 But of course, you know, that type of manual annotating, no one really wants to do.
 You know, as machine learning scientists, we'd rather spend that time trying to, you know, code up a solution to do that instead of doing it ourselves.
 But what we did, we tried to make it as easy as possible by, you know, we're not HCI experts,
 but, you know, we tried to come up with a kind of intuitive labeling interface to make it as quick as possible to kind of, you know,
 like one demonstration that's three minutes long at a, you know, a FPS of 20 frames per second.
 You know, that's a lot of images. And we try to take advantage of the fact that the images are somewhat correlated to time.
 Right. So the way we designed our labeling interface is kind of just a step through each image through the trajectory.
 And if you hold down a button, let's say one of the buttons is, you know, there's there's nothing ahead.
 It's just open fields. So you can just hold down that button and it's going to traverse, you know,
 through the demonstration until something else comes up and then you can just move a different button.
 So very quickly, you know, you can, you know, label 5000 images in one trajectory in like less than a minute
 because you're just holding down these buttons instead of like, you know, showing an individual image
 and then selecting the label and then the next image and select the label.
 I think that really allowed us to get it sacrifices a little bit of accuracy.
 Maybe when you're transitioning, you might miss, you know, get a few misclassifications,
 but you're able to get a lot more more labeled images.
 I think this is a recurring theme sort of in real world tasks, the efficiency of data labeling when you include humans.
 I've just recently watched sort of Elon Musk's appearance on Lex Friedman.
 And before that, I've commented on Karpati's talk about the autopilot there.
 It's a thing that you see again and again that the easier you make it for humans to annotate data,
 the more benefit you have later.
 Like it's almost an unfair multiplier that you have on your system.
 I think it's neglected currently by academia.
 So it's pretty cool that you thought about this as well.
 Yeah, I think it is neglected because it is not easy and takes a lot of time.
 Like manual labor, nobody wants to do manual labor,
 but definitely having like high quality labeled data labeled by humans makes totally the difference.
 So and now we'll let's let's go to the to the navigation subroutine.
 How do you how do you navigate?
 Wait, that is here.
 So you have a navigation policy which essentially says the agent needs to go from A to B
 and what does it take to build that?
 Like it seems very complicated in a game so complicated as Minecraft.
 So well, so the behavioral cloning part, right?
 So that part is, you know, unfortunately, just very simple.
 It's not any secret sauce or anything complicated.
 You know, we again, just prefacing by this, you know, was a competition and we had a deadline.
 We had so much more that we wanted to do with this particular part, right?
 For the solar navigation part, we wanted to do something, you know, way more than just standard behavioral cloning.
 You know, things like generative adversarial imitation learning, you know, trying to have better architectures.
 In the end, we didn't have enough time.
 We were scrambling and for this component, we just did behavioral cloning.
 The way that we did that is, you know, as you can see in this model, it's like, OK, the agent only has the image as input
 and its output, you know, are more or less just the direction key.
 So it can go forward, it can turn left, it can turn right, it can strafe left, strafe right, and then it can move its camera.
 And really the way that we did that is we just we had all these demonstrations for each of these tasks.
 We kind of the only kind of trick that we applied was that we realized this is just a navigation component.
 So we only want to learn to imitate the part of the demonstrations that we're navigating.
 Right. So let's just chop off that demonstration just to that navigation part and then feed that into our navigation policy.
 And so that's that's basically what we did was, you know, any any time where the agent was building,
 like building the pen or the village or the waterfall, we cut those segments out.
 The remaining segments are where the agent is just trying to go from one point to the next.
 We kept those in and use that as our training data for the behavioral cloning module.
 And in this in this model here, it says image input.
 Do you also give the model access to, let's say, the the results of your state classifier and maybe the current state machine state or something like this?
 So the agent knows where to go or do you rely on behavior cloning for the entirety of navigation?
 Yeah, that's a really good point.
 So again, it's our this particular navigation policy is just terribly simple.
 It's really just the the image input being driven by the state classifier in the sense that it allow, you know,
 the state classifier decides when to start and stop the navigation policy.
 But we're not feeding in any information directly from the state classifier or other other more interesting information that that certainly would help.
 If we had more time, we could probably do that.
 It would make sense to do that.
 But right now, the state classifier just decides when to start that navigation policy and when to terminate the.
 I think so.
 No, I just just want to add a little bit on top of that.
 The main reason we didn't add anything else on this is because we didn't have.
 So like the so this navigation sub task policy was trained from the demonstrations provided by the competition.
 So that data didn't have any like state machine.
 So the state machine was everything on our side.
 So we really only had access to the actions that the agent took right and the camera data.
 And and again, like I think the using that demonstration data provided by the competition to train only the navigation sub task made sense because let's say think about it.
 Let's say we want to do end to end behavior cloning, right? And then you were doing the fine cave task and the fine cave task.
 At some point, the human will throw a snowball when the agent is inside the cave.
 And that's only one data sample.
 And the whole episode has about two to three thousand.
 So you have one sample to throw in the snowball on over three thousand samples.
 And to find the cave, it took a lot of steps and this is all really useful for navigation.
 So we did this like Nick said, this preprocess to remove all those actions, leave only the navigation part and use that to train this navigation sub task.
 And I think that was pretty helpful to in our approach.
 So is it fair to say that, for example, you're here and you you are your house mountain classifier says yes, then the state machine would simply activate the navigation.
 Does it? But it doesn't it doesn't necessarily tell it where to go.
 You just rely on the fact that your demonstration in your demonstration, people have generally gone towards the mountain and therefore the navigation policy would have learned that implicitly.
 Exactly. Let me I guess let me explain this diagram a little bit.
 So what you said is correct. So the green diamonds are decision notes, right?
 And that's that's based on the output of the state classifier. Right.
 So like has my mountains, you know, if it's over, let's say 90 percent confidence, we'll take that as a yes. Right.
 And then we go to those blue rectangles and each blue rectangle is a sub task and those sub tasks can be either learned or coded or like hard coded.
 So, for example, go to go or find go actually find go was learned from the human demonstration.
 So we would not say like something like, oh, go to this coordinate like we didn't have. Right.
 We would just use the human the policy that was trained from human demonstrations to navigate, let's say, going up the mountain. Right.
 And then let's say on that part of the diagram where you have the dashed line, you know, there's a green diamond there written at the top.
 So let's say if the state classifier detect that we're on top of the mountain, right, then we would switch to this place waterfall sub task and this place waterfall sub task was hard coded.
 So that was not learned from the human demonstrations.
 And what the sub task does is basically point your camera down, keep the water bucket and throw it.
 You know, that's kind of placing the waterfall. So those blows are our mix of learned sub tasks and hard coded.
 Yeah. What my question is a little bit. You have, for example, this danger ahead state. Right.
 But you don't feed any state to the navigation policy. Where is the danger ahead used inside the state classifier somewhere?
 Like you say, if there's danger ahead, then we don't even want to activate navigation.
 Exactly. So that's something that it's like a safe critical sub task that takes priority over everything.
 So it doesn't matter if you're looking at the mounting, whatever you need to do. If there's danger ahead, just avoid it. Right.
 So it's like a sort of a safe override that's always on no matter which sub task we're doing, if you're following the human or not.
 Because, you know, just avoid danger because our first iterations of Asian and even the final one is still there sometimes.
 When you fall on one of those lakes, you just can't escape. It's just too hard.
 Like sometimes there are like two blocks tall, then it's hard to like teach the Asian to break the blocks and jump.
 Like do all those things that us humans do pretty well for the Asian is pretty hard.
 So our Asian got stuck a bunch of times. Then we had to add like some safety sub tasks to help a little bit the Asian to escape those things.
 And at some point you also built in this odometry estimation because you only had the image and you thought it would be...
 Maybe you can explain this. What led you... Because it's not a straightforward thing to include, right?
 If I think about how would I solve this task, what is the odometry estimation? What is it for? And why did you include it?
 I can talk about it. So like you mentioned at the beginning of the video, we could not...
 Like in Minecraft we do know where the Asian is. Like when you're playing the game, you can press F3, you can see everything, right?
 But in the competition we were not allowed to use that.
 So we had some ideas, okay let's use the simulator, but we were not allowed to do that.
 But we're thinking like what do we know about this problem? So we do have access to the actions that the Asian took.
 And we do have access to the image. Not only that, we know a little bit of Minecraft. So we know that the simulator runs at 20 frames per second.
 So each frame is 1 over 20, 0.05 seconds. So we know this time interval between each frame, right?
 And from Minecraft we know that for example the walking distance is actually I think 4.32 meters per second.
 So we had this information from the wiki. So let's say if the Asian sent the command to move forward, right?
 And not considering inertia or anything, right? We could assume that in one frame the Asian walked 4.32 times 0.05.
 So like this velocity times this dt, this time interval. So we know how much the Asian walked in the X direction, right?
 And then we had the actions, we had access to the actions for the camera control. So we could estimate the heading.
 So just based on the actions that the Asian took and knowledge of the simulator, right?
 We were able to sort of estimate velocity X, Y and heading. And then you integrate that over time because you know your time interval.
 So you can come up with estimates of X, Y and heading for the agent.
 And that's what you see on this kind of this black diagram on the right, which I can explain everything in more details too.
 So I mean you build this sort of map almost. Like this is an overhead map of the agent in its environment.
 And annotated with first of all what you've done so far, right? Your position that's been going on.
 Maybe if this here loads, this here is different trajectories. But you also annotate this map with various things that you find.
 Like whenever your state classifier says something. Where is this information used?
 I guess it's you said it's not in the navigation because that it doesn't get any additional features.
 Where is the information that you estimate from this overhead map? Where is it used?
 The best example for this is to make waterfall task. So when the agent places a waterfall,
 you know something we're thinking is maybe we'll try the behavioral cloning, but often, you know,
 the behavioral cloning doesn't really stay still very often because it really learned the navigation sub policy.
 So instead we sort of use that heading estimation to move the agent away a fixed amount and then rotate around to look at it.
 So there are just certain tasks that it's really important that whatever the final view is,
 the line with some landmark in the environment that we don't have a ground truth information for.
 Yeah, so it's really the odometry is mainly used in various places in the state classifier.
 I mean, the state machine in some of the subtasks like David was saying. Another example is the animal pen, right?
 The challenging part of that task is you really have to build. You first got to find an open location, then build the pen.
 And then you have to leave that pen and go find the animal somewhere, right? They could be anywhere.
 And then lure them back to the pen. So you have to remember where you built that pen.
 And so that's, you know, the odometry comes into play for that place. So we were using the state classifier to kind of classify.
 OK, here's an open location. Now we switch to pen building mode. OK, the pen is built. Let's go find some animals.
 We remember the location of that pen, you know, based on our estimated odometry.
 And then once we find some animals, then we try to go back to that location.
 And just to say that the try to go back will be a hard coded policy that takes as an input the remembered location of the pen and your guess of where you are in relation to that pen.
 Exactly. Yeah. So at that stage you have a XY coordinate of the pen and you have an XY and headings estimates of your position, right?
 So you can basically compute the angle between where you're looking and where the pen is. You can compute this angle, right?
 And the policy was literally kind of close this angle and then keep moving to kind of reduce this distance over time and go back to that location.
 So it's a simple policy. There are a few limitations though on the odometry side, which I just want to comment just to don't say this was like a god-tier approach for that.
 So, for example, since we only use the actions, right? If you think about it, the odometry is just seeing the actions, right?
 And then, OK, the agent is moving forward. So we're seeing this moving forward action, right?
 So we're integrating that over time, increasing the distance and everything, right? But what if the agent gets stuck, like behind the rock, behind the tree, and it is still moving forward?
 Like in Minecraft you can still kind of walk forward sort of sliding, right? But you're still stuck in place. But the odometry does not know that.
 We had some ideas to integrate differently in the pixels, right? Using this camera data to know when the agent is stuck. So we ignore that.
 But we didn't have time to do that at the end. But this approach, our current approach, still works for short distance, right?
 So, of course, the longer you walk, you know, like the drift will be just higher on this estimation. But for short distances, it actually works pretty well.
 And I guess it's sorry. I was going to say that a slam approach in a 64 by 64 image that's only RGB is incredibly challenging and probably not the right approach for this particular challenge.
 And it might also be fair to say that you said you had a lot of ideas. I guess if you were to go further, you'd probably let's say, try to come up with a navigation policy that's both learned but also controllable in some way.
 Try to come up with an odometry estimation that takes into account the picture, which could recognize when you're stuck and so on.
 I think there's there's a lot of stuff to improve. But I'm very impressed by sort of your your pragmatism of okay, this works well enough. Let's go on.
 Was there was there moments when I guess there's moments in every project when when you're or what was the moment when you most thought, ah, this is not going to work.
 Let's give up. Like, did you have a moment like this? And and what did you do?
 You guys want to comment on that?
 Well, there's there were, I guess, a lot of those moments.
 We, if you go back to the main overall diagram, we definitely like had, you know, went back and forth on, you know, what should the solution be?
 You know, we were still toying around at some points with with, you know, a more, you know, end to end approach in some places and whether we should put our eggs in that basket or whether we should do this current approach.
 Ultimately, you know, this is the one that we landed on. And we we designed this.
 The next thing about this approach is it's it's hierarchical, but it's very modular. Right. And the idea is that each of these sub tasks, you know, their individual models that we can improve upon or replace.
 And so, like, you know, if we had more time, some of the things that we would do is start to try to replace some of these hand engineered sub tasks with more learning based sub tasks and or, you know, replace the navigation module with a more advanced learning module that uses more information.
 One of the things we spent a lot of time on that never made into or was was kind of using generative adversarial limitation learning as our core algorithm for learning the navigation module.
 And, you know, with Gale, it's it's basically using a GAN. And as we found out, like everybody knows, GANs are notoriously difficult to stabilize, including GANs for Minecraft. And it ultimately didn't end up making it.
 So we had to revert back. So that was one of our centers. We're like, oh, this is this is definitely not going to work. You know, we spent a ton of time doing that and we had to kind of, you know, replace with our with our backup, which is just, you know, standard behavior.
 So go ahead. Also, the one point my brothers are very good at Minecraft and the Minecraft speed running community is a pretty big thing. So at one point we were considering, why don't we just get somebody to play Minecraft really well?
 But that stupid Minecraft simulator limitation and also, you know, it's it's one thing to get a bunch of people to play the game better than maybe the demonstrators were playing. But that also means that, you know, that data won't necessarily be very rich because they can't play the game well and label the data at the same time.
 And I think it comes back to this problem of labeling data really conveniently is difficult, especially when you're driving the agent simultaneously. So it becomes a very difficult challenge to use human data when the the amount of data you can actually collect is small.
 And this being Minecraft, I think I like I'm fascinated by this because I wonder how much world knowledge is in inside a human when they play Minecraft and how much is sort of learned because the world is different, like literally different every time.
 And I can learn Minecraft by just watching someone do it a few times, right? I can perfectly not perfectly, but I can well generalize to other worlds. Is that because I've watched someone I've done it a bunch of times or is that because I know from my life what sand is and what water is and how it behaves?
 And I think that I don't know. Yeah, I think I guess the main advantage of like, you know, humans is that, you know, we've lived, you know, 20, 30, 70 years already right in the real world.
 And then Minecraft tries to mimic that. So we humans have a huge kind of baggage that we can use. But we have to always remember like those Asians, they start from scratch. They literally start from nothing.
 Right. We had to collect data to teach what danger was for those agents like, like to teach, oh, don't jump in the water, you know, don't don't drown there, you know, things like that. So that's very challenging as well.
 And I have I have your sort of so for videos that you uploaded, and they have side by side the agent view, the classifier, but also the odometry estimation.
 Do you want to maybe so this is for example, do you have one that is your favorite of these four?
 Yeah, probably the waterfall I think will look pretty nice. So this is build build house was pretty challenging.
 This is 30 seconds I'm gonna I'm gonna slow it down to like point to five right here.
 Do you maybe. Oh, yeah, I can. Oh yeah, I can like comment like a comment a little bit on what's happening right here. So which state is it in what's happening.
 Yeah, so so this is a video of the agent solving the make waterfall task right and then you mainly see in the screen in the screen two panels.
 So on the left side, that's the RGB. So this is like a camera view of the agent right and on the right side, this black panel is the estimated odometry.
 So if we start there on top left, you see like action and then use tensor right. So that's the I think 12 or 13 actions that the agent was performing.
 So they're mostly binaries. So like move forward or not move back or not, you know, things like that.
 And below that you see the raw output of the state classifier. So we had 12 classes or I guess 13 with, you know, the known class.
 And you see like the confidence of the classifier, you know, for classifying the state like this camera, this camera image.
 So you see like right now, you know, facing wall is pretty much almost 100 percent.
 I think it is from all the stone that the agent is seeing. So it thinks it is a wall. Right.
 And on the right side, the odometry. So we can start there on the on the top part there.
 You see a X, a Y and a heading. So X, Y. So that's the estimated position of the agent.
 So that's not the ground truth. So again, we didn't have the ground truth. Same with the heading.
 So that's estimated and that camera angle there is like a vertical angle. Right.
 And then on the right side, you have like some time. So we kind of just have a track, keep track of time.
 And then you have a legend. So the legend there is for all the colors you see in the odometry.
 So the red one, the red dot is the agent. So right now it is down at the bottom of the screen.
 Whenever the way the agent walks around, it leaves like this trace.
 So that's the Y dashed line that you see on the screen.
 And then like right now you see, for example, it just saw that cyan, I think, blob at the bottom there.
 That's when the state classifier detect that we were on the top of the waterfall.
 So you see that that's the last thing on the legend there.
 So basically, yeah, the agent walks around and some of the relevant states that we classify,
 we sort of drop a pin in the map kind of just to keep track of it.
 In the video, the first like 25 seconds or so, what you know, this is the map.
 You know, it starts off basically with the navigation policy, right?
 The go to goal. So the behavioral cloning module that we trained is in control and it's driving.
 And it's basically, you know, trying to mimic all of the other human demonstrators that did this task, you know,
 which is more or less kind of walk around and look for a good spot.
 And then when the state classifier detects like, OK, this is a decent spot, that's when you saw it switch to the,
 all right, let's build the waterfall. And then after build the waterfall,
 the state classifier switch to the now go take a picture sub task.
 And so that's basically what you see in this video.
 And one thing I'll say with this, the interesting thing with the navigation policy is, you know,
 this is something we kind of noticed and it's just a theory.
 We don't have any proof on it. But like, you know, the, you know, the agent jumps around a lot.
 But we think that's because the agent is mimicking the human demonstrators.
 So like, so jumping for the sake of jumping, not necessarily to jump over stuff like, you know, there's some players.
 You're faster if you jump.
 Yeah, yeah, exactly. And that's seen in the demonstrations or some players like me, I just jump idly, you know,
 just a fixation. So I'm just like randomly jumping that not to particularly jump over anything.
 You kind of see that in the agents behavior. So it's almost, you know, makes it more human like,
 at least in our opinion, versus, you know, a hard coded navigation policy, which mainly, you know,
 you might expect it to just walk without jumping unless it needs to jump right over something here.
 You know, the agent is kind of just more pseudo randomly jumping like a human would.
 And we thought that was pretty cool because, you know, another part of this competition that we haven't talked about yet,
 it's not just, you know, developing agents that can do the task the best, but also there was a sub thread
 to the competition of who can build the most human like agent, which we also won that prize.
 So, you know, this potentially, I mean, really our whole system, you know, is sort of aimed at the human like
 because we added a lot of human knowledge to it. But like the behavioral cloning part, you know,
 that might also add to that because it kind of moves around more or less like it, like a human would move around.
 And it looks a little less robotic, like if it were kind of a more hand engineered.
 Except like here when it's like a good spot for a waterfall, you immediately point down and start like,
 I guess this is the hard coded part, like you see right now, immediately point down, build a bunch of blocks, place the bucket.
 And then it's interesting. So this part here is hard coded as well. It's just like move the agent away.
 And we see the agent kind of slide on the left a little bit because I've noticed that later when it turns around,
 it sort of almost misses a little bit the angle. Right. So this could be this drift that you have in the odometry estimation.
 So it's trying to make a picture of the waterfall directly, misses like a little bit.
 So I guess that would be that would sort of be the problems that you get in just having the just having the estimation from the action, which you mentioned.
 Yeah. So for example, when you throw the water down, right, sometimes the agent will float in the water and that will turn the agent a little bit left and right.
 But the odometry doesn't see that because the agent didn't command the camera movement.
 So it doesn't update your headings. So that can also cause problems later. But yeah, like you said, that part was hard coded like the place waterfall subtask was hard coded.
 But all the way up to that thing, up to that part was learned from human demonstrations, which is the navigation subtask.
 What I think what you what you need to do is you just need to train the navigation thing on, you know, dream.
 So you just want to you just want to train it on like a bunch of videos of dream and then just see what happens.
 I would be so curious to see what happens.
 Well, that's what we wanted to do that initially is we thought, oh, look at all of this awesome data on YouTube that we could maybe try to learn from.
 But there's no actions associated with it. Yes. OK, true.
 You'd sort of have to estimate the actions almost a little bit.
 And you'd also have to like there's a lot of things you'd have to guess at what's actually going on, which where do we crop the video?
 Right. There's all this stuff they have overlaid and it becomes more challenging to use YouTube data.
 But I see. OK, you you wait. What was I was I gonna?
 One thing that yeah, one thing that I was a little bit like a tiny bit dissatisfied with with this competition.
 Obviously, it's already super duper challenging. Right.
 And Minecraft is so much more complicated than this thing.
 But there were these four tasks and you knew them ahead of time. Right.
 That's why you were able to sort of build the state machine.
 The descriptions were very clear ahead of time.
 Let's say that I come and I'm the organizer and I change the challenge for next year and next year.
 It's still the same thing. It's human rated.
 It's described in just like a simple string, but I won't tell you what the string is. Right.
 I won't tell you ahead ahead of time.
 How would you how would you go about designing a system like this?
 Like what would you would you do? Would you try to go the same route?
 Or let's say you also had very limited resources like you had now.
 You can't train like a giant or else system.
 I think we would definitely be forced to go a different route, which I think would be good.
 You know, one of the things I like about this competition again is that it's you know, I think it's important for the field because you know, it's these tasks again that you can't just, you know, do this black box optimization over because there's no objective function.
 So you're forced to really try to learn from a human. Right. Or do something. Right.
 And and and you know, we really took that to heart. We knew like, OK, in order to do wellness competition, we cannot just use the human provided demonstrations like the majority of the other teams.
 We had to add our own additional human input and feedback.
 And we did that with the design of our state machine and in the labeling, the human exhaustive human labeling that we added. But, you know, to take it a step further, really, I think the interesting thing would be to have a system where you have you learn from real time human feedback, which our system didn't do.
 Because, you know, well, one is that's more challenging and we didn't have time. And because all the the tasks are known ahead of time, you don't have to have real time human feedback.
 You can, you know, collect your human feedback or human labeling beforehand and then use it.
 But if you have now a new iteration of this competition where you do not know the the tasks ahead of time, then you now might need a system where your agent needs to learn from human feedback in real time and kind of interact with the human to kind of get that learning.
 Because, you know, you're just seeing what you need to do the task at competition time. So I think that would be really interesting. And that would force more solutions to use something that that uses real time human feedback.
 What set you apart? If you you've probably seen sort of the other teams that competed and so on, and I'm sure they were also they were also engaged and motivated and tried a bunch of things. What do you think was sort of the or maybe the most defining factor that let you win?
 Was it I'm sure there was a level of stochasticity in the evaluation, but you know, you won, I think not one but two of the three subcategories even.
 So it must mean that you had a considerable, let's say edge over most of the competition. What in your estimation was that?
 I have a guess you guys can comment on that. I think in my opinion, I think our edge was actually using human feedback data.
 So like the other teams, if I remember correctly, I think number two used sort of improved algorithm that would improve on Gale. So that was kind of sort of full RL approach.
 The third team tried to use some of kind of learning from human preference, if you remember that paper, but they didn't use a human to rate the trajectories.
 They used like heuristic, right? And we were the only team that actually use human data. So we, you know, we label a bunch of data, you know, we added kind of our knowledge, our bias on the task and everything.
 So I think really using the human, I think was the key factor that allowed us to win two or three of the awards.
 100%. Like, you know, yeah, we had a state machine approach with, you know, these modular hierarchical design, but really we wouldn't have been able to do that if we didn't have, you know, this classifier that was generated with additional, you know, human feedback and human labeling.
 And so it's really the thing that Sturzauper and like we said, it was, you know, the other teams, they just use the human demonstrations and even the third place team, they used a simulated human, right?
 Instead of, you know, doing the hard work of actually getting that human feedback, they just defined this simple heuristic. And I think that right there is like, you know, the important thing, like the field, you know, sometimes can just like, oh, well, let's just, it's easier to kind of simulate out the human.
 Let's, you know, come up with a better algorithm, but it really just shows like we should do a better job trying to incorporate human feedback because it's definitely, you know, valuable information and can really improve the way we develop our AI algorithms.
 I think it's important as well to, you know, when you look at Minecraft, it's very much feels like an open world sandbox problem, very similar to using a robot in the real world.
 And collecting real world data is about as difficult as I would say, well, it's a little more challenging in some ways, but challenging to collect lots of good rich human demonstrations in this particular environment.
 And so, if we were looking at this as a generalized approach to solving this kind of navigation problem, I think we would have used a similar approach for handling this on a robot, where, you know, robot going to go pick something up somewhere can be broken down into a bunch of discrete steps, and we solve each of those steps really well.
 Whereas an end to end approach, we risk having situations where the neural network is doing something that we can't debug at all.
 And I think that hierarchical approach really let us debug each step really well, as opposed to the monolithic approach.
 Now, just to say in on the on the leaderboard website, there is a team that has like a better score than you was that is that an artifact of the one leaderboard or is it late entry after the competition.
 So that's the that's the public leaderboard, right. And it's not officially award. This is, yeah, this highlights the other difficulty of this competition is like, again, there's nothing to just automatically grade everything that you have to just get volunteers.
 To literally just sit down and look at pairs of videos of different agents and see which one is better. Very, very arduous task, right.
 And the public leaderboard is just any random person with a web browser can go on and start rating all the people you know we we provided some ratings.
 It's completely unofficial, but it was just used to kind of determine who would go to the next round.
 So the top 10 teams and then the competition organizers actually hired professional contractors, you know, you know, but actually had, you know, not just random people, but like contractors go and do official valuations to determine the winners.
 And on that one, that's that's where we won first place. But on the public leaderboard, we're not showing us first place because of the stochasticity of all the human raiders.
 I love that the professional contractors were probably like they had to know Minecraft, right. So they're like the most competent people in it were probably like some 13 year olds.
 Kids to watch some videos, give some ratings.
 Excellent. Yeah, is there is there anything you you'd like to that was my exhaustive list of of questions that I had about this. Is there anything you feel is important to to add for people to know if they want to do something like this themselves or
 I think I think during the presentation we had this slide about that. So so this competition might happen again next year or I guess this year already 2022.
 So if you're really interested on that, make sure to go ahead and start playing with the mine RL package now because it took us a long time to to figure that out.
 I think I think I can speak for all all three here. I think that was our first time working with the Minecraft package like the reinforcement learning package.
 So it took us some time to to learn all the you know how to work with that their action space observation space and everything.
 So if you want to like an extra edge this next year you can maybe start playing with the package now. And I think I think that's it. Maybe play a lot of Minecraft. I think that that helped.
 Yeah, I mean you mentioned the paper that we have but we also made our code available for anybody that wants to try it themselves or improve upon our solution.
 Awesome. I think the paper got the link to the code. Yeah, I'm pretty sure. Yeah, it's there. So yeah, go ahead to play with our code. Maybe make it better. Let us know. Maybe make some pull requests.
 Cool. Awesome. Well, in this case, thank you so much for being here and sharing this. It's really I love I like it. I think it's really cool when when things like this get get out into the well not real world but Minecraft world which is close enough.
 It's incredibly hard task and for just from the videos I saw it I was surprised by you know just how far you can get with how little sort of resources and data.
 And just one last thing like the definitely, you know, for this first year's competition, the, you know, this is far from solved, and I think the competition organizers realize that too. So out of the four tasks which are, you know that you already mentioned, you know, basically advancing
 the fine cave in the make waterfall the easiest. Those are pretty much solved. The create animal pen and especially the build the village. None of those solutions came even close to really solving that, you know, I'm sure the human raiders are just looking at to really
 junk agents doing random stuff and trying to pick which one's better. Right. But, you know, it's still like on that build village tasks but still very simple tasks out of the range of tasks that you can conceive in Minecraft is still far from from salt.
 And, I mean, yeah, there's, there's no crafting yet there is no fighting there is no exploring. And this isn't even like this, this is where Minecraft starts the actual game of Minecraft is where you sort of set your own goals right and you try to achieve something new.
 Yeah, it's, it's cool to see that there's still a lot of a lot of stuff to do. Awesome. Thank you so much for being here. And, yeah, I hope to see you next year again.
 Thank you very much for having us Yannick. Like I said, I watched a bunch of your videos I really like your channel I'm excited to see.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel.
 Let me know if you like this video, leave a like if you did and leave a comment if you have comments, suggestions, anything at all. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.
 Hey there it's Yannick, I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators, so you can see what the human saw and what it takes to win such a competition will show you all the submissions for each of the tasks in parallel. See you next time.