Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper minerl minecraft ai diamond pickaxe ai diamond pickaxe openai minecraft deep learning projects what is deep learning deep learning tutorial introduction to deep learning gpt 3 gpt-3 vpt video pretraining video pre-training openai vpt vpt minecraft minecarft
#openai #vpt #minecraft 

Minecraft is one of the harder challenges any RL agent could face. Episodes are long, and the world is procedurally generated, complex, and huge. Further, the action space is a keyboard and a mouse, which has to be operated only given the game's video input. OpenAI tackles this challenge using Video PreTraining, leveraging a small set of contractor data in order to pseudo-label a giant corpus of scraped footage of gameplay. The pre-trained model is highly capable in basic game mechanics and can be fine-tuned much better than a blank slate model. This is the first Minecraft agent that achieves the elusive goal of crafting a diamond pickaxe all by itself.

OUTLINE:
0:00 - Intro
3:50 - How to spend money most effectively?
8:20 - Getting a large dataset with labels
14:40 - Model architecture
19:20 - Experimental results and fine-tuning
25:40 - Reinforcement Learning to the Diamond Pickaxe
30:00 - Final comments and hardware

Blog: https://openai.com/blog/vpt/
Paper: https://arxiv.org/abs/2206.11795
Code & Model weights: https://github.com/openai/Video-Pre-Training

Abstract:
Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.

Authors: Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll talk about video pre-training, learning to act by watching unlabeled online videos. This is by a team out of OpenAI and is the first system that successfully crafts a diamond pickaxe in Minecraft. So apart from humans, obviously. So Minecraft has been sort of a test bed for reinforcement learning algorithms all of these years. But it's notoriously hard if you don't know what Minecraft is, even if you do, it is a hard, hard problem. So you're in this open world, and you can essentially deconstruct any block. So the first thing is you want to punch a tree, right? This gets you wood, and then you want to craft that wood to these logs, and you will craft these logs to that table. Crafting is done in a menu like this, like the top right here. The crafting interface means that you have to arrange the items you have to create new items. There is a recipe book, but sometimes you also have to know what you're doing. Then you walk around in this open world. This is not a very competent player right here. And you can see there's a menuing interface and so on. So this is hard, even if you have like predefined actions. But if you don't, and you just want to use the mouse and the keyboard as this system does right here, it becomes nearly impossible. There is a progression of things to build, you know, given wooden planks and crafting tables and sticks, sticks are missing here, you can build the wooden pickaxe with the wooden pickaxe. You can you can use that to mine cobblestone with the cobblestone. You can then build a stone pickaxe with the stone pickaxe. You can go even further and further. Here you can see a bunch of stuff that this agent learns. This is tapped on mute. Well I did it. In any case, this agent here learned to raid a village like to look around in a village. You can see just how complex these worlds are right there are these villages, it's an open world, the terrain is randomly generated. And it's a completely new terrain every single time you start the game. And this is why it's so incredible. Look at the amount of the items in this in this chest right here. So just to give you sort of an idea of now it's an idea of how difficult this game is. No agent has yet managed to successfully kind of progress through these things, especially no agent that is not like has hard coded things in in it like that. So here would be the full progression to the diamond pickaxe week before we saw you get into the stone pickaxe, you can use the stone pickaxe to mine iron ore. From that you can smell the iron ore in a furnace to produce iron you need something that's burnable. From that you can craft an iron pickaxe and with the iron pickaxe you can mine the diamond if you find the diamond. Now the episodes, the episodes here run for 10 minutes, I believe or 15. We have tried this so on our discord we discussed this paper and thank you very much to everyone who participated. I've tried it. And it was pretty hard. I got to two diamonds once within within two diamonds within 10 minutes or 15. And the diamond pickaxe needs three diamonds. So for a human it's already pretty hard for a system like this. It is actually it's pretty darn hard. So you can see it right here. If you were to train this from a randomly initialized model just with reinforcement learning, it doesn't work. So the entire question is, how do we get this to work in a like in the cheapest way possible? And that's where this paper comes in. So I think the fundamental question, even though it's called video, video pre training, which essentially means we have a model that's pre trained on videos. The main question is here, where do we spend our money most effectively? So let's say we have a bunch of money, right? So let's say here is a bucket. Well, it's more like a box, okay. And the box is the box has dollars in it. Now these aren't as worth as much anymore as they used to in the good old days. But in any case, how would you spend that money, right? You can go and collect label data, for example. So you can go to contractors and they can play the game. All right, so oopsie. You can tell them you can say, okay, this much of my money, that's kind of playing. I pay people to play the game, I record their actions, right? So and then I have a video together with the labels, the labels being the inputs of the humans. And then I have at least a data set where I can do something like behavior cloning, right? The other thing could be I could spend the money on getting unlabeled data. Now if I spend the same money on unlabeled data, let's say this this slice right here, unlabeled. I suck at writing. I'm going to get much more data, but they don't have labels. So can I do something with the unlabeled data? And then lastly, I can spend money on labeling itself. So let's say that the chunk here may be spent on labeling. I can also do other stuff, right? But the question is, what's the best distribution of getting your money spent and getting an agent that performs as well as possible? Okay, I also have to spend some money on training the actual system. But well, it's open AI, they have the compute. So the way that this paper does it, which I find is quite cool, and is a good recipe for sort of future applications of if you have any problem that's in this domain, you might want to give this approach here a try. They are by no means the first people who do it like this. But they are the first to show that this significantly reduces your cost in getting a capable Minecraft agent. And it's such a general method that it's pretty much applicable almost anywhere where you have this type of problem. So what are they doing? They recognize a simple fact, namely that if you have a video sequence, video, frame, frame, frame, frame, right, and if you want to infer kind of what's the next action, let's say, this is the past, right, you are here, and you want to infer what is the next action that the agent is taking, essentially, that requires you to learn from the past to look back into the past, right, determine the next actions, although regressive, it's a causal model and you know, what you essentially need to do if you let's say you watch a video of someone playing, you have to predict what's the next action, what's the next mouse movement, what's the next key press, you have to understand what they're thinking, you have to sort of look ahead like what might they want to do next, right, and then you can sort of predict the next action. This paper recognizes it's much simpler. If you already have the entire video sequence of past and future frames, to then from all of this, look back and forward, so you integrate all the information in hindsight, you can determine much more easily what action was in between those two frames, right, because you see the future, you see the effects of the action, you might even see a little bit ahead of what the person, you know, is actually doing, and then you might infer their plans and so on, so that is a much easier task to infer the action from the hindsight situation than doing for the action just from the causal situation. And this is the basis of their method. We've seen this in other places before. I've once analyzed a talk by Andrej Karpati on Tesla labeling, and they're doing exactly the same thing. They're saying, wait, if you actually have the whole video sequence, and the car is hidden and then appears again, right, if you look back in hindsight, you can determine much more easily where that car was the entire time. Same idea here. So what are they doing? They are doing two things. They're collecting labeled data first in two different ways. So the first way they collect labeled data is they simply tell contractors, what color is good here, they tell contractors to play the game, as we said, they sit them down, and they play for 2000 hours of video game, 2000 hours of Minecraft, they just play it while their key presses and their mouse movements are all recorded, right? So that, sorry, that gives you a data set where you can train a system. Now you could run sort of behavior cloning directly on that system and try to get a good agent out of that labeled data. But no, they actually train this purple system right here. So they train a system that takes into account future and past in a given window, and then tries to determine the action of one of the frames in the middle. They call this the inverse dynamics model. Now they have now a model that you can't really build an agent with it because the agent can never see the future. But what you can do is you can go out into the internet and you can collect unlabeled data. YouTube, in case you have noticed, happens to be full of Minecraft videos, even I made a Minecraft video. So you know, you can go out and you can collect tons and tons and tons of Minecraft data. The only thing they have to do is they have to collect what they call clean data. So very often there is like a streamer in the picture, like, you know, me right here. So this is not sorry, this is not a clean paper review video. It's actually it has me inside of it, or there'd be like a subscribe button somewhere or some something like this. So they also collect a bunch of labeled data from from crowd workers to classify frames to clean Minecraft footage, which is Minecraft footage that has just the Minecraft interface, including the hot bar and the health bars and so on. But not any of the streamer information and is in survival mode. If you don't know what that means, just forget about it. It's one of the game modes of Minecraft that most people play in the others will be like creative mode. And I don't even know what exists. Other than that. So you want to go, you want to collect frame labels to classify clean data, you can do that pretty cheaply. In fact, I think they from the labeled data, they I think they run them through a resonant pre trained resonant and then just train a support vector machine to classify clean frames from like non non clean frames, which, you know, is pretty simple, but it works. So all the better for that. But then they essentially have here 70,000 hours of clean, but unlabeled data. And then the trick is they just use this inverse dynamic model to label the unlabeled data to have pseudo labels. Now this obviously requires you to have very, very accurate inverse dynamics model. And in fact, they do verify and and I believe they get over like a 90% accuracy in inferring the actions. So that's kind of a requirement. But once you have that, you can pseudo label all of this unlabeled video data. So you label that's what they say here, you label the videos with the inverse dynamics model, and that leads you to 70,000 hours of labeled data. And then you can do the behavior cloning, then you can run your classic, it's not reinforcement learnings, behavior cloning, essentially learning from expert demonstrations, but they're only pseudo expert demonstrations, because the labels have been essentially propagated from a smaller set of expert demonstrations. They will show in their results that this strategy is like way cheaper, you have to collect a lot less labeled data than if you were to go the route of behavior cloning directly. And I think that's the thing that's applicable throughout sort of many, many, many problems. Not only that they can, you know, so they can then train this behavior cloning model, this causal model right here. And then they can do multiple things, they can fine tune it on like subsets of their data. They can also fine tune it with reinforcement learning to achieve certain goals. And this all becomes possible right here, because this prior, just the prior of movement, right, these videos that they collect right here, they have no goal. It's just people playing the game. But this prior of how to move in this world of things that you can do and skills acquired is so versatile that then you can do like reinforcement learning, given a certain task with some regularization, actually get some good results. So we're going to dive into a little bit more detail what they do right here. But this is the basic idea. It's very simple on its face. But it is very, very effective. Now one thing I have to point out here is that they keep using this term foundation model. So they have different models right here, right? They have this inverse dynamics model here, they have the classifier for the clean data. And the model that they train, the behavior cloning model that they train on the pseudo labeled data, the large data, that's what they call the foundation model. I don't know how much money Stanford has given them in order to call it the foundation model. But this is essentially the pre trained model that then you can either use for zero shot application or you can use for fine tuning or further behavior cloning on sub data sets. But it's just like I have nothing. Okay, I like the name is a different debate, but just the amount of times if you read this paper, the amount of times they make sure to use the name foundation model or the word foundation is it's a bit over the top, I have to admit, you know, but to each their own. So if you don't know, like the GPT series of models and so on, then it might be a good time to look up on on that I have several videos on that. I'll just continue and assume that you kind of know what's going on in the causal or autoregressive natural language modeling world. One notable difference right here if we're talking about causal models, non causal models and so on is that here they don't go from the same domain to the same domain. So this is not a because GPT three is like text as an input and then text as an output. So you can sort of do this autoregressive thing. In this case, it's frame data as input like short video sequences, and as an output, you get actions. So it's not predicting the next frames or anything like this. But you do get the actions as an output. And then you have to work together with the game or with the simulator in order to actually get sequence. Alright, so what what should we dive in first, maybe the model architecture would be another good place or a good place to start. So I already told you that the labeling model of clean versus non clean data is a support vector machine on pre trained features. That's pretty simple. The inverse dynamics model, the purple one right here, and the behavior cloning model, the green one are essentially the same model, except one gets to look into the future and one does not. So how does that model look? Let me see where I get some space. Again, let's say you have frames of video. So I'm going to draw them like this. Okay, I probably need to draw a lot of them. So yada, yada, yada, yada. Okay, this was not a good idea. I hope you can recognize these are sequential frames of videos. I'm only going to draw the inverse dynamic model for the behavior cloning model exactly the same except it can't look into the future. So let's say we want to predict the action for this frame right here. What we do first is, so at the end we want we want the action. So what we do first is we run over the thing with a 3d convolution. The convolution usually is in 2d on images. But if you extend the same principle to 3d, you can you can also convolve in time. So there's a 3d convolution, I believe it's a kernel size of five in the time domain. So that would be a five by k by k filter that runs over the individual like every five neighboring frames and runs over them in a convolution fashion. So this runs over the whole thing. So what you get are essentially another sequence of frames because if you know from a conv net, if I let it run over a sequence or over an image, I get out an image, you might have different amount of channels and so on, which is the same here. I've not drawn the channels actually every image here is one channel but imagine this in four dimension. Okay. So you have this, then I believe each of these frames is passed individually through a feed forward layer or a sequence of feed forward layer so that you get embeddings. So each frame now has just single vector embeddings or this is not frame per se. So each one of these frames is obviously a combination of five frames around it. But each combination of five frames and they are overlapping, of course, you know, if you see how convolutions work. Each one of those is made into an embedding and then obviously how else you have a big transformer model. Big transformer model that processes all of this kind of stuff and spits out, you know, essentially whatever you want in this case, the action to be taken. They have a bit of an action encoding scheme, which is hierarchical, which I don't want to go into because it's very Minecraft specific, but they do something that the amount of classes that you have here doesn't blow up but also excludes like mutually exclusive actions and so on. But that's very Minecraft specific. This part right here is essentially the video part of video pre training. Like that's how you handle or that's how they handle video data by doing convolutions in time mapping to embeddings, then feeding into a transformer model. If you don't know what a transformer model is, I have a good video. It's called Attention is All You Need and you can learn all about it there. So the results are pretty astounding, as I said. Here you can see on the left, you see the performance of the inverse dynamic model. You can see that the accuracy in the accuracy in actually do they get the correct actions out of their model? Like can their model that gets to look into the future predict the correct actions? And yes, it is actually it is actually pretty good. You can see the accuracies rising up right here. The mouse distance also getting better and better. And here is the here is the good one I say, here is one of the main results. So you can see the validation loss of the model. Now if you were to use just behavioral cloning on the contractor data right here is this is a function of data set size. If you were to just use the contractor data, you would improve, but you get much better loss if you use the inverse dynamics model, because it gets to look into the future, right? It's fairly, but want to say it's fairly intuitive that if you do get to look into the future, you become much better at predicting these things. So that it makes total sense to train the inverse dynamics model first and use that to label the data. So now we have some results right here, and they always give the results in sort of this form. So at the bottom, you have something like you know, the progress of training. And these lines represent different items. So for example, this one right here is a crafting table. If you remember a crafting for a crafting table, you need to go collect wood, you need to craft wood into planks, and then you need to craft the planks into the crafting table. So all of this requires movement in the real world, holding the action to punch. Yes, you punch a tree in Minecraft, then opening the crafting menu, crafting twice by arranging different items in different ways. So they tell you sort of how often these things happen, or you know, how much the agent achieves these things. So this line here would be representing of this item right here. Obviously, the higher it goes, the more the better the agent is at crafting that thing, or the more often the agent actually has achieved crafting that thing during evaluation. So if we look at a few, yeah, a few more results, they then take that foundation model, and the way they call it, at some point, they call, they even call it foundation data, which I found funny. Just using the word foundation all the time. So they now take, oh, I can do this when I'm in the picture. So they can now take this foundation model. And as I said, they can just measure how often the agent achieves, either collects or crafts a given item. So the blue thing here is just the foundation model that they train, you know, just on this data, this data has no goal. It's just people playing Minecraft. They just put the agent into the world and they say, and they say, what can you achieve? Okay, it can achieve something like, well, what's that basic mining, basic mining, it just means, I guess it collects some blocks, pretty often, the blue bars here, logs pretty often planks, what kind of sort of often, but you can already see this is a log scale, by the way, right here. There are other agents that do it much, much better. So what are these other agents? Well, one of them, as you can see here, is fine tuned on the keyword early game. So they go to YouTube again, and they simply filter Minecraft videos by the ones that are also having the title or with the keyword early game, which are usually beginner tutorials that kind of show you how to get off the ground at the beginning, which for a model like this, if you fine tune on that, and the items that we have right here, they are very basic items. They're the items that you get at the very beginning of the game. So that data set is much more representative of that gameplay. And you can see that from the blue to the green bar, there's like one order of magnitude in some of these items, which is pretty huge. And then the last thing is they train, they collect another set of contractor data. And this time, they tell them to build a house. So in Minecraft, you can build a house, which is also one of the first things you'll do. But now it's not early game aimless, right, every YouTuber does whatever. Now every contractor is tasked to build a house. So we are now in the really behavior cloning setting with a goal. And yeah, that's what we do. So the data set is targeted towards building a house. And naturally, the items that you need to build a house, I guess the stone tools, yeah, it's pretty good to have stone tools, not necessary, but pretty good. But at least the like the wooden tools are also pretty handy when building a house. And you can see that all of the items that you need right here are much higher, there's like an increase of 213 X in crafting tables. All of this essentially means that if your data set is more appropriate, you'll get sort of more behavior like the data set, I guess. However, all of this is fine tuned or behavior cloned on top of the foundation model. So they first train that pre trained model, I keep saying foundation model myself, see that the marketing gets me. They train on this first thing. And then after that, on top of that, they either do the fine tuning to the early game data set or the fine tuning to the house building. Or as we shall see, they do reinforcement learning. So on top of I believe this is on top of the early game model, they now do fine tuning. So the early game model gets to somewhere, maybe here, I think it gets to like the stone tools, right? And then they do reinforcement learning, while giving rewards for collecting each each of the items in the sequence right here with different weights and so on. There's a fair bit of reward shaping going on right here. So I guess you can criticize that. But reward shaping has always been the case in Minecraft. People have done much harder reward shaping for Minecraft than this and they've never achieved anything, right? So the ability of this model to actually get to the diamond pickaxe over here is astounding. So this here is what happens. If you simply this, this, this plot right here is it's just flexing, right? It's pretty useless. If you just have a randomly initialized model, and you just do reinforcement learning with their reward shaping and all, you're at zero, all the lines are at zero, it achieves absolutely nothing. If you actually re reinforcement learn from that pre trained model that's been pre trained on just the full data set of Minecraft footage, you see that you get pretty far right you get even you get to the furnace actually right here, but the higher tools are still not in reach even after reinforcement learning. So if you then reinforcement learn from the early game model, so you do pre training, you do behavioral cloning on early game filtered keyword videos. And on top of that you do reinforcement learning with the reward shaping, you can see that you actually do get to diamonds and to the diamond pickaxe, which is you need three diamonds for in 2.5% of the evaluation runs. And keep in mind, as far as I understand, although I have not seen this in the paper, maybe it's in the appendix, or maybe I've missed it, but this is random seed. So the world, as I said, is different for every episode. That's really the hard part right here, that the world is so complex and different. So that is is pretty cool. Now we can draw a bunch of conclusions from this, I think, you know, the fact that there is such the fact that there is a big difference between this and this or this and the bottom two, it does speak highly for, you know, this approach, where you want to have a lot of labeled data in order to pre train a model. And on the basis of that, you can do reinforcement learning. And from before, we know that it's way cheaper if you first collect small set of labeled data, use the fact that you can look into the future to label unlabeled data and then use that as your bigger label data set. However, there is also a difference between this one and this one right here, right? Because just pre training, and then doing reinforcement learning doesn't seem to be enough to reach the highest tools right here. It also pays off to really have an appropriate pre training. So when you do further pre training, essentially on early game footage, then that is much more conducive on the way to getting a diamond pickaxe, which I guess to some Minecraft players is late game, but to most is still also kind of early game to get your first diamond tools. And that is also pretty, pretty interesting. So it is not, it is not the case that you can just go out and get any sort of data that you want, obviously, more is always better. But having the appropriate data is also very, very important. So whatever you can do to get that and maybe add that then on top of the of the full random data, that's kind of the best strategy, at least from this from this chart right here. So they do a bunch of more experiments right here to, for example, see the effect of the 3d convolutions, see the effect of the inverse dynamics model of the quality of that, like what if you train it better or with more data and so on. But essentially, that's the paper in a nutshell. And yeah, as I said, it's pretty simple. It's certainly not something that no one has done before in principle. However, it is a pretty good demonstration of something in practice like making a capable Minecraft agent. No one has done that. This is quite a significant jump. I have, I believe. And the idea here, not only to do that, because I'm pretty sure open AI could have just paid for like tons and tons of data in order to do that. But in order like doing that, while giving us a recipe, you know, here is how you can kind of save a ton of money. Again, they're not the first to do it. But they demonstrate quite nicely that in a situation like this, it can make quite the difference. Yeah, and lastly, I do believe they make their model available. There is a there's the competition Mine RL. If you're interested in that, that's a Minecraft reinforcement learning competition. And you can take their model and you can fine tune that at your heart's content. So you don't have to do that whole video pre training because that's like the training itself is pretty expensive. I thought somewhere. So the inverse Okay, I've lost that. But I think the inverse dynamics model training was already quite a bit vroom vroom. But then let's see fine tuning. I'm not gonna find it. I'm not gonna find it. Oh, there we go. Oh, it took nine days on 720 v 100 GPUs. That's a big number. That's a lot of v 100 GPUs. Geez. Yeah, so they've done that for you. You can take their model, you can fine tune it, you can modify it and so on. So please do that. And if you have if you happen to have spare GPUs, you can you can send me you can send them to me. No problem. All right, that was it for me. Stay hydrated. See you around. корп
 Hi there, today we'll talk about video pre-training, learning to act by watching unlabeled online
 videos.
 This is by a team out of OpenAI and is the first system that successfully crafts a diamond
 pickaxe in Minecraft.
 So apart from humans, obviously.
 So Minecraft has been sort of a test bed for reinforcement learning algorithms all of these
 years.
 But it's notoriously hard if you don't know what Minecraft is, even if you do, it is a
 hard, hard problem.
 So you're in this open world, and you can essentially deconstruct any block.
 So the first thing is you want to punch a tree, right?
 This gets you wood, and then you want to craft that wood to these logs, and you will craft
 these logs to that table.
 Crafting is done in a menu like this, like the top right here.
 The crafting interface means that you have to arrange the items you have to create new
 items.
 There is a recipe book, but sometimes you also have to know what you're doing.
 Then you walk around in this open world.
 This is not a very competent player right here.
 And you can see there's a menuing interface and so on.
 So this is hard, even if you have like predefined actions.
 But if you don't, and you just want to use the mouse and the keyboard as this system
 does right here, it becomes nearly impossible.
 There is a progression of things to build, you know, given wooden planks and crafting
 tables and sticks, sticks are missing here, you can build the wooden pickaxe with the
 wooden pickaxe.
 You can you can use that to mine cobblestone with the cobblestone.
 You can then build a stone pickaxe with the stone pickaxe.
 You can go even further and further.
 Here you can see a bunch of stuff that this agent learns.
 This is tapped on mute.
 Well I did it.
 In any case, this agent here learned to raid a village like to look around in a village.
 You can see just how complex these worlds are right there are these villages, it's an
 open world, the terrain is randomly generated.
 And it's a completely new terrain every single time you start the game.
 And this is why it's so incredible.
 Look at the amount of the items in this in this chest right here.
 So just to give you sort of an idea of now it's an idea of how difficult this game is.
 No agent has yet managed to successfully kind of progress through these things, especially
 no agent that is not like has hard coded things in in it like that.
 So here would be the full progression to the diamond pickaxe week before we saw you get
 into the stone pickaxe, you can use the stone pickaxe to mine iron ore.
 From that you can smell the iron ore in a furnace to produce iron you need something
 that's burnable.
 From that you can craft an iron pickaxe and with the iron pickaxe you can mine the diamond
 if you find the diamond.
 Now the episodes, the episodes here run for 10 minutes, I believe or 15.
 We have tried this so on our discord we discussed this paper and thank you very much to everyone
 who participated.
 I've tried it.
 And it was pretty hard.
 I got to two diamonds once within within two diamonds within 10 minutes or 15.
 And the diamond pickaxe needs three diamonds.
 So for a human it's already pretty hard for a system like this.
 It is actually it's pretty darn hard.
 So you can see it right here.
 If you were to train this from a randomly initialized model just with reinforcement
 learning, it doesn't work.
 So the entire question is, how do we get this to work in a like in the cheapest way possible?
 And that's where this paper comes in.
 So I think the fundamental question, even though it's called video, video pre training,
 which essentially means we have a model that's pre trained on videos.
 The main question is here, where do we spend our money most effectively?
 So let's say we have a bunch of money, right?
 So let's say here is a bucket.
 Well, it's more like a box, okay.
 And the box is the box has dollars in it.
 Now these aren't as worth as much anymore as they used to in the good old days.
 But in any case, how would you spend that money, right?
 You can go and collect label data, for example.
 So you can go to contractors and they can play the game.
 All right, so oopsie.
 You can tell them you can say, okay, this much of my money, that's kind of playing.
 I pay people to play the game, I record their actions, right?
 So and then I have a video together with the labels, the labels being the inputs of the
 humans.
 And then I have at least a data set where I can do something like behavior cloning,
 right?
 The other thing could be I could spend the money on getting unlabeled data.
 Now if I spend the same money on unlabeled data, let's say this this slice right here,
 unlabeled.
 I suck at writing.
 I'm going to get much more data, but they don't have labels.
 So can I do something with the unlabeled data?
 And then lastly, I can spend money on labeling itself.
 So let's say that the chunk here may be spent on labeling.
 I can also do other stuff, right?
 But the question is, what's the best distribution of getting your money spent and getting an
 agent that performs as well as possible?
 Okay, I also have to spend some money on training the actual system.
 But well, it's open AI, they have the compute.
 So the way that this paper does it, which I find is quite cool, and is a good recipe
 for sort of future applications of if you have any problem that's in this domain, you
 might want to give this approach here a try.
 They are by no means the first people who do it like this.
 But they are the first to show that this significantly reduces your cost in getting a capable Minecraft
 agent.
 And it's such a general method that it's pretty much applicable almost anywhere where you
 have this type of problem.
 So what are they doing?
 They recognize a simple fact, namely that if you have a video sequence, video, frame,
 frame, frame, frame, right, and if you want to infer kind of what's the next action, let's
 say, this is the past, right, you are here, and you want to infer what is the next action
 that the agent is taking, essentially, that requires you to learn from the past to look
 back into the past, right, determine the next actions, although regressive, it's a causal
 model and you know, what you essentially need to do if you let's say you watch a video of
 someone playing, you have to predict what's the next action, what's the next mouse movement,
 what's the next key press, you have to understand what they're thinking, you have to sort of
 look ahead like what might they want to do next, right, and then you can sort of predict
 the next action.
 This paper recognizes it's much simpler.
 If you already have the entire video sequence of past and future frames, to then from all
 of this, look back and forward, so you integrate all the information in hindsight, you can
 determine much more easily what action was in between those two frames, right, because
 you see the future, you see the effects of the action, you might even see a little bit
 ahead of what the person, you know, is actually doing, and then you might infer their plans
 and so on, so that is a much easier task to infer the action from the hindsight situation
 than doing for the action just from the causal situation.
 And this is the basis of their method.
 We've seen this in other places before.
 I've once analyzed a talk by Andrej Karpati on Tesla labeling, and they're doing exactly
 the same thing.
 They're saying, wait, if you actually have the whole video sequence, and the car is hidden
 and then appears again, right, if you look back in hindsight, you can determine much
 more easily where that car was the entire time.
 Same idea here.
 So what are they doing?
 They are doing two things.
 They're collecting labeled data first in two different ways.
 So the first way they collect labeled data is they simply tell contractors, what color
 is good here, they tell contractors to play the game, as we said, they sit them down,
 and they play for 2000 hours of video game, 2000 hours of Minecraft, they just play it
 while their key presses and their mouse movements are all recorded, right?
 So that, sorry, that gives you a data set where you can train a system.
 Now you could run sort of behavior cloning directly on that system and try to get a good
 agent out of that labeled data.
 But no, they actually train this purple system right here.
 So they train a system that takes into account future and past in a given window, and then
 tries to determine the action of one of the frames in the middle.
 They call this the inverse dynamics model.
 Now they have now a model that you can't really build an agent with it because the agent can
 never see the future.
 But what you can do is you can go out into the internet and you can collect unlabeled
 data.
 YouTube, in case you have noticed, happens to be full of Minecraft videos, even I made
 a Minecraft video.
 So you know, you can go out and you can collect tons and tons and tons of Minecraft data.
 The only thing they have to do is they have to collect what they call clean data.
 So very often there is like a streamer in the picture, like, you know, me right here.
 So this is not sorry, this is not a clean paper review video.
 It's actually it has me inside of it, or there'd be like a subscribe button somewhere or some
 something like this.
 So they also collect a bunch of labeled data from from crowd workers to classify frames
 to clean Minecraft footage, which is Minecraft footage that has just the Minecraft interface,
 including the hot bar and the health bars and so on.
 But not any of the streamer information and is in survival mode.
 If you don't know what that means, just forget about it.
 It's one of the game modes of Minecraft that most people play in the others will be like
 creative mode.
 And I don't even know what exists.
 Other than that.
 So you want to go, you want to collect frame labels to classify clean data, you can do
 that pretty cheaply.
 In fact, I think they from the labeled data, they I think they run them through a resonant
 pre trained resonant and then just train a support vector machine to classify clean frames
 from like non non clean frames, which, you know, is pretty simple, but it works.
 So all the better for that.
 But then they essentially have here 70,000 hours of clean, but unlabeled data.
 And then the trick is they just use this inverse dynamic model to label the unlabeled data
 to have pseudo labels.
 Now this obviously requires you to have very, very accurate inverse dynamics model.
 And in fact, they do verify and and I believe they get over like a 90% accuracy in inferring
 the actions.
 So that's kind of a requirement.
 But once you have that, you can pseudo label all of this unlabeled video data.
 So you label that's what they say here, you label the videos with the inverse dynamics
 model, and that leads you to 70,000 hours of labeled data.
 And then you can do the behavior cloning, then you can run your classic, it's not reinforcement
 learnings, behavior cloning, essentially learning from expert demonstrations, but they're only
 pseudo expert demonstrations, because the labels have been essentially propagated from
 a smaller set of expert demonstrations.
 They will show in their results that this strategy is like way cheaper, you have to
 collect a lot less labeled data than if you were to go the route of behavior cloning directly.
 And I think that's the thing that's applicable throughout sort of many, many, many problems.
 Not only that they can, you know, so they can then train this behavior cloning model,
 this causal model right here.
 And then they can do multiple things, they can fine tune it on like subsets of their
 data.
 They can also fine tune it with reinforcement learning to achieve certain goals.
 And this all becomes possible right here, because this prior, just the prior of movement,
 right, these videos that they collect right here, they have no goal.
 It's just people playing the game.
 But this prior of how to move in this world of things that you can do and skills acquired
 is so versatile that then you can do like reinforcement learning, given a certain task
 with some regularization, actually get some good results.
 So we're going to dive into a little bit more detail what they do right here.
 But this is the basic idea.
 It's very simple on its face.
 But it is very, very effective.
 Now one thing I have to point out here is that they keep using this term foundation
 model.
 So they have different models right here, right?
 They have this inverse dynamics model here, they have the classifier for the clean data.
 And the model that they train, the behavior cloning model that they train on the pseudo
 labeled data, the large data, that's what they call the foundation model.
 I don't know how much money Stanford has given them in order to call it the foundation model.
 But this is essentially the pre trained model that then you can either use for zero shot
 application or you can use for fine tuning or further behavior cloning on sub data sets.
 But it's just like I have nothing.
 Okay, I like the name is a different debate, but just the amount of times if you read this
 paper, the amount of times they make sure to use the name foundation model or the word
 foundation is it's a bit over the top, I have to admit, you know, but to each their own.
 So if you don't know, like the GPT series of models and so on, then it might be a good
 time to look up on on that I have several videos on that.
 I'll just continue and assume that you kind of know what's going on in the causal or autoregressive
 natural language modeling world.
 One notable difference right here if we're talking about causal models, non causal models
 and so on is that here they don't go from the same domain to the same domain.
 So this is not a because GPT three is like text as an input and then text as an output.
 So you can sort of do this autoregressive thing.
 In this case, it's frame data as input like short video sequences, and as an output, you
 get actions.
 So it's not predicting the next frames or anything like this.
 But you do get the actions as an output.
 And then you have to work together with the game or with the simulator in order to actually
 get sequence.
 Alright, so what what should we dive in first, maybe the model architecture would be another
 good place or a good place to start.
 So I already told you that the labeling model of clean versus non clean data is a support
 vector machine on pre trained features.
 That's pretty simple.
 The inverse dynamics model, the purple one right here, and the behavior cloning model,
 the green one are essentially the same model, except one gets to look into the future and
 one does not.
 So how does that model look?
 Let me see where I get some space.
 Again, let's say you have frames of video.
 So I'm going to draw them like this.
 Okay, I probably need to draw a lot of them.
 So yada, yada, yada, yada.
 Okay, this was not a good idea.
 I hope you can recognize these are sequential frames of videos.
 I'm only going to draw the inverse dynamic model for the behavior cloning model exactly
 the same except it can't look into the future.
 So let's say we want to predict the action for this frame right here.
 What we do first is, so at the end we want we want the action.
 So what we do first is we run over the thing with a 3d convolution.
 The convolution usually is in 2d on images.
 But if you extend the same principle to 3d, you can you can also convolve in time.
 So there's a 3d convolution, I believe it's a kernel size of five in the time domain.
 So that would be a five by k by k filter that runs over the individual like every five neighboring
 frames and runs over them in a convolution fashion.
 So this runs over the whole thing.
 So what you get are essentially another sequence of frames because if you know from a conv net,
 if I let it run over a sequence or over an image, I get out an image, you might have
 different amount of channels and so on, which is the same here.
 I've not drawn the channels actually every image here is one channel but imagine this
 in four dimension.
 Okay.
 So you have this, then I believe each of these frames is passed individually through a feed
 forward layer or a sequence of feed forward layer so that you get embeddings.
 So each frame now has just single vector embeddings or this is not frame per se.
 So each one of these frames is obviously a combination of five frames around it.
 But each combination of five frames and they are overlapping, of course, you know, if you
 see how convolutions work.
 Each one of those is made into an embedding and then obviously how else you have a big
 transformer model.
 Big transformer model that processes all of this kind of stuff and spits out, you know,
 essentially whatever you want in this case, the action to be taken.
 They have a bit of an action encoding scheme, which is hierarchical, which I don't want
 to go into because it's very Minecraft specific, but they do something that the amount of classes
 that you have here doesn't blow up but also excludes like mutually exclusive actions and
 so on.
 But that's very Minecraft specific.
 This part right here is essentially the video part of video pre training.
 Like that's how you handle or that's how they handle video data by doing convolutions in
 time mapping to embeddings, then feeding into a transformer model.
 If you don't know what a transformer model is, I have a good video.
 It's called Attention is All You Need and you can learn all about it there.
 So the results are pretty astounding, as I said.
 Here you can see on the left, you see the performance of the inverse dynamic model.
 You can see that the accuracy in the accuracy in actually do they get the correct actions
 out of their model?
 Like can their model that gets to look into the future predict the correct actions?
 And yes, it is actually it is actually pretty good.
 You can see the accuracies rising up right here.
 The mouse distance also getting better and better.
 And here is the here is the good one I say, here is one of the main results.
 So you can see the validation loss of the model.
 Now if you were to use just behavioral cloning on the contractor data right here is this
 is a function of data set size.
 If you were to just use the contractor data, you would improve, but you get much better
 loss if you use the inverse dynamics model, because it gets to look into the future, right?
 It's fairly, but want to say it's fairly intuitive that if you do get to look into the future,
 you become much better at predicting these things.
 So that it makes total sense to train the inverse dynamics model first and use that
 to label the data.
 So now we have some results right here, and they always give the results in sort of this
 form.
 So at the bottom, you have something like you know, the progress of training.
 And these lines represent different items.
 So for example, this one right here is a crafting table.
 If you remember a crafting for a crafting table, you need to go collect wood, you need
 to craft wood into planks, and then you need to craft the planks into the crafting table.
 So all of this requires movement in the real world, holding the action to punch.
 Yes, you punch a tree in Minecraft, then opening the crafting menu, crafting twice by arranging
 different items in different ways.
 So they tell you sort of how often these things happen, or you know, how much the agent achieves
 these things.
 So this line here would be representing of this item right here.
 Obviously, the higher it goes, the more the better the agent is at crafting that thing,
 or the more often the agent actually has achieved crafting that thing during evaluation.
 So if we look at a few, yeah, a few more results, they then take that foundation model, and
 the way they call it, at some point, they call, they even call it foundation data, which
 I found funny.
 Just using the word foundation all the time.
 So they now take, oh, I can do this when I'm in the picture.
 So they can now take this foundation model.
 And as I said, they can just measure how often the agent achieves, either collects or crafts
 a given item.
 So the blue thing here is just the foundation model that they train, you know, just on this
 data, this data has no goal.
 It's just people playing Minecraft.
 They just put the agent into the world and they say, and they say, what can you achieve?
 Okay, it can achieve something like, well, what's that basic mining, basic mining, it
 just means, I guess it collects some blocks, pretty often, the blue bars here, logs pretty
 often planks, what kind of sort of often, but you can already see this is a log scale,
 by the way, right here.
 There are other agents that do it much, much better.
 So what are these other agents?
 Well, one of them, as you can see here, is fine tuned on the keyword early game.
 So they go to YouTube again, and they simply filter Minecraft videos by the ones that are
 also having the title or with the keyword early game, which are usually beginner tutorials
 that kind of show you how to get off the ground at the beginning, which for a model like this,
 if you fine tune on that, and the items that we have right here, they are very basic items.
 They're the items that you get at the very beginning of the game.
 So that data set is much more representative of that gameplay.
 And you can see that from the blue to the green bar, there's like one order of magnitude
 in some of these items, which is pretty huge.
 And then the last thing is they train, they collect another set of contractor data.
 And this time, they tell them to build a house.
 So in Minecraft, you can build a house, which is also one of the first things you'll do.
 But now it's not early game aimless, right, every YouTuber does whatever.
 Now every contractor is tasked to build a house.
 So we are now in the really behavior cloning setting with a goal.
 And yeah, that's what we do.
 So the data set is targeted towards building a house.
 And naturally, the items that you need to build a house, I guess the stone tools, yeah,
 it's pretty good to have stone tools, not necessary, but pretty good.
 But at least the like the wooden tools are also pretty handy when building a house.
 And you can see that all of the items that you need right here are much higher, there's
 like an increase of 213 X in crafting tables.
 All of this essentially means that if your data set is more appropriate, you'll get sort
 of more behavior like the data set, I guess.
 However, all of this is fine tuned or behavior cloned on top of the foundation model.
 So they first train that pre trained model, I keep saying foundation model myself, see
 that the marketing gets me.
 They train on this first thing.
 And then after that, on top of that, they either do the fine tuning to the early game
 data set or the fine tuning to the house building.
 Or as we shall see, they do reinforcement learning.
 So on top of I believe this is on top of the early game model, they now do fine tuning.
 So the early game model gets to somewhere, maybe here, I think it gets to like the stone
 tools, right?
 And then they do reinforcement learning, while giving rewards for collecting each each of
 the items in the sequence right here with different weights and so on.
 There's a fair bit of reward shaping going on right here.
 So I guess you can criticize that.
 But reward shaping has always been the case in Minecraft.
 People have done much harder reward shaping for Minecraft than this and they've never
 achieved anything, right?
 So the ability of this model to actually get to the diamond pickaxe over here is astounding.
 So this here is what happens.
 If you simply this, this, this plot right here is it's just flexing, right?
 It's pretty useless.
 If you just have a randomly initialized model, and you just do reinforcement learning with
 their reward shaping and all, you're at zero, all the lines are at zero, it achieves absolutely
 nothing.
 If you actually re reinforcement learn from that pre trained model that's been pre trained
 on just the full data set of Minecraft footage, you see that you get pretty far right you
 get even you get to the furnace actually right here, but the higher tools are still not in
 reach even after reinforcement learning.
 So if you then reinforcement learn from the early game model, so you do pre training,
 you do behavioral cloning on early game filtered keyword videos.
 And on top of that you do reinforcement learning with the reward shaping, you can see that
 you actually do get to diamonds and to the diamond pickaxe, which is you need three diamonds
 for in 2.5% of the evaluation runs.
 And keep in mind, as far as I understand, although I have not seen this in the paper,
 maybe it's in the appendix, or maybe I've missed it, but this is random seed.
 So the world, as I said, is different for every episode.
 That's really the hard part right here, that the world is so complex and different.
 So that is is pretty cool.
 Now we can draw a bunch of conclusions from this, I think, you know, the fact that there
 is such the fact that there is a big difference between this and this or this and the bottom
 two, it does speak highly for, you know, this approach, where you want to have a lot of
 labeled data in order to pre train a model.
 And on the basis of that, you can do reinforcement learning.
 And from before, we know that it's way cheaper if you first collect small set of labeled
 data, use the fact that you can look into the future to label unlabeled data and then
 use that as your bigger label data set.
 However, there is also a difference between this one and this one right here, right?
 Because just pre training, and then doing reinforcement learning doesn't seem to be
 enough to reach the highest tools right here.
 It also pays off to really have an appropriate pre training.
 So when you do further pre training, essentially on early game footage, then that is much more
 conducive on the way to getting a diamond pickaxe, which I guess to some Minecraft players
 is late game, but to most is still also kind of early game to get your first diamond tools.
 And that is also pretty, pretty interesting.
 So it is not, it is not the case that you can just go out and get any sort of data that
 you want, obviously, more is always better.
 But having the appropriate data is also very, very important.
 So whatever you can do to get that and maybe add that then on top of the of the full random
 data, that's kind of the best strategy, at least from this from this chart right here.
 So they do a bunch of more experiments right here to, for example, see the effect of the
 3d convolutions, see the effect of the inverse dynamics model of the quality of that, like
 what if you train it better or with more data and so on.
 But essentially, that's the paper in a nutshell.
 And yeah, as I said, it's pretty simple.
 It's certainly not something that no one has done before in principle.
 However, it is a pretty good demonstration of something in practice like making a capable
 Minecraft agent.
 No one has done that.
 This is quite a significant jump.
 I have, I believe.
 And the idea here, not only to do that, because I'm pretty sure open AI could have just paid
 for like tons and tons of data in order to do that.
 But in order like doing that, while giving us a recipe, you know, here is how you can
 kind of save a ton of money.
 Again, they're not the first to do it.
 But they demonstrate quite nicely that in a situation like this, it can make quite the
 difference.
 Yeah, and lastly, I do believe they make their model available.
 There is a there's the competition Mine RL.
 If you're interested in that, that's a Minecraft reinforcement learning competition.
 And you can take their model and you can fine tune that at your heart's content.
 So you don't have to do that whole video pre training because that's like the training
 itself is pretty expensive.
 I thought somewhere.
 So the inverse Okay, I've lost that.
 But I think the inverse dynamics model training was already quite a bit vroom vroom.
 But then let's see fine tuning.
 I'm not gonna find it.
 I'm not gonna find it.
 Oh, there we go.
 Oh, it took nine days on 720 v 100 GPUs.
 That's a big number.
 That's a lot of v 100 GPUs.
 Geez.
 Yeah, so they've done that for you.
 You can take their model, you can fine tune it, you can modify it and so on.
 So please do that.
 And if you have if you happen to have spare GPUs, you can you can send me you can send
 them to me.
 No problem.
 All right, that was it for me.
 Stay hydrated.
 See you around.
 корп