Can Wikipedia Help Offline Reinforcement Learning? (Author Interview)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper
#wikipedia #reinforcementlearning #languagemodels

Original paper review here: https://youtu.be/XHGh19Hbx48

Machel Reid and Yutaro Yamada join me to discuss their recent paper on langauge model pre-training for decision transformers in offline reinforcement learning.

OUTLINE:
0:00 - Intro
1:00 - Brief paper, setup & idea recap
7:30 - Main experimental results & high standard deviations
10:00 - Why is there no clear winner?
13:00 - Why are bigger models not a lot better?
14:30 - What’s behind the name ChibiT?
15:30 - Why is iGPT underperforming?
19:15 - How are tokens distributed in Reinforcement Learning?
22:00 - What other domains could have good properties to transfer?
24:20 - A deeper dive into the models' attention patterns
33:30 - Codebase, model sizes, and compute requirements
37:30 - Scaling behavior of pre-trained models
40:05 - What did not work out in this project?
42:00 - How can people get started and where to go next?

Paper: https://arxiv.org/abs/2201.12122
Code: https://github.com/machelreid/can-wikipedia-help-offline-rl
My Video on Decision Transformer: https://youtu.be/-buULmf7dec

Abstract:
Fine-tuning reinforcement learning (RL) models has been challenging because of a lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models on other domains (vision, language) when finetuned on offline RL tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms of both convergence speed and reward on a variety of environments, accelerating training by 3-6x and achieving state-of-the-art performance in a variety of tasks using Wikipedia-pretrained and GPT2 language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trained models for RL, but also inspires future work on sharing knowledge between generative modeling tasks of completely different domains.

Authors: Machel Reid, Yutaro Yamada, Shixiang Shane Gu

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hey, this is the interview part of the video, Can Wikipedia Help Offline Reinforcement Learning? If you haven't seen it, I've made a comprehensive review of this research paper in the previous video. So be sure to check that out. The authors that I speak to today are the authors of this paper. They've seen my review and they're ready to dive in and tackle all of my criticisms. It's a big privilege to have the authors on and to be able to ask them any questions. So please let me know how I'm doing. Let me know how I can improve these videos for you. And as always, if you like, leave a like, and I'll see you around. Bye. Hi, everyone. Today I'm here with Michelle Reed and Yutaro Yamada, who are the authors of the paper, Can Wikipedia Help Offline Reinforcement Learning? First of all, both of you, welcome, and thank you very much for being here and discussing the paper with me. Thank you for having me. So obviously, the basic ideas of the paper I've mentioned, what would interest me is just how would you pitch the paper? If you had to pitch the paper, let's say someone comes up to you at a poster presentation or something like this, what would be your initial pitch, like whatever, 30 second or a minute, the basics of what you do? I'll give it a shot. Let's see. So here in our paper, we look at seeing whether, say, Wikipedia or language retraining can help other sequence modeling tests. And in this case, we focus on offline reinforcement learning. And I found this to be personally like a pretty cool project because essentially, the reasons are not completely clear, to be honest. But we see that with this language retraining, we can actually see quite substantial gains in certain areas over like random initialization. And I think even more interesting is that these models manage to converge faster, which shows that there is some sort of information there that is helpful. And personally, I'm pretty interested in this line of research because it really begs the question, how are these seemingly unrelated tests similar? Is there a way to see how similar they are? And maybe even encourage a new paradigm for transfer learning where you don't even need conventionally related data. How did you? You mentioned it a little bit, why it's interesting. And I completely agree. And the results are astounding, I would say. How did you get the idea to do this? Because initially, if someone told me, you just pre-train something on language and then use it for reinforcement learning or something like this, you dismiss it quite quickly, let's say, of all the ideas that you could choose from. So did you have some indication that this could work or a hunch or did you just try it at some Saturday morning? How did it come about? Sort of a mix of all three. So I guess as a background, we have that, like say in multilingual learning, it's been demonstrated by a couple of papers now that say you can transfer an English BERT to a Spanish BERT, for example. Or you can add new languages to say a model where it wasn't pre-trained on those languages. Or even there's an experiment in the MBART paper, I think, where they have this ablation where they pre-train on six languages. And then they test on some unseen languages, if I remember correctly. And that works too. So in the multilingual setting, this sort of intuition has been demonstrated, though you could argue, oh, it's language to language. And then I was talking with the other author in this paper, Shane. One day we were just chatting and we ended up talking about pre-training for RL. And I was like, oh, there's no pre-training for RL. They haven't had their BERT moment or their GPT moment yet. And we were discussing. He was discussing the limitations. And then I was like, why don't we try doing a language model? And then it became sort of like the Saturday morning experimentation session, which you alluded to, which is that day I was like, OK, let me just try putting in a language model there and see what happens. And the initial results were actually quite surprising in a good way. So we decided to continue doing that. Oh, I was going to just add on to, I remember you and Marshall were saying that when Shane's first reaction was like, there's no way that's going to work. And that sort of thing. I don't think he was really excited about the idea. But when Marshall actually did experiments and showed the results, he was like really excited. And yeah. The basic concept here is, I think it is very simple. And therefore, the sort of the setup of the paper is very simple. You pre-train on this language modeling objective. And you make a point that it is the autoregressivity that might be somewhat important right here in what you do. And then there is this decision transformer on the right-hand side. Now, I don't know how much you've seen of my introductory video, but did I get anything wrong in the setup here? Or did you want to highlight a specific part of this? Why could language models be particularly useful for this kind of reinforcement learning offline? Offline reinforcement learning with decision transformers. Right. Yeah, I think you captured it pretty well. I guess we'll go deeper into maybe the reasons why this could work as we go deeper into the questions. But as a high-level idea, yeah. I think you captured it pretty well. I was always, just maybe as a side note, I was always a bit astounded by these decision transformers, by the whole approach of doing this as this sequence modeling with this fixed context size and these returns to go. And then I essentially say, well, I just want a really high return. Just get me there. It seems very special, but it seems to work. I don't know if you have any thoughts on this. Not necessarily related to your paper, but I do find it a very special model for reinforcement learning specifically. Yeah, for sure. Actually, I was experimenting with trying some higher returns. I don't think we included it in the paper. But sometimes, especially during early stages of training, you could get free returns almost by just using an artificially large returns to go value. And then suddenly, the model would get better at play time, for example. Yeah, I think it's pretty amazing, honestly. Maybe shows something about the power of transformers to gather ideas like states together and combine them in interesting ways. I think we can directly go a little into the results. Because as I said, the setup is quite simple. Now, you test on two different data sets. So just to remind people, we have the decision transformer, which serves as the baseline for what we're trying to do. That's a same model with the same technique and the same inputs, just not pre-trained on language. And then there is this, if I pronounce this correctly, chibi-T model that is the same size, but has been pre-trained on language. And then there's GPT-2, which is a lot larger and obviously has been pre-trained on language. And then you have some baselines over here that are just for offline reinforcement learning. Now, you mentioned that your models consistently outperform or the language pre-trained models consistently outperform the decision transformer. But one of my worries here was that the standard deviations, especially in this experiment, they seem ginormous. How can we be sure we're not just measuring? It's better in the bottom table right here, but on this DQN benchmark, how can we be sure we're not just measuring noise in these cases? I would say, well, A, we can't be sure. But I would say that the trends across experiments do tend to point towards a certain direction. And also, I'm generally a language person. So when I was coming to RL and I was saying, oh, wow, we just changed a random seed. And it changed by this much. It was quite surprising to me. But after running experiments many times, it seems the trends were towards one direction. But I guess we could clarify that with some significance tests and things like that. I think I was mentioning that the trend is in one direction. I think that's much more convincing than anything being inside or outside of some standard deviation. What surprised me also is that I think that's just a property of reinforcement learning as such. For example, the Qbert environment, all of a sudden, you see, for example, there are baselines that just fail. They're just nothing, right? But all of a sudden, these models also aren't as good. But then this model is really good. Like, how do you? And also in the bottom table, I think a lot of times, which model is better than which other model is all over the place. Sometimes these are better. Sometimes these are better. Do you have an explanation of what's going on here? Why is there such a, let's say, a diversity of which approach wins in which circumstance? No. But I would say this is pretty interesting. Now, again, I'm coming from a language perspective. And I'm sure an RL person could give you a much better explanation. But even when I was experimenting, I noticed for some environments, the transformer tended to do, even early on, the language pre-training tended to do significantly better than the, say, the not language pre-training models, or even the other models we have here. And this is just, honestly, it's my intuition. But I feel like some of these techniques are very specialized, or maybe very specialized to the sense that maybe we don't know exactly what it is. But there are some properties of the environments that really go nicely with certain techniques, but then don't go nicely with certain others. And it's sort of like this random puzzle game that's being played here. That was my intuition when I was playing with it. I was like, oh, wow, this is pretty weird, actually. But yeah, that's my intuition. Yeah, even if you look at a GPT2, a GPT columns, I think it varies across the environment as well. So I think that sort of speaks to it. I also feel in reinforcement learning, a lot of times these algorithms are almost designed with a problem in mind. They are formulated as these general algorithms. But I think a lot of times people go and they see, what's the problem? I felt like this, like go explore, that the first algorithm that solved Montezuma's revenge. I looked at it and I was like, you just essentially hard coded the game into the algorithm. Even with their, they had two versions, even with their non-human designed feature space, I was just like, you looked at what fails and you just hard coded the solution. And you just, I'm trying to tell me that this is a general, maybe something like this is happening here too, where people, they analyze what goes wrong in particular environments. And then they make an algorithm that would specifically address those problems. I find this to be, I find reinforcement learning to be an interesting field because it seems like it's so not solved yet. When we just look at your models, there is a discrepancy. First of all, I've noticed that a lot of times the GPT-2 here doesn't significantly, sometimes it outperforms, but oftentimes it doesn't significantly outperform the much smaller model. Do you have an intuition as to maybe what's, why don't we see a bigger benefit of large models here? You say somewhere it's over a hundred times larger. My intuition is, so like, I think with like the certain papers we've shown that like larger models can fit like larger amounts of data better. Maybe you can even extrapolate from those larger amounts of data better. But if we think about what we're transferring here, and it's not, again, it's not completely clear as of yet, but if we assume that it's say maybe a smaller set of features or properties rather than like language as a whole, but maybe like some properties of language, then we can maybe say that, okay, if GPT and GPT-2, despite their like very different sizes, have learned sort of the same sort of maybe some element of the structure, some notion of hierarchy or something like that, and they're both learned like relatively equally, so to say, then maybe size doesn't matter as much here given that we're fine tuning on the same like relatively small amount of like trajectory data. So that's what I think. Is it called GPT because it sounds like GPT? No. Okay. Because, well, it was sort of related, but chibi is like, it means like sort of small mini type of thing in Japanese. So it was like a joke because initially, so initially I was calling it chibi-lm actually, like when I was just referring to it because I needed a name, I couldn't write like the small pre-trained language model every time. And then Shane was like, you know what, let's make it chibi-t. So then that's what I think. And you mentioned that clip often, it performs a little bit worse. And to note, you only use the text encoder or sorry, the text model from clip, which is a sequence model like the other ones. And also there is I-GPT, image GPT, that performs a lot worse. We can see it in this table. It just gets nowhere, right? And you had some hypotheses, do you wanna maybe, especially for the image GPT, what is your hypotheses on why that is just kind of a failure case? Yeah, I think Yutaro can answer this one because he was like master running these experiments. Yeah, so well, I think the image, like the structure that's in the image, so image GPT is trained on basically you could unroll pixels from images. And I think the structure that's there in the image is really different from the structure that you've seen in language. And in a way that if you only have a static image, and if you only have pixels out there, it's really hard to even group, which pixels group together into a discrete, like unit of objects, like discrete, I guess discrete objects. First of all, I-GPT or image GPT sort of like has to figure out that sort of like discreteness like before you can actually has ability to transfer to these RL settings where it has more discrete structures. Yeah. So yeah, that's I think one of the main reasons why the current version of image GPT that are trained on static images are not really good at transferring from their domain to RL task. And I think if we can actually train the sequential modeling or sequential models for like a video data, where it'll be much easier to extract these like discreteness because if you only look at images or static images, it's really, and if you don't have any prior information about objects, like it's really hard to extract objects only from static images. But if you have a temporal dimension, if you have a video information, then it becomes much easier to extract these objects because if you look at like frame T and frame T plus one, you look at like pixels that transform from T and T plus one, there is a difference in terms of perspectives. So that sort of gives you a strong sense or strong cue regarding like which pixels group together. And that's a really difference I think that will make, eventually I think if we invest more into video research and if sequential modeling in the video domain, I think it'll be a really big difference. Though I think I'm really excited about like the future of like a structural modeling that uses a video. And I'm excited to see how the pre-training model on the video will be transferred to like a different domains like RL in the future. And possibly the sort of the direction into vector quantized models might also help a little bit because not working on, as you say, it's really hard to even get what pixels belong together. But if we had more of token-based approaches, maybe that could help decouple from the pixel level just a bit. But I guess that's just speculation by me. And one speculation I also had was with respect to your alignment modules right here. So you have these linear projections that try to make the token embeddings of the RL problem as close as possible to the token embeddings that were seen during language pre-training, which makes sense because you kind of get to reuse, let's say the paths that are already there for the language models. In your ablations, you show that these, it also works without them, which was good for me to see because sometimes it's little things like this that only make stuff work. But there is a difference between the distribution of language tokens, which is usually like a zip distribution or some sort of very heavy-tailed, but sharp distribution, and image tokens, which by construction tend to be more uniform, especially if you think like pixels, but also the vector quantized models there by design uniform. And with the RL problem, could it be that it's also a matter of how the tokens are distributed? Maybe the RL tokens are again, more zip-in distributed and that's why it might fit a lot better, or did you investigate the appropriateness of this, how the embeddings look like? No, we didn't actually look into how the embeddings looked like. That was like, we actually planned to do this because I think, personally, I think it would be really cool, for example, if we found out that it actually, these embeddings turned into a sentence or something like that. But I do agree with your hypothesis about maybe how the tokens are distributed or how frequent things are. And I think this also sort of relates to sort of the structure in language or like this natural tendency to express things in a certain way. And you may want to express certain concepts more often than others. And then there's also like sort of this conditional nature, like maybe only if this concept appears, which is represented by a certain set of tokens, then you wanna talk about this, which in a sense, you could say mirrors RL or like just any sort of activities that you would do. Versus image modeling, personally, I feel it's cool, like as a topic, but I also do feel it's very force in a sense. It doesn't feel very natural to me, if that makes sense. Do you feel that there are other disciplines that would transfer well to reinforcement learning? I don't know if you've thought about this. You do include language and images. So maybe you thought of even other things. There are, I don't know, protein modeling, genetic sequences, there is sound and so on. Do you have any hypotheses or any plans to try out other modalities? Yes, we do wanna try other things. I think like some interesting things, like in addition to what you mentioned, could even be like, this is a natural language, but it's usually grouped in together with like the NLP community, but like code, for example, or even like testing out different languages, simpler languages, controlling for complexity, really maybe even music. I definitely think speech could be something else to try as well, as you tarrow look at to video. I think there's so many things in sort of our, I don't know about saying like daily life, but there are a lot of things around us which sort of have like a natural sequential nature of things, and it would be interesting to see if somehow, especially in like a low data regime, if these things are able to transfer to each other well, and if they're like some maybe underlying principles, or maybe like some like biases that are learned that correspond to like a large majority of sequential data, or maybe certain types of sequential data and might also help us like group sequential data types, maybe learn more about how they relate to each other. And I think if we're able to do that, then I think we'd be able to study this even more in depth and maybe build models based on those findings. It's a pretty special world, right? That all our models converge from all the different modalities that even allow us to do things like this. I find it to be a very special time because it would not have been possible if all the image models are ConvNet, right? And all the speech models are somehow Fourier transformed, transformed some things, everything sort of converging to transformers. Some people might not like it, but it does enable sort of a bigger picture on what it means to process data, or if you wanna look at it like this. So these attention plots right here, I found to be very interesting. Now, to be clear, this, you say this is on Hopper. So this is one of these gym tasks, one of these continuous control tasks. Is this one particular sample or is this like an aggregate over the data set? Or how do we, what is displayed here? So this is an attention map basically given a single trajectory. A single one, okay. So it's a single trajectory, yeah. But we can assume it's kind of representative of kind of what happens in general. So I have made a bunch of observations here in my video, some of which you also state in the paper, for example, this structure of three, like the models often looking back three steps back, which makes total sense because the decision transformer input comes in these tuples of three, right? And I'm gonna guess, if I want to predict the next return to go, it's probably very related to the last one, especially if the reward is more sparse, I can just predict like the same number again, I'm gonna be correct most of the time. And maybe the same with actions, given that in the continuous control frame by frame, I don't wanna switch my action around too much, maybe, right? But it's a pace to look mostly at these things. What I found interesting is the image GPT had a sort of just a recency bias. Like it just seemed to look just two or three tokens back in time, which I think supports very well what you claimed that image modeling might be different from language modeling in that, yeah, it might be that the image transformer just sort of looks at a local neighborhood and then just goes on, doesn't care too much about big structure. I don't know, it's just hypotheses. And then I think the most shady thing I said was with respect to the randomly initialized decision transformer. So this would be the baseline model, a transformer that from scratch is trained on this RL data. And I claimed what we can also see this sort of pattern of three, but much more strongly than in something like GPT-2, which does have a more diffuse attention. So here it's really super duper hard attention. And I claimed that might hinder the model from learning proper connections between things in the future because it already kind of discards in the early layers, everything that would connect sort of a state and a reward. Does this come close to what you concluded or do you have like different insights into these attention maps or what's happening here? It's actually very, very close to what we were thinking after looking at these attention maps. I think one thing actually after watching your video that I didn't really notice until you pointed it out was like those yellow blocks of two. I didn't actually notice that they were actually two, which I think is actually pretty cool to see like maybe like for those ones that weights like two of them together, maybe with different weightings. But overall, I think the interesting thing is that it's pretty consistent. Like it doesn't necessarily change, like the patterns don't change significantly, which is sort of unlike language, for example, where you can see things, like generally there is a recency bias to some degree, but you can see things like depending on the token go like pretty far if it's like attending to similar tokens from far back. But then again, if you do think about it that way, you could argue like action representations would probably be similar to action representation, state to state representations and so on. So maybe actually the language models and even the randomly initialized model are mirroring that. Yeah, I found it to be very special how hard the attention patterns are is right here. But also there is always in distance of three rows, there is one that is just only looking at three steps back and six and nine and so on. And then the ones in between, there is one that has, as you say, that has two and one that even has like, it seems like almost it has three but just one is a bit stronger. It'd be interesting to figure out which one is which. I don't think I can tell from this thing, but yeah. So I think the one that's only looking at like three behind, if I remember correctly is the returns to go. And then the ones between that are, let's say the state representations and then the action. Yeah, so the order is basically world state action. Yeah, that makes a bit of sense. And I think the sort of the result right here, I think in the middle layer, it's really nicely shown that something like GPT, it will start to focus on maybe kind of the important things in the past. It will select some of them to focus on. And so no matter which time step, it will kind of look back at maybe what it determines to be important states, whereas the randomly initialized one, it will almost be like stuck in this mode of how it looks back. And so my question here, and you can clearly see it in the last layer in that in GPT-2, there's still this sort of focus and attention on maybe what it determines to be important things in the episode. And the other ones, they just have like a diffuse attention matrix. And my question would be, might it be possible that we could achieve the effect between let's say GPT-2 and the random one, like this benefit through a much simpler procedure of just kind of regularizing, just saying like, you know, don't make your attention so hard. Like make, you know, just kind of keep your options open. Try to look back a bit further. Don't try to be so sure yet. Is that, you know, is that something that's reasonable or do you think there's reason to discard that idea? I think it's reasonable to try, but I still do feel that I think the, if we do something like this, then maybe we again, fall into the trap of what we were like talking about earlier is like this essentially like putting a bandaid on like a very specific problem per se. But I think like the cool thing about transformers is they can learn a lot of different things. So I think if you say like with a language model, for example, it's an initialization, you can fine tune it however you'd like to. And I think it's more like flexible in that sense. Unless like say we were trying to tackle like a very specific issue, then I think, yeah, it would be for sure something to try. Like I think there's this recent paper for language mumbling by like Ofir Press from UW. And he, they were looking at like say how they can bias the like basically enforce a recency bias towards a language model and that like improves like extrapolation towards longer sequences and so on. So I think in this case in language modeling, it's like one specific task that they're trying to solve. But here, if we like just talk about like offline reinforcement learning, it's very, very broad. And I think, for example, if you tried like Ofir's trick in like say for pre-training BERT or something like that, now again, this is just conjecture, but I have a feeling it may not work as well given like there's, I would say a lesser, like there was also another paper by, I don't know who it was, but I think from Dhanthi Chen's group at Princeton recently about like the masking rate in BERT models and things like that and perplexity doesn't necessarily correlate with downstream performance and so on. So yeah, if we're tackling a specific task, I would say sure, but I think the one nice thing about the language model pre-training is how flexible it can be. Yeah, I was, I mean, I was the same. I'm probably, as you say, falling in the same trap that I criticized the field of reinforcement learning, say, you know, looking at one thing and saying, can I make up something that would just solve this one thing? Yeah, and I think, you know, the difference is also to clip, show a little bit that it's not just, I can't just do any architecture or anything. There might actually be something to language modeling. In this table, you specifically show that the language model pre-trained ones converge faster. And I had one question here, and that was that, how different is this code base? Like how much of the difference in convergence can I attribute to you just being better at implementing stuff? And how much is really due to these two things being pre-trained? Is it the same code base or did you re-implement or implement from scratch? I wish I could say I was like this amazing programmer that can make things so much more efficient, but no, we use the same code base. Yeah, so this is legit, legit speed up that is due to the pre-training. Nice. I guess like one caveat that mentioned like about GPT-2 is that the faster training speed is due to like faster conversions, even though it's pretty big. But like say when you're doing your roll-outs and stuff like that inference time, it is definitely slower as to be expected by a larger model. Yeah, that makes sense. I was also surprised because in reinforcement learning, usually the conventional wisdom is that it needs a lot of resources. And here you mentioned something like, you have a single V100 and you have a single V2, and the time here is, I mean, even for the decision transformers, it's a couple of hours. It's not I have to train on eight GPUs for a couple of days. I was just positively surprised by just sort of the requirements and this makes it more accessible. Yeah, I think that's the cool thing about offline RL. You just, well, you just have to like say fit a certain set of trajectories. And there've been like a lot of pretty efficient models recently as well. So yeah, I think it's when you get into the online setting then things get pretty like computationally expensive. You also mentioned that context size doesn't really matter. In fact, more context seems to make stuff worse a little bit, right? Like how significant this really is. But do you have an idea here? Is that, is it just because there's more noise or is there something wrong with the objective of the decision transformer? I think partially more noise. And two, I think because of like say the tasks that are tested in gym, it's like you see a teeter running for example, or you have like this hopper, which is literally just hopping. And those emotions are relatively repetitive. Like in Atari, for example, the context is, I think quite a bit larger. I don't remember exactly what the value was, but maybe like 50 or maybe even a bit bigger than that. But it's like, okay, for Atari, maybe you need more information because I guess like the actions that are being performed are more diverse and like sort of what can happen is more diverse, but then for these tasks, then maybe that much context is not as necessary. But this is just my intuition. Maybe an RL person would be able to give a better idea of why. So the last thing that was here very special is just the scaling behavior of these models, namely with the language model pre-training, you could scale to much larger models. Do you have a feeling of how that continues? Like does it continue dropping off and just not giving you returns anymore? Or would it eventually also say you have like a model that's too large and it would drop in performance again versus a smaller model? Because my hypothesis was that language modeling, you have infinite data essentially. So you can never overfit on the pre-training. And therefore, there might never be really an opportunity to overfit on a fine tuning data set. I don't know, do you have an intuition? I'm gonna guess, maybe you didn't wanna go up to too high parameter models. Yeah, for like computational reasons, but I do generally agree with you. Like if we have, I think if we have a decent initialization like from the like language modeling on say like, like quote unquote like infinite data, then I think we should be able to arguably at least retain the same performance or get like very close to it. Perhaps there is a time, like a point where it just gets too big that it starts overfitting, but I would say that would probably happen when it like not close to the parameters we tested. Now you, oh, sorry. So I think, oh yeah, sorry. So that's like one thing, one good thing about like offline RLs. So you can also collect a lot more trajectory data from just running agents and then train on offline data. So I think there's that perspective in this figure. Like we can also train like a larger model and larger trajectory data. And then if you have like a really good language initialization, then you can also try that sort of direction of thinking that way. Do you have an idea how that trades off? Like would I rather invest into pre-training my model on language data or would I rather invest into gathering more offline RL data? Personally, I think if you're working with a fixed, like say, okay, say if we fix the amount of offline RL data and say we're gonna like use that versus like designing like a better algorithm or something, I would say pre-train your language model. But then again, as we see with like GPT versus GPT experiment, making it that much bigger, like sure it does help, like by some margin, but it's not like that super significant. So based on that, if we're gonna assume that language transfer is only like a certain set of maybe limited properties to these RL tasks, then I would say, yeah, collect more RL data, I would say. You said at the beginning, you tried it out, you thought about it, it kind of worked out of, or initially you got some promising results. Was there ever a thing that didn't work? Like the something in this project you tried and just didn't work at all or it didn't work at first? Any sort of avenues you got stuck in? I would say that what was interesting was that the cosine loss that we added, especially like towards like later stages, everything sort of smooths out, but this more has to do with how fast the model converges. So that's actually, maybe we should have ablated this, but the cosine loss actually allows the model to converge much faster. And one thing that was interesting is especially in the early stages is that the cosine, so say we weren't using the cosine embedding loss initially, and we just saw like GPT and GPT, or GPT, and GPT was like quite a bit lower than GPT, but then like say GPT without this extra loss, and then GPT with the loss, GPT managed to catch up to GPT, which is like pretty mind blowing to me. So like something like that was interesting. I wouldn't say like a hiccup because it actually worked like pretty well, like straight off the bat, but it was pretty interesting to see. And another thing was without say like the positional embeddings, for example, I would, you would general, like I think we ablated this, but we would generally see like quite lower returns and things like that. So maybe even like the position transferred from language is also quite important. Is there anything else you'd like to get out about this paper? Can people get into this themselves? Your code, is it available? Yeah. So actually it's in the footnote of the first page. So yeah, I think this stuff personally is super interesting to see how we can transfer different sequence modeling tasks to each other, sort of unite. So like say one big model that handles all the sequences or something like that. Another thing that was actually pretty cool is with like the language modeling co-training that we did. When we did it, the language, like it was, we actually had a model that was able to language model and was able to handle trajectories at the same time. And like the language modeling performance didn't degrade significantly, which was also pretty cool because it means that we essentially have the capacity even at a small scale to do both of these tasks at once. And if we have like these models that are able to handle these separately, then it begs the question, okay, what can we do together? Like, can we model everything all together? Like basically I think with, what was it? The, like say like with multilingual pre-training that we have, it's sort of like until I guess, and for maybe like a few papers before that, we didn't really feed all languages just together at once and see what happens. And then on top of that, we see like, oh, we have like this zero-shot transfer. Whether it's truly zero-shot is a different question, but still it's pretty cool. And I think if we can sort of replicate that, say we have like, I don't know, a remotely related language modeling, like a domain and language. And if we fine tune on this domain and language, suddenly we can do like trajectory modeling on this domain that say has to do with what was talked about in language and things like that. Like it opens a new set of possibilities for maybe like generalization and just like zero-shot. I don't like using that word, but like that sort of performance in general, like these new behaviors and stuff. Cool, excellent. Well, Michelle and Jutaro, thank you very much for being here and sharing the projects. I hope to see you again very soon with more modalities and more. I think this is, I'm still amazed sort of by the results. I find them really cool and yeah, good luck in the future.不知道
 Hey, this is the interview part of the video,
 Can Wikipedia Help Offline Reinforcement Learning?
 If you haven't seen it, I've made a comprehensive review
 of this research paper in the previous video.
 So be sure to check that out.
 The authors that I speak to today
 are the authors of this paper.
 They've seen my review and they're ready to dive in
 and tackle all of my criticisms.
 It's a big privilege to have the authors on
 and to be able to ask them any questions.
 So please let me know how I'm doing.
 Let me know how I can improve these videos for you.
 And as always, if you like, leave a like,
 and I'll see you around.
 Bye.
 Hi, everyone.
 Today I'm here with Michelle Reed and Yutaro Yamada,
 who are the authors of the paper,
 Can Wikipedia Help Offline Reinforcement Learning?
 First of all, both of you, welcome,
 and thank you very much for being here
 and discussing the paper with me.
 Thank you for having me.
 So obviously, the basic ideas of the paper I've mentioned,
 what would interest me is just how would you pitch the paper?
 If you had to pitch the paper, let's say someone comes up
 to you at a poster presentation or something like this,
 what would be your initial pitch,
 like whatever, 30 second or a minute,
 the basics of what you do?
 I'll give it a shot.
 Let's see.
 So here in our paper, we look at seeing whether, say,
 Wikipedia or language retraining can help other sequence
 modeling tests.
 And in this case, we focus on offline reinforcement learning.
 And I found this to be personally
 like a pretty cool project because essentially, the reasons
 are not completely clear, to be honest.
 But we see that with this language retraining,
 we can actually see quite substantial gains
 in certain areas over like random initialization.
 And I think even more interesting
 is that these models manage to converge faster, which
 shows that there is some sort of information there
 that is helpful.
 And personally, I'm pretty interested
 in this line of research because it really
 begs the question, how are these seemingly unrelated tests
 similar?
 Is there a way to see how similar they are?
 And maybe even encourage a new paradigm for transfer learning
 where you don't even need conventionally related data.
 How did you?
 You mentioned it a little bit, why it's interesting.
 And I completely agree.
 And the results are astounding, I would say.
 How did you get the idea to do this?
 Because initially, if someone told me,
 you just pre-train something on language
 and then use it for reinforcement learning
 or something like this, you dismiss it quite quickly,
 let's say, of all the ideas that you could choose from.
 So did you have some indication that this could work or a hunch
 or did you just try it at some Saturday morning?
 How did it come about?
 Sort of a mix of all three.
 So I guess as a background, we have that,
 like say in multilingual learning,
 it's been demonstrated by a couple of papers now
 that say you can transfer an English BERT to a Spanish BERT,
 for example.
 Or you can add new languages to say a model where it wasn't
 pre-trained on those languages.
 Or even there's an experiment in the MBART paper,
 I think, where they have this ablation where they pre-train
 on six languages.
 And then they test on some unseen languages,
 if I remember correctly.
 And that works too.
 So in the multilingual setting, this sort of intuition
 has been demonstrated, though you could argue,
 oh, it's language to language.
 And then I was talking with the other author in this paper,
 Shane.
 One day we were just chatting and we ended up
 talking about pre-training for RL.
 And I was like, oh, there's no pre-training for RL.
 They haven't had their BERT moment or their GPT moment yet.
 And we were discussing.
 He was discussing the limitations.
 And then I was like, why don't we
 try doing a language model?
 And then it became sort of like the Saturday morning
 experimentation session, which you alluded to,
 which is that day I was like, OK,
 let me just try putting in a language model there
 and see what happens.
 And the initial results were actually
 quite surprising in a good way.
 So we decided to continue doing that.
 Oh, I was going to just add on to,
 I remember you and Marshall were saying
 that when Shane's first reaction was like,
 there's no way that's going to work.
 And that sort of thing.
 I don't think he was really excited about the idea.
 But when Marshall actually did experiments
 and showed the results, he was like really excited.
 And yeah.
 The basic concept here is, I think it is very simple.
 And therefore, the sort of the setup of the paper
 is very simple.
 You pre-train on this language modeling objective.
 And you make a point that it is the autoregressivity
 that might be somewhat important right here in what you do.
 And then there is this decision transformer
 on the right-hand side.
 Now, I don't know how much you've
 seen of my introductory video, but did I get anything wrong
 in the setup here?
 Or did you want to highlight a specific part of this?
 Why could language models be particularly
 useful for this kind of reinforcement learning offline?
 Offline reinforcement learning with decision transformers.
 Right.
 Yeah, I think you captured it pretty well.
 I guess we'll go deeper into maybe the reasons
 why this could work as we go deeper into the questions.
 But as a high-level idea, yeah.
 I think you captured it pretty well.
 I was always, just maybe as a side note,
 I was always a bit astounded by these decision transformers,
 by the whole approach of doing this as this sequence
 modeling with this fixed context size and these returns to go.
 And then I essentially say, well, I just
 want a really high return.
 Just get me there.
 It seems very special, but it seems to work.
 I don't know if you have any thoughts on this.
 Not necessarily related to your paper,
 but I do find it a very special model for reinforcement
 learning specifically.
 Yeah, for sure.
 Actually, I was experimenting with trying some higher
 returns.
 I don't think we included it in the paper.
 But sometimes, especially during early stages of training,
 you could get free returns almost
 by just using an artificially large returns to go value.
 And then suddenly, the model would get better at play time,
 for example.
 Yeah, I think it's pretty amazing, honestly.
 Maybe shows something about the power of transformers
 to gather ideas like states together and combine them
 in interesting ways.
 I think we can directly go a little into the results.
 Because as I said, the setup is quite simple.
 Now, you test on two different data sets.
 So just to remind people, we have the decision transformer,
 which serves as the baseline for what we're trying to do.
 That's a same model with the same technique
 and the same inputs, just not pre-trained on language.
 And then there is this, if I pronounce this correctly,
 chibi-T model that is the same size,
 but has been pre-trained on language.
 And then there's GPT-2, which is a lot larger
 and obviously has been pre-trained on language.
 And then you have some baselines over here
 that are just for offline reinforcement learning.
 Now, you mentioned that your models consistently outperform
 or the language pre-trained models consistently outperform
 the decision transformer.
 But one of my worries here was that the standard deviations,
 especially in this experiment, they seem ginormous.
 How can we be sure we're not just measuring?
 It's better in the bottom table right here,
 but on this DQN benchmark, how can we
 be sure we're not just measuring noise in these cases?
 I would say, well, A, we can't be sure.
 But I would say that the trends across experiments
 do tend to point towards a certain direction.
 And also, I'm generally a language person.
 So when I was coming to RL and I was saying, oh, wow,
 we just changed a random seed.
 And it changed by this much.
 It was quite surprising to me.
 But after running experiments many times,
 it seems the trends were towards one direction.
 But I guess we could clarify that with some significance
 tests and things like that.
 I think I was mentioning that the trend is in one direction.
 I think that's much more convincing than anything
 being inside or outside of some standard deviation.
 What surprised me also is that I think
 that's just a property of reinforcement learning as such.
 For example, the Qbert environment, all of a sudden,
 you see, for example, there are baselines that just fail.
 They're just nothing, right?
 But all of a sudden, these models also aren't as good.
 But then this model is really good.
 Like, how do you?
 And also in the bottom table, I think a lot of times,
 which model is better than which other model
 is all over the place.
 Sometimes these are better.
 Sometimes these are better.
 Do you have an explanation of what's going on here?
 Why is there such a, let's say, a diversity
 of which approach wins in which circumstance?
 No.
 But I would say this is pretty interesting.
 Now, again, I'm coming from a language perspective.
 And I'm sure an RL person could give you
 a much better explanation.
 But even when I was experimenting,
 I noticed for some environments, the transformer
 tended to do, even early on, the language pre-training
 tended to do significantly better than the, say,
 the not language pre-training models,
 or even the other models we have here.
 And this is just, honestly, it's my intuition.
 But I feel like some of these techniques
 are very specialized, or maybe very specialized to the sense
 that maybe we don't know exactly what it is.
 But there are some properties of the environments that really
 go nicely with certain techniques,
 but then don't go nicely with certain others.
 And it's sort of like this random puzzle game
 that's being played here.
 That was my intuition when I was playing with it.
 I was like, oh, wow, this is pretty weird, actually.
 But yeah, that's my intuition.
 Yeah, even if you look at a GPT2, a GPT columns,
 I think it varies across the environment as well.
 So I think that sort of speaks to it.
 I also feel in reinforcement learning,
 a lot of times these algorithms are almost designed
 with a problem in mind.
 They are formulated as these general algorithms.
 But I think a lot of times people go and they see,
 what's the problem?
 I felt like this, like go explore,
 that the first algorithm that solved Montezuma's revenge.
 I looked at it and I was like, you just
 essentially hard coded the game into the algorithm.
 Even with their, they had two versions,
 even with their non-human designed feature space,
 I was just like, you looked at what fails
 and you just hard coded the solution.
 And you just, I'm trying to tell me that this is a general,
 maybe something like this is happening here too,
 where people, they analyze what goes wrong
 in particular environments.
 And then they make an algorithm that would specifically
 address those problems.
 I find this to be, I find reinforcement learning
 to be an interesting field because it seems like
 it's so not solved yet.
 When we just look at your models,
 there is a discrepancy.
 First of all, I've noticed that a lot of times
 the GPT-2 here doesn't significantly,
 sometimes it outperforms, but oftentimes
 it doesn't significantly outperform the much smaller model.
 Do you have an intuition as to maybe what's,
 why don't we see a bigger benefit of large models here?
 You say somewhere it's over a hundred times larger.
 My intuition is, so like, I think with like the
 certain papers we've shown that like larger models
 can fit like larger amounts of data better.
 Maybe you can even extrapolate from those larger amounts
 of data better.
 But if we think about what we're transferring here,
 and it's not, again, it's not completely clear as of yet,
 but if we assume that it's say maybe a smaller set of
 features or properties rather than like language as a whole,
 but maybe like some properties of language,
 then we can maybe say that, okay, if GPT and GPT-2,
 despite their like very different sizes,
 have learned sort of the same sort of maybe some element
 of the structure, some notion of hierarchy
 or something like that, and they're both learned
 like relatively equally, so to say,
 then maybe size doesn't matter as much here given that
 we're fine tuning on the same like relatively small
 amount of like trajectory data.
 So that's what I think.
 Is it called GPT because it sounds like GPT?
 No.
 Okay. Because, well, it was sort of related,
 but chibi is like, it means like sort of small mini
 type of thing in Japanese.
 So it was like a joke because initially,
 so initially I was calling it chibi-lm actually,
 like when I was just referring to it
 because I needed a name,
 I couldn't write like the small pre-trained language model
 every time.
 And then Shane was like, you know what,
 let's make it chibi-t.
 So then that's what I think.
 And you mentioned that clip often,
 it performs a little bit worse.
 And to note, you only use the text encoder
 or sorry, the text model from clip,
 which is a sequence model like the other ones.
 And also there is I-GPT, image GPT,
 that performs a lot worse.
 We can see it in this table.
 It just gets nowhere, right?
 And you had some hypotheses, do you wanna maybe,
 especially for the image GPT,
 what is your hypotheses on why that is just
 kind of a failure case?
 Yeah, I think Yutaro can answer this one
 because he was like master running these experiments.
 Yeah, so well, I think the image,
 like the structure that's in the image,
 so image GPT is trained on basically
 you could unroll pixels from images.
 And I think the structure that's there in the image
 is really different from the structure
 that you've seen in language.
 And in a way that if you only have a static image,
 and if you only have pixels out there,
 it's really hard to even group,
 which pixels group together into a discrete,
 like unit of objects, like discrete,
 I guess discrete objects.
 First of all, I-GPT or image GPT
 sort of like has to figure out that sort of like discreteness
 like before you can actually has ability to transfer
 to these RL settings where it has more discrete structures.
 Yeah.
 So yeah, that's I think one of the main reasons why
 the current version of image GPT
 that are trained on static images
 are not really good at transferring
 from their domain to RL task.
 And I think if we can actually train
 the sequential modeling or sequential models
 for like a video data,
 where it'll be much easier to extract these like discreteness
 because if you only look at images
 or static images, it's really,
 and if you don't have any prior information about objects,
 like it's really hard to extract objects
 only from static images.
 But if you have a temporal dimension,
 if you have a video information,
 then it becomes much easier to extract these objects
 because if you look at like frame T and frame T plus one,
 you look at like pixels that transform from T and T plus one,
 there is a difference in terms of perspectives.
 So that sort of gives you a strong sense
 or strong cue regarding like which pixels group together.
 And that's a really difference I think that will make,
 eventually I think if we invest more into video research
 and if sequential modeling in the video domain,
 I think it'll be a really big difference.
 Though I think I'm really excited about like
 the future of like a structural modeling
 that uses a video.
 And I'm excited to see how the pre-training model
 on the video will be transferred
 to like a different domains like RL in the future.
 And possibly the sort of the direction
 into vector quantized models might also help a little bit
 because not working on, as you say,
 it's really hard to even get what pixels belong together.
 But if we had more of token-based approaches,
 maybe that could help decouple from the pixel level
 just a bit.
 But I guess that's just speculation by me.
 And one speculation I also had was with respect
 to your alignment modules right here.
 So you have these linear projections
 that try to make the token embeddings of the RL problem
 as close as possible to the token embeddings
 that were seen during language pre-training,
 which makes sense because you kind of get to reuse,
 let's say the paths that are already there
 for the language models.
 In your ablations, you show that these,
 it also works without them,
 which was good for me to see
 because sometimes it's little things like this
 that only make stuff work.
 But there is a difference
 between the distribution of language tokens,
 which is usually like a zip distribution
 or some sort of very heavy-tailed,
 but sharp distribution,
 and image tokens, which by construction
 tend to be more uniform,
 especially if you think like pixels,
 but also the vector quantized models there by design uniform.
 And with the RL problem,
 could it be that it's also a matter
 of how the tokens are distributed?
 Maybe the RL tokens are again, more zip-in distributed
 and that's why it might fit a lot better,
 or did you investigate the appropriateness of this,
 how the embeddings look like?
 No, we didn't actually look into
 how the embeddings looked like.
 That was like, we actually planned to do this
 because I think, personally,
 I think it would be really cool, for example,
 if we found out that it actually,
 these embeddings turned into a sentence
 or something like that.
 But I do agree with your hypothesis
 about maybe how the tokens are distributed
 or how frequent things are.
 And I think this also sort of relates to
 sort of the structure in language
 or like this natural tendency to express things
 in a certain way.
 And you may want to express certain concepts
 more often than others.
 And then there's also like sort of this conditional nature,
 like maybe only if this concept appears,
 which is represented by a certain set of tokens,
 then you wanna talk about this,
 which in a sense, you could say mirrors RL
 or like just any sort of activities that you would do.
 Versus image modeling, personally, I feel it's cool,
 like as a topic, but I also do feel it's very force
 in a sense.
 It doesn't feel very natural to me, if that makes sense.
 Do you feel that there are other disciplines
 that would transfer well to reinforcement learning?
 I don't know if you've thought about this.
 You do include language and images.
 So maybe you thought of even other things.
 There are, I don't know, protein modeling,
 genetic sequences, there is sound and so on.
 Do you have any hypotheses or any plans
 to try out other modalities?
 Yes, we do wanna try other things.
 I think like some interesting things,
 like in addition to what you mentioned,
 could even be like, this is a natural language,
 but it's usually grouped in together
 with like the NLP community, but like code, for example,
 or even like testing out different languages,
 simpler languages, controlling for complexity,
 really maybe even music.
 I definitely think speech could be something else
 to try as well, as you tarrow look at to video.
 I think there's so many things in sort of our,
 I don't know about saying like daily life,
 but there are a lot of things around us
 which sort of have like a natural sequential nature
 of things, and it would be interesting to see
 if somehow, especially in like a low data regime,
 if these things are able to transfer to each other well,
 and if they're like some maybe underlying principles,
 or maybe like some like biases that are learned
 that correspond to like a large majority of sequential data,
 or maybe certain types of sequential data
 and might also help us like group sequential data types,
 maybe learn more about how they relate to each other.
 And I think if we're able to do that,
 then I think we'd be able to study this even more in depth
 and maybe build models based on those findings.
 It's a pretty special world, right?
 That all our models converge
 from all the different modalities
 that even allow us to do things like this.
 I find it to be a very special time
 because it would not have been possible
 if all the image models are ConvNet, right?
 And all the speech models are somehow Fourier transformed,
 transformed some things,
 everything sort of converging to transformers.
 Some people might not like it,
 but it does enable sort of a bigger picture
 on what it means to process data,
 or if you wanna look at it like this.
 So these attention plots right here,
 I found to be very interesting.
 Now, to be clear, this, you say this is on Hopper.
 So this is one of these gym tasks,
 one of these continuous control tasks.
 Is this one particular sample
 or is this like an aggregate over the data set?
 Or how do we, what is displayed here?
 So this is an attention map
 basically given a single trajectory.
 A single one, okay.
 So it's a single trajectory, yeah.
 But we can assume it's kind of representative
 of kind of what happens in general.
 So I have made a bunch of observations here in my video,
 some of which you also state in the paper,
 for example, this structure of three,
 like the models often looking back three steps back,
 which makes total sense
 because the decision transformer input
 comes in these tuples of three, right?
 And I'm gonna guess,
 if I want to predict the next return to go,
 it's probably very related to the last one,
 especially if the reward is more sparse,
 I can just predict like the same number again,
 I'm gonna be correct most of the time.
 And maybe the same with actions,
 given that in the continuous control frame by frame,
 I don't wanna switch my action around too much, maybe, right?
 But it's a pace to look mostly at these things.
 What I found interesting is the image GPT
 had a sort of just a recency bias.
 Like it just seemed to look just two or three tokens
 back in time, which I think supports very well
 what you claimed that image modeling might be different
 from language modeling in that,
 yeah, it might be that the image transformer
 just sort of looks at a local neighborhood
 and then just goes on,
 doesn't care too much about big structure.
 I don't know, it's just hypotheses.
 And then I think the most shady thing I said
 was with respect to the randomly initialized
 decision transformer.
 So this would be the baseline model,
 a transformer that from scratch is trained on this RL data.
 And I claimed what we can also see this sort of pattern
 of three, but much more strongly than in something
 like GPT-2, which does have a more diffuse attention.
 So here it's really super duper hard attention.
 And I claimed that might hinder the model
 from learning proper connections between things
 in the future because it already kind of discards
 in the early layers, everything that would connect
 sort of a state and a reward.
 Does this come close to what you concluded
 or do you have like different insights
 into these attention maps or what's happening here?
 It's actually very, very close to what we were thinking
 after looking at these attention maps.
 I think one thing actually after watching your video
 that I didn't really notice until you pointed it out
 was like those yellow blocks of two.
 I didn't actually notice that they were actually two,
 which I think is actually pretty cool to see like maybe
 like for those ones that weights like two of them together,
 maybe with different weightings.
 But overall, I think the interesting thing
 is that it's pretty consistent.
 Like it doesn't necessarily change,
 like the patterns don't change significantly,
 which is sort of unlike language, for example,
 where you can see things, like generally
 there is a recency bias to some degree,
 but you can see things like depending on the token
 go like pretty far if it's like attending
 to similar tokens from far back.
 But then again, if you do think about it that way,
 you could argue like action representations
 would probably be similar to action representation,
 state to state representations and so on.
 So maybe actually the language models
 and even the randomly initialized model are mirroring that.
 Yeah, I found it to be very special
 how hard the attention patterns are is right here.
 But also there is always in distance of three rows,
 there is one that is just only looking at three steps back
 and six and nine and so on.
 And then the ones in between,
 there is one that has, as you say, that has two
 and one that even has like, it seems like almost it has three
 but just one is a bit stronger.
 It'd be interesting to figure out which one is which.
 I don't think I can tell from this thing, but yeah.
 So I think the one that's only looking at like three behind,
 if I remember correctly is the returns to go.
 And then the ones between that are,
 let's say the state representations and then the action.
 Yeah, so the order is basically world state action.
 Yeah, that makes a bit of sense.
 And I think the sort of the result right here,
 I think in the middle layer, it's really nicely shown
 that something like GPT, it will start to focus
 on maybe kind of the important things in the past.
 It will select some of them to focus on.
 And so no matter which time step,
 it will kind of look back at maybe what it determines
 to be important states,
 whereas the randomly initialized one,
 it will almost be like stuck in this mode
 of how it looks back.
 And so my question here,
 and you can clearly see it in the last layer
 in that in GPT-2, there's still this sort of focus
 and attention on maybe what it determines to be important
 things in the episode.
 And the other ones, they just have like a diffuse
 attention matrix.
 And my question would be,
 might it be possible that we could achieve the effect
 between let's say GPT-2 and the random one,
 like this benefit through a much simpler procedure
 of just kind of regularizing, just saying like,
 you know, don't make your attention so hard.
 Like make, you know, just kind of keep your options open.
 Try to look back a bit further.
 Don't try to be so sure yet.
 Is that, you know, is that something that's reasonable
 or do you think there's reason to discard that idea?
 I think it's reasonable to try,
 but I still do feel that I think the,
 if we do something like this, then maybe we again,
 fall into the trap of what we were like talking about earlier
 is like this essentially like putting a bandaid
 on like a very specific problem per se.
 But I think like the cool thing about transformers is
 they can learn a lot of different things.
 So I think if you say like with a language model,
 for example, it's an initialization,
 you can fine tune it however you'd like to.
 And I think it's more like flexible in that sense.
 Unless like say we were trying to tackle
 like a very specific issue, then I think, yeah,
 it would be for sure something to try.
 Like I think there's this recent paper for language mumbling
 by like Ofir Press from UW.
 And he, they were looking at like say how they can bias
 the like basically enforce a recency bias
 towards a language model and that like improves
 like extrapolation towards longer sequences and so on.
 So I think in this case in language modeling,
 it's like one specific task
 that they're trying to solve.
 But here, if we like just talk about like
 offline reinforcement learning, it's very, very broad.
 And I think, for example, if you tried like Ofir's trick
 in like say for pre-training BERT or something like that,
 now again, this is just conjecture,
 but I have a feeling it may not work as well
 given like there's, I would say a lesser,
 like there was also another paper by, I don't know who it was,
 but I think from Dhanthi Chen's group at Princeton recently
 about like the masking rate in BERT models
 and things like that and perplexity doesn't necessarily
 correlate with downstream performance and so on.
 So yeah, if we're tackling a specific task,
 I would say sure, but I think the one nice thing
 about the language model pre-training
 is how flexible it can be.
 Yeah, I was, I mean, I was the same.
 I'm probably, as you say, falling in the same trap
 that I criticized the field of reinforcement learning,
 say, you know, looking at one thing and saying,
 can I make up something that would just solve this one thing?
 Yeah, and I think, you know, the difference is also to clip,
 show a little bit that it's not just,
 I can't just do any architecture or anything.
 There might actually be something to language modeling.
 In this table, you specifically show
 that the language model pre-trained ones converge faster.
 And I had one question here, and that was that,
 how different is this code base?
 Like how much of the difference in convergence
 can I attribute to you just being better
 at implementing stuff?
 And how much is really due to these two things
 being pre-trained?
 Is it the same code base or did you re-implement
 or implement from scratch?
 I wish I could say I was like this amazing programmer
 that can make things so much more efficient,
 but no, we use the same code base.
 Yeah, so this is legit, legit speed up
 that is due to the pre-training.
 Nice.
 I guess like one caveat that mentioned like about GPT-2
 is that the faster training speed
 is due to like faster conversions,
 even though it's pretty big.
 But like say when you're doing your roll-outs
 and stuff like that inference time,
 it is definitely slower as to be expected by a larger model.
 Yeah, that makes sense.
 I was also surprised because in reinforcement learning,
 usually the conventional wisdom is that
 it needs a lot of resources.
 And here you mentioned something like,
 you have a single V100 and you have a single V2,
 and the time here is,
 I mean, even for the decision transformers,
 it's a couple of hours.
 It's not I have to train on eight GPUs for a couple of days.
 I was just positively surprised by just sort of
 the requirements and this makes it more accessible.
 Yeah, I think that's the cool thing about offline RL.
 You just, well, you just have to like say fit
 a certain set of trajectories.
 And there've been like a lot of pretty efficient models
 recently as well.
 So yeah, I think it's when you get into the online setting
 then things get pretty like computationally expensive.
 You also mentioned that context size doesn't really matter.
 In fact, more context seems to make stuff worse
 a little bit, right?
 Like how significant this really is.
 But do you have an idea here?
 Is that, is it just because there's more noise
 or is there something wrong with the objective
 of the decision transformer?
 I think partially more noise.
 And two, I think because of like say the tasks
 that are tested in gym,
 it's like you see a teeter running for example,
 or you have like this hopper,
 which is literally just hopping.
 And those emotions are relatively repetitive.
 Like in Atari, for example, the context is,
 I think quite a bit larger.
 I don't remember exactly what the value was,
 but maybe like 50 or maybe even a bit bigger than that.
 But it's like, okay, for Atari,
 maybe you need more information
 because I guess like the actions that are being performed
 are more diverse and like sort of what can happen
 is more diverse, but then for these tasks,
 then maybe that much context is not as necessary.
 But this is just my intuition.
 Maybe an RL person would be able to give a better idea of why.
 So the last thing that was here very special
 is just the scaling behavior of these models,
 namely with the language model pre-training,
 you could scale to much larger models.
 Do you have a feeling of how that continues?
 Like does it continue dropping off
 and just not giving you returns anymore?
 Or would it eventually also say you have like a model
 that's too large and it would drop in performance again
 versus a smaller model?
 Because my hypothesis was that language modeling,
 you have infinite data essentially.
 So you can never overfit on the pre-training.
 And therefore, there might never be really an opportunity
 to overfit on a fine tuning data set.
 I don't know, do you have an intuition?
 I'm gonna guess, maybe you didn't wanna go up
 to too high parameter models.
 Yeah, for like computational reasons,
 but I do generally agree with you.
 Like if we have, I think if we have a decent initialization
 like from the like language modeling on say like,
 like quote unquote like infinite data,
 then I think we should be able to arguably
 at least retain the same performance
 or get like very close to it.
 Perhaps there is a time, like a point
 where it just gets too big that it starts overfitting,
 but I would say that would probably happen
 when it like not close to the parameters we tested.
 Now you, oh, sorry.
 So I think, oh yeah, sorry.
 So that's like one thing, one good thing
 about like offline RLs.
 So you can also collect a lot more trajectory data
 from just running agents and then train on offline data.
 So I think there's that perspective in this figure.
 Like we can also train like a larger model
 and larger trajectory data.
 And then if you have like a really good language
 initialization, then you can also try that sort of direction
 of thinking that way.
 Do you have an idea how that trades off?
 Like would I rather invest into pre-training my model
 on language data or would I rather invest
 into gathering more offline RL data?
 Personally, I think if you're working with a fixed,
 like say, okay, say if we fix the amount of offline RL data
 and say we're gonna like use that
 versus like designing like a better algorithm or something,
 I would say pre-train your language model.
 But then again, as we see with like GPT versus GPT experiment,
 making it that much bigger, like sure it does help,
 like by some margin, but it's not like that
 super significant.
 So based on that, if we're gonna assume
 that language transfer is only like a certain set
 of maybe limited properties to these RL tasks,
 then I would say, yeah, collect more RL data, I would say.
 You said at the beginning, you tried it out,
 you thought about it, it kind of worked out of,
 or initially you got some promising results.
 Was there ever a thing that didn't work?
 Like the something in this project you tried
 and just didn't work at all or it didn't work at first?
 Any sort of avenues you got stuck in?
 I would say that what was interesting
 was that the cosine loss that we added,
 especially like towards like later stages,
 everything sort of smooths out,
 but this more has to do with how fast the model converges.
 So that's actually, maybe we should have ablated this,
 but the cosine loss actually allows the model
 to converge much faster.
 And one thing that was interesting
 is especially in the early stages
 is that the cosine, so say we weren't using the cosine
 embedding loss initially, and we just saw like GPT and GPT,
 or GPT, and GPT was like quite a bit lower than GPT,
 but then like say GPT without this extra loss,
 and then GPT with the loss, GPT managed to catch up to GPT,
 which is like pretty mind blowing to me.
 So like something like that was interesting.
 I wouldn't say like a hiccup
 because it actually worked like pretty well,
 like straight off the bat,
 but it was pretty interesting to see.
 And another thing was without say like
 the positional embeddings, for example,
 I would, you would general, like I think we ablated this,
 but we would generally see like quite lower returns
 and things like that.
 So maybe even like the position transferred from language
 is also quite important.
 Is there anything else you'd like to get out
 about this paper?
 Can people get into this themselves?
 Your code, is it available?
 Yeah.
 So actually it's in the footnote of the first page.
 So yeah, I think this stuff personally is super interesting
 to see how we can transfer different sequence modeling
 tasks to each other, sort of unite.
 So like say one big model that handles all the sequences
 or something like that.
 Another thing that was actually pretty cool
 is with like the language modeling co-training that we did.
 When we did it, the language, like it was,
 we actually had a model that was able to language model
 and was able to handle trajectories at the same time.
 And like the language modeling performance
 didn't degrade significantly,
 which was also pretty cool because it means that
 we essentially have the capacity even at a small scale
 to do both of these tasks at once.
 And if we have like these models that are able to handle
 these separately, then it begs the question,
 okay, what can we do together?
 Like, can we model everything all together?
 Like basically I think with, what was it?
 The, like say like with multilingual pre-training
 that we have, it's sort of like until I guess,
 and for maybe like a few papers before that,
 we didn't really feed all languages just together at once
 and see what happens.
 And then on top of that, we see like,
 oh, we have like this zero-shot transfer.
 Whether it's truly zero-shot is a different question,
 but still it's pretty cool.
 And I think if we can sort of replicate that,
 say we have like, I don't know,
 a remotely related language modeling,
 like a domain and language.
 And if we fine tune on this domain and language,
 suddenly we can do like trajectory modeling on this domain
 that say has to do with what was talked about in language
 and things like that.
 Like it opens a new set of possibilities
 for maybe like generalization and just like zero-shot.
 I don't like using that word,
 but like that sort of performance in general,
 like these new behaviors and stuff.
 Cool, excellent.
 Well, Michelle and Jutaro,
 thank you very much for being here and sharing the projects.
 I hope to see you again very soon
 with more modalities and more.
 I think this is, I'm still amazed sort of by the results.
 I find them really cool and yeah, good luck in the future.
不知道