CM3: A Causal Masked Multimodal Model of the Internet (Paper Explained w/ Author Interview)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper cm3 facebook ai fair meta ai language model language modelling gpt-3 gpt 3 gpt3 dall-e ru-dalle text to image ai image generation ai internet language model html transformer html large language models transformer autoregressive causal masking causally masked language model bidirectional bert masked language modelling
#cm3 #languagemodel #transformer

This video contains a paper explanation and an incredibly informative interview with first author Armen Aghajanyan.
Autoregressive Transformers have come to dominate many fields in Machine Learning, from text generation to image creation and many more. However, there are two problems. First, the collected data is usually scraped from the web and uni- or bi-modal and throws away a lot of structure of the original websites, and second, language modelling losses are uni-directional. CM3 addresses both problems: It directly operates on HTML and includes text, hyperlinks, and even images (via VQGAN tokenization) and can therefore be used in plenty of ways: Text generation, captioning, image creation, entity linking, and much more. It also introduces a new training strategy called Causally Masked Language Modelling, which brings a level of bi-directionality into autoregressive language modelling. In the interview after the paper explanation, Armen and I go deep into the how and why of these giant models, we go over the stunning results and we make sense of what they mean for the future of universal models.

OUTLINE:
0:00 - Intro & Overview
6:30 - Directly learning the structure of HTML
12:30 - Causally Masked Language Modelling
18:50 - A short look at how to use this model
23:20 - Start of interview
25:30 - Feeding language models with HTML
29:45 - How to get bi-directionality into decoder-only Transformers?
37:00 - Images are just tokens
41:15 - How does one train such giant models?
45:40 - CM3 results are amazing
58:20 - Large-scale dataset collection and content filtering
1:04:40 - More experimental results
1:12:15 - Why don't we use raw HTML?
1:18:20 - Does this paper contain too many things?

Paper: https://arxiv.org/abs/2201.07520

Abstract:
We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM. We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

Authors: Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer


Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Today, we'll talk about CM3, which is a model that directly ingests websites, learns the HTML, it uses a novel objective that does left-to-right language modeling, but with a twist that essentially allows it to incorporate bi-directional information into the language modeling. It incorporates text, structure, images, hyperlinks, and with clever prompting, it can do almost anything. It can do what Dali does, generating images from text. It can caption images. It can do text summarization. It can do entity linking, and it can do much more. I like this paper because of the idea of incorporating the structure of HTML. And also, the new objective is very cool. So we're briefly going to go over what the paper is and does and how it works. And then we're going to jump into an interview with Arman, who joined me in talking about this paper. This is a very informative interview, and I suggest that you give it a listen. So this is just going to be a short introduction. Again, I have to rely on you to tell me how I make the best use of authors coming on, because I think it's so cool. I want to talk to them about the paper, and I want to get the most information out there for you that is possible. So please tell me short intros, long intros, how to structure it and all. Leave a comment down. If you like videos like this, leave a like as well. If you leave a dislike, you know, that's kind of useless now on YouTube. But you know, feel free. I'm still going to see it. So CM3, a causal masked multimodal model of the internet by researchers at Meta. I'm going to guess this is now. So this model is, it's a family of models, actually, and a family of causally masked generative models trained over a large corpus of structured multimodal documents that can contain both text and image tokens. In fact, much more. So what this model does, it's a language model. And the language model ingests HTML, a cleaned up version of HTML, but still HTML. If you don't know what HTML is, HTML is essentially the language your websites are written in. And it consists of tags. So for example, one tag is a div tag, that is, it's it has it had I think it had a meaning at some point. But right now, it just serves as kind of a container tag. So div might be something like a container, and you close it by saying slash div. Anything in between is the content of that div. Other popular elements are, for example, a paragraph. So inside a paragraph, you can have some text. Hello. There. And then what you can also have is hyperlinks. So hyperlinks start with an a tag. So you can see these tags can be nested. These tags can have attributes. So the a tag can have an attribute, like an href. So that is a URL, so www dot something, and so on. So it can have URLs, it can also have URLs within the document. Then there is the text of the link. Now we close the a tag. Oops. Then we may continue the paragraph or we may close the paragraph. A forward slash. And the last thing that we're also going to need in these documents right here are images. So there can also be images and I'm gonna write this over here. After all, whitespace doesn't matter in HTML. So images can have a so called source. The two most important attributes are the source. And the source is it's usually usually it's a URL, it can be a base 64 blob. But usually it's also a URL, like, I don't know, like imgur.com slash something something dot jpg. So the browser would actually go and fetch that image and display it at this position. And also, an important thing is the alt text, which you put there for screen readers and other sort of assistive technology that cannot directly make use of the image to see what's in the image. So you can already see here that there's a lot of information in HTML. Now previous work, what they would have done is if it's a language model, for example, GPT-3, they would simply only take the text bits of that they would take, for example, here, hello there, they would probably also take the text of the link right here. And and that would be it, they would scrape the websites for the containing text to do language modeling. Other models such as Dali, Dali, I've made a video about Dali, if you don't know what it is, but essentially a model that you put in text, and it gives you an image. And the reverse of that is is sort of clip, not the reverse, but clip is a model where that says whether or not an image or a piece of text go together well. And the reverse of Dali would be like a captioning model, you put in an image and you get a text describing that all of that you can get by also scraping the internet and always taking the following two things you take the alt text of a an image tag, and you take that source image. And these are pairs of images and text that go together, right. So you can train this is kind of like weak supervision, there are some problems with that. But it's weak supervision. Likewise, there are other tasks if you are, for example, doing entity linking or entity disambiguation or something, what you would do is you would go to Wikipedia. And on Wikipedia, you would always take the text of a link and the link itself if it points to another Wikipedia article. And you know, in this case here, it says like, Romans were captured by Alexander the Great, Alexander the Great would be a thing you could click on. And then that link would sort of tell you what entity that is it lead to the Wikipedia page of Alexander the Great. So people have parsed websites for a long time in various ways to achieve different tasks to collect data for different tasks. However, there is this new direction. And it's not the first paper that does this. But it is the first that I've come across. And the previous work is also by largely the same authors. So I'm just going to give them credit for some at least some of this. Basically, the the novel idea here is that why don't we use the entire structure of HTML directly in instead of just scraping subset of them. Now, again, they do clean the HTML because a lot of HTML is kind of like visual elements, cascading style sheets and so on. There definitely would be information there. But it is a good step to say, hey, the whole thing, you know, the entire thing here, the structure that is actually super duper important. It has so much structure that we would throw away otherwise. For example, the image right here, you know, it could be not only described by the alt text, it could also be described by like the surrounding text like this stuff right here. Of course, if there's an image on a website, reasonable to assume that the surrounding text might also have to do something with it, right? It is reasonable to assume that in order to disambiguate this entity right here, you might want to take a look at the text around it. You might want to take a look at the images around it and so on. So if we had a model that could directly learn the structure of HTML, we could exploit all the work that went into creating that HTML, which is essentially what front end programmers and website programmers do all day. This is human ingenuity that goes into creating these structures, even if it's a framework, right? That there's something, someone that has to come up with, you know, what are the elements? How is the structure? And that is really good data. And exploiting that data to me, when I saw this, it made perfect sense to say, you know, we should just keep the HTML and just learn the language model over the HTML, right? So what can you do if you have such a language model? Well, if I have trained such a language model, I can maybe, you know, start a paragraph, start a paragraph, I put like a piece of text right here. All right. And then I just start an image tag. And I say source equals, and then I'll let the model generate whatever is here. Right. Now, there is a there is a there is a trick right here. I can't obviously put a URL, I actually have to put the image itself there. And if the model is good enough, it will look at this, it will generate an appropriate image. Or you know, I could do the same thing by simply having an image tag. And first generating the alt first putting the alt text, I put something here that I want and then source and I say equals and then I let the model continue. It will generate me an image, I can reverse that I can put the image first and then say, please generate me the alt text, I can put an entity and say, please generate me the link to the entity, and so on. So you can see how powerful this is. We can do many, many different tasks if we have a model like this. This is one thing that this paper does. And I said it's inspired by previous work. However, it pushes it a bit further. So first we have to discuss this and then we have to discuss the novel objective, which makes it even more powerful. The only thing to discuss right here actually is how do they treat images because language modeling is fine. I can just have an appropriate tokenizer for HTML, which needs to be I guess a little bit of a different tokenizer than for regular text because you have to handle these tags correctly. But essentially, I have to have a tokenizer and transformers are pretty good at learning to open sort of appropriate tags and then close appropriate tags again and so on. The only part really are the images. So we don't want to have URLs of images in there. Instead, what they do whenever they encounter an image tag, so whenever they encounter image with a source that equals some URL, www dot something, what they do is they would go, they would fetch that image, they would put it through a, I think a VQ GAN model, some vector quantized GAN model that is pre-trained. They would extract the latent embedding from that and they would put that embedding here. So these models, these vector quantized models, they would take some image and have like a neural network and they would encode that into a series of tokens, which are going to be something like, I believe it results in 256 tokens, latent tokens. So these are essentially because it's vector quantized, every one of these is part of a vocabulary. And so these are essentially tokens like language model tokens, like letters that I can build images from. I can simply unroll, oops, I simply unroll the tokens in these images that the VQ GAN gives me, right? I can have some scheme of how I go through here and I can replace the source property here just with these tokens or I mean appropriately the embeddings of these tokens. All right, this goes here and so on. So once I have these tokens, right, I can train the language model and then the language model will generate these tokens again. Again, they're not continuous values because it's a vector quantized model. They come from a fixed vocabulary and that's what I ingest and that's what I predict and therefore I can treat it exactly the same as the language model. There is a bit of a difference with how these things are distributed. They do talk about this in the paper as language tokens are zypion distributed and image tokens are by design uniformly distributed but I mean essentially from a conceptual standpoint it's the same. The second thing they do is they have a different objective than language modeling. Language modeling usually goes left to right. So that means the language model whenever it generates a token it looks at what it's generated so far and then from that it will generate the next token. What it cannot do is it cannot look at the like right like the head. It cannot look ahead. You can't tell it, you know, here is a piece of text and here is a piece of text. Please fill in this piece of text. That would be a masked language model like BERT. But some a model like BERT isn't really good at autoregressively generating text. For that the left to right causally masked language models are much, much better and you know, higher performing. So is there a way we can get the best of both worlds or at least some kind of a trade-off? Turns out yes there is with the following objective. So as I said we have an example right here in a standard language model. We have the following thing which is a way we can do entity linking. So imagine we'd have to predict this piece right here. As you can see this is the link. It's an anchor tag. This is the link to the page, the Wikipedia page for Armenian nationalism. So Armenian nationalism, we want to predict that link which is essentially solving entity linking for this sentence. If we only have a causally masked language model all we can do is input this piece of text to the left. So this would be our entire context. Now this example is constructed such that this thing right here, this word right here is really important to classifying, to seeing what is there. Therefore if we only had a causally masked language model, if we only ever trained left to right, we couldn't make use of the word that was behind right here. If we had something like a masked language model we could absolutely do that. So that is this example right here. If we had a masked language model then we could absolutely do that. We could input this and we could input this and we could say, you know, here is a masked token. Please generate what's in the masked token. However we already discussed the weaknesses of that approach. Instead they have a new objective which they call a causally masked language model. Now I called this before a causally masked language model because there's also this sort of causal mask inside of it. I'm sorry. The causally masked language model is the thing they are going to propose. Inside of these language models usually there is something like causal masking. So it's a bit confusing if I look at this right now. What they do is during training. So during training what the masked language model would do is it would just mask out these parts and then it would try to fill them in. This limits training because you can only mask out so much. You can't train in parallel and so on. Whereas with the autoregressive language models you can train a lot of stuff in parallel. There is none of these noise and so on. Everything is decomposed nicely. Here what we would do is we would take the things during training. We would simply have a span that we mask but we don't just leave it away. We actually put it at the end. And there is an identifier token right here to show. You can see that this token right here and this token right here are the same. So we tell the language model. We tell it, look here is a sentence. There is a mask right here. There's something missing. It could be one or many tokens. And then here we want you to generate that thing again. And the model simply has to generate the thing back here. There can be one mask tokens. There can be many of these mask tokens in which case we just, you know, if we mask something else like this right here, we just put the corresponding token right here and ask the model to generate it on. The model will learn if there are two mask tokens. The model will learn to after it finished the first thing that it's supposed to produce to automatically put the next mask token there. So that is the objective. It still benefits from this left to right thing. As you can see, we can train this left to right. Once we reorder the sentence, we can just input the whole thing here into training. We can train it like a decoder only language model and we get all the performance off of that. Yet we can still do kind of like masking. So we get bidirectionality by design, because now if we want to predict this mask right here, we have seen all of this context. So essentially we have seen the whole data point. We do sacrifice like a little bit of performance because, well, inherently this part here is still left to right. So there's that. Like in itself, it's still left to right. Also, we do take stuff out of order. So there is the question of, you know, how long can I memorize stuff and so on with transformers maybe a bit less, but we do take stuff out of order, which introduces some noise and so on. So it is definitely a trade off wherein pure language modeling is still going to be more powerful. But this now enables us, this enables bidirectional context essentially into the things that we generate. And that has a lot of advantages for many, many different tasks. There is a whole scheme. It seems to be really important how exactly, oh yeah, 256 tokens for each image. Sorry. It seems to be quite important how you generate these masks during training, how long they are. They try to make them quite long in order for the model to learn important structure and so on. We'll go through all of this in the interview. The scaling laws are pretty astonishing in that they're large model right here. And these are large models, right? These are like the scale of this. It was trained on 384 A100 GPUs. No, I think that's even the baseline. That is even the baseline. Where is their model? Yeah, I don't currently find it. But you can just see sort of the scale here of what they're going for. So these are not small models. But if you make them sufficiently large, you can see that largest models, they're not done training yet. Even after they put sufficient or put enormous amounts of resources through them, you can see they're not even the same ahead. Like the same advanced inside of the training. So yeah, this is very promising. I think this is a very promising direction to make use of that, to make use of the HTML structure. You can see a little bit here. So essentially, if you just put this as a prompt, you can have the model generate the alt text and the image at the same time, right? It interestingly chooses to put the alt text in front, like it chooses to generate a little description before it generates the images, which is interesting. You can also force it to first generate the image by just putting the source tag directly. So then it needs to generate the image. And it's interesting because the quality of the images when you force it to generate image before alt text, it is a lot lower, as you can see here, than if you just let it generate the image, in which case it chooses to generate the alt text first. You can do many things. You can do image inpainting by masking out a portion of the tokens of the image. You have to mask out entire tokens, but still you can do like crude image infilling. You can do conditional infilling by providing alt text first and then do infilling. You can do conditional generation by providing alt text. So the possibilities are very, very great right here. You can see this is infilling, conditional infilling, and so on. The possibilities are great. And remember, this is a very particular data sets and very particular cleaning methods of HTML. I believe if we extend this to even more structure and so on, maybe even take cascading style sheets into account, take all of the structural elements of websites into account, title tags, headers, footers, and so on, this could be really powerful beyond the applications that we see right here. They can also do pure text modality data sets. As we said, entity disambiguation by predicting hyperlinks. They also do get new state of the art in zero-shot summarization by simply generating like the title or the meta tag, the description tag of the website. They give it a fake website with the text they want to summarize and they generate these tags. They do say for completeness below is an example of a prompt that can do basic summarization. I did not find that prompt anywhere. So yeah, maybe I didn't look enough or maybe LaTeX screwed up where some kind of a figure is. In any case, I don't want to go too much into the results right here, but I think the direction of using that structured content is pretty cool. The new objective is also pretty cool. I do criticize a little bit that these two things are kind of decoupled from each other. Like they could all be their own paper. And that's also something that we talk about in the interview. So in the interview, we're going to go briefly over the model again, over the research process, over what it means, what it could enable and what difficulties there were and also over the results, which are extremely, extremely interesting. I enjoyed the interview a lot. I hope you do too. Tell me what you think of it and now I'll leave it up for the interview. Thank you very much and have fun. Welcome everyone. Today I have with me Armin Aghajanyan and I've practiced that name 10 seconds ago and I think I got it down. Armin is the first author of the CM3 paper. Welcome Armin to the channel. Thank you for having me. So I saw this paper and of course you have like some big names here. There's lots of authors, there's Facebook AI research. But still, like given all of that, it was still impressive. Like I was impressed by what it could do and sort of the results it gave. Like it seems to be, wow, there's zero shot, there's image generation, there is like a new objective, there's HTML in there. So there seems to be a lot in one pot. If you gave the pitch, I will have made an introduction, but if you gave the pitch to the paper, what is it mainly about? The goal here was kind of to have a single multimodal model that can do everything. Image generation, image captioning, image infilling, to even pure text tasks like summarization, but mostly focusing on this zero shot setting, specifically this popping setting. And how did you, like, were you, this is a very popular thing. I think in the last few years, this came up, maybe starting with something like GPT-3 where people could really say, okay, stuff is possible zero shot if we train on large enough data. Then came things like Dali and so on where, you know, we saw for the first time, okay, maybe stuff is even possible in other modalities than text. This goes even further. This is multimodal. There have been a lot of other approaches to multimodal. There is like this Rudolph even model. I don't know if you've seen that. It goes like image to text to image and so on. And they all work, let's say, with very cleaned up data. It's very, you know, I want text, I want images that go with the text, which makes sense, right? So do you get, how did you get the idea to use, let's say relatively unstructured HTML for this? Like, how did your thought process go until you came to this idea? So usually there are pros and cons having super strong alignment, right? So like Dali, for example, they have like a very specific alignment of like, you know, text on the left side and then you have like 1024 image tokens on the right side, right? Super strong alignment. And in general, it's easy for the models to kind of learn this type of single alignment, but then you're incredibly limited on the prompting side. And I think it's incredibly creative. If you have a general model, it takes a little bit of creativity to extract out the prompt. So the key here is we don't want to have any strict alignment in terms of the modalities. So the goal was like, what is the weakest alignment that we can go for that would still give us the ability to prompt in non-trivial ways? So actually this is kind of a follow-up to an older paper that we published. It was just accepted in ICLR actually, which was this HTLM paper. And the core idea of this paper is that we argued that document structure is really, really important. So what we did there is we took BART large and then we pretty much trained it on just web data, like minimized HTML. So minimal HTML is we pretty much do multiple passes over the DOM and take out anything that we don't think is semantically important. So in that paper, we showed really strong results. So for example, for zero-shot summarization in a structured language like HTML, this is pretty much just generating the title or generating the meta tag where the attribute is the headline. So in some sense, we could exactly replicate how CNN and Daily Mail was collected, which was they looked for headlines. So in the prompt, you can actually describe the way that the data was collected. So we saw that there was some rich structure available to be used in HTML. So after Dali came out, we thought, okay, there are some fundamental restrictions with Dali. So the first one being the causal approach. So they train a decoder only left to right model. So in some sense, you can't do things like generate the text given the image, right, just because of the positioning of the image. It's on the right side of the image. You can't really do image infilling either, which means conditioning on both the prefix and postfix of the image. Or you'd have to train specifically one particular type of infilling. You could rearrange stuff such that you could infill one part, but you can't dynamically infill something. Exactly. Yeah. So those were kind of the first weaknesses that we saw there. The approach was very clever though, right? So pretty much taking continuous data, discretizing it, and just doing sequence modeling. It seems to work very, very well. So the idea that we kind of combined the two from the HTML paper, which was that document structure through HTML is really important, but let's also encode images there and see if we can recover something like Dali. So here you're kind of looking at the data that we collected. So the data set size is actually quite good. I mean, we're around like the 200 billion tokens, which is a relatively good size if you're training large models. But one kind of downside that we have here is because we don't have the strict alignment, we can't artificially increase the amount of images that we have available in the documents. If you actually look, I think we have 25 million unique images. I don't know about Dali. Dali was trained on 400 million. I don't know how many of them are unique, but regardless, they still have an order of magnitude more images than we do. But then we have the other benefits, which is we're also training on a ton of text. So we can do a lot of text only tasks. And I think the rest of the paper will show that we can do not only text only tasks, but we're actually competitive to T5, which is actually really hard to do. And I can explain why we think this is the case in a little bit. So the very first thing was, okay, so now we kind of have this data, but HTML is also very localized, right? Like the title always comes first. It's in the head, right? Or like the meta tags always pop up first, right? So if you want to generate meta tags or generate title, right, condition on the rest of the text, it's kind of non-trivial how you would do this in decoder only setting. And so we kind of started thinking, there are multiple ways around this, right? So the first thing is using encoder decoder architecture, right? And then with some masking, you can kind of recover this type of bidirectionality. This is true, but there are pros and cons to this. So encoder decoder only architectures, they're really good for fine tuning, but they're not so good for prompting, is at least what we noticed. And also training them is a little bit more non-trivial. So decoder only models are quite nice because you get per token generation. So you pretty much generate every token for the source. Whereas for encoder decoder, most of the time you're generating, I think like 15% is what Bert and Bart or Roberta do. It's all around that 15%. So most of the times you have to go through the data multiple times. For some reason, they don't prompt super well. And the kind of the other big thing is if you want to do score-based prompting, it's kind of hard to do with encoder decoder only architecture, right? If you want to ask what's the log probability of this sequence with the mass language model, it's kind of tough to do, right? So we knew that we wanted to go kind of this decoder only route. So we introduced this new objective that we called causal masking. And so the idea behind causal masking, if you want to scroll down, I think there's a figure there. This one. Yeah. So the idea there is relatively straightforward, right? So pretty much think of mass language modeling, where you place in the mask, but take the mask and put what the mask represents simply at the very end of the sequence. So if you do this, you kind of get, it's very, very simple, right? But you get a lot of the benefits, which is you still get per token generation. You optionally allow for bidirectionality, which is actually a really, really big thing to have, right? And the other thing that we noticed is that depending on the sending, prompting versus fine tuning, the size of the mask is really important. So for fine tuning, localized information is really important. You want to have a lot of small masks. For prompting, we saw kind of the opposite, which is you want to have very, very few masks, but they can be very long. So the strategy that we use here is for every document, we sample from a Poisson distribution centered around one. So the majority of times, right, and we clip it to one. So if you get zero, it becomes one, right? So majority of times, you're only going to get a single mask, right? Over 50% of the time, you're only going to get a single mask. And then you pick, you uniformly sample a subset of the document of any size, and you kind of place that in the end. So you get these very, very long kind of infilling naturally. And so this objective turned out to be quite strong. So it's competitive to language modeling in the sense that when you get per token generation, our perplexities were not that much higher than just a language modeling objective. You get optional bidirectionality whenever you want it, right? You can score probabilities of sequences super, super easily. So we're kind of going all in on this objective. And so we have some follow-up work looking at causal masked scaling loss for text. So this is some ongoing work that we have now. So we're pushing heavily on this. So the general argument that we're trying to build is that if you're doing language modeling, deconormally language modeling, you should be doing causal masked language modeling. So that's kind of my... Yeah. I mean, it is intuitively a good trade-off. So I think here you make the case, if I interpret this correctly, that this word nationalist right here is really important to fill in this mask. And if it were just sort of left to right, it would be very difficult to fill this in yet since you move it to the end, right? And the model has to extra learn kind of to keep these tokens in context to sort of realize what's there. So it has to waste kind of some extra memory to remember the context of each of the mask tokens and so on. But yeah, I think it is very intuitive. It is also a good trade-off between, I want to say, left to right has, at least for most there are right to left languages, but for left to right languages, left to right objective actually makes sense, right? That is how we generate language when we write it down. So there is something to left to right that I was never happy. There are other approaches like XL net or so. They were saying, well, we just train on all possible paths of decoding, like all possible sequence of masking out tokens. And it was never really satisfying because I always thought, but there is something to left to right. However, sometimes as you say, it's really important to know what's after. And I think this is like a really good trade-off. Yeah, like specifically in this example, in the zero-shot prompting case, let's say we want to tag nationalist with some entity link. If it appears beforehand in the sequence, there's no way to prompt the language model to generate an entity link before the entity appears. So that was kind of another reason that we had because like I said, HTML data is very localized. In Wikipedia, this a tag which represents the entity link always appears before the entity. We have the option of training two models, one left to right, one right to left. Or you can kind of do this kind of clever rotation of the document. Yeah, the XL net approach is definitely interesting, which is having different permutations of the source document. But like you said, I think there's a lot of inductive bias for left to right, which is why I think left to right models are kind of de facto now. Just for my understanding, is there a reason behind these arrows? Why do the arrows are like double arrows, then there's a line and there's like a double arrow again? Does that have a specific meaning? And here the arrows are only here? Yeah, so arrows pretty much was the tokens that you actually generate. So in the language model, you're generating every token in the mass model. So you go like this, okay, I see, I see. Because I was like, okay, is there some meaning? But yes, there is. And this shows that in the mass language model objective, you only actually generate very small number of tokens and you wouldn't even get like a loss for the other tokens. You said before that you had a certain number of tokens, right? And you said, well, that's actually good or bad for, you know, that's actually in a good order for language modeling. Yet a special thing about your model is that images are also tokens. You push images through a VQGAN encoder, right? Which is pre-trained. And these just become tokens in whatever sequence. And this results obviously in larger data because some of it is images. So you say you have a terabyte of data in this data set, which is obviously way larger than for example, a text only data set. Do you find there is a difference? Like do you find the number of tokens is really what matters in the size of the data? Or is there a qualitative difference between image data and text data, even though both are tokens? Yeah, so there's a couple of ways to approach this. So the very first thing is that modeling, and I think we mentioned this quickly in the paper, but modeling image tokens versus text tokens, it's quite different actually. So for like text usually follows like textual tokens follow like a Zipfian distribution, right? Whereas I think in Appendix we have a figure, it's pretty much uniform for images. So there's different like in terms of the distributions that you have to predict, they're actually quite different. So we saw a little bit of challenges and we saw some kind of weird behavior during training. We didn't mention this in the paper, but the one weird behavior that we saw was that there were regimes during the training, like parts of the training that only optimized for text. So on our image evaluations, like it pretty much would be flat. And then there were times that it was quite the opposite where, you know, images would be being optimized for the text kind of stayed flat. So we don't really have explanations for why this is happening. I think there needs to be future like scaling laws looking at multimodal sequence modeling. And when I say multimodal, I'm not just talking about like images and like natural language text. I meant like you can even include code as a different modality, right? So the scaling laws there I think are a little bit different than what we're used to with the text. The reason for using tokens is purely because of a compute thing, right? So you know, we're given some amount of GPUs, right, for some amount of times. So what we do is we take the number of tokens that we have, we take the amount of compute that we have and try to find a larger size model that we can train. It's kind of an optimization problem to find the largest architecture. So that's kind of why we used number of tokens as the guiding principle. I mean, it seems to also align with what others... Yeah, for example, this Rudolph paper. So it seems to be a common approach to lift images into like the space of textual tokens, which is, I guess, a bit surprising because a couple of years ago, no one would have gone that route. Even if you were to inject images into a sequence model, you'd probably inject like a single vector, right? So I find that to be a bit surprising, but also, yeah, it seems appropriate that an image could be expressed in something like a sequence of tokens. It's just a bit... I'm not too big of a fan of how this is currently done because the tokens, they also... They seem to be a bit localized in the image and so on. I think there's a better way, if you're a human, that's not really what you do with an image. You see more like the different layers maybe or what's there. In any case, I was surprised by these scaling plots. These are brutal. We scale it up and the loss goes down for the largest model. It seems you're nowhere near done, right? You said you had some different experiences during training, yet also, I think in the paper somewhere you hinted at, well, we didn't really see any pathologies. What was the process like? You had the data, you trained the thing, did it immediately work? It took a little bit of handholding to work, especially the 13 billion parameter model took a little bit of handholding to work. A lot of the times the pathologies we see are things like gradient, underflow or overflow. Gradient explosions happen, although they usually happen in much bigger models like the 100 billion scale. But the surprising thing was that we almost used exactly the same hyperparameters as this paper that came out from Vesto in those group. So the surprising thing is it kind of just worked out of the box apart from having to tune, I think we tune like learning rate, we had to tune weight decay and batch size. Apart from tuning those things, it just worked almost straight out of the box. And what you said is actually correct, which is if you look at the large model, it's actually not done training. So the good news is once CM3 is released, we're going to release the checkpoint that we use for this model. I think the model that we have now is continuing training. So we'll really release that one too. So people will be able to play around with both. Excellent. But one thing I'd like to point out is that the multimodal scaling laws are a little bit different than text scaling laws. One thing seems to be that scale plays a slightly larger role in multimodal than it does in text. So I think the quantitative thing that we saw is that if you look at the data efficiency jumps between like, I'm forgetting the exact numbers, but like let's make them up, like the 1.3 billion model and the 13 billion model from Vess's paper. And the data efficiency there, let's say it was like the larger model was five times more efficient in terms of data. So in order to reach the same perplexity, it would need five times less data. Using the same exact models, we saw that in the multimodal case, it was 10x. So there was almost a two times difference for some reason. And that's why I think it's really important to kind of chase these multimodal scaling laws and fundamentally understand what's going on here. There's a lot of unknowns here. When you say we had to do a little bit of hand holding, what does that even mean in these large models? Like, can you afford to restart training? Or is it more like, you know, you have checkpoint, checkpoint, and then something goes wrong and you go back to the last checkpoint and you do something there? Like what does the process of training these very large models look like? It's just really, really tedious. So one of the main things is, you know, whenever you have a ton of nodes that you're running, there's infrastructure issues that pop up, right? So like if one GPU goes down, right, then all of training is paused, right? So infrastructure issues are kind of a big thing and we have some automated systems in place to take care of that. Other things are like, for example, like we didn't set a high enough warm up period in the beginning. So we saw that we actually had to pause training, increase the warm up, load up the last checkpoint and go there. And so we also kind of tuned learning rate a little bit as training goes on. Although with the large models, I think it might have been just a handful of times. So failures- Do you always have like multiple models running ahead and then you choose the one that looks best or is it really like you change and you train one model and you see how it develops? Yeah, because of the computer is one model. So it really comes down to intuition. So both Mike Lewis and Naman Goyal who are on the paper have trained these really, really big models before. So they had a ton of great intuition about how to get things to work in terms of these very large models. Cool. I mean, yeah, I'm excited and it is very cool that you actually are going to release these things. I think people will love to play around with them. In order to do now the tasks, you tackled some tasks. How did you decide? Wait, there are some natural tasks, let's say there are some that are more, you know, you have to come up with something. Did you have some targets of tasks that you want to tackle? Or was it more like the model came first and then you sat down and saw what can you actually do with it and whatnot? And what worked and were there also tasks that you tried that maybe didn't work at all? Yeah. Yeah, that's a great question. So I think at the beginning of the project, the push was really to have a single model that can do any image task in the zero shot case. And so kind of the story that we built around it is, can we describe all the tasks that we're interested in through some prompt, through some HTML prompt, even before we train the models we got about this. So we came up with a ton, right? And some prompts were very complicated, like style transfer for one. So you can have an image that has a picture of the mountains in the summer. And then you have another image tag that says the same picture, but in the winter. And then you ask them all to predict the image tokens, right? So you can get this kind of zero shot style transfer. So you have some kind of complex prompts. So some of them didn't work. Some of them only worked at scale. And we can kind of go through this. Specifically like one thing is that like the captioning only worked at scale. So their team building model was the only model that could caption well. And the captioning, you go mainly with the alt text of the image. Alter the title, either one. Yeah. But like the figure that you're on now, I think is kind of interesting. So we can kind of get unconditional image generation by just asking the model to generate a sequence of tokens after the image tag. So we saw one interesting behavior is that the model for some reason almost always wanted to first generate the alt text before generating the image. For it was actually easier to condition on the text before generating an image than doing this type of free form generation. When you say it wanted to, that's just what it did. Yeah. Like when you sampled, did you like, I mean, when you say it wanted to, it could also be that in the internet, humans most of the time write alt first and then the source. Yeah. So we actually looked into this. So a lot of text does have alt, but it's around like, I want to say like 70 to 80% mark, if I recall correctly. So it wouldn't explain why the model almost always wants to generate alt text. Now the theory that we kind of have is that without alt text, you have much higher perplexities for images. So the model, because we're doing like sampling, right? So it's going to pick out high probability, low perplexity tokens, which most of the case means picking out the alt just because it appears so often. So that could be it. But overall, I think if you look at these images, they're rather like, they're semi-coherent, especially the ones conditioned on the text. And the same thing I think you see with, you can kind of force the model not to generate the alt text by giving a prompt and generate the image tokens immediately. And do you think, so the VQGAN tokens, naturally they are predicted as one, right? There's some encoder, they're not, as far as I understand, they're not in the image encoder that makes the tokens, they're not predicted autoregressively. So there is no inherent sequence nature to these tokens. Could that be like some sort of a reason why there's also a difference? Because text naturally is sequential, whereas these tokens, the only thing they have is they're kind of localized, but there's no inherent sequential nature. Yeah, that's true. For VQGAN, there isn't something explicit, but I think the way that the layers are constructed, we do still get some implicit dependencies across the tokens. And so I think this is what the transformers kind of pulling apart here. And to be honest, I think there's still a lot of work to be done on the discretizing images front. So one thing about VQGAN is that it blurs a lot of fine detail, so like human faces. In our case, this is kind of good because it's privacy preserving, you're not going to generate like a person's face unless it's a really, really popular, like close up face. So in our case, it kind of worked out. But in the future, I think we need to get much, much higher fidelity image tokens if we think that the way of doing things is to treat everything as a token. Of course, I think there are a ton of new approaches that are not token based. I think Glide was fantastic from OpenAI. The diffusion models are doing great generative work. But if you want to maintain the same benefits of generative models, so being able to generate trivially, being able to compute log probabilities, I think tokens are probably the easiest way to go. And one thing is you can naturally increase the resolution of tokens images just by increasing how many tokens you use per image. So in some sense, if you have enough compute, you can scale up to arbitrary resolutions, right? Yeah. So probably, you could at some point get more tokens than pixels. I wouldn't know what that would mean. But I guess the resolution isn't even limited by the resolution of the image itself. So there's this interesting thing you can do, as you said, infilling by letting the model generate sort of middle tokens. Now you could probably do arbitrary infilling, but you have to have multiple mask tokens. So I guess the natural thing to do is just to infill, since the tokens kind of go left to right, top to bottom, is to infill one of these stripes, which you've demonstrated right here. Did you try infilling arbitrary things? Or was this sort of the natural thing to do? Yeah, so actually, because of our objective, because we sampled the number of masks, right? You can actually mask out like five, six, seven masks, and it still work. I don't think there was any specific reason that we stuck to masking out a single thing. I'm sure it would work with multiple as well. I mean, if you were to infill, let's say, if I infill a square like this, and it covers sort of multiple token lines, this would already result in like if it covers three token lines, it would already result in like three mask tokens, right? So I mean, there is some with just with the sequential nature. But I think that can be can be worked around. So what here we see, so left is source image, then you mask out something in the middle. Then you also give the ground truth, which is here on the right. And then there's one model that does infilling unconditional. So just looking at the image. And then there is one model that does it conditionally. And the conditional is conditioned with this thing right here as the the alt text. So you understand, okay, so understand it correctly. I was, yeah, I mean, I was surprised, for example, by this one right here, this, the park bench, because obviously, if you see the the model that does infilling conditionally, it can do it quite well. However, the unconditional one, it kind of warps the bench or something like this. Like it's it's a bit I'm not I'm not sure the unconditionality has something much to do with it, because there is no this doesn't look like natural, you know, you know what I mean a little bit like, yes, this shouldn't be like, just because it's not conditioned on it. If it's not conditioned on text, I would expect it to be maybe a red bench, right, or, or something, you know, something that is conceivable in nature, but is not according to the text, like there is an ambiguity of what's behind the mask. However, here it really seems to degrade in performance when you don't give it the text. Yeah. So so one theory that we kind of had here is that the the model needs to understand the continued continuation of the the horizontal lines, right? That requires some semantic understanding that this is, for example, a bench, right? And actually, if you look at the the massed out input, the horizontal lines are not completely horizontal. The top of the bench is at a different angle than the top of the bench. So I think the model has a tough time understanding the high level semantic content of the image, which is fixed by feeding in text. Yeah. Now, I think, of course, if you have I think if you have a larger model that's trained for longer with a higher resolution, this probably should not be an issue. VQV, again, it blurs out a lot of things. Number one. Number two, it's just if you change the tokens even a little bit, the blurring aspect happens very, very quickly with VQV again, compared to, for example, the VQV from Dali, which requires more tokens. So 1024 tokens versus the 256 we use here. But it's more direct in some sense. Yeah. So, yeah, I think the main thing here is just that you need to get some like high level semantic information about what's going on in the image. And it's hard to do if you're only looking at like the VQV GAM tokens. Yeah. Okay. I mean, that makes sense. You go on and you have some examples of conditional image generation. On the left side here is a prompt and then you sample images from that with the same technique, right? You give the alt text and then you sample the image. So the avocado chair is like forever going to be to stick in history, right? I think that's just a given. Was there something that surprised you with conditional image generation? Yeah. So the models are quite good at actually generating something that's somewhat coherent. So for example, like the red car, you can see it generates two red cars. That one looks like a truck or a tractor. Sometimes the model tries to cheat and generate something that's easy. For example, in the case that it doesn't generate a car at all, it just generates mountains, right? Just because the landscapes are easier to generate. The other thing that we saw kind of tough compared to Dali is the data that we used only came from Wikipedia or Common Crawl News. So none of it was fictional in some sense, right? We don't have any like art. Yeah. So like our images always try to be as non-fictional as possible, which is it acts weird if you try to give it like really fantasy based prompts. Yeah. So that's kind of one downside. And actually this is one criticism I have of the evaluation that we did for the FID matrix, which is a way to measure the quality of images, which is we actually took the table from Glide for the FID numbers on the conditional generation. One thing was is that MS Coco is almost all non-fiction, like non-fantasy images. So this is really like it's under-representing Dali. So I think if you casted a wider net here and had something that included a wider array, a bigger distribution of images, I think Dali's results here would be much, much stronger. Which is why I think we're kind of comparable, our largest model is comparable to Dali on MS Coco. But in terms of image generation, it's not as good on the fantasy front at all. You did discuss a little bit. You also said you sub-sampled web data and you cited some concerns as well. But there is also quality issue with sort of the wider you cast the net, the sort of more the quality goes down, I guess the alt tags quality go down, whether or not the images even have alt tags, whether or not they're ads or something like this. Why did you limit to this subset of the data and not bigger or smaller? I think at the beginning we had some ethical concerns. Like I said, we have very weak alignment, so you can prompt with anything, right? We had some ethical concerns about images that you can generate if you were just trained on all of Common Crawl. So we try to think about what are large scale data sets that we can get that are somewhat filtered. Wikipedia is definitely one of them. But even then actually Wikipedia itself has a gender bias and I think this is a new, I think other papers have showed this before. And Common Crawl News, which probably is not going to have the terrible content that we don't want to pick up. So we kind of picked those two and it was okay at the scale that we wanted to. So we stuck with those two. But yeah, I think it's hard. I don't know what the solution is. Like the lay on 400 million data set that was released. I don't know if you've heard of it, but this data set, I think there was a critique paper written like a month about it, right? That showed that it was like a highly, highly problematic data set. So in terms of the ethical approach, I'm not really sure what the right answer is for collecting at scale. There are tricks you can do, right? So like if you look at the CC100 data set that Facebook collected, they use this trick that they train a language model on Wikipedia and then use it to score Common Crawl and then take only like medium perplexed from Common Crawl. So you could probably do something like this here. I questioned the efficacy just because very large models, they only need to see a data point a couple of times in order to pick it up. So I think there's like some very fundamental engineering work that's being done for scaling up these data sets to like trillions of tokens essentially. Yeah, I mean, I guess it casts much wider questions such as, you know, I as a human, I'm perfectly capable of going to 4chan and seeing kind of the worst of humanity and it doesn't instantly make me like, you know, I don't know, a terrible, terrible, like it doesn't make me want to repeat everything or something like this. And there's various considerations like shouldn't we be able to build model that also ingests stuff but kind of may also a bit distinguish between things? Like if the models are able to distinguish, it might help them to ingest more of this critical data. But on the other hand, I can absolutely understand that, especially if you're the maker of a model, you don't want your model to output, you know, that I think that's why for example, OpenAI keeps such a tight grip on GPT-3. If you want to build anything with it, right, you have to go through approval processes and whatnot. And it's, it's, yeah, it's, I think it's tricky topic. I also don't know what exactly to do. I'm happy that there are models that are filtered, like say on filtered data. I'm happy that there also exist models that aren't. Yeah, I think the, maybe the sort of the, let's say diversity makes is, is probably the best. So you can always choose which one you want to, you want to use. I don't know. I'm sorry, this is just a rand by now. You do have some, sorry, go ahead. I was going to say, with respect to what you're saying, there's, the solution doesn't necessarily have to lie on the language model side. Yeah. So one thing is you can think of language modeling as just pure density estimation over tokens, right? So if you're doing that, like, of course you're going to model like 4chan, for example, right? But it's up to your generative sampling strategy to remove that part of the density and only sample from parts of the density estimation that you know are safe, for example. And so we're actually seeing, I think, a lot of movement from having a singular model that does generative work and to having like multiple models. So a great example is like Dali, right? So they do density estimation over, you know, text and image tokens, right? But the way they generate images is they sample like 128 candidates and, or whatever number of candidates, and then they use CLIP, a secondary model, to kind of select in some sense the mode of the slice of the density, right? And so something probably similarly can be done here. Like a great example is like take Codex, for example, right? I think in the Codex paper what they do is they generate a ton of samples and then they re-rank the samples in terms of perplexity, so average probability, and then they take the mode. So essentially the exact mode of that density estimation, right? So one thing to argue is that, you know, you could train language models that do pure density estimation over all the text that we have and then have smarter generation algorithms that are able to select subsets of that density that are safe. So like you said, in terms of research, I think there's pros and cons to having unfiltered and filtered models, but that's kind of the way I've been thinking about it recently. Yeah, and it's probably a good approach because the sort of the handle we have on, let's say, discriminative models like CLIP is a lot larger than the handles we have really on generative models like, yeah, the only handle really we have there is kind of data. You also do some experiments on text pure, I don't want to say pure text data because it's more than that, right? It's entity disambiguation, entity linking and so on. Now, is that purely a result of the fact like of you use Wikipedia as a data source and Wikipedia is essentially, it's not really only text, it's kind of a huge entity link and database. Is that kind of, is it fair to say that it works really well because you use Wikipedia as data or is there something more to it? Yeah, no, that's exactly it. So actually, there's this work that we sent in this paper a couple of times, the genre paper. So in the genre paper, I think the paper is called auto-aggressive entity linking or entity disambiguation. So the idea there was exactly that, which is if you take all of Wikipedia and then you train a language model that tries to predict entity link post entity, you get a model that does really, really good entity linking, right? So in some sense, the genre objective was a subset of our much more general objective, right? And it's not too surprising we beat out genre just because our models are bigger in our fine-tuning case. But the really, really cool thing I think was that we can do the zero shot, which is exactly what I showed in the first figure. If you mask out the entity, if you know that you want this entity, you want to disambiguate this entity, you can place a mask there with this a tag, right? And then our model will fill in what it thinks the disambiguation is. So that's kind of cool. I couldn't find any zero shot baselines like this. So I think this is kind of the first paper to do this type of zero shot entity linking and disambiguation. And so, I mean, you also have other tasks like summarization. We also didn't look at the alt text generation and so on. Is there one result that we didn't talk about that you want to highlight in particular, like what maybe one surprised you the most or so? Yeah, so the captioning one was interesting. I think we can look at that. So the captioning is, this is pretty much the dual of Dolly, right? So what we're doing is saying, okay, now that you have an image, generate the alt text for me given the image, right? So in some sense, we can exactly describe the captioning task in HTML, which is again kind of solidifies the argument that you want some level of document structure for prompting. So the results are quite good actually, at least from a semantic level. So one problem is that we don't actually generate in the style of, I think, MSCoco here. So we didn't report like blue four numbers or like the standard numbers. But if you look at the semantic similarity using BERT score, the CM3 captioning with clip as a re-ranker is actually a very, very strong baseline. And so you can kind of see the style here is weird. It tries to explicitly state what type of airplane it is. Yeah. But that's kind of an interesting behavior. So I think definitely at scale, you could get a single model that I think could be competitive with MSCoco with caption only models. If you do things like increase the resolution of the tokenized images, I think scale is really important here. So if you just scale up so that you have a similar amount of samples that are trained using MSCoco. You've said this a couple of times now, this sort of, you know, with scale, we could beat this or that. And I guess you see this work a little bit as a maybe a signpost, you know, to like later work that actually achieves this scale. Do you think the scale you're talking about, the scale at which, you know, this is competitive with on MSCoco, where the image generation is competitive with Dali, do you think that scale is currently achievable or is it so large that it's kind of, well, you know, we need entirely new hardware? Yeah, I think it is achievable. So let me tell you about the result that we just got a couple of days back. That's not in the paper here. So one reason that we also changed, chased this kind of multimodal setup is because we're interested or at least I'm very personally interested in the grounding aspect of language. So we kind of defined grounding as can you improve document level perplexity on text by extra conditioning on images? So that's one kind of way to measure grounding. The other way to measure grounding is we call it symmetrical grounding. So what you do is given a pretty much given a piece of text, generate an image from that piece of text and then condition on that image, generate back that piece of text, right? And I look at the perplexity differences between the two texts and that will give you the informational content of that image that is generated, right? So you can measure grounding that way. The unfortunate thing is that even the 13 billion parameter model that we have here did doesn't ground. But if you look at the scaling laws from, you know, or I think our 100 million parameter model to our 13 billion parameter model, around the 60 billion mark is where we'll see grounding in this setup. Okay. So our expectation is that if you scale this up to 60 billion, that you should be able to achieve, I think, language image grounding, which is kind of a cool result that I think a lot of people have been chasing here. And that's insane that you can make these predictions, right? This is like this is something I think in machine learning is something new. Because right now, no one could tell the most people could tell was like GPT three is going to be like somewhat better than GPT two. But now you're you're able and you know, I am confident that this is a you know, maybe it might be whatever 50 or 80 billion parameters, but you can actually make these predictions, which is which is, you know, it's it's cool. Like I'm amazed by this. Yeah, I definitely don't think we're going to be like order of magnitude off, right? Oh, so I think with the 100 billion parameter, 100 billion or 175 billion, like GPT three size, we can get very, very nontrivial behavior to the point of being competitive across all tasks. And I think the future in general is having a single multimodal model that can prompt in an instructable way, kind of like instruct GPT, but with all modalities. So I think that's kind of the north star that everyone is chasing right now. But I think we have a good I think we have a solid base for this work. But yeah, I think the captioning surprised me. And one thing that I want to call out here is that it only worked at a 13 billion scale. I might have mentioned this earlier. So there are fundamental stepwise changes in behavior from scaling up the model. It's not something smooth, right? So something that a 13 billion model can do is something that, you know, like a 2.7 billion model will not be able to do at all. So you won't, it's just going to generate random stuff. So it's interesting to see what the next, you know, stepwise changes in behavior will be, if you scale this up. With respect to the HTML, right, that you use, which is, I thought it was it was pretty cool because it is data that is, you know, so available. And your argument is a little bit that if you clean the HTML too much, right, these other these other data sets, they just pull out the text content, maybe the image, they try to align it and so on. You know, if you clean that up, there's so much structure missing, right, you're missing on all of this valuable information. Yet, you also do cleaning, right, you do quite a lot of HTML cleaning, you say somewhere up here in the data section. We strip this, we strip that any any sort of non non whatever elements we strip out, all headers, all footers, copyrights, forms, dialog boxes, we merge consecutive div elements and so on. Couldn't the same argument be made against you saying, well, you're losing so much of the structure, there's so much information there, like, why are you doing this? Do you think there is a valid direction to go in actually taking in even more context of these HTML documents? Yeah, so there are different constraints here, right. So one thing that I mentioned is that we can only model x amount of tokens, right, 300 billion tokens, for example, right. So if the majority of those tokens, right, like, I think the average document is like, 95% of the document we removed. So yeah, in some still right, you know, even though you're the ones that remove way less than the other ones. Yeah. So, so in some sense, do, do we want to model every single token? So in the case that you have infinite compute shirt, right. But here, there's kind of a min max problem that you have to solve, right, which is you want to kind of, you want to maximize the amount of semantic information that is available while minimizing the amount of tokens that you have, right. And this is kind of complex to do. So I think we found a good enough balance of the two. Like, in most cases, like, you don't want to repeat the same copyright like 400 million times, right. I mean, there's, there's probably a lot of information in the fact that jQuery is imported in this website, right. Right. So things like that. But we also do things that might break document structure, like the merging of elements, right. There's probably something there as to why the person has multiple developments, right. Regardless, we remove it. The other thing that we remove is attributes. So we remove all the attributes except those that are structured. So like open graph schema, I think Twitter has a like a structured graph as well. And the reason there was that the attributes were just, first of all, they were way too long most of the time, and they were not informationally rich enough. So you kind of have to balance compute here with how much structural information you want to maintain. Yeah, I see. And so there's no fundamental reason to use HTML, right. It's just something that's there, right. There's, I mean, for example, you can use markdown as well, right. And you can kind of recover a lot of the same things, right. Like generating the title you can do in markdown, right. High links you can do in markdown, right. So maybe the future direction is explicitly codifying this min max problem, right. And coming up with the document structure that the document structure is described in the minimal set of tokens. So maybe that's a pure engineering project as well. When you think of HTML and the DOM, it is a tree, right. Which is different from a linear sequence. Do you think there is, do you think there's value in treating the tree as a tree? Do you think it's mainly a limitation of the models we have? They go, let's say, like, see token by token or left to right or something like this. Do you think, you know, maybe it's still good to treat it as a sequence because there's text in there and text is left to right? Like what keeps us from building tree based models, which would be much more appropriate for something like this? Yeah. So one thing about transformers is it seems that they can learn the inductive bias of the data fairly well and it's not necessarily encoded. So my argument to this is that usually for these large scale runs, the best thing is just to keep it as simple as possible. Mostly just because they're risky, right. You get one chance. But the other reason is that transformers are actually highly capable of picking up this type of structure. So this isn't in the paper, but we looked at the attention scores and then you can see very clearly that the model knows what are like boundaries between HTML elements, for example. But again, there's also a ton of work to be done as well. So like some exciting work is, I think you also interviewed like Ofer for the alibi work, right? That work is really clever, right? Because it introduces an explicit inductive bias that the further away a token is, the probably less likely that you are to look at it. And it gets rid of the need for positional representations. So you can imagine like an extension of alibi here that would directly encode a tree like structure, right? So there's a ton of work to be done here. And then other thing is we didn't do too much for the images, right? In terms of attending, the positional representations for images are different than of text. So future work should consider specifically embedding images in such a way that you maintain locality of positions, right? So this is all stuff that needs to be done in the future as well. But that being said, I think if you have enough compute, these models can learn anything. It mostly becomes an efficiency angle. So about this paper, so what I have a bit of a trouble with is too many things in one paper, which in this case is this idea of using HTML and so on, although there was a previous paper of that, but then there's also the new loss and so on. Have you tested the new loss on pure text generation? Something like this, can you parse out what the different things contribute to the success of these models? Yeah. And that's a great criticism of the paper, actually. So fundamentally, I think if we wanted to do those like the proper science way, this would be like four or five papers, just teasing things apart. But at the same time, when you're training these large language models, ablation studies are pretty much impossible, right? No one has much compute to do these ablation studies. But the answer is yes. So we're looking at causal mass scaling loss for text only. This is a project that we're working on. We've trained a code model using the causal mass objective that's outperforming, I think both Google and Codex of similar sizes while being able to have a bidirectional option. So there are a couple of teams within Facebook that are trying out this objective with some success. So there will be future work about this. Excellent. And apart from what you just mentioned and scale, what's sort of next in this direction? Are you like, what are you excited about? Maybe it's not even you working on it, but what kind of is your exciting stuff that's happening? So one thing is figuring out a way to have higher fidelity. So the question to ask here is how do you represent continuous data in a discrete domain? And I don't think we're there yet, right? So that's some fundamental work that needs to move forward. The other thing that I'm kind of interested in looking is can we start joining more modalities, right? So Hubert that also came from Facebook had speech tokens, right? Very simple. I think they use k-means. I might be wrong though, just to find discrete tokens for speech. So imagine that you have a single model that has video images, text, speech, everything kind of put into one, right? Like what level of grounding and what level of zero-shot prompting can you get here? And I think a lot of people are kind of chasing this at the bigger companies. I'm kind of excited about that. On the analysis front, I think there's still a lot of unknowns about transformers. Like fundamentally we're still using the four-year-old implementation, right? The only difference is just pre-layer norm, right, from the original transformer. So I think better fundamentally understanding transformers. And I have some qualms with scaling laws. Like I don't think perplexity is necessarily the measure that we should be using. So internally we've been discussing like what does like memory-based scaling laws look like. So if you use memory as the fundamental unit of transformers, what do those scaling laws look like? So there's some more fundamental work to be done there. And the other thing is bridging, fine-tuning, and prompting performance. So far it's kind of orthogonal, which is, you know, if you want to get a better fine-tuning model, you have to do something that will hurt prompting and vice versa. So figuring out like is it just because we don't have like bi-directional like masks? Is that why? Is it because we only mask for like causal models and upper triangular matrix? Is there something more fundamental there? I think kind of peeling that apart and figuring out what's going on there is kind of important too. But I think we're very early on. I think this year is going to be the year of multimodal. I know they kind of kick stuff off. So I'm kind of excited to see what other groups are working on. It seems like it. Yeah. Is there anything else about the paper or the research direction you want to shout out? You want people to know that we haven't mentioned so far? Yeah. I mean, we'll be releasing all this code really, really soon. We're just waiting on some internal approvals so people will get to play around with it. I think we'll release three billion model, but the 13 billion model is the one that really shines. Yeah. So if people get that running, I think it's really cool. I spent hours just playing around with it. What does it take to just to forward propagate? What's the minimal configuration? So with the recent deep speed stuff that was released for inference, I'm not really sure because I think they said that you can use one GPU for like a 6.7 billion model. So if you do model parallelism, I think you need two GPUs. But without that, just give us a ballpark, what would it be like forward propping through this model? Yeah. So one thing is you could do it on a CPU if you have a strong enough CPU. But for inference, I think what I used was four V100s. Yeah. Model parallel. So less than a known. Cool. Excellent. Well, Armen, thank you so much for being here. This was really cool. Really valued the like also the kind of behind the scenes and insights we got here. And I hope to see you again very soon with even like CM4. Yeah, thank you for having me. Excellent.
 Today, we'll talk about CM3, which is a model that directly ingests websites, learns the
 HTML, it uses a novel objective that does left-to-right language modeling, but with
 a twist that essentially allows it to incorporate bi-directional information into the language
 modeling.
 It incorporates text, structure, images, hyperlinks, and with clever prompting, it can do almost
 anything.
 It can do what Dali does, generating images from text.
 It can caption images.
 It can do text summarization.
 It can do entity linking, and it can do much more.
 I like this paper because of the idea of incorporating the structure of HTML.
 And also, the new objective is very cool.
 So we're briefly going to go over what the paper is and does and how it works.
 And then we're going to jump into an interview with Arman, who joined me in talking about
 this paper.
 This is a very informative interview, and I suggest that you give it a listen.
 So this is just going to be a short introduction.
 Again, I have to rely on you to tell me how I make the best use of authors coming on,
 because I think it's so cool.
 I want to talk to them about the paper, and I want to get the most information out there
 for you that is possible.
 So please tell me short intros, long intros, how to structure it and all.
 Leave a comment down.
 If you like videos like this, leave a like as well.
 If you leave a dislike, you know, that's kind of useless now on YouTube.
 But you know, feel free.
 I'm still going to see it.
 So CM3, a causal masked multimodal model of the internet by researchers at Meta.
 I'm going to guess this is now.
 So this model is, it's a family of models, actually, and a family of causally masked
 generative models trained over a large corpus of structured multimodal documents that can
 contain both text and image tokens.
 In fact, much more.
 So what this model does, it's a language model.
 And the language model ingests HTML, a cleaned up version of HTML, but still HTML.
 If you don't know what HTML is, HTML is essentially the language your websites are written in.
 And it consists of tags.
 So for example, one tag is a div tag, that is, it's it has it had I think it had a meaning
 at some point.
 But right now, it just serves as kind of a container tag.
 So div might be something like a container, and you close it by saying slash div.
 Anything in between is the content of that div.
 Other popular elements are, for example, a paragraph.
 So inside a paragraph, you can have some text.
 Hello.
 There.
 And then what you can also have is hyperlinks.
 So hyperlinks start with an a tag.
 So you can see these tags can be nested.
 These tags can have attributes.
 So the a tag can have an attribute, like an href.
 So that is a URL, so www dot something, and so on.
 So it can have URLs, it can also have URLs within the document.
 Then there is the text of the link.
 Now we close the a tag.
 Oops.
 Then we may continue the paragraph or we may close the paragraph.
 A forward slash.
 And the last thing that we're also going to need in these documents right here are images.
 So there can also be images and I'm gonna write this over here.
 After all, whitespace doesn't matter in HTML.
 So images can have a so called source.
 The two most important attributes are the source.
 And the source is it's usually usually it's a URL, it can be a base 64 blob.
 But usually it's also a URL, like, I don't know, like imgur.com slash something something
 dot jpg.
 So the browser would actually go and fetch that image and display it at this position.
 And also, an important thing is the alt text, which you put there for screen readers and
 other sort of assistive technology that cannot directly make use of the image to see what's
 in the image.
 So you can already see here that there's a lot of information in HTML.
 Now previous work, what they would have done is if it's a language model, for example,
 GPT-3, they would simply only take the text bits of that they would take, for example,
 here, hello there, they would probably also take the text of the link right here.
 And and that would be it, they would scrape the websites for the containing text to do
 language modeling.
 Other models such as Dali, Dali, I've made a video about Dali, if you don't know what
 it is, but essentially a model that you put in text, and it gives you an image.
 And the reverse of that is is sort of clip, not the reverse, but clip is a model where
 that says whether or not an image or a piece of text go together well.
 And the reverse of Dali would be like a captioning model, you put in an image and you get a text
 describing that all of that you can get by also scraping the internet and always taking
 the following two things you take the alt text of a an image tag, and you take that
 source image.
 And these are pairs of images and text that go together, right.
 So you can train this is kind of like weak supervision, there are some problems with
 that.
 But it's weak supervision.
 Likewise, there are other tasks if you are, for example, doing entity linking or entity
 disambiguation or something, what you would do is you would go to Wikipedia.
 And on Wikipedia, you would always take the text of a link and the link itself if it points
 to another Wikipedia article.
 And you know, in this case here, it says like, Romans were captured by Alexander the Great,
 Alexander the Great would be a thing you could click on.
 And then that link would sort of tell you what entity that is it lead to the Wikipedia
 page of Alexander the Great.
 So people have parsed websites for a long time in various ways to achieve different
 tasks to collect data for different tasks.
 However, there is this new direction.
 And it's not the first paper that does this.
 But it is the first that I've come across.
 And the previous work is also by largely the same authors.
 So I'm just going to give them credit for some at least some of this.
 Basically, the the novel idea here is that why don't we use the entire structure of HTML
 directly in instead of just scraping subset of them.
 Now, again, they do clean the HTML because a lot of HTML is kind of like visual elements,
 cascading style sheets and so on.
 There definitely would be information there.
 But it is a good step to say, hey, the whole thing, you know, the entire thing here, the
 structure that is actually super duper important.
 It has so much structure that we would throw away otherwise.
 For example, the image right here, you know, it could be not only described by the alt
 text, it could also be described by like the surrounding text like this stuff right here.
 Of course, if there's an image on a website, reasonable to assume that the surrounding
 text might also have to do something with it, right?
 It is reasonable to assume that in order to disambiguate this entity right here, you might
 want to take a look at the text around it.
 You might want to take a look at the images around it and so on.
 So if we had a model that could directly learn the structure of HTML, we could exploit all
 the work that went into creating that HTML, which is essentially what front end programmers
 and website programmers do all day.
 This is human ingenuity that goes into creating these structures, even if it's a framework,
 right?
 That there's something, someone that has to come up with, you know, what are the elements?
 How is the structure?
 And that is really good data.
 And exploiting that data to me, when I saw this, it made perfect sense to say, you know,
 we should just keep the HTML and just learn the language model over the HTML, right?
 So what can you do if you have such a language model?
 Well, if I have trained such a language model, I can maybe, you know, start a paragraph,
 start a paragraph, I put like a piece of text right here.
 All right.
 And then I just start an image tag.
 And I say source equals, and then I'll let the model generate whatever is here.
 Right.
 Now, there is a there is a there is a trick right here.
 I can't obviously put a URL, I actually have to put the image itself there.
 And if the model is good enough, it will look at this, it will generate an appropriate image.
 Or you know, I could do the same thing by simply having an image tag.
 And first generating the alt first putting the alt text, I put something here that I
 want and then source and I say equals and then I let the model continue.
 It will generate me an image, I can reverse that I can put the image first and then say,
 please generate me the alt text, I can put an entity and say, please generate me the
 link to the entity, and so on.
 So you can see how powerful this is.
 We can do many, many different tasks if we have a model like this.
 This is one thing that this paper does.
 And I said it's inspired by previous work.
 However, it pushes it a bit further.
 So first we have to discuss this and then we have to discuss the novel objective, which
 makes it even more powerful.
 The only thing to discuss right here actually is how do they treat images because language
 modeling is fine.
 I can just have an appropriate tokenizer for HTML, which needs to be I guess a little bit
 of a different tokenizer than for regular text because you have to handle these tags
 correctly.
 But essentially, I have to have a tokenizer and transformers are pretty good at learning
 to open sort of appropriate tags and then close appropriate tags again and so on.
 The only part really are the images.
 So we don't want to have URLs of images in there.
 Instead, what they do whenever they encounter an image tag, so whenever they encounter image
 with a source that equals some URL, www dot something, what they do is they would go,
 they would fetch that image, they would put it through a, I think a VQ GAN model, some
 vector quantized GAN model that is pre-trained.
 They would extract the latent embedding from that and they would put that embedding here.
 So these models, these vector quantized models, they would take some image and have like a
 neural network and they would encode that into a series of tokens, which are going to
 be something like, I believe it results in 256 tokens, latent tokens.
 So these are essentially because it's vector quantized, every one of these is part of a
 vocabulary.
 And so these are essentially tokens like language model tokens, like letters that I can build
 images from.
 I can simply unroll, oops, I simply unroll the tokens in these images that the VQ GAN
 gives me, right?
 I can have some scheme of how I go through here and I can replace the source property
 here just with these tokens or I mean appropriately the embeddings of these tokens.
 All right, this goes here and so on.
 So once I have these tokens, right, I can train the language model and then the language
 model will generate these tokens again.
 Again, they're not continuous values because it's a vector quantized model.
 They come from a fixed vocabulary and that's what I ingest and that's what I predict and
 therefore I can treat it exactly the same as the language model.
 There is a bit of a difference with how these things are distributed.
 They do talk about this in the paper as language tokens are zypion distributed and image tokens
 are by design uniformly distributed but I mean essentially from a conceptual standpoint
 it's the same.
 The second thing they do is they have a different objective than language modeling.
 Language modeling usually goes left to right.
 So that means the language model whenever it generates a token it looks at what it's
 generated so far and then from that it will generate the next token.
 What it cannot do is it cannot look at the like right like the head.
 It cannot look ahead.
 You can't tell it, you know, here is a piece of text and here is a piece of text.
 Please fill in this piece of text.
 That would be a masked language model like BERT.
 But some a model like BERT isn't really good at autoregressively generating text.
 For that the left to right causally masked language models are much, much better and
 you know, higher performing.
 So is there a way we can get the best of both worlds or at least some kind of a trade-off?
 Turns out yes there is with the following objective.
 So as I said we have an example right here in a standard language model.
 We have the following thing which is a way we can do entity linking.
 So imagine we'd have to predict this piece right here.
 As you can see this is the link.
 It's an anchor tag.
 This is the link to the page, the Wikipedia page for Armenian nationalism.
 So Armenian nationalism, we want to predict that link which is essentially solving entity
 linking for this sentence.
 If we only have a causally masked language model all we can do is input this piece of
 text to the left.
 So this would be our entire context.
 Now this example is constructed such that this thing right here, this word right here
 is really important to classifying, to seeing what is there.
 Therefore if we only had a causally masked language model, if we only ever trained left
 to right, we couldn't make use of the word that was behind right here.
 If we had something like a masked language model we could absolutely do that.
 So that is this example right here.
 If we had a masked language model then we could absolutely do that.
 We could input this and we could input this and we could say, you know, here is a masked
 token.
 Please generate what's in the masked token.
 However we already discussed the weaknesses of that approach.
 Instead they have a new objective which they call a causally masked language model.
 Now I called this before a causally masked language model because there's also this sort
 of causal mask inside of it.
 I'm sorry.
 The causally masked language model is the thing they are going to propose.
 Inside of these language models usually there is something like causal masking.
 So it's a bit confusing if I look at this right now.
 What they do is during training.
 So during training what the masked language model would do is it would just mask out these
 parts and then it would try to fill them in.
 This limits training because you can only mask out so much.
 You can't train in parallel and so on.
 Whereas with the autoregressive language models you can train a lot of stuff in parallel.
 There is none of these noise and so on.
 Everything is decomposed nicely.
 Here what we would do is we would take the things during training.
 We would simply have a span that we mask but we don't just leave it away.
 We actually put it at the end.
 And there is an identifier token right here to show.
 You can see that this token right here and this token right here are the same.
 So we tell the language model.
 We tell it, look here is a sentence.
 There is a mask right here.
 There's something missing.
 It could be one or many tokens.
 And then here we want you to generate that thing again.
 And the model simply has to generate the thing back here.
 There can be one mask tokens.
 There can be many of these mask tokens in which case we just, you know, if we mask something
 else like this right here, we just put the corresponding token right here and ask the
 model to generate it on.
 The model will learn if there are two mask tokens.
 The model will learn to after it finished the first thing that it's supposed to produce
 to automatically put the next mask token there.
 So that is the objective.
 It still benefits from this left to right thing.
 As you can see, we can train this left to right.
 Once we reorder the sentence, we can just input the whole thing here into training.
 We can train it like a decoder only language model and we get all the performance off of
 that.
 Yet we can still do kind of like masking.
 So we get bidirectionality by design, because now if we want to predict this mask right
 here, we have seen all of this context.
 So essentially we have seen the whole data point.
 We do sacrifice like a little bit of performance because, well, inherently this part here is
 still left to right.
 So there's that.
 Like in itself, it's still left to right.
 Also, we do take stuff out of order.
 So there is the question of, you know, how long can I memorize stuff and so on with transformers
 maybe a bit less, but we do take stuff out of order, which introduces some noise and
 so on.
 So it is definitely a trade off wherein pure language modeling is still going to be more
 powerful.
 But this now enables us, this enables bidirectional context essentially into the things that we
 generate.
 And that has a lot of advantages for many, many different tasks.
 There is a whole scheme.
 It seems to be really important how exactly, oh yeah, 256 tokens for each image.
 Sorry.
 It seems to be quite important how you generate these masks during training, how long they
 are.
 They try to make them quite long in order for the model to learn important structure
 and so on.
 We'll go through all of this in the interview.
 The scaling laws are pretty astonishing in that they're large model right here.
 And these are large models, right?
 These are like the scale of this.
 It was trained on 384 A100 GPUs.
 No, I think that's even the baseline.
 That is even the baseline.
 Where is their model?
 Yeah, I don't currently find it.
 But you can just see sort of the scale here of what they're going for.
 So these are not small models.
 But if you make them sufficiently large, you can see that largest models, they're not done
 training yet.
 Even after they put sufficient or put enormous amounts of resources through them, you can
 see they're not even the same ahead.
 Like the same advanced inside of the training.
 So yeah, this is very promising.
 I think this is a very promising direction to make use of that, to make use of the HTML
 structure.
 You can see a little bit here.
 So essentially, if you just put this as a prompt, you can have the model generate the
 alt text and the image at the same time, right?
 It interestingly chooses to put the alt text in front, like it chooses to generate a little
 description before it generates the images, which is interesting.
 You can also force it to first generate the image by just putting the source tag directly.
 So then it needs to generate the image.
 And it's interesting because the quality of the images when you force it to generate image
 before alt text, it is a lot lower, as you can see here, than if you just let it generate
 the image, in which case it chooses to generate the alt text first.
 You can do many things.
 You can do image inpainting by masking out a portion of the tokens of the image.
 You have to mask out entire tokens, but still you can do like crude image infilling.
 You can do conditional infilling by providing alt text first and then do infilling.
 You can do conditional generation by providing alt text.
 So the possibilities are very, very great right here.
 You can see this is infilling, conditional infilling, and so on.
 The possibilities are great.
 And remember, this is a very particular data sets and very particular cleaning methods
 of HTML.
 I believe if we extend this to even more structure and so on, maybe even take cascading style
 sheets into account, take all of the structural elements of websites into account, title tags,
 headers, footers, and so on, this could be really powerful beyond the applications that
 we see right here.
 They can also do pure text modality data sets.
 As we said, entity disambiguation by predicting hyperlinks.
 They also do get new state of the art in zero-shot summarization by simply generating like the
 title or the meta tag, the description tag of the website.
 They give it a fake website with the text they want to summarize and they generate these
 tags.
 They do say for completeness below is an example of a prompt that can do basic summarization.
 I did not find that prompt anywhere.
 So yeah, maybe I didn't look enough or maybe LaTeX screwed up where some kind of a figure
 is.
 In any case, I don't want to go too much into the results right here, but I think the direction
 of using that structured content is pretty cool.
 The new objective is also pretty cool.
 I do criticize a little bit that these two things are kind of decoupled from each other.
 Like they could all be their own paper.
 And that's also something that we talk about in the interview.
 So in the interview, we're going to go briefly over the model again, over the research process,
 over what it means, what it could enable and what difficulties there were and also over
 the results, which are extremely, extremely interesting.
 I enjoyed the interview a lot.
 I hope you do too.
 Tell me what you think of it and now I'll leave it up for the interview.
 Thank you very much and have fun.
 Welcome everyone.
 Today I have with me Armin Aghajanyan and I've practiced that name 10 seconds ago and
 I think I got it down.
 Armin is the first author of the CM3 paper.
 Welcome Armin to the channel.
 Thank you for having me.
 So I saw this paper and of course you have like some big names here.
 There's lots of authors, there's Facebook AI research.
 But still, like given all of that, it was still impressive.
 Like I was impressed by what it could do and sort of the results it gave.
 Like it seems to be, wow, there's zero shot, there's image generation, there is like a
 new objective, there's HTML in there.
 So there seems to be a lot in one pot.
 If you gave the pitch, I will have made an introduction, but if you gave the pitch to
 the paper, what is it mainly about?
 The goal here was kind of to have a single multimodal model that can do everything.
 Image generation, image captioning, image infilling, to even pure text tasks like summarization,
 but mostly focusing on this zero shot setting, specifically this popping setting.
 And how did you, like, were you, this is a very popular thing.
 I think in the last few years, this came up, maybe starting with something like GPT-3 where
 people could really say, okay, stuff is possible zero shot if we train on large enough data.
 Then came things like Dali and so on where, you know, we saw for the first time, okay,
 maybe stuff is even possible in other modalities than text.
 This goes even further.
 This is multimodal.
 There have been a lot of other approaches to multimodal.
 There is like this Rudolph even model.
 I don't know if you've seen that.
 It goes like image to text to image and so on.
 And they all work, let's say, with very cleaned up data.
 It's very, you know, I want text, I want images that go with the text, which makes sense,
 right?
 So do you get, how did you get the idea to use, let's say relatively unstructured HTML
 for this?
 Like, how did your thought process go until you came to this idea?
 So usually there are pros and cons having super strong alignment, right?
 So like Dali, for example, they have like a very specific alignment of like, you know,
 text on the left side and then you have like 1024 image tokens on the right side, right?
 Super strong alignment.
 And in general, it's easy for the models to kind of learn this type of single alignment,
 but then you're incredibly limited on the prompting side.
 And I think it's incredibly creative.
 If you have a general model, it takes a little bit of creativity to extract out the prompt.
 So the key here is we don't want to have any strict alignment in terms of the modalities.
 So the goal was like, what is the weakest alignment that we can go for that would still
 give us the ability to prompt in non-trivial ways?
 So actually this is kind of a follow-up to an older paper that we published.
 It was just accepted in ICLR actually, which was this HTLM paper.
 And the core idea of this paper is that we argued that document structure is really,
 really important.
 So what we did there is we took BART large and then we pretty much trained it on just
 web data, like minimized HTML.
 So minimal HTML is we pretty much do multiple passes over the DOM and take out anything
 that we don't think is semantically important.
 So in that paper, we showed really strong results.
 So for example, for zero-shot summarization in a structured language like HTML, this is
 pretty much just generating the title or generating the meta tag where the attribute is the headline.
 So in some sense, we could exactly replicate how CNN and Daily Mail was collected, which
 was they looked for headlines.
 So in the prompt, you can actually describe the way that the data was collected.
 So we saw that there was some rich structure available to be used in HTML.
 So after Dali came out, we thought, okay, there are some fundamental restrictions with
 Dali.
 So the first one being the causal approach.
 So they train a decoder only left to right model.
 So in some sense, you can't do things like generate the text given the image, right,
 just because of the positioning of the image.
 It's on the right side of the image.
 You can't really do image infilling either, which means conditioning on both the prefix
 and postfix of the image.
 Or you'd have to train specifically one particular type of infilling.
 You could rearrange stuff such that you could infill one part, but you can't dynamically
 infill something.
 Exactly.
 Yeah.
 So those were kind of the first weaknesses that we saw there.
 The approach was very clever though, right?
 So pretty much taking continuous data, discretizing it, and just doing sequence modeling.
 It seems to work very, very well.
 So the idea that we kind of combined the two from the HTML paper, which was that document
 structure through HTML is really important, but let's also encode images there and see
 if we can recover something like Dali.
 So here you're kind of looking at the data that we collected.
 So the data set size is actually quite good.
 I mean, we're around like the 200 billion tokens, which is a relatively good size if
 you're training large models.
 But one kind of downside that we have here is because we don't have the strict alignment,
 we can't artificially increase the amount of images that we have available in the documents.
 If you actually look, I think we have 25 million unique images.
 I don't know about Dali.
 Dali was trained on 400 million.
 I don't know how many of them are unique, but regardless, they still have an order of
 magnitude more images than we do.
 But then we have the other benefits, which is we're also training on a ton of text.
 So we can do a lot of text only tasks.
 And I think the rest of the paper will show that we can do not only text only tasks, but
 we're actually competitive to T5, which is actually really hard to do.
 And I can explain why we think this is the case in a little bit.
 So the very first thing was, okay, so now we kind of have this data, but HTML is also
 very localized, right?
 Like the title always comes first.
 It's in the head, right?
 Or like the meta tags always pop up first, right?
 So if you want to generate meta tags or generate title, right, condition on the rest of the
 text, it's kind of non-trivial how you would do this in decoder only setting.
 And so we kind of started thinking, there are multiple ways around this, right?
 So the first thing is using encoder decoder architecture, right?
 And then with some masking, you can kind of recover this type of bidirectionality.
 This is true, but there are pros and cons to this.
 So encoder decoder only architectures, they're really good for fine tuning, but they're not
 so good for prompting, is at least what we noticed.
 And also training them is a little bit more non-trivial.
 So decoder only models are quite nice because you get per token generation.
 So you pretty much generate every token for the source.
 Whereas for encoder decoder, most of the time you're generating, I think like 15% is what
 Bert and Bart or Roberta do.
 It's all around that 15%.
 So most of the times you have to go through the data multiple times.
 For some reason, they don't prompt super well.
 And the kind of the other big thing is if you want to do score-based prompting, it's
 kind of hard to do with encoder decoder only architecture, right?
 If you want to ask what's the log probability of this sequence with the mass language model,
 it's kind of tough to do, right?
 So we knew that we wanted to go kind of this decoder only route.
 So we introduced this new objective that we called causal masking.
 And so the idea behind causal masking, if you want to scroll down, I think there's a
 figure there.
 This one.
 Yeah.
 So the idea there is relatively straightforward, right?
 So pretty much think of mass language modeling, where you place in the mask, but take the
 mask and put what the mask represents simply at the very end of the sequence.
 So if you do this, you kind of get, it's very, very simple, right?
 But you get a lot of the benefits, which is you still get per token generation.
 You optionally allow for bidirectionality, which is actually a really, really big thing
 to have, right?
 And the other thing that we noticed is that depending on the sending, prompting versus
 fine tuning, the size of the mask is really important.
 So for fine tuning, localized information is really important.
 You want to have a lot of small masks.
 For prompting, we saw kind of the opposite, which is you want to have very, very few masks,
 but they can be very long.
 So the strategy that we use here is for every document, we sample from a Poisson distribution
 centered around one.
 So the majority of times, right, and we clip it to one.
 So if you get zero, it becomes one, right?
 So majority of times, you're only going to get a single mask, right?
 Over 50% of the time, you're only going to get a single mask.
 And then you pick, you uniformly sample a subset of the document of any size, and you
 kind of place that in the end.
 So you get these very, very long kind of infilling naturally.
 And so this objective turned out to be quite strong.
 So it's competitive to language modeling in the sense that when you get per token generation,
 our perplexities were not that much higher than just a language modeling objective.
 You get optional bidirectionality whenever you want it, right?
 You can score probabilities of sequences super, super easily.
 So we're kind of going all in on this objective.
 And so we have some follow-up work looking at causal masked scaling loss for text.
 So this is some ongoing work that we have now.
 So we're pushing heavily on this.
 So the general argument that we're trying to build is that if you're doing language
 modeling, deconormally language modeling, you should be doing causal masked language
 modeling.
 So that's kind of my...
 Yeah.
 I mean, it is intuitively a good trade-off.
 So I think here you make the case, if I interpret this correctly, that this word nationalist
 right here is really important to fill in this mask.
 And if it were just sort of left to right, it would be very difficult to fill this in
 yet since you move it to the end, right?
 And the model has to extra learn kind of to keep these tokens in context to sort of realize
 what's there.
 So it has to waste kind of some extra memory to remember the context of each of the mask
 tokens and so on.
 But yeah, I think it is very intuitive.
 It is also a good trade-off between, I want to say, left to right has, at least for most
 there are right to left languages, but for left to right languages, left to right objective
 actually makes sense, right?
 That is how we generate language when we write it down.
 So there is something to left to right that I was never happy.
 There are other approaches like XL net or so.
 They were saying, well, we just train on all possible paths of decoding, like all possible
 sequence of masking out tokens.
 And it was never really satisfying because I always thought, but there is something to
 left to right.
 However, sometimes as you say, it's really important to know what's after.
 And I think this is like a really good trade-off.
 Yeah, like specifically in this example, in the zero-shot prompting case, let's say we
 want to tag nationalist with some entity link.
 If it appears beforehand in the sequence, there's no way to prompt the language model
 to generate an entity link before the entity appears.
 So that was kind of another reason that we had because like I said, HTML data is very
 localized.
 In Wikipedia, this a tag which represents the entity link always appears before the
 entity.
 We have the option of training two models, one left to right, one right to left.
 Or you can kind of do this kind of clever rotation of the document.
 Yeah, the XL net approach is definitely interesting, which is having different permutations of
 the source document.
 But like you said, I think there's a lot of inductive bias for left to right, which is
 why I think left to right models are kind of de facto now.
 Just for my understanding, is there a reason behind these arrows?
 Why do the arrows are like double arrows, then there's a line and there's like a double
 arrow again?
 Does that have a specific meaning?
 And here the arrows are only here?
 Yeah, so arrows pretty much was the tokens that you actually generate.
 So in the language model, you're generating every token in the mass model.
 So you go like this, okay, I see, I see.
 Because I was like, okay, is there some meaning?
 But yes, there is.
 And this shows that in the mass language model objective, you only actually generate very
 small number of tokens and you wouldn't even get like a loss for the other tokens.
 You said before that you had a certain number of tokens, right?
 And you said, well, that's actually good or bad for, you know, that's actually in a good
 order for language modeling.
 Yet a special thing about your model is that images are also tokens.
 You push images through a VQGAN encoder, right?
 Which is pre-trained.
 And these just become tokens in whatever sequence.
 And this results obviously in larger data because some of it is images.
 So you say you have a terabyte of data in this data set, which is obviously way larger
 than for example, a text only data set.
 Do you find there is a difference?
 Like do you find the number of tokens is really what matters in the size of the data?
 Or is there a qualitative difference between image data and text data, even though both
 are tokens?
 Yeah, so there's a couple of ways to approach this.
 So the very first thing is that modeling, and I think we mentioned this quickly in the
 paper, but modeling image tokens versus text tokens, it's quite different actually.
 So for like text usually follows like textual tokens follow like a Zipfian distribution,
 right?
 Whereas I think in Appendix we have a figure, it's pretty much uniform for images.
 So there's different like in terms of the distributions that you have to predict, they're
 actually quite different.
 So we saw a little bit of challenges and we saw some kind of weird behavior during training.
 We didn't mention this in the paper, but the one weird behavior that we saw was that there
 were regimes during the training, like parts of the training that only optimized for text.
 So on our image evaluations, like it pretty much would be flat.
 And then there were times that it was quite the opposite where, you know, images would
 be being optimized for the text kind of stayed flat.
 So we don't really have explanations for why this is happening.
 I think there needs to be future like scaling laws looking at multimodal sequence modeling.
 And when I say multimodal, I'm not just talking about like images and like natural language
 text.
 I meant like you can even include code as a different modality, right?
 So the scaling laws there I think are a little bit different than what we're used to with
 the text.
 The reason for using tokens is purely because of a compute thing, right?
 So you know, we're given some amount of GPUs, right, for some amount of times.
 So what we do is we take the number of tokens that we have, we take the amount of compute
 that we have and try to find a larger size model that we can train.
 It's kind of an optimization problem to find the largest architecture.
 So that's kind of why we used number of tokens as the guiding principle.
 I mean, it seems to also align with what others...
 Yeah, for example, this Rudolph paper.
 So it seems to be a common approach to lift images into like the space of textual tokens,
 which is, I guess, a bit surprising because a couple of years ago, no one would have gone
 that route.
 Even if you were to inject images into a sequence model, you'd probably inject like a single
 vector, right?
 So I find that to be a bit surprising, but also, yeah, it seems appropriate that an image
 could be expressed in something like a sequence of tokens.
 It's just a bit...
 I'm not too big of a fan of how this is currently done because the tokens, they also...
 They seem to be a bit localized in the image and so on.
 I think there's a better way, if you're a human, that's not really what you do with
 an image.
 You see more like the different layers maybe or what's there.
 In any case, I was surprised by these scaling plots.
 These are brutal.
 We scale it up and the loss goes down for the largest model.
 It seems you're nowhere near done, right?
 You said you had some different experiences during training, yet also, I think in the
 paper somewhere you hinted at, well, we didn't really see any pathologies.
 What was the process like?
 You had the data, you trained the thing, did it immediately work?
 It took a little bit of handholding to work, especially the 13 billion parameter model
 took a little bit of handholding to work.
 A lot of the times the pathologies we see are things like gradient, underflow or overflow.
 Gradient explosions happen, although they usually happen in much bigger models like
 the 100 billion scale.
 But the surprising thing was that we almost used exactly the same hyperparameters as this
 paper that came out from Vesto in those group.
 So the surprising thing is it kind of just worked out of the box apart from having to
 tune, I think we tune like learning rate, we had to tune weight decay and batch size.
 Apart from tuning those things, it just worked almost straight out of the box.
 And what you said is actually correct, which is if you look at the large model, it's actually
 not done training.
 So the good news is once CM3 is released, we're going to release the checkpoint that
 we use for this model.
 I think the model that we have now is continuing training.
 So we'll really release that one too.
 So people will be able to play around with both.
 Excellent.
 But one thing I'd like to point out is that the multimodal scaling laws are a little bit
 different than text scaling laws.
 One thing seems to be that scale plays a slightly larger role in multimodal than it does in
 text.
 So I think the quantitative thing that we saw is that if you look at the data efficiency
 jumps between like, I'm forgetting the exact numbers, but like let's make them up, like
 the 1.3 billion model and the 13 billion model from Vess's paper.
 And the data efficiency there, let's say it was like the larger model was five times more
 efficient in terms of data.
 So in order to reach the same perplexity, it would need five times less data.
 Using the same exact models, we saw that in the multimodal case, it was 10x.
 So there was almost a two times difference for some reason.
 And that's why I think it's really important to kind of chase these multimodal scaling
 laws and fundamentally understand what's going on here.
 There's a lot of unknowns here.
 When you say we had to do a little bit of hand holding, what does that even mean in
 these large models?
 Like, can you afford to restart training?
 Or is it more like, you know, you have checkpoint, checkpoint, and then something goes wrong
 and you go back to the last checkpoint and you do something there?
 Like what does the process of training these very large models look like?
 It's just really, really tedious.
 So one of the main things is, you know, whenever you have a ton of nodes that you're running,
 there's infrastructure issues that pop up, right?
 So like if one GPU goes down, right, then all of training is paused, right?
 So infrastructure issues are kind of a big thing and we have some automated systems in
 place to take care of that.
 Other things are like, for example, like we didn't set a high enough warm up period in
 the beginning.
 So we saw that we actually had to pause training, increase the warm up, load up the last checkpoint
 and go there.
 And so we also kind of tuned learning rate a little bit as training goes on.
 Although with the large models, I think it might have been just a handful of times.
 So failures-
 Do you always have like multiple models running ahead and then you choose the one that looks
 best or is it really like you change and you train one model and you see how it develops?
 Yeah, because of the computer is one model.
 So it really comes down to intuition.
 So both Mike Lewis and Naman Goyal who are on the paper have trained these really, really
 big models before.
 So they had a ton of great intuition about how to get things to work in terms of these
 very large models.
 Cool.
 I mean, yeah, I'm excited and it is very cool that you actually are going to release these
 things.
 I think people will love to play around with them.
 In order to do now the tasks, you tackled some tasks.
 How did you decide?
 Wait, there are some natural tasks, let's say there are some that are more, you know,
 you have to come up with something.
 Did you have some targets of tasks that you want to tackle?
 Or was it more like the model came first and then you sat down and saw what can you actually
 do with it and whatnot?
 And what worked and were there also tasks that you tried that maybe didn't work at all?
 Yeah.
 Yeah, that's a great question.
 So I think at the beginning of the project, the push was really to have a single model
 that can do any image task in the zero shot case.
 And so kind of the story that we built around it is, can we describe all the tasks that
 we're interested in through some prompt, through some HTML prompt, even before we train the
 models we got about this.
 So we came up with a ton, right?
 And some prompts were very complicated, like style transfer for one.
 So you can have an image that has a picture of the mountains in the summer.
 And then you have another image tag that says the same picture, but in the winter.
 And then you ask them all to predict the image tokens, right?
 So you can get this kind of zero shot style transfer.
 So you have some kind of complex prompts.
 So some of them didn't work.
 Some of them only worked at scale.
 And we can kind of go through this.
 Specifically like one thing is that like the captioning only worked at scale.
 So their team building model was the only model that could caption well.
 And the captioning, you go mainly with the alt text of the image.
 Alter the title, either one.
 Yeah.
 But like the figure that you're on now, I think is kind of interesting.
 So we can kind of get unconditional image generation by just asking the model to generate
 a sequence of tokens after the image tag.
 So we saw one interesting behavior is that the model for some reason almost always wanted
 to first generate the alt text before generating the image.
 For it was actually easier to condition on the text before generating an image than doing
 this type of free form generation.
 When you say it wanted to, that's just what it did.
 Yeah.
 Like when you sampled, did you like, I mean, when you say it wanted to, it could also be
 that in the internet, humans most of the time write alt first and then the source.
 Yeah.
 So we actually looked into this.
 So a lot of text does have alt, but it's around like, I want to say like 70 to 80% mark, if
 I recall correctly.
 So it wouldn't explain why the model almost always wants to generate alt text.
 Now the theory that we kind of have is that without alt text, you have much higher perplexities
 for images.
 So the model, because we're doing like sampling, right?
 So it's going to pick out high probability, low perplexity tokens, which most of the case
 means picking out the alt just because it appears so often.
 So that could be it.
 But overall, I think if you look at these images, they're rather like, they're semi-coherent,
 especially the ones conditioned on the text.
 And the same thing I think you see with, you can kind of force the model not to generate
 the alt text by giving a prompt and generate the image tokens immediately.
 And do you think, so the VQGAN tokens, naturally they are predicted as one, right?
 There's some encoder, they're not, as far as I understand, they're not in the image
 encoder that makes the tokens, they're not predicted autoregressively.
 So there is no inherent sequence nature to these tokens.
 Could that be like some sort of a reason why there's also a difference?
 Because text naturally is sequential, whereas these tokens, the only thing they have is
 they're kind of localized, but there's no inherent sequential nature.
 Yeah, that's true.
 For VQGAN, there isn't something explicit, but I think the way that the layers are constructed,
 we do still get some implicit dependencies across the tokens.
 And so I think this is what the transformers kind of pulling apart here.
 And to be honest, I think there's still a lot of work to be done on the discretizing
 images front.
 So one thing about VQGAN is that it blurs a lot of fine detail, so like human faces.
 In our case, this is kind of good because it's privacy preserving, you're not going
 to generate like a person's face unless it's a really, really popular, like close up face.
 So in our case, it kind of worked out.
 But in the future, I think we need to get much, much higher fidelity image tokens if
 we think that the way of doing things is to treat everything as a token.
 Of course, I think there are a ton of new approaches that are not token based.
 I think Glide was fantastic from OpenAI.
 The diffusion models are doing great generative work.
 But if you want to maintain the same benefits of generative models, so being able to generate
 trivially, being able to compute log probabilities, I think tokens are probably the easiest way
 to go.
 And one thing is you can naturally increase the resolution of tokens images just by increasing
 how many tokens you use per image.
 So in some sense, if you have enough compute, you can scale up to arbitrary resolutions,
 right?
 Yeah.
 So probably, you could at some point get more tokens than pixels.
 I wouldn't know what that would mean.
 But I guess the resolution isn't even limited by the resolution of the image itself.
 So there's this interesting thing you can do, as you said, infilling by letting the
 model generate sort of middle tokens.
 Now you could probably do arbitrary infilling, but you have to have multiple mask tokens.
 So I guess the natural thing to do is just to infill, since the tokens kind of go left
 to right, top to bottom, is to infill one of these stripes, which you've demonstrated
 right here.
 Did you try infilling arbitrary things?
 Or was this sort of the natural thing to do?
 Yeah, so actually, because of our objective, because we sampled the number of masks, right?
 You can actually mask out like five, six, seven masks, and it still work.
 I don't think there was any specific reason that we stuck to masking out a single thing.
 I'm sure it would work with multiple as well.
 I mean, if you were to infill, let's say, if I infill a square like this, and it covers
 sort of multiple token lines, this would already result in like if it covers three token lines,
 it would already result in like three mask tokens, right?
 So I mean, there is some with just with the sequential nature.
 But I think that can be can be worked around.
 So what here we see, so left is source image, then you mask out something in the middle.
 Then you also give the ground truth, which is here on the right.
 And then there's one model that does infilling unconditional.
 So just looking at the image.
 And then there is one model that does it conditionally.
 And the conditional is conditioned with this thing right here as the the alt text.
 So you understand, okay, so understand it correctly.
 I was, yeah, I mean, I was surprised, for example, by this one right here, this, the
 park bench, because obviously, if you see the the model that does infilling conditionally,
 it can do it quite well.
 However, the unconditional one, it kind of warps the bench or something like this.
 Like it's it's a bit I'm not I'm not sure the unconditionality has something much to
 do with it, because there is no this doesn't look like natural, you know, you know what
 I mean a little bit like, yes, this shouldn't be like, just because it's not conditioned
 on it.
 If it's not conditioned on text, I would expect it to be maybe a red bench, right, or, or
 something, you know, something that is conceivable in nature, but is not according to the text,
 like there is an ambiguity of what's behind the mask.
 However, here it really seems to degrade in performance when you don't give it the text.
 Yeah.
 So so one theory that we kind of had here is that the the model needs to understand
 the continued continuation of the the horizontal lines, right?
 That requires some semantic understanding that this is, for example, a bench, right?
 And actually, if you look at the the massed out input, the horizontal lines are not completely
 horizontal.
 The top of the bench is at a different angle than the top of the bench.
 So I think the model has a tough time understanding the high level semantic content of the image,
 which is fixed by feeding in text.
 Yeah.
 Now, I think, of course, if you have I think if you have a larger model that's trained
 for longer with a higher resolution, this probably should not be an issue.
 VQV, again, it blurs out a lot of things.
 Number one.
 Number two, it's just if you change the tokens even a little bit, the blurring aspect happens
 very, very quickly with VQV again, compared to, for example, the VQV from Dali, which
 requires more tokens.
 So 1024 tokens versus the 256 we use here.
 But it's more direct in some sense.
 Yeah.
 So, yeah, I think the main thing here is just that you need to get some like high level
 semantic information about what's going on in the image.
 And it's hard to do if you're only looking at like the VQV GAM tokens.
 Yeah.
 Okay.
 I mean, that makes sense.
 You go on and you have some examples of conditional image generation.
 On the left side here is a prompt and then you sample images from that with the same
 technique, right?
 You give the alt text and then you sample the image.
 So the avocado chair is like forever going to be to stick in history, right?
 I think that's just a given.
 Was there something that surprised you with conditional image generation?
 Yeah.
 So the models are quite good at actually generating something that's somewhat coherent.
 So for example, like the red car, you can see it generates two red cars.
 That one looks like a truck or a tractor.
 Sometimes the model tries to cheat and generate something that's easy.
 For example, in the case that it doesn't generate a car at all, it just generates mountains,
 right?
 Just because the landscapes are easier to generate.
 The other thing that we saw kind of tough compared to Dali is the data that we used
 only came from Wikipedia or Common Crawl News.
 So none of it was fictional in some sense, right?
 We don't have any like art.
 Yeah.
 So like our images always try to be as non-fictional as possible, which is it acts weird if you
 try to give it like really fantasy based prompts.
 Yeah.
 So that's kind of one downside.
 And actually this is one criticism I have of the evaluation that we did for the FID
 matrix, which is a way to measure the quality of images, which is we actually took the table
 from Glide for the FID numbers on the conditional generation.
 One thing was is that MS Coco is almost all non-fiction, like non-fantasy images.
 So this is really like it's under-representing Dali.
 So I think if you casted a wider net here and had something that included a wider array,
 a bigger distribution of images, I think Dali's results here would be much, much stronger.
 Which is why I think we're kind of comparable, our largest model is comparable to Dali on
 MS Coco.
 But in terms of image generation, it's not as good on the fantasy front at all.
 You did discuss a little bit.
 You also said you sub-sampled web data and you cited some concerns as well.
 But there is also quality issue with sort of the wider you cast the net, the sort of
 more the quality goes down, I guess the alt tags quality go down, whether or not the images
 even have alt tags, whether or not they're ads or something like this.
 Why did you limit to this subset of the data and not bigger or smaller?
 I think at the beginning we had some ethical concerns.
 Like I said, we have very weak alignment, so you can prompt with anything, right?
 We had some ethical concerns about images that you can generate if you were just trained
 on all of Common Crawl.
 So we try to think about what are large scale data sets that we can get that are somewhat
 filtered.
 Wikipedia is definitely one of them.
 But even then actually Wikipedia itself has a gender bias and I think this is a new, I
 think other papers have showed this before.
 And Common Crawl News, which probably is not going to have the terrible content that we
 don't want to pick up.
 So we kind of picked those two and it was okay at the scale that we wanted to.
 So we stuck with those two.
 But yeah, I think it's hard.
 I don't know what the solution is.
 Like the lay on 400 million data set that was released.
 I don't know if you've heard of it, but this data set, I think there was a critique paper
 written like a month about it, right?
 That showed that it was like a highly, highly problematic data set.
 So in terms of the ethical approach, I'm not really sure what the right answer is for collecting
 at scale.
 There are tricks you can do, right?
 So like if you look at the CC100 data set that Facebook collected, they use this trick
 that they train a language model on Wikipedia and then use it to score Common Crawl and
 then take only like medium perplexed from Common Crawl.
 So you could probably do something like this here.
 I questioned the efficacy just because very large models, they only need to see a data
 point a couple of times in order to pick it up.
 So I think there's like some very fundamental engineering work that's being done for scaling
 up these data sets to like trillions of tokens essentially.
 Yeah, I mean, I guess it casts much wider questions such as, you know, I as a human,
 I'm perfectly capable of going to 4chan and seeing kind of the worst of humanity and it
 doesn't instantly make me like, you know, I don't know, a terrible, terrible, like it
 doesn't make me want to repeat everything or something like this.
 And there's various considerations like shouldn't we be able to build model that also ingests
 stuff but kind of may also a bit distinguish between things?
 Like if the models are able to distinguish, it might help them to ingest more of this
 critical data.
 But on the other hand, I can absolutely understand that, especially if you're the maker of a
 model, you don't want your model to output, you know, that I think that's why for example,
 OpenAI keeps such a tight grip on GPT-3.
 If you want to build anything with it, right, you have to go through approval processes
 and whatnot.
 And it's, it's, yeah, it's, I think it's tricky topic.
 I also don't know what exactly to do.
 I'm happy that there are models that are filtered, like say on filtered data.
 I'm happy that there also exist models that aren't.
 Yeah, I think the, maybe the sort of the, let's say diversity makes is, is probably
 the best.
 So you can always choose which one you want to, you want to use.
 I don't know.
 I'm sorry, this is just a rand by now.
 You do have some, sorry, go ahead.
 I was going to say, with respect to what you're saying, there's, the solution doesn't necessarily
 have to lie on the language model side.
 Yeah.
 So one thing is you can think of language modeling as just pure density estimation over
 tokens, right?
 So if you're doing that, like, of course you're going to model like 4chan, for example, right?
 But it's up to your generative sampling strategy to remove that part of the density and only
 sample from parts of the density estimation that you know are safe, for example.
 And so we're actually seeing, I think, a lot of movement from having a singular model that
 does generative work and to having like multiple models.
 So a great example is like Dali, right?
 So they do density estimation over, you know, text and image tokens, right?
 But the way they generate images is they sample like 128 candidates and, or whatever number
 of candidates, and then they use CLIP, a secondary model, to kind of select in some sense the
 mode of the slice of the density, right?
 And so something probably similarly can be done here.
 Like a great example is like take Codex, for example, right?
 I think in the Codex paper what they do is they generate a ton of samples and then they
 re-rank the samples in terms of perplexity, so average probability, and then they take
 the mode.
 So essentially the exact mode of that density estimation, right?
 So one thing to argue is that, you know, you could train language models that do pure density
 estimation over all the text that we have and then have smarter generation algorithms
 that are able to select subsets of that density that are safe.
 So like you said, in terms of research, I think there's pros and cons to having unfiltered
 and filtered models, but that's kind of the way I've been thinking about it recently.
 Yeah, and it's probably a good approach because the sort of the handle we have on, let's say,
 discriminative models like CLIP is a lot larger than the handles we have really on generative
 models like, yeah, the only handle really we have there is kind of data.
 You also do some experiments on text pure, I don't want to say pure text data because
 it's more than that, right?
 It's entity disambiguation, entity linking and so on.
 Now, is that purely a result of the fact like of you use Wikipedia as a data source and
 Wikipedia is essentially, it's not really only text, it's kind of a huge entity link
 and database.
 Is that kind of, is it fair to say that it works really well because you use Wikipedia
 as data or is there something more to it?
 Yeah, no, that's exactly it.
 So actually, there's this work that we sent in this paper a couple of times, the genre
 paper.
 So in the genre paper, I think the paper is called auto-aggressive entity linking or entity
 disambiguation.
 So the idea there was exactly that, which is if you take all of Wikipedia and then you
 train a language model that tries to predict entity link post entity, you get a model that
 does really, really good entity linking, right?
 So in some sense, the genre objective was a subset of our much more general objective,
 right?
 And it's not too surprising we beat out genre just because our models are bigger in our
 fine-tuning case.
 But the really, really cool thing I think was that we can do the zero shot, which is
 exactly what I showed in the first figure.
 If you mask out the entity, if you know that you want this entity, you want to disambiguate
 this entity, you can place a mask there with this a tag, right?
 And then our model will fill in what it thinks the disambiguation is.
 So that's kind of cool.
 I couldn't find any zero shot baselines like this.
 So I think this is kind of the first paper to do this type of zero shot entity linking
 and disambiguation.
 And so, I mean, you also have other tasks like summarization.
 We also didn't look at the alt text generation and so on.
 Is there one result that we didn't talk about that you want to highlight in particular,
 like what maybe one surprised you the most or so?
 Yeah, so the captioning one was interesting.
 I think we can look at that.
 So the captioning is, this is pretty much the dual of Dolly, right?
 So what we're doing is saying, okay, now that you have an image, generate the alt text for
 me given the image, right?
 So in some sense, we can exactly describe the captioning task in HTML, which is again
 kind of solidifies the argument that you want some level of document structure for prompting.
 So the results are quite good actually, at least from a semantic level.
 So one problem is that we don't actually generate in the style of, I think, MSCoco here.
 So we didn't report like blue four numbers or like the standard numbers.
 But if you look at the semantic similarity using BERT score, the CM3 captioning with
 clip as a re-ranker is actually a very, very strong baseline.
 And so you can kind of see the style here is weird.
 It tries to explicitly state what type of airplane it is.
 Yeah.
 But that's kind of an interesting behavior.
 So I think definitely at scale, you could get a single model that I think could be competitive
 with MSCoco with caption only models.
 If you do things like increase the resolution of the tokenized images, I think scale is
 really important here.
 So if you just scale up so that you have a similar amount of samples that are trained
 using MSCoco.
 You've said this a couple of times now, this sort of, you know, with scale, we could beat
 this or that.
 And I guess you see this work a little bit as a maybe a signpost, you know, to like later
 work that actually achieves this scale.
 Do you think the scale you're talking about, the scale at which, you know, this is competitive
 with on MSCoco, where the image generation is competitive with Dali, do you think that
 scale is currently achievable or is it so large that it's kind of, well, you know, we need
 entirely new hardware?
 Yeah, I think it is achievable.
 So let me tell you about the result that we just got a couple of days back.
 That's not in the paper here.
 So one reason that we also changed, chased this kind of multimodal setup is because we're
 interested or at least I'm very personally interested in the grounding aspect of language.
 So we kind of defined grounding as can you improve document level perplexity on text
 by extra conditioning on images?
 So that's one kind of way to measure grounding.
 The other way to measure grounding is we call it symmetrical grounding.
 So what you do is given a pretty much given a piece of text, generate an image from that
 piece of text and then condition on that image, generate back that piece of text, right?
 And I look at the perplexity differences between the two texts and that will give you the informational
 content of that image that is generated, right?
 So you can measure grounding that way.
 The unfortunate thing is that even the 13 billion parameter model that we have here
 did doesn't ground.
 But if you look at the scaling laws from, you know, or I think our 100 million parameter
 model to our 13 billion parameter model, around the 60 billion mark is where we'll see grounding
 in this setup.
 Okay.
 So our expectation is that if you scale this up to 60 billion, that you should be able
 to achieve, I think, language image grounding, which is kind of a cool result that I think
 a lot of people have been chasing here.
 And that's insane that you can make these predictions, right?
 This is like this is something I think in machine learning is something new.
 Because right now, no one could tell the most people could tell was like GPT three is going
 to be like somewhat better than GPT two.
 But now you're you're able and you know, I am confident that this is a you know, maybe
 it might be whatever 50 or 80 billion parameters, but you can actually make these predictions,
 which is which is, you know, it's it's cool.
 Like I'm amazed by this.
 Yeah, I definitely don't think we're going to be like order of magnitude off, right?
 Oh, so I think with the 100 billion parameter, 100 billion or 175 billion, like GPT three
 size, we can get very, very nontrivial behavior to the point of being competitive across all
 tasks.
 And I think the future in general is having a single multimodal model that can prompt
 in an instructable way, kind of like instruct GPT, but with all modalities.
 So I think that's kind of the north star that everyone is chasing right now.
 But I think we have a good I think we have a solid base for this work.
 But yeah, I think the captioning surprised me.
 And one thing that I want to call out here is that it only worked at a 13 billion scale.
 I might have mentioned this earlier.
 So there are fundamental stepwise changes in behavior from scaling up the model.
 It's not something smooth, right?
 So something that a 13 billion model can do is something that, you know, like a 2.7 billion
 model will not be able to do at all.
 So you won't, it's just going to generate random stuff.
 So it's interesting to see what the next, you know, stepwise changes in behavior will
 be, if you scale this up.
 With respect to the HTML, right, that you use, which is, I thought it was it was pretty
 cool because it is data that is, you know, so available.
 And your argument is a little bit that if you clean the HTML too much, right, these
 other these other data sets, they just pull out the text content, maybe the image, they
 try to align it and so on.
 You know, if you clean that up, there's so much structure missing, right, you're missing
 on all of this valuable information.
 Yet, you also do cleaning, right, you do quite a lot of HTML cleaning, you say somewhere
 up here in the data section.
 We strip this, we strip that any any sort of non non whatever elements we strip out,
 all headers, all footers, copyrights, forms, dialog boxes, we merge consecutive div elements
 and so on.
 Couldn't the same argument be made against you saying, well, you're losing so much of
 the structure, there's so much information there, like, why are you doing this?
 Do you think there is a valid direction to go in actually taking in even more context
 of these HTML documents?
 Yeah, so there are different constraints here, right.
 So one thing that I mentioned is that we can only model x amount of tokens, right, 300
 billion tokens, for example, right.
 So if the majority of those tokens, right, like, I think the average document is like,
 95% of the document we removed.
 So yeah, in some still right, you know, even though you're the ones that remove way less
 than the other ones.
 Yeah.
 So, so in some sense, do, do we want to model every single token?
 So in the case that you have infinite compute shirt, right.
 But here, there's kind of a min max problem that you have to solve, right, which is you
 want to kind of, you want to maximize the amount of semantic information that is available
 while minimizing the amount of tokens that you have, right.
 And this is kind of complex to do.
 So I think we found a good enough balance of the two.
 Like, in most cases, like, you don't want to repeat the same copyright like 400 million
 times, right.
 I mean, there's, there's probably a lot of information in the fact that jQuery is imported
 in this website, right.
 Right.
 So things like that.
 But we also do things that might break document structure, like the merging of elements, right.
 There's probably something there as to why the person has multiple developments, right.
 Regardless, we remove it.
 The other thing that we remove is attributes.
 So we remove all the attributes except those that are structured.
 So like open graph schema, I think Twitter has a like a structured graph as well.
 And the reason there was that the attributes were just, first of all, they were way too
 long most of the time, and they were not informationally rich enough.
 So you kind of have to balance compute here with how much structural information you want
 to maintain.
 Yeah, I see.
 And so there's no fundamental reason to use HTML, right.
 It's just something that's there, right.
 There's, I mean, for example, you can use markdown as well, right.
 And you can kind of recover a lot of the same things, right.
 Like generating the title you can do in markdown, right.
 High links you can do in markdown, right.
 So maybe the future direction is explicitly codifying this min max problem, right.
 And coming up with the document structure that the document structure is described in
 the minimal set of tokens.
 So maybe that's a pure engineering project as well.
 When you think of HTML and the DOM, it is a tree, right.
 Which is different from a linear sequence.
 Do you think there is, do you think there's value in treating the tree as a tree?
 Do you think it's mainly a limitation of the models we have?
 They go, let's say, like, see token by token or left to right or something like this.
 Do you think, you know, maybe it's still good to treat it as a sequence because there's
 text in there and text is left to right?
 Like what keeps us from building tree based models, which would be much more appropriate
 for something like this?
 Yeah.
 So one thing about transformers is it seems that they can learn the inductive bias of
 the data fairly well and it's not necessarily encoded.
 So my argument to this is that usually for these large scale runs, the best thing is
 just to keep it as simple as possible.
 Mostly just because they're risky, right.
 You get one chance.
 But the other reason is that transformers are actually highly capable of picking up
 this type of structure.
 So this isn't in the paper, but we looked at the attention scores and then you can see
 very clearly that the model knows what are like boundaries between HTML elements, for
 example.
 But again, there's also a ton of work to be done as well.
 So like some exciting work is, I think you also interviewed like Ofer for the alibi work,
 right?
 That work is really clever, right?
 Because it introduces an explicit inductive bias that the further away a token is, the
 probably less likely that you are to look at it.
 And it gets rid of the need for positional representations.
 So you can imagine like an extension of alibi here that would directly encode a tree like
 structure, right?
 So there's a ton of work to be done here.
 And then other thing is we didn't do too much for the images, right?
 In terms of attending, the positional representations for images are different than of text.
 So future work should consider specifically embedding images in such a way that you maintain
 locality of positions, right?
 So this is all stuff that needs to be done in the future as well.
 But that being said, I think if you have enough compute, these models can learn anything.
 It mostly becomes an efficiency angle.
 So about this paper, so what I have a bit of a trouble with is too many things in one
 paper, which in this case is this idea of using HTML and so on, although there was a
 previous paper of that, but then there's also the new loss and so on.
 Have you tested the new loss on pure text generation?
 Something like this, can you parse out what the different things contribute to the success
 of these models?
 Yeah.
 And that's a great criticism of the paper, actually.
 So fundamentally, I think if we wanted to do those like the proper science way, this
 would be like four or five papers, just teasing things apart.
 But at the same time, when you're training these large language models, ablation studies
 are pretty much impossible, right?
 No one has much compute to do these ablation studies.
 But the answer is yes.
 So we're looking at causal mass scaling loss for text only.
 This is a project that we're working on.
 We've trained a code model using the causal mass objective that's outperforming, I think
 both Google and Codex of similar sizes while being able to have a bidirectional option.
 So there are a couple of teams within Facebook that are trying out this objective with some
 success.
 So there will be future work about this.
 Excellent.
 And apart from what you just mentioned and scale, what's sort of next in this direction?
 Are you like, what are you excited about?
 Maybe it's not even you working on it, but what kind of is your exciting stuff that's
 happening?
 So one thing is figuring out a way to have higher fidelity.
 So the question to ask here is how do you represent continuous data in a discrete domain?
 And I don't think we're there yet, right?
 So that's some fundamental work that needs to move forward.
 The other thing that I'm kind of interested in looking is can we start joining more modalities,
 right?
 So Hubert that also came from Facebook had speech tokens, right?
 Very simple.
 I think they use k-means.
 I might be wrong though, just to find discrete tokens for speech.
 So imagine that you have a single model that has video images, text, speech, everything
 kind of put into one, right?
 Like what level of grounding and what level of zero-shot prompting can you get here?
 And I think a lot of people are kind of chasing this at the bigger companies.
 I'm kind of excited about that.
 On the analysis front, I think there's still a lot of unknowns about transformers.
 Like fundamentally we're still using the four-year-old implementation, right?
 The only difference is just pre-layer norm, right, from the original transformer.
 So I think better fundamentally understanding transformers.
 And I have some qualms with scaling laws.
 Like I don't think perplexity is necessarily the measure that we should be using.
 So internally we've been discussing like what does like memory-based scaling laws look like.
 So if you use memory as the fundamental unit of transformers, what do those scaling laws
 look like?
 So there's some more fundamental work to be done there.
 And the other thing is bridging, fine-tuning, and prompting performance.
 So far it's kind of orthogonal, which is, you know, if you want to get a better fine-tuning
 model, you have to do something that will hurt prompting and vice versa.
 So figuring out like is it just because we don't have like bi-directional like masks?
 Is that why?
 Is it because we only mask for like causal models and upper triangular matrix?
 Is there something more fundamental there?
 I think kind of peeling that apart and figuring out what's going on there is kind of important
 too.
 But I think we're very early on.
 I think this year is going to be the year of multimodal.
 I know they kind of kick stuff off.
 So I'm kind of excited to see what other groups are working on.
 It seems like it.
 Yeah.
 Is there anything else about the paper or the research direction you want to shout out?
 You want people to know that we haven't mentioned so far?
 Yeah.
 I mean, we'll be releasing all this code really, really soon.
 We're just waiting on some internal approvals so people will get to play around with it.
 I think we'll release three billion model, but the 13 billion model is the one that really
 shines.
 Yeah.
 So if people get that running, I think it's really cool.
 I spent hours just playing around with it.
 What does it take to just to forward propagate?
 What's the minimal configuration?
 So with the recent deep speed stuff that was released for inference, I'm not really sure
 because I think they said that you can use one GPU for like a 6.7 billion model.
 So if you do model parallelism, I think you need two GPUs.
 But without that, just give us a ballpark, what would it be like forward propping through
 this model?
 Yeah.
 So one thing is you could do it on a CPU if you have a strong enough CPU.
 But for inference, I think what I used was four V100s.
 Yeah.
 Model parallel.
 So less than a known.
 Cool.
 Excellent.
 Well, Armen, thank you so much for being here.
 This was really cool.
 Really valued the like also the kind of behind the scenes and insights we got here.
 And I hope to see you again very soon with even like CM4.
 Yeah, thank you for having me.
 Excellent.