BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding&Generation
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper zeta alpha blip language vision pre training language vision pre-training deep learning pre-training clip pre-training blip pretraining parameter sharing sequence to sequence image captioning vqa visual question answering fine-tuning vit vision transformer salesforce
#blip #review #ai

Cross-modal pre-training has been all the rage lately in deep learning, especially training vision and language models together. However, there are a number of issues, such as low quality datasets that limit the performance of any model trained on it, and also the fact that pure contrastive pre-training cannot be easily fine-tuned for most downstream tasks. BLIP unifies different tasks and objectives in a single pre-training run and achieves a much more versatile model, which the paper immediately uses to create, filter, clean and thus bootstrap its own dataset to improve performance even more!

Sponsor: Zeta Alpha
https://zeta-alpha.com
Use code YANNIC for 20% off!

OUTLINE:
0:00 - Intro
0:50 - Sponsor: Zeta Alpha
3:40 - Paper Overview
6:40 - Vision-Language Pre-Training
11:15 - Contributions of the paper
14:30 - Model architecture: many parts for many tasks
19:50 - How data flows in the model
26:50 - Parameter sharing between the modules
29:45 - Captioning & Filtering bootstrapping
41:10 - Fine-tuning the model for downstream tasks

Paper: https://arxiv.org/abs/2201.12086
Code: https://github.com/salesforce/BLIP
Demo: https://huggingface.co/spaces/Salesforce/BLIP

Abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at this https URL.

Authors: Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hey, y'all, this is a comprehensive paper review of the paper on blip. This is a model and a technique for bootstrapping one's own data set in vision and language pre training, which is pretty cool. So the video is a comprehensive review, we'll dive into the paper, we'll see what the paper is about, I'll explain you what's in it. And by the end of the video, you should have a good understanding of what's in the paper. In the next video, which I'm going to release tomorrow, there's going to be an interview with the authors of the paper. So also be sure to check that out because that answers a few very, very interesting questions that I had while reading the paper itself. So I wish you a lot of fun. Let me know what you think in the comments and I'll see you around. Bye bye. Hey there, this video is sponsored by Zeta Alpha, which is a new neural discovery and recommendation engine for papers. Yes, for scientific papers for trends in research and code in AI. Their goal is to become your research assistant and streamline how you organize, share and stay up to date on the latest R&D. This is really cool because the flood of papers in machine learning is sheer overwhelming in recent months. Zeta Alpha uses neural embedding based search and can give you the best recommendation of research that matches your interest and that you don't want to miss. And what better way than to just try it out. So first I start off searching for today's paper, which is the blip paper. And this is really cool because not only do I get the paper, I also get the GitHub code implementation and I can directly see the impact on social media that this paper has. This is much better than something like Google Scholar, which would just give me a few links to the paper itself. I can now save this paper under a tagging category that I'm just going to invent right now. And I can use Zeta Alpha to find similar research. Here I'm going to limit my search to the last three months. So I make sure that I don't miss anything that has recently been going on that I should know about when reviewing this paper. Now I also like a bunch of those other papers. So I'm going to save them as well to the same category. Once I have a bunch of papers in my category, I can use again Zeta Alpha's recommendation engine to give me more suggested papers to add to the same category based on what I have already in there. And I can also share this entire category with my teammates because everything Zeta Alpha does is not only for individuals, but also for teams. This is really powerful and can dramatically accelerate your discovery of new and relevant research. Now this doesn't only work for categories that you define. Once you interact with the search engine, Zeta Alpha is going to be able to give you a list a feed of recommendations from archive, from conferences, from blogs, from GitHub, and much more. This saves you a ton of time and lets you stay up to date with whatever is happening. If you're at all into ML research, this is hyper relevant for you. And I definitely invite you to check it out. Now they do have a free tier, but I got you a great deal. If you go over there right now and use code Yannick, you'll get 20% off a personal assistant subscription. Again, go to Zeta-Alpha.com, use code Yannick for 20% off right now. Thanks again so much to Zeta Alpha for sponsoring today's video. And now let's get into it. See ya. Hello there. Today we'll look at Blip Bootstrapping Language Image Pre-Training for Unified Vision Language Understanding and Generation by Junan Li, Dongxu Li, Taiming Xiong, Stephen Hoy. Yeah, that's it. Of Salesforce Research. So this paper proposes two things. One is a new architecture. And I want to say a new conglomeration of existing things. So an arrangement of modules for multitask pre-training. This model will take in an image text pair and perform multiple tasks on it. It has multiple losses and therefore ends up being able to do multiple things. Now that being said, this is a pre-training method. So the idea is that for any of these modules, you'll take them, you recompose them downstream and you fine tune them on a task, although they do have some zero shot results. So this is one thing. And this could be really cool if this alone turns out to be successful because it leads the path to a future where we have much more dynamic compositions of models and where we would pre-train these models with a lot of different tasks in one thing rather than pre-training them on just a single task like language modeling. The other thing is a bootstrapping method for the data. And these two things are not necessarily disconnected, although I do lament the fact that it's two things in one paper a little bit. But there's a bootstrapping method for these image text data set that includes training captioners and filters, which means that there is a part that learns to synthetically generate data and then there is a part that learns to distinguish good from bad data. And that allows them to collect lots and lots of data from the internet and filter out bad, badly poorly labeled images, which there exists a lot on the internet, and also allows them to augment the data set by labeling images themselves. So this is also really interesting and it feeds really well back into their model because their model is uniquely capable of doing this, being the multitask model that it is. So we're going to go through the architecture through the data set bootstrapping method. And keep in mind that I think if this catches on, there could be a recipes in here for future research that lead us to a much more dynamic world where we compose these modules, much like we compose different modules, low level modules in deep learning. We could compose these higher level modules and losses and do lots more multitask pre-training, maybe even dynamically configured. But let's dive in. So vision language pre-training, they say, has recently been the hit. For example, if you think of something like clip, and that's not even pre-training, but there are lots of architectures that do vision language pre-training, meaning they take pairs of images and text. So you'll have like some sort of an image and you'll have like some sort of text that goes with it. And you'll try to come up with a system that connects the two in any way. They say the major, the existing methods have two major limitations. So first of all, the, what they call the model perspective, they say they are either the existing methods are either encoder based or an encoder decoder architecture. So in an encoder based setup, what you would do is you would take in both of these things and you would try to come up with probably a number that represents how well they fit together. So are they good together or not? This is the clip architecture essentially. So in encoder based models, they criticize that encoder based are less straightforward to directly transfer to text generation tasks. So it's not, it's not simple to take clip and actually make it produce something. Remember if we have to, if you have to produce an actual image with clip, we need to do this diffusion clip guided diffusion or clip guided GANs, VQ GANs. So it's really cumbersome to make clip generate an image and it's probably even more cumbersome to make it generate text because it's not trained on that. So they criticize on these methods. It's not easy to make them do generation tasks. Whereas encoder decoder models have not been successfully adopted for image text retrieval tasks. So an encoder decoder model is where you would take the image probably and then make it produce the text. And then you train it as a language model to autoregressively produce the caption. And that's really neat for producing captions, but you cannot necessarily do this task up here very easily with such a model. You will, you will be able to do some things, but they're not necessarily successful because the task is really a different task. So both, both approaches for doing this currently are not ideal. The other thing is the data perspective. They criticize that these models are pre-trained on image text pairs that are essentially scraped from the internet. So collected from the internet. And they say noisy web text is suboptimal for vision language learning. We've known for a long time that there is a trade off between scale of data and quality of data. And ideally you'd have both. However, if you scrape from the internet, so let's say you scrape websites and there is like some text and there is an image somewhere and the image will have alt text. And that's what's usually used as the label in these systems. So if you don't know in the HTML, if you have an image tag, that's how, that's how the browser knows it's an image. You have the image tag, you have the source attribute, which leads, it's a URL usually that leads to the image, but then you also have an alt attribute. And it's really recommended that you put an alt, an alt property to the point where frameworks and linters and so on, they will yell at you if you don't have it. So what does this do? This specifically is for visually impaired people, for screen readers, but also for bots to know what is in the image. So you put the description there. However, a lot of people don't do that. And I think it makes it actually worse that linters and so on almost require you to do it. Because if you don't want to do it, you're just going to put like some some dumb stuff there like image, or people do lots of search engine optimizations in there. So since you know, the search engines don't usually look at the image itself, but at the alt text, they try to come up with buzzwordy things, so that it's ranked high in search results. So not necessarily the best quality data. And their bootstrapping, their bootstrapping method right here is is helping in that of getting higher quality data out of the internet. So how do they do this? The first thing they propose is this model, the multimodal mixture of encoder decoder. They say it can operate either as a unimodal encoder, or an image grounded text, the encoder or an image grounded text decoder. So yeah, we're going to look at these things. But I think here they say can operate either as one or this or that. It's not like this. It's not like that exact same model can do this. It's just that they put all of these models into one big model. And then they just use the part of the model that does the particular thing. So it's not necessarily super duper unified is what I wanted to say. Yeah, they train the three, the three sub parts of their models with three objectives, which we're also going to look at. The second part is this captioning and filtering. This is what this is what boosts the data set quality. They say they learn from noisy image text pairs by cleaning them by producing more and cleaning them. They train a captioner, which whose goal is to produce synthetic captions given web images and a filter to remove noisy captions from both the original web text and synthetic text. So the captioner will get images produce labels for these images or produce alt text. And then the filter goes over both the generated ones and the collected ones and just filters out everything that it deems to be qualitatively low standard. Of course, this needs to be trained on a high quality data set. But these sort of bootstrapping methods we've seen a number of times in the recent past that they actually work. In fact, this model, this paper here seems to be a good accumulation of sort of recognitions and good practices over the last few years. And we're going to point those out as we go through their their contributions. Here they say we show that the caption and the filter work together to achieve substantial performance improvement, which, okay, I don't know what substantial means in these kinds of tasks, but it's I it's an improvement. They are they achieve state of the art performance in a wide range of vision language tasks. And interestingly, also, this is a property of maybe synthetic data generation, they show more diverse captions yield larger gains. This might also be a good lesson for people who want to go and apply these methods. Lastly, they say next to having state of the art in downstream fine tune tasks, they also achieve zero short performance when directly transferring our models to two video language tasks. So they were they were never trained on video language tasks, never pre trained, never fine tuned, yet still they have a good zero short performance, which is okay. Like if you understand images, then there are going to be some video tasks that are your that you're particularly good at. Right. So let's dive into the model. And I've already shown you a diagram of the model. They quickly go through this here. They have three parts. They have actually, well, I want to say four parts to their model. One part one is a visual transformer, a VIT as the image encoder. So again, they take an image and they take a piece of text and now they do stuff with it. And the first part is they encode the image using a visual transformer. That's all they do with the image they encoded using a bit with the text, they do three, three different things. The first thing is they also just encode the text unimodally. So put the text through an encoder. And that with those two things already, they've essentially reproduced clip. Except they say it's the same as BERT. Yeah. So they've reproduced clip with those two things, because now they can set it up this visual transformer and the unimodal encoder, they can set it up as a similarity metric. So the unimodal encoder will give you some vector in an embedding space, the visual transformer will give you some vector in an embedding space, you can set up a contrastive loss to check whether these two things go together and whether they are apart from let's say any other encoded image or text. You can do this via contrastive learning, you can do it via regularized methods. But essentially, this is what we've come to known as encoder only models. The second thing they have is this image grounded text encoder. So the image grounded text encoder does almost the same thing as the unimodal text encoder. However, it doesn't encode the text separately. It jointly encodes the text while incorporating attention into the visual transformer. We're going to see how that goes in a second. But essentially, it produces a vector, let's say this one. And while producing that on the path, as it produces that, it incorporates information from the visual transformer. So it will, this here is the output of the visual transformer, it will incorporate that at multiple layers here via cross attention into the process. So this here is really a joint kind of encoding of the text given the image. That's why it's called image grounded text encoder. What this can do is you can build a classifier on top of this, like a binary classifier, because it is a representation of the text that has but that has already the information of the image inside of it. So it's kind of a joint representation of the image and the text. So you can build a classifier, for example, whether or not the two things go together again, but you don't have to use a contrastive loss, you can in fact use a supervised loss and classify and build a classifier. The third thing is this image grounded text decoder. Now again, being image grounded, that is a long, what is going on? Something's up here. There's an image grounded text decoder. The image grounded text decoder is much like the image grounded text encoder in that it incorporates cell across attention. However, it's a text decoder. So what it will do is it will actually produce text. So it will auto aggressively produce the text while incorporating again, information via cross attention from the visual representation. You can see that they have a different section on the pre training objectives. These just map to these three parts. So there's the image text contrastive loss, which is the loss for the first part. There is the image, the image text matching loss, which is the loss for the second part. And again, this is just a binary classification task where the model uses a linear layer head, they call it an ITM, an image text, text matching head, but it's a linear layer to predict whether an image text pair is positive, which means matched or negative unmatched given their multi modal feature. The special thing here is they do have a hard negative mining strategy. So they go to the top part here, they go to the joint, no, sorry, to the disjoint encoding to this part, and they look which ones are the hard negatives, which means that negatives that have a high contrastive similarity, and they use those specifically to train this loss here. The last loss is a language modeling loss, which is obviously relevant for the third part. This is a cross entropy loss, it maximizes the likelihood of the text in an autoregressive manner. If we put all of this together, we get this model right here. Again, if we go through it, the input data are two things, the input data are the image down here, and the piece of text here. Again, we know these go together because we've scraped them from the web. So these two, we know they go together. This is not an unsupervised training. This is essentially supervised learning for two things that we know go together. The first thing is we're going to encode the image through the image encoder. That's the image encoder. This is the image representation. This is just a bit. This is a visual transformer. I don't think they freeze it, but they may start from a checkpoint. All of this is jointly trained. So all of these losses, as I understand them, are jointly trained. So then we have the vision representation. What we can do is we can put the text first of all through the text encoder. You can see we can append different tokens right here to let the encoder know what we're currently doing because we also have some parameter sharing going on. So the text encoder gets the input text. It will also compute an encoding. And then we have this contrastive loss between the two encodings. They need to be close for pairs that we know go together, and they need to be far apart for other pairs. You can do something like in-batch negatives, or you can, as we said, mine hard negatives from this part. Well that makes no sense. You can mine hard negatives for that part over here, given this part over here. Which makes me believe, okay, maybe I haven't read closely enough. Maybe they also just train one of the losses maybe for each batch because they have to sample differently for the things. It doesn't make too much of a difference whether they train it really all jointly, jointly, or always activate one of the three text pathways. This would be interesting to figure out. So the last thing, the second thing they do is they give it to this image grounded text encoder. Again, this gets the text and a little token to show what's going on. It will encode, and now you can see that it has this cross attention module. And the cross attention module, as it encodes, it incorporates information that comes from all the way over here, comes all the way over here from the image. So the image representation is part of the encoding here, which means this thing has information about both the text and the image. Now yeah, of course, it's still a, it's still, it's not symmetric, right? We don't, the joint encoding is asymmetric in the sense that it is the text that is encoded based on the image. And that allows them to, you to only compute the image representation once. So they only need to do this pathway on the left here once, and then they can reuse that representation for all of the, for all of the different paths in the text here. Yeah, you can see that on the left, this is the difference on the left here. This is skipped, the cross attention is skipped. We don't have cross attention, it's just an encoding of the text itself. And here it's really a joint encoding, which means that this thing here contains information on both the image and the text. And we can perform any sort of task that we want with this joint encoding. In our case, we simply train it on a very similar objective as the contrastive loss in that it's a binary classification. It needs to figure out whether or not the two things actually go together or not. The third thing, again, almost the same is this decoder, the text decoder, same input except there's a little decode token. There is a difference in that this is bidirectional. The other two modules have bidirectional self-attention because they are encoders, so they get to use bidirectionality. Here we use causal self-attention, which essentially means that in the text you only get to attend things. So if you produce a particular token right here, you only get to attend to tokens that are behind yourself. This is a bit of a hack, because otherwise we couldn't train these things with batches or in parallel. It is definitely possible to use bidirectional self-attention as long as you cap, as long as you mask whatever comes next. So you want to mask sort of the future, but within the past you could totally use bidirectional self-attention. Again, this is just a hack to make training easier, but it's come to be a popular hack, so everyone's doing it. Again, you can see there's cross-attention coming from the image, and here you can really see that it's necessary. If I want to actually produce text, I need some sort of information of what I want to produce. So this language modeling loss here really needs the cross-attention, really needs the input from the image. So again, this comes from here, from the image representation. So there you have it. It's an unholy concoction of many different things in one. And this is all trained jointly. And yeah, I'm excited about this because I think not necessarily this particular arrangement. I have lots of stuff to criticize or lots of choices here that are kind of arbitrary. Why this asymmetry in, you know, I have the image encoded once and I have cross-attention into all the text encoders. Why not the other way around? Why don't we do image generation tasks? Why don't we do any sort of masked modeling, like masked language modeling? This could even be in the image. There's lots of stuff, let's say, to criticize. But I think what this thing shows is that a good recipe for the future could be to combine lots of these different methods together, combine lots of them into one big thing. Reusing parts intelligently and then train them jointly. We could even think of frameworks that do this automatically or that allow you to really easily set this up with a few lines of code and it will figure out by itself, like the framework would figure out itself, what it can compose and how it could reuse. What you can also see right here is I've overshadowed it a little bit with my thing right here, but there's color and the color indicates shared parameters, which is also really interesting. So you can see that essentially the text encoders aren't three separate encoders, but they largely share parameters. For example, the feedforward parameters are shared. The cross-attention parameters, they're all shared, except of course they're not active in this encoder. The bidirectional self-attention parameters are shared. The causal self-attention, those ones are separate over here, but if we had some sort of other autoregressive module, they would be shared too. So you'd share whatever you could in these architectures and that reduces the overhead, but also in their evaluations really helps, which I guess makes sense. Well, I don't know. If the tasks are too distant, you might get this catastrophic forgetting, but in their case it does help. Yes, which I could guess, right? For example, the bidirectional self-attention right here, since these two modules are almost doing the same task, it's reasonable that they would share parameters. So we've gone through a whole lot of things that they say down here. They do reason through their choices a little bit, even though I think these choices, they are either arbitrary or they're guided by experiments, just seeing what works better. They do bring up some hypotheses of what they think, why do things work and why do things don't work. They say that text encoder and decoder share all parameters except for the self-attention layer. The reason is that the differences between the encoding and decoding tasks are best captured by the self-attention layers. So they're essentially saying that whether you want to encode or decode, that is mostly going to be different in the attention layers, not from the architectural perspective, but from sort of the how the task is done perspective. And that I don't think necessarily you can say this, right? Like you can't necessarily say the feed forward layers have a similar job in or have similar features and perform similar functions, whether you're encoding or decoding. I don't just don't think that's out of the box, really evident that we need to be supported by evidence. So yeah. But it seems to work well in empirical evaluations and so I'm going to I'm going to with them sharing the parameters, but the reasoning are more hypotheses. So the second part they go into is this cap field. Again, this is a bit disconnected, although it plays well into their model. Here they criticize how these data sets are usually collected. They say alt text often do not accurately describe the visual content of the images that are scraped from the web. And that's why they have a bootstrapping method. So what they do is they collect a data set from the internet. And yeah, well, I find this diagram here to be a little bit complicated. So we're just going to make our own. So they have the internet, I'm going to this is a globe with, you know, the lines and so on. So we're going to collect a big chunk of data of pairs of images and text, images and alt text from the web, really noisy. And what we're going to do with this stuff is we're going to train a first blip architecture or a first now how they call it MED architecture, multi something something, whatever their model is on top. We're just going to train that with this noisy data, and that's going to be our first iteration model. Now this is really noisy so far and so on. But what we're going to do then is we're going to fine tune this. We're going to fine tune a filter and a captioner. So we're going to fine tune a filter and a captioner on supervised data. There exist some supervised data sets. And one of them, I believe, is the Coco data set. Yes, the Coco data set. So this step here, we need supervised data and supervised data of image text pairs. So human made captions for existing images, which it's a sort of a proxy for quality. So of these things, we can be sure that the quality is relatively high. If we could find some sort of an automated way to get really high quality image text pair data, it doesn't necessarily need to be human labeled. It just needs to be high in quality. So they use that to train a filter and a captioner. Now what is the filter and the captioning model? Now these are going to be fine tuned versions of their MED models. For example, the captioner takes in an image and gives you a caption, a synthetic caption. Now this is something our model can do. If we just take two parts, so we take this part and we take this part right here. This is now a captioning model. So the idea here, the general idea of BLIP of this MED model is that we pre train all of these things together and we sub select or we rearrange even the different sub components and then fine tune them on a downstream task. And one easy way is to take two components, simply deactivate all others and let them run in inference mode. So now we have a captioning model. The captioning, the filtering model on the other hand, very similar, but it takes an image and a piece of text both inside and it will output a score of whether the two things go together or not. Now this, of course we can achieve in multiple ways, but we can achieve this in the probably the most high quality way by taking the image encoder and taking this part right here that is specifically trained to jointly encode. You might ask, why don't we use this module right here and then use this contrastive estimation? We could also do that, definitely. But usually there are always multiple ways of determining similarity. You can have sort of the two stack encoder. So here is the image and here is the text. You can have separate encoders for them and then at the end determine whether they go together. And that's usually good if you want to do something like a search index because you can pre-compute a lot of these things. You can pre-compute all the embeddings for the images and then at inference time, if you have a query using text, you want to search an image via text, you only need to encode the text. Whereas with a joint encoding, it's really different. You need to input both into the encoder and that will give you a score at the end. And if you want to build a search engine like this, then for every single time you issue a query, what you need to do is you need to go through the whole data set and encode the query here together with all of the images, get the score for each one and then evaluate that. And you can see there is a trade-off, the left side is way friendlier computation-wise if you have an existing data set. The right side is qualitatively higher because during computation through these layers, the two things can already attend to one another, whereas really the only interaction here is the end over here. So this is qualitatively better estimate of whether the two things match or don't match. And that's why we're going to have the filter here. Since we're working, since we're filtering the data set, we can jointly encode the two things anyway. So we're going to fine tune that part to become our filter. So now we have a fine tuned part, one captioner, one filter. What can we do now? Well, we can take our data set, this thing right here, and we can use the captioner to produce another data set by just taking the images. So we just take the images here, we put them through the captioner and we get another data set. So we get another data set, it's going to have the same images, right? And it's going to have different texts. So I'm going to put this. So this is a synthetic data set. We can then join the two data sets together. So join the two data sets, and then we can put them both through the filter. So we're going to put them both through the filter. And the filter will simply filter out any image text pair that is not adequate, which means that it will filter out any image text pair which doesn't match well together, given the fine tuning of the filter on the supervised or high quality data set. So then we end up with a data set of, and we can restrict it like to only have one caption for each image or something like this. And we end up with a data set of image text pairs, which is large because we've augmented it with synthetic data, but also is of high quality because we have done the filtering. Now all of this being said, again, this highly relies on the quality of the data set that we fine tune on and of the diversity of that data set as well. Because you can also imagine if that data set isn't containing much of the domain that you're looking at, then your filter will learn to essentially down rank everything because it says, well, my data set says these two things don't go well together because I actually have just no data in that region. So there's a bit of danger in doing this. You really need to pay attention at what data set you're fine tuning. But this is how you bootstrap a good data set. So you can see go from here to here. And you can think of multiple things. Again, I think this paper is less about the particular method they choose. And I think more about what could be recipes for the future. And I think in the recent times, we've seen a lot of synthetic data generation, first of all, being really helpful. We've seen this in a number of reinforcement learning applications, a number of even NLP applications. So synthetic data is really, really picking up, I want to say, with advances in SIM to real and so on. And then also this approach of filtering. This has come up more and more in recent years, where generative models are paired with discriminative models that either rerank their outputs or filter their outputs for quality. This seems to be a very good recipe for achieving generative tasks in general. Not only train a generator, but train a ranker or filter on top of that. It's pretty computationally efficient. It's easy to implement. And yeah, I think it's a good recipe for the future. And one can think of various ways here to improve this, like to do this bootstrapping multiple times, to collect the supervised data set in a different manner and so on. I think there's a lot of possibilities here that are not yet explored, which I find to be pretty, pretty cool. So that's essentially all. Yeah. Okay, no, I was actually wrong here. You can see the filter is actually fine tuned on both of the objectives to learn whether a text matches the image. So this it's both the contrastive and the the single classifier loss. So I do think I do think the filter like what they actually pay attention to at the end is going to be this thing right here is going to be the classification head. But I guess it doesn't hurt to use both losses as you fine tune it. And since all parameters are shared, essentially, you really don't have you really don't have you can like it's it's easy to try and it's not too much of an overhead. So that's the methods. Again, they have this concoction of modules that they all pre train jointly with their respective losses. And then on the other hand, they have this bootstrapping method where they can directly use their model, right? That's the way these integrate these two. Since they have a model that can do all of these different things, they can fine tune that model to become a filter or to become a captioner. And the same thing holds for the results downstream. Here they have some examples, by the way, of generated. And so the bottom text is always a generated one. The top text is one from the data set. Anything that's red is filtered out by the filter. Anything that's green is accepted by the filter. Yeah, so they they also discuss a little bit of the dangers of doing this, of training the filtering and the captioning on this from the same pre training state on the same data set, which is that like there is some going to be some confirmation bias in that the filter will up rank things that the captioner produces because they're essentially learn from the same data. That's why they don't share. They fine tune them separately to combat this a little bit. But I still think that you're going to have some of that in there definitely. But you know, it's this is, you know, this is a real data from bridge near my house, which might be true, right? But it's not very descriptive and the filter realizes it. Yet a flock of birds flying over a lake at sunset. That's pretty descriptive. Another interesting thing is that they use nucleus sampling here, which is a common strategy. But they do find that using nucleus sampling leads to better performance and that because it generates more diverse and surprising captions, which contain more new information that the model could benefit from this, they compare this to beam search and beam search essentially goes for the highest likelihood sample. It tends to generate safe captions that are common in the data set, hence offering less extra knowledge. I think that's also really cool recognition right here that if we sample things from generative models, we might have different goals. And therefore it might not always be good to like it might be good to have an objective or a sampling method that encourages diversity. We've already seen this in alpha code. And my question there was already a little bit. Do we even have the correct training procedures for this? Because we train maximum likelihood? Or do we have the correct sampling procedures for this? All of these are interesting questions. And I think this kind of research validates that it's not all the same, like, depending on what we want to do, our training and sampling procedures need to adjust. I don't want to dive too deep into the results. They are outperforming other things by some margin. Like I don't necessarily agree that they outperform things so heavily as they advertise. But you know, that's research currently. Again, they allude to the fact that they share parameters here. And why that is, they say, sharing all the layers except for the self attention leads to better performance compared to not sharing. That's the part I believe, right? Totally. You share numbers go up good. But then they say, if the shared attention layers are shared, the model's performance would degrade to the conflict between the encoding and the decoding tasks. And this, I think, yeah, this stuff needs needs evidence. Because I mean, yeah, I'm fine with just going with the numbers. Here you can see the various ways they combine the things, for example, for visual question answering, they first encode the image, then they feed that to the text encoder, then they feed that to the decoder. So you can see, you can not only sub select modules, but you can rearrange them, right? Because you fine tune, you can adjust the parameters. So this connection already exists in the previous model, this connection doesn't. So you can sort of rearrange and recombine these modules to do various things. You can see here, we have two image or a double image encoder, or I guess the image encoder get just gets two samples. And then we also have two, one, a duplication of these cross attention modules. And then we output that into a newly trained merge layer. So this is the exciting part right here. And I feel I feel really don't want to necessarily go into this because we might go into this in the interview. But I feel a future where we have frameworks, coding frameworks, where this kind of stuff could be supported in an automatic fashion where I don't have to, you know, go and really hand define exactly how I want these things combined. But I could have a more high level descriptive language that allows me to do this whole pre training arrangements and this recombination for downstream fine tuning. That's really exciting. All right, I'm going to leave it at that. I hope you had a good overview. If you want to dive into the results, you know, feel free, there's lots of tables in here. And then we have a pro evaluation, which is really cool because it lends a lot of credence to their methods. And with that, let me know what you think in the comments and bye bye.
 Hey, y'all, this is a comprehensive paper review of the paper on blip.
 This is a model and a technique for bootstrapping one's own data set in vision and language
 pre training, which is pretty cool.
 So the video is a comprehensive review, we'll dive into the paper, we'll see what the paper
 is about, I'll explain you what's in it.
 And by the end of the video, you should have a good understanding of what's in the paper.
 In the next video, which I'm going to release tomorrow, there's going to be an interview
 with the authors of the paper.
 So also be sure to check that out because that answers a few very, very interesting
 questions that I had while reading the paper itself.
 So I wish you a lot of fun.
 Let me know what you think in the comments and I'll see you around.
 Bye bye.
 Hey there, this video is sponsored by Zeta Alpha, which is a new neural discovery and
 recommendation engine for papers.
 Yes, for scientific papers for trends in research and code in AI.
 Their goal is to become your research assistant and streamline how you organize, share and
 stay up to date on the latest R&D.
 This is really cool because the flood of papers in machine learning is sheer overwhelming
 in recent months.
 Zeta Alpha uses neural embedding based search and can give you the best recommendation of
 research that matches your interest and that you don't want to miss.
 And what better way than to just try it out.
 So first I start off searching for today's paper, which is the blip paper.
 And this is really cool because not only do I get the paper, I also get the GitHub code
 implementation and I can directly see the impact on social media that this paper has.
 This is much better than something like Google Scholar, which would just give me a few links
 to the paper itself.
 I can now save this paper under a tagging category that I'm just going to invent right
 now.
 And I can use Zeta Alpha to find similar research.
 Here I'm going to limit my search to the last three months.
 So I make sure that I don't miss anything that has recently been going on that I should
 know about when reviewing this paper.
 Now I also like a bunch of those other papers.
 So I'm going to save them as well to the same category.
 Once I have a bunch of papers in my category, I can use again Zeta Alpha's recommendation
 engine to give me more suggested papers to add to the same category based on what I have
 already in there.
 And I can also share this entire category with my teammates because everything Zeta
 Alpha does is not only for individuals, but also for teams.
 This is really powerful and can dramatically accelerate your discovery of new and relevant
 research.
 Now this doesn't only work for categories that you define.
 Once you interact with the search engine, Zeta Alpha is going to be able to give you
 a list a feed of recommendations from archive, from conferences, from blogs, from GitHub,
 and much more.
 This saves you a ton of time and lets you stay up to date with whatever is happening.
 If you're at all into ML research, this is hyper relevant for you.
 And I definitely invite you to check it out.
 Now they do have a free tier, but I got you a great deal.
 If you go over there right now and use code Yannick, you'll get 20% off a personal assistant
 subscription.
 Again, go to Zeta-Alpha.com, use code Yannick for 20% off right now.
 Thanks again so much to Zeta Alpha for sponsoring today's video.
 And now let's get into it.
 See ya.
 Hello there.
 Today we'll look at Blip Bootstrapping Language Image Pre-Training for Unified Vision Language
 Understanding and Generation by Junan Li, Dongxu Li, Taiming Xiong, Stephen Hoy.
 Yeah, that's it.
 Of Salesforce Research.
 So this paper proposes two things.
 One is a new architecture.
 And I want to say a new conglomeration of existing things.
 So an arrangement of modules for multitask pre-training.
 This model will take in an image text pair and perform multiple tasks on it.
 It has multiple losses and therefore ends up being able to do multiple things.
 Now that being said, this is a pre-training method.
 So the idea is that for any of these modules, you'll take them, you recompose them downstream
 and you fine tune them on a task, although they do have some zero shot results.
 So this is one thing.
 And this could be really cool if this alone turns out to be successful because it leads
 the path to a future where we have much more dynamic compositions of models and where we
 would pre-train these models with a lot of different tasks in one thing rather than pre-training
 them on just a single task like language modeling.
 The other thing is a bootstrapping method for the data.
 And these two things are not necessarily disconnected, although I do lament the fact that it's two
 things in one paper a little bit.
 But there's a bootstrapping method for these image text data set that includes training
 captioners and filters, which means that there is a part that learns to synthetically generate
 data and then there is a part that learns to distinguish good from bad data.
 And that allows them to collect lots and lots of data from the internet and filter out bad,
 badly poorly labeled images, which there exists a lot on the internet, and also allows them
 to augment the data set by labeling images themselves.
 So this is also really interesting and it feeds really well back into their model because
 their model is uniquely capable of doing this, being the multitask model that it is.
 So we're going to go through the architecture through the data set bootstrapping method.
 And keep in mind that I think if this catches on, there could be a recipes in here for future
 research that lead us to a much more dynamic world where we compose these modules, much
 like we compose different modules, low level modules in deep learning.
 We could compose these higher level modules and losses and do lots more multitask pre-training,
 maybe even dynamically configured.
 But let's dive in.
 So vision language pre-training, they say, has recently been the hit.
 For example, if you think of something like clip, and that's not even pre-training, but
 there are lots of architectures that do vision language pre-training, meaning they take pairs
 of images and text.
 So you'll have like some sort of an image and you'll have like some sort of text that
 goes with it.
 And you'll try to come up with a system that connects the two in any way.
 They say the major, the existing methods have two major limitations.
 So first of all, the, what they call the model perspective, they say they are either the
 existing methods are either encoder based or an encoder decoder architecture.
 So in an encoder based setup, what you would do is you would take in both of these things
 and you would try to come up with probably a number that represents how well they fit
 together.
 So are they good together or not?
 This is the clip architecture essentially.
 So in encoder based models, they criticize that encoder based are less straightforward
 to directly transfer to text generation tasks.
 So it's not, it's not simple to take clip and actually make it produce something.
 Remember if we have to, if you have to produce an actual image with clip, we need to do this
 diffusion clip guided diffusion or clip guided GANs, VQ GANs.
 So it's really cumbersome to make clip generate an image and it's probably even more cumbersome
 to make it generate text because it's not trained on that.
 So they criticize on these methods.
 It's not easy to make them do generation tasks.
 Whereas encoder decoder models have not been successfully adopted for image text retrieval
 tasks.
 So an encoder decoder model is where you would take the image probably and then make it produce
 the text.
 And then you train it as a language model to autoregressively produce the caption.
 And that's really neat for producing captions, but you cannot necessarily do this task up
 here very easily with such a model.
 You will, you will be able to do some things, but they're not necessarily successful because
 the task is really a different task.
 So both, both approaches for doing this currently are not ideal.
 The other thing is the data perspective.
 They criticize that these models are pre-trained on image text pairs that are essentially scraped
 from the internet.
 So collected from the internet.
 And they say noisy web text is suboptimal for vision language learning.
 We've known for a long time that there is a trade off between scale of data and quality
 of data.
 And ideally you'd have both.
 However, if you scrape from the internet, so let's say you scrape websites and there
 is like some text and there is an image somewhere and the image will have alt text.
 And that's what's usually used as the label in these systems.
 So if you don't know in the HTML, if you have an image tag, that's how, that's how the browser
 knows it's an image.
 You have the image tag, you have the source attribute, which leads, it's a URL usually
 that leads to the image, but then you also have an alt attribute.
 And it's really recommended that you put an alt, an alt property to the point where frameworks
 and linters and so on, they will yell at you if you don't have it.
 So what does this do?
 This specifically is for visually impaired people, for screen readers, but also for bots
 to know what is in the image.
 So you put the description there.
 However, a lot of people don't do that.
 And I think it makes it actually worse that linters and so on almost require you to do
 it.
 Because if you don't want to do it, you're just going to put like some some dumb stuff
 there like image, or people do lots of search engine optimizations in there.
 So since you know, the search engines don't usually look at the image itself, but at the
 alt text, they try to come up with buzzwordy things, so that it's ranked high in search
 results.
 So not necessarily the best quality data.
 And their bootstrapping, their bootstrapping method right here is is helping in that of
 getting higher quality data out of the internet.
 So how do they do this?
 The first thing they propose is this model, the multimodal mixture of encoder decoder.
 They say it can operate either as a unimodal encoder, or an image grounded text, the encoder
 or an image grounded text decoder.
 So yeah, we're going to look at these things.
 But I think here they say can operate either as one or this or that.
 It's not like this.
 It's not like that exact same model can do this.
 It's just that they put all of these models into one big model.
 And then they just use the part of the model that does the particular thing.
 So it's not necessarily super duper unified is what I wanted to say.
 Yeah, they train the three, the three sub parts of their models with three objectives,
 which we're also going to look at.
 The second part is this captioning and filtering.
 This is what this is what boosts the data set quality.
 They say they learn from noisy image text pairs by cleaning them by producing more and
 cleaning them.
 They train a captioner, which whose goal is to produce synthetic captions given web images
 and a filter to remove noisy captions from both the original web text and synthetic text.
 So the captioner will get images produce labels for these images or produce alt text.
 And then the filter goes over both the generated ones and the collected ones and just filters
 out everything that it deems to be qualitatively low standard.
 Of course, this needs to be trained on a high quality data set.
 But these sort of bootstrapping methods we've seen a number of times in the recent past
 that they actually work.
 In fact, this model, this paper here seems to be a good accumulation of sort of recognitions
 and good practices over the last few years.
 And we're going to point those out as we go through their their contributions.
 Here they say we show that the caption and the filter work together to achieve substantial
 performance improvement, which, okay, I don't know what substantial means in these kinds
 of tasks, but it's I it's an improvement.
 They are they achieve state of the art performance in a wide range of vision language tasks.
 And interestingly, also, this is a property of maybe synthetic data generation, they show
 more diverse captions yield larger gains.
 This might also be a good lesson for people who want to go and apply these methods.
 Lastly, they say next to having state of the art in downstream fine tune tasks, they also
 achieve zero short performance when directly transferring our models to two video language
 tasks.
 So they were they were never trained on video language tasks, never pre trained, never fine
 tuned, yet still they have a good zero short performance, which is okay.
 Like if you understand images, then there are going to be some video tasks that are
 your that you're particularly good at.
 Right.
 So let's dive into the model.
 And I've already shown you a diagram of the model.
 They quickly go through this here.
 They have three parts.
 They have actually, well, I want to say four parts to their model.
 One part one is a visual transformer, a VIT as the image encoder.
 So again, they take an image and they take a piece of text and now they do stuff with
 it.
 And the first part is they encode the image using a visual transformer.
 That's all they do with the image they encoded using a bit with the text, they do three,
 three different things.
 The first thing is they also just encode the text unimodally.
 So put the text through an encoder.
 And that with those two things already, they've essentially reproduced clip.
 Except they say it's the same as BERT.
 Yeah.
 So they've reproduced clip with those two things, because now they can set it up this
 visual transformer and the unimodal encoder, they can set it up as a similarity metric.
 So the unimodal encoder will give you some vector in an embedding space, the visual transformer
 will give you some vector in an embedding space, you can set up a contrastive loss to
 check whether these two things go together and whether they are apart from let's say
 any other encoded image or text.
 You can do this via contrastive learning, you can do it via regularized methods.
 But essentially, this is what we've come to known as encoder only models.
 The second thing they have is this image grounded text encoder.
 So the image grounded text encoder does almost the same thing as the unimodal text encoder.
 However, it doesn't encode the text separately.
 It jointly encodes the text while incorporating attention into the visual transformer.
 We're going to see how that goes in a second.
 But essentially, it produces a vector, let's say this one.
 And while producing that on the path, as it produces that, it incorporates information
 from the visual transformer.
 So it will, this here is the output of the visual transformer, it will incorporate that
 at multiple layers here via cross attention into the process.
 So this here is really a joint kind of encoding of the text given the image.
 That's why it's called image grounded text encoder.
 What this can do is you can build a classifier on top of this, like a binary classifier,
 because it is a representation of the text that has but that has already the information
 of the image inside of it.
 So it's kind of a joint representation of the image and the text.
 So you can build a classifier, for example, whether or not the two things go together
 again, but you don't have to use a contrastive loss, you can in fact use a supervised loss
 and classify and build a classifier.
 The third thing is this image grounded text decoder.
 Now again, being image grounded, that is a long, what is going on?
 Something's up here.
 There's an image grounded text decoder.
 The image grounded text decoder is much like the image grounded text encoder in that it
 incorporates cell across attention.
 However, it's a text decoder.
 So what it will do is it will actually produce text.
 So it will auto aggressively produce the text while incorporating again, information via
 cross attention from the visual representation.
 You can see that they have a different section on the pre training objectives.
 These just map to these three parts.
 So there's the image text contrastive loss, which is the loss for the first part.
 There is the image, the image text matching loss, which is the loss for the second part.
 And again, this is just a binary classification task where the model uses a linear layer head,
 they call it an ITM, an image text, text matching head, but it's a linear layer to predict whether
 an image text pair is positive, which means matched or negative unmatched given their
 multi modal feature.
 The special thing here is they do have a hard negative mining strategy.
 So they go to the top part here, they go to the joint, no, sorry, to the disjoint encoding
 to this part, and they look which ones are the hard negatives, which means that negatives
 that have a high contrastive similarity, and they use those specifically to train this
 loss here.
 The last loss is a language modeling loss, which is obviously relevant for the third
 part.
 This is a cross entropy loss, it maximizes the likelihood of the text in an autoregressive
 manner.
 If we put all of this together, we get this model right here.
 Again, if we go through it, the input data are two things, the input data are the image
 down here, and the piece of text here.
 Again, we know these go together because we've scraped them from the web.
 So these two, we know they go together.
 This is not an unsupervised training.
 This is essentially supervised learning for two things that we know go together.
 The first thing is we're going to encode the image through the image encoder.
 That's the image encoder.
 This is the image representation.
 This is just a bit.
 This is a visual transformer.
 I don't think they freeze it, but they may start from a checkpoint.
 All of this is jointly trained.
 So all of these losses, as I understand them, are jointly trained.
 So then we have the vision representation.
 What we can do is we can put the text first of all through the text encoder.
 You can see we can append different tokens right here to let the encoder know what we're
 currently doing because we also have some parameter sharing going on.
 So the text encoder gets the input text.
 It will also compute an encoding.
 And then we have this contrastive loss between the two encodings.
 They need to be close for pairs that we know go together, and they need to be far apart
 for other pairs.
 You can do something like in-batch negatives, or you can, as we said, mine hard negatives
 from this part.
 Well that makes no sense.
 You can mine hard negatives for that part over here, given this part over here.
 Which makes me believe, okay, maybe I haven't read closely enough.
 Maybe they also just train one of the losses maybe for each batch because they have to
 sample differently for the things.
 It doesn't make too much of a difference whether they train it really all jointly, jointly,
 or always activate one of the three text pathways.
 This would be interesting to figure out.
 So the last thing, the second thing they do is they give it to this image grounded text
 encoder.
 Again, this gets the text and a little token to show what's going on.
 It will encode, and now you can see that it has this cross attention module.
 And the cross attention module, as it encodes, it incorporates information that comes from
 all the way over here, comes all the way over here from the image.
 So the image representation is part of the encoding here, which means this thing has
 information about both the text and the image.
 Now yeah, of course, it's still a, it's still, it's not symmetric, right?
 We don't, the joint encoding is asymmetric in the sense that it is the text that is encoded
 based on the image.
 And that allows them to, you to only compute the image representation once.
 So they only need to do this pathway on the left here once, and then they can reuse that
 representation for all of the, for all of the different paths in the text here.
 Yeah, you can see that on the left, this is the difference on the left here.
 This is skipped, the cross attention is skipped.
 We don't have cross attention, it's just an encoding of the text itself.
 And here it's really a joint encoding, which means that this thing here contains information
 on both the image and the text.
 And we can perform any sort of task that we want with this joint encoding.
 In our case, we simply train it on a very similar objective as the contrastive loss
 in that it's a binary classification.
 It needs to figure out whether or not the two things actually go together or not.
 The third thing, again, almost the same is this decoder, the text decoder, same input
 except there's a little decode token.
 There is a difference in that this is bidirectional.
 The other two modules have bidirectional self-attention because they are encoders, so they get to use
 bidirectionality.
 Here we use causal self-attention, which essentially means that in the text you only get to attend
 things.
 So if you produce a particular token right here, you only get to attend to tokens that
 are behind yourself.
 This is a bit of a hack, because otherwise we couldn't train these things with batches
 or in parallel.
 It is definitely possible to use bidirectional self-attention as long as you cap, as long
 as you mask whatever comes next.
 So you want to mask sort of the future, but within the past you could totally use bidirectional
 self-attention.
 Again, this is just a hack to make training easier, but it's come to be a popular hack,
 so everyone's doing it.
 Again, you can see there's cross-attention coming from the image, and here you can really
 see that it's necessary.
 If I want to actually produce text, I need some sort of information of what I want to
 produce.
 So this language modeling loss here really needs the cross-attention, really needs the
 input from the image.
 So again, this comes from here, from the image representation.
 So there you have it.
 It's an unholy concoction of many different things in one.
 And this is all trained jointly.
 And yeah, I'm excited about this because I think not necessarily this particular arrangement.
 I have lots of stuff to criticize or lots of choices here that are kind of arbitrary.
 Why this asymmetry in, you know, I have the image encoded once and I have cross-attention
 into all the text encoders.
 Why not the other way around?
 Why don't we do image generation tasks?
 Why don't we do any sort of masked modeling, like masked language modeling?
 This could even be in the image.
 There's lots of stuff, let's say, to criticize.
 But I think what this thing shows is that a good recipe for the future could be to combine
 lots of these different methods together, combine lots of them into one big thing.
 Reusing parts intelligently and then train them jointly.
 We could even think of frameworks that do this automatically or that allow you to really
 easily set this up with a few lines of code and it will figure out by itself, like the
 framework would figure out itself, what it can compose and how it could reuse.
 What you can also see right here is I've overshadowed it a little bit with my thing right here, but
 there's color and the color indicates shared parameters, which is also really interesting.
 So you can see that essentially the text encoders aren't three separate encoders, but they largely
 share parameters.
 For example, the feedforward parameters are shared.
 The cross-attention parameters, they're all shared, except of course they're not active
 in this encoder.
 The bidirectional self-attention parameters are shared.
 The causal self-attention, those ones are separate over here, but if we had some sort
 of other autoregressive module, they would be shared too.
 So you'd share whatever you could in these architectures and that reduces the overhead,
 but also in their evaluations really helps, which I guess makes sense.
 Well, I don't know.
 If the tasks are too distant, you might get this catastrophic forgetting, but in their
 case it does help.
 Yes, which I could guess, right?
 For example, the bidirectional self-attention right here, since these two modules are almost
 doing the same task, it's reasonable that they would share parameters.
 So we've gone through a whole lot of things that they say down here.
 They do reason through their choices a little bit, even though I think these choices, they
 are either arbitrary or they're guided by experiments, just seeing what works better.
 They do bring up some hypotheses of what they think, why do things work and why do things
 don't work.
 They say that text encoder and decoder share all parameters except for the self-attention
 layer.
 The reason is that the differences between the encoding and decoding tasks are best captured
 by the self-attention layers.
 So they're essentially saying that whether you want to encode or decode, that is mostly
 going to be different in the attention layers, not from the architectural perspective, but
 from sort of the how the task is done perspective.
 And that I don't think necessarily you can say this, right?
 Like you can't necessarily say the feed forward layers have a similar job in or have similar
 features and perform similar functions, whether you're encoding or decoding.
 I don't just don't think that's out of the box, really evident that we need to be supported
 by evidence.
 So yeah.
 But it seems to work well in empirical evaluations and so I'm going to I'm going to with them
 sharing the parameters, but the reasoning are more hypotheses.
 So the second part they go into is this cap field.
 Again, this is a bit disconnected, although it plays well into their model.
 Here they criticize how these data sets are usually collected.
 They say alt text often do not accurately describe the visual content of the images
 that are scraped from the web.
 And that's why they have a bootstrapping method.
 So what they do is they collect a data set from the internet.
 And yeah, well, I find this diagram here to be a little bit complicated.
 So we're just going to make our own.
 So they have the internet, I'm going to this is a globe with, you know, the lines and so
 on.
 So we're going to collect a big chunk of data of pairs of images and text, images and alt
 text from the web, really noisy.
 And what we're going to do with this stuff is we're going to train a first blip architecture
 or a first now how they call it MED architecture, multi something something, whatever their
 model is on top.
 We're just going to train that with this noisy data, and that's going to be our first iteration
 model.
 Now this is really noisy so far and so on.
 But what we're going to do then is we're going to fine tune this.
 We're going to fine tune a filter and a captioner.
 So we're going to fine tune a filter and a captioner on supervised data.
 There exist some supervised data sets.
 And one of them, I believe, is the Coco data set.
 Yes, the Coco data set.
 So this step here, we need supervised data and supervised data of image text pairs.
 So human made captions for existing images, which it's a sort of a proxy for quality.
 So of these things, we can be sure that the quality is relatively high.
 If we could find some sort of an automated way to get really high quality image text
 pair data, it doesn't necessarily need to be human labeled.
 It just needs to be high in quality.
 So they use that to train a filter and a captioner.
 Now what is the filter and the captioning model?
 Now these are going to be fine tuned versions of their MED models.
 For example, the captioner takes in an image and gives you a caption, a synthetic caption.
 Now this is something our model can do.
 If we just take two parts, so we take this part and we take this part right here.
 This is now a captioning model.
 So the idea here, the general idea of BLIP of this MED model is that we pre train all
 of these things together and we sub select or we rearrange even the different sub components
 and then fine tune them on a downstream task.
 And one easy way is to take two components, simply deactivate all others and let them
 run in inference mode.
 So now we have a captioning model.
 The captioning, the filtering model on the other hand, very similar, but it takes an
 image and a piece of text both inside and it will output a score of whether the two
 things go together or not.
 Now this, of course we can achieve in multiple ways, but we can achieve this in the probably
 the most high quality way by taking the image encoder and taking this part right here that
 is specifically trained to jointly encode.
 You might ask, why don't we use this module right here and then use this contrastive estimation?
 We could also do that, definitely.
 But usually there are always multiple ways of determining similarity.
 You can have sort of the two stack encoder.
 So here is the image and here is the text.
 You can have separate encoders for them and then at the end determine whether they go
 together.
 And that's usually good if you want to do something like a search index because you
 can pre-compute a lot of these things.
 You can pre-compute all the embeddings for the images and then at inference time, if
 you have a query using text, you want to search an image via text, you only need to encode
 the text.
 Whereas with a joint encoding, it's really different.
 You need to input both into the encoder and that will give you a score at the end.
 And if you want to build a search engine like this, then for every single time you issue
 a query, what you need to do is you need to go through the whole data set and encode the
 query here together with all of the images, get the score for each one and then evaluate
 that.
 And you can see there is a trade-off, the left side is way friendlier computation-wise
 if you have an existing data set.
 The right side is qualitatively higher because during computation through these layers, the
 two things can already attend to one another, whereas really the only interaction here is
 the end over here.
 So this is qualitatively better estimate of whether the two things match or don't match.
 And that's why we're going to have the filter here.
 Since we're working, since we're filtering the data set, we can jointly encode the two
 things anyway.
 So we're going to fine tune that part to become our filter.
 So now we have a fine tuned part, one captioner, one filter.
 What can we do now?
 Well, we can take our data set, this thing right here, and we can use the captioner to
 produce another data set by just taking the images.
 So we just take the images here, we put them through the captioner and we get another data
 set.
 So we get another data set, it's going to have the same images, right?
 And it's going to have different texts.
 So I'm going to put this.
 So this is a synthetic data set.
 We can then join the two data sets together.
 So join the two data sets, and then we can put them both through the filter.
 So we're going to put them both through the filter.
 And the filter will simply filter out any image text pair that is not adequate, which
 means that it will filter out any image text pair which doesn't match well together, given
 the fine tuning of the filter on the supervised or high quality data set.
 So then we end up with a data set of, and we can restrict it like to only have one caption
 for each image or something like this.
 And we end up with a data set of image text pairs, which is large because we've augmented
 it with synthetic data, but also is of high quality because we have done the filtering.
 Now all of this being said, again, this highly relies on the quality of the data set that
 we fine tune on and of the diversity of that data set as well.
 Because you can also imagine if that data set isn't containing much of the domain that
 you're looking at, then your filter will learn to essentially down rank everything because
 it says, well, my data set says these two things don't go well together because I actually
 have just no data in that region.
 So there's a bit of danger in doing this.
 You really need to pay attention at what data set you're fine tuning.
 But this is how you bootstrap a good data set.
 So you can see go from here to here.
 And you can think of multiple things.
 Again, I think this paper is less about the particular method they choose.
 And I think more about what could be recipes for the future.
 And I think in the recent times, we've seen a lot of synthetic data generation, first
 of all, being really helpful.
 We've seen this in a number of reinforcement learning applications, a number of even NLP
 applications.
 So synthetic data is really, really picking up, I want to say, with advances in SIM to
 real and so on.
 And then also this approach of filtering.
 This has come up more and more in recent years, where generative models are paired with discriminative
 models that either rerank their outputs or filter their outputs for quality.
 This seems to be a very good recipe for achieving generative tasks in general.
 Not only train a generator, but train a ranker or filter on top of that.
 It's pretty computationally efficient.
 It's easy to implement.
 And yeah, I think it's a good recipe for the future.
 And one can think of various ways here to improve this, like to do this bootstrapping
 multiple times, to collect the supervised data set in a different manner and so on.
 I think there's a lot of possibilities here that are not yet explored, which I find to
 be pretty, pretty cool.
 So that's essentially all.
 Yeah.
 Okay, no, I was actually wrong here.
 You can see the filter is actually fine tuned on both of the objectives to learn whether
 a text matches the image.
 So this it's both the contrastive and the the single classifier loss.
 So I do think I do think the filter like what they actually pay attention to at the end
 is going to be this thing right here is going to be the classification head.
 But I guess it doesn't hurt to use both losses as you fine tune it.
 And since all parameters are shared, essentially, you really don't have you really don't have
 you can like it's it's easy to try and it's not too much of an overhead.
 So that's the methods.
 Again, they have this concoction of modules that they all pre train jointly with their
 respective losses.
 And then on the other hand, they have this bootstrapping method where they can directly
 use their model, right?
 That's the way these integrate these two.
 Since they have a model that can do all of these different things, they can fine tune
 that model to become a filter or to become a captioner.
 And the same thing holds for the results downstream.
 Here they have some examples, by the way, of generated.
 And so the bottom text is always a generated one.
 The top text is one from the data set.
 Anything that's red is filtered out by the filter.
 Anything that's green is accepted by the filter.
 Yeah, so they they also discuss a little bit of the dangers of doing this, of training
 the filtering and the captioning on this from the same pre training state on the same data
 set, which is that like there is some going to be some confirmation bias in that the filter
 will up rank things that the captioner produces because they're essentially learn from the
 same data.
 That's why they don't share.
 They fine tune them separately to combat this a little bit.
 But I still think that you're going to have some of that in there definitely.
 But you know, it's this is, you know, this is a real data from bridge near my house,
 which might be true, right?
 But it's not very descriptive and the filter realizes it.
 Yet a flock of birds flying over a lake at sunset.
 That's pretty descriptive.
 Another interesting thing is that they use nucleus sampling here, which is a common strategy.
 But they do find that using nucleus sampling leads to better performance and that because
 it generates more diverse and surprising captions, which contain more new information that the
 model could benefit from this, they compare this to beam search and beam search essentially
 goes for the highest likelihood sample.
 It tends to generate safe captions that are common in the data set, hence offering less
 extra knowledge.
 I think that's also really cool recognition right here that if we sample things from generative
 models, we might have different goals.
 And therefore it might not always be good to like it might be good to have an objective
 or a sampling method that encourages diversity.
 We've already seen this in alpha code.
 And my question there was already a little bit.
 Do we even have the correct training procedures for this?
 Because we train maximum likelihood?
 Or do we have the correct sampling procedures for this?
 All of these are interesting questions.
 And I think this kind of research validates that it's not all the same, like, depending
 on what we want to do, our training and sampling procedures need to adjust.
 I don't want to dive too deep into the results.
 They are outperforming other things by some margin.
 Like I don't necessarily agree that they outperform things so heavily as they advertise.
 But you know, that's research currently.
 Again, they allude to the fact that they share parameters here.
 And why that is, they say, sharing all the layers except for the self attention leads
 to better performance compared to not sharing.
 That's the part I believe, right?
 Totally.
 You share numbers go up good.
 But then they say, if the shared attention layers are shared, the model's performance
 would degrade to the conflict between the encoding and the decoding tasks.
 And this, I think, yeah, this stuff needs needs evidence.
 Because I mean, yeah, I'm fine with just going with the numbers.
 Here you can see the various ways they combine the things, for example, for visual question
 answering, they first encode the image, then they feed that to the text encoder, then they
 feed that to the decoder.
 So you can see, you can not only sub select modules, but you can rearrange them, right?
 Because you fine tune, you can adjust the parameters.
 So this connection already exists in the previous model, this connection doesn't.
 So you can sort of rearrange and recombine these modules to do various things.
 You can see here, we have two image or a double image encoder, or I guess the image encoder
 get just gets two samples.
 And then we also have two, one, a duplication of these cross attention modules.
 And then we output that into a newly trained merge layer.
 So this is the exciting part right here.
 And I feel I feel really don't want to necessarily go into this because we might go into this
 in the interview.
 But I feel a future where we have frameworks, coding frameworks, where this kind of stuff
 could be supported in an automatic fashion where I don't have to, you know, go and really
 hand define exactly how I want these things combined.
 But I could have a more high level descriptive language that allows me to do this whole pre
 training arrangements and this recombination for downstream fine tuning.
 That's really exciting.
 All right, I'm going to leave it at that.
 I hope you had a good overview.
 If you want to dive into the results, you know, feel free, there's lots of tables in
 here.
 And then we have a pro evaluation, which is really cool because it lends a lot of credence
 to their methods.
 And with that, let me know what you think in the comments and bye bye.