One Model For All The Tasks - BLIP (Author Interview)
#blip #interview #salesforce

Paper Review Video: https://youtu.be/X2k7n4FuI7c
Sponsor: Assembly AI
https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=yannic2

This is an interview with Junnan Li and Dongxu Li, authors of BLIP and members of Salesforce research.
Cross-modal pre-training has been all the rage lately in deep learning, especially training vision and language models together. However, there are a number of issues, such as low quality datasets that limit the performance of any model trained on it, and also the fact that pure contrastive pre-training cannot be easily fine-tuned for most downstream tasks. BLIP unifies different tasks and objectives in a single pre-training run and achieves a much more versatile model, which the paper immediately uses to create, filter, clean and thus bootstrap its own dataset to improve performance even more!

OUTLINE:
0:00 - Intro
0:40 - Sponsor: Assembly AI
1:30 - Start of Interview
2:30 - What's the pitch?
4:40 - How did data bootstrapping come into the project?
7:10 - How big of a problem is data quality?
11:10 - Are the captioning & filtering models biased towards COCO data?
14:40 - Could the data bootstrapping be done multiple times?
16:20 - What was the evolution of the BLIP architecture?
21:15 - Are there additional benefits to adding language modelling?
23:50 - Can we imagine a modular future for pre-training?
29:45 - Diving into the experimental results
42:40 - What did and did not work out during the research?
45:00 - How is research life at Salesforce?
46:45 - Where do we go from here?

Paper: https://arxiv.org/abs/2201.12086
Code: https://github.com/salesforce/BLIP
Demo: https://huggingface.co/spaces/Salesforce/BLIP

Abstract:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at this https URL.

Authors: Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, this is an interview with the authors of the blip paper. If you haven't seen it, I've made a review video of the paper itself. Be sure to check that out. The authors have seen that and are directly able to respond to it. So we all start on an even footing. It's very cool to have the authors on and this interview particularly was really interesting to me. I hope it is to you. As always, thank you for everyone who leaves a like who leaves a comment. Thanks to all the patreons and the support I get on Twitter and on YouTube itself. It's really cool. And I wish you a lot of fun. Thank you. Hey there, a quick shout out to today's sponsor. Assembly AI is an AI company that offers accurate API's for speech to text. As a developer, you can use these API's to automatically transcribe and understand audio and video data in just a few lines of code. Assembly AI automatically converts asynchronous and even live audio streams into text. They have so many features that help you understand your audio data. For example, summarization, content moderation, topic detection, and much more. Please check them out using the link in the description to let them know I sent you. Now let's get on with the video. Hi everyone. Today I'm here with Junnan Li and Dongxu Li, who are two of the researchers of the blip paper. It's a very big honor to have you here. Welcome both of you. Thanks for having us. Really happy to share our work here. Yeah, this paper was really cool. I think when it came out, everyone saw it and it generated quite a bit of buzz because it is a new approach to incorporating images and language and it can do a lot of things at the same time. It is a big system and yeah, I was super happy when I saw it. And when I read the paper, I was also pretty happy after I read the paper, which sometimes isn't the case anymore after you read the paper. And if you would just to dive in maybe, if you would pitch your idea to someone, like someone comes to you in a poster session or so, maybe for people who haven't seen the paper review just extremely briefly, what does your paper say or what do you propose? So maybe I can take this question. I think the major point of our paper, the setting point is that we propose a unified framework for visual language pre-training where we can pre-train this model that has the capability of doing both visual language understanding and visual language generation. So what understanding means is that it can jointly understand the two modalities, namely image and text, and produce some kind of multimodal features that can be used such as for classification tasks. And what generation means here is that it can generate text based on some image input. For example, for image captioning, it's one of a typical generation task. So I think this is the main idea of our model. In terms of the technical, in terms of how do we achieve that, I think there is one big point that I would like to highlight is we do have this data set bootstrapping to tackle the challenge of noisy web training data. Because existing works, a lot of them pre-train on those data that are collected from the web, which contains the image and all text pairs, which can be noisy. I think you mentioned in the review video. So what we do here is we want to synthetically generate captions and also to use a filter to try to remove the noisy captions. And by doing so, we can significantly improve the quality of the data set. And I think one of the key message we want to send in the paper is that the quality of the data really matters, it's as important as if not more important than the quantity. So a lot of passwords have focused on scaling up the model with big data. But here we do scale up, but we also focus on the quality of the data. I want to dive into this data bootstrapping right away, because it is almost a bit of an independent thing from the system itself. We've long known that we can trade off quality for quantity, but usually it is in an exponential fashion. So to get the same amount more quality, we need exponentially more data if we want to achieve it with less quality data. Which came first, the idea of building the vision language model or the idea of filtering or the data set, because they both play nicely into one another in your paper. And I'm just a bit wondering, how did this come to be? Which came first? Why one or the other? Yeah. So actually, for my research, for my past papers, I focused some papers on this weekly supervised learning or learning from the noisy data. So I've always been quite interested in how do people train models with imperfect data, which is a very practical scenario. And I think this field may deserve more attention. It's not as popular as some of the other fields, but it's really a very practical issue. And it does exist for vision language pre-training. So actually, one of my previous papers in vision language pre-training, which we call it LBF model, it was published in NeurIPS last year, we have this kind of self-training scheme where we want to clean the noise in the data set. But it's in a relatively more simpler way than what we do here. So rather than generating synthetic captions, we were doing some self-dissolation thing. So then we take it to the next step in the brief paper, where we first look at the data set and we see a lot of noise. And here, noise basically means that the caption is not really describing the visual content of the image. It may still be a good human-written text. It's not the text is grammatically wrong, it's grammatically correct. It's just that it's not aligned with the image. So what we try to solve is how do we generate texts that are more aligned with the image such that our pre-training can benefit from this. I think this left picture here illustrates it well, where it just says, from a bridge near my house, right? Which is a weird thing to put in an alt text, you would put that usually in some sort of a social media post or so. But this is one of the examples where the alt text doesn't really describe the image. I thought that was really well. Were you always aware of this weakness? How do you even find out that that is a large-scale problem? Yeah, so I think I first come find out this problem when going through some of the Pergena data set. So I think what people previously used, a quite standard web data set, was this conceptual caption 3 million, which is a relatively medium scale. It's not too small, but not very huge. And there do exist a lot of captions like this in that data set. And I found this problem even exaggerated as I tried to use a bigger data set. For example, in this paper, we used a line data set, which was a very newly released data set. And the noisy problem was even more, like, happens a lot more frequent when you try to scale up the data to include more web images with alt text. So we feel like this is something that if we can solve it, it could really change the model's performance. Have you seen that there's a recent paper called something like, vision models are more robust and fair when trained on uncurated data or something like this? So this here, you seem to say we need better quality data. And that group is saying essentially, no, our models work better when we have less quality, but we just go out and collect data. Can you maybe establish a bit of a connection between the two views? Like how do they agree? Yeah, so I think maybe there's two different aspects. One is the quality, the other is the diversity. And I think what that paper tried to maybe claim is, I haven't read the detail, it's just my impression was that they tried to claim if you have this huge web data set that is more diverse maybe than your maybe human-curated data set, you can bring better advantage to the model. I think that doesn't contradict with what we say here. So actually in our experiment, we show that the diversity of captions do matter a lot. When we try to generate synthetic captions, we try to generate a diverse set of captions that covers a whole bunch of different concepts rather than a very common and safe description of the image. I think maybe these two approaches seem to me to not contradict but complementary to each other. On one aspect, when you have more data, of course, you can always scale up the size of your data as you are always having more samples. That gives you better capacity for the model. But on the other side, we have more focus on the quality side. If you really look at the number of images we are using here for the pre-training, compared with some of the other works, it's not a lot. It's not too much, too large a scale. But since the quality of our pre-training corpus is better, we are now with better performance. So I really think the skill and the quality, they are complementary and they do not contradict, I believe. Let's stay on the captioning and filtering for just one more second. You first pre-train the entire model on this uncurated dataset and then you use fine-tuning on a human-generated captioning dataset in order to get these filter and captioning models. My worry there would be a little bit exactly what we talked about right now. What my filter and captioning models learn is really dependent on, let's assume the quality of the human-generated dataset is good, but the diversity of it really matters. Because it needs to cover all the images that come from the uncurated dataset. Otherwise it is going to misjudge, misfilter or not being able to caption this dataset. How do you control for that? Maybe you can also comment on if I now, let's say I want to expand my dataset to areas that I know that the human one doesn't cover, what could be a method of still going and researching on this new type of data? Yeah, I think that's a very good question. I think it's a valid concern that this fine-tuning may be biased models to our certain domains. I think one of the reasons we achieve performance improvement is because a lot of these downstream tasks are similar to the Coco domain image. So I think that's a valid point. But in the meantime, I would say that this fine-tuning doesn't destroy the model's capability to generate diverse captions. Because the fine-tuning is really a very lightweight procedure. So for Peretrainion, we're peretrain on this huge dataset for 220 epochs, which would take a few days, maybe even a week. But this fine-tuning, we only fine-tune for five epochs on a very small scale of Coco dataset, which can finish within a few hours. So this fine-tuning would not make the model forget about what it has previously saw. It only slightly modified the model so that it can generate captions that are more like human-written ones. But we do find that even after fine-tuning, the model can generate captions that are not within the vocabulary of Coco dataset. So it's not like the fine-tuning completely destroyed the model's diversity capability. So that's your answer to our first question. And for the second question, if someone wants to try to expand the model to a different domain where there doesn't exist human annotations, I would say first, if you can collect some, it would be good. And if you cannot, maybe one solution is there might be some similar images from this huge web dataset that maybe you can retrieve. So let's say if you can retrieve some similar images associated with web captions, then maybe you can slightly fine-tune the model on those subsets so that the model becomes slightly more biased towards your domain and more suitable to your downstream task. You suggest with this arrow right here, almost you suggest like a loop, like suggesting that this could be done multiple times, right? I could go multiple times through this stage. Is this anything? Okay, I've maybe not seen this in the experiment. If this is anything you've tried, or would anything change in the loop number two or number three or number four, what would be the difference? There's no new data introduced. Yeah, so first of all, I would say it's definitely possible to do multiple rounds of iterations of this bootstrapping. And in our future work, we mentioned this as one of the future work. And in terms of extra knowledge, each round of bootstrapping, we can add in new captions. So if the model becomes better, it can generate better synthetic captions. And there might be a diminishing return if we do multiple rounds. I would say my intuition is the first round will probably help the most, and maybe the second or third will help less. But unfortunately, due to the time and computation constraint, we didn't really have the resource to produce the experiment before the paper. So that's definitely one of the future plans that we have. So let's shift maybe. Sorry. Good. Okay, this model here is quite big. Was my first impression when I saw it. There's a lot of stuff. Okay, I have also drawn a lot of stuff on it. Sorry, I can make this go away. So the model here is relatively big and relatively, you know, there's modules going around, there's parameter sharing going on. What was the evolution of this model? Is this version one that we're looking at right here? Or is this like, you know, version 50 after you've tried a bunch of other things? Yeah, yeah. Definitely not version one. So actually, this model is heavily inspired by our previous LBF model, which is an encoder-only model. So if you look at the model, there's not too much difference between LBF and BLEEP, except the fact that now we add the generation capability to BLEEP with the language modeling loss. So the reason why we want to add this is first that because the encoder model doesn't really transfer that well to image captioning task and other generation tasks, so it's better that we can pre-train it to have this capability. That's why we add in this new decoder module. And then after we add in the decoder module, we thought, since we are doing multitask learning, can we share some parameters? Because first of all, it's more efficient to share parameters. And secondly, it may bring some advantage from the multitask training by jointly optimizing those few losses. So we tried different sharing strategies. First, we started with not sharing any parameters at all. And then we tried to share maybe the... So we tried to decouple maybe some...the cross-attention layer or the self-attention layer or the feed-forward layer. And we find that decoupling the self-attention layer from the encoder and decoder is a more efficient and effective way. So that's why we choose this strategy. But there is a possibility that because we are doing this experiment on a relatively smaller scale pre-training, so we were using the 40 million images for pre-training, but our final model was pre-trained on 100 million images. So maybe this sharing strategy is not optimal for if you scale up the dataset. So I would imagine if you want to have the best possible performance, you may want to scale up the dataset and try to decouple the parameters more. But that would, of course, sacrifice some of the efficiencies brought by the parameter sharing. Yeah. Another point I probably want to add here is like this architecture is not like ad hoc design because remember that one of our starting point is to eliminate the noise levels in this pre-training datasets. So from there, on one side we need to identify what are the noisy ones, whether the image and the caption match with each other. And that ends up with this design of encoder model. On the other side, we want even more that when we find that the caption does not align well with the image itself, we don't want to simply discard the training data point. We want to generate some useful captions, surprising captions that can further help us. So from that, I really want to say that it's not like we want to put everything together, glue different models into a single model to make it big. It really serves very well for this caption filter algorithm. And I think that kind of, yeah. Yeah. Just one additional comment is that our model is really actually not big if you compare to some other models. So basically our model is a VIT plus a bird. So it's a base version of the bird. So in terms of the number of parameters, I would say it's a standard parameter deep learning model. It's not that crazy huge. So even we draw it in the current figure, actually there is because of this parameter sharing going on, the number of parameters and the training computation load is not that heavy. Yeah. I like the fact that this really arises from sort of the goal of cleaning the data set. I also thought the more I read it and the more I talked about it, it became more evident that the things really played together nicely. So you use the contrastive loss to get the hard negatives for the, I want to say, matching loss or ranker loss. And then that gives you the filter. And then the language model here gives you the captioning. With respect to parameter sharing, you said, okay, the matching head or the contrastive heads, they're not really good at captioning themselves. So we'd rather pre-train or train a captioning or a language generation model. Do you find that adding the task of language generation also helps the tasks that the other models would be good at? Like, do you find an additional benefit, except for our model can also do captioning, do you find an additional benefit for the already existing or the already tackled tasks by adding, let's say, the language model? Yes, yes. We find that there is an advantage brought by this language model loss. So this language model loss, if you think about it, is really quite similar to the mass language model loss, except that now it's an autoregressive version. So in our previous IOBF work and in some other papers, what people usually do is mass language learning to try to improve the model's capability to understand the text in a more fine-grained granularity, because the image text matching and image text contrastive learning is more like a global matching. You are trying to match the image and text. But the language model is more fine-grained. You want to generate the word based on the image. And by achieving so, you need to better understand maybe some details of the image and align it with the textual concept to be able to generate the word. Do you have, let's say, more extensive goals in mind here? You just said it's actually not that big. If it's really nice, I agree with all of that. Yet, I foresee a future where you could bring together lots of these modules. Essentially, what I'd like to have is, first of all, we could obviously think of doing the same with the image side right here. You just have an encoder here right now. But we could think of breaking out here, doing image generation, doing whatever we can do with images. But on the other hand, maybe an even bigger future vision would be I bring a data set and I say, look, these are pairs of images and text. Now please, system, make me a model that includes all of these losses that I can think of, like all of these different combinations. And the system would figure out, oh, okay, I can share parameters here and I can build that and so on. And maybe that would, given your findings, which I totally believe that adding more of these tasks and sharing the parameters actually mutually benefits each other, the representations, they become more capable, they become maybe more broadly meaningful and so on. So I think that might be a cool future to work against. I don't know how feasible it is though. Is that anything on your roadmap or what does the future look like of these models? Yeah, I think that's a very cool idea. Maybe a very ambitious goal. So we have considered to add in some image generation capability, but we didn't because it doesn't fit very well with our current framework. So we don't want to make the framework to be very huge and messy. We try to keep it more cleaner. And regarding your point that can we have automatic system that can maybe combine different modules and losses? I think that's a possible goal. It's just there could be a lot of obstacles in how to achieve that. For example, if we borrow some idea from the NAS community and maybe we borrow some reinforcement learning idea, maybe there are some ways we can train a policy to do that. But it's not entirely clear to me how can we achieve that because I think the main problem is this per training is how to evaluate a per training is a big problem. So you cannot just say that lower per training loss means that your model is better downstream task. If there is a correlation between per training loss and downstream task, then it may be easier. You just find the optimal module that you can minimize your per training loss. But usually it's not the case. It also depends on how well aligned is your per training task and downstream task. I think that's one of the major issues of why it may take some trial and error to find the best strategy for the per training. Maybe I can add a few sentence to that. I think being able to figure out how to combine these different modules together automatically would be super cool and futuristic. I think there are a couple of practical messages that we want to convey here, which is the first I think if you really look at how this we fine tune this MED model to make them a captioner, a filter, and also how we combine these different modules together in order to tackle the downstream tasks. There are really some dedicated ways to do that. And usually if you look at some per training works on the market, their strategies will be pretty simplistic in the sense that in most of occasions they just add the task specific heads. But in this particular work, we just move one step further than that. We are rethinking how to rearrange these modules and what are the best strategies for this parameter sharing strategy. Another message we may want to say here is a lot of people, they blindly do this multitasking by aggregating hundreds of different data sets and tasking to one per training model. And maybe by bleep we want people to revisit this decision next time they do this multitasking because not necessarily every task they complement with each other. And you may want to carefully look into what to share, what not to share. I think these are the two things we want to remind for future works. And I have one additional comment to follow what Dongxu said is that you can see a lot of other works, they really combine really like maybe eight or ten objectives together. So there are some strategies for visual language training is you bring in object detection objective to improve your localization capability. So we think that's a way to that's a valid way to improve performance. But here what we try to say is that we want to keep things very nice and simple. So we have these three laws where each law serves a very clear purpose and can be transferred to a very specific Dongxuan task. And all we need is just image text pairs. We don't need any bounding box or anything else. So I think that's one of the message we want to also convey. Cool. And yeah, and I especially I like the fact that with pre-training with the aspect of fine tuning, then you're able to recombine these different modules in very creative ways. So even though you have these modules, they have their purposes for the pre-training, for the captioning, for the filtering. But then they can be it seems it seems many, many tasks can now be tackled by some sort of combination of these models and a little bit of fine tuning, which is something that I find really cool. You have done extensive and like there are there are lots of lots of tables means means you had to run like and collect lots of numbers, which is is very nice because it gives a bit also of a broad overview than just having, you know, four numbers or so comparing with one baseline. Although could you maybe highlight some of the of the standing out results that you got or one of some of the more important results? Like how would you summarize or what would you highlight about your experimental evaluation of this? Yeah, sure. I think the most important one would be table one, where we demonstrate the performance gain achieved by how do we bootstrap our data set. And yeah, so this is table basically, if you look at the first column, it shows how many images you are using. So we have two settings, one is a 40 million images, another we scale up with small noisy image taxpayers. And the second column is how do we perform the bootstrapping? C stands for captioning and F stands for filtering. It means whether we do captioning to generate synthetic captions, or we do filtering to remove the noisy captions, or we do both together. So if you look at the first row, second row, third and fourth row, you can see that both the captioning and the filtering can help individually. And if you combine them together, they really have complemented each other. So by generating synthetic captions, and at the same time, try to remove the noise, we can achieve, I would say a quite good amount of gain in these two different, four different data sets covering both the retrieval task and the captioning task. So I think that's one of the key results we have here. And also maybe then it goes to the second table is how do we do the bootstrapping of the captions? So do we use beam search? Or do we use nuclear sampling? So the difference between those two approaches is that beam search is a deterministic sampling, not sampling, deterministic decoding strategy, where you try to find the most likely sentence associated with the image. And nuclear sampling is a stochastic approach where you try to sample according to some probability distribution. We find that surprisingly, if you compare beam search with no generation, there is a good gain achieved by beam search. But by moving beam search to nuclear sampling, there is a similar amount of gain. So this is something that we didn't expect at the first time we see the results. And after we really deep dive into what the captions look like, how does beam search and nuclear sampling generate different captions, we found out that the beam search will generate a kind of a safe caption that accurately describes the image most of the time, but it's not surprising. So you can commonly see those captions in the data set. And that doesn't add a lot of extra knowledge for the model to learn. But the nuclear sampling really introduces some really diverse captions that are more like human written ones. Humans don't write a very boring distribution like a man is with a dog in a park. So it's a very boring caption. But nuclear sampling can give you more diverse captions. And if you look at a noise ratio, which is actually how much of those captions were filtered out by our filter, you can also see that beam search is less noisy. But even though it's less noisy, it's not as beneficial as nuclear sampling here. And this really raises another question, which I think is a very interesting future work, is that is nuclear sampling the best way? So because those models are pertrained with the language modeling laws, which is kind of deterministic laws, you try to maximize the likelihood of your captions. And we are just doing that, and we try to do something in the decoding side to try to give more diverse captions. But this nuclear sampling was used in mostly NLP papers. So does there exist some better diverse captioning strategy for image captioning tasks? So I think that's a very interesting question. I think in recent times, this has been shining through in a lot of works that the fact that maybe we don't need to go maximum likelihood in our inference step, but maybe it's a better approach to go diverse with the sampling. And then exactly what you do have some sort of a classifier or some sort of a filter to just scrap out the noise. I think that's a really, really good approach. And we saw this anywhere. I think Dolly famously had Clip re-ranking all the outputs. And I think more and more models go towards this. It's really cool finding that you're essentially finding exactly the same thing. When I look at these numbers, all of the numbers, it's very convincing to see that everything uniformly almost uniformly gets better. You support whatever you say really well. This trend right here, it really works across all of the data sets. You uniformly almost get better in all the tables. However, the difference is always, the maximum difference is whatever. This from here to here is like two points in what is this? What's TR? It's the true... It's a recall, text recall. Text recall, sorry. Oh yeah, it's down here. Text recall, image recall. That's like 2%. Right here, again, it's like one point something percent. So there's a uniformly getting better. My question is, given that the getting better is convincing, but the scale of it is like yeah, 2% or so, when is it worth to do this week long pre-training you mentioned? This is a big procedure. The pre-training is big. And then to fine tune the pre-training again, when is it worth it? From what scale or for what applications does it become actually worth to do something like this? Yeah, I think that's a very good question. And first of all, I would say it is worth doing if your data is really... If you observe a large amount of noise in the data and maybe your data is incomplete in some of the domains. For example, here, the web data is primarily dominated by those alt text, which can be different from what human would write to describe an image. So if there is a noisy scenario or a domain gap, I think it's worth to do so. And secondly, actually, we have also released our dataset after bootstrapping so that if you are just trying to do regionally pre-training in a similar domain, I think you can just download our version and use that as a starting point to avoid the first round of pre-training. And maybe certainly about your previous comment that we have really unanimous improvement for those tasks. Actually in one of the tasks, maybe you can scroll down the paper. Let me try to find... I think it's the NLVR task. Table eight, maybe? Yeah, yeah, table eight. Yeah, actually for this task, this is where we find the better quality of captions doesn't necessarily give you a better game if you compare here. And actually by scaling up the number of pre-training image, it doesn't correlate very straightforwardly to a downstream performance game. So I think it still depends on your alignment between your pre-training and your downstream objective. So for most of the tasks, it is well aligned. And that's why improving your pre-training data quality can improve your downstream task. Yeah, maybe I can add a few sentences in terms of whether it is worthwhile to improve that much. I think if you really imagine the big picture here in terms of the multimodal retrieval, let's say if you deploy this retrieval algorithm, and that manages to improve their profit by 1%, that's a huge achievement. You won a lot. So at Salesforce, we also have the retrieval. We also work with clients for their retrieval services. So in terms of that, if you just let your GPU run for one week and improve by 1%, that's a huge improvement, I would say. And I would also like to say that these numbers, they kind of, I think, under hype what BLEAP has achieved. Because I think BLEAP, beyond this relative advantage over its competitors, is also qualitatively better in terms of how easy it is to use BLEAP. If you really look at the demo we created there on the web, and it just freely asks any questions in natural language rather easily. In contrast, a lot of these image question answering models, they are not doing the free form generation. They are kind of doing classification in order to tackle this question answering task. This point is, however, not fully demonstrated, I believe, in the current manuscript. So if you really want to get impressed, we really suggest you check out our demo and put whatever photos you like and questions. Cool. It's really neat, by the way, that you have a demo to go along with it, because I think it makes it more accessible and it demonstrates also the capabilities of this. It's almost like we're moving into the world that GPT-3 maybe has created for text with these image language models, because we got the same feeling from GPT-3. Oh no, I can just go and I can put any text, right, and I can interact with the system in a sort of free form way. And it's really cool to see that we're also moving in this direction with the image models. In terms of just the process of how this research went about, you ended up with a cool system with a nice way of bootstrapping data and so on. Can you maybe tell us a little bit about stuff that didn't necessarily work out during the research? Was there any point where you were maybe disheartened a little bit, things that didn't work out? What were your low and your high points during the creation of this paper? Yeah, actually, one of the experiments we had was when we first tried to scale up the potential with small web images using this line data set that we have downloaded, which takes quite some time. It doesn't help that much. So then it feels really feel like why scaling up the data is not benefiting the model. So then I did some more analysis and after that I realized that a lot of those images are very, very small in the resolution. Some are just icons or some brand names. And if I remove those, then it begins to show the gains. But I think that's one of the kind of the blockers we faced. And I think after we first get the bootstrapping, especially the nuclear sampling to give a big performance gain, then at that point, we are quite confident that this should be a good solution. And I think that point is when I realized, okay, this method should work well and we can write a paper about it. Great. Dongxin, do you want to say something? Yeah, I believe some of these strategies, they also arise from the internal discussions with other group members as well. So it's really a lot of crowd intelligence behind the scenes. How is the research organized at Salesforce? I have a bit of insight into, let's say, the big tech giants like Google and Facebook and so on, and they have their research divisions. At a company like Salesforce, who is more customer, I want to say customer, all these companies are customer oriented, obviously. But how is research organized there? What do you do while the model is pre-training for a week? Do you have other stuff to do or are you mainly researchers or what's life like there? Yeah. So first of all, I would say that AI is a big part of Salesforce, what they try to achieve, to use AI to better help the customers. So we have this separate research division, maybe not as large as Google or Facebook, but I think everything works quite well in our research team. In terms of our day-to-day operation, I think it's mostly similar to other industrial researchers. We can be quite flexible to do research or do some more product oriented work. We are motivated to do research that can generate high impact, that can really change the field in a more substantial way. And while we wait for the GPU to finish training, we already just do other research stuff or read some papers involving some internal discussions or maybe try to solve some real production problems. Cool. Is there anything else you want to get out about this paper? You already said people can go to the web, to your repo, and you have a demo also available. Is there anything you'd want to get out? What's the easiest for people to get started with this research? Yes. I think first, again, welcome to try out our demo and welcome to visit our GitHub. We do have, I think, quite detailed instructions on how to download and train our fine-tuned model. And also, I welcome any suggestions or questions you might have about our model that we can use that to improve our model or the code. That would be great. Dongxu, anything, any last messages? Our team is expanding, so if you are interested, just let you know. Yeah, we are looking for an intern position in the visual language research. Cool. Who can apply? Anyone that is at university? Yeah, anyone can apply. We hire globally, so we can do remote working now. Cool. Excellent. Okay, Dongxu and Jinan, thank you very much for being here. This was a lot of fun. Thank you for having us. Thank you. Have a great day of preparation.
 Hello, this is an interview with the authors of the blip paper.
 If you haven't seen it, I've made a review video of the paper itself.
 Be sure to check that out.
 The authors have seen that and are directly able to respond to it.
 So we all start on an even footing.
 It's very cool to have the authors on and this interview particularly was really interesting
 to me.
 I hope it is to you.
 As always, thank you for everyone who leaves a like who leaves a comment.
 Thanks to all the patreons and the support I get on Twitter and on YouTube itself.
 It's really cool.
 And I wish you a lot of fun.
 Thank you.
 Hey there, a quick shout out to today's sponsor.
 Assembly AI is an AI company that offers accurate API's for speech to text.
 As a developer, you can use these API's to automatically transcribe and understand audio
 and video data in just a few lines of code.
 Assembly AI automatically converts asynchronous and even live audio streams into text.
 They have so many features that help you understand your audio data.
 For example, summarization, content moderation, topic detection, and much more.
 Please check them out using the link in the description to let them know I sent you.
 Now let's get on with the video.
 Hi everyone.
 Today I'm here with Junnan Li and Dongxu Li, who are two of the researchers of the blip
 paper.
 It's a very big honor to have you here.
 Welcome both of you.
 Thanks for having us.
 Really happy to share our work here.
 Yeah, this paper was really cool.
 I think when it came out, everyone saw it and it generated quite a bit of buzz because
 it is a new approach to incorporating images and language and it can do a lot of things
 at the same time.
 It is a big system and yeah, I was super happy when I saw it.
 And when I read the paper, I was also pretty happy after I read the paper, which sometimes
 isn't the case anymore after you read the paper.
 And if you would just to dive in maybe, if you would pitch your idea to someone, like
 someone comes to you in a poster session or so, maybe for people who haven't seen the
 paper review just extremely briefly, what does your paper say or what do you propose?
 So maybe I can take this question.
 I think the major point of our paper, the setting point is that we propose a unified
 framework for visual language pre-training where we can pre-train this model that has
 the capability of doing both visual language understanding and visual language generation.
 So what understanding means is that it can jointly understand the two modalities, namely
 image and text, and produce some kind of multimodal features that can be used such as for classification
 tasks.
 And what generation means here is that it can generate text based on some image input.
 For example, for image captioning, it's one of a typical generation task.
 So I think this is the main idea of our model.
 In terms of the technical, in terms of how do we achieve that, I think there is one big
 point that I would like to highlight is we do have this data set bootstrapping to tackle
 the challenge of noisy web training data.
 Because existing works, a lot of them pre-train on those data that are collected from the
 web, which contains the image and all text pairs, which can be noisy.
 I think you mentioned in the review video.
 So what we do here is we want to synthetically generate captions and also to use a filter
 to try to remove the noisy captions.
 And by doing so, we can significantly improve the quality of the data set.
 And I think one of the key message we want to send in the paper is that the quality of
 the data really matters, it's as important as if not more important than the quantity.
 So a lot of passwords have focused on scaling up the model with big data.
 But here we do scale up, but we also focus on the quality of the data.
 I want to dive into this data bootstrapping right away, because it is almost a bit of
 an independent thing from the system itself.
 We've long known that we can trade off quality for quantity, but usually it is in an exponential
 fashion.
 So to get the same amount more quality, we need exponentially more data if we want to
 achieve it with less quality data.
 Which came first, the idea of building the vision language model or the idea of filtering
 or the data set, because they both play nicely into one another in your paper.
 And I'm just a bit wondering, how did this come to be?
 Which came first?
 Why one or the other?
 Yeah.
 So actually, for my research, for my past papers, I focused some papers on this weekly
 supervised learning or learning from the noisy data.
 So I've always been quite interested in how do people train models with imperfect data,
 which is a very practical scenario.
 And I think this field may deserve more attention.
 It's not as popular as some of the other fields, but it's really a very practical issue.
 And it does exist for vision language pre-training.
 So actually, one of my previous papers in vision language pre-training, which we call
 it LBF model, it was published in NeurIPS last year, we have this kind of self-training
 scheme where we want to clean the noise in the data set.
 But it's in a relatively more simpler way than what we do here.
 So rather than generating synthetic captions, we were doing some self-dissolation thing.
 So then we take it to the next step in the brief paper, where we first look at the data
 set and we see a lot of noise.
 And here, noise basically means that the caption is not really describing the visual content
 of the image.
 It may still be a good human-written text.
 It's not the text is grammatically wrong, it's grammatically correct.
 It's just that it's not aligned with the image.
 So what we try to solve is how do we generate texts that are more aligned with the image
 such that our pre-training can benefit from this.
 I think this left picture here illustrates it well, where it just says, from a bridge
 near my house, right?
 Which is a weird thing to put in an alt text, you would put that usually in some sort of
 a social media post or so.
 But this is one of the examples where the alt text doesn't really describe the image.
 I thought that was really well.
 Were you always aware of this weakness?
 How do you even find out that that is a large-scale problem?
 Yeah, so I think I first come find out this problem when going through some of the Pergena
 data set.
 So I think what people previously used, a quite standard web data set, was this conceptual
 caption 3 million, which is a relatively medium scale.
 It's not too small, but not very huge.
 And there do exist a lot of captions like this in that data set.
 And I found this problem even exaggerated as I tried to use a bigger data set.
 For example, in this paper, we used a line data set, which was a very newly released
 data set.
 And the noisy problem was even more, like, happens a lot more frequent when you try to
 scale up the data to include more web images with alt text.
 So we feel like this is something that if we can solve it, it could really change the
 model's performance.
 Have you seen that there's a recent paper called something like, vision models are more
 robust and fair when trained on uncurated data or something like this?
 So this here, you seem to say we need better quality data.
 And that group is saying essentially, no, our models work better when we have less quality,
 but we just go out and collect data.
 Can you maybe establish a bit of a connection between the two views?
 Like how do they agree?
 Yeah, so I think maybe there's two different aspects.
 One is the quality, the other is the diversity.
 And I think what that paper tried to maybe claim is, I haven't read the detail, it's
 just my impression was that they tried to claim if you have this huge web data set that
 is more diverse maybe than your maybe human-curated data set, you can bring better advantage to
 the model.
 I think that doesn't contradict with what we say here.
 So actually in our experiment, we show that the diversity of captions do matter a lot.
 When we try to generate synthetic captions, we try to generate a diverse set of captions
 that covers a whole bunch of different concepts rather than a very common and safe description
 of the image.
 I think maybe these two approaches seem to me to not contradict but complementary to
 each other.
 On one aspect, when you have more data, of course, you can always scale up the size of
 your data as you are always having more samples.
 That gives you better capacity for the model.
 But on the other side, we have more focus on the quality side.
 If you really look at the number of images we are using here for the pre-training, compared
 with some of the other works, it's not a lot.
 It's not too much, too large a scale.
 But since the quality of our pre-training corpus is better, we are now with better performance.
 So I really think the skill and the quality, they are complementary and they do not contradict,
 I believe.
 Let's stay on the captioning and filtering for just one more second.
 You first pre-train the entire model on this uncurated dataset and then you use fine-tuning
 on a human-generated captioning dataset in order to get these filter and captioning models.
 My worry there would be a little bit exactly what we talked about right now.
 What my filter and captioning models learn is really dependent on, let's assume the quality
 of the human-generated dataset is good, but the diversity of it really matters.
 Because it needs to cover all the images that come from the uncurated dataset.
 Otherwise it is going to misjudge, misfilter or not being able to caption this dataset.
 How do you control for that?
 Maybe you can also comment on if I now, let's say I want to expand my dataset to areas that
 I know that the human one doesn't cover, what could be a method of still going and researching
 on this new type of data?
 Yeah, I think that's a very good question.
 I think it's a valid concern that this fine-tuning may be biased models to our certain domains.
 I think one of the reasons we achieve performance improvement is because a lot of these downstream
 tasks are similar to the Coco domain image.
 So I think that's a valid point.
 But in the meantime, I would say that this fine-tuning doesn't destroy the model's capability
 to generate diverse captions.
 Because the fine-tuning is really a very lightweight procedure.
 So for Peretrainion, we're peretrain on this huge dataset for 220 epochs, which would take
 a few days, maybe even a week.
 But this fine-tuning, we only fine-tune for five epochs on a very small scale of Coco
 dataset, which can finish within a few hours.
 So this fine-tuning would not make the model forget about what it has previously saw.
 It only slightly modified the model so that it can generate captions that are more like
 human-written ones.
 But we do find that even after fine-tuning, the model can generate captions that are not
 within the vocabulary of Coco dataset.
 So it's not like the fine-tuning completely destroyed the model's diversity capability.
 So that's your answer to our first question.
 And for the second question, if someone wants to try to expand the model to a different
 domain where there doesn't exist human annotations, I would say first, if you can collect some,
 it would be good.
 And if you cannot, maybe one solution is there might be some similar images from this huge
 web dataset that maybe you can retrieve.
 So let's say if you can retrieve some similar images associated with web captions, then
 maybe you can slightly fine-tune the model on those subsets so that the model becomes
 slightly more biased towards your domain and more suitable to your downstream task.
 You suggest with this arrow right here, almost you suggest like a loop, like suggesting that
 this could be done multiple times, right?
 I could go multiple times through this stage.
 Is this anything?
 Okay, I've maybe not seen this in the experiment.
 If this is anything you've tried, or would anything change in the loop number two or
 number three or number four, what would be the difference?
 There's no new data introduced.
 Yeah, so first of all, I would say it's definitely possible to do multiple rounds of iterations
 of this bootstrapping.
 And in our future work, we mentioned this as one of the future work.
 And in terms of extra knowledge, each round of bootstrapping, we can add in new captions.
 So if the model becomes better, it can generate better synthetic captions.
 And there might be a diminishing return if we do multiple rounds.
 I would say my intuition is the first round will probably help the most, and maybe the
 second or third will help less.
 But unfortunately, due to the time and computation constraint, we didn't really have the resource
 to produce the experiment before the paper.
 So that's definitely one of the future plans that we have.
 So let's shift maybe.
 Sorry.
 Good.
 Okay, this model here is quite big.
 Was my first impression when I saw it.
 There's a lot of stuff.
 Okay, I have also drawn a lot of stuff on it.
 Sorry, I can make this go away.
 So the model here is relatively big and relatively, you know, there's modules going around, there's
 parameter sharing going on.
 What was the evolution of this model?
 Is this version one that we're looking at right here?
 Or is this like, you know, version 50 after you've tried a bunch of other things?
 Yeah, yeah.
 Definitely not version one.
 So actually, this model is heavily inspired by our previous LBF model, which is an encoder-only
 model.
 So if you look at the model, there's not too much difference between LBF and BLEEP, except
 the fact that now we add the generation capability to BLEEP with the language modeling loss.
 So the reason why we want to add this is first that because the encoder model doesn't really
 transfer that well to image captioning task and other generation tasks, so it's better
 that we can pre-train it to have this capability.
 That's why we add in this new decoder module.
 And then after we add in the decoder module, we thought, since we are doing multitask learning,
 can we share some parameters?
 Because first of all, it's more efficient to share parameters.
 And secondly, it may bring some advantage from the multitask training by jointly optimizing
 those few losses.
 So we tried different sharing strategies.
 First, we started with not sharing any parameters at all.
 And then we tried to share maybe the...
 So we tried to decouple maybe some...the cross-attention layer or the self-attention layer or the feed-forward
 layer.
 And we find that decoupling the self-attention layer from the encoder and decoder is a more
 efficient and effective way.
 So that's why we choose this strategy.
 But there is a possibility that because we are doing this experiment on a relatively
 smaller scale pre-training, so we were using the 40 million images for pre-training, but
 our final model was pre-trained on 100 million images.
 So maybe this sharing strategy is not optimal for if you scale up the dataset.
 So I would imagine if you want to have the best possible performance, you may want to
 scale up the dataset and try to decouple the parameters more.
 But that would, of course, sacrifice some of the efficiencies brought by the parameter
 sharing.
 Yeah.
 Another point I probably want to add here is like this architecture is not like ad hoc
 design because remember that one of our starting point is to eliminate the noise levels in
 this pre-training datasets.
 So from there, on one side we need to identify what are the noisy ones, whether the image
 and the caption match with each other.
 And that ends up with this design of encoder model.
 On the other side, we want even more that when we find that the caption does not align
 well with the image itself, we don't want to simply discard the training data point.
 We want to generate some useful captions, surprising captions that can further help
 us.
 So from that, I really want to say that it's not like we want to put everything together,
 glue different models into a single model to make it big.
 It really serves very well for this caption filter algorithm.
 And I think that kind of, yeah.
 Yeah.
 Just one additional comment is that our model is really actually not big if you compare
 to some other models.
 So basically our model is a VIT plus a bird.
 So it's a base version of the bird.
 So in terms of the number of parameters, I would say it's a standard parameter deep learning
 model.
 It's not that crazy huge.
 So even we draw it in the current figure, actually there is because of this parameter
 sharing going on, the number of parameters and the training computation load is not that
 heavy.
 Yeah.
 I like the fact that this really arises from sort of the goal of cleaning the data set.
 I also thought the more I read it and the more I talked about it, it became more evident
 that the things really played together nicely.
 So you use the contrastive loss to get the hard negatives for the, I want to say, matching
 loss or ranker loss.
 And then that gives you the filter.
 And then the language model here gives you the captioning.
 With respect to parameter sharing, you said, okay, the matching head or the contrastive
 heads, they're not really good at captioning themselves.
 So we'd rather pre-train or train a captioning or a language generation model.
 Do you find that adding the task of language generation also helps the tasks that the other
 models would be good at?
 Like, do you find an additional benefit, except for our model can also do captioning, do you
 find an additional benefit for the already existing or the already tackled tasks by adding,
 let's say, the language model?
 Yes, yes.
 We find that there is an advantage brought by this language model loss.
 So this language model loss, if you think about it, is really quite similar to the mass
 language model loss, except that now it's an autoregressive version.
 So in our previous IOBF work and in some other papers, what people usually do is mass language
 learning to try to improve the model's capability to understand the text in a more fine-grained
 granularity, because the image text matching and image text contrastive learning is more
 like a global matching.
 You are trying to match the image and text.
 But the language model is more fine-grained.
 You want to generate the word based on the image.
 And by achieving so, you need to better understand maybe some details of the image and align
 it with the textual concept to be able to generate the word.
 Do you have, let's say, more extensive goals in mind here?
 You just said it's actually not that big.
 If it's really nice, I agree with all of that.
 Yet, I foresee a future where you could bring together lots of these modules.
 Essentially, what I'd like to have is, first of all, we could obviously think of doing
 the same with the image side right here.
 You just have an encoder here right now.
 But we could think of breaking out here, doing image generation, doing whatever we can do
 with images.
 But on the other hand, maybe an even bigger future vision would be I bring a data set
 and I say, look, these are pairs of images and text.
 Now please, system, make me a model that includes all of these losses that I can think of, like
 all of these different combinations.
 And the system would figure out, oh, okay, I can share parameters here and I can build
 that and so on.
 And maybe that would, given your findings, which I totally believe that adding more of
 these tasks and sharing the parameters actually mutually benefits each other, the representations,
 they become more capable, they become maybe more broadly meaningful and so on.
 So I think that might be a cool future to work against.
 I don't know how feasible it is though.
 Is that anything on your roadmap or what does the future look like of these models?
 Yeah, I think that's a very cool idea.
 Maybe a very ambitious goal.
 So we have considered to add in some image generation capability, but we didn't because
 it doesn't fit very well with our current framework.
 So we don't want to make the framework to be very huge and messy.
 We try to keep it more cleaner.
 And regarding your point that can we have automatic system that can maybe combine different
 modules and losses?
 I think that's a possible goal.
 It's just there could be a lot of obstacles in how to achieve that.
 For example, if we borrow some idea from the NAS community and maybe we borrow some reinforcement
 learning idea, maybe there are some ways we can train a policy to do that.
 But it's not entirely clear to me how can we achieve that because I think the main problem
 is this per training is how to evaluate a per training is a big problem.
 So you cannot just say that lower per training loss means that your model is better downstream
 task.
 If there is a correlation between per training loss and downstream task, then it may be easier.
 You just find the optimal module that you can minimize your per training loss.
 But usually it's not the case.
 It also depends on how well aligned is your per training task and downstream task.
 I think that's one of the major issues of why it may take some trial and error to find
 the best strategy for the per training.
 Maybe I can add a few sentence to that.
 I think being able to figure out how to combine these different modules together automatically
 would be super cool and futuristic.
 I think there are a couple of practical messages that we want to convey here, which is the
 first I think if you really look at how this we fine tune this MED model to make them a
 captioner, a filter, and also how we combine these different modules together in order
 to tackle the downstream tasks.
 There are really some dedicated ways to do that.
 And usually if you look at some per training works on the market, their strategies will
 be pretty simplistic in the sense that in most of occasions they just add the task specific
 heads.
 But in this particular work, we just move one step further than that.
 We are rethinking how to rearrange these modules and what are the best strategies for this
 parameter sharing strategy.
 Another message we may want to say here is a lot of people, they blindly do this multitasking
 by aggregating hundreds of different data sets and tasking to one per training model.
 And maybe by bleep we want people to revisit this decision next time they do this multitasking
 because not necessarily every task they complement with each other.
 And you may want to carefully look into what to share, what not to share.
 I think these are the two things we want to remind for future works.
 And I have one additional comment to follow what Dongxu said is that you can see a lot
 of other works, they really combine really like maybe eight or ten objectives together.
 So there are some strategies for visual language training is you bring in object detection
 objective to improve your localization capability.
 So we think that's a way to that's a valid way to improve performance.
 But here what we try to say is that we want to keep things very nice and simple.
 So we have these three laws where each law serves a very clear purpose and can be transferred
 to a very specific Dongxuan task.
 And all we need is just image text pairs.
 We don't need any bounding box or anything else.
 So I think that's one of the message we want to also convey.
 Cool.
 And yeah, and I especially I like the fact that with pre-training with the aspect of
 fine tuning, then you're able to recombine these different modules in very creative ways.
 So even though you have these modules, they have their purposes for the pre-training,
 for the captioning, for the filtering.
 But then they can be it seems it seems many, many tasks can now be tackled by some sort
 of combination of these models and a little bit of fine tuning, which is something that
 I find really cool.
 You have done extensive and like there are there are lots of lots of tables means means
 you had to run like and collect lots of numbers, which is is very nice because it gives a bit
 also of a broad overview than just having, you know, four numbers or so comparing with
 one baseline.
 Although could you maybe highlight some of the of the standing out results that you got
 or one of some of the more important results?
 Like how would you summarize or what would you highlight about your experimental evaluation
 of this?
 Yeah, sure.
 I think the most important one would be table one, where we demonstrate the performance
 gain achieved by how do we bootstrap our data set.
 And yeah, so this is table basically, if you look at the first column, it shows how many
 images you are using.
 So we have two settings, one is a 40 million images, another we scale up with small noisy
 image taxpayers.
 And the second column is how do we perform the bootstrapping?
 C stands for captioning and F stands for filtering.
 It means whether we do captioning to generate synthetic captions, or we do filtering to
 remove the noisy captions, or we do both together.
 So if you look at the first row, second row, third and fourth row, you can see that both
 the captioning and the filtering can help individually.
 And if you combine them together, they really have complemented each other.
 So by generating synthetic captions, and at the same time, try to remove the noise, we
 can achieve, I would say a quite good amount of gain in these two different, four different
 data sets covering both the retrieval task and the captioning task.
 So I think that's one of the key results we have here.
 And also maybe then it goes to the second table is how do we do the bootstrapping of
 the captions?
 So do we use beam search?
 Or do we use nuclear sampling?
 So the difference between those two approaches is that beam search is a deterministic sampling,
 not sampling, deterministic decoding strategy, where you try to find the most likely sentence
 associated with the image.
 And nuclear sampling is a stochastic approach where you try to sample according to some
 probability distribution.
 We find that surprisingly, if you compare beam search with no generation, there is a
 good gain achieved by beam search.
 But by moving beam search to nuclear sampling, there is a similar amount of gain.
 So this is something that we didn't expect at the first time we see the results.
 And after we really deep dive into what the captions look like, how does beam search and
 nuclear sampling generate different captions, we found out that the beam search will generate
 a kind of a safe caption that accurately describes the image most of the time, but it's not surprising.
 So you can commonly see those captions in the data set.
 And that doesn't add a lot of extra knowledge for the model to learn.
 But the nuclear sampling really introduces some really diverse captions that are more
 like human written ones.
 Humans don't write a very boring distribution like a man is with a dog in a park.
 So it's a very boring caption.
 But nuclear sampling can give you more diverse captions.
 And if you look at a noise ratio, which is actually how much of those captions were filtered
 out by our filter, you can also see that beam search is less noisy.
 But even though it's less noisy, it's not as beneficial as nuclear sampling here.
 And this really raises another question, which I think is a very interesting future work,
 is that is nuclear sampling the best way?
 So because those models are pertrained with the language modeling laws, which is kind
 of deterministic laws, you try to maximize the likelihood of your captions.
 And we are just doing that, and we try to do something in the decoding side to try to
 give more diverse captions.
 But this nuclear sampling was used in mostly NLP papers.
 So does there exist some better diverse captioning strategy for image captioning tasks?
 So I think that's a very interesting question.
 I think in recent times, this has been shining through in a lot of works that the fact that
 maybe we don't need to go maximum likelihood in our inference step, but maybe it's a better
 approach to go diverse with the sampling.
 And then exactly what you do have some sort of a classifier or some sort of a filter to
 just scrap out the noise.
 I think that's a really, really good approach.
 And we saw this anywhere.
 I think Dolly famously had Clip re-ranking all the outputs.
 And I think more and more models go towards this.
 It's really cool finding that you're essentially finding exactly the same thing.
 When I look at these numbers, all of the numbers, it's very convincing to see that everything
 uniformly almost uniformly gets better.
 You support whatever you say really well.
 This trend right here, it really works across all of the data sets.
 You uniformly almost get better in all the tables.
 However, the difference is always, the maximum difference is whatever.
 This from here to here is like two points in what is this?
 What's TR?
 It's the true...
 It's a recall, text recall.
 Text recall, sorry.
 Oh yeah, it's down here.
 Text recall, image recall.
 That's like 2%.
 Right here, again, it's like one point something percent.
 So there's a uniformly getting better.
 My question is, given that the getting better is convincing, but the scale of it is like
 yeah, 2% or so, when is it worth to do this week long pre-training you mentioned?
 This is a big procedure.
 The pre-training is big.
 And then to fine tune the pre-training again, when is it worth it?
 From what scale or for what applications does it become actually worth to do something
 like this?
 Yeah, I think that's a very good question.
 And first of all, I would say it is worth doing if your data is really...
 If you observe a large amount of noise in the data and maybe your data is incomplete
 in some of the domains.
 For example, here, the web data is primarily dominated by those alt text, which can be
 different from what human would write to describe an image.
 So if there is a noisy scenario or a domain gap, I think it's worth to do so.
 And secondly, actually, we have also released our dataset after bootstrapping so that if
 you are just trying to do regionally pre-training in a similar domain, I think you can just
 download our version and use that as a starting point to avoid the first round of pre-training.
 And maybe certainly about your previous comment that we have really unanimous improvement
 for those tasks.
 Actually in one of the tasks, maybe you can scroll down the paper.
 Let me try to find...
 I think it's the NLVR task.
 Table eight, maybe?
 Yeah, yeah, table eight.
 Yeah, actually for this task, this is where we find the better quality of captions doesn't
 necessarily give you a better game if you compare here.
 And actually by scaling up the number of pre-training image, it doesn't correlate very straightforwardly
 to a downstream performance game.
 So I think it still depends on your alignment between your pre-training and your downstream
 objective.
 So for most of the tasks, it is well aligned.
 And that's why improving your pre-training data quality can improve your downstream task.
 Yeah, maybe I can add a few sentences in terms of whether it is worthwhile to improve that
 much.
 I think if you really imagine the big picture here in terms of the multimodal retrieval,
 let's say if you deploy this retrieval algorithm, and that manages to improve their profit by
 1%, that's a huge achievement.
 You won a lot.
 So at Salesforce, we also have the retrieval.
 We also work with clients for their retrieval services.
 So in terms of that, if you just let your GPU run for one week and improve by 1%, that's
 a huge improvement, I would say.
 And I would also like to say that these numbers, they kind of, I think, under hype what BLEAP
 has achieved.
 Because I think BLEAP, beyond this relative advantage over its competitors, is also qualitatively
 better in terms of how easy it is to use BLEAP.
 If you really look at the demo we created there on the web, and it just freely asks
 any questions in natural language rather easily.
 In contrast, a lot of these image question answering models, they are not doing the free
 form generation.
 They are kind of doing classification in order to tackle this question answering task.
 This point is, however, not fully demonstrated, I believe, in the current manuscript.
 So if you really want to get impressed, we really suggest you check out our demo and
 put whatever photos you like and questions.
 Cool.
 It's really neat, by the way, that you have a demo to go along with it, because I think
 it makes it more accessible and it demonstrates also the capabilities of this.
 It's almost like we're moving into the world that GPT-3 maybe has created for text with
 these image language models, because we got the same feeling from GPT-3.
 Oh no, I can just go and I can put any text, right, and I can interact with the system
 in a sort of free form way.
 And it's really cool to see that we're also moving in this direction with the image models.
 In terms of just the process of how this research went about, you ended up with a cool system
 with a nice way of bootstrapping data and so on.
 Can you maybe tell us a little bit about stuff that didn't necessarily work out during the
 research?
 Was there any point where you were maybe disheartened a little bit, things that didn't work out?
 What were your low and your high points during the creation of this paper?
 Yeah, actually, one of the experiments we had was when we first tried to scale up the
 potential with small web images using this line data set that we have downloaded, which
 takes quite some time.
 It doesn't help that much.
 So then it feels really feel like why scaling up the data is not benefiting the model.
 So then I did some more analysis and after that I realized that a lot of those images
 are very, very small in the resolution.
 Some are just icons or some brand names.
 And if I remove those, then it begins to show the gains.
 But I think that's one of the kind of the blockers we faced.
 And I think after we first get the bootstrapping, especially the nuclear sampling to give a
 big performance gain, then at that point, we are quite confident that this should be
 a good solution.
 And I think that point is when I realized, okay, this method should work well and we
 can write a paper about it.
 Great.
 Dongxin, do you want to say something?
 Yeah, I believe some of these strategies, they also arise from the internal discussions
 with other group members as well.
 So it's really a lot of crowd intelligence behind the scenes.
 How is the research organized at Salesforce?
 I have a bit of insight into, let's say, the big tech giants like Google and Facebook and
 so on, and they have their research divisions.
 At a company like Salesforce, who is more customer, I want to say customer, all these
 companies are customer oriented, obviously.
 But how is research organized there?
 What do you do while the model is pre-training for a week?
 Do you have other stuff to do or are you mainly researchers or what's life like there?
 Yeah.
 So first of all, I would say that AI is a big part of Salesforce, what they try to achieve,
 to use AI to better help the customers.
 So we have this separate research division, maybe not as large as Google or Facebook,
 but I think everything works quite well in our research team.
 In terms of our day-to-day operation, I think it's mostly similar to other industrial researchers.
 We can be quite flexible to do research or do some more product oriented work.
 We are motivated to do research that can generate high impact, that can really change the field
 in a more substantial way.
 And while we wait for the GPU to finish training, we already just do other research stuff or
 read some papers involving some internal discussions or maybe try to solve some real production
 problems.
 Cool.
 Is there anything else you want to get out about this paper?
 You already said people can go to the web, to your repo, and you have a demo also available.
 Is there anything you'd want to get out?
 What's the easiest for people to get started with this research?
 Yes.
 I think first, again, welcome to try out our demo and welcome to visit our GitHub.
 We do have, I think, quite detailed instructions on how to download and train our fine-tuned
 model.
 And also, I welcome any suggestions or questions you might have about our model that we can
 use that to improve our model or the code.
 That would be great.
 Dongxu, anything, any last messages?
 Our team is expanding, so if you are interested, just let you know.
 Yeah, we are looking for an intern position in the visual language research.
 Cool.
 Who can apply?
 Anyone that is at university?
 Yeah, anyone can apply.
 We hire globally, so we can do remote working now.
 Cool.
 Excellent.
 Okay, Dongxu and Jinan, thank you very much for being here.
 This was a lot of fun.
 Thank you for having us.
 Thank you.
 Have a great day of preparation.