How has contrastive learning been used to pretraining multi-modal models?
Recent years have seen significant success for large-
scale pre-trained vision-language models (Tan and
Bansal, 2019; Chen et al., 2020c; Huang et al.,
2020; Li et al., 2020; Yu et al., 2021; Li et al.,
2021; Wang et al., 2021b; Li et al., 2022a; Xu et al.,
2021; Kim et al., 2021) in a variety of cross-modal
tasks. Self-supervised Multi-modal Contrastive
Learning (SMCL) has lately sparked significant ad-
vancements. (Li et al., 2022b; Radford et al., 2021;
Yao et al., 2021; Li et al., 2021, 2022a) by con-
ducting cross-modal alignment. SMCL consists of
image-to-text and text-to-image contrastive learn-
ing, e.g., with the InfoNCE (Oord et al., 2018) loss.
However, traditional cross-modal contrasting strat-
egy can be problematic given the many-to-many
correspondences between images and texts but few
works notice this issue. Recently, to solve the issue
of noisy image-text pairing in VLP pre-training
datasets, some pioneering work has fed the con-
trastive loss with soft labels in a self-distillation
method (Li et al., 2022b, 2021; Andonian et al.,
2022). Even while these techniques may help re-
duce the number of false negatives, their harmful
effect has not been carefully explored.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
The learning objective of vision-language approach of CLIP does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets, which contributes to its compute and data inefficiency. To address this challenge, we introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data. Our model distills its own knowledge to dynamically generate soft-alignment targets for a subset of images and captions in every minibatch, which are then used to update its parameters. Extensive evaluation across 14 benchmark datasets shows that our method consistently outperforms its CLIP counterpart in multiple settings, including: (a) zero-shot classification, (b) linear probe transfer, and (c) image-text retrieval, without incurring added computational cost. Analysis using an ImageNet-based robustness test-bed reveals that our method offers better effective robustness to natural distribution shifts compared to both ImageNet-trained models and CLIP itself. Lastly, pretraining with datasets spanning two orders of magnitude in size shows that our improvements over CLIP tend to scale with number of training examples.
[{"first":"Alex","last":"Andonian","middle":[]},{"first":"Shixing","last":"Chen","middle":[]},{"first":"Raffay","last":"Hamid","middle":[]}]
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and pre-trained models are available at https://github.com/salesforce/ALBEF/.
[{"first":"Junnan","last":"Li","middle":[]},{"first":"Ramprasaath","last":"Selvaraju","middle":["R."]},{"first":"Akhilesh","last":"Gotmare","middle":["Deepak"]},{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]},{"first":"Steven","last":"Hoi","middle":[]}]
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this paper, we reconsider the problem of (partial) false negative samples from the Mutual Information (MI) Maximization perspective, the traditional contrastive loss (like InfoNCE loss) will equally push away the anchor of all positive samples and negative samples regardless of their possible semantic similarities. We theoretically show that InfoNCE loss will not only maximize the MI between the anchor and positive samples but minimize the MI between the anchor and false negative samples even though they share similar semantic which could provide a possible theoretical explanation for the observation of the existence of false negative samples in the cross-modal contrastive learning will decrease the downstream task performance of VLP models. Above analysis motivate us to propose the VLP model with a novel Semantic Awared Contrastive Learning framework named SACL where different negative samples are assigned with different contrastive weights according to the semantic similarity between them and the anchor.
[{"first":"Chaoya","last":"Jiang","middle":[]},{"first":"Wei","last":"Ye","middle":[]},{"first":"Haiyang","last":"Xu","middle":[]},{"first":"Songfang","last":"Huang","middle":[]},{"first":"Fei","last":"Huang","middle":[]},{"first":"Shikun","last":"Zhang","middle":[]}]
FILIP: Fine-grained Interactive Language-Image Pre-Training
Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.
[{"first":"Lewei","last":"Yao","middle":[]},{"first":"Runhui","last":"Huang","middle":[]},{"first":"Lu","last":"Hou","middle":[]},{"first":"Guansong","last":"Lu","middle":[]},{"first":"Minzhe","last":"Niu","middle":[]},{"first":"Hang","last":"Xu","middle":[]},{"first":"Xiaodan","last":"Liang","middle":[]},{"first":"Zhenguo","last":"Li","middle":[]},{"first":"Xin","last":"Jiang","middle":[]},{"first":"Chunjing","last":"Xu","middle":[]}]
Learning Transferable Visual Models From Natural Language Supervision
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.
[{"first":"Alec","last":"Radford","middle":[]},{"first":"Jong","last":"Kim","middle":["Wook"]},{"first":"Chris","last":"Hallacy","middle":[]},{"first":"Aditya","last":"Ramesh","middle":[]},{"first":"Gabriel","last":"Goh","middle":[]},{"first":"Sandhini","last":"Agarwal","middle":[]},{"first":"Girish","last":"Sastry","middle":[]},{"first":"Amanda","last":"Askell","middle":[]},{"first":"Pamela","last":"Mishkin","middle":[]},{"first":"Jack","last":"Clark","middle":[]},{"first":"Gretchen","last":"Krueger","middle":[]},{"first":"Ilya","last":"Sutskever","middle":[]}]
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.
[{"first":"Junnan","last":"Li","middle":[]},{"first":"Dongxu","last":"Li","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]},{"first":"Steven","last":"Hoi","middle":[]}]
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
The convergence of self-supervised pretraining techniques in natural language processing and computer vision have brought about a renaissance of cross-modal representation learning methods [1,19,30,39,52,63,68,75] where largescale weakly correlated multimodal data (e.g., image-text pairs) is used to learn cross-modal representations using contrastive learning techniques. In particular, the recently proposed CLIP [63] model has garnered significant attention due to its impressive zero-shot recognition ability and excellent transfer performance on downstream tasks.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
However, despite their recent success, multimodal pretraining methods like CLIP [63] are data and compute in-* This work was done when the author was an intern at Amazon. learns a joint vision-language embedding space by bringing corresponding image-text representation together (green links), while repelling unpaired instances away from each other (red links). This formulation does not account for potential semantic similarity between negative samples. We address this issue by learning to predict a distribution of soft-alignment targets (dotted blue edges) in a given minibatch, thereby allowing our model to learn more robust representations. This robustness is evident when comparing predicted distributions on an out-of-distribution image from ImageNet-R dataset [25], where unlike CLIP, our method can correctly classify a goldfish rendered in a stained glass pane. efficient. Much of CLIP's success can be attributed to its voracious appetite for training data, utilizing 400M imagetext pairs and an estimated 3, 584 GPU days for pretraining. As the scale of data increases, pretraining requirements of these methods become increasingly expensive, thereby limiting their widespread adoption in a sustainable manner. This data and compute inefficiency of CLIP [63] can be partially attributed to the underlying assumptions it makes about the web-harvested data it uses for training. Several mainstream vision-language datasets utilize the alttext HTML attribute of images scraped from archived web pages [9,67,71] where captions can often have words unrelated to their corresponding image-content [67]. However, CLIP [63] models the caption for each image to be accurately and exclusively related to only that image (see Figure 1). Moreover, when using larger batch sizes (32K used for CLIP), the likelihood of observing negatives with high semantic similarity increases which can further degrade the learned representations especially those associated with shared semantics between faulty negatives [2].
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To address this challenge, we propose to model the many-to-many relationships between images of webharvested datasets and their corresponding captions more accurately using soft probabilities rather than hard pairing labels. Specifically, we propose a simple yet effective framework for robust contrastive language-image pretraining that uses progressive self-distillation and soft image-text alignment targets to more efficiently learn from noisy data. Instead of explicitly finding, correcting or even pruning noisy correspondences [75,88], our joint student-teacher model dynamically generates a new set of soft-alignments for a random subset of images and captions in every minibatch. This enables our method to model many-to-many relationships while simultaneously re-calibrating potentially poorly matched instances without needing to identify them. Over the course of training, our network generates softalignments for increasingly large subsets of a minibatch, effectively becoming its own teacher. We identify several key elements that allow the student network to predict its targets without representation collapse or reinforcing its mistakes.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We use multiple pretraining datasets to extensively compare our approach to CLIP [63] evaluated on 14 benchmark datasets, where our approach consistently outperforms CLIP under multiple settings. Analysis using an ImageNet-based robustness test-bed [70] shows that our method offers better effective robustness to natural distribution shifts compared to both ImageNet-trained models as well as CLIP. Pretraining with datasets spanning two orders of magnitude in size shows that our improvements over CLIP tend to scale with number of training examples. Lastly, the simplicity of our approach allows it to be readily incorporated into existing and future methods.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
2. Related Work a. Self-Supervised Representation Learning: Selfsupervised learning (SSL) approaches use a pretext task to automatically generate a supervision signal from the data itself, thereby eliminating the dependence on expensive manual data-labeling [31]. Pretext tasks in computer vision include spatial reasoning [15,20,32,54,59], temporal context [23,32,36,48,49], and other visual properties such as hue [13,35,81,82], brightness [29,72] or optical flow [29,62,73,79], reconstruction of modified inputs [59,74,81], and classifying inputs with pseudo-labels [15,16,58] or pseudo-clusters [7,8,86,87]. A promising subset of SSL methods uses a variant of instance discrimination framework [17,77] which learns to align augmented versions of features while distinguishing them from features of other instances using a contrastive loss [8,11,24,47].
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
b. Vision-Language Pretraining: Joint vision-language pretraining (VLP) is an active research area [1,19,39,63,68] where the availability of large-scale image-text datasets e.g., YFCC100M [71] and Conceptual Captions [9,67] has played a key role in its progress. Although multiple concurrent works are being proposed to further improve VLP models [75], our work is different from them in a few important ways. Specifically, unlike EfficientCLIP [75] that proposes an ensemble approach to obtain a less noisy data subset for cross-modal training, our method attempts to sidestep this problem altogether by re-purposing as opposed to completely removing noisy data. Similarly, DeCLIP [39] improves on the data-efficiency of CLIP [63] by leveraging intra-model contrastive learning along with a nearestneighbor feature bank to augment negatives. However, incorporating these supervision sources can be computationally expensive. In contrast, our approach offers a simple yet effective way to improve the data-efficiency of CLIP [63] without incurring additional computational cost.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
c. Learning from Noisy Data: Several techniques have been developed to increase the robustness of label noise especially in the supervised context [22,40,60,65,83]. These techniques include loss functions that reduce impact of outliers [76,83], metalearning procedures that learn how to correct sources of label noise [1,37,69,84], loss correction approaches that model label noise [50,51,65], regularization techniques aimed at lowering the impact of noise [61], and noise filtering processes that iteratively refine dataset labels and retrain models to obtain a more robust final model [55]. However, these works investigate noise-robust methods in the context of common object detection and classification tasks and cannot directly be applied effectively to crossmodal pretraining tasks. As it stands, noise-robust VLP pretraining methods are still a relatively unexplored topic. d. Knowledge Distillation: Approaches for knowledge distillation (KD) [27] aim to transfer knowledge from one model (i.e., a teacher) to another (i.e., a student). While KD techniques are often motivated by certain performance and efficiency goals [6,10,38,66], researchers have also found that KD methods serve as an effective regularization technique that can reduce model overfitting and improve generalization capabilities [14,40,43,53]. Our approach is motivated by the recent success of self-knowledge distillation approaches [21,78,80] that use the student network as a teacher under supervised settings to achieve high accuracy with reduced computational cost. To the best of our knowledge, we are among the first to investigate progressive selfdistillation in the context of vision-language pretraining.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Unlike CLIP [63], we view the problem of learning aligned vision-language representations from web-scale weaklyannotated data as the challenge of learning many-to-many relationships from noisy image-text correspondences. To address this challenge, we propose a novel vision-language pretraining method that progressively distills a model's own knowledge to soften its initially hard-target alignments, thereby enabling it to learn more transferable representations from the same amount of training data (see Figure 2). In the following, we first establish cross-modal contrastive learning objective and identify some of its limitations. We then introduce our novel progressive self-distillation approach and explain how it addresses these limitations.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We consider a batch of N semantically paired image-text tuples {(v i , t i )} i=1:N drawn from a cross-modal dataset. The goal of cross-modal contrastive pretraining is to learn encoders f v for image data and f t for text data such that for a given semantically related instance
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
are close together (i.e., "aligned") under some distance metric, while the unpaired image and text embeddings are farther apart (i.e., "unaligned").
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Recall that CLIP [63] trains these image and text encoders with a contrastive loss by minimizing the InfoNCE [56] loss
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
where L v is the loss for aligning images to text and L t is the loss for aligning text to images. Specifically, L v is defined as:
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
where sim(v i , t j ) = v T i t j is typically chosen to be the dot product (cosine similarity), τ is a learnable softmax temperature parameter and I ij is an element from the identity matrix I N . Since InfoNCE is symmetric, L t and P t are defined in a similar manner.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
For convenience, let V, T ∈ R N×d be matrices that contain a batch of image and text embeddings whose rows are populated with v 1:N and t 1:N , respectively. Then, the In-foNCE loss can be re-written compactly in matrix form as:
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
where H is the batched (row-wise) cross-entropy function with mean reduction and ρ is the standard softmax function applied row-wise such that each row sums to one. Equation 3 shows that InfoNCE loss is simply the cross entropy between a one-hot distribution I ij and estimated probability P v (v i , t j ; τ ). It enforces the strict assumption that an image v i selected from a batch should be paired exclusively with text t i while being repelled from all other t j .
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
However, this assumption generally does not hold for two important reasons. First, it is likely that a given image would be aligned to several text captions to different degrees especially under large batch-size settings. Second, the ground truth pairings in large-scale weakly-annotated datasets may be simply incorrect or describe loose correlations between images and their corresponding texts.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To address the aforementioned limitations of using the In-foNCE loss to train on noisy cross-modal data, we propose to adopt a knowledge distillation framework where the predictive probabilities produced by a teacher network are used as soft target distributions to train a student network.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Our framework offers two key advantages. First, in the process of generating target image-text alignments, a well-trained teacher can combat poorly captioned images by re-pairing them with stronger semantic matches from the batch, thereby providing a cleaner learning signal for the student network. Second, by providing soft targets, the teacher can convey many-to-many relationships in a batch.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Specifically, to estimate the correspondence between image v i and text t i , our teacher model employs image and text encodersf v andf t to compute 2 -normalized teacher embeddingsṽ i andt j which are similarly batched as rows in the matricesṼ andT respectively. Our method uses a swapped prediction strategy to produce soft target distributions A v and A t to supervise student training. These target distributions are defined as:
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
where ρ is the standard softmax function now using a secondary teacher temperatureτ that transforms and re-scales raw logits into probabilities. Swapped prediction improves on the well established forward bootstrapping approach [65] by using predictions from opposite modality. Unlike bootstrapping, swapped prediction computes the image alignment scores A v from the text encoder posterior probabilities and vice-versa, thus aggregating information over all other instances from the opposite modality. Intuitively, the strength of alignment from image v i to text t j is based on the probability that text t j should be matched with image v i compared to all other v j . This strategy has shown promise in related contrastive learning settings [50], which is consistent with our results.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Replacing the I N targets in Equation 3 with estimated soft-alignment probabilities A v and A t allows the teacher to re-calibrate the attractive and repulsive forces between image and text embeddings in the representation space based on its estimated similarity between instances. For instance, a faulty negative pair (v i , t j ), which may have high semantic similarity is assigned a similarity score of zero by
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
A cute kitten looking at the camera ⌊-N⌋ aligned instances Figure 2. Approach Overview -We partition a minibatch into aligned instances (rows above the dotted red line) and unaligned instances (rows below the dotted red line). Our teacher network is trained on the aligned instances using the standard InfoNCE loss. By normalizing the predictions from the opposite modality (columns to the right of the dotted yellow line), the teacher provides the estimated soft-alignments for the unaligned data to supervise the student. Over the course of training, the proportion of soft-alignments provided to the student increases as the teacher's representations become more reliable.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
the InfoNCE loss, whereas our method provides A ij as a target which should be larger given a well trained teacher.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We now explain how to start such a teacher network and how its contribution to learning process evolves over time.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Conventional KD and SSL methods offer numerous potential teacher choices, e.g., larger but static pretrained teacher networks [27], or networks that share the same model architecture but use weights from a previous epoch [42], or as an exponential moving average [24]. The primary drawback to these approaches is reduced computational and memory efficiency as they require a secondary inference stage using additional model weights that must be kept in memory.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To circumvent these issues, we adopt a self-distillation strategy where the student network acts as its own teacher (i.e., f v =f v , f t =f t ). The idea here is to update the targets of the student contrastive objective using the current state of the model. Intuitively, as the learning improves over time, its representation can be trusted to make more accurate predictions. This mitigates the negative effects of noisy pairings as incorrect pairs are increasingly likely to be inconsistent with the consensus learned from the rest of the data as training progresses. By refining inconsistent alignments, the model can develop more coherent representations, which further improves its ability to evaluate the consistency of noisy image-text pairs.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
As our objective relies on some basic level of alignment between corresponding image and text representations, we introduce a novel procedure that progressively increases the contribution of self-distillation to the contrastive learning process over the course of training. Our model therefore dynamically evolves into its own teacher as training progresses, which differs from the standard knowledge distillation setting where the teacher is often static and separate.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We achieve this dynamic progression by randomly partitioning a batch of N image-text pairs into N a = αN "aligned" instances and N u = N − αN "unaligned" instances where α ∈ [0, 1] determines their relative proportions. The aligned subset is used to train the teacher network using the hard ground truth pairings and the standard InfoNCE loss. The teacher network then employs our aforementioned swapped prediction strategy to estimate soft-alignments on the unaligned instances to supervise the student. We refer to this random minibatch partitioning as dynamic as opposed to static as the global partitioning of instances into aligned and unaligned subsets is refreshed for each training epoch.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To increase the strength of the teacher's influence on learning, we decrease the value of α gradually, in the same way that the learning rate can be scheduled. While there are several strategies to decrease α as a function of the training iteration, e.g., step-wise, linear, etc., we use a cosineannealing schedule [38] specified by a start and end value.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To summarize our overall learning procedure, we first compute the batched student-teacher embedding with V = V and T =T. Next, we extract the first N a rows to form aligned subset of teacher embeddingsṼ a ,T a , and the last N u rows for the unaligned student embeddings V u , T u . Altogether, our final objective function is defined as:
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
where I Na ∈ R Na×N is the zero-padded identity matrix, while A v , A t are indexed to match the unaligned student embeddings.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We start by describing our experimental setup which aims to match CLIP [63] as closely as possible for fair comparison. We then demonstrate the advantages of our method over CLIP [63] for: (a) zero-shot classification, (b) finetuning (i.e., linear probe), and (c) image-text retrieval.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We apply our pretraining method to three image-text datasets varying in scale, scope and noise: 
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In the following experiments, the image encoders follow ViT-B/32 vision transformer architecture proposed in CLIP [63], while the text encoder's transformer-based architecture follows modification proposed in [63]. Image and text features are projected to a shared 512-D space and L2 normalized before participating in contrastive loss. Models are trained from scratch for 100 epochs using the Adam optimizer [33] with weight decay and a cosine annealing learning rate schedule with warmup [44]. As done in [63], the learnable temperature parameter τ is initialized to 0.07 and clamped to values less than 100. Automatic mixed-precision [45] training is used to save on memory and achieve minibatch sizes of 4096. Input images are randomly cropped and resized to 224 × 224 resolution during pretraining and the maximum length of the text is limited to 77 tokens via random sub-sequence sampling similar to [63]. Training is conducted on as many as 8 Nvidia A-100 GPUs with the longest experiments spanning up to several days. The partitioning factor α is decayed from 0.8 to 0.2 over the course of training using cosine annealing. De-tails regarding hyperparameter values for different datasets and models are provided in the supplementary material.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We compare our approach to a re-implementation of CLIP based on the methods and pseudo-code described in the original paper as well as open source implementations such as OpenCLIP [28]. The performance of our reimplementation is consistent with that achieved by Open-CLIP and unreleased CLIP models trained on reduced subsets of the 400M dataset; thus, we retrain a baseline CLIP model for each pretraining dataset to serve as a proxy.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
a. Evaluation Protocol -We measure zero-shot and linear probe classification performance using Top-1 accuracy. For the linear probe experiments, we train the linear classifier with the L-BFGS optimizer on extracted visual features as described in [63]. We use standard retrieval metrics: recall at rank K (R@K, higher is better) and mean rank (MnR, lower is better), to evaluate the retrieval performance of our model. R@K (Recall at K) calculates the percentage of test samples for which the correct result is found in the top-K retrieved points to the query sample.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
b. Benchmark Datasets -We evaluate the zero-shot and linear probe classification performance of proposed approach on a suite of benchmark evaluation datasets, including ImageNet [12], Places365 [85], ObjectNet [3], and several recent variants of ImageNet aimed at evaluating the robustness of trained models to natural (as opposed to synthetic) distribution shifts [25,26,64].
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
c. Prompt Ensembling with Templates -Consistent with previous work, we find that our approach benefits from prompt ensembling to augment the original class label for downstream tasks. For fair comparison, we use the same set of prompt templates published in CLIP [63], which generally take on the form of "a photo of a {label}."
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Following the pretraining stage, we evaluate our method on zero-shot image classification achieved through natural language input, and compare to its CLIP counterpart trained under identical settings. Table 1 lists the top-1 accuracy (%) of zero-shot image classification across a suite of benchmark datasets. Given a fixed amount of pretraining data, our method considerably outperforms its CLIP counterparts in terms of average top-1 accuracy over all datasets, achieving as much as a 6.19% absolute improvement in the CC3M pretraining regime. Notably, our method surpasses CLIP on Im-ageNet and all of the ImageNet variants. Our method also achieves substantial performance gains on ImageNet-Renditions (ImageNet-R) [25], a dataset specifically aimed at evaluating out-of-distribution robustness, with an average  Our method consistently learns representations more amenable to zero-shot classification than CLIP given same amount of pretraining data.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
performance gain of nearly 8.5% when considering Conceptual Captions datasets [9,67]. Figure 1 shows how our method is able to provide more accurate predictions for a goldfish rendered in a stained glass pane.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
To better understand the relationship between the amount of pretraining data and relative differences in downstream performance between our method and CLIP, we perform a sweep of experiments that restrict pretraining to 5, 10, 25, 50, and 100% of the CC12M dataset. Figure 3 plots the average zero-shot classification score for both methods, and shows that our method maintains higher performance throughout this range. Interestingly, our method appears to provide the largest performance increases at the boundaries of the plot, suggesting that our method can improve dataefficiency in data-limited regimes, in addition to providing larger performance gains as the dataset size increases. . Effective Robustness Evaluation -Our method produces features more robust to natural distribution shift compared to CLIP features. When comparing models with similar ImageNet performance, our method produces representations that offer better performance on naturally shifted distributions. Mean Transfer Accuracy is computed over the ImageNet-R/O/A/V2 test sets. Bestfit trend lines suggests that the effective robustness of our method (blue) outpaces CLIP (red) as ImageNet accuracy increases.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
CLIP models [63,70] have been found to be more robust to natural distribution shifts when compared to standard models trained on ImageNet. This phenomena is illustrated in Figure 4 with ImageNet accuracy on the x-axis and mean accuracy on ImageNet-A, ImageNet-O, ImageNet-R, and ImageNetV2, on the y-axis. Previous work [46,70] has found that the in-distribution and out-of-distribution accuracies of ImageNet models follow a predictable linear trend (plotted in green), and CLIP models established a trend (plotted in red) of improved effective robustness. Note that the slope of linear-fit to our model is higher than that for the CLIP model, suggesting that our effective robustness improves over CLIP with increasing scale.  Table 2. Linear Probe Performance -Linear probe top-1 accuracy (%) of our method compared to baseline CLIP on four benchmark datasets is given. Our method learns visual features that consistently achieve improved finetuning performance, suggesting that our loss helps to improve within-modal feature alignment.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We report our linear probe performance on 4 downstream datasets in Table 2. Our method outperforms its CLIP counterpart in every case, suggesting that our learned visual features alone (i.e., considering within-modal alignment only) are more transferable than CLIP.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Given the cross-modal nature of models under investigation, image-text retrieval consists of two sub-tasks: image-to-text and text-to-image. We evaluate our method on COCO test set with 5K images, each with 5 unique captions. Table 3 shows our zero-shot retrieval performance relative to baseline CLIP under COCO, CC3M and CC12M pretraining datasets. Relative to CLIP, our method consistently outperforms in both text-to-image and image-to-text subtasks, with both methods generally achieving slightly higher performance in text-to-image sub-task. Note that, due to data domain differences, pretraining directly on COCO produces models with higher downstream performance than pretraining on CC12M despite the 100× increase in datasize. The last four rows show that, as expected, finetuning Conceptual Captions pretrained models on COCO produces retrieval performance that exceeds corresponding baselines and the performance from training on COCO alone.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We now investigate the contributions of individual components of our method, specifically the swapped prediction strategy for estimating soft-alignments over the forward bootstrapping strategy, dynamic minibatch partitioning, and progressive self-distillation to downstream performance. Following pretraining on COCO, we measure models' performance on COCO zero-shot image-text retrieval and ImageNet zero-shot classification.  Table 4 shows that directly using the model's posterior (bootstrapping) to assign alignments to a static subset of inputs leads to a degradation of learned representations and decreased performance compared to the baseline. Here the teacher network is likely reinforcing its own mistakes without penalty from the loss, leading to partial representation collapse for the unaligned examples. Utilizing the swapped prediction strategy restores zero-shot performance to slightly above baseline suggesting effective method for addressing this issue; however, retrieval performance is still lower than the baseline. By incorporating a dynamic partitioning strategy, our method exceeds baseline performance.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
a. Distribution of Similarity Scores: Our method is largely motivated by the observation that previous VLP approaches neglect potential semantic similarity between negative samples and that accounting for this phenomenon can improve learned representations. In Figure 5, we plot the distribution of similarity scores for both positive and negative samples drawn from the COCO testing set. The left subplot reveals that our method consistently yields larger similarity scores for positive pairs compared to its CLIP counterpart and the OpenAI pretrained CLIP. Interestingly, the histogram of negative similarity scores shows that our method also assigns higher similarity scores to negative pairs. While it may seem counter intuitive to assign greater similarity scores to negative samples, we argue that doing so is the very reason our method captures greater similarity between positive pairs. By allowing some degree of alignment between the right set of negative examples, our method is able to minimize the inconsistencies between shared context of related positives and negatives. This in turn allows us to learn an overall more coherent representation space, resulting in increased robustness and downstream performance.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
b. Visualizing Text-Image Retrievals: In Figure 6, we show the comparative list of top ten retrieved images for 5 example text queries. Overall, these retrievals suggest that  Table 3. Image-Text Retrieval -Zero-shot retrieval performance on the COCO 5K testing set as measured by recall at 1,5,10 (higher is better) and mean rank (lower is better). The last four rows (marked with * ) report the zero-shot retrieval results of the same models further finetuned on the COCO captions training set. OpenAI CLIP Baseline CLIP Our CLIP Figure 5. Similarity Scores -Similarity scores distribution for positive and negative pairs in joint cross-modal space after training with original CLIP loss and our proposed loss is provided. Compared to our baseline (COCO) pretrained CLIP and OpenAI's pretrained CLIP model, our method yields similarity scores with higher mean and lower variance for positive pairs. While COCO pretrained CLIP model concentrates negatives' similarity scores around zero, our method concentrates them at higher levels as it allows for some degree of semantic similarity between negatives.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
our learned features more comprehensively capture all potential similarities between a text snippet and image. In contrast, baseline CLIP features tend to more narrowly focus on one specific commonality while neglecting others. For example, in the last row of Figure 6, CLIP strongly targets "living room" but misses the cat. Our approach on the other hand is successfully able to extract all the key aspects of the query including the cat, the chair and the living room. Similar trends are found in other examples.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
We proposed a novel cross-modal contrastive learning framework with progressive self-distillation and soft imagetext alignments. Our approach distills its own knowledge to dynamically generate soft-alignment targets for a subset of samples in every minibatch, which enables it to efficiently learn robust representations from noisy data. Exten- Figure 6. Example Text-Image Retrievals -Given a text query, we display the top ten most semantically related images (ranked left to right) retrieved by CLIP and our method. Compared to CLIP, our method continues to retrieve images that more holistically match the text description, even after the ground truth image has appeared in the ranking.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
sive evaluations across 14 benchmark datasets showed that our method consistently outperforms its CLIP counterpart in multiple settings. Moreover, our method provides better effective robustness to natural distribution shifts compared to existing state-of-the-art methods. Going forward, we plan to further improve the efficiency of our approach by investigating data redundancy, network architectures and optimization algorithms during pretraining, so that we can generalize to even larger scale with fewer resources. 
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In Table 2, we list additional details for the evaluation datasets in this study including the number of classes and the sizes of the training-testing splits. For the last 5 rows (ObjectNet and ImageNet-{A,O,R,V2}), we list only the number of classes and the testing set size as these have been designated as "testing-only" datasets.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Our pretraining implementation largely follows CLIP [8] with significant deviations motivated by computational constraints or empirical observations.  Table 3. Common hyperparametersused for both baseline CLIP pretraining and our method.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In order to underscore how our method improves dataefficiency without incurring additional computation cost, we note that the number of trainable parameters in the ViT-B/32 (151.3M) and RN50 (102.0M) CLIP models are precisely the same under our method as the architectures remain unmodified with no additional parameters required. Also, the average time per training iteration (e.g., 0.253s for our COCO experiments using 8 Nvidia V100 GPUs) and memory requirements are effectively identical as well. Finally, we contrast our self-distillation approach to traditional knowledge-distillation methods and other choices of teacher network (e.g., a momentum teacher design), which may provide similar data-efficiency improvements, but incur significant computational costs of at least 2x compute and memory requirements or more. Table 4 lists the prompt templates used for zero-shot classification on the evaluation datasets. For each template, string interpolation replaces the placeholder symbol {} with a text representation of the category name, and a grammatical correction is applied to the preceding article, i.e., a → an for categories that start with a vowel.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
The linear probe evaluation involves training a logistic regression classifier on the frozen visual features extracted using the model's image encoder. Following CLIP [8], we train the logistic regression for a maximum of 1000 iterations using the L-BFGS optimization algorithm provided by scikit-learn [7]. We use the train/test split sizes listed in Table 2.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In this section, we provide a comparison of our work to unpublished concurrent works. While the concurrent works discussed in this section are aimed at improving on CLIP, they altogether vary along several dimensions including pretraining dataset, backbone architecture, hyperparameters and training details. Due to the large number of unique experimental configurations, it is not feasible to precisely replicate the setting of each concurrent work; therefore, we provide our most closely matched experiments while acknowledging that the comparisons are not exactly one-to-one analogs. Table 5 lists the commonly reported ImageNet zero-shot Top 1 accuracy achieved by concurrent methods aimed at reproducing CLIP and/or addressing its limitations. Each work additionally provides a re-implementation of CLIP (listed as CLIP impl. followed by a citation). Even for architecture and dataset matched experiments, the difference in accuracy can differ by as much as 4.8% (compare [6] and [8]), further highlighting the challenges of providing meaningful comparison. From the perspective of raw performance, our method achieves the highest absolute top 1 accuracy on zero-shot ImageNet classification out of all approaches and architectures that use at most 15M pretraining examples.  Table 5. Zero-Shot ImageNet Classification Comparison -Zero-shot Top1 accuracy (%) of our method compared to concurrent works.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In order to demonstrate the effectiveness of our method to a broader experimental setting, we present results from experiments that utilize the widely adopted ResNet50 (RN50) visual backbone in Table 6. These results are highly consistent with performance trends observed with the ViT backbone (even with minimal hyperparameter tuning), suggesting that our method is amenable to CNN-based backbones as well.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Our method is fairly robust to the choice of α scheduling. For example, replacing cosine annealing with a simple linear schedule produces a network with near identical performance on downstream evaluations. Specifically, for COCO the two scheduling methods result in an absolute mean difference in top1 classification of 0.13% and relative difference of 0.67% for our method using a ViT backbone. Similarly, for CC3M we observe an absolute mean difference of −0.22% and relative difference of −0.63%.  Table 6. Zero-Shot Image Classification Comparison with RN50 backbone -Zero-shot Top1 accuracy (%) of our method compared to baseline CLIP on numerous downstream benchmark datasets. Note: Results are derived from published hyperparameters for the baseline and minimal hyperparameter tuning of our method to account for differences in backbone architecture.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In Figures 1 and 2, we provide additional text-image retrieval results computed on the COCO test set. Consistent with the retrievals shown in Figure 6 from our main paper, our method more consistently captures the full semantic extent of the query caption, whereas the baseline CLIP model tends to narrowly focus on one particular aspect. For example, in the first row of Figure ??, CLIP does retrieve a "black and white cat," but also retrieves a black and white dog, a black and white zebra and a black and white photograph of boxes, whereas our method retrieves images containing a cat in 9 out of the 10 top retrievals.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In Figure 3, we show qualitative examples of the instances where our method shows improved robustness over its CLIP counterpart for out-of-distribution images drawn from the ImageNet-R dataset. While CLIP struggles to handle certain artistic styles (angular shark depicted in row 1, goldfinch in row 4, tattooed tree frog in row 6), strong color patterns (black and white colors in row 3), or subjects in unusual contexts (rows 5,7,8), our method is able to more consistently provide a reasonable set of top predictions, which is consistent with the quantitative improvement of nearly 8.5% on average when considering Conceptual Captions pretraining as reported in Table 1 of our main paper.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
In Figure 4, we show a matrix of pairwise cosine similarity scores assigned to a batch of images and corresponding text snippets by our method compared to its CLIP counterpart. These similarity matrices present concrete examples of the trends captured by the distributions shown in Figure 5 of the main paper. Namely, it shows that CLIP has been optimized to assign high similarity scores along the diagonal (positive pairs) and low similarity to off diagonal elements (negatives), even when there is non-negligible semantic similarity between unpaired instances (e.g., the text "a black-and-white silhouette..." and the black-and-white image of a photographer dressed in black clothing) . In contrast, our method yields elevated scores for negative pairings that show this amount of secondary similarity. As a consequence, we empirically observe that our learned representations produce larger scores for ground truth positive pairs and lead to more robust zero-shot classification performance.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
A primary motivation driving this work is to increase the robustness and efficiency of a vision-language pretraining (VLP) method that has recently given rise to a set of socalled foundation models [1]. Due to the anticipated role that foundation models are to play in the immediate de- Figure 1. Example Text-Image Retrievals -Given a text query, we display the top ten most semantically related images (ranked left to right) retrieved by CLIP and our method. Compared to CLIP, our method continues to retrieve images that more holistically match the text description, even after the ground truth image has appeared in the ranking.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
velopment of AI systems, contributions to advancing the core training method will have far-reaching impacts on the field and downstream application areas by definition. Since improving the efficiency and lowering the computational/environmental cost associated with this VLP method is a primary objective of our work, we would like our work to assist in providing greater accessibility to the study, development and deployment of these VLP methods. Figure 2. Additional Text-Image Retrievals -Given a text query, we display the top ten most semantically related images (ranked left to right) retrieved by CLIP and our method. Compared to CLIP, our method continues to retrieve images that more holistically match the text description, even after the ground truth image has appeared in the ranking.
Robust Cross-Modal Representation Learning with Progressive Self-Distillation
Robustness to challenging, novel, and even adversarial examples is rapidly becoming an extremely important part of modern computer vision systems, which are now starting to be deployed in sensitive contexts such as autonomous vehicles [5] and medical applications [2] with life and death consequences. Additionally, the increasing diversity of data sources, ranging from massive and cumbersome datasets to extremely limited and highly sensitive information, poses several practical and environmental challenges to consistently training robust and reliable machine learning systems. Our proposed framework aims to address these aspects jointly.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Vision-and-Language Pre-training (VLP) aims to learn multimodal representations from large-scale image-text pairs that can improve downstream Vision-and-Language (V+L) tasks. Most existing VLP methods (e.g. LXMERT [1], UNITER [2], OSCAR [3]) rely on pre-trained object detectors to extract region-based image features, and employ a multimodal encoder to fuse the image features with word tokens. The multimodal encoder is trained to solve tasks that require joint understanding of image and text, such as masked language modeling (MLM) and image-text matching (ITM).
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
While effective, this VLP framework suffers from several key limitations: (1) The image features and the word token embeddings reside in their own spaces, which makes it challenging for the multimodal encoder to learn to model their interactions; (2) The object detector is both annotation-expensive and compute-expensive, because it requires bounding box annotations during pre-training, and highresolution (e.g. 600×1000) images during inference; (3) The widely used image-text datasets [4,5] are collected from the web and are inherently noisy, and existing pre-training objectives such as MLM may overfit to the noisy text and degrade the model's generalization performance.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We propose ALign BEfore Fuse (ALBEF), a new VLP framework to address these limitations. We first encode the image and text independently with a detector-free image encoder and a text encoder. Then we use a multimodal encoder to fuse the image features with the text features through crossmodal attention. We introduce an intermediate image-text contrastive (ITC) loss on representations from the unimodal encoders, which serves three purposes: (1) it aligns the image features and the text features, making it easier for the multimodal encoder to perform cross-modal learning; (2) it 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
improves the unimodal encoders to better understand the semantic meaning of images and texts; (3) it learns a common low-dimensional space to embed images and texts, which enables the image-text matching objective to find more informative samples through our contrastive hard negative mining.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
To improve learning under noisy supervision, we propose Momentum Distillation (MoD), a simple method which enables the model to leverage a larger uncurated web dataset. During training, we keep a momentum version of the model by taking the moving-average of its parameters, and use the momentum model to generate pseudo-targets as additional supervision. With MoD, the model is not penalized for producing other reasonable outputs that are different from the web annotation. We show that MoD not only improves pre-training, but also downstream tasks with clean annotations.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We provide theoretical justifications on ALBEF from the perspective of mutual information maximization. Specifically, we show that ITC and MLM maximize a lower bound on the mutual information between different views of an image-text pair, where the views are generated by taking partial information from each pair. From this perspective, our momentum distillation can be interpreted as generating new views with semantically similar samples. Therefore, ALBEF learns vision-language representations that are invariant to semantic-preserving transformations.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We demonstrate the effectiveness of ALBEF on various downstream V+L tasks including image-text retrieval, visual question answering, visual reasoning, visual entailment, and weakly-supervised visual grounding. ALBEF achieves substantial improvements over existing state-of-the-art methods. On image-text retrieval, it outperforms methods that are pre-trained on orders of magnitude larger datasets (CLIP [6] and ALIGN [7]). On VQA and NLVR 2 , it achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art method VILLA [8], while enjoying much faster inference speed. We also provide quantitative and qualitative analysis on ALBEF using Grad-CAM [9], which reveals its ability to perform accurate object, attribute and relationship grounding implicitly.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Most existing work on vision-language representation learning fall into two categories. The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10,11,12,13,1,14,15,2,3,16,8,17,18]. Methods in this category achieve superior performance on downstream V+L tasks that require complex reasoning over image and text (e.g. NLVR 2 [19], VQA [20]), but most of them require high-resolution input images and pre-trained object detectors. A recent method [21] improves inference speed by removing the object detector, but results in lower performance. The second category focuses on learning separate unimodal encoders for image and text [22,23,6,7]. The recent CLIP [6] and ALIGN [7] perform pre-training on massive noisy web data using a contrastive loss, one of the most effective loss for representation learning [24,25,26,27]. They achieve remarkable performance on image-text retrieval tasks, but lack the ability to model more complex interactions between image and text for other V+L tasks [21].
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
ALBEF unifies the two categories, leading to strong unimodal and multimodal representations with superior performance on both retrieval and reasoning tasks. Furthermore, ALBEF does not require object detectors, a major computation bottleneck for many existing methods [1,2,3,8,17].
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Knowledge distillation [28] aims to improve a student model's performance by distilling knowledge from a teacher model, usually through matching the student's prediction with the teacher's. While most methods focus on distilling knowledge from a pre-trained teacher model [28,29,30,31,32], online distillation [33,34] simultaneously trains multiple models and use their ensemble as the teacher. Our momentum distillation can be interpreted as a form of online self-distillation, where a temporal ensemble of the student model is used as the teacher. Similar ideas have been explored in semi-supervised learning [35], label noise learning [36], and very recently in contrastive learning [37]. Different from existing studies, we theoretically and experimentally show that momentum distillation is a generic learning algorithm that can improve the model's performance on many V+L tasks.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
In this section, we first introduce the model architecture (Section 3.1). Then we delineate the pretraining objectives (Section 3.2), followed by the proposed momentum distillation (Section 3.3). Lastly we describe the pre-training datasets (Section 3.4) and implementation details (Section 3.5).  We propose an image-text contrastive loss to align the unimodal representations of an image-text pair before fusion. An image-text matching loss (using in-batch hard negatives mined through contrastive similarity) and a masked-language-modeling loss are applied to learn multimodal interactions between image and text. In order to improve learning with noisy data, we generate pseudo-targets using the momentum model (a moving-average version of the base model) as additional supervision during training.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
As illustrated in Figure 1, ALBEF contains an image encoder, a text encoder, and a multimodal encoder. We use a 12-layer visual transformer ViT-B/16 [38] as the image encoder, and initialize it with weights pre-trained on ImageNet-1k from [31]. An input image I is encoded into a sequence of embeddings:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where v cls is the embedding of the [CLS] token. We use a 6-layer transformer [39] for both the text encoder and the multimodal encoder. The text encoder is initialized using the first 6 layers of the BERT base [40] model, and the multimodal encoder is initialized using the last 6 layers of the BERT base . The text encoder transforms an input text T into a sequence of embeddings {w cls , w 1 , ..., w N }, which is fed to the multimodal encoder. The image features are fused with the text features through cross attention at each layer of the multimodal encoder.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We pre-train ALBEF with three objectives: image-text contrastive learning (ITC) on the unimodal encoders, masked language modeling (MLM) and image-text matching (ITM) on the multimodal encoder. We improve ITM with online contrastive hard negative mining.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Image-Text Contrastive Learning aims to learn better unimodal representations before fusion. It learns a similarity function s = g v (v cls ) g w (w cls ), such that parallel image-text pairs have higher similarity scores. g v and g w are linear transformations that map the [CLS] embeddings to normalized lower-dimensional (256-d) representations. Inspired by MoCo [24], we maintain two queues to store the most recent M image-text representations from the momentum unimodal encoders. The normalized features from the momentum encoders are denoted as g v (v cls ) and g w (w cls ). We define
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
For each image and text, we calculate the softmax-normalized image-to-text and text-to-image similarity as:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where τ is a learnable temperature parameter. Let y i2t (I) and y t2i (T ) denote the ground-truth one-hot similarity, where negative pairs have a probability of 0 and the positive pair has a probability of 1. The image-text contrastive loss is defined as the cross-entropy H between p and y:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
GT: standing Figure 2: Examples of the pseudo-targets for MLM (1st row) and ITC (2nd row). The pseudo-targets can capture visual concepts that are not described by the ground-truth text (e.g. "beautiful waterfall", "young woman").
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Masked Language Modeling utilizes both the image and the contextual text to predict the masked words. We randomly mask out the input tokens with a probability of 15% and replace them with the special token [MASK] 1 . LetT denote a masked text, and p msk (I,T ) denote the model's predicted probability for a masked token. MLM minimizes a cross-entropy loss:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where y msk is a one-hot vocabulary distribution where the ground-truth token has a probability of 1.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Image-Text Matching predicts whether a pair of image and text is positive (matched) or negative (not matched). We use the multimodal encoder's output embedding of the [CLS] token as the joint representation of the image-text pair, and append a fully-connected (FC) layer followed by softmax to predict a two-class probability p itm . The ITM loss is:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where y itm is a 2-dimensional one-hot vector representing the ground-truth label.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We propose a strategy to sample hard negatives for the ITM task with zero computational overhead. A negative image-text pair is hard if they share similar semantics but differ in fine-grained details. We use the contrastive similarity from Equation 1 to find in-batch hard negatives. For each image in a mini-batch, we sample one negative text from the same batch following the contrastive similarity distribution, where texts that are more similar to the image have a higher chance to be sampled. Likewise, we also sample one hard negative image for each text.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
The full pre-training objective of ALBEF is:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
The image-text pairs used for pre-training are mostly collected from the web and they tend to be noisy. Positive pairs are usually weakly-correlated: the text may contain words that are unrelated to the image, or the image may contain entities that are not described in the text. For ITC learning, negative texts for an image may also match the image's content. For MLM, there may exist other words different from the annotation that describes the image equally well (or better). However, the one-hot labels for ITC and MLM penalize all negative predictions regardless of their correctness.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
To address this, we propose to learn from pseudo-targets generated by the momentum model. The momentum model is a continuously-evolving teacher which consists of exponential-moving-average versions of the unimodal and multimodal encoders. During training, we train the base model such that its predictions match the ones from the momentum model. Specifically, for ITC, we first compute the image-text similarity using features from the momentum unimodal encoders as s (I, T ) = g v (v cls ) g w (w cls ) and s (T, I) = g w (w cls ) g v (v cls ). Then we compute soft pseudotargets q i2t and q t2i by replacing s with s in Equation 1. The ITC MoD loss is defined as: Similarly, for MLM, let q msk (I,T ) denote the momentum model's prediction probability for the masked token, the MLM MoD loss is:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
In Figure 2, we show examples of the top-5 candidates from the pseudo-targets, which effectively capture relevant words/texts for an image. More examples can be found in Appendix.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We also apply MoD to the downstream tasks. The final loss for each task is a weighted combination of the original task's loss and the KL-divergence between the model's prediction and the pseudo-targets. For simplicity, we set the weight α = 0.4 for all pre-training and downstream tasks 2 .
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Following UNITER [2], we construct our pre-training data using two web datasets (Conceptual Captions [4], SBU Captions [5]) and two in-domain datasets (COCO [41] and Visual Genome [42]). The total number of unique images is 4.0M, and the number of image-text pairs is 5.1M. To show that our method is scalable with larger-scale web data, we also include the much noisier Conceptual 12M dataset [43], increasing the total number of images to 14.1M 3 . Details are in Appendix.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Our model consists of a BERT base with 123.7M parameters and a ViT-B/16 with 85.8M parameters.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We pre-train the model for 30 epochs using a batch size of 512 on 8 NVIDIA A100 GPUs. We use the AdamW [44] optimizer with a weight decay of 0.02. The learning rate is warmed-up to 1e −4 in the first 1000 iterations, and decayed to 1e −5 following a cosine schedule. During pre-training, we take random image crops of resolution 256 × 256 as input, and also apply RandAugment 4 [45]. During fine-tuning, we increase the image resolution to 384 × 384 and interpolate the positional encoding of image patches following [38]. The momentum parameter for updating the momentum model is set as 0.995, and the size of the queue used for image-text contrastive learning is set as 65,536. We linearly ramp-up the distillation weight α from 0 to 0.4 within the 1st epoch.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
In this section, we provide an alternative perspective of ALBEF and show that it maximizes a lower bound on the mutual information (MI) between different "views" of an image-text pair. ITC, MLM, and MoD can be interpreted as different ways to generate the views.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Formally, we define two random variables a and b as two different views of a data point. In selfsupervised learning [24,25,46], a and b are two augmentations of the same image. In vision-language representation learning, we consider a and b as different variations of an image-text pair that capture its semantic meaning. We aim to learn representations invariant to the change of view. This can be achieved by maximizing the MI between a and b. In practice, we maximize a lower bound on MI(a, b) by minimizing the InfoNCE loss [47] defined as:
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where s(a, b) is a scoring function (e.g., a dot product between two representations), andB contains the positive sample b and |B| − 1 negative samples drawn from a proposal distribution.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Our ITC loss with one-hot labels (Equation 2) can be re-written as: 
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Minimizing L itc can be seen as maximizing a symmetric version of InfoNCE. Hence, ITC considers the two individual modalities (i.e., I and T ) as the two views of an image-text pair, and trains the unimodal encoders to maximize the MI between the image and text views for the positive pairs. 2 our experiments show that α = 0.3, 0.4, 0.5 yield similar performance, with α = 0.4 slightly better 3 some urls provided by the web datasets have become invalid 4 we remove color changes from RandAugment because the text often contains color information As shown in [48], we can also interpret MLM as maximizing the MI between a masked word token and its masked context (i.e. image + masked text). Specifically, we can re-write the MLM loss with one-hot labels (Equation 3) as
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
where ψ(y) : V → R d is a lookup function in the multimodal encoder's output layer that maps a word token y into a vector and V is the full vocabulary set, and f (I,T ) is a function that returns the final hidden state of the multimodal encoder corresponding to the masked context. Hence, MLM considers the two views of an image-text pair to be: (1) a randomly selected word token, and (2) the image + the contextual text with that word masked.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Both ITC and MLM generate views by taking partial information from an image-text pair, through either modality separation or word masking. Our momentum distillation can be considered as generating alternative views from the entire proposal distribution. Take ITC MoD in Equation 6 as an example, minimizing KL(p i2t (I), q i2t (I)) is equivalent to minimizing the following objective: 
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
It maximizes MI(I, T m ) for texts that share similar semantic meaning with the image I because those texts would have larger q i2t m (I). Similarly, ITC MoD also maximizes MI(I m , T ) for images that are similar to T . We can follow the same method to show that MLM MoD generates alternative views y ∈ V for the masked word y msk , and maximizes the MI between y and (I,T ). Therefore, our momentum distillation can be considered as performing data augmentation to the original views. The momentum model generates a diverse set of views that are absent in the original image-text pairs, and encourages the base model to learn representations that capture view-invariant semantic information.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We adapt the pre-trained model to five downstream V+L tasks. We introduce each task and our fine-tuning strategy below. Details of the datasets and fine-tuning hyperparameters are in Appendix.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Image-Text Retrieval contains two subtasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). We evaluate ALBEF on the Flickr30K [49] and COCO benchmarks, and fine-tune the pretrained model using the training samples from each dataset. For zero-shot retrieval on Flickr30K, we evaluate with the model fine-tuned on COCO. During fine-tuning, we jointly optimize the ITC loss (Equation 2) and the ITM loss (Equation 4). ITC learns an image-text scoring function based on similarity of unimodal features, whereas ITM models the fine-grained interaction between image and text to predict a matching score. Since the downstream datasets contain multiple texts for each image, we change the ground-truth label of ITC to consider multiple positives in the queue, where each positive has a ground-truth probability of 1/#positives. During inference, we first compute the feature similarity score s itc for all image-text pairs. Then we take the top-k candidates and calculate their ITM score s itm for ranking. Because k can be set to be very small, our inference speed is much faster than methods that require computing the ITM score for all image-text pairs [2,3,8].
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Visual Entailment (SNLI-VE 5 [51]) is a fine-grained visual reasoning task to predict whether the relationship between an image and a text is entailment, neutral, or contradictory. We follow UNITER [2] and consider VE as a three-way classification problem, and predict the class probabilities using a multi-layer perceptron (MLP) on the multimodal encoder's representation of the [CLS] token.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Visual Question Answering (VQA [52]) requires the model to predict an answer given an image and a question. Different from existing methods that formulate VQA as a multi-answer classification problem [53,2], we consider VQA as an answer generation problem, similar to [54]. Specifically, we use a 6-layer transformer decoder to generate the answer. As shown in Figure 3a, the auto-regressive answer decoder receives the multimodal embeddings through cross attention, and a start-of-sequence token ([CLS]) is used as the decoder's initial input token. Likewise, an end-of-sequence token ([SEP]) is appended to the end of decoder outputs which indicates the completion of generation.  Figure 3: The model architecture for VQA and NLVR 2 . For VQA, we append an auto-regressive decoder to generate the answer given the image-question embeddings. For NLVR 2 , we replicate the transformer block within each layer of multimodal encoder to enable reasoning over two images.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
The answer decoder is initialized using the pre-trained weights from the multimodal encoder, and finetuned with a conditional language-modeling loss. For a fair comparison with existing methods, we constrain the decoder to only generate from the 3,192 candidate answers [55] during inference.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Natural Language for Visual Reasoning (NLVR 2 [19]) requires the model to predict whether a text describes a pair of images. We extend our multimodal encoder to enable reasoning over two images. As shown in Figure 3b, each layer of the multimodal encoder is replicated to have two consecutive transformer blocks, where each block contains a self-attention layer, a cross-attention layer, and a feed-forward layer (see Figure 1). The two blocks within each layer are initialized using the same pre-trained weights, and the two cross-attention layers share the same linear projection weights for the keys and values. During training, the two blocks receive two sets of image embeddings for the image pair. We append a MLP classifier on the multimodal encoder's [CLS] representation for prediction.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
For NLVR 2 , we perform an additional pre-training step to prepare the new multimodal encoder for encoding an image-pair. We design a text-assignment (TA) task as follows: given a pair of images and a text, the model needs to assign the text to either the first image, the second image, or none of them. We consider it as a three-way classification problem, and use a FC layer on the [CLS] representation to predict the assignment. We pre-train with TA for only 1 epoch using the 4M images (Section 3.4).
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Visual Grounding aims to localize the region in an image that corresponds to a specific textual description. We study the weakly-supervised setting, where no bounding box annotations are available. We perform experiments on the RefCOCO+ [56] dataset, and fine-tune the model using only imagetext supervision following the same strategy as image-text retrieval. During inference, we extend Grad-CAM [9] to acquire heatmaps, and use them to rank the detected proposals provided by [53].
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
First, we evaluate the effectiveness of the proposed methods (i.e. image-text contrastive learning, contrastive hard negative mining, and momentum distillation). Table 1     all tasks. The proposed hard negative mining improves ITM by finding more informative training samples. Furthermore, adding momentum distillation improves learning for both ITC (row 4), MLM (row 5), and on all downstream tasks (row 6). In the last row, we show that ALBEF can effectively leverage more noisy web data to improve the pre-training performance. Table 2 and Table 3 report results on fine-tuned and zero-shot image-text retrieval, respectively. Our ALBEF achieves state-of-the-art performance, outperforming CLIP [6] and ALIGN [7] which are trained on orders of magnitude larger datasets. Given the considerable amount of improvement of ALBEF when the number of training images increases from 4M to 14M, we hypothesize that it has potential to further grow by training on larger-scale web image-text pairs. Table 4 reports the comparison with existing methods on other V+L understanding tasks. With 4M pre-training images, ALBEF already achieves state-of-the-art performance. With 14M pre-training images, ALBEF substantially outperforms existing methods, including methods that additionally use object tags [3] or adversarial data augmentation [8]. Compared to VILLA [8], ALBEF achieves absolute improvements of 2.37% on VQA test-std, 3.84% on NLVR 2 test-P, and 1.88% on SNLI-VE test. Because ALBEF is detector-free and requires lower resolution images, it also enjoys much faster inference speed compared to most existing methods (>10 times faster than VILLA on NLVR 2 ).  "man with head down" "girl with black tank" "green shirt" "a little girl holding a kitten next to a blue fence" "girl" "holding" "kitten" "next" "blue" visualizations on the self-attention maps in the last layer of the image encoder, where the gradients are acquired by maximizing the image-text similarity s itc . The ALBEF itm variant computes Grad-CAM on the cross-attention maps in the 3rd layer of the multimodal encoder (which is a layer specialized in grounding), where the gradients are acquired by maximizing the image-text matching score s itm . Figure 4 provides a few visualizations. More analysis is in Appendix.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
We provide the Grad-CAM visualizations for VQA in Figure 5. As can be seen in Appendix, the Grad-CAM visualizations from ALBEF are highly correlated with where humans would look when making decisions. In Figure 6, we show per-word visualizations for COCO. Notice how our model not only grounds objects, but also their attributes and relationships.     
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
This paper proposes ALBEF, a new framework for vision-language representation learning. ALBEF first aligns the unimodal image representation and text representation before fusing them with a multimodal encoder. We theoretically and experimentally verify the effectiveness of the proposed image-text contrastive learning and momentum distillation. Compared to existing methods, ALBEF offers better performance and faster inference speed on multiple downstream V+L tasks.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
While our paper shows promising results on vision-language representation learning, additional analysis on the data and the model is necessary before deploying it in practice, because web data may contain unintended private information, unsuitable images, or harmful texts, and only optimizing accuracy may have unwanted social implications.
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Here we describe the implementation details for fine-tuning the pre-trained model. For all downstream tasks, we use the same RandAugment, AdamW optimizer, cosine learning rate decay, weight decay, and distillation weight as during pre-training. All downstream tasks receive input images of resolution "the woman is working on her computer at the desk" "a young boy with an umbrella who is touching the horn of a cow" "horn" "touching" "woman" "working" "computer" "desk" "umbrella" "boy" "an old vehicle painted light blue and dark blue" "old" "vehicle" "light" "dark"
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
"a woman on a motorcycle wearing a bag and passing a car" "on" "wearing" "passing" "car" 
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
In Figure 9, we show more visualizations of per-word Grad-CAM to demonstrate the ability of our model to perform visual grounding of objects, actions, attributes, and relationships.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks (Tan and Bansal, 2019; Chen et al., 2020c;Huang et al., 2020;Li et al., 2020;Wang et al., 2021b;Li et al., 2022a;Kim et al., 2021). Self-supervised learning (SSL) (Jaiswal et al., 2020;Liu et al., 2020) have impressively contributed to vision-language pretraining (VLP) due to its capability of leveraging large-scale image-text pairs without annotations. More recently, Self-supervised Multi-modal Contrastive Learning (SMCL) triggered great progress (Li et al., 2022b;Radford et al., 2021;Yao et al., 2021;Li et al., , 2022a by conducting crossmodal alignment. SMCL consists of image-totext and text-to-image contrastive learning, e.g., * corresponding author.  Figure 1: Conceptual illustration of cross-modal similarity regularization. Sub-figure (a) demonstrates that (partial) false negatives commonly exist in cross-modal training data. In sub-figure (b), negative samples will be equally pushed away from the anchor in conventional cross-modal contrastive learning, leading to data deficiency given these false ones. Instead, we take the first step to contrast negatives according to cross-modal similarity (represented by a set of concentric circles in sub-figure (c)), keeping good while removing harmful effects of (partial) false negative samples.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
with the InfoNCE (Oord et al., 2018) loss. Taking the text-to-image one as an example, given a text-image pair (T, I), I will be treated as the positive sample for the anchor T, and other images in a mini-batch of text-image pairs will be regarded as negatives. The training objective is to attract the positive to the anchor while repelling all the negative samples. However, this contrasting strategy can be problematic given the many-to-many correspondences between images and texts. As shown in Figure 1 (a), a text can be semantically paired with multiple images. In this scenario, though images I 4 and I 5 are treated as negatives, they are actually semantically consistent (or partially consistent) with the text anchor "A bird in the tree." The (partial) false negatives like I 4 and I 5 will inevitably hinder the contrasting effect, yielding sub-optimal crossmodal representations.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Some pioneering efforts have addressed the noisy image-text pairing problem in VLP pretraining datasets Andonian et al., 2022), by feeding the contrastive loss with soft labels in a self-distillation manner. Though these methods can address the problem of false negatives to some extent, the specific harmful effect of false negatives remains far from being systematically studied. For example, based on these methods (e.g., ALBEF (Li et al., 2021) ), we can easily improve the performances of downstream tasks by simply filtering false negatives, as shown in Table 1.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this paper, we investigate the problem of false negatives from the perspective of Mutual Information (MI) optimization. The InfoNCE loss used in contrastive learning has been proved to maximize the lower bound of MI between anchors and their positives (Oord et al., 2018). We revisit the theoretical proof in the presence of non-negligible false negatives. Defining the MI between anchors and positives as MI-P, and the counterpart between anchors and negatives as MI-N, we derive a more general conclusion (see the appendix A.2) that optimizing InfoNCE is equivalent to maximizing the lower bound of (MI-P − MI-N). The finding suggests that MI-N will be minimized (e.g., as close to zero as possible), even though some negatives may semantically match the anchor. The theoretical analyses explain the deficiency of the vanilla contrasting strategy on the one hand, and inspire us with another derivation (appendix A.3) that guarantees proper MI optimization for negative samples on the other hand.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Guided by these theoretical analyses, we propose a novel contrasting strategy regulated by crossmodal similarity. We hypothesize that the MI between an image and text positively correlates with their semantic similarity. Therefore, we introduce a contrastive weight, which is derived based on cross-modal similarity and progressively refined with training, for each negative sample as a contrasting regulator. This regulator will guide the model to optimize MI-N properly, keeping it from being unexpectedly minimized and thus yielding a more semantically structural representation space. We equip our proposed contrasting strategy on AL-BEF    Cross-modal Retrieval, Zero-shot Cross-modal Retrieval, and Natural Language for Visual Reasoning (NLVR). The experimental results show that our adjusted contrastive learning significantly improves their performances.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In summary, our contributions are:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
• We investigate the issue of false negatives in cross-modal contrastive learning from the perspective of Mutual Information (MI) optimization. We deduce a more general form of MI's lower bound for InfoNCE loss in the presence of non-negligible false negatives, revealing that the MI between (partial) false negatives and anchors is improperly minimized.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
• Based on a theoretical derivation that guarantees appropriate MI optimization for negative samples, we propose a novel contrasting strategy by attaching each negative sample with a progressively refined contrastive weight based on cross-modal similarity.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
• Applying the contrasting strategy to VLP methods yields impressive performance improvement on various downstream tasks, and demonstrates our contrasting strategy systematically balances the positive and negative impacts of false negatives.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Mutual Information (MI) is designed to measure the relationship between random variables or determine the amount of shared information (Becker, 1996(Becker, , 1993. Oord et al. (2018) has proven that the InfoNCE loss function widely used in contrastive learning can be seen as a lower bound of MI between anchors and positives. Note that Li et al.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
(2021) provides a conceptual yet more intuitive discussion of the correspondence between InfoNCE and MI in the VLP scenario. In this paper, we go one step further to revisit the proof of Oord et al.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
(2018) under a cross-modal contrastive learning context.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
The standard InfoNCE loss in VLP consists of two parts: L Inf oN CE = L v Inf oN CE +L t Inf oN C , where the former corresponds to image-to-text alignment and the latter corresponds to text-to-image alignment. For the following discussion, we will take L v Inf oN CE as an example. Suppose we randomly sample N semantically paired image-text tuples
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Inf oN CE is defined as:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
(1) where f (v i , t i ) measures the distance between v i and t i in a semantic space. According to Oord et al. (2018), the function f (v i , t i ) can be utilized to model the density ratio, which preserves the mutual information between v i and t i and we can rewrite the f (v i , t i ) to P (t i |v i ) P (t i ) . Then we can derive the well-known lower bound of MI between t i and v i :
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
where the I (t i , v i ) is the mutual information between t i and v i . The details of this copy-to-VLP derivation can be found in appendix A.1.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
The derivation process in Appendix A.1 implicitly assumes that t j (the negative sample) and v i are independent, which is reasonable given a large enough number of negatives with little noise. So the expectation of density ratio P (t j |v i ) P (t j ) is equal to 1 and eliminated (e.g., from Equation 12 to Equation 13). In the presence of non-negligible false negatives, t j and v i may not be independent. Therefore, we revisit this derivation and deduce a more general conclusion (see detailed derivation in appendix A.2):
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Equation 3 provides a more general lower bound form that the InfoNCE loss optimizes. The first term on the left side of this equation is MI between an anchor and the positive, and the second term is MI expectation between an anchor and negatives. Equation 3 reveals that optimizing InfoNCE is equivalent to maximizing the lower bound of the difference between the former and the latter.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Combining Equations 2 and 3, we can find that in addition to maximizing MI between an anchor and the positive (say MI-P), InfoNCE loss will also minimize the MI expectation between an anchor and negatives (say MI-N), e.g., to be as close to zero as possible, despite the existence of the (partial) false negative samples. Since they may semantically match the anchor, over-minimizing MI-N could produce less structural cross-modal representation space.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
To optimize MI-N to a proper value, we first need to provide a prior estimation of MI-N as a target. Here we exploit cross-modal similarity to approximate MI between an image and text. The second problem is integrating this prior estimation into the optimization process. Based on the derivation of Equation 3, we further theoretically prove that assigning a positive weight w i,j to each f (v i , t i ) can push MI expectation between an anchor and negatives to a controllable positive value, given the following two conditions:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
• Condition 1. The covariance between w i,j and P (t i |v i ) P (t i ) is negative. • Condition 2. The expectation of w i,j among all negatives is equal to 1.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
With this theoretical guidance (see complete proof in Appendix A.3), we propose to improve InfoNCE loss by applying each negative with a contrastive weight, which is inversely proportional to its cross-modal similarity with the anchor.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this section, we will first introduce our model architecture, and then introduce our Similarity-  Figure 2: The pipeline of our method. The proposed model consists of two unimodal encoders for images and text separately and a multi-modal encoder for the fusion of the cross-modal information. After feeding the input to the unimodal encoders, we take the representation of [CLS] token as the global representation and use Similarity-Regulated Contrastive Learning (SRCL) to align the unimodal representations of an image-text pair. We also apply an image-text matching loss and a masked-language-modeling loss to learn multimodal interactions between image and text.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Regulated Contrastive Learning (SRCL), followed by the details of other pre-training objectives. Figure 2 shows an overview of our model, our model consists of two unimodal encoders for image and text independently and a multi-modal encoder. To better model the inherent modality bias information, we first use two unimodal encoders to encode the image and text separately. Following (Dou et al., 2021;Shen et al., 2021), we use a visual transformer (Dosovitskiy et al., 2020) directly on the image patches as the visual encoder, which is more computation-friendly than using pretrained object detectors for visual feature extraction (Anderson et al., 2018;Zhang et al., 2021). The visual encoder divides an input image into patches and encodes them as a sequence of embeddings {v cls , v 1 , v 2 , ..., v m } with an additional [CLS] token. The input text is fed to the text encoder and represented as a sequence of embeddings {t cls , t 1 , t 2 , ..., t n }, where t cls is the embedding of the [CLS] token and used to summarize the input text. Then, the visual and linguistic representations are fed into the multi-modal encoder, which consists of multiple transformer layers.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In section 2, we reveal that vanilla InfoNCE loss will treat negative samples equally without consid-ering their semantic similarity with anchors. Thus the MI between the (partial) false negative samples and the anchor is over-reduced, limiting the performance of pre-training models.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We propose a novel contrasting strategy regulated by cross-modal similarity. We hypothesize that the MI between an image and text positively correlates with their semantic similarity. Therefore, we introduce a contrastive weight, which is derived based on cross-modal similarity and progressively refined with training, for each negative sample as a contrasting regulator. This regulator drives the model to optimize MI-N properly rather than simply minimizing it.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Formally, with a batch of N semantically paired image-text tuples
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
of each image and text in the batch, the image-to-text contrastive loss is:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
i,j indicate the contrastive weight of j-th negative text sample in the contrastive framework. Similarly, the contrastive loss from text to image can be 14663 written as follow:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
i,j indicate the contrastive weight of j-th negative image sample in the contrastive framework.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this subsection, we introduce how to calculate the regulation weight of the negative samples in contrastive learning. As the regulation weights are inversely proportional to the semantic similarity between anchors and negatives, we need first to calculate the semantic similarity to estimate the regulation weight. Due to the capacity of the VLP model to align images and texts, the VLP model could be utilized to measure cross-modal semantic similarity. However, we notice that the VLP model in the earlier training stages is unreliable since the semantic structure of the embedding space is still under optimization.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Therefore, in the beginning, we use the highquality human-annotated dataset (Chen et al., 2015) to train another model denoted as H β which shares the same structure with our VLP model S γ . This model H β is optimized by InfoNCE loss and is used to estimate the semantic similarity of the image text pairs at early pre-training stages.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
During the pre-training our VLP model S γ , the parameters of model H β are frozen. The final semantic similarity between anchors and negatives is derived by taking a weighted average of similarity computed from S γ and H β .
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
At the beginning of the pre-training stages, the weight of the VLP model S γ for calculating the final similarity is set to 0, and the weight of H β is set to 1. As the number of training epochs rises, we progressively increase the weights of F γ and decrease the weights of H β .
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Formally, given a mini-batch {(T 1 , I 1 ) , . . . , (T N , I N )} which contains N image-text pairs, for an text anchor T i and a negative image sample I j , the similarityŝ i,j calculated from the H β is:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
wheret i cls is the [CLS] representation of the text T i extracted from the text encoder of H β andv j cls is the [CLS] representation of the Image T i extracted from the image encoder of H β . Similariy, the similarityṡ i,j calculated from the V γ is:ṡ
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Then the finally semantic similarity between T i and V j :
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
where α is a hyper-parameter and will continue to decrease with the increase of pretraining steps. The contrastive weight w t i,j can be driven as follow:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Where δ is a scaling factor. Notably, w t i,j is inversely proportional to the similarity to meet Condition 1 described in Section 2.3, and the N orm function makes the mean value of all negatives' weights to be 1 to meet Condition 2.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Similarly, given an image anchor and its text negative samples, we can also calculate the imageto-text contrastive weight.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We construct our pre-training data using two web datasets (Conceptual Captions (Sharma et al., 2018), SBU Captions (Ordonez et al., 2011)) and two in-domain datasets (MSCOCO (Chen et al., 2015) and Visual Genome (Krishna et al., 2017)). The total number of unique images is 4.0M, and the number of image-text pairs is 5.1M.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We implement SRCL based on ALBEF (Li et al., 2021) framework and evaluate it in four widely used downstream tasks: image-text retrieval, zeroshot image-text retrieval (ZSR), visual question answering (VQA), and natural language for visual reasoning (NLVR).
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We conduct experiments for both image-to-text retrieval (TR) and text-to-image retrieval (IR) on MSCOCO (Chen et al., 2015) 
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
To investigate the semantic structure of the learned representation space, we examine the SRCL on the zero-shot image-text retrieval task on Flickr30K(Plummer et al., 2015). The results are shown in Table 4 where SRCL outperforms AL-BEF, indicating SRCL could yield a better semantic structural representation space. SRCL also achieves better performance than the previous stateof-the-art models (e.g., CLIP, ALIGN, and Florence) pre-trained with more image-text pairs.  (1) shows the zero-shot crossmodal retrieval (ZCR) results on Flickr30K by explicitly removing negatives based on different contrastive weight thresholds. IR means image retrieval, and TR means text retrieval. The horizontal line represents the performances of SRCL. Sub-figure (2) shows the contrastive weights (CW) distribution between anchors and negatives by averaging 10000 mini-batches.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
An astute reader may notice that (partial) negatives will somewhat overlap with hard negatives. It is non-trivial to accurately define hard or false negatives in vision-language contrasting since the cross-modal semantic boundary is blurry. But we do face a paradox here: we want to alleviate the contrastive effect of false negatives that contain a certain number of hard ones, while many works about hard negative mining (HEM) (Hu et al., 2020;Xiong et al., 2020;Kalantidis et al., 2020) try to learn with more hard negative samples.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
To investigate this problem, we experiment with different proportions of false negatives (or hard negatives, approximately). Specifically, we use the contrastive weights, negatively correlated with cross-modal similarity, to roughly approximate whether a negative sample is false. If the weight is lower than a threshold, the corresponding sample is regarded as false, and true otherwise. We explicitly remove the identified false negatives when contrasting, and then check the performance of the pre-trained ALBEF on zero-shot cross-modal retrieval.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
As shown in Figure 3(a), there is a general trend that the performances of downstream tasks initially boost as the threshold increases and then begin to decline when a certain threshold (e.g., 0.2 and 0.3) is reached. We statistic the distribution of contrastive weight by averaging 10000 mini-batches and visualize it in Figure 3(b). We can estimate that with the threshold of 0.7, about 20% negative samples will be discarded. Combining Figure 3 dox: in vanilla contrastive learning, too many false negatives (or hard negatives) could bring harmful impacts, so removing some of them deliver performance improvements; but they are indispensable for a promising contrasting effect, so overly removing them also hinder performances, which is also the reason why hard negative mining methods will increase hard negatives in the absence of them.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
From another perspective, the above explanation validates our method's merits. With the crossmodal similarity regulation, we drive the model to optimize the MI between negatives and their anchor more appropriately rather than simply minimizing it, systematically balancing false negatives' beneficial and harmful effects.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
To better understand the correlation between pretraining data size and downstream performance, we experiment with pretraining data of 4M, 6M, 8M,10M, and 12M. Figure 4 plots the zero-shot cross-modal retrieval and VQA results for SRCL and ALBEF. We can observe that our SRCL continuously maintains higher performance, and the gap becomes more evident with the data size increase. This observation verifies that SRCL promisingly addresses the harmful effect of false negatives and thus enhances data efficiency.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this section, we conduct a qualitative analysis by visualizing the zero-shot text-image retrieval results of ALBEF and our method. We choose this zero-shot task to directly examine the model's representation capacity without fine-tuning impacts. In Figure 5, we find that ALBEF tends to focus more narrowly on one specific commonality while neglecting others. For example, in the second case, ALBEF intensely targets "a woman with blond a security officer with a tiny face and big glasses leans on a metal gate looking into the camera Our Baseline Our Baseline a woman with blond hair is sitting in a booth with a drink working on her laptop Figure 5: An visualization of zero-shot text-to-image retrieval result. We compare the baseline ALBEF and our model in this figure. hair" but misses the critical information "working on her laptop." On the other hand, our approach can successfully extract all the essential aspects of the query. These retrievals suggest that our learned features more comprehensively capture potential similarities between a text caption and the image. Meanwhile, our method's result ranking reflects a trend from full alignment to partial alignment between the retrieved images and the query. These observations clearly verify that our contrasting strategy produces better cross-modal representations for downstream tasks. Note that these two examples are not cherrypicked. The phenomenon in these two examples is commonly observed among other samples. We demonstrate more cases in Appendix 6. Meanwhile, other qualitative analyses can be found in Appendix F.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Recently, self-supervised learning has made significant progress thanks to contrastive learning (Chen et al., 2020a;Oord et al., 2018;He et al., 2019;Chen et al., 2020b;Radford et al., 2021). InfoNCE (Oord et al., 2018) is commonly used in traditional contrasting learning, which optimizes the similarity of positive pairings and minimizes the similarity of negative pairs. In the contrastive learning framework, the negative pairs play a vital role as they prevent shortcuts and collapse solutions. However, Chen et al. (2021) shows the unfavorable effect of false negatives and proposes to incrementally detect and explicitly remove the false negative samples in the contrastive learning framework. Compared with Chen et al. (2021), we propose a more solid method by regulating the false negative samples rather than directly omitting them.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Recent years have seen significant success for largescale pre-trained vision-language models (Tan and Bansal, 2019; Chen et al., 2020c;Huang et al., 2020;Li et al., 2020;Wang et al., 2021b;Li et al., 2022a;Kim et al., 2021) in a variety of cross-modal tasks. Self-supervised Multi-modal Contrastive Learning (SMCL) has lately sparked significant advancements. (Li et al., 2022b;Radford et al., 2021;Yao et al., 2021;Li et al., , 2022a by conducting cross-modal alignment. SMCL consists of image-to-text and text-to-image contrastive learning, e.g., with the InfoNCE (Oord et al., 2018) loss. However, traditional cross-modal contrasting strategy can be problematic given the many-to-many correspondences between images and texts but few works notice this issue. Recently, to solve the issue of noisy image-text pairing in VLP pre-training datasets, some pioneering work has fed the contrastive loss with soft labels in a self-distillation method (Li et al., 2022bAndonian et al., 2022). Even while these techniques may help reduce the number of false negatives, their harmful effect has not been carefully explored.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We have presented our cross-modal contrastive learning method that addresses the problem of (partial) false negatives with vision-language semantic similarity guidance. A series of mathematical proofs based on InfoNCE loss provides a more general lower bound for contrastive optimization and inspires us with a novel contrasting strategy that theoretically guarantees the mitigation of false negatives. Empirically, our method demonstrates performance superiority on four downstream crossmodal tasks. Meanwhile, by comparing false negatives and hard negatives, we reveal that balancing the beneficial and harmful effects of (partial) false negatives is crucial to learn robust cross-modal representations.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We verify our method mainly based on the recent robust VLP model ALBEF . Evaluating it more broadly by incorporating it into other VLP models can further highlight our contribution. Given the solid theoretical foundation of our method, the main conclusion regarding its effectiveness and performance will not be affected, but there can be more inspirational findings in a broader research context. Meanwhile, comparing false negatives and hard negatives is worth further exploration. We leave these problems for future work. 
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We rewrite the proof provided by Oord et al. (2018) in the context of image-to-text contrastive learning, where v i represents an image anchor and t i and t j are positive and negative samples, respectively.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Therefore, we have
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
where N is the number of batch size.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In the presence of non-negligible false negatives, we re-derive the above A.1 derivation as follows:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Note the false negatives account for a relatively small proportion of the overall negatives, so the expectation E t j P (t j |v i ) P (t j ) is less than the density ratio P (t i ) P (t i |v i ) , thus we have:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
therefore we can safely derive the inequality from equation 18 to equation 19. Now we can get :
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
According to Jensen's inequality, we have:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Therefore, we have
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In this section, we prove that assigning a positive weight w i,j to each f (v i , t i ) can push MI expectation between an anchor and negatives to a controllable positive value, under specific conditions. Using image-to-text contrasting as an example, the loss can be written as follow: Following (Oord et al., 2018), the function f (v i , t i ) can be seen as density ratio which preserves the mutual information between v i and t i and could be written as P (t i |v i ) P (t i ) and we can rewrite 14671 the equation 28 as:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Here we set the regulated weigth w i,j inversely proportional to P (t i ) P (t i |v i ) (Condition 1 in Section 2.3), so the covariance between w i,j and P (t i ) P (t i |v i ) is less than 0. Thus, we have:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
we have :
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Combine inequality 21 and 32, we have:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Therefore, we can derive that
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Similar with the inequality 26, we get:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
When optimizing the loss, the last term on the left side of the inequality will be minimized, which means
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Then we can get
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Thus we have:
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
As w i,j is inversely proportional to the semantic similarity between anchor v i and the negative sample t j , the MI expectation v i and t j will be optimized to a controllable positive value negative correlated with the average similarities between v i and t j .
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
B Comparison Methods LXMERT (Tan and Bansal, 2019): is the first twostream region-based VLP model, which consists of an object relationship encoder, a language encoder and a cross-modality encoder. E2E-VLP : proposes the first endto-end VLP method for both V+L understanding and generation, with a unified Transformer encoderdecoder architecture.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
VILT ( UNITER (Chen et al., 2020c): proposes a new word-region alignment pre-training task via the use of optimal transport to help fine-grained alignment between words and image regions.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
ViLBERT (Lu et al., 2019): proposes one of the first work that extend the BERT architecture to a multi-modal two-stream region-based VLP model.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We pre-train our model with three standard objectives: Image-Text Contrastive learning (ITC), Image-Text Matching (ITM) and Masked Language Modeling (MLM). Since we have introduced ITC in the previous subsections, in the following, we will only introduce two other pre-training tasks. Image-Text Matching (ITM) The goal of imagetext matching is to predict whether the input image and text are matched. We follow the design of (Li et al., 2021) and select hard negative image-text pairs based on the contrastive text-image similarity.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We take the text [CLS] embedding of the multimodal encoder's output as the joint representation, followed by a Multi-Layer Perceptron (MLP) layer for prediction. Masked Language Modeling (MLM) The task setup is basically the same as in BERT (Devlin et al., 2018), where we randomly mask 15% of tokens in text and the model is asked to predict these masked words with the cross-modal representations.
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We implement our method based on the ALBEF ( 
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
We evaluate SRCL on the three downstream visionlanguage tasks. The hyperparameters that we use for finetuning on the downstream tasks are listed in  VQA. The VQA task (Antol et al., 2015) requires the model to answer natural language questions given an image. We conduct experiment on the VQA2.0 dataset (Antol et al., 2015), which contains 83k/41k/81k images for training/validation/test. Following (Li et al., 2021), we use both training and validation splits for training, and incorporate additional training data from Visual Genome (Krishna et al., 2017).
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
Image-Text Retrieval. We conduct experiments for both image-to-text retrieval (TR) and textto-image retrieval (IR) on COCO (Chen et al., 2015) and Flickr30K 
Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation
In Figure 7, we plot the distribution of text-toimage contrastive weight in the mini-batch drawn from the Flickr30K testing set. As shown in the Figure 7, for false negative samples, our method can effectively assign them with low contrastive weights. For examples, in the sixth row and fifth column of the first case, for the text anchor "this is a cute cat.", the false negative sample is the sixth image which also contains a cat and the contrastive weight of it is 0.12. Beside, we can observe that most negatives have a high contrastive weight as semantic similarity between them and anchors are low. To further investigate the effectiveness of contrastive weight for regulating the (partial) false negative samples in contrastive learning, we visualize the false negative samples and their contrastive weights. As shown in Figure 8, for the false negative samples, they are all assigned with low contrastive weights (not more than 0.2). This also supply the results of the experiments in subsection 4.3 that masking the negatives whose contrastive weight is less than 0.2 can gets a remarkable improvement.   
FILIP: Fine-grained Interactive Language-Image Pre-Training
Large-scale Vision-Language Pre-training (VLP) models like CLIP  and ALIGN  have recently demonstrated success across various downstream tasks. They learn visual and textual representations from millions of image-text pairs collected from the Internet and show superior zero-shot ability and robustness. The core technique of these models lies in the global contrastive alignment of the images and texts through a dual-stream model. Such architecture is inference-efficient for downstream tasks like retrieval because the encoders for the two modalities can be decoupled and the image or text representations can be pre-computed offline. However, CLIP and ALIGN model the cross-modal interaction via solely the similarity of the global feature of each modality, lacking the ability of capturing finer-level information like the relationship between visual objects and textual words. In this paper, we develop a simple yet efficient cross-modal finer-grained interaction mechanism for large-scale VLP.
FILIP: Fine-grained Interactive Language-Image Pre-Training
To achieve finer-grained cross-modal interaction, previous methods mainly exploited two kinds of methods. (1) One line of work Li et al., 2020b;Li et al., 2021b; uses a pre-trained object detector to extract region-of-interest (ROI) features from images, and then fuses it with the paired text through a VLP model. This design complicates the pre-training due to pre-computing and storing a large number of ROI features. In addition, the zero-shot ability of these approaches is usually limited by the predefined number of classes and their performance is also restricted by the quality of the detector. (2) Another line of work (Li et al., 2021a; enforces the token-wise or patch-wise representations from both modalities into the same space and models these finer-grained interactions via cross-attention (Li et al., 2021a) or self-attention . However, these methods are usually less efficient in terms of both training and inference. In particular, during training, cross-attention in (Li et al., 2021a) requires to be performed in an encoder-decoder structure, while the complexity of the self-attention  grows quadratically with the length of the prolonged concatenated sequences of both modalities. During inference, the data from both modalities are intertwined to compute the cross-attention or self-attention, and can not be pre-computed offline as dual-stream models like CLIP and ALIGN. This can be less efficient for downstream tasks like image/text retrieval and image classification.
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we propose a large-scale Fine-grained Interactive Language-Image Pre-training framework named FILIP to address these limitations. Inspired by Khattab & Zaharia (2020), we model the fine-grained semantic alignment through a novel cross-modal late interaction mechanism in the contrastive loss, instead of using cross or self-attention. Specifically, our fine-grained contrastive learning uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. In this way, FILIP successfully leverages the finer-grained expressiveness among image patches and textual words while simultaneously gaining the ability to pre-compute image and text representations offline. Unlike Khattab & Zaharia (2020), we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment, which enhances the cross-modal representation learning and stabilizes training. Furthermore, we construct a large-scale pre-training dataset named FILIP300M from the Internet. Data cleaning and image-text data augmentation are also explored and proved useful in this work.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Extensive experiments show that by effectively learning fine-grained representations, FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks, including zero-shot image classification and image-text retrieval. For example, FILIP reaches 77.1% top-1 accuracy for zero-shot ImageNet classification, surpassing CLIP with less training data. Visualizations on wordpatch alignment further show that FILIP learns meaningful finer-grained features with promising localization ability.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Vision-Language Pre-training Models. The pre-train-and-fine-tune scheme has achieved great success in the domains of natural language processing (Devlin et al., 2019;Brown et al., 2020) and computer vision (Dosovitskiy et al., 2020). It is then naturally extended to a joint cross-modal domain of Vision-and-Language Pre-training (VLP). The pre-training datasets of recent VLP models include publically available datasets like YFCC100M (Thomee et al., 2016) and CC12M (Changpinyo et al., 2021), as well as larger-scale datasets with more than 100M samples in CLIP  and ALIGN , which are shown to be even more powerful. The pretraining tasks of VLP models can be categorized into two categories: image-text contrastive learning task and Language Modeling (LM) based tasks: (i) CLIP , ALIGN  and UNIMO (Li et al., 2021b) make use of cross-modal contrastive learning which aligns the textual and visual information into a unified semantic space; (ii) VisualBERT (Li et al., 2019), UNITER , M6 , and DALL-E  employ LM-like objectives, including both masked LM (e.g., Masked Language/Region Modeling), and autoregressive LM (e.g., image captioning, text-grounded image generation). On the other hand, some methods rely on a pre-trained object detection model such as Faster- RCNN (Ren et al., 2015) to extract image regional features offline, which requires extra labeled bounding-box data and makes the approach less scalable. Recent efforts such as SOHO (Huang et al., 2021) and SimVLM  try to eliminate this burden via visual dictionary or PrefixLM (Raffel et al., 2020). In this paper, we directly learn fine-grained vision-language representations in an end-to-end and simpler manner while maintaining the benefit of inference efficiency.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Multi-Modality Interaction Mechanism. The core of vision-language pre-training models lies in modeling the interaction between the two modalities. There are mainly two types of cross-modal interaction architectures: single-stream and dual-stream models. Single-stream models like Visual-BERT (Li et al., 2019) and ViLT  directly concatenate the patch-wise or regional visual features and textual embeddings and feed them to the transformer-based model. Dual-stream 
FILIP: Fine-grained Interactive Language-Image Pre-Training
Text-to-image Contrastive A novel fine-grained contrastive learning equipped with cross-modal late interaction is proposed, which uses a token-wise maximum similarity between visual and textual tokens. models such as ViLBERT (Lu et al., 2019) and CLIP  have separate encoders for different modalities. This allows flexible use of different models for different modalities, and efficient inference for downstream tasks like image-text retrieval, through the ability of decoupling the encoders and pre-compute image/text features offline. In this paper, while following the dual-stream approach for its flexible and efficient inference, we further propose a new multi-modal interaction mechanism to capture the fine-grained representations.
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this paper, we propose a new cross-modal pre-training model that excels in fine-grained interaction between image encoder and text encoder for mining more detailed semantic alignment, named as FILIP, as shown in Figure 1 (Radford et al., 2019). On top of the image and text encoders, the representations of textual tokens and visual tokens are linearly projected to the multi-modal common space, and are separately L2-normalized. Different from existing dual-stream models (e.g., CLIP and ALIGN) which models cross-modal interaction via only the global features of the entire image and text sequence, we introduce a novel fine-grained contrastive learning objective equipped with cross-modal late interaction which takes into account the fine-grained interaction between image patches and textual tokens, detailed in Section 3.1.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Contrastive representation learning has recently been found to learn better representations than its predictive counterpart in both visual (Tian et al., 2020) and vision-language cross-modal pre-training . Under a general formulation of cross-modal contrastive learning , we want to learn encoders f θ for image data I and g φ for text data T such that, given an image x I ∈ I, and a text x T ∈ T , the encoded representations f θ (x I ) and g φ (x T ) are close if they are related and far apart if not, under a distance metric. In each training batch, we sample b
FILIP: Fine-grained Interactive Language-Image Pre-Training
x T k is its positive, while the other texts will be used as in-batch negatives. The image-to-text contrastive loss L I k for x I k can then be formulated as
FILIP: Fine-grained Interactive Language-Image Pre-Training
where s I k,j denotes the similarity of the k-th image to the j-th text. Similarly, the text-to-image contrastive loss for x T k is
FILIP: Fine-grained Interactive Language-Image Pre-Training
.
FILIP: Fine-grained Interactive Language-Image Pre-Training
The total loss of this mini-batch can be represented by
FILIP: Fine-grained Interactive Language-Image Pre-Training
From the contrastive loss (1), the cross-modal interaction is reflected in how we compute the similarities s I i,j and s T i,j for the i-th image and j-th text. Previous methods like CLIP  and ALIGN  simply encode each image or text separately to a global feature i.e., f θ (x I i ) ∈ R d and g φ (x T j ) ∈ R d , and compute these two similarities as
FILIP: Fine-grained Interactive Language-Image Pre-Training
neglecting finer-grained interactions (e.g., word-patch alignment) between the two modalities. To alleviate this problem, while simultaneously maintain the training and inference efficiency of dualstream models, we apply a cross-modal late interaction inspired by Khattab & Zaharia (2020) to model the token-wise cross-modal interaction.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Specifically, denote n 1 and n 2 as the number of (non-padded) tokens of the i-th image and j-th text, respectively, and the corresponding encoded features are f θ (x I i ) ∈ R n1×d and g φ (x T j ) ∈ R n2×d . For the k-th visual token, we compute its similarities with all textual tokens of x T j , and use the largest one max
FILIP: Fine-grained Interactive Language-Image Pre-Training
as its token-wise maximum similarity with x T j . We then use the average token-wise maximum similarity of all non-padded tokens in the image (resp. text) as the similarity of an image to a text (resp. a text to an image). The similarity of the i-th image to the j-th text can thus be formulated as:
FILIP: Fine-grained Interactive Language-Image Pre-Training
Similarly, the similarity of the j-th text to the i-th image is (5).
FILIP: Fine-grained Interactive Language-Image Pre-Training
Remark 1 Intuitively, the token-wise maximum similarity in Equation (3) means that for each image patch, we find its most similar textual token. Similarly, for each textual token, we also find its closest image patch. By applying this to the similarity calculation in (4) and (5) for contrastive loss (1), the dual-stream model learns fine-grained alignment between image patches and textual tokens.
FILIP: Fine-grained Interactive Language-Image Pre-Training
The original late interaction mechanism in (Khattab & Zaharia, 2020) computes the relevance score of a document to a query padded with mask tokens, as a sum of token-wise maximum similarities, and is optimized via a pairwise softmax cross-entropy loss. Though inspired from Khattab & Zaharia (2020), our proposed cross-modal late interaction differs in several aspects. Firstly, we exclude the padded textual tokens when computing the similarity, as they harm the performance. We speculate that this is because these padded tokens also learn textual representations and will mislead the model to align image patches to these meaningless padded tokens rather than meaningful non-padded words. Secondly, when computing similarities (4) and (5), we use the average of the token-wise maximum similarities instead of summation in (Khattab & Zaharia, 2020). This is because the number of non-padded tokens varies from text to text, and this summation over all non-padded tokens can have quite different magnitudes, leading to less stabilized training and worse final performance. Thirdly, we optimize the late interaction mechanism via a contrastive loss (1) which is found powerful vision-language pre-training  instead of the original pairwise loss in (Khattab & Zaharia, 2020).
FILIP: Fine-grained Interactive Language-Image Pre-Training
Training Efficiency. Though the cross-modal late interaction is able to capture finer-grained features compared with the original loss, it relies on the token-wise representations of both modalities, and can be inefficient in terms of communication, memory and computation, especially when the batch size is large. To alleviate this problem, we utilize several methods. Firstly, we reduce the embedding size to 256. Besides, we reduce the precision of the last-layer features of both modalities from fp32 to fp16 before node communication in a distributed learning setting, and perform the multiplication in Equations (4) and (5) under the reduced precision. In addition, since the complexity of similarity calculation scales with the sequence length of textual tokens and image patches, for each image (resp. text), we select the 25% tokens with the highest token-wise maximum similarity score (Equation (3)) among all texts (resp. images) in the same local worker before node communication, based on the intuition that each sample can be represented by a few of the most representative tokens. Effects of these modifications are studied in Section 4.4.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Due to the problem of polysemy and inconsistency with the pre-training process, following , we also use prompt templates to augment the original label for some downstream tasks. For visualizations, for simplicity, we use only one prompt template across the paper, i.e. "a photo of a {label}." as . For other experiments, we report results using prompt ensemble following . When multiple prompts are allowed, the token-wise representations of different prompt templates for the same class label are different, and can not be summed together to form a mean textual representation as in . Thus, instead of ensembling different prompt templates by their mean textual representation, we ensemble them by their mean token-wise similarity. Specifically, suppose there are C prompt templates, each label is augmented to C different texts x T 1 , x T 2 , · · · , x T C . The similarity between an image x I and this label is computed as 1 C C c=1 s I ·,· (x I , x T c ), where s I ·,· is defined in Equation (4). We use a unified rule-based method inspired by Radford et al. (2018) to construct prompt templates for image classification tasks. Specifically, each template consists of four components:
FILIP: Fine-grained Interactive Language-Image Pre-Training
(6) Here, the "[prefix]" is an in-context description like "a photo of a" similar as ; "label" is a class label of the dataset; "[category description]" describes the category which is found helpful for some fine-grained image classification datasets , e.g., " a type of pet" for dataset Oxford-IIIT Pets. An interesting finding is that, adding a suffix that includes the reference word "it" (e.g., "I like it.") at the end of the prompt empirically improves the zero-shot classification performance of the proposed model. We speculate this is because the reference word "it" strengthens the fine-grained cross-modal alignment, as it can also be aligned to image patches of the target object. Detailed prompt templates for different datasets can be found in Appendix A.4.
FILIP: Fine-grained Interactive Language-Image Pre-Training
To obtain better generalization and data-efficiency of the model, we perform data augmentation on both images and texts during the pre-training phase to construct more image-text pairs. We apply AutoAugment (Krizhevsky et al., 2012;Sato et al., 2015;Cubuk et al., 2019;Hoffer et al., 2020)  for image augmentation, following the SOTA vision recognition methods (Touvron et al., 2021;Xie et al., 2020b). To ensure the augmented texts are semantically similar as the original one, for text augmentation, we rewrite the original text using back-translation (Xie et al., 2020a;Sennrich et al., 2016a). Specifically, the texts are first translated to the target language and then translated back to the source language. We choose German and Russian as the target language and get extra two texts for each image-text pair. When constructing a batch of image-text pairs during the pre-training, the text of each image-text pair is randomly sampled from the three candidate texts, i.e., the original text and two back-translated texts.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Model Architectures. We train two models from scratch, i.e., FILIP base and FILIP large . The model architectures follow CLIP , i.e., the image encoder is ViT-B/32 for FILIP base and ViT-L/14 for FILIP large . More details can be found in Appendix A.2.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Pre-training Details. To save memory and scale up the batch size, automatic mixed-precision (Micikevicius et al., 2018) and gradient checkpoint (Griewank & Walther, 2000;Chen et al., 2016) are used The input images are resized to 224 × 224 resolution during pre-training and the maximum length of the text is limited to 77 tokens following . The training is mainly conducted on Nvidia V100 GPUs and Ascend Cards. FILIP base is trained on 128 cards about 9 days and FILIP large takes about 24 days to train on 192 cards. Unless otherwise specified, we use FILIP large to compare with other methods and FILIP base for ablation. We train both models using the LAMB optimizer (You et al., 2020) and cosine learning rate schedule (Loshchilov & Hutter, 2016) with a linear warmup. Weight decay regularization is applied to all parameters except bias, layer normalization, token embedding, positional embedding and temperature in contrastive loss. Detailed values of hyperparameters for different datasets and models can be found in Appendix A.2. 
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this section, we evaluate our proposed FILIP on the zero-shot image classification task. We compare our FILIP with CLIP (Radford et al., 2021) on 12 downstream classification datasets, using the same evaluation setting as in CLIP. As described in Section 3.1.2, we apply a set of prompts for each dataset and ensemble them to get the final results, see Appendix A.4 for details. We only compare the zero-shot performance with CLIP here as ALIGN does not release its model and the related performances are not reported in their paper. Table 1 shows the results on 12 datasets. Despite using less training data (340M vs. 400M), both FILIP base and FILIP large considerably outperform their CLIP counterparts in terms of average top-1 accuracy over 12 datasets, i.e., achieving absolute improvements of 5.6% and 3.0%, respectively. In particular, our FILIP surpasses CLIP on ImageNet, the largest dataset among 12 datasets. FILIP also achieves substantial performance gains on some domain-specific datasets, e.g., for Aircrafts, the two FILIP models reach a 30% improvement over CLIP on average. We speculate this is because, unlike CLIP which aggregates the information of the whole image into the representation of the [CLS] token, our proposed FILIP model focuses more on the target object by directly aligning the image patches corresponding to the target object with the textual tokens corresponding to the class label (visualizations of word-patch alignment are in Section 4.5).
FILIP: Fine-grained Interactive Language-Image Pre-Training
Image-text retrieval consists of two sub-tasks: image-to-text retrieval and text-to-image retrieval.
FILIP: Fine-grained Interactive Language-Image Pre-Training
We evaluate our FILIP model on two retrieval benchmark datasets: Flickr30K (Plummer et al., 2015) and MSCOCO (Lin et al., 2014), under both zero-shot and fine-tuned settings. More details of experimental setting can be found in Appendix A.2.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Tables 2 and 3 show the results of zero-shot and fine-tuned image-text retrieval, respectively. We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER , VILLA , ERNIE-ViL , Oscar (Li et al., 2020b), VinVL , ALBEF (Li et al., 2021a), and methods trained on larger-scale image-text datasets including CLIP  and ALIGN . As we can see, FILIP achieves state-of-the-art performances under all metrics on both Flickr30K and MSCOCO datasets, except for zero-shot text-to-image retrieval on Flickr30K, where FILIP achieves competitive performance with SOTA. For zero-shot image-to-text retrieval on MSCOCO dataset, the absolute R@1 of our proposed FILIP is 2.7% higher than ALIGN, which is trained on a much larger dataset.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Effectiveness of Each Component. We study the effectiveness of each component in FILIP, i.e., image/text augmentations and cross-modal late interaction. Experiments are conducted on FILIP base , with a filtered subset of YFCC100M as the training dataset (as described in Section 3.3), on both zero-shot retrieval and classification tasks. We measure models' performance on MSCOCO zero-   Table 5: Efficiency study of the cross-modal late interaction. "orig" and "late" stand for the contrastive loss based on the original cosine similarity in CLIP and our proposed cross-modal late interaction, respectively. "ZS" means zero-shot performance. We report results for ViT-B/32 trained on filtered YFCC100M with 8 V100 GPUs, with a batch size of 512 per GPU. Training time and memory consumption are tested using the same gradient checkpoint configuration. * denotes our final setting used in other experiments. shot image-text retrieval and ImageNet zero-shot classification, which are two effective indicators for the quality of the learned vision-language representations. Table 4 reports the results. As can be seen, all three components are beneficial for both tasks. Despite the simple design, cross-modal late interaction brings significant performance improvements over the baseline (the vanilla CLIP ViT-B/32), with an absolute R@1 gain of 5.5% (resp. 3.8%) for image-to-text (resp. text-to-image) retrieval on MSCOCO and an absolute top-1 accuracy gain of 3.9% for zero-shot classification on ImageNet. Further improvements are observed when all components are combined together.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Efficiency Study of Cross-modal Late Interaction. Since the late interaction mechanism in Section 3.1.1 requires to calculate the similarity between all visual and textual tokens, its efficiency can be a problem when employed in large-scale distributed training. As described in Section 3.1.1, we make several attempts to address the issue. Table 5 shows the efficiency improvement on zero-shot classification on ImageNet when these attempts are applied. As can be seen, these attempts improve  the efficiency of late interaction without accuracy drop. Combining all three attempts achieves only slightly slower training and larger memory consumption than the original loss in CLIP.
FILIP: Fine-grained Interactive Language-Image Pre-Training
In this section, we visualize FILIP's capability of capturing fine-grained cross-modal correspondence using the method of word-patch alignment. To make a fair comparison, we use our FILIP base trained on YFCC100M and CLIP's ViT-B/32, which are of the same size, for visualization. Each image is patchified to 7×7 image patches. More visualization results can be found in Appendix A.3.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Visualization Method. The word-patch alignment is performed based on the token-wise similarity between the image patches and textual tokens. Specifically, for the k-th image patch, the location index of textual token with the largest similarity with it (m I k in Equation (4)) is considered as its predicted label, and is placed at the center of it. Take class "balloon" as an example. There are 8 tokens in the tokenized textual sequence "[BOS] a photo of a balloon. [EOS]", and the location index of the class label "balloon" is "5". Note that one class label may be tokenized to more than one token. Location indices of textual tokens corresponding to the class label are highlighted in red, while the others are marked in white. A desired model that learns fine-grained representations would predict image patches of the target object to red indices.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Observations. Figure 2 shows the word-patch alignment results for FILIP and CLIP on 4 classes from the ImageNet dataset. As can be seen, FILIP exhibits the finer-grained understanding of an image in the following aspects. (i) A single object: From the visualization of class "small white butterfly", the image patches covering the object are all classified correctly; (ii) Same object in different shapes: From the visualizations of class "balloon" and "lifeboat", image patches corresponding to all target objects with different shapes and locations are correctly classified; (iii) Key Components of an object: For class "electric locomotive", there are two key components crucial to correctly classifying the image, i.e., "electric" and "locomotive", whose corresponding textual token indices are "5" and "6", respectively. As can be seen, image patches matching these two key components are respectively correctly classified. On the other hand, CLIP can not correctly align image patches with corresponding textual tokens. Compared with  which uses an extra optimal transport to align the textual word and image patch distributions, the word-patch alignment can be simply automatically learned by our method.
FILIP: Fine-grained Interactive Language-Image Pre-Training
This paper introduces FILIP, a simple yet generic framework towards fine-grained vision-language pre-training. By using a token-wise maximum similarity, our method learns fine-grained represen-tation for patches in the images and words in the sentences. While it achieves competitive results against several large-scale multi-modal pre-training on various downstream tasks, both its architecture and training procedure can still be optimized to improve its performance. In the future, a more advanced image encoder as well as a well-designed interaction layer can be used to boost the performance. Furthermore, we can further add more masked language/image loss to support more generation tasks. To this end, we hope to extend FILIP as a generic and unified interface for solving a large variety of vision-language tasks.
FILIP: Fine-grained Interactive Language-Image Pre-Training
A.1 DATASETS SUMMARY Table 6 shows the number of image-text pairs of each datasets used in different pre-training methods.  Model Architectures. We follow the same architecture design as CLIP, for both FILIP base and FILIP large , except that we reduce the embedding dimension from 512/768 to 256 for the efficiency of loss computation. Table 7 describes the details of architectures.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Details for Pre-training and Hyperparameters. For the implementation of the contrastive loss, following CLIP  and ALIGN , we also set the temperature in the softmax function to be a learnable parameter and initialize it as 0.07. For the pre-training, we use the LAMB optimizer implemented by the cybertronai's open-source repository (https: //github.com/cybertronai/pytorch-lamb). For the learning rate scheduler, we first assign a base learning rate and then linearly warm it up to the peak learning rate according to the effective total batch size by a square root strategy, peak lr = base lr × total bs 512 . We note that a large weight decay is crucial to stabilize training and improve generalization. Specifically, we found that the training stability is a challenging issue when applying mix-precision training to large-scale models, i.e., the training is extremely unstable and the NaN loss easily happens. Recent works DALL-E  and Cogview  also notice this issue and provide their solutions. However, we found that simply increasing the weight decay and applying the trick of removing the weight decay of specific parameters as described in Section 4.1 work for our case. The base learning rate and weight decay are selected manually via observing the performance at the early training stage. Table 8 summarizes the common hyperparameters and Table 9 shows the model-and dataset-specific hyperparameters for FILIP pre-training.  Details for Image-text Retrieval. Following previous works Li et al., 2021a), for Flickr30K, we test on the 1K test set with or without fine-tuning on the 30K training set, while for MSCOCO, we test on the 5K test set with or without fine-tuning on the 113K training set. We use the similarity between image and text for ranking and use the contrastive loss for fine-tuning.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Since there are multiple texts for each image in these two datasets, we change the ground-truth label of contrastive loss to consider multiple positives, by assigning a probability of 1/#positive to each positive following ALBEF (Li et al., 2021a). Besides, we also use prompts during evaluation for both datasets, see Appendix A.4 for details. Table 10 shows the hyperparameters for image-text retrieval fine-tuning. 
FILIP: Fine-grained Interactive Language-Image Pre-Training
In Figure 3, we visualize the cross-modal alignment of the proposed method for more images, in terms of both word-patch alignment as described in Section 4.5 and Grad-CAM heatmaps (Selvaraju et al., 2017). We compute the Grad-CAM heatmaps based on the average self-attention maps over the image patches classified to targeted textual tokens (i.e., the textual token(s) corresponding to the class label in the ImageNet dataset) in the last layer of the image encoder. We average the heatmaps over all attention heads. As can be seen, our proposed model learns meaningful alignment between image patches and textual tokens.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Image Classification.  (6). There are three components to be determined in the template, i.e., the prefix, the category description and the suffix. For each component, we select several well-performed ones for each dataset. Then we use the full combinations of all three components as the set of prompt templates for ensemble. For instance, we use 5 prefixes, no category descriptions, and 6 suffixes for dataset ImageNet. Then the total number of prompt templates for this dataset is: 5 × 1 × 6 = 30.
FILIP: Fine-grained Interactive Language-Image Pre-Training
Image-text Retrieval. Following CLIP , we use prompt in zero-shot imagetext retrieval for both Flickr30K and MSCOCO datasets. The prompt is selected by the same rule as described in Section 3.1.2, except that we do not use "[category description]" here. Table 12 shows the prompt templates for zero-shot image-text retrieval on Flickr30K and MSCOCO datasets.    Flickr30K image-to-text retrieval "a good photo of the" "I hate it." text-to-image retrieval "a good photo of" None MSCOCO image-to-text retrieval "a good photo of" "It is ugly." text-to-image retrieval None None
Learning Transferable Visual Models From Natural Language Supervision
Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015;Peters et al., 2018;Howard & Ruder, 2018;Radford et al., 2018;Devlin et al., 2018;Raffel et al., 2019). * Equal contribution 1 OpenAI, San Francisco, CA 94110, USA. Correspondence to: <{alec, jongwook}@openai.com>.
Learning Transferable Visual Models From Natural Language Supervision
Task-agnostic objectives such as autoregressive and masked language modeling have scaled across many orders of magnitude in compute, model capacity, and data, steadily improving capabilities. The development of "text-to-text" as a standardized input-output interface (McCann et al., 2018;Radford et al., 2019;Raffel et al., 2019) has enabled taskagnostic architectures to zero-shot transfer to downstream datasets removing the need for specialized output heads or dataset specific customization. Flagship systems like GPT-3 (Brown et al., 2020) are now competitive across many tasks with bespoke models while requiring little to no dataset specific training data.
Learning Transferable Visual Models From Natural Language Supervision
These results suggest that the aggregate supervision accessible to modern pre-training methods within web-scale collections of text surpasses that of high-quality crowd-labeled NLP datasets. However, in other fields such as computer vision it is still standard practice to pre-train models on crowd-labeled datasets such as ImageNet (Deng et al., 2009). Could scalable pre-training methods which learn directly from web text result in a similar breakthrough in computer vision? Prior work is encouraging.
Learning Transferable Visual Models From Natural Language Supervision
Over 20 years ago Mori et al. (1999) explored improving content based image retrieval by training a model to predict the nouns and adjectives in text documents paired with images. Quattoni et al. (2007) demonstrated it was possible to learn more data efficient image representations via manifold learning in the weight space of classifiers trained to predict words in captions associated with images. Srivastava & Salakhutdinov (2012) explored deep representation learning by training multimodal Deep Boltzmann Machines on top of low-level image and text tag features. Joulin et al. (2016) modernized this line of work and demonstrated that CNNs trained to predict words in image captions learn useful image representations. They converted the title, description, and hashtag metadata of images in the YFCC100M dataset (Thomee et al., 2016) into a bag-ofwords multi-label classification task and showed that pretraining AlexNet (Krizhevsky et al., 2012) to predict these labels learned representations which preformed similarly to ImageNet-based pre-training on transfer tasks. Li et al. (2017) then extended this approach to predicting phrase ngrams in addition to individual words and demonstrated the ability of their system to zero-shot transfer to other image I 1 ·T 2 I 1 ·T 3 …  classification datasets by scoring target classes based on their dictionary of learned visual n-grams and predicting the one with the highest score. Adopting more recent architectures and pre-training approaches, VirTex (Desai & Johnson, 2020), ICMLM (Bulent Sariyildiz et al., 2020), and Con-VIRT  have recently demonstrated the potential of transformer-based language modeling, masked language modeling, and contrastive objectives to learn image representations from text.
Learning Transferable Visual Models From Natural Language Supervision
While exciting as proofs of concept, using natural language supervision for image representation learning is still rare. This is likely because demonstrated performance on common benchmarks is much lower than alternative approaches. For example, Li et al. (2017) reach only 11.5% accuracy on ImageNet in a zero-shot setting. This is well below the 88.4% accuracy of the current state of the art (Xie et al., 2020). It is even below the 50% accuracy of classic computer vision approaches (Deng et al., 2012). Instead, more narrowly scoped but well-targeted uses of weak supervision have improved performance. Mahajan et al. (2018) showed that predicting ImageNet-related hashtags on Instagram images is an effective pre-training task. When fine-tuned to ImageNet these pre-trained models increased accuracy by over 5% and improved the overall state of the art at the time. Kolesnikov et al. (2019) and Dosovitskiy et al. (2020) have also demonstrated large gains on a broader set of transfer benchmarks by pre-training models to predict the classes of the noisily labeled JFT-300M dataset.
Learning Transferable Visual Models From Natural Language Supervision
This line of work represents the current pragmatic middle ground between learning from a limited amount of supervised "gold-labels" and learning from practically unlimited amounts of raw text. However, it is not without compro-mises. Both works carefully design, and in the process limit, their supervision to 1000 and 18291 classes respectively. Natural language is able to express, and therefore supervise, a much wider set of visual concepts through its generality. Both approaches also use static softmax classifiers to perform prediction and lack a mechanism for dynamic outputs. This severely curtails their flexibility and limits their "zero-shot" capabilities.
Learning Transferable Visual Models From Natural Language Supervision
A crucial difference between these weakly supervised models and recent explorations of learning image representations directly from natural language is scale. While Mahajan et al. (2018) and Kolesnikov et al. (2019) trained their models for accelerator years on millions to billions of images, VirTex, ICMLM, and ConVIRT trained for accelerator days on one to two hundred thousand images. In this work, we close this gap and study the behaviors of image classifiers trained with natural language supervision at large scale. Enabled by the large amounts of publicly available data of this form on the internet, we create a new dataset of 400 million (image, text) pairs and demonstrate that a simplified version of ConVIRT trained from scratch, which we call CLIP, for Contrastive Language-Image Pre-training, is an efficient method of learning from natural language supervision. We study the scalability of CLIP by training a series of eight models spanning almost 2 orders of magnitude of compute and observe that transfer performance is a smoothly predictable function of compute (Hestness et al., 2017;Kaplan et al., 2020). We find that CLIP, similar to the GPT family, learns to perform a wide set of tasks during pre-training including OCR, geo-localization, action recognition, and many others. We measure this by benchmarking the zero-shot transfer performance of CLIP on over 30 existing datasets and find . CLIP is much more efficient at zero-shot transfer than our image caption baseline. Although highly expressive, we found that transformer-based language models are relatively weak at zero-shot ImageNet classification. Here, we see that it learns 3x slower than a baseline which predicts a bag-of-words (BoW) encoding of the text (Joulin et al., 2016). Swapping the prediction objective for the contrastive objective of CLIP further improves efficiency another 4x.
Learning Transferable Visual Models From Natural Language Supervision
it can be competitive with prior task-specific supervised models. We also confirm these findings with linear-probe representation learning analysis and show that CLIP outperforms the best publicly available ImageNet model while also being more computationally efficient. We additionally find that zero-shot CLIP models are much more robust than equivalent accuracy supervised ImageNet models which suggests that zero-shot evaluation of task-agnostic models is much more representative of a model's capability. These results have significant policy and ethical implications, which we consider in Section 7.
Learning Transferable Visual Models From Natural Language Supervision
At the core of our approach is the idea of learning perception from supervision contained in natural language. As discussed in the introduction, this is not at all a new idea, however terminology used to describe work in this space is varied, even seemingly contradictory, and stated motivations are diverse. Zhang et al. (2020), Gomez et al. (2017), Joulin et al. (2016), and Desai & Johnson (2020) all introduce methods which learn visual representations from text paired with images but describe their approaches as unsupervised, self-supervised, weakly supervised, and supervised respectively.
Learning Transferable Visual Models From Natural Language Supervision
We emphasize that what is common across this line of work is not any of the details of the particular methods used but the appreciation of natural language as a training signal. All these approaches are learning from natural language super-vision. Although early work wrestled with the complexity of natural language when using topic model and n-gram representations, improvements in deep contextual representation learning suggest we now have the tools to effectively leverage this abundant source of supervision (McCann et al., 2017).
Learning Transferable Visual Models From Natural Language Supervision
Learning from natural language has several potential strengths over other training methods. It's much easier to scale natural language supervision compared to standard crowd-sourced labeling for image classification since it does not require annotations to be in a classic "machine learning compatible format" such as the canonical 1-of-N majority vote "gold label". Instead, methods which work on natural language can learn passively from the supervision contained in the vast amount of text on the internet. Learning from natural language also has an important advantage over most unsupervised or self-supervised learning approaches in that it doesn't "just" learn a representation but also connects that representation to language which enables flexible zero-shot transfer. In the following subsections, we detail the specific approach we settled on.
Learning Transferable Visual Models From Natural Language Supervision
Existing work has mainly used three datasets, MS-COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), and YFCC100M (Thomee et al., 2016). While MS-COCO and Visual Genome are high quality crowd-labeled datasets, they are small by modern standards with approximately 100,000 training photos each. By comparison, other computer vision systems are trained on up to 3.5 billion Instagram photos (Mahajan et al., 2018). YFCC100M, at 100 million photos, is a possible alternative, but the metadata for each image is sparse and of varying quality. Many images use automatically generated filenames like 20160716 113957.JPG as "titles" or contain "descriptions" of camera exposure settings. After filtering to keep only images with natural language titles and/or descriptions in English, the dataset shrunk by a factor of 6 to only 15 million photos. This is approximately the same size as ImageNet.
Learning Transferable Visual Models From Natural Language Supervision
A major motivation for natural language supervision is the large quantities of data of this form available publicly on the internet. Since existing datasets do not adequately reflect this possibility, considering results only on them would underestimate the potential of this line of research. To address this, we constructed a new dataset of 400 million (image, text) pairs collected form a variety of publicly available sources on the Internet. To attempt to cover as broad a set of visual concepts as possible, we search for (image, text) pairs as part of the construction process whose text includes one of a set of 500,000 queries. 1 We approximately class balance the results by including up to 20,000 (image, text) pairs per query. The resulting dataset has a similar total word count as the WebText dataset used to train GPT-2. We refer to this dataset as WIT for WebImageText.
Learning Transferable Visual Models From Natural Language Supervision
State-of-the-art computer vision systems use very large amounts of compute. Mahajan et al. (2018) required 19 GPU years to train their ResNeXt101-32x48d and Xie et al. (2020) required 33 TPUv3 core-years to train their Noisy Student EfficientNet-L2. When considering that both these systems were trained to predict only 1000 ImageNet classes, the task of learning an open set of visual concepts from natural language seems daunting. In the course of our efforts, we found training efficiency was key to successfully scaling natural language supervision and we selected our final pre-training method based on this metric.
Learning Transferable Visual Models From Natural Language Supervision
Our initial approach, similar to VirTex, jointly trained an image CNN and text transformer from scratch to predict the caption of an image. However, we encountered difficulties efficiently scaling this method. In Figure 2 we show that a 63 million parameter transformer language model, which already uses twice the compute of its ResNet-50 image encoder, learns to recognize ImageNet classes three times slower than a much simpler baseline that predicts a bag-ofwords encoding of the same text.
Learning Transferable Visual Models From Natural Language Supervision
Both these approaches share a key similarity. They try to predict the exact words of the text accompanying each image. This is a difficult task due to the wide variety of descriptions, comments, and related text that co-occur with images. Recent work in contrastive representation learning for images has found that contrastive objectives can learn better representations than their equivalent predictive objective (Tian et al., 2019). Other work has found that although generative models of images can learn high quality image representations, they require over an order of magnitude more compute than contrastive models with the same performance (Chen et al., 2020a). Noting these findings, we explored training a system to solve the potentially easier proxy task of predicting only which text as a whole is paired with which image and not the exact words of that text. Starting with the same bag-of-words encoding baseline, we swapped the predictive objective for a contrastive objective in Figure 2 and observed a further 4x efficiency improvement in the rate of zero-shot transfer to ImageNet. Given a batch of N (image, text) pairs, CLIP is trained to predict which of the N × N possible (image, text) pairings across a batch actually occurred. To do this, CLIP learns a with high pointwise mutual information as well as the names of all Wikipedia articles above a certain search volume. Finally all WordNet synsets not already in the query list are added. multi-modal embedding space by jointly training an image encoder and text encoder to maximize the cosine similarity of the image and text embeddings of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the N 2 − N incorrect pairings. We optimize a symmetric cross entropy loss over these similarity scores. In Figure 3 we include pseudocode of the core of an implementation of CLIP. To our knowledge this batch construction technique and objective was first introduced in the area of deep metric learning as the multi-class N-pair loss Sohn (2016), was popularized for contrastive representation learning by Oord et al. (2018) as the InfoNCE loss, and was recently adapted for contrastive (text, image) representation learning in the domain of medical imaging by Zhang et al. (2020).
Learning Transferable Visual Models From Natural Language Supervision
Due to the large size of our pre-training dataset, over-fitting is not a major concern and the details of training CLIP are simplified compared to the implementation of Zhang et al. (2020). We train CLIP from scratch without initializing the image encoder with ImageNet weights or the text encoder with pre-trained weights. We do not use the non-linear projection between the representation and the contrastive embedding space, a change which was introduced by Bachman et al. (2019) and popularized by Chen et al. (2020b). We instead use only a linear projection to map from each encoder's representation to the multi-modal embedding space. We did not notice a difference in training efficiency between the two versions and speculate that non-linear projections may be co-adapted with details of current image only in self-supervised representation learning methods. We also remove the text transformation function t u from Zhang et al. (2020) which samples a single sentence at uniform from the text since many of the (image, text) pairs in CLIP's pretraining dataset are only a single sentence. We also simplify the image transformation function t v . A random square crop from resized images is the only data augmentation used during training. Finally, the temperature parameter which controls the range of the logits in the softmax, τ , is directly optimized during training as a log-parameterized multiplicative scalar to avoid turning as a hyper-parameter.
Learning Transferable Visual Models From Natural Language Supervision
We consider two different architectures for the image encoder. For the first, we use ResNet-50 (He et al., 2016a) as the base architecture for the image encoder due to its widespread adoption and proven performance. We make several modifications to the original version using the ResNet-D improvements from He et al. (2019) and the antialiased rect-2 blur pooling from Zhang (2019). We also replace the global average pooling layer with an attention pooling mechanism. The attention pooling is implemented as a single layer of "transformer-style" multi-head QKV attention where the query is conditioned on the global average-pooled  representation of the image. For the second architecture, we experiment with the recently introduced Vision Transformer (ViT) (Dosovitskiy et al., 2020). We closely follow their implementation with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme.
Learning Transferable Visual Models From Natural Language Supervision
The text encoder is a Transformer (Vaswani et al., 2017) with the architecture modifications described in Radford et al. (2019). As a base size we use a 63M-parameter 12layer 512-wide model with 8 attention heads. The transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015). For computational efficiency, the max sequence length was capped at 76.  (Loshchilov & Hutter, 2017) applied to all weights that are not gains or biases, and decay the learning rate using a cosine schedule (Loshchilov & Hutter, 2016). Initial hyperparameters were set using a combination of grid searches, random search, and manual tuning on the baseline ResNet-50 model when trained for 1 epoch. Hyper-parameters were then adapted heuristically for larger models due to computational constraints. The learnable temperature parameter τ was initialized to the equivalent of 0.07 from (Wu et al., 2018) and clipped to prevent scaling the logits by more than 100 which we found necessary to prevent training instability. We use a very large minibatch size of 32,768. Mixed-precision (Micikevicius et al., 2017) was used to accelerate training and save memory. To save additional memory, gradient checkpointing (Griewank & Walther, 2000;Chen et al., 2016), half-precision Adam statistics , and half-precision stochastically rounded text encoder weights were used. The calculation of embedding similarities was also sharded with individual GPUs computing only the subset of the pairwise similarities necessary for their local batch of embeddings. The largest ResNet model, RN50x64, took 18 days to train on 592 V100 GPUs while the largest Vision Transformer took 12 days on 256 V100 GPUs. For the ViT-L/14 we also pre-train at a higher 336 pixel resolution for one additional epoch to boost performance similar to FixRes (Touvron et al., 2019). We denote this model as ViT-L/14@336px. Unless otherwise specified, all results reported in this paper as "CLIP" use this model which we found to perform best.
Learning Transferable Visual Models From Natural Language Supervision
In computer vision, zero-shot learning usually refers to the study of generalizing to unseen object categories in image classification (Lampert et al., 2009). We instead use the term in a broader sense and study generalization to unseen datasets. We motivate this as a proxy for performing unseen tasks, as aspired to in the zero-data learning paper of Larochelle et al. (2008). While much research in the field of unsupervised learning focuses on the representation learning capabilities of machine learning systems, we motivate studying zero-shot transfer as a way of measuring the tasklearning capabilities of machine learning systems. In this view, a dataset evaluates performance on a task on a specific distribution. However, many popular computer vision datasets were created by the research community primarily as benchmarks to guide the development of generic image classification methods rather than measuring performance on a specific task. While it is reasonable to say that the SVHN dataset measures the task of street number transcription on the distribution of Google Street View photos, it is unclear what "real" task the CIFAR-10 dataset measures. It is clear, however, what distribution CIFAR-10 is drawn from -TinyImages (Torralba et al., 2008). On these kinds of datasets, zero-shot transfer is more an evaluation of CLIP's robustness to distribution shift and domain generalization rather than task generalization. Please see Section 3.3 for analysis focused on this.
Learning Transferable Visual Models From Natural Language Supervision
To our knowledge, Visual N-Grams (Li et al., 2017) first studied zero-shot transfer to existing image classification datasets in the manner described above. It is also the only other work we are aware of that has studied zero-shot transfer to standard image classification datasets using a generically pre-trained model and serves as the best reference point for contextualizing CLIP. Their approach learns the parameters of a dictionary of 142,806 visual n-grams (spanning 1-to 5-grams) and optimizes these n-grams using a differential version of Jelinek-Mercer smoothing to maximize the probability of all text n-grams for a given image. In order to perform zero-shot transfer, they first convert the text of each of the dataset's class names into its n-gram representation and then compute its probability according to their model, predicting the one with the highest score.
Learning Transferable Visual Models From Natural Language Supervision
Our focus on studying zero-shot transfer as an evaluation of task learning is inspired by work demonstrating task learning in the field of NLP. To our knowledge Liu et al. (2018) first identified task learning as an "unexpected side-effect" when a language model trained to generate Wikipedia articles learned to reliably transliterate names between languages. While GPT-1 (Radford et al., 2018) focused on pre-training as a transfer learning method to improve supervised fine-tuning, it also included an ablation study demonstrating that the performance of four heuristic zero-shot transfer methods improved steadily over the course of pre-training, without any supervised adaption. This analysis served as the basis for GPT-2  which focused exclusively on studying the task-learning capabilities of language models via zero-shot transfer.
Learning Transferable Visual Models From Natural Language Supervision
3.1.2. USING CLIP FOR ZERO-SHOT TRANSFER CLIP is pre-trained to predict if an image and a text snippet are paired together in its dataset. To perform zero-shot classification, we reuse this capability. For each dataset, we use the names of all the classes in the dataset as the set of potential text pairings and predict the most probable (image, text) pair according to CLIP. In a bit more detail, we first compute the feature embedding of the image and the feature embedding of the set of possible texts by their respective encoders. The cosine similarity of these embeddings is then calculated, scaled by a temperature parameter τ , and normalized into a probability distribution via a softmax. Note that this prediction layer is a multinomial logistic regression classifier with L2-normalized inputs, L2-normalized weights, no bias, and temperature scaling. When interpreted this way, the image encoder is the computer vision backbone which computes a feature representation for the image and the text encoder is a hypernetwork (Ha et al., 2016) which generates the weights of a linear classifier based on the text specifying the visual concepts that the classes represent. Lei Ba et al. (2015) first introduced a zero-shot image classifier of this form while the idea of generating a classifier from natural language dates back to at least Elhoseiny et al. (2013). Continuing with this interpretation, every step of CLIP pre-training can be viewed as optimizing the performance of a randomly created proxy to a computer vision dataset which contains 1 example per class and has 32,768 total classes defined via natural language descriptions. For zero-shot evaluation, we cache the zero-shot classifier once it has been computed by the text encoder and reuse it for all subsequent predictions. This allows the cost of generating it to be amortized across all the predictions in a dataset.
Learning Transferable Visual Models From Natural Language Supervision
In Table 1 we compare Visual N-Grams to CLIP. The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 million crowd-labeled training examples available for this dataset. Additionally, the top-5 accuracy of CLIP models are noticeably higher than their top-1, and this model has a 95% top-5 accuracy, matching Inception-V4 (Szegedy et al., 2016). The ability to match the performance of a strong, fully supervised baselines in a zero-shot setting suggests CLIP is a significant step towards flexible and practical zero-shot computer vision classifiers. As mentioned above, the comparison to Visual N-Grams is meant for contextualizing the performance of CLIP and should not be interpreted as a direct methods comparison between CLIP and Visual N-Grams as many performance relevant differences between the two systems were not controlled for. For instance, we train on a dataset that is 10x larger, use a vision model that requires nearly 100x more compute per prediction, likely used over 1000x their training compute, and use a transformer-based model which did not exist when Visual N-Grams was published. As a closer comparison, we trained a CLIP ResNet-50 on the same YFCC100M dataset that Visual N-Grams was trained on and found it matched their reported ImageNet performance within a V100 GPU day. This baseline was also trained from scratch instead of being initialized from pre-trained ImageNet weights as in Visual N-Grams.
Learning Transferable Visual Models From Natural Language Supervision
CLIP also outperforms Visual N-Grams on the other 2 reported datasets. On aYahoo, CLIP achieves a 95% reduction in the number of errors, and on SUN, CLIP more than doubles the accuracy of Visual N-Grams. To conduct a more comprehensive analysis and stress test, we implement a much larger evaluation suite detailed in Appendix A. In total we expand from the 3 datasets reported in Visual N-Grams to include over 30 datasets and compare to over 50 existing computer vision systems to contextualize results.
Learning Transferable Visual Models From Natural Language Supervision
Most standard image classification datasets treat the information naming or describing classes which enables natural language based zero-shot transfer as an afterthought. The vast majority of datasets annotate images with just a numeric id of the label and contain a file mapping these ids back to their names in English. Some datasets, such as Flowers102 and GTSRB, don't appear to include this mapping at all in their released versions preventing zero-shot transfer entirely. 2 For many datasets, we observed these labels may be 2 Alec learned much more about flower species and German traffic signs over the course of this project than he originally anticipated.  . Prompt engineering and ensembling improve zeroshot performance. Compared to the baseline of using contextless class names, prompt engineering and ensembling boost zero-shot classification performance by almost 5 points on average across 36 datasets. This improvement is similar to the gain from using 4 times more compute with the baseline zero-shot method but is "free" when amortized over many predictions. chosen somewhat haphazardly and do not anticipate issues related to zero-shot transfer which relies on task description in order to transfer successfully.
Learning Transferable Visual Models From Natural Language Supervision
A common issue is polysemy. When the name of a class is the only information provided to CLIP's text encoder it is unable to differentiate which word sense is meant due to the lack of context. In some cases multiple meanings of the same word might be included as different classes in the same dataset! This happens in ImageNet which contains both construction cranes and cranes that fly. Another example is found in classes of the Oxford-IIIT Pet dataset where the word boxer is, from context, clearly referring to a breed of dog, but to a text encoder lacking context could just as likely refer to a type of athlete.
Learning Transferable Visual Models From Natural Language Supervision
Another issue we encountered is that it's relatively rare in our pre-training dataset for the text paired with the image to be just a single word. Usually the text is a full sentence describing the image in some way. To help bridge this distribution gap, we found that using the prompt template "A photo of a {label}." to be a good default that helps specify the text is about the content of the image. This often improves performance over the baseline of using only the label text. For instance, just using this prompt improves accuracy on ImageNet by 1.3%. Similar to the "prompt engineering" discussion around GPT-3 (Brown et al., 2020;Gao et al., 2020), we have also observed that zero-shot performance can be significantly improved by customizing the prompt text to each task. A few, non exhaustive, examples follow. We found on several fine-grained image classification datasets that it helped to specify the category. For example on Oxford-IIIT Pets, using "A photo of a {label}, a type of pet." to help provide context worked well. Likewise, on Food101 specifying a type of food and on FGVC Aircraft a type of aircraft helped too. For OCR datasets, we found that putting quotes around the text or number to be recognized improved performance. Finally, we found that on satellite image classification datasets it helped to specify that the images were of this form and we use variants of "a satellite photo of a {label}.".
Learning Transferable Visual Models From Natural Language Supervision
We also experimented with ensembling over multiple zeroshot classifiers as another way of improving performance. These classifiers are computed by using different context prompts such as 'A photo of a big {label}" and "A photo of a small {label}". We construct the ensemble over the embedding space instead of probability space. This allows us to cache a single set of averaged text embeddings so that the compute cost of the ensemble is the same as using a single classifier when amortized over many predictions. We've observed ensembling across many generated zero-shot classifiers to reliably improve performance and use it for the majority of datasets. On ImageNet, we ensemble 80 different context prompts and this improves performance by an additional 3.5% over the single default prompt discussed above. When considered together, prompt engineering and ensembling improve ImageNet accuracy by almost 5%. In Figure 4 we visualize how prompt engineering and ensembling change the performance of a set of CLIP models compared to the contextless baseline approach of directly embedding the class name as done in Li et al. (2017).
Learning Transferable Visual Models From Natural Language Supervision
Since task-agnostic zero-shot classifiers for computer vision have been understudied, CLIP provides a promising opportunity to gain a better understanding of this type of model. In this section, we conduct a study of various properties of CLIP's zero-shot classifiers. As a first question, we look simply at how well zero-shot classifiers perform. To contextualize this, we compare to the performance of a simple off-the-shelf baseline: fitting a fully supervised, regularized, logistic regression classifier on the features of the canonical ResNet-50. In Figure 5 we show this comparison across 27 datasets. Please see Appendix A for details of datasets and setup.
Learning Transferable Visual Models From Natural Language Supervision
Zero-shot CLIP outperforms this baseline slightly more of- +28.9 StanfordCars Figure 5. Zero-shot CLIP is competitive with a fully supervised baseline. Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet. ten than not and wins on 16 of the 27 datasets. Looking at individual datasets reveals some interesting behavior. On fine-grained classification tasks, we observe a wide spread in performance. On two of these datasets, Stanford Cars and Food101, zero-shot CLIP outperforms logistic regression on ResNet-50 features by over 20% while on two others, Flowers102 and FGVCAircraft, zero-shot CLIP underperforms by over 10%. On OxfordPets and Birdsnap, performance is much closer. We suspect these difference are primarily due to varying amounts of per-task supervision between WIT and ImageNet. On "general" object classification datasets such as ImageNet, CIFAR10/100, STL10, and PascalVOC2007 performance is relatively similar with a slight advantage for zero-shot CLIP in all cases. On STL10, CLIP achieves 99.3% overall which appears to be a new state of the art despite not using any training examples. Zeroshot CLIP significantly outperforms a ResNet-50 on two datasets measuring action recognition in videos. On Kinet-ics700, CLIP outperforms a ResNet-50 by 14.5%. Zeroshot CLIP also outperforms a ResNet-50's features by 7.7% on UCF101. We speculate this is due to natural language providing wider supervision for visual concepts involving verbs, compared to the noun-centric object supervision in ImageNet.
Learning Transferable Visual Models From Natural Language Supervision
Looking at where zero-shot CLIP notably underperforms, we see that zero-shot CLIP is quite weak on several specialized, complex, or abstract tasks such as satellite image classification (EuroSAT and RESISC45), lymph node tumor detection (PatchCamelyon), counting objects in synthetic scenes (CLEVRCounts), self-driving related tasks such as German traffic sign recognition (GTSRB), recognizing distance to the nearest car (KITTI Distance). These results highlight the poor capability of zero-shot CLIP on more complex tasks. By contrast, non-expert humans can robustly perform several of these tasks, such as counting, satellite image classification, and traffic sign recognition, suggesting significant room for improvement. However, we caution that it is unclear whether measuring zero-shot transfer, as opposed to few-shot transfer, is a meaningful evaluation for difficult tasks that a learner has no prior experience with, such as lymph node tumor classification for almost all humans (and possibly CLIP).
Learning Transferable Visual Models From Natural Language Supervision
While comparing zero-shot performance to fully supervised models contextualizes the task-learning capabilities of CLIP, comparing to few-shot methods is a more direct comparison, since zero-shot is its limit. In Figure 6, we visualize how zero-shot CLIP compares to few-shot logistic regression on the features of many image models including the best publicly available ImageNet models, self-supervised learning methods, and CLIP itself. While it is intuitive to expect zero-shot to underperform one-shot, we instead find that zero-shot CLIP matches the performance of 4-shot logistic regression on the same feature space. This is likely due to an important difference between the zero-shot and few-shot approach. First, CLIP's zero-shot classifier is generated via natural language which allows for visual concepts to be directly specified ("communicated"). By contrast, "normal" supervised learning must infer concepts indirectly from training examples. Context-less example-based learning has the drawback that many different hypotheses can be consistent with the data, especially in the one-shot case.
Learning Transferable Visual Models From Natural Language Supervision
A single image often contains many different visual concepts. Although a capable learner is able to exploit visual cues and heuristics, such as assuming that the concept being demonstrated is the primary object in an image, there is no guarantee.
Learning Transferable Visual Models From Natural Language Supervision
A potential resolution of this discrepancy between zeroshot and few-shot performance is to use CLIP's zero-shot classifier as a prior for the weights of the few-shot classifier. While adding an L2 penalty towards the generated weights is a straightforward implementation of this idea, we found that hyperparameter optimization would often select for such a large value of this regularizer that the resulting fewshot classifier was "just" the zero-shot classifier. Research into better methods of combining the strength of zero-shot transfer with flexibility of few-shot learning is a promising direction for future work.
Learning Transferable Visual Models From Natural Language Supervision
When comparing zero-shot CLIP to few-shot logistic regression on the features of other models, zero-shot CLIP roughly matches the performance of the best performing 16-shot classifier in our evaluation suite, which uses the features of a BiT-M ResNet-152x2 trained on ImageNet-21K. We are certain that a BiT-L model trained on JFT-300M would perform even better but these models have not been publicly released. That a BiT-M ResNet-152x2 performs best in a 16-shot setting is somewhat surprising since, as analyzed in Section 3.2, the Noisy Student EfficientNet-L2 outperforms it in a fully supervised setting by almost 5% on average across 27 datasets.
Learning Transferable Visual Models From Natural Language Supervision
In addition to studying the average performance of zero-shot CLIP and few-shot logistic regression, we also examine performance on individual datasets. In Figure 7, we show estimates for the number of labeled examples per class that a logistic regression classifier on the same feature space requires to match the performance of zero-shot CLIP. Since zero-shot CLIP is also a linear classifier, this estimates the effective data efficiency of zero-shot transfer in this setting. In order to avoid training thousands of linear classifiers, we estimate the effective data efficiency based on a loglinear interpolation of the performance of a 1, 2, 4, 8, 16shot (when possible), and a fully supervised linear classifier trained on each dataset. We find that zero-shot transfer can If we assume that evaluation datasets are large enough that the parameters of linear classifiers trained on them are well estimated, then, because CLIP's zero-shot classifier is also a linear classifier, the performance of the fully supervised classifiers roughly sets an upper bound for what zero-shot transfer can achieve. In Figure 8 we compare CLIP's zeroshot performance with fully supervised linear classifiers across datasets. The dashed, y = x line represents an "optimal" zero-shot classifier that matches the performance of its fully supervised equivalent. For most datasets, the performance of zero-shot classifiers still underperform fully supervised classifiers by 10% to 25%, suggesting that there is still plenty of headroom for improving CLIP's task-learning and zero-shot transfer capabilities.
Learning Transferable Visual Models From Natural Language Supervision
There is a positive correlation of 0.82 (p-value < 10 −6 ) between zero-shot performance and fully supervised perfor-  Figure 8. Zero-shot performance is correlated with linear probe performance but still mostly sub-optimal. Comparing zero-shot and linear probe performance across datasets shows a strong correlation with zero-shot performance mostly shifted 10 to 25 points lower. On only 5 datasets does zero-shot performance approach linear probe performance (≤3 point difference). mance, suggesting that CLIP is relatively consistent at connecting underlying representation and task learning to zeroshot transfer. However, zero-shot CLIP only approaches fully supervised performance on 5 datasets: STL10, CI-FAR10, Food101, OxfordPets, and Caltech101. On all 5 datasets, both zero-shot accuracy and fully supervised accuracy are over 90%. This suggests that CLIP may be more effective at zero-shot transfer for tasks where its underlying representations are also high quality. The slope of a linear regression model predicting zero-shot performance as a function of fully supervised performance estimates that for every 1% improvement in fully supervised performance, zero-shot performance improves by 1.28%. However, the 95th-percentile confidence intervals still include values of less than 1 (0.93-1.79).
Learning Transferable Visual Models From Natural Language Supervision
Over the past few years, empirical studies of deep learning systems have documented that performance is predictable as a function of important quantities such as training compute and dataset size (Hestness et al., 2017;Kaplan et al., 2020). The GPT family of models has so far demonstrated consistent improvements in zero-shot performance across a 1000x increase in training compute. In Figure 9, we check whether the zero-shot performance of CLIP follows a similar scaling pattern. We plot the average error rate of the 5 ResNet CLIP models across 39 evaluations on 36 different datasets and find that a similar log-log linear scaling trend holds for CLIP across a 44x increase in model compute. While the overall trend is smooth, we found that performance on individual evaluations can be much noisier. We are unsure whether RN50x64 Figure 9. Zero-shot CLIP performance scales smoothly as a function of model compute. Across 39 evals on 36 different datasets, average zero-shot error is well modeled by a log-log linear trend across a 44x range of compute spanning 5 different CLIP models. Lightly shaded lines are performance on individual evals, showing that performance is much more varied despite the smooth overall trend. this is caused by high variance between individual training runs on sub-tasks (as documented in D'Amour et al. (2020)) masking a steadily improving trend or whether performance is actually non-monotonic as a function of compute on some tasks.
Learning Transferable Visual Models From Natural Language Supervision
While we have extensively analyzed the task-learning capabilities of CLIP through zero-shot transfer in the previous section, it is more common to study the representation learning capabilities of a model. There exist many ways to evaluate the quality of representations as well as disagreements over what properties an "ideal" representation should have (Locatello et al., 2020). Fitting a linear classifier on a representation extracted from the model and measuring its performance on various datasets is a common approach. An alternative is measuring the performance of end-to-end fine-tuning of the model. This increases flexibility, and prior work has convincingly demonstrated that fine-tuning outperforms linear classification on most image classification datasets (Kornblith et al., 2019;Zhai et al., 2019). While the high performance of fine-tuning motivates its study for practical reasons, we still opt for linear classifier based evaluation for several reasons. Our work is focused on developing a high-performing task and dataset-agnostic pre-training approach. Fine-tuning, because it adapts representations to each dataset during the fine-tuning phase, can compensate for and potentially mask failures to learn general and robust representations during the pre-training phase. Linear classifiers, because of their limited flexibility, instead highlight these failures and provide clear feedback during development. For CLIP, training supervised linear classifiers has the added benefit of being very similar to the approach used for its zero-shot classifiers which enables extensive comparisons and analysis in Section 3.1. Finally, we aim to compare CLIP to a comprehensive set of existing models across many tasks. Studying 66 different models on 27 different datasets requires tuning 1782 different evaluations. Fine-tuning opens up a much larger design and hyperparameter space, which makes it difficult to fairly evaluate and computationally expensive to compare a diverse set of techniques as discussed in other large scale empirical studies (Lucic et al., 2018;Choi et al., 2019). By comparison, linear classifiers require minimal hyper-parameter tuning and have standardized implementations and evaluation procedures. Please see Appendix A for further details on evaluation.
Learning Transferable Visual Models From Natural Language Supervision
Figure 10 summarizes our findings. To minimize selection effects that could raise concerns of confirmation or reporting bias, we first study performance on the 12 dataset evaluation suite from Kornblith et al. (2019). While small CLIP models such as a ResNet-50 and ResNet-101 outperform other ResNets trained on ImageNet-1K (BiT-S and the originals), they underperform ResNets trained on ImageNet-21K (BiT-M). These small CLIP models also underperform models in the EfficientNet family with similar compute requirements. However, models trained with CLIP scale very well and the largest model we trained (ResNet-50x64) slightly outperforms the best performing existing model (a Noisy Student EfficientNet-L2) on both overall score and compute efficiency. We also find that CLIP vision transformers are about 3x more compute efficient than CLIP ResNets, which allows us to reach higher overall performance within our compute budget. These results qualitatively replicate the findings of Dosovitskiy et al. (2020) which reported that vision transformers are more compute efficient than convnets when trained on sufficiently large datasets. Our best overall model is a ViT-L/14 that is fine-tuned at a higher resolution of 336 pixels on our dataset for 1 additional epoch. This model outperforms the best existing model across this evaluation suite by an average of 2.6%.
Learning Transferable Visual Models From Natural Language Supervision
As Figure 21 qualitatively shows, CLIP models learn a wider set of tasks than has previously been demonstrated in a single computer vision model trained end-to-end from random initialization. These tasks include geo-localization, optical character recognition, facial emotion recognition, and action recognition. None of these tasks are measured in the evaluation suite of Kornblith et al. (2019). This could be argued to be a form of selection bias in Kornblith et al. (2019)'s study towards tasks that overlap with ImageNet. To address this, we also measure performance on a broader 27 dataset evaluation suite. This evaluation suite, detailed in Appendix A includes datasets representing the aforementioned tasks, German Traffic Signs Recognition Benchmark (Stallkamp et al., 2011), as well as several other datasets adapted from VTAB (Zhai et al., 2019).  (Right) Scores are averaged over 27 datasets that contain a wider variety of distributions. Dotted lines indicate models fine-tuned or evaluated on images at a higher-resolution than pre-training. See Table 10 for individual scores and Figure 20 for plots for each dataset.
Learning Transferable Visual Models From Natural Language Supervision
On this broader evaluation suite, the benefits of CLIP are more clear. All CLIP models, regardless of scale, outperform all evaluated systems in terms of compute efficiency. The improvement in average score of the best model over previous systems increases from 2.6% to 5%. We also find that self-supervised systems do noticeably better on our broader evaluation suite. For instance, while SimCLRv2 still underperforms BiT-M on average on the 12 datasets of Kornblith et al. (2019), SimCLRv2 outperforms BiT-M on our 27 dataset evaluation suite. These findings suggest continuing to expand task diversity and coverage in order to better understand the "general" performance of systems. We suspect additional evaluation efforts along the lines of VTAB to be valuable.
Learning Transferable Visual Models From Natural Language Supervision
In addition to the aggregate analysis above, we visualize per-dataset differences in the performance of the best CLIP model and the best model in our evaluation suite across all 27 datasets in Figure 11. CLIP outperforms the Noisy Student EfficientNet-L2 on 21 of the 27 datasets. CLIP improves the most on tasks which require OCR (SST2 and HatefulMemes), geo-localization and scene recognition (Country211, SUN397), and activity recognition in videos (Kinetics700 and UCF101). In addition CLIP also does much better on fine-grained car and traffic sign recognition (Stanford Cars and GTSRB). This may reflect a problem with overly narrow supervision in ImageNet. A result such as the 14.7% improvement on GTSRB could be indicative of an issue with ImageNet-1K, which has only a single label for all traffic and street signs. This could encourage a supervised representation to collapse intra-class details and hurt accuracy on a fine-grained downstream task. As mentioned, CLIP still underperforms the EfficientNet on several datasets. Unsurprisingly, the dataset that the Effi-cientNet does best relative to CLIP on is the one it was trained on: ImageNet. The EffcientNet also slightly outperforms CLIP on low-resolution datasets such as CIFAR10 and CIFAR100. We suspect this is at least partly due to the lack of scale-based data augmentation in CLIP. The Effi-cientNet also does slightly better on PatchCamelyon and CLEVRCounts, datasets where overall performance is still
Learning Transferable Visual Models From Natural Language Supervision
+6.5 SUN397 +14.7 GTSRB +15.9 StanfordCars +18.8 HatefulMemes +22.7 Country211 +23.6 SST2 Figure 11. CLIP's features outperform the features of the best ImageNet model on a wide variety of datasets. Fitting a linear classifier on CLIP's features outperforms using the Noisy Student EfficientNet-L2 on 21 out of 27 datasets. low for both approaches.
Learning Transferable Visual Models From Natural Language Supervision
In 2015, it was announced that a deep learning model exceeded human performance on the ImageNet test set (He et al., 2015). However, research in the subsequent years has repeatedly found that these models still make many simple mistakes (Dodge & Karam, 2017;Geirhos et al., 2018;Alcorn et al., 2019), and new benchmarks testing these systems has often found their performance to be much lower than both their ImageNet accuracy and human accuracy Barbu et al., 2019). What explains this discrepancy? Various ideas have been suggested and studied (Ilyas et al., 2019;Geirhos et al., 2020). A common theme of proposed explanations is that deep learning models are exceedingly adept at finding correlations and patterns which hold across their training dataset and thus improve in-distribution performance. However many of these correlations and patterns are actually spurious and do not hold for other distributions and result in large drops in performance on other datasets.
Learning Transferable Visual Models From Natural Language Supervision
We caution that, to date, most of these studies limit their evaluation to models trained on ImageNet. Recalling the topic of discussion, it may be a mistake to generalize too far from these initial findings. To what degree are these failures attributable to deep learning, ImageNet, or some combination of the two? CLIP models, which are trained via natural language supervision on a very large dataset and are capable of high zero-shot performance, are an opportunity to investigate this question from a different angle.
Learning Transferable Visual Models From Natural Language Supervision
Taori et al. (2020) is a recent comprehensive study moving towards quantifying and understanding these behaviors for ImageNet models. Taori et al. (2020) study how the performance of ImageNet models change when evaluated on natural distribution shifts. They measure performance on a set of 7 distribution shifts: ImageNetV2 , ImageNet Sketch , Youtube-BB and ImageNet-Vid , ObjectNet (Barbu et al., 2019), ImageNet Adversarial (Hendrycks et al., 2019), and ImageNet Rendition (Hendrycks et al., 2020a). They distinguish these datasets, which all consist of novel images collected from a variety of sources, from synthetic distribution shifts such as ImageNet-C (Hendrycks & Dietterich, 2019), Stylized ImageNet (Geirhos et al., 2018), or adversarial attacks (Goodfellow et al., 2014) which are created by perturbing existing images in various ways. They propose this distinction because in part because they find that while several techniques have been demonstrated to improve performance on synthetic distribution shifts, they often fail to yield consistent improvements on natural distributions. 3 Across these collected datasets, the accuracy of ImageNet models drop well below the expectation set by the Ima-geNet validation set. For the following summary discussion we report average accuracy across all 7 natural distribution shift datasets and average accuracy across the corresponding class subsets of ImageNet unless otherwise specified. Additionally, for Youtube-BB and ImageNet-Vid, which have two different evaluation settings, we use the average of pm-0 and pm-10 accuracy.
Learning Transferable Visual Models From Natural Language Supervision
A ResNet-101 makes 5 times as many mistakes when evaluated on these natural distribution shifts compared to the ImageNet validation set. Encouragingly however, Taori et al.
Learning Transferable Visual Models From Natural Language Supervision
(2020) find that accuracy under distribution shift increases predictably with ImageNet accuracy and is well modeled as a linear function of logit-transformed accuracy. Taori et al. (2020) use this finding to propose that robustness analysis should distinguish between effective and relative robustness. Effective robustness measures improvements in accuracy under distribution shift above what is predicted by the documented relationship between in-distribution and out-of-distribution accuracy. Relative robustness captures any improvement in out-of-distribution accuracy. Taori et al. Linear probe average over 26 datasets Figure 12. CLIP's features are more robust to task shift when compared to models pre-trained on ImageNet. For both dataset splits, the transfer scores of linear probes trained on the representations of CLIP models are higher than other models with similar ImageNet performance. This suggests that the representations of models trained on ImageNet are somewhat overfit to their task. or fine-tuned on the ImageNet dataset. Returning to the discussion in the introduction to this section -is training or adapting to the ImageNet dataset distribution the cause of the observed robustness gap? Intuitively, a zero-shot model should not be able to exploit spurious correlations or patterns that hold only on a specific distribution, since it is not trained on that distribution. 4 Thus it is reasonable to expect zero-shot models to have much higher effective robustness. In Figure 13, we compare the performance of zero-shot CLIP with existing ImageNet models on natural distribution shifts. All zero-shot CLIP models improve effective robustness by a large amount and reduce the size of the gap between ImageNet accuracy and accuracy under distribution shift by up to 75%.
Learning Transferable Visual Models From Natural Language Supervision
While these results show that zero-shot models can be much more robust, they do not necessarily mean that supervised learning on ImageNet causes a robustness gap. Other details of CLIP, such as its large and diverse pre-training dataset or use of natural language supervision could also result in much more robust models regardless of whether they are zero-shot or fine-tuned. As an initial experiment to potentially begin narrowing this down, we also measure how the performance of CLIP models change after adapting to the ImageNet distribution via a L2 regularized logistic regression classifier fit to CLIP features on the ImageNet training set. We visualize how performance changes from the zero-shot classifier in Figure 14. Although adapting CLIP to the ImageNet distribution increases its ImageNet accuracy by 9.2% to 85.4% overall, and ties the accuracy of the 2018 SOTA from Mahajan et al. (2018), average accuracy under distribution shift slightly decreases.
Learning Transferable Visual Models From Natural Language Supervision
It is surprising to see a 9.2% increase in accuracy, which corresponds to roughly 3 years of improvement in SOTA, fail to translate into any improvement in average performance under distribution shift. We also break down the differences between zero-shot accuracy and linear classifier accuracy per dataset in Figure 14 and find performance still increases significantly on one dataset, ImageNetV2. ImageNetV2 closely followed the creation process of the original Ima-geNet dataset which suggests that gains in accuracy from supervised adaptation are closely concentrated around the ImageNet distribution. Performance decreases by 4.7% on   Figure 13. Zero-shot CLIP is much more robust to distribution shift than standard ImageNet models. (Left) An ideal robust model (dashed line) performs equally well on the ImageNet distribution and on other natural image distributions. Zero-shot CLIP models shrink this "robustness gap" by up to 75%. Linear fits on logit transformed values are shown with bootstrap estimated 95% confidence intervals. (Right) Visualizing distribution shift for bananas, a class shared across 5 of the 7 natural distribution shift datasets. The performance of the best zero-shot CLIP model, ViT-L/14@336px, is compared with a model that has the same performance on the ImageNet validation set, ResNet-101.
Learning Transferable Visual Models From Natural Language Supervision
ImageNet-R, 3.8% on ObjectNet, 2.8% on ImageNet Sketch, and 1.9% on ImageNet-A. The change in accuracy on the two other datasets, Youtube-BB and ImageNet Vid, is insignificant.
Learning Transferable Visual Models From Natural Language Supervision
How is it possible to improve accuracy by 9.2% on the Im-ageNet dataset with little to no increase in accuracy under distribution shift? Is the gain primarily from "exploiting spurious correlations"? Is this behavior unique to some combination of CLIP, the ImageNet datatset, and the distribution shifts studied, or a more general phenomena? Does it hold for end-to-end finetuning as well as linear classifiers? We do not have confident answers to these questions at this time. Prior work has also pre-trained models on distributions other than ImageNet, but it is common to study and release models only after they have been fine-tuned to ImageNet. As a step towards understanding whether pre-trained zero-shot models consistently have higher effective robustness than fine-tuned models, we encourage the authors of Mahajan et al. (2018) We also investigate another robustness intervention enabled by flexible zero-shot natural-language-based image classifiers. The target classes across the 7 transfer datasets are not always perfectly aligned with those of ImageNet. Two datasets, Youtube-BB and ImageNet-Vid, consist of superclasses of ImageNet. This presents a problem when trying to use the fixed 1000-way classifier of an ImageNet model to make predictions. Taori et al. (2020) handle this by max-pooling predictions across all sub-classes according to the ImageNet class hierarchy. Sometimes this mapping is much less than perfect. For the person class in Youtube-BB, predictions are made by pooling over the ImageNet classes for a baseball player, a bridegroom, and a scuba diver. With CLIP we can instead generate a custom zero-shot classifier for each dataset directly based on its class names. In Figure 14 we see that this improves average effective robustness by 5% but is concentrated in large improvements on only a few datasets. Curiously, accuracy on ObjectNet also increases by 2.3%. Although the dataset was designed to closely overlap with ImageNet classes, using the names provided for each class by ObjectNet's creators still helps a small amount compared to using ImageNet class names and pooling predictions when necessary.
Learning Transferable Visual Models From Natural Language Supervision
While zero-shot CLIP improves effective robustness, Figure  14 shows that the benefit is almost entirely gone in a fully supervised setting. To better understand this difference, we investigate how effective robustness changes on the continuum from zero-shot to fully supervised. In Figure 15 we visualize the performance of 0-shot, 1-shot, 2-shot, 4-shot ..., 128-shot, and fully supervised logistic regression classifiers on the best CLIP model's features. We see that while few-shot models also show higher effective robustness than existing models, this benefit fades as in-distribution performance increases with more training data and is mostly, though not entirely, gone for the fully supervised model. Additionally, zero-shot CLIP is notably more robust than a few-shot model with equivalent ImageNet performance. 
Learning Transferable Visual Models From Natural Language Supervision
+26.9 Youtube-BB Adapt to class shift Figure 14. While supervised adaptation to ImageNet increases ImageNet accuracy by 9.2%, it slightly reduces average robustness.
Learning Transferable Visual Models From Natural Language Supervision
(Left) Customizing zero-shot CLIP to each dataset improves robustness compared to using a single static zero-shot ImageNet classifier and pooling predictions across similar classes as in Taori et al. (2020). CLIP models adapted to ImageNet have similar effective robustness as the best prior ImageNet models. (Right) Details of per dataset changes in accuracy for the two robustness interventions. Adapting to ImageNet increases accuracy on ImageNetV2 noticeably but trades off accuracy on several other distributions. Dataset specific zero-shot classifiers can improve accuracy by a large amount but are limited to only a few datasets that include classes which don't perfectly align with ImageNet categories.
Learning Transferable Visual Models From Natural Language Supervision
Across our experiments, high effective robustness seems to result from minimizing the amount of distribution specific training data a model has access to, but this comes at a cost of reducing dataset-specific performance.
Learning Transferable Visual Models From Natural Language Supervision
Taken together, these results suggest that the recent shift towards large-scale task and dataset agnostic pre-training combined with a reorientation towards zero-shot and fewshot benchmarking on broad evaluation suites (as advocated by Yogatama et al. (2019) and Linzen (2020)) promotes the development of more robust systems and provides a more accurate assessment of performance. We are curious to see if the same results hold for zero-shot models in the field of NLP such as the GPT family. While Hendrycks et al. 
Learning Transferable Visual Models From Natural Language Supervision
How does CLIP compare to human performance and human learning? To get a better understanding of how well humans perform in similar evaluation settings to CLIP, we evaluated humans on one of our tasks. We wanted to get a sense of how strong human zero-shot performance is at these tasks, and how much human performance is improved if they are shown one or two image samples. This can help us to compare task difficulty for humans and CLIP, and identify correlations and differences between them.
Learning Transferable Visual Models From Natural Language Supervision
We had five different humans look at each of 3669 images in the test split of the Oxford IIT Pets dataset (Parkhi et al., 2012) and select which of the 37 cat or dog breeds best matched the image (or 'I don't know' if they were completely uncertain). In the zero-shot case the humans were given no examples of the breeds and asked to label them to the best of their ability without an internet search. In the one-shot experiment the humans were given one sample image of each breed and in the two-shot experiment they were given two sample images of each breed. 5
Learning Transferable Visual Models From Natural Language Supervision
One possible concern was that the human workers were not sufficiently motivated in the zero-shot task. High human accuracy of 94% on the STL-10 dataset    Figure 15. Few-shot CLIP also increases effective robustness compared to existing ImageNet models but is less robust than zero-shot CLIP. Minimizing the amount of ImageNet training data used for adaption increases effective robustness at the cost of decreasing relative robustness. 16-shot logistic regression CLIP matches zero-shot CLIP on ImageNet, as previously reported in Figure 7, but is less robust. and 97-100% accuracy on the subset of attention check images increased our trust in the human workers.
Learning Transferable Visual Models From Natural Language Supervision
Interestingly, humans went from a performance average of 54% to 76% with just one training example per class, and the marginal gain from an additional training example is minimal. The gain in accuracy going from zero to one shot is almost entirely on images that humans were uncertain about. This suggests that humans "know what they don't know" and are able to update their priors on the images they are most uncertain in based on a single example. Given this, it seems that while CLIP is a promising training strategy for zero-shot performance ( Figure 5) and does well on tests of natural distribution shift (Figure 13), there is a large difference between how humans learn from a few examples and the few-shot methods in this paper.
Learning Transferable Visual Models From Natural Language Supervision
This suggests that there are still algorithmic improvements waiting to be made to decrease the gap between machine and human sample efficiency, as noted by Lake et al. (2016) and others. Because these few-shot evaluations of CLIP don't make effective use of prior knowledge and the humans do, we speculate that finding a method to properly integrate prior knowledge into few-shot learning is an important step in algorithmic improvements to CLIP. To our knowledge, using a linear classifier on top of the features of a high-  Table 2. Comparison of human performance on Oxford IIT Pets. As in Parkhi et al. (2012), the metric is average per-class classification accuracy. Most of the gain in performance when going from the human zero shot case to the human one shot case is on images that participants were highly uncertain on. "Guesses" refers to restricting the dataset to where participants selected an answer other than "I don't know", the "majority vote" is taking the most frequent (exclusive of ties) answer per image.
Learning Transferable Visual Models From Natural Language Supervision
quality pre-trained model is near state-of-the-art for few shot learning , which suggests that there is a gap between the best few-shot machine learning methods and human few-shot learning.
Learning Transferable Visual Models From Natural Language Supervision
If we plot human accuracy vs CLIP's zero shot accuracy (Figure 16), we see that the hardest problems for CLIP are also hard for humans. To the extent that errors are consistent, our hypothesis is that this is due to at least a two factors: noise in the dataset (including mislabeled images) and out of distribution images being hard for both humans and models.
Learning Transferable Visual Models From Natural Language Supervision
A concern with pre-training on a very large internet dataset is unintentional overlap with downstream evals. This is important to investigate since, in a worst-case scenario, a complete copy of an evaluation dataset could leak into the pre-training dataset and invalidate the evaluation as a meaningful test of generalization. One option to prevent this is to identify and remove all duplicates before training a model. While this guarantees reporting true hold-out performance, it requires knowing all possible data which a model might be evaluated on ahead of time. This has the downside of limiting the scope of benchmarking and analysis. Adding a new evaluation would require an expensive re-train or risk reporting an un-quantified benefit due to overlap.
Learning Transferable Visual Models From Natural Language Supervision
Instead, we document how much overlap occurs and how performance changes due to these overlaps. To do this, we use the following procedure: Zero-Shot CLIP One-Shot Human Zero-Shot Human Figure 16. The hardest problems for CLIP also tend to be the hardest problems for humans. Here we rank image categories by difficulty for CLIP as measured as probability of the correct label.
Learning Transferable Visual Models From Natural Language Supervision
contains all examples that are below this threshold. We denote the unaltered full dataset All for reference. From this we first record the degree of data contamination as the ratio of the number of examples in Overlap to the size of All.
Learning Transferable Visual Models From Natural Language Supervision
2) We then compute the zero-shot accuracy of CLIP RN50x64 on the three splits and report All -Clean as our main metric. This is the difference in accuracy due to contamination. When positive it is our estimate of how much the overall reported accuracy on the dataset was inflated by over-fitting to overlapping data.
Learning Transferable Visual Models From Natural Language Supervision
3) The amount of overlap is often small so we also run a binomial significance test where we use the accuracy on Clean as the null hypothesis and compute the one-tailed (greater) p-value for the Overlap subset. We also calculate 99.5% Clopper-Pearson confidence intervals on Dirty as another check.
Learning Transferable Visual Models From Natural Language Supervision
A summary of this analysis is presented in Figure 17. Out of 35 datasets studied, 9 datasets have no detected overlap at all. Most of these datasets are synthetic or specialized making them unlikely to be posted as normal images on the internet (for instance MNIST, CLEVR, and GTSRB) or are guaranteed to have no overlap due to containing novel data from after the date our dataset was created (ObjectNet and Hateful Memes). This demonstrates our detector has a low-false positive rate which is important as false positives would under-estimate the effect of contamination in our analysis. There is a median overlap of 2.2% and an average overlap of 3.2%. Due to this small amount of overlap, overall accuracy is rarely shifted by more than 0.1% with only 7 datasets above this threshold. Of these, only 2 are statistically significant after Bonferroni correction. The max detected improvement is only 0.6% on Birdsnap which has the second largest overlap at 12.1%. The largest overlap is for Country211 at 21.5%. This is due to it being constructed out of YFCC100M, which our pre-training dataset contains a filtered subset of. Despite this large overlap there is only a 0.2% increase in accuracy on Country211. This may be because the training text accompanying an example is often not related to the specific task a downstream eval measures. Country211 measures geo-localization ability, but inspecting the training text for these duplicates showed they often do not mention the location of the image.
Learning Transferable Visual Models From Natural Language Supervision
We are aware of two potential concerns with our analysis. First our detector is not perfect. While it achieves near 100% accuracy on its proxy training task and manual inspection + threshold tuning results in very high precision with good recall among the found nearest-neighbors, we can not tractably check its recall across 400 million examples. Another potential confounder of our analysis is that the underlying data distribution may shift between the Overlap and Clean subsets. For example, on Kinetics-700 many "overlaps" are in fact all black transition frames. This explains why Kinetics-700 has an apparent 20% accuracy drop on Overlap. We suspect more subtle distribution shifts likely exist. One possibility we noticed on CIFAR-100 is that, due to the very low resolution of its images, many duplicates were false positives of small objects such as birds or planes. Changes in accuracy could instead be due to changes in the class distribution or difficulty of the duplicates. Unfortunately, these distribution and difficulty shifts could also mask the effects of over-fitting.
Learning Transferable Visual Models From Natural Language Supervision
However, these results closely follow the findings of similar duplicate analysis in previous work on large scale pretraining. Mahajan et al. (2018) and Kolesnikov et al. (2019) detected similar overlap rates and found minimal changes in overall performance. Importantly, Kolesnikov et al. (2019) also compared the alternative de-duplication strategy discussed in the introduction to this section with the approach we settled on and observed little difference between the two approaches.
Learning Transferable Visual Models From Natural Language Supervision
There are still many limitations to CLIP. While several of these are discussed as part of analysis in various sections, we summarize and collect them here.
Learning Transferable Visual Models From Natural Language Supervision
On datasets with training splits, the performance of zeroshot CLIP is on average competitive with the simple su- pervised baseline of a linear classifier on top of ResNet-50 features. On most of these datasets, the performance of this baseline is now well below the overall state of the art. Significant work is still needed to improve the task learning and transfer capabilities of CLIP. While scaling has so far steadily improved performance and suggests a route for continued improvement, we estimate around a 1000x increase in compute is required for zero-shot CLIP to reach overall state-of-the-art performance. This is infeasible to train with current hardware. Further research into improving upon the computational and data efficiency of CLIP will be necessary. Analysis in Section 3.1 found that CLIP's zero-shot performance is still quite weak on several kinds of tasks. When compared to task-specific models, the performance of CLIP is poor on several types of fine-grained classification such as differentiating models of cars, species of flowers, and variants of aircraft. CLIP also struggles with more abstract and systematic tasks such as counting the number of objects in an image. Finally for novel tasks which are unlikely to be included in CLIP's pre-training dataset, such as classifying the distance to the nearest car in a photo, CLIP's performance can be near random. We are confident that there are still many, many, tasks where CLIP's zero-shot performance is near chance level.
Learning Transferable Visual Models From Natural Language Supervision
While zero-shot CLIP generalizes well to many natural image distributions as investigated in Section 3.3, we've observed that zero-shot CLIP still generalizes poorly to data that is truly out-of-distribution for it. An illustrative example occurs for the task of OCR as reported in Appendix E.
Learning Transferable Visual Models From Natural Language Supervision
CLIP learns a high quality semantic OCR representation that performs well on digitally rendered text, which is common in its pre-training dataset, as evidenced by performance on Rendered SST2. However, CLIP only achieves 88% accuracy on the handwritten digits of MNIST. An embarrassingly simple baseline of logistic regression on raw pixels outperforms zero-shot CLIP. Both semantic and near-duplicate nearest-neighbor retrieval verify that there are almost no images that resemble MNIST digits in our pre-training dataset. This suggests CLIP does little to address the underlying problem of brittle generalization of deep learning models. Instead CLIP tries to circumvent the problem and hopes that by training on such a large and varied dataset that all data will be effectively in-distribution. This is a naive assumption that, as MNIST demonstrates, is easy to violate.
Learning Transferable Visual Models From Natural Language Supervision
Although CLIP can flexibly generate zero-shot classifiers for a wide variety of tasks and datasets, CLIP is still limited to choosing from only those concepts in a given zero-shot classifier. This is a significant restriction compared to a truly flexible approach like image captioning which could generate novel outputs. Unfortunately, as described in Section 2.3 we found the computational efficiency of the image caption baseline we tried to be much lower than CLIP. A simple idea worth trying is joint training of a contrastive and generative objective with the hope of combining the efficiency of CLIP with the flexibility of a caption model. As another alternative, search could be performed at inference time over many natural language explanations of a given image, similar to approach proposed in Learning with Latent Language Andreas et al. (2017).
Learning Transferable Visual Models From Natural Language Supervision
CLIP also does not address the poor data efficiency of deep learning. Instead CLIP compensates by using a source of supervision that can be scaled to hundreds of millions of training examples. If every image seen during training of a CLIP model was presented at a rate of one per second, it would take 405 years to iterate through the 12.8 billion images seen over 32 training epochs. Combining CLIP with self-supervision (Henaff, 2020; Chen et al., 2020c) and self-training (Lee; Xie et al., 2020) methods is a promising direction given their demonstrated ability to improve data efficiency over standard supervised learning.
Learning Transferable Visual Models From Natural Language Supervision
Our methodology has several significant limitations. Despite our focus on zero-shot transfer, we repeatedly queried performance on full validation sets to guide the development of CLIP. These validation sets often have thousands of examples, which is unrealistic for true zero-shot scenarios. Similar concerns have been raised in the field of semi-supervised learning (Oliver et al., 2018). Another potential issue is our selection of evaluation datasets. While we have reported results on Kornblith et al. (2019)'s 12 dataset evaluation suite as a standardized collection, our main results use a somewhat haphazardly assembled collection of 27 datasets that is undeniably co-adapted with the development and capabilities of CLIP. Creating a new benchmark of tasks designed explicitly to evaluate broad zero-shot transfer capabilities, rather than re-using existing supervised datasets, would help address these issues.
Learning Transferable Visual Models From Natural Language Supervision
CLIP is trained on text paired with images on the internet. These image-text pairs are unfiltered and uncurated and result in CLIP models learning many social biases. This has been previously demonstrated for image caption models (Bhargava & Forsyth, 2019). We refer readers to Section 7 for detailed analysis and quantification of these behaviors for CLIP as well as discussion of potential mitigation strategies.
Learning Transferable Visual Models From Natural Language Supervision
While we have emphasized throughout this work that specifying image classifiers through natural language is a flexible and general interface, it has its own limitations. Many complex tasks and visual concepts can be difficult to specify just through text. Actual training examples are undeniably useful but CLIP does not optimize for few-shot performance directly. In our work, we fall back to fitting linear classifiers on top of CLIP's features. This results in a counter-intuitive drop in performance when transitioning from a zero-shot to a few-shot setting. As discussed in Section 4, this is notably different from human performance which shows a large increase from a zero to a one shot setting. Future work is needed to develop methods that combine CLIP's strong zero-shot performance with efficient few-shot learning.
Learning Transferable Visual Models From Natural Language Supervision
CLIP has a wide range of capabilities due to its ability to carry out arbitrary image classification tasks. One can give it images of cats and dogs and ask it to classify cats, or give it images taken in a department store and ask it to classify shoplifters-a task with significant social implications and for which AI may be unfit. Like any image classification system, CLIP's performance and fitness for purpose need to be evaluated, and its broader impacts analyzed in context. CLIP also introduces a capability that will magnify and alter such issues: CLIP makes it possible to easily create your own classes for categorization (to 'roll your own classifier') without a need for re-training. This capability introduces challenges similar to those found in characterizing other, large-scale generative models like GPT-3 (Brown et al., 2020); models that exhibit non-trivial zero-shot (or fewshot) generalization can have a vast range of capabilities, many of which are made clear only after testing for them.
Learning Transferable Visual Models From Natural Language Supervision
Our studies of CLIP in a zero-shot setting show that the model displays significant promise for widely-applicable tasks like image retrieval or search. For example, it can find relevant images in a database given text, or relevant text given an image. Further, the relative ease of steering CLIP toward bespoke applications with little or no additional data or training could unlock a variety of novel applications that are hard for us to envision today, as has occurred with large language models over the past few years.
Learning Transferable Visual Models From Natural Language Supervision
In addition to the more than 30 datasets studied in earlier sections of this paper, we evaluate CLIP's performance on the FairFace benchmark and undertake exploratory bias probes. We then characterize the model's performance in a downstream task, surveillance, and discuss its usefulness as compared with other available systems. Many of CLIP's capabilities are omni-use in nature (e.g. OCR can be used to make scanned documents searchable, to power screen reading technologies, or to read license plates). Several of the capabilities measured, from action recognition, object classification, and geo-localization, to facial emotion recognition, can be used in surveillance. Given its social implications, we address this domain of use specifically in the Surveillance section.
Learning Transferable Visual Models From Natural Language Supervision
We have also sought to characterize the social biases inherent to the model. Our bias tests represent our initial efforts to probe aspects of how the model responds in different scenarios, and are by nature limited in scope. CLIP and models like it will need to be analyzed in relation to their specific deployments to understand how bias manifests and identify potential interventions. Further community exploration will be required to develop broader, more contextual, and more robust testing schemes so that AI developers can better characterize biases in general purpose computer vision models.
Learning Transferable Visual Models From Natural Language Supervision
Race Gender Age 
Learning Transferable Visual Models From Natural Language Supervision
Algorithmic decisions, training data, and choices about how classes are defined and taxonomized (which we refer to informally as "class design") can all contribute to and amplify social biases and inequalities resulting from the use of AI systems (Noble, 2018;Bechmann & Bowker, 2019;Bowker & Star, 2000). Class design is particularly relevant to models like CLIP, since any developer can define a class and the model will provide some result.
Learning Transferable Visual Models From Natural Language Supervision
In this section, we provide preliminary analysis of some of the biases in CLIP, using bias probes inspired by those outlined in Buolamwini & Gebru (2018)  FairFace is a face image dataset designed to balance age, gender, and race, in order to reduce asymmetries common in previous face datasets. It categorizes gender into 2 groups: female and male and race into 7 groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. There are inherent problems with race and gender classifications, as e.g. Bowker & Star (2000) as an initial bias probe, then probe the model further to surface additional biases and sources of biases, including class design.
Learning Transferable Visual Models From Natural Language Supervision
We evaluated two versions of CLIP on the FairFace dataset: a zero-shot CLIP model ("ZS CLIP"), and a logistic regression classifier fitted to FairFace's dataset on top of CLIP's features ("LR CLIP"). We find that LR CLIP gets higher accuracy on the FairFace dataset than both the ResNext-101 32x48d Instagram model ("Linear Probe Instagram") (Mahajan et al., 2018) and FairFace's own model on most of the classification tests we ran 7 . ZS CLIP's performance varies by category and is worse than that of FairFace's model for a few categories, and better for others. (See Table 3 and Table  4). and Keyes (2018) have shown. While FairFace's dataset reduces the proportion of White faces, it still lacks representation of entire large demographic groups, effectively erasing such categories. We use the 2 gender categories and 7 race categories defined in the FairFace dataset in a number of our experiments not in order to reinforce or endorse the use of such reductive categories, but in order to enable us to make comparisons to prior work. 7 One challenge with this comparison is that the FairFace model uses binary classes for race ("White" and "Non-White"), instead of breaking down races into finer-grained sub-groups. Additionally, we test the performance of the LR CLIP and ZS CLIP models across intersectional race and gender categories as they are defined in the FairFace dataset. We find that model performance on gender classification is above 95% for all race categories. Table 5 summarizes these results.
Learning Transferable Visual Models From Natural Language Supervision
While LR CLIP achieves higher accuracy than the Linear Probe Instagram model on the FairFace benchmark dataset for gender, race and age classification of images by intersectional categories, accuracy on benchmarks offers only one approximation of algorithmic fairness, as Raji et al. (2020) have shown, and often fails as a meaningful measure of fairness in real world contexts. Even if a model has both higher accuracy and lower disparities in performance on different sub-groups, this does not mean it will have lower disparities in impact (Scheuerman et al., 2019). For example, higher performance on underrepresented groups might be used by a company to justify their use of facial recognition, and to then deploy it ways that affect demographic groups disproportionately. Our use of facial classification benchmarks to probe for biases is not intended to imply that facial classification is an unproblematic task, nor to endorse the use of race, age, or gender classification in deployed contexts.
Learning Transferable Visual Models From Natural Language Supervision
We also probed the model using classification terms with high potential to cause representational harm, focusing on denigration harms in particular (Crawford, 2017). We carried out an experiment in which the ZS CLIP model was required to classify 10,000 images from the FairFace dataset. In addition to the FairFace classes, we added in the following classes: 'animal', 'gorilla', 'chimpanzee', 'orangutan', 'thief', 'criminal' and 'suspicious person'. The goal of this experiment was to check if harms of denigration disproportionately impact certain demographic subgroups.
Learning Transferable Visual Models From Natural Language Supervision
We found that 4.9% (confidence intervals between 4.6% and 5.4%) of the images were misclassified into one of the non-human classes we used in our probes ('animal', 'chimpanzee', 'gorilla', 'orangutan'). Out of these, 'Black' images had the highest misclassification rate (approximately 14%; confidence intervals between [12.6% and 16.4%]) while all other races had misclassification rates under 8%. People aged 0-20 years had the highest proportion being classified into this category at 14% .
Learning Transferable Visual Models From Natural Language Supervision
We also found that 16.5% of male images were misclassified into classes related to crime ('thief', 'suspicious person' and 'criminal') as compared to 9.8% of female images. Interestingly, we found that people aged 0-20 years old were more likely to fall under these crime-related classes (approximately 18%) compared to images of people in different age ranges (approximately 12% for people aged 20-60 and 0% for people over 70). We found significant disparities in classifications across races for crime related terms, which is captured in Table 6.
Learning Transferable Visual Models From Natural Language Supervision
Given that we observed that people under 20 were the most likely to be classified in both the crime-related and nonhuman animal categories, we carried out classification for the images with the same classes but with an additional category 'child' added to the categories. Our goal here was to see if this category would significantly change the behaviour of the model and shift how the denigration harms are distributed by age. We found that this drastically reduced the number of images of people under 20 classified in either crime-related categories or non-human animal categories (Table 7). This points to how class design has the potential to be a key factor determining both the model performance and the unwanted biases or behaviour the model may exhibit while also asks overarching questions about the use of face images to automatically classify people along such lines (y Arcas et al., 2017).
Learning Transferable Visual Models From Natural Language Supervision
The results of these probes can change based on the class categories one chooses to include as well as the specific language one uses to describe each class. Poor class design can lead to poor real world performance; this concern is particularly relevant to a model like CLIP, given how easily developers can design their own classes.
Learning Transferable Visual Models From Natural Language Supervision
We also carried out experiments similar to those outlined by Schwemmer et al. (2020) to test how CLIP treated images of men and women differently using images of Members of Congress. As part of these experiments, we studied how certain additional design decisions such as deciding thresholds for labels can impact the labels output by CLIP and how biases manifest.
Learning Transferable Visual Models From Natural Language Supervision
We carried out three experiments -we tested for accuracy on gender classification and we tested for how labels were differentially distributed across two different label sets. For our first label set, we used a label set of 300 occupations and for our second label set we used a combined set of labels that Google Cloud Vision, Amazon Rekognition and Microsoft Azure Computer Vision returned for all the images.
Learning Transferable Visual Models From Natural Language Supervision
We first simply looked into gender prediction performance of the model on the images of Members of Congress, in order to check to see if the model correctly recognized men as men and women as women given the image of a person who appeared to be in an official setting/position of power. We found that the model got 100% accuracy on the images. This is slightly better performance than the model's performance on the FairFace dataset. We hypothesize that one of the reasons for this is that all the images in the Members of Congress dataset were high-quality and clear, with the people clearly centered, unlike those in the FairFace dataset.
Learning Transferable Visual Models From Natural Language Supervision
In order to study how the biases in returned labels depend on the thresholds set for label probability, we did an experiment in which we set threshold values at 0.5% and 4.0%. We found that the lower threshold led to lower quality of labels. However, even the differing distributions of labels under this threshold can hold signals for bias. For example, we find that under the 0.5% threshold labels such as 'nanny' and 'housekeeper' start appearing for women whereas labels such as 'prisoner' and 'mobster' start appearing for men. This points to gendered associations similar to those that have previously been found for occupations (Schwemmer et al., 2020) (Nosek et al., 2002) (Bolukbasi et al., 2016).
Learning Transferable Visual Models From Natural Language Supervision
At the higher 4% threshold, the labels with the highest probability across both genders include "lawmaker", "legislator" and "congressman". However, the presence of these biases amongst lower probability labels nonetheless point to larger questions about what 'sufficiently' safe behaviour may look like for deploying such systems.
Learning Transferable Visual Models From Natural Language Supervision
When given the combined set of labels that Google Cloud Vision (GCV), Amazon Rekognition and Microsoft returned for all the images, similar to the biases Schwemmer et al. (2020) found in GCV systems, we found our system also disproportionately attached labels to do with hair and appearance in general to women more than men. For example, labels such as 'brown hair', 'blonde' and 'blond' appeared significantly more often for women. Additionally, CLIP attached some labels that described high status occupations disproportionately more often to men such as 'executive' and 'doctor'. Out of the only four occupations that it attached more often to women, three were 'newscaster', 'television presenter' and 'newsreader' and the fourth was 'Judge'. This is again similar to the biases found in GCV and points to historical gendered differences (Schwemmer et al., 2020).
Learning Transferable Visual Models From Natural Language Supervision
Interestingly, when we lowered the threshold to 0.5% for this set of labels, we found that the labels disproportionately describing men also shifted to appearance oriented words such as 'suit', 'tie' and 'necktie' (Figure 18). Many occupation oriented words such as 'military person' and 'executive' -which were not used to describe images of women at the higher 4% threshold -were used for both men and women at the lower 0.5% threshold, which could have caused the change in labels for men. The reverse was not true. Descriptive words used to describe women were still uncommon amongst men.
Learning Transferable Visual Models From Natural Language Supervision
Design decisions at every stage of building a model impact how biases manifest and this is especially true for CLIP given the flexibility it offers. In addition to choices about training data and model architecture, decisions about things like class designs and thresholding values can alter the labels a model outputs and as a result heighten or lower certain kinds of harm, such as those described by Crawford (2017). People designing and developing models and AI systems have considerable power. Decisions about things like class design are a key determiner not only of model performance, but also of how and in what contexts model biases manifest.
Learning Transferable Visual Models From Natural Language Supervision
These experiments are not comprehensive. They illustrate potential issues stemming from class design and other sources of bias, and are intended to spark inquiry.
Learning Transferable Visual Models From Natural Language Supervision
We next sought to characterize model performance in relation to a downstream task for which there is significant societal sensitivity: surveillance. Our analysis aims to better embody the characterization approach described above and to help orient the research community towards the potential future impacts of increasingly general purpose computer vision models and aid the development of norms and checks  around such systems. Our inclusion of surveillance is not intended to indicate enthusiasm for this domain -rather, we think surveillance is an important domain to try to make predictions about given its societal implications (Zuboff, 2015;Browne, 2015). We measure the model's performance on classification of images from CCTV cameras and zero-shot celebrity identification. We first tested model performance on low-resolution images captured from surveillance cameras (e.g. CCTV cameras). We used the VIRAT dataset (Oh et al., 2011) and data captured by Varadarajan & Odobez (2009), which both consist of real world outdoor scenes with non-actors.
Learning Transferable Visual Models From Natural Language Supervision
Given CLIP's flexible class construction, we tested 515 surveillance images captured from 12 different video sequences on self-constructed general classes for coarse and fine grained classification. Coarse classification required the model to correctly identify the main subject of the image (i.e. determine if the image was a picture of an empty parking lot, school campus, etc.). For fine-grained classification, the model had to choose between two options constructed to determine if the model could identify the presence/absence of smaller features in the image such as a person standing in the corner.
Learning Transferable Visual Models From Natural Language Supervision
For coarse classification, we constructed the classes by handcaptioning the images ourselves to describe the contents of the image and there were always at least 6 options for the model to choose from. Additionally, we carried out a 'stress test' where the class set included at least one more caption for something that was 'close' to the image (for example, 'parking lot with white car' vs. 'parking lot with red car'). We found that the model had a top-1 accuracy of 91.8% on the CCTV images for the initial evaluation. The accuracy dropped significantly to 51.1% for the second evaluation, with the model incorrectly choosing the 'close' answer 40.7% of the time.
Learning Transferable Visual Models From Natural Language Supervision
For fine-grained detection, the zero-shot model performed poorly, with results near random. Note that this experiment was targeted only towards detecting the presence or absence of small objects in image sequences.
Learning Transferable Visual Models From Natural Language Supervision
We also tested CLIP's zero-shot performance for 'in the wild' identity detection using the CelebA dataset 8 . We did this to evaluate the model's performance for identity detection using just the publicly available data it was pre-trained on. While we tested this on a dataset of celebrities who have a larger number of images on the internet, we hypothesize that the number of images in the pre-training data needed for the model to associate faces with names will keep decreasing as models get more powerful (see Table 8), which has significant societal implications (Garvie, 2019 Table 8. CelebA Zero-Shot Top-1 Identity Recognition Accuracy mirrors recent developments in natural language processing, in which recent large language models trained on Internet data often exhibit a surprising ability to provide information related to relatively minor public figures (Brown et al., 2020).
Learning Transferable Visual Models From Natural Language Supervision
We found that the model had 59.2% top-1 accuracy out of 100 possible classes for 'in the wild' 8k celebrity images. However, this performance dropped to 43.3% when we increased our class sizes to 1k celebrity names. This performance is not competitive when compared to production level models such as Google's Celebrity Recognition (Google). However, what makes these results noteworthy is that this analysis was done using only zero-shot identification capabilities based on names inferred from pre-training data -we didn't use any additional task-specific dataset, and so the (relatively) strong results further indicate that before deploying multimodal models, people will need to carefully study them for behaviors in a given context and domain.
Learning Transferable Visual Models From Natural Language Supervision
CLIP offers significant benefit for tasks that have relatively little data given its zero-shot capabilities. However, large datasets and high performing supervised models exist for many in-demand surveillance tasks such as facial recognition. As a result, CLIP's comparative appeal for such uses is low. Additionally, CLIP is not designed for common surveillance-relevant tasks like object detection and semantic segmentation. This means it has limited use for certain surveillance tasks when models that are designed with these uses in mind such as Detectron2  are widely available.
Learning Transferable Visual Models From Natural Language Supervision
However, CLIP does unlock a certain aspect of usability given how it removes the need for training data. Thus, CLIP and similar models could enable bespoke, niche surveillance use cases for which no well-tailored models or datasets exist, and could lower the skill requirements to build such applications. As our experiments show, ZS CLIP displays nontrivial, but not exceptional, performance on a few surveillance relevant tasks today.
Learning Transferable Visual Models From Natural Language Supervision
This preliminary analysis is intended to illustrate some of the challenges that general purpose computer vision models pose and to give a glimpse into their biases and impacts.
Learning Transferable Visual Models From Natural Language Supervision
We hope that this work motivates future research on the characterization of the capabilities, shortcomings, and biases of such models, and we are excited to engage with the research community on such questions.
Learning Transferable Visual Models From Natural Language Supervision
We believe one good step forward is community exploration to further characterize the capabilities of models like CLIP and -crucially -identify application areas where they have promising performance and areas where they may have reduced performance 9 . This process of characterization can help researchers increase the likelihood models are used beneficially by:
Learning Transferable Visual Models From Natural Language Supervision
• Identifying potentially beneficial downstream uses of models early in the research process, enabling other researchers to think about applications.
Learning Transferable Visual Models From Natural Language Supervision
• Surfacing tasks with significant sensitivity and a large set of societal stakeholders, which may call for intervention by policymakers.
Learning Transferable Visual Models From Natural Language Supervision
• Better characterizing biases in models, alerting other researchers to areas of concern and areas for interventions.
Learning Transferable Visual Models From Natural Language Supervision
• Creating suites of tests to evaluate systems like CLIP on, so we can better characterize model capabilities earlier in the development cycle.
Learning Transferable Visual Models From Natural Language Supervision
• Identifying potential failure modes and areas for further work.
Learning Transferable Visual Models From Natural Language Supervision
We plan to contribute to this work, and hope this analysis provides some motivating examples for subsequent research.
Learning Transferable Visual Models From Natural Language Supervision
We have investigated whether it is possible to transfer the success of task-agnostic web-scale pre-training in NLP to another domain. We find that adopting this formula results in similar behaviors emerging in the field of computer vision and discuss the social implications of this line of research. In order to optimize their training objective, CLIP models learn to perform a wide variety of tasks during pretraining. This task learning can then be leveraged via natural language prompting to enable zero-shot transfer to many existing datasets. At sufficient scale, the performance of this approach can be competitive with task-specific supervised models although there is still room for much improvement.
Learning Transferable Visual Models From Natural Language Supervision
We'd like to thank the millions of people involved in creating the data CLIP is trained on. We'd also like to thank Susan Zhang for her work on image conditional language models while at OpenAI, Ishaan Gulrajani for catching an error in the pseudocode, and Irene Solaiman, Miles Brundage, and Gillian Hadfield for their thoughtful feedback on the broader impacts section of the paper. We are also grateful to the Acceleration and Supercomputing teams at OpenAI for their critical work on software and hardware infrastructure this project used. Finally, we'd also like to thank the developers of the many software packages used throughout this project including, but not limited, to Numpy (Harris et al., 2020), SciPy (Virtanen et al., 2020), ftfy (Speer, 2019), Tensor-Flow (Abadi et al., 2016), PyTorch (Paszke et al., 2019), pandas (pandas development team, 2020), and scikit-learn (Pedregosa et al., 2011).  Li, X., Yin, X., Li, C., Hu, X., Zhang, P., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Objectsemantics aligned pre-training for vision-language tasks. arXiv preprint arXiv:2004.06165, 2020b.
Learning Transferable Visual Models From Natural Language Supervision
Liang, W., Zou, J., and Yu, Z. Alice: Active learning with contrastive natural language explanations. arXiv preprint arXiv:2009.10259, 2020.
Learning Transferable Visual Models From Natural Language Supervision
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Learning Transferable Visual Models From Natural Language Supervision
Linzen, T. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint arXiv:2005.00955, 2020.
Learning Transferable Visual Models From Natural Language Supervision
Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., and Yannakoudakis, H. A multimodal framework for the detection of hateful memes. arXiv preprint arXiv:2012.12871, 2020.
Learning Transferable Visual Models From Natural Language Supervision
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018.
Learning Transferable Visual Models From Natural Language Supervision
Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. A sober look at the unsupervised learning of disentangled representations and their evaluation. arXiv preprint arXiv:2010.14766, 2020.
Learning Transferable Visual Models From Natural Language Supervision
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
Learning Transferable Visual Models From Natural Language Supervision
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Learning Transferable Visual Models From Natural Language Supervision
Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for visionand-language tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Learning Transferable Visual Models From Natural Language Supervision
Lu, Z., Xiong, X., Li, Y., Stroud, J., and Ross, D. Leveraging weakly supervised data and pose representation for action recognition, 2020. URL https://www.youtube. com/watch?v=KOQFxbPPLOE&t=1390s. Mishra, A., Alahari, K., and Jawahar, C. Scene text recognition using higher order language priors. 2012.
Learning Transferable Visual Models From Natural Language Supervision
Mithun, N. C., Panda, R., Papalexakis, E. E., and Roy-Chowdhury, A. K. Webly supervised joint embedding for cross-modal image-text retrieval. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1856-1864, 2018.
Learning Transferable Visual Models From Natural Language Supervision
Mori, Y., Takahashi, H., and Oka, R. Image-to-word transformation based on dividing and vector quantizing images with words. Citeseer, 1999.
Learning Transferable Visual Models From Natural Language Supervision
Mu, J., Liang, P., and Goodman, N. Shaping visual representations with language for few-shot classification. arXiv preprint arXiv:1911.02683, 2019. 
Learning Transferable Visual Models From Natural Language Supervision
We provide additional details for linear probe experiments presented in this paper, including the list of the datasets and models used for evaluation.
Learning Transferable Visual Models From Natural Language Supervision
We use the 12 datasets from the well-studied evaluation suite introduced by ( , and the ImageNet-1k dataset (Deng et al., 2012). For the two video datasets (UCF101 and Ki-netics700), we use the middle frame of each video clip as the input image. STL-10 and UCF101 have multiple predefined train/validation/test splits, 10 and 3 respectively, and we report the average over all splits. Details on each dataset and the corresponding evaluation metrics are provided in Table 9.
Learning Transferable Visual Models From Natural Language Supervision
Additionally, we created two datasets that we call Coun-try211 and Rendered SST2. The Country211 dataset is designed to assess the geolocation capability of visual representations. We filtered the YFCC100m dataset (Thomee et al., 2016) to find 211 countries (defined as having an ISO-3166 country code) that have at least 300 photos with GPS coordinates, and we built a balanced dataset with 211 categories, by sampling 200 photos for training and 100 photos for testing, for each country.
Learning Transferable Visual Models From Natural Language Supervision
The Rendered SST2 dataset is designed to measure the optical character recognition capability of visual representations.
Learning Transferable Visual Models From Natural Language Supervision
To do so, we used the sentences from the Stanford Sentiment Treebank dataset (Socher et al., 2013) and rendered them into images, with black texts on a white background, in a 448×448 resolution. Two example images from this dataset are shown in Figure 19.
Learning Transferable Visual Models From Natural Language Supervision
In combination with the datasets listed above, we evaluate the following series of models using linear probes.
Learning Transferable Visual Models From Natural Language Supervision
LM RN50 This is a multimodal model that uses an autoregressive loss instead of a contrastive loss, while using the ResNet-50 architecture as in the smallest contrastive model. To do so, the output from the CNN is projected into four tokens, which are then fed as a prefix to a language model autoregressively predicting the text tokens. Apart from the training objective, the model was trained on the same dataset for the same number of epochs as other CLIP models.
Learning Transferable Visual Models From Natural Language Supervision
CLIP-RN Five ResNet-based contrastive CLIP models are included. As discussed in the paper, the first two models follow ResNet-50 and ResNet-101, and we use EfficientNetstyle (Tan & Le, 2019) scaling for the next three models which simultaneously scale the model width, the number of layers, and the input resolution to obtain models with roughly 4x, 16x, and 64x computation. Instagram-pretrained ResNeXt We use the four models (32x8d, 32x16d, 32x32d, 32x48d) released by (Mahajan et al., 2018), as well as their two FixRes variants which use higher input resolutions (Touvron et al., 2019).
Learning Transferable Visual Models From Natural Language Supervision
Big Transfer (BiT) We use BiT-S and BiT-M models (Kolesnikov et al., 2019), trained on the ImageNet-1k and ImageNet-21k datasets. The model weights for BiT-L is not publicly available.
Learning Transferable Visual Models From Natural Language Supervision
Vision Transformer (ViT) We also include four ViT (Dosovitskiy et al., 2020) checkpoints pretrained on the ImageNet-21k dataset, namely ViT-B/32, ViT-B/16, ViT-L/16, and ViT-H/14. We note that their best-performing models, trained on the JFT-300M dataset, are not available publicly.
Learning Transferable Visual Models From Natural Language Supervision
SimCLRv2 The SimCLRv2 (Chen et al., 2020c) project released pre-trained and fine-tuned models in various settings. We use the seven pretrain-only checkpoints with selective kernels.
Learning Transferable Visual Models From Natural Language Supervision
BYOL We use the recently released model weights of BYOL (Grill et al., 2020), specifically their 50x1 and 200x2 Figure 19. Two example images from the Rendered SST2 dataset checkpoints.
Learning Transferable Visual Models From Natural Language Supervision
Momentum Contrast (MoCo) We include the MoCo-v1 (He et al., 2020) and the MoCo-v2 (Chen et al., 2020d) checkpoints.
Learning Transferable Visual Models From Natural Language Supervision
VirTex We use the pretrained model of VirTex (Desai & Johnson, 2020). We note that VirTex has a similar model design to CLIP-AR but is trained on a 1000x smaller dataset of high-quality captions from MSCOCO.
Learning Transferable Visual Models From Natural Language Supervision
ResNet We add the original ResNet checkpoints released by (He et al., 2016b), namely ResNet-50, ResNet-101, and ResNet152.
Learning Transferable Visual Models From Natural Language Supervision
We use image features taken from the penultimate layer of each model, ignoring any classification layer provided. For CLIP-ViT models, we used the features before the linear projection to the embedding space, which corresponds to I f in Figure 3. We train a logistic regression classifier using scikit-learn's L-BFGS implementation, with maximum 1,000 iterations, and report the corresponding metric for each dataset. We determine the L2 regularization strength λ using a hyperparameter sweep on the validation sets over the range between 10 −6 and 10 6 , with 96 logarithmically spaced steps. To save compute required for the sweeps, we perform a parametric binary search that starts with λ = [10 −6 , 10 −4 , 10 −2 , 1, 10 2 , 10 4 , 10 6 ] and iteratively halves the interval around the peak until it reaches a resolution of 8 steps per decade. The hyperparameter sweeps are performed on a validation split of each dataset. For the datasets that contain a validation split in addition to a test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, we split the training dataset to perform the hyperparameter search. For the final result, we combine the validation split back with the training split and report the performance on the unused split.
Learning Transferable Visual Models From Natural Language Supervision
The individual linear probe scores are provided in Table 10 and plotted in Figure 20. The best-performing CLIP model, using ViT-L/14 archiecture and 336-by-336 pixel images, achieved the state of the art in 21 of the 27 datasets, i.e. included in the Clopper-Pearson 99.5% confidence interval around each dataset's top score. For many datasets, CLIP performs significantly better than other models, demonstrating the advantage of natural language supervision over traditional pre-training approaches based on image classification. See Section 3.2 for more discussions on the linear probe results.       Figure 21. Visualization of predictions from 36 CLIP zero-shot classifiers. All examples are random with the exception of reselecting Hateful Memes to avoid offensive content. The predicted probability of the top 5 classes is shown along with the text used to represent the class. When more than one template is used, the first template is shown. The ground truth label is colored green while an incorrect prediction is colored orange.  Table 11. Zero-shot performance of CLIP models over 27 datasets. 
Learning Transferable Visual Models From Natural Language Supervision
To provide a qualitative summary / overview of CLIP's zeroshot performance we visualize a randomly selected prediction for 36 different zero-shot CLIP classifiers in Figure  21. In addition, Table 11 and Figure 22 show the individual zero-shot performance scores for each dataset.
Learning Transferable Visual Models From Natural Language Supervision
Our early attempts at duplicate detection and analysis used nearest neighbors in the model's learned embedding space. While it is intuitive to use a model's own notion of similarity, we encountered issues. We found the model's feature space is weighted very heavily towards semantic similarity. Many false positives occurred due to distinct objects that would be described similarly (soccer balls, flowers of the same species, etc...) having almost perfect similarity. We also observed the model was quite poor at assigning certain kinds of near-duplicates high similarity scores. We noticed repeatedly that images with high-frequency textures (such as fur or stripe patterns) pre-processed by different resizing algorithms (nearest neighbor vs bi-linear) could have surprisingly low similarity. This resulted in many false negatives.
Learning Transferable Visual Models From Natural Language Supervision
We built our own near-duplicate detector to fix this issue. We created a synthetic data augmentation pipeline that combined a variety of common image manipulations. The augmentation pipeline combines random cropping and zooming, aspect ratio distortion, downsizing and upscaling to different resolutions, minor rotations, jpeg compression, and HSV color jitter. The pipeline also randomly selects from different interpolation algorithms for all relevant steps. We then trained a model to maximize the similarity of an image and its transformed variant while minimizing similarity to all other images in a training batch. We used the same n-pair / InfoNCE loss as CLIP but with a fixed temperature of 0.07.
Learning Transferable Visual Models From Natural Language Supervision
We selected a ResNet-50 as the model architecture. We modified the base ResNet-50 with the anti-alias improvements from (Zhang, 2019) and used weight norm (Salimans & Kingma, 2016) instead of batch norm (Ioffe & Szegedy, 2015) to avoid leaking information about duplicates via batch statistics -a problem previously noted in (Henaff, 2020). We also found the GELU activation function (Hendrycks & Gimpel, 2016) to perform better for this task. We trained the model with a total batch size of 1,712 for approximately 30 million images sampled from our pretraining dataset. At the end of training it achieves nearly 100% accuracy on its proxy training task. 
Learning Transferable Visual Models From Natural Language Supervision
To study whether our custom dataset is critical to the performance of CLIP, we trained a model on a filtered subset of the YFCC100M dataset (details described in Section 2.2) and compared its performance to the same model trained on an equally sized subset of WIT. We train each model for 32 epochs at which point transfer performance begins to plateau due to overfitting. Results are shown in Table 12. Across our whole eval suite, YFCC and WIT perform similarly on average for both zero-shot and linear probe settings. However, performance on specific fine-grained classification datasets can vary widely -sometimes by over 10%. Our speculation is that these differences in performance reflect the relative density of relevant data in each pre-training dataset. For instance, pre-training on YFCC100M, which might contain many photos of birds and flowers (common subjects for photographers), results in better performance on Birdsnap and Flowers102, while pre-training on WIT results in better car and pet classifiers (which appear common in our dataset).
Learning Transferable Visual Models From Natural Language Supervision
Overall, these results are encouraging as they suggest our approach can use any reasonably filtered collection of paired (text, image) data. This mirrors recent work which reported positive results using the same contrastive pre-training objective on the relatively different domain of medical imaging . It also is similar to the findings of noisy student self-training which reported only slight improvements when using their JFT300M dataset over YFCC100M (Xie et al., 2020). We suspect the major advantage of our dataset over the already existing YFCC100M is its much larger size.
Learning Transferable Visual Models From Natural Language Supervision
Finally, we caution that WIT includes this filtered subset of YFCC100M. This could result in our ablation underestimating the size of performance differences between YFCC100M and the rest of WIT. We do not think this is likely as YFCC100M is only 3.7% of the overall WIT data blend and it did not noticeably change the performance of models when it was added to the existing data blend during the creation of WIT.
Learning Transferable Visual Models From Natural Language Supervision
Due to the large variety of datasets and experiments considered in this work, the main body focuses on summarizing and analyzing overall results. In the following subsections we report details of performance for specific groups of tasks, datasets, and evaluation settings.
Learning Transferable Visual Models From Natural Language Supervision
CLIP pre-trains for the task of image-text retrieval on our noisy web-scale dataset. Although the focus of this paper is on representation learning and task learning for the purpose of transfer to a wide variety of downstream datasets, validating that CLIP is able to achieve high transfer performance transfer on exactly what it is pre-trained for is an important sanity check / proof of concept. In Table 13 we check the zero-shot transfer performance of CLIP for both text and image retrieval on the Flickr30k and MSCOCO datsets. Zero-shot CLIP matches or outperforms all prior zero-shot results on these two datasets. Zero-shot CLIP is also competitive with the current overall SOTA for the task of text retrieval on Flickr30k. On image retrieval, CLIP's performance relative to the overall state of the art is noticeably lower. However, zero-shot CLIP is still competitive with a fine-tuned Unicoder-VL. On the larger MS-COCO dataset fine-tuning improves performance significantly and zero-shot CLIP is not competitive with the most recent work. For both these datasets we prepend the prompt "a photo of" to the description of each image which we found boosts CLIP's zero-shot R@1 performance between 1 and 2 points.
Learning Transferable Visual Models From Natural Language Supervision
Although visualizations have shown that ImageNet models contain features that respond to the presence of text in an image (Zeiler & Fergus, 2014), these representations are not sufficiently fine-grained to use for the task of optical character recognition (OCR). To compensate, models are augmented with the outputs of custom OCR engines and features to boost performance on tasks where this capability is required (Singh et al., 2019;Yang et al., 2020). Early during the development of CLIP, we noticed that CLIP began to learn primitive OCR capabilities which appeared to steadily improve over the course of the project. To evaluate this qualitatively noticed behavior, we measured performance on 5 datasets requiring the direct and indirect use of OCR. Three of these datasets MNIST (LeCun), SVHN (Netzer et al., 2011), andIIIT5K (Mishra et al., 2012) directly check the ability of a model to perform low-level character and word recognition, while Hateful Memes (Kiela et al., 2020) and SST-2 (Socher et al., 2013) check the ability of a model to use OCR to perform a semantic task. Results are reported in Table 14.
Learning Transferable Visual Models From Natural Language Supervision
CLIP's performance is still highly variable and appears to be sensitive to some combination of the domain (rendered or natural images) and the type of text to be recognized (numbers or words). CLIP's OCR performance is strongest Hateful Memes and SST-2 -datasets where the text is digitally rendered and consists mostly of words. On IIIT5K, which is natural images of individually cropped words, zero-shot CLIP performs a bit more respectively and its performance is similar to Jaderberg et al. (2014) early work combining deep learning and structured prediction to perform openvocabulary OCR. However, performance is noticeably lower on two datasets involving recognition of hand written and street view numbers. CLIP's 51% accuracy on full number SVHN is well below any published results. Inspection suggests CLIP struggles with repeated characters as well as the low resolution and blurry images of SVHN. CLIP's zeroshot MNIST performance is also poor and is outperformed by supervised logistic regression on raw pixels, one of the simplest possible machine learning baselines.
Learning Transferable Visual Models From Natural Language Supervision
SST-2 is a sentence level NLP dataset which we render into images. We include SST-2 in order to check whether CLIP is able to convert low level OCR capability into a higher level representation. Fitting a linear classifier on CLIP's representation of rendered sentences achives 80.5% accuracy. This is on par with the 80% accuracy of a continuous bag of words baseline using GloVe word vectors pre-trained on 840 billion tokens (Pennington et al., 2014). While this is a simple NLP baseline by today's standard, and well below the 97.5% of the current SOTA, it is encouraging to see that CLIP is able to turn an image of rendered text into a non-trivial sentence level representation. Fully supervised CLIP is also surprisingly strong on Hateful Meme detection, where CLIP is only 0.7 points behind the current single model SOTA and several points above the best baseline from the original paper. Similar to SST-2, these other results on Hateful Memes use the ground truth text which CLIP does not have access to. Finally, we note that zero-shot CLIP outperforms the best results using fully supervised linear probes across all other 56 models included in our evaluation suite. This suggests CLIP's OCR capability is at least somewhat unique compared to existing work on self-supervised and supervised representation learning. 
Learning Transferable Visual Models From Natural Language Supervision
For the purpose of learning, a potentially important aspect of natural language is its ability to express, and therefore supervise, an extremely wide set of concepts. A CLIP model, since it is trained to pair semi-arbitrary text with images, is likely to receive supervision for a wide range of visual concepts involving both common and proper nouns, verbs, and adjectives. ImageNet-1K, by contrast, only labels common nouns. Does the lack of broader supervision in ImageNet result in weaker transfer of ImageNet models to tasks involving the recognition of visual concepts that are not nouns?
Learning Transferable Visual Models From Natural Language Supervision
To investigate this, we measure and compare the performance of CLIP and ImageNet models on several video UCF101 K700 RareAct  (Alayrac et al., 2020) action classification datasets which measure the ability of a model to recognize verbs. In Table 15 we report results on UCF-101 (Soomro et al., 2012) and Kinetics-700 (Carreira et al., 2019), two common datasets for the task. Unfortunately, our CPU based linear classifier takes a prohibitively long time to evaluate on a video dataset due to the very large number of training frames. To deal with this, we aggressively sub-sample each video to only a single center frame, effectively turning it into an image classification dataset. As a result, our reported performance in a linear evaluation setting likely under estimates performance by a moderate amount.  (Touvron et al., 2019) Despite this handicap, CLIP features transfer surprisingly well to this task. CLIP matches the best prior result on UCF-101 in a linear probe evaluation setting and also outperforms all other models in our evaluation suite. On Kinetics-700, CLIP also outperforms the fine-tuned I3D baseline from the original paper. Since it does not require a training stage, we report CLIP's zero-shot performance when averaging predictions across all frames. CLIP also performs well in this setting and on Kinetics-700 its performance is within 1% of the fully supervised I3D baseline which is trained on 545000 labeled videos. Encouraged by these results, we also measure CLIP's performance on the recently introduced RareAct dataset (Miech et al., 2020a) which was designed to measure zero-shot recognition of unusual actions like "hammering a phone" and "drilling an egg". CLIP improves over the prior state of the art, a S3D model trained on automatically extracted captions from 100 million instructional videos, by 10 points.
Learning Transferable Visual Models From Natural Language Supervision
While CLIP has encouragingly strong performance on the task of action recognition, we note that there are many differences between the models being compared beyond just their form of supervision such as model architecture, training data distribution, dataset size, and compute used. Further work is needed to more precisely determine what specific design decisions contribute to achieving high performance on this task. 
Learning Transferable Visual Models From Natural Language Supervision
Another behavior we noticed during the development of CLIP was its ability to recognize many places and locations.
Learning Transferable Visual Models From Natural Language Supervision
To quantify this we created the Country211 dataset as described in Appendix A and report results on it throughout the paper. However it is a new benchmark so to compare with prior work on geolocalization we also report results on the IM2GPS test set from Hays & Efros (2008) in Table  17. Since IM2GPS is a regression benchmark, we guess the GPS coordinates of the nearest image in a set of reference images using CLIP's embedding space. This is not a zeroshot result since it uses nearest-neighbor regression. Despite querying only 1 million images, which is much less than prior work, CLIP performs similarly to several task specific models. It is not, however, competitive with the current state of the art.
Learning Transferable Visual Models From Natural Language Supervision
Section 3.3 provides a high level summary and analysis of ImageNet-related robustness results. We briefly provide some additional numerical details in this appendix. Performance results per dataset are provided in Table 16 and compared with the current state of the art results reported in Taori et al. (2020)'s evaluation suite. Zero-shot CLIP improves the state of the art on 5 of the 7 datasets, ImageNet-R, ObjectNet, ImageNet-Sketch, ImageNet-Vid, and Youtube-BB. CLIP's improvements are largest on ImageNet-Vid and Youtube-BB due to its flexible zero-shot capability and on ImageNet-R, which likely reflects CLIP's pre-training distribution including significant amounts of creative content. A similar behavior has been documented for the Instagram pre-trained ResNeXt models as discussed in Taori et al.  
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Vision-language pre-training has recently received tremendous success on various multimodal downstream tasks. However, existing methods have two major limitations:
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(1) Model perspective: most methods either adopt an encoder-based model (Radford et al., 2021;Li et al., 2021a), or an encoder-decoder (Cho et al., 2021;Wang et al., 2021) model. However, encoder-based models are less straightforward to directly transfer to text generation tasks (e.g. image captioning), whereas encoder-decoder models have not been successfully adopted for image-text retrieval tasks.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(2) Data perspective: most state-of-the-art methods (e.g., CLIP (Radford et al., 2021), ALBEF (Li et al., 2021a), SimVLM ) pre-train on image-text pairs collected from the web. Despite the performance gain obtained by scaling up the dataset, our paper shows that the noisy web text is suboptimal for vision-language learning.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
To this end, we propose BLIP: Bootstrapping Language-Image Pre-training for unified vision-language understanding and generation. BLIP is a new VLP framework which enables a wider range of downstream tasks than existing methods. It introduces two contributions from the model and data perspective, respectively: (a) Multimodal mixture of Encoder-Decoder (MED): a new model architecture for effective multi-task pre-training and flexible transfer learning. An MED can operate either as a unimodal encoder, or an image-grounded text encoder, or an image-grounded text decoder. The model is jointly pre-trained with three vision-language objectives: imagetext contrastive learning, image-text matching, and imageconditioned language modeling.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(b) Captioning and Filtering (CapFilt): a new dataset boostrapping method for learning from noisy image-text pairs. We finetune a pre-trained MED into two modules: a captioner to produce synthetic captions given web images, and a filter to remove noisy captions from both the original web texts and the synthetic texts.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We perform extensive experiments and analysis, and make the following key observations.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• We show that the captioner and the filter work together to achieve substantial performance improvement on various downstream tasks by bootstrapping the captions. We also find that more diverse captions yield larger gains.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• BLIP achieves state-of-the-art performance on a wide range of vision-language tasks, including image-text re-  Figure 2. Pre-training model architecture and objectives of BLIP (same parameters have the same color). We propose multimodal mixture of encoder-decoder, a unified vision-language model which can operate in one of the three functionalities: (1) Unimodal encoder is trained with an image-text contrastive (ITC) loss to align the vision and language representations.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(2) Image-grounded text encoder uses additional cross-attention layers to model vision-language interactions, and is trained with a image-text matching (ITM) loss to distinguish between positive and negative image-text pairs. (3) Image-grounded text decoder replaces the bi-directional self-attention layers with causal self-attention layers, and shares the same cross-attention layers and feed forward networks as the encoder. The decoder is trained with a language modeling (LM) loss to generate captions given images.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
trieval, image captioning, visual question answering, visual reasoning, and visual dialog. We also achieve state-ofthe-art zero-shot performance when directly transferring our models to two video-language tasks: text-to-video retrieval and videoQA.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Vision-language pre-training (VLP) aims to improve performance of downstream vision and language tasks by pretraining the model on large-scale image-text pairs. Due to the prohibitive expense of acquiring human-annotated texts, most methods Li et al., 2020;2021a;Wang et al., 2021;Radford et al., 2021) use image and alt-text pairs crawled from the web (Sharma et al., 2018;Changpinyo et al., 2021;Jia et al., 2021), Despite the use of simple rule-based filters, noise is still prevalent in the web texts. However, the negative impact of the noise has been largely overlooked, shadowed by the performance gain obtained from scaling up the dataset. Our paper shows that the noisy web texts are suboptimal for vision-language learning, and proposes CapFilt that utilizes web datasets in a more effective way.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
There have been many attempts to unify various vision and language tasks into a single framework (Zhou et al., 2020;Cho et al., 2021;Wang et al., 2021). The biggest challenge is to design model architectures that can perform both understanding-based tasks (e.g. image-text retrieval) and generation-based tasks (e.g. image captioning). Neither encoder-based models (Li et al., 2021a;Radford et al., 2021) nor encoder-decoder models (Cho et al., 2021;Wang et al., 2021) can excel at both types of tasks, whereas a single unified encoder-decoder (Zhou et al., 2020) also limits the model's capability. Our proposed multimodal mixture of encoder-decoder model offers more flexibility and better performance on a wide range of downstream tasks, in the meantime keeping the pre-training simple and efficient.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Knowledge distillation (KD) (Hinton et al., 2015) aims to improve the performance of a student model by distilling knowledge from a teacher model. Self-distillation is a special case of KD where the teacher and student have equal sizes. It has been shown to be effective for image classification (Xie et al., 2020), and recently for VLP (Li et al., 2021a). Different from mostly existing KD methods which simply enforce the student to have the same class predictions as the teacher, our proposed CapFilt can be interpreted as a more effective way to perform KD in the context of VLP, where the captioner distills its knowledge through semantically-rich synthetic captions, and the filter distills its knowledge by removing noisy captions.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
While data augmentation (DA) has been widely adopted in computer vision (Shorten & Khoshgoftaar, 2019), DA for language tasks is less straightforward. Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al., 2020;Anaby-Tavor et al., 2020;Puri et al., 2020;Yang et al., 2020). Different from these methods which focus on the low-resource language-only tasks, our method demonstrates the advantage of synthetic captions in large-scale vision-language pre-training.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We propose BLIP, a unified VLP framework to learn from noisy image-text pairs. This section first introduces our new model architecture MED and its pre-training objectives, and then delineates CapFilt for dataset bootstrapping.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We employ a visual transformer (Dosovitskiy et al., 2021) as our image encoder, which divides an input image into patches and encodes them as a sequence of embeddings, with an additional [CLS] token to represent the global image feature. Compared to using pre-trained object detectors for visual feature extraction , using a ViT is more computation-friendly and has been adopted by the more recent methods (Li et al., 2021a;Kim et al., 2021).
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In order to pre-train a unified model with both understanding and generation capabilities, we propose multimodal mixture of encoder-decoder (MED), a multi-task model which can operate in one of the three functionalities:
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(1) Unimodal encoder, which separately encodes image and text. The text encoder is the same as BERT (Devlin et al., 2019), where a [CLS] token is appended to the beginning of the text input to summarize the sentence.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(2) Image-grounded text encoder, which injects visual information by inserting one additional cross-attention (CA) layer between the self-attention (SA) layer and the feed forward network (FFN) for each transformer block of the text encoder. A task-specific [Encode] token is appended to the text, and the output embedding of [Encode] is used as the multimodal representation of the image-text pair.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(3) Image-grounded text decoder, which replaces the bidirectional self-attention layers in the image-grounded text encoder with causal self-attention layers. A [Decode] token is used to signal the beginning of a sequence, and an end-of-sequence token is used to signal its end.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We jointly optimize three objectives during pre-training, with two understanding-based objectives and one generationbased objective. Each image-text pair only requires one forward pass through the computational-heavier visual transformer, and three forward passes through the text transformer, where different functionalities are activated to compute the three losses as delineated below.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Image-Text Contrastive Loss (ITC) activates the unimodal encoder. It aims to align the feature space of the visual trans-former and the text transformer by encouraging positive image-text pairs to have similar representations in contrast to the negative pairs. It has been shown to be an effective objective for improving vision and language understanding (Radford et al., 2021;Li et al., 2021a). We follow the ITC loss by Li et al. (2021a), where a momentum encoder is introduced to produce features, and soft labels are created from the momentum encoder as training targets to account for the potential positives in the negative pairs.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Image-Text Matching Loss (ITM) activates the imagegrounded text encoder. It aims to learn image-text multimodal representation that captures the fine-grained alignment between vision and language. ITM is a binary classification task, where the model uses an ITM head (a linear layer) to predict whether an image-text pair is positive (matched) or negative (unmatched) given their multimodal feature. In order to find more informative negatives, we adopt the hard negative mining strategy by Li et al. (2021a), where negatives pairs with higher contrastive similarity in a batch are more likely to be selected to compute the loss.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Language Modeling Loss (LM) activates the imagegrounded text decoder, which aims to generate textual descriptions given an image. It optimizes a cross entropy loss which trains the model to maximize the likelihood of the text in an autoregressive manner. We apply a label smoothing of 0.1 when computing the loss. Compared to the MLM loss that has been widely-used for VLP, LM enables the model with the generalization capability to convert visual information into coherent captions.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In order to perform efficient pre-training while leveraging multi-task learning, the text encoder and text decoder share all parameters except for the SA layers. The reason is that the differences between the encoding and decoding tasks are best captured by the SA layers. In particular, the encoder employs bi-directional self-attention to build representations for the current input tokens, while the decoder employs causal self-attention to predict next tokens. On the other hand, the embedding layers, CA layers and FFN function similarly between encoding and decoding tasks, therefore sharing these layers can improve training efficiency while benefiting from multi-task learning,
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Due to the prohibitive annotation cost, there exist a limited number of high-quality human-annotated image-text pairs {(I h , T h )} (e.g., COCO (Lin et al., 2014)). Recent work (Li et al., 2021a;Wang et al., 2021) utilizes a much larger number of image and alt-text pairs {(I w , T w )} that are automatically collected from the web. However, the alt-texts often do not accurately describe the visual content of the images, making them a noisy signal that is suboptimal for learning vision-language alignment.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Multimodal Mixture of Encoder-Decoder
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
To model
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
To data ! : web images " : human-annotated images ! : web texts ! : filtered web texts # : synthetic texts # : filtered synthetic texts " : human-annotated texts Figure 3. Learning framework of BLIP. We introduce a captioner to produce synthetic captions for web images, and a filter to remove noisy image-text pairs. The captioner and filter are initialized from the same pre-trained model and finetuned individually on a small-scale human-annotated dataset. The bootstrapped dataset is used to pre-train a new model.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We propose Captioning and Filtering (CapFilt), a new method to improve the quality of the text corpus. Figure 3 gives an illustration of CapFilt. It introduces two modules: a captioner to generate captions given web images, and a filter to remove noisy image-text pairs. Both the captioner and the filter are initialized from the same pre-trained MED model, and finetuned individually on the COCO dataset. The finetuning is a lightweight procedure.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Specifically, the captioner is an image-grounded text decoder. It is finetuned with the LM objective to decode texts given images. Given the web images I w , the captioner generates synthetic captions T s with one caption per image.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
The filter is an image-grounded text encoder. It is finetuned with the ITC and ITM objectives to learn whether a text matches an image. The filter removes noisy texts in both the original web texts T w and the synthetic texts T s , where a text is considered to be noisy if the ITM head predicts it as unmatched to the image. Finally, we combine the filtered image-text pairs with the human-annotated pairs to form a new dataset, which we use to pre-train a new model.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In this section, we first introduce pre-training details. Then we provide a detailed experimental analysis on our method.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Our models are implemented in PyTorch (Paszke et al., 2019) and pre-trained on two 16-GPU nodes. The image transformer is initialized from ViT pre-trained on Ima-geNet (Touvron et al., 2020;Dosovitskiy et al., 2021), and the text transformer is initialized from BERT base (Devlin et al., 2019). We explore two variants of ViTs: ViT-B/16 and ViT-L/16. Unless otherwise specified, all results reported in this paper as "BLIP" uses ViT-B. We pre-train the model for 20 epochs using a batch size of 2880 (ViT-B) / 2400 (ViT-L). We use AdamW (Loshchilov & Hutter, 2017) optimizer with a weight decay of 0.05. The learning rate is warmed-up to 3e-4 (ViT-B) / 2e-4 (ViT-L) and decayed linearly with a rate of 0.85. We take random image crops of resolution 224 × 224 during pre-training, and increase the image resolution to 384 × 384 during finetuning. We use the same pre-training dataset as Li et al. (2021a) with 14M images in total, including two human-annotated datasets (COCO and Visual Genome (Krishna et al., 2017)), and three web datasets (Conceptual Captions (Changpinyo et al., 2021), Conceptual 12M (Changpinyo et al., 2021), SBU captions (Ordonez et al., 2011)). We also experimented with an additional web dataset, LAION (Schuhmann et al., 2021), which contains 115M images with more noisy texts 1 . More details about the datasets can be found in the appendix.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In Table 1, we compare models pre-trained on different datasets to demonstrate the efficacy of CapFilt on downstream tasks, including image-text retrieval and image captioning with finetuned and zero-shot settings.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
When only the captioner or the filter is applied to the dataset with 14M images, performance improvement can be observed. When applied together, their effects compliment each other, leading to substantial improvements compared to using the original noisy web texts.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
CapFilt can further boost performance with a larger dataset and a larger vision backbone, which verifies its scalability in both the data size and the model size. Furthermore, by using a large captioner and filter with ViT-L, performance of the base model can also be improved.     Table 3. Comparison between different parameter sharing strategies for the text encoder and decoder during pre-training.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In Figure 4, we show some example captions and their corresponding images, which qualitatively demonstrate the effect of the captioner to generate new textual descriptions, and the filter to remove noisy captions from both the original web texts and the synthetic texts. More examples can be found in the appendix.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In CapFilt, we employ nucleus sampling (Holtzman et al., 2020) to generate synthetic captions. Nucleus sampling is a stochastic decoding method, where each token is sampled from a set of tokens whose cumulative probability mass exceeds a threshold p (p = 0.9 in our experiments). In Table 2, we compare it with beam search, a deterministic decoding method which aims to generate captions with the highest probability. Nucleus sampling leads to evidently better performance, despite being more noisy as suggested by a higher noise ratio from the filter. We hypothesis that the reason is that nucleus sampling generates more diverse and surprising captions, which contain more new information that the model could benefit from. On the other hand, beam search tends to generate safe captions that are common in the dataset, hence offering less extra knowledge.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
During pre-training, the text encoder and decoder share all parameters except for the self-attention layers. In Table 3 layers except for SA leads to better performance compared to not sharing, while also reducing the model size thus improveing training efficiency. If the SA layers are shared, the model's performance would degrade due to the conflict between the encoding task and the decoding task.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
During CapFilt, the captioner and the filter are end-to-end finetuned individually on COCO. In Table 4, we study the effect if the captioner and filter share parameters in the same way as pre-training. The performance on the downstream tasks decreases, which we mainly attribute to confirmation bias. Due to parameter sharing, noisy captions produced by the captioner are less likely to be filtered out by the filter, as indicated by the lower noise ratio (8% compared to 25%).
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In this section, we compare BLIP to existing VLP methods on a wide range of vision-language downstream tasks 2 . Next we briefly introduce each task and finetuning strategy. More details can be found in the appendix.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We evaluate BLIP for both image-to-text retrieval (TR) and text-to-image retrieval (IR) on COCO and Flickr30K (Plummer et al., 2015) datasets. We finetune the pre-trained model using ITC and ITM losses. To enable faster inference speed, we follow Li et al. (2021a) and first select k candidates based on the image-text feature similarity, and then rerank the selected candidates based on their pairwise ITM scores. We set k = 256 for COCO and k = 128 for Flickr30K.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
As shown in Table 5, BLIP achieves substantial performance improvement compared with existing methods. Using the same 14M pre-training images, BLIP outperforms the previous best model ALBEF by +2.7% in average recall@1 on COCO. We also perform zero-shot retrieval by directly transferring the model finetuned on COCO to Flickr30K. The result is shown in Table 6, where BLIP also outperforms existing methods by a large margin.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We consider two datasets for image captioning: No-Caps (Agrawal et al., 2019) and COCO, both evaluated using the model finetuned on COCO with the LM loss. Similar as Wang et al. (2021), we add a prompt "a picture of" at the beginning of each caption, which leads to slightly better results. As shown in  Table 7. Comparison with state-of-the-art image captioning methods on NoCaps and COCO Caption. All methods optimize the crossentropy loss during finetuning. C: CIDEr, S: SPICE, B@4: BLEU@4. BLIPCapFilt-L is pre-trained on a dataset bootstrapped by captioner and filter with ViT-L. VinVL † and LEMON † require an object detector pre-trained on 2.5M images with human-annotated bounding boxes and high resolution (800×1333) input images. SimVLM huge uses 13× more training data and a larger vision backbone than ViT-L.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Question Encoder  200M images. Note that LEMON requires a computationalheavy pre-trained object detector and higher resolution (800×1333) input images, leading to substantially slower inference time than the detector-free BLIP which uses lower resolution (384×384) input images.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
VQA (Antol et al., 2015) requires the model to predict an answer given an image and a question. Instead of formulating VQA as a multi-answer classification task (Chen et al.,  Li et al., 2020), we follow Li et al. (2021a) and consider it as an answer generation task, which enables open-ended VQA. As shown in Figure 5(a), during finetuning, we rearrange the pre-trained model, where an image-question is first encoded into multimodal embeddings and then given to an answer decoder. The VQA model is finetuned with the LM loss using ground-truth answers as targets.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
The results are shown in Table 8. Using 14M images, BLIP outperforms ALBEF by +1.64% on the test set. Using 129M images, BLIP achieves better performance than SimVLM which uses 13× more pre-training data and a larger vision backbone with an additional convolution stage.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
NLVR 2 (Suhr et al., 2019) asks the model to predict whether a sentence describes a pair of images. In order to enable rea-  Table 9. Comparison with state-of-the-art methods on VisDial v1.0 validation set. VD-ViLBERT † (Murahari et al., 2020) pre-trains ViLBERT (Lu et al., 2019) with additional VQA data.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
soning over two images, we make a simple modification to our pre-trained model which leads to a more computationalefficient architecture than previous approaches (Li et al., 2021a;Wang et al., 2021). As shown in Figure 5(b), for each transformer block in the image-grounded text encoder, there exist two cross-attention layers to process the two input images, and their outputs are merged and fed to the FFN. The two CA layers are intialized from the same pre-trained weights. The merge layer performs simple average pooling in the first 6 layers of the encoder, and performs concatenation followed by a linear projection in layer 6-12. An MLP classifier is applied on the output embedding of the [Encode] token. As shown in Table 8, BLIP outperforms all existing methods except for ALBEF which performs an extra step of customized pre-training. Interestingly, performance on NLVR 2 does not benefit much from additional web images, possibly due to the domain gap between web data and downstream data.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
VisDial (Das et al., 2017) extends VQA in a natural conversational setting, where the model needs to predict an answer not only based on the image-question pair, but also considering the dialog history and the image's caption. We follow the discriminative setting where the model ranks a pool of answer candidates (Gan et al., 2019;Wang et al., 2020;Murahari et al., 2020). As shown in Figure 5(c), we concatenate image and caption embeddings, and pass them to the dialog encoder through cross-attention. The dialog encoder is trained with the ITM loss to discriminate whether the answer is true or false for a question, given the entire dialog history and the image-caption embeddings. As shown in Table 9, our method achieves state-of-the-art performance on VisDial v1.0 validation set.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Our image-language model has strong generalization ability to video-language tasks. In Table 10 and Table 11, we perform zero-shot transfer to text-to-video retrieval and video question answering, where we directly evaluate the models trained on COCO-retrieval and VQA, respectively. To process video input, we uniformly sample n frames per video (n = 8 for retrieval and n = 16 for QA), and concatenate the frame features into a single sequence. Note that this simple approach ignores all temporal information.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
ActBERT (Zhu & Yang, 2020) 8.6 23.4 33.1 36 SupportSet (Patrick et al., 2021) 8.7 23.0 31.1 31 MIL-NCE (Miech et al., 2020) 9.9 24.0 32.4 29.5 VideoCLIP (Xu et al., 2021) 10.4 22.2 30.0 -FiT (Bain et al., 2021) 18.7 39.5 51.6 10 BLIP 43.3 65.6 74.7 2 finetuning ClipBERT  22.0 46.8 59.9 6 VideoCLIP (Xu et al., 2021) 30.9 55.4 66.8 -   35.6 36.1 VQA-T  41.5 46. 3   Table 11. Comparisons with state-of-the-art methods for video question answering. We report top-1 test accuracy on two datasets.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Despite the domain difference and lack of temporal modeling, our models achieve state-of-the-art performance on both video-language tasks. For text-to-video retrieval, zeroshot BLIP even outperforms models finetuned on the target video dataset by +12.4% in recall@1. Further performance improvement can be achieved if the BLIP model is used to initialize a video-language model with temporal modeling (e.g. replace our ViT with a TimeSformer (Bertasius et al., 2021)) and finetuned on video data.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In this section, we provide additional ablation experiments on CapFilt.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Improvement with CapFilt is not due to longer training.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Since the bootstrapped dataset contains more texts than the original dataset, training for the same number of epochs takes longer with the bootstrapped dataset. To verify that the effectiveness of CapFilt is not due to longer training, we replicate the web text in the original dataset so that it has the same number of training samples per epoch as the bootstrapped dataset. As shown in from the previous pre-trained model, using the bootstrapped dataset. Table 13 hows that continue training does not help. This observation agrees with the common practice in knowledge distillation, where the student model cannot be initialized from the teacher.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
We propose BLIP, a new VLP framework with stateof-the-art performance on a wide range of downstream vision-language tasks, including understanding-based and generation-based tasks. BLIP pre-trains a multimodal mixture of encoder-decoder model using a dataset bootstrapped from large-scale noisy image-text pairs by injecting diverse synthetic captions and removing noisy captions. Our bootstrapped dataset are released to facilitate future visionlanguage research.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
There are a few potential directions that can further enhance the performance of BLIP: (1) Multiple rounds of dataset bootstrapping;
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
(2) Generate multiple synthetic captions per image to further enlarge the pre-training corpus; (3) Model ensemble by training multiple different captioners and filters and combining their forces in CapFilt. We hope that our paper motivates future work to focus on making improvements in both the model aspect and the data aspect, the bread and butter of vision-language research. VQA. We experiment with the VQA2.0 dataset (Goyal et al., 2017), which contains 83k/41k/81k images for training/validation/test. Following Li et al. (2021a), we use both training and validation splits for training, and include additional training samples from Visual Genome. During inference on VQA, we use the decoder to rank the 3,128 candidate answers (Li et al., 2021a;Kim et al., 2018).
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
NLVR 2 . We conduct experiment on the official split (Suhr et al., 2019).
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
VisDial. We finetune on the training split of VisDial v1.0 and evaluate on its validation set. 
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
In Figure 6, we show additional examples of images and texts where the web captions are filtered out, and the synthetic captions are kept as clean training samples.  Table 15. Statistics of the pre-training datasets.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
! : "a week spent at our rented beach house in Sandbridge" " : "an outdoor walkway on a grass covered hill" ! : "that's what a sign says over the door" " : "the car is driving past a small old building" ! : "hand held through the glass in my front bedroom window" " : "a moon against the night sky with a black background" ! : "stunning sky over walney island, lake district, july 2009" " : "an outdoor walkway on a grass covered hill" ! : "living in my little white house" " : "a tiny white flower with a bee in it" ! : "the pink rock from below" " : "some colorful trees that are on a hill in the mountains" Figure 6. Examples of the web text Tw and the synthetic text Ts. Green texts are accepted by the filter, whereas red texts are rejected.