How has prior work study and evaluate robustness of multimodal models?
Robustness of Multimodal Models: Previous
studies independently introduce unimodal pertur-
bations in the visual or textual part of the input to
study multimodal robustness. This could involve
introducing imperceptible adversarial noise in the
images and independently modifying the text using
the augmentation approaches discussed earlier (Li
et al., 2021a; Ribeiro et al., 2020; Wei and Zou,
2019). For instance, Chen et al. (2020) synthesize
counterfactual samples of multimodal data using
language models to modify the text. To ensure the
preservation of the semantic meaning in the aug-
mented text, Sheng et al. (2021) and Li et al.
(2021b) employ humans to perturb the textual ques-
tions to fool the state-of-the-art models for Visual
Question Answering (Antol et al., 2015). In a step
towards using cross-modal interactions in image-
text data to generate realistic variations, Verma et al.
(2022) proposed adding relevant information from
the image to expand the original textual description,
and assess the robustness of multimodal classifiers.
Our work proposes a different approach to leverage
cross-modal associations to augment multimodal data. Instead of expanding the original text, we
insert attributes of objects in the image that are
also mentioned in the corresponding text to modify
the original text. Additionally, our work considers
more multimodal tasks by studying text-to-image
retrieval and cross-modal entailment.
Human-Adversarial Visual Question Answering
Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, it is clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against human-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model's predicted answer is incorrect. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.
[{"first":"Sasha","last":"Sheng","middle":[]},{"first":"Amanpreet","last":"Singh","middle":[]},{"first":"Vedanuj","last":"Goswami","middle":[]},{"first":"Jose","last":"Magana","middle":["Alberto","Lopez"]},{"first":"Wojciech","last":"Galuba","middle":[]},{"first":"Devi","last":"Parikh","middle":[]},{"first":"Douwe","last":"Kiela","middle":[]}]
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Benefiting from large-scale pre-training, we have witnessed significant performance boost on the popular Visual Question Answering (VQA) task. Despite rapid progress, it remains unclear whether these state-of-the-art (SOTA) models are robust when encountering examples in the wild. To study this, we introduce Adversarial VQA, a new large-scale VQA benchmark, collected iteratively via an adversarial human-and-model-in-the-loop procedure. Through this new benchmark, we discover several interesting findings. (i) Surprisingly, we find that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully. (ii) Both large-scale pre-trained models and adversarial training methods achieve far worse performance on the new benchmark than over standard VQA v2 dataset, revealing the fragility of these models while demonstrating the effectiveness of our adversarial dataset. (iii) When used for data augmentation, our dataset can effectively boost model performance on other robust VQA benchmarks. We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.
[{"first":"Linjie","last":"Li","middle":[]},{"first":"Jie","last":"Lei","middle":[]},{"first":"Zhe","last":"Gan","middle":[]},{"first":"Jingjing","last":"Liu","middle":[]}]
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Despite Visual Question Answering (VQA) has realized impressive progress over the last few years, today's VQA models tend to capture superficial linguistic correlations in the train set and fail to generalize to the test set with different QA distributions. To reduce the language biases, several recent works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on VQA-CP. However, since the complexity of design, current methods are unable to equip the ensemble-based models with two indispensable characteristics of an ideal VQA model: 1) visual-explainable: the model should rely on the right visual regions when making decisions. 2) question-sensitive: the model should be sensitive to the linguistic variations in question. To this end, we propose a model-agnostic Counterfactual Samples Synthesizing (CSS) training scheme. The CSS generates numerous counterfactual training samples by masking critical objects in images or words in questions, and assigning different ground-truth answers. After training with the complementary samples (ie, the original and generated samples), the VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. In return, the performance of these models is further boosted. Extensive ablations have shown the effectiveness of CSS. Particularly, by building on top of the model LMH, we achieve a record-breaking performance of 58.95% on VQA-CP v2, with 6.5% gains.
[{"first":"Long","last":"Chen","middle":[]},{"first":"Xin","last":"Yan","middle":[]},{"first":"Jun","last":"Xiao","middle":[]},{"first":"Hanwang","last":"Zhang","middle":[]},{"first":"Shiliang","last":"Pu","middle":[]},{"first":"Yueting","last":"Zhuang","middle":[]}]
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
The robustness of multimodal deep learning models to realistic changes in the input text is critical for applicability on important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but without leveraging the cross-modal information present in multimodal data. Such information from the visual modality, such as color, size, and shape, provides additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., “girl on a chair” to “little girl on a wooden chair”). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of ~15% in MRR and ~20% in F1 score, respectively. Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples. We release the code to encourage robustness evaluations of deep vision-and-language models: https://github.com/claws-lab/multimodal-robustness-xmai
[{"first":"Shivaen","last":"Ramshetty","middle":[]},{"first":"Gaurav","last":"Verma","middle":[]},{"first":"Srijan","last":"Kumar","middle":[]}]
Human-Adversarial Visual Question Answering
Visual question answering (VQA) is widely recognized as an important evaluation task for vision and language research. Besides direct applications such as helping the visually impaired or multimodal content understanding on the web, it offers a mechanism for probing machine understanding of images via natural language queries. Making progress on VQA requires bringing together different subfields in AI -combining advances from natural language processing (NLP) and computer vision together with those in multimodal fusion -making it an exciting task in AI research.
Human-Adversarial Visual Question Answering
Over the years, the performance of VQA models has started to plateau on the popular VQA v2 dataset [18] -approaching inter-human agreement -as evidenced by Fig. 1. This raises important questions for the field: To what extent have we solved the problem? If we haven't, what are we still missing? How good are we really?
Human-Adversarial Visual Question Answering
An intriguing method for investigating these questions is dynamic data collection, where human annotators and state-of-the-art models are put "in the loop" together to collect data adversarially. Annotators are tasked with and rewarded for finding model-fooling examples, which are then verified by other humans. The easier it is to find such examples, the worse the model's performance can be said to be. The collected data can be used to "stress test" current VQA models and can serve as the next iteration of the VQA benchmark that can help drive further progress.
Human-Adversarial Visual Question Answering
The commonly used VQA dataset [18] dataset was collected by instructing annotators to "ask a question about this scene that [a] smart robot probably can not answer" [3]. One way of thinking about our proposed human-adversarial data collection is that it explicitly ensures that the questions can not be answered by today's "smartest" models. Table 1: Contrastive examples from VQA and AdVQA. Predictions are given for the VisualBERT, ViLBERT and UniT models, respectively. Models can answer VQA questions accurately, but consistently fail on AdVQA questions. This work is, to the best of our knowledge, the first to apply this human-adversarial approach to any multimodal problem. We introduce Adversarial VQA (AdVQA), a large evaluation dataset of 28,522 examples in total, all of which fooled the VQA 2020 challenge winner, MoViE+MCAN [42] model.
Human-Adversarial Visual Question Answering
We evaluate a wide range of existing VQA models on AdVQA and find that their performance is significantly lower than on the commonly used VQA v2 dataset [18] (see Table 1). Furthermore, we conduct an extensive analysis of AdVQA characteristics, and contrast with the VQA v2 dataset.
Human-Adversarial Visual Question Answering
We hope that this new benchmark can help advance the state of the art by shedding important light on the current model shortcomings. Our findings suggest that there is still considerable room for continued improvement, with much more work remaining to be done.
Human-Adversarial Visual Question Answering
In this phase, annotators on Amazon MTurk are shown an image and are tasked with asking questions about this image. The interface has a state-of-the-art VQA model in the loop. For each question the annotator asks, the model produces an answer. Annotators are asked to come up with questions that fool the model. Annotators are done (i.e., can submit the HIT) once they successfully fooled the model or after they tried a minimum of 10 questions (whichever occurs first). To account for the fact that it may be hard to think of many questions for some images, annotators are given the option to skip an image after providing non-fooling three questions. We use the VQA Challenge 2020 winner -  MoViE+MCAN [42] -trained on the COCO 2017 train set as the model in the loop. This is to ensure that the adversarial benchmark we collect is challenging for the current state-of-the-art in VQA.
Human-Adversarial Visual Question Answering
The interface provides a digital magnifier to zoom into an area of the image when the mouse hovers over it, allowing workers to examine the image closely if necessary. See Figure 3b. Keyboard shortcuts were provided for convenience.
Human-Adversarial Visual Question Answering
Note that in the question collection phase, annotators self-report when they have identified a question that is incorrectly answered by the model. Whether or not this was actually the case is verified in the question validation phase. Two different annotators are shown the image, question, and answer predicted by the model, and asked whether the model's answer is correct. As an additional quality control, the annotator is also asked whether the question is "valid". A valid question is one where the image is necessary and sufficient to answer the question. Examples of invalid questions are: "What is the capital of USA?" (does not need an image) or "What is the person doing?" when the image does not contain any people or where there are multiple people doing multiple things. If the two validators disagree, a third annotator is used to break the tie. Examples are added to the dataset only if at least two annotators agree that the question is valid.
Human-Adversarial Visual Question Answering
In the final stage of data collection, following [18,52], we collect 10 answers per question providing instructions similar to those used in [18] making sure that no annotator sees the same question twice.
Human-Adversarial Visual Question Answering
In addition, as an extra step of caution, to filter bad questions that might have passed through last two stages and to account for ambiguity that can be present in questions, we allow annotators to select "unanswerable" as an answer. Further, to ensure superior data quality, we occasionally provide annotators with hand-crafted questions for which we know the non-ambiguous single true answer, as a means to identify and filter out annotators providing poor quality responses.
Human-Adversarial Visual Question Answering
The statistics for the first two stages (question collection and validation) are shown in Table 2. We find that annotators took about 5 tries and on average around 4 minutes to find a model-fooling example. For computing the model error rate, we can look at the instances where annotators claimed they had fooled the model, and where annotators were verified by other annotators to have fooled the model. The latter has been argued to be a particularly good metric for measuring model performance [27]. We also further confirm that the model was indeed fooled by running the model-in-the-loop on a subset of examples in which human agreement was 100% and found the accuracy to be 0.08% on val, and 0.1% on test splits of AdVQA respectively. Interestingly, the "claimed" model error rate, based on self-reported model-fooling questions, is similar to that of text-based adversarial question answering, which was at 44.0% for RoBERTa and 47.1% for BERT [4]. The validated error rate for our task is much higher than e.g. for ANLI [43], which was 9.52% overall, suggesting that fooling models is a lot easier for VQA than for NLI. It appears to be not too difficult for annotators to find examples that the model fails to predict correctly.
Human-Adversarial Visual Question Answering
We analyze the performance of several baselines and a wide variety of state-of-the-art VQA models on the AdVQA dataset. We evaluate the same set of models on VQA v2 dataset as a direct comparison.
Human-Adversarial Visual Question Answering
Prior baselines. We start by evaluating two prior baselines: answering based on the overall majority answer, or the per answer type majority answer in the validation dataset. We use the same answer types as in [3]. The overall majority answer in AdVQA is no. The majority answer is no for "yes/no", 2 for "numbers" and unanswerable for "others". See Section 4 for more details on answer types.
Human-Adversarial Visual Question Answering
Unimodal baselines. Next, we evaluate two unimodal pretrained models: i) ResNet-152 [20] pretrained on Imagenet [12]; and ii) BERT [13], both finetuned on the task using the visual (image) or textual (question) modality respectively, while ignoring the other modality. We observe that the unimodal text model performs better than unimodal image for both VQA and AdVQA.
Human-Adversarial Visual Question Answering
Multimodal methods. We evaluate two varieties of multimodal models: i) unimodal pretrained and ii) multimodal pretrained. In the unimodal pretrained category we explore MMBT [28], MoViE+MCAN [42] and UniT [22]. These models are initialized from unimodal pretrained weights: BERT pretraining for MMBT; Imagenet + Visual Genome [32] detection pretraining for MoViE+MCAN; and Imagenet + COCO [37] detection pretraining for the image encoder part and BERT pretraining for the text encoder part in UniT. In the multimodal pretrained category, we explore VisualBERT [33], VilBERT [39,40], VilT [30], UNITER [10] and VILLA [16]. These models are first initialized from pretrained unimodal models and then pretrained on different multimodal datasets on proxy self-supervised/semi-supervised tasks before finetuning on VQA. VisualBERT is pretrained on COCO Captions [9]; VilBERT is pretrained on Conceptual Captions [48] We find that multimodal models in general perform better than unimodal models, as we would expect given that both modalities are important for the VQA task.
Human-Adversarial Visual Question Answering
Multimodal OCR methods. As we will see in Section 4, a significant amount of questions in AdVQA can be answered using scene text.
Human-Adversarial Visual Question Answering
We test a state-of-the-art TextVQA [50] model, M4C [23] on AdVQA. We evaluate two versions: (i) trained on VQA 2.0 dataset, and (ii) trained on TextVQA [50] and STVQA [5]. In both cases, we use OCR tokens extracted using the Rosetta OCR system [6]. We also use the same answer vocabulary used by other models for fair comparison. 
Human-Adversarial Visual Question Answering
Our model evaluation results show some surprising findings. We discuss our observations and hypotheses around these findings in this section.
Human-Adversarial Visual Question Answering
Baseline comparison. Surprisingly, most multimodal models are unable to outperform a unimodal baseline (BERT [13]) and simple majority answer (per answer type) prior baseline which only predicts the most-frequently occurring word for the question's category (no for "yes/no", 2 for "numbers" and unanswerable for "others"). A detailed breakdown of category-wise performance provided in Table 4 suggests that even though "yes/no" questions are often considered to be easy, the baseline multimodal models we evaluate are unable to beat the majority answer prediction. We also observe varied but close to majority answer performance in the "numbers" category, which is a question type known to be difficult for VQA models [42]. In contrast, the models outperform the majority answer in the "others" category even though "unanswerable" (the majority answer) is not in the answer vocabulary used. Interestingly, M4C outperforms all on "numbers" and "others" categories possibly thanks to its text reading capabilities. These trends showcase the difficulty of AdVQA and suggest that we have a long way to go still, given that we are as yet apparently unable to beat such simple baselines.
Human-Adversarial Visual Question Answering
Model rankings. First, M4C [23] performs the best among the evaluated models. Interestingly, it is smaller than many of the more sophisticated model architectures that score higher on VQA. This is probably due to the importance of the ability to read and reason about text in the image for answering some AdVQA questions. Second, among models that can't read text, VisualBERT [33] is the best model, despite (or perhaps because of?) it being simple and having fewer parameters compared to other models. Third, the adversarially-trained VILLA [16] model performs surprisingly poorly.
Human-Adversarial Visual Question Answering
While it may be more robust to statistically-generated adversarial examples, it appears to be less so against human-adversarial examples. Fourth, we find that all of these models perform poorly compared to humans, while model performance on VQA is much closer to that of humans. Human performance. Another surprising finding is that inter-human agreement is higher on the AdVQA dataset than on VQA. This could be due to different data annotation procedures, requirements on annotators or just statistical noise. Human-adversarial questions may also be more specific due to annotators having to make crisp decisions about the model failing or not. 
Human-Adversarial Visual Question Answering
We train all the models in Table 3 on the VQA train + val split excluding the COCO 2017 validation images. We also add questions from Visual Genome [32] corresponding to the images that overlap with the VQA training set to our training split. The VQA val set in our results contains all questions associated with the images in the COCO 2017 validation split. This is also consistent with the training split used for the model in the loop. We collect our AdVQA validation set on the images from the COCO 2017 val split, ensuring there is no overlap with the training set images. We choose the best checkpoint for each model by validating on the AdVQA validation set.
Human-Adversarial Visual Question Answering
For all our experiments, we use the standard architecture for the models as provided by their authors. For the models that are initialized from pretrained models (whether unimodal or multimodal pretrained), we use off-the-shelf pretrained model weights and then finetune on our training set. We do not do any hyperparamter search for these models and use the best hyperparams as provided by respective authors. We finetune each model with three different seeds and report average accuracy.
Human-Adversarial Visual Question Answering
We run most of our experiments on NVIDIA V100 GPUs. The maximum number of GPUs used for training is 8 for larger models. Maximum training time is 2 days. More details about hyperparameters, number of devices and time for training each model are provided in the supplementary material.
Human-Adversarial Visual Question Answering
Questions. We first analyze the question diversity in AdVQA and compare them with popular VQA datasets. AdVQA contains 28,522 questions (5,123 in val and 23,399 in test) each with 10 answers. Fig. 4a shows the distribution of question length compared with [18,52]. The average question length in AdVQA is 7.8, which is higher than the VQA v2 (6.3), and TextVQA (7.2). The workers often need to get creative and more specific to fool the model-in-the-loop, leading to somewhat longer questions (e.g. specifying a particular person to ask about). Fig. 6a shows the top 15 most occurring We see that the average question length (7.82) in AdVQA is higher than the prior work. (b) Percentage of questions solvable when a particular number of top k most occurring answers are selected from each dataset. The plot suggests that AdVQA has a much more diverse answer vocabulary compared to [18] but also not quite as challenging as [52]. (c) Cumulative human agreement scores. We see that human agreement is better and higher on AdVQA compared to [18].
Human-Adversarial Visual Question Answering
questions from the val set, showcasing that questions involving text (e.g. time, sign) and counting (e.g. how many) are major failures for current state-of-the-art VQA models, corroborating the findings of prior work in [52,19,53,26]. Fig. 5 shows a sunburst plot for the first 4 words in the AdVQA val set questions. We can observe that questions in AdVQA often start with "what" or "how" frequently inquiring about aspects like "many" (count) and "brand" (text).  Answers. In AdVQA val set, only 66.6% (3,856) answers occur more than twice compared to 94.8% in VQA v2 [18], suggesting that the diversity of possible answers is much larger in AdVQA compared to VQA v2. Fig. 4b shows the percentage of questions that are solvable with a vocabulary of top k most occurring answers. We observe that more questions in VQA v2 while fewer question in TextVQA are solvable with smaller vocabulary compare to AdVQA. This suggests that AdVQA is more diverse and difficult than VQA v2 but not as narrowly focused as TextVQA, making it a great testbed for future VQA models. We also showcase more qualitative examples in our supplementary to demonstrate that AdVQA's diversity doesn't lead to unnatural questions. Fig. 4c shows the cumulative percentage of questions where more than a particular number of annotators agree with each other. Fig. 6b shows the top 500 most occurring answers in the AdVQA val set; starting from very common answers such as "no" and counts ("1", "9"), to gradually more specific and targeted answers like "21", "teddy bear" and "center".
Human-Adversarial Visual Question Answering
Answer Vocabulary. To showcase the challenge of AdVQA, we take the original VQA v2 vocabulary used in the Pythia v0.3 model [50]. We find that 77.2% of the AdVQA val set's questions are answerable using this vocabulary, suggesting that a model with powerful reasoning capability won't be heavily limited by vocabulary on AdVQA. But, we also note that for high performance on AdVQA, a model will need to understand and reason about rare concepts, as 50.9% of the answers in AdVQA val and test sets don't occur in VQA v2 train set. Question Types. Table 5 shows the categorywise distribution for AdVQA questions compared with VQA v2 [18]. We can observe a shift from more easy questions of the "yes/no" category in VQA v2 dataset to more difficult questions in "numbers" category (as suggested in prior work [42]) in AdVQA.
Human-Adversarial Visual Question Answering
Human Agreement. In the AdVQA val set, 27.2% human annotators agree on all 10 answers while 3 or more annotators agree an answer for 97.7%, which is higher compared to 93.4% on VQA v2 [18], even though as discussed AdVQA contains a large number of rare concepts.
Human-Adversarial Visual Question Answering
Relationship to TextVQA [50]. To understand if the ability to read text is crucial for AdVQA, we extract Rosetta [6] tokens on the AdVQA val set and determine how many questions can be answered using OCR tokens at an edit distance of 1 to account for OCR errors. We find that 15.4% of the AdVQA val questions are solvable using OCR tokens suggesting that ability to read scene text would play a crucial role in solving AdVQA.
Human-Adversarial Visual Question Answering
We introduced the AdVQA dataset, a novel human-adversarial multimodal dataset designed for accelerating progress on Visual Question Answering (VQA). Current VQA datasets have started plateauing and are approaching saturation with respect to human performance. In this work, we demonstrate that the problem is far from solved.
Human-Adversarial Visual Question Answering
In particular, our analysis and model evaluation results suggest that current state-of-the-art models underperform on AdVQA due to a reasoning gap incurred from a combination of (i) inability to read text; (ii) inability to count; (iii) heavy bias towards the VQA v2 question and answer distribution; (iv) external knowledge; (v) rare unseen concepts; and (vi) weak multimodal understanding. We've shown the gap is unlikely to be due to (i) limited answer vocabulary; (ii) language representation (BERT performance compared to other); (iii) no pretraining (UniT); or (iv) lack of adversarial training (VILLA performance). We will provide an evaluation benchmark for community to evaluate on AdVQA and hope that AdVQA will help bridge the gap by serving as a dynamic new benchmark for visual reasoning with a large amount of headroom for further progress in the field.
Human-Adversarial Visual Question Answering
In future work, it would be interesting to continue AdVQA as a dynamic benchmark. If a new state of the art emerges, those models can be put in the loop to examine how we can improve even further.
Human-Adversarial Visual Question Answering
This work analyzed state-of-the-art Visual Question Answering (VQA) models via a dataset constructed using a dynamic human-adversarial approach. We hope that this work can help make VQA models more robust.
Human-Adversarial Visual Question Answering
VQA datasets contain biases, both in the distribution of images represented in the datasets, as well as the corresponding questions and answers. These biases are likely amplified by the VQA models trained on these datasets. Biases studied in the context of image captioning [7,58] are also relevant for VQA. English is the only language represented in this work.
Human-Adversarial Visual Question Answering
VQA models can be useful for aiding visually impaired users. The commonly used VQA datasets are not representative of the needs of visually impaired users -both in terms of the distribution of images (typically consumer photographs from the web), and in terms of the questions contained in the datasets (typically asked by sighted individuals while looking at the image). In contrast, the VizWiz dataset [19] contains questions asked by visually impaired users on images taken by them to accomplish day-to-day tasks. It would be beneficial for the community to collect larger datasets in this context to enable progress towards relevant technology. It would be important to do this with input from all relevant stakeholders, and in a responsible and privacy-preserving manner. VQA models are currently far from being accurate enough to be uesful or safe in these contexts.
Human-Adversarial Visual Question Answering
Except UNITER [10] and VILLA [16], we trained all models using MMF [50]. The evaluation results with the standard deviation over three different runs with three different seeds are provided in  MMBT [28] We trained MMBT from scratch with a batch size 64 without any pretraining following [28] for 88K updates. The training takes an average of 17 hours.
Human-Adversarial Visual Question Answering
UniT [22] We initialized from the model pretrained on all 8 datasets [22] with COCO initialization. We set the batch size to 8, weight decay as 1e − 4 and train the model on 8 GPUs for 2 days.
Human-Adversarial Visual Question Answering
VisualBERT [33] We trained VisualBERT from the best pretrained model on COCO using MLM loss using a batch size of 64 which takes an average of 8 hours.
Human-Adversarial Visual Question Answering
ViLBERT [39] We trained ViLBERT from the best pretrained model on Conceptual Captions using MLM loss using a batch size of 64 which takes an average of 13 hours.
Human-Adversarial Visual Question Answering
ViLT [30] We trained ViLT with 44K updates, initial learning rate of 1e − 4, eps as 1e − 8 and weight decay as 0.01 for an average of 7 hours.
Human-Adversarial Visual Question Answering
UNITER/VILLA [10,16] We used the author-provided pretrained checkpoint for UNITER 4 and VILLA 5 with a confidence threshold of 0.075. The rest of the hyper parameters were consistent with the configuration provided in the repository. The training takes about 2 hours.
Human-Adversarial Visual Question Answering
M4C [23] We used the same training schedule and hyper parameters as [23] used for TextVQA training [52] for both TextVQA + STVQA and VQA v2. We didn't use extra Visual Genome [32] QA data due to lack of OCR text and used the OCR data from [23] for both COCO and TextVQA.
Human-Adversarial Visual Question Answering
In addition to the main results in Table 3, we also evaluated existing OSCAR [35] / VinVL [57] models on AdVQA, since both models are known to perform well on VQA. Note that the training set of the off-the-shelf OSCAR and VinVL models includes COCO val2014 data, which overlaps with our validation set (COCO 2017). Even then, we find that the validation performance is still much lower (more than 2x) compared to its performance on VQA v2. See Table B.1. 
Human-Adversarial Visual Question Answering
Before annotators were able to proceed with the main task, they had to pass an on-boarding phase (Section C.1). We restricted the interface to be only accessible on non-mobile devices, to annotators in the US with at least 100 approved hits and with an approval rate higher than 98%. The annotators were paid a bonus if their self-claimed fooling question was verified to be fooling and valid by two other annotators.
Human-Adversarial Visual Question Answering
The annotators were asked to go through a two-stage qualification phase. In the first stage, they were shown 11 examples that include both valid and invalid questions. For each question, we collect ten answers from ten different annotators using this interface. We provide explicit instructions following [18] and [52] to avoid ambiguity and collect short relevant answers. The annotators are also provided a checkbox to select "unanswerable" in case the question is ambiguous or can't be answered which annotators are suggested to use sparingly. Finally, we use and show a set of hand-crafted already annotated questions without ambiguity randomly to filter out bad annotators by comparing their answers to ground truth. An annotator is prevented from doing the task if they fail the test three times.   
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Visual Question Answering (VQA) [4] is a task where given an image and a question about it, the model provides an open-ended answer. A successful VQA system can be applied to real-life scenarios such as a chatbot that assists visually impaired people. In these applications, the VQA models are expected to handle diverse question types from recognition to reasoning, and answer questions faithfully based on the evidence in the image.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
While model performance on the popular VQA dataset [14] has been advanced in recent years [4,19,3,50,9,43,54], with better visual representations [18,54], more sophisticated model designs [12,27], large-scale pretraining [30,41,7,42,55] and adversarial training [11], today's VQA models are still far from being robust enough for practical use. There are some works studying the robustness of VQA models, such as their sensitivity to visual content manipulation [1], answer distribution shift [2], linguistic variations in input questions [39], and reasoning capabilities [13,38]. However, current robust VQA benchmarks mostly suffer from three main limitations: (i) designed with heuristic rules [13,2,1]; (ii) focused on a single type of robustness [38,39,13]; (iii) based on VQA v2 [14] images (or questions), which state-of-the-art (SOTA) VQA models are trained on [13,2,1,38,39]. The images [1] or questions [13,17] are often synthesized, not provided by human.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In addition, previous data collection procedures on VQA benchmarks are often static, meaning that the data samples in these datasets do not evolve, and model performance can saturate on the fixed dataset without good generalization. For example, model accuracy on VQA v2 has been improved from 50% [4] to 76% [54] since inception. Similarly, on robust VQA benchmarks, a recent study [28] has found that pre-trained models can greatly lift state of the art. Yet it remains unclear whether such high performance can be maintained when encountering examples in the wild.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
To build an organically evolving benchmark, we introduce Adversarial VQA (AVQA), a new large-scale VQA dataset dynamically collected with Human-And-Model-inthe-Loop Enabled Training (HAMLET) [47]. AVQA is built on images from different domains, including web images from Conceptual Captions [40], user-generated images from Fakeddit [32], and movie images from VCR [52]. Our data collection is iterative and can be perpetually going. We first ask human annotators to create examples that current best models cannot answer correctly ( Figure 1). These newly annotated examples expose the model's weaknesses, and are added to the training data for training a stronger model. The re-trained model is subjected to the same process, and the collection can iterate for several rounds. After each round, we train a new model and set aside a new test set. In this way, not only is the resultant dataset more challenging than existing benchmarks, but this process also yields a "moving post" target for VQA systems, rather than a static benchmark that will eventually saturate.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
With this new benchmark, we present a thorough quantitative evaluation on the robustness of VQA models along multiple dimensions. First, we provide the first study on the vulnerability of VQA models when under adversarial attacks by human. Second, we benchmark several SOTA VQA models on the proposed dataset to reveal the fragility of VQA models. We observe a significant and universal performance drop when compared to VQA v2 and other robust VQA benchmarks, which corroborates our belief that existing VQA models are not robust enough. Meanwhile, this also demonstrates the transferability of these adversarial examples -data samples collected using one set of models are also challenging for other models. Third, as our annotators can ask different types of questions for different types of robustness, our analyses show that SOTA models suffer across various questions types, especially counting and reasoning.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Our main contributions are summarized as follows. (i) For better evaluation of VQA model robustness, we introduce a new VQA benchmark dynamically collected with a Human-and-Model-in-the-Loop procedure. (ii) Despite rapid advances on VQA v2 and robust VQA benchmarks, the evaluation on our new dataset shows that SOTA models are far from being robust. In fact, they are extremely vulnerable when attacked by human annotators, who can succeed within 2 trials on average. (iii) We provide a thorough analysis to share insights on the shortcomings of current models as well as comparison with other robust VQA benchmarks.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In this section, we introduce the AVQA dataset in detail. Sec. 3.1 explains the data collection pipeline. Sec. 3.2 and Sec. 3.3 present data statistics and the comparison with other datasets.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
The HAMLET data collection procedure can be considered as a game played by two parties: a human annotator and a well-trained model. The human annotator competes against the model as an adversary and tries to design adversarial examples to identify its vulnerabilities. After collecting enough examples, the model augments its training with the collected data to defend similar attacks. For VQA, we define the adversarial example as an adversarial question on a natural image that the model answers incorrectly.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
As shown in Figure 2, given an image, the human annotator tries to write a tricky question that the VQA model may fail. Once the question is submitted, an online model prediction will be displayed immediately to the workers. The model answer is then judged by the same annotator as either "definitely correct", "definitely wrong", or "not sure". If the model prediction is "definitely wrong", then the attack is successful, and we further ask the annotator to provide a correct answer. Otherwise, the annotator needs to write another question until the model predicts a wrong answer, or the number of tries exceeds a threshold (5 tries). To avoid obviously invalid questions caused by the annotator taking shortcuts (e.g., untruthful judgement on model predictions, questions irrelevant to the image content), we also launch an answer annotation task. Successfully attacked questions are provided to 9 other annotators to collect extra answers, as well as their confidence level ("confident", "maybe" and "not confident") of their answer. The questions that receive less than 6 "confident" answers and have no agreement in answers among 10 annotators are removed during post-processing. In the end, each image is presented to 3 workers for question collection, and each image-question pair is shown to 10 annotators for answer collection.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
This procedure can be continuously deployed for multiple rounds. At each round, we strengthen the models as we re-train them with extra data collected from previous rounds. This "dynamic" evolution of attacked models allows the collection of "harder" questions in the later rounds. In our setup, we launch the data collection for 3 rounds on Amazon Mechanical Turk. However, this data collection can be a never-ending process, as we can always replace the attacked model with a stronger model trained on newly collected data or better architectures developed in the future. Round 1 (R1) For the first round, we employ VQA models trained on examples from VQA v2 [14] and VGQA [23] as our starting point. To avoid the collected questions overfitting to the vulnerabilities of a single model or a single architecture, for each user question, we randomly sample one model from LXMERT [43], UNITER-B [9] and UNITER-L [9] as the attacked model to generate the answer. We choose LXMERT and UNITER as representatives of twostream and single-stream pre-trained V+L models, due to their strong performance on VQA v2. We use images sampled from Conceptual Captions [40] for annotation. In total, we collected 38.7K verified 1 questions and 28.2K unverfied questions over 13.7K images, and split the verified examples into 60%/10%/30% for train/val/test splits. All unverified examples are also added to the training split. Round 2 (R2) For the second round, we re-train our models with questions from VQA v2, VGQA and R1's train split, and select the best model checkpoints of LXMERT,   
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
For the third round, we include more diverse images from different domains: (i) web images from Conceptual Captions [40]; (ii) user-generated images from Fakeddit [32]; and (iii) movie frame images from VCR [52]. The attacked model is still randomly sampled from LXMERT, UNITER-B and UNITER-L, but we add the training set from R1 and R2 to the training data.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Summary Finally, combining data collected in R1, R2 and R3 produces our proposed AVQA dataset. In the end, we collected 243.0K questions over 37.9K images, with 142.1K/8.7K/26.4K images in the train/val/test split.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
The data statistics of the new dataset are summarized in Table 1. The number of examples we collected per image varies per round, starting with approximately 6.8 questions/image for R1, to around 5.4 for R2 and 7.2 for R3. Under the same image domain for R1 and R2, we suspect that the annotators learn to identify model vulnerabilities more rapidly than the models learn to defend itself from the adversarial examples. We provide analyses in Sec. 4.1 and 4.4 for further investigation. On the one hand, the annotators are getting better at identifying vulnerabilities of these models. Analyses of question types per round in Sec. 4.4 show that the workers tend to ask more questions in certain categories, such as "count", "OCR" and "commonsense reasoning", that the model is more likely to fail. On the other hand, although the attacked model is strengthened through data augmentation, the model does not seem to learn from the adversarial examples effectively.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
For each round, we report the model error rate, both on verified and all examples. The model error rate reported under "Total" captures the percentage of examples where the writer disagrees with the model's answer during question collection, but where we are not yet sure that the example is correct. The verified model error rate is the percentage of model errors from examples that we further collected 9 additional answers from other workers. We observe an increase in model error rate from R1 to R2. Assuming constant image domain difficulty in R1 and R2, the higher model rate suggests that the models in the later rounds are not significantly stronger, or the annotators are getting better at fooling the state-of-the-art models. In R3, where we included images from more diverse domains, the model error rate decreases from 49.0% to 34.4%. We suspect it is because the movie images from VCR are mostly humancentric, which is commonly observed in COCO.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We also report the average number of attempts ("#Tries" in Table 1) that a worker needed to complete the annotation process for each image, i.e., to successfully attack the model or exceed the limits on number of tries. Surprisingly, although the VQA models used in the later rounds are trained with more data, the number of tries needed to successfully attack them does not increase. On average, it takes less than 2 tries to successfully attack a VQA model. Similarly, the average time needed per successful attack decreases by 15 seconds as data collection progresses.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Our Adversarial VQA dataset sets a new benchmark for evaluating the robustness of VQA models. It improves upon existing robust VQA benchmarks in several ways. First, the dataset by design is more difficult than previous datasets. During collection, we do not constrain the worker to ask questions that only fall into a single robustness type (Sec. 4.4). As a result, our dataset is helpful in defending model robustness against several robust VQA benchmarks (Sec. 4.3). Second, most robust VQA datasets are based on VQA v2 validation set, which state-of-the-art models use for training or hyper-parameter tuning. Thus, it is difficult to analyze the robustness of the best-performing models due to this data leakage. Our dataset is built on non-overlapping images from diverse domains, which naturally resolves it. Lastly, our dataset is composed of human-written questions on natural images, rather than rule-based questions in [13,17] or manipulated images in [1]. A detailed comparison on data statistics is provided in Table 1.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Our work is inspired by ANLI [34]. While ANLI focuses on the pure text task of natural language inference, our work targets at the multi-modal task of visual question answering. However, due to the open-ended nature of VQA problem, the construction of AVQA is more challenging. Instead of giving the worker a target label when collecting adversarial questions, we first ask the worker to judge whether the model prediction is correct, then provide a ground-truth answer. Our verification process is also different from ANLI. In order to evaluate model performance under the same criteria as VQA v2 [14], we collect 10 answers from workers in total. Unlike the observations on ANLI, where the adversarial robustness of NLI models can be improved in a large extent through data augmentation of ANLI, our analysis on AVQA in Sec. 4 will show that it is more difficult to defend against adversarial attacks for VQA models.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In this section, we conduct extensive experiments to study the AVQA dataset. Specifically, Sec. 4 Table 2 reports the main results. In addition to UNITER-B, UNITER-L [9] and LXMERT [43], we also include BUTD [3]   Overall, these three models achieve similar performance across rounds and on AVQA. When trained with "ALL" data, the performance gain from UNITER-L over BUTD is only +2.00 on AVQA, even though UNITER-L is pre-trained with extensive amount of image-text pairs.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We dive deeper into the key factors behind the low performance of state-of-the-art models on AVQA, and try to answer the following questions. Q1: Is the language in AVQA biased? Starting from VQA-CP [2], concerns have been raised about the propensity of models to pick up on spurious artifacts that are present just in the co-occurrence of question-answer pairs, without actually paying attention to the image content. We compare full models trained with both images and questions to models trained only on questions by zeroing out image features in Table 4a. The results show that language-only models perform poorly on AVQA, and similarly on VQA v2.  Table 5: Evaluation of grid-feature-based method ClipBERT [25], and adversarial-training-based method VILLA [11]. 'ALL' refers to VQA v2+VGQA+AVQA.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Language-only model performance decreases over rounds for AVQA. However, UNITER-B is not much better than language-only on AVQA. Obviously, without manual intervention, some bias remains in how annotators phrase questions. For example, there might be more counting questions with answers other than 2, which is the majority answer in VQA v2. Therefore, models trained on AVQA only performs slightly higher for both UNITER-B and Languageonly model. However, we also observe the significant drop in VQA v2 performance is out of proportion to the slight performance improvement on AVQA. We further investigate if the low performance is due to the difference in answer distribution between training and testing split. Due to the large number of answer candidates (more than 3000 for VQA v2), it is impossible to evenly balance the possibility of each answer. Therefore, we test out this hypothesis by adopting a simple yet effective baseline method on VQA-CP [44]: adding a regularization term by replacing the image with a randomly sampled one. The intuition is that the answer to a question corresponding to a given image is very unlikely to be correct for a randomly sampled image. As reported in Table 4b, although effective on VQA-CP, adding such regularization hurts the performance on AVQA for both BUTD and UNITER-B. In addition, when applied to a stronger model on VQA-CP, i.e., UNITER-B, the regularization term is less effective. Q2: Is AVQA transferrable to different visual features? The AVQA dataset is collected with the assistance of models trained on Faster R-CNN [36] region features [3]. To investigate whether these collected adversarial examples are transferrable to different image features, we conduct experiments using another type of feature, i.e., grid features [18] from CNNs, which have shown to be effective for VQA tasks [18,16,33,25]. Specifically, we consider Clip-BERT [25], an end-to-end pre-trained model that directly takes in raw images and questions, and the images are represented by grid features as in [18]. Meanwhile, ClipBERT's end-to-end training strategy may also help to defend potential attacks to fixed feature representations widely used in previous work [9,43,3]. Table 5   We examine the effectiveness of adversarial training by adopting PGD-based adversarial training method VILLA in [11]. VILLA-B is both adversarially pre-trained on large-scale image-text data and adversarially finetuned on the respective dataset. We compare its performance against UNITER-B on both AVQA and VQA v2 in Table 5. Adversarial training brings slight performance improvement. However, the performance gap between AVQA and VQA v2 is still very significant. Note that VILLA-B crafts adversarial examples during training by adding adversarial perturbations to the embedding space. These adversarial perturbations can hardly change the intrinsic statistics of training data, such as the distribution of question types and relevant objects in the image. Our analysis of question types and visual recognition concepts in Sec. 4.4 will show that AVQA is hard because it requires the model to have the ability to reason, count and recognize different visual concepts.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We also test models on recent robust VQA benchmarks including: VQA-Rephrasings [39] for linguistic variations, VQA-LOL [13] Complement/Supplement for logical reasoning, VQA-Introspect [38] for consistency of model predictions in perceptual sub-questions and main reasoning questions, CV-VQA [1] and IV-VQA [1] for model robustness to image manipulations. Results are summarized in Table 6. We observe that UNITER-B can already outperform previous models for most of the benchmarks, which is consistent with observations in [28]. Training on AVQA is helpful in improving model performance on robustness benchmarks. Particularly, AVQA helps to boost model reasoning capability across 3 datasets. It is likely that AVQA exposes the model training to more diverse question templates, hence improves on VQA-Rephrasings. On IV-VQA, which focuses on counting questions, AVQA helps to improve performance despite of the significant performance gain UNITER-B has already achieved.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We manually annotate 300 randomly sampled examples from each round to investigate: which types of questions do workers employ to fool the models, and how they evolve as the rounds progress.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Results are summarized in Table 7. Questions are categorized into 4 meta-categories: counting, OCR, reasoning, and visual concept recognition. Although OCR and counting can be considered as visual concept and quantitative reasoning, we separate them out as they contribute a large portion per round, to almost 50% in the later rounds. There are three main reasoning questions: positional reasoning (i.e., the relative/absolute position of an object), relational reasoning (i.e., semantic relationship between two or more objects), and commonsense reasoning (i.e., visual commonsense reasoning, e.g., "Is the water more likely to be a lake or an ocean", given an image showing a body of water surrounded by mountains.). Other reasoning questions include comparative reasoning (e.g., "which person is taller?") and logical reasoning (e.g., negation). For visual concept recognition, we roughly divide them into low-level visual concepts (e.g., color, shape, texture), action (e.g., "what is the person doing"), small objects, occluded objects, and abstract objects (e.g., objects in painting).
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We observe that annotators rely heavily on counting questions to attack the models -nearly 30% of the sampled   [37], Textfooler [20] and Sememe+PSO [51]. Green (red) indicates a correct (wrong) answer.  questions across all rounds fall into this category. While R1 questions are mostly on objects that are of normal sizes and less occluded, we found that the counting questions become harder in R2 and R3 as many of them are about small and occluded objects. There is also a surge in abstract and OCR questions for R2, due to the increase in the number of abstract images and images that contain scene text. The percentage of reasoning questions, especially relational reasoning and commonsense reasoning, increases drastically from R1 to R2 and R3. Visualizations in Figure 3a show that questions in later rounds are indeed more complicated, with more detailed relational and positional descriptions when referring to an object. Overall, these findings are compatible with the idea that VQA models are not robust enough to various types of questions.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Textual adversarial attack methods [31,20,51] have been widely explored in NLP. The goal is to alter model predictions with minor changes to the input textual queries, so that adversarial examples can be generated and model vulnerabilities can be identified automatically. We investigate whether we can directly apply these methods to generate adversarial examples in high quality and compare the generated examples to AVQA. In total, we consider 3 different textual adversarial attack methods, including Sears [37] via bask-translation for sentence-level attacks, Textfooler [20] and Sememe+PSO [51] by replacing words with its synonyms or words that share the same sememe annotations for word-level attacks. The adversarial attacks are performed to all questions on 5000 images in the Karpathy split [21].
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We visualize examples in Figure 12. Without human-inthe-loop, the generated adversarial questions share similar problems: (i) the adversarial question does not share the same answer with the original question, therefore additional answer annotations may need to be collected; (ii) model prediction to the adversarial question is not necessarily incorrect when it is different from answers to the original question; (iii) word similarity may not hold when it needs to be grounded to the image (e.g., window vs. skylights). In addition, we compare these methods against the AVQA dataset quantitatively in Table 8. Generally, humans take much fewer tries and have a higher successful rate when attacking VQA models. How to design effective adversarial attack methods to generate high-quality VQA examples can be an interesting future research direction.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In this work, we collect a new benchmark Adversarial VQA (AVQA) to evaluate the robustness of VQA models. It is collected iteratively for 3 rounds via a human-andmodel-in-the-loop enabled training paradigm, on images from different domains. AVQA questions cover diverse robustness types, enabling a more comprehensive evaluation on model robustness. Our analysis shows that state-of-theart models cannot maintain decent performance on AVQA, despite of large-scale pre-training, adversarial training, sophisticated model architecture design, and stronger visual features. AVQA brings a new challenge to the community on how to design more robust VQA models that are ready to deploy in real-life applications.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
A. More Discussions on AVQA Future Practices. We recommend future models to report performance on both VQA v2 and AVQA. AVQA is designed to test VQA model robustness under human adversarial attacks. It is complementary to VQA v2 (naturallycollected questions), rather than a replacement. In addition, we believe it is beneficial to evaluate on other robust VQA benchmarks as well. While AVQA encompasses broader robustness types and image domains with higher data quality, existing robustness benchmarks can in addition provide useful analysis tailored to individual robustness types. An ideal VQA system should perform well on all VQA benchmarks. Further, we encourage future work to apply humanin-the-loop adversarial attack to their proposed models to identify potential vulnerabilities. For AVQA, we expect to provide a dynamically evolving VQA benchmark as models grow more robust, to alleviate the drawbacks of static benchmarks (e.g., performance saturation and overfitting).
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Constraints/Rules For Data Collection. As our goal is to examine VQA models' robustness when encountering test examples in the wild, we do not constrain the questions to specific types, to avoid unconscious bias from dataset creators. As a result, we have found that models make more mistakes on Count/OCR/Relation/Commonsense questions ( Table 7 in the main text).
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
To obtain high-quality adversarial questions, we enforce a set of rules to ensure the questions are objective, relevant to the image, and have exact answers (see detailed instructions in Figure 16). We also manually filter out questions with repetitive patterns for each annotator during collection. Our answer annotation process validates the collected questions to some extent, which are answerable by human but not always answerable by model.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Bias in Model Choices. Our current choice of models was guided by the assumption that newer models are more likely to be transformer-based with currently-proven most effective features. We plan to include a broader choice of models in future collection, as the benchmark evolves.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Type of Questions. Following [4], given the structure of questions generated in English, we cluster questions into different types based on the words that start the question. Figure 4 shows the distribution of questions based on the first four words of the questions in AVQA. Interestingly, the variety of question types are quite similar to those in [4], including "What is", "How many" and "Is there". Quantitatively, we also categorize the questions into "Y/N", "Num", "OOV" and "Other". The percentage of questions for different categories is shown in Table 9. "OOV" questions refer to questions that cannot be answered by VQA v2 [14] an- swer vocabularies. We also include two upper bounds, one based on VQA v2 answer vocabularies, and the other on open vocabularies. Moreover, we estimate human performance on AVQA by sampling 1 human answer as prediction and use the rest 9 answers as references. We repeat the process 10 times and average the score. Comparing model performance reported in the main text, there is still a huge gap, with about 50 points lower than the upper bounds or the estimated human performance.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Question Lengths. Figure 5 shows the distribution of question lengths. We see that most questions range from four to ten words.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Dataset Properties Across Rounds. Figure 6 shows a histogram of the number of tries per verified example across the three different rounds. We observe a consistent trend for all three rounds, over 80% of examples are successfully collected within 2 tries. Figure 7 shows the time taken per verified example. As the round progresses, we observe that more and more examples are collected within 100 seconds (less than 2 minutes). Figure 8 shows the proportion of different types of collected examples across three rounds. Comparing to R1 and R2, R3 contains more "not sure" judgements to model answers during question collection (type B), which indicates that the task is getting harder.  Answer Confidence and Inter-human Agreement. During answer collection (see interface in Figure 15), the annotators are required to provide both a correct answer to the question given the image content and a self-judgment on how confident they feel about the answer. Specifically, we ask "Do you think you were able to answer the question correctly?", and the annotator need to choose from "yes" (confident with score 1), "maybe" or "no" (not confident with score 0). Figure 9 shows the distribution of responses (black lines). A majority of the answers were labeled as confident. More than 9 annotators are confident about their answers on over 60% questions on average.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In addition, we investigate how the self-judgment confidence corresponds to the answer agreement between annotators across three rounds of data collection. Color bars in Figure 9 show the percentage of questions in which (i) 7 or more, (ii) 3-7, or (iii) less than 3 workers agree on the answers given their average confidence score. Across all rounds, the agreement between subjects increases with confidence. We do observe that workers are more confident about their answers in R2 and R3, comparing to R1.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Answer Distribution. Figure 10 shows the distribution of answers for several question types. We can see that a number of question types, such as "Is . . . ", "Can. . . ", and "Does. . . " are typically answered using "yes" and "no" as answers. Other questions such as "What is/are. . . " and "What kind/type. . . " have a rich diversity of responses. Other question types such as "What color. . . " or "Which. . . " have more specialized responses, such as colors, or "left" and "right". These observations are similar to those in VQA v2.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We include more visualization examples of collected data across three rounds in Figure 11. We show adversarial questions from 4 categories: Count, OCR, Reasoning and Visual Concept Recognition. Note that questions may belong to multiple categories. For example, counting question from R3 ("How many natural satellites are in the sky?") requires commonsense about "natural satellites". OCR question from R1 ("What company is on the back of the referee?") not only requires commonsense about "referee" but relational reasoning about "on the back of". Reasoning questions include positional/relational reasoning (e.g., "What is the woman closest to the camera holding in her hand?"), commonsense reasoning (e.g., "Is the egg yolk cooked?") and comparative reasoning ("Who is taller?"). There are also questions that require recognition of both low-level visual concepts (e.g., color/shape) and high-level visual concepts (e.g., action, relation).
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We also visualize more examples generated via textual adversarial attack methods (Sears [37], Textfooler [20] and Sememe+PSO [51]) in Figure 12. The first two columns show invalid examples, and the last column includes valid examples, based on our manual examination. Recall that our goal is to collect high-quality adversarial questions that can be used to accurately, thoroughly evaluate and examine the weakness of VQA models. Automatically generated adversarial questions are often incorrect (requiring additional human efforts to validate their correctness), and limited to linguistic variations to existing questions, thereby they are unlikely to provide a comprehensive analysis.  
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
Recall that questions in R3 are collected on images from various domains, including web images from Conceptual Captions [40] (CC, used in R1 and R2), user-generated images from Fakeddit [32] and movie video frames from VCR [52]. Hence, we can study how model performance can be transferable across different domains. We create a new split of R3 (R3 ) according to the image source, with CC images for training and Fakeddit/VCR images for evaluation. We also observe that model performance on VCR is significantly higher than those on the original R3 val and Fakeddit splits across all training settings. Images from VCR are often human-centric, which may be "easier" than complex or abstract scenes depicted in CC/Fakeddit images.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
In addition, we include detailed results from BUTD [3], ClipBERT [25], VILLA-B and VILLA-L [11] in Table 11.  These results are consistent with observations we summarized in Section 4 of the main text.  Wrong") during question collection and sent to 9 annotators for verification during answer collection. Specifically, C=Examples that more than 3 verifiers overruled the question author's decision of "Definitely Wrong" and agree with the model's answer. D=Examples for which there is no agreement among verifiers, E=Examples where at least two verifiers agree with each other during answer collection. We split E by images into training, validation, and testing sets. Examples on training images in A and C are added to the training set, the rest are discarded. B and D are excluded due to low quality.    Table 11: Detailed results from BUTD [3], ClipBERT [25], VILLA-B and VILLA-L [11] under different settings. AVQA = R1+R2+R3, ALL = VQA v2+VGQA+AVQA.   [37], Textfooler [20] and Sememe+PSO [51]. Green (red) indicates a correct (wrong) answer. Blue highlights the changes made in adversarial questions. Figure 13: UI for question collection. Given an image, the annotator is required to write a tricky answer to fool our "smart VQA robot" (well-trained VQA models). After clicking the "Get Robot Ansner", the annotated question will be sent to our online model for evaluation, and a feedback will be returned immediately. See Figure 14 for an example of model feedback. If the model answer is definitely wrong, the annotator is prompted to enter a correct answer. Figure 15: UI for answer collection. Given an image and a question, an annotator is asked to write a concise answer to the question, and choose a confidence level for the answer ("Yes", "No", or "Maybe").  
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Visual Question Answering (VQA), i.e., answering natural language questions about the visual content, is one of the core techniques towards complete AI. With the release of multiple large scale VQA datasets (e.g., VQA v1 [6] and v2 [17]), VQA has received unprecedented attention and * Long Chen and Xin Yan are co-first authors with equal contributions. † Corresponding author. 1 Codes: https://github.com/yanxinzju/CSS-VQA
Counterfactual Samples Synthesizing for Robust Visual Question Answering
What is the color of the bus to predict correct answer (e.g., "surfing"), but also relies on the right reference regions when making this prediction. (b) questionsensitive ability: the model should be sensitive to the linguistic variations, e.g., after replacing the critical word "luggage" with "bus", the predicted answers of two questions should be different.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
hundreds of models have been developed. However, since the inevitable annotation artifacts in the real image datasets, today's VQA models always over-rely on superficial linguistic correlations (i.e., language biases) [2,42,23,17].
Counterfactual Samples Synthesizing for Robust Visual Question Answering
For example, a model answering "2" for all "how many X" questions can still get satisfactory performance regardless of the X. Recently, to disentangle the bias factors and clearly monitor the progress of VQA research, a diagnostic benchmark VQA-CP (VQA under Changing Priors) [3] has been proposed. The VQA-CP deliberately has different questionanswer distributions in the train and test splits. The performance of many state-of-the-art VQA models [5,15,40,4] drop significantly on VQA-CP compared to other datasets. Currently, the prevailing solutions to mitigate the bias issues are ensemble-based methods: they introduce an aux- iliary question-only model to regularize the training of targeted VQA model. Specifically, these methods can further be grouped into two sub-types: 1) adversary-based [33,18,7]: they train two models in an adversarial manner [16,12], i.e., minimizing the loss of VQA model while maximizing the loss of question-only model. Since the two models are designed to share the same question encoder, the adversarybased methods aim to reduce the language biases by learning a bias-neutral question representation. Unfortunately, the adversarial training scheme brings significant noise into gradients and results in an unstable training process [18]. 2) fusion-based [10,14,27]: they late fuse the predicted answer distributions of the two models, and derive the training gradients based on the fused answer distributions. The design philosophy of the fusion-based methods, is to let the targeted VQA model focuses more on the samples, which cannot be answered correctly by the question-only model. Although the ensemble-based methods have dominated the performance on VQA-CP, it is worth noting that current methods fail to equip them with two indispensable characteristics of an ideal VQA model: 1) visual-explainable: the model should rely on the right visual regions when making decisions, i.e., right for the right reasons [34]. As shown in Figure 1 (a), although both two models can predict the correct answer "surfing", they actually refer to totally different reference regions when making this answer prediction. 2) question-sensitive: the model should be sensitive to the linguistic variations in question. As shown in Figure 1 (b), for two questions with similar sentence structure (e.g., only replacing word "luggage" with "bus"), if the meanings of two questions are different, the model should perceive the discrepancy and make corresponding predictions.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
In this paper, we propose a novel model-agnostic Coun-terfactual Samples Synthesizing (CSS) training scheme. The CSS serves as a plug-and-play component to improve the VQA models' visual-explainable and question-sensitive abilities, even for complex ensemble-based methods. As shown in Figure 2, CSS consists of two different types of samples synthesizing mechanisms: V-CSS and Q-CSS. For V-CSS, it synthesizes a counterfactual image by masking critical objects in the original image. By "critical", we mean that these objects are important in answering a certain question (e.g., object for the question "what color is the man's tie"). Then, the counterfactual image and original question compose a new image-question (VQ) pair. For Q-CSS, it synthesizes a counterfactual question by replacing critical words in the original question with a special token "[MASK]". Similarly, the counterfactual querstion and original image compose a new VQ pair. Given a VQ pair (from V-CSS or Q-CSS), a standard VQA training sample triplet still needs the corresponding ground-truth answers. To avoid the expensive manual annotations, we design a dynamic answer assigning mechanism to approximate ground-truth answers for all synthesized VQ pairs (e.g., "not green" in Figure 2). Then, we train the VQA models with all original and synthesized samples. After training with numerous complementary samples, the VQA models are forced to focus on critical objects and words.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Extensive ablations including both qualitative and quantitative results have demonstrated the effectiveness of CSS. The CSS can be seamlessly incorporated into the ensemblebased methods, which not only improves their both visualexplainable and question-sensitive abilities, but also consistently boosts the performance on VQA-CP. Particularly, by building of top on model LMH [14], we achieve a new record-breaking performance of 58.95% on VQA-CP v2.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We consider the common formulation of VQA task as a multi-class classification problem. Without loss of generality, given a dataset D = {I i , Q i , a i } N i consisting of triplets of images I i ∈ I, questions Q i ∈ Q and answers a i ∈ A, VQA task learns a mapping f vqa : I ×Q → [0, 1] |A| , which produces an answer distribution given image-question pair. For simplicity, we omit subscript i in the following sections.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
In this section, we first introduce the base bottom-up topdown model [4], and the ensemble-based methods for debiasing in Section 3.1. Then, we introduce the details of the Counterfactual Samples Synthesizing (CSS) in Section 3.2. V ← e v (I) 3:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
question-only model 6:P vqa (a) ← M (P vqa (a), P q (a)) 7:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
update parameters 8: if cond then 9: return V , Q, P vqa (a) 10: end if 11: end function w j is j-th word feature. Then both V and Q are fed into the model f vqa to predict answer distributions:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
(1)
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Model f vqa typically contains an attention mechanism [13,29,41], and it is trained with cross-entropy loss [38,11]. Ensemble-Based Models. As we discussed in Section 1, the ensemble-based models can be grouped into two subtypes: adversary-based and fusion-based. Since adversarybased models [33,18,7] suffer severe unstable training and relatively worse performance, in this section, we only introduce the fusion-based models [10,14,27]. As shown in Algorithm 1, they introduce an auxiliary question-only model f q which takes Q as input and predicts answer distribution:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Then, they combine the two answer distributions and obtain a new answer distributionP vqa (a) by a function M :
Counterfactual Samples Synthesizing for Robust Visual Question Answering
In the training stage, the XE loss is computed based on the fused answer distributionP vqa (a) and the training gradients are backpropagated through both f vqa and f q . In test stage, only model f vqa is used as the plain VQA models.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
The overall structure of CSS training scheme is shown in Algorithm 2. Specifically, for any VQA model, given a training sample (I, Q, a), CSS consists of three main steps:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
1. Training VQA model with original sample (I, Q, a); 2. Synthesizing a counterfactual sample (I − , Q, a − ) by V-CSS or (I, Q − , a − ) by Q-CSS; 3. Training VQA model with the counterfactual sample.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
In the following, we introduce the details of V-CSS and Q-CSS (i.e., the second step). As shown in Algorithm 2, for each training sample, we only use one certain synthesizing mechanism, and δ is the trade-off weight (See Figure 4 (c) for more details about the influence of different δ). 
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We sequentially introduce all steps of V-CSS following its execution path (line 5 to 8 in Algorithm 2), which consists of four main steps: initial objects selection (IO SEL), object local contributions calculation, critical objects selection (CO SEL), and dynamic answer assigning (DA ASS).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
1. Initial Objects Selection (IO SEL). In general, for any specific QA pair (Q, a), only a few objects in image I are related. To narrow the scope of critical objects selection, we first construct a smaller object set I, and assume all objects in I are possibly important in answering this question. Since we lack annotations about the critical objects for each sample, we followed [39] to extract the objects which are highly related with the QA. Specifically, we first assign POS tags to each word in the QA using the spaCy POS tagger [19] and extract nouns in QA. Then, we calculate the cosine similarity between the GloVe [31] embedding of object categories and the extracted nouns, the similarity scores between all objects in I and the QA are denoted as SIM.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We select |I| objects with the highest SIM scores as I.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
2. Object Local Contributions Calculation. After obtaining the object set I, we start to calculate the local contribution of each object to the predicted probability of groundtruth answer. Following recent works [22,36,39] which utilize the modified Grad-CAM [35] to derive the local contribution of each participant, we calculate the contribution of i-th object feature to the ground-truth answer a as:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
where P vqa (a) is the predicted answer probability of ground truth answer a, v i is i-th object feature, and 1 is an allones vector. Obviously, if the score s(a, v i ) is higher, the contributions of object v i to answer a is larger. VQA.eval() don't update parameters 3: , , P + vqa (a) ← VQA(I + , Q + , a, True) 4: a + ← top-N(argsort ai∈A (P + vqa (a i ))) 5:
Counterfactual Samples Synthesizing for Robust Visual Question Answering
a is gt answer set 6: return a − 7: end function  
Counterfactual Samples Synthesizing for Robust Visual Question Answering
After obtaining the private contribution scores s(a, v i ) for all objects in I, we select the top-K objects with highest scores as the critical object set I + . The K is a dynamic number for each image, which is the smallest number meets Eq. (5):
Counterfactual Samples Synthesizing for Robust Visual Question Answering
vi∈I + exp(s(a, v i ))/ vj ∈I exp(s(a, v j )) > η, (5) where η is a constant, we set η = 0.65 in all experiments (See Figure 4 for more details about the dynamic K setting).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Then, the counterfactual visual input I − is the absolute complement of set I + in set I, i.e., I − = I\I + . We show an example of I, I + , and I − in Figure 3.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
4. Dynamic Answer Assigning (DA ASS). Given the counterfactual visual input I − and original question Q, we compose a new VQ pair (I − , Q). To assign ground truth answers for VQ pair (I − , Q), we design a dynamic answer assigning (DA ASS) mechanism. The details of DA ASS are shown in Algorithm 3. Specifically, we first feed another VQ pair (I + , Q) into the VQA model, and obtain the predicted answer distribution P + vqa (a). Based on P + vqa (a), we select the top-N answers with highest predicted probabilities as a + . Then we define a − := {a i |a i ∈ a, a i / ∈ a + }. In an extreme case, if the model predicts all ground truth answer correctly for VQ pair (I + , Q), i.e., a ⊂ a + , then a − is a ∅, i.e., zero for all answer candidates. The basic motivation is that if current model can predict ground truth answer for (I + , Q) (i.e., I + contains critial objects and I − not), the ground truth for (I − , Q) should not contain original ground truth answers anymore, e.g., "not green" in Figure 2.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
All steps in Q-CSS are similar to V-CSS. Following its execution path (line 11 to 13 in Algorithm 2), it consists of word local contribution calculation, critical words selection (CW SEL), and dynamic answer assigning (DA ASS).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
1. Word Local Contribution Calculation. Similar with the V-CSS (cf. Eq. (4)), we calculate the contribution of i-th word feature to the ground-truth answer a as: s(a, w i ) = S(P vqa (a), w i ) := (∇ wi P vqa (a)) T 1. (6) 2. Critical Words Selection (CW SEL.) In this step, we first extract question-type words for each question Q 2 (e.g., "what color" in Figure 3). Then, we select top-K words with highest scores from the remaining sentence (except the question-type words) as critical words. The counterfactual question Q − is the sentence by replacing all critical words in Q with a special token "[MASK]". Meanwhile, the Q + is the sentence by replacing all other words (except question-type and critical words) with "[MASK]". We show an example of Q, Q + , and Q − in Figure 3.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
3. Dynamic Answer Assigning (DA ASS.) This step is identical to the DA ASS in V-CSS, i.e., Algorithm 3. For Q-CSS, the input for DA ASS is the VQ pair (I, Q + ).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. We evaluated the proposed CSS for VQA mainly on the VQA-CP test set [3]. We also presented experimental results on the VQA v2 validation set [17] for completeness. For model accuracies, we followed the standard VQA evaluation metric [6]. For fair comparisons, we did all the same data preprocessing steps with the widely-used UpDn model [4] using the publicly available reimplementation 3 .
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We run a number of ablations to analyze the influence of different hyperparameters of V-CSS and Q-CSS. Specifically, we conducted all ablations by building on top of ensemblebased model LMH [14]. Results are illustrated in Figure 4. The size of I in V-CSS. The influence of different size of I is shown in Figure 4 (a). We can observe that the model's performance gradually decreases with the increase of |I|. The size of critical objects in V-CSS. The influence of masking different numbers of critical objects is shown in Figure 4 (a). We compared the dynamic K (Eq. (5)) with some fixed constants (e.g., 1,3,5). From the results, we can observe that the dynamic K achieves the best performance. The size of critical words in Q-CSS. The influence of replacing different sizes of critical words is shown in Figure 4 2 We use the default question-type annotations in VQA-CP dataset. 3  (b). From the results, we can observe that replacing only one word (i.e., top-1) achieves the best performance. The proportion δ of V-CSS and Q-CSS. The influence of different δ is shown in Figure 4 (c). From the results, we can observe that the performance is best when δ = 0.5 .
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. Since the proposed CSS is a model-agnostic training scheme, which can be seamlessly incorporated into different VQA architectures. To evaluate the effectiveness of CSS to boost the debiasing performance of different backbones, we incorporated the CSS into multiple architectures including: UpDn [4], PoE (Product of Experts) [14,27], RUBi [10], LMH [14]. Especially, PoE, RUBi, LMH are ensemble-based methods. All results are shown in Table 1.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Results. Compared to these baseline models, the CSS can consistently improve the performance for all architectures. The improvement is more significant in the ensemble-based models (e.g., 6.50% and 9.79% absolute performance gains in LMH and PoE). Furthermore, when both two types of CSS are used, models often achieve the best performance.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. We incorporated the CSS into model LMH [14], which is dubbed as LMH-CSS, and compared it with the state-of-the-art models on both VQA-CP v2 and VQA v2. According to the backbone of these models, we group them into: 1) AReg [33], MuRel [9], GRL [18], RUBi [10], SCR [39], LMH [14], HINT [36]. These models utilize the UpDn [4] as their backbone. 2) HAN [28], GVQA [3], ReGAT [25], NSM [20]. These models utilize other different backbones, e.g., BLOCK [8], BAN [24] etc. Especially, the AReg, GRL, RUBi, LMH are ensemble-based models.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Results. The results are reported in Table 3. When trained and tested on the VQA-CP v2 dataset (i.e., left side of Table 3), the LMH-CSS achieves a new state-of-the-art performance over all question categories. Particularly, CSS improves the performance of LMH with a 6.50% absolution performance gains (58.95% vs. 52.45%). When trained and tested on the VQA v2 dataset (i.e., middle side of Table 3), the CSS results in a minor drop in the performance by 1.74% for LMH. For completeness, we further compared the performance drop between the two benchmarks. Different from previous models that suffer severe performance drops (e.g., 23.74% in UpDn, and 9.19% in LMH), the LMH-CSS can significantly decrease the performance drop into 0.96%, which demonstrate that the effectiveness of CSS to further reduce the language biases in VQA.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. We further compared the LMH-CSS with state-ofthe-art models on VQA-CP v1. Similarly, we group these baseline models into: 1) GVQA with SAN [40] backbone, 2) AReg, GRL, RUBi, and LMH with UpDn backbone.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Results. Results are reported in Table 2. Compared to these baseline models, the LMH-CSS achieves a new state-of-theart performance on VQA-CP v1. Particularly, the CSS improves the performance of LMH with a 5.68% absolution performance gains (60.95% vs. 55.27%).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We will validate the effectiveness of CSS to improve the visual-explainable ability by answering the following questions: Q1: Can existing visual-explainable models be incorporated into the ensemble-based framework? Q2: How  Settings. We equipped the existing state-of-the-art visualexplainable model SCR [39] into the LMH framework, and compared it with CSS. Results are reported in Table 4 (a).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Results. Since the training of all SOTA visual-explainable models (e.g., SCR, HINT) are not end-to-end, for fair comparisons, we used a well-trained LMH (i.e., 52.45% accuracies on VQA-CP v2) as the initial model. However, we observe that its performance continues to decrease from the start, which shows that the existing visual-explainable models can not be easily incorporated into the ensemble-based framework. In contrast, CSS can improve the performance.   For qualitative results, we illustrated in Figure 5 (a).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Results. From Table 4 (b), we can observe that CSS dramatically improves the AI scores, which means the actually influential objects are more related to the QA pair. From Figure 5 (a), we can find that the CSS helps the model to make predictions based on critical objects (i.e., green boxes), and suppress the influence of irrelevant objects (i.e., red boxes).
Counterfactual Samples Synthesizing for Robust Visual Question Answering
We will validate the effectiveness of CSS to improve the question-sensitive ability by answering the following questions: Q3: Does CSS helps to improve the robustness to diverse rephrasings of questions? Q4: How does CSS improve the model's question-sensitive abilities?
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. As discussed in previous work [37], being robust to diverse rephrasing of questions is one of key behaviors of a question-sensitive model. To more accurately evaluate the robustness, we re-splited the existing dataset VQA-Rephrasings [37] with the same splits as VQA-CP, and denoted it as VQA-CP-Rephrasings. For evaluation, we used the standard metric Consensus Score CS(k). Results are reported in Table 4 (c) (left). We refer readers to [37] for more details about the VQA-Rephrasings and metric CS(k). Table 4 (c), we can observe that Q-CSS dramatically improves the robustness to diverse rephrasings of questions. Furthermore, V-CSS can help to further improve the robustness, i.e., CSS achieves the best performance.
Counterfactual Samples Synthesizing for Robust Visual Question Answering
Settings. We evaluate the effectiveness of CSS to improve the question-sensitive ability on both quantitative and qualitative results. For quantitative results, since there is no standard evaluation metric, we design a new metric Confidence Improvement (CI): Given a test sample (I, Q, a), we remove a critical noun in question Q, and obtain a new test sample (I, Q * , a) 4 . Then we feed both two samples into evaluated model, and calcluate the confidence decreses of the ground-truth answer. We formally define CI in Eq. 7: CI = (I,Q) (Pvqa(a|I, Q) − Pvqa(a|I, Q * )) · 1(a =â) (I,Q) 1 4 The auxiliary test set is released in: github.com/yanxinzju/CSS-VQA
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
The ability to model the interaction of information in vision and language modalities powers several web applications -text-to-image search (He et al., 2016), summarizing multimodal content (Zhu et al., 2018), visual question answering (Antol et al.,
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A street with a mechanical sign with directions.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Corresponding Image A street with a mechanical sign with directions.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
small neon Augmented Caption using XMAI (Our) Image is retrieved at rank 104
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Image is retrieved at rank 506 ↓ A street with a mechanical sign with directions.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
busy Augmented Caption using CLARE (Text-only) Image is retrieved at rank 104 Figure 1: We propose Cross-Modal Attribute Insertions (XMAI) -an approach that leverages cross-modal interactions in multimodal data to obtain meaningful text augmentations that methods using text-only information (e.g., CLARE) cannot provide. These augmentations highlight vulnerabilities of multimodal models; in this case, the corresponding image is retrieved at a worse rank (104 → 506) for the modified caption.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
2015), and editing images using language commands (Shi et al., 2021). Ensuring satisfactory user experience within such applications necessitates the development of multimodal models that can robustly process text and image data, jointly. Existing research has demonstrated the brittle reasoning mechanism of text-only and image-only models by introducing variations in the inputs (Evtimov et al., 2020;Li et al., 2021a). Furthermore, prior work have established controlled generation methods for text (Ross et al., 2022), including counterfactuals for model assessment (Madaan et al., 2021;Wu et al., 2021). However, beyond applying modality-specific perturbations to multimodal (image + text) data (Qiu et al., 2022) , existing research has not studied the robustness of models to likely augmentations in text that leverage cross-modal interactions. Specifically, current research on text augmentation considers the following likely variations: skipping certain words, introducing typographical errors, inserting noun or verb modifiers, or using synonyms. Consequently, to study the robustness of deep models, several automated methods have been developed to intro-duce these variations in the text. However, while these text-only perturbations can cover more variations, they are by no means exhaustive with respect to multimodal data. In the context of multimodal data, the text accompanying an image can be meaningfully perturbed to include information from the image. For instance, users can issue a query on a search engine that specifies attributes of the desired image(s); 'a male driver posing with a red car' instead of 'a driver posing with a car.' Existing augmentation approaches can only model text-only data and cannot introduce relevant crossmodal information (like 'male' and 'red' in the above example) while generating augmentations.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We propose novel text variations that leverage the image modality to insert relevant information into the text, which we call cross-modal attribute insertions. Our method inserts attributes of objects that are both present in the image and mentioned in the text. To do so, cross-modal attribute insertion uses object detection to capture objects and their attributes in the image (Anderson et al., 2018), and masked-language modeling to place those attributes prior to the object's mentions in the text (Devlin et al., 2019) (see Figure 1). Additionally, we use embedding similarities to expand the search space of possible augmentations, and introduce an adversarial component to estimate the robustness of multimodal models.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Our proposed approach is highly modular, controllable, and task-agnostic. Different modules govern attribute selection from images, cross-modal object matching, attribute insertion in text, and adversarial strength of the augmented example. The contribution of these modules toward the final augmented text can be controlled using weights that can be tuned as hyper-parameters. Finally, our approach for generating augmentations does not involve any parameter training, which makes it task-agnostic and broadly applicable.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We demonstrate the applicability of our crossmodal attribute insertion approach by generating augmentations for assessing the robustness of models for two different multimodal tasks -(a) text-toimage retrieval and (b) cross-modal entailment. Together, these two tasks are representative of ranking and classification multimodal tasks. Our evaluation comprises assessing the robustness of state-of-theart multimodal learning approaches for these tasks to our augmentations as well as quantifying the relevance of generated augmentations to unmodified examples. We contrast our cross-modal attribute insertions with several baseline approaches that model text-only information.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Our key contributions and findings are: • We propose cross-modal attribute insertions as a new realistic variation in multimodal data. Our proposed approach introduces these variations in a modular, controllable, and task-agnostic manner.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
• We demonstrate that state-of-the-art approaches for text-to-image retrieval and cross-modal entailment are not robust to cross-modal attribute insertions, demonstrating relative drops of ∼ 15% and ∼ 20% in MRR and F 1 score, respectively.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
• While being as effective as existing text-only augmentation methods in highlighting model vulnerabilities, our approach produces augmentations that human annotators perceive to be of better quality than the most competitive text-only augmentation method. Furthermore, our method matches the quality of unmodified textual examples, while being at least 9× faster than the most competitive baseline across the two multimodal tasks.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Overall, we find that cross-modal attribute insertions produce novel, realistic, and human-preferred text augmentations that are complementary to current text-only perturbations, and effectively highlight the vulnerabilities of multimodal models. Future work could employ our augmentation strategy to evaluate and develop more robust multimodal models.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Previous studies independently introduce unimodal perturbations in the visual or textual part of the input to study multimodal robustness. This could involve introducing imperceptible adversarial noise in the images and independently modifying the text using the augmentation approaches discussed earlier (Li et al., 2021a;Ribeiro et al., 2020;Wei and Zou, 2019). For instance, Chen et al. (2020) synthesize counterfactual samples of multimodal data using language models to modify the text. To ensure the preservation of the semantic meaning in the augmented text, Sheng et al. (2021) and Li et al. (2021b) employ humans to perturb the textual questions to fool the state-of-the-art models for Visual Question Answering (Antol et al., 2015). In a step towards using cross-modal interactions in imagetext data to generate realistic variations, Verma et al. (2022) proposed adding relevant information from the image to expand the original textual description, and assess the robustness of multimodal classifiers. Our work proposes a different approach to leverage cross-modal associations to augment multimodal data. Instead of expanding the original text, we insert attributes of objects in the image that are also mentioned in the corresponding text to modify the original text. Additionally, our work considers more multimodal tasks by studying text-to-image retrieval and cross-modal entailment.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Our approach for augmenting text in multimodal data involves identifying objects in the image that are also mentioned in the text, and inserting words similar to their attributes in the text at relevant places. An overview of our approach (XMAI) is depicted in Figure 2.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We denote paired multimodal units as (I, T ), where I represents the input image and T is the text corresponding to that image. Our goal is to transform T into T ′ such that the text includes relevant information from I while effectively highlighting the target model's vulnerabilities. Our method to infuse object attributes in the text can be broken into four parts: (a) object and attribute detection in I, (b) BERT-based [MASK] prediction in T while ensuring (c) similarity of inserted tokens with detected object attributes, and (d) enforcing dissimilarity between modified text T ′ and I to obtain robustness estimates of multimodal models.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Object and Attribute Detection: For each image I we use a pre-trained bottom-up attention model (Anderson et al., 2018;Yu et al., 2020) to extract objects and their associated attributes. The bottom-up attention model identifies objects and corresponding attributes with a one-to-one mapping. We use these objects and attributes to modify T , by intro-ducing masks (i.e., the [MASK] token), in front of the mentions of the objects in T . However, a strict matching criterion would ignore similar objects or alternatively named objects in T . To address this, whenever the text does not have any direct object matches we use a Parts of Speech (PoS) tagger to identify nouns that could represent objects in the image. These identified nouns are compared to objects using cosine similarity between the word embeddings. If the cosine similarity between a noun T and a detected object in I is above some threshold, t, then a [MASK] token is placed before that noun. Overall, this step ensures that insertions are made only for objects in T that are seen in the corresponding image I to obtain T ′ .
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Mask Prediction: Next, we aim to fill in the [MASK] tokens with contextually relevant object attributes. To do so, we use the pre-trained language model BERT (Devlin et al., 2019). We sample top-k predictions from the BERT model based on probability scores that also meet the following criteria: the predicted word should not be a stop word and should not exist in the 3-hop neighborhood of the current [MASK]. Furthermore, since T may contain more than one [MASK] token, we carry out this process sequentially for each [MASK] to utilize newly introduced contexts. Following this process, we obtain k candidate insertions that are contextually relevant for each of the identified objects in T that also exists in I. In the next step, to maintain cross-modal relevance, we consider the similarity of these candidate attributes with the attributes detected in I.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Attribute Similarity: To better select a word for a specific mask that aligns well with information in I, we only consider predicted tokens similar to the attributes of the associated object detected in I. In order to do so, we utilize embedding-based similarities between each predicted token and the detected attribute string. The image attributes can describe the shape, size, color, or other characteristics (like 'floral dress') of detected objects.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Cross-Modal Dissimilarity for Estimating Robustness: Finally, to estimate the robustness of multimodal models, we explicitly include a component that encourages dissimilarity in the embeddings of the image I and the modified text T ′ . For each possible modified text T ′ , we compute the cosine distance between its embedding obtained using the CLIP model (Radford et al., 2021) and that of the corresponding image's CLIP embedding.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
While the mask prediction and attribute similarity steps ensure that the attribute insertions are semantically meaningful and maintain cross-modal relevance, the cross-modal dissimilarity between T ′ and I ensures that we leverage the vulnerabilities in the encoding mechanism of multimodal models. We use CLIP as the encoder for this step as it is a strong representative of the state-of-the-art vision-language models.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Text Augmentation Strategy: We now choose the final augmentation of T by combining the above four components ---object and attribute detection, mask prediction, attribute similarity, and cross-modal dissimilarity for estimating robustness. After placing [MASK] tokens in front of the identified objects mentions or similar nouns in T , we consider the top-k BERT predictions for each of the [MASK] words, denoted by w i ∀ i ∈ {1, . . . , k}. We take the predicted probability scores of these k words and normalize them to sum to one, denoting each by p i . The attribute similarity step computes the similarities for w i with the corresponding attribute, which are then normalized to sum to one and denoted by s i . Finally, we create k augmentations of T , each denoted by T ′ i , and compute the cosine distance of their CLIP embeddings with that of the corresponding image I. The distances are also normalized to sum to one and denoted by d i . Mathematically, the cumulative score for a predicted word w i is given as,
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
where, λ 1 , λ 2 , and λ 3 are hyper-parameters that control the contribution of mask prediction using BERT, attribute similarity, and cross-modal dissimilarity, respectively. The word w i with the maximum score S is the word that is inserted in the place of the [MASK]. For text with multiple [MASK] tokens, we repeat this process iteratively in the order of their occurrence in T .
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
By design, our cross-modal attribute insertion approach is modular as different components serve complementary functions toward the final objective of introducing semantically meaningful augmentations. It is also controllable using hyper-parameters λ 1 , λ 2 , λ 3 , k, and t. Finally, our approach is training-free and, therefore, can be applied to investigate several tasks and models.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We study the effect of cross-modal attribute insertions on two multimodal tasks: text-to-image retrieval (i.e., retrieving images for a textual description) and cross-modal entailment (i.e., predicting the relationship between textual hypothesis and visual premise).
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Task: Given a set of text and image pairs as input, the goal is to retrieve the associated image for each text. The retrieval occurs for each text over a set of images, in our case we use a subset of 1000 textimage pairs, with the objective being to rank the original/ground-truth image the highest. Axiom for Retrieval: Given an image I in the search repository and two search queries Q 1 and Q 2 , such that Q 1 contains more specific details of objects than Q 2 , I should be retrieved at the same or higher rank for query Q 1 than for Q 2 . Dataset: For this task, we use the MSCOCO dataset (Lin et al., 2014), which contains imagescaption pairs. Specifically, we use the imagecaption pairs from 2017's validation split. This subset of the data contains 5, 000 unique images and 25, 010 captions, where each image can have multiple captions. To assess robustness, we perform augmentations on 25, 010 captions one by one while ranking all the images for each caption.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Model Under Investigation: For this task we consider the CLIP model (ViT-B/32) (Radford et al., 2021). CLIP is pretrained on 400 million imagecaption pairs using contrastive learning, resulting in image and text encoders that produce unimodal embeddings that lie in a common latent space. The CLIP model has demonstrated great generalizability to various downstream tasks, including zeroshot and few-shot image classification and crossmodal retrieval. We obtain the CLIP embeddings for each image and caption in the MSCOCO dataset and rank all the images for a given caption based on their cosine similarities. We then contrast the ranking performance of the CLIP model using the original and augmented captions as textual queries.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Task: Cross-modal entailment aims to determine whether the relationship between a visual premise and a textual hypothesis is 'entailment,' 'contradiction,' 'neutral.' Specifically, 'entailment' is observed when the textual hypothesis is logically im-plied (true) by the image while 'contradiction' indicates that the textual hypothesis is not implied (false) by the visual premise. Finally, 'neutral' represents an inconclusive or uncertain relationship between the hypothesis and the premise. Axiom for Entailment: If the relationship between a visual premise and a textual hypothesis is 'entailment,' it should not change to 'contradictory' if the textual hypothesis is enriched with the information from the visual modality. . In addition, METER is fine-tuned on SNLI-VE in order to achieve competitive performance. We contrast the performance of the model on the unmodified SNLI-VE dataset with the performance on the augmented version of SNLI-VE dataset.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We compare our cross-modal attribute insertion approach (XMAI) with competitive baselines that are capable of introducing perturbations based on textonly information. We utilize the TextAttack (Morris et al., 2020) 1 framework for implementing all the baseline perturbation strategies. Deletion: A perturbation strategy that randomly removes words from the text. EDA (Wei and Zou, 2019): This approach combines random deletion, random swapping, random insertion, and synonym replacement to modify each caption. We keep all parameters as default and set the percentage of words to swap to 20%. CheckList (Ribeiro et al., 2020): Developed to generate a diverse set of evaluation examples, CheckList works by coalescing name replacement, location replacement, number alteration, and word contraction/extension. CLARE (Li et al., 2021a): This perturbation strategy uses language models to replace, insert, and merge tokens in captions. We use TextAttack's default fast implementation of CLARE.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We choose k = 3 for the number of top-k predicted BERT words for each [MASK] token and flair/pos-english-fast for PoS tagging of text. Next, to compare the nouns in the text with the objects identified in the image, we use word embeddings produced by a Transformer-based model (bert-base-nli-mean-tokens on HuggingFace (Wolf et al., 2020)). We set the threshold, t, for cosine similarity between nouns in T and objects in I to be 0.7. For [MASK] filling, we use the bert-base-cased model on HuggingFace and the list of stopwords is adopted from NLTK. 2 To compute the similarity between attributes detected in I and BERT predictions, we employ SpaCy's pretrained tok2vec model (en_core_web_md), which contains 300dimensional embeddings for ∼ 500k words (Honnibal et al., 2020). Lastly, the pre-trained CLIP model (ViT-B/32) is used to compute image and text embeddings in a common latent space. For our main experiments, we set the values of λ i as λ 1 = 1, λ 2 = 5, and λ 3 = 5.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We measure the impact of perturbations in the text on the capabilities of multimodal models using task-specific metrics. We quantify text-to-image retrieval performance using mean reciprocal rank (MRR). For cross-modal entailment, we report standard classification metrics (accuracy, precision, recall, and F 1 score).
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
While the effectiveness of perturbations is important for highlighting model vulnerabilities, it is also imperative to measure the relevance of augmented text T ′ with original text T and image I. To this end, we compute mean cosine similarity Sim T −T ′ between original and modified texts (i.e., T & T ′ , respectively) using a sentence Transformer model (all-mpnet-base-v2) (Reimers and Gurevych, 2019). Similarly, we report BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005), using NLTK to further compare the texts (considering n-grams of size upto 4 in the case of BLEU). Additionally, we compute the mean cosine similarity Sim I−T ′ using CLIP (ViT-B/32) embeddings.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Recall that our primary goal is to use XMAI to obtain a complementary and novel set of text augmentations that can highlight the vulnerabilities of multimodal models. To this end, we contrast the performance of the models under investigation on the original and the modified examples, and quantify the relevance of the modified text with respect to the original text and image. We recruit human annotators to compare the quality of the augmentations generated using our approach with (i) the ones generated using the most competitive baseline, and (ii) the original text. Table 1 shows that for the text → image retrieval task, our cross-modal attribute insertions cause the greatest drop in observed MRR; the MRR drops from an original value of 0.632 to 0.536. Similarly, Table 2 shows that for the cross-modal entailment task our approach performs second only to CLARE -an observation that is consistent across all the metrics, accuracy, precision, recall, and F 1 . It is worth noting that our approach is the only one that uses the information from the image modality to introduce textual perturbations and hence the resultant perturbed examples are characteristically different than the ones that are produced using baseline methods like CLARE. We will revisit this using qualitative examples. Overall, results demonstrate that state-of-the-art vision-and-language learning approaches for text-to-image retrieval and crossmodal entailment tasks are not robust to our proposed augmentations. Tables 1 and 2 show that XMAI produces augmentations T ′ that maintain high-relevance with the original text T and the image I, in terms of Sim T −T ′ and Sim I−T ′ . It is interesting to note that the BLEU scores for augmentations generated by XMAI are notably lower than that for the baseline augmentations. On the contrary, METEOR scores show that XMAI's augmentations are "closer" to the original texts compared to most baselines. XMAI's poor BLEU scores can be largely attributed to BLEU's tendency to penalize novel insertions severely compared to removals or replacements, as it is a precision-   We thus attribute the poor performance of XMAI in terms of BLEU scores to BLEU's inability to handle insertions appropriately. This is further substantiated by the human assessments.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We recruit annotators using Amazon Mechanical Turk (AMT) to answer the following two questions: (i) do cross-modal attribute insertions lead to better text augmentations than the most competitive baseline (i.e., CLARE), and (ii) are cross-modal attribute insertions as good as the original text accompanying the image. Please see Appendix A.1 for details on the recruitment filters and compensation on AMT. XMAI versus CLARE: We randomly sampled 100 examples from the validation set of the MSCOCO dataset and showed the modified captions using CLARE and XMAI. 5 annotators annotated each example. We asked annotators to indicate their agreement to the following question after seeing two captions for a given image using a 5point Likert scale (1: Strongly disagree, ..., 5: Strongly agree): Caption 2 is a better description of the shown image than Caption 1 in terms of its quality and accuracy. Caption 1 and Caption 2 were randomly flipped between CLARE and XMAI to avoid any position bias. Furthermore, to ensure quality annotations, we randomly inserted some "attention check" examples that instructed annotators to ignore all previous instructions and mark specified responses on the Likert scale. We discarded responses from annotators who marked the attention-check examples incorrectly and re-collected the annotations.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
For 63% of the examples, a majority of annotators (i.e., at least 3 out of 5) preferred the captions modified using XMAI over CLARE. The captions modified using CLARE were preferred for 26% examples. The rest were either marked as 'equivalent' (i.e., 3: Neither disagree nor agree) or had ambiguous majority votes. Overall, the results demonstrate that the annotators preferred the captions modified using XMAI over the ones modified using CLARE, in terms of their accuracy in describing the image and their quality. We next assess how XMAI modified captions compare against the original captions. XMAI versus Original: We randomly sampled 100 examples from the validation set of the MSCOCO dataset and randomly chose 50 of them to be modified using XMAI while leaving the other 50 unmodified. We first primed the annotators to view 5 original image caption pairs, noting them as reference examples. 4 We then asked the annotators to view a list of image-caption pairs and evaluate the caption quality using the following prompt: Rate the caption quality for the given image based on the reference examples shown earlier. A response of 1 on the 5-point Likert scale indicated 'extremely poor quality' whereas that of 5 indicated 'extremely good quality.' The shown list comprised randomly shuffled original image-caption pairs and modified image-caption pairs using XMAI, and a few attention-check examples. Each example received annotations from 5 different annotators.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
The unmodified captions received an average score of 4.12 (±0.37) whereas that for the modified caption using XMAI was 4.07 (±0.33). The observed inter-rater agreement was strong, with a Krippendorf's α of 0.78. Additionally, a two-sided t-test with unequal variances assumption failed to reject the null hypothesis (p > 0.05) that the average Likert scores for the original and modified captions are from different distributions. In sum, the perceived quality of the modified captions using XMAI is not statistically different from that of the original captions. Computational Efficiency: In Appendix Fig. 7 we demonstrate that our approach for inserting crossmodal attributes is 14.8× and 9.4× faster than the most competitive baseline approach (i.e., CLARE) on MSCOCO and SNLI-VE, respectively. Combined with the fact that XMAI augmentations are perceived to be of better quality than CLARE augmentations and are effective at highlighting model vulnerabilities, the increased computational efficiency allows for more rapid and realistic model validation. In the following section, we demonstrate via qualitative examples that, being the only approach that leverages cross-modal interactions in multimodal data, the augmentations produced by XMAI are novel to those produced by the text-only baselines.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
In Figure 3 we show illustrative examples of the insertions introduced using our approach and contrast them with existing text-only perturbations. We observe that our cross-modal insertions lead to a complementary set of augmentations that are not covered by text-only approaches.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We note that our method does not remove any information present in the original caption/hypothesis. This prevents our method from drastically changing the original semantics, which has been a known shortcoming of text-only perturbations (Wang et al., 2021). In Figure 3(A), we note that EDA produces a grammatically incoherent augmentation ("Image of next front the a house of to...") and CLARE inserts an inaccurate attribute ("round   Figure 4: Varying λ i to isolate their effect on the text-to-image retrieval task. Ablations on independent effects of lambda values, where the default lambdas are: λ 1 = 1, λ 2 = 5, and λ 3 = 5. Each line plot represents changing the specified λ while keeping the others as default. We observe the variation in task-specific performance as well as the similarity metrics.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
serts relevant attributes to the original text ("The Image of the window front of a large house next to an outdoor image of a woman holding a small wooden table."). In Figure 3(B&D) we see that XMAI modifies the text using the information in the corresponding images -for instance, our approach identifies the neon LEDs and inserts 'neon' in front of 'sign.' However, EDA and CLARE introduce inaccurate details. XMAI is also capable of multiple meaningful insertions. Our work is the first to enable cross-modal insertion capabilities to obtain meaningful augmentations of multimodal (image + text) data.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
In Figure 4, we visualize the change in retrieval performance with respect to independent changes in λ 1 , λ 2 , and λ 3 . In other words, we vary a given λ i while keeping other hyper-parameters at their aforementioned values. We find that increasing λ 1 and λ 2 improves the relevance of augmentations but reduces their effectiveness in highlighting vulnerabilities. Intuitively, these components increase the likelihood that our approach picks insertions with high BERT prediction scores (controlled by λ 1 ) and similarities with the identified image attribute (controlled by λ 2 ). On the other hand, increasing λ 3 , which controls the contributions of the robustness assessment component, generates less relevant augmentations that are highly effective. This observation also aligns with our goal of exploiting the lack of robust encoding mechanisms to highlight model vulnerabilities.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Overall, these results demonstrate that the individual components of our approach play significant roles, and can be controlled using the λ i hyperparameters. Similar trends are observed for the cross-modal entailment task; see Appendix Fig. 6.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We discuss the ablations pertaining to the similarity threshold for matching image objects and text nouns in Appendix A.2.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A robust understanding of vision-and-language data is crucial for powering several applications. We propose cross-modal attribute insertions -i.e., adding the visual attributes to text, as a new variation that is likely in multimodal data and to which multimodal models should be robust. Our approach produces novel augmentations that are complementary to existing methods that model text-only data, and are preferred over them by human annotators. Using our augmentations we effectively highlight the vulnerabilities of state-of-the-art multimodal models for text-to-image retrieval and cross-modal entailment. In the future, we aim to empirically study the effect of including XMAI augmented data in task-specific training sets and expand to a broader set of multimodal tasks and metrics.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Limitations and bias of pre-trained models: Our work uses detected objects and their attributes in the images to introduce novel insertions in the corresponding text. To this end, it is important to address the limitations of the state-of-the-art object and attribute detection methods. The undesired artifacts of these methods could be categorized as inaccurate or biased. The detected objects could be incorrect, but since we only consider objects that are also mentioned in the text, the effect of incorrect object detections is non-existent in our augmentations. However, we notice that some of the detected attributes in images and BERT predictions reflect stereotypical associations and have been documented in prior works (Li and Xu, 2021;Kaneko and Bollegala, 2022). We acknowledge that the current state of deep learning research is limited, and the consequential shortcomings are reflected in our augmentations to some extent. Broader social impact: The authors do not foresee any negative social impacts of this work. We believe our cross-modal augmentations will enable an exhaustive evaluation of the robustness of visionand-language models, leading to more reliable multimodal systems. We release the code for our experiments to aid reproducibility and enable future research on this topic. Annotations, IRB approval, and datasets: The annotators for evaluations done in this study were recruited via Amazon Mechanical Turk. We specifically recruited 'Master' annotators located in the United States; and paid them at an hourly rate of 12 USD for their annotations. The human evaluation experiments were approved by the Institutional Review Board (IRB) at the authors' institution. The datasets used in this study are publicly available and were curated by previous research. We abide by their terms of use.  
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
For our annotation tasks, we recruited annotators using Amazon Mechanical Turk. We set the cri-teria to 'Master' annotators who had at least 99% approval rate and were located in the United States. The rewards were set by assuming an hourly rate of 12 USD for all the annotators. In addition, the annotators were informed that the aggregate statistics of their annotations would be used and shared as part of academic research. Previous research has demonstrated the role of providing priming examples in obtaining highquality annotations (Khashabi et al., 2021). Therefore, we showed unmodified examples from the MSCOCO corpus to help annotators establish a reference for quality and accuracy. For both the crowdsourced evaluations, we inserted some "attentioncheck" examples to ensure the annotators read the text carefully before responding. This was done by explicitly asking the annotators to mark a randomlychosen score on the Likert scale regardless of the actual content. We discard the annotations from annotators who did not correctly respond to all the attention-check examples.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Extended Variations in t: To illustrate the effect of changing the threshold, t, we plot our described MSCOCO metrics with respect to variations in t in Figure 5. We see that as the criterion for matching nouns in the text and objects in the image is made more stringent, the quality and relevance of the augmentations improve further. However, the effectiveness of the resulting augmentations in highlighting model vulnerabilities decreases. It is worth noting as the model becomes more selective in inserting attributes (due to fewer matched nouns and objects), we witness a stark increase in BLUE scores. Variations in t effectively capture the trade-off between maintaining the relevance of augmentations and effectively highlighting model vulnerabilities. Even though it is possible to construct augmentations that will be more effective in making the multimodal models perform poorly, it would sacrifice the relevance and quality of resulting augmentations. Our main results demonstrate that with t = 0.7, we obtain high-quality and human-preferred augmentations that are also effective in highlighting vulnerabilities.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Variations in k: We perform another ablation by increasing the value of k for the top-k predictions made by pre-trained BERT model. Table 3 shows that increasing the search space for possible insertion tokens leads to a notable drop in the retrieval         : Varying λ i to isolate their effect on the cross-mdodal entailment task. Ablations on independent effects of lambda values, where the default lambdas are: λ 1 = 1, λ 2 = 5, and λ 3 = 5. Each line plot represents changing the specified λ while keeping the others as default. We observe the variation in task-specific performance as well as the similarity metrics.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
could produce highly dissimilar cross-modal representations, thereby helping the adversarial component of our framework, but with the compromise being the relevancy of the augmentations. We also note that exploring more insertion possibilities increases the per-caption augmentation time taken by XMAI.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Varying λ 2 and λ 3 : To comprehensively understand the effect of variations in λ i , specifically λ 2 and λ 3 , we set each to 1 or −1 (one at a time) while setting the other two lambdas to 0. Results in Table 3 empirically show that both the attribute similarity and robustness assessment components are essential and can serve dual purposes; i.e., negative values of λ 2 and λ 3 allow the associated components to serve the opposite purpose. In contrast to their original objective, when the associated λ values are set to negative values, attribute similarity decreases performance and robustness assessment increases it.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
We report the number of insertions or replacements each augmentation method makes to the original text as well as the number of texts modified for each dataset. The results are reported in Table 4 and 5. We find that our XMAI approach introduces more novel words in the augmentation than any other approach, while also augmenting nearly the same amount of captions as most competitive baseline approaches. This observation, combined with the fact that human annotators prefer XMAI augmentations over baseline augmentations, shows that cross-modal insertions can be used to introduce new information without causing semantic deterioration of the text. Additionally, these results allow us to attribute the low BLEU scores observed in Table 1 to the higher number of insertions that XMAI makes. Note that BLEU computation is precision-based and hence penalizes novel insertions more severely. It is interesting to note that even though 'Deletion' is expected to have no insertions or replacements, we found that in very few cases, due to the adopted implementation as well as the noise in the text could result in fragmented or fused words that were being considered novel compared to the original text and therefore counted as an insertion/replacement.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
In Figure 7 we show the average time to augment the input text for each of the methods. The results are plotted using a logarithmic scale to ensure a clearer depiction across methods that vary considerably in terms of computational time.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Simpler approaches such as Deletion, EDA, and CheckList can modify tens or hundreds of samples each second. Intuitively, the reliance on simple  rules makes these approaches very efficient. On the other hand, context-aware methods such as CLARE and XMAI are slower due to their reliance on large language models and involved selection processes. However, XMAI can augment the text a magnitude faster than CLARE, even after using the fast implementation of CLARE from TextAttack. We don't consider the time to compute objects and attributes for XMAI for two reasons. First, the cost to perform this step was < 1 hour for our datasets and the relationship between methods remains the same. Secondly, objects and attributes only need to be computed once, so there is no additive cost for augmentation unless changes are made to the detection component.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Our experiments were split between a single Tesla V100 for object detection (∼ 1 hour) and NVIDIA Tesla T4 GPUs for our augmentation (∼ 3 hours).
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A For every submission:
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A1. Did you describe the limitations of your work?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A2. Did you discuss any potential risks of your work?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
A3. Do the abstract and introduction summarize the paper's main claims?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Abstract and Section 1 A4. Have you used AI writing assistants when working on this paper?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Left blank.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
Section 3 B1. Did you cite the creators of artifacts you used?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.