@inproceedings{geigle-etal-2023-one,
    title = "One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks",
    author = "Geigle, Gregor  and
      Liu, Chen  and
      Pfeiffer, Jonas  and
      Gurevych, Iryna",
    editor = "Can, Burcu  and
      Mozes, Maximilian  and
      Cahyawijaya, Samuel  and
      Saphra, Naomi  and
      Kassner, Nora  and
      Ravfogel, Shauli  and
      Ravichander, Abhilasha  and
      Zhao, Chen  and
      Augenstein, Isabelle  and
      Rogers, Anna  and
      Cho, Kyunghyun  and
      Grefenstette, Edward  and
      Voita, Lena",
    booktitle = "Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.repl4nlp-1.9",
    doi = "10.18653/v1/2023.repl4nlp-1.9",
    pages = "97--117",
    abstract = "Current multimodal models, aimed at solving Vision and Language (V+L) tasks, predominantly repurpose Vision Encoders (VE) as feature extractors. While many VEs{---}of different architectures, trained on different data and objectives{---}are publicly available, they are not designed for the downstream V+L tasks. Nonetheless, most current work assumes that a \textit{single} pre-trained VE can serve as a general-purpose encoder. In this work, we focus on analysis and aim to understand whether the information stored within different VEs is complementary, i.e. if providing the model with features from multiple VEs can improve the performance on a target task, and how they are combined. We exhaustively experiment with three popular VEs on six downstream V+L tasks and analyze the attention and VE-dropout patterns. Our analyses suggest that diverse VEs complement each other, resulting in improved downstream V+L task performance, where the improvements are not due to simple ensemble effects (i.e. the performance does not always improve when increasing the number of encoders). We demonstrate that future VEs, which are not \textit{repurposed}, but explicitly \textit{designed} for V+L tasks, have the potential of improving performance on the target V+L tasks.",
}

Current multimodal models, aimed at solving Vision and Language (V+L) tasks, predominantly repurpose Vision Encoders (VE) as feature extractors. While many VEs{---}of different architectures, trained on different data and objectives{---}are publicly available, they are not designed for the downstream V+L tasks. Nonetheless, most current work assumes that a \textit{single} pre-trained VE can serve as a general-purpose encoder. In this work, we focus on analysis and aim to understand whether the information stored within different VEs is complementary, i.e. if providing the model with features from multiple VEs can improve the performance on a target task, and how they are combined. We exhaustively experiment with three popular VEs on six downstream V+L tasks and analyze the attention and VE-dropout patterns. Our analyses suggest that diverse VEs complement each other, resulting in improved downstream V+L task performance, where the improvements are not due to simple ensemble effects (i.e. the performance does not always improve when increasing the number of encoders). We demonstrate that future VEs, which are not \textit{repurposed}, but explicitly \textit{designed} for V+L tasks, have the potential of improving performance on the target V+L tasks.
Geigle, Gregor Liu, Chen Pfeiffer, Jonas Gurevych, Iryna
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks