@inproceedings{subramanian-etal-2023-modular,
    title = "Modular Visual Question Answering via Code Generation",
    author = "Subramanian, Sanjay  and
      Narasimhan, Medhini  and
      Khangaonkar, Kushal  and
      Yang, Kevin  and
      Nagrani, Arsha  and
      Schmid, Cordelia  and
      Zeng, Andy  and
      Darrell, Trevor  and
      Klein, Dan",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.65",
    doi = "10.18653/v1/2023.acl-short.65",
    pages = "747--761",
    abstract = "We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3{\%} and on the GQA dataset by 2{\%} compared to the few-shot baseline that does not employ code generation.",
}

We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3{\%} and on the GQA dataset by 2{\%} compared to the few-shot baseline that does not employ code generation.
Subramanian, Sanjay Narasimhan, Medhini Khangaonkar, Kushal Yang, Kevin Nagrani, Arsha Schmid, Cordelia Zeng, Andy Darrell, Trevor Klein, Dan
Modular Visual Question Answering via Code Generation