@inproceedings{fu-etal-2023-generate,
    title = "Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge",
    author = "Fu, Xingyu  and
      Zhang, Sheng  and
      Kwon, Gukyeong  and
      Perera, Pramuditha  and
      Zhu, Henghui  and
      Zhang, Yuhao  and
      Li, Alexander Hanbo  and
      Wang, William Yang  and
      Wang, Zhiguo  and
      Castelli, Vittorio  and
      Ng, Patrick  and
      Roth, Dan  and
      Xiang, Bing",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.147",
    doi = "10.18653/v1/2023.findings-acl.147",
    pages = "2333--2346",
    abstract = "The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge. Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. However, these methods suffer from low knowledge coverage caused by PLM bias {--} the tendency to generate certain tokens over other tokens regardless of prompt changes, and high dependency on the PLM quality {--} only models using GPT-3 can achieve the best result. To address the aforementioned challenges, we propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. Rather than following the de facto standard to train a multi-modal model that directly generates the VQA answer, {pasted macro {`}MODEL{'}}name first adopts PLM to generate all the possible answers, and then trains a lightweight answer selection model for the correct answer. As proved in our analysis, RASO expands the knowledge coverage from in-domain training data by a large margin. We provide extensive experimentation and show the effectiveness of our pipeline by advancing the state-of-the-art by 4.1{\%} on OK-VQA, without additional computation cost.",
}

The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge. Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. However, these methods suffer from low knowledge coverage caused by PLM bias {--} the tendency to generate certain tokens over other tokens regardless of prompt changes, and high dependency on the PLM quality {--} only models using GPT-3 can achieve the best result. To address the aforementioned challenges, we propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time. Rather than following the de facto standard to train a multi-modal model that directly generates the VQA answer, {pasted macro {`}MODEL{'}}name first adopts PLM to generate all the possible answers, and then trains a lightweight answer selection model for the correct answer. As proved in our analysis, RASO expands the knowledge coverage from in-domain training data by a large margin. We provide extensive experimentation and show the effectiveness of our pipeline by advancing the state-of-the-art by 4.1{\%} on OK-VQA, without additional computation cost.
Fu, Xingyu Zhang, Sheng Kwon, Gukyeong Perera, Pramuditha Zhu, Henghui Zhang, Yuhao Li, Alex er Hanbo Wang, William Yang Wang, Zhiguo Castelli, Vittorio Ng, Patrick Roth, Dan Xiang, Bing
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge