@inproceedings{katyal-etal-2023-teampn,
    title = "team{PN} at {S}em{E}val-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot {M}ulti{M}odal Approach",
    author = "Katyal, Nikita  and
      Rajpoot, Pawan  and
      Tamilarasu, Subhanandh  and
      Mustafi, Joy",
    editor = {Ojha, Atul Kr.  and
      Do{\u{g}}ru{\"o}z, A. Seza  and
      Da San Martino, Giovanni  and
      Tayyar Madabushi, Harish  and
      Kumar, Ritesh  and
      Sartori, Elisa},
    booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.semeval-1.63",
    doi = "10.18653/v1/2023.semeval-1.63",
    pages = "457--461",
    abstract = "Visual Word Sense Disambiguation shared task at SemEval-2023 aims to identify an image corresponding to the intended meaning of a given ambiguous word (with related context) from a set of candidate images. The lack of textual description for the candidate image and the corresponding word{'}s ambiguity makes it a challenging problem. This paper describes teamPN{'}s multi-modal and modular approach to solving this in English track of the task. We efficiently used recent multi-modal pre-trained models backed by real-time multi-modal knowledge graphs to augment textual knowledge for the images and select the best matching image accordingly. We outperformed the baseline model by {\textasciitilde}5 points and proposed a unique approach that can further work as a framework for other modular and knowledge-backed solutions.",
}

Visual Word Sense Disambiguation shared task at SemEval-2023 aims to identify an image corresponding to the intended meaning of a given ambiguous word (with related context) from a set of candidate images. The lack of textual description for the candidate image and the corresponding word{'}s ambiguity makes it a challenging problem. This paper describes teamPN{'}s multi-modal and modular approach to solving this in English track of the task. We efficiently used recent multi-modal pre-trained models backed by real-time multi-modal knowledge graphs to augment textual knowledge for the images and select the best matching image accordingly. We outperformed the baseline model by {\textasciitilde}5 points and proposed a unique approach that can further work as a framework for other modular and knowledge-backed solutions.
Katyal, Nikita Rajpoot, Pawan Tamilarasu, Subhan h Mustafi, Joy
teamPN at SemEval-2023 Task 1: Visual Word Sense Disambiguation Using Zero-Shot MultiModal Approach