@inproceedings{taghavi-etal-2023-ebhaam,
    title = "Ebhaam at {S}em{E}val-2023 Task 1: A {CLIP}-Based Approach for Comparing Cross-modality and Unimodality in Visual Word Sense Disambiguation",
    author = "Taghavi, Zeinab  and
      Naeini, Parsa Haghighi  and
      Sadraei Javaheri, Mohammad Ali  and
      Gooran, Soroush  and
      Asgari, Ehsaneddin  and
      Rabiee, Hamid Reza  and
      Sameti, Hossein",
    editor = {Ojha, Atul Kr.  and
      Do{\u{g}}ru{\"o}z, A. Seza  and
      Da San Martino, Giovanni  and
      Tayyar Madabushi, Harish  and
      Kumar, Ritesh  and
      Sartori, Elisa},
    booktitle = "Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.semeval-1.269",
    doi = "10.18653/v1/2023.semeval-1.269",
    pages = "1960--1964",
    abstract = "This paper presents an approach to tackle the task of Visual Word Sense Disambiguation (Visual-WSD), which involves determining the most appropriate image to represent a given polysemous word in one of its particular senses. The proposed approach leverages the CLIP model, prompt engineering, and text-to-image models such as GLIDE and DALL-E 2 for both image retrieval and generation. To evaluate our approach, we participated in the SemEval 2023 shared task on {``}Visual Word Sense Disambiguation (Visual-WSD){''} using a zero-shot learning setting, where we compared the accuracy of different combinations of tools, including {``}Simple prompt-based{''} methods and {``}Generated prompt-based{''} methods for prompt engineering using completion models, and text-to-image models for changing input modality from text to image. Moreover, we explored the benefits of cross-modality evaluation between text and candidate images using CLIP. Our experimental results demonstrate that the proposed approach reaches better results than cross-modality approaches, highlighting the potential of prompt engineering and text-to-image models to improve accuracy in Visual-WSD tasks. We assessed our approach in a zero-shot learning scenario and attained an accuracy of 68.75{\textbackslash}{\%} in our best attempt.",
}

This paper presents an approach to tackle the task of Visual Word Sense Disambiguation (Visual-WSD), which involves determining the most appropriate image to represent a given polysemous word in one of its particular senses. The proposed approach leverages the CLIP model, prompt engineering, and text-to-image models such as GLIDE and DALL-E 2 for both image retrieval and generation. To evaluate our approach, we participated in the SemEval 2023 shared task on {``}Visual Word Sense Disambiguation (Visual-WSD){''} using a zero-shot learning setting, where we compared the accuracy of different combinations of tools, including {``}Simple prompt-based{''} methods and {``}Generated prompt-based{''} methods for prompt engineering using completion models, and text-to-image models for changing input modality from text to image. Moreover, we explored the benefits of cross-modality evaluation between text and candidate images using CLIP. Our experimental results demonstrate that the proposed approach reaches better results than cross-modality approaches, highlighting the potential of prompt engineering and text-to-image models to improve accuracy in Visual-WSD tasks. We assessed our approach in a zero-shot learning scenario and attained an accuracy of 68.75{\textbackslash}{\%} in our best attempt.
Taghavi, Zeinab Naeini, Parsa Haghighi Sadraei Javaheri, Mohammad Ali Gooran, Soroush Asgari, Ehsaneddin Rabiee, Hamid Reza Sameti, Hossein
Ebhaam at SemEval-2023 Task 1: A CLIP-Based Approach for Comparing Cross-modality and Unimodality in Visual Word Sense Disambiguation