@inproceedings{sharma-etal-2023-memex,
    title = "{MEMEX}: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization",
    author = "Sharma, Shivam  and
      S, Ramaneswaran  and
      Arora, Udit  and
      Akhtar, Md. Shad  and
      Chakraborty, Tanmoy",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.289",
    doi = "10.18653/v1/2023.acl-long.289",
    pages = "5272--5290",
    abstract = "Memes are a powerful tool for communication over social media. Their affinity for evolving across politics, history, and sociocultural phenomena renders them an ideal vehicle for communication. To comprehend the subtle message conveyed within a meme, one must understand the relevant background that facilitates its holistic assimilation. Besides digital archiving of memes and their metadata by a few websites like knowyourmeme.com, currently, there is no efficient way to deduce a meme{'}s context dynamically. In this work, we propose a novel task, MEMEX - given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. At first, we develop MCC (Meme Context Corpus), a novel dataset for MEMEX. Further, to benchmark MCC, we propose MIME (MultImodal Meme Explainer), a multimodal neural framework that uses external knowledge-enriched meme representation and a multi-level approach to capture the cross-modal semantic dependencies between the meme and the context. MIME surpasses several unimodal and multimodal systems and yields an absolute improvement of 4{\%} F1-score over the best baseline. Lastly, we conduct detailed analyses of MIME{'}s performance, highlighting the aspects that could lead to optimal modeling of cross-modal contextual associations.",
}

Memes are a powerful tool for communication over social media. Their affinity for evolving across politics, history, and sociocultural phenomena renders them an ideal vehicle for communication. To comprehend the subtle message conveyed within a meme, one must understand the relevant background that facilitates its holistic assimilation. Besides digital archiving of memes and their metadata by a few websites like knowyourmeme.com, currently, there is no efficient way to deduce a meme{'}s context dynamically. In this work, we propose a novel task, MEMEX - given a meme and a related document, the aim is to mine the context that succinctly explains the background of the meme. At first, we develop MCC (Meme Context Corpus), a novel dataset for MEMEX. Further, to benchmark MCC, we propose MIME (MultImodal Meme Explainer), a multimodal neural framework that uses external knowledge-enriched meme representation and a multi-level approach to capture the cross-modal semantic dependencies between the meme and the context. MIME surpasses several unimodal and multimodal systems and yields an absolute improvement of 4{\%} F1-score over the best baseline. Lastly, we conduct detailed analyses of MIME{'}s performance, highlighting the aspects that could lead to optimal modeling of cross-modal contextual associations.
Sharma, Shivam S, Ramaneswaran Arora, Udit Akhtar, Md. Shad Chakraborty, Tanmoy
MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization