@inproceedings{wang-etal-2023-benchmarking,
    title = "Benchmarking Diverse-Modal Entity Linking with Generative Models",
    author = "Wang, Sijia  and
      Li, Alexander Hanbo  and
      Zhu, Henghui  and
      Zhang, Sheng  and
      Perera, Pramuditha  and
      Hang, Chung-Wei  and
      Ma, Jie  and
      Wang, William Yang  and
      Wang, Zhiguo  and
      Castelli, Vittorio  and
      Xiang, Bing  and
      Ng, Patrick",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.497",
    doi = "10.18653/v1/2023.findings-acl.497",
    pages = "7841--7857",
    abstract = "Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenge of DMEL, facilitating future researches on this task.",
}

Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenge of DMEL, facilitating future researches on this task.
Wang, Sijia Li, Alex er Hanbo Zhu, Henghui Zhang, Sheng Perera, Pramuditha Hang, Chung-Wei Ma, Jie Wang, William Yang Wang, Zhiguo Castelli, Vittorio Xiang, Bing Ng, Patrick
Benchmarking Diverse-Modal Entity Linking with Generative Models