@inproceedings{yuasa-etal-2023-multimodal,
    title = "Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model",
    author = "Yuasa, Ryoya  and
      Tamura, Akihiro  and
      Kajiwara, Tomoyuki  and
      Ninomiya, Takashi  and
      Kato, Tsuneo",
    editor = "Padmakumar, Vishakh  and
      Vallejo, Gisela  and
      Fu, Yao",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-srw.12",
    doi = "10.18653/v1/2023.acl-srw.12",
    pages = "76--82",
    abstract = "This study proposes a new multimodal neural machine translation (MNMT) model using synthetic images transformed by a latent diffusion model. MNMT translates a source language sentence based on its related image, but the image usually contains noisy information that are not relevant to the source language sentence. Our proposed method first generates a synthetic image corresponding to the content of the source language sentence by using a latent diffusion model and then performs translation based on the synthetic image. The experiments on the English-German translation tasks using the Multi30k dataset demonstrate the effectiveness of the proposed method.",
}

This study proposes a new multimodal neural machine translation (MNMT) model using synthetic images transformed by a latent diffusion model. MNMT translates a source language sentence based on its related image, but the image usually contains noisy information that are not relevant to the source language sentence. Our proposed method first generates a synthetic image corresponding to the content of the source language sentence by using a latent diffusion model and then performs translation based on the synthetic image. The experiments on the English-German translation tasks using the Multi30k dataset demonstrate the effectiveness of the proposed method.
Yuasa, Ryoya Tamura, Akihiro Kajiwara, Tomoyuki Ninomiya, Takashi Kato, Tsuneo
Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model