# Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

## Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

[Original source](https://google.github.io/crossmodal-3600/)
task_categories:image-to-text task_ids:image-captioning source_datasets:wikipedia source_datasets:google language:ar language:bn language:cs language:da language:de language:el language:en language:es language:fa language:fi language:fil language:fr language:hi language:hr language:hu language:id language:it language:he language:ja language:ko language:mi language:nl language:no language:pl language:pt language:quz language:ro language:ru language:sv language:sw language:te language:th language:tr language:uk language:vi language:zh license:other crossmodal-3600 region:us
{"language": ["ar", "bn", "cs", "da", "de", "el", "en", "es", "fa", "fi", "fil", "fr", "hi", "hr", "hu", "id", "it", "he", "ja", "ko", "mi", "nl", false, "pl", "pt", "quz", "ro", "ru", "sv", "sw", "te", "th", "tr", "uk", "vi", "zh"], "license": "other", "source_datasets": ["wikipedia", "google"], "task_categories": ["image-to-text"], "task_ids": ["image-captioning"], "pretty_name": "Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset", "tags": ["crossmodal-3600"]}
TAGS
#task_categories-image-to-text #task_ids-image-captioning #source_datasets-wikipedia #source_datasets-google #language-Arabic #language-Bengali #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Persian #language-Finnish #language-Filipino #language-French #language-Hindi #language-Croatian #language-Hungarian #language-Indonesian #language-Italian #language-Hebrew #language-Japanese #language-Korean #language-Maori #language-Dutch #language-Norwegian #language-Polish #language-Portuguese #language-Cusco Quechua #language-Romanian #language-Russian #language-Swedish #language-Swahili (macrolanguage) #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Vietnamese #language-Chinese #license-other #crossmodal-3600 #region-us 

# Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset

## Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Original source
# Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset ## Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Original source
TAGS
#task_categories-image-to-text #task_ids-image-captioning #source_datasets-wikipedia #source_datasets-google #language-Arabic #language-Bengali #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Persian #language-Finnish #language-Filipino #language-French #language-Hindi #language-Croatian #language-Hungarian #language-Indonesian #language-Italian #language-Hebrew #language-Japanese #language-Korean #language-Maori #language-Dutch #language-Norwegian #language-Polish #language-Portuguese #language-Cusco Quechua #language-Romanian #language-Russian #language-Swedish #language-Swahili (macrolanguage) #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Vietnamese #language-Chinese #license-other #crossmodal-3600 #region-us 
 # Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset ## Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Original source
passage: TAGS
#task_categories-image-to-text #task_ids-image-captioning #source_datasets-wikipedia #source_datasets-google #language-Arabic #language-Bengali #language-Czech #language-Danish #language-German #language-Modern Greek (1453-) #language-English #language-Spanish #language-Persian #language-Finnish #language-Filipino #language-French #language-Hindi #language-Croatian #language-Hungarian #language-Indonesian #language-Italian #language-Hebrew #language-Japanese #language-Korean #language-Maori #language-Dutch #language-Norwegian #language-Polish #language-Portuguese #language-Cusco Quechua #language-Romanian #language-Russian #language-Swedish #language-Swahili (macrolanguage) #language-Telugu #language-Thai #language-Turkish #language-Ukrainian #language-Vietnamese #language-Chinese #license-other #crossmodal-3600 #region-us 
# Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset## Abstract

Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets. In this paper we present the Crossmodal-3600 dataset (XM3600 in short), a geographically-diverse set of 3600 images annotated with human-generated reference captions in 36 languages. The images were selected from across the world, covering regions where the 36 languages are spoken, and annotated with captions that achieve consistency in terms of style across all languages, while avoiding annotation artifacts due to direct translation. We apply this benchmark to model selection for massively multilingual image captioning models, and show strong correlation results with human evaluations when using XM3600 as golden references for automatic metrics.

Original source