- John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, D. Wijaya, Chris Callison-Burch. 2018. Supplemental Materials : Learning Translations via Images with a Massively Multilingual Image Dataset. Abstract: In order to ascertain that the images in our dataset were high-quality, it was necessary to manually validate that the images associated with a word were indeed related to that word. We limited our analysis to French, Indonesian, and English. We selected a random sample of 955 English words that had both French and Indonesian translations in our dataset as well as groundtruth concreteness scores. We gathered the first 10 images for each of these words in each of the three languages. We then crowdsourced annotations from workers on Amazon Mechanical Turk. We did so by providing workers each English word, and the associated 10 images, and asking them to identify which images did NOT relate to that word. For French and Indonesian, we provided workers the translated English word, rather than the original foreign word. Finally, alongside the 10 images, we provided two additional images associated with a different word in our dataset, to provide quality control. An example of the interface we showed the Mechanical Turkers is provided in Figure 2.
