- J. Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, D. Wijaya, Chris Callison-Burch. 2018. Learning Translations via Images: A Large Multilingual Dataset and Comprehensive Study. Abstract: We conduct the most comprehensive study to date into translating words via images. To facilitate research on the task, we introduce a large-scale multilingual corpus of images, each labeled with the word it represents. Past datasets have been limited to only a few high-resource languages and unrealistically easy translation settings. In contrast, we have collected by far the largest available dataset for this task, with images for approximately 10,000 words in each of 100 languages. We run experiments on a dozen high resource languages and 20 low resources languages, demonstrating the effect of word concreteness and part-of-speech on translation quality. To improve image-based translation, we introduce a novel method of predicting word concreteness from images, which improves upon previously state-of-the-art unsupervised techniques. This allows us to predict when image-based translation may be effective, enabling consistent improvements to a state-of-the-art text-based word translation system. Our code and dataset will be made available.
