The task of automatically generating sentential descriptions of image content has become increasingly popular in recent years, resulting#R##N#in the development of large-scale image description datasets and the proposal of various metrics for evaluating image description#R##N#generation systems. However, not much work has been done to analyse and understand both datasets and the metrics. In this paper,#R##N#we propose using a leave-one-out cross validation (LOOCV) process as a means to analyse multiply annotated, human-authored image#R##N#description datasets and the various evaluation metrics, i.e. evaluating one image description against other human-authored descriptions#R##N#of the same image. Such an evaluation process affords various insights into the image description datasets and evaluation metrics,#R##N#such as the variations of image descriptions within and across datasets and also what the metrics capture. We compute and analyse#R##N#(i) human upper-bound performance; (ii) ranked correlation between metric pairs across datasets; (iii) lower-bound performance by#R##N#comparing a set of descriptions describing one image to another sentence not describing that image. Interesting observations are made#R##N#about the evaluation metrics and image description datasets, and we conclude that such cross-validation methods are extremely useful#R##N#for assessing and gaining insights into image description datasets and evaluation metrics for image descriptions.
['Josiah Wang', 'Robert J. Gaizauskas']
Cross-validating Image Description Datasets and Evaluation Metrics.