This paper discusses the challenges in carrying out fair comparative evaluations of sentiment analysis systems. Firstly, these are due to#R##N#differences in corpus annotation guidelines and sentiment class distribution. Secondly, different systems often make different assumptions#R##N#about how to interpret certain statements, e.g. tweets with URLs. In order to study the impact of these on evaluation results, this paper#R##N#focuses on tweet sentiment analysis in particular. One existing and two newly created corpora are used, and the performance of four#R##N#different sentiment analysis systems is reported; we make our annotated datasets and sentiment analysis applications publicly available.#R##N#We see considerable variations in results across the different corpora, which calls into question the validity of many existing annotated#R##N#datasets and evaluations, and we make some observations about both the systems and the datasets as a result.
['Diana Maynard', 'Kalina Bontcheva']
Challenges of Evaluating Sentiment Analysis Tools on Social Media