- Fernando Diaz. 2014. Experimentation Standards for Crisis Informatics. Abstract: Easy access to online media has led to an escalation of researchers and companies developing information analysis systems for use in crisis response. These systems use state of the art text mining algorithms to extract information from online media to support crisis responders and other users. Published research in this area has primarily focused on systems leveraging public social media feeds such as Twitter. These results often are supported by samples of social media manually analyzed using a combination of hired assessors and data mining algorithms. Unfortunately, despite leveraging open data like Twitter, the current experimental environment for crisis informatics does not rely on open test collections or evaluation methodologies. Social media samples are often collected by individual research sites and not shared. Assessment is often conducted by crowd-workers or in-house editors, at times with little attention paid to inter-site consistency or to real use cases. Consequently, crisis informatics results are difficult to compare across research publications and to apply to real situations. This article makes the argument that information access algorithms that support crisis response require standard evaluation metrics and experimental corpora. There are two objectives for such a initiative: (a) encourage open and reproducible experimentation on crisis informatics, and (b) develop standard evaluation metrics for use by decision-makers. Information retrieval researchers are in a unique position to assist in the development of algorithms, methodology, and working systems for use in crisis response.
