- Jaime Arguello, Fernando Diaz, Jimmy J. Lin, A. Trotman. 2015. SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR). Abstract: Many, if not most, published research papers in Information Retrieval (IR) describe the following process: the authors identify an opportunity to improve on a particular IR task, implement an experimental system, and compare its performance against one or more baselines (or a control condition, in the case of a user study). The quality of the research is judged based on the magnitude of the improvement and whether the methodological choices suggest external validity and generalizability, for example, whether the experimental setup is “realistic” or whether the baseline methods reflect the state of the art. Unfortunately, research demonstrating the failure to reproduce or generalize previous results does not have a similar publication venue. This sort of result—often referred to as a ‘negative result’—serves to control the quality of published research in a scientific discipline and to better understand the limits of previously published methods. Publication venues for such research exist in fields such as ecology, biomedicine, pharmacy,, and social science. The SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR) aims to provide a venue for publication and discussion of IR research that fails to reproduce a previously published result under the same or similar experimental conditions (e.g., same test collection and system configuration) and research that demonstrates the failure to generalize an existing approach to a new domain. To this end, we have developed a set of categories covering different ways in which a result may
