- T. Sakai, N. Kando, Chuan-Jie Lin, Ruihua Song, Hideki Shima, T. Mitamura. 2008. NTCIR-7 ACLIA IR4QA Results based on Qrels Version 2. Abstract: This document is a postscript to the Overview of the NTCIR-7 ACLIA IR4QA Task [2]. At the NTCIR7 Workshop Meeting (December 2008), participating systems of IR4QA were evaluated based on “qrels version 1,” which covered the depth-30 pool for every topic and went further down the pool for a limited number of topics. Here, we report on revised results based on “qrels version 2” which covers the depth-100 pool for every topic. While the version 1 and version 2 results are generally in agreement, some differences in system rankings and significance test results suggest that the additional effort was worthwhile.
