@inproceedings{santhanam-etal-2023-moving,
    title = "Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking",
    author = "Santhanam, Keshav  and
      Saad-Falcon, Jon  and
      Franz, Martin  and
      Khattab, Omar  and
      Sil, Avi  and
      Florian, Radu  and
      Sultan, Md Arafat  and
      Roukos, Salim  and
      Zaharia, Matei  and
      Potts, Christopher",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.738",
    doi = "10.18653/v1/2023.findings-acl.738",
    pages = "11613--11628",
    abstract = "Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks. Unfortunately, some dimensions of this progress are illusory: the majority of the popular IR benchmarks today focus exclusively on downstream task accuracy and thus conceal the costs incurred by systems that trade away efficiency for quality. Latency, hardware cost, and other efficiency considerations are paramount to the deployment of IR systems in user-facing settings. We propose that IR benchmarks structure their evaluation methodology to include not only metrics of accuracy, but also efficiency considerations such as a query latency and the corresponding cost budget for a reproducible hardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we show how the best choice of IR system varies according to how these efficiency considerations are chosen and weighed. We hope that future benchmarks will adopt these guidelines toward more holistic IR evaluation.",
}

Neural information retrieval (IR) systems have progressed rapidly in recent years, in large part due to the release of publicly available benchmarking tasks. Unfortunately, some dimensions of this progress are illusory: the majority of the popular IR benchmarks today focus exclusively on downstream task accuracy and thus conceal the costs incurred by systems that trade away efficiency for quality. Latency, hardware cost, and other efficiency considerations are paramount to the deployment of IR systems in user-facing settings. We propose that IR benchmarks structure their evaluation methodology to include not only metrics of accuracy, but also efficiency considerations such as a query latency and the corresponding cost budget for a reproducible hardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we show how the best choice of IR system varies according to how these efficiency considerations are chosen and weighed. We hope that future benchmarks will adopt these guidelines toward more holistic IR evaluation.
Santhanam, Keshav Saad-Falcon, Jon Franz, Martin Khattab, Omar Sil, Avi Florian, Radu Sultan, Md Arafat Roukos, Salim Zaharia, Matei Potts, Christopher
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking