CiteBench: A benchmark for Scientific Citation Text Generation

The publication rates are skyrocketing across many ﬁelds of science, and it is difﬁcult to stay up to date with the latest research. This makes automatically summarizing the latest ﬁndings and helping scholars to synthesize related work in a given area an attractive research objective. In this paper we study the problem of citation text generation , where given a set of cited papers and citing context the model should generate a citation text. While citation text generation has been tack-led in prior work, existing studies use different datasets and task deﬁnitions, which makes it hard to study citation text generation systematically. To address this, we propose C ITE B ENCH : a benchmark for citation text generation that uniﬁes the previous datasets and enables standardized evaluation of citation text generation models across task settings and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into task def-inition and evaluation to guide the future research in citation text generation. We make C ITE B ENCH publicly available at https:// github.com/UKPLab/citebench .