Are there NLP datasets that study the problem of retrieving multiple answers given a query?
Multi-Answer Retrieval Related work (Min
et al., 2021; Amouyal et al., 2022) also studies
the problem of multi-answer retrieval, where sys-
tems are required to predict multiple distinct an-
swers for a query. Min et al. (2021) adapt existing
datasets (for example, WebQuestionsSP (Yih et al.,
2016)) to study this setting and propose a new met-
ric, MRecall@K, to evaluate exhaustive recall of
multiple answers. We also consider the problem of
multi-answer set retrieval, but consider queries that implicitly contain set constraints.
In concurrent work, RomQA (Zhong et al., 2022)
proposes an open-domain QA dataset, focusing on
combinations of constraints extracted from Wiki-
data. RomQA shares our motivation to enable an-
swering queries with multiple constraints, which
have possibly large answer sets. To make attribu-
tion to evidence feasible without human annotation,
RomQA focuses on questions whose component
constraints can be verified from single entity-linked
sentences from Wikipedia abstracts, annotated with
relations automatically through distant supervision,
with high precision but possibly low recall (T-Rex
corpus). In QUEST, we broaden the scope of query-
evidence matching operations by allowing for at-
tribution through more global, document-level in-
ference. To make human annotation for attribution
feasible, we limit the answer set size and the evi-
dence for an answer to a single document
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Existing benchmarks for open-domain question answering (ODQA) typically focus on questions whose answers can be extracted from a single paragraph. By contrast, many natural questions, such as"What players were drafted by the Brooklyn Nets?"have a list of answers. Answering such questions requires retrieving and reading from many passages, in a large corpus. We introduce QAMPARI, an ODQA benchmark, where question answers are lists of entities, spread across many paragraphs. We created QAMPARI by (a) generating questions with multiple answers from Wikipedia's knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. We train ODQA models from the retrieve-and-read family and find that QAMPARI is challenging in terms of both passage retrieval and answer generation, reaching an F1 score of 32.8 at best. Our results highlight the need for developing ODQA models that handle a broad range of question types, including single and multi-answer questions.
[{"first":"Samuel","last":"Amouyal","middle":["Joseph"]},{"first":"Tomer","last":"Wolfson","middle":[]},{"first":"Ohad","last":"Rubin","middle":[]},{"first":"Ori","last":"Yoran","middle":[]},{"first":"Jonathan","last":"Herzig","middle":[]},{"first":"Jonathan","last":"Berant","middle":[]}]
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Formulating selective information needs results in queries that implicitly specify set operations, such as intersection, union, and difference. For instance, one might search for “shorebirds that are not sandpipers” or “science-fiction films shot in England”. To study the ability of retrieval systems to meet such information needs, we construct QUEST, a dataset of 3357 natural language queries with implicit set operations, that map to a set of entities corresponding to Wikipedia documents. The dataset challenges models to match multiple constraints mentioned in queries with corresponding evidence in documents and correctly perform various set operations. The dataset is constructed semi-automatically using Wikipedia category names. Queries are automatically composed from individual categories, then paraphrased and further validated for naturalness and fluency by crowdworkers. Crowdworkers also assess the relevance of entities based on their documents and highlight attribution of query constraints to spans of document text. We analyze several modern retrieval systems, finding that they often struggle on such queries. Queries involving negation and conjunction are particularly challenging and systems are further challenged with combinations of these operations.
[{"first":"Chaitanya","last":"Malaviya","middle":[]},{"first":"Peter","last":"Shaw","middle":[]},{"first":"Ming-Wei","last":"Chang","middle":[]},{"first":"Kenton","last":"Lee","middle":[]},{"first":"Kristina","last":"Toutanova","middle":[]}]
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We study multi-answer retrieval, an under-explored problem that requires retrieving passages to cover multiple distinct answers for a given question. This task requires joint modeling of retrieved passages, as models should not repeatedly retrieve passages containing the same answer at the cost of missing a different valid answer. Prior work focusing on single-answer retrieval is limited as it cannot reason about the set of passages jointly. In this paper, we introduce JPR, a joint passage retrieval model focusing on reranking. To model the joint probability of the retrieved passages, JPR makes use of an autoregressive reranker that selects a sequence of passages, equipped with novel training and decoding algorithms. Compared to prior approaches, JPR achieves significantly better answer coverage on three multi-answer datasets. When combined with downstream question answering, the improved retrieval enables larger answer generation models since they need to consider fewer passages, establishing a new state-of-the-art.
[{"first":"Sewon","last":"Min","middle":[]},{"first":"Kenton","last":"Lee","middle":[]},{"first":"Ming-Wei","last":"Chang","middle":[]},{"first":"Kristina","last":"Toutanova","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]}]
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Open-domain question answering (ODQA) is a core language understanding task concerned with answering factoid questions over large document collections (Voorhees and Tice, 2000;Brill et al., 2002). Due to its wide applicability, ODQA has received substantial attention in recent years (Chen et al., 2017;. Typically, systems solving ODQA tasks follow the "retrieve-and-read" paradigm, where a retriever first retrieves a set of candidate passages, followed by a reader which receives the retrieved passages and produces the final answer.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The retrieve-and-read paradigm has been shown to be effective for benchmarks such as Natural Questions (NQ) (Kwiatkowski et al., 2019) and  TriviaQA (Joshi et al., 2017), where the answer is typically a single phrase from a single passage. However, in many cases, a question might have many answers that are spread across multiple passages. Consider the example in Fig. 1. Eric Newman produced multiple movies, so finding them along with their directors requires incorporating information from many passages. Such questions pose two main challenges to retrieve-and-read systems. First, as there are multiple answers, that can be far apart, the reader model must reason over a long text sequence to generate all of the correct answers. Second, since the reader is computationally constrained to process at most K passages, the retriever must score all necessary passages at its top-K results, which is challenging and even impossible when the number of such passages is ≥K.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
While recent works explored questions that involve reading multiple passages, their overall number of passages was quite small. AMBIGQA  studied ambiguous questions from NQ with several plausible answers. However, as 70% of its questions have at most 2 answers, retrieveand-read models can be easily adapted to the AM-BIGQA task. The HOTPOTQA (Yang et al., 2018) dataset focused on multi-hop reasoning, but its questions require no more than 2 passages to answer. Last, WIKINLDB (Thorne et al., 2021) was proposed as a benchmark for testing reasoning over multiple facts. However, WIKINLDB restricted its text corpus to databases of 1,000 facts at most, making it significantly smaller than standard ODQA corpora. Moreover, these facts are model-generated utterances rather than natural language passages.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In this work, we present QAMPARI, a benchmark for Questions with many Answers over Multiple Paragraphs, Indeed. All questions in QAMPARI have at least 5 answers, with an average of 13 answers per question. Examples are semi-automatically generated using two data sources, Wikidata (Vrandečić and Krötzsch, 2014) and Wikipedia tables. We automatically generate multi-answer questions of the form "What/Who has [relation] with [entity]?" and convert these into pseudo-language using manually defined templates. We then verify our questions are answerable, given Wikipedia sentences, by automatically extracting evidence passages for all answers. Finally, we use crowdsourcing to validate example correctness, and to paraphrase questions from pseudo-language into natural language (Wang et al., 2015). To further increase the richness of questions, we also generate composition questions, that compose two relations (as in Fig. 1), and intersection questions, such as "What movies were produced and directed by Clint Eastwood?". Overall, QAMPARI contains 2K test questions and more than 60K training examplessee Tab. 1 for some examples. We evaluate models from the retrieve-and-read family and find that they struggle on QAMPARI. Specifically, we use a BM25 (Robertson and Zaragoza, 2009) retriever followed by one of two readers: (1) a RAG-style reader  that decodes an answer or abstains given each passage independently, and (2) an FiD reader , that directly decodes the answer list given encoded representations of many passages. To evaluate, we compare the set of answers predicted by a model to the gold set of answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
When training models in a multi-task setting of NQ and QAMPARI we observe that QAMPARI is challenging in terms of both passage retrieval and answer generation. Models reach an F 1 score of 26.6 at most. In addition, models are able to return over 80% of the correct answers only for 30% of our examples, well below typical performance on single-answer datasets such as NQ.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To summarize, we present QAMPARI, a challenging benchmark for evaluating the ability of ODQA models to handle questions with many answers over multiple passages from Wikipedia. We advocate to evaluate ODQA models not on QAM-PARI alone, but alongside benchmarks such as NQ and TriviaQA. This joint evaluation would better test for ODQA models ability to handle both singleand multi-answer questions, tests which are conspicuously absent from current benchmarks.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The QAMPARI benchmark, models and relevant codebase are available at: https:// samsam3232.github.io/qampari/.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We present our process of generating examples for QAMPARI. Each example in QAMPARI is a triple (q, A, P), where q is a question, A is a set of answers and P is a set of passages from our target corpus. Each answer a ∈ A has 1-2 evidence passages from P (see Fig. 1). We define passages to be consecutive sentences from our corpus (Wikipedia), that span at most 100 words. As our focus is on questions with many answers, all examples in QAMPARI have |A| ≥ 5.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We generate examples using two steps. First, we generate simple questions that involve a single entity and a single relation, e.g. "Who was drafted by the Brooklyn Nets?" ( §2.1). Then, we expand such questions in order to generate complex questions that involve intersection and composition operations ( §2.2).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To increase diversity, our questions are generated using two data sources, Wikidata and Wikipedia tables. We first describe the example generation over Wikidata, then briefly present the generation process from Wikipedia tables in §2.3. In both cases, we ensure all answers can be derived from evidence passages in Wikipedia. 1 Tab. 1 presents examples from each data source and question type.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Notation We introduce notation for formal queries over Wikidata, used for explaining our example generation process. Wikidata is a knowledge graph, K, that can be viewed as a set of labeled edges (e 1 , r, e 2 ). Graph nodes e 1 , e 2 ∈ E are entities which are connected by an edge labeled with the relation r ∈ R. For example, one possible labeled edge is (BarackObama, ReceivedAward, NobelPeacePrize). What are the museums found in Concord, Massachusetts?
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The Wayside V. One can query K by applying a relation r over an entity e, resulting in a simple query r(e) whose denotation (answer set) is r(e) = {e i | (e i , r, e) ∈ K}. Composition queries can be formed by applying a relation over the result of a simple query. We denote a composition query by r 2 (r 1 (e)), and its denotation is r 2 (r 1 (e)) = {e i | ∃e j s.t (e i , r 2 , e j ) ∈ K ∧ (e j , r 1 , e) ∈ K}. Last, an intersection query r 1 (e 1 ) r 2 (e 2 ) corresponds to the intersection of two simple queries, that is, r 1 (e 1 ) r 2 (e 2 ) = {e i | (e i , r 1 , e 1 ) ∈ K ∧ (e i , r 2 , e 2 ) ∈ K}.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Fig. 2 provides an overview of our semi-automatic procedure for creating simple question examples: (i) We manually define query templates, (ii) automatically populate query templates using K to create queries with a sufficiently large number of answers in K, (iii) automatically identify evidence passages for the answers and filter out noisy examples, (iv) map query templates to question templates to obtain pseudo-language questions, and (v) validate answers and paraphrase pseudo-language questions through crowdsourcing. Next, we describe each of these steps in detail.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Generating query templates We manually select a set of 135 relationsR ⊂ R, which will be used in our query templates. We select relations going through the most frequent relations in Wikidata and choosing ones for which denotations often contain many entities (e.g., ReceivedAward). The full list of relations is provided in App. A. For each relation, we manually write a template that will be used to map queries to pseudo-language questions. For example, the template for ReceivedAward is "Who received the award X?" Some relations are underspecified -for example, LocatedIn can describe the location of buildings, geographical features, and cities. When generating synthetic questions, this leads to vague questions such as "What is located in Paris?". To address this issue we manually split these to typed relations that specify the semantic type of their answers/denotations. This split is done using the type hierarchy given in Wikidata and given the type t of answer entities. We denote typed relations by r t , and the denotation of r t (e) comprises all entities of type t returned by r(e). 2 For example, the entity The Louvre has type cultural organization, and we can map the relevant query template to the pseudo-language question "Which cultural organization is located in Paris?".
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Simple query generation We instantiate all possible simple queries using all r ∈R and entities e in Wikidata. For a relation r (or r t ), we keep the query r(e) iff |r(e)| ≥ 5. We denote this set of instantiated simple queries by S, which contains 1,431,268 simple queries.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Finding evidence sentences As our goal is to create an ODQA benchmark, we must verify that every answer is indeed found in our target text corpus. We do this by (a) identifying candidate evidence sentences from Wikipedia, and (b) verifying that they entail the answer, using a Natural Language Inference (NLI) model.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Specifically, every simple query-answer pair can be viewed as a triple (e 1 , r, e 2 ). We use a "distant supervision" approach (Mintz et al., 2009), similar to KELM (Agarwal et al., 2021), and define any sentence in the Wikipedia page of entity e 1 that contains the entity e 2 , or one of its Wikidata aliases, as a candidate evidence sentence (and vice versa in the Wikipedia page of e 2 ). For example, in Fig. 2, the evidence for the triple (BarackObama, ReceivedAward, NobelPeacePrize) appears on the Wikipedia page of Barack Obama, where the phrase Nobel Peace Prize appears.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Aligning Wikipedia sentences to Wikidata can lead to false positives. For example, for the triple (TheGoonies, HasScreenwriter, StevenSpielberg), most mentions of Spielberg in the page TheGoonies are not as a screenwriter. To account for this, we use an off-the-shelf NLI model. 3 For every answer, we consider each candidate evidence sentence along with its two preceding sentences, and check whether they entail the hypothesis phrase describing triple (e 1 , r, e 2 ). We use templates to phrase triples as short declarative sentences ("The Goonies has Steven Spielberg as screenwriter"
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
). An answer is validated if there is an evidence sentence that entails the triple. Manual analysis shows this process eliminates 70% of false positives (sentences not entailing the triple), while removing only 7.5% of the correct alignments.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Query filtering After finding evidence sentences, we only keep queries that at least 80% of their answers were validated and their number of validated answers lies between 5 and 200. The resulting set contains 60,792 simple queries, where each query has a set of validated answers, A, and of passages P that contain the identified evidence sentences. 4 We now describe how simple queries are expanded to complex queries.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To increase the diversity of QAMPARI, we automatically expand simple queries to composition and intersection queries, for which answers require reading two passages.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Intersection Intersection queries are generated by finding two simple queries such that the size of the intersection of their denotations is at least 5. To avoid improbable questions such as "Which competition was won by Manchester City and had Manchester City as a participant?", we add a constraint that the denotation of one of the simple queries cannot be a subset of the other. Formally, the set of intersection queries are all queries r 1 (e 1 ) r 2 (e 2 ) such that | r 2 (e 2 ) r 1 (e 1 ) | ≥ 5, r 1 (e 1 ) r 2 (e 2 ) and r 2 (e 2 ) r 1 (e 1 ) . Pseudo-language questions are generated by heuristically combining the two simple questions, for example "Which television program had Chris Carter as screenwriter and had Frank Spotnitz as screenwriter?". There is no need to perform answer validation since all of the underlying intersecting answers were already validated.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Composition To create composition queries, we manually handpick a set of 423 relations R comp ⊂ R (list in our codebase), in a process similar to simple queries. Then, we generate all the possible composition queries r 2 (r 1 (e)) such that r 1 (e) ∈ S, r 2 ∈ R comp , and | r 2 (r 1 (e)) | ≥ 5. An example composition query is "What is the height of buildings located in Dubai?".
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Unlike intersection queries, in composition queries we need to validate that our new triples (e i , r 2 , e j ), where e j ∈ r 1 (e) , are indeed supported by Wikipedia sentences. We use the same procedure to find evidence sentences for triples (e i , r 2 , e j ), and consider an answer e i as validated if both (e i , r 2 , e j ) and (e j , r 1 , e) can be aligned to Wikipedia. We keep all complex queries where 80% of the answers are validated. Finally, we manually define templates for relations in R comp to generate pseudo-language questions.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To further diversify QAMPARI, we create an analogous pipeline for generating simple and composition questions from Wikipedia tables, with more open-ended relations compared to Wikidata. We briefly describe this pipeline.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We look at all Wikipedia tables with title "List of X" that have at least 5 rows, in total, 1,897 tables. We find the "key" column, c key in each table using the table classifier from Talmor et al. (2021), which outputs the column of entities that the table describes. For example, in the table List of nuclear whistle blowers, c key is 'name' and specifies the whistle-blower names. This naturally creates simple questions of the form "Who or what is X?".
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Simple questions are expanded to composition questions by looking at non-key columns, c non-key and asking what rows in the table have the value v in column c non-key . For example, what is the value in the column 'Year' for nuclear whistle-blowers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Questions from Wikipedia are validated using a procedure similar to Wikidata. For each answer entity e, we validate that the Wikipedia page for e contains the relevant words that are part of the name of the table as well as the value (for composition questions), and only keep questions where 80% of the table rows are validated and the number of validated answers is at least 5. Overall, we generate 170 simple questions and 6,036 composition questions using this process.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We provide a training set with QAMPARI, whose goal is to teach the model to handle multi-answer questions. However, we do not want the model to use the training set to memorize how particular Wikidata relations map to text patterns, as our goal is to test language understanding regardless of Wikidata relations.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Consequently, we perform a relation split, randomly splitting the setR into two equally-sized setsR train andR test . Simple queries are assigned to the train/test set based on their relation, composition queries r 2 (r 1 (e)) are assigned to the test set iff either r 1 or r 2 are inR test , and intersection queries r 1 (e 1 ) r 2 (e 2 ) are placed in the test set iff both r 1 and r 2 are inR test .
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
At this point, we can create the final train/development/test split (see also Tab. 2). The main bottleneck in our example generation pipeline is validation of the test set through crowdsourcing ( §2.5), since each question requires validating all of the answers in the list. Thus, we pre-determine the test set to contain 1,000 simple questions (830 from Wikidata and 170 from Wikipedia tables) and 1,000 complex questions (400 Wikidata composition questions, 400 Wikidata intersection questions, and 200 Wikipedia tables composition questions). For simple Wikidata questions, we sample 830 questions such that the distribution over relations fromR test is roughly uniform. All Wikipedia tables simple questions are placed in the test set, and for complex questions we randomly sample the pre-determined number from the set of generated questions. Last, the test set is randomly split in half to a development set and test set. We also sub-sample training set examples, such that each relation appears in at most 1,000 examples.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We validate the correctness of development and test examples and paraphrase them into natural language through crowdsourcing.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Correctness validation For every question and answer, we present a crowdsourcing worker with the question, the answer, and links to the Wikipedia page (or pages for complex questions) with the evidence passage. We then ask the worker to check if the question can be answered from the given pages, using the text only (no infoboxes or tables).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
As the grand majority of examples are correct, we control for quality by injecting wrong answers in 10% of the cases and reject workers that fail to identify those wrong answers. Moreover, we manually verify 5% of examples marked as correct and all examples marked as incorrect, and again reject low-performing workers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Overall, 24 annotators validated 30,259 answers for an average pay of 12.5$ per hour. We find that our process for generating examples is accurate, with 96.6% of the answers validated. Nonvalidated questions were replaced until 2,000 questions were validated. A question is defined nonvalidated if its number of distinct answers goes bellow 5. Snapshots from the presented tasks are in App. B.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Paraphrasing Since our questions are in pseudolanguage, we follow past work (Wang et al., 2015)   and ask workers to re-phrase the questions in the development/test sets. We restrict this task to workers from the US or the UK who pass a qualification test. We randomly verified half of the paraphrases for each worker for quality assurance. Tab. 2 provides key statistics on QAMPARI (development and test statistics are aggregated). Test examples in QAMPARI have 13.23 answers on average and a median of 7 answers. This is substantially higher than, e.g., AmbigQA, where the median is 2. Simple questions tend to have more answers than complex questions and are typically shorter than complex questions. Test examples are shorter than training examples since they were rephrased by annotators who used more concise and natural phrasings. Figure 3a shows a binned distribution over the number of answers on the development and test sets. We can see that roughly half of the questions have 8 answers or more, a non-negligible fraction (20%) have more than 15 answers, and 3.5% have more than 50.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Tab. 3 shows the frequency of the top-10 relations in the training and test sets of QAMPARI, as well as the top-10 semantic types of answers, which can be inferred from relation. Mirroring Wikipedia, we observe that the relations and types are focused on people, locations, and the entertainment world.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Manual analysis As mentioned, we use crowdsourcing to validate that gold answers have evi-dence in Wikipedia. However, Wikipedia can contain additional correct answers that are not in Wikidata/Wikipedia tables. Since manually annotating all correct answers on Wikipedia is virtually impossible, we estimate the frequency of this phenomenon. We sample 50 examples from QAM-PARI, and ask an expert annotator to find more correct answers on Wikipedia within 5-10 min. Fig. 3b shows the results of the manual analysis. In 20% of the questions, no additional answers were found, and for more than 60% of the questions, the gold set of answers contains at least half of the answers found by the annotator. Overall, precision estimates of models against the gold set of answers should be taken with a grain of salt.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We now turn to our experimental evaluation.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
ODQA models typically fall into either the retrieveand-read framework or the 'closed-book' framework (Roberts et al., 2020), where the model provides answers using knowledge encoded in its parameters. Here, we use baseline models from the retrieve-and-read family as they reach state-of-theart performance and are more parameter-efficient.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Retriever We use BM25 (Robertson and Zaragoza, 2009) to index Wikipedia passages and query the index. As mentioned ( §2), we chunk Wikipedia into passages, each containing consecutive sentences with at most 100 tokens, similar to DPR .
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
BM25 in a strong sparse retrieval baseline that scores question-passage pairs based on lexical similarity. BM25 is notoriously hard to beat using unsupervised methods Ram et al., 2022), and obtains respectable performance even compared to supervised methods. We leave training a dense passage retriever  for the near future. Specifically, we return from BM25 the top-200 passages for each question.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Reader We evaluate two encoder-decoder readers, both initialized from T5 (Raffel et al., 2019). a "RAG-like" model ) that generates answers from each passage independently and Fusion-in-Decoder (FiD) , which encodes multiple passages and decodes a list of answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
For RAG, the model encodes each of the 200 retrieved passages independently and outputs either   "Not Relevant" for no answer, or "Answer: X" for some X. The predicted list of answers is the union of all answers predicted across passages. We train RAG in the following manner. For simple questions with answers A = {a i } |A| i=1 , we take the evidence passage p i for each a i and train the model to decode a i given the passage p i and the question q. For complex questions, where each answer a i requires 1-2 passages of evidence, we create a positive examples from each of the two evidence passages. Last, to train RAG to emit "Not Relevant", we sample for each positive passage a negative passage p neg by selecting the top-scoring BM25 passage that is not an evidence passage and does not contain the answer. We then train the model to emit "Not Relevant" given q and p neg .
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
FiD  uses an encoder to encode each of the retrieved passages (along with the input question) and the decoder attends to the encoded representations to decode a list of answers. Since each example is now an input-output pair, we train with standard maximum likelihood.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
FiD is computationally expensive as the decoder attends to a large number of encoded tokens and the generated output is long. Thus, we can only fit the top-50 passages from BM25 using T5-large on a single A100 GPU. We apply teacher forcing (Williams and Zipser, 1989) at training time, i.e., the model is given the gold passages and then the top scoring passages according to BM25. If |P| > 50, the model sees a random sample of size 50 from the set of gold passages.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
As explained in §1, we create QAMPARI as a benchmark to be evaluated alongside other ODQA benchmarks such as NQ. Since QAMPARI is semiautomatically generated, one can develop models tailored for QAMPARI, but our goal should be to have a single model that can perform well across a wide variety of question types. Thus, we train and test models on QAMPARI only, but also in a multi-task setup, where models are trained on both NQ and QAMPARI.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Our main metrics are recall, precision, and F 1 . Specifically, for a test example (q, P, A), and a predicted set of answers A pred , recall, precision, and F 1 are computed in the typical manner, comparing A and A pred , while allowing for aliases, that is, a gold answer is considered covered if it or one of its aliases appears as a predicted answer. A model score is given by averaging recall, precision, and F 1 across all examples. To get a sense of the average accuracy across examples we measure the fraction of examples where F 1 is at least 0.5 (%F 1 ≥0.5) and the fraction where recall is at least 0.8 (%Recall≥0.8). We focus on recall and F 1 since, as shown in §3, precision is a only approximate due to additional answers not covered by Wikidata. For NQ, we use the standard exact match (EM) metric.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We evaluate the retriever with RECALL@K, that is, the fraction of answers that appear in the top-K retrieved passages, averaged across examples. This metric comes in two variants: (a) Answer RECALL@K ( ARECALL@K): for every gold answer whether it or one of its aliases appear in the   top-K retrieved passages. This is a loose metric since an answer can appear even if the evidence does not support the answer; (b) Evidence RECALL@K (ERECALL@K): since we have evidence paragraphs for every answer, we consider for every gold answer the fraction of evidence passages in the top-K retrieved passages. This is a strict metric since an answer can sometimes be answered by passages other than the ones we identified.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Tab. 5 shows test retrieval results of BM25 on QAMPARI. We observe that even given 200 retrieved passages average recall is only 55.2-62.7. Moreover, for low values of K, average recall is quite low (17.4-25.9 for K=10, 28.8-38.1 for K=25), and there is still substantial improvement from K=100 to K=200. These results illustrate the challenge in retrieving a large number of relevant passages into the "beam" of retrieved results. Tab. 4 shows test results on QAMPARI. Overall, performance on QAMPARI is low, where %F 1 ≥0.5 is at most 20.1 (FiD-Large, trained on QAM-PARI only (QO)) and %Recall≥0.8 is at most 30.9 (RAG-large, multi-task (MT) training). When training on both NQ and QAMPARI (MT), results on QAMPARI are lower than NQ (despite the more permissive evaluation metric), illustrating the challenge in answering multi-answer questions.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
RAG is recall-oriented, while FiD is precisionoriented. This can be attributed to the number of negative passages at test time, which is larger than training time for RAG, and also to RAG taking 200 passages as input, while FiD only takes 50. F 1 for both models is around 25-27, and %Recall≥0.8 is extremely low for FiD (3.9% at most).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The MT setup benefits RAG, but not FiD. We hypothesize this is because training on NQ acts as a regularizer that reduces the number of predicted answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Question type analysis We break down by question type the test performance of both FiD-Large and RAG-Large trained on QAMPARI only (Tab. 6). The mean number of answers predicted by RAG is dramatically higher than FiD (30.5 vs 5.1), which agrees with our observation that RAG is recall-oriented, while FiD is precision oriented.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Surprisingly, for FiD, performance on simple questions is lower than performance on complex questions, and specifically, intersection questions seem easiest. Possible explanations are: (a) simple questions have on average more answers (see Tab. 2), which makes them harder, and (b) Models can predict the right answer, even with only one of the evidence passages, due to either "shortcuts" (Chen and Durrett, 2019), or knowledge encoded in the parameters (Longpre et al., 2021).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Unlike FiD, the performance of RAG on Wikidata composition questions is lower than simple questions, potentially since it cannot reason over multiple passages and can only emit answers from each passage independently. Specifically, the recall of RAG on composition questions is less than half the recall on intersection questions. An analogous analysis for the MT setup is in Fig. 8  Model precision As mentioned ( §3), precision is a lower bound due to potential additional correct answers on Wikipedia. To estimate the effect of this, we randomly sampled 30 questions from the development set and manually computed "true" precision by checking whether answers are correct on Wikipedia. For FiD-Large (QO), estimated precision on this set is 36.0%, while true precision is 67.3%. For RAG-Large (MT) estimated precision is 21.6%, but true precision is 42.1%. This demonstrates that precision should be used to rank models, but not as an absolute measure of true precision.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Datasets Work on ODQA has mostly focused on datasets where an answer is a single phrase from a single passage, such as NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), WebQuestions (Berant et al., 2013), Cu-ratedTREC (Baudiš and Šedivý, 2015), SQuAD (Rajpurkar et al., 2016), and EntityQuestions (Sciavolino et al., 2021)). Multi-hop datasets, such as HotpotQA (Yang et al., 2018), WikiHop (Welbl et al., 2018), and Multi-evidence FEVER (Thorne et al., 2018) output a single phrase but require reasoning over more than one passage. AmbigQA  deals with questions that have multiple answers due to ambiguity. QAMPARI has questions with many answers and thus requires retrieving a much larger number of passages and generating answers from them. WikiNLDB (Thorne et al., 2021) tests the ability of models to reason over sets of facts. However, retrieval is restricted to at most 1,000 facts, which are model-generated. Here, our corpus is Wikipedia, and we retrieve much longer passages, which makes the setup more realistic, and retrieval and generation more challenging.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The retrieve-and-read paradigm is currently the prevailing approach in ODQA, due to its ability to scale to large corpora (Chen et al., 2017;Yang et al., 2019;Sachan et al., 2021). However, when the number of evidence passages is large, retrieve-and-read models need to fetch all relevant passages to generate the correct answer. An alternative approach is to use closed-book models, where information is encoded in the model parameters (Roberts et al., 2020;Tay et al., 2022), however this entails using very high-capacity models. Last, a less explored model family that is potentially suitable for large answer sets are virtual knowledge-bases, which encode a corpus into a differentiable knowledge-base that is amenable for retrieval and logical operations (Sun et al., 2021;Dhingra et al., 2020).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We presented QAMPARI, an ODQA benchmark which focuses on the ability of models to handle questions that have many answers and thus require reading multiple text passages. QAMPARI is semi-automatically generated, where examples are generated from WikiData and Wikipedia tables, and manual work is done only to prepare pseudolanguage templates, to validate examples, and to re-phrase questions. We evaluate strong baselines on QAMPARI and show that the need to retrieve a large number of passages and generate long lists is challenging for state-of-the-art models from the retrieve-and-read family.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We view multi-answer questions as an integral part of the ODQA problem that has thus far been neglected, and invite the research community to develop models that can simultaneously answer a wide range of question types, include single-and multi-answer questions.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In Tab. 7. we gathered all the 135 relations we used to create our simple questions. The 423 relations used to create our composition questions can be found in our code base.  Table 8 breaks down the performance of RAG-Large and FiD-Large in the multi-task setup.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In Tab. 9 we present results analogous to those in Tab. 4 for the development set.    
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
People often express their information needs with multiple preferences or constraints. Queries corresponding to such needs typically implicitly express set operations such as intersection, difference, and union. For example, a movie-goer might be looking for a science-fiction film from the 90s which does not feature aliens and a reader might be interested in a historical fiction novel set in France. Similarly, * Work done during an internship at Google. 1 The dataset is available at https://github.com/ google-research/language/tree/master/language/ quest.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Birds found in the Venezuelan Andes but not in Colombia a botanist attempting to identify a species based on their recollection might search for shrubs that are evergreen and found in Panama. Further, if the set of entities that satisfy the constraints is relatively small, a reader may like to see and explore an exhaustive list of these entities. In addition, to verify and trust a system's recommendations, users benefit from being shown evidence from trusted sources .
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Addressing such queries has been primarily studied in the context of question answering with structured knowledge bases (KBs), where query constraints are grounded to predefined predicates and symbolically executed. However, KBs can be incomplete and expensive to curate and maintain. Meanwhile, advances in information retrieval may enable developing systems that can address such queries without relying on structured KBs, by matching query constraints directly to supporting evidence in text documents. However, queries that combine multiple constraints with implicit set operations are not well represented in existing retrieval benchmarks such as MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019). Also, such datasets do not focus on retrieving an exhaustive document set, instead limiting annotation to the top few results of a baseline information retrieval system. To analyze retrieval system performance on such queries, we present QUEST, a dataset with natural language queries from four domains, that are mapped to relatively comprehensive sets of entities corresponding to Wikipedia pages. We use categories and their mapping to entities in Wikipedia as a building block for our dataset construction approach, but do not allow access to this semistructured data source at inference time, to simulate text-based retrieval. Wikipedia categories represent a broad set of natural language descriptions of entity properties and often correspond to selective information need queries that could be plausibly issued by a search engine user. The relationship between property names and document text is often subtle and requires sophisticated reasoning to determine, representing the natural language inference challenge inherent in the task.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Our dataset construction process is outlined in Figure 1. The base queries are semi-automatically generated using Wikipedia category names. To construct complex queries, we sample category names and compose them by using pre-defined templates (for example, A ∩ B \ C). Next, we ask crowdworkers to paraphrase these automatically generated queries, while ensuring that the paraphrased queries are fluent and clearly describe what a user could be looking for. These are then validated for naturalness and fluency by a different set of crowdworkers, and filtered according to those criteria. Finally, for a large subset of the data, we collect scalar relevance labels based on the entity documents and fine-grained textual attributions mapping query constraints to spans of document text. Such annotation could aid the development of systems that can make precise inferences from trusted sources.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Performing well on this dataset requires systems that can match query constraints with corresponding evidence in documents and handle set operations implicitly specified by the query (see erations are not well represented in existing re-61 trieval benchmarks such as MSMarco (Nguyen 62 et al., 2016) 
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Retrieval System Figure 2: An example of a query and relevant entity from QUEST. The attribution for different query constraints can come from different parts of the document.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Figure 2), while also efficiently scaling to large collections of entities. We evaluate several retrieval systems by finetuning pretrained models on our dataset. Systems are trained to retrieve multidocument sets given a query. We find that current dual encoder and cross-attention models up to the size of T5-Large (Raffel et al., 2020) are largely not effective at performing retrieval for queries with set operations. Queries with conjunctions and negations prove to be especially challenging for models and systems are further challenged with combinations of set operations. Our error analysis reveals that non-relevant false positive entities are often caused by the model ignoring negated constraints, or ignoring the conjunctive constraints in a query.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Many opendomain QA benchmarks, which consider QA over unstructured text corpora, have been proposed in prior work. Some of these, such as TREC (Craswell et al., 2020), MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019) are constructed using "found data", using real user queries on search engines. Thakur et al. (2021) present a benchmark where they consider many such existing datasets. Datasets such as Hot-potQA (Yang et al., 2018), and MultiRC (Khashabi et al., 2018) have focused on multi-hop question answering. Other work has explored e-commerce datasets (for example, (Kong et al., 2022)), but these have not been released publicly. Notably, the focus of these datasets differs from ours as we focus on queries that contain implicit set operations over exhaustive answer sets. Such queries are not well represented in existing datasets because they occur in the tail of the query distributions considered.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Multi-Answer Retrieval Related work (Min et al., 2021;Amouyal et al., 2022) also studies the problem of multi-answer retrieval, where systems are required to predict multiple distinct answers for a query. Min et al. (2021) adapt existing datasets (for example, WebQuestionsSP (Yih et al., 2016)) to study this setting and propose a new metric, MRecall@K, to evaluate exhaustive recall of multiple answers. We also consider the problem of multi-answer set retrieval, but consider queries that implicitly contain set constraints.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
In concurrent work, RomQA (Zhong et al., 2022) proposes an open-domain QA dataset, focusing on combinations of constraints extracted from Wikidata. RomQA shares our motivation to enable answering queries with multiple constraints, which have possibly large answer sets. To make attribution to evidence feasible without human annotation, RomQA focuses on questions whose component constraints can be verified from single entity-linked sentences from Wikipedia abstracts, annotated with relations automatically through distant supervision, with high precision but possibly low recall (T-Rex corpus). In QUEST, we broaden the scope of queryevidence matching operations by allowing for attribution through more global, document-level inference. To make human annotation for attribution feasible, we limit the answer set size and the evidence for an answer to a single document.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
QUEST consists of 3357 queries paired with up to 20 corresponding entities. Each entity has an associated document derived from its Wikipedia page. The dataset is divided into 1307 queries for training, 323 for validation, and 1727 for testing.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
The task for a system is to return the correct set of entities for a given query. Additionally, as the collection contains 325,505 entities, the task requires retrieval systems that can scale efficiently. We do not allow systems to access additional information outside of the text descriptions of entities at inference time. Category labels are omitted from all entity documents.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
The base atomic queries (i.e., queries without any introduced set operations) in our dataset are derived from Wikipedia category names 2 . These are handcurated natural language labels assigned to groups of related documents in Wikipedia 3 . Category assignments to documents allow us to automatically determine the set of answer entities for queries with high precision and relatively high recall. We compute transitive closures of all relevant categories to determine their answer sets.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
However, repurposing these categories for constructing queries poses challenges: 1) lack of evi-  dence in documents: documents may not contain sufficient evidence for judging their relevance to a category, potentially providing noisy signal for relevance attributable to the document text, 2) low recall: entities may be missing from categories to which they belong. For about half of the dataset, we crowdsource relevance labels and attribution based on document text, and investigate recall through manual error analysis ( §5). We select four domains to represent some diversity in queries: films, books, animals and plants.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Focusing on four rather than all possible domains enables higher quality control. The former two model a general search scenario, while the latter two model a scientific search scenario.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
To construct queries with set operations, we define templates that represent plausible combinations of atomic queries. Denoting atomic queries as A, B and C, our templates and corresponding examples from different domains are listed in Table 1. Templates were constructed by composing three basic set operations (intersection, union and difference). They were chosen to ensure unambiguous interpretations of resulting queries by omitting those combinations of set operations that are non-associative.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Below we describe the logic behind sampling atomic queries (i.e., A, B, C) for composing com-plex queries, with different set operations. In all cases, we ensure that answer sets contain between 2-20 entities so that crowdsourcing relevance judgements is feasible. We sample 200 queries per template and domain, for a total of 4200 initial queries. The dataset is split into train + validation (80-20 split) and testing equally. In each of these sets, we sampled an equal number of queries per template. Intersection. The intersection operation for a template A∩B is particularly interesting and potentially challenging when both A and B have large answer sets but their intersection is small. We require the minimum answer set sizes of each A and B to be fairly large (>50 entities), while their intersection to be small (2-20 entities). Difference. Similar to intersection, we require the answer sets for both A and B to be substantial (>50 entities), but also place maximum size constraints on both A (<200 entities) and B (<10000 entities) as very large categories tend to suffer from recall issues in Wikipedia. We also limit the intersection of A and B (see reasoning in Appendix B). Union. For the union operation, we require both A and B to be well-represented through the entities in the answer set for their union A ∪ B. Hence, we require both A and B to have at least 3 entities. Further, we require their intersection to be non-zero but less than 1/3rd of their union. This is so that A and B are somewhat related queries.  For all other templates that contain compositions of the above set operations, we apply the same constraints recursively. For example, for A∩B \C, we sample atomic queries A and B for the intersection operation, then sample C based on the relationship between A ∩ B and C.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Automatically generating queries based on templates results in queries that are not always fluent and coherent. Further, entities mapped to a query may not actually be relevant and don't always have attributable evidence for judging their relevance. We conduct crowdsourcing to tackle these issues. The annotation tasks aim at ensuring that 1) queries are fluent, unambiguous and contain diverse natural language logical connectives, (2) entities are verified as being relevant or non-relevant and (3) relevance judgements are attributed to document text for each relevant entity. Crowdsourcing is performed in three stages, described below. More annotation details and the annotation interfaces can be found in Appendix C.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Crowdworkers were asked to paraphrase a templatically generated query so that the paraphrased query is fluent, expresses all constraints in the original query, and clearly describes what a user could be looking for. This annotation was done by one worker per query.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
This stage is aimed at validating the queries we obtain from the paraphrasing stage. Crowdworkers were given queries from the first stage and asked to label whether the query is 1) fluent, 2) equivalent to the original templatic query in meaning, and 3) rate its naturalness (how likely it is to be issued by a real user). This annotation was done by 3 workers per query. We excluded those queries which were rated as not fluent, unnatural or having a different meaning than the original query, based on a ma-jority vote. Based on the validation, we removed around around 11% of the queries from stage 1.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Next, crowdworkers were asked to provide relevance judgements for the automatically determined answer sets of queries. Specifically, they were given a query and associated entities/documents, and asked to label their relevance on a scale of 0-3 (definitely not relevant, likely not relevant, likely relevant, definitely relevant). They were asked to ensure that relevance should mostly be inferred from the document, but they could use some background knowledge and do minimal research. We also asked them to provide attributions for document relevance. Specifically, we ask them to first label whether the document provides sufficient evidence for the relevance of the entity (complete/partial/no). Then, for different phrases in the query (determined by the annotator), we ask them to mark sentence(s) in the document that indicate its relevance. The attribution annotation is broadly inspired by Rashkin et al. (2021). For negated constraints, we ask annotators to mark attributable sentences if they provide counter-evidence. Since this annotation was time-intensive, we collected these annotations for two domains (films and books). We found that relevance labeling was especially difficult for the plants and animals domains, as they required more specialized scientific knowledge. In our pilot study prior to larger scale data collection, we collected 3 relevance ratings from different annotators for 905 query and document pairs from the films domain. In 61.4% of cases, all 3 raters judged the document to be "Definitely relevant" or "Likely relevant" or all 3 raters judged the document to be "Definitely not relevant" or "Likely not relevant". The Fleiss' kappa metric on this data was found to be K=0.43. We excluded all entities which were marked as likely or definitely not relevant to a query based on the document text from its answer set. Around 23.7% of query-document pairs from stage 2 were excluded.  Figure 3: We compare several systems consisting of a retriever for efficiently selecting a set of candidates from the document corpus and a document relevance classifier for determining the final predicted document set.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Basic dataset statistics are reported in Table 2. The dataset contains more entities from the films domain, because this domain is more populated in Wikipedia. The average length of queries is 8.6 words and the average document length is 452 words. Documents from the films and books domains are longer on average, as they often contain plots and storylines. Around ∼69% of entities have complete evidence and ∼30% have partial evidence. Evidence was labeled as partial when not all phrases in the query had explicit evidence in the document (i.e., they may require background knowledge or reasoning). There are on average 33.2 words attributed for each entity with the maximum attribution text span ranging up to length 1837 words. Finally, the average answer set size is 10.5 entities.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Beyond the annotated data, we generated additional synthetic examples for training. We found including such examples improved model performance, and we include these examples for the experiments in §4. To generate these examples, we sample 5000 atomic queries from all domains, ensuring that they do not already appear as sub-queries in any of the queries in QUEST and use their corresponding entities in Wikipedia as their relevant entity set.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We evaluate modern retrieval systems to establish baseline performances. We also perform extensive error analysis to understand patterns of model errors and the quality of the labels in QUEST.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We consider a corpus, E, that contains entities across all domains in the dataset. Each entity is accompanied with a document based on its Wikipedia page. An example in our dataset consists of a query,
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
x, and an annotated set of relevant entities, y ⊂ E. As described in §3, for all examples |y| < 20. Our task is to develop a system that, given E and a query x, predicts a set of relevant entities,ŷ ⊂ E.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Our primary evaluation metric is average F 1 , which averages per-example F 1 scores. We compute F 1 for each example by comparing the predicted set of entities,ŷ, with the annotated set, y.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We evaluated several combinations of retrievers and classifiers, as shown in Figure 3. For the retriever component, we consider a sparse BM25 retriever (Robertson et al., 2009) and a dense dual encoder retriever (denoted DE). Following , we initialize our dual encoder from a T5 (Raffel et al., 2020) encoder and train with an in-batch sampled softmax loss (Henderson et al., 2017). Once we have a candidate set, we need to determine a set of relevant entities. To classify relevance of each candidate document for the given query, we consider a cross-attention model which consists of a T5 encoder and decoder. 4 We train the cross-attention classifier using a binary cross-entropy loss with negative examples based on non-relevant documents in top 1,000 documents retrieved by BM25 and random non-relevant documents (similarly to Nogueira and Cho (2019)). As cross-attention classification for a large number of candidates is computationally expensive, we restrict BM25 and the dual encoder to retrieve 100 candidates which are then considered by the crossattention classifier. As our T5-based dual encoder can only efficiently accommodate up to 512 tokens,   we truncate document text. We discuss the impact of this and alternatives in §5. Further, since T5 was pre-trained on Wikipedia, we investigate the impact of memorization in Appendix D. Additional details and hyperparameter settings are in Appendix A.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
For the best overall system, we sampled errors and manually annotated 1145 query-document pairs from the validation set. For the retriever, we sampled relevant documents not included in the top-100 candidate set and non-relevant documents ranked higher than relevant ones. For the classifier, we sampled false positive and false negative errors made in the top-100 candidate set. This annotation process included judgements of document relevance (to assess agreement with the annotations in the dataset) and whether the document (and the truncated version considered by the dual encoder or classifier) contained sufficient evidence to reasonably determine relevance. We also annotated relevance for each constraint within a query. We discuss these results in §5.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We report the performance of our baseline systems on the test set in Table 3. In this section, we summarize the key findings from our analysis of these results and the error annotation described in §4.4. Dual encoders outperform BM25. As shown in Table 3, the best overall system uses a T5-Large Dual Encoder instead of BM25 for retrieval. The performance difference is even more significant when comparing recall of Dual Encoders and BM25 directly. We report average recall (average per-example recall of the full set of relevant documents) and MRecall (Min et al., 2021) (the percentage of examples where the candidate set contains all relevant documents), over various candidate set sizes in Table 4. Retrieval and classification are both challenging. As we consider only the top-100 candidates from the retriever, the retriever's recall@100 sets an upper bound on the recall of the overall system. Recall@100 is only 0.476 for the T5-Large Dual Encoder, and the overall recall is further reduced by the T5-Large classifier to 0.368, despite achieving only 0.165 precision. This suggests that there is room for improvement from both stages to improve overall scores. As performance improves for larger T5 sizes for both retrieval and classification, further model scaling could be beneficial. Models struggle with intersection and difference. We also analyzed results across different templates and domains, as shown in Table 5. Different constraints lead to varying distributions over answer set sizes and the atomic categories used. Therefore, it can be difficult to interpret differences in F1 scores across templates. Nevertheless, we found the queries with set union have the highest average F1 scores. Queries with set intersection have the lowest average F1 scores, and queries with set difference also appear to be challenging.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
To analyze why queries with conjunction and negation are challenging, we labeled the relevance of individual query constraints ( §4.4), where a system incorrectly judges relevance of a non-relevant document. The results are summarized in Table 6. For a majority of false positive errors involving intersection, at least one constraint is satisfied. This could be interpreted as models incorrectly treating intersection as union when determining relevance. Similarly, for a majority of examples with set difference, the negated constraint is not satisfied. This suggests that the systems are not sufficiently sensitive to negations.  There is significant headroom to improve both precision and recall. As part of our manual error analysis ( §4.4), we made our own judgements of relevance and measured agreement with the relevance annotations in QUEST. As this analysis focused on cases where our best system disagreed with the relevance labels in the dataset, we would expect agreement on these cases to be significantly lower than on randomly selected query-document pairs in the dataset. Therefore, it provides a focused way to judge the headroom and annotation quality of the dataset. For false negative errors, we judged 91.1% of the entities to be relevant for the films and books domains, and 81.4% for plants and animals. Notably, we collected relevance labels for the films and books domains and removed some entities based on these labels, as described in §3, which likely explains the higher agreement for false negatives from these domains. This indicates significant headroom for improving recall as defined by QUEST, especially for the domains where we collected relevance labels.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
For false positive errors, we judged 28.8% of the entities to be relevant, showing a larger disagreement with the relevance labels in the dataset. This is primarily due to entities not included in the entity sets derived from the Wikipedia category taxonomy (97.7%), rather than entities removed due to relevance labeling. This is a difficult issue to fully resolve, as it is not feasible to exhaustively label relevance for all entities to correct for recall issues in the Wikipedia category taxonomy. Future work can use pooling to continually grow the set   bergen, 1975). Despite this, our analysis suggests there is significant headroom for improving precision, as we judged a large majority of the false positive predictions to be non-relevant.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Truncating document text usually provides sufficient context. In our experiments, we truncate document text to 512 tokens for the dual encoder, and 384 tokens for the classifier to allow for the document and query to be concatenated. Based on our error analysis ( §4.4), out of the documents with sufficient evidence to judge relevance, evidence occurred in this truncated context 93.2% of the time for the dual encoder, and 96.1% of the time for the classifier. This may explain the relative success of this simple baseline for handling long documents. We also evaluated alternative strategies but these performed worse in preliminary experiments 5 . Future work can evaluate efficient transformer variants (Guo et al., 2022;Beltagy et al., 2020).
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We present QUEST, a new benchmark of queries which contain implicit set operations with corresponding sets of relevant entity documents. Our experiments indicate that such queries present a challenge for modern retrieval systems. Future work could consider approaches that have better inductive biases for handling set operations in natural language expressions (for example, Vilnis et al. (2018)). The attributions in QUEST can be leveraged for building systems that can provide finegrained attributions at inference time. The potential of pretrained generative LMs and multi-evidence aggregation methods to answer set-seeking selective queries, while providing attribution to sources, can also be investigated.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Naturalness. Since our dataset relies on the Wikipedia category names and semi-automatically generated compositions, it does not represent an unbiased sample from a natural distribution of real search queries that contain implicit set operations. Further, we limit attention to non-ambiguous queries and do not address the additional challenges that arise due to ambiguity in real search scenarios. However, the queries in our dataset were judged to plausibly correspond to real user search needs and system improvements measured on QUEST should correlate with improvements on at least a fraction of natural search engine queries with set operations.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Recall. We also note that because Wikipedia categories have imperfect recall of all relevant entities (that contain sufficient evidence in their documents), systems may be incorrectly penalised for predicted relevant entities assessed as false positive. We quantify this in section 5. We have also limited the trusted source for an entity to its Wikipedia document but entities with insufficient textual evidence in their documents may still be relevant. Ideally, multiple trusted sources could be taken into account and evidence could be aggregated to make relevance decisions. RomQA (Zhong et al., 2022) takes a step in this latter direction although the evidence attribution is not manually verified.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Answer Set Sizes. To ensure that relevance labels are correct and verifiable, we seek the help of crowdworkers. However, this meant that we needed to restrict the answer set sizes to 20 for the queries in our dataset, to make annotation feasible.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
On one hand, this is realistic for a search scenario because users may only be interested in a limited set of results. On the other hand, our dataset does not model a scenario where the answer set sizes are much larger.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
All models were fine-tuned starting from T5 1.1 checkpoints 6 . We fine-tune T5 models on 32 Cloud TPU v3 cores 7 . Fine-tuning takes less than 8 hours for all models.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Dual Encoder. We used the t5x_retrieval library 8 for implementing dual encoder models. We tuned some parameters based on results on the validation set. Relevant hyperparameters for training the dual encoder are:
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
• For the T5 input we concatenated the query and truncated document text. The T5 output is the string "relevant" or "not relevant". To classify document relevance at inference time, we applied a threshold to the probability assigned to the "relevant" label, which we tuned on the validation set. When classifying BM25 candidates we used a threshold of 0.9 and when classifying the dual encoder candidates we used a threshold of 0.95.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Other relevant hyperparameters for training the classifier are: We quantify the degree of missing entities by denoting recall as r A and r B , such that |Â| = r A * |A| and |B| = r B * |B|. We quantify the fraction of elements in A that are also in B as r ∩ , such that |A ∩ B| = r ∩ * |A|. For simplicity, we also assume that the overlap between A andB is such that |Â ∩ B| = r A * |A ∩ B| and |Â ∩B| = r A * r B * |A ∩ B|.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Derivation What is the recall (r) and precision (p) ofÂ \B relative to A \ B as a function of r A , r B , and r ∩ ? First, we derive this function for recall: 9
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
And for precision:
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We note some useful properties of pairs of sets X and Y :
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Discussion While recall is simply equal to r A , precision is a more complicated function of r B and r ∩ , and can be very low for large values of r ∩ . Intuitively, if subtractingB fromÂ removes most ofÂ, then the precision of the resulting set will be dominated by the relevant entities missing fromB. This motivates limiting the intersection of the two sets used to construct queries involving set intersection. For example, if r B = 0.95, then with r ∩ < 0.8, we can ensure p > 0.83.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
The annotation tasks in QUEST were carried out by participants who were paid contractors. They are based in Austin, TX and either have a bachelor's degree (55%) or equivalent work experience (45%). They were paid by the hour for their work and were recruited from a vendor who screened them for knowledge of US English. They were informed of how their work would be used and could opt out. They received a standard contracted wage, which complies with living wage laws in their country of employment. The annotation interfaces presented to the annotators are shown in Figures 4, 5 and 6.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Since the T5 checkpoints we use to initialize our models were pre-trained on the C4 corpus (which includes Wikipedia), we investigate whether these models have memorized aspects of the Wikipedia category graph. We compare recall of the T5-based dual encoder model for Wikipedia documents that were created prior to the pre-training date of the T5 checkpoint compared with documents that were added after pre-training. We report these in Table 7, along with the recalls for the same sets of documents with a BM25 retriever, for a baseline  comparison. We note that the ratio of scores between the documents added before pre-training to documents added after pre-training is similar for both systems, which suggests factors other than memorization may explain the difference. For example, the documents created before vs. after the pre-training date have average lengths of 759.7 vs. 441.2 words, respectively.   A2. Did you discuss any potential risks of your work? Not applicable. We did not identify any risks.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
A3. Do the abstract and introduction summarize the paper's main claims?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Abstract and section 1 A4. Have you used AI writing assistants when working on this paper?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Left blank.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
B Did you use or create scientific artifacts?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We will release the code and our dataset publicly.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Appendix A B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Yes, we will use an MIT license to release our dataset.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A D Did you use human annotators (e.g., crowdworkers) or research with human participants?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Appendix C D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix C
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? Appendix C D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix C D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Left blank.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix C
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Passage retrieval is the problem of retrieving a set of passages relevant to a natural language question from a large text corpus. Most prior work focuses on single-answer retrieval, which scores passages independently from each other according to their relevance to the given question, assuming there is a single answer (Voorhees et al., 1999;Chen et al., 2017;Lee et al., 2019). However, questions posed by humans are often open-ended and ambiguous, leading to multiple valid answers . For example, for the question in Figure 1, "What was Eli Whitney's job?", an ideal retrieval system should provide passages covering all professions of Eli Whitney. This introduces the problem of multianswer retrieval-retrieval of multiple passages * Work done while interning at Google.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Question Eli Whitney was an American inventor, widely known for inventing the cotton gin … Whitney worked as a farm laborer and school teacher to save money.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
What was Eli Whitney's job?
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Prediction Target Figure 1: The problem of multi-answer retrieval. A retrieval system must retrieve a set of k passages (k = 5 in the figure) which has maximal coverage of diverse answers to the input question: inventor, farm laborer and school teacher in this example. This requires modeling the joint probability of the passages in the output set: P (p 1 ...p k |q). Our proposed model JPR achieves this by employing an autoregressive model. with maximal coverage of all distinct answerswhich is a challenging yet understudied problem.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Multi-answer retrieval poses two challenges that are not well represented in single-answer retrieval. First, the task requires scoring passages jointly to optimize for retrieving multiple relevant-yetcomplementary passages. Second, the model needs to balance between two different goals: retrieving passages dissimilar to each other to increase the recall, and keeping passages relevant to the question.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
In this work, we introduce Joint Passage Retrieval (JPR), a new model that addresses these challenges. To jointly score passages, JPR employs an encoder-decoder reranker and autoregressively generates passage references by modeling the probability of each passage as a function of previously retrieved passages. Since there is no ground truth ordering of passages, we employ a new training method that dynamically forms supervision to drive the model to prefer passages with answers not already covered by previously selected passages. Furthermore, we introduce a new tree-decoding algorithm to allow flexibility in the degree of diversity.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
In a set of experiments on three multi-answer datasets-WEBQSP (Yih et al., 2016), AM-BIGQA (Min et al., 2019) and TREC (Baudiš and Šedivỳ, 2015), JPR achieves significantly improved recall over both a dense retrieval baseline (Guu et al., 2020;Karpukhin et al., 2020) and a state-of-the-art reranker that independently scores each passage (Nogueira et al., 2020). Improvements are particularly significant on questions with more than one answer, outperforming dense retrieval by up to 12% absolute and an independent reranker by up to 6% absolute.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We also evaluate the impact of JPR in downstream question answering, where an answer generation model takes the retrieved passages as input and generates short answers. Improved reranking leads to improved answer accuracy because we can supply fewer, higher-quality passages to a larger answer generation model that fits on the same hardware. This practice leads to a new stateof-the-art on three multi-answer QA datasets and NQ (Kwiatkowski et al., 2019). To summarize, our contributions are as follows:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
1. We study multi-answer retrieval, an underexplored problem that requires the top k passages to maximally cover the set of distinct answers to a natural language question.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
2. We propose JPR, a joint passage retrieval model that integrates dependencies among selected passages, along with new training and decoding algorithms.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
3. On three multi-answer QA datasets, JPR significantly outperforms a range of baselines with independent scoring of passages, both in retrieval recall and answer accuracy.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
In a typical single-answer retrieval problem, a model is given a natural language question q and retrieves k passages {p 1 ...p k } from a large text corpus C (Voorhees et al., 1999;Ramos et al., 2003;Robertson and Zaragoza, 2009;Chen et al., 2017;Lee et al., 2019;Karpukhin et al., 2020;Luan et al., 2020). The goal is to retrieve at least one passage that contains the answer to q. During training, question-answer pairs (q, a) are given to the model.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Evaluation Intrinsic evaluation directly evaluates the retrieved passages. The most commonly used metric is RECALL @ k which considers re-  Table 1: A comparison of single-answer and multianswer retrieval tasks. Previous work has used independent ranking models P (p i |q) for multi-answer retrieval because the inference-time inputs and outputs are the same. We propose JPR as an instance of P (p 1 ...p k |q).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
trieval successful if the answer a is included in {p 1 ...p k }. Extrinsic evaluation uses the retrieved passages as input to an answer generation model such as the model in Izacard and Grave (2021) and evaluates final question answering performance.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Reranking Much prior work (Liu, 2011;Asadi and Lin, 2013;Nogueira et al., 2020) found an effective strategy in using a two-step approach of (1) retrieving a set of candidate passages B from the corpus C (k < |B| |C|) and (2) using another model to rerank the passages, obtaining a final top k. A reranker could be more expressive than the first-stage model (e.g. by using cross-attention), as it needs to process much fewer candidates. Most prior work in reranking, including the current stateof-the-art (Nogueira et al., 2020), scores each passage independently, modeling P (p|q).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We now formally define the task of multi-answer retrieval. A model is given a natural language question q and needs to find k passages {p 1 ...p k } from C that contain all distinct answers to q. Unlike in single-answer retrieval, question-and-answer-set pairs (q, {a 1 ...a n }) are given during training.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Evaluation Similar to single-answer retrieval, the intrinsic evaluation directly evaluates a set of k passages. As the problem is underexplored, metrics for it are less studied. We propose to use MRE-CALL @ k, a new metric which considers retrieval to be successful if all answers or at least k answers in the answer set {a 1 ...a n } are recovered by {p 1 ...p k }. Intuitively, MRECALL is an extension of RECALL that considers the completeness of the retrieval; the model must retrieve all n answers Training with dynamic oracle (Section 3.2)
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Inference with TREEDECODE (Section 3.3)
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Prefix Positive passages not in the prefix Autoregressive generation of passage references (Section 3.1) Figure 2: An overview of JPR. We focus on reranking and propose: autoregressive generation of passage references (Section 3.1), training with dynamic oracle (Section 3.2), and inferece with TREEDECODE (Section 3.3).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
when n ≤ k, or at least k answers when n > k. 1 The extrinsic evaluation inputs the retrieved passages into an answer generation module that is designed for multiple answers, and measures multianswer accuracy using an appropriate metric such as the one in .
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Comparing to single-answer retrieval We compare single-answer retrieval and multi-answer retrieval in Table 1. Prior work makes no distinctions between these two problems, since they share the same interface during inference. However, while independently scoring each passage (P (p i |q)) may be sufficient for single-answer retrieval, multi-answer retrieval inherently requires joint passage scoring P (p 1 ...p k |q). For example, models should not repeatedly retrieve the same answer at the cost of missing other valid answers, which can only be done by a joint model.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Choice of k for downstream QA Previous stateof-the-art models typically input a large number (k ≥ 100) of passages to the answer generation model. For instance, Izacard and Grave (2021) claim the importance of using a larger value of k to improve QA accuracy. In this paper, we argue that with reranking, using a smaller value of k (5 or 10) and instead employing a larger answer generation model is advantageous given a fixed hardware budget. 2 We show in Section 5 that, as retrieval performance improves, memory is better spent on larger answer generators rather than on more passages, ultimately leading to higher QA accuracy. answer retrieval. JPR uses an approach consisting of first-stage retrieval followed by reranking: the first-stage retrieval obtains candidate passages B from the corpus C, and a reranker processes B to output {p 1 ...p k } ⊂ B. We refer to Section 4.2 for the first-stage retrieval, and focus on the reranking component of the model, which allows (1) efficiently modeling the joint probability P (p 1 ...p k |q), and (2) processing candidate passages with a more expressive model.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
The overview of JPR is illustrated in Figure 2. The reranker of JPR leverages the encoder-decoder architecture for an autoregressive generation of passage references (Section 3.1). Unlike typical use cases of the encoder-decoder, (1) the ordering of passages to retrieve is not given as supervision, and (2) it is important to balance between exploring passages about new answers and finding passages that may cover previously selected answers. To this end, we introduce a new training method (Section 3.2) and a tree-based decoding algorithm (Section 3.3).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
JPR makes use of the encoder-decoder architecture, where the encoder processes candidate passages and the decoder autoregressively generates a sequence of k passage references (indexes). Intuitively, dependencies between passages can be modeled by the autoregressive architecture.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We extend the architecture from Izacard and Grave (2021); we reuse the encoder but modify the decoder. Each candidate passage p i is concatenated with the question q and the number i (namely index). It is fed into the encoder to be transformed to p i ∈ R L×h , where L is the length of the input text and h is a hidden size. Next, p 1 ...p |B| are concatenated to formp ∈ R L|B|×h , and then fed into the decoder. The decoder is trained to autoregressively output a sequence of indexes i 1 ...i k , 
Joint Passage Ranking for Diverse Multi-Answer Retrieval
O={a, b} O={a, b, e} O={a, b, e, d} S={empty, a} S={empty, a, a::b} S={empty, a, a::b, a::e} S={empty, a, a::b, a::e, a::b::d}
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Step 1
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Step 2
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Step 3
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Step 4 d t=3 i f z Figure 3: An illustration of TREEDECODE, where passages that are chosen and passages that are being considered are indicated in orange and blue, respectively. See Section 3.3 and Algorithm 1 for details.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
representing a sequence of passages p 1 ...p k . As the generation at step t (1 ≤ t ≤ k) is dependent on the generation at step 1 . . . t − 1, it can naturally capture dependencies between selected passages. As each index occupies one token, the length of the decoded sequence is k.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
A standard way of training the encoder-decoder is teacher forcing which assumes a single correct sequence. However, in our task, a set of answers can be retrieved through many possible sequences of passages, and it is unknown which sequence is the best. To this end, we dynamically form the supervision data which pushes the model to assign high probability to a dynamic oracle-any positive passage covering a correct answer that is not in the prefix, i.e., previously selected passages. We first precompute a set of positive passages O and a prefixp 1 ...p k . A set of positive passages O includes up to k passages with maximal coverage of the distinct answers. 3 A prefixp 1 ...p k is a simulated prediction of the model, consisting of O and k − |Õ| sampled negatives. At each step t (1 ≤ t ≤ k) , given a set of positive passagesÕ and a prefixp 1 ...p t (denoted as P t ), JPR is trained to assign high probabilities to the dynamic oraclẽ O − P t . The objective is defined as follows:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
A typical autoregressive decoder makes the top 1 prediction at each step to decode a sequence of k (SEQDECODE in Algorithm 1), 4 which, based 3 |Õ| < k if fewer than k passages are sufficient to cover all distinct answers; |Õ| = k otherwise. 4 We explored beam search decoding but it gives results that are the same as or marginally different from SEQDECODE.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Algorithm 1 Decoding algorithms for JPR.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
return Set(O) 7: procedure TREEDECODE(k, P (p|p1...pn), l) 8:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
O ← ∅ // a set of selected passages 9:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
S ← [Empty] // a tree 10:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
while |O| < k do 11: P (p|s) ← P (p|s)I[s :: p / ∈ S] 12:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
(ŝ,p) ← argmax p∈B,s∈S l(|s| + 1)logP (p|s) 13:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
O ← O ∪ {p}, S ← S.append(ŝ ::p) 14:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
return O on our training scheme, asks the decoder to find a new answer at every step. However, when k is larger than the number of correct answers, it would be counter-productive to ask for k passages that each covers a distinct answer. Instead, we want the flexibility of decoding fewer timesteps and take multiple predictions from each timestep. In this context, we introduce a new decoding algorithm TREEDECODE, which decodes a tree from an autoregressive model. TREEDECODE iteratively chooses between the depth-wise and the width-wise expansion of the tree-going forward to the next step and taking the next best passage within the same step, respectively-until it reaches k passages (Figure 3). Intuitively, if the model believes that there are many distinct answers covered by different passages, it will choose to take the next step, being closer to SEQDECODE. On the other hand, if the model believes that there are very few distinct answers, it will choose to take more predictions from the same step, resulting in behavior closer to independent scoring.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
The formal algorithm is as follows. We represent a tree S as a list of ordered lists [s 1 ...s T ] where s 1 is an empty list and s i is one element appended to any of s 1 ...s i−1 . The corresponding set O is  ∪ s∈S Set(s). We define a score of a tree S as
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We form S and O through an iterative process by (1) starting from O = ∅ and S = [null], and (2) updating O and S by finding the best addition of an element that maximizes the gain in f (S), until |O| = k, as described in Algorithm 1.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We compare JPR with multiple baselines in a range of multi-answer QA datasets. We first present an intrinsic evaluation of passage retrieval by reporting MRECALL based on answer coverage in the retrieved passages (Section 5.1). We then present an extrinsic evaluation through experiments in downstream question answering (Section 5.2).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We train and evaluate on three datasets that provide a set of distinct answers for each question. Statistics of each dataset are provided in Table 2. WEBQSP (Yih et al., 2016) consists of questions from Google Suggest API, originally from Berant et al. (2013). The answer is a set of distinct entities in Freebase; we recast this problem as textual question answering based on Wikipedia. AMBIGQA  consists of questions mined from Google search queries, originally from NQ (Kwiatkowski et al., 2019). Each question is paired with an annotated set of distinct answers that are equally valid based on Wikipedia. TREC (Baudiš and Šedivỳ, 2015) contains questions curated from TREC QA tracks, along with regular expressions as answers. Prior work uses this data as a task of finding a single answer (where retrieving any of the correct answers is sufficient), but we recast the problem as a task of finding all answers, and approximate a set of distinct answers. Details are described in Appendix B.1.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
JPR can obtain candidate passages B from any firststage retrieval model. In this paper, we use DPR + , our own improved version of DPR (Karpukhin et al., 2020) combined with REALM (Guu et al., 2020). DPR and REALM are dual encoders with a supervised objective and an unsupervised, language modeling objective, respectively. We initialize the dual encoder with REALM and train on supervised datasets using the objective from DPR. More details are provided in Appendix A.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We compare JPR with three baselines, all of which are published models or enhanced versions of them. All baselines independently score each passage. DPR + only uses DPR + without a reranker.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
DPR + +Nogueira et al. (2020) uses DPR + followed by Nogueira et al. (2020), the state-of-theart document ranker. It processes each passage p i in B independently and is trained to output yes if p i contains any valid answer to q, otherwise no. At inference, the probability for each p i is computed by taking a softmax over the logit of yes and no. The top k passages are chosen based on the probabilities assigned to yes. INDEPPR is our own baseline that is a strict nonautoregressive version of JPR in which prediction of a passage is independent from other passages in the retrieved set. It obtains candidate passages B through DPR + and the encoder of the reranker processes q and B, as JPR does. Different from JPR, the decoder is trained to output a single token i (1 ≤ i ≤ |B|) rather than a sequence. The objective is the sum of −logP (p|q, B) of the passages including any valid answer to q. At inference, INDEPPR outputs the top k passages based the logit values of the passage indices. We compare mainly to IN-DEPPR because it is the strict non-autoregressive version of JPR, and is empirically better than or comparable to Nogueira et al. (2020) (Section 5.1).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use the English Wikipedia from 12/20/2018 as the retrieval corpus C, where each article is split into passages with up to 288 wordpieces, All rerankers are based on T5 (Raffel et al., 2020), a pretrained encoder-decoder model; T5-base is used unless otherwise specified. We use |B| = 100, k = {5, 10}. Models are first trained on NQ (Kwiatkowski et al., 2019) and then finetuned  Table 3: Results on passage retrieval in MRECALL. The two numbers in each cell indicate performance on all questions and on questions with more than one answer, respectively. Test-set metrics on AMBIGQA are not available as its test set is hidden, but we report the test results on question answering in Section 5.2.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Note: it is possible to have higher MRECALL @ 5 than MRECALL @ 10 based on our definition of MRECALL (Section 2.2).   on multi-answer datasets, which we find helpful since all multi-answer datasets are relatively small. During dynamic oracle training, k − |Õ| negatives are sampled from B −Õ based on s(p i ) + γg i , where s(p i ) is a prior logit value from INDEPPR, g i ∼ Gumbel(0, 1) and γ is a hyperparameter. In TREEDECODE, to control the trade-off between the depth and the width of the tree, we use a length penalty function l(y) = 5+y 5+1 β , where β is a hyperparameter, following Wu et al. (2016). More details are in Appendix B.2. Table 3 reports MRECALL on all questions and on questions with more than one answer. Independent vs. joint ranking JPR consistently outperforms both DPR + +Nogueira et al. (2020) and INDEPPR on all datasets and all values of k. Gains are especially significant on questions with more than one answer, outperforming two reranking baselines by up to 11% absolute and up to 6% absolute, respectively. WEBQSP sees the largest gains out of the three datasets, likely because the average number of answers is the largest.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Training methods Table 4 compares dynamic oracle training with alternatives. 'Dynamic oracle w/o negatives' is the same as dynamic oracle training except the prefix only has positive passages. 'Teacher forcing' is a standard method in training an autoregressive model: given a target sequence o 1 ...o k , the model is trained to maximize Π 1≤t≤k P (o t |o 1 ...o t−1 ). We form a target sequence using a set of positive passagesÕ, where the order is determined by following the ranking from INDEPPR. Table 4 shows that our dynamic oracle training, which uses both positives and negatives, significantly outperforms the other methods. Table 5 compares JPR with SEQDECODE and with TREEDECODE. We find that TREEDECODE consistently improves the performance on both WEBQSP and AM-BIGQA, with both k = 5 and 10. Gains are especially significant on AMBIGQA, since the choice of whether to increase diversity is more challenging on AMBIGQA where questions are more specific and have fewer distinct answers, which TREEDE-  The average depth of the tree is larger on WEBQSP, likely because its average number of distinct answers is larger and thus requires more diversity.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
An example prediction Table 6 shows predictions from INDEPPR and JPR given an example question from AMBIGQA, "Who plays Mark on the TV show Roseanne?" One answer Glenn Quinn is easy to retrieve because there are many passages in Wikipedia providing evidence, while the other answer Ames McNamara is harder to find. While INDEPPR repeatedly retrieves passages that mention Glenn Quinn and fails to cover Ames McNamara, JPR successfully retrieves both answers. More analysis can be found in Appendix C.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
This section discusses experiments on downstream question answering: given a question and a set of passages from retrieval, the model outputs all valid answers to the question. We aim to answer two research questions: (1) whether the improvements in passage retrieval are transferred to improvements in downstream question answering, and (2) whether using a smaller number of passages through reranking is better than using the largest possible number of passages given fixed hardware memory.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use an answer generation model based on Izacard and Grave (2021) which we train to generate a sequence of answers, separated by a [SEP] token, given a set of retrieved passages. Our main model uses JPR to obtain passages fed into the answer generation model. The baselines obtain passages from either DPR + only or INDEPPR, described in Section 4.3.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We compare different models that fit on the same hardware by varying the sizes of T5 (base, large, 3B) and use the maximum number of passages (k). 5 This results in three settings: {k = 140, base}, {k = 40, large} and {k = 10, 3B}. Table 7 reports the performance on three multianswer datasets in F1, following .
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Impact of reranking With {k = 10, 3B}, JPR outperforms both baselines, indicating that the improvements in retrieval are successfully transferred to improvements in QA performance. We however find that our sequence-to-sequence answer generation model tends to undergenerate answers, presumably due to high variance in the length of the output sequence. This indicates the model is not fully benefiting from retrieval of many answers, and we expect more improvements when combined with an answer generation model that is capable of generating many answers.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
More passages vs. bigger model With fixed memory during training, using fewer passages equipped with a larger answer generation model outperforms using more passages. This is only true when reranking is used; otherwise, using more passages is often better or comparable. This demonstrates that, as retrieval improves, memory is better spent on larger answer generators rather than more passages, leading to the best performance.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Finally, JPR establishes a new state-of-the-art, outperforming the previous state-of-the-art on AM-BIGQA (Gao et al., 2021) with extensive reranking and the answer generation model trained using x3 more resources than ours. 6
Joint Passage Ranking for Diverse Multi-Answer Retrieval
While our main contributions are in multi-answer retrieval, we experiment on NQ to demonstrate  Table 7: Question Answering results on multi-answer datasets. The two values in each cell indicate F1 on all questions and F1 on questions with multiple answers only, respectively. Mem compares the required hardware memory during training. Note that Gao et al. (2021) reranks 1000 passages instead of 100, and trains an answer generation model using x3 more memory than ours. Better retrieval enables using larger answer generation models on fewer retrieved passages.  Table 8: Question Answering results on NQ. We report Exact Match (EM) accuracy. The first five rows are from our own experiments, which all use the same hardware resources for training. The last row is the previous state-of-the-art which requires x5 more resources than ours to train the model. that the value of good reranking extends to the single-answer scenario. Table 8 indicates two observations consistent to the findings from multianswer retrieval: (1) when compared within the same setting (same T5 and k), INDEPPR always outperforms DPR + only, and (2) with reranking, {k = 10, 3B} outperforms {k = 40, large}. Finally, our best model outperforms the previous state-of-the-art (Izacard and Grave, 2021) which uses x5 more training resources. Altogether, this result (1) justifies our choice of focusing on reranking, and (2) shows that INDEPPR is very competitive and thus our JPR results in multi-answer retrieval are very strong.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We introduce JPR, an autoregressive passage reranker designed to address the multi-answer retrieval problem. On three multi-answer datasets, JPR significantly outperforms a range of baselines in both retrieval recall and downstream QA accuracy, establishing a new state-of-the-art. Future work could extend the scope of the problem to other tasks that exhibit specific information need while requiring diversity.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use a pretrained dual encoder model from REALM (Guu et al., 2020) and further finetune it on the QA datasets using the objective from DPR (Karpukhin et al., 2020):
Joint Passage Ranking for Diverse Multi-Answer Retrieval
where f q and f p are trainable encoders for the questions and passages, respectively, p + is a positive passage (i.e., a passage containing the answer), and B − is a set of negative passages (i.e., passages without the answer). As shown in Karpukhin et al. (2020), a choice of B − is significant for the performance. We explore two methods: Distant negatives follows DPR (Karpukhin et al., 2020) in using distantly obtained negative passages as B − . We obtain two distant negative passages per question: one hard negative, a top prediction from REALM without finetuning, and one random negative, drawn from a uniform distribution, both not containing the answer.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Full negatives considers all passages in Wikipedia expect p + as B − , and instead freezes the passage encoder f p and only finetunes the question encoder f q . This is appealing because (a) the number and the quality of the negatives, which both are the significant factors for training, are the strict maximum, and (b) f p from REALM is already good, producing high quality passage representations without finetuning. Implementation of this method is feasible by exploiting extensive model parallelism.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use distant negatives for multi-answer datasets and full negatives for NQ as this combination gave the best result. B Experiment Details B.1 Data processing for TREC TREC from Baudiš and Šedivỳ (2015) contains regular expressions as the answers. We approximate a set of semantically distinct answers as follows. We first run regular expressions over Wikipedia to detect valid answer text. If there is no valid answer found from Wikipedia, or there are more than 100 valid answers 8 , we discard the question. We then only keep the answers with up to five tokens, following the notion of short answers from Lee et al. (2019). Finally, we group the answers  that are the same after normalization and white space removal. We find that this gives a reasonable approximation of a set of semantically distinct answers. Note that the data we use is the subset of the original data because we discarded a few questions. Statistics are reported in Section 4.1.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Here is an example: a regular expression from the original data is Long Island|New\s?York|Roosevelt Field. All matching answers over Wikipedia include roosevelt field, new york, new\xa0york, new\nyork, newyork, long island.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Once the grouping is done, we have three semantically distinct answers: (1) roosevelt field, (2) new york|new\xa0york|new\nyork|newyork, and (3) long island.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
All implementations are based on Tensorflow (Abadi et al., 2015) and Mesh Tensorflow (Shazeer et al., 2018). All experiments are done in Google Cloud TPU. We use batch size that is the maximum that fits one instance of TPU v3-32 (for WEBQSP and AMBIGQA) or TPU v3-8 (TREC). We use the same batch size for INDEPPR; for Nogueira et al. (2020), we use the batch size of 1024. We use the encoder length of 360 and the decoder length of k (JPR) or 1 (all others). We use k = {5, 10} for all experiments. We train JPR with γ = {0, 0.5, 1.0, 1.5} and choose the one with the best accuracy on the development data. We use a flat learning rate of 1 × 10 −3 with warm-up for the first 500 steps. Full hyperparameters are reported in Table 9.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
For training INDEPPR and JPR, instead of using all of |B| passages, we use |B|/4 passages by sampling k positive passages and |B|/4 − k negative passages. We find that this trick allows larger batch size when using the same hardware, ultimately leading to substantial performance gains. We also find   Algorithm 2 An algorithm to obtainÕ from the answer set and B. Aleft ← Aleft− answers in b 8:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
if |Õ| == k then 9: break 10:
Joint Passage Ranking for Diverse Multi-Answer Retrieval
return O.toSet() that assigning indexes of the passages based on a prior, e.g., ranking from dense retrieval, leads to significant bias, e.g., in 50% of the cases, the top-1 passage from dense retrieval contains a correct answer. We therefore randomly assign the indexes, and find this gives significantly better performance. Algorithm 2 describes how a set of positive pas-sagesÕ used in Section 3.2 is computed during preprocessing.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We train the models using a batch size of 32. We use a decoder length of 20 and 40 for NQ and multi-answer datasets, respectively. We decode answers only when they appear in the retrieved passages, as we want the generated answers to be grounded by Wikipedia passages. Answers in the output sequence follow the order they appear in the passages, except on WEBQSP, where shuffling the order of the answers improves the accuracy. All other training details are the same as details of reranker training.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We additionally report retrieval performance in α-NDCG @ k, one of the metrics for diverse retrieval in IR (Clarke et al., 2008;Sakai and Zeng, 2019). It is a variant of NDCG (Järvelin and Kekäläinen, 2002), but penalizes retrieval of the same answer. We refer to Clarke et al. (2008) for a complete definition. We use α = 0.9.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Results are reported in Table 10. JPR consistently outperforms INDEPPR across all datasets, although the gains are less significant than the gains in MRECALL. We note that we report α-NDCG following IR literatures, but we think of MRECALL as a priority, because α-NDCG does not use an explicit notion of completeness of retrieval of all answers. It is also a less strict measure than recall because it gives partial credits to retrieving a subset of the answers.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Gains with respect to the number of answers Figure 4 shows gains over INDEPPR on three datasets with respect to the number of answers. Overall, gains are larger when the number of answers is larger, especially for WEBQSP and TREC. For AMBIGQA, the largest gains are when the number of answers is 2, which is responsible for over half of multi-answer questions.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Passage retrieval is the problem of retrieving a set of passages relevant to a natural language question from a large text corpus. Most prior work focuses on single-answer retrieval, which scores passages independently from each other according to their relevance to the given question, assuming there is a single answer (Voorhees et al., 1999;Chen et al., 2017;Lee et al., 2019). However, questions posed by humans are often open-ended and ambiguous, leading to multiple valid answers . For example, for the question in Figure 1, "What was Eli Whitney's job?", an ideal retrieval system should provide passages covering all professions of Eli Whitney. This introduces the problem of multianswer retrieval-retrieval of multiple passages * Work done while interning at Google.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
In a set of experiments on three multi-answer datasets-WEBQSP (Yih et al., 2016), AM-BIGQA (Min et al., 2019) and TREC (Baudiš and Šedivỳ, 2015), JPR achieves significantly improved recall over both a dense retrieval baseline (Guu et al., 2020;Karpukhin et al., 2020) and a state-of-the-art reranker that independently scores each passage (Nogueira et al., 2020). Improvements are particularly significant on questions with more than one answer, outperforming dense retrieval by up to 12% absolute and an independent reranker by up to 6% absolute.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
In a typical single-answer retrieval problem, a model is given a natural language question q and retrieves k passages {p 1 ...p k } from a large text corpus C (Voorhees et al., 1999;Ramos et al., 2003;Robertson and Zaragoza, 2009;Chen et al., 2017;Lee et al., 2019;Karpukhin et al., 2020;Luan et al., 2020). The goal is to retrieve at least one passage that contains the answer to q. During training, question-answer pairs (q, a) are given to the model.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
trieval successful if the answer a is included in {p 1 ...p k }. Extrinsic evaluation uses the retrieved passages as input to an answer generation model such as the model in Izacard and Grave (2021) and evaluates final question answering performance.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Reranking Much prior work (Liu, 2011;Asadi and Lin, 2013;Nogueira et al., 2020) found an effective strategy in using a two-step approach of (1) retrieving a set of candidate passages B from the corpus C (k < |B| |C|) and (2) using another model to rerank the passages, obtaining a final top k. A reranker could be more expressive than the first-stage model (e.g. by using cross-attention), as it needs to process much fewer candidates. Most prior work in reranking, including the current stateof-the-art (Nogueira et al., 2020), scores each passage independently, modeling P (p|q).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Choice of k for downstream QA Previous stateof-the-art models typically input a large number (k ≥ 100) of passages to the answer generation model. For instance, Izacard and Grave (2021) claim the importance of using a larger value of k to improve QA accuracy. In this paper, we argue that with reranking, using a smaller value of k (5 or 10) and instead employing a larger answer generation model is advantageous given a fixed hardware budget. 2 We show in Section 5 that, as retrieval performance improves, memory is better spent on larger answer generators rather than on more passages, ultimately leading to higher QA accuracy. answer retrieval. JPR uses an approach consisting of first-stage retrieval followed by reranking: the first-stage retrieval obtains candidate passages B from the corpus C, and a reranker processes B to output {p 1 ...p k } ⊂ B. We refer to Section 4.2 for the first-stage retrieval, and focus on the reranking component of the model, which allows (1) efficiently modeling the joint probability P (p 1 ...p k |q), and (2) processing candidate passages with a more expressive model.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We train and evaluate on three datasets that provide a set of distinct answers for each question. Statistics of each dataset are provided in Table 2. WEBQSP (Yih et al., 2016) consists of questions from Google Suggest API, originally from Berant et al. (2013). The answer is a set of distinct entities in Freebase; we recast this problem as textual question answering based on Wikipedia. AMBIGQA  consists of questions mined from Google search queries, originally from NQ (Kwiatkowski et al., 2019). Each question is paired with an annotated set of distinct answers that are equally valid based on Wikipedia. TREC (Baudiš and Šedivỳ, 2015) contains questions curated from TREC QA tracks, along with regular expressions as answers. Prior work uses this data as a task of finding a single answer (where retrieving any of the correct answers is sufficient), but we recast the problem as a task of finding all answers, and approximate a set of distinct answers. Details are described in Appendix B.1.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
JPR can obtain candidate passages B from any firststage retrieval model. In this paper, we use DPR + , our own improved version of DPR (Karpukhin et al., 2020) combined with REALM (Guu et al., 2020). DPR and REALM are dual encoders with a supervised objective and an unsupervised, language modeling objective, respectively. We initialize the dual encoder with REALM and train on supervised datasets using the objective from DPR. More details are provided in Appendix A.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
DPR + +Nogueira et al. (2020) uses DPR + followed by Nogueira et al. (2020), the state-of-theart document ranker. It processes each passage p i in B independently and is trained to output yes if p i contains any valid answer to q, otherwise no. At inference, the probability for each p i is computed by taking a softmax over the logit of yes and no. The top k passages are chosen based on the probabilities assigned to yes. INDEPPR is our own baseline that is a strict nonautoregressive version of JPR in which prediction of a passage is independent from other passages in the retrieved set. It obtains candidate passages B through DPR + and the encoder of the reranker processes q and B, as JPR does. Different from JPR, the decoder is trained to output a single token i (1 ≤ i ≤ |B|) rather than a sequence. The objective is the sum of −logP (p|q, B) of the passages including any valid answer to q. At inference, INDEPPR outputs the top k passages based the logit values of the passage indices. We compare mainly to IN-DEPPR because it is the strict non-autoregressive version of JPR, and is empirically better than or comparable to Nogueira et al. (2020) (Section 5.1).
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use the English Wikipedia from 12/20/2018 as the retrieval corpus C, where each article is split into passages with up to 288 wordpieces, All rerankers are based on T5 (Raffel et al., 2020), a pretrained encoder-decoder model; T5-base is used unless otherwise specified. We use |B| = 100, k = {5, 10}. Models are first trained on NQ (Kwiatkowski et al., 2019) and then finetuned  Table 3: Results on passage retrieval in MRECALL. The two numbers in each cell indicate performance on all questions and on questions with more than one answer, respectively. Test-set metrics on AMBIGQA are not available as its test set is hidden, but we report the test results on question answering in Section 5.2.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use an answer generation model based on Izacard and Grave (2021) which we train to generate a sequence of answers, separated by a [SEP] token, given a set of retrieved passages. Our main model uses JPR to obtain passages fed into the answer generation model. The baselines obtain passages from either DPR + only or INDEPPR, described in Section 4.3.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Finally, JPR establishes a new state-of-the-art, outperforming the previous state-of-the-art on AM-BIGQA (Gao et al., 2021) with extensive reranking and the answer generation model trained using x3 more resources than ours. 6
Joint Passage Ranking for Diverse Multi-Answer Retrieval
While our main contributions are in multi-answer retrieval, we experiment on NQ to demonstrate  Table 7: Question Answering results on multi-answer datasets. The two values in each cell indicate F1 on all questions and F1 on questions with multiple answers only, respectively. Mem compares the required hardware memory during training. Note that Gao et al. (2021) reranks 1000 passages instead of 100, and trains an answer generation model using x3 more memory than ours. Better retrieval enables using larger answer generation models on fewer retrieved passages.  Table 8: Question Answering results on NQ. We report Exact Match (EM) accuracy. The first five rows are from our own experiments, which all use the same hardware resources for training. The last row is the previous state-of-the-art which requires x5 more resources than ours to train the model. that the value of good reranking extends to the single-answer scenario. Table 8 indicates two observations consistent to the findings from multianswer retrieval: (1) when compared within the same setting (same T5 and k), INDEPPR always outperforms DPR + only, and (2) with reranking, {k = 10, 3B} outperforms {k = 40, large}. Finally, our best model outperforms the previous state-of-the-art (Izacard and Grave, 2021) which uses x5 more training resources. Altogether, this result (1) justifies our choice of focusing on reranking, and (2) shows that INDEPPR is very competitive and thus our JPR results in multi-answer retrieval are very strong.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use a pretrained dual encoder model from REALM (Guu et al., 2020) and further finetune it on the QA datasets using the objective from DPR (Karpukhin et al., 2020):
Joint Passage Ranking for Diverse Multi-Answer Retrieval
where f q and f p are trainable encoders for the questions and passages, respectively, p + is a positive passage (i.e., a passage containing the answer), and B − is a set of negative passages (i.e., passages without the answer). As shown in Karpukhin et al. (2020), a choice of B − is significant for the performance. We explore two methods: Distant negatives follows DPR (Karpukhin et al., 2020) in using distantly obtained negative passages as B − . We obtain two distant negative passages per question: one hard negative, a top prediction from REALM without finetuning, and one random negative, drawn from a uniform distribution, both not containing the answer.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We use distant negatives for multi-answer datasets and full negatives for NQ as this combination gave the best result. B Experiment Details B.1 Data processing for TREC TREC from Baudiš and Šedivỳ (2015) contains regular expressions as the answers. We approximate a set of semantically distinct answers as follows. We first run regular expressions over Wikipedia to detect valid answer text. If there is no valid answer found from Wikipedia, or there are more than 100 valid answers 8 , we discard the question. We then only keep the answers with up to five tokens, following the notion of short answers from Lee et al. (2019). Finally, we group the answers  that are the same after normalization and white space removal. We find that this gives a reasonable approximation of a set of semantically distinct answers. Note that the data we use is the subset of the original data because we discarded a few questions. Statistics are reported in Section 4.1.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
All implementations are based on Tensorflow (Abadi et al., 2015) and Mesh Tensorflow (Shazeer et al., 2018). All experiments are done in Google Cloud TPU. We use batch size that is the maximum that fits one instance of TPU v3-32 (for WEBQSP and AMBIGQA) or TPU v3-8 (TREC). We use the same batch size for INDEPPR; for Nogueira et al. (2020), we use the batch size of 1024. We use the encoder length of 360 and the decoder length of k (JPR) or 1 (all others). We use k = {5, 10} for all experiments. We train JPR with γ = {0, 0.5, 1.0, 1.5} and choose the one with the best accuracy on the development data. We use a flat learning rate of 1 × 10 −3 with warm-up for the first 500 steps. Full hyperparameters are reported in Table 9.
Joint Passage Ranking for Diverse Multi-Answer Retrieval
We additionally report retrieval performance in α-NDCG @ k, one of the metrics for diverse retrieval in IR (Clarke et al., 2008;Sakai and Zeng, 2019). It is a variant of NDCG (Järvelin and Kekäläinen, 2002), but penalizes retrieval of the same answer. We refer to Clarke et al. (2008) for a complete definition. We use α = 0.9.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Open-domain question answering (ODQA) is a core language understanding task concerned with answering factoid questions over large document collections (Voorhees and Tice, 2000;Brill et al., 2002). Due to its wide applicability, ODQA has received substantial attention in recent years . Typically, systems tackling ODQA tasks follow the "retrieve-and-read" paradigm, where a retriever first retrieves a set of candidate passages, followed by a reader which receives the retrieved passages and produces the final answer.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Open-domain question answering (ODQA) is a core language understanding task concerned with answering factoid questions over large document collections (Voorhees and Tice, 2000;Brill et al., 2002). Due to its wide applicability, ODQA has received substantial attention in recent years . Typically, systems tackling ODQA tasks follow the "retrieve-and-read" paradigm, where a retriever first retrieves a set of candidate passages, followed by a reader which receives the retrieved passages and produces the final answer.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The retrieve-and-read paradigm has been effective for benchmarks such as Natural Questions  (NQ)  and TriviaQA , where the answer is a single phrase from a single passage. However, in many cases, a question might have many answers, spread across multiple passages. Consider the example in Fig. 1. Eric Newman produced multiple movies, so finding them, along with their directors, requires incorporating information from many passages. Such questions pose two main challenges to retrieve-andread systems. First, as there are multiple answers that can be far apart, the reader must reason over a long text sequence to generate all correct answers. Second, since the reader is computationally constrained to process at most K passages, the retriever must score all necessary passages at its top-K results, which is challenging and even impossible when the number of relevant passages exceeds K.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Nevertheless, research on multi-answer questions has largely been underexplored. While previous works proposed questions that involve reading multiple passages, the number of passages was quite small. AMBIGQA  studied ambiguous questions from NQ with several answers. However, as 70% of its questions have at most two answers, retrieve-and-read models can be adapted to AMBIGQA. HOTPOTQA  focused on multi-hop reasoning, but its questions require no more than two passages to answer. WIKINLDB (Thorne et al., 2021) is a benchmark for testing reasoning over multiple facts. However, WIKINLDB restricted its text corpus to databases of 1,000 facts at most, making it significantly smaller than standard ODQA corpora. Moreover, these facts are model-generated utterances rather than natural language passages. Multianswer questions are also rare in real-world user questions (Bajaj et al., 2016;, which can be attributed to the performance bias of existing systems. Namely, people mostly pose questions that they can successfully get answers to with current technology. This does not diminish the importance of multi-answer questions ('Which drugs are effective against skin cancer?'; 'Which plants can be grown in an apartment?'), which constitute an important research challenge.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Nevertheless, research on multi-answer questions has largely been underexplored. While previous works proposed questions that involve reading multiple passages, the number of passages was quite small. AMBIGQA  studied ambiguous questions from NQ with several answers. However, as 70% of its questions have at most two answers, retrieve-and-read models can be adapted to AMBIGQA. HOTPOTQA  focused on multi-hop reasoning, but its questions require no more than two passages to answer. WIKINLDB (Thorne et al., 2021) is a benchmark for testing reasoning over multiple facts. However, WIKINLDB restricted its text corpus to databases of 1,000 facts at most, making it significantly smaller than standard ODQA corpora. Moreover, these facts are model-generated utterances rather than natural language passages. Multianswer questions are also rare in real-world user questions (Bajaj et al., 2016;, which can be attributed to the performance bias of existing systems. Namely, people mostly pose questions that they can successfully get answers to with current technology. This does not diminish the importance of multi-answer questions ('Which drugs are effective against skin cancer?'; 'Which plants can be grown in an apartment?'), which constitute an important research challenge.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In this work, we present QAMPARI, a benchmark for Questions with many Answers over Multiple Paragraphs, Indeed. All questions in QAMPARI have at least 5 answers, with an average of 13 answers. Examples are semi-automatically generated using two data sources, Wikidata (Vrandečić and Krötzsch, 2014) and Wikipedia tables. We automatically generate multi-answer questions of the form "What/Who has [relation] with [entity]?" and convert these into pseudo-language using manually defined templates. Then, we verify that our questions are answerable from Wikipedia by automatically extracting evidence passages for all their answers. Finally, we use crowdsourcing to validate example correctness, and paraphrase questions into natural language (Wang et al., 2015). To further enrich our data we also generate composition questions, that compose two relations (as in Fig. 1), and intersection questions, such as "What movies were produced and directed by Clint Eastwood?". Overall, QAMPARI contains 2K development and test questions and more than 60K training examples -see Tab. 1 for some examples. We evaluate a large suite of baselines, including models from the retrieve-and-read family as well as a closed-book question answering model , and find that they struggle on QAM-PARI. In the retrieve-and-read setup, we experiment with both BM25 (Robertson and Zaragoza, 2009) and DPR  retrievers, followed by either (a) a RAG-like reader ) that given each retrieved passage either decodes an answer or abstains, or (b) an FiD reader ) that takes the encoded representations of multiple passages and decodes the list of answers directly.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In this work, we present QAMPARI, a benchmark for Questions with many Answers over Multiple Paragraphs, Indeed. All questions in QAMPARI have at least 5 answers, with an average of 13 answers. Examples are semi-automatically generated using two data sources, Wikidata (Vrandečić and Krötzsch, 2014) and Wikipedia tables. We automatically generate multi-answer questions of the form "What/Who has [relation] with [entity]?" and convert these into pseudo-language using manually defined templates. Then, we verify that our questions are answerable from Wikipedia by automatically extracting evidence passages for all their answers. Finally, we use crowdsourcing to validate example correctness, and paraphrase questions into natural language (Wang et al., 2015). To further enrich our data we also generate composition questions, that compose two relations (as in Fig. 1), and intersection questions, such as "What movies were produced and directed by Clint Eastwood?". Overall, QAMPARI contains 2K development and test questions and more than 60K training examples -see Tab. 1 for some examples. We evaluate a large suite of baselines, including models from the retrieve-and-read family as well as a closed-book question answering model , and find that they struggle on QAM-PARI. In the retrieve-and-read setup, we experiment with both BM25 (Robertson and Zaragoza, 2009) and DPR  retrievers, followed by either (a) a RAG-like reader ) that given each retrieved passage either decodes an answer or abstains, or (b) an FiD reader ) that takes the encoded representations of multiple passages and decodes the list of answers directly.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
When training models on QAMPARI alone, or in a multi-task setup with NQ, we observe that QAMPARI is challenging in terms of both passage retrieval and answer generation. Namely, the best model reaches an F 1 score of 32.8. Moreover, models return more than 80% of the correct answers in only 31.2% of the test examples, well below performance on single-answer datasets like NQ.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To summarize, QAMPARI is a challenging benchmark for evaluating the ability of ODQA models to handle questions with many answers over multiple passages. We advocate to evaluate ODQA models not on QAMPARI alone, but alongside benchmarks such as NQ and TriviaQA. Such joint evaluation will test models' ability to handle both single-and multi-answer questions, an evaluation that the community is currently lacking. The QAMPARI benchmark, models and relevant codebase are available at: https://samsam3232. github.io/qampari/.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Each example in QAMPARI is a triple (q, A, P), where q is a question, A is a set of answers and P is a set of passages from our target corpus. An answer a ∈ A has 1-2 evidence passages from P (see Fig. 1).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We define passages as consecutive sentences from our corpus (Wikipedia), that span on average 100 words. As our focus is multi-answer questions, examples in QAMPARI have |A| ≥ 5.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We generate examples in two steps. First, we generate simple questions that involve a single entity and relation, e.g., "Who was drafted by the Brooklyn Nets?" ( §2.1). Then, we expand such questions to generate complex questions with intersection and composition operations ( §2.2).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To increase diversity, questions are generated from two data sources, Wikidata and Wikipedia tables. We first describe example generation over Wikidata, then briefly present the generation process from Wikipedia tables in §2.3. In both cases, we ensure answers can be derived from evidence passages in Wikipedia. 1 Tab. 1 presents examples from each data source and question type.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Notation We introduce notation for formal queries over Wikidata to explain example generation. Wikidata is a knowledge graph, K, that can be viewed as a set of labeled edges (e 1 , r, e 2 ). Graph nodes e 1 , e 2 ∈ E are entities connected by an edge labeled by the relation r ∈ R. For example, a possible labeled edge is (BarackObama, ReceivedAward, NobelPeacePrize).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
One can query K by applying a relation r over an entity e, resulting in a simple query r(e) whose denotation (answer set) is r(e) = {e i | (e i , r, e) ∈ K}. Composition queries are formed by applying a relation over the result of a simple query. We denote a composition query by r 2 (r 1 (e)), and its denotation is r 2 (r 1 (e)) = {e i | ∃e j s.t (e i , r 2 , e j ) ∈ K ∧ (e j , r 1 , e) ∈ K}. Last, an intersection query r 1 (e 1 ) ⊓ r 2 (e 2 ) corresponds to the intersection of two simple queries, r 1 (e 1 ) ⊓ r 2 (e 2 ) = {e i | (e i , r 1 , e 1 ) ∈ K ∧ (e i , r 2 , e 2 ) ∈ K}.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Fig. 2 provides an overview of our procedure for creating simple question examples: (i) We manually define query templates, (ii) populate query templates using K to create queries with a sufficiently large number of answers in K, (iii) automatically identify evidence passages for the answers and filter out noisy examples, (iv) map query templates to question templates to obtain pseudo-language questions, and (v) validate answers and paraphrase pseudo-language questions through crowdsourcing. Next, we describe each of these steps in detail.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Generating query templates We manually select a set of 135 relationsR ⊂ R, which will be used in our query templates. We select frequent relations from Wikidata for which denotations contain many entities (e.g., ReceivedAward). The list of relations is in App. A. For each relation, we manually write a template to map queries to pseudo-language questions. For example, the template for ReceivedAward is "Who received the award X?" Some relations are underspecified -for example, LocatedIn can describe the location of buildings, geographical features, and cities. When generating synthetic questions, this leads to vague questions such as "What is located in Paris?". To address this, we manually split these to typed relations that specify the semantic type of their an-swers/denotations. This is done using the type hierarchy given in Wikidata and given the type t of answer entities. We denote typed relations by r t , and the denotation of r t (e) comprises all entities of type t returned by r(e). For example, the entity The Louvre has type cultural organization, and we can map the relevant query template to the pseudo-language question "Which cultural organization is located in Paris?".
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Finding evidence sentences For an ODQA benchmark, we must verify that every answer is found in our target corpus. We do this by identifying candidate evidence sentences from Wikipedia, and verifying they entail the answer, using a Natural Language Inference (NLI) model.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Specifically, every simple query-answer pair can be viewed as a triple (e 1 , r, e 2 ). We use a "distant supervision" approach (Mintz et al., 2009), similar to KELM (Agarwal et al., 2021), and define any sentence in the Wikipedia page of entity e 1 that contains the entity e 2 , or one of its Wikidata aliases, as a candidate evidence sentence (and vice versa in the page of e 2 ). E.g., in Fig. 2, the evidence for (BarackObama, ReceivedAward, NobelPeacePrize) appears on the page Barack Obama, where 'Nobel Peace Prize' appears.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Specifically, every simple query-answer pair can be viewed as a triple (e 1 , r, e 2 ). We use a "distant supervision" approach (Mintz et al., 2009), similar to KELM (Agarwal et al., 2021), and define any sentence in the Wikipedia page of entity e 1 that contains the entity e 2 , or one of its Wikidata aliases, as a candidate evidence sentence (and vice versa in the page of e 2 ). E.g., in Fig. 2, the evidence for (BarackObama, ReceivedAward, NobelPeacePrize) appears on the page Barack Obama, where 'Nobel Peace Prize' appears.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Aligning Wikipedia sentences to Wikidata can lead to false positives. E.g., for the triple (TheGoonies, HasScreenwriter, StevenSpielberg), most mentions of Spielberg in the page TheGoonies are not as a screenwriter. To account for this, we use an off-theshelf NLI model. 2 For every answer, we consider each candidate evidence sentence along with its two preceding sentences, and check whether they entail the hypothesis phrase describing the triple (e 1 , r, e 2 ). We use templates to phrase triples as short declarative sentences ("The Goonies has Steven Spielberg as screenwriter"
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
). An answer is validated if there is an evidence sentence that entails the triple. Manual analysis shows this  process eliminates 70% of false positives, while removing only 7.5% of the correct alignments.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Query filtering After finding evidence sentences, we only keep queries that at least 80% of their answers were validated and their number of validated answers is between 5 and 200. The resulting set contains 60,792 simple queries, where each query has a set of validated answers, A, and of passages P that contain the identified evidence sentences. 3
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
To increase diversity, we expand simple queries to composition and intersection queries, for which answers require reading two passages.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Intersection Intersection queries are generated by finding two simple queries such that the size of the intersection of their denotations is at least 5. To avoid improbable questions such as "Which competition was won by Manchester City and had Manchester City as a participant?", we add a constraint that the denotation of one of the simple queries cannot be a subset of the other. Formally, the set of intersection queries are all queries r 1 (e 1 ) ⊓ r 2 (e 2 ) such that | r 2 (e 2 ) ⊓ r 1 (e 1 ) | ≥ 5, r 1 (e 1 ) ⊈ r 2 (e 2 ) , and r 2 (e 2 ) ⊈ r 1 (e 1 ) . Pseudo-language questions are generated by heuristically combining the two simple questions, for example "Which television program had Chris Carter as screenwriter and had Frank Spotnitz as screenwriter?". There is no need to perform an- 3 We keep a single evidence passage for every triple.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
swer validation since all of the underlying intersecting answers were already validated.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We look at all Wikipedia tables with title "List of X" that have at least 5 rows, in total, 1,897 tables. We find the "key" column, c key in each table using the table classifier from Talmor et al. (2021), which outputs the column of entities that the table describes. For example, in the table List of nuclear whistle blowers, c key is 'name' and specifies the whistle-blower names. This naturally creates simple questions of the form "Who or what is X?".
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
QAMPARI contains a training set, whose goal is to teach the model to handle multi-answer questions. However, we do not want the model to memorize how particular Wikidata relations map to text patterns. Consequently, we perform a relation split, randomly splitting the setR into two equally-sized setsR train andR test . Simple queries are assigned to the train/test set based on their relation, composition queries r 2 (r 1 (e)) are assigned to the test set iff either r 1 or r 2 are inR test , and intersection queries r 1 (e 1 ) ⊓ r 2 (e 2 ) are placed in the test set iff both r 1 and r 2 are inR test .
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We now create the train/development/test split (Tab. 2). The main bottleneck in our example generation pipeline is validation of the test set through crowdsourcing ( §2.5), since each question requires validating all of the answers. Thus, we pre-determine the test set to contain 1,000 simple questions (830 from Wikidata, 170 from Wikipedia tables) and 1,000 complex questions (400 Wikidata composition questions, 400 Wikidata intersection questions, 200 Wikipedia tables composition questions). For simple Wikidata questions, we sample 830 questions such that the distribution over relations fromR test is roughly uniform. All Wikipedia tables simple questions are placed in the test set, and for complex questions we randomly sample the pre-determined number from the set of generated questions. Last, the test set is randomly split in half to a development set and test set. We also sub-sample training set examples, such that each relation appears in at most 1,000 examples.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Correctness validation For every question and answer, we present a crowdsourcing worker with the question, the answer, and links to the Wikipedia page (or pages for complex questions) with the evidence passage. We ask the worker to check if the question can be answered from the given pages, using the text only (no infoboxes or tables).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Since the vast majority of examples are correct, we test worker performance by injecting wrong answers in 10% of the cases and reject workers that fail to identify wrong answers. Moreover, we manually verify 5% of examples marked as correct and all examples marked as incorrect, and again reject low-performing workers. Overall, 24 annotators validated 30,259 answers for an average pay of 12.5$ per hour. We find that our process for generating examples is accurate, with 96.6% of the answers validated. Non-validated questions were replaced until 2,000 questions were validated. A question is defined non-validated if its number of distinct answers goes below 5. Snapshots from the presented tasks are in App. C.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Paraphrasing Since our questions are in pseudolanguage, we follow past work (Wang et al., 2015) and ask workers to re-phrase 3,000 questions in the training set and the entire development/test set. We restrict this task to US or UK workers who pass a qualification test. We randomly verified half of the paraphrases for each worker for quality assurance.  , where the median is 2. On average, simple questions have more answers than complex ones while being shorter in length. We note that since test and development questions were manually re-phrased by annotators they are generally shorter than the training questions. Figure 3a presents a binned distribution of the number of answers per question in the development and test sets. Roughly half of the questions have 8   or more answers, with 20% having more than 15 answers and 3.5% with over 50 answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Paraphrasing Since our questions are in pseudolanguage, we follow past work (Wang et al., 2015) and ask workers to re-phrase 3,000 questions in the training set and the entire development/test set. We restrict this task to US or UK workers who pass a qualification test. We randomly verified half of the paraphrases for each worker for quality assurance.  , where the median is 2. On average, simple questions have more answers than complex ones while being shorter in length. We note that since test and development questions were manually re-phrased by annotators they are generally shorter than the training questions. Figure 3a presents a binned distribution of the number of answers per question in the development and test sets. Roughly half of the questions have 8   or more answers, with 20% having more than 15 answers and 3.5% with over 50 answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Extended set As discussed in §2.5, we manually validate each answer in QAMPARI is supported by sentences from Wikipedia. However, Wikipedia might contain additional correct answers. To alleviate this issue, we manually annotate additional gold answers for a subset of test questions, and name it the ExtendedSet. We randomly sampled 200 questions from the test set and had an author manually annotate as many additional answers as possible in 12 minutes, per question. This process is not guaranteed to be complete, as it would require manually reviewing all of Wikipedia. Moreover, questions with hundreds of gold answers ("Who worked for Burton F. C?") would incur hours of annotation, which is too expensive. This is similar to work in open information extraction (Vo and Bagheri, 2017), where creating the full gold set of triples is not feasible. Fig. 3 plots the number of added answers per question on the extended set. In 30% of the questions, we did not add any answer, and the median/average/maximum number of added answers are 2/3.13/16 respectively. Evaluation on the test set and the extended set in §4.3 shows that model precision on the extended set is somewhat higher, but does not alter model ranking, illustrating the reliability our test set.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Retriever For retrieval, we experiment with both sparse and dense retrieval models on Wikipedia. As discussed in §2, we chunk Wikipedia into passages of consecutive sentences, using NLTK's sentence tokenizer, where each passage is 100 words on average. For all retrievers, we evaluate retrieval accuracy of the top-200 passages returned per question.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We use BM25 (Robertson and Zaragoza, 2009) as a strong sparse retrieval model. BM25 scores question-passage pairs based on their lexical similarity. It has been shown that BM25 is notoriously hard to beat using unsupervised retrieval methods Ram et al., 2022), and achieves comparable performance to that of supervised methods (Thakur et al., 2021). As our dense retriever we finetune on QAMPARI a DPR model  trained on NQ. We finetune DPR in the typical contrastive manner (inbatch training), with one positive and one negative passage per question. Positives are sampled from the evidence passages, and negatives are sampled from the top-10 highest scoring passages, according to BM25, which do not contain the answer.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We use BM25 (Robertson and Zaragoza, 2009) as a strong sparse retrieval model. BM25 scores question-passage pairs based on their lexical similarity. It has been shown that BM25 is notoriously hard to beat using unsupervised retrieval methods Ram et al., 2022), and achieves comparable performance to that of supervised methods (Thakur et al., 2021). As our dense retriever we finetune on QAMPARI a DPR model  trained on NQ. We finetune DPR in the typical contrastive manner (inbatch training), with one positive and one negative passage per question. Positives are sampled from the evidence passages, and negatives are sampled from the top-10 highest scoring passages, according to BM25, which do not contain the answer.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Reader We experiment with two readers -a Passage-Independent Generator (PIG), which reads each passage independently (a-là RAG ), and a Fusion-in-Decoder (FiD) model , which reads multiple passages simultaneously.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
PIG is an encoder-decoder model that takes each of the retrieved passages as input and decodes a single answer or outputs "Not Relevant" to indicate there is no answer. The final output is the union of all decoded answers across retrieved passages. We initialize PIG with T5-large (Raffel et al., 2019) and train with standard maximum likelihood. We use evidence passages as positive examples and the top scoring retrieved passage that is not an evidence passage and does not contain an answer (or its aliases) as a negative example.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
FiD encodes each of the retrieved passages along with the input question. Its decoder then attends to the encoded representation and outputs a list of answers. We initialize FiD using a pretrained T5-Large model (Raffel et al., 2019) and train with standard maximum likelihood.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
FiD is computationally expensive, as its decoder attends to a large number of encoded tokens and the generated output is long. Thus, we can only fit the top-50 passages returned by the retriever on a single A100 GPU.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Closed-book question answering We also experiment with a closed-book setting, where the QA model generates answers from knowledge encoded in its parameters without any evidence passages. We initialize our closed-book QA model with T5-SSM with 3B parameters , and train it with standard maximum likelihoodthe question is provided as input, and the model is trained to generate the gold set of answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Zero-shot We test the zero-shot ability of Open AI's text-davinci-003, from the Instruct-GPT family (Ouyang et al., 2022). We use GPT-3 in: (a) closed-book QA setup; (b) as a multi-passage reader. In the closed-book setup, the model receives only the question and is asked to provide a list of answers. In the reader setup, the model gets the question and the 15 highest-ranking passages from BM25 (the maximal number that fits in the context) and is asked to output a list of answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Zero-shot We test the zero-shot ability of Open AI's text-davinci-003, from the Instruct-GPT family (Ouyang et al., 2022). We use GPT-3 in: (a) closed-book QA setup; (b) as a multi-passage reader. In the closed-book setup, the model receives only the question and is asked to provide a list of answers. In the reader setup, the model gets the question and the 15 highest-ranking passages from BM25 (the maximal number that fits in the context) and is asked to output a list of answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We created QAMPARI as a benchmark to be evaluated alongside additional ODQA benchmarks, such as NQ. Since it is semi-automatically generated, one can develop models tailored for QAMPARI. However, our goal is to have a single model that performs well across a wide variety of question types. Thus, we train and test models in a multitask setup, on both NQ and QAMPARI, in addition to a QAMPARI only setting. We also train our models on NQ only and evaluate them on QAM-PARI, to verify QAMPARI's training set indeed improves answering questions with many answers.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Our main metrics are recall, precision, and F 1 . Specifically, for test example (q, P, A), and a predicted set of answers A pred , recall, precision, and F 1 are standardly computed by comparing A and  A pred , allowing for aliases (i.e., a gold answer is covered if it or one of its aliases are in A pred ). The model scores are averaged across examples. To get a sense of the average accuracy across examples, we measure the fraction of examples with F 1 of at least 0.5 (%F 1 ≥0.5) and the fraction with recall of at least 0.8 (%Recall≥0.8). For NQ, we report the standard exact match (EM) metric. We evaluate the retriever with RECALL@K, that is, the fraction of answers that appear in the top-K retrieved passages, averaged across examples. This metric comes in two flavors: (a) Answer RE-CALL@K (ARECALL@K): for every gold answer whether it or one of its aliases appear in the top-K retrieved passages. It is a loose metric since an answer can appear in a passage that does not provide any evidence to support the answer; (b) Evidence RECALL@K (ERECALL@K): since we have evidence paragraphs for every answer, we consider for every gold answer the fraction of evidence passages in the top-K retrieved passages. This is a strict metric since an answer can sometimes be answered by passages other than the ones we identified.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Tab. 3 presents passage retrieval results on QAM-PARI test. Scores for ARecall@200 for BM25 and DPR are 61.0% and 55.2%, respectively. As for ERecall@K, results are unsurprisingly lower. BM25 retrieves 55.6% of the evidence passages with K=200, while DPR retrieves only 30.2% of evidence passages. 4 Overall, DPR pretrained on NQ and finetuned on QAMPARI performs worse than BM25. This is in line with Sciavolino et al. (2021) who showed that, when tested on questions with rare entities, DPR performs worse than BM25. We hypothesize that rare entities in QAMPARI questions may account for DPR's lower performance.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Tab. 3 presents passage retrieval results on QAM-PARI test. Scores for ARecall@200 for BM25 and DPR are 61.0% and 55.2%, respectively. As for ERecall@K, results are unsurprisingly lower. BM25 retrieves 55.6% of the evidence passages with K=200, while DPR retrieves only 30.2% of evidence passages. 4 Overall, DPR pretrained on NQ and finetuned on QAMPARI performs worse than BM25. This is in line with Sciavolino et al. (2021) who showed that, when tested on questions with rare entities, DPR performs worse than BM25. We hypothesize that rare entities in QAMPARI questions may account for DPR's lower performance.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Tab. 4 lists results on the test sets of QAMPARI and NQ. Overall, performance on QAMPARI is  low. FiD-DPR and PIG-DPR are more precisionoriented with FiD-DPR achieving precision of 41.3 and PIG-DPR a precision of 44.8. PIG-BM25 is recall-oriented, achieving recall of 47.9. Overall, PIG variants perform best, with small differences between PIG-BM25 and PIG-DPR, and both are slightly higher than FiD-BM25.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
When training on both NQ and QAMPARI (MT), performance on NQ (47.2 with BM25 and 53.1 with DPR) is similar to that reported by Izacard and Grave (2021) (44.1 with BM25 and 51.4 with DPR). When training on NQ only, results on QAMPARI are significantly lower than when training also on QAMPARI, showing that training on QAMPARI improves performance on multi-answer questions, as expected. The lower performance on QAMPARI compared to NQ, despite the fact that NQ's EM evaluation metric is much more strict than the metrics used for QAMPARI, illustrates the challenge in answering multi-answer questions.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
PIG-DPR has much higher recall than FiD-DPR, showing that going over 200 passages independently (PIG) leads to higher recall than jointly reasoning over 50 passages (FiD). Moreover, the solid performance of PIG-DPR indicates that QA performance is more correlated with ARecall@K than ERecall@K (Tab. 3).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Finetuned closed-book performance is low with an F 1 of 2.6 for QAMPARI, which we attribute to the relation-based train/test split ( §2.4). This guarantees that there is no overlap between train and test questions. Lewis et al. (2021) have shown that mitigating such train-test overlap causes a drop in QA performance, with a drastic drop being observed in closed-book models.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Finetuned closed-book performance is low with an F 1 of 2.6 for QAMPARI, which we attribute to the relation-based train/test split ( §2.4). This guarantees that there is no overlap between train and test questions. Lewis et al. (2021) have shown that mitigating such train-test overlap causes a drop in QA performance, with a drastic drop being observed in closed-book models.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
The performance of zero-shot models is lower than finetuned retrieve-and-read models, as expected. However, text-davinci-003's performance in the closed book setup is impressive and significantly better than finetuned T5-3B.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
ExtendedSet results We report results for FiD and PIG on the ExtendedSet (see §3) in §F. As expected, considering additional correct answers improves the precision of all models. Since changes to recall are small, the overall F 1 is higher when considering manual annotations. Importantly, ranking across models does not change, and the absolute performance remains low, suggesting that our test set can be safely used for evaluation.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Oracle analysis To disentangle retrieval from answer extraction, we run PIG and FiD in an oracle setup, where we assume a perfect retriever and run our readers on the gold evidence passages only. Performance of both models greatly improves in this setup, with larger gains for PIG. This shows that developing better retrieval mechanisms for multianswer questions can greatly benefit QAMPARI. FiD's recall is still limited (47.5), illustrating the challenge of reading a large number of documents. Full oracle results are in §G (Tab. §9).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We release QAMPARI, a dataset targeting ODQA models ability to answer multi-answer questions, and show that it is challenging for current stateof-the-art models. QAMPARI will aid develop models that answer a wide range of question types, including single-and multi-answer questions.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
A key limitation of QAMPARI is that the gold set of answers is incomplete. Thus, predicted answers might be correct but missing from the gold answer set. The ExtendedSet addresses this problem partially, allowing a more accurate model ranking, but even in this set all the correct answers are not part of the gold set. A second limitation is that our data generation process is mostly automatic and is thus amenable to reverse-engineering. Hence, we recommend evaluating models on QAMPARI along with additional benchmarks created with a different generation process. Last, our data generation process can only generate answers based on relations from Wikidata and relations that are in Wikipedia tables, and thus its scope does not generalize to arbitrary relations.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In Tab. 5. we gathered all the 135 relations we used to create our simple questions. The 423 relations used to create our composition questions can be found in our code base.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
Composition questions overall template is: What is the <comp_property> of <subtype> who/which <base_property>?. All the templates are in our code base. Fig. 4 shows two screenshots of the task crowdsourcing workers performed.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
For both readers (FiD and PIG), we used T5-large which has 770 million parameters. We used an A100 to train both of them, FiD with a batch size of 8 and PIG with a batch size of 32 for a single GPU. We trained each of them for around 48 hours on two GPUs. For FiD, we concatenated the answers using # as a separator. At evaluation time, there is no importance to the order of the answers. For both PIG and FiD, all aliases of a given gold entity provided by Wikidata are used as additional correct answers. When verifying whether our model predicted an answer A, we verify whether it predicted A or any of its aliases. We performed an hyper parameter search around the learning rate, the number of training steps, the ratio of positive to negative (for PIG) and the number of times an NQ example will appear in each epoch (for multi task). Tab. 6 presents the parameters of the reported results. We report the results of a single run with seed 0.
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We break test performance of FiD-BM25 (MT) by question type (Tab. 7). Surprisingly, performance on simple questions is lower than complex questions, and intersection questions seem easiest. Possible explanations are: (a) simple questions have more answers (see Tab. 2), which makes them harder, and (b) models can predict the answer given just one evidence passage, due to "shortcuts" (Chen and Durrett, 2019), or parametric knowledge (Longpre et al., 2021).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
We break test performance of FiD-BM25 (MT) by question type (Tab. 7). Surprisingly, performance on simple questions is lower than complex questions, and intersection questions seem easiest. Possible explanations are: (a) simple questions have more answers (see Tab. 2), which makes them harder, and (b) models can predict the answer given just one evidence passage, due to "shortcuts" (Chen and Durrett, 2019), or parametric knowledge (Longpre et al., 2021).
QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
In Tab 
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Addressing such queries has been primarily studied in the context of question answering with structured knowledge bases (KBs), where query constraints are grounded to predefined predicates and symbolically executed. However, KBs can be incomplete and expensive to curate and maintain. Meanwhile, advances in information retrieval may enable developing systems that can address such queries without relying on structured KBs, by matching query constraints directly to supporting evidence in text documents. However, queries that combine multiple constraints with implicit set operations are not well represented in existing retrieval benchmarks such as MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019). Also, such datasets do not focus on retrieving an exhaustive document set, instead limiting annotation to the top few results of a baseline information retrieval system. To analyze retrieval system performance on such queries, we present QUEST, a dataset with natural language queries from four domains, that are mapped to relatively comprehensive sets of entities corresponding to Wikipedia pages. We use categories and their mapping to entities in Wikipedia as a building block for our dataset construction approach, but do not allow access to this semistructured data source at inference time, to simulate text-based retrieval. Wikipedia categories represent a broad set of natural language descriptions of entity properties and often correspond to selective information need queries that could be plausibly issued by a search engine user. The relationship between property names and document text is often subtle and requires sophisticated reasoning to determine, representing the natural language inference challenge inherent in the task.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Figure 2), while also efficiently scaling to large collections of entities. We evaluate several retrieval systems by finetuning pretrained models on our dataset. Systems are trained to retrieve multidocument sets given a query. We find that current dual encoder and cross-attention models up to the size of T5-Large (Raffel et al., 2020) are largely not effective at performing retrieval for queries with set operations. Queries with conjunctions and negations prove to be especially challenging for models and systems are further challenged with combinations of set operations. Our error analysis reveals that non-relevant false positive entities are often caused by the model ignoring negated constraints, or ignoring the conjunctive constraints in a query.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Many opendomain QA benchmarks, which consider QA over unstructured text corpora, have been proposed in prior work. Some of these, such as TREC (Craswell et al., 2020), MSMarco (Nguyen et al., 2016) and Natural Questions (Kwiatkowski et al., 2019) are constructed using "found data", using real user queries on search engines. Thakur et al. (2021) present a benchmark where they consider many such existing datasets. Datasets such as Hot-potQA (Yang et al., 2018), and MultiRC (Khashabi et al., 2018) have focused on multi-hop question answering. Other work has explored e-commerce datasets (for example, (Kong et al., 2022)), but these have not been released publicly. Notably, the focus of these datasets differs from ours as we focus on queries that contain implicit set operations over exhaustive answer sets. Such queries are not well represented in existing datasets because they occur in the tail of the query distributions considered.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Multi-Answer Retrieval Related work (Min et al., 2021;Amouyal et al., 2022) also studies the problem of multi-answer retrieval, where systems are required to predict multiple distinct answers for a query. Min et al. (2021) adapt existing datasets (for example, WebQuestionsSP (Yih et al., 2016)) to study this setting and propose a new metric, MRecall@K, to evaluate exhaustive recall of multiple answers. We also consider the problem of multi-answer set retrieval, but consider queries that implicitly contain set constraints.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We evaluated several combinations of retrievers and classifiers, as shown in Figure 3. For the retriever component, we consider a sparse BM25 retriever (Robertson et al., 2009) and a dense dual encoder retriever (denoted DE). Following , we initialize our dual encoder from a T5 (Raffel et al., 2020) encoder and train with an in-batch sampled softmax loss (Henderson et al., 2017). Once we have a candidate set, we need to determine a set of relevant entities. To classify relevance of each candidate document for the given query, we consider a cross-attention model which consists of a T5 encoder and decoder. 4 We train the cross-attention classifier using a binary cross-entropy loss with negative examples based on non-relevant documents in top 1,000 documents retrieved by BM25 and random non-relevant documents (similarly to Nogueira and Cho (2019)). As cross-attention classification for a large number of candidates is computationally expensive, we restrict BM25 and the dual encoder to retrieve 100 candidates which are then considered by the crossattention classifier. As our T5-based dual encoder can only efficiently accommodate up to 512 tokens,   we truncate document text. We discuss the impact of this and alternatives in §5. Further, since T5 was pre-trained on Wikipedia, we investigate the impact of memorization in Appendix D. Additional details and hyperparameter settings are in Appendix A.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We report the performance of our baseline systems on the test set in Table 3. In this section, we summarize the key findings from our analysis of these results and the error annotation described in §4.4. Dual encoders outperform BM25. As shown in Table 3, the best overall system uses a T5-Large Dual Encoder instead of BM25 for retrieval. The performance difference is even more significant when comparing recall of Dual Encoders and BM25 directly. We report average recall (average per-example recall of the full set of relevant documents) and MRecall (Min et al., 2021) (the percentage of examples where the candidate set contains all relevant documents), over various candidate set sizes in Table 4. Retrieval and classification are both challenging. As we consider only the top-100 candidates from the retriever, the retriever's recall@100 sets an upper bound on the recall of the overall system. Recall@100 is only 0.476 for the T5-Large Dual Encoder, and the overall recall is further reduced by the T5-Large classifier to 0.368, despite achieving only 0.165 precision. This suggests that there is room for improvement from both stages to improve overall scores. As performance improves for larger T5 sizes for both retrieval and classification, further model scaling could be beneficial. Models struggle with intersection and difference. We also analyzed results across different templates and domains, as shown in Table 5. Different constraints lead to varying distributions over answer set sizes and the atomic categories used. Therefore, it can be difficult to interpret differences in F1 scores across templates. Nevertheless, we found the queries with set union have the highest average F1 scores. Queries with set intersection have the lowest average F1 scores, and queries with set difference also appear to be challenging.
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
Truncating document text usually provides sufficient context. In our experiments, we truncate document text to 512 tokens for the dual encoder, and 384 tokens for the classifier to allow for the document and query to be concatenated. Based on our error analysis ( §4.4), out of the documents with sufficient evidence to judge relevance, evidence occurred in this truncated context 93.2% of the time for the dual encoder, and 96.1% of the time for the classifier. This may explain the relative success of this simple baseline for handling long documents. We also evaluated alternative strategies but these performed worse in preliminary experiments 5 . Future work can evaluate efficient transformer variants (Guo et al., 2022;Beltagy et al., 2020).
QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations
We present QUEST, a new benchmark of queries which contain implicit set operations with corresponding sets of relevant entity documents. Our experiments indicate that such queries present a challenge for modern retrieval systems. Future work could consider approaches that have better inductive biases for handling set operations in natural language expressions (for example, Vilnis et al. (2018)). The attributions in QUEST can be leveraged for building systems that can provide finegrained attributions at inference time. The potential of pretrained generative LMs and multi-evidence aggregation methods to answer set-seeking selective queries, while providing attribution to sources, can also be investigated.