How do language models leverage parametric and non-parametric knowledge?
Parametric and non-parametric knowledge. Petroni et al. (2019) demonstrate that large pre-trained LMs such as BERT (Devlin et al., 2019) memorize the significant amount of world knowledge in their parameters (parametric knowledge), and Roberts et al. (2020) show that fine-tuned T5 without any reference documents (closed-book QA) can achieve competitive performance on open-domain QA. More recent and powerful LMs (Brown et al., 2020; Chowdhery et al., 2022) further improve performance on diverse knowledge-intensive tasks, leveraging their strong parametric memories (Kandpal et al., 2022; Yu et al., 2022). However, relying solely on their parameters to encode a wealth of world knowledge requires a prohibitively large number of parameters and the knowledge can become obsolete quickly (Kasai et al., 2022; Jang et al., 2022). Recent work shows that augmenting LMs with non-
parametric memories (i.e., retrieved text chunks) enables much smaller models to match the performance of larger models (Izacard et al., 2022b; Khandelwal et al., 2020; Min et al., 2022), although Chen et al. (2022) and Longpre et al. (2021) show that even those models can ignore non-parametric knowledge and rely on parametric knowledge.
Nonparametric Masked Language Modeling
Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.
[{"first":"Sewon","last":"Min","middle":[]},{"first":"Weijia","last":"Shi","middle":[]},{"first":"Mike","last":"Lewis","middle":[]},{"first":"Xilun","last":"Chen","middle":[]},{"first":"Wen-tau","last":"Yih","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]}]
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.
[{"first":"Wenhao","last":"Yu","middle":[]},{"first":"Dan","last":"Iter","middle":[]},{"first":"Shuohang","last":"Wang","middle":[]},{"first":"Yichong","last":"Xu","middle":[]},{"first":"Mingxuan","last":"Ju","middle":[]},{"first":"Soumya","last":"Sanyal","middle":[]},{"first":"Chenguang","last":"Zhu","middle":[]},{"first":"Michael","last":"Zeng","middle":[]},{"first":"Meng","last":"Jiang","middle":[]}]
Large Language Models Struggle to Learn Long-Tail Knowledge
The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.
[{"first":"Nikhil","last":"Kandpal","middle":[]},{"first":"Haikang","last":"Deng","middle":[]},{"first":"Adam","last":"Roberts","middle":[]},{"first":"Eric","last":"Wallace","middle":[]},{"first":"Colin","last":"Raffel","middle":[]}]
Language Models as Knowledge Bases?
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as “fill-in-the-blank” cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.
[{"first":"Fabio","last":"Petroni","middle":[]},{"first":"Tim","last":"Rocktäschel","middle":[]},{"first":"Sebastian","last":"Riedel","middle":[]},{"first":"Patrick","last":"Lewis","middle":[]},{"first":"Anton","last":"Bakhtin","middle":[]},{"first":"Yuxiang","last":"Wu","middle":[]},{"first":"Alexander","last":"Miller","middle":[]}]
Generalization through Memorization: Nearest Neighbor Language Models
We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.
[{"first":"Urvashi","last":"Khandelwal","middle":[]},{"first":"Omer","last":"Levy","middle":[]},{"first":"Dan","last":"Jurafsky","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]},{"first":"Mike","last":"Lewis","middle":[]}]
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.
[{"first":"Gautier","last":"Izacard","middle":[]},{"first":"Patrick","last":"Lewis","middle":[]},{"first":"Maria","last":"Lomeli","middle":[]},{"first":"Lucas","last":"Hosseini","middle":[]},{"first":"Fabio","last":"Petroni","middle":[]},{"first":"Timo","last":"Schick","middle":[]},{"first":"Jane","last":"Dwivedi-Yu","middle":[]},{"first":"Armand","last":"Joulin","middle":[]},{"first":"Sebastian","last":"Riedel","middle":[]},{"first":"Edouard","last":"Grave","middle":[]}]
Nonparametric Masked Language Modeling
Current large language models, despite their wide use and impressive performance, are expensive to scale, difficult to update, and struggle with long-tail knowledge and patterns (Kandpal et al., 2022). Recent work follows a retrieve-and-generate approach to partially address these issues (Lewis et al., 2020;Izacard et al., 2022); however, their final predictions are still made by a parametric model. In particular, they still include a softmax over a finite vocabulary, which limits expressivity (Yang et al., 2018;Pappas et al., 2020) and can make them reluctant to predict rare or unseen tokens (e.g., Thessaloniki in Figure 1).
Nonparametric Masked Language Modeling
In this paper, we introduce NPM, the first NonParametric Masked Language Model that predicts tokens solely based on a nonparametric distribution over phrases in a text corpus (Figure 1). NPM consists of an encoder that maps the text Item delivered broken. Very cheaply made and does not even function.
Nonparametric Masked Language Modeling
10/10, would buy this cheap awesome gaming headset again.
Nonparametric Masked Language Modeling
The Church of Saint Demetrius, or Hagios Demetrios, is the main sanctuary dedicated to Saint Demetrius, the patron saint of Thessaloniki.
Nonparametric Masked Language Modeling
The Banpo Bridge (Korean:
Nonparametric Masked Language Modeling
) is a major bridge in downtown Seoul.
Nonparametric Masked Language Modeling
Encoder cheaper than an iPod. It was <mask>.
Nonparametric Masked Language Modeling
broken cheap construction. It was <mask>.
Nonparametric Masked Language Modeling
awesome Hagios Demetrios is located in <mask>.
Nonparametric Masked Language Modeling
The Korean translation of Banpo Brige is <mask>.
Nonparametric Masked Language Modeling
(12 tokens) Figure 1: An illustration of NPM. The encoder maps a masked sentence into a dense vector, and retrieves the nearest phrase from a reference corpus. NPM can fill in the [MASK] with multiple tokens, e.g., Thessaloniki (4 BPE tokens) and unseen words, e.g., ᄇ ᅡ ᆫᄑ ᅩᄃ ᅢᄀ ᅭ (12 BPE tokens).
Nonparametric Masked Language Modeling
into a fixed-sized vector, and a reference corpus from which NPM retrieves a phrase and fills in the [MASK]. It, crucially, does not have a softmax over a fixed vocabulary, but instead has a fully nonparametric distribution over phrases. This is in contrast to a recent body of work that incorporates nonparametric components in a parametric model (Borgeaud et al., 2022;Izacard et al., 2022;Zhong et al., 2022b).
Nonparametric Masked Language Modeling
Training such a nonparametric model introduces two key challenges: (1) full corpus retrieval during training is expensive, and (2) learning to predict an arbitrary length phrase without a decoder is nontrivial. We address the first challenge by using inbatch approximations to full corpus retrieval (Wu et al., 2020;Zhong et al., 2022b), and the second by extending span masking (Joshi et al., 2020) and a phrase-level contrastive objective (Oord et al., 2018;Lee et al., 2021).
Nonparametric Masked Language Modeling
We perform zero-shot evaluation on 16 tasks including classification, fact probing and question answering. They include temporal shift and wordlevel translation tasks that highlight the need to predict new facts or rare phrases. We compare with a range of competitive baselines including encoderonly (Liu et al., 2019), encoder-decoder (Raffel et al., 2020, and decoder-only models (Zhang et al., 2022;Brown et al., 2020). We also compare with a retrieve-and-generate approach that feeds a concatenation of the input and passages to parametric models using off-the-shelf retrieval. Results show that NPM is significantly more parameter-efficient, outperforming up to 500x larger parametric models and up to 37x larger retrieve-and-generate models. It is particularly good at (1) predicting rare words (e.g., an entity split into multiple BPE tokens such as Thessaloniki) and (2) disambiguating word senses (e.g., cheap may indicate inexpensive or of very poor quality; Figure 1). Finally, our evaluation on an entity translation task demonstrates that NPM can predict a word consisting of characters that are extremely rare if not unseen (e.g., non-Latin script; Figure 1).
Nonparametric Masked Language Modeling
In summary, our contributions are as follows.
Nonparametric Masked Language Modeling
1. We introduce NPM, the first nonparametric masked language model that fills in the [MASK] solely from a phrase-level nonparametric distribution over a corpus. 2. We introduce a novel training scheme to train NPM on unlabeled data. We completely remove the softmax over the output vocabulary, enabling an effectively unbounded output space by predicting any n-gram. 3. Zero-shot evaluation on 16 downstream tasks shows that NPM outperforms significantly larger parametric models, are better on rare patterns, scale well, can be efficiently updated at test time, and can predict extremely rare if not unseen tokens (e.g., words in non Latin script).
Nonparametric Masked Language Modeling
Most if not all language models use a softmax function that gives a categorical probability distribution over a finite vocabulary. Yang et al. (2018) showed that this softmax is a low-rank approximation of a high-rank output space, making the model less expressive. Pappas et al. (2020) discussed that a fixed output vocabulary makes language models resistant to adaptation to new domains and tasks. We share the motivation with such prior work and propose to use a nonparametric output space to address these issues. Moreover, although not explicitly explored in this paper, our work that completely removes the softmax over the vocabulary can make training more efficient, especially when the vocabulary is large (e.g., multilingual models (Conneau et al., 2020)).
Nonparametric Masked Language Modeling
Nonparametric models. In nonparametric models, the data distribution is not defined by a fixed set of parameters, but is rather a function of the available data (Siegel, 1957;Hollander et al., 2013). Having complexity that grows as the data grows, they are differentiated from parametric models whose complexity is bounded as a priori. Freeman et al. (2002) noted that the term nonparametric does not imply that they have no parameters, but rather that the number and nature of the effective parameters are flexible and can depend on the data.
Nonparametric Masked Language Modeling
Recent work in NLP has explored nonparametric inference without training (Khandelwal et al., 2020;He et al., 2021;Xu et al., 2022), or trained the nonparametric model on the labeled data for a specific downstream task (Seo et al., 2018(Seo et al., , 2019Lee et al., 2021). In contrast, our work trains a fully q start q end c 2 c 1 c 10/10, would buy this cheap awesome gaming headset again.  Figure 2: Inference of NPM (Section 3.1). Each token in the reference corpus C is mapped into a dense vector space. At test time, a query is represented as two vectors, q start and q end , each in the same vector space. We use a nearest neighbor search to retrieve the start and the end of the phrase using q start and q end , respectively. nonparametric language model without the labeled data and performs a range of tasks zero-shot.
Nonparametric Masked Language Modeling
We introduce NPM, the first NonParametric Masked Language Model. NPM consists of an encoder and a reference corpus, and models a nonparametric distribution over a reference corpus ( Figure 1). The key idea is to map all the phrases in the corpus into a dense vector space using the encoder and, when given a query with a [MASK] at inference, use the encoder to locate the nearest phrase from the corpus and fill in the [MASK].
Nonparametric Masked Language Modeling
Encoder-only models are competitive representation models (Patel et al., 2022), outperforming the other two classes of models in classification tasks (Section 5.4). However, existing encoderonly models are unable to make a prediction whose number of tokens is unknown, making their use cases limited without fine-tuning. NPM addresses this issue, since it can fill in the [MASK] with an arbitrary number of tokens by retrieving a phrase.
Nonparametric Masked Language Modeling
We first describe inference of NPM assuming a learned encoder (Section 3.1), and then describe how we train the encoder to map the text into a good vector space (Section 3.2).
Nonparametric Masked Language Modeling
Overview. The encoder maps every distinct phrase in a reference corpus C into a dense vec-tor space. At test time, the encoder maps the masked query into the same vector space and retrieves phrases from C to fill in the [MASK]. Here, C does not have to be the same as the training corpus, and can be replaced or scaled at test time without re-training the encoder.
Nonparametric Masked Language Modeling
In practice, there is a significant number of phrases in the corpus, and it is expensive to index all of them. We therefore use a technique from Lee et al. (2021) that represents a phrase with token representations of the start and the end of the phrase. In this approach, we index representations of each distinct token in C, and then at test time, use a k nearest neighbor search for the start and the end of the phrase, separately. Consider Figure 2 as an example. We represent a query with two vectors, q start and q end . We then use each to retrieve the start and the end of the plausible phrases-in this case, c 1 and c 4 , which are the start and the end of Thessaloniki, respectively.
Nonparametric Masked Language Modeling
Method. Formally, let C = {c 1 , · · · , c N } be a reference corpus with N tokens. We first map each token c i into a contextualized, h-dimensional vector c i ∈ R h by feeding the text into the encoder and take the vector that corresponds to each token:
Nonparametric Masked Language Modeling
At inference time, NPM is given a query whose tth token is masked: q 1 ...q t−1 , [MASK], q t+1 ...q L . We replace [MASK] with two special tokens [MASK s ][MASK e ] and feed it into the encoder to obtain a list of h-dimensional vectors:
Nonparametric Masked Language Modeling
We then take the vector corresponding to [MASK s ] and [MASK e ] as q start and q end , respectively. 1
Nonparametric Masked Language Modeling
We then make a prediction via:
Nonparametric Masked Language Modeling
where V * is a set of possible n-grams defined by the vocabulary V and sim is a pre-defined similarity function that maps a pair of vectors into a scalar  value. In practice, iterating over N tokens is infeasible. We thus use an approximation using a fast nearest neighbor search for the start and the end separately. Details are provided in Appendix A.1.
Nonparametric Masked Language Modeling
Similarity function. The choice of similarity function can be flexible. We follow Zhong et al. (2022b) in using a scaled inner product
Nonparametric Masked Language Modeling
where h is a dimension of the token vectors.
Nonparametric Masked Language Modeling
NPM is trained on unlabeled text data. We describe the masking strategy first (Section 3.2.1), and then the training objective (Section 3.2.2).
Nonparametric Masked Language Modeling
We extend span masking (Joshi et al., 2020), which masks spans (consecutive tokens) whose length is sampled from a geometric distribution. Our span masking differs from Joshi et al. (2020) in two ways. First, we mask spans if they co-occur in the other sequences in the batch to guarantee inbatch positives during training (Section 3.2.2). For instance, masked spans in Figure 4  Obtaining vector representations. Consider the i-th sequence in the batch that consists of L tokens,
Nonparametric Masked Language Modeling
..x i L as a consequence of span masking over x i . Both x i andx i are fed into the encoder, and each token is mapped into an h-dimensional vector: 2
Nonparametric Masked Language Modeling
Training objective. We consider a masked span in x i , represented with [MASK s ][MASK e ], denoted asx i t ,x i t+1 . We then denote g i t as the original n-gram that were replaced byx i t ,x i t+1 . We now define the objective for this masked span, and the final objective is summed over all masked spans. The training objective for this masked span is defined as
Nonparametric Masked Language Modeling
.
Nonparametric Masked Language Modeling
Here, sim(·, ·) is a similarity function defined in Section 3.1, and
Nonparametric Masked Language Modeling
are start positives, start negatives, end positives and end negatives of g i t , respectively, which are defined in the next paragraph. This objective follows a phrase-level contrastive learning objectives in prior work (Lee et al., 2021;Ram et al., 2021;Deng et al., 2021;Kulkarni et al., 2022) with an extension that allows multiple positives.
Nonparametric Masked Language Modeling
In-batch positives and negatives. The start positives and the end positives are the start and the end of the spans to be retrieved. The start negatives and the end negatives are tokens that are not the start positives and not the end positives, respectively. More formally: Training. We use the model architecture and initial weights of RoBERTa large (Liu et al., 2019), consisting of 354M parameters. Training is done for 100,000 steps, using thirty-two 32GB GPUs. One batch consists of 512 sequences (131,072 tokens). We use an Adam optimizer (Kingma and Ba, 2014) with a learning rate of 3 × 10 −5 , weight decay of 0.01 and 4, 000 steps of warm-up.
Nonparametric Masked Language Modeling
Batching. The choice of batching is important in in-batch approximations, as it determines the quality of positives and negatives. For instance, Zhong et al. (2022b) uses BM25 to ensure the sequences in the same batch are likely to share the same topic.
Nonparametric Masked Language Modeling
With a pretraining corpus with billions of tokens, it can be significantly expensive to build a BM25 index. Therefore, we instead construct the batch by grouping sequences from the same document and assigning them to the same batch. 3 This trick ensures that (a) positives (spans that share the string) are likely to share the context, reducing false positives, and (b) negatives are those that the model is likely to be confused with, thus training against them helps the model better identify positives. During training, we gather all sequences from multiple GPUs to increase the size of the effective batch and make in-batch approximation more effective.
Nonparametric Masked Language Modeling
We perform zero-shot evaluation on closed-set tasks where a small set of candidates is given.
Nonparametric Masked Language Modeling
We include nine classification datasets that are known for not necessarily requiring factual knowledge: AGNews (Zhang et al., 2015), Yahoo (Zhang et al., 2015), 
Nonparametric Masked Language Modeling
We compare with the encoder-only, the decoderonly and the encoder-decoder models with various sizes (354M to 175B parameters). We include RoBERTa ( Shi et al. (2022) who use kNN inference using GPT-2 with PMI. In particular, (1) GPT-2 kNN uses kNN inference without training, and (2) GPT-2 kNN-LM interpolates distributions from GPT-2 and GPT-2 kNN.
Nonparametric Masked Language Modeling
We use the templates and verbalizers from Shi et al. (2022) for all models. When available, we use fuzzy verbalizers from Shi et al. (2022). We use a domain-specific reference corpus: a union of the English Wikipedia and CC News for AGN, Yahoo and RTE, a subjectivity corpus for Subj, and a review corpus for sentiment classification datasets. Their sizes vary from 15M tokens to 126M tokens. Details are in Appendix B. Fast similarity search is done using FAISS (Johnson et al., 2019) with the HNSW index. We use k = 4096 for inference.
Nonparametric Masked Language Modeling
NPM outperforms baselines in the zero-shot setting (Table 1). We discuss the results in detail below.
Nonparametric Masked Language Modeling
Comparison between baselines. Among parametric models, RoBERTa achieves the best performance, outperforming larger models including RoBERTa cheaper than an iPod. It was <mask>. cheap construction. It was <mask>.
Nonparametric Masked Language Modeling
cheaper than an iPod. It was <mask>. cheap construction. It was <mask>.
Nonparametric Masked Language Modeling
Positive Negative
Nonparametric Masked Language Modeling
Sim(cheap, <m>) Sim(cheap, <m>) Sim(cheap, cheap) Sim(<m>, <m>)
Nonparametric Masked Language Modeling
= 28.8 = 28.5 = 15.9 = 15.7
Nonparametric Masked Language Modeling
Retrieved context for <mask>: 10/10, would buy this cheap awesome gaming headset again.
Nonparametric Masked Language Modeling
Item delivered broken. Very cheaply made and does not even function. GPT-3. This is perhaps surprising, and is likely because bidirectionality of the encoder-only model plays a vital role, as claimed in Patel et al. (2022).
Nonparametric Masked Language Modeling
The kNN-LM approach from Shi et al. (2022), which incorporates the nonparametric component to the parametric model, outperforms all other baselines. Nonetheless, solely relying on retrieval (kNN) performs poorly with GPT-2, suggesting that using kNN at inference only is limited.
Nonparametric Masked Language Modeling
Baselines versus NPM. NPM significantly outperforms all baselines, achieving consistently competitive performance over all datasets. This indicates that, even for tasks that do not explicitly require external knowledge, nonparametric models are very competitive.
Nonparametric Masked Language Modeling
Qualitative analysis. Figure 5 depicts predictions from RoBERTa and NPM on a sentiment  Table 8 in Appendix C for the raw numbers.
Nonparametric Masked Language Modeling
analysis task. The first example uses cheap to indicate inexpensive, and the second example uses cheap to indicate of very poor quality. RoBERTa predicts Positive to both, while NPM makes correct predictions by retrieving the context that uses cheap in the same context as the input. We also find that representations from NPM lead to better word sense disambiguation. For instance, RoBERTa assigns a high similarity score between cheap (inexpensive) and cheap (of very poor quality). On the other hand, NPM successfully assigns a low similarity score between cheap and cheap, even though their surface forms are the same.
Nonparametric Masked Language Modeling
We include zero-shot evaluation on open-set tasks whose answer can be any arbitrary-length string.
Nonparametric Masked Language Modeling
We 
Nonparametric Masked Language Modeling
We compare with T5 (Raffel et al., 2020) as the encoder-decoder, and GPT-3 (Brown et al., 2020) and OPT (Zhang et al., 2022) as the decoder-only models. The encoder-only models are not applicable for open-set tasks since the number of tokens to predict is unknown.
Nonparametric Masked Language Modeling
Prior work found that a "retrieve-and-generate" approach that concatenates the input and passages from an off-the-shelf retrieval system is often helpful in knowledge-dependent tasks (Kandpal et al., 2022). We add them as baselines, using up to five passages from BM25 (Robertson et al., 2009).  Figure 7: Ablation on the size of the reference corpus, from 41M tokens (5%) to 810M tokens (100%). There is a strong correlation between the size of the corpus and downstream performance.
Nonparametric Masked Language Modeling
For all datasets, we report Exact Match (EM). The LAMA test data is biased toward frequent entities because they are filtered to only include answers that are single tokens based on BERT (Devlin et al., 2019). Since we do not want our evaluation to be biased toward overly frequent entities, we report a micro-averaged accuracy over the data whose answers are 1, 2, 3 and 4+ grams, respectively. Other datasets do not have such filtering, therefore we report average EM. As a reference corpus, we use the English Wikipedia from 08/01/2019, consisting of 810M tokens. For TempLAMA 22 19 , we use the English Wikipedia from 08/01/2022, consisting of 858M tokens.
Nonparametric Masked Language Modeling
For NPM, we find combining with sparse retrieval significantly helps, likely because dense retrieval and sparse retrieval capture complementary features (Karpukhin et al., 2020;Seo et al., 2019). In particular, we reduce the search space to the top 3 passages based on BM25 and perform dense search as done in Kassner and Schütze (2020). Figure 6 show results on five knowledge tasks.
Nonparametric Masked Language Modeling
First, performance of parametric models largely depends on the number of parameters, as it has been claimed in much of prior work (Brown et al., 2020;Kandpal et al., 2022). The retrieve-and-generate approach that combines parametric models with BM25 significantly improves performance.
Nonparametric Masked Language Modeling
NPM outperforms or is on par with significantly larger baselines across all datasets. It substantially outperforms all models on two LAMA datasets, including 500x larger GPT-3 either with or without BM25. On KML, TQA and NQ, NPM consistently outperforms 37x larger models with or  , on an unchanged set, a changed set, and a macro-average over two, respectively. xx→xx indicates performance when using the outdated and the updated Wikipedia, respectively. without BM25. This is impressive given that NPM is not trained on data with questions.
Nonparametric Masked Language Modeling
It is also worth noting that sparse retrieval is critical in NPM, e.g., without sparse retrieval, performance on LAMA-TREx drops from 34.5 to 16.1. We think this is because (1) sparse retrieval and dense retrieval capture complementary features, and (2) the removal of approximation in search improves search quality. We think future work can explore completely removing sparse retrieval, as has been done in Lee et al. (2021)  Impact of the reference corpus size. Figure 7 reports the impact of the size of the reference corpus, from 41M tokens (5%) to 810M tokens (100%). Performance of NPM is highly correlated with the size of the reference corpus, strongly suggesting that using a larger reference corpus is important.
Nonparametric Masked Language Modeling
Results on temporal knowledge tasks. Table 2 reports results on TempLAMA. NPM retains its performance on the unchanged set (18.9 →19.5) and successfully updates its answers on the changed set (2.9 → 17.5). Its performance is significantly better than the performance of parametric models with up to 13B parameters, and is on par with a larger model with the retrieve-and-generate approach, which also successfully updates its answer by leveraging the updated corpus. This is in agreement with prior work that shows the model with a nonparametric component adapts to temporal updates by replacing the reference corpus at test time (Izacard et al., 2022). Nonetheless, the retrieve-and-generate approach is still significantly worse than NPM when the target entities are rare, which we show in the next paragraph.   Table 3: Results on the entity translation task. See Table 10 in Appendix C for per-language results. #L indicates the number of languages multilingual models are trained on. Bold and Bold indicate the best among monolingual models and the best including multilingual models, respectively. NPM significantly outperforms all existing monolingual models, and approaches or outperforms larger multilingual models.
Nonparametric Masked Language Modeling
Performance on rare entities. We break down the instances on LAMA and TempLAMA based on the number of BPE splits of the target entity, e.g., Thessaloniki is one word that is split into 4 BPE tokens, thus the number of splits is 3. Since BPE splits a word if they are rare, the number of BPE splits indicates the rarity of the entity. We compare NPM with GPT-3 and BM25+GPT-3 on LAMA, and BM25+T5 (770M and 3B) on TempLAMA, the two most competitive baselines on each dataset. Figure 8 reports results. On LAMA, NPM outperforms GPT-3 fairly consistently, with larger gains as the number of BPE splits increases. On TempLAMA, while BM25+T5 is competitive on frequent entities with zero BPE split, it consistently lags behind NPM with ≥ 1 BPE splits. This suggests that NPM is particularly good at addressing rare entities, compared to not only parametric models without retrieval but also the retrieve-andgenerate approach.
Nonparametric Masked Language Modeling
Results in Entity Translation. Results on the entity translation task are shown in Table 3 (perlanguage results are reported in Table 10 of Appendix C). T5 and OPT struggle to perform the task, both with and without BM25 retrieval. In contrast, NPM performs well across all languages.
Nonparametric Masked Language Modeling
In order to better calibrate performance of NPM, we provide reference performance of models that are purposely trained on the multilingual data-mT5 (Xue et al., 2021) and BLOOM (Scao et al., 2022). NPM outperforms 3.4x larger mT5 and 20x larger BLOOM, and approaches 11x larger mT5, even though it is trained on English. We think strong cross-lingual transferability of NPM is likely because it can retrieve a phrase based on its surrounding context, even if it has not seen the exact word during training.
Nonparametric Masked Language Modeling
We introduced NPM, a nonparametric masked language model that replaces a softmax over the output vocabulary with a nonparametric distribution over a reference corpus. NPM can be efficiently trained using a contrastive objective and an in-batch approximation to a full corpus. Zero-shot evaluation on 16 tasks shows that NPM outperforms significantly larger parametric models. NPM is particularly good at rare patterns (word senses or facts), scaling and updating at test time, and predicting extremely rare if not unseen characters.
Nonparametric Masked Language Modeling
Scaling through the inference corpus. The size of the reference corpus is an additional dimension for model scale in nonparametric models. In this paper, we scale the corpus up to nearly 1B tokens, which is still smaller than the training data of very large language models (Brown et al., 2020;Rae et al., 2021). We think future work can scale it fur-ther using tools such as Distributed FAISS (Johnson et al., 2019) or ScaNN (Guo et al., 2020).
Nonparametric Masked Language Modeling
Significant memory usage. Using NPM saves GPU compute and memory compared to using models with more parameters. However, NPM requires more RAM and disk memory due to embeddings of a reference corpus. For instance, the largest corpus in our experiments (full English Wikipedia) requires 70GB of RAM and 1.4TB of disk memory. Future work can build more efficient NPM as done in prior work in nearest neighbor search (Jegou et al., 2010;Norouzi et al., 2012;Ge et al., 2014;Izacard et al., 2020;Yamada et al., 2021).
Nonparametric Masked Language Modeling
Exploration of larger vocabulary. Large vocabulary is known to lead performance gains (Conneau et al., 2020) but is bounded in memory costs. Previous work explored more efficient softmax approximations (Morin and Bengio, 2005;Chen et al., 2016;Grave et al., 2017). Our nonparametric training offers an alternative by removing the softmax over the vocabulary. With the RoBERTa architecture, increasing the vocab size by 2x makes the baseline training 50% more memory expensive, but does not increase the memory in training NPM. However, this paper does not include more systematic evaluation on the effect of large vocabulary. Future work can explore training NPM with a significantly larger vocabulary to further boost performance.
Nonparametric Masked Language Modeling
Extension for generation. Our paper evaluates NPM only on prediction tasks. It is currently nontrivial to use NPM for generation, since it is the encoder-only model. Future work can explore autoregressive generation as done in Patel et al. (2022) or use NPM for editing (Schick et al., 2022;Gao et al., 2022).
Nonparametric Masked Language Modeling
Extension to few-shot learning and fine-tuning. Our paper focuses on zero-shot evaluation only. Future work can extend NPM to a few-shot learning setup. In fact, fine-tuning NPM is significantly easier than fine-tuning larger models such as T5, OPT and GPT-3 which we compare NPM with, and can be explored in future work.
Nonparametric Masked Language Modeling
Better cross-lingual transfer. Our work explored cross-lingual transfer in a limited setup where the model is trained on monolingual data. We think future work can train multilingual NPM, and explore more comprehensive cross-lingual evaluation. In fact, nonparametric training may alleviate the burden of collecting large-scale multilingual  Table 4: Inference speed measured on Subj with |C| = 15M (the first block) and NQ with |C| = 810M (the second block). A single GPU used (Quadro GP100). ‡ indicates the corpus is used. 'FS' and 'SP' indicate that a FAISS index is used and a sparse index (+ exact inner product search in case of NPM) is used, respectively. NPM is slower than the same-sized parametric model, but is faster than larger models (either with or without retrieval) while outperforming or matching performance.
Nonparametric Masked Language Modeling
corpora since it makes the model less sensitive to the language coverage in the training data, and may lead to significantly better cross-lingual transfer, as we demonstrate in the entity translation task.
Nonparametric Masked Language Modeling
Limitation in speed. We find that search makes inference considerably slower than the counterpart without search. We think that (1) search can significantly be faster with better engineering (we use the default hyperparameters of the FAISS index with no tuning) or better index, and (2) the speed of NPM is still on par with the speed of significantly larger parametric models that NPM outperforms (see Table 4 
Nonparametric Masked Language Modeling
A.1 Details of NPM Approximation at inference. Given q start and q end , we take the top k tokens with the highest similarity scores with each of them, and compute scores over spans composed by these tokens. Let c * i:j be a span in C from the i-th token to the j-th token, and E(c) ∈ R h be a vector corresponding to a token c ∈ C. We find the top k tokens for the start and the end: c s 1 , c s 2 , · · · , c s k = argTopk c∈C sim(q start , E(c)), c e 1 , c e 2 , · · · , c e k = argTopk c∈C sim(q end , E(c)) using a fast nearest neighbor search. We then define a set of candidate phrasesC * as:
Nonparametric Masked Language Modeling
and predict:
Nonparametric Masked Language Modeling
where E(c * ) ∈ R 2h is a vector corresponding to c * , and V * is a set of any possible n-grams defined by the vocabulary V.
Nonparametric Masked Language Modeling
All implementation was done with PyTorch (Paszke et al., 2019), PyTorch Lightning 4 and Huggingface Transformers (Wolf et al., 2020).
Nonparametric Masked Language Modeling
Masking. We use a masking ratio of 15% for all models, following the standard in prior work (Devlin et al., 2019;Liu et al., 2019;Joshi et al., 2020). We implement masking as follows: (1) we first identify all possible candidate spans (spans that positives are found from other sequences in the batch), (2) sample the length of spans to mask from a geometric distribution with a hyperparameter p = 0.5, and (3) mask the spans with respect to the sampled length until the masking budget has been spent. We do not mask more than 128 spans from one sequence, and do not mask the span if the same span has been masked for more than ten times within the batch in order to prevent repeatedly masking overly frequent spans. 
Nonparametric Masked Language Modeling
Along with NPM, we introduce NPM SINGLE, which outputs a nonparametric distribution over every single token in C, instead of a phrase. To some extent, NPM is a strict generalization of NPM SIN-GLE, and NPM SINGLE still has a problem that existing encoder-only models have, e.g., can only fill in the [MASK] with a single token. We however think NPM SINGLE can be useful for some applications, e.g., for fine-tuning, as existing encoder-only models are used for.
Nonparametric Masked Language Modeling
Inference. Given a reference corpus C = {c 1 , · · · , c N }, we construct N number of hdimensional vectors c 1 , · · · , c N ∈ R h by feeding the text into the encoder. At inference time, given a query whose t-th token is [MASK], we feed it into the encoder:
Nonparametric Masked Language Modeling
We take q t as a vector that represents the [MASK] token in the query. Finally, the prediction is made by aggregating the similarity scores to the tokens in C:
Nonparametric Masked Language Modeling
where E(c) ∈ R h is a vector corresponding to c, and V is the vocabulary set. In practice, since computing scores over all tokens in C is infeasible, an approximation is made by computing scores for the top k nearest neighbors only, and treating other tokens to have a similarity score of −Inf. More precisely: c 1 , c 2 , · · · , c k = argTopk c∈C sim(q t , E(c)) are obtained by using an index (e.g., FAISS (Johnson et al., 2019)), and the following is returned as a prediction:
Nonparametric Masked Language Modeling
Training. Let x i 1 ...x i L be the i-th sequence in the batch, whose subset is replaced with [MASK] and converted tox i 1 ...x i L . Both the unmasked sequence and the masked sequence are fed into the encoder, and each token is mapped into an h-dimensional vector:
Nonparametric Masked Language Modeling
The training objective is then defined as:
Nonparametric Masked Language Modeling
.
Nonparametric Masked Language Modeling
Here, sim(·, ·) is a similarity function defined in Section 3.1, and Y + (x i t ) and Y − (x i t ) are positives and negatives of x i t -tokens from other sequences in the batch that share and do not the vocab, respectively.
Nonparametric Masked Language Modeling
When applying NPM and NPM SINGLE on closedsetk tasks, we closely follow Shi et al. (2022) who adapts kNN-LM for zero-shot inference on classification tasks. We assume a fuzzy verbalizer: f : Y →Ṽ, where Y is a set of labels in the task andṼ ∈ V is a subset of the vocabulary V. The fuzzy verbalizer maps a label to a set of tokens that express the label, e.g., in a sentiment classification task, f (Positive) includes awesome or great, and f (Negative) includes terrible or broken.
Nonparametric Masked Language Modeling
NPM SINGLE is given a query vector q ∈ R h and predicts:
Nonparametric Masked Language Modeling
where E(c) ∈ R h is a vector corresponding to c, and τ is a hyperparameter.
Nonparametric Masked Language Modeling
NPM is given a query vector q ∈ R 2h and predicts:
Nonparametric Masked Language Modeling
where E(c * ) ∈ R 2h is a vector corresponding to c * . Note that this is essentially equivalent to argmax y∈Y c∈C
Nonparametric Masked Language Modeling
We use τ = 5.0 for both NPM SINGLE and NPM. Table 5 reports statistics and templates on each downstream task, and Table 6 reports statistics of the retrieval corpus used in experiments.
Nonparametric Masked Language Modeling
For closed-set tasks, we use templates and verbalizers provided by Shi et al. (2022) for most datasets, except two datasets. For RTE, we use the template from Artetxe et al. (2022). For Subj, we write our own template, motivated by Zhong et al. (2022a) that found Subj is mainly about differentiating a review and a summary. For open-set tasks, we use templates provided by the original authors, except NQ and TQA for which we use the templates from GPT-3 (Brown et al., 2020). Due to limited computation resource, we subsample the data to include up to 3,000 examples, following the standard from prior work (Zhao et al., 2021;Shi et al., 2022). For closed-set tasks, we use exactly the same set of data as Shi et al. (2022), and for open-set tasks, we use the same script to subsample the data. For LAMA T-REx and Google RE, we subsample up to 1,000 examples for each of 1, 2, 3 and 4+ grams. For the entity translation task, we subsample up to 1,000 examples per language.
Nonparametric Masked Language Modeling
The following is a more detailed description of open-set tasks used in Section 6.    and Apple) are excluded. We also consider the hard subset of T-Rex from Zhong et al. (2021). Note that Petroni et al. (2019) only include triples whose object is one token based on BERT (Devlin et al., 2019); however, with a different pretrained model like RoBERTa, entities could be multiple BPE tokens. Entities that are splitted into multiple BPE tokens are more rare entities. KAMEL (Kalo and Fichtel, 2022) is another factual probing task as LAMA but with a few key differences to make it more general and broad: (1) it includes a broader coverage of triples, (2) it removes the constraint that the object is one token based on BERT, (3) it includes objects with literal values, and (4) it has a question answering format. TempLAMA 22
Nonparametric Masked Language Modeling
19 is a task that requires probing knowledge with temporal updates. The task is first introduced by Dhingra et al. (2022) and Jang et al. (2022); however, we could not use either of existing data as their time split do not match our training. We therefore create the data by using a script provided by Dhingra et al. (2022) but using the 2019 and the 2022 dumps. We take Wikipedia triples whose relations are available for a template from either Petroni et al. (2019) or Dhingra et al. (2022). We then include triples whose object entities differ between the 2019 dump and the 2022 dump (due to the entity being updated), or only appear in the 2022 dump (due to the subject or the relation being added) to the changed set. Otherwise, triples are included in the unchanged set. We additionally find that many triples are overly difficult because the fact is extremely niche and not really known. We thus filter the data to only include facts that appear in Wikipedia. Specifically, we include triples if the subject has a corresponding Wikipedia page and the object entity appears in that Wikipedia page.
Nonparametric Masked Language Modeling
Entity translation requires translating an entity from English to other languages that are not Latin based. While this is mainly to evaluate if the model can generate rare or unseen characters that are not in English, the entity translation task itself is a vital and challenging task in real applications such as machine translation (Babych and Hartley, 2003;Yan et al., 2018) and cross-lingual question answering (Clark et al., 2020;Asai et al., 2021). It is often beyond a series of simple translations of each word, or spelling out its pronunciation (Moore, 2003;Hassan et al., 2007;Sun et al., 2017). For instance, the Korean translation of Banpo Bridge in Figure 1 (ᄇ ᅡ ᆫ ᄑ ᅩᄃ ᅢᄀ ᅭ) is not the concatenation of the translations of Banpo and Bridge (ᄇ ᅡ ᆫᄑ ᅩ ᄃ ᅡᄅ ᅵ).
Nonparametric Masked Language Modeling
We first identify a list of 15 non-Latin languages: Arabic (ar), Czech (cs), Greek (el), Hindi (hi), Hebrew (iw), Japanese (jp), Korean (ko), Malayalam (ml), Mongolian (mn), Polish (pl), Russian (ru), Tamil (ta), Thai (th), Turkish (tr), and Chinese (zh). We then implement heuristics to identify entities and their translations from English Wikipedia. Specifically, we parse the first paragraph of each Wikipedia article and pair the found translation with a topic en-tity of the article. For instance, a Korean translation of Banpo Bridge is found from the first sentence of https://en.wikipedia.org/ wiki/Banpo_Bridge. Per-language statistics are reported in Table 7.
Nonparametric Masked Language Modeling
Full results on knowledge tasks. Table 8 reports full results on five knowledge tasks. See Figure 6 for an illustration, and Section 6.4 for discussion.
Nonparametric Masked Language Modeling
Comparison to few-shot GPT-3. Table 9 compares zero-shot NPM SINGLE and NPM with zeroand four-shot GPT-3. Our zero-shot models outperform 500x larger zero-shot GPT-3 and 7.6x larger 4-shot GPT-3, but lag behind 4-shot GPT-3 that is 19x or larger. We think future work can explore extending our models to a few-shot setup.
Nonparametric Masked Language Modeling
Additional qualitative results. Figure 9 depicts predictions from RoBERTa and NPM in topic classification, choosing a label between four candidates: health, computer, travel and politics. All three examples contain the word torch, but with different meanings, e.g., an infectious diseases, a tool, and a computer library. RoBERTa predicts health for all of them, while NPM predicts health, travel and computer, which are all correct predictions.
Nonparametric Masked Language Modeling
As in Figure 5, we find that representations from NPM enable better word sense disambiguation: the pairwise similarities between between different meanings of torch are significantly lower than the pairwise similarities between other tokens that share the meaning.
Nonparametric Masked Language Modeling
Entity translation given an oracle passage. We evaluate models on the entity translation task where an oracle passage-a passage that is guaranteed to contain the translation information-is provided to the model. Baselines prepend oracle passages to the input, as it does with the retrieve-and-generate approach. NPM uses oracle passages to restrict the search space. Table 11 reports results. While performance overall increases compared to when the oracle passage is not provided, the overall comparison between models does not change from Table 10: (1) all monolingual models significantly suffer, except for a couple of languages that are derived from Latin; (2) NPM significantly outperforms all monolingual models; (3) NPM even outperforms 3.4x larger mT5 and 20x larger BLOOM, and approaches 11x larger mT5.  Table 8: Results on open-set tasks (numbers used in Figure 6). # Params indicates the relative number of model parameters compared to RoBERTa large (354M), and C indicates whether a text corpus is used. For LAMA (T-REx and Google RE), the macro-averaged EM over 1, 2, 3 and 4+ grams are reported. All models are zero-shot. NPM significantly outperforms larger parameters models, either with and without a retrieval-and-generate approach that uses BM25.  Table 9: Comparison to GPT-3 on AG News and SST-2. # Params indicates the relative number of model parameters compared to RoBERTa large (354M). All GPT-3 numbers are taken from previous work. k-shot indicates that the model performs in-context learning with k labeled examples with no gradient updates. We report on SST-2 and AGN, because they are all datasets shared between our paper and previous papers that report GPT-3 results (Zhao et al., 2021;Holtzman et al., 2021). Our zero-shot models outperform 500x larger zero-shot GPT-3 and 7.6x larger 4-shot GPT-3, but lag behind 4-shot GPT-3 that is 19x or larger. Figure 9: Predictions from RoBERTa (baseline) and NPM on a topic classification task (classes={health, computer, travel, politics}). The bottom indicates the context NPM retrieves to fill in [MASK]. On the right, we indicate the token-wise similarity scores. NPM assigns significantly lower scores to the token pairs with distinct meanings than to the token pairs with the similar meaning, e.g., torch (a disease) and torch (a tool).  Table 10: Results on the entity translation task. #L indicates the number of languages multilingual models are trained on. Bold and Bold indicate the best among monolingual models and the best including multilingual models, respectively. NPM significantly outperforms all existing monolingual models, and approaches or outperforms larger multilingual models.  Table 11: Results on the entity translation task given an oracle passage. #L indicates the number of languages multilingual models are trained on. Bold and Bold indicate the best excluding multilingual models and the best including multilingual models, respectively.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Knowledge-intensive tasks, such as open-domain question answering (QA) and fact checking, require access to a large amount of world or domain knowledge (Petroni et al., 2021). These tasks are even challenging for humans without access to an external knowledge source such as Wikipedia. A common thread of existing methods for knowledge-intensive tasks employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from Wikipedia and then conditions the prediction of the answer on these documents along with the question (Karpukhin et al., 2020;Lewis et al., 2020;Izacard & Grave, 2021). Nevertheless, these methods mainly suffer from three drawbacks. First, candidate documents for retrieval are chunked (e.g., 100 words) and fixed, so the retrieved documents might contain noisy information that is irrelevant to the question. Second, the representations of questions and documents are typically obtained independently in modern two-tower dense retrieval models (Karpukhin et al., 2020), leading to only shallow interactions captured between them (Khattab et al., 2021). Third, document retrieval over a large corpus requires the retriever model to first encode all candidate documents and store representations for each document. These two operations limit the parameters of dense retrievers and the size of embedding vectors, and thus cannot enjoy the world knowledge or deduction capabilities of large language models (Levine et al., 2022).
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In this paper, we propose to leverage large language models, such as InstructGPT (Ouyang et al., 2022), to directly generate contextual documents for a given question, instead of retrieving relevant documents from an external corpus, such as Wikipedia. Our approach has two main advantages. First, we show that generated contextual documents contain the correct answer more often than the top retrieved documents. We believe this is because large language models generate contextual documents by performing deep token-level cross-attention between all the question and document contents, resulting in generated documents that are more specific to the question than retrieved documents. Second, we show that our approach significantly outperforms directly generating answers from large language models despite not incorporating any new external information. This is mainly because the task of generating document-level contexts is close to the objective of causal language modeling pre-training, so the world knowledge stored in the model parameters can be better utilized.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We show, on multiple datasets, that generated documents are more likely to contain correct answers than the top retrieved documents. Notably, in dense retrieval methods, as more documents are retrieved, the recall of documents containing the correct answer increases (Karpukhin et al., 2020). However, the recall performance does not scale as well with generated documents because even with sampling methods, generated documents tend to contain duplicate information. In order to improve the recall performance of generated documents, we propose a novel clustering-based prompt method. We synthesize a prompt with in-context demonstrations of question-document pairs sampled from diverse clusters. These prompts result in generated documents that cover different perspectives of the question and improve the scaling of performance as more documents are generated per question.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In contrast to the retrieve-then-read pipeline, our method is essentially a generate-then-read pipeline. Specifically, it first prompts a large language model to generate contextual documents based on a given question, and then reads the generated document to produce the final answer. The reader can still be a large model (e.g., InstructGPT (Ouyang et al., 2022)) used under a zero-shot setting, or a small one (e.g., FiD (Izacard & Grave, 2021)) fine-tuned with generated documents on the training split of the target dataset. We evaluate our proposed method on three different knowledge-intensive tasks and demonstrate its effectiveness on both zero-shot and supervised settings.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Overall, our main contributions can be summarized as follows:
Generate rather than Retrieve: Large Language Models are Strong Context Generators
1. We propose a novel generate-then-read pipeline for solving knowledge-intensive tasks, i.e., replacing the process of retrieving documents from Wikipedia or searching for related documents on Google, by prompting a large language model to generate relevant contextual documents.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
2. We propose a novel clustering-based prompting approach to generate multiple diverse contextual documents that increases the likelihood of covering the correct answer. We demonstrate this approach can significantly improve performance on end QA and other downstream tasks.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
3. We conduct extensive experiments with three knowledge-intensive NLP tasks under both zeroshot and supervised settings. Notably, our method can match or even outperform retrieve-then-read pipeline methods, without retrieving any documents from any external knowledge source. Mainstream methods for solving knowledge-intensive NLP tasks employ a retrieve-then-read model pipeline. Given a question, this model first leverages a retriever over a large evidence corpus (e.g. Wikipedia) to fetch a set of relevant documents that may contain the answer. A reader is then used to peruse the retrieved documents and predict an answer. Recent follow-up work has mainly focused on improving the retriever (Karpukhin et al., 2020;Qu et al., 2021;Sachan et al., 2022) or the reader (Izacard & Grave, 2021;Cheng et al., 2021;Yu et al., 2022), or training the system end-toend (Lewis et al., 2020;Singh et al., 2021). Early retrieval methods mainly employed sparse retrievers, such as BM25 (Chen et al., 2017). Recently, ORQA (Lee et al., 2019) and DPR (Karpukhin et al., 2020) have revolutionized the field by utilizing dense contextualized vectors for document indexing, leading to superior performance to traditional approaches. We propose an alternative approach which forgoes retrieval, instead extracting the knowledge from the model parameters of a large language model. We show that our approach is can be combine with dense retrievers to outperform both methods independently. Our method can also be combined with any reader mechanism, allowing generated context documents to be plugged into any current knowledge-intensive NLP pipelines.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Recent works have investigated using auto-regressive language models to generate identifier strings for documents, as an intermediate target for retrievals, such as entity names (De Cao et al., 2020) or distinctive n-grams that can be mapped to full passages (Bevilacqua et al., 2022). However, one needs to create the identifiers, hence the structure was not thoroughly evaluated on a large-scale benchmark (Bevilacqua et al., 2022). Other works have demonstrated that the knowledge stored in the parameters of pre-trained language models could be "retrieved" to some extent by directly generating text (Petroni et al., 2019;. However, the previous work only used generation for query expansion (Mao et al., 2021), which did not exploit the potential of directly generating contextual documents for open-domain questions. Different from the above approaches that aimed to train a generator model to produce contextual document identifiers (which is still using the original Wikipedia text) or provide data augmentation to retrievers, our work directly generates contextual documents for given questions.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
A line of recent work has shown that relevant knowledge can be elicited from large language models, especially for those domains that lack appropriate knowledge bases with sufficient coverage (Liu et al., 2022b;Fang et al., 2022). For example, Liu et al. (2022b) proposed leveraging GPT-3 to generate relevant contexts, then providing the contexts as additional input when answering a commonsense question. Another line of work focused on prompting a large language model to generate a series of intermediate reasoning steps, often referred to as chain-of-thought (Wei et al., 2022b;Kojima et al., 2022;. The prompt consists of an instruction (e.g., Let's think step by step!), a few demonstrations that are fixed for each task, and a new-question placeholder. The demonstrations are human-written, and each consists of a question in the style of the task and a series of intermediate reasoning steps that is helpful for answering the question. Our work does not require any human annotation, but adds to this line of work of leveraging model generated text to guide further generations. In our case, we apply this approach to knowledge-intensive tasks, which have not been explored by previous work.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In this section, we present details of our proposed novel generate-then-read (GENREAD) pipeline for solving various knowledge-intensive tasks. Specifically, it first prompts a large language model to generate contextual documents with respect to a given query, then reads the generated documents to predict the final answer. The reader can either be a large model (e.g., InstructGPT) used for the zero-shot setting, or a small one (e.g., FiD) fine-tuned with generated documents on the training split of the target dataset. We introduce the zero-shot setting in §3.1 and supervised setting in §3.2.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Under the zero-shot setting, there is no training data -neither questions nor contextual documents. When tested on the open-domain QA task, most existing large language models directly encode the given question and predict the answer (Brown et al., 2020;Du et al., 2022;Chowdhery et al., 2022). Specifically, the question q, associated with some text prompt, is input to the model, which then generates the answer, denoted as p(a|q, θ), where θ represents the pre-trained model parameters. In practice, the maximum a posteriori estimation (MAP) is the final answer, i.e.,â = arg max a p(a|q, θ). However, this way of directly asking large language models to output answers often leads to poor performance, as it leaves a considerable amount of additional world knowledge unexploited (Levine et al., 2022). On the contrary, the zero-shot retrieve-then-read pipeline first uses an off-the-shelf retriever to fetch relevant documents from an external knowledge source such as Wikipedia, then asks the large language model to read the documents and predict the answer.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In this work, we improve the performance by introducing an additional auxiliary generated document variable d, and then extend the model to have the form p(a|q) = i p(a|d i , q)p(d i |q). In practice, we cannot sum over all possible documents d. Therefore, the most common approach is to compute What does Monsanto own? (WebQ)
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Step 1: Get one document for each question via retrieval or generation.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Step 2: Get embeddings, and cluster them by K-means.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Monsanto is a multinational agrochemical and agricultural biotechnology corporation … It is one of the world's leading producers of roundup, a glyphosate herbicide. (63 words)
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Step 3: Given question for training or inference, for each cluster ∈ 1 … :
Generate rather than Retrieve: Large Language Models are Strong Context Generators
• sample !# , !# , = 1 … , whose cluster id is ;
Generate rather than Retrieve: Large Language Models are Strong Context Generators
• create prompt ! = " !" ; !" ; … ; … ; !$ ; !$ ";
Generate rather than Retrieve: Large Language Models are Strong Context Generators
• generate document ! with ! using a large language model. Using a reader (e.g., FiD), with and the diverse documents { " , % , … , & }, find answers .
Generate rather than Retrieve: Large Language Models are Strong Context Generators
• agricultural chemicals • seed (also correct)
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Monsanto Company is an American multinational agrochemical and agricultural biotechnology corporation … It is a leading producer of genetically engineered seed and … (70 words)
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Monsanto is a multinational agricultural biotechnology corporation. … The company also manufactures other agricultural chemicals, such as insecticides … (36 words) Figure 1: An overall framework of clustering-based prompting method. It leverages distinct questiondocument pairs sampled from each embedding cluster as in-context demonstrations to prompt a large language model to generate diverse documents, then read the documents to predict an answer.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
the MAP estimated = arg maxp(d) using beam search, and then to approximate the sum over d with this single value. This two step approach, we label it as a generate-then-read pipeline.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
STEP1: GENERATE. In this step, we first prompt a large language model (e.g., InstructGPT (Ouyang et al., 2022)) to generate documents based on the given question. For example, the input to the language model could be "Generate a background document to answer the given question. {question placeholder}". We can use any decoding strategy (e.g., greedy decoding, beam search), but we used greedy decoding throughout the zero-shot experiments for simplicity and reproducibility.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
STEP 2: READ. In the second step, we use generated sentenced along with the input question to produce the final answer from the large language model. This is actually the same setting as "zeroshot" reading comprehension, as widely studied in existing works (Brown et al., 2020;Lazaridou et al., 2022). We choose appropriate prompts from P3 (Bach et al., 2022), such as "Refer to the passage below and answer the following question. Passage: {background placeholder} Question: {question placeholder}". Finally, the language model is fed the prompted text to generate an answer.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Although large language models demonstrate impressive performance on zero-shot learning abilities, their performance still lag behind the supervised setting. Therefore, we also explore how the generated documents from large language models can benefit the supervised setting. As directly fine-tuning large language models on downstream datasets could be prohibitively expensive, we leverage a small reader model such as FiD to peruse the generated documents under the supervised setting.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Under the supervised setting, scaling the size of retrieved documents can lead to better performance (Karpukhin et al., 2020;Izacard & Grave, 2021). This is mainly because retrieving more documents can cover more relevant information and knowledge, i.e., a higher recall score. Nevertheless, asking a large language model to generate multiple high-quality contextual documents is a challenging task. Dense retrieval methods can fetch multiple documents covering different perspectives of the answer. Compared to dense retrievers, simply prompting a large language model to generate multiple contextual documents often leads to low knowledge coverage, since the contents generated by multiple decoding passes from the same input tend to be similar. Sampling decoding methods, such as nucleus sampling 1 (Holtzman et al., 2020) can diversify the generation process to some extent, but the knowledge content of generated texts still tends to be highly repetitive when used to generate documents for a given question. We further propose two novel solutions, including diverse human prompts and clustering-based prompts, which will be elaborated on in this section.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In order to avoid similar token distributions under a single prompt, we ask human annotators to provide different prompts, in order to make the generated document diverse. This method is simple, but can effectively vary the token distribution during generation. In the experiments, we empirically found this method can bring improvement to the retrieval performance ( Figure 2). However, this method suffers from two drawbacks. On one hand, it requires human annotators to write different prompts, which cannot be easily generalized to different knowledge-intensive tasks. On the other hand, different large language models might be sensitive to different prompt words, which might cause a set of good prompt words not work on a different large language model.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
To increase knowledge coverage in generated documents, we propose a novel clustering-based prompt method. It first clusters the representations of a set of documents into K classes (K = 2 in Figure  1), where the number of classes is equal to the number of documents that need to be generated in the end. Next, it randomly selects n question-document pairs (n = 5 in Figure 1) from each cluster. Lastly, a large language model presents the different n question-document pairs as in-context demonstrations for generating documents to a given question. In this way, large language models are based on different distributions of examples, hence resulting in generated documents covering different perspectives. We show this in Figure 1 and illustrate the details of each step as follows.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
STEP 1: GET ONE INITIAL DOCUMENT PER QUESTION. Similar to the zero-shot setting, we first ask a large language model to generate one contextual document d for each question q ∈ Q, where Q is the set of questions in the training split. Alternatively, we can use an unsupervised retriever (e.g., BM25) to obtain a document from Wikipedia. We now have a question-document pair set
Generate rather than Retrieve: Large Language Models are Strong Context Generators
. STEP 2: ENCODE EACH DOCUMENT, DO K-MEANS CLUSTERING. We then use a large language model (i.e., GPT-3) to encode each question-document pair, i.e., e i = GPT-3([q i , d i ]), resulting in a 12,288-dimensional vector per document. Then, we use K-means to cluster all embedding vectors {e i } |Q| i=1 into K sets, so each question-document pair is assigned a unique cluster id c ∈ {1, ..., K}. We vary the number of K in the experiments, which will be illustrated in Figure 2.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
STEP 3: SAMPLE AND GENERATE K DOCUMENTS. Lastly, we sample n question-document pairs from each cluster c, denoted as {q c1 , d c1 ; q c2 , d c2 ; ...; q cn , d cn }, in which n is a hyperparameter 2 . Then, the n sampled question-document pairs from the same cluster serve as in-context demonstrations for the large language model to generate a contextual document. For example, the input to the large language model could be "{q c1 placeholder} {d c1 placeholder} ... {q cn placeholder} {d cn placeholder} {input question placeholder}". By enumerating the sampled documents in these K clusters, we can finally get K-generated documents. By conditioning on different sampled in-context demonstrations collected from different clusters, the large language model has been biased for different perspectives. Although these different perspectives exist in a latent manner, we empirically show it works well in practice, by comparing it with sampling methods, diverse human prompts ( Figure 2 and Table 2) and randomly sampling n pairs from the entire dataset (Table 11) WebQ (Berant et al., 2013)), fact checking (FEVER (Thorne et al., 2018) andFM2 (Eisenschlos et al., 2021)) and open-domain dialogue system (WoW (Dinan et al., 2019)). More detailed dataset information can be found in Appendix A.1. To evaluate the model performance, we use exact match (EM) score for evaluating open-domain QA (Zhu et al., 2021). An answer is considered correct if and only if its normalized form has a match in the acceptable answer list. We also employ Recall@K (R@K) as an intermediate evaluation metric, measured as the percentage of top-K retrieved or generated documents that contain the answer. This metric is commonly used in evaluations of previous works (Karpukhin et al., 2020;Izacard & Grave, 2020;Sachan et al., 2022). For other knowledge-intensive tasks, we follow the KILT benchmark (Petroni et al., 2021) to use accuracy (ACC) for fact checking and F1 / Rouge-L (R-L) score for open-domain dialogue system.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Open  (Wei et al., 2021)) under this setting without using any external document. Our GENREAD can achieve comparable or even better performance than zero-shot retrieve-then-read models that use a retriever or search engine to first obtain contextual documents.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
To ensure reproducibility, we use greedy search in decoding. All prompts used are shown in the §B.1. Note: fix numbers in v2 by adding average performance of different prompts, see details in Table 20.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We first compare our proposed GENREAD approach with various large language models proposed in recent years, including GPT-3 ( (Chowdhery et al., 2022) and InstructGPT (Ouyang et al., 2022). Due to the space limitation, we only put the best performance on each dataset in Table 1, in which the line is called previous SoTA methods. In addition, their corresponding model parameters and performance are listed in Table 9 in Appendix. All of these baseline methods use the same input formats, i.e., [prompt words; question].
Generate rather than Retrieve: Large Language Models are Strong Context Generators
GENREAD is based on InstructGPT with 175B parameters. In order to fully evaluate the effectiveness of our proposed method, we also compare with InstructGPT augmented with retrieved documents from Wikipedia or Google search. The baseline methods (1) BM25 / Contriever + In-structGPT;
Generate rather than Retrieve: Large Language Models are Strong Context Generators
(2) Google + InstructGPT; (3) DPR + InstructGPT have the same input format as our GENREAD , i.e., [prompt words; contextual document; question]. BM25 is a traditional sparse retrieval method. Contriever (Izacard et al., 2022a) is a state-of-the-art unsupervised dense retrieval model. DPR (Karpukhin et al., 2020) is a supervised dense retrieval model directly trained on NQ, TriviaQA and WebQ datasets. We note that comparing with above three methods is challenging because our method only relies on the large language model itself, without using any external corpus.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In the experiments, we use InstructGPT as our backbone model. As shown in Table 1, compared with state-of-the-art large language models, our proposed GENREAD with the InstructGPT reader improves its performance by generating contextual documents and conditioning on the generated documents, even though no new data is introduced, and the generator and reader have the exact same parameters. Specifically, GENREAD can improve the EM score by +6.9 on three open-domain QA benchmarks, compared to the original InstructGPT. We also make a similar observation on fact checking and open-domain dialogue system. Our proposed GENREAD can consistently outperform the baseline InstructGPT model without retrieving any contextual documents.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
To further validate the effectiveness of GENREAD , we compare against zero-shot retrieve-then-read pipeline models, which first use a retrieval model or the Google search engine to get a relevant contextual document, then use InstructGPT to read the texts and produce the final answer. As shown in Table 1, GENREAD can achieve on-par performance with zero-shot retrieve-then-read pipeline models on the NQ and FM2 datasets, and outperform them on all other benchmarks. The knowledge learned by the large language models can be retrieved via autoregressive text generation. Without   Figure 2: Recall@K on test sets, measured as the percentage of top-K documents that contain the answer. Our proposed clustering-based prompting method can outperform DPR and Google search, also two variants of using LLMs to generate documents. Exact numbers are reported in Table 6.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
seeing any examples from these datasets, GENREAD can outperform using the supervised retrieval model (i.e., DPR) to recover relevant contextual documents.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We compare our proposed GENREAD with retrieve-then-read models, including DPR (Karpukhin et al., 2020), RAG (Lewis et al., 2020), and FiD (Izacard & Grave, 2021). In addition, we compared with obtaining relevant documents from the internet using the Google search engine.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
For our proposed method, we replace the retriever with a large language model to directly generate contextual documents. In the experiments, we use InstructGPT (Ouyang et al., 2022). After contextual documents are retrieved or generated, we employ a FiD reader with 770M parameter models (i.e., FiD-l) and 3B parameter models (i.e., FiD-xl) that are fine-tuned on the training split of target datasets. We note that we only use 10 documents during reading for the following reasons.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Why do we choose to use only 10 documents instead of 100 when reading?
Generate rather than Retrieve: Large Language Models are Strong Context Generators
As noted in Section 6.2 in DPR (Karpukhin et al., 2020) and Figure 3 in FiD (Izacard & Grave, 2021), increasing the number of documents can lead to better model performance and achieve state-of-the-art when using 100 documents. However, there are two major drawbacks to using 100 documents during the reading step. First, the operation is very expensive, leading to a significant increase in memory consumption and training time. As reported by Izacard & Grave (2021), the training process requires 64 Tesla V100 32GB running for around one day. Second, generating documents by using a large language model is slow and expensive, so only using 10 documents can be a significant cost saving in our method. Therefore, in our experiments, we choose to use 10 documents during the reading process. When using FiD-770M (i.e., FiD-large), the training process can be easily performed even on a single Tesla V100 32GB GPU. Meanwhile, when only using 10 documents, we can also increase the size of FiD model from 770M to 3B, which takes about the same amount of GPU memory as using 100 documents on a 770M model, but at the same time significantly shortens the training time.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We note that training T5-3B model needs a bigger cluster such as 8 Tesla V100 or A100 GPUs.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We first use Recall@K to compare the retrieval accuracy of different models. As shown in Figure 2, GENREAD can significantly outperform DPR and Google search for under 10 retrieved or generated documents. Compared to different GENREAD variants, including nucleus sampling, human written prompts, and clustering-based prompts, clustering-based prompts achieve the best performance. At the same time, we notice that the language model inevitably has the problem that the slope of the curve decreases as the number of generated documents increases. On one hand, this is due to the similarity of token distributions when large language models generate multiple documents. On the other hand, due to the shallow interaction characteristics of the dense retrieval model itself, the retrieved documents might not be completely relevant to the given question, so that the increase in recall might come from false positive documents, as also mentioned by Sachan et al. (2022).  Table 2: Supervised open-domain QA performance. By only using generated documents from In-structGPT, our GENREAD with FiD reader (named GENREAD (FiD)) can achieve better performance than baseline methods on TriviaQA and WebQ. Through our detailed analysis of NQ, we found the performance gap mainly due to the temporality issue, which will be elaborated in §A.7.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
As shown in Table 2, we can first observe the FiD model performs the best among all baseline models. Using FiD-xl with only 10 documents achieves comparable performance with using FiD-l with 100 documents. The average gap is less than 1% on three benchmarks. Compared with both close-book models and Wikipedia-based retrieve-then-read pipelines, our proposed GENREAD can achieve state-of-the-art performance. Furthermore, compared with using sampling methods to generate documents, the clustering-based prompt method can improve the EM score by +2.2 on average. This indicates that the clustering-based prompt method is effectively increasing the knowledge coverage of generated documents, and also leading to better downstream QA performance. We also show that GENREAD can outperform Google search on all benchmarks. We observe both our method and Google search perform worse than DPR, mainly due to the significant portion of time-dependent questions in the dataset, which is described in the following analysis.  We demonstrate the experimental results in Table 3. Under the supervised setting, GENREAD can achieve on par performance on the fact checking task and superior performance on the dialogue system task, indicating that large language model can be seen as a strong knowledge generator.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
The main reason that GENREAD performs worse than the dense retriever for fact checking is that the task provides sufficient semantic information to reach strong performance on this binary decision task. So, there is a smaller semantic gap between the given factual statement and contextual documents than that of question and document pairs in open-domain QA, which is an easier retrieval setting for modern dense retrieval methods that are mainly based on vector similarity.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Generated documents can be combined with retrieved documents to outperform both. Even with a very large number of retrieved documents, including few samples of generated knowledge leads to large improvements. As shown in Table 2, merging retrieved documents with generated documents can  Figure 3: Combining DPR retrieved documents and large language model (LLM) generated documents can achieve significantly better performance than using DPR retrieved documents only. For a fair comparison, instead of adding LLM generated documents to the model, we replace 10 documents retrieved by DPR with 10 documents generated by LLM so the total number of documents is the same.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In this experiment, we use FiD-l (i.e., FiD-large) as the reader model because when the documents scale to more than 20, FiD-xl (i.e., FiD-3B) causes out-of-memory issues on A100 GPUs.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
achieve state-of-the-art performance compared to all baseline methods listed in the 
Generate rather than Retrieve: Large Language Models are Strong Context Generators
The improvement in open-domain QA performance is due to the fact that correct answers are included more frequently in the generated text Recall@K is the most commonly used metric in existing works to measure the retrieval performance, which computes the percentage of top-K retrieved or generated documents that contain any possible answer at least once. than in the retrieved documents. However, as many questions contain multiple correct answers, recall@K cannot fully reflect the diversity of generated or retrieved documents. Each question in the WebQ has 2.39 correct answers, 1.79 correct answers in NQ and 14.02 (including all entity alias) in the TriviaQA. NQ and WebQ do not include alias names in the labels.   After we manually compare some retrieved documents from DPR and generated documents from InstructGPT, we observe that the readability of different documents, when they contain the correct answer string, is different. In other words, documents containing answers might also contain noisy information that is irrelevant to the question, which could affect both the model and human reading.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
In order to further validate the readability of retrieved documents and generated documents, we extracted a subset of data examples from NQ, TriviaQA and WebQ datasets, in which both retrieved and generated documents contain the correct answer. As shown in Table 5, when both retrieved and generated documents contain the correct answer, the FiD reader can produce more correct answers when reading the generated documents from large language models (e.g., InstructGPT).
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We also provide some case studies in Tables 16-19. For example, in Table 18, the question is "What city was Zeus the patron god of?". The first document retrieved by DPR is "Like the other Panhellenic Games, the ancient Olympic Games were a religious festival, held at the sanctuary of Zeus at Olympia.". Although it contains the correct answer, it is hard to infer the answer "Olympia" from it. On the contrary, InstructGPT generates the document "Zeus was the patron god of the city of Olympia, which was located in the northwestern Peloponnese region of Greece. Olympia was the site of the Olympic Games, held every four years in honor of Zeus.", which is much easier to read.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
CONCLUSION. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing the dense retrieval models with large language model generators. We call it generate-thenread, which first prompts a large language model to generate contextual documents, then read the generated document to infer the final answer. Notably, without retrieving any documents, it reaches 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the current retrieval-reader model DPR-FiD, as well as on other two knowledge-intensive tasks.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
LIMITATION AND FUTURE WORK. Despite the strong performance on the presented datasets, our approach is limited in its ability to update knowledge state and adapt to new domains. A major feature of retrieve-then-read is the ability to swap in new documents when new information is learned, such as temporally more recent documents, or adding in documents from a new domain to quickly adapt to a new downstream task. Our approach relies on a large language model to contain all this knowledge and adding new knowledge would likely require some retraining. Future work will explore how to efficiently incorporate new knowledge into our generate-then-read method. Besides, generated documents might suffer from hallucination error, resulting in incorrect predictions. We demonstrated case study in Table 15. Consideration in combination with recent approaches (Creswell & Shanahan, 2022) to boost generative faithfulness is a also direction worthy of future research.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Large language models have a wide range of beneficial applications for society, but they also have potentially harmful applications. Previous work has shown various forms of bias, such as racial and gender bias, in large language models like GPT-3, even after explicit efforts to reduce toxic language (Chan, 2022). The importance of addressing these societal harms is acknowledged by OpenAI themselves in their 2020 paper introducing GPT-3 (Brown et al., 2020), which stated "we focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 ... and issues of bias, fairness, and representation within models like GPT-3." on page 34.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
The goal of this paper is to utilize knowledge stored in the parameters of large language models to answer open-domain questions and solve knowledge-intensive tasks. Unlike retrieve-then-read where an external corpus can be curated to be trustworthy, the use of a model to generate contextual documents may further permeate existing biases in common models. First, our work shows that generated documents suffer from challenges of stale information from outdated documents used for training. Second, we show that generated documents tend to be less diverse, potentially biasing answers towards more common entities and terms from the training data. Finally, we conducted experiments on only three large language models. It is possible that some of our conclusions or observations may not necessarily hold for other models trained with different data or objectives.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Regarding ethical solutions, future work includes (i) further exploring potential bias and intentional or unintentional harm that may result from using generated contextual documents; (ii) better aligning language models with user intent to generate less biased contents and fewer fabricated facts.   -FEVER (Thorne et al., 2018) is one of the largest datasets for fact checking that requires retrieving evidence from external corpus to support if a statement is supported or refuted.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
-FOOL ME TWICE (FM2) (Eisenschlos et al., 2021) is a challenging fact checking dataset collected by gamification. Players write challenging claims either entailed or refuted by evidence from Wikipedia. They are then tasked to spot the refuted claim among a group.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
-WIZARD OF WIKIPEDIA (WoW) (Dinan et al., 2019) is an open-domain dialogue task for training agents that can converse knowledgeably about open-domain topics. One speaker in the conversation must ground their utterances in a specific knowledge sentence from a Wikipedia page.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We use the same train / dev / test splits in KILT challenge (Petroni et al., 2021) for the FEVER and WoW datasets. Their test labels are hidden, so the performance can only be evaluated through https://ai.facebook.com/tools/kilt. For FM2, we use its official dataset splits.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We use T5-770M  and T5-3B as our backbone models to implement FiD (Izacard & Grave, 2021). We use AdamW as the optimizer, with 2,000 warm-up steps. We set the dropout probability to 0.1 and weight decay to 0.01. We use one A100 for running T5-770M and set the batch size of 16. We use 8 A100 for running T5-3B and set the per GPU batch as 2, leading to the total batch size as 16. We searched different learning rates, ranging from 5e-6 to 4e-5, and we found 3e-5 to 6e-5 performed the best under the T5-3B setting and 5e-5 to 1e-4 performed the best under the T5-770M setting. We refer to more individual implementation details in Table 7.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We implement other baseline methods by using repositories:   Table 2), fact checking and dialogue system (numbers reported in Table 3). The upper part numbers are from GENREAD (FiD-l) and the lower part numbers are from GENREAD (FiD-xl).   62.1 51.8 InstructGPT (Ouyang et al., 2022) 71.3 54.5 Codex (OpenAI, 2022 72.6 55.4   Figure 4 shows the scaling of performance with In-structGPT generator parameters, including Ada-150M, Babbage-1.3B, Curie-6.7B and Davinci-175B. We note that for both FiD and our GENREAD , we use the FiD-xl with 10 input documents either retrieved from Wikipedia or generated by InstructGPT. The performance of both TriviaQA and WebQ continues to improve as the generator model parameters increase, as does the slope. Only with the largest size InstructGPT, GENREAD can outperform the DPR-FiD. This indicates using large language model to generate contextual documents is an "emergent ability" of scaling, which is not present in smaller models but is only present in larger language models (Wei et al., 2022a).      We now compare the costs of using DPR and InstructGPT to retrieve or generate contextual documents. We consider DPR using the BERT-base (Devlin et al., 2019) version with 110M parameters and InstructGPT using its largest version with 175B parameters. For simplicity, we use the FLOPs-pertoken estimates for Transformer-based language models, which is introduced by Kaplan et al. (2020). It should be noted that FLOPs are not a direct measure of real-world computing costs, as latency, power consumption, and other costs can vary widely based on other factors (Liu et al., 2022a).
Generate rather than Retrieve: Large Language Models are Strong Context Generators
For the DPR model, all Wikipedia documents (around 21M) only need to be encoded once. Therefore, as the number of input questions increases, the marginal computational cost gradually decreases. For fair comparison, we first use DPR to encode all 21M Wikipedia documents once. Encoding all Wikipedia documents requires 110e6 (BERT-base parameters) × 21e6 (total number of documents) × 100 (tokens per document) = 2.3e17 FLOPs. When the embedding of all candidate documents are produced, retrieving documents for a given question requires 110e6 (BERT-base parameters) × 20 (tokens per question) +21e6 (total number of documents) × (768 + 768 − 1) = 3.2e10 FLOPs.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
For InstructGPT, it requires 175e9 (InstructGPT parameters) × 10 (number of documents) × 55 (generated tokens per document) = 9.6e13 FLOPs to generate 10 documents for a given question.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Therefore, the equation for the total cost Y DPR-cost to retrieve 10 documents using DPR versus the number of input questions X is: Y DPR-cost = 3.2e10X + 2.3e17. Besides, the equation for the total cost Y GPT3-cost to generate 10 documents using InstructGPT versus the number of input questions X is: Y GPT3-cost = 9.6e13X. When Y DPR-cost = Y GPT3-cost , X ≈ 2473. In conclusion, if the number of input questions is less than 2473, the total cost of InstructGPT is lower than the DPR; if the number of input questions is greater than 2473, the total cost of InstructGPT exceeds the DPR.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
As stated in Zhang & Choi (2021), NQ contains a significant proportion, roughly 16.5%, of questions that have time-dependent answers. Similarly, Izacard et al. (2022b) observed using the latest version of Wikipedia (12 / 2021) could lead to 4.4 drops of the EM score, compared to the Wikipedia version (12 / 2018) that the NQ questions are created from. We provide case studies in Table 13 in Appendix.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
We did case studies of 100 examples from the NQ dataset. The results are shown in Table 12. Among these 100 examples, we found that 29 examples have data collection and annotation mistakes, mainly including the temporal question issue (13 / 29) and the incomplete answer issue (16 / 29). A typical temporal-dependent question is that no specific time condition is provided. For example, "Who won the MVP for the National League?" could have different answers in different years. In 2017, the MVP is Giancarlo Stanton, and in 2018, the MVP is Christian Yelich. Besides, some answer labels provided in the NQ dataset are not complete. For example, person names in the NQ dataset usually consist of first, middle, and last names, but most names in the generated documents are first and last names. For the question "who played lionel in as time goes by?", the labeled answer is "Geoffrey Dyson Palmer". DPR-FiD produces "Geoffrey Dyson Palmer" but GENREAD produces "Geoffrey Palmer", both of which should be considered correct. More examples are provided in Table 14.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Besides, GENREAD produced correct answers for 49 questions. Among the 22 incorrect predictions, 12 of them could be classified as retrieval errors (i.e., step-I error) and 12 as reading errors (i.e., step-II error). In all cases of retrieval errors, none of the generated documents contain the correct answer. In all cases of reading errors, at least one generated document contains the correct answer but the reader model failed to infer the correct answer from the documents..  We further categorized 51 incorrect predictions of our GENREAD, including errors caused by data collection and annotation, and errors caused by model prediction. In addition, we provide more case studies in Tables 13-15 (Table 13 for the temporal  question issue; Table 14 for the incomplete answer issue; Table 15 for the hallucination issue).  Table 13: Case studies of temporality issues of the NQ dataset. All these questions are drawn from Zhang & Choi (2021), which contains a subset of NQ data examples with time-dependent questions.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Original question DPR-FiD predictions GENREAD predictions Q: Who played lionel in as time goes by? Geoffrey Dyson Palmer Geoffrey Palmer Explanation: The labeled answer is "Geoffrey Dyson Palmer", however, "Geoffrey Palmer" is also correct. DPR retrieved documents: Geoffrey Dyson Palmer, (born 4 June 1927) is an English actor known for his roles in British television sitcoms playing Jimmy Anderson in "The Fall and Rise of Reginald Perrin", Ben Parkinson in "Butterflies" and Lionel Hardcastle in "As Time Goes By". His film appearances include "A Fish Called Wanda", "The Madness of King George", "Mrs. Brown", and "Tomorrow Never Dies". GPT generated documents: As Time Goes By is a British sitcom that aired on BBC One from 1992 to 2005. The show starred Geoffrey Palmer and Judi Dench as Lionel and Jean Pargetter, a middle-aged couple who reunite after many years apart. Lionel was played by Palmer, who was also a writer on the show. Q: How many cracker barrels in the united states? 645 over 630 Explanation: The labled answer is "639" or "over 600", so "over 630" is also a reasonable answer. Q: Where do the greasers live in the outsiders?
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Tulsa, Oklahoma Oklahoma Explanation: The labled answer is "Tulsa, Oklahoma", but "Oklahoma" is also a correct answer. DPR retrieved documents: The movie received mostly positive reviews from critics, and performed well at the box office, grossing 33 million on a 10 million budget. In Tulsa, Oklahoma, greasers are a gang of tough, low-income working-class teens. They include Ponyboy Curtis and his two older brothers, Sodapop and Darrel, as well as Johnny. GPT generated documents: The Outsiders is a novel by S.E. Hinton. It is about a gang of greasers in Oklahoma in the 1960s. The greasers are from the poor side of town and constantly in trouble with the law. Q: Where are unipolar neurons found in spinal cord? the granule region dorsal root ganglia Explanation: The labled answer is "the distal dorsal root", but the output "dorsal root ganglia" is the same. Table 14: Case studies of the incomplete answers issue of the NQ dataset. Since the labels in NQ are spans identified from Wikipedia passages, it is easier for DPR-FiD to predict correct answers.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Question: Who wrote the first declaration of human rights? Answer: Cyrus Cylinder Generated document: The first declaration of human rights was the Virginia Declaration of Rights, which was written by George Mason in 1776. Retrieved document: John Peters Humphrey, OC (April 30, 1905-2013March 14, 1995  Retrieved document: ... ... was the character of Barb, the nerdy friend and classmate of Nancy who is taken and killed by the monster early in the season. According to actress Shannon Purser, Barb wasn't supposed to be a big deal, and the Duffer Brothers had not gone into great detail about the character since the focus was on finding Will. ... ... (Wikipedia page: Stranger Things) Golden Wikipedia evidence: One area of focus from these fans was the character of Barb, the nerdy friend and classmate of Nancy who is taken and killed by the monster early in the season. According to actress Shannon Purser, Barbẅasn't supposed to be a big deal, and the Duffer Brothers had not gone into great detail about the character since the focus was on finding Will. ... ... (Wikipedia page: Stranger Things) The owner, Benny, learns from a tattoo of "011" on her arm that her name is Eleven. Brenner, monitoring the phone lines, sends agents to the diner after Benny calls social services. The agents kill Benny, but Eleven manages to escape using telekinetic abilities. Joyce's phone short circuits after receiving a mysterious phone call that she believes is from Will. While searching for Will in the woods, Mike, Dustin and Lucas come across Eleven. (Wikipedia page: Stranger Things) Hallucination: In GPT generated documents, Will Byers is said to be dead, which is actually incorrect. Will Byers is actually a fake deaths in season 1 and he is still alive at the end of the season. Model predictions: DPR-FiD Barb; GENREAD Will Question: Who played the judge in good will hunting? Answer: James P. Flynn Generated document: The character of the Judge in Good Will Hunting was played by actor Robert Sean Leonard. Leonard is best known for his role as Dr. James Wilson in the television series House.  Table 15: Case studies of hallucination errors in InstructGPT generated documents. The documents contain contents that contradict to the facts and world knowledge, resulting in wrong predictions.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Query: What type of music did Mozart composed? Answers: opera, classical music, art song, chamber music, ballet DPR #1: The following is a list of surviving works that Mozart composed for performance at gatherings of Masons. The story and music of his opera "The Magic Flute" is also considered to have strong Masonic influences. The following is a partial list of family members, patrons, and colleagues who were Masons. Mozart's grandfather Johann Georg, a bookbinder, was raised among the extended Mozart family in Augsburg, in the house of Johann's own grandfather David Mozart. David and his children were distinguished architects and master ... (Wikipedia id: 10988168 -"Mozart and Freemasonry") DPR #2: Progressively, and in large part at the hands of Mozart himself, the contrapuntal complexities of the late Baroque emerged once more, moderated and disciplined by new forms, and adapted to a new aesthetic and social milieu. Mozart was a versatile composer, and wrote in every major genre, including symphony, opera, the solo concerto, chamber music including string quartet and string quintet, and the piano sonata. These forms were not new, but Mozart advanced their technical sophistication and emotional reach. He almost single-handedly developed and popularized the Classical piano concerto. He wrote a great deal ... (Wikipedia id: 451968 -"Wolfgang Amadeus Mozart") Among top-10 retrieved documents: opera: 2 times chamber music: 1 time GPT nucleus #1: Mozart was a prolific composer and wrote in many different genres. He composed symphonies, operas, concertos, chamber music, and more. Many of his works are considered some of the greatest ever written in those genres. GPT nucleus #2: Mozart composed a variety of music, including operas, symphonies, chamber music, concertos, and sonatas. His early works were influenced by the music of Haydn and Bach, and his later works show the influence of the French and Italian composers of his time. Mozart's music is characterized by its beauty, elegance, and precision, and he is considered one of the greatest composers of all time. GPT nucleus #3: Mozart composed a wide variety of music, including operas, symphonies, concertos, solo pieces, and chamber music. He also wrote a number of religious works, including masses and motets. Among top-10 generated documents: chamber music: 9 times GPT cluster #1: Mozart found work in Vienna as a piano teacher and composer for opera and other works. There he met a fellow Mason, Joseph Haydn, who was to become an important mentor. Mozart married Constanze Weber in 1782; their marriage is thought to have been happy, in contrast to his earlier unhappy arranged marriage to his cousin, Maria AnnaNannerlMozart. Mozart composed more than 600 works, including works widely. GPT cluster #2: Mozart was a prolific composer, and wrote in a wide variety of genres. He composed over 600 works, including operas, symphonies, concertos, solo works, and chamber music. His operas include The Marriage of Figaro, Don Giovanni, and Cosi fan tutte. His symphonies include the well-known Symphony No. 40 in G minor, and his concertos include the Piano Concerto No. 21 in C major, known as theËlvira Madiganconcerto. GPT cluster #3: Mozart was a prolific composer who wrote music in a wide variety of genres. He is best known for his work in the classical genre, which includes such works as his Symphony No. 40 in G minor and his piano concertos. He also wrote operas, including The Marriage of Figaro and The Magic Flute. In addition to his work in classical music, Mozart also composed works in the popular genre, such as his Eine kleine Nachtmusik. Among top-10 generated documents: opera: 2 times chamber music: 7 times classical music: 1 time Table 16: Case study of retrieved documents and generated documents from language models (1/4).
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Published as a conference paper at ICLR 2023 Query: What is the collection of the districts to the east of the Jordan River? Answers: Jordan, Golan Heights DPR #1: Water politics in the Jordan River basin Water politics in the Jordan River basin refers to political issues of water within the Jordan River drainage basin, including competing claims and water usage, and issues of riparian rights of surface water along transnational rivers, as well as the availability and usage of ground water. Water resources in the region are scarce, and these issues directly affect the five political subdivisions (Israel, the West Bank, Lebanon, Syria and Jordan) located within and bordering the basin, which were created since the collapse, , during World War I, of the former single controlling entity, the ... (Wikipedia id: 12822413 -"Water politics in the Jordan River basin") DPR #2: Jordan's western border follows the bottom of the rift. Although an earthquake-prone region, no severe shocks had been recorded for several centuries. By far the greatest part of the East Bank is desert, displaying the land forms and other features associated with great aridity. Most of this land is part of the Syrian Desert and northern Arabian Desert. There are broad expanses of sand and dunes, particularly in the south and southeast, together with salt flats. Occasional jumbles of sandstone hills or low mountains support only meager and stunted vegetation that ... (Wikipedia id: 210052 -"Geography of Jordan") Among top-10 retrieved documents: Jordan: 8 times GPT nucleus #1: The collection of the districts to the east of the Jordan River is known as the West Bank. The West Bank is a landlocked territory situated in Western Asia. It is bordered by Jordan to the east, Egypt to the southwest, and Israel to the west and north. GPT nucleus #2: The collection of the districts to the east of the Jordan River is the West Bank. The West Bank is a landlocked territory near the Mediterranean coast of Western Asia, bordered by Lebanon to the north, Syria to the northeast, Jordan to the east, and Egypt to the southwest. It is the smaller of two political entities that comprise the Palestinian territories, the other being the Gaza Strip. Among top-10 generated documents: Jordan: 10 times GPT cluster #1: The collection of the districts to the east of the Jordan River is known as the West Bank. The West Bank is a landlocked territory situated in Western Asia. It is bordered by Jordan to the east, Egypt to the southwest, and Israel to the west and north. GPT cluster #2: The Jordan River is a major river in the Middle East, the source of which is in the northern part of Israel. The river flows southward through Israel and then Jordan, emptying into the Dead Sea. East of the river is the collection of districts known as the East Bank. GPT cluster #3: There is no single answer to this question as the east bank of the Jordan River is home to a number of different districts and regions, each with its own unique history, culture, and customs. However, some of the more well-known districts on the east bank include the West Bank, the Gaza Strip, and the Golan Heights. Among top-10 generated documents: Jordan: 10 times Golan Heights: 2 times We observed the prompts (i.e., "Q: {query}\n\nA:") used in GPT-3 paper (Brown et al., 2020) perform poorly on its text-davinci-002 version. Therefore, we experimented with multiple prompts and found the following two prompts work best on open-domain QA datasets.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
-(1) "{query}\n\nThe answer is" (no space between {query} and \n) -(2) "{query} \n\n The answer is" (performance reported in Table 1) For fact checking and dialogue system, we used the following prompts. We collected the prompt from P3 (Bach et al., 2022), which includes over 2,000 open-source prompts for roughly 170 datasets. For zero-shot QA, we experimented with three different reading comprehension prompts. We reported the performance for each prompt in Table 20. -(1) "Refer to the passage below and answer the following question with just a few words. Passage: {background} \n\n Question: {query} \n\n The answer is" -(2) "Passage: {background} \n\n Question: {query} \n\n Referring to the passage above, the correct answer (just one entity) to the given question is" -(3) "Refer to the passage below and answer the following question with just one entity. \n\n Passage: background \n\n Question: query \n\n The answer is" For fact checking and dialogue system, we chose the simplest prompt from P3. In order to get a better prompt for large language models to generate better contextual documents, we asked 30 students in the computer science department to write different prompts. We first constructed a small validation set with 200 examples by combining 50 random question-answer pairs from NQ, 100 random pairs from TriviaQA and 50 random pairs from WebQ. When an annotator wrote down a prompt, our system can immediately evaluate the prompt by using the validation set and return the performance to the annotator. Then, the annotator can modify the previous prompt until the recall performance reaches a threshold, which is set as 50 in our experiments. Finally, we got 29 prompts from human annotators due to two of them are the same. We used the top-10 prompts (shown in Table 21 and Table 22) in the human prompt setting, as described in §3.2.1.  Prepare a background document to support the given question.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
To support the given question, prepare a background document. 62.5 #8
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Create a background document that supports the given question.
Generate rather than Retrieve: Large Language Models are Strong Context Generators
Retrieve a document from Wikipedia to answer the given question. 60.5 #10 Retrieve a Wikipedia article to address the posed question. 59.5  
Large Language Models Struggle to Learn Long-Tail Knowledge
Large language models (LLMs) trained on text from the Internet capture many facts about the world, ranging from well-known factoids to esoteric domain-specific information. These models implicitly store knowledge in their parameters   (Petroni et al., 2019;Roberts et al., 2020), and given the scale of today's pre-training datasets and LLMs, one would hope that they can learn a huge amount of information from web-sourced text. However, not all of the knowledge on the Internet appears equally often-there is a long-tail of information that appears rarely or only once.
Large Language Models Struggle to Learn Long-Tail Knowledge
In this work, we explore the relationship between the knowledge learned by an LLM and the information in its pretraining dataset. Specifically, we study how an LLM's ability to answer a question relates to how many documents associated with that question were seen during pretraining. We focus on factoid QA datasets (Joshi et al., 2017;Kwiatkowski et al., 2019), which lets us ground questionanswer pairs into concrete subject-object co-occurrences.
Large Language Models Struggle to Learn Long-Tail Knowledge
As an example, for the QA pair (In what city was the poet Dante born?, Florence), we consider documents where the entities Dante and Florence co-occur as highly relevant.
Large Language Models Struggle to Learn Long-Tail Knowledge
To identify these entity co-occurrences we apply a highlyparallelized entity linking pipeline to trillions of tokens from datasets such as C4 , The Pile (Gao et al., 2020), ROOTS (Laurençon et al., 2022), OpenWeb-Text (Gokaslan & Cohen, 2019), and Wikipedia.
Large Language Models Struggle to Learn Long-Tail Knowledge
We observe a strong correlation between an LM's ability to answer a question and the number of pre-training documents relevant to that question for numerous QA datasets, pretraining datasets, and model sizes (e.g., Figure 1). For example, the accuracy of BLOOM-176B  jumps from 25% to above 55% when the number of relevant pre-training documents increases from 10 1 to 10 4 .
Large Language Models Struggle to Learn Long-Tail Knowledge
We also conduct a counterfactual re-training experiment, where we train a 4.8B-parameter LM with and without certain documents. Model accuracy drops significantly on questions whose relevant documents were removed, which validates our entity linking pipeline and shows that the observed correlational trends are likely causal in nature.
Large Language Models Struggle to Learn Long-Tail Knowledge
Finally, we analyze ways to better capture knowledge that rarely appears in the pre-training data: model scaling and retrieval-augmentation. For model scaling, we find a strong log-linear relationship between parameter count and QA accuracy. These trends show that while scaling up LMs improves knowledge learning, models would need to be scaled dramatically (e.g., to one quadrillion parameters) to achieve competitive QA accuracy on long-tail questions. Retrieval-augmented systems are more promising-when a retriever succeeds in finding a relevant document, it reduces an LLM's need to have a large amount of relevant pretraining text. Nevertheless, retrieval systems themselves still exhibit a mild dependence on relevant document count.
Large Language Models Struggle to Learn Long-Tail Knowledge
Overall, our work is one of the first to study how LLM knowledge is influenced by pre-training data. To enable future research, we release our code as well as the entity data for ROOTS, The Pile, C4, OpenWebText, and Wikipedia at https://github.com/nkandpa2/long tail knowledge.
Large Language Models Struggle to Learn Long-Tail Knowledge
Background and Research Question Numerous NLP tasks are knowledge-intensive: they require recalling and synthesizing facts from a knowledge source (e.g. Wikipedia or the web). Results on knowledge-intensive tasks have been dramatically improved using LLMs, as these models have been shown to leverage the vast amounts of knowledge they learn from their pre-training corpora Petroni et al., 2019;De Cao et al., 2021). However, it remains unclear as to what kind of knowledge LMs actually capture-for example, do they simply learn "easy" facts that frequently appear in their pre-training data?
Large Language Models Struggle to Learn Long-Tail Knowledge
We study this question using closed-book QA evaluations  of LLMs in the few-shot setting (Brown et al., 2020). Models are prompted with incontext training examples (QA pairs) and a test question without any relevant background text. The goal of our work is to investigate the relationship between an LM's ability to answer a question and the number of times information relevant to that question appears in the pre-training data.
Large Language Models Struggle to Learn Long-Tail Knowledge
The key challenge is to efficiently identify all of the documents that are relevant to a particular QA pair in pre-training datasets that are hundreds of gigabytes in size. To tackle this, we begin by identifying the salient entities that are contained in a question and its set of ground-truth answer aliases. We then identify relevant pre-training documents by searching for instances where the salient question entity and the answer entity co-occur.
Large Language Models Struggle to Learn Long-Tail Knowledge
For example, consider the question In what city was the poet Dante born? with the valid answers Florence, City of Florence, and Italy (e.g., Figure 2). We extract the salient question and answer entities, Dante Alighieri and Florence, and count the documents that contain both entities.
Large Language Models Struggle to Learn Long-Tail Knowledge
Our approach is motivated by Elsahar et al. (2018), who show that when only the subject and object of a subjectobject-relation triple co-occur in text, the resulting triple is often also present. In addition, we conduct human studies that show our document counting pipeline selects relevant documents a majority of the time (Section 2.3). Moreover, we further validate our pipeline by training an LM without certain relevant documents and showing that this reduces accuracy on the associated questions (Section 3.2). Based on these findings, we refer to documents that contain the salient question and answer entities as relevant documents.
Large Language Models Struggle to Learn Long-Tail Knowledge
To apply the above method, we must entity link massive pre-training corpora, as well as downstream QA datasets. We accomplish this by building a parallelized pipeline for entity linking (Section 2.1), which we then customize for downstream QA datasets (Section 2.2).
Large Language Models Struggle to Learn Long-Tail Knowledge
We perform entity linking at scale using a massively distributed run of the DBpedia Spotlight Entity Linker (Mendes et al., 2011), which uses traditional entity linking methods to link entities to DBpedia or Wikidata IDs. We entity link the following pre-training datasets, which were chosen based on their use in the LLMs we consider:
Large Language Models Struggle to Learn Long-Tail Knowledge
• The Pile: an 825GB dataset that contains a mix of 22 different primarily English-language sources (Gao et al., 2020).
Large Language Models Struggle to Learn Long-Tail Knowledge
• ROOTS (En): the 490GB English subset of the ROOTS corpus (Laurençon et al., 2022). Note that we do not study if models trained on the non-English subsets of ROOTS are able to leverage cross-lingual factual knowledge.
Large Language Models Struggle to Learn Long-Tail Knowledge
• C4: a 305GB English corpus that was collected by filtering CommonCrawl .
Large Language Models Struggle to Learn Long-Tail Knowledge
• OpenWebText: a 39GB English corpus that contains the text of web pages that were linked on the website Reddit (Gokaslan & Cohen, 2019 For each document in these pre-training datasets, we record the linked entities in a data structure that enables quickly counting individual entity occurrences and entity co-occurrences. This pipeline took approximately 3 weeks to entity link 2.1TB of data on a 128-CPU-core machine.
Large Language Models Struggle to Learn Long-Tail Knowledge
We next entity link two standard open-domain QA datasets: Natural Questions (Kwiatkowski et al., 2019) and Trivi-aQA (Joshi et al., 2017). To expand our sample sizes, we use both the training and validation data, except for a small set of examples used for few-shot learning prompts.
Large Language Models Struggle to Learn Long-Tail Knowledge
We first run the DBPedia entity linker on each example. Because there can be multiple annotated answers for a single example, we concatenate the question and all valid answers, as this enabled more accurate entity linking. We use the most common entity found in the set of ground truth answers as the salient answer entity. We then iterate over all entities found in the question and select the entity that co-occurs the most with the salient answer entity in the pre-training data. In cases where no entity is found in the question, answer, or both, we discard the example. If the resulting number of relevant documents is zero, we discard the example, as this is likely due to an entity linking error.
Large Language Models Struggle to Learn Long-Tail Knowledge
Here, we conduct a human evaluation of our document identification pipeline. Note that a document can vary in the extent to which it is "relevant" to a particular QA pair. For instance, consider the QA pair (William Van Allan designed which New York building-the tallest brick building in the world in 1930?, Chrysler Building). The documents that we identify as relevant may (1) contain enough information to correctly answer the question, (2) contain information relevant to the question but not enough to correctly answer it, or (3) contain no relevant information. For example, a document that mentions that the Chrysler building was designed by William Van Allan, but not that it was the tallest brick building in 1930, would fall into the second category.
Large Language Models Struggle to Learn Long-Tail Knowledge
We randomly sample 300 QA pairs from TriviaQA and selected one of their relevant documents at random. We then manually labeled the documents into one of the three categories: 33% of documents contained enough information to answer the question and an additional 27% of contained some relevant information. Thus, our pipeline has ∼60% precision at identifying relevant documents for TriviaQA.
Large Language Models Struggle to Learn Long-Tail Knowledge
Our pipeline is imperfect as (1) the entity linker sometimes mis-identifies entities and (2) not all documents containing the salient question and answer entity are relevant. However, when applied at the scale of large-scale pre-training datasets, this pipeline is efficient and achieves enough precision and recall to observe correlational (Section 3.1) and causal (Section 3.2) relationships to QA performance. . We plot accuracy on TriviaQA versus relevant document count for GPT-Neo. The trends match those seen for BLOOM ( Figure 1). We also include a histogram that shows how many QA examples fall into each bucket; TriviaQA often asks about knowledge represented 10 2 to 10 5 times in the pre-training data.
Large Language Models Struggle to Learn Long-Tail Knowledge
In this section, we measure the relationship between an LLM's ability to answer a question and the number of relevant documents in the pre-training corpus. We use popular Transformer decoder-only LMs (Vaswani et al., 2017) that span three orders of magnitude in size:
Large Language Models Struggle to Learn Long-Tail Knowledge
• GPT-Neo: The GPT-Neo, GPT-NeoX, and GPT-J LMs trained by EleutherAI on the Pile (Gao et al., 2020) that range in size from 125M to 20B parameters (Black et al., 2021;Wang & Komatsuzaki, 2021;Black et al., 2022). We refer to these models collectively as GPT-Neo models.
Large Language Models Struggle to Learn Long-Tail Knowledge
• BLOOM: Models trained by the BigScience initiative on the ROOTS dataset . The BLOOM models are multi-lingual; we analyze their English performance only. The models range in size from 560M to 176B parameters.
Large Language Models Struggle to Learn Long-Tail Knowledge
• GPT-3: Models trained by OpenAI that range in size from ≈350M (Ada) to ≈175B parameters (Davinci). Since the pre-training data for these models is not public, we estimate relevant document counts by scaling up the counts from OpenWebText to simulate if the dataset was the same size as GPT-3's pre-training data. We recognize that there is uncertainty around these models' pre-training data, their exact sizes, and whether they have been fine-tuned. We therefore report these results in the Appendix for readers to interpret with these sources of error in mind. . . .
Large Language Models Struggle to Learn Long-Tail Knowledge
A: [In-Context Answer n]
Large Language Models Struggle to Learn Long-Tail Knowledge
We generate answers by greedy decoding until the models generate a newline character, and we evaluate answers using the standard Exatch Match (EM) metric against the groundtruth answer set (Rajpurkar et al., 2016).
Large Language Models Struggle to Learn Long-Tail Knowledge
We first evaluate the BLOOM and GPT-Neo model families on TriviaQA and plot their QA accuracies versus the number of relevant documents in Figures 1 and 3. For improved readability, we average the accuracy for QA pairs using logspaced bins (e.g., the accuracy for all questions with 1 to 10 relevant documents, 10 to 100 relevant documents, etc.). Below each plot, we also include a histogram that shows how many QA examples fall into each bin. We trim the plots when the bins contain fewer than 500 QA examples to avoid reporting accuracies for small sample sizes. There is a strong correlation between question answering accuracy and relevant document count for all tested models. Correspondingly, when the number of relevant documents is low, models are quite inaccurate, e.g., the accuracy of BLOOM-176B jumps from 25% to above 55% when the number relevant documents increases from 10 1 to 10 4 . Model size is also a major factor in knowledge learning: as the number of model parameters is increased, the QA performance substantially improves. For example, BLOOM-176B has over 4× higher accuracy than BLOOM-560M on TriviaQA questions with more than 10 5 relevant documents.
Large Language Models Struggle to Learn Long-Tail Knowledge
We repeat this experiment using the Natural Questions QA dataset and find similar trends for all model families (see Figure 4 for GPT-Neo, and Figures 10 and 11 in the Appendix for BLOOM and GPT-3 results).
Large Language Models Struggle to Learn Long-Tail Knowledge
Are Less Effective In the experiments above, we identify relevant documents by searching for co-occurrences of salient question and answer entities. To evaluate whether this process is necessary, we compare against two baseline document identification methods: counting documents that contain the salient question entity and counting documents that contain the salient answer entity (as done in Petroni et al. 2019).
Large Language Models Struggle to Learn Long-Tail Knowledge
We show in Figure 13 that all three document identification methods are correlated with QA accuracy. However, when only considering QA examples where the question and answer entities co-occur few (< 5) times, the two baseline methods no longer correlate with QA accuracy. This indicates that counting documents with just the answer entity or question entity alone is insufficient for explaining why LMs are able to answer certain questions. This validates our definition of relevant documents as those that contain both the question entity and answer entity.
Large Language Models Struggle to Learn Long-Tail Knowledge
Humans Show Different Trends Than LMs An alternate explanation for our results is that questions with lower document counts are simply "harder", which causes the drop in model performance. We show that this is not the case by measuring human accuracy on Natural Questions. We use a leave-one-annotator-out metric, where we take questions that are labeled by 5 different human raters (all of whom can see the necessary background text), hold out one of the raters, and use the other four as the ground-truth answer set. We plot the human accuracy versus relevant document count in the top of Figure 7. Human accuracy is actually highest for the questions with few relevant documents, the opposite trend of models. We hypothesize that humans are better on questions with few relevant documents because (1) questions about rarer facts are more likely to be simple factoids compared to common entities, and (2) the Wikipedia documents are that are provided to the annotators are shorter for rarer entities, which makes reading comprehension easier and increases inner-annotator agreement.
Large Language Models Struggle to Learn Long-Tail Knowledge
Our results thus far are correlational in nature: there may be unknown confounds that explain them away, i.e., the rarer questions are more difficult for LMs for other reasons.
Large Language Models Struggle to Learn Long-Tail Knowledge
Here we establish a causal relationship by removing certain documents in the training data and re-training the LM.
Large Language Models Struggle to Learn Long-Tail Knowledge
We first train a baseline 4.8 billion parameter LM on C4, following the setup from Wang et al. (2022). We then measure the effect of deleting certain documents from the training set. For each log-scaled bin of relevant document count (e.g., 10 0 to 10 1 relevant documents, 10 1 to 10 2 , ...) we sample 100 questions from Trivia QA and remove all relevant documents for those questions in C4. In total, this removes about 30% of C4. Finally, we train a "counterfactual" LM on this modified pre-training dataset and compare its performance to the baseline model. For both the baseline model and the counterfactual model, we train for a single epoch. Note that the counterfactual model was trained for 30% fewer steps, which makes it slightly worse in performance overall. To account for this, we only study the performance on questions whose relevant documents were removed.
Large Language Models Struggle to Learn Long-Tail Knowledge
We show the difference in performance between the two LMs on questions whose documents were removed in Fig . Scaling trends for fact learning. We plot BLOOM accuracy on rare instances from Natural Questions (< 100 relevant docs) as a function of the log of the model size. Extrapolating from the empirical line of best fit-which approximates the trend well at R 2 = 0.98-implies that immensely large models would be necessary to get high accuracy.
Large Language Models Struggle to Learn Long-Tail Knowledge
ure 5. For questions with few relevant documents in the original C4 dataset, performance is poor for both the baseline and the counterfactual LM, i.e., their performance difference is small. However, for questions with many relevant documents, performance is significantly worse for the counterfactual LM. This suggests a causal link between the number of relevant documents and QA performance.
Large Language Models Struggle to Learn Long-Tail Knowledge
Thus far, we showed that LLMs have a strong dependence on relevant document count. Here, we investigate methods to mitigate this dependence: increasing data scale, increasing model scale, and adding an auxiliary retrieval module.
Large Language Models Struggle to Learn Long-Tail Knowledge
Today's largest LLMs are pre-trained on hundreds of billions of tokens. One naïve approach for improving accuracy on questions about less-prevalent knowledge is to collect larger quantities of data. Our results suggest that this would not significantly improve accuracy as scaling datasets by moderate factors (e.g., 5x) usually results in small accuracy gains. An alternative idea would be to increase the diversity of the pre-training data. However, we also believe this would provide minimal benefit because many data sources are surprisingly correlated . Although each of the pre-training datasets considered were collected indepen- dently, the amount of supporting information they provide for different TriviaQA examples is highly consistent as seen by the rank correlations between their relevant document counts in Table 1.
Large Language Models Struggle to Learn Long-Tail Knowledge
Using larger models consistently produces better QA performance. However, our results suggest that one would need immensely large LMs to achieve high accuracy on long-tail questions. In Figure 6 we plot a scaling trend line for rare fact learning, where we show BLOOM accuracy on rare instances from Natural Questions (< 100 relevant docs) as a function of the log of the model size. The empirical log-linear trend-which approximates the scaling extremely well (R 2 = 0.98)-shows that in order to match a strong supervised baseline (Izacard & Grave, 2021) or human performance, one would need a BLOOM model with over 10 18 (one quintillion) parameters. 1 We see similar trends for other models and datasets (see Figure 12 in the Appendix).
Large Language Models Struggle to Learn Long-Tail Knowledge
Modifying the Training Objective Another option similar to scaling up models is to directly modify the training objective to encourage memorization. One simple method to accomplish this is to increase the number of training epochs. All of the LMs that we study do limited epochs, as it is generally seen as preferable to use large enough pre-training datasets so that the LM completes one epoch of training when the compute budget is exhausted . However, in the context of QA, it may be preferable to increase epochs and reduce data size to ensure models memorize as much as possible. Alternatively, one could consider modifying the training loss to encourage the model to focus on salient facts (Guu et al., 2020) or designing a curriculum to minimize forgetting (Jagielski et al., 2023). 
Large Language Models Struggle to Learn Long-Tail Knowledge
Thus far, we use LMs as isolated systems that do not leverage external information. However, for knowledge-intensive tasks, a natural alternative is to make LMs retrievalaugmented, i.e., combine them with a retrieval module that returns relevant textual contexts Guu et al., 2020;Karpukhin et al., 2020). Here, we study whether retrieval-augmented models can mitigate the dependence on the amount of relevant knowledge in the pre-training data.
Large Language Models Struggle to Learn Long-Tail Knowledge
Oracle Retrieval We first study an oracle setting where we provide LMs with a gold paragraph from Wikipedia that supports each answer in Natural Questions . We use the 300-word segment that surrounds the ground-truth answer from the gold Wikipedia page and evaluate the 2-shot accuracy of GPT-Neo. Figure 7 shows that oracle retrieval-augmentation dramatically boosts accuracy over closed-book models, especially on rarer instances. Similar to Liu et al. (2022), we also find that QA accuracy actually goes down as the number of relevant documents increases-the opposite trend of closed-book LLMs. As discussed in Section 3.1, humans exhibit the same trend, likely because rare questions are easier on average when relevant context information.
Large Language Models Struggle to Learn Long-Tail Knowledge
BM25 Retrieval We next follow a common retrievalaugmented baseline, where we use a BM25 retriever (Robertson & Zaragoza, 2009) to select paragraphs from Wikipedia. We add the top-3 highest scoring paragraphs into the prompt for both the in-context training examples and the test question. We verify that at least one of the retrieved paragraphs contains the answer for each in-context training example, to ensure that the LM learns to utilize on the documents.
Large Language Models Struggle to Learn Long-Tail Knowledge
We first evaluate the BM25 retriever's top-k recall on its knowledge corpus (Wikipedia) as a function of relevant document count, and plot the results in Figure 8. We find that BM25 attains reasonably high recall, especially for larger values of k. However, the BM25 retriever still shows a mild dependence on relevant document count. We next evaluate the accuracy of BM25-augmented GPT-Neo models on Natural Questions and plot the results in Figure 9. Overall, retrieval-augmented models outperform their closed-book counterparts across all ranges of relevant document counts, and especially on rare examples. These results suggest that retrieval augmentation provides a promising path towards improving performance on questions with few relevant documents in the pre-training dataset.
Large Language Models Struggle to Learn Long-Tail Knowledge
Past work studies training data memorization from the perspective of privacy, i.e., how LMs inadvertently reveal private text (Carlini et al., 2019;Lee et al., 2021). These works focus on how LMs memorize and repeat verbatim text samples, and the effect of duplicating those texts in the training set (Kandpal et al.,  Memorization and Fact Learning Existing work also analyzes the relationship between the pre-training data and the factual knowledge of LLMs. Akyürek et al. (2022) look to automatically identify which documents were most influential for a language model's QA predictions. Our work instead directly identifies and estimates the number of relevant documents via entity linking large corpora. Other work notices a correspondence between model accuracy and data frequency for different knowledge-intensive tasks (Petroni et al., 2019;Kassner et al., 2020;De Cao et al., 2021;Wei et al., 2021;Févry et al., 2020) and for domains outside of NLP (Rao et al., 2021). Our paper reports similar findings, but scales this analysis to massive LM pre-training datasets and model sizes.
Large Language Models Struggle to Learn Long-Tail Knowledge
In concurrent and independent work, Mallen et al. (2022) study how QA performance correlates with frequency in the pre-training data. Unlike our work, they do not use entity linking methods to count occurrences and instead use proxies such as entity popularity on Wikipedia. They also find QA accuracy is highly correlated with pre-training data frequency and show that retrieval models can improve long-tail knowledge. Our work differs in that we conduct causal re-training experiments and find that model scaling is highly beneficial to long-tail QA performance.
Large Language Models Struggle to Learn Long-Tail Knowledge
Large language models demonstrate impressive few-shot learning capabilities that arise from simply training on largescale internet text. With the open-source release of LLMsand their associated pre-training datasets-the research community can now begin to understand the origins of these capabilities. Our work is one of the first to relate an observed phenomenon in LLMs back to the pre-training data itself. In our case, our results are negative: while LLMs achieve moderate performance on open-domain QA benchmarks, they are mainly successful on questions that probe knowledge that appears widely in their pre-training datasets.
Large Language Models Struggle to Learn Long-Tail Knowledge
Our work raises numerous directions for further inquiry, namely, how to improve retention of long-tail knowledge given that simply scaling up model and dataset size will likely be insufficient. We are personally excited about improving retrieval-augmented LMs, especially with regards to their efficiency and retrieval accuracy. Moreover, our work focuses on knowledge learning as it relates to factoid question answering, but we leave open the question as to whether similar relationships exist for other types of tasks, be it knowledge-intensive or otherwise. Relatedly, even though our work analyzes the impact of memorization on question answering, our results may have implications for other tasks that require using (or avoiding) memorized knowledge, e.g., analyzing private text, performing commonsense reasoning, or predicting source code. Finally, we hope that future evaluations of few-shot learning can continue to shed light into model behavior by tracing accuracy back to properties of the pre-training data. In particular, our work shows that by performing such an analysis, one can help elucidate the successes and failures of existing models, as well as help to identify possible paths forward to improve today's systems.
Large Language Models Struggle to Learn Long-Tail Knowledge
Here we show how QA performance is related to the number of relevant pre-training documents for the BLOOM on Natural Questions ( Figure 10) and the GPT-3 model family on TriviaQA and Natural Questions (Figure 11). Like the results in the main text, models are significantly better at answering questions about facts that are well supported in the pre-training data and model scale improves knowledge acquisition.
Large Language Models Struggle to Learn Long-Tail Knowledge
Note that our estimates for the number of relevant pre-training documents for the GPT-3 model family may be inaccurate since the training data for GPT-3 is not public. Instead, we estimate these relevant document counts using the open source dataset OpenWebText, which was collected with a similar process to the reported collection methodology for the GPT-3 pre-training dataset.  Figure 11. We present the QA results for GPT-3, with Natural Questions shown in (a) and TriviaQA shown in (b). The trends match those seen in BLOOM and GPT-Neo. Note that our estimates for the number of relevant pre-training documents may be inaccurate because the training data for GPT-3 is not public.
Large Language Models Struggle to Learn Long-Tail Knowledge
In this section we show additional results for how long-tail QA accuracy scales with model size for the BLOOM model family on TriviaQA and the GPT-Neo model family on Natural Questions and TriviaQA ( Figure 12). The log-linear trend matches the results shown in the main text. All the trends are similar-we will need to scale up models dramatically to reach high QA accuracy-but the exact degree to how much we would need to scale models changes across the different settings.
Large Language Models Struggle to Learn Long-Tail Knowledge
In this section, we analyze the difference between our relevant document heuristic, which counts documents where the salient question and answer entity co-occur, compared to two simple baselines: counting documents containing the question entity and documents containing the answer entity. In Figure 13(a) we show that all three document counting heuristics are correlated with QA accuracy. However, as seen in Figure 13(b) the correlation of the two baseline counting methods with QA accuracy disappears when only considering QA examples where the question and answer entity co-occur few (< 5) times in the pre-training data. Thus, these baseline counting heuristics appear correlated with QA accuracy simply because they are simply correlated with question and answer entity co-occurrence (i.e., common entities tend to co-occur with other entities more frequently) rather than causally related to QA performance.  Figure 13. In (a), we plot the relationship between model accuracy and the count of the question entity alone, as well as the answer entity alone. QA accuracy increases as both of these counts increase. In (b), we consider only f QA pairs with few question and answer entity co-occurrences (< 5 documents). For this subpopulation of QA pairs, neither of the baseline heuristics are correlated with QA accuracy.
Language Models as Knowledge Bases?
Recently, pretrained high-capacity language models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018a) have become increasingly important in NLP. They are optimised to either predict the next word in a sequence or some masked word anywhere in a given sequence (e.g. "Dante was born in [Mask] in the year 1265."). The parameters of these models appear to store vast amounts of linguistic knowledge (Peters et al., 2018b;Goldberg, 2019;Tenney et al., 2019) useful for downstream tasks. This knowledge is usually accessed either by conditioning on latent context representations produced by the original model or by using the original model weights to initialize a task-specific model which is then further fine-tuned. This type of knowledge transfer is crucial for current state-of-the-art results on a wide range of tasks.
Language Models as Knowledge Bases?
In contrast, knowledge bases are effective solutions for accessing annotated gold-standard relational data by enabling queries such as (Dante, born-in, X). However, in practice we often need to extract relational data from text or other modalities to populate these knowledge bases. This requires complex NLP pipelines involving entity extraction, coreference resolution, entity linking and relation extraction (Surdeanu and Ji, 2014)components that often need supervised data and fixed schemas. Moreover, errors can easily propagate and accumulate throughout the pipeline. Instead, we could attempt to query neural language models for relational data by asking them to fill in masked tokens in sequences like "Dante was born in [Mask]", as illustrated in Figure 1. In this setting, language models come with various attractive properties: they require no schema engineering, do not need human annotations, and they support an open set of queries.
Language Models as Knowledge Bases?
Given the above qualities of language models as potential representations of relational knowledge, we are interested in the relational knowledge already present in pretrained off-the-shelf language models such as ELMo and BERT. How much relational knowledge do they store? How does this differ for different types of knowledge such as facts about entities, common sense, and general question answering? How does their performance without fine-tuning compare to symbolic knowledge bases automatically extracted from text? Beyond gathering a better general understanding of these models, we believe that answers to these questions can help us design better unsupervised knowledge representations that could transfer factual and commonsense knowledge reliably to downstream tasks such as commonsense (visual) question answering (Zellers et al., 2018;Talmor et al., 2019) or reinforcement learning (Branavan et al., 2011;Chevalier-Boisvert et al., 2018;Bahdanau et al., 2019;Luketina et al., 2019).
Language Models as Knowledge Bases?
For the purpose of answering the above questions we introduce the LAMA (LAnguage Model Analysis) probe, consisting of a set of knowledge sources, each comprised of a set of facts. We define that a pretrained language model knows a fact (subject, relation, object) such as (Dante, born-in, Florence) if it can successfully predict masked objects in cloze sentences such as "Dante was born in " expressing that fact. We test for a variety of types of knowledge: relations between entities stored in Wikidata, common sense relations between concepts from ConceptNet, and knowledge necessary to answer natural language questions in SQuAD. In the latter case we manually map a subset of SQuAD questions to cloze sentences.
Language Models as Knowledge Bases?
Our investigation reveals that (i) the largest BERT model from Devlin et al. (2018b) (BERT-large) captures (accurate) relational knowledge comparable to that of a knowledge base extracted with an off-the-shelf relation extractor and an oracle-based entity linker from a corpus known to express the relevant knowledge, (ii) factual knowledge can be recovered surprisingly well from pretrained language mod-els, however, for some relations (particularly N-to-M relations) performance is very poor, (iii) BERT-large consistently outperforms other language models in recovering factual and commonsense knowledge while at the same time being more robust to the phrasing of a query, and (iv) BERT-large achieves remarkable results for open-domain QA, reaching 57.1% precision@10 compared to 63.5% of a knowledge base constructed using a task-specific supervised relation extraction system.
Language Models as Knowledge Bases?
In this section we provide background on language models. Statistics for the models that we include in our investigation are summarized in Table 1.
Language Models as Knowledge Bases?
Given an input sequence of tokens w = [w 1 , w 2 , . . . , w N ], unidirectional language models commonly assign a probability p(w) to the sequence by factorizing it as follows
Language Models as Knowledge Bases?
(1)
Language Models as Knowledge Bases?
A common way to estimate this probability is using neural language models (Mikolov and Zweig, 2012;Melis et al., 2017;Bengio et al., 2003) with
Language Models as Knowledge Bases?
where h t ∈ R k is the output vector of a neural network at position t and W ∈ R |V| × k is a learned parameter matrix that maps h t to unnormalized scores for every word in the vocabulary V. Various neural language models then mainly differ in how they compute h t given the word history, e.g., by using a multi-layer perceptron (Bengio et al., 2003;Mikolov and Zweig, 2012), convolutional layers (Dauphin et al., 2017), recurrent neural networks (Zaremba et al., 2014;Merity et al., 2016;Melis et al., 2017) or self-attention mechanisms (Radford et al., 2018;Dai et al., 2019;Radford et al., 2019). So far, we have looked at language models that predict the next word given a history of words. However, in many downstream applications we mostly care about having access to contextual representations of words, i.e., word representations that are a function of the entire context of a unit of text such as a sentence or paragraph, and not only conditioned on previous words. Formally, given an input sequence w = [w 1 , w 2 , . . . , w N ] and a position 1 ≤ i ≤ N, we want to estimate p(w i ) = p(w i | w 1 , . . . , w i−1 , w i+1 , . . . , w N ) using the left and right context of that word. ELMo: To estimate this probability, Peters et al. (2018a) propose running a forward and backward LSTM (Hochreiter and Schmidhuber, 1997), resulting in − → h i and ← − h i which consequently are used to calculate a forward and backward language model log-likelihood. Their model, ELMo, uses multiple layers of LSTMs and it has been pretrained on the Google Billion Word dataset. Another version of the model, ELMo 5.5B, has been trained on the English Wikipedia and monolingual news crawl data from WMT 2008-2012. BERT: Instead of a standard language model objective, Devlin et al. (2018a) propose to sample positions in the input sequence randomly and to learn to fill the word at the masked position. To this end, they employ a Transformer architecture and train it on the BookCorpus (Zhu et al., 2015) as well as a crawl of English Wikipedia. In addi-tion to this pseudo language model objective, they use an auxiliary binary classification objective to predict whether a particular sentence follows the given sequence of words.
Language Models as Knowledge Bases?
We introduce the LAMA (LAnguage Model Analysis) probe to test the factual and commonsense knowledge in language models. It provides a set of knowledge sources which are composed of a corpus of facts. Facts are either subject-relationobject triples or question-answer pairs. Each fact is converted into a cloze statement which is used to query the language model for a missing token. We evaluate each model based on how highly it ranks the ground truth token against every other word in a fixed candidate vocabulary. This is similar to ranking-based metrics from the knowledge base completion literature (Bordes et al., 2013;Nickel et al., 2016). Our assumption is that models which rank ground truth tokens high for these cloze statements have more factual knowledge. We discuss each step in detail next and provide considerations on the probe below.
Language Models as Knowledge Bases?
To assess the different language models in Section 2, we cover a variety of sources of factual and commonsense knowledge. For each source, we describe the origin of fact triples (or questionanswer pairs), how we transform them into cloze templates, and to what extent aligned texts exist in Wikipedia that are known to express a particular fact. We use the latter information in supervised baselines that extract knowledge representations directly from the aligned text.
Language Models as Knowledge Bases?
The Google-RE corpus 3 contains ∼60K facts manually extracted from Wikipedia. It covers five relations but we consider only three of them, namely "place of birth", "date of birth" and "place of death". We exclude the other two because they contain mainly multi-tokens objects that are not supported in our evaluation. We manually define a template for each considered relation, e.g., "[S] was born in [O]" for "place of birth". Each fact in the Google-RE dataset is, by design, manually aligned to a short piece of Wikipedia text supporting it.
Language Models as Knowledge Bases?
The T-REx knowledge source is a subset of Wikidata triples. It is derived from the T-REx dataset (Elsahar et al., 2018) and is much larger than Google-RE with a broader set of relations. We consider 41 Wikidata relations and subsample at most 1000 facts per relation. As with the Google-RE corpus, we manually define a template for each relation (see Table 3 for some examples). In contrast to the Google-RE knowledge source, T-REx facts were automatically aligned to Wikipedia and hence this alignment can be noisy. However, Elsahar et al. (2018) report an accuracy of 97.8% for the alignment technique over a test set.
Language Models as Knowledge Bases?
ConceptNet (Speer and Havasi, 2012) is a multilingual knowledge base, initially built on top of Open Mind Common Sense (OMCS) sentences. OMCS represents commonsense relationships between words and/or phrases. We consider facts from the English part of ConceptNet that have single-token objects covering 16 relations. For these ConceptNet triples, we find the OMCS sentence that contains both the subject and the object. We then mask the object within the sentence and use the sentence as template for querying language models. If there are several sentences for a triple, we pick one at random. Note that for this knowledge source there is no explicit alignment of facts to Wikipedia sentences.
Language Models as Knowledge Bases?
SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset. We select a subset of 305 context-insensitive questions from the SQuAD development set with single token answers. We manually create cloze-style questions from these questions, e.g., rewriting "Who developed the theory of relativity?" as "The theory of relativity was developed by ". For each question and answer pair, we know that the corresponding fact is expressed in Wikipedia since this is how SQuAD was created.
Language Models as Knowledge Bases?
We consider the following pretrained casesensitive language models in our study (see Table 1): fairseq-fconv (Fs), Transformer-XL large (Txl), ELMo original (Eb), ELMo 5.5B (E5B), BERT-base (Bb) and BERT-large (Bl). We use the natural way of generating tokens for each model by following the definition of the training objective function.
Language Models as Knowledge Bases?
Assume we want to compute the generation for the token at position t. For unidirectional language models, we use the network output (h t−1 ) just before the token to produce the output layer softmax. For ELMo we consider the output just before ( − → h t−1 ) for the forward direction and just after ( ← − h t+1 ) for the backward direction. Following the loss definition in (Peters et al., 2018a), we average forward and backward probabilities from the corresponding softmax layers. For BERT, we mask the token at position t, and we feed the output vector corresponding to the masked token (h t ) into the softmax layer. To allow a fair comparison, we let models generate over a unified vocabulary, which is the intersection of the vocabularies for all considered models (∼21K case-sensitive tokens).
Language Models as Knowledge Bases?
To compare language models to canonical ways of using off-the-shelf systems for extracting symbolic knowledge and answering questions, we consider the following baselines. Freq: For a subject and relation pair, this baseline ranks words based on how frequently they appear as objects for the given relation in the test data. It indicates the upper bound performance of a model that always predicts the same objects for a particular relation. RE: For the relation-based knowledge sources, we consider the pretrained Relation Extraction (RE) model of Sorokin and Gurevych (2017). This model was trained on a subcorpus of Wikipedia annotated with Wikidata relations. It extracts relation triples from a given sentence using an LSTMbased encoder and an attention mechanism. Based on the alignment information from the knowledge sources, we provide the relation extractor with the sentences known to express the test facts. Using these datasets, RE constructs a knowledge graph of triples. At test time, we query this graph by finding the subject entity and then rank all objects in the correct relation based on the confidence scores returned by RE. We consider two versions of this procedure that differ in how the entity linking is implemented: RE n makes use of a naïve entity linking solution based on exact string matching, while RE o uses an oracle for entity linking in addition to string matching. In other words, assume we query for the object o of a test subjectrelation fact (s, r, o) expressed in a sentence x. If RE has extracted any triple (s , r, o ) from that sen-tence x, s will be linked to s and o to o. In practice, this means RE can return the correct solution o if any relation instance of the right type was extracted from x, regardless of whether it has a wrong subject or object. DrQA: Chen et al. (2017) introduce DrQA, a popular system for open-domain question answering. DrQA predicts answers to natural language questions using a two step pipeline. First, a TF/IDF information retrieval step is used to find relevant articles from a large store of documents (e.g. Wikipedia). On the retrieved top k articles, a neural reading comprehension model then extracts answers. To avoid giving the language models a competitive advantage, we constrain the predictions of DrQA to single-token answers.
Language Models as Knowledge Bases?
We consider rank-based metrics and compute results per relation along with mean values across all relations. To account for multiple valid objects for a subject-relation pair (i.e., for N-M relations), we follow Bordes et al. (2013) and remove from the candidates when ranking at test time all other valid objects in the training data other than the one we test. We use the mean precision at k (P@k). For a given fact, this value is 1 if the object is ranked among the top k results, and 0 otherwise.
Language Models as Knowledge Bases?
There are several important design decisions we made when creating the LAMA probe. Below we give more detailed justifications for these decisions.
Language Models as Knowledge Bases?
Manually Defined Templates For each relation we manually define a template that queries for the object slot in that relation. One can expect that the choice of templates has an impact on the results, and this is indeed the case: for some relations we find both worse and better ways to query for the same information (with respect to a given model) by using an alternate template. We argue that this means we are measuring a lower bound for what language models know. We make this argument by analogy with traditional knowledge bases: they only have a single way of querying knowledge for a specific relation, namely by using the relation id of that relation, and this way is used to measure their accuracy. For example, if the relation ID is works-For and the user asks for is-working-for, the accuracy of the KG would be 0.
Language Models as Knowledge Bases?
Single Token We only consider single token objects as our prediction targets. The reason we include this limitation is that multi-token decoding adds a number of additional tuneable parameters (beam size, candidate scoring weights, length normalization, n-gram repetition penalties, etc.) that obscure the knowledge we are trying to measure. Moreover, well-calibrated multi-token generation is still an active research area, particularly for bidirectional models (see e.g. Welleck et al. (2019)).
Language Models as Knowledge Bases?
We choose to only query object slots in triples, as opposed to subject or relation slots. By including reverse relations (e.g. contains and contained-by) we can also query subject slots. We do not query relation slots for two reasons. First, surface form realisations of relations will span several tokens, and as we discussed above, this poses a technical challenge that is not in the scope of this work. Second, even if we could easily predict multi-token phrases, relations can generally be expressed with many different wordings, making it unclear what the gold standard pattern for a relation should be, and how to measure accuracy in this context.
Language Models as Knowledge Bases?
The models that we considered are trained with different vocabularies. For instance, ELMo uses a list of ∼800K tokens while BERT considers only ∼30K tokens. The size of the vocabulary can influence the performance of a model for the LAMA probe. Specifically, the larger the vocabulary the harder it would be to rank the gold token at the top. For this reason we considered a common vocabulary of ∼21K case-sensitive tokens that are obtained from the intersection of the vocabularies for all considered models. To allow a fair comparison, we let every model rank only tokens in this joint vocabulary.
Language Models as Knowledge Bases?
We summarize the main results in Table 2, which shows the mean precision at one (P@1) for the different models across the set of corpora considered. In the remainder of this section, we discuss the results for each corpus in detail.
Language Models as Knowledge Bases?
Google-RE We query the LMs using a standard cloze template for each relation. The base and large versions of BERT both outperform all other models by a substantial margin. Furthermore, they  obtain a 2.2 and 2.9 respective average accuracy improvement over the oracle-based RE baseline. This is particularly surprising given that with the gold-aligned Google-RE source we know for certain that the oracle RE baseline has seen at least one sentence expressing each test fact. Moreover, the RE baseline was given substantial help through an entity linking oracle.
Language Models as Knowledge Bases?
It is worth pointing out that while BERT-large does better, this does not mean it does so for the right reasons. Although the aligned Google-RE sentences are likely in its training set (as they are part of Wikipedia and BERT has been trained on Wikipedia), it might not "understand" them to produce these results. Instead, it could have learned associations of objects with subjects from co-occurrence patterns.
Language Models as Knowledge Bases?
The knowledge source derived from Google-RE contains relatively few facts and only three relations. Hence, we perform experiments on the larger set of facts and relations in T-REx. We find that results are generally consistent with Google-RE. Again, the performance of BERT in retrieving factual knowledge are close to the performance obtained by automatically building a knowledge base with an off-the-shelf relation extraction system and oracle-based entity linking. Broken down by relation type, the performance of BERT is very high for 1-to-1 relations (e.g., capital of ) and low for N-to-M relations.
Language Models as Knowledge Bases?
Note that a downstream model could learn to make use of knowledge in the output representations of a language model even if the correct answer is not ranked first but high enough (i.e. a hint about the correct answer can be extracted from the output representation). Figure 2 shows the mean P@k curves for the considered models. For BERT, the correct object is ranked among the top ten in around 60% of the cases and among the top 100 in 80% of the cases. To further investigate why BERT achieves such strong results, we compute the Pearson correlation coefficient between the P@1 and a set of metrics that we report in Figure 3. We notice, for instance, that the number of times an object is mentioned in the training data positively correlates with performance while the same is not true for the subject of a relation. Furthermore, the log probability of a prediction is strongly positively correlated with P@1. Thus, when BERT has a high confidence in its prediction, it is often correct. Performance is also positively correlated with the cosine similarity between subject and object vectors, and  Figure 3: Pearson correlation coefficient for the P@1 of the BERT-large model on T-REx and a set of metrics: SM and OM refer to the number of times a subject and an object are mentioned in the BERT training corpus 4 respectively; LPFP is the log probability score associated with the first prediction; SOCS is the cosine similarity between subject and object vectors (we use spaCy 5 ); ST and SWP are the number of tokens in the subject with a standard tokenization and the BERT WordPiece tokenization respectively.
Language Models as Knowledge Bases?
slightly with the number of tokens in the subject. Table 3 shows randomly picked examples for the generation of BERT-large for cloze template queries. We find that BERT-large generally predicts objects of the correct type, even when the predicted object itself is not correct.
Language Models as Knowledge Bases?
To understand how the performance of a pretrained language model varies with different ways of querying for a particular fact, we analyze a maximum of 100 random facts per relation for which we randomly select 10 aligned sentences in Wikipedia from  In each of the sentences, we mask the object of the fact, and ask the model to predict it. For several of our language models this also tests their ability to memorize and recall sentences from the training data since as the models have been trained on Wikipedia (see Table 1). Figure 4 shows the average distribution of the rank for ten queries per fact. The two BERT models and ELMo 5.5B exhibit the lowest variability while ranking the correct object close to the top on average. Surprisingly, the performance of ELMo original is not far from BERT, even though this model did not see Wikipedia during training. Fairseq-fconv and Transformer-XL experi-5 The original training corpus is not available, we created our version using the same sources. 5 https://spacy.io 6 We exclude all facts with less than 10 alignments. ence a higher variability in their predictions. Note that BERT and ELMo 5.5B have been trained on a larger portion of Wikipedia than fairseq-fconv and Transformer-XL and may have seen more sentences containing the test queries during training.
Language Models as Knowledge Bases?
The results on the ConceptNet corpus are in line with those reported for retrieving factual knowledge in Google-RE and T-REx. The BERT-large model consistently achieves the best performance, and it is able to retrieve commonsense knowledge at a similar level to factual knowledge. The lower half of Table 3 shows generations by BERT-large for randomly sampled examples. Some of the concepts generated by the language models are surprisingly reasonable in addition to being syntactically correct.
Language Models as Knowledge Bases?
SQuAD Next we evaluate our system on opendomain cloze-style question answering and compare against the supervised DrQA model. Table  2 shows a performance gap between BERT-large and the DrQA open-domain QA system on our cloze SQuAD task. Again, note that the pretrained language model is completely unsupervised, it is not fine-tuned, and it has no access to a dedicated information retrieval system. Moreover, when comparing DrQA and BERT-large in terms of P@10, we find that gap is remarkably small (57.1 for BERT-large and 63.5 for DrQA).
Language Models as Knowledge Bases?
We presented a systematic analysis of the factual and commonsense knowledge in publicly available pretrained language models as is and found Rome [-1.8] , Florence [-1.8] , Naples [-1.9] , Milan [-2.4] , Bologna [-2.5] P20
Language Models as Knowledge Bases?
Adolphe Adam died in . Paris Paris [-0.5] , London [-3.5] , Vienna [-3.6] , Berlin [-3.8] , Brussels [-4.0] P279
Language Models as Knowledge Bases?
English bulldog is a subclass of . dog dogs [-0.3] , breeds [-2.2] , dog [-2.4] , cattle [-4.3] , sheep [-4.5] P37
Language Models as Knowledge Bases?
The official language of Mauritius is . English English [-0.6] , French [-0.9] , Arabic [-6.2] , Tamil [-6.7] , Malayalam [-7.0] P413 Patrick Oboya plays in position. midfielder centre [-2.0] , center [-2.2] , midfielder [-2.4] , forward [-2.4] , midfield [-2.7
Language Models as Knowledge Bases?
Hamburg Airport is named after . Hamburg Hess [-7.0] , Hermann [-7.1] , Schmidt [-7.1] , Hamburg [-7.5] , Ludwig [-7.5] P364
Language Models as Knowledge Bases?
The original language of Mon oncle Benjamin is . French French [-0.2] , Breton [-3.3] , English [-3.8] , Dutch [-4.2] , German [-4.9] P54
Language Models as Knowledge Bases?
Dani Alves plays with . Barcelona Santos [-2.4] , Porto [-2.5] , Sporting [-3.1] , Brazil [-3.3] , Portugal [-3.7] P106 Paul Toungui is a by profession . politician lawyer [-1.1] , journalist [-2.4] , teacher [-2.7] , doctor [-3.0] , physician [-3.7] P527
Language Models as Knowledge Bases?
Sodium sulfide consists of . sodium water [-1.2] , sulfur [-1.7] , sodium [-2.5] , zinc [-2.8] , salt [-2.9] P102
Language Models as Knowledge Bases?
Gordon Scholes is a member of the political party. Labor Labour [-1.3] , Conservative [-1.6] , Green [-2.4] , Liberal [-2.9] , Labor [-2.9] P530
Language Models as Knowledge Bases?
Kenya maintains diplomatic relations with . Uganda India [-3.0] , Uganda [-3.2] , Tanzania [-3.5] , China [-3.6] , Pakistan [-3.6] P176
Language Models as Knowledge Bases?
iPod Touch is produced by . Apple
Language Models as Knowledge Bases?
Apple [-1.6] , Nokia [-1.7] , Sony [-2.0] , Samsung [-2.6] , Intel [-3.1] P30
Language Models as Knowledge Bases?
Bailey Peninsula is located in . Antarctica Antarctica [-1.4] , Bermuda [-2.2] , Newfoundland [-2.5] , Alaska [-2.7] , Canada [-3.1] P178 JDK is developed by . Oracle IBM [-2.0] , Intel [-2.3] , Microsoft [-2.5] , HP [-3.4] , Nokia [-3.5] P1412
Language Models as Knowledge Bases?
Carl III used to communicate in . Swedish German [-1.6] , Latin [-1.9] , French [-2.4] , English [-3.0] , Spanish [-3.0] P17 Sunshine Coast, British Columbia is located in . Canada Canada [-1.2] , Alberta [-2.8] , Yukon [-2.9] , Labrador [-3.4] , Victoria [-3.4] P39
Language Models as Knowledge Bases?
Pope Clement VII has the position of . pope cardinal [-2.4] , Pope [-2.5] , pope [-2.6] , President [-3.1] , Chancellor [-3.2] P264 Joe Cocker is represented by music label . Capitol EMI [-2.6] , BMG [-2.6] , Universal [-2.8] , Capitol [-3.2] , Columbia [-3.3] P276
Language Models as Knowledge Bases?
London Jazz Festival is located in . London
Language Models as Knowledge Bases?
London [-0.3] , Greenwich [-3.2] , Chelsea [-4.0] , Camden [-4.6] , Stratford [-4.8] P127
Language Models as Knowledge Bases?
Border TV is owned by . ITV Sky [-3.1] , ITV [-3.3] , Global [-3.4] , Frontier [-4.1] , Disney [-4.3] P103
Language Models as Knowledge Bases?
The native language of Mammootty is .
Language Models as Knowledge Bases?
Malayalam Malayalam [-0.2] , Tamil [-2.1] , Telugu [-4.8] , English [-5.2] , Hindi [-5.6] P495
Language Models as Knowledge Bases?
The Sharon Cuneta Show was created in . Philippines Manila [-3.2] , Philippines [-3.6] , February [-3.7] , December [-3.8] , Argentina [-4.0] ConceptNet AtLocation You are likely to find a overflow in a . drain sewer [-3.1] , canal [-3.2] , toilet [-3.3] , stream [-3.6] , drain [-3.6] CapableOf
Language Models as Knowledge Bases?
Ravens can . fly fly [-1.5] , fight [-1.8] , kill [-2.2] , die [-3.2] , hunt [-3.4] CausesDesire
Language Models as Knowledge Bases?
Joke would make you want to . laugh cry [-1.7] , die [-1.7] , laugh [-2.0] , vomit [-2.6] , scream [-2.6] Causes Sometimes virus causes . infection disease [-1.2] , cancer [-2.0] , infection [-2.6] , plague [-3.3] , fever [-3.4] HasA
Language Models as Knowledge Bases?
Birds have . feathers wings [-1.8] , nests [-3.1] , feathers [-3.2] , died [-3.7] , eggs [-3.9] HasPrerequisite
Language Models as Knowledge Bases?
Typing requires . speed patience [-3.5] , precision [-3.6] , registration [-3.8] , accuracy [-4.0] , speed [-4.1] HasProperty
Language Models as Knowledge Bases?
Time is . finite short [-1.7] , passing [-1.8] , precious [-2.9] , irrelevant [-3.2] , gone [-4.0] MotivatedByGoal You would celebrate because you are . alive happy [-2.4] , human [-3.3] , alive [-3.3] , young [-3.6] , free [-3.9] ReceivesAction Skills can be . taught acquired [-2.5] , useful [-2.5] , learned [-2.8] , combined [-3.9] , varied [-3.9] UsedFor A pond is for . fish swimming [-1.3] , fishing [-1.4] , bathing [-2.0] , fish [-2.8] , recreation [-3.1]  that BERT-large is able to recall such knowledge better than its competitors and at a level remarkably competitive with non-neural and supervised alternatives. Note that we did not compare the ability of the corresponding architectures and objectives to capture knowledge in a given body of text but rather focused on the knowledge present in the weights of existing pretrained models that are being used as starting points for many researchers' work. Understanding which aspects of data our commonly-used models and learning algorithms are capturing is a crucial field of research and this paper complements the many studies focused on the learned linguistic properties of the data.
Language Models as Knowledge Bases?
We found that it is non-trivial to extract a knowledge base from text that performs on par to directly using pretrained BERT-large. This is despite providing our relation extraction baseline with only data that is likely expressing target facts, thus reducing potential for false negatives, as well as using a generous entity-linking oracle. We suspected BERT might have an advantage due to the larger amount of data it has processed, so we added Wikitext-103 as additional data to the relation extraction system and observed no significant change in performance. This suggests that while relation extraction performance might be difficult to improve with more data, language mod-els trained on ever growing corpora might become a viable alternative to traditional knowledge bases extracted from text in the future.
Language Models as Knowledge Bases?
In addition to testing future pretrained language models using the LAMA probe, we are interested in quantifying the variance of recalling factual knowledge with respect to varying natural language templates. Moreover, assessing multi-token answers remains an open challenge for our evaluation setup.
Generalization through Memorization: Nearest Neighbor Language Models
Neural language models (LMs) typically solve two subproblems: (1) mapping sentence prefixes to fixed-sized representations, and (2) using these representations to predict the next word in the text (Bengio et al., 2003;Mikolov et al., 2010). We present a new language modeling approach that is based on the hypothesis that the representation learning problem may be easier than the prediction problem. For example, any English speaker knows that Dickens is the author of and Dickens wrote will have essentially the same distribution over the next word, even if they do not know what that distribution is. We provide strong evidence that existing language models, similarly, are much better at the first problem, by using their prefix embeddings in a simple nearest neighbor scheme that significantly improves overall performance.
Generalization through Memorization: Nearest Neighbor Language Models
We introduce kNN-LM, an approach that extends a pre-trained LM by linearly interpolating its next word distribution with a k-nearest neighbors (kNN) model. The nearest neighbors are computed according to distance in the pre-trained embedding space and can be drawn from any text collection, including the original LM training data. This approach allows rare patterns to be memorized explicitly, rather than implicitly in model parameters. It also improves performance when the same training data is used for learning the prefix representations and the kNN model, strongly suggesting that the prediction problem is more challenging than previously appreciated.
Generalization through Memorization: Nearest Neighbor Language Models
To better measure these effects, we conduct an extensive empirical evaluation. Applying our kNN augmentation to a strong WIKITEXT-103 LM using only the original dataset achieves a new stateof-the-art perplexity of 15.79 -a 2.86 point improvement over the base model (Baevski & Auli, 2019) -with no additional training. We also show that the approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore. Training a model on 100-million tokens and using kNN search over a 3-billion token dataset can outperform training the same model on all 3-billion tokens, opening a  Figure 1: An illustration of kNN-LM. A datastore is constructed with an entry for each training set token, and an encoding of its leftward context. For inference, a test context is encoded, and the k most similar training contexts are retrieved from the datastore, along with the corresponding targets. A distribution over targets is computed based on the distance of the corresponding context from the test context. This distribution is then interpolated with the original model's output distribution.
Generalization through Memorization: Nearest Neighbor Language Models
new path for efficiently using large datasets in language models. Similarly, adding out-of-domain data to the datastore makes a single LM useful across multiple domains, again without further training. Qualitatively, we find the model is particularly helpful for long-tail patterns, such as factual knowledge, which might be easier to access via explicit memory.
Generalization through Memorization: Nearest Neighbor Language Models
Language models (LMs) assign probabilities to sequences. Given a context sequence of tokens c t = (w 1 , . . . w t−1 ), autoregressive LMs estimate p(w t |c t ), the distribution over the target token w t .
Generalization through Memorization: Nearest Neighbor Language Models
The kNN-LM involves augmenting such a pre-trained LM with a nearest neighbors retrieval mechanism, without any additional training (the representations learned by the LM remain unchanged). This can be done with a single forward pass over a text collection (potentially including the original LM training set), where the resulting context-target pairs are stored in a key-value datastore that is queried during inference, as illustrated in Figure 1.
Generalization through Memorization: Nearest Neighbor Language Models
Datastore Let f (·) be the function that maps a context c to a fixed-length vector representation computed by the pre-trained LM. For instance, in a Transformer LM, f (c) could map c to an intermediate representation that is output by an arbitrary self-attention layer. Then, given the i-th training example (c i , w i ) ∈ D, we define the key-value pair (k i , v i ), where the key k i is the vector representation of the context f (c i ) and the value v i is the target word w i . The datastore (K, V) is thus the set of all key-value pairs constructed from all the training examples in D:
Generalization through Memorization: Nearest Neighbor Language Models
Inference At test time, given the input context x the model generates the output distribution over next words p LM (y|x) and the context representation f (x). The model queries the datastore with f (x) to retrieve its k-nearest neighbors N according to a distance function d(·, ·) (squared L 2 distance in our experiments, making the similarity function an RBF kernel).Then, it computes a distribution over neighbors based on a softmax of their negative distances, while aggregating probability mass for each vocabulary item across all its occurrences in the retrieved targets (items that do not appear in the retrieved targets have zero probability):
Generalization through Memorization: Nearest Neighbor Language Models
Finally, we follow Grave et al. (2017a) and interpolate the nearest neighbor distribution p kNN with the model distribution p LM using a tuned parameter λ to produce the final kNN-LM distribution:
Generalization through Memorization: Nearest Neighbor Language Models
Implementation The datastore contains an entry for each target in the training set, which for LMs can be up to billions of examples. To search over this large datastore, we use FAISS (Johnson et al., 2017), an open source library for fast nearest neighbor retrieval in high dimensional spaces. FAISS speeds up search by clustering the keys and looking up neighbors based on the cluster centroids, while reducing memory usage by storing compressed versions of the vectors. We found in preliminary experiments that using L 2 distance for FAISS retrieval results in better performance for kNN-LM, compared to inner product distance.
Generalization through Memorization: Nearest Neighbor Language Models
Related Cache Models Prior work (Grave et al., 2017c;Merity et al., 2017) used a similar approach to compute similarity to the previous hidden states of test documents, making it easier to copy rare vocabulary items from the recent past. Such techniques have been less popular since the development of Transformers (Vaswani et al., 2017), which can learn to copy recent words using self-attention; in Section 4.1, we observe relatively small gains from caching recent items in the same test documentà la Grave et al. (2017c). Most relatedly, Grave et al. (2017a) describe an online language model using nearest neighbor search over all previous hidden states, to improve domain adaptation. In our work, we only save training data, with the goal of explicitly memorizing training examples to better generalize to similar cases at test time.
Generalization through Memorization: Nearest Neighbor Language Models
Data Experiments in this paper use the following English corpora: Except for WIKITEXT-103, text is tokenized using the byte-pair encoding (Sennrich et al., 2015) with the 29K subword vocabulary from BERT (Devlin et al., 2019).
Generalization through Memorization: Nearest Neighbor Language Models
Model Architecture kNN-LM is compatible with any model that produces fixed size context representations. We use decoder-only Transformers (Vaswani et al., 2017) for language modeling, which are the current state of the art. Since the kNN-LM makes no changes to the underlying LM, we take the exact architecture and optimization described by Baevski & Auli (2019)  Evaluation LMs are trained to minimize the negative log-likelihood of the training corpus, and evaluated by perplexity (exponentiated negative log-likelihood) on held out data. Following Baevski & Auli (2019), 512 tokens are scored per test example, but up to 2560 tokens of extra prior context is provided for WIKITEXT-103 and up to 512 tokens of extra prior context is provided for the rest of the corpora.
Generalization through Memorization: Nearest Neighbor Language Models
kNN-LM The keys used for kNN-LM are the 1024-dimensional representations fed to the feedforward network in the final layer of the Transformer LM (after self-attention and layernorm; see Section 5 for further explanation). We perform a single forward pass over the training set with the trained model, in order to save the keys and values. During this forward pass, each target token is provided a minimum of 1536 tokens of prior context for WIKITEXT-103 and a minimum of 512 We first experiment with creating a datastore from the same data used to train the LM. Table 1 shows that kNN-LM improves perplexity on WIKITEXT-103 from 18.65 (Baevski & Auli, 2019) to a new state-of-the-art of 16.12. We also provide reported perplexities from two other recent models that also build upon Baevski and Auli's, suggesting that further improvements may be possible by augmenting the kNN-LM with these techniques. We compare with models trained only on the standard training set, but recent work has shown performance can be improved by training on additional data, from either the test set (Krause et al., 2019) or large amounts of web text (Shoeybi et al., 2019).
Generalization through Memorization: Nearest Neighbor Language Models
We also experiment with a continuous cache model, a related but orthogonal technique from Grave et al. (2017c), in which the model saves and retrieves neighbors from earlier in the test document,    rather than the training set. Gains from interpolating with the continuous cache are smaller than reported in the original setting that used LSTMs, perhaps because self-attentive language models can learn to perform such queries. Improvements from the continous cache are additive with the kNN-LM, pushing our state-of-the-art result to 15.79, a gain of 2.86 over the base model.
Generalization through Memorization: Nearest Neighbor Language Models
Finally, we repeat the experiment using text from a different domain, BOOKS, to control for the possibility that encyclopedic Wikipedia text is somehow uniquely good for caching. Table 2 shows an improvement in test set perplexity from 11.89 to 10.89, suggesting that this is not the case.   Table 5: WIKITEXT-103 validation results using different states from the final layer of the LM as the representation function f (·) for keys and queries. We retrieve k=1024 neighbors and λ is tuned for each.
Generalization through Memorization: Nearest Neighbor Language Models
To understand how the amount of data used for kNN retrieval affects performance, we use the WIKI-100M model to create datastores using different amounts of randomly sampled data from WIKI-3B. Figure 2a shows that using only 1.6B examples for the datastore already surpasses the performance of the model trained on all of WIKI-3B. In addition, performance does not saturate at 3B examples in the datastore, suggesting that growing the datastore more could lead to further gains. Figure 2b shows the model relies more on the kNN component as the size of the datastore increases.
Generalization through Memorization: Nearest Neighbor Language Models
We also experiment with domain adaptation by creating a datastore on the target domain training set. Table 4 shows that an in-domain LM on BOOKS has a relatively low perplexity (11.89), while a model trained on WIKI-3B performs poorly on the BOOKS domain (34.84 perplexity). Adding kNN search over BOOKS to the WIKI-3B model reduces perplexity by 14 points (to 20.47), demonstrating that kNN-LM allows a single model to be useful in multiple domains, by simply adding a datastore per domain.
Generalization through Memorization: Nearest Neighbor Language Models
While the kNN-LM is conceptually straightforward, and requires no additional training, a number of hyperparameters are introduced for nearest neighbor search. We experiment with different choices here.
Generalization through Memorization: Nearest Neighbor Language Models
Key Function For similarity search, we extract a representation of context c using an intermediate state of the LM f (c). Transformers compute a number of different intermediate states, and we compare several choices depicted in Figure 3, with results shown in Table 5. While all the instantiations of f we tried are helpful, we achieved the largest improvement by using the input to the final layer's feedforward network. We also observe that normalized representations (i.e. taken immediately after the layer norm) perform better. Repeating the experiment on the second-last transformer layer showed similar trends with slightly worse results (not shown), suggesting that the feedforward layer might be focusing more on the prediction problem, while the onus of representing the input falls more on the self-attention layer. 
Generalization through Memorization: Nearest Neighbor Language Models
Each query returns the top-k neighbors. Figure 4 shows that performance monotonically improves as more neighbors are returned, and suggests that even larger improvements may be possible with a higher value of k. Nonetheless, even a small number of neighbors (k = 8) is enough to achieve a new state of the art.
Generalization through Memorization: Nearest Neighbor Language Models
Interpolation Parameter We use a parameter λ to interpolate between the base model distribution and the distribution from kNN search over the dataset. Figure 5 shows that λ = 0.25 is optimal on WIKITEXT-103. However, λ = 0.65 works best for domain adaptation results ( Figure 5).
Generalization through Memorization: Nearest Neighbor Language Models
In FAISS, the nearest neighbor search computes L 2 distances against quantized keys. We found results were improved from 16.5 perplexity on WIKITEXT-103 to 16.06 by computing squared L 2 distances with full precision keys for Equation 2.
Generalization through Memorization: Nearest Neighbor Language Models
Qualitative Analysis To understand why kNN-LM improves performance, we manually examine cases in which p kNN was significantly better than p LM . Table 6 shows one such example, along with several others in Appendix A. The example shows an interesting case where the model matches the trigram impact on the in several retrieved neighbors, but puts almost all weight on the most relevant neighbor, thus adding more value than an n-gram LM.
Generalization through Memorization: Nearest Neighbor Language Models
In general, we find that examples where kNN-LM is most helpful typically contain rare patterns. Examples include factual knowledge, names, and near-duplicate sentences from the training set. In these cases, assigning train and test instances similar representations (via f (·)) appears to be an easier problem than implicitly memorizing the next word in model parameters.
Generalization through Memorization: Nearest Neighbor Language Models
We observe that many long-tail phenomena manifest as rare n-grams (e.g. names). Is it therefore possible to interpolate an n-gram model with a Transformer LM, as an alternative to our kNN approach? Figure 7 shows little improvement from using n-gram LMs -0.2 perplexity points (similarly to Bakhtin et al. (2018)). This result highlights the need to use the learned representation function f (·) to measure similarity between more varied contexts.
Generalization through Memorization: Nearest Neighbor Language Models
If a neural representation function is crucial for kNN-LM, could implicitly memorizing the training dataset in the neural network parameters replace the explicit memory in the datastore? To test this, we train a Transformer LM with no dropout. Figure 8 shows that this model eventually reaches zero training loss, indicating that it can make perfect predictions for all examples in the training set; the model has memorized the dataset. Naturally, the memorizing LM overfits, i.e. the training loss drops to 0 while the best validation perplexity is much higher at 28.59. For comparison, the vanilla Transformer LM (with dropout) has a much higher training loss (shown in Figure 8), but also generalizes better with a validation perplexity of 17.96. This result shows that the Transformer has sufficient capacity to memorize the training set.
Generalization through Memorization: Nearest Neighbor Language Models
Test Context (pkNN = 0.998, pLM = 0.124) Test Target it was organised by New Zealand international player Joseph Warbrick, promoted by civil servant Thomas Eyton, and managed by James Scott, a publican. The Natives were the first New Zealand team to perform a haka, and also the first to wear all black. They played 107 rugby matches during the tour, as well as a small number of Victorian Rules football and association football matches in Australia. Having made a significant impact on the...
Generalization through Memorization: Nearest Neighbor Language Models
Training Set Context Training Set Target
Generalization through Memorization: Nearest Neighbor Language Models
As the captain and instigator of the 1888-89 Natives -the first New Zealand team to tour the British Isles -Warbrick had a lasting impact on the... We consider whether the memorizing LM can be an effective substitute for nearest neighbor search.
Generalization through Memorization: Nearest Neighbor Language Models
Interpolating the memorizing LM with the original LM improves validation perplexity by just 0.1 -compared to 1.9 from kNN-LM. This result suggests that although the Transformer is expressive enough to memorize all training examples, learning to do so does not result in context representations that generalize. In contrast, kNN-LM memorizes training data while improving generalization.
Generalization through Memorization: Nearest Neighbor Language Models
From these experiments, we conjecture that kNN-LM improves performance because (1) the Transformer LM is very good at learning a representation function for contexts with an implicit notion of similarity, and (2) while the Transformer has capacity to memorize all training examples, doing so causes its representation to generalize less effectively, but (3) the kNN-LM allows the model to memorize the training data while retaining an effective similarity function.
Generalization through Memorization: Nearest Neighbor Language Models
We discuss related uses of caches for language modeling in Section 2.
Generalization through Memorization: Nearest Neighbor Language Models
Similar kNN models to ours have been proposed for computer vision tasks (Papernot & McDaniel, 2018;Orhan, 2018;Zhao & Cho, 2018), primarily motivated by improving interpretability and robustness to adversarial attacks. We hypothesize that our method may be particularly effective for language modeling, because plentiful unlabeled data allows datastores of billions of tokens, and language modeling often requires world knowledge to be learnt from few examples.
Generalization through Memorization: Nearest Neighbor Language Models
Nearest neighbor models have been applied to a number of NLP problems in the past, such as part of speech tagging (Daelemans et al., 1996) and morphological analysis (Bosch et al., 2007), but the use of learned representations makes the similarity function much more effective in the case of neural models. More recently,  have used a similarly differentiable memory that is learned and updated during training, and is applied to one-shot learning tasks.
Generalization through Memorization: Nearest Neighbor Language Models
Several models have also improved language generation by using training examples directly at test time. Guu et al. (2018) propose a model that samples training sentences at random and edits them with a sequence-to-sequence model, but does not use a retrieval mechanism such as kNN. Gu et al. (2018) introduce a translation model that attends over retrieved training set examples. Weston et al. (2018) improve a dialogue response generation model by refining similar instances from the training set. kNN-LM differs from these approaches by working at the level of individual tokens instead of whole training sentences, as well as not incorporating the retrieval mechanism into the training pipeline.
Generalization through Memorization: Nearest Neighbor Language Models
A general trend in machine learning, and in language modeling in particular, is that adding more data consistently improves performance (Devlin et al., 2019;Radford et al., 2019;Liu et al., 2019;Zellers et al., 2019;Shoeybi et al., 2019). Our work offers an alternative method for scaling language models, in which relatively small models learn context representations, and a nearest neighbour search acts as a highly expressive classifier.
Generalization through Memorization: Nearest Neighbor Language Models
We have introduced kNN-LMs, which can significantly outperform standard language models by directly querying training examples at test time. The approach can be applied to any neural language model. The success of this method suggests that learning similarity functions between contexts may be an easier problem than predicting the next word from some given context. Future work should explore explicitly training similarity functions, and reducing the size of the datastore.
Generalization through Memorization: Nearest Neighbor Language Models
This section provides several examples where p kNN places higher probability mass on the true target, compared to p LM .    Mycena maculata bears some resemblance to M. <unk>, but is only associated with decaying hardwood logs and stumps, and is found in eastern North America, and sometimes on oak on the West Coast. In age, it...
Generalization through Memorization: Nearest Neighbor Language Models
Training Set Context Training Set Target
Generalization through Memorization: Nearest Neighbor Language Models
Morchella tridentina (=Morchella frustrata) is also rufescent and very similar to M. rufobrunnea. It is found in mountainous forests and maquis and forms a marked sinus at the attachment of the cap with the stem, which is pure white. At maturity, it...
Generalization through Memorization: Nearest Neighbor Language Models
The winter bonnet (M. tintinnabulum) is a northern European species that is much smaller (cap diameter up to 2.6 cm (1.0 in) across) and has a brown cap, and has ragged hairs at the base. It... generally 0.029
Generalization through Memorization: Nearest Neighbor Language Models
The "bleeding" will distinguish Mycena atkinsoniana from most other Mycena species commonly encountered. The common and widely distributed M. sanguinolenta is another "bleeder", but it is smaller than M. atkinsonia, with a cap diameter ranging from 3 to 15 mm (0.1 to 0.6 in).  Table 9: This is an example where the p kNN distribution is relatively flat, as several words are plausible continuations. However, the nearest neighbors search assigns the highest probability to the correct target and a corresponding context that is particularly relevant. In contrast, the LM probability on the correct target is lower.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Large language models (LLMs) are impressive few-shot learners Rae et al., 2021;Hoffmann et al., 2022;Chowdhery et al., 2022). They are able to learn new tasks with very few examples or even from instructions alone. For this generalisation ability to emerge, the key ingredients are scaling both the parameter count of the model, and the size of the training data. Large language models owe this improvement to both a larger computational budget, enabling more complex reasoning, and the ability to memorize more information related to downstream tasks from the larger training data. While it is intuitive to assume that increased reasoning abilities lead to better generalisation, and hence few-shot learning, the same is not true for in-parameter memorisation. Specifically, it is unclear to what extent effective few-shot learning requires vast knowledge in the parameters of the model.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this paper, we investigate whether few-shot learning requires models to store a large amount of information in their parameters, and if memorisation can be decoupled from generalisation. To do so, we leverage the fact that memory can be outsourced and replaced by an external non-parametric knowledge source by employing a retrieval-augmented architecture. These models employ a non-parametric memory, e.g. a neural retriever over a large, external, potentially non-static knowledge source to enhance a parametric language model. In addition to their memorisation abilities, such architectures are attractive due to a number of other established advantages in terms of adaptability, interpretability and efficiency (Guu et al., 2020;Borgeaud et al., 2021, inter alia). However, retrieval-augmented models have yet to … … Figure 1: We introduce Atlas, a retrieval-augmented language model that exhibits strong few-shot performance on knowledge tasks, and uses retrieval during both pre-training and fine-tuning.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
demonstrate compelling few-shot learning capabilities. In this work we address this gap, and present Atlas, a retrieval-augmented language model capable of strong few-shot learning, despite having lower parameter counts than other powerful recent few-shot learners.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Atlas retrieves relevant documents based on the current context by using a general-purpose dense retriever using a dual-encoder architecture, based on the Contriever . The retrieved documents are processed, along with the current context, by a sequence-to-sequence model using the Fusion-in-Decoder architecture ) that generates the corresponding output. We study the impact of different techniques to train Atlas on its few-shot performance on a range of downstream tasks, including question answering and fact checking. We find that jointly pre-training the components is crucial for few-shot performance, and we carefully evaluate a number of existing and novel pre-training tasks and schemes for this purpose. Atlas achieves strong downstream performance in both few-shot and resource-rich settings. For example, with only 11B parameters, Atlas achieves an accuracy of 42.4% on NaturalQuestions using 64 training examples (45.1% with a Wikipedia-only index), outperforming PaLM (Chowdhery et al., 2022), a 540B parameter model by almost 3 points, and 64.0% in a full-dataset setting with a Wikipedia index, establishing a new state of the art by 8 points.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In summary we make the following contributions:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• A thorough study on how to design and train retrieval-augmented language models, with a focus on downstream few-shot learning and sample efficiency.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• The findings of this study lead to a retrieval-augmented language model, called Atlas, that exhibits few-shot abilities that emerge at lower scale than standard LLM.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• We provide an exploration of fine-tuning strategies to efficiently adapt both the retriever and the language model to the task at hand.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Thorough downstream experiments in few-shot settings, demonstrating state-of-the-art results on few-shot NaturalQuestions (+2.8%), TriviaQA (+3.3%), FEVER (+5.1%), and results on par or stronger than models with 15× more parameters on MMLU.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Experiments investigating full-dataset finetuning, setting new state-of-the-art results in NaturalQuestions (+8.1%), TriviaQA (+9.3%) and 5 KILT Tasks.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Experiments demonstrating the updatability and interpretability characteristics of Atlas.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Experiments demonstrating that a compressed index using product quantisation achieves comparable performance as an uncompressed index while resulting in a 5x memory reduction.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Our code, pretrained Atlas checkpoints, and various supporting data are available at https://github.com/ facebookresearch/atlas
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this section, we discuss four different loss functions to train the retriever jointly with the language model. We consider loss functions that leverage the language model to provide supervisory signal to train the retriever. In other words, if the language model finds a document useful when generating the output, the retriever objective should encourage the retriever to rank said document higher. This allows us to train models using only query and output pairs from the task of interest, without relying on document annotations. For example, in the case of fact checking, a model only requires pairs of claims and corresponding verdicts but no documents containing the evidence to back up the verdict. In practice, we can apply this approach on any task, including self-supervised pre-training. As shown in the experimental section, pre-training is critical for obtaining models that exhibit few-shot learning abilities.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Attention Distillation (ADist). The first loss that we consider is based on the attention scores of the language model, and is heavily inspired by . The main idea is that the cross-attention scores between the input documents and the output, can be used as a proxy of the importance of each input document when generating the output. In particular,  showed that these scores can be aggregated across attention heads, layers and tokens for a given document to obtain a single score for each document. Then, these scores can be distilled into the retriever by minimizing the KL-divergence with the probability distribution p retr over the top-K documents {d k } 1,...,K obtained from the retriever:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
where s is the dot-product between the query and documents vectors and θ is a temperature hyper-parameter.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In the original paper, it was proposed to use the pre-softmax scores from the decoder cross-attentions, and average across heads, layers and tokens. Here, we propose an alternative which gives slightly stronger results, which relies on the following observation. In the attention mechanism, as defined by y = N n=1 α n v n , the contribution to the output y of a particular token n cannot be evaluated from the attention score α n alone, but should also take the norm of the value v n into account. Hence, we use the quantity α n v n 2 as the measure of relevance for token n. Following , we average these scores over all attention heads, layers, and tokens to obtain a score for each document. We apply the Softmax operator over the resulting scores, to obtain a distribution p attn (d k ) over the top-K retrieved documents. We then minimize the KL-divergence between p attn (d k ), and the distribution p retr from the retriever defined in Equation 1:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Here, this loss is only used to optimize the parameters of the retriever, and not the language model. When using recent deep learning frameworks, this is achieved by applying a StopGradient operator on p attn .
Atlas: Few-shot Learning with Retrieval Augmented Language Models
End-to-end training of Multi-Document Reader and Retriever (EMDR 2 ). Next, we consider the method introduced by Sachan et al. (2021), which is inspired by the expectation-maximization algorithm, treating retrieved documents as latent variables. Given a query q, the corresponding output a and the set D K of top-K retrieved documents with the current retriever, the EMDR 2 loss to train the retriever is
Atlas: Few-shot Learning with Retrieval Augmented Language Models
where p retr is again the probability over the top-K documents obtained with the retriever, as defined by Equation 1. Again, only the parameters of the retriever are updated by applying a StopGradient operator around p lm . One should note that the probability distribution over documents that maximizes this loss function is an indicator of the document corresponding to the highest probability of the output according to the language model. Finally, in practice, the EMDR 2 loss function is applied at the token level, and not at the sequence level.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Perplexity Distillation (PDist). Third, we discuss a simpler loss function which is loosely inspired by the objectives from the attention distillation and EMDR 2 methods Sachan et al., 2021). More precisely, we want to train the retriever to predict how much each document would improve the language model perplexity of the output, given the query. To this end, we minimize the KL-divergence between the documents distribution of the retriever (Eqn. 1), and the documents posterior distribution according to the language model, using a uniform prior:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Using the Softmax operator, we have that
Atlas: Few-shot Learning with Retrieval Augmented Language Models
.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Leave-one-out Perplexity Distillation (LOOP). Finally, we propose an objective based on how much worse the prediction of the language model gets, when removing one of the top-k retrieved documents. To do so, we compute the log probability of the output for each subset of k-1 documents, and use the negative value as relevance score for each document. Following the previous loss function, we use the softmax operator to obtain a probability distribution over documents:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
As before, we then minimize the KL-divergence between this distribution, and the one obtained with retriever. This loss is more expensive to compute than PDist and EMDR, but, like ADist, employs the language model more closely to the way it is trained i.e. the LM is trained to be conditioned on a set of K documents. For LOOP, the language model is conditioned on (K − 1) documents, rather than a single document as in EMDR 2 and PDist.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For all losses, we can also use a temperature hyper-parameter when computing the target or retriever distributions to control the distribution's peakiness of, which might be important for some tasks or losses. Indeed, for PDist and LOOP, the perplexity of the output may not vary much when conditioning on different documents, especially in the case of long outputs.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this section, we describe pretext tasks that can be used to jointly pre-train the retriever and the language model using only unsupervised data.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Prefix language modeling. First, we consider a standard language modeling task as potential pre-training objective. To cast language modeling in the text-to-text framework, we consider a chunk of N words, and split this chunk in two sub-sequences of equal length N/2. Then, the first sub-sequence is used as the query, and the second corresponds to the output. We thus retrieve relevant documents by using the first sub-sequence of N/2 tokens, to generate the output.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Masked language modeling. Second, we consider masked language modeling, as formulated by Raffel et al. (2019). Again, starting from a chunk of N words, we sample k spans of average length 3 tokens, leading to a masking ratio of 15%. We then replace each span by a different special token. The model is then trained to generate the masked spans, each span beginning with the special sentinel mask token that was inserted in the input sequence. We retrieve documents using the masked query, but replace the special mask tokens with a mask token supported by the retriever vocabulary.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Title to section generation. Finally, we consider a more abstractive generation task, generating sections from Wikipedia articles, given the article and section title. Here, the query corresponds to the title of the article, together with the title of the section, and the output corresponds to the text of the section. We exclude sections "See also", "References", "Further reading" and "External links".
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Retrieval is facilitated by using a document index, which is a pre-computed collection of the document embeddings for all the documents in the retrieval corpus. When jointly training the retriever and language model, the index needs to be updated regularly, otherwise, the embeddings of the documents stored in the index become stale relative to the updated retriever. This means that we need to recompute the embeddings for the full collection of documents regularly during training to keep the index fresh, which can be computationally expensive for large indices. This is particularly true at fine-tuning time, where the number of training examples could be small relative to the number of documents in the index. Training the retriever could thus add an important computational overhead compared to standard language model finetuning. In this section, we analyse strategies that might make this process more efficient, alleviating the need to re-compute the embeddings of all the documents too often.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Full index update. Let us start by analysing the overhead due to updating the index, compared to using a fixed retriever. To compare the computation time of different models, we will make the following assumption: the time required to perform a forward pass on a document with a model of P parameters is O(P ). While this computation model may seem naive, the main assumption is that document sizes are constant. 1 Since we split long documents into passages with similar number of words, and use padding when processing documents of different sizes, this assumption is reasonable in practice. Let K be the number of documents that are retrieved and processed by the language model, P lm be the number of parameters of the language model and B the batch size. Each training step has a complexity of 4 × B × K × P lm . 2
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Next, let N be the number of documents in the index, and P retr be the number of parameters of the retriever. Then, re-computing the full index has a complexity of N × P retr . If we refresh the index every R training steps, we obtain the following overhead:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
If we use the BERT-base architecture for our retriever and T5-XL for our language model, we get Pretr Plm ≈ 1 25 , lading to the overhead:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
If we use an index containing 37M documents (the size of our Wikipedia index), train with a batch size of 64 with 20 retrieved documents and refresh the index every 1000 steps, this results in an overhead of ∼ 30%.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
A second strategy is to retrieve a larger number of documents L with the retriever, and to re-embed and rerank these documents with the up-to-date retriever, and pass the resulting top-K to the language model. In that case, the overhead of reranking the top-L documents is equal to B × L × P retr . Since we perform this operation at every time step, the overhead is equal to
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Using the same assumption as before, we finally get that the overhead is of the order of L 100×K . If we re-rank 10x more documents than what the language model processes (i.e., L = 10 × K), we get an overhead of 10%. However, note that if many updates are performed on the retriever, the index might still need to be fully updated, as the true top-k documents may not be retrieved in the top-L results from the stale index. In practice, it is possible to track the positions of the top-K re-ranked documents in the top-L, and estimate when the index needs to be updated.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Query-side fine-tuning. Finally, the last strategy is to decouple the encoding of the queries and documents. In this case, we fix the parameters corresponding to the document encoder, and only train the parameters corresponding to the query encoder. Thus, the embeddings of documents are fixed, and we do not need to refresh the index, and thus there is no computational overhead. As we will see in practice, the impact of fixing the documents encoder varies greatly for different tasks when a large training dataset is available. For most of the few-shot settings that we consider, query-side finetuning does not have large performance impact, and sometimes even slightly improves performance.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Retrieval for knowledge intensive tasks. Previous work has shown that retrieval improves performance across a variety of tasks such as question answering (Voorhees et al., 1999;Chen et al., 2017;Kwiatkowski et al., 2019), fact checking (Thorne et al., 2018), dialogue (Dinan et al., 2019) or citation recommendation . Historically, this information retrieval step was implemented using term-matching methods, such as TF-IDF or BM25 (Jones, 1972;Robertson et al., 1995). For open-domain question answering (Voorhees et al., 1999), documents are often retrieved from Wikipedia (Chen et al., 2017). Recently, dense retrievers based on neural networks have become popular. These usually follow a dual-encoder architecture (Yih et al., 2011;Huang et al., 2013;Shen et al., 2014), where queries and passages are encoded independently as vectors, and relevance is computed using the inner product or Euclidean distance. Popular supervised retrievers include DPR , which is trained to discriminate the relevant passage among negative passages, and extensions such as ANCE (Xiong et al., 2020) which improved the hard negatives mining process. We refer the reader to Yates et al. (2021) for a survey of dense retrieval techniques.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
After retrieval, the relevant documents are processed to produce the final output. In open-domain QA, models can extract a span of text from retrieved documents as the answer (Chen et al., 2017;Clark & Gardner, 2018;Wang et al., 2019;, a method inspired by reading comprehension (Richardson, 2013;Rajpurkar et al., 2016). Recently, generating the answer as free-form text, using a seq2seq model conditioned on retrieved documents have become prevalent . These architectures have also been shown to reduce hallucination in dialogue agents (Shuster et al., 2021).
Atlas: Few-shot Learning with Retrieval Augmented Language Models
The need for expensive query-document annotations for training the retriever can be bypassed, by leveraging signals from the language model, or using unsupervised learning. REALM (Guu et al., 2020) and RAG  jointly train the retriever and language model by modelling documents as latent variable, and minimizing the objective with gradient descent. REALM pre-trains end-to-end with an MLM approach but uses an extractive BERT-style model . Guu et al. (2020) also explore a query-side finetuning at finetuning time to avoid index refreshes, which is also explored in the context of phrase-based retrieval by Lee et al. (2021b).  proposed to use cross-attention scores as supervision with knowledge distillation. Sachan et al. (2021) perform joint training of the reader and the retriever by leveraging the perplexity of the output generated by the reader. Sachan et al. (2021) and Lee et al. (2021a) both employ salient span masking to pre-train retrievers, leveraging the perplexity and attention scores from the language model. The inverse cloze task was proposed by  to pre-train dense retrievers in an unsupervised way. Paranjape et al. (2021) propose a method to train retrieval-augmented generators using a second "informed" retriever with access to the output, which the test-time retriever can be distilled from, and Hofstätter et al. (2022) recently proposed a training set filtering/weighting approach to train stronger retrieval-augmented generators.  explored different contrastive learning methods to train retrievers, while Ram et al. (2022) used recurring spans within a document to create pseudo-positive query-document pairs. Retrieval-augmented language models. Continuous cache models (Grave et al., 2017b) defines a probability distribution over recent tokens, by computing the similarity between previous and current representations of tokens. This distribution is then interpolated with the distribution of the language model, to improve predictions. Later, the amount of tokens used to compute this distribution was extended to a much larger memory by leveraging approximate nearest neighbors search (Grave et al., 2017a). The related kNN-LM model (Khandelwal et al., 2020) replaced LSTMs by transformer networks, and scaled the memory to billions of tokens, leading to strong performance improvements. More recently, RETRO (Borgeaud et al., 2021) extended these by scaling the retrieval memory to trillions of tokens, and changing the model architecture to take retrieved documents as input.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Recently, different works have proposed to train large language models to interact with a search engine, by generating text queries, and using the retrieved documents as additional context (Nakano et al., 2021;Thoppilan et al., 2022;Shuster et al., 2022). In the context of few-shot question answering, Lazaridou et al. (2022) used the question to perform a search query, and retrieved documents are added to the prompt of a large language model performing in-context learning.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Few-shot learning, the task of learning from very few examples, has been studied for decades (Thrun & Pratt, 1998;Fink, 2005;Vinyals et al., 2016), but has recently seen an explosion of interest in NLP with the arrival of large pre-trained models, which exhibit emergent few-shot learning abilities (Wei et al., 2022).
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In-context Learning with large Language models. Providing language models with natural language descriptions of tasks, as proposed by Radford et al. (2019) has led to significant developments in few-shot learning. GPT-3  demonstrated the ability of large language models to perform few-shot predictions, where the model is given a description of the task in natural language with few examples. Scaling model size, data and compute is crucial to enable this learning ability, leading to the further development of large models (Lieber et al., 2021;Rae et al., 2021;Smith et al., 2022;Chowdhery et al., 2022;Smith et al., 2022). Hoffmann et al. (2022) revisited the scaling law from , suggesting that training on more data with a smaller model may be more effective, resulting in Chinchilla, a 70B parameter model with improved parameter efficiency.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Few-shot finetuning and prompt-based learning. The above models perform few-shot learning with in-context instructions without training the parameters of the language model. Few-shot learning can also be accomplished by combining textual templates ("prompts") and various forms of model finetuning, either fully updating a model's parameters, e.g. for classification (Schick & Schütze, 2021a;Schick & Schutze, 2021;Gao et al., 2021;Tam et al., 2021) or generation (Schick & Schütze, 2021b). Prompts themselves can be optimized, for example by search (Jiang et al., 2020;Shin et al., 2020) or by only updating parts of the model (Logan et al., 2021), or learning "soft-prompts" (Lester et al., 2021;Li & Liang, 2021). Due to its simplicity, in this work we either employ simple prompts or simply feed in inputs without preprocessing, and perform full-model finetuning, a method similar to Le Scao & Rush (2021).
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this section, we report empirical evaluations of our language models on few-shot learning. We start by introducing our experimental setup, describing our evaluation benchmarks in section 4.1, and giving the training details of our models in section 4.2. Then, we perform an ablation study to compare the different technical choices leading to our main model. We finally evaluate this model, called Atlas, on different natural language understanding tasks in few-shot and full dataset settings.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
To evaluate our retrieval-augmented language models we consider the following benchmarks, which include different tasks.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
First, we use the KILT evaluation suite , containing 11 datasets corresponding to 5 tasks: fact checking, question answering, dialog generation, entity linking and slot-filling. These different tasks require knowledge about the world to be solved, which can be found in Wikipedia. We evaluate our model on the following tasks and datasets included in KILT: question answering: NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018); slot filling: Zero Shot RE (Levy et al., 2017) and T-REx (Elsahar et al., 2018); entity linking: AIDA CoNLL-YAGO (Hoffart et al., 2011); dialogue: Wizard of Wikipedia (Dinan et al., 2019); and fact checking: FEVER (Thorne et al., 2018). The KILT versions of these datasets differ from their original versions, as instances requiring knowledge not present in the August 2019 Wikipedia dump have been removed.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Our second main evaluation benchmark is MMLU (Hendrycks et al., 2021), which contains 57 multi-choice question answering datasets (referred to as domains), sourced from real examinations designed for humans. These cover a very broad range of topics, e.g. high school mathematics, professional law, logical fallacies and clinical knowledge and can be broadly categorized in four subsets: humanities, social sciences, STEM and "other". We focus on few-shot learning, and the authors of the benchmarks suggest to use 5 training examples per domain. Beyond the 5-shot setting, We also consider three additional settings. The first is a zero-shot setting, with no training data at all. The second, which we call multi-task few-shot, is where we train a single model on the 6-shot data from all tasks, hence leading to a training set of 285 examples. The last, which we call transfer learning, leverages additional training examples from other multiple-choice QA tasks provided by the MMLU authors, namely MCTest (Richardson, 2013), RACE (Lai et al., 2017), ARC  and OBQA (Mihaylov et al., 2018) leading to a training set of 95k examples.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Additional benchmarks. Additionally, we report results on the original open-domain versions of the popular NaturalQuestions (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017) datasets. We also evaluate our model on the original version of FEVER (Thorne et al., 2018), which presents fact checking as a three-way classification problem for textual claims (either "Supported": the text is supported by evidence in Wikipedia, "refuted": the claim is not consistent with evidence in Wikipedia, or "not enough info", where there is insufficient evidence to make a judgement). We also perform experiments to assess temporal sensitivity of our models. Here, we construct a dataset from TempLAMA (Dhingra et al., 2022), consisting of a set of time-sensitive cloze questions on a range of topics, where the answer changes from 2017 to 2020. We assess the accuracy of our models when supplied with a index from 2017 vs 2020 to assess to what degree models faithfully reflect the content of the index supplied to them at test time, and how effective updating the index is as a continual learning or model updateability method.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We now describe the procedure for pre-training and fine-tuning our models. We focus on the setting used for the ablation studies performed in Section 4.3 and Section 4.4. We give more details about the hyperparameters used for our final model later.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For the pre-training, we initialize the retriever module using the unsupervised Contriever model, which uses the BERT-base architecture. We initialize the language model with the T5 pre-trained weight. As the original T5 pre-trained model included supervised data in the training set, we use the version 1.1 models which were trained on unlabeled text only. Specifically, we initialize from the T5-lm-adapt variants due to their improved stability.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For the ablation studies performed in Section 4.3 and Section 4.4, we use T5-XL which contains 3B weights. We pre-train all our models for 10,000 iterations, using AdamW with a batch size of 64 and a learning rate of 10 −4 for the reader and 10 −5 for the retriever with linear decay and 1,000 warmup steps. We refresh the index every 1,000 steps. This means that the index is recomputed 10 times during the pre-training, leading to an overhead of around 30%, compared to training with a fixed retriever. We set the number of retrieved documents to 20. We detail the hyperparameters used for the training of our final model at the beginning of Section 4.5. Fine-tuning. When performing a downstream task, either in a few-shot setting or with a large training set, we employ fine-tuning to adapt our models to these tasks. For the few-shot KILT ablation experiments, we perform a fixed number of fine-tuning iterations, instead of using early-stopping. More precisely, we decided to use 50 iterations for the 64-shot setting and 200 iterations in the 1024-shot setting. In both cases, we use a batch size of 32 examples, a learning rate of 4 × 10 −5 with linear decay and 5 warmup steps for both the reader and the retriever.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Unlabeled datasets. Finally, we discuss the unlabeled text datasets that we use to train our models, which form the retrieval index. First, we consider the Dec. 20, 2021 Wikipedia dump, for which we keep the lists and infoboxes, which are linearized by adding a semi-colon separator between the entries. We split articles by section, and split long sections into passages of equal sizes and containing less than 200 words. This leads to a total of 37M passages, containing 78 words in average. We also use documents from the 2020-10 common crawl dump, preprocessed with the CCNet pipeline (Wenzek et al., 2020). We perform additional document filtering, in a similar fashion to Gopher . More precisely, we filter documents based on document length, average word length, ratio of alphanumeric characters and number of repeated tokens. This leads to a total of 350M passages. The same passages are used for the index and model pre-training. During pre-training, we ensure the passage we are training on is filtered out from the retrieved documents, to prevent the model from simply retrieving the passage it is de-nosing/generating, and trivially using it to solve the pre-training task.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We start our ablation study by comparing different pre-training tasks, and objective functions to jointly train the retriever and the language model. Our goal here is to answer the following research questions:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
(RQ 1) Does jointly pre-training the whole model lead to better few-shot performance?
Atlas: Few-shot Learning with Retrieval Augmented Language Models
(RQ 2) What is the best objective function for the retriever, and the best pretext task?
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We start by comparing the training objectives of the retriever, introduced in Section 2.2, by pre-training models using the masked language modelling task. We evaluate these models on a subset of the 64-shot and 1024-shot KILT benchmark: NaturalQuestions, FEVER and Wizard of Wikipedia, along with two baselines: a 'closed-book" (i.e. non-augmented T5) baseline, pre-trained on the same data, and initialized from Contriever and T5-lm-adapt. We report results in Table 1. First, we note the poor performance of the closed-book baseline, indicating the importance of augmentation. Next, we observe that pre-training our model with retrieval is important to obtain good performance on few-shot tasks. Indeed, all models that include retrieval during pre-training strongly outperform the baseline without joint pre-training. Next,  we compare a model that was pre-trained with a fixed retriever, and models using the various retriever training objectives. On the MLM validation metric corresponding to the pre-training objective, we observe that jointly training the retriever leads to strong improvements. This effect tends to be less marked on 64-shot downstream tasks, and almost non-existent for 1024-shot. We believe that this is evidence that the biggest impact of pre-training is on the language model, which learns to use and aggregate information from the retrieved documents. Lastly, we do not observe significant systematic differences between the different retriever training objectives. We thus decide adopt use Perplexity Distillation for subsequent experiments, as it tends to be more stable than EMDR 2 or ADist, and more computationally efficient than LOOP.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Next, we compare the different self-supervised pretext tasks introduced in Section 2.3 in Table 2. Here we observe similar results for all three tasks, with a small advantage for masked language modelling. Thus, in what follows, we adopt masked language modelling for pre-training.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Finally, we consider different combinations of data sources-Wikipedia and common crawl-for the index and training data during pre-training. In all cases, we use the Wikipedia 2021 dump as the index when performing few-shot fine-tuning. We report results in Table 3. First, we observe that using a Wikipedia-based index leads to better downstream performance. There could be two explanations for this: first, as we use Wikipedia for the few-shot tasks, the model might be better adapted when trained using the same data. Another explanation might be that Wikipedia is a higher-quality and denser source of knowledge than common crawl. Second, when using a common crawl index, we observe that pre-training on Wikipedia data leads to lower performance than using common crawl data. We believe that the primary reason is that the distribution mismatch between the two domains leads to generally-less relevant retrieved documents. In turn, this probably means that the pre-training is less efficient, because the language model does not leverage as much information from the documents. In the following, we thus decide to combine the data from both domains for both the index and the pre-training data.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this section, we perform an ablation study on how to apply our models on downstream tasks, which relies on fine-tuning. In particular, we want to investigate the following research question:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
(RQ 3) How to efficiently fine-tune Atlas on tasks with limited training data? To answer this question, we compare the different strategies to fine-tune the retriever module, described in Section 2.4. We report results in Table 4. First, as for pre-training, we observe that keeping the retriever fixed during fine-tuning leads to a significant performance drops, for both 64-and 1024-shot settings. Second, the re-ranking strategy (row 2) leads to very similar results to fully updating the index (row 1), while being significantly more efficient. Lastly, fine-tuning only the query encoder also leads to strong results: in particular, in the 64-shot setup, this is slightly stronger than performing full fine-tuning, which we attribute to there being less opportunity for over-fitting. On the other hand, on the 1024-shot setting, performing a full fine-tuning leads to stronger results, especially on NaturalQuestions. In the following, we thus use query-side fine-tuning for experiments with small numbers of examples, and standard fine-tuning for larger datasets.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this section, we apply the findings from the ablations of the previous sections to train a family of Atlas models, ranging from 770M to 11B parameters. More specifically, we use the Perplexity Distillation objective function, along with the masked language modelling pretext task. We pre-train these models using a mix of Wikipedia and Common Crawl data, for both the training data and content of the index. We retrieve 20 documents, and update the index every 2,500 steps and perform re-ranking of the top-100 documents. We pre-train models for 10,000 iterations using AdamW with a batch size of 128.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
As mentioned in section 4.1, we consider four setting for MMLU: 1) a zero-shot setting where we directly apply the pretrained model with no few-shot finetuning 2) a 5-shot setting, where we finetune a model using 5 training examples for each of the 57 domains 3) a 5-shot multitask setting, where, rather than finetuning a model independently for each domain, we train a single model to perform all tasks and 4) a setting with access to a number of auxiliary datasets, with 95K total training examples. We train the models to generate the letter corresponding to the correct answer option ('A', 'B', 'C' or 'D'), and pick the answer with the most likely of the 4 letters at test time. Full technical details can be found in appendix A.1.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We start by comparing Atlas to closed-book models of different sizes for 5-shot, 5-shot multitask and the full setting, and report results in Table 5. Across these settings, Atlas outperforms the closed-book baselines by between 6.6 and 15.6 points, demonstrating consistent utility of retrieval for few-shot language understanding across 57 domains. The closed-book T5 struggles to perform significantly better than random (25%) in few-shot settings with 770M parameters, whereas the equivalent Atlas achieves around 40%, significantly better than random, despite its small size. All models improve with more data, but interestingly, the 770M models do not benefit as much from few-shot multitask learning compared to larger models (for closed-book, it actually loses 3 points) suggesting smaller models struggle to grasp the synergies between the tasks in the few-shot setting. Larger models exploit the multi-task setting well, with Atlas improving more than closed-book. For example, Atlas-11B improves by 13 points (43.4 → 56.4), but equivalent closed-book only improves by 7 (36.1 → 43.5). Finally, on the transfer learning setting, all models improve, but the relative gaps between closed-book at Atlas models remain similar.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
De-biasing. When finetuning, we permute which answer option appears with which answer letter to reduce over-fitting and encourage a uniform prior over answer letters. However, the model may still exhibit a bias towards some letters, especially in few-shot settings, so we also include a second 'de-biased' inference mode in addition the standard inference used above. Here, we run 4 forward passes, one for each cyclic permutation of the answer letter-answer option assignment in the question, e.g. the answer option assigned to letter 'A' becomes 'B', what was 'B' becomes 'C' etc. 3 We then sum the 4 probabilities to obtain the final prediction, which reduces spurious bias towards one of the answer letters (further details in appendix A.1). The results are shown in Table 6. We find that in zero-shot and 5-shot settings, de-biasing is very effective, improving results by 10.3 and 4.5 points respectively. When more training data is available, the need for de-biasing decreases, leading to only 0.2 point improvement in the multi-task and full data settings.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Comparison to published works Next, we compare our Atlas-11B results with de-biasing to recently reported results with state-of-the-art large language models such as GPT-3 or Chinchilla, which required significantly more amount of computation to train. We report results in Table 7. We find that Atlas is able to perform significantly better than random in zero-shot, and in conjunction with de-biased inference, achieves zero-shot scores that exceed 5-shot results reported with GPT3 in the literature (47.1% vs 43.9%) (Hendrycks et al., 2021). For the 5-shot setting, Atlas outperforms GPT-3 by 4%, while using 15× less parameters, and 10× less pre-training compute. 4 When multitask-training on the combined 5-shot data, Atlas improves to 56.6% close to the 5-shot performance of Gopher (60.0%). Finally, on the full data setting, where we train on auxiliary data recommended by the MMLU authors, Atlas reaches an overall accuracy of 65.6%, close to the state-of-the-art. Interestingly, in this setup, Atlas significantly outperforms GPT-3, while on the 5-shot setting, their performance is similar.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Next we evaluate Atlas on two open-domain question answering benchmarks: NaturalQuestions and TriviaQA. We compare to prior work, both in a few-shot setting using 64 examples, and using the full training set, and report results in Table 8. On these benchmarks, which require high-degree of memorisation, we clearly see the benefits of retrieval-augmentation. Atlas-11B obtains state-of-the-art results on 64-shot question answering, for both NaturalQuestions and TriviaQA. In particular, it outperforms significantly larger models, such as PaLM, or models that required significantly more training compute such as Chinchilla. When using the full training set, Atlas also obtains state-of-the-art results, for example improving the accuracy on NaturalQuestions from 55.9% to 60.4%. This result is obtained using an index comprised of CCNet and  the December 2021 Wikipedia corpora, our default setting for the index. In section 5.2 we consider using indexes composed of Wikipedia corpus archived at different dates, and demonstrate an additional +3.6% on NaturalQuestions when using an index which is temporally matched to NaturalQuestions. We report performance as a function of model size as well as detailed hyperparameters in Appendix A.2.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Atlas also compares favorably to recent work exploring retrieval-augmented few-shot question answering with very large models. Lazaridou et al. (2022) explore NaturalQuestions in a 15-shot setup using Gopher, augmenting questions with 50 passages retrieved using Google Search. This method consists of generating 4 candidate answers from each retrieved passages, and then re-ranking using either a score inspired by RAG   
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We report results on the original 3-class FEVER fact checking test set in Table 9. We consider a 64-shot setting, with training examples uniformly sampled from the full training set. Unlike the development and test sets, the train set is imbalanced, with more positive labels than negative, posing a challenge for few-shot  learning. In this setting, we achieve an accuracy of 64.3%. We also report a 15-shot setting, with 5 examples uniformly sampled from each class to compare with published results from Gopher , where Atlas scores 56.2%, outperforming Gopher by 5.1 points. Lastly we fine-tune our model on the full training set, and achieve a score of 78%, within 1.5% of the ProoFVer, which uses a specialized architecture, a retriever trained with sentence-level annotations, and is supplied with the Wikipedia corpus released with FEVER, whereas Atlas retrieves from CCNet and the December 2021 Wikipedia dump. If we give Atlas an index comprised of the FEVER Wikipedia corpus, we set a new state-of-the-art of 80.1%
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Finally we evaluate Atlas on KILT, a benchmark composed of several different knowledge intensive tasks, which was described in section 4.1. We report results on test sets in Table 10 for which evaluation is available online 5 . The KILT versions of datasets are filtered, and thus results for datasets we have evaluated elsewhere are not directly comparable on KILT (i.e. FEVER, NQ and TQA). We consider both a 64-shot setting and a full fine-tuning setting, in both cases we train Atlas individually on each dataset. More details on the hyperparameters and development set results are reported in Appendix A.3. For 64-shot, we greatly exceed random performance, and are even competitive with some fully-finetuned models on the leaderboard, such as for FEVER, where our 64-shot Atlas is only 2-2.5 points behind Sphere, SEAL and Re2G, and outperforms Sphere and SEAL on zero-shot RE. In the full dataset setting, Atlas is within 3% to the state-of-the-art for 3 datasets, and sets the state-of-the-art in the remaining five datasets. 
Atlas: Few-shot Learning with Retrieval Augmented Language Models
An advantage of semi-parametric models like Atlas is the ability to inspect retrieved items to aid interpretability. To better understand how well Atlas retrieves, and how it uses retrieved passages, we examine the retrieved passages for multi-task few-shot MMLU. As shown in the left panel of Figure 3, the model retrieves the majority of its passages from CCNet (85% on average). Wikipedia makes up about 15% of retrieved passages, which is higher than we would expect under a uniform prior, given Wikipedia only makes up about 10% of the index. The fraction of Wikipedia retrieval varies between MMLU domains, with the model using Wikipedia to a greater extent for STEM domains, and least for social sciences. The domain making the greatest use of Wikipedia is "abstract algebra" (73%), and the least is "moral scenarios" (3%). We also note that the MMLU-finetuned Atlas does not make significant use of Wikipedia infobox passages.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We can also analyse the content of passages to assess how they may useful for accomplishing the downstream task. The middle panel of Figure 3 shows how often retrieved documents contain the text of the correct answer option. There being at least one mention of the correct answer choice in 30% of test questions in the top 25 passages. 6 The right panel shows that the accuracy on MMLU increases when the correct answer option text occurs more frequently in retrieved passages, rising from 55% for questions when the answer option does not appear, to 77% for questions mentioned more than 15 times.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
A human analysis of retrieved documents revealed that documents are helpful for answering questions in a number of different ways. Manual inspection of a sample of 50 correctly-answered questions revealed that 44% contained at least partially useful background information. These are documents that would improve the likelihood of a non-expert human answering correctly, such as contextual clues surrounding a quotation from a question, or helpful numerical figures for quantity-based questions, which help to narrow down the answer options to a smaller range. In a further 26% of cases, a passage contained all the necessary information to answer the question, stated in a straightforward way. If read competently, such passages make the question simple to answer, and often include information such as canonical definitions, or the exact numerical answer requested in the question. 28% of retrieval sets did not contain obvious information which would make the question easier. Finally, 2% contained the verbatim question in a passage, together with its answer.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Given that MMLU has been created from pre-existing exams, it is possible that these questions appear on the open web. Models trained on web data (or, in our case, retrieving from it) run the risk of answering correctly not through generalisation, but by verbatim memorisation, which could lead to misleadingly high scores. In some very large language models, which can verbatim memorize and recall large parts of their pre-training data (Carlini et al., 2021), efforts have sometimes been made to filter occurrences of downstream instances from pre-training data, but this has not been performed for MMLU in the literature. In order to assess the prevalence of MMLU leakage in our index, we manually checked retrieval results for questions where the longest n-gram overlap between the question (without answer options) and a passage was at least 75% the length of the question. This resulted in an estimate of leakage of 2.8% of questions from our CC-Net corpus.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
A benefit of retrieval-augmented models such as Atlas is the editability of its knowledge (see section 5.2 for additional analysis). To estimate pure, non-leaked performance, we can filter out any potentially-leaked passages from retrieved results and rerun the language model. The MMLU score drops slightly when controlling for this leakage from 56.4 to 55.8% (-.5%).We note that our CC-net corpus is relatively small compared to the pre-trained corpora of recent very large models, which are trained on up to 1.4 trillion tokens (Hoffmann et al., 2022), 35x the size of our index, making it likely that models trained on corpora of that size would observe more MMLU leaked examples, but detecting such leakage is challenging in non-retrieval augmented models.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
A benefit of retrieval-augmented models is that they can be kept up-to-date without retraining, by updating or swapping their index at test time. To assess the effectiveness of this mechanism in Atlas, we first construct a dataset of time-sensitive questions derived from TempLAMA (Dhingra et al., 2022). TempLAMA is a collection of templated cloze questions derived from Wikidata and Wikidata where the correct answer changes over time. We select a subset of questions from this dataset which have a different answer in 2017 and 2020, for example, Question: Theo Walcott plays for ___ Answer: Arsenal F.C. ( ), Everton F.C. (2020, and form a small training set of 248 training, 112 development and 806 test questions.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Using this dataset, we finetune closed-book T5-XXL and Atlas using the questions and the 2017 answers, supplying Atlas with a 2017 Wikipedia index, and then measure exact match accuracy on the 2017 test set. The results can be found in the first row and first two columns of Table 11. We first observe that, as expected, Atlas greatly outperforms T5 (57.7% c.f. 12.1%). We also note that, as desired, both T5 and Atlas almost never generate an answer from 2020 when trained with the 2017 answers, scoring 2.8% and 1.5% respectively (first row, second two columns of Table 11). However, as shown in row 2, we can swap the Atlas index to a 2020 Wikipedia index, without retraining, and find that Atlas updates its predictions accordingly, with 2020 accuracy rising to a similar level to its 2017 performance (53.1%), whereas the purely parametric T5 has no such updateability mechanism.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
This demonstrates that Atlas can be faithful and condition strongly on its supplied index. Furthermore, this zero-shot updateability mechanism has the useful property of staying up-to-date without requiring up-to-date annotated data, or continuous, lifelong pre-training, as would be may required for a large parametric-only model. Rows 3 and 4 of Table 11 complete the picture, where this time we train with 2020 answers, and demonstrate Atlas can zero-shot transfer backwards in time to 2017 effectively too (50.1%). Interestingly, T5 is unable answer questions from 2020 well, even when trained with 2020 answers (3.6%), likely because it was pre-trained on data pre-dating 2020 (Dodge et al., 2021).
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We also examine temporal effects for NaturalQuestions. NaturalQuestions is a dataset composed of search queries collected via the Google search engine in a short period of time. Thus data have a strong temporal bias, with a lot of questions about the 2018 World Cup for example. Moreover some questions are ambiguous without specification of the temporal context. For instance, for the question "when did ireland last beat england at twickenham", the expected answer is 2018 in NaturalQuestions, while Ireland also beat England at Twickenham in 2022 as well as many other times before. In Table 12, we report results obtained by finetuning Atlas using different Wikipedia dumps for the index. We observe that the 2018 December Wikipedia dump, which is close to the date of data collection, leads to the best results for both few-shot and full fine-tuning. In particular, it leads to a new state-of-the-art of 64 EM on NaturalQuestions. Table 11: Results on our TempLAMA-derived dataset. We report performance for a static, closed-book T5-11B, as well as Atlas-11B supplied with a test-time Wikipedia index from 2017 or 2020. We evaluate models finetuned on a small training set of 248 time-sensitive cloze-question-answer pairs, using answers either from 2017 or 2020. Good models should score highly when the test set year matches the year of the test-time index, and score low otherwise.  
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Maintaining dense retrieval indices can be memory-intensive, especially as the number of indexed items is scaled. In this section, we briefly analyse the memory requirements of Atlas's index in the case of a) a Wikipedia index and b) the combined CC and Wikipedia index used in most of the experiments above.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
There are two sources of memory pressure for Atlas's retrieval component -the passages themselves, and the document embedding index. The tokenized passages, once binarized, require 11GB and 130GB of storage for the Wikipedia and combined indices respectively. These passages do not need to be stored in expensive GPU RAM, and could even be memory-mapped to disk, sharded across nodes or compressed if required, and thus do not represent a limiting hardware challenge in this context. The embedding index itself, however, must be stored in GPU RAM for fast search, and thus its size is more sensitive. In the above experiments, we perform exact search over our index, which is achieved by sharding the index over all the the available GPUs, and computing the search in parallel. The index is stored at fp16 precision, resulting in a total GPU memory requirement of 49 GB and 587 GB for the Wikipedia and combined indices, respectively.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
This large GPU memory requirement for the index limits accessibility and ease of deployment. However, many index compression techniques are available for nearest neighbour search, which can often dramatically reduce memory requirements at the cost of some retrieval accuracy. Following , we explore the effect of Product Quantization (PQ, Jégou et al., 2011), a popular lossy compression technique on Atlas-3B's accuracy for the 64-shot NQ task at different compression levels.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
The results are shown in Figure 4. We find that substantial compression is possible before the onset of significant performance degradation. Namely, the Wikipedia index can be compressed from 49GB to 4GB with negligible drop in retrieval precision and exact match. Likewise, the combined index can be compressed from 587GB to 50GB without serious degradation, indicating that the combined index could be loaded onto a single 80GB GPU.  
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In this paper, we introduce Atlas, a large retrieval-augmented language model. By jointly pre-training the retriever module and the language model, we show that Atlas has strong few-shot learning capabilities on a wide range of knowledge intensive tasks, including NaturalQuestions, TriviaQA, FEVER, 8 KILT tasks and 57 MMLU tasks. For example, Atlas-11B reaches more than 42% accuracy on NaturalQuestions and 84.7% on TriviaQA when training on 64 examples, which is an improvement of almost 3 points compared to PaLM, a 540B parameters model, which required 50x more pre-training compute. We also provided detailed ablations and analyses for what factors are important when training such retrieval-augmented models, and demonstrated Atlas's updateability, interpretability and controlability capabilities. Lastly, we demonstrated that Atlas is also powerful in full-dataset settings obtaining a new state-of-the-art results on NaturalQuestions, TriviaQA, FEVER, and 5 KILT tasks. 
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Featurization MMLU consists of multiple choice questions with four possible lexicalized answer options. We represent the input using the following template:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
and train the model to generate the mask token followed by the letter of the correct answer:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
This format closely matches the format of MLM pre-training objective, aiding few-shot learning. When training, we permute the order of the answer options, i.e. shuffling which answer option appears as letter A etc. This helps reduce overfitting, and encourages a uniform prior on the letters.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Once trained we obtain predictions from the model by selecting the pre-softmax logits for the tokens A, B, C and D, and performing a softmax over them to obtain a distribution over the 4 answer options. For standard inference, we then simply return the answer corresponding to the argmax of this distribution.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
As mentioned in the main text, even though our model is finetuned with data that encourages a uniform prior over answer letters (by permuting which answer option letter is used with which lexical answer option text in training data), this may not be enough to ensure the model has no residual bias towards specific letters. Consider answers a, questions q and a nuisance variable z ∈ Z, which represents the ordering of the answer options or, equivalently, which answer letter gets assigned to which answer option text. There are 4 answer options in MMLU, and thus |Z| = 24 unique ways they can be ordered, or assigned to given letters. Running our model with our standard inference for a question q, corresponds to calculating p(a|q = q, z = z) for the answer ordering z that happens to appear in the dataset. We can control for z by running the model with all possible answer orderings in the input, and marginalizing: p(a|q = q) = z ∈Z p(a|q = q, z = z )p(z = z |q = q), and assuming p(z = z |q = q) is uniform (no answer ordering is more likely than another), this reduces to simply p(a|q = q) ∝ z ∈Z p(a|q = q, z = z ). This procedure requires 24 forward passes, one for each answer ordering, so is 24× slower than standard inference. Table 13 shows the result of applying the full permutation de-biasing, which leads to an 12% improvement zero-shot and 6% in 5-shot performance overall. Empirically, using only the cyclic permutations of the answer order provided in the original dataset (of which there are 4) works nearly as well, which is what we report in the main paper, and only increases inference compute by a factor of 4, rather than 24. Cyclic permutation de-biasing improves over standard inference by 10% in zero-shot and 5% in 5-shot. Empirically, de-biased inference is largely unnecessary when training in the 5-shot multitask or full dataset setting, as there is enough data for the model to learn a more uniform prior over the letters. Evaluation We evaluate by following the method of Hendrycks et al. (2021), namely, micro-averaging across all 57 domains to obtain overall accuracy. We quote the results of GPT3  and UnifiedQA (Khashabi et al., 2020) from the MMLU leaderboard at https://github.com/hendrycks/test. For Chinchilla and Gopher, we calculate the scores on the categories using the full MMLU results from Hoffmann et al. (2022).
Atlas: Few-shot Learning with Retrieval Augmented Language Models
The index used for MMLU for all MMLU experiments in the main paper comprised of concatenation of the Wikipedia passages, Wikipedia info boxes and Common Crawl indices, for a total of 387M passages. We can assess the importance of the index by running a model without the common crawl data, leading to a 5-shot multitask result of 52.8%, compared to 56.4% for the full model, a drop of 3.6%. This indicates that whilst the Wikipedia data is sufficient do well on the task, the addition of the CC data improves results further.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Hyperparameters and development data Selecting hyperparameters is challenging in few-shot settings. We do not assume access to an in-domain development set for the 5-shot task. Instead, we determine a set of hyperparameters for the 5-shot task using data from RACE, one of the auxiliary datasets provided by MMLU. Here, we sample 5 sets of 5-shot training data, and for each model size, we explore batch size {32, 64}, learning rates for the language model and retriever {(5e-5, 1e-5), (4e-5, 4e-5)}, retriever temperature {0.1, 0.01} and a fixed number of training steps {16, 32, 64, 128}, picking the setting that achieves strongest RACE validation scores. Having determined these hyperparameters, we apply them directly to the 5-shot MMLU task. For the 5-shot multi-task and full/transfer settings, we use the same batch size, temperatures and learning rates as the 5-shot task, but use a set of 285 MMLU validation examples (5 per domain) in order to determine the total number of training steps and for early stopping. The hyperparameters selected in the MMLU experiments can be found in table 14. We use query-side finetuning for the 5-shot and 5-shot multitask settings, and top-128 reranking for the full setting. For all MMLU runs we retrieve 30 documents
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Inter-run Variance few-shot learning is well-known to suffer from high variance. In the main paper, we quote the result obtained with our first run. In order to assess the effect of noise and variance, we ran the 5-shot experiment with Atlas 5 times. 7 We observe high variance for individual domains, sometimes as high as 20%, however, once aggregated across all 57 domains, the inter-run variance is low. The overall scores for these different runs, when using the same hyperparameters are shown in table 15. Due the effects of averaging over the many domains that comprise MMLU, the inter-run variance is quite modest on the aggregated metrics, with a std deviation of 0.5 in this experiment.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
The closed book baselines we compare Atlas to in table 5 are initialized from the same T5 model as their respective Atlas, and then pre-trained with MLM for the same number of steps (10K) using the same pre-training data as Atlas, for fairer comparison. The same procedure as for Atlas was used to determine hyperparameters for MMLU for the closed-book model.s 45.0 ± 0.5 40.5 ± 0.6 53.6 ± 1.6 37.4 ± 0.5 51.1 ± 0.8
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Tables 16 and 17 shows the full MMLU scores for each domain for Atlas and the closed book T5 respectively. The full results for the cyclic-permutation-de-biased Atlas-XXL can be found in Table 18.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For question answering, similarly to the MMLU experiments, we format the input using the following template:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
and train the model to generate the mask token followed by the answer:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
[MASK_0] {answer}.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We generate answers using greedy decoding. For both training and testing, we retrieve 40 passages, and truncate the result of the concatenation between the query and the passages to 384 tokens.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For few-shot fine-tuning we train Atlas for 30 steps using 64 random samples from the train sets. The retriever is trained using query-side fine-tuning. We select the model after 30 training steps. We use AdamW with a batch size of 32 and a learning rate of 4 × 10 −5 with linear decay and 5 iterations of warmup for both the language model and the retriever.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For the fine-tuning on the full datasets, we train the model for 5k gradient steps and refresh the index every 500 steps for the first 1,000 training steps and every 2k training steps afterwards. We use AdamW with a batch size of 64 and a learning rate of 4 × 10 −5 with linear decay and 5 iterations of warmup for both the language model and the retriever. We evaluate models every 500 steps and select the best one on the validation set based on the exact match score.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
In Table 19, we report performance on NaturalQuestions and TriviaQA as a function of the number of parameters in the reader module. Both for few-shot learning and full fine-tuning we observe strong improvements by scaling the size of the reader module. However we can notice sign of saturation when finetuning on full datasets, with limited gains when scaling from 3B to 11B parameters (+0.6% on NaturalQuestions, +0.5% on TriviaQA). While performance improves substantially when scaling from 3B to 11B parameters with 64 training samples, with +3.7% and +1.2% improvement on NaturalQuestions and TriviaQA respectively. For these experiments we use a setup similar to the one use in Table 8, except that we use an index composed of the December 2018 Wikipedia dump processed as described in section 4.2.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For the results on KILT reported in Table 10 we fine-tune Atlas individually on each dataset. We format the input using a template similar to the one used for question answering:     and train the model to generate the mask token followed by the expected output:
Atlas: Few-shot Learning with Retrieval Augmented Language Models
[MASK_0] {output}.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We retrieve 20 passages and generate answer using greedy decoding. In KILT, FEVER is a two-way classification task of claims. We lexicalize the "SUPPORTS" (resp. 'REFUTES") label into "true" (respectively "false").
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For few-shot fine-tuning we train Atlas for 30 steps using 64 random samples from the train sets. The retriever is trained using query-side fine-tuning. We evaluate models every 5 steps and select the best one on the development set based on the reported metric. We use AdamW with a batch size of 32 and a learning rate of 4 × 10 −5 with linear decay and 5 iterations of warmup for both the language model and the retriever.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
For the fine-tuning on the full datasets, the model is trained for 5k gradient steps. We evaluate models every 500 steps and select the best one on the development set based on the reported metric. The index is refreshed every 500 step for the first 1000 iterations, and every 2k steps afterwards. We use AdamW with a batch size of 64 and a learning rate of 4 × 10 −5 with linear decay and 500 iterations of warmup for both the language model and the retriever.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
We report results on the development sets in Table 20.