How does pretraining data affect a language model's in-context learning performance?
Pretraining data Razeghi et al. (2022) find on
numerical reasoning tasks, a language model’s ICL
performance is highly correlated with the term fre-
quency of the input data in the pretraining corpus.
Shin et al. (2022) investigate how ICL can be af-
fected when the pretraining dataset varies. They
discover that ICL heavily depends on the corpus do-
main source, but pretraining with a corpus related
to a downstream task does not always translate to
a competitive ICL performance on the task. Chan
et al. (2022) experiment on a synthetic image-label
pairs dataset. They show certain distributional prop-
erties of the synthetic pretraining data, such as the burstiness of classes and large numbers of rarely
occurring classes, promote the emergence of ICL.
Our work belongs to this line of work, but offers
a first step towards understanding ICL in realistic
NLP tasks through analyzing instance-level pre-
training data. Additionally, concurrent to our work,
Gu et al. (2023) propose a method that groups pre-
training data by their instrinsic tasks, enhancing
instead of interpreting existing language models’
ICL ability.
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning (ICL) improves language models’ performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations. Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via investigating the pretraining data. Specifically, we first adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL. We observe that a continued pretraining on this small subset significantly improves the model’s ICL ability, by up to 18%. We then compare the supportive subset constrastively with random subsets of pretraining data and discover: (1) The supportive pretraining data to ICL do not have a higher domain relevance to downstream tasks. (2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens. (3) The supportive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encourages ICL. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data. Our insights have a potential to enhance the ICL ability of language models by actively guiding the construction of pretraining data in the future.
[{"first":"Xiaochuang","last":"Han","middle":[]},{"first":"Daniel","last":"Simig","middle":[]},{"first":"Todor","last":"Mihaylov","middle":[]},{"first":"Yulia","last":"Tsvetkov","middle":[]},{"first":"Asli","last":"Celikyilmaz","middle":[]},{"first":"Tianlu","last":"Wang","middle":[]}]
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper, we investigate how well these models reason with terms that are less frequent in the pretraining data. In particular, we examine the correlations between the model performance on test instances and the frequency of terms from those instances in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models (pretrained on the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and unit conversion). Our results consistently demonstrate that models are more accurate on instances whose terms are more prevalent, in some cases above $70\%$ (absolute) more accurate on the top 10\% frequent terms in comparison to the bottom 10\%. Overall, although LMs exhibit strong performance at few-shot numerical reasoning tasks, our results raise the question of how much models actually generalize beyond pretraining data, and we encourage researchers to take the pretraining data into account when interpreting evaluation results.
[{"first":"Yasaman","last":"Razeghi","middle":[]},{"first":"IV","last":"RobertL.Logan","middle":[]},{"first":"Matt","last":"Gardner","middle":[]},{"first":"Sameer","last":"Singh","middle":[]}]
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in a wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In our initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models were unable to achieve both simultaneously. However, our later experiments uncovered that the two modes of learning could co-exist in a single model when it was trained on data following a skewed Zipfian distribution -- another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. In sum, our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and how future work might encourage both in-context and in-weights learning in domains beyond language.
[{"first":"Stephanie","last":"Chan","middle":["C.Y."]},{"first":"Adam","last":"Santoro","middle":[]},{"first":"Andrew","last":"Lampinen","middle":["K."]},{"first":"Jane","last":"Wang","middle":["X."]},{"first":"Aaditya","last":"Singh","middle":[]},{"first":"Pierre","last":"Richemond","middle":["H."]},{"first":"Jay","last":"McClelland","middle":[]},{"first":"Felix","last":"Hill","middle":[]}]
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.
[{"first":"Seongjin","last":"Shin","middle":[]},{"first":"Sang-Woo","last":"Lee","middle":[]},{"first":"Hwijeen","last":"Ahn","middle":[]},{"first":"Sungdong","last":"Kim","middle":[]},{"first":"HyoungSeok","last":"Kim","middle":[]},{"first":"Boseop","last":"Kim","middle":[]},{"first":"Kyunghyun","last":"Cho","middle":[]},{"first":"Gichang","last":"Lee","middle":[]},{"first":"Woomyoung","last":"Park","middle":[]},{"first":"Jung-Woo","last":"Ha","middle":[]},{"first":"Nako","last":"Sung","middle":[]}]
Pre-Training to Learn in Context
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models’ in-context learning ability by pre-training the model on a large collection of “intrinsic tasks” in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.
[{"first":"Yuxian","last":"Gu","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Furu","last":"Wei","middle":[]},{"first":"Minlie","last":"Huang","middle":[]}]
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning in NLP has drawn tremendous attention recently (Dong et al., 2022). Unlike traditional learning paradigms that rely on training or finetuning models, in-context learning only provides a handful of demonstration examples to language models as a prefix to the test input, without any parameter updates. In-context learning has shown superior performance on a range of NLP tasks (Brown et al., 2020;Zhang et al., 2022b; * Work done during an internship at Meta AI. Figure 1: An example from the pretraining data of OPT (Zhang et al., 2022b) and an illustrative in-context learning example of topic classification. The in-context learning task data can be drastically different from pretraining instances, both in content and format. Chowdhery et al., 2022;Hoffmann et al., 2022), but the origin and reason of this emergent ability remain under-investigated. In-context learning is surprising since language models have not been explicitly trained to learn from demonstration examples (Xie et al., 2022). As shown in an illustrative scenario in Figure 1, a typical pretraining data instance is highly different from an in-context learning example for downstream tasks, in both content and format.
Understanding In-Context Learning via Supportive Pretraining Data
Prior work have attempted to answer what incontext learning is, through empirically investigating useful and irrelevant attributes of the demonstration examples (Min et al., 2022;Zhang et al., 2022a), or theoretically proving certain synthetic language models implicitly do Bayesian inference with demonstrations (Xie et al., 2022). Furthermore, recent work have drawn connections between the mechanism of in-context learning and standard learning algorithms, such as regression, nearest neighbor, and gradient descent (Olsson et al., 2022;Akyürek et al., 2022;Dai et al., 2022;von Oswald et al., 2022).
Understanding In-Context Learning via Supportive Pretraining Data
Differently, in this work we are interested in understanding from where the in-context learning ability is acquired, through a perspective of pre-training data. Although not many, some recent work have investigated this direction. For instance, Shin et al. (2022) pretrain a variety of language models on different corpora. They study correlations between attributes of pretraining datasets and in-context learning performance, at a relatively coarse dataset-level. Chan et al. (2022) construct pretraining data with different attributes and discover that some distributional properties of the data drive the emergence of in-context learning. However, their experiment is limited to synthetic data of image-label pairs.
Understanding In-Context Learning via Supportive Pretraining Data
In this work, we investigate a large language model OPT (Zhang et al., 2022b) and its pretraining data. We first hypothesize that there exists some specific pretraining data instances that are particularly helpful to the model's in-context learning ability. As an attempt to find such instances, we adapt an iterative, gradient-based method ORCA (Han and Tsvetkov, 2022) to search within OPT's pretraining corpus. The process is guided by the gradients of the in-context learning data from downstream tasks, and we refer to the identified subset as supportive pretraining data to in-context learning following Han and Tsvetkov (2022). Furthermore, we quantitatively verify through a perturbative continued pretraining, that the supportive subset does improve the model's in-context learning performance on downstream tasks, while not affecting a spurious zero-shot performance ( §2).
Understanding In-Context Learning via Supportive Pretraining Data
We then analyze the identified supportive data in contrast to the general pretraining data, to obtain data features particularly relevant to in-context learning. We specifically approach from three aspects: the domain relevance to downstream tasks, the token frequency distribution, and the information gain of incorporating long-range pretraining context. Our major findings include: (1) Compared to general pretraining data, the supportive data do not have a higher domain relevance to the downstream tasks.
Understanding In-Context Learning via Supportive Pretraining Data
(2) The supportive pretraining data contain a relatively higher amount of rarely occurring, long-tail tokens.
Understanding In-Context Learning via Supportive Pretraining Data
(3) The supportive pretraining data are challenging examples in incorporating long-range context for language modeling ( §3).
Understanding In-Context Learning via Supportive Pretraining Data
Our work offers a first step towards interpreting in-context learning in NLP tasks via analyzing instance-level pretraining data. We believe it can help improve the transparency and interpretability of language models' in-context learning behavior. Our analysis can also pave the way to improved in-context learning in the future by informing pretraining data construction.
Understanding In-Context Learning via Supportive Pretraining Data
2 Finding supportive pretraining data for in-context learning Han and Tsvetkov (2022) propose an iterative, gradient-based method ORCA to find supportive pretraining data of BERT (Devlin et al., 2019) under a vanilla zero-shot prompting setup. In this section, we provide some background and adapt ORCA for large language models in a setting of incontext learning (ICL), finding supportive pretraining data for downstream tasks with demonstration examples. 1
Understanding In-Context Learning via Supportive Pretraining Data
Assume we have a pretrained language model (LM) θ and data pairs (x, y) representing the inputs and ground truth outputs of task D task . Both x and y are in natural language. For classification tasks, the target labels can be converted to natural language via verbalizers (Schick and Schütze, 2021).
Understanding In-Context Learning via Supportive Pretraining Data
Zero-shot prompting A pretrained language model can be applied to perform downstream tasks via zero-shot prompting (e.g., Petroni et al., 2019). For classification tasks, the language model θ outputs a candidate answer with top probability, argmax y ′ ∈Y p θ (y ′ | x) = argmax y ′ ∈Y
Understanding In-Context Learning via Supportive Pretraining Data
where Y contains all candidate answers y ′ . For generation tasks, outputs can be obtained by sampling autoregressively from θ conditioned on x (e.g., Holtzman et al., 2019). This is a zero-shot scenario with no demonstration examples.
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning Instead of modeling p θ (y | x), ICL estimates p θ (y | {(x demo , y demo )}, x), prepending the original model input with several demonstration examples (x demo , y demo ) sampled from the target task D task . The language model θ is never trained on the task data with demonstrations. However, we can form a loss on the in-context data as a surrogate for θ's ICL performance, which will be used for a later guidance step,
Understanding In-Context Learning via Supportive Pretraining Data
Pretraining The pretraining data of θ often consists of texts w from large, general-domain corpora. During pretraining, the LM θ is updated via stochastic gradient descent with a loss to reconstruct w given a prefixing context, L PT θ (w) = − log t<|w| t=0 p θ (w t | w <t ). Supportive pretraining data Our goal is to locate what pretraining data w if upweighted would be most helpful to the LM θ's ICL ability. Following ORCA (Han and Tsvetkov, 2022), we use the similarity between gradients ∇ θ L PT θ (w) and ∇ θ L ICL θ (x, y) iteratively to find such supportive pretraining data. We show details of our adapted algorithm ORCA-ICL in Figure 2. The algorithm finds pretraining data that exert a gradient to θ similarly as a group of guidance ICL task data would. ∇ θ L ICL θ (x, y) provides a guidance for the direction the model parameters should be updated towards to be better at ICL, while ∇ θ L PT θ (w) approximates how the direction the model parameters would be updated based on individual pretraining instances. We conduct a multi-iteration process (a total of M iterations each selecting k supportive instances) to mitigate noise. 2 SGD denotes an one-pass stochastic gradient descent to mimick an incremental upweight to the selected data, with a minimum number of steps to prevent overfitting. The resulting supportive set S has a very small size (under 2000 in this work). 3
Understanding In-Context Learning via Supportive Pretraining Data
Verifying supportiveness To quantitatively evaluate the supportiveness of the selected set of pretraining data, we perform an one-pass gradient descent on the original LM with the selected set S, which mimics a perturbative continued pretraining with a minimum number of updates: θ M ← SGD S (θ 0 ). We then benchmark this perturbed model (θ M ) with the original model (θ 0 ) and a model perturbed with a random set of pretraining data. We expect the perturbed model using our selected supportive pretraining data to achieve a better ICL performance.
Understanding In-Context Learning via Supportive Pretraining Data
Algorithm 1 ORCA-ICL 1: Load a pretrained language model as θ0
Understanding In-Context Learning via Supportive Pretraining Data
9: end if 10: end for 11: Return supportive pretraining data S ← ∪ M i=1 Si Figure 2: ORCA-ICL, an iterative gradient-based selection of supportive pretraining data for ICL.
Understanding In-Context Learning via Supportive Pretraining Data
Language model Throughout the work, we use a pretrained, autoregressive OPT-6.7B (Zhang et al., 2022b) as our LM θ.
Understanding In-Context Learning via Supportive Pretraining Data
Tasks In this work, we focus on classification problems and first retrieve 48 classification-based tasks from Natural Instructions v2 (NI-v2, . We apply the LM on the tasks with both a zero-shot and in-context learning setup. We extract tasks that achieve at least 10% better performance with in-context demonstrations. We group 17 tasks that satisfies the constraint and further select 6 typical tasks among them: SST-2: Movie review sentiment classification (Socher et al., 2013). AG News: News topic classification (Zhang et al., 2015). Story Cloze Test: Story coherence classification (Mostafazadeh et al., 2017). SMS Spam Collection: Spam classification (Almeida et al., 2011). Sentiment 140: Tweet sentiment classification (Go et al., 2009). TweetQA: Answer verification (Xiong et al., 2019).
Understanding In-Context Learning via Supportive Pretraining Data
For each task, we randomly sample 500 examples with a balanced class distribution as D task , guiding the ORCA-ICL algorithm. The quantitative evaluation is performed on the full dataset. For ICL, for each instance in the task data, we randomly sample 4 demonstration examples under each candidate class defined in the task. 4 The order of demonstration examples in the context is randomly shuffled. The template and verbalizer of each task follows the original NI-v2 dataset, though we did not include the task instructions, as the focus of this work is in-context learning with demonstration examples.
Understanding In-Context Learning via Supportive Pretraining Data
Pretraining Considering the size of pretraining data D PT , we include an as large portion of OPT's pretraining data as possible under a reasonable budget. Specifically, in this work we use a total of 2.5M pretraining instances each consists of 2048 tokens. 5 For computing efficiency, we use intralayer model parallelism (Shoeybi et al., 2019) and fully sharded data parallel (Ott et al., 2021). 6 Implementation Details We run ORCA-ICL with a maximum of M = 5 iterations. In each iteration we extract k = 400 pretraining instances with top gradient similarity with the ICL task data. We use a batch size of 16 and learning rate of 2e-5 for the one-pass gradient descent with an Adam optimizer (Kingma and Ba, 2014). This results in a total of 125 updates 7 to the original LM after all iterations as the perturbative continued pretraining.
Understanding In-Context Learning via Supportive Pretraining Data
Perturbative continued pretraining As the main evaluation of the supportive pretraining data obtained by ORCA-ICL, we perform perturbative continued pretraining on both the selected supportive data and random pretraining data as a control. Table 1 shows the main results of task accuracy. The leftmost column shows a source task D task guiding the selection of supportive pretraining data. At each row, we evaluate the perturbed model (SGD S (θ 0 )) on all 6 tasks. The ICL performance of the original LM is reported in the headers of the table.
Understanding In-Context Learning via Supportive Pretraining Data
In each cell of the table, the top number shows the continued pretraining result with the supportive data we identified. We consider M ∈ [1, 5] iterations as a hyperparameter and report result with a best M . We want to know at a same size of selection, how our identified subset performs compared to random pretraining data. We therefore run random selection with 5 seeds, and the bottom number of the cell shows the continued pretraining result with random data at a same size of our selection, accompanied by a standard deviation. The performance of our selection is bolded when the performance difference with random selection exceeds one standard deviation.
Understanding In-Context Learning via Supportive Pretraining Data
The diagonal cells show the performance of perturbed models on the same task used for selecting supportive data. We observe on 4 of the 6 source tasks, our selection of supportive pretraining data is effective. For the cross-task performance, we observe on 5 of the 6 source tasks, our selection is effective for at least three tasks. 8 We conclude that our identified supportive pretraining data is overall effective for ICL, though the cross-task results show a portion of the ICL behavior can be task-specific and not universal across tasks.
Understanding In-Context Learning via Supportive Pretraining Data
Control evaluation on zero-shot data Being effective on the ICL data does not necessarily mean a direct support for a model's ICL ability, which is to learn from the demonstration examples. The test input can be a confounding factor: if our selection is effective as well on zero-shot test input without demonstrations, then the selection is not specific to the ICL ability. Therefore, we further confirm the supportiveness of our selected supportive pretraining data to ICL, contrastively in a zeroshot setup. We evaluate our models after perturbative continued pretraining in Table 1 on the same tasks but without the in-context demonstrations. We present the results in Table 2. The two columns show the zero-shot prompting performance of the original LM and the model after continued pretraining with our ICL-supportive selection, respectively. We do not observe performance gain for most tasks, indicating our selection is specific to the ICL ability without benefiting the zero-shot, no-demonstration task performance.
Understanding In-Context Learning via Supportive Pretraining Data
3 Analyzing supportive pretraining data for in-context learning
Understanding In-Context Learning via Supportive Pretraining Data
In the previous section, we identify a small subset of pretraining data that supports the ICL ability of language models. In this section, we analyze the selected supportive pretraining data to understand what makes them useful to ICL. Specifically, we compare the supportive pretraining data contrastively with randomly sampled pretraining instances, investigating three aspects of the pretraining data: the domain relevance to downstream  Table 1: Evaluation of supportive pretraining data to ICL. We obtain supportive pretraining data using the guidance of a source task and evaluate ICL on all tasks. In the headers, we show the ICL performance of the original LM. We perform perturbative continued pretraining with both our selected supportive data (top number in cells) and an equal number of randomly sampled pretraining data (bottom number in cells). Diagonal cells indicate same-task evaluation and are marked purple. Our performance is bolded when the difference exceeds one standard deviation. On 4 of 6 tasks, the same-task ICL performance gain is observed (diagonal). On 5 of 6 tasks, the corresponding supportive pretraining data improves ICL on at least three tasks (rows).  tasks, the token frequency distribution, and the information gain of incorporating long-range context.
Understanding In-Context Learning via Supportive Pretraining Data
Xie et al. (2022) and Min et al. (2022) imply that in-context demonstration is useful since it helps locate a particular domain or concept of the test input the LM already learned through the pretraining data. On the other hand, Olsson et al. (2022) imply that in-context demonstration is useful because the decision over the test input may be done through a soft-copy mechanism from the demonstration examples. These lead to two different expectations of the role of supportive pretraining data: (1) Inferred from Xie et al. (2022) and Min et al. (2022), the supportive pretraining data should be from a same domain as the demonstration and test examples, providing direct supporting knowledge to solve the downstream task.
Understanding In-Context Learning via Supportive Pretraining Data
(2) Inferred from Olsson et al. (2022), the supportive pretraining data should be beneficial to the soft-copy mechanism, providing meta support for the abstract ability, unconstrained with the concrete data domain. 9 We aim to measure the domain relevance between supportive pretraining data and downstream tasks.
Understanding In-Context Learning via Supportive Pretraining Data
Method To quantify domain relevance, we use MAUVE score (Pillutla et al., 2021) to measure an information divergence between two text distributions. We compute two MAUVE scores, between the target task data and our selected supportive pretraining data, and between the task data and ran- Figure 3: The MAUVE score between the supportive pretraining data and target task data, subtracted by the MAUVE score between random data and target task data. The error bars indicate the 95% confidence interval. No tasks show the supportive data has a significant higher domain relevance compared to random data.
Understanding In-Context Learning via Supportive Pretraining Data
dom pretraining data. We then compute and report their difference. A positive MAUVE difference indicates a higher domain relevance of our supportive pretraining data. 10 We use RoBERTa (Liu et al., 2019) as MAUVE's embedding model following He et al. (2022).
Understanding In-Context Learning via Supportive Pretraining Data
Results We show the difference of MAUVE scores in Figure 3. The error bar shows the 95% confidence interval using 32 random seeds. We find that for 5 of the 6 tasks, there is no significant difference between the MAUVE scores of supportive pretraining data and random data. For SST-2, the supportive pretraining data even shows a lower MAUVE score. Therefore, the supportive pretraining data to ICL do not have a higher domain relevance to the task, compared to general pretraining data. This result aligns with the domain relevance finding in Shin et al. (2022) where dataset-level analyses were performed. This implies the improved ICL behavior of our models may be a meta ability, aided by pretraining data unrelated to the specific domain knowledge for solving the task, but related to a domain-invariant mechanism to learn from a data's context. §3.3 continues this discussion.
Understanding In-Context Learning via Supportive Pretraining Data
Providing demonstrations to a task input under an ICL setup creates repetitions (e.g., of label tokens), which changes the token frequency distribution of the ICL task data. Therefore, we are interested in Figure 4: The difference in average Zipf's coefficients of the token frequency distribution of supportive pretraining instances and random examples. The error bars indicate the 95% confidence interval. We find a lower Zipf's coefficient for supportive pretraining data, indicating a flatter frequency distribution, with a relatively higher mass on the rare, long-tail tokens.
Understanding In-Context Learning via Supportive Pretraining Data
whether the supportive pretraining data possess a different token frequency distribution from general pretraining data. Experimented with sequences of image-label pairs, Chan et al. (2022) find that a skewed class distribution (high burstiness) and a large number of rarely occurring classes in training data promote the ICL ability of Transformer models (Vaswani et al., 2017). However, it is unknown whether the findings on the synthetic image-label data can transfer to the natural language pretraining data, a gap we address in this subsection.
Understanding In-Context Learning via Supportive Pretraining Data
We fit a Zipfian distribution over each supportive and random pretraining instance that consists of 2048 tokens. The Zipf's coefficient is the negative slope of a linear regression over the tokens' log-rank v.s. log-frequency. A higher Zipf's coeffcient indicates a higher mass on the frequent tokens (i.e., more skewed distribution). A lower Zipf's coefficient indicates a higher mass on the rare, long-tail tokens (i.e., flatter distribution).
Understanding In-Context Learning via Supportive Pretraining Data
In Figure 4, we show the difference in average Zipf's coefficients between supportive and random pretraining data, each with a group size of 2000. The error bar shows the 95% confidence interval with 32 random seeds. We find that for all tasks, the Zipf's coefficient of the supportive pretraining data is significantly lower than that of the random pretraining data. This indicates a flatter Zipfian distribution with a relatively higher mass over the long-tail tokens. In other words, though the overall burstiness of data is lower, there is a relatively higher amount of rarely occurring, long-tail tokens in the supportive pretraining data for ICL. Flatter frequency distribution also indicates higher entropy over the tokens, presumably making the supportive pretraining data challenging examples to fit by the model, a concept we explore further in the next subsection.
Understanding In-Context Learning via Supportive Pretraining Data
In §3.1, we find that the domain relevance of the supportive pretraining data to downstream tasks is not higher than that of random pretraining data. This is comprehendible if we follow the aforementioned perspective of Olsson et al. (2022), hypothesizing that there exists a soft-copy mechanism between the in-context demonstrations and test input. The supportive pretraining data may provide meta support for the abstract soft-copy mechanism rather than task-specific knowledge. We further hypothesize that to facilitate such meta support, the incorporation of long-range context during language modeling in supportive pretraining data should be different from random pretraining data, since the demonstration examples in the ICL setup is a form of long-range context. We propose a novel information gain measure to quantify this feature of incorporating long-range context.
Understanding In-Context Learning via Supportive Pretraining Data
Recall that the canonical definition of information gain (IG) is IG(T, a) = H(T ) − H(T | a), where T is a target variable, a is an attribute conditioned on by T , and H(·) computes entropy. It measures the decrease of entropy (thus the gain of information) in T if conditioned on a. We adapt the canonical IG to measure the decrease of cross entropy for each token (w i ) in a pretraining dataset when conditioned on a long (l) context over a short (s) context:
Understanding In-Context Learning via Supportive Pretraining Data
Ideally the length of long or short context should remain constant across different tokens w i , but it would be a very expensive computation due to a lack of parallelism. We approximate the computation by splitting a full sequence of pretraining tokens (e.g., 2048 tokens) to smaller blocks and calculate cross entropy with the boundary of blocks:
Understanding In-Context Learning via Supportive Pretraining Data
With the above definition, the average length of context for all w i is s and l, respectively. In the experiments below, we keep s = 128 for the length of short context and increase the length of long context at l = {256, 512, 1024}. We report the difference in the average information gain (across w i ) of incorporating long-range context for a language modeling objective, in supportive pretraining data over random pretraining data. Additionally, we want to use the defined information gain measure as a standalone feature of data, so we use a different LM to compute the cross entropy than the LM on which we perform ICL. Below we report results using OPT-1.3B, while experiments using OPT-350M shows a similar trend.
Understanding In-Context Learning via Supportive Pretraining Data
In Figure 5, we see for all of the experimented tasks, there is a significant trend that increasing the length l for the long-range context for supportive pretraining data has a lower relative information gain compared to random pretraining data. Though seeming counterintuitive at first glance, this suggests that the supportive pretraining data are more challenging examples in incorporating the long-range context information. 11 A possible explanation for this is that such challenging examples contain confounding spans that harms the information gain measure. The language model has to learn to decide which part of the longrange context is truly relevant to the prediction of next tokens. This would resemble more and thus helpful to the ICL task scenario where there are multiple demonstrations from different classes.
Understanding In-Context Learning via Supportive Pretraining Data
Despite our aforementioned findings, we mainly conduct correlational analyses throughout the work. Despite the potential confounding factors, future work can try converting the correlational findings to causal ones. For example, to actively refine or construct pretraining data to improve existing models' ICL performance, with a metric of token frequency distribution (i.e., find data with a higher mass of long-tail tokens) or context information gain (i.e., find difficult examples in incorporating long-range context). Additionally, we only investigate classification tasks in this work. However, the ORCA-ICL method can be applicable to generation tasks as well in the future, if the ICL loss is defined over a sequence probability of the generation. 
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning has shown superior performance on a range of NLP tasks, yet it remained unclear from where language models acquired this ability. We approach the problem by identifying a small subset of pretraining data that particularly supports language models to do in-context learning on downstream tasks. We analyze common features of the supportive instances in contrast to general pretraining data and find that: (1) The supportive pretraining data do not have a higher domain relevance to the downstream tasks. (2) The supportive data contain a relatively larger amount of rare, longtail tokens. (3) The supportive pretraining data are more challenging instances in incorporating longrange context in language modeling. Our findings may be beneficial to future work that refine or construct pretraining data, in order to actively improve existing models' in-context learning performance.
Understanding In-Context Learning via Supportive Pretraining Data
It is worth noting that the supportive pretraining data we investigated throughout the work is w.r.t. the current LM, such that a perturbative continued pretraining with the supportive data would improve the final LM checkpoint deployed to downstream tasks. It is possible that for some data which we did not determine as supportive, they had been supportive w.r.t. early checkpoints of the LM. With more computing resources, future work may investigate the trend of supportive patterns across multiple checkpoints of a LM throughout the pretraining process.
Understanding In-Context Learning via Supportive Pretraining Data
Additionally, another significant limitation of our work is the amount of involved computing resource. The ORCA-ICL method is gradient-based that requires back-propagation. Since we iterate through a large size of pretraining data, the cost of computation is similar to training a language model with a batch size of 1 on the considered pretraining data. On our 4 nodes each consists of 8 Nvidia V100 GPUs, finding the supportive pretraining data for each source task in our experiment would take about a week. One mitigating aspect of such computation is that the gradient calculation can be done asynchronously, therefore enabling the use of idle, leftover GPUs scattered across a cluster of nodes. We plan to explore efficient computation of gradient similarity or move from a paradigm of extracting supportive data to generating supportive data in future work.
Understanding In-Context Learning via Supportive Pretraining Data
Supportive pretraining data to ICL ... Samsung's new Odyssey+ headset could fix its muddled VR vision As one of the world's most technologically innovative companies, Samsung should be leading the pack in VR -one of the decade's top transformative technologies. Instead, it has largely let Microsoft and Facebook determine its role in the VR space, leading to its current situation as an also-ran. If I was betting on whether that will change anytime soon, an FCC leak of the company's new Odyssey+ VR headset (discovered by RoadtoVR) would point to "no." Most of the specs are staying the same as its prior, Windowsdependent Odyssey model: Each eye still gets a 3.5-inch screen with 1,440 by 1,600 resolution, combining for a 110-degree field of view, and AMOLED technology will be used to guarantee dark blacks and rich colors. There's one mystery in the new specs, namely a reference to the AMOLED screens now including something called "SFS." ...
Understanding In-Context Learning via Supportive Pretraining Data
... Bangladesh authorities and intelligence officials have long been saying that many of the refugees are involved in illicit drug trade, smuggling, robbery and ransom-seeking. Earlier Tuesday, the elite security agency Rapid Action Battalion arrested nine refugees suspected of being involved in various criminal activities. They had firearms, bullets and sharp weapons, Islam said. Local media reported that Tuesday' s chaos began after the arrest of the suspects as one group blamed another for helping the security agency in detaining them. Human rights groups that are involved in the camps acknowledge there are criminal elements among the Rohingya refugees. ... Table 3: Qualitative examples of the supportive pretraining data to ICL in the task of SMS spam detection. We also show an example of random pretraining data for comparison. As our finding on domain relevance suggested, neither of the examples are about SMS spam, so the language model may not learn direct knowledge about the task from supportive pretraining data to ICL. Compared to the random data, the supportive data to ICL has some relatively low-frequency tokens appear multiple times (e.g., VR, Odyssey, AMOLED) and the language model may learn some meta-knowledge about ICL (e.g., copying behaviors from the context) based on them. However, such patterns are sparse, noisy, and hard to analyze through manual inspections. We therefore present the quantitative analyses in the main paper.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Large language models have demonstrated outstanding performance in zero-and few-shot learning settings on numerous tasks, from simple classifications such as sentiment analysis to complex reasoning-related task like natural language inference and arithmetic (Brown et al., 2020;Radford et al., 2019). These results suggest that models may have gained the ability to perform simple inductive reasoning through a combination of pretraining and model size. shot accuracy on multiplication (averaged over multiple multiplicands and training instances) against the frequency of the equation's first term in the pretraining corpus. Each point represents the average performance for that term (e.g., 24) multiplied by numbers 1-50 and 5 choices of random seeds. As in the example, the performance difference for the numbers 24 and 23 is more than 20%. We find a strong correlation between accuracy and frequency.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
However, current evaluation schemes for the reasoning of large language models, often neglect or underestimate the impact of data leakage from pretraining data when assessing their reasoning ability. Although the overlap between the training and evaluation splits of public datasets and their effect on the generalization of the language models have been studied (Elangovan et al., 2021;Lewis et al., 2020), the effect of the pretraining data has gotten less attention. Traditionally, a model that has learned to reason in the training phase should be able to generalize outside of the narrow context that it was trained in. Specifically, if the model has learned to reason, its performance on instances with less frequent terms (based on pretraining data) should not be significantly lower than its performance on the instances with more common terms.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
As an illustration, consider the arithmetic task of multiplying two integers (shown in Figure 1). A model that has learned proper arithmetic skills should be able to answer the queries irrespective of the frequencies of the operands in the pretraining data. Therefore, it should have roughly equiv-arXiv:2202.07206v2 [cs.CL] 24 May 2022 alent performance when answering the queries Q: what is 24 times X? and Q: what is 23 times X?, when aggregated over various values of X. This is not the case with current LMs and we will study the effect of frequency terms in details through this paper. To show the effect of frequency, in this example, we plot the average accuracy of GPT-J-6B  on the numbers 0-100 (averaged over 1-50 as the other operand) against the frequency of the number in the pretraining data in Figure 1. We find a strong correlation between the term frequency and the model performance indicating that the model reasoning is not robust to these frequencies. Note that even "rare" terms are overall frequent (on the order of millions) in the pretraining data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
In this work, we investigate this impact of the frequency of test instance terms in the model's pretraining data on model's performance. We focus our analysis on numerical reasoning tasks, including addition, multiplication, and unit conversion. For each of these tasks, we identify relevant terms from each instance; for these tasks, terms are the numbers and units involved. We count occurrences of these terms in the pretraining data, including co-occurrences of term pairs or triples within a fixed window. This procedure allows us to aggregate over instances in which these terms appear and observe the relationship between term frequency and model accuracy on instances that include those terms. We summarize this behavior through the performance gap between instances that have the most frequent terms and instances that have the least frequent terms. Intuitively, models that exhibit a high performance gap are more accurate on instances that are more common in the pretraining data; this indicates that the model does not generalize appropriately and is likely affected by dataset overlap.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
We present analysis on these numerical reasoning tasks for three sizes of the EleutherAI/GPT models pretrained on the Pile (Gao et al., 2020) dataset, which has been publicly released and thus permits this kind of analysis (in contrast to the data that, e.g., GPT-3 (Brown et al., 2020) was trained on). Our results show a consistently large performance gap between highest-frequency terms and lowest-frequency terms in all of our experiments; in some cases there is more than 70% of average accuracy gap between the first and last 10% indicating that even simple unigram statistics are highly correlated with models performance which should not happen if the model is performing reasoning. These observations suggest that any evaluation of reasoning that does not take the pretraining data into account is difficult to interpret, and that we need to revisit evaluation of language models with respect to their pretraining data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Reasoning ability has long been considered as a proxy for intelligence (Johnson-Laird, 2010). Thus, developing models with this skill has been also an essential goal of AI and natural language processing (NLP) (Bommasani et al., 2021). Recently, large language models have exhibited an ability to perform reasoning-related tasks in few-shot settings without requiring any modifications to their parameters through a method called in-context learning. Our goal is to evaluate this reasoning skill in-depth for numerical induction tasks. This section provides background information on in-context learning and introduces our method for measuring the performance gap of the models on numerical reasoning tasks based on differences in pretraining term frequency.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
2.1. In-context Learning Brown et al. (2020) show that the large GPT-3 model is able to perform well on few-shot reasoning tasks without requiring any changes to its internal parameters, through the usage of a technique called in-context learning. In place of a typical learning procedure, in-context learning instead places training examples in a prompt format, which is subsequently fed to a language model as its input.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Among numerous experiments, Brown et al. (2020) show that GPT3 performs well on a variety of arithmetic questions such as addition and subtraction with 2-5 digit numbers. For example, they show that their largest model can perform zero-shot 2-digit addition with 76.9% accuracy. Although impressive, due to the large volume of data GPT-3 is trained on, it is possible that the model is merely repeating answers seen during pretraining. To attribute this performance to the model's reasoning capabilities, we need to make sure that the model is not affected by statistical overlaps between the terms of the arithmetic questions and the pretraining data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
In the following sections, we introduce metrics that we use to investigate the relationship between the frequency of terms in the pretraining data and the model performance on reasoning instances containing those terms. To assess this relation, we first define an approach for measuring term frequencies in a large pretraining dataset (Section 2.2). We connect these frequencies to reasoning performance by introducing the performance gap ∆ (Section 2.3).
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
We consider numerical reasoning tasks (Table 1) whose instances consist of input terms, x = (x 1 , . . . , x i , . . . x n ), and a derived output term y, where the x i 's are either positive integers or units of time (e.g., 1, 2, hour, etc.) and y is a positive integer. For example, for the task of multiplication, an instance might be x = (23, 18) and y = 414, representing the equation 23 × 18 = 414.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
For each instance, we extract counts of the number of times that a subset of its terms X ⊆ {x 1 , . . . , x n , y} appear within a specified window in the pretraining data. We refer to this count as the frequency, ω X , of X.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
In this paper, we restrict our attention to frequencies involving three or less input terms, e.g., x = (x 1 ) or (x 1 , x 2 ) or (x 1 , x 2 , x 3 ) and optionally the output term y, e.g.:
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
• ω {x1} : the number of times that x 1 (e.g., 23) appears in the pretraining data. • ω {x1,x2} : the number of times that the input terms x 1 (e.g., 23) and x 2 (e.g., 18) appear in the pretraining data within a specific window size. • ω {x1,y} : the number of times that the first input term x 1 (e.g., 23) and the output term y (e.g., 414) appear in the pretraining data within a specific window size.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Note that our usage of set notation in the subscript is deliberate; although x = (x 1 , x 2 ) and x = (x 2 , x 1 ) are not necessarily the same (e.g., order is important when representing the task instance), frequency is symmetric (e.g.,
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
We want to measure how much more accurate the model is on instances containing more versus less frequent terms in the pretraining data. We do this by calculating the differences in average accuracies of the instances in the top and bottom quantiles of the distribution over term frequencies, which we call the performance gap.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
, be a set of terms for a task and their associated term frequencies in the pretraining corpus. Given a task (e.g. addition), we create reasoning instances for each element of this set by instantiating uninstantiated values of x i , and deriving y if y / ∈ X (n) . We then measure the LM's accuracy a (n) over the set of instances, and repeat this process for all n ∈ [1, N ], producing a set Ω = {(ω (n) X , a (n) )}. The formula for the performance gap is then given by:
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
where Ω >90% is the top 10% of elements in Ω ordered by frequency, Ω <10% is the bottom 10%, and Acc(Ω ) is the average accuracy of elements in Ω . We introduce the following convenient abuses of notation ∆ 1 , ∆ 1,2 , ∆ 1,y , . . . , to denote the performance gap over the frequency distributions of ω {x1} , ω {x1,x2} , ω {x1,y} , . . . , respectively.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Concretely, for the multiplication example from Figure 1, x = (x 1 , x 2 ) and we consider the performance gap over frequencies ω {x1} . For each number (say 23), we count the number of times it appears in the pretraining corpus (ω {23} ), and compute the average accuracy of the model over all instances where the first operand is 23. The performance gap w.r.t. to ω {x1} for this task is the difference between the average accuracy over the top 10% and the bottom 10% most frequent numbers in the pretraining corpus. 
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
In this section, we describe our setup to measure the effect of pretraining data on the few-shot evaluation of a number of numerical reasoning tasks for different language models. For reproducibility, we will release the complete source code of our experiments.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Language Models We experiment on models from EleutherAI i.e., GPT-J-6B , and GPT-Neo-1.3B, GPT-Neo-2.7B (Black et al., 2021). These models are publicly available, but more importantly, the corpus used to pretrain them has also been released. We use the HuggingFace 1 Transformer integration of the models in our experiments.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Pretraining Corpus Language models studied in this work are trained on the Pile dataset (Gao et al., 2020), a large-scale language modeling dataset (800GB) consisting of English documents in 22 academic or other professional data sources. Since our reasoning tasks focus on numbers, we count the frequency of all integers with less than seven digits using a white-space tokenizer. To calculate the frequencies of the numbers we use Amazon Elastic Map Reduce (EMR) platform. We will release the code and statistics of the frequencies.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Numerical Reasoning Tasks We create three types of datasets that target the mathematical capabilities of language models since solving mathematical questions is a useful reasoning capability of the models (Brown et al., 2020).
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
• Arithmetic, 2 tasks As the first task, we consider simple arithmetic operations: addition x 1 + x 2 y and multipli-  Figure 2. Pipeline for Data Construction: We use the term counts processed from the pretraining data to develop the reasoning queries and render them with prompts templates to a proper language model input format (illustrated using the example from Figure 1). cation x 1 × x 2 y. In both cases, the first operands (x 1 ) are numbers less than 100 (that are in the top 200 most frequent numbers) and the second operands (x 2 ) are the numbers in the range (1 − 50).
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
• Operation Inference, 2 tasks Instead of directly specifying the operation, we also create a variation where the model needs to infer, from a few examples, the operation itself, as well as the result, as introduced in the evaluation of Megatron-Turing model 2 . We replace the arithmetic operation with a "#", with the same operations and operands as previous, to create these datasets.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
• Time Unit Conversion, 7 tasks Apart from directly executing arithmetic expressions, we are also interested in evaluating model capability to implicitly reason about these operations. To this end, we construct a unit conversion dataset by identifying the most frequent numbers that co-occur with time unit words ("second", "minute", "hour", "day", "week", "month", "year", and "decade") as the primary operand x 1 , the time units themselves as additional operands (x 2 x 3 ), i.e. converting 24 hours to minutes is represented as (24, "hours", 60). We expect converting time values to be mathematically more straightforward than two-digit multiplication since the model need only multiply with the same (implicit) second operand, e.g., ×60 for converting hours to minutes.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
The overall pipeline for creating natural language instances for these datasets is illustrated in Figure 2. We compute occurrences and co-occurrences (if they are within a window of 5 words) of the terms in the corpus, i.e. the time units and numbers. We then generate instances for each of our reasoning datasets using the most frequent terms (in the top 200) as operands with less than 3 digits. We focus on the top terms since we expect the models to have a fairly reliable and robust representations for these words. Each reasoning instance is rendered as a natural language query using the prompt templates from Table 1, and fed to the language model to generate the answer. For example, to create a multiplication instance given the argument (x 1 = 23, x 2 = 18), we use the instance template to create a natural language 2 https://turing.microsoft.com/ input for the model as "Q: What is 23 times 18? A: ", with the goal of producing "414" (y = 23 × 18 = 414).
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
For few-shot evaluation, we prompt the language models with k = 0, 2, 4, 8, 16 shots, and average performance over five random selection of the prompt instances, we randomly select k samples from the dataset as the training examples provided in the prompt and use the rest of the instances from the model as the test data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
With the three types of reasoning tasks (consisting of 11 total datasets), we present an evaluation of the effect of pretraining frequency on the performance of the language models. For each dataset, we measure the performance gap on instances that consist of rarer (relatively) terms, for a few different choices of what to compute frequency over i.e different combinations of the instance terms. To observe this effect with larger language models, we also investigate the effect of the model size on this performance gap.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Arithmetic We first study the performance on simple addition and multiplication of numbers. The results for the GPT-J-6B model is provided in Table 2, with performance gap computed just for x 1 , for (x 1 , x 2 ), and for (x 1 , y). In multiplication, we observe a very high performance gap for all these definitions of frequencies, suggesting a strong effect of frequency in the pretraining data on the model's ability to perform multiplication. For better illustration of the performance gap, we plot the mean accuracy across the frequency of x 1 in Figure 3b. The plot demonstrates the strong correlation between the models accuracy on specific instances, and the instance element frequency in the pretraining data. For addition, we observe an overall higher performance of the GPT-J-6B model in comparison to the multiplication experiments. However, the performance gap on all of the definitions of the instance frequencies still shows an strong effect on the models accuracy. As shown in Figure 3a, the average accuracy of the model still has a positive slope, indicating the effect of instance frequencies.  Operation Inference These tasks aim to assess the model capability to both infer the math operation and to perform the actual computation. Overall, as we see in Table 2, the model is much less accurate here as compared to the arithmetic experiments. However, the model has better performance on the frequent instances even for these low performance tasks (see detailed trend in Figures 3d and 3c). The performance gap here suggests that the effect of pretraining is not only for tasks that the model is accurate on, but even for operation inference that is more challenging and require deeper reasoning. An additional interesting trend is in the last column of Table 2 with the steady growth of performance gap with respect to ω {x1,y} (co-occurrence of the instance input element and the true answer) as the number of shots increases and the model gets better in performance. Moreover, the lower accuracy here as compared to addition experiments in the previous section suggests that the model is unable to infer the operation from the few-shot prompts, and it may be performing some form of pattern matching based on the pretraining data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
The performance gap evaluated on all the time unit conversion experiments is provided in Table 3. We first observe a relatively high performance gap on all the tasks except the conversion from decade to year. For example, Figure 4 illustrates the trend of increased model performance with the frequency of instance elements for converting the time values from years to months. We also observe a general pattern of increase in the performance gap as the number of shots (training examples in the prompt) increases (results are in table 3). These results suggest that even though the model gets more accurate, the improvements focus on more frequent instances of the task. Decades to years: As we can observe in Figure 5, The model performs nearly perfectly on this task with as few as 8 shots, and we only see very small performance gap s. This is likely due to the task being quite simple (appending a "0" to the input number) and the model is able to generalize in the manner we are evaluating it. However, it is also possible that we are simply not identifying the right frequency statistics for this task, and there is an effect that our current evaluation setup does not capture.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Studying the Size of Language Models To further study the impact of language models sizes on the performance gap caused by the instance frequencies, we perform the Table 3. GPT-J-6B results on Time-Unit Conversion: ∆1,2, ∆1,2,3 and ∆1,2,y represent the performance gap over the frequency distributions of ω {x 1 ,x 2 } , ω {x 1 ,x 2 ,x 3 } and ω {x 1 ,x 2 ,y} respectively, where x1 is the number operand, x2 is the source time unit, x3 is the second implicit number operand needed for performing the conversion and the y is the true answer.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
k
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Week Day
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Acc. ∆1,2 ∆1,2,3 ∆1,2,y Acc. ∆1,2 ∆1,2,3 ∆1,2,y Acc. ∆1,2 ∆1,2,3 ∆1,2,y Acc. ∆1,2 ∆1,2,3 ∆1,2,y    arithmetic experiments for 2 and 8 shots using a variety of models (including the smaller versions of models GPT-Neo-1.3B and GPT-Neo-2.7B). We can see the trends of the average accuracy of the models in Figures 6a and 6b. The smaller models overall are less accurate on the arithmetic tasks, which is consistent with observations in related work (Brown et al., 2020). However, their success is still focused on the more frequent terms from the pretraining corpus, suggesting that even the smaller models show the effect of reliance on the pretraining data, although to a much lower extent than the larger ones.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Summary Overall, we observe high positive performance gap for almost all of the experiments on the three definition levels of the frequency for each task. This suggests a strong effect of frequency of the instances in the pretraining data on the model performance. In particular, evaluation using performance gap with ω {x1} shows that even the unigram statistics of the instances have strong correlation with the models performance on the instance.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Other than some exceptional cases, we observe an increasing trend in the performance gap as we put more training instances in the prompt (the number of shots); this can be a further indication that the model is directed through the patterns in the pretraining data to answer the reasoning questions. Our experiments with the smaller sizes of the model also show that they can only solve the frequent instances of the tasks, which further supports our observation that model performance is correlated with the term frequencies. . The effect of model size on performance: Smaller models only perform well on instances with more frequent terms in the pretraining data. k represents the number of shots.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Memorization Feldman (2020) provide a theoretical definition of memorization as the difference between the accuracy of a model on a training data point when that point is included vs. excluded from training. In subsequent work, they develop an approach for approximating memorization using influence functions Feldman & Zhang (2020). This framework is applied to study memorization in language models by Zhang et al. (2021), who find that training examples that are memorized by the LM tend to have high influence of LM predictions on similar validation instances. Their result may provide a plausible explanation that the frequency effects observed in this work are due to memorization.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Training Artifacts Challenge Evaluation Our results raise the issue that in-context learning probes may overestimate an LM's ability generalize from few examples when biases are present in the training data. This is consistent with prior work that has exposed the similar effects of biases from: lexical cues in natural language inference datasets (Gururangan et al., 2018;Poliak et al., 2018;McCoy et al., 2019), question-passage overlap and entity cues in reading comprehension datasets (Chen et al., 2016;Sugawara et al., 2018;Jia & Liang, 2017;Lewis et al., 2021), gender cues in coreference resolution datasets , popularity in named entity disambiguation (Chen et al., 2021), similarity between training and test instances in information extraction and sentiment analysis datasets (Elangovan et al., 2021), and effects of how data is split (Gorman & Bedrick, 2019;Søgaard et al., 2021). Relatedly, data poisoning research studies how to adversarially introduce artifacts into training data to produce unwanted model behaviors (Nelson et al., 2008;Chan et al., 2020;Wallace et al., 2021). A general statistical procedure to test for artifacts is presented in Gardner et al. (2021), who also theoretically show that large datasets are almost certain to contain artifacts under reasonable assumptions. Techniques for mitigating biases in the presence of dataset artifacts are covered by Romanov et al. (2019) andKarimi Mahabadi et al. (2020).
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
To better understand the risks of dataset artifacts, there has been a call to better document the characteristics and intended uses of datasets Bender et al., 2021). However, due to the sheer size of LM pretraining datasets-which range from 100's of GBs to 10's of TBs-doing so can pose a substantial challenge. Despite this, researchers have been able to estimate word frequencies, topics, and genres of documents (Sharoff, 2020), as well as proportions of toxic text (Gehman et al., 2020) appearing in OpenWebText (Gokaslan & Cohen, 2019). Similar efforts have been made to characterize the top-level domains, amount of hate speech, and censured text appearing in the C4 corpus (Raffel et al., 2020;Dodge et al., 2021;Luccioni & Viviano, 2021 Patel et al. (2021) also showed that NLP math solvers rely on simple heuristics to answer math questions. We expect that the performance gap metric proposed in this work will be useful to better understand the impact of such schemes.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
In this work, we consider how to conduct few-shot evaluations in light of the analysis with respect to the pretraining data. Prior work has attempted to control for overlap between training or pretraining data and the test instances, but as we have seen, those methods are insufficient. For example, Brown et al. (2020) measure the impact of removing instances from evaluation datasets that share 13-gram overlap with their pretraining data on GPT-3's accuracy, and also argue that the low occurrence of exact phrases such as "NUM1 + NUM2 =" and "NUM1 plus NUM2" in the pretraining data indicate that the model's strong performance on arithmetic tasks is likely due to factors other than memorization. However, we have seen that model performance is impacted by much simpler statistical patterns, as small as unigram overlaps with the pretraining data.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
For these reasons, we strongly recommend that evaluation of reasoning capabilities should take the pretraining corpus into account, and any claims of reasoning can only be made after demonstrating robustness to the effect of pretraining.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
LM benchmarks typically have no reference to the model's pretraining data. However, it is impossible to interpret fewshot performance on any benchmark without reference to information from the data that the LM was trained on. One possible addition to future evaluation is the performance gap between high-frequency and low-frequency terms, perhaps including only terms that have been seen more than some threshold value. It is worth mentioning that, even a performance gap of 0 is likely not sufficient to demonstrate a claim of reasoning capabilities-what exactly constitutes "reasoning" remains ill-defined-but it may be a necessary condition, and one that current models do not meet.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
There are a few limitations to our study that open up avenues for future research. We are not making a causal claim here, and in general, there may be confounders that we have not eliminated in our study. We recommend further research in investigating methods in causal inference and interventions during training to provide finer-grained analysis of the effect of pretraining. Since our approach aggregates fairly simple patterns, the effect we observe might be stronger if a wider variety and complexity of patterns is considered in the pretraining corpus. Similarly, our work was also limited to numerical reasoning tasks, and it would be worthwhile to consider how much other reasoning capability evaluations are impacted by the same effect, which could be measured using the performance gap metric introduced here. Defining appropriate instance terms for other reasoning tasks such as commonsense reasoning will be a challenging but important direction for future work. With the insights in this work and these recommendations, we hope to inspire further studies into the effect of pretraining on LM performance.
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
We show that in-context language model performance on numerical reasoning tasks can be impacted significantly by low-order co-occurrence statistics in the pretraining data, raising questions on the extent to which these models are actually reasoning to solve these tasks. These observations suggest the necessity for reconsidering and redefining the reasoning evaluation schemes for the large language models. Further characterizing the impacting factors on the models reasoning capacities is also an important tasks for the community. Most importantly, we suggest that the NLP community should not treat the pretraining data of the large language models as unknown black boxes. Overlooking the impact of the pretraining data can be misleading in evaluating the model reasoning skills.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Large transformer-based language models show an intriguing ability to perform in-context learning (Brown et al., 2020). This is the ability to generalize rapidly from a few examples of a new concept on which they have not been previously trained, without gradient updates to the model. In-context learning is a special case of few-shot learning in which the output is conditioned on examples from a 'context', and where there are no gradient updates. It contrasts with 'in-weights' learning, which is (a) Model, inputs, and outputs.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(b) Sequences for training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(c) Sequences to evaluate in-context learning.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(d) Sequences to evaluate in-weights learning. Figure 1: Experimental design, as described in Section 2. (a) For each experiment, a transformer model is trained on sequences of image-label pairs. The model is trained to minimize the loss on predicting the label corresponding to the final 'query' image. (b) In training, image-label mappings are fixed across sequences, in contrast to few-shot meta-training. The training data consist of a mix of 'bursty' and 'non-bursty' sequences. Bursty sequences, featuring multiple occurrences of the same classes, can be solved by learning labels across sequences (in-weights learning), or referring back to the context (in-context learning). Non-bursty sequences were composed of i.i.d. images. (c) To evaluate few-shot in-context learning, the model is presented with a standard few-shot sequence. The holdout image classes were never encountered in training, and are randomly assigned to labels {0,1}. Thus the model must use the context to predict the query label. (d) To evaluate in-weights learning, the model is presented with sequences where the labels are the same as in training. However, the query class does not appear in the context. Thus, the model must used information stored in weights to predict the query label. In the example sequences, we add colors and use only Latin characters for visualization purposes.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
the standard setting for supervised learning -this is slow (requiring many examples), and depends on gradient updates to the weights. Earlier work in the context of 'meta-learning' showed how neural networks can perform few-shot learning without the need for weight updates (Santoro et al., 2016;Vinyals et al., 2016;Wang et al., 2016). To achieve this, the researchers explicitly designed the training regime to incentivize in-context learning, a process sometimes called 'meta-training'. In the case of transformer language models, however, the capacity for in-context learning is emergent.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Neither the model's transformer architecture nor its learning objective are explicitly designed with in-context learning in mind.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Here, we consider the question of how transformer language models are able to acquire this impressive ability, without it being explicitly targeted by the training setup or learning objective. The emergence of in-context learning in language models was observed as recurrent models were supplanted by transformers, e.g. in GPT3. Was the novel architecture the critical factor behind this emergence? In this work we explore this possibility, as well as a second: that a capacity for in-context learning depends on the distributional qualities of the training data.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
This hypothesis was inspired by the observation that many natural data sources -including natural language -differ from typical supervised datasets due to a few notable features. For example, natural data is temporally 'bursty". That is, a given entity (word, person, object, etc) may have a distribution that is not uniform across time, instead tending to appear in clusters (Altmann et al., 2009;Alvarez-Lacalle et al., 2006;Lambiotte et al., 2013;Neuts, 2007;Sarkar et al., 2005;Serrano et al., 2009). Natural data also often has the property that the marginal distribution across entities is highly skewed, following a Zipfian (power law) distribution with a long tail of infrequent items (Piantadosi, 2014;Smith et al., 2018;Zipf, 1949). Finally, the 'meaning' of entities in natural data (such as words in natural language) is often dynamic rather than fixed. That is, a single entity can have multiple possible interpretations (polysemy and homonymy, in language) and multiple entities can map to the same interpretation (synonymy, in language), usually in a context-dependent way. The combination (a) In-context learning on holdout classes.  Figure 2: Effects of burstiness. P (bursty) indicates the proportion of training sequences that were bursty vs non-bursty, and models are evaluated on the two types of evaluation sequences, over the course of training. Burstiness in the training data increases in-context learning, and decreases inweights learning. Also, over the course of training, in-context learning tends to decrease while inweights learning increases.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
of these properties may result in training data that occupies some middle-ground between the data used in canonical supervised learning and that used for few-shot meta-training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In particular, standard supervised training typically consists of item classes that recur with uniform regularity, and with item-label mappings that are fixed throughout training -these properties allow a model to gradually learn over time, by encoding information into its weights, e.g. via gradient descent. By contrast, few-shot or in-context meta-training generally involves training a model directly on specially crafted sequences of data where item classes only recur and/or item-label mappings are only fixed within episodes -they do not recur and are not fixed across episodes (Santoro et al., 2016;Vinyals et al., 2016). Naturalistic data, such as language or first-person experience, has characteristics of both of these data types. As in supervised training, items (words) do recur, and the relationship between an entity and its interpretation (or meaning) is fixed, to some degree at least. At the same time, the skewed and long-tailed distribution of natural data means that some entities recur very frequently while a large number recur much more rarely. Importantly, however, these rare items are often bursty, making them disproportionately likely to occur multiple times within a given context window, somewhat like a sequence of 'meta-training' data. We can also see the dynamic relationship between entities and their interpretation (epitomized by synonyms, homonyms, and polysemy, in the case of language) as weaker versions of the completely dynamic item-label mappings that are used in few-shot meta-training, where the mappings are randomly permuted on every episode.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In this paper, we experimentally manipulated the distributional properties of the training data and measured the effects on in-context few-shot learning. We performed our experiments over data sequences sampled from a standard image-based few-shot dataset (the Omniglot dataset; Lake et al., 2019). At training, we fed each model (such as a transformer or recurrent network) with input sequences of Omniglot images and labels, varying the natural data-inspired distributional properties of choice. At evaluation, we assessed whether these properties gave rise to in-context learning abilities.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Our results showed that, indeed, in-context learning emerges in a transformer model only when trained on data that includes both burstiness and a large enough set of rarely occurring classes.We also tested two instantiations of the kinds of dynamic item interpretation observed in natural datahaving many labels per item as well as within-class variation. We found that both interventions on the training data could bias the model more strongly towards in-context learning. The models we tested typically exhibited a tradeoff between rapid in-context learning vs. relying on information that was stored through slow, gradient-based updates ('in-weights" learning). However, we found that models could simultaneously exhibit both in-context learning and in-weights learning when trained on a skewed marginal distribution over classes (akin to the Zipfian distribution of natural data).
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
At the same time, architecture is also important. Unlike transformers, recurrent models like LSTMs and RNNs (matched on number of parameters) were unable to exhibit in-context learning when trained on the same data distribution. It is important to note, however, that transformer models trained on the wrong data distributions still did fail to exhibit in-context learning. Thus, attention is not all you need -architecture and data are both key to the emergence of in-context learning. To investigate the factors that lead to in-context few-shot learning, we created training and evaluation sequences using the Omniglot dataset (Lake et al., 2019, MIT License), a standard image-label dataset for few-shot learning. Omniglot consists of 1623 different character classes from various international alphabets, with each class containing 20 handwritten examples. Using the Omniglot dataset allowed us to apply evaluation procedures that are standard in the study of few-shot learning.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
The few-shot challenge is to classify an example of a character class that was never seen in training, based only on a few examples of that class and some alternate classes.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
The training data consisted of sequences of images and labels (Fig 1b). The first 16 elements of each sequence comprised the 'context', and consisted of 8 image-label pairs (where each image was always followed immediately by its corresponding label). The final element was the 'query' image, and the aim of the model was to predict the correct label for the query.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Images were allowed to recur throughout training, and the integer label for each image class was unique and fixed across training, as in typical supervised datasets. We emphasize that this is a major departure from conventional few-shot training, where item-label mappings are completely novel on each episode, or the items themselves are novel on each episode.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In our standard experiments, we trained the model on a mixture of 'bursty' and 'non-bursty' sequences.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In the bursty sequences, the query class appeared 3 times in the context. There are many possible ways to quantify or instantiate burstiness (e.g., Altmann et al., 2009;Alvarez-Lacalle et al., 2006;Lambiotte et al., 2013;Neuts, 2007;Sarkar et al., 2005;Serrano et al., 2009), but the 'bursty' sequences in our experiment were designed to reflect the within-context burstiness that is observed in e.g. language. To prevent the model from simply outputting the most common label in the sequence, a second imagelabel pair also appeared 3 times in the context. For the non-bursty sequences, the image-label pairs were drawn randomly and uniformly from the full Omniglot set. We can continuously vary the overall degree of burstiness in a dataset by changing the proportion of 'bursty' vs 'non-bursty' sequences.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Each element of a sequence was first passed through an embedder (a standard embedding layer for the integer labels, and a ResNet for the images; He et al., 2015). These embedded tokens were passed into a causal transformer model (Fig 1a) (Vaswani et al., 2017). Unless stated otherwise, we used a transformer with 12 layers and embedding size 64. The model was trained on a softmax cross-entropy loss on the prediction for the final (query) image.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
We evaluated trained models on two types of sequences, to measure (1) in-context learning and (2) in-weights learning. As in the training sequences, the evaluation sequences also consisted of 8 pairs of 'context' image and label tokens, followed by a single 'query' image token.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
To measure a trained model's ability for in-context few-shot learning, we used a standard few-shot setup. The context consisted of a random ordering of 2 different image classes with 4 examples each, and the query was randomly selected from one of the two image classes (a '4-shot 2-way' problem, in few-shot nomenclature). Unlike in training, where the labels were fixed across all sequences, the labels for these two image classes were randomly re-assigned for each sequence. One image class was assigned to 0, and the other to 1 (Fig 1c). Because the labels were randomly re-assigned for each sequence, the model must use the context in the current sequence in order to make a label prediction for the query image. Unless stated otherwise, in-context learning was always evaluated on holdout image classes that were never seen in training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Although the model is always required to perform a full multi-class classification over all possible output labels (as in training), few-shot accuracy is computed by considering the model outputs only for the two labels seen in the few-shot sequence (0 and 1), with chance at 1/2. This ensures that performance above chance cannot be due to e.g. randomly selecting one of the labels from the context. Note also that the model was evaluated for in-context learning on novel image classes, but not novel labels (see the appendix for further discussion).
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
To measure in-weights learning of trained classes in a model, evaluation sequences consisted of image classes that were selected uniformly without replacement, with the same labels that were used in training (Fig 1d). Because the image classes were forced to be unique within each sequence, the query had no support in the context. Thus, the only way for a model to correctly predict the label was to rely on information stored in the model weights. For this problem, where the correct query label could be any of the labels seen in training, chance was usually 1/1600.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
3.1 What kinds of training data promote in-context learning?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Burstiness. In our first experiments, we vary levels of burstiness in the training data by varying the proportion of bursty vs non-bursty sequences in the training data (as described in Section 2.1). These experiments replicate the finding that transformers can acquire in-context few-shot learning even without explicit meta-training. They further show that, as hypothesized, the model displays better in-context learning with more burstiness in training (Fig 2a). We also see that in-context learning trades off against in-weights learning -greater burstiness simultaneously leads to lower weight-based learning (Fig 2b). Interestingly, the models can in some cases lose an initial bias towards in-context learning, moving towards in-weights learning over the course of training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
A large number of rarely occurring classes. Our second set of experiments show that in-context learning performance depends on the number of training classes (keeping the level of burstiness fixed at p(bursty) = 0.9). As we increase the number of classes from 100 to 1600 (and correspondingly decrease the frequency of each class), we see improvement of in-context learning (Fig 3a). As before, we also see an accompanying decrease in in-weights learning (Fig 3b). This accords with our hypothesis about the importance of having a long tail in the distribution, or a large vocabulary. Note that the bias against in-weights learning cannot be explained by the number of exposures to each class -even controlling for the number of exposures, the model trained with 1600 classes is much slower to achieve similar levels of in-weights learning. Importantly, we need both burstiness and a large number of classes for in-context learning to emerge. In order to further increase the number of classes beyond the 1623 available in the original Omniglot dataset, we rotated (0 • , 90 • , 180 • , 270 • ) and flipped (left-right) the images, obtaining 8× more image classes. We ensured that the holdout set did not include transformed versions of train images. Training on these 12800 classes further improved in-context learning (and reduced in-weights learning) (Fig 3). However, some images in Omniglot have rotational or mirror symmetries, so that the models trained on 12800 classes may additionally be pushed towards in-context learning by a label-multiplicity effect, described next.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Multiplicity of labels. Our third set of experiments explored the effect of dynamic meanings, with training distributions where images did not have completely fixed labels. Each image class was assigned to multiple possible labels and, in the data sequences, the label shown after each image was randomly selected among the possible labels. If a class appeared more than once in the same sequence, the label was consistent for all presentations within that sequence (this is commonly the case in natural data such as language, too; Gale et al., 1992). In Fig 4, we see that increasing the 'label multiplicity' (the number of labels per class) also increases in-context learning. Again, burstiness was fixed for these experiments at p(bursty) = 0.9.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(a) In-context learning on holdout classes.  Within-class variation. We then explored another source of dynamic variation of meaning -the amount of variation within image classes themselves. In the lowestvariation condition, each image class consists of only a single image, i.e. the images for a given class were always identical. In the medium-variation conditions, we added Gaussian pixel noise to the images (resampled for each presentation). In the highvaration condition, we used the full Omniglot classes (each class consists of 20 different images drawn by 20 different people). To our surprise, we found that greater within-class variation leads to greater incontext learning (Fig 5). In other words, making the generalization problem harder actually made in-context learning emerge more strongly -it preferentially hampered in-weights learning more than it hampered in-context learning. Across all the above experiments (Figs 2-5), we also evaluated in-context learning on training classes (rather than holdout classes), again randomly assigning the classes to labels 0 and 1 (rather than using the ones seen in training). Evaluations looked similar in all cases, with only slightly higher performance (Appendix C.3. 3.2 What kinds of training data enable in-context learning and in-weights learning to co-exist in the same model?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In the previous section, we saw a consistent tradeoff between in-context learning and in-weights learning -no models could maintain both. However, it is useful for a model to have both capabilities -to remember information about classes that will re-appear in evaluation, while also being able to perform rapid in-context learning on new classes that appear only in holdout. Large language models certainly do have both of these capabilities. How might we achieve this?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
For all prior experiments, the training data were marginally distributed uniformly over classes, even if the data were non-uniform in other ways. I.e., each class was equally likely to appear, marginalizing across the dataset. We postulated that we might achieve both types of learning in the same model by instead training on marginally-skewed distributions. In this case, some classes appear very commonly, while most classes appear very rarely. Many natural phenomena such as word distributions take this form, and are classically described as a Zipfian (power law) distribution (Zipf, 1949):
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Here, X is the rank of the class (e.g. 1 for the most common class), and the exponent α ∈ [0, ∞) determines the degree of skew. Fig 6a shows Kucera, 1979). 2 This type of skewed distribution could allow a model to learn common classes in its weights, while the long tail of rare classes simultaneously induces an ability for in-context learning.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
To test this hypothesis, we trained on Zipfian distributions, varying the Zipf exponent and hence the degree of skew. We used the same training sequences as before, with 12800 classes and p(bursty) = 0.9. Our results are shown in Figs 6c-e. We evaluate in-weights learning separately on common classes (the 10 classes seen most often in training) and on rare classes (the remaining classes). When there is no skew, all classes are relatively rare, and we see high levels of in-context learning but no in-weights learning. Increasing the skew leads to the loss of in-context learning and increased in-weights learning of common classes. 3 In between the two extremes, we observe a sweet spot at Zipf exponent = 1, where the model maintains high levels of both in-context learning and in-weights learning of common classes. Intriguingly, natural languages are best described by a Zipfian distribution with an exponent of approximately 1 (Piantadosi, 2014). Note though that the sweet spot for simultaneously maintaining in-weights and in-context learning in transformers may differ, depending on the training regime.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(a) Transformer. We compare architectures while holding fixed the number of layers, hidden layer size, and number of parameters. Only a transformer is able to attain in-context learning; the Vanilla RNN and LSTM never perform above chance. One run was performed for each set of hyperparameters in a hyperparameter sweep. Each color denotes one run, but not any particular hyperparameter values.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
To investigate whether these results are specific to transformer models, we performed similar experiments using recurrent sequence models. For these models, we simply replaced the transformer with either a vanilla recurrent neural network (RNN; David E. Rumelhart et al., 1985) or a long short-term memory network (LSTM; Hochreiter and Schmidhuber, 1997). We used the same training sequences as before, with 1600 classes and p(bursty) = 0.9. We also used the same image and label encoders, and cross-entropy classification loss. The recurrent models were matched to the transformer for depth, number of parameters, and hidden layer size. We performed a comprehensive hyperparameter search for all models (see Appendix for details).
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In these experiments, we see that the recurrent models are never able to achieve in-context learning, despite the parity in training setup (Fig 7). Interestingly, the transformer actually outperforms the recurrent models on in-weights learning as well (see Fig 8 in the Appendix), indicating that we cannot explain these results by proposing that recurrent models are simply more biased towards in-weights learning than transformers.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In summary, we find that both data and architectures contribute significantly to the emergence of in-context learning in transformers.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Data properties that promote in-context learning. We identify several features of training data that can promote in-context learning -burstiness, number and rarity of training classes, and dynamic meaning (as instantiated by multiple labels per class or within-class variation). These data properties allow in-context learning to emerge despite differing significantly from the data used in standard few-shot meta-training, in that we allow items and item-label mappings to recur throughout training. These properties are also central features of natural data including language, and thus may explain the remarkable emergence of in-context learning in large language models without explicit meta-training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Effects of architecture. We find that architecture does matter as well. Transformers show a significantly greater capacity for in-context learning than recurrent models -we were completely unable to elicit in-context learning in recurrent models, even with the training procedure, number of parameters, and model architecture otherwise matched to the transformer experiments. We emphasize however that the transformer architecture alone was insufficient for eliciting in-context learning -it was necessary for the training data to exhibit at least burstiness and large numbers of classes, too.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In-context vs. in-weights learning. In most cases, we found that transformers exhibited a tradeoff in their bias towards either in-context learning or in-weights learning, and could not maintain both in the same model. We characterize this behavior as a 'bias', because neither type of learning is 'correct' per se. For our training data, an in-context learning strategy and an in-weights learning strategy will give the same answer, since the labels are fixed. Thus, in the in-context evaluation sequences, it is ambiguous (by design) whether the model should use the labels seen in training or in the current context, allowing us to measure the model's bias. We also note that even models with an initial bias towards in-context learning can often move towards in-weights learning with enough repetition.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
However, it is often important and useful for a model to exhibit both capabilities -to perform slow, gradient-based in-weights learning of class information that is presented during training, while also being able to quickly learn (without weight updates) about new classes that appear only in evaluation. Indeed, large language models exhibit both of these capabilities (Brown et al., 2020).
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
In our experiments, we discovered that an additional language-like distributional property could allow models to maintain both capabilities as well -a skewed, Zipfian distribution over classes. This allowed the models to retain information in their weights about common classes, while simultaneously developing in-context learning abilities that were presumably induced by the long tail of rare classes.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Implications for understanding language models. Our findings have a few noteworthy implications. First, by pointing to specific distributional properties of training data that both exist in language and also promote in-context learning, these results may help us reach a more scientific understanding of why in-context learning emerges in transformer-based language models. This is an area of increasing interest (e.g. Min et al., 2022;Razeghi et al., 2022;Webson and Pavlick, 2021;Xie et al., 2021).
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
We emphasize that the transformers in our experiments successfully performed in-context evaluation on holdout classes, and only performed slightly better with in-context evaluation on trained classes. These results are counter to an emerging narrative that large language models may not actually be performing genuine in-context learning, and simply draw on examples seen in training (Min et al., 2022;Razeghi et al., 2022;Xie et al., 2021) -our experiments show that naturalistic distributional properties can give rise to a capacity for in-context learning on classes that were never seen in training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Broader implications. This understanding may also help us design and collect datasets to achieve in-context learning in domains outside of language, an area of ongoing research (e.g. Finn et al., 2017;Wang et al., 2016). Given that reinforcement learning environments are generally designed to be uniformly distributed (Chan et al., 2022), or that supervised datasets are frequently rebalanced to have more uniform distributions (Chawla et al., 2002;Katharopoulos and Fleuret, 2019;Van Hulse et al., 2007), we may be missing an opportunity to endow non-language models with a powerful capability. We may need to consider data distributions more carefully when pre-training in non-language domains, as well. For example, recent work has shown that pre-training on language data was useful for offline reinforcement learning, but pre-training on vision data was not (Reid et al., 2022) -could this difference be due to the non-uniform, structured distribution of the language data?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Cognition and neuroscience. Our experiments could also potentially inspire research on the role of non-uniformity in human cognitive development. Infants rapidly learn statistical properties of language (Saffran and Kirkham, 2018) -could these distributional features help infants to acquire an ability for rapid learning, or serve as useful pretraining for later learning? And could non-uniform distributions in other domains (e.g., vision) also contribute to this development (cf. Smith et al., 2018)?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Our results may also relate to complementary learning systems theory Mc-Clelland and O'Reilly, 1995) and its application to language understanding in the brain (McClelland et al., 2020). According to this theory, the neocortical part of the language system bears similarities to the weights of neural networks, in that both systems learn gradually through the accumulated influence of large amounts of experience. The hippocampal system plays a role similar to the context window in a transformer model, by representing the associations encountered most recently (the hip-pocampus generally has a time-limited window; Squire, 1992). 4 In this light, it is possible to see the human hippocampal system as a system that provides the architectural advantage of the transformer's context representations for in-context learning.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Future directions. The above results suggest exciting lines of future research. How do these data distributional properties interact with reinforcement learning vs. supervised losses? How might results differ in experiments that replicate other aspects of language and language modeling, e.g. using symbolic inputs, training on next-token or masked-token prediction, and having the meaning of words determined by their context? For models that display both in-context and in-weights learning, it would be interesting to understand contextual cuing of already learned information -does this increase with more exposure? There is also a lot more to understand about the behaviors and biases of transformers vs. recurrent architectures -why do transformers seem to be more capable of in-context learning?
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Non-uniformity. Finally, we hope to emphasize the dual nature of non-uniformity in training data. While it can impair both supervised and reinforcement learning (Chan et al., 2022;Van Hulse et al., 2007), we show here that non-uniform training distributions can induce the emergence of at least one useful and interesting capability, and thus can be an opportunity as well as a challenge.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
An important constraint of the model implementation and evaluation procedure is that we do not require the models to handle novel image labels, only novel image classes. Thus, in-context learning is evaluated on labels that were previously seen in training, i.e. 0 and 1 (on the Zipfian-skewed experiments, these corresponded to two most common labels). Note that, if anything, this causes in-context learning to be more difficult for the model, since it must overcome existing image-label associations that were learned in training.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
However, as future extensions, it would be possible to extend the model to handle novel labels as well.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
For example, we might tie the input and output embedding layers (sometimes done in large language models, though mainly for computational efficiency), or to generate novel labels as combinations of already-seen tokens (akin to language models that use the SentencePiece family of tokenization Transformers exhibited similar or slightly higher in-weights learning than the recurrent models (Fig  8), indicating that their superior in-context learning performance (as seen in Fig 7) cannot simply be explained by a bias towards in-context learning and against in-weights learning.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Fig 9 shows results of evaluating in-context learning on classes that were seen in training, rather than on holdout classes (the standard evaluation setting for few-shot learning, as described in Sec 2.3. The pattern of results is very similar between the two settings, with just slightly higher performance when evaluating on training classes.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
For completeness, we also report the in-context evaluation results by computing accuracy fully multiclass across all possible outputs of the model (Fig. 10). This is in contrast to the evaluations that were reported in the main text (as described Sec 2.3), across just the two labels that appeared in context; the two-choice evaluation provides a more sensitive measure of performance, ensuring that all experimental conditions have the same levels of chance, and also ensuring that the model cannot achieve above-chance performance simply by randomly selecting from the labels in context. Note that, across training and both types of evaluation (in-context and in-weights), the model is the sameit is trained to perform multi-class classification.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Multi-class evaluation shows the same patterns of results as the two-way evaluation from the main text. Note that the multi-class evaluation results showing the effects of the number of training classes (Fig 10b) and dynamic meanings (Fig 10d) need to be interpreted with caution, because the number of model outputs changes in the different conditions, so that task difficulty and chance levels differ for each.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
The multi-class evaluation uncovers one additional interesting result, for the models trained on Zipfian distributions (Fig 10e). As in the two-choice evaluation setting, a Zipfian distribution with an exponent of 1 is the only one able to elicit significantly above chance accuracy on both in-context evaluation and in-weights evaluation on common classes. However, Zipf 1 models have relatively lower few-shot performance when evaluated in the fully multi-class setting. Further investigation revealed that this was because those models have overall less tendency to output labels from context (Fig 11). Nonetheless, the Zipf 1 models do perform significantly above chance in both settings, and when forced to choose between the two labels that are shown in context, the model performs very well on these sequences. This indicates that, interestingly, a model can attain both in-weights and in-context learning abilities and process an input sequence in both ways, even if it is unsure which of those two processes it should output the result for.
Data Distributional Properties Drive Emergent In-Context Learning in Transformers
(a) Transformer.    Figure 8: In-weights learning in transformers vs. recurrent architectures. We compare architectures while holding fixed the number of layers, hidden layer size, and number of parameters. One run was performed for each set of hyperparameters in a hyperparameter sweep. Each color denotes one run, but not any particular hyperparameter values. 
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
NLP community has been surprised by emergence of in-context learning ability of a large-scale language model (LM) such as GPT-3 (Brown et al., 2020) despite no duplication between downstream task data and the pretraining corpus. Indeed, incontext learning uses a natural language description and a few examples to prime a language model. Then the language model can predict the answer of a new example without updating the language model's parameters. Since the release of GPT-3, * Equal contribution. various large-scale in-context language models have been proposed (Black et al., 2021;Kim et al., 2021;Zeng et al., 2021;Rae et al., 2021;Hoffmann et al., 2022;Chowdhery et al., 2022).
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
There still remain many questions on language models' in-context learning capability despite these successful reports. For example, the relationship between the choice of a pretraining corpus and downstream in-context learning task accuracy is unknown. Previous studies argue pretraining with the corpus similar to the downstream task improves the downstream performance, but these observations are often limited to the case where a pretrained language model is finetuned for the downstream task (Gururangan et al., 2020;Lee et al., 2020;Micheli et al., 2020).
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In addition, analysis on the relation between the validation perplexity of a language model and incontext learning performance is still less investigated. Previous research on in-context learning implicitly assumes that perplexity is predictive of incontext learning performance by showing scaling law property of their model (Kaplan et al., 2020;Brown et al., 2020;Kim et al., 2021). Rae et al. (2021) also use perplexity for the hyperparameter selection on corpus reweighting in the pretraining of their in-context learner. However, their explicit correlations are less discovered.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Motivated by this lack of in-depth analysis on the relationship between in-context learning and corpus properties, we vary the sources and sizes of pretraining corpora and analyze their impact on in-context learning, using HyperCLOVA, which is a Korean-centric large LM. (Kim et al., 2021). We mainly discover in-context few-shot learning as in the previous work (Kim et al., 2021) but also explore in-context zero-shot learning. We use Hyper-CLOVA corpus, which is a large-scale pretraining corpus mainly in Korean collected by Kim et al. (2021), as a base corpus from which we derive pretraining corpora for our experiments.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Our major findings include:
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
• Corpus Source: In-context learning performance depends heavily on corpus sources, and with some sources, in-context learning does not work effectively. For example, the model trained only on a subcorpus of blog (Blog) achieves competitive in-context fewshot learning performance, but training on a subcorpus of community website (Cafe) or online news articles (News) hardly yields incontext few-shot learning ability.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
• Corpus Combination: In-context learning ability can emerge by fusing two corpora, even when each on its own does not result in incontext learning. For example, while training only on KiN corpus, which consists of QnA websites, or Ency corpus, which consists of Encyclopedia websites, in-context few-shot learning ability was not observed, but training on both corpora makes the emergence of in-context few-shot learning.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
• Domain Relevance: Pretraining with a corpus related to a downstream task seems to help incontext zero-shot learning performance, but is not indicative of the competitive in-context few-shot learning performance. For example, training on only News corpus makes a relatively good in-context zero-shot learning ability on a news-related downstream task, e.g., news topic classification based on its title, KLUE-YNAT (Park et al., 2021), but does not yield in-context few-shot learning ability.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
• Perplexity: Although perplexity and incontext learning accuracies correlate well when training a single model, perplexity alone does not reflect the difference in in-context learning accuracies across different language models. This is prominent particularly when they were trained using different pretraining corpora.  (Zhao et al., 2021;Holtzman et al., 2021). On the other hand, another group of papers tries to explain the mechanism of in-context few-shot learning (Min et al., 2022;Xie et al., 2022).
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Previous studies argue a better downstream accuracy is observed with a pretraining corpus more similar to the downstream task corpus (Gururangan et al., 2020;Lee et al., 2020;Micheli et al., 2020). However, these observations are limited to the case where a pretrained language model is finetuned for the downstream task.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
There are a few studies on the effects of different corpus on the relationship between pretraining and in-context learning. A notable example is Codex, where GPT-3 is trained on Github corpus so that the model can generate code from comments (Chen et al., 2021a). However, the corpus used for Codex is limited to code comments and the corresponding code. We study the effect of pretraining corpus on in-context learning performance using various domains.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
There have been several studies on the quantity and quality of pretraining data. Raffel et al. (2020) conduct an ablation study on different pretraining corpus on T5, and their filtered C4 corpus makes T5 perform better in downstream tasks. As with GPT-3, researchers generally improve the quality of their language model through data filtering (Brown et al., 2020;Kim et al., 2021). Our research differs from the existing work in that we focus on in-depth analysis of how the amount of data and the corpus source affect in-context learning.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Multi-task learning approaches, which explicitly finetune on the in-context learning objective by using numerous NLP tasks, are proposed recently to tackle zero/few-shot transfer to the unseen task at test time (Wei et al., 2021;Sanh et al., 2021;Chen et al., 2021b;Min et al., 2021). Unlike the studies in a finetuning paradigm, many properties of the in-context learning related to pretraining corpus are still unknown. As the previous multi-task studies show that diverse tasks improve the ability of in-context learning, our study shows that diverse pretraining corpora strengthen the ability of in-context learning.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We use the variants of HyperCLOVA with various parameter sizes and pretraining corpus. We mainly experiment with models with 1.3B parameters, but we also include the result for 6.9B-sized models. All models have a maximum sequence length of 2,048.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We emphasize that all models use the same vocabulary across all our experiments. We use the morpheme-aware byte-level BPE tokenizer trained with HyperCLOVA corpus (Kim et al., 2021) for all models. We train multiple models with different portions of HyperCLOVA corpus to investigate the effects of the source and size of the corpus on in-context learning ability.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We analyze the effect of seven subcorpora in the HyperCLOVA corpus: Blog, Cafe, News, Comments, KiN, Modu, and Ency. Table 1 summarizes the characteristics of the subcorpora. Blog, Cafe, and News are taken from blog, community sites, and online news articles of NAVER 1 , a Korean web portal service, respectively. Comments is the comment threads related to the three subcorpora mentioned above. KiN comes 1 https://www.naver.com/ from NAVER's online community QnA service similar to Quora. Ency is a collection of encyclopedic texts including Korean Wikipedia. Modu consists of five public datasets constructed by National Institute of the Korean Language 2 , including 3.2B of news, 2.1B of written language, 0.4B of spoken language, 0.2B of web corpus, and 0.02B tokens of messenger. Others was excluded to investigate the explicit effects of domain corpus sources on in-context learning because Others is the corpus where various subcorpora are taken from multiple heterogeneous sources. Tables 12 and 13 in Appendix show the examples of seven pretraining corpus in Korean and English, respectively. ALL denotes the original HyperCLOVA corpus including Others.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
For corpora with less than 150B tokens, we assign 99% of each corpus to the pretraining corpus and randomly extract 10,000 examples from the remaining 1% to the validation corpus for measuring validation perplexity. For corpora with more than 150B tokens, we make the training corpora 150B tokens via random sampling and construct a validation set with 10,000 examples randomly sampled from the remaining. As a result, the maximum training set size of each corpus is 150B tokens.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
The validation set for each corpus consists of 10,000 examples and is used for the early stopping of models trained with each corpus. However, we combine all validation set to make the entire validation set contains 70,000 examples for seven domains, and the entire validation set is used for calculating perplexity, as described in Section 3.5.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We evaluate in-context learning performance of each corpus-specific model on four Korean downstream task datasets used in Kim et al. (2021): NSMC 3 , KorQuAD (Lim et al., 2019), AI Hub translation 4 , and YNAT (Park et al., 2021). NSMC is a binary sentiment classification dataset on movie review. KorQuAD is a machine reading comprehension dataset similar to SQuAD 1.0 (Rajpurkar et al., 2016). AI Hub translation dataset consists of Korean-English parallel sentences from news, government websites, legal documents, etc. YNAT is a topic classification problem with seven classes.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We think that three datasets for downstream tasks are closely related to the HyperCLOVA cor- Figure 1: Vocabulary overlap ratio between pretraining corpus and downstream task. Top 1,000 nouns are used to calculate the ratio. Nouns are extracted using our inhouse part-of-speech tagger.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
pus. Passages which construct KorQuAD are taken from Korean Wikipedia, which is also a part of the Ency. YNAT is a topic classification task of news headlines, so the downstream task is deeply related to the News corpus. A significant portion of parallel sentences for AI Hub translation dataset also comes from news articles. KiN corpus is also related to the translation task. About 2.5% of QnA data in KiN includes Korean questions on the English language, as a foreign language. These question-style passages often include Korean-English sentence pairs in the passage. Vocabulary overlap between downstream tasks and HyperCLOVA corpus is depicted in Figure 1.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We try our best to make the same hyperparameter of Kim et al. (2021), including global batch size, training step, maximum sequence length, learning rate, and so on. In our experiments, the models are trained for 72K steps with a global batch size of 1,024. We note that under this setting, the number of tokens that were actually used in pretraining is 150B. Therefore, we set the maximum size of training corpus to 150B as in Section 3.2.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In most experiments, validation perplexity decreases monotonically as training goes on. Thus, we use the checkpoint at 72K step. The only exception is the Ency model. The Ency model has a minimum validation loss at 12K steps, which is likely to be caused by overfitting to pretraining data due to a small size of the data. Therefore, we use early-stopping checkpoints at the 12K steps for the report.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
For optimization, AdamW (Loshchilov and Hutter, 2019) with the learning rate of 2.0e-4 and the cosine learning rate scheduling are used. We use the mixed precision training. Models are trained on the Nvidia Superpod which consists of 1,024 A100 GPUs spread across 128 nodes. Using Superpod, it spends around 18 hours to train 1.3B model with 72K steps.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
For classification tasks such as NSMC and YNAT, we use a rank classification approach (Wei et al., 2021), where we compare pre-defined outputs ("positive" and "negative") and take the one with higher probability. KorQuAD and AI Hub are free-form completion tasks, where we directly generate output tokens using the greedy decoding.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In the few-shot experiments, the number of shots is set to 70, 4, 4, and 70 for NSMC, KorQuAD, AI Hub, and YNAT, respectively. Downstream tasks are performed 12, 1, 3, and 6 times with different random seeds for NSMC, KorQuAD, AI Hub, and YNAT, respectively. We report the average performance. Random seed influences the sampling of shots from training data and their order. The reason KorQuAD has only one random seed is described in Appendix D. Appendix D also includes the examples of the few-shot prompts used in our experiments. These all experimental settings in the few-shot experiments, from the number of shots to the number of random trials, basically come from the experimental setting of Kim et al. (2021). However, we change the number of trials of YNAT from 3 to 6, because we found that the standard derivation of YNAT is relatively high.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We report validation perplexity in various tables and figures to verify our argument. We use the term "PPL" to denote validation perplexities on the validation set. The validation set consists of 70,000 examples from seven corpus sources, as described in Section 3.2. We emphasize that, for calculating PPL, all experiments use the same vocabulary and validation set.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In Tables 2 and 4, we use Italic font for the results from a multi-domain model, which is pretrained with two or more mixed corpora. Since a multi-domain model trains more domains than a single-domain model, the PPLs of multi-domain models are generally lower than those of singledomain models. To keep readers from directly comparing PPLs between a single-domain and a multi-domain model, we use italic font for the results of a multi-domain model.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We perform intensive experiments to answer these four main questions:
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
1. How large do the source and the size of pretraining corpora have the effects on emerging in-context learning ability? (Section 4.2 and 4.3)
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
2. What is the effect of combining various corpora? (Section 4.4)
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
3. How large does domain relevance of corpus influence on model performances of the downstream task? (Section 4.5)
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
4. How strong is the correlation between validation perplexity and in-context learning of language models? (Section 4.6)
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Tables 2 and 4 show the in-context few-shot results on various pretraining corpus sources and different corpus combination, respectively. Tables 3 and 5 depict the in-context zero-shot results of some models in Tables 2 and 4, respectively. All results in Tables 2, 3, 4, and 5 come from models with 1.3B parameters. Tables 8 and 9 in Appendix A present the standard derivation values on the results of Tables 2 and 4. In Tables 2, 3, 4, and 5, Purple-underline denotes the score is below the mean performance value of ALL and Majority baseline in Table 2, and Tealbold denotes the score is above. We use this mean value of Majority and ALL in Table 2 as the performance basis to prevent the in-context learning performance of each model from being distorted by the high basis performance of two classification tasks such as NSMC and YNAT.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Tables 2 and 6 include in-context few-shot results on various pretraining corpus sizes. In Table  6, for example, 56B and 6B correspond to the 1/10 and 1/100 of the original HyperCLOVA corpus with 560B tokens, respectively. The 56B tokens and 6B tokens models are trained with around 3 and 25 epochs, respectively, so that both models can be trained with 72K training steps. On the other hand, Table 2 compares 27B tokens models trained with different corpus sources to show the results in controlled corpus size.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
It is noticeable that in-context learning ability emerges differently depending on pretraining corpus sources, as shown in Tables 2, 3, 4, and 5. For example, Blog model makes competitive incontext few-shot learning performance to ALL model, while each of Cafe and News models hardly shows in-context few-shot learning ability from Table 2. It is also noticeable that Modu model performs better than Cafe and News model although the size of Modu corpus is less than 1/10 of Cafe or News corpus, showing the corpus size is not the only factor to predict in-context learning performance. Likewise, it is also interesting that Cafe+News model also shows poor performance despite the same size to Blog and ALL, as shown in Table 4.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
These differences in in-context learning are dramatic compared to the finetuning results we expect in general. For a comparative experiment between in-context learning and finetuning in our setting, we also finetuned the experimented models with LoRA (Hu et al., 2021). As Table 11 in Appendix C shows, the performance differences in finetuning are much smaller than in the case of in-context learning. Table 6 shows that reducing the corpus size from 150B to 56B does not decrease the performance severely despite training with 1/10 of corpus. However, the performance degradation of 6B tokens model is remarkable comparing to ALL model. Nevertheless, it is noticeable that 6B tokens model still performs much better than Cafe+News model, which trains 150B tokens of Cafe and News corpus.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We can also see the similar results for three Blog models of different sizes in Table 2. Blog and Blog 54B achieve similar performance. However, like in ALL 6B, Blog 27B performs quite worse than Blog 54B. Figure 3 shows the comparison between 1.3Bsized model and 6.9B-sized model. In the 6.9Bsized models, the in-context few-shot performance with 56B tokens does not decrease significantly compared to 150B tokens, as in the 1.3B-sized models.   
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
One of our main goals is to investigate the effects of combining multiple corpora from various sources on in-context learning performance. Table  4 shows that in-context few-shot learning ability can be emerged by combining two corpora, even if each of both corpora cannot provide in-context few-shot learning ability. For example, KiN+Ency model succeeds to make in-context learning ability in most tasks, while each of KiN and Ency fails in most tasks. Likewise, Cafe+KiN model succeeds to make in-context few-shot learning ability, while each of Cafe and KiN fails in most tasks. Incontext zero-shot abilities of these models follow similar patterns as shown in Table 5.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
This phenomenon is related to the argument that in-context learning emerges by multi-task learning.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
According to the argument, as the language modeling objective function requires a language model to learn variety of next word prediction tasks, the generalization pushes in-context learning ability on unseen tasks. In the example of KiN+Ency model, KiN+Ency may learn in-context learning ability of MRC task, by learning next word prediction tasks of both Ency (Wikipedia) and KiN (QnA).
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Unlike these positive cases, we observe that combining corpora does not assure the emergence of competitive in-context learning. For example, from the case of Cafe+News in Table 4, even if the mixed corpus model shows slightly better performance on KorQuAD than each of two corpora, its in-context few-shot performances on NSMC, Ko-rQuAD, and YNAT are still below the basis. Furthermore, the performances on NSMC and YNAT even decrease.    
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Speaking of the few-shot results, Table 2 shows that the close relationship between a pretraining corpus and a downstream task does not always guarantee in-context few-shot learning ability on the downstream task. KiN and Ency do not perform well on KorQuAD task, although KorQuAD is an MRC task from Korean Wikipedia, Ency includes Korean Wikipedia, and KiN consists of question answering pair, respectively. Likewise, News does not perform well on YNAT task, although YNAT consists of news headline queries. Table 4 further shows that News+KiN+Ency model shows more degenerated F1 score on YNAT than KiN+Ency, even though a large amount of News corpus is added to News+KiN+Ency model.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
For further investigation, we analyze vocabulary statistics of each corpus. Figure 1 shows the vocabulary overlapping ratio between pretraining corpora and downstream tasks. The result shows that high vocabulary overlap between a pretraining corpus and a downstream task does not indicate high downstream task performance. Although the Modu corpus has a large vocabulary overlapping ratio to AI Hub, in-context learning performances of the Modu model on the translation tasks are much lower than Blog and KiN.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
The counter example of above supports is AI Hub task performance of KiN model. KiN model learned the pattern of Korean-English sentence pairs, since the corpus includes a lot of Korean questions on English language. While KiN model does not work well in other downstream tasks, the performance on AI Hub translation is competitive and makes the best performance in Ko→En among seven pretraining corpora.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In the zero-shot setting, on the other hand, do- Figure 2: In-context few-shot learning performance of various corpus models and their PPL. A score of the model is divided by that of ALL to calculate the normalized performance. Blue and red lines denote the performance of ALL model and majority baseline, and the purple line represents the average of both defined in the caption of Table 4. main relevance seems to affect more positively. For example, training the News corpus helps in-context zero-shot learning in KLUE-YNAT consistently. As shown in Tables 3 and 5, the models whose training corpus includes the News corpus (i.e., News, Cafe+News, and News+KiN+Ency) even perform better than the model trained whole Hyper-CLOVA corpus.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
In the case of KiN and AI Hub, zero-shot performance increase for AI Hub tasks of the KiN model is less significant than few-shot. However, adding KiN corpus into the pretraining corpus in the experiments of Table 5 (i.e., KiN+Ency, Cafe+KiN, and News+KiN+Ency) makes a consistent performance increase, and the model outperform ALL. Figure 2 presents the scatter plots of PPL (x-axis) and in-context few-shot learning performance (y- Figure 3: Comparison on two model sizes and two corpus sizes of the original HyperCLOVA corpus such as 1.3B-sized model and 6.9B-sized model, and 56B and 150B tokens. Few-shot results are reported. Green rectangle denotes 1.3B-sized model and blue rectangle denotes 6.9B-sized model. axis) on five downstream tasks for single corpus models and the ALL model. In Figure 2, we normalized in-context few-shot learning performance by dividing ALL model performance for calibrating various task metrics. Because we observe less explicit tendency of correlation between validation perplexity and in-context performance, we argue that it is difficult to hypothesize better perplexity assures emerging of in-context few-shot learning ability.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
According to Table 2, Blog model shows both the lowest PPL and the best in-context learning performance, and Ency model shows both the highest PPL and the worst in-context learning performance. On the contrary, while Cafe model and KiN model shows the second and third lowest PPL, in-context few-shot learning ability was not observed. These results show that the perplexity does not serve as a strong predictor of in-context few-shot learning performance in comparing models trained using different corpora. Table 2 also shows that the corpus size affects in-context fewshot learning performance more than PPL. Blog 27B performs notably worse than Blog, but PPL relatively does not decrease as much.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Speaking of zero-shot results, it seems Table  3 shows that in-context zero-shot learning performances relatively more correlate with perplexity than the few-shot cases. Nevertheless, Modu still has both relatively high perplexity and relatively high in-context zero-shot learning performances.  Table 7: Validation perplexity scores per each subcorpus. All denotes the validation perplexity on our main validation set from seven corpus sources. Italic font denotes the validation PPL of their corpus domain, and Bold denotes second best after own corpus. Overall, Blog has the best overall validation perplexity in most tasks. Figure 4: Relation between validation PPL and incontext few-shot learning performance for five downstream tasks as pretraining steps proceed. The results come from the 1.3B-sized ALL model. each subcorpus. Each row corresponds to the model and each column corresponds to the validation set's subcorpus. Each validation set except All in Table  7 consists of 10,000 instances, and is the part of our main validation sets, consists of 70,000 instances. On the other hand, Figure 4 shows that PPL and in-context learning performance correlate well in the perspective of training a single model. We can find that the correlation trends between the cases in the training and the cases between the corpus domain are different.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Our knowledge can be used to increase the performance of in-context learning when the corpus is small or/and there exists demand for collecting more corpus. In the case of XGLM (Lin et al., 2021), which is a concurrent work on multilingual GPT-3, achieved better in-context learning performance for many languages. However, it does not reach the performance of a single language model. We hope our observation can give insight into what types of pretraining to be collected more, both for multilingual model and low-resource language model.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Another notable example comes from Gopher (Rae et al., 2021), which is a concurrent work on state-of-the-art in-context learner. Rae et al. (2021) determine the ratio between subcorpora based on the perplexity of the validation corpus. They implicitly claim that this ratio results in better downstream task performance, but do not address explicit evidence for this. On the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting. We hope our findings contribute to making better in-context learners along with other research.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
This paper investigates the effects of the source and the size of the training corpus on in-context learning ability, using the HyperCLOVA corpus. Our discoveries include that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model. One direction for future work is to investigate linguistic properties of corpus sources which make a competitive in-context learning model. For example, quantifying the difference between two corpora can shed light on how to select suitable corpora for NLP practitioners who build large-scale language models. In addition, intensive studies on different corpus sources other than the HyperCLOVA corpus can help understand the properties of in-context learning.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We present multiple pieces of evidence that models using only a part of the pretraining corpus are comparable with those trained with the entire corpus in terms of in-context performances. Although we leave the validation on larger-scale models, such as tens of billion parameters, to future work, our analysis presents a hint to effectively training LMs with smaller corpora. This approach can contribute to alleviating severe energy consumption issues caused by large-scale LMs.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Meanwhile, our study relates to the misuse and fairness of large-scale LMs. For example, reweighting domain-specific corpus might cause LMs to be biased inherent in the domain corpus. Therefore, alleviating domain corpus bias would be a valuable future direction.  Tables 8 and 9 show standard derivation value on  Tables 2 and 4. Table 10 shows score difference with ALL in addition to in-context learning scores on Table 2. Figure 5, supporting Table 7, shows the validation perplexity of different model from different corpus.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Tables 12 and 13 show example instances of seven pretraining corpus in Korean and English, respectively.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
For preprocessing steps of our pretraining corpus, we use HyperCLOVA corpus which is also used in (Kim et al., 2021) as described in Section 3.1. Therefore, we share the preprocessing steps of Kim et al. (2021). Appendix A in (Kim et al., 2021) describes their preprocessing methods on data descriptoin, data clearning, data anonymization, and data postprocessing.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
We additionally introduce the deduplication preprocess of HyperCLOVA corpus, which is used in (Kim et al., 2021). The deduplication preprocess was applied to construct HyperCLOVA corpus to prevent explicit duplication within and between subcorpora (Kim et al., 2021). According to the response of Kim et al. (2021), they use an in-house search engine and an in-house engineering trick to detect document pairs that are very similar to each other. There are two pipelined steps: (1) removing duplicates within subparts of the corpus, and then (2) removing duplicates between subparts of the corpus. Therefore, documents with high overlap do not exist throughout the documents. Here, the number of subparts is 29. These 29 subparts are categorized into the eight domains we deal with in the paper (i.e., Blog, News, Cafe, Comments, KiN, Modu, Ency, and Others). Overall, there is no explicit overlap between each corpus, since very similar documents have already been removed from the corpus. The overlap between eight Hy-perCLOVA subcorpora is quite small. There were many overlaps within the subpart of the corpus. However, the overlap between subparts of the corpus was only 0.024% of the total, according to the counts in the second pipelined step of deduplication between subparts. 
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Tables 14, 16, 18, and 19 show the example fewshot prompt of NSMC, KorQuAD, AI Hub, and YNAT, respectively. Tables 15, 17, and 20 show the translated version for NSMC, KorQuAD, and YNAT, respectively.
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
On the other hand, the number of random seed is one for KorQuAD. We explain why evaluation on KorQuAD with many random seeds is difficult, from the perspective of prompt design. The way we make randomness on trials is to change fewshot examples in the prompt. However, in the case of Kim et al. (2021) and in our case, there are no alternative examples to put into the prompt. The prompt examples of KorQuAD are one document and a few question-answer pairs, and not a few document-question-answer triples. In other words, in the prompt of KorQuAD, the number of the document is one. Thus, the document is used for both few-shot question-answer pairs and a query question for the inference. In KorQuAD, there are five corresponding question-answer pairs in each document. In the experimental setting of ours and Kim et al. (2021), four question-answers are put into the prompt and one question is used for the test. Therefore, there are no other question-answer pairs to replace the four pairs.               
Pre-Training to Learn in Context
Pre-trained language models (PLMs; Han et al., 2021;Qiu et al., 2020) have shown strong abilities of learning and performing unseen tasks conditioning on several task examples or instructions in its context, which is called in-context learning (ICL;Brown et al., 2020). Compared to conventional fine-tuning methods, ICL adapts PLMs to downstream tasks only through inference, without parameter updates, which is computationally cheaper in practice and is closer to general AI. † Corresponding author. * Contribution during internship at Microsoft Research.
Pre-Training to Learn in Context
However, PLMs trained on massive corpora to predict the next word given previous words are not explicitly taught to learn in the context. This makes ICL a surprising emergent ability but also indicates that the ICL ability of PLMs is not fully exploited. Garg et al. (2022) has shown that by directly training to do ICL in a meta-learning paradigm, models show strong performance on learning simple function classes in the context. In practical NLP scenarios, previous works (Min et al., 2022b;Chen et al., 2022b) also enhance the ICL performance by metafine-tuning PLMs on a large collection of downstream tasks and evaluating them on unseen tasks. However, the low diversity of human-annotated downstream tasks restricts the performance of the meta-tuned model. Direct training on downstream tasks also brings undesired bias on specific input formats, label spaces, or domains, which hurts the generalization of PLMs.
Pre-Training to Learn in Context
To enhance the ICL ability while maintaining generalization, we propose PICL (Pre-training for In-Context Learning), a framework that exploits the PLM's ICL ability by pre-training models on data automatically constructed from the general plain-text corpus. Our framework is based on a simple observation that many paragraphs in the text documents contain "intrinsic tasks". As shown in the left part of Figure 1, each paragraph in the document contains an intrinsic task. When doing language modeling on each paragraph, models implicitly perform the corresponding intrinsic tasks simultaneously. This shares a similar idea with the prompt-learning paradigm (Liu et al., 2021), where downstream data examples from NLP tasks are transformed into text sequences, and the model learns to perform the original tasks when trained on the text sequences with language modeling. Different from the downstream data, text paragraphs contain more diverse intrinsic tasks and have little bias on input formats, label spaces, or domains because they are free-form texts from the large- The overall framework of PICL. For each paragraph z 0 in the corpus C, we retrieve k paragraphs that share the same intrinsic task (Sentiment Analysis) as demonstrations and then concatenate them with z 0 to construct a pre-training instance. We compute the language modeling loss on the whole instance to train the model. scale general corpus. By gathering and concatenating paragraphs with the same intrinsic tasks (right part of Figure 1), we can construct a meta-training dataset to pre-train the model to perform the intrinsic task conditioning on paragraphs in the context, and thereby improve the ICL ability.
Pre-Training to Learn in Context
We adopt a retrieval-based approach to gather paragraphs sharing the same intrinsic tasks from a general corpus. We first train an encoder to represent each paragraph in a vector space where paragraphs with the same intrinsic task have close embeddings. The encoder is trained with contrastive learning (Khosla et al., 2020) on a collection of downstream datasets by taking examples from the same tasks as positive pairs and those from different tasks as negative pairs. Then, treating any paragraph in the corpus as a query, we retrieve the paragraphs with close representations to the query, namely, sharing the same intrinsic task with the query. Finally, we concatenate the query and the retrieved paragraphs to get a pre-training instance. Note that although we use downstream datasets, the model is trained on instances constructed from the general corpus, which ensures its generalization.
Pre-Training to Learn in Context
We evaluate the ICL performance of the model pre-trained with PICL on seven widelyused text classification datasets and SUPER-NATURALINSTRUCTIONS (Wang et al., 2022), a benchmark whose test split contains more than 100 tasks formulated into text generation. Empirical results show the effectiveness of PICL, enabling the model to reach or even outperform larger models with nearly 4x parameters. Besides, we find that the PICL-trained model is more generalizable on various tasks than previous meta-fine-tuning methods. We also conduct extensive experiments to analyze several key factors of PICL.
Pre-Training to Learn in Context
We first present an overview of PICL and then describe the details in the following sections. As shown in the right part of Figure 1, we construct the pre-training instances from a corpus C consisting of paragraphs split from full documents by "\n". For each paragraph z 0 in C, we first use a retriever R to find k paragraphs {z 1 , z 2 , · · · , z k } sharing the same intrinsic task (Sentiment Analysis) with z 0 . Then the retrieved paragraphs are treated as demonstrations and concatenated with z 0 to form a pre-training instance: z k ⊕ z k−1 ⊕ · · · ⊕ z 1 ⊕ z 0 . Finally, we adopt a language modeling objective to pre-train the model on the constructed instances.
Pre-Training to Learn in Context
In this way, the pre-training stage can be regarded as a meta-training process, where the model learns to solve the intrinsic task in z 0 conditioning on its context z k ⊕ z k−1 ⊕ · · · ⊕ z 1 . Since C is a large-scale general corpus, it contains a variety of intrinsic tasks and little domain bias, which ensures the generalization of the pre-trained model.
Pre-Training to Learn in Context
The main component of the retriever R is a tasksemantics encoder E that represents a text paragraph as a d-dimensional vector in a space V , where paragraphs with the same intrinsic tasks have similar representations. We define the similarity between two paragraphs z 0 and z using the dot product of their representations: E(z 0 ) · E(z). Encoder We use RoBERTa BASE (Liu et al., 2019) as the base model of E. The output vector is computed by averaging the last-layer representation of each token in the input paragraph.
Pre-Training to Learn in Context
Retrieval We approximate that paragraphs whose representations are close to each other in V share the same intrinsic task. Therefore, for every paragraph z 0 in C, R searches for k paragraphs with embeddings closest to E(z 0 ):
Pre-Training to Learn in Context
We employ the FAISS library (Johnson et al., 2019) for efficient searching.
Pre-Training to Learn in Context
Contrastive Training We adopt contrastive learning (Khosla et al., 2020;Karpukhin et al., 2020) to train the task-semantics encoder E. As shown in Figure 2, we take two paragraphs with the same intrinsic task as positive pairs and those from different tasks as negative pairs. However, the annotation of a paragraph's intrinsic task is usually unavailable. To this end, we use a collection of downstream NLP datasets from various tasks whose examples are converted into text sequences with human-written prompts to train E. In this way, treating each text sequence as a paragraph, we can regard the corresponding downstream task as the intrinsic task annotation. We assume that the instances from all downstream tasks form a dataset D. For each z 0 ∈ D, we have a positive instance z + sharing the same task with z 0 and a set N (z 0 ) consisting of negative instances with different tasks than z 0 , the loss function takes the form:
Pre-Training to Learn in Context
(2)
Pre-Training to Learn in Context
Positive and Negative Instances For each z 0 ∈ D, we randomly sample a positive instance z + belonging to the same task with z 0 from D\{z 0 }. As shown in Figure 2, N (z 0 ) contains two kinds of negative instances: (1) Easy Negatives z − easy sampled from D and belonging to different tasks than z 0 . (2) Hard Negatives z + hard sharing the same prompt with z 0 but containing mismatched tasks. For instance, in Figure 2, we apply the prompt from the sentiment task to the summarization task to create the hard negative instance z − hard . This prevents the model from hacking the contrastive objective using prompts like "Guess the sentiment" and learning a trivial pattern matching but forces the model to extract task semantics from the whole paragraph.
Pre-Training to Learn in Context
For each paragraph z 0 ∈ C, we concatenate the retrieved paragraphs {z 1 , z 2 , · · · , z k } = R(z 0 ) with z 0 to get a pre-training instance z k ⊕ z k−1 ⊕ · · · ⊕ z 1 ⊕ z 0 . To improve the quality of the constructed data, we derive an approach to filter out instances that are less informative to ICL. We consider the following score to measure the informativeness of an instance based on the perplexity difference of the paragraphs in the instance before and after they are concatenated as a sequence:
Pre-Training to Learn in Context
where | · | is the length of a sequence and P (·) is the language modeling probability based on any uni-direct PLMs. Given a manually set threshold δ, we retain the instances that satisfy s > δ. This criterion leverages the original ICL ability of the PLM. If concatenating the paragraphs results in lower perplexity, they are more correlated and may be more informative for ICL. We finally construct a pre-training corpus containing N instances
Pre-Training to Learn in Context
We pre-train the model with auto-regressive language modeling on C pre-train . Unlike previous works (Min et al., 2022b;Chen et al., 2022b), which only compute the language modeling loss on the label tokens, we compute the loss on the whole sequence. There are two reasons for this choice. First, the intrinsic tasks are already in the natural language format, and it is unnecessary to split the input and the label. Second, we argue that computing loss on the whole sequence ensures a large token number in a forward batch, which is critical to maintaining the basic in-weights ability (Chan et al., 2022). Therefore, the loss function is:
Pre-Training to Learn in Context
where θ is the parameters of the model. In addition, we find that adding a language modeling loss L LM (θ) on the original full documents before being split into paragraphs benefits the performance. Therefore, the final optimization objective is:
Pre-Training to Learn in Context
where we set α = 0.5 in our main experiments.
Pre-Training to Learn in Context
We merge OPENWEBTEXT (Gokaslan et al., 2019), WIKICORPUS (Foundation, 2022), and BOOKCOR-PUS (Zhu et al., 2015) to construct the pre-training data, where full documents are split into paragraphs by "\n". The corpus C consists of 80M paragraphs, totaling about 30GB. For each paragraph, we search for k = 20 demonstrations and concatenate them until 1024 tokens, the maximum input length constraint of the language model we used. This ensures that the model sees various demonstration numbers during pre-training. We use GPT2-Large (Radford et al., 2019) to compute P (·) in Equation 3 and set δ = 0.0 for filtering. More details of data processing and statistics are shown in Appendix A.
Pre-Training to Learn in Context
We consider four baselines in our experiments:
Pre-Training to Learn in Context
• VanillaICL directly prompts a PLM with the concatenation of training examples to do ICL. • ExtraLM further pre-trains the PLM on the original full documents before being split into paragraphs with the language modeling objective. • Self-Sup (Chen et al., 2022a) designs four selfsupervised pre-training objectives, including Next Sentence Generation, Masked Word Prediction, Last Phrase Prediction, and Classification, to enhance the ICL performance. We conduct the self-supervised pre-training on our merged corpus for a fair comparison. • MetaICL (Min et al., 2022b) meta-trains the model on a large collection of downstream human-annotated datasets for learning to learn in context. The meta-training instances are constructed by concatenating several training examples in each dataset to a single text sequence. We replicate the method on the training set of our task-semantics encoder for a fair comparison.
Pre-Training to Learn in Context
We evaluate the model trained with PICL on two kinds of downstream tasks. , and AG-News (Zhang et al., 2015) to evaluate the few-shot ICL performance of the trained models (see Appendix B.1 for more details). Note that these tasks are not included in the training set of the tasksemantics encoder. We randomly sample 4 or 8 demonstrations from the official training sets of each dataset. Effects of other demonstration numbers can be found in Section 4.3. We compute the average accuracy scores on at most 1000 samples from the validation split of each dataset across five random seeds for selecting demonstrations.
Pre-Training to Learn in Context
Instruction Following To test the generalization of PICL, we also evaluate the trained model on a larger range of tasks with more free-form inputs, including both human instructions and fewshot examples. We use the test split of SUPER-NATURALINSTRUCTIONS (Wang et al., 2022) as the benchmark and exclude the tasks that appear in the training set of the task-semantics encoder, resulting in 105 evaluation tasks (see Appendix B.2 for a full list of tasks). Each task is specified with a human-written instruction and two or three demonstrations. We follow Wang et al. (2022) to formulate all tasks to the text generation format and score the outputs with ROUGE-L (Lin, 2004).
Pre-Training to Learn in Context
Retriever  Pre-Training We set the maximum learning rate to 1×10 −6 and use the "inverse square root" scheduler (Vaswani et al., 2017) with 1000 steps warmup. The model sees 131K tokens in a step and is pretrained for 100K steps. It takes less than a day to finish pre-training on 64 V100 32G GPUs. Table 1 shows the results of few-shot text classification, from which we have 3 observations. First, among the baselines with 770M parameters, simply further training the model on our corpus with language modeling improves the performance (ExtraLM). This is likely due to the higher domain diversity of our corpus. MetaICL is helpful on most datasets, which verifies the effectiveness of meta-training for ICL. Self-Sup fails to bring benefits on most datasets against VanillaICL, probably because the constrained label space of the Classification training task (only contains "True" and "False") brings bias to the model's output. This emphasizes the importance of using training objectives with little bias.
Pre-Training to Learn in Context
Second, we observe that the PICL-trained model outperforms the baselines with the same model sizes by a large margin on most datasets across different shots, verifying the effectiveness of PICL. An exception is RTE, where MetaICL performs the best. We speculate the reason is that some training tasks of MetaICL share the same label space with RTE ("Yes"/"No"), such as paraphrase identification. Min et al. (2022c) has shown that the label space plays a vital role in ICL, which explains the good performance of MetaICL on RTE.
Pre-Training to Learn in Context
Thrid, comparing models across different sizes, we find that increasing the model parameters boosts the performance, but PICL enables the 770M model to beat a 2.7B counterpart. This indicates that the ICL ability can be enhanced not only through scaling up the parameters. Improving the structure of the pre-training data is also beneficial. In Appendix E.1, we can see that PICL is also effective when applied to a 1.5B model.
Pre-Training to Learn in Context
The results on SUPER-NATURALINSTRUCTIONS are shown in Table 2. We can see that PICL  (Wang et al., 2022). Each bar represents an evaluation task. The y-axis means the ROUGE-L score difference between the two methods.
Pre-Training to Learn in Context
Param. ROUGE-L  achieves higher overall instruction following performance than the baselines, outperforming a larger model with about 4x parameters.
Pre-Training to Learn in Context
In Figure 3, we compare the per-task performance of PICL and MetaICL because they share the most similar setting where human-annotated downstream datasets are used. We observe that PICL outperforms MetaICL on about 3/4 of evaluation tasks, indicating that compared to fine-tuning directly on downstream tasks, pre-training on intrinsic tasks constructed from the general plain-text corpus brings better ICL ability and ensures higher generalization performance across a broad range of tasks (see Appendix E.2 for more details).
Pre-Training to Learn in Context
Most tasks where MetaICL beats PICL belong to text classification whose output spaces are "Yes/No" or "True/False". This matches the second observation in Section 4.1, where MetaICL predicts "Yes/No" well because of training on tasks that share the same label spaces. On the other hand, PICL performs much better on text generation, or tasks whose output spaces share the same semantics with "Yes/No" but use label words not in the training tasks of MetaICL (e.g., "Correct/Wrong"). This indicates that direct training on downstream datasets causes overfitting to specific labels. There are also tasks where PICL performs similarly to MetaICL, such as reasoning and word analogy. We notice that the improvements of PICL and MetaICL on these tasks are also marginal against VanillaICL probably because these tasks rely more on the "inweights learning" ability (Chan et al., 2022), rather than in-context learning.
Pre-Training to Learn in Context
Effect of Retriever We compare different approaches to retrieve paragraphs and test the final model performance. We try randomly selecting paragraphs (Random), retrieving using the nonparametric approach (BM25), encoding each paragraph with the original pre-trained encoder as it is (RoBERTa), or using the encoder for sentence similarity (Reimers and Gurevych, 2019) (SRoBERTa). We also study different numbers of hard negatives (0, 1, 4) and downstream tasks (7, 24, 37) to train the task-semantics encoder in PICL. From the results in Table 3, we can see that all retrieval methods except Random bring improvements against VanillaICL on both text classification and instruction following settings, indicating that improving the coherence of the paragraphs in the pre-training data benefits ICL. Using the task-semantics encoder in PICL achieves the best performance, showing the importance of retrieving paragraphs based on task semantics rather than word overlap or sentence meanings. Comparing different settings to train the task-semantics encoder, we observe that increasing the number of hard negatives and training tasks improves the final performance. This is in line with previous works (Karpukhin et al., 2020;Chen et al., 2020;He et al., 2020) that more challenging hard negatives benefit contrastive learning.
Pre-Training to Learn in Context
Training with PICL brings two benefits: (1) PLMs learn a format where demonstrations from the same task are concatenated as the prefix, which is beneficial when the model is evaluated under the same number of demonstrations. (2)   and n Tasks means the number of hard negatives and downstream tasks to train the task-semantics encoder in PICL. "CLS Accuracy" means the average accuracy scores on text classification tasks. "SUP-NI ROUGE-L" means the average ROUGE-L scores across the tasks in SUPER-NATURALINSTRUCTIONS. ability to infer and perform tasks from the context, even when the demonstration numbers in evaluation and pre-training do not match. To differentiate these effects, we conduct pre-training on instances containing only 4, 8, or 16 demonstrations and test the trained models under different text classification shots. Results in Figure 4 show that when pre-trained with different demonstration numbers, the models generalize well to unseen demonstration numbers in evaluation, achieving similar performance with the default setup where the model sees various demonstration numbers in pre-training (PICL-default). This indicates that the models learn more than the input formats in PICL.
Pre-Training to Learn in Context
We try different threshold values δ for filtering and report the scores on text classification tasks in Figure 5(a), while controlling the sizes of the constructed pre-training data the same. We find that δ = 0 yields the best performance, which means we retain an instance if and only if the perplexity of individual paragraphs is higher than that of the concatenated sequence (Equation 3). For lower δ, the pre-training data contain too many uninformative instances for ICL. For larger δ, we speculate that the filtering process relies on the original GPT2-Large too much. Since we also pretrain based on GPT2-Large, the filtering process reduces the signals in the constructed data beyond the base model's ability.
Pre-Training to Learn in Context
Effect of Full Documents In Figure 5(b), we report the model performance on text classification tasks when using different choices of α, which controls the proportion of the full-document data. We find that balancing the constructed and fulldocument data performs the best (α = 0.5). When α is too large, the model is trained mostly on our constructed data and overfits its bias inevitably introduced by the task-semantics encoder in the data construction process. When α is too small, our method degenerates into ExtraLM.
Pre-Training to Learn in Context
We study the size effect of the corpus used to construct the pre-training data in PICL and report the performance on text classification tasks in Figure 6(a). We conduct the data construction on 0.01%, 0.1%, 1%, and 10% of the original 80M paragraphs (100%) and pre-train models for at most 100K steps until the validation losses begin to increase. From the results, we conclude that when the corpus is small, pre-training with the constructed data hurts the performance because the search library is too small to find paragraphs sharing the same intrinsic tasks. Training on small data for multiple epochs also causes overfitting. When the corpus contains more than 80K paragraphs (0.1%), adding more data constantly improves the performance, which is consistent with the scaling law (Kaplan et al., 2020).
Pre-Training to Learn in Context
We compare the usefulness of different pre-training data to enhance the ICL ability. In addition to the final model performance, we borrow the thoughts for designing the filtering criterion in Section 2.2 to measure the usefulness of a pre-training instance by computing the perplexity using a reference large PLM: GPT-J (Wang and Komatsuzaki, 2021) with 6B parameters. Lower perplexity means the correlation within the instance is higher and is intuitively more helpful for enhancing the ICL ability. In Figure 6(b), we show the perplexity and the final model performance of 3 pretraining data: original full documents before being split into paragraphs (Full Doc), concatenation of randomly selected paragraphs (Rand ⊕ z 0 ), and the concatenated same-intrinsic-task paragraphs gathered using the retrieval method in PICL before filtering (R(z 0 ) ⊕ z 0 ). We can see that the data constructed by retrieval has much lower perplexity and correspondingly yields higher accuracy scores, which verifies its usefulness. In Appendix F, we present several examples of the retrieved paragraphs and the corresponding intrinsic tasks.
Pre-Training to Learn in Context
This paper presents PICL, a framework that exploits the in-context learning ability of PLMs by pre-training models on concatenations of text paragraphs sharing the same "intrinsic tasks" gathered from the large-scale general corpus. In PICL, models learn to perform various intrinsic tasks conditioning on their context while preserving their generalization due to the little bias of the pre-training data. Extensive experiments show that PICL improves the ICL performance on various datasets against several baselines, enabling a 770 M model to outperform a larger model with about 4x parameters while maintaining good generalization across a wide range of tasks. For future work, we would like to consider adding human instructions to our pre-training framework to enhance more abilities of PLMs like zero-shot instruction following.
Pre-Training to Learn in Context
One limitation of our paper is that the exact distribution of the intrinsic tasks in the original corpus and the constructed data is still unknown. Knowing the distribution can offer a better interpretation of the effectiveness of PICL, even of the strong performance of large language models. Besides, although we can find many constructed instances that share obvious intrinsic tasks (see Appendix F), there still exist some instances where the intrinsic tasks are hard to identify. How to better evaluate the contribution of these instances to the ICL ability or designing better filtering approaches to select more informative data for ICL is worth studying. Our task-semantics encoder inevitably contains some bias because it is trained on downstream datasets, although we have tried to ensure a large number and diversity of the dataset collection. However, the final language model is pre-trained on the general corpus, and we add the full document loss, which eliminates the bias to some extent.
Pre-Training to Learn in Context
Regarding computing power, we acknowledge that our framework takes relatively large training resources in the retrieval and pre-training process. Therefore, we did not conduct experiments based on extra-large language models. 
Pre-Training to Learn in Context
This section presents details of the data processing of the pre-training corpus and its statistics.
Pre-Training to Learn in Context
Data Processing Our pre-training corpus is a merge of OPENWEBTEXT (Gokaslan et al., 2019), WIKICORPUS (Foundation, 2022), and BOOKCOR-PUS (Zhu et al., 2015), downloaded from the Hug-gingFace datasets repository 1 . We first split each document in the corpus into paragraphs with "\n". To avoid training with too short paragraphs, we concatenate a paragraph with previous paragraphs if the token number after concatenation is lower than 128. We also exclude paragraphs longer than 500 tokens because they are not likely to fit into an instance with more than 1 paragraph. The filtering process in Section 2.3 drops about 24% instances. The licenses of all corpora allow for scientific research.
Pre-Training to Learn in Context
Statistics We plot the distribution of the mean paragraph length per instance in Figure 7(a) and the distribution of the paragraph number per instance in Figure 7(b). The average paragraph length is 150.0, and the average paragraph number in an instance is 11.7. We can see that the model sees various demonstration numbers in PICL pre-training.
Pre-Training to Learn in Context
The details of each text classification dataset and the corresponding prompt in evaluation are listed in Table 6. All datasets are downloaded from the HuggingFace datasets repository 1 . We simplify the evaluation prompts as much as possible to reduce the effect of prompt engineering. Following previous works (Brown et al., 2020;, the model is evaluated by the ranking score strategy, where we compare the perplexity of each classification label under the model and choose the label with the lowest perplexity. The licenses of all datasets allow for scientific research.
Pre-Training to Learn in Context
The original test split of the benchmark SUPER-NATURALINSTRUCTIONS (Wang et al., 2022) contains 119 tasks. We exclude tasks that appear in the training tasks of the task-semantics encoder or whose input length is too long to fit in the context of our model. Our final evaluation includes 105 tasks. A full list of the tasks is shown in Table 8. Pre-train learning rate 1e-5, 5e-6, 1e-6, 2e-7 batch size 64, 128, 256, 512 warmup 0, 1000, 5000 We use the same template to combine few-shot examples with task instructions as Wang et al. (2022). The license of this benchmark is Apache License 2.0.
Pre-Training to Learn in Context
The downstream datasets we use to train the tasksemantics encoder are a merge of the training data used in  and the HR→LR setting in (Min et al., 2022b). All datasets are downloaded from the HuggingFace datasets repository 1 and all prompts come from the PromptSource library (Bach et al., 2022) 2 . We exclude datasets from the sentiment classification task, the topic classification task, and the natural language inference task because they are included in our text classification evaluation. We finally get a collection of 37 datasets, as listed in Table 5. The licenses of all datasets allow for scientific research.
Pre-Training to Learn in Context
All model checkpoints we used come from the Hug-gingFace models repository 3 . The searching interval of each hyper-parameter is listed in  Figure 7, we can see that PICL is also applicable to larger models, outperforming the baselines based on the same-sized model on most datasets.
Pre-Training to Learn in Context
We present the performance comparison between PICL and MetaICL per evaluation task in Figure  8. PICL outperforms MetaICL on 77 / 105 tasks, indicating that PICL ensures the better generalization of the trained model. The name of each task is also listed in Figure 8. We can see that the top three tasks where MetaICL performs the best are:
Pre-Training to Learn in Context
which are all "Yes/No" classification tasks. The top three tasks where PICL performs the best are:
Pre-Training to Learn in Context
which are text generation, text generation, and "Correct/Wrong" classification tasks respectively. The four tasks where PICL and MetaICL have the same scores are:
Pre-Training to Learn in Context
• bard_analogical_reasoning_containers, • copa_commonsense_cause_effect, • winogrande_answer_generation, • bard_analogical_reasoning_trash_or_treasure, which belong to commonsense reasoning and word analogy tasks.
Pre-Training to Learn in Context
In Table 9 and 10, we present several cases of the retrieved paragraphs and the corresponding intrinsic tasks. We can see that there exists a large range of intrinsic tasks in the constructed data and many of them do not appear in the training data of the tasksemantics encoder, which shows the generalization of the encoder.  task1442_doqa_movies_isanswerable  task623_ohsumed_yes_no_answer_generation  task642_esnli_classification  task329_gap_classification  task1612_sick_label_classification  task1195_disflqa_disfluent_to_fluent_conversion  task1624_disfl_qa_question_yesno_classification  task1728_web_nlg_data_to_text  task199_mnli_classification  task743_eurlex_summarization  task640_esnli_classification  task520_aquamuse_answer_given_in_passage  task1409_dart_text_generation  task1516_imppres_naturallanguageinference  task970_sherliic_causal_relationship  task738_perspectrum_classification  task304_numeric_fused_head_resolution  task641_esnli_classification  task1154_bard_analogical_reasoning_travel  task1622_disfl_qa_text_modication  task1156_bard_analogical_reasoning_tools  task1157_bard_analogical_reasoning_rooms_for_containers  task1155_bard_analogical_reasoning_trash_or_treasure  task033_winogrande_answer_generation  task828_copa_commonsense_cause_effect  task1159_bard_analogical_reasoning_containers  task827_copa_commonsense_reasoning  task1534_daily_dialog_question_classification  task1158_bard_analogical_reasoning_manipulating_items  task290_tellmewhy_question_answerability  task392_inverse_causal_relationship  task891_gap_coreference_resolution  task677_ollie_sentence_answer_generation  task1152_bard_analogical_reasoning_causation  task249_enhanced_wsc_pronoun_disambiguation  task288_gigaword_summarization  task1586_scifact_title_generation  task769_qed_summarization  task201_mnli_neutral_classification  task937_defeasible_nli_social_classification  task879_schema_guided_dstc8_classification  task1342_amazon_us_reviews_title  task202_mnli_contradiction_classification  task1161_coda19_title_generation  task1640_aqa1.0_answerable_unanswerable_question_classification  task1390_wscfixed_coreference  task220_rocstories_title_classification  task418_persent_title_generation  task1391_winogrande_easy_answer_generation  task200_mnli_entailment_classification  task020_mctaco_span_based_question  task619_ohsumed_abstract_title_generation  task1529_scitail1.1_classification  task620_ohsumed_medical_subject_headings_answer_generation  task035_winogrande_question_modification_person  task281_points_of_correspondence  task1540_parsed_pdfs_summarization  task1394_meta_woz_task_classification  task1439_doqa_cooking_isanswerable  task233_iirc_link_exists_classification  task510_reddit_tifu_title_summarization  task226_english_language_answer_relevance_classification  task349_squad2.0_answerable_unanswerable_question_classification  task050_multirc_answerability  task1153_bard_analogical_reasoning_affordance  task036_qasc_topic_word_to_generate_related_fact  task1388_cb_entailment  task1407_dart_question_generation  task391_causal_relationship  task890_gcwd_classification  task957_e2e_nlg_text_generation_generate  task613_politifact_text_generation  task219_rocstories_title_answer_generation  task362_spolin_yesand_prompt_response_sub_classification  task880_schema_guided_dstc8_classification  task569_recipe_nlg_text_generation  task935_defeasible_nli_atomic_classification  task1531_daily_dialog_type_classification  task442_com_qa_paraphrase_question_generation  task1533_daily_dialog_formal_classification  task614_glucose_cause_event_detection  task1554_scitail_classification  task039_qasc_find_overlapping_words  task648_answer_generation  task936_defeasible_nli_snli_classification  task1659_title_generation  task500_scruples_anecdotes_title_generation  task602_wikitext-103_answer_generation  task401_numeric_fused_head_reference  task893_gap_fill_the_blank_coreference_resolution  task1385_anli_r1_entailment  task892_gap_reverse_coreference_resolution  task1387_anli_r3_entailment  task1631_openpi_answer_generation  task1615_sick_tclassify_b_relation_a  task330_gap_answer_generation  task190_snli_classification  task1386_anli_r2_entailment  task1393_superglue_copa_text_completion  task1664_winobias_text_generation  task133_winowhy_reason_plausibility_detection task393_plausible_result_generation task034_winogrande_question_modification_object PICL > MetaICL MetaICL > PICL   Table 6: Details of the text classification datasets. "Sent. CLS" stands for "Sentiment Classification". "NLI" stands for "Natural Language Inference". "Subj. CLS" stands for "Subjectivity Classification". "Topic CLS" stands for "Topic Classification".   • How did Japan stumble on for another nine years, borrowing trillions of yen and squandering those trillions on make-work bridges to nowhere and lavish social spending? Answer: its citizens self-funded its deficits by saving trillions and investing those trillions in government debt. • How did our country thrive without income taxes for 126 years? Answer: federal spending was significantly lower than it is today. In the early 1900s, government spending accounted for roughly 7% of our GDP; today, federal spending accounts for around 35% of our GDP. • What was Trump's biggest persuasion problem in the election? Answer: His opponents did a great job of framing him as some kind of Hitler.  4867 7 • Now we can log into our Twilio account and set the Message Request URL to our sms route via ngrok: Try the app out by texting into your new Twilio number and you'll get the response back. Displaying Our Messages We're now passing our message to the arduino. The next step is to write the code that examines that message and displays it on our LCD. Let's lay the foundation for our app: # include < Wire. h > # include "rgb_lcd.h" rgb_lcd lcd ; void setup () { Serial. begin ( 9600 ); // set up the LCD's number of columns and rows: lcd. begin ( 16, 2 ); lcd. setCursor ( 0, 1 ); // Print a message to the LCD. lcd. print ( "Ricky's Pager" ); delay ( 1000 ); } void loop () { } • Now we're ready to track allocations. The first step is to "hijack" our 3 memory functions we defined in the first part (lines 4, 11 and 17): void* _Malloc(tU32 Size, tU32 Alloc-Type, const tChar* Desc, const tChar* File, tU32 Line) { void* Result = malloc(Size); RegisterAlloc(Result, Size, AllocType, Desc, File, Line); return Result; } void* _Realloc(void* Ptr, tU32 Size, const tChar* File, tU32 Line) { void* Result = realloc(Ptr, Size); UpdateAlloc(Ptr, Result, Size, File, Line); return Result; } void _Free(void* Ptr) { UnregisterAlloc(Ptr); return free(Ptr); } • Here we use the gulp.src API to specify our input files. One thing to note is that we need to specify a reporter for JSHint. I'm using the default reporter, which should be fine for most people. More on this can be found on the JSHint website. Compress Images Next, we'll set up image compression: gulp. task ( 'images', function () { return gulp. src ('src/images/**/*' ). pipe ( imagemin ({ optimizationLevel : 3, progressive : true, interlaced : true })). pipe ( gulp. dest ( 'dist/assets/img' )). pipe ( notify ({ message : 'Images task complete' })); }); • He strode toward her, barely slowly when he reached her. One arm slid around her waist and the other along her shoulders. She'd barely registered his touch before his mouth descended upon hers. • He lifted his head and stared at her. His face paled, and for the first time she noticed a spattering of orange freckles on his nose and across his cheekbones. He didn't speak. Just stared. • He continued rocking her gently, steadily. Her body's tremors calmed and her sobs quietened. He removed his white handkerchief from his trouser pocket, wiping her face. She didn't look at him and kept her eyes lowered.
Pre-Training to Learn in Context
Vivid Description 10 • Even the sugary cereals, they said, are of nutritional value because they contain vitamins and minerals. Research shows that 40 percent of U.S. children consume their milk via cereal, said Sutherland of Kellogg. General Mills cites data from the Journal of the American Dietetic Association that says people who frequently eat cereal, including kids who eat sweetened ones, tend to have healthier body weights than those that don't. • Lead's toxicity has long been known, and most of the uses that led to human exposure, like the manufacture of lead paint, have been banned for decades. Lead ammunition consumed only about 3 percent of the 6.4 million tons of lead used worldwide in 2000, according to a 2003 report by the Nordic Council of Ministers. • One reason Tesla has pushed the technology so aggressively is that its battery packs store more than three times the energy of its competitors' electric-car batteries. As a result, they require more power to charge quickly, says Arindam Maitra, a senior project manager at the Electric Power Research Institute. A1. Did you describe the limitations of your work?
Pre-Training to Learn in Context
The section named "Limitations" A2. Did you discuss any potential risks of your work?
Pre-Training to Learn in Context
The section named "Limitations" A3. Do the abstract and introduction summarize the paper's main claims?
Pre-Training to Learn in Context
Section 1 A4. Have you used AI writing assistants when working on this paper?
Pre-Training to Learn in Context
Left blank. B2. Did you discuss the license or terms for use and / or distribution of any artifacts?