Why does in-context learning work?
A number of empirical studies have attempted to characterize the behavior of ICL and explain why it works. These studies have found that ICL can be overly sensitiveo certain aspects of the prompt, such as the order of the demonstration examples (Lu et al., 2022) and choice of label words (Zhao et al., 2021), but also surprisingly insensitive to others. In particular, Min et al. (2022) show that LLMs can largely ignore the relationship between inputs and labels in the demonstration example and Webson and Pavlick (2022) show that the performance of ICL can perform well on NLI even if the prompt is irrelevant or misleading. Relatedly, Wei et al. (2023); Pan et al. (2023) show that LLMs can perform ICL well with flipped labels or semantically-unrelated labels, but such ability of overriding semantic priors emerges with scale. Some theoretical work has also attempted to explain why prompt-based methods work, by drawing connections between the prompting setting and properties of the pretraining distribution (Saunshi et al., 2021; Wei et al., 2021; Xie et al., 2022) or by arguing that Transformers can act as meta-learners, implicitly performing gradient descent on the in-context examples (von Oswald et al., 2022; Akyürek et al., 2023).
An Explanation of In-context Learning as Implicit Bayesian Inference
Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples. The LM learns from these examples without being explicitly pretrained to learn. Thus, it is unclear what enables in-context learning. In this paper, we study how in-context learning can emerge when pretraining documents have long-range coherence. Here, the LM must infer a latent document-level concept to generate coherent next tokens during pretraining. At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt. We prove when this occurs despite a distribution mismatch between prompts and pretraining data in a setting where the pretraining distribution is a mixture of HMMs. In contrast to messy large-scale datasets used to train LMs capable of in-context learning, we generate a small-scale synthetic dataset (GINC) where Transformers and LSTMs both exhibit in-context learning. Beyond the theory, experiments on GINC exhibit large-scale real-world phenomena including improved in-context performance with model scaling (despite the same pretraining loss), sensitivity to example order, and instances where zero-shot is better than few-shot in-context learning.
[{"first":"Sang","last":"Xie","middle":["Michael"]},{"first":"Aditi","last":"Raghunathan","middle":[]},{"first":"Percy","last":"Liang","middle":[]},{"first":"Tengyu","last":"Ma","middle":[]}]
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Large language models (LMs) are able to in-context learn—perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required—randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of endtask performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.
[{"first":"Sewon","last":"Min","middle":[]},{"first":"Xinxi","last":"Lyu","middle":[]},{"first":"Ari","last":"Holtzman","middle":[]},{"first":"Mikel","last":"Artetxe","middle":[]},{"first":"Mike","last":"Lewis","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]}]
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are “fantastic” and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.
[{"first":"Yao","last":"Lu","middle":[]},{"first":"Max","last":"Bartolo","middle":[]},{"first":"Alastair","last":"Moore","middle":[]},{"first":"Sebastian","last":"Riedel","middle":[]},{"first":"Pontus","last":"Stenetorp","middle":[]}]
What learning algorithm is in-context learning? Investigations with linear models
Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples $(x, f(x))$ presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners' late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations are released at https://github.com/ekinakyurek/google-research/blob/master/incontext.
[{"first":"Ekin","last":"Akyurek","middle":[]},{"first":"Dale","last":"Schuurmans","middle":[]},{"first":"Jacob","last":"Andreas","middle":[]},{"first":"Tengyu","last":"Ma","middle":[]},{"first":"Denny","last":"Zhou","middle":[]}]
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).
[{"first":"Tianyi","last":"Zhang","middle":[]},{"first":"Tatsunori","last":"Hashimoto","middle":[]}]
Calibrate Before Use: Improving Few-Shot Performance of Language Models
GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as"N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.
[{"first":"Tony","last":"Zhao","middle":["Z."]},{"first":"Eric","last":"Wallace","middle":[]},{"first":"Shi","last":"Feng","middle":[]},{"first":"Dan","last":"Klein","middle":[]},{"first":"Sameer","last":"Singh","middle":[]}]
Larger language models do in-context learning differently
We study how in-context learning (ICL) in language models is affected by semantic priors versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with semantically-unrelated labels-across various model families (GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM). First, experiments on ICL with flipped labels show that overriding semantic priors is an emergent ability of model scale. While small language models ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining, large models can override semantic priors when presented with in-context exemplars that contradict priors, despite the stronger semantic priors that larger models may hold. We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e.g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task. The ability to do SUL-ICL also emerges primarily with scale, and large-enough language models can even perform linear classification in a SUL-ICL setting. Finally, we evaluate instruction-tuned models and find that instruction tuning strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former.
[{"first":"Jerry","last":"Wei","middle":[]},{"first":"Jason","last":"Wei","middle":[]},{"first":"Yi","last":"Tay","middle":[]},{"first":"Dustin","last":"Tran","middle":[]},{"first":"Albert","last":"Webson","middle":[]},{"first":"Yifeng","last":"Lu","middle":[]},{"first":"Xinyun","last":"Chen","middle":[]},{"first":"Hanxiao","last":"Liu","middle":[]},{"first":"Da","last":"Huang","middle":[]},{"first":"Denny","last":"Zhou","middle":[]},{"first":"Tengyu","last":"Ma","middle":[]}]
Transformers learn in-context by gradient descent
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .
[{"first":"Johannes","last":"Oswald","middle":["von"]},{"first":"Eyvind","last":"Niklasson","middle":[]},{"first":"Ettore","last":"Randazzo","middle":[]},{"first":"Joao","last":"Sacramento","middle":[]},{"first":"Alexander","last":"Mordvintsev","middle":[]},{"first":"Andrey","last":"Zhmoginov","middle":[]},{"first":"Max","last":"Vladymyrov","middle":[]}]
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Autoregressive language models pretrained on large corpora have been successful at solving downstream tasks, even with zero-shot usage. However, there is little theoretical justification for their success. This paper considers the following questions: (1) Why should learning the distribution of natural language help with downstream classification tasks? (2) Why do features learned using language modeling help solve downstream tasks with linear classifiers? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as next word prediction tasks, thus making language modeling a meaningful pretraining task. For (2), we analyze properties of the cross-entropy objective to show that $\epsilon$-optimal language models in cross-entropy (log-perplexity) learn features that are $\mathcal{O}(\sqrt{\epsilon})$-good on natural linear classification tasks, thus demonstrating mathematically that doing well on language modeling can be beneficial for downstream tasks. We perform experiments to verify assumptions and validate theoretical results. Our theoretical insights motivate a simple alternative to the cross-entropy objective that performs well on some linear classification tasks.
[{"middle":[],"last":"Saunshi","first":"Nikunj"},{"middle":[],"last":"Malladi","first":"Sadhika"},{"middle":[],"last":"Arora","first":"Sanjeev"}]
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Pretrained language models have achieved state-of-the-art performance when adapted to a downstream NLP task. However, theoretical analysis of these models is scarce and challenging since the pretraining and downstream tasks can be very different. We propose an analysis framework that links the pretraining and downstream tasks with an underlying latent variable generative model of text -- the downstream classifier must recover a function of the posterior distribution over the latent variables. We analyze head tuning (learning a classifier on top of the frozen pretrained model) and prompt tuning in this setting. The generative model in our analysis is either a Hidden Markov Model (HMM) or an HMM augmented with a latent memory component, motivated by long-term dependencies in natural language. We show that 1) under certain non-degeneracy conditions on the HMM, simple classification heads can solve the downstream task, 2) prompt tuning obtains downstream guarantees with weaker non-degeneracy conditions, and 3) our recovery guarantees for the memory-augmented HMM are stronger than for the vanilla HMM because task-relevant information is easier to recover from the long-term memory. Experiments on synthetically generated data from HMMs back our theoretical findings.
[{"first":"Colin","last":"Wei","middle":[]},{"first":"Sang","last":"Xie","middle":["Michael"]},{"first":"Tengyu","last":"Ma","middle":[]}]
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Large language models (LLMs) exploit in-context learning (ICL) to solve tasks with only a few demonstrations, but its mechanisms are not yet well-understood. Some works suggest that LLMs only recall already learned concepts from pre-training, while others hint that ICL performs implicit learning over demonstrations. We characterize two ways through which ICL leverages demonstrations. Task recognition (TR) captures the extent to which LLMs can recognize a task through demonstrations -- even without ground-truth labels -- and apply their pre-trained priors, whereas task learning (TL) is the ability to capture new input-label mappings unseen in pre-training. Using a wide range of classification datasets and three LLM families (GPT-3, LLaMA and OPT), we design controlled experiments to disentangle the roles of TR and TL in ICL. We show that (1) models can achieve non-trivial performance with only TR, and TR does not further improve with larger models or more demonstrations; (2) LLMs acquire TL as the model scales, and TL's performance consistently improves with more demonstrations in context. Our findings unravel two different forces behind ICL and we advocate for discriminating them in future ICL research due to their distinct nature.
[{"first":"Jane","last":"Pan","middle":[]},{"first":"Tianyu","last":"Gao","middle":[]},{"first":"Howard","last":"Chen","middle":[]},{"first":"Danqi","last":"Chen","middle":[]}]
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Autoregressive language models pretrained on large corpora have been successful at solving downstream tasks, even with zero-shot usage. However, there is little theoretical justification for their success. This paper considers the following questions: (1) Why should learning the distribution of natural language help with downstream classification tasks? (2) Why do features learned using language modeling help solve downstream tasks with linear classifiers? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as next word prediction tasks, thus making language modeling a meaningful pretraining task. For (2), we analyze properties of the cross-entropy objective to show that $\epsilon$-optimal language models in cross-entropy (log-perplexity) learn features that are $\mathcal{O}(\sqrt{\epsilon})$-good on natural linear classification tasks, thus demonstrating mathematically that doing well on language modeling can be beneficial for downstream tasks. We perform experiments to verify assumptions and validate theoretical results. Our theoretical insights motivate a simple alternative to the cross-entropy objective that performs well on some linear classification tasks.
[{"first":"Nikunj","last":"Saunshi","middle":[]},{"first":"Sadhika","last":"Malladi","middle":[]},{"first":"Sanjeev","last":"Arora","middle":[]}]
An Explanation of In-context Learning as Implicit Bayesian Inference
Large language models (LMs) such as GPT-3 (Brown et al., 2020, Lieber et al., 2021, Radford et al., 2019, Wang and Komatsuzaki, 2021 are pretrained on massive text corpora to predict the next word given previous words. They demonstrate the surprising ability to do in-context learning, where an LM "learns" to do a task simply by conditioning on a prompt containing input-output pairs, achieving SOTA results on LAMBADA (Paperno et al., 2016) and TriviaQA (Joshi et al., 2017) tasks (18% and 3% over previous SOTA (Brown et al., 2020)). For example, consider the task of predicting nationalities from names. A prompt (Figure 1) is constructed by concatenating independent "training" examples (e.g., "Albert Einstein was German") followed by a "test example" ("Marie Curie was"). Conditioning on this prompt, GPT-3 places the largest probability on the correct output p("Polish" | "Albert Einstein was German \n Mahatma Gandhi was Indian \n Marie Curie was") 1 The code, data, and experiments are located on GitHub and CodaLab. Figure 1: In-context learning can emerge from modeling long-range coherence in the pretraining data. During pretraining, the language model (LM) implicitly learns to infer a latent concept (e.g., wiki bios, which typically transition between name (Albert Einstein) → nationality (German) → occupation (physicist) → ...) shared across sentences in a document. Although prompts are unnatural sequences that concatenate independent examples, in-context learning occurs if the LM can still infer the shared concept across examples to do the task (name → nationality, which is part of wiki bios). by inferring the task from examples. Intruigingly, GPT-3 was not explicitly pretrained to learn from examples, and the distribution of prompts (which concatenate independent examples) is quite different from natural language. Our understanding of in-context learning is limited since (i) real pretraining data is messy and (ii) in-context learning has so far required large-scale datasets and models. In this paper, we introduce a simple pretraining distribution where in-context learning emerges. To generate a document, we first draw a latent concept θ, which parameterizes the transitions of a Hidden Markov Model (HMM) (Baum and Petrie, 1966), then sample a sequence of tokens from the HMM (Figure 9). This latent variable structure is common in topic models such as LDA (Blei et al., 2003, Gruber et al., 2007. During pretraining, the LM must infer the latent concept across multiple sentences to generate coherent continuations. When conditioning on a prompt, in-context learning occurs when the LM also infers a shared prompt concept across examples to make a prediction. We assume the LM fits the pretraining distribution p exactly with enough data and expressivity, so that the question of in-context learning becomes characterizing the conditional distribution of completions given prompts p(output|prompt) under the pretraining distribution, where the prompt is generated from a different distribution p prompt . This conditional distribution, which is the posterior predictive distribution, marginalizes out the latent concepts: p(output|prompt) = concept p(output|concept, prompt)p(concept|prompt)d(concept).
An Explanation of In-context Learning as Implicit Bayesian Inference
(1)
An Explanation of In-context Learning as Implicit Bayesian Inference
If p(concept|prompt) concentrates on the prompt concept with more examples, then the LM learns via marginalization by "selecting" the prompt concept. Thus, in-context learning can be viewed as the LM implicitly performing Bayesian inference. The main challenge is that prompts are sampled from a different distribution than the pretraining distribution. The canonical Bayesian asymptotic tool is the Bernstein-von Mises theorem (Gunst and Shcherbakova, 2008, Kleijn and van der Vaart, 2012, van der Vaart, 1998, which asserts (under regularity conditions) that the posterior distribution of a latent variable concentrates on the maximum likelihood estimate. However, Bernstein-von Mises typically assumes observations are independent and/or drawn from the same distribution as the model, both of which are not satisfied. We prove that despite the distribution mismatch, the asymptotic prediction error of in-context learning is optimal when the signal about the latent concept in each prompt example is larger than the error due to the distribution mismatch. Additionally, we prove that the in-context learning error decreases with the length of each example-thus, information in the inputs, not just the input-output mapping, can be useful for in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
As a companion to this theory, we created the Generative IN-Context learning dataset (GINC), which is a small-scale synthetic dataset for studying in-context learning. We find that both Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter and Schmidhuber, 1997) trained on GINC exhibit in-context learning. We verify intuitions from the theory, showing that the accuracy of incontext learning improves with the number of examples and example length. Ablations of the GINC dataset show that the latent concept structure in the pretraining distribution is crucial to the emergence of in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
The experiments also bring up open questions which go beyond our theory, which only studies the pretraining distribution. We find that scaling up the number of model parameters steadily improves the in-context accuracy despite achieving the same pretraining loss, showing that larger models may improve in-context learning beyond increasing the capacity for memorizing the training data better. Previously observed in-context learning phenomena such as sensitivity to example ordering (Zhao et al., 2021) and the existence of settings where zero-shot is better than one/fewshot learning (Brown et al., 2020) are also mirrored in GINC.
An Explanation of In-context Learning as Implicit Bayesian Inference
Pretraining distribution. In our framework, a latent concept θ from a family of concepts Θ defines a distribution over observed tokens o from a vocabulary O. To generate a document, we first sample a concept from a prior p(θ) and then sample the document given the concept. Each pretraining document is a length T sequence:
An Explanation of In-context Learning as Implicit Bayesian Inference
x i = O i [1 : k − 1] (the first k − 1 tokens) followed by an output token y i = O i [k] at the end 2 . The i-th training example is independently generated as follows:
An Explanation of In-context Learning as Implicit Bayesian Inference
1. Generate a start hidden state h start i from a prompt start distribution p prompt .
An Explanation of In-context Learning as Implicit Bayesian Inference
, the pretraining distribution conditioned on a prompt concept θ * .
An Explanation of In-context Learning as Implicit Bayesian Inference
The test input x test = x n+1 is sampled similarly. Between each example, there is a special delimiter token o delim . The prompt consists of a sequence of training examples (S n ) followed by the test example x test :
An Explanation of In-context Learning as Implicit Bayesian Inference
Mismatch between prompt and pretraining distributions. Since transitions between independent examples can be unnatural, the prompts are low probability sequences under the pretraining distribution. We provide a simple illustration using the names to nationalities example. Suppose that wiki bio documents in the pretraining data typically transition between name → nationality → occupation → . . . . In the prompt, the examples transition between name → nationality → name → nationality → . . . , which contains low-probability transitions such as "German" → "Mahatma Gandhi". The prompt formatting (e.g., choice of delimiter) can also be a source of mismatch. We aim to show that despite this mismatch, large LMs can infer the prompt concept from examples.
An Explanation of In-context Learning as Implicit Bayesian Inference
In-context predictor and task. For in-context learning, the output target y for each example x is sampled according to p prompt (y|x):
An Explanation of In-context Learning as Implicit Bayesian Inference
where h start test denotes the hidden state corresponding to the first token of x test . We analyze the in-context predictor f n (x test ) = arg max y p(y|S n , x test ), which outputs the most likely prediction over the pretraining distribution conditioned on the prompt from the prompt distribution 3 . We study the in-context predictor and its expected 0-1 error with n examples
An Explanation of In-context Learning as Implicit Bayesian Inference
We detail the assumptions in our framework, including the structure of delimiters and regularity assumptions. We first assume that there exists a subset of delimiter hidden states D which generates the special delimiter token o delim deterministically.
An Explanation of In-context Learning as Implicit Bayesian Inference
Assumption 1 (Delimiter hidden states). Let the delimiter hidden states D be a subset of H. For any h delim ∈ D and θ ∈ Θ, p(o delim |h delim , θ) = 1 and for any h / ∈ D, p(o delim |h, θ) = 0.
An Explanation of In-context Learning as Implicit Bayesian Inference
Thus, observing the delimiter o delim reveals that the corresponding hidden state is in D, but does not reveal which element of D it is. The delimiter is usually a token that can appear in a broad range of contexts (e.g., newline). The delimiter ideally does not distract from the examples -for example, an adversarial delimiter could look like part of the input x. To mitigate these scenarios, we assume that no delimiter (e.g., newline) is significantly more likely under one concept rather than another.
An Explanation of In-context Learning as Implicit Bayesian Inference
2 The example length k is fixed for simplicity -we leave extending our analysis to variable k as future work. 3 In practice, greedy decoding or nucleus sampling (Holtzman et al., 2020) are used for likely completions.
An Explanation of In-context Learning as Implicit Bayesian Inference
Assumption 2 (Bound on delimiter transitions). For any delimiter state h delim ∈ D and any hidden state h ∈ H, the probability of transitioning to a delimiter hidden state under θ is upper bounded p(h delim |h, θ) < c 2 for any θ ∈ Θ \ {θ * }, and is lower bounded p(h delim |h, θ * ) > c 1 > 0 for θ * . Additionally, the start hidden state distribution for delimiter hidden states is bounded as p(h delim |θ) ∈ [c 3 , c 4 ].
An Explanation of In-context Learning as Implicit Bayesian Inference
The choice of prompt start distribution can be a source of distribution shift which is separate from the distribution shift from concatenating independent examples. We make an assumption that limits how much distribution shift is introduced by the prompt start distribution.
An Explanation of In-context Learning as Implicit Bayesian Inference
Assumption 3 (Distribution shift from prompt start distribution). We assume that the prompt start distribution p prompt is close in TV distance to all hidden transition distributions (under θ * ) starting from a delimiter hidden state:
An Explanation of In-context Learning as Implicit Bayesian Inference
is the margin between the most likely label y max = arg max y p prompt (y|x test ) and the second most likely label.
An Explanation of In-context Learning as Implicit Bayesian Inference
Note that even when the maximum TV distance is 0, there is still distribution shift from concatenating independent examples. We also assume the prompt concept θ * is in the family Θ, which is a broad set of concepts.
An Explanation of In-context Learning as Implicit Bayesian Inference
Assumption 4 (Well-specification). The prompt concept θ * is in Θ.
An Explanation of In-context Learning as Implicit Bayesian Inference
Even though the pretraining distribution is broad, the prompt is still low probability under the pretraining distribution since it concatenates independent examples. Finally, if the prompt has zero probability under the prompt concept θ * , then Bayesian inference will not be able to infer the prompt concept as in Section 3.1. The following are regularity assumptions which mainly ensure that the prompt is not zero probability under θ * .
An Explanation of In-context Learning as Implicit Bayesian Inference
Assumption 5 (Regularity). The pretraining distribution p satisfies: 1) Lower bound on transition probability for the prompt concept θ * : for any pair of hidden states h, h ∈ H, p(h|h , θ * ) > c 5 > 0. 2) Start hidden state is lower bounded: for any h ∈ H, p(h|θ * ) ≥ c 8 > 0. 3) All tokens can be emitted: for every symbol o, there is some hidden state h ∈ H such that p(o|h, θ * ) > c 6 > 0, 4) The prior p(θ) has support over the entire concept family Θ and is bounded above everywhere.
An Explanation of In-context Learning as Implicit Bayesian Inference
We prove that in the limit of infinite examples, the error of the in-context predictor is optimal if a distinguishability condition holds -the prompt concept θ * is distinct enough from the other concepts in Θ (e.g., when Θ is a discrete set). When distinguishability does not hold (e.g, Θ is continuousvalued), we show that the expected error still decreases with the length of each example, showing that information in both the inputs and the input-output mapping contribute to in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
Our goal is to show that arg max y p(y|S n , x test ) → arg max y p prompt (y|x test ) as the number of examples n grows. In the following, assume that the prompt has non-zero probability under the pretraining distribution p given θ * , meaning that p(S n , x test |θ * ) > 0. We expand p(y|S n , x test ) to analyze its limit:
An Explanation of In-context Learning as Implicit Bayesian Inference
(Law of total prob, Markov property, divide by p(S n , x test |θ * ) (a constant))
An Explanation of In-context Learning as Implicit Bayesian Inference
where r n (θ) = 1 n log p(Sn,xtest|θ) p(Sn,xtest|θ * ) . In Theorem 1, we prove that under a distinguishability condition, exp(n · r n (θ)) → 0 for all concepts θ except the prompt concept θ * , where exp(n · r n (θ * )) = 1. The only nonzero term in the integral is when θ = θ * , and thus the prompt concept is "selected" as a consequence of Bayesian inference 4 . Lemma 1 shows that the argmax after restricting to θ * is the same as the most likely label under p prompt (y|x test ) (using Assumption 3). Putting these together with Equation 6, the in-context predictor infers the prompt concept θ * :
An Explanation of In-context Learning as Implicit Bayesian Inference
Thus, the in-context predictor is optimal as the number of in-context examples increases.
An Explanation of In-context Learning as Implicit Bayesian Inference
Recall from Section 3.1 that if exp(n · r n (θ)) → 0 for all θ = θ * , then Bayesian inference "selects" the prompt concept through marginalization. To do this, we focus on showing that r n (θ), the average log-likelihood ratio between θ and θ * , converges to a negative constant, and thus nr n goes to −∞. 
An Explanation of In-context Learning as Implicit Bayesian Inference
To show this, we expand p(S n |θ) with the chain rule, and with Assumption 5 (to bound p(x test |S n , θ) by O(1)) it can be shown that
An Explanation of In-context Learning as Implicit Bayesian Inference
We then marginalize p(O ex i |O ex 1:i−1 , θ) over the hidden state h delim i−1 corresponding to the delimiter in 4 We can exchange limits and integrals since the probabilities are bounded (dominated convergence).
An Explanation of In-context Learning as Implicit Bayesian Inference
While summing over H above would be a trivial equality, we can replace H with the set of delimiter hidden states D since p(h|O ex 1:i−1 , θ) = 0 for non-delimiter hidden states h / ∈ D (Assumption 1). We used in the first equality that O ex 1:i−1 → h delim i−1 → O ex i forms a Markov chain and p(o delim i−1 |h delim i−1 ) = 1 (Assumption 1) to change O ex i to O i . Finally, we can show using properties of delimiter hidden states (Assumption 2) that p(h delim i−1 |O ex
An Explanation of In-context Learning as Implicit Bayesian Inference
in the second step. Therefore, we can upper bound r n (θ) as
An Explanation of In-context Learning as Implicit Bayesian Inference
The expectation term can be written as the difference of two KL divergences,
An Explanation of In-context Learning as Implicit Bayesian Inference
. We bound the first KL term by a constant using Assumption 5 -intuitively for one example, p prompt and p(·|θ * ) are close. We break the second term into a sum of negative KL divergences over k tokens. There are O(k) KL terms and only O(1) other error terms, which come from the distribution mismatch between the prompt and pretraining distributions. If the KL terms are larger than the error terms, then r n (θ) has a negative limit. If this holds for all θ = θ * , then we have exp(n · r n (θ)) → 0 for all θ = θ * , enabling in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
We define a distinguishability condition which formalizes when in-context learning occurs. Letting
An Explanation of In-context Learning as Implicit Bayesian Inference
, θ) be the output distribution of the j-th token given the previous tokens and p j prompt (o) := p prompt (O[j] = o|O[1 : j − 1]) be the analogous distribution under the prompt distribution, the distinguishability condition depends on the KL divergence between p j prompt (which represents θ * ) and p j θ as well as error terms θ start and θ delim coming from the distribution mismatch between the prompt and pretraining distributions at the start and delimiter token for each example:
An Explanation of In-context Learning as Implicit Bayesian Inference
θ delim := 2(log(c 2 ) − log(c 1 )) + log(c 4 ) − log(c 3 ), θ start := log(1/c 8 ).
An Explanation of In-context Learning as Implicit Bayesian Inference
Condition 1 (Distinguishability). We define θ * to be distinguishable if for all θ ∈ Θ, θ = θ * ,
An Explanation of In-context Learning as Implicit Bayesian Inference
When the signal from KL divergence (LHS) is larger than the error terms, Equation 14 is satisfied ( Figure 2). For larger example lengths k, the LHS increases, improving distinguishability. Intuitively, larger example lengths increase the proportion of the prompt sampled from the pretraining distribution by providing more evidence for Bayesian inference. Under Condition 1, the in-context predictor asymptotically achieves the optimal expected error.
An Explanation of In-context Learning as Implicit Bayesian Inference
Theorem 1. Assume the assumptions in Section 2.1 hold. If Condition 1 holds, then as n → ∞ the prediction according to the pretraining distribution is arg max
An Explanation of In-context Learning as Implicit Bayesian Inference
Thus, the in-context predictor f n achieves the optimal 0-1 risk: lim n→∞ L 0-1 (f n ) = inf f L 0-1 (f ).
An Explanation of In-context Learning as Implicit Bayesian Inference
When the signal about the prompt concept within each example (green) is greater than the error from low-probability transitions between examples, in-context learning succeeds in our latent concept setting (Theorem 1). Increasing the example length k increases the signal. The signal for in-context learning comes from tokens in both the inputs and the input-output mapping.
An Explanation of In-context Learning as Implicit Bayesian Inference
The distinguishability condition (Condition 1) fails when there is some θ = θ * for which the KL divergence between θ and θ * is less than the error terms. However, this also means that the output distributions of θ and θ * are close in KL. We leverage this to prove that the expected 0-1 error decreases with the example length k under two different settings where distinguishability does not hold.
An Explanation of In-context Learning as Implicit Bayesian Inference
Our first result relies on a continuity assumption between the concept parameter and its corresponding output distribution. Our assumption is based on prior works (Kleijn and van der Vaart, 2012), where the KL divergence is assumed to have a 2nd-order Taylor expansion.
An Explanation of In-context Learning as Implicit Bayesian Inference
Theorem 2. Let the set of θ which does not satisfy Equation 14 in Condition 1 to be B. Assume that KL divergences have a 2nd-order Taylor expansion around θ * :
An Explanation of In-context Learning as Implicit Bayesian Inference
where I j,θ * is the Fisher information matrix of the j-th token distribution with respect to θ * . Let γ θ * = maxj λmax(I j,θ * ) min jλ min (I j,θ * ) where λ max , λ min return the largest and smallest eigenvalues. Then for k ≥ 2 and as n → ∞, the 0-1 risk of the in-context learning predictor f n is bounded as
An Explanation of In-context Learning as Implicit Bayesian Inference
is a calibration function (Steinwart, 2007, Ávila Pires and Szepesvári, 2016) for the multiclass logistic loss for δ ∈ [0, 1), assuming that the minimizers of the 0-1 risk and multiclass logistic risk are the same.
An Explanation of In-context Learning as Implicit Bayesian Inference
Since the inverse calibration function g −1 is roughly linear in for ≤ 0.7, the excess risk roughly decreases as O(1/k). When the "worst-case condition number" γ θ * of the Fisher information matrices is smaller (well-conditioned), the error decreases. Intuitively, this means that there is no direction to vary θ * in which the output distribution will sharply change. As a consequence, the concepts θ that are not distinguishable from the prompt concept θ * parameterize distributions that produce similar outputs to the prompt concept and thus achieve a small error.  
An Explanation of In-context Learning as Implicit Bayesian Inference
In the setting where the length of x test is random (uniformly from 2 to k), we can give a similar error guarantee without continuity.
An Explanation of In-context Learning as Implicit Bayesian Inference
Theorem 3. Let the set of θ which does not satisfy Equation 14 in Condition 1 to be B. Let the length of the test example x test be uniformly distributed between 2 and k, for k ≥ 2. Then for k ≥ 2 and as n → ∞, the 0-1 risk of the in-context learning predictor f n is bounded as
An Explanation of In-context Learning as Implicit Bayesian Inference
assuming that the minimizers of the 0-1 risk and multiclass logistic risk are the same.
An Explanation of In-context Learning as Implicit Bayesian Inference
Instead of measuring only the error at the k-th token, we average the prediction error on the 2nd to k-th tokens. However, we leave bridging the mismatch between training examples, which are consistently length k, and test examples, which have random length, to future work.
An Explanation of In-context Learning as Implicit Bayesian Inference
We generate the GINC dataset and show that Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter and Schmidhuber, 1997) trained on GINC exhibit in-context learning. In the theory, we assumed that the pretrained LM fits the pretraining distribution exactly. Here, we pretrain LMs to approximate the pretraining distribution, showing that the in-context learning properties of the pretraining distribution transfer to the LM.
An Explanation of In-context Learning as Implicit Bayesian Inference
GINC dataset. We construct the GINC dataset according to our theory (see Appendix F.1). For pretraining, we define a uniform mixture of HMMs over a family Θ of 5 concepts to generate 1000 pretraining documents with ∼10 million tokens total. For prompting, we generate prompts with 0 to 64 training examples and example lengths k ∈ {3, 5, 8, 10} (2500 prompts for each setting). The target token y test is taken to be the most likely output arg max y p prompt (y|x test ) instead of sampling so that the intrinsic error is 0.
An Explanation of In-context Learning as Implicit Bayesian Inference
Main result. We train GPT-2-based Transformers (Radford et al., 2019) and LSTMs on three versions of the GINC dataset with vocabulary sizes 50, 100, and 150, then evaluate the in-context accuracy (see Appendix F.2, F.3). We average all results over 5 pretraining runs. Figure 3 shows that for both Transformer and LSTMs, in-context accuracy improves as the number of prompt examples n and the example length k increase, verifying our theory.
An Explanation of In-context Learning as Implicit Bayesian Inference
Ablations on the latent concept structure. We ablate the role of the mixture-of-concepts structure in GINC. In Figure 4 (left), we pretrain a 4 layer Transformer on data with only one concept (removing the prior) from Θ, resulting in flat in-context learning curves. Figure 4 (middle) shows that pretraining on random pretraining data, which contains all possible token transitions, in-context learning also fails. Therefore, the mixture-of-concepts structure is important and simply seeing diverse token transitions does not enable in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
Extrapolation to unseen concepts. Full generative control of GINC allows for experimentation with latent variables in the pretraining distribution. For example, in large-scale datasets, it is difficult to test whether a concept or task is in the pretraining data. We test this in GINC by testing the in-context accuracy of a 4 layer Transformer on prompts generated from 5 random concepts that are not in the pretraining family of concepts. Figure 4 (right) shows that in-context learning also fails for these novel concepts.
An Explanation of In-context Learning as Implicit Bayesian Inference
Effect of model size and architecture. Figure 5 shows that increasing the size of the Transformer (4, 12, 16 layers) steadily increases the in-context accuracy, corroborating the results of Brown et al.
An Explanation of In-context Learning as Implicit Bayesian Inference
(2020). Table 6 shows that even though larger Transformers may have the same pretraining loss (e.g., 12 and 16 layer Transformers both get 1.33 validation loss for vocab size 50), the in-context accuracy still improves (81% to 85% from 12 to 16 layers), suggesting that larger models can improve in-context learning beyond improving pretraining perplexity. This may be related to phenomena from overparameterization and overtraining (Power et al., 2021, Zhang et al., 2017. Finally, the model architecture also plays a role -LSTMs consistently outperform Transformers on GINC despite having fewer parameters, perhaps due to the similarity between HMMs and LSTMs. We leave analysis of the effect of model scaling and model architecture as open questions.
An Explanation of In-context Learning as Implicit Bayesian Inference
Sensitivity to example ordering. In Figure  Zero-shot is sometimes better than few-shot. In some settings in GINC, we find that zero-shot performance can be better than few-shot performance. This mirrors GPT-3 on some datasets (e.g., LAMBADA, HellaSwag, PhysicalQA, RACE-m, CoQA/SAT analogies for smaller models (Brown  Figure 6: In-context accuracies (95% intervals) on GINC with vocab sizes (50, 100, 150) for Transformers and LSTMs. Accuracy improves with scale even though the pretraining loss may be the same.  Zero-shot performance can be higher than one/few-shot performance in some settings in GINC, mirroring the behavior of GPT-3 on some datasets such as LAMBADA (Brown et al., 2020). The few-shot setting introduces the distracting prompt structure, which can initially lower accuracy. et al., 2020)). This occurs especially when the transition probabilities in GINC are lower entropy (controlled via a temperature parameter). For this experiment, we consider GINC with transition matrix temperature parameter 0.01 (instead of 0.1), 12 concepts, and vocabulary size 100. Figure 7 (right) shows that here, few-shot accuracy is initially worse than zero-shot accuracy, but can recover with more examples. We hypothesize that the distracting prompt structure initially decreases the accuracy in this setting.
An Explanation of In-context Learning as Implicit Bayesian Inference
Learning via Bayesian inference and extrapolation. The canonical Bernstein-von Mises theorem (van der Vaart, 1998) does not apply for in-context learning since the prompt examples are not independent under the pretraining distribution. Gunst and Shcherbakova (2008) show a Bernsteinvon Mises-type result for observations from an HMM, but do not handle observations from a dif-ferent distribution. Future directions include more precise asymptotic results about the posterior distribution and results under misspecification/extrapolation (Kleijn and van der Vaart, 2012). A possible avenue for extrapolation to some types of unseen concepts is to factorize the latent concept into semantics and syntax. While the pretraining data may contain only some semantics-syntax pairs, the language model could generalize to unseen pairs if it learns generalizable syntactical operations such as copying or reordering.
An Explanation of In-context Learning as Implicit Bayesian Inference
Topic models and HMMs. Topic models such as LDA (Blei et al., 2003) also have document-level latent variables, but learning is typically relies on algorithms such as EM (Dempster et al., 1977), variational inference (Jordan et al., 1999), or MCMC (Hastings, 1970, Metropolis et al., 1953. We focus on learning as a natural result of Bayesian inference without an explicit inference algorithm. Wei et al. (2021a) also use an HMM model in their pretraining analysis. However, they analyze how pre-trained representations learned with masked LMs (Clark et al., 2020, Devlin et al., 2019, Lewis et al., 2020, Liu et al., 2019 can improve optimization-based downstream learning (Lester et al., 2021, Li andLiang, 2021) rather than in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
Bridging the mismatch between pretraining and prompting. Prior works support our theoretical intuitions that reducing the prompt distribution mismatch would improve in-context learning. Finetuning LMs on text with a prompting format improves its zero-shot performance (Sanh et al., 2021, Wei et al., 2021b  Meta-learning. Meta-learning methods can also train a sequence model to learn from examples (Ravi and Larochelle, 2017). However, meta-learning models are trained to learn, while incontext learning emerges from LM pretraining.
An Explanation of In-context Learning as Implicit Bayesian Inference
Studying large-scale phenomena at a small scale. We can study in-context learning, a large scale phenomenon, at a small scale in GINC because the complexity of the pretraining distribution (HMM hidden state size, number of latent concepts) is small, such that the data and models are relatively larger. Since GINC is synthetic, we can also control the latent data properties (e.g., unseen concepts) to make predictions about large LMs while working at a small scale.
An Explanation of In-context Learning as Implicit Bayesian Inference
We cast in-context learning as implicit Bayesian inference, where the pretrained LM implicitly infers a concept when making a prediction. We show that in-context learning occurs when the pretraining distribution is a mixture of HMMs. Our work provides a first step towards understanding in-context learning, which we hope will provide insight for improving pretraining and prompting.
An Explanation of In-context Learning as Implicit Bayesian Inference
Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 37 (6) 
An Explanation of In-context Learning as Implicit Bayesian Inference
Prompt distribution details. For in-context learning, we sample a prompt from a new distribution p prompt , which consists of n independent training examples and 1 test example. We first sample n hidden segments H of length k by sampling the first element h start = H[1] from a prompt start distribution p prompt . Then, we sample the rest of the segment H seg = H[2 : k] from the hidden transition distribution of the pretraining distribution p corresponding to a particular concept θ * :
An Explanation of In-context Learning as Implicit Bayesian Inference
To end each example (except the test example), we sample n delimiters h delim ∈ D from p delim prompt : 
An Explanation of In-context Learning as Implicit Bayesian Inference
The "input" for each example is
An Explanation of In-context Learning as Implicit Bayesian Inference
. Taking S to be the sequence of training examples (without the test example), the resulting prompt sequence is
An Explanation of In-context Learning as Implicit Bayesian Inference
is sampled via the same process but with k − 1 elements.
An Explanation of In-context Learning as Implicit Bayesian Inference
The following propositions, which lower bound the probability of a delimiter token and probability of an example under θ * , are direct corollaries of the assumptions. Proof. By Assumption 2,
An Explanation of In-context Learning as Implicit Bayesian Inference
Similarly,
An Explanation of In-context Learning as Implicit Bayesian Inference
Proposition 2. The probability of an example is lower bounded for θ * : there is some c 7 > 0 such that p(O i |h start i , h j,l , θ * ) > c 7 for all i and future hidden states h j,l , for any l and j > i.
An Explanation of In-context Learning as Implicit Bayesian Inference
Proof. By Assumption 5, we have
An Explanation of In-context Learning as Implicit Bayesian Inference
for some H i . We have
An Explanation of In-context Learning as Implicit Bayesian Inference
which lower bounds the terms in the numerator by c 5 (marginalizing over previous hidden states), and upper bounding the denominator by 1. Setting c 7 = (c 6 ) k c 2 5 finishes the proof.
An Explanation of In-context Learning as Implicit Bayesian Inference
Under Assumption 3, we show that the in-context predictor f n (x test ) = arg max y p(y|S n , x test ) converges when abstracting away the Bayesian inference component (the selection of θ * from Θ) of the in-context predictor. We will complete the argument for the convergence of the in-context predictor in the proof of Theorem 1.
An Explanation of In-context Learning as Implicit Bayesian Inference
Lemma 1. Suppose the prompt S n and the test input x test are given. Under Assumption 3, we show that the argmax of the averaged predictive distribution conditioned on θ * and a prompt S n is the same as the argmax of the prompt predictive distribution:
An Explanation of In-context Learning as Implicit Bayesian Inference
Proof. First, we note by definition that
An Explanation of In-context Learning as Implicit Bayesian Inference
Expanding the last term, we have
An Explanation of In-context Learning as Implicit Bayesian Inference
which is proportional to a constant in x test . On the other hand, analyzing one term inside the LHS of the lemma statement, we have
An Explanation of In-context Learning as Implicit Bayesian Inference
which is proportional to a constant in x test and S n . The quantities differ in the last term, which we expand below and put in matrix form. Let T ∈ R |H|×|D| be the matrix that represents the transition probabilities starting from a delimiter state: p(h start test |h delim ) for h start test ∈ H and h delim ∈ D. As a result,
An Explanation of In-context Learning as Implicit Bayesian Inference
where h delim n is the delimiter hidden state before h start test .
An Explanation of In-context Learning as Implicit Bayesian Inference
Let W ∈ R |Y|×|H| be the matrix that represents the probabilities p(y|x test , h start test , θ * )p(x test |h start test , θ * ) for all the possible y ∈ Y and h start test ∈ H. Overall, we can write
An Explanation of In-context Learning as Implicit Bayesian Inference
where u ∈ R |H| is the vector of probabilities that corresponds to the prompt start distribution p prompt . Bounding the difference between the two predictive distributions,
An Explanation of In-context Learning as Implicit Bayesian Inference
Using Assumption 3, we can further bound this by ∆/2:
An Explanation of In-context Learning as Implicit Bayesian Inference
Since the probability of any output does not change by more than ∆/2 and the margin between the most likely label and the second most likely label is ∆, the argmax's are the same, showing the result.
An Explanation of In-context Learning as Implicit Bayesian Inference
Proof. We analyze the most likely prediction over the pretraining distribution conditioned on the prompt arg max y p(y|S n , x test ).
An Explanation of In-context Learning as Implicit Bayesian Inference
Defining the following quantity,
An Explanation of In-context Learning as Implicit Bayesian Inference
we will show that under distinguishability for all θ = θ * , r n (θ) converges to a negative constant such that
An Explanation of In-context Learning as Implicit Bayesian Inference
for θ = θ * , whereas this ratio is always 1 for θ = θ * . This will then "select" the desired prompt concept through marginalization. Supposing that Equation 53 holds, we show that the theorem statement holds. Let
An Explanation of In-context Learning as Implicit Bayesian Inference
and let < (∆/2 − ∆ )p(θ * ). Then for n large enough (due to Equation 53),
An Explanation of In-context Learning as Implicit Bayesian Inference
where θ (y) ≤ /2 for all y ∈ Y. By Lemma 1, the argmax of the first term of Equation 57 is the same as arg max y p prompt (y|x test ), where the margin between the most likely label and the second most likely is at least ∆/2 − ∆ .
An Explanation of In-context Learning as Implicit Bayesian Inference
Since
An Explanation of In-context Learning as Implicit Bayesian Inference
for all y ∈ Y, the argmax of Equation 57 is also the same as arg max p prompt (y|x test ).
An Explanation of In-context Learning as Implicit Bayesian Inference
Now it remains to show that r n (θ) converges to a negative constant for θ = θ * . Let O ex i = [o delim i−1 , O i ] be the i-th observation segment and the previous delimiter together for i > 1 and define O ex 1 = O 1 . Expanding the numerator of the ratio in r n (θ), we have
An Explanation of In-context Learning as Implicit Bayesian Inference
Note that in the last line, the inner sum is over the set of delimiter states D by using the assumption that observing a delimiter o delim implies that the corresponding hidden state h delim must be in D.
An Explanation of In-context Learning as Implicit Bayesian Inference
We also see that h delim n p(h delim n |O ex 1:n , θ) = 1. We restrict our attention to θ where p(S n , x test |θ) > 0, since otherwise θ does not affect the prediction. Expanding r n (θ), we have the following upper bound:
An Explanation of In-context Learning as Implicit Bayesian Inference
In the above steps, we used both Propositions 1 and 2 in the terms involving c 2 , c 1 (bounding the probability of h delim hidden states) and c 7 (bounding the probability of x test ). Note that in the second line, the sum can must be over the set of delimiter states D by using the assumption that observing a delimiter o delim implies that the corresponding hidden state h delim must be in D.
An Explanation of In-context Learning as Implicit Bayesian Inference
Focusing on the numerator of the ratio term and summing over the start hidden state for the i-th example,
An Explanation of In-context Learning as Implicit Bayesian Inference
where the last step applies Bayes' rule. We can lower and upper bound the following quantity for any θ using Assumption 2:
An Explanation of In-context Learning as Implicit Bayesian Inference
This implies that
An Explanation of In-context Learning as Implicit Bayesian Inference
Plugging in these bounds, we have
An Explanation of In-context Learning as Implicit Bayesian Inference
where we set
An Explanation of In-context Learning as Implicit Bayesian Inference
Next, we convert the expectation in the bound into a KL divergence. We have
An Explanation of In-context Learning as Implicit Bayesian Inference
= KL(p prompt p(·|θ * )) − KL(p prompt p(·|θ)).
An Explanation of In-context Learning as Implicit Bayesian Inference
We will upper bound the first KL term:
An Explanation of In-context Learning as Implicit Bayesian Inference
Expanding the numerator and denominator of the ratio inside, we have
An Explanation of In-context Learning as Implicit Bayesian Inference
which differ in only the hidden start distribution. Using Assumption 5, we have that p(h|θ * ) ≥ c 8 for any h ∈ H, which implies that
An Explanation of In-context Learning as Implicit Bayesian Inference
Finally, this implies that the KL term is bounded as KL(p prompt p(·|θ * )) ≤ − log(c 8 ).
An Explanation of In-context Learning as Implicit Bayesian Inference
(88) This term is non-negative since c 8 ≤ 1.
An Explanation of In-context Learning as Implicit Bayesian Inference
Aiming to decompose the second KL term into a sum over the k tokens, we write p j
An Explanation of In-context Learning as Implicit Bayesian Inference
Then we have that
An Explanation of In-context Learning as Implicit Bayesian Inference
The second term (set θ start = log( 1 c8 )) is an error term that depends on how different the starting prompt distribution p prompt (which is part of p prompt ) is to the pretraining distribution. The third term is an error term that comes from the delimiter transitions. The bound is negative when the sum of KL terms is larger in magnitude than the error terms. Note that as k becomes larger, the number of observations of θ * "overpowers" the distracting transitions in the prompt distribution. This condition is equivalent to the disinguishability condition (Condition 1). By assumption, for θ = θ * the Condition 1 holds, and thus lim n→∞ p(S n , x test |θ) p(S n , x test |θ * ) = lim n→∞ exp(n · r n (θ)) = 0
An Explanation of In-context Learning as Implicit Bayesian Inference
since r n (θ) has a negative, constant limit. Note that exp(n · r n (θ * )) = 1 for θ * .
An Explanation of In-context Learning as Implicit Bayesian Inference
When Condition 1 is unsatisfied, Equation 14), gives an upper bound on the sum of KL divergences for the next token distributions given different-length histories. In contrast, the in-context task only measures the accuracy of the last (k-th) token. The main challenge is to relate the different-length histories to each other to give a more precise bound for the error on the in-context task (last token). Before addressing this challenge, we give the following lemma, which leverages the result of Steinwart (2007), Ávila Pires and Szepesvári (2016) to relate a bound on the KL divergence to 0-1 loss.
An Explanation of In-context Learning as Implicit Bayesian Inference
Lemma 2. Let the set of θ which does not satisfy Condition 1 to be B. Assume that KL(p prompt (y test |x test ) p(y test |x test , θ) is bounded above for all θ and that θ * minimizes the multiclass logistic
An Explanation of In-context Learning as Implicit Bayesian Inference
is a calibration function for the multiclass logistic loss for δ ∈ [0, 1].
An Explanation of In-context Learning as Implicit Bayesian Inference
Proof. First, we note that we can study the 0-1 risk of the limiting predictor: 
An Explanation of In-context Learning as Implicit Bayesian Inference
where in the last step we use that since the output space of f n is discrete and the probabilities that the in-context predictor takes an argmax over converges, then for N large enough, f N (x test ) = lim n→∞ f n (x test ). Note that for every input x test , the limiting in-context learning predictor outputs the argmax of a predictive distribution which can be a mixture of predictive distributions over B:
An Explanation of In-context Learning as Implicit Bayesian Inference
for some distribution q over B. The KL divergence between this mixture and the prompt concept is bounded by the KL divergence of any one θ ∈ B, due to the convexity of KL:
An Explanation of In-context Learning as Implicit Bayesian Inference
where we can exchange the order of expectations since the KL is bounded (dominated convergence).
An Explanation of In-context Learning as Implicit Bayesian Inference
From the KL bound KL(p prompt (y test |x test ) p(y test |x test , θ), we thus have
An Explanation of In-context Learning as Implicit Bayesian Inference
where L CE (θ) = −E xtest∼pprompt [p prompt (y test |x test ) log p(y test |x test , θ)] is the multiclass logistic risk, and L CE (θ * ) is the optimal risk over θ ∈ Θ by assumption. Applying Theorem 2.2 and 5.11 of Ávila Pires and Szepesvári (2016), g is a calibration function for the multiclass logistic loss, and allows us to convert the surrogate risk bound to a bound on the 0-1 loss, giving the result. Note that we have zero approximation error here, since θ * ∈ Θ.
An Explanation of In-context Learning as Implicit Bayesian Inference
Note that g −1 is roughly linear in for smaller than 0.7, where the bound is non-vacuous.
An Explanation of In-context Learning as Implicit Bayesian Inference
Proof. By the continuity assumption, we have for any θ in B that
An Explanation of In-context Learning as Implicit Bayesian Inference
We use this to bound the last KL term by plugging it in below:
An Explanation of In-context Learning as Implicit Bayesian Inference
Rearranging and noting that KL k (θ * θ) = E xtest∼pprompt [KL(p prompt (y test |x test ) p(y test |x test , θ))], we have
An Explanation of In-context Learning as Implicit Bayesian Inference
Plugging into Lemma 2 gives the result.
An Explanation of In-context Learning as Implicit Bayesian Inference
Note that Condition 1 ensures that the sum of KL divergences between positions within a k-length input is bounded. This means that we have a bound over not only the last-position KL divergence, but also for all the intermediate tokens. Intuitively, the random length test example allows the incontext predictor to "take credit" for fitting the intermediate tokens. The proof is immediate given the KL bound and Lemma 2, given that the length of x test is uniformly random between 2 to k. Proof. Let the set of θ that does not satisfy Condition 1 to be B. We have for any θ in B that
An Explanation of In-context Learning as Implicit Bayesian Inference
by Theorem 1 and Condition 1. Plugging this into Lemma 2 gives the result.
An Explanation of In-context Learning as Implicit Bayesian Inference
Pretraining distribution. We consider a pretraining distribution from a mixture of HMMs with an interpretable hidden state structure and emission distribution. The HMM hidden state h t = [s t , v t ] at time t is composed of an entity v t ∈ {1, . . . , |V|} (e.g., Einstein) and a property s t ∈ {1, . . . , |S|} (e.g., nationality, first name, last name, other grammatical tokens). We model the entities and properties as independent Markov chains (i.e., a factorial HMM (Ghahramani and Jordan, 1997)), while the emissions depend on both. In pretraining documents, we expect that the entities (e.g., Einstein) change slowly over time while and the properties of the entity (e.g., their nationality) change quickly with some pattern to generate natural sentences. We implement this by ensuring that the probability of transitioning to the same entity index in the next step is at least 0.9. The emission distribution depends on a memory matrix M with |V| rows and |S| columns (Figure 9). At step t, we use the entity v t and property s t to index into the memory matrix. In particular, the observed tokens are deterministic with p(o
An Explanation of In-context Learning as Implicit Bayesian Inference
This construction satisfies the structure on delimiter states (Assumption 1). We ensure that all the transitions have nonzero probability and use a uniform prior over concepts, satisfying Assumptions 2 and 5. Figure 9: The GINC dataset generates sequences from a mixture of HMMs. The HMM hidden states consist of entities (v) and properties (s), which index into a memory matrix to produce the observed token. The entity and property sequences are sampled from independent Markov chains. The concept parameter θ is the transition matrix for properties, which defines relations between properties. In this example, the sequence of properties [2,3,5,4] relates names to nationalities, defining the in-context task. The blue color represents hidden states/observations sampled from the prompt distribution, and the purple color represents hidden states/observations sampled from the pretraining distribution.
An Explanation of In-context Learning as Implicit Bayesian Inference
Concept parameter. The concept parameter is the property transition matrix, while the entity transition matrix is fixed for all concepts. The prompt start distribution and the concept together determine the in-context task. We define a uniform mixture of HMMs over a family Θ of 5 concepts to generate 1000 documents with ∼10 million tokens total.
An Explanation of In-context Learning as Implicit Bayesian Inference
Vocabulary. The GINC dataset is generated from a mixture of HMMs. These HMMs output tokens from a vocabulary of size in {50, 100, 150}. The vocabulary contains a special delimiter token (backslash -see Figure 8, designated to be index 1. The vocabulary is generated as combinations of letters starting from a to z, then aa to az, and so on. All sequences are tokenized by splitting on whitespaces.
An Explanation of In-context Learning as Implicit Bayesian Inference
Memory matrix. The shared memory matrix has 10 entities and 10 properties, totaling 100 entries (corresponding to 100 hidden states). The first column of the memory matrix is fixed to be the delimiter token, while each remaining entry of the shared memory matrix is populated with a token sampled uniformly from the vocabulary.
An Explanation of In-context Learning as Implicit Bayesian Inference
Transition matrix for properties. We generate 5 property transition matrices, one for each component of the HMM mixture. We generate each transition matrix via a convex combination of 100 random permutation matrices. The weights of the convex combination are randomly generated as
An Explanation of In-context Learning as Implicit Bayesian Inference
where u ∈ R 100 has uniform random entries in [0, 1] and t is a temperature parameter, set to 0.1. Transition matrix for entities. The entity transition matrix is shared between all the HMMs that consistute the mixture. The entity transition matrix is generated in the same way as the property transition matrices, except with one additional step. Letting T be a transition matrix sampled in the same way as a property transition matrix, In pretraining documents, we expect that the entities (e.g., Einstein) change slowly over time while and the properties of the entity (e.g., their occupation) change quickly with some pattern to generate natural sentences. We implement this by ensuring that the probability of transitioning to the same entity index in the next step is at least 0.9. The final entity transition matrix is then 0.1T + 0.9I where I is the identity matrix. Although we add the diagonal component for added realism, we also consider not adding this component. Figure 10 shows in-context learning curves for a small (4 layer) Transformer trained on data that does not add the diagonal component (we check this for vocabulary sizes 50, 100, and 150). In-context learning still works in this case, although not as well for the 50 vocab size case.
An Explanation of In-context Learning as Implicit Bayesian Inference
Start distribution. The starting distribution for the hidden states in all HMMs in the mixture are close to uniform. We generate the start distribution as softmax((u − 0.5)/t) for random vector u with entries uniformly from [0, 1] and temperature t = 10. In the pretraining documents, we only sample from the start distribution in the beginning of the document.
An Explanation of In-context Learning as Implicit Bayesian Inference
Prompt distribution. We generate prompts with 0 to 64 training examples and example lengths k ∈ {3, 5, 8, 10} (2500 prompts for each setting). The target token y test is taken to be the most likely output arg max y p prompt (y|x test ) instead of sampling so that the intrinsic error is 0.
An Explanation of In-context Learning as Implicit Bayesian Inference
To generate the prompts, we first sample a concept θ uniformly at random from Θ (well-specification, Assumption 4), then use it to generate all the prompt examples. The prompt start distribution is chosen to be uniform over entities but with a fixed starting property that is chosen randomly for each prompt, for consistency in the task. This may not satisfy Assumption 3, but we found this to still work empirically and is simpler. Given the starting property, we sample k tokens from the HMM defined by the concept θ. Finally, we append the delimiter token for the example. We repeat this process for each example in the prompt, concatenating all examples. The label is generated as
An Explanation of In-context Learning as Implicit Bayesian Inference
under the prompt concept θ * . This differs from the theory, which samples y test instead of taking it to be the most likely token. However, there can be a large amount of intrinsic error that sampling introduces. We define the label this way in the simulations to remove the intrinsic error from sampling.
An Explanation of In-context Learning as Implicit Bayesian Inference
Example of prompt generation. In the example in Figure 8 (right), the starting property is fixed to be 5 (for example). The first token (l) is generated by sampling a random entity index (3), and indexing into the memory matrix returns l. Running the hidden state chain of the HMM forward gives the next pair of property and entity. Since the entity Markov chain changes slowly, the entity is still 3 in the next step -however, the property has changed to 4, and indexing into the memory matrix outputs the next token (aw). Following this same process to generate the third token (the output for the first example), we finish generating one example. To end the example, we append a delimiter (backslash). We repeat this example generation process for all the examples, except for the test example at the end, where we do not generate the last token. We condition the HMM on the generated prompt to compute the posterior distribution over the next token p prompt (y|x test ). We take the argmax of this distribution to be the ground truth label. 
An Explanation of In-context Learning as Implicit Bayesian Inference
Our Transformer models are based on the GPT-2 architectures with 4, 12, and 16 layers respectively, with 12 attention heads, 768 dimensional embeddings, residual/embedding/attention dropout set to 0.1, and a context window of 1024. Other than the number of layers, the other parameters are the default settings from the HuggingFace library (Wolf et al., 2019). We train for 5 epochs using the AdamW optimizer (Kingma andBa, 2015, Loshchilov andHutter, 2019) with a batch size of 8 and a linear learning rate schedule (with 1000 step warmup) up to a learning rate of 8e-4 for the 4 layer and 12 layer model, while for the 16 layer model we start with a constant learning rate of 8e-4 and reduce by a factor of 0.25 whenever the best validation loss does not improve. We tried both learning rate strategies for all models and take the most consistent. We tuned these models so that the training loss curves between seeds have smaller variability between the runs in terms of the curve shape and when the loss decreases -we found that this is an important indication of stable results. The models took 50 minutes, 2 hours, 3 hours to train respectively. The hardware was mainly Titan Xp GPUs, trained and evaluated using 16-bit precision. All the results are reported with 5 pretraining runs (5 different seeds).
An Explanation of In-context Learning as Implicit Bayesian Inference
We train an LSTM language model with embedding size 768, hidden layer size 768, and 6 layers. We use dropout 0.2 and weight decay 1e-5. The optimizer is AdamW starting with a learning rate of 1e-3, then reducing by a factor of 0.25 whenever the best validation loss does not go down. We train for a total of 10 epochs, with gradient clipping at norm 1.0. We use a batch size of 8 and backpropagate through time for 1024 steps (each pretraining data segment is also 1024 tokens). Each model takes roughly 2 hours to train on Titan Xp GPUs.
An Explanation of In-context Learning as Implicit Bayesian Inference
To do well on the in-context learning task, the model must both infer the prompt concept and the last HMM hidden state. In general, increasing the number of observable symbols makes the incontext task easier by making the inference of the HMM hidden state easier. With more symbols, each hidden state is more likely to output a different symbol, making the inference problem easier. This improvement comes despite the number of output classes in the problem (same as the vocabulary size) increasing. Figures 11, 12, 13, 14 show in-context learning curves for vocabulary sizes 50, 100, and 150, keeping other hyperparmeters of the dataset the same.   
An Explanation of In-context Learning as Implicit Bayesian Inference
We conduct an additional experiment which shows that longer examples improve in-context learning in GPT-3 on the LAMBADA (Paperno et al., 2016) completion task.
An Explanation of In-context Learning as Implicit Bayesian Inference
In this experiment, we define a short version of the LAMBADA test dataset (LAMBADA test-short) which contains only test examples with up to 200-300 characters in length. We also define two "training" datasets from which to sample examples for the in-context prompts from. The short training dataset (LAMBADA train-short) contains examples from the training set that are 200-300 characters in length, which matches the distribution of test-short. The long training dataset (LAMBADA train-long) contains training examples that are 500-600 characters long. We cut the number of examples in the larger of the two training datasets so that the two training datasets are equally sized (47 examples). For each test example, we sample 5 random training examples (5-shot learning). We also consider equalizing the total length of the prompts in two ways. First, we consider duplicating the 5 short examples (if the examples are [1,2,3,4,5], duplicating refers to [1,2,3,4,5,1,2,3,4,5]). This allows for equalizing the total length without increasing the number of examples. As a skyline comparison, we also consider sampling 10 independent short examples, which contains more input-output pairs for the task.
An Explanation of In-context Learning as Implicit Bayesian Inference
Result. Table 1 shows that when evaluating only on LAMBADA test-short, 5-shot in-context learning using LAMBADA train-long improves the test accuracy by almost 1% compared to LAMBADA train-short, despite the long/short distribution mismatch between train and test. This supports intuitions from our theory. In comparison, simply increasing the total prompt length by duplicating the short examples does not improve the accuracy. Intuitively, the longer examples have additional information that is not directly related to mapping between the input and output, but can be leveraged to improve incontext learning by helping the model infer the latent concept. Using 5 long examples (as opposed to 5 short examples) closes about 56% of the gap between using 5 short examples and 10 independent short examples despite not adding additional examples or task-related information.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Large language models (LMs) have shown impressive performance on downstream tasks by simply conditioning on a few input-label pairs (demonstrations); this type of inference has been referred to as in-context learning (Brown et al., 2020). Despite incontext learning consistently outperforming zeroshot inference on a wide range of tasks (Zhao et al., 2021;Liu et al., 2021), there is little understanding of how it works and which aspects of the demonstrations contribute to end task performance.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
In this paper, we show that ground truth demonstrations are in fact not required for effective incontext learning (Section 4). Specifically, replacing the labels in demonstrations with random labels barely hurts performance in a range of classification and multi-choice tasks (Figure 1). The result Figure 1: Results in classification (top) and multi-choice tasks (bottom), using three LMs with varying size. Reported on six datasets on which GPT-3 is evaluated; the channel method is used. See Section 4 for the full results. In-context learning performance drops only marginally when labels in the demonstrations are replaced by random labels.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
is consistent over 12 different models including the GPT-3 family (Radford et al., 2019;Min et al., 2021b;Wang and Komatsuzaki, 2021;Artetxe et al., 2021;Brown et al., 2020). This strongly suggests, counter-intuitively, that the model does not rely on the input-label mapping in the demonstrations to perform the task.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Further analysis investigates which parts of demonstrations actually do contribute to the performance. We identify possible aspects of demonstrations (e.g., the label space and the distribution of the input text) and evaluate a series of variants of the demonstrations to quantify the impact of each (Section 5). We find that: (1) the label space and the distribution of the input text specified by the demonstrations are both key to in-context learning (regardless of whether the labels are correct for individual inputs); (2) specifying the overall format is also crucial, e.g., when the label space is unknown, using random English words as labels is significantly better than using no labels; and
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
(3) meta-training with an in-context learning objective (Min et al., 2021b) magnifies these effects-the models almost exclusively exploit simpler aspects of the demonstrations like the format rather than the input-label mapping.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
In summary, our analysis provides a new way of understanding the role of the demonstrations in in-context learning. We empirically show that the model (1) counter-intuitively does not rely on the ground truth input-label mapping provided in the demonstrations as much as we thought (Section 4), and (2) nonetheless still benefits from knowing the label space and the distribution of inputs specified by the demonstrations (Section 5). We also include a discussion of broader implications, e.g., what we can say about the model learning at test time, and avenues for future work (Section 6).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We describe the experimental setup used in our analysis (Section 4 and 5).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Models. We experiment with 12 models in total. We include 6 language models ( 
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
No Demos Demos w/ gold labels Demos w/ random labels Figure 3: Results when using no-demonstrations, demonstrations with gold labels, and demonstrations with random labels in classification (top) and multi-choice tasks (bottom). The first eight models are evaluated on 16 classification and 10 multi-choice datasets, and the last four models are evaluated on 3 classification and 3 multi-choice datasets. See Figure 11 for numbers comparable across all models. Model performance with random labels is very close to performance with gold labels (more discussion in Section 4.1).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
largest dense LM (GPT-3) and the largest publicly released dense LM (fairseq 13B) at the time of conducting experiments. We also include MetaICL, which is initialized from GPT-2 Large and then meta-trained on a collection of supervised datasets with an in-context learning objective, and ensure that our evaluation datasets do not overlap with those used at meta-training time.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Evaluation Data. We evaluate on 26 datasets, including sentiment analysis, paraphrase detection, natural language inference, hate speech detection, question answering, and sentence completion (full list and references provided in Appendix A). 1 All datasets are classification and multi-choice tasks. We use these datasets because they (1) are true low-resource datasets with less than 10K training examples, (2) include well-studied benchmarks from GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019a), and (3) cover diverse domains including science, social media, finance, and more.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Other Details. We use k = 16 examples as demonstrations by default for all experiments in the paper, unless otherwise specified. Examples are sampled at uniform from the training data. We choose a set of k training examples using 5 different random seeds and run experiments 5 times. For fairseq 13B and GPT-3, due to limited resources, we experiment with a subset of 6 datasets 2 and 3 random seeds. We report Macro-F1 3 for classification tasks and Accuracy for multichoice tasks. We compute per-dataset average over seeds, and then report macro-average over datasets. We use the minimal templates in forming an input sequence from an example. We refer to Appendix B for more details. All experiments are reproducible from github.com/Alrope123/ rethinking-demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
To see the impact of correctly-paired inputs and labels in the demonstrations-which we call the ground truth input-label mapping-we compare the following three methods. 4
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
No demonstrations is a typical zero-shot method that does not use any labeled data. A prediction is made via argmax y∈C P (y|x), where x is the test input and C is a small discrete set of possible labels.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Demonstrations w/ gold labels are used in a typical in-context learning method with k labeled examples (x 1 , y 1 )...(x k , y k ). A concatenation of k input-label pairs is used to make a prediction via argmax y∈C P (y|x 1 , y 1 ...x k , y k , x).  Demonstrations w/ random labels are formed with random labels, instead of gold labels from the labeled data.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
is paired withỹ i that is randomly sampled at uniform from C. A concatenation of (x 1 ,ỹ 1 )...(x k ,ỹ k ) is then used to make a prediction via argmax y∈C P (y|x 1 ,ỹ 1 ...x k ,ỹ k , x).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Results are reported in Figure 3. First, using the demonstrations with gold labels significantly improves the performance over no demonstrations, 5 as it has been consistently found in much of prior work (Brown et al., 2020;Zhao et al., 2021;Liu et al., 2021). We then find that replacing gold labels with random labels only marginally hurts performance. The trend is consistent over nearly all models: models see performance drop in the range of 0-5% absolute. There is less impact in replacing labels in multi-choice tasks (1.7% on average) than in classification tasks (2.6% absolute).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
This result indicates that the ground truth inputlabel pairs are not necessary to achieve performance gains. This is counter-intuitive, given that correctly paired training data is critical in typical supervised training-it informs the model of the expected input-label correspondence required to perform the downstream task. Nonetheless, the models do achieve non-trivial performance on the downstream tasks. This strongly suggests that the models are capable of recovering the expected inputlabel correspondence for the task; however, it is not directly from the pairings in the demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
It is also worth noting that there is particularly little performance drop in MetaICL: 0.1-0.9% absolute. This suggests that meta-training with an explicit in-context learning objective actually encourages the model to essentially ignore the input-5 There are some exceptions, e.g., in the classification tasks, Direct GPT-2, Direct GPT-J and Direct fairseq 6.7B models are not significantly better than random guessing on many datasets; Channel fairseq 13B has significantly better nodemonstrations performance compared to demonstrations with gold labels. We thus discuss the results from these models less significantly for the rest of analysis. 
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Demos w/ gold Demos w/ random label mapping and exploit other components of the demonstrations (more discussion in Section 5.4). In Appendix C.2, we provide additional results showing that (1) selecting random labels from a true distribution of labels (instead of a uniform distribution) reduces the gap even further, and (2) the trends may depend on the dataset, although the overall trend is consistent over most datasets.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
For additional ablations, we experiment with 5 classification and 4 multi-choice datasets. 6 Does the number of correct labels matter? To further examine the impact of correctness of labels in the demonstrations, we conduct an ablation study by varying the number of correct labels in the demonstrations. We evaluate "Demonstrations w/ a% correct labels" (0 ≤ a ≤ 100) which consist of k × a/100 correct pairs and k × (1 − a/100) incorrect pairs (see Algorithm 1 in Appendix B). Here, a = 100 is the same as typical in-context learning, i.e., demonstrations w/ gold labels.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Results are reported in Figure 4. Model performance is fairly insensitive to the number of correct labels in the demonstrations. In fact, always using incorrect labels significantly outperforms no- -choice)   25  30  35  40  45  50  55  60  65 Accuracy (%)
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
No demos Gold labels Random labels No demos + T Gold labels + T Random labels + T Figure 6: Results with minimal templates and manual templates. '+T' indicates that manual templates are used. Channel and Direct used for classification and multi-choice, respectively.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
demonstrations, e.g., preserving 92%, 100% and 97% of improvements from using the demonstrations with MetaICL in classification, MetaICL in multi-choice, and GPT-J in multi-choice, respectively. In contrast, GPT-J in classification sees relatively significant performance drop with more incorrect labels, e.g., nearly 10% drop in performance when always using incorrect labels. Still, always using incorrect labels is significantly better than no demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Is the result consistent with varying k? We study the impact of the number of input-label pairs (k) in the demonstrations. Results are reported in Figure 5. First, using the demonstrations significantly outperforms the no demonstrations method even with small k (k = 4), and performance drop from using gold labels to using random labels is consistently small across varying k, in the range of 0.8-1.6%. 7 Interestingly, model performance does not increase much as k increases when k ≥ 8, both with gold labels and with random labels. This is in contrast with typical supervised training where model performance rapidly increases as k increases, especially when k is small. We hypothesize that larger labeled data is beneficial mainly for supervising the input-label correspondence, and other components of the data like the example inputs, example labels and the data format are easier to recover from the small data, which is potentially a reason for minimal performance gains from larger k (more discussion in Section 5).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Is the result consistent with better templates? While we use minimal templates by default, we also explore manual templates, i.e., templates that are manually written in a dataset-specific manner, taken from prior work (details in Appendix B). Figure 6 shows that the trend-replacing gold labels with random labels barely hurting performanceholds with manual templates. It is worth noting 7 With an exception of 4.4% in classification with k = 4, likely due to a high variance with a very small value of k. Circulation revenue has increased by 5% in Finland.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
\n Positive
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
=
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Input-label mapping Panostaja did not disclose the purchase price. \n Neutral
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Paying off the national debt will be extremely painful. \n Negative
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
The acquisition will have an immediate positive impact. \n ? Figure 7: Four different aspects in the demonstrations: the input-label mapping, the distribution of the input text, the label space, and the use of input-label pairing as the format of the demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
that using manual templates does not always outperform using minimal templates.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Section 4 shows that the ground truth input-label mapping in the demonstrations has little impact to performance gains from in-context learning. This section further examines what other aspects of the demonstrations lead to good performance of incontext learning. We identify four aspects of the demonstrations (x 1 , y 1 )...(x k , y k ) that potentially provide learning signal (depicted in Figure 7).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
1. The input-label mapping, i.e., whether each input x i is paired with a correct label y i .
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
2. The distribution of the input text, i.e., the underlying distribution that x 1 ...x k are from.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
3. The label space, i.e., the space covered by y 1 ...y k . 4. The format-specifically, the use of inputlabel pairing as the format.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
As Section 4 does for the input-label mapping, we design a series of variants of the demonstrations that quantify the impact of each aspect in isolation (Section 5.1-5.3). We then additionally discuss the trend of the models meta-trained with an in-context learning objective (Section 5.4). For all experiments, models are evaluated on five classification  Figure 8: Impact of the distribution of the inputs. Evaluated in classification (top) and multi-choice (bottom). The impact of the distribution of the input text can be measured by comparing ■ and ■. The gap is substantial, with an exception in Direct MetaICL (discussion in Section 5.1). and four multi-choice datasets as in Section 4.2. See Appendix B and Table 4 for implementation details and example demonstrations, respectively.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We experiment with OOD demonstrations which include out-of-distribution (OOD) text instead of the inputs from unlabeled training data. Specifically, a set of k sentences {x i,rand } k i=1 are randomly sampled from an external corpus, and replace x 1 ...x k in the demonstrations. This variant assesses the impact of the distribution of the input text, while keeping the label space and the format of the demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Results. Figure 8 shows that using out-ofdistribution inputs instead of the inputs from the training data significantly drops the performance when Channel MetaICL, Direct GPT-J or Channel GPT-J are used, both in classification and multichoice, by 3-16% in absolute. In the case of Direct GPT-J in multi-choice, it is even significantly worse than no demonstrations. Direct MetaICL is an exception, which we think is the effect of meta-training (discussion in Section 5.4).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
This suggests that in-distribution inputs in the demonstrations substantially contribute to performance gains. This is likely because conditioning on the in-distribution text makes the task closer to language modeling, since the LM always conditioned on the in-distribution text during training.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We also experiment with demonstrations w/ random English words that use random English words as labels for all k pairs. Specifically, we sample a random subset of English words C rand where |C rand | = |C|, and randomly pairỹ i ∈ C rand with x i . This variant assesses the impact of the label space, while keeping the distribution of the input text and the format of the demonstrations.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Results. Based on Figure 9, direct models and channel models exhibit different patterns. With direct models, the performance gap between using random labels within the label space and using random English words is significant, ranging between 5-16% absolute. This indicates that conditioning on the label space significantly contributes to performance gains. This is true even for multi-choice tasks where there is no fixed set of labels-we hypothesize that multi-choice tasks still do have a particular distribution of the choices (e.g., objects like "Bolts" or "Screws" in the OpenBookQA dataset) that the model uses.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
On the other hand, removing the output space does not lead to significant drop in the channel models: there is 0-2% drop in absolute, or sometimes even an increase. We hypothesize that this is because the channel models only condition on the labels, and thus are not benefiting from knowing the label space. This is in contrast to direct models which must generate the correct labels.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Section 5.1 and 5.2 focus on variants which keep the format of the demonstrations as much as possible. This section explores variants that change the format. While there are many aspects of the format, we make minimal modifications to remove the pairings of inputs to labels. Specifically, we evaluate demonstrations with no labels where the LM is conditioned on the concatenation of x 1 ...x k , and  Figure 9: Impact of the label space. Evaluated in classification (top) and multi-choice (bottom). The impact of the label space can be measured by comparing ■ and ■. The gap is significant in the direct models but not in the channel models (discussion in Section 5.2).  Figure 10: Impact of the format, i.e., the use of the input-label pairs. Evaluated in classification (top) and multichoice (bottom). Variants of demonstrations without keeping the format (■ and ■) are overall not better than no demonstrations (■). Keeping the format is especially significant when it is possible to achieve substantial gains with the label space but without the inputs (■ vs. ■ in Direct MetaICL), or with the input distribution but without the labels (■ vs. ■ in Channel MetaICL and Channel GPT-J). More discussion in Section 5.3.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
demonstrations with labels only where the LM is conditioned on the concatenation of y 1 ...y k . These ablations provide the no-format counterparts of the 'demonstrations with random English words' and 'demonstrations with OOD inputs', respectively.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Results. Based on Figure 10, removing the format is close to or worse than no demonstrations, indicating the importance of the format. This is likely because conditioning on a sequence of inputlabel pairs triggers the model to mimic the overall format and complete the new example as expected when the test input is given. More interestingly, keeping the format plays a significant role in retaining a large portion of performance gains by only using the inputs or only using the labels. For instance, with Direct MetaICL, it is possible to retain 95% and 82% of improvements from in-context learning (demonstrations with gold labels) by simply sampling random sen-tences from a corpus and randomly pairing them with the label set (■ in Figure 10) in classification and multi-choice, respectively. Similarly, with the channel models, it is possible to retain 82%, 87%, 86% and 75% of improvements from in-context learning by simply pairing each input from the unlabeled training data with a random English word (■ in Figure 10) in MetaICL classification, GPT-J classification, MetaICL multi-choice and GPT-J multi-choice, respectively. For all of these cases, removing inputs instead of using OOD inputs, or removing labels instead of using random English words is significantly worse, indicating that keeping the format of the input-label pairs is key.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Different from other models, MetaICL is trained with an in-context learning objective, in line with recent work that uses multi-task training on a large collection of supervised datasets (called metatraining) for generalization to new tasks (Aghajanyan et al., 2021;Khashabi et al., 2020;Wei et al., 2022a;Sanh et al., 2022). We aim to better understand the role of this meta-training in relation with our findings by closely examining the result of MetaICL. In particular, we observe that the patterns we see so far are significantly more evident with MetaICL than with other models. For instance, the ground truth input-label mapping matters even less, and keeping the format of the demonstrations matters even more. There is nearly zero influence of the input-label mapping and the input distribution in Direct MetaICL, and the input-label mapping and the output space in Channel MetaICL.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Based on this observation, we hypothesize that meta-training encourages the model to exclusively exploit simpler aspects of the demonstrations and to ignore others. This is based on our intuition that (1) the input-label mapping is likely harder to exploit, (2) the format is likely easier to exploit, and (3) the space of the text that the model is trained to generate is likely easier to exploit than the space of the text that the model conditions on. 8
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
In this paper, we study the role of the demonstrations with respect to the success of in-context learning. We find that the ground truth input-label mapping in the demonstrations matters significantly less than one might think-replacing gold labels with random labels in the demonstrations only marginally lowers the performance. We then identify a series of aspects in the demonstrations and examine which aspect actually contributes to performance gains. Results reveal that (1) gains are mainly coming from independent specification of the input space and the label space, (2) the models can still retain up to 95% of performance gains by using either the inputs only or the label set only if the right format is used, and (3) meta-training with an in-context learning objective magnifies these trends. Together, our findings lead to a set of broader indications about in-context learning, as well as avenues for future work.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Does the model learn at test time? If we take a strict definition of learning: capturing the inputlabel correspondence given in the training data, then our findings suggest that LMs do not learn new tasks at test time. Our analysis shows that the model may ignore the task defined by the demonstrations and instead use prior from pretraining.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
However, learning a new task can be interpreted more broadly: it may include adapting to specific input and label distributions and the format suggested by the demonstrations, and ultimately getting to make a prediction more accurately. With this definition of learning, the model does learn the task from the demonstrations. Our experiments indicate that the model does make use of aspects of the demonstrations and achieve performance gains.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Capacity of LMs. The model performs a downstream task without relying on the input-label correspondence from the demonstrations. This suggests that the model has learned the (implicit notion of) input-label correspondence from the language modeling objective alone, e.g., associating a positive review with the word 'positive'. This is in line with Reynolds and McDonell (2021) who claim that the demonstrations are for task location and the intrinsic ability to perform the task is obtained at pretraining time. 9 On one hand, this suggests that the language modeling objective has led to great zero-shot capacity, even if it is not always evident from the naive zero-shot accuracy. On the other hand, this suggests that in-context learning may not work on a task whose input-label correspondence is not already captured in the LM. This leads to the research question of how to make progress in NLP problems that in-context learning does not solve: whether we need a better way of extracting the input-label mappings that are already stored in the LM, a better variant of the LM objective that learns a wider range of task semantics, or explicit supervision through fine-tuning on the labeled data.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Connection to instruction-following models. Prior work has found it promising to train the model that reads the natural language description of the task (called instructions) and performs a new task at inference (Mishra et al., 2021b;Efrat and Levy, 2020;Wei et al., 2022a;Sanh et al., 2022). We think the demonstrations and instructions largely have the same role to LMs, and hypothesize that our findings hold for instruction-following models: the instructions prompt the model to recover the capacity it already has, but do not supervise the model to learn novel task semantics. This has been partially verified by Webson and Pavlick (2022) who showed that the model performance does not degrade much with irrelevant or misleading instructions. We leave more analysis on instruction-following models for future work.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Significantly improved zero-shot performance. One of our key findings is that it is possible to achieve nearly k-shot performance without using any labeled data, by simply pairing each unlabeled input with a random label and using it as the demonstrations. This means our zero-shot baseline level is significantly higher than previously thought. 10 Future work can further improve the zero-shot performance with relaxed assumptions in access to the unlabeled training data.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Effect of types of tasks and datasets. This paper focuses on the tasks from established NLP benchmarks that have real natural language inputs. Synthetic tasks with more limited inputs may actually use the ground truth labels more, as observed by Rong (2021).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We report macro-level analysis by examining the average performance over multiple NLP datasets, but different datasets may behave differently. Appendix C.2 discusses this aspect, including findings that there are larger gaps between using the ground truth labels and using the random labels in some dataset-model pairs (e.g., in the most extreme case, nearly 14% absolute on the finan-cial_phrasebank dataset with GPT-J). Since the first version of our paper, Kim et al. (2022) showed that using negated labels substantially lowers the performance in classification. 11 We believe it is important to understand to what extend the model needs the ground truth labels to successfully perform in-context learning.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Extensions to generation. Our experiments are limited to classification and multi-choice tasks. We hypothesize that ground truth output may not be necessary for in-context learning in the open-set 10 We take the perspective that using the unlabeled training data is permitted (Kodirov et al., 2015;Wang et al., 2019b;Schick and Schütze, 2021). 11 Note that Kim et al. (2022) estimate the random label performance by interpolating with the performance using negated labels, while our paper samples the random labels at uniform. tasks such as generation, but leave this to future work. Extending of our experiments to such tasks is not trivial, because it requires a variation of the output which has incorrect input-output correspondence while keeping the correct output distribution (which is important based on our analysis in Section 5).
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Since the first version of our paper, Madaan and Yazdanbakhsh (2022) conducted a similar analysis with the chain of thought prompting (Wei et al., 2022b) which generates a rationale to perform complex tasks such as math problems. Madaan and Yazdanbakhsh (2022) show that, while simply using a random rationale in the demonstrations (e.g., pairing with a rationale from a different example) significantly degrades the performance, other types of counterfactual rationales (e.g., wrong equations) do not degrade the performance as much as we thought. We refer to Madaan and Yazdanbakhsh (2022) for more discussions on what aspects of the rationale matter or do not matter.  IV et al. (2021) in using the minimal format to transform the input to a sequence (e.g. a concatenation of multiple inputs) and using the label words from each dataset as it is. We also explore manual templates taken from prior work (Holtzman et al., 2021;Zhao et al., 2021) as reported in Section 4.2, although we find that using these templates is not consistently better than using minimal templates. We thus run main experiments with minimal templates. Example templates are provided in Table 3.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Format of the demonstrations We follow the standard of each model for formatting the demonstrations, either from exploration in prior work or the example code provided in the official tutorial. For GPT-2, we separate the input and the label,  and each demonstration example with a space. For MetaICL, GPT-J and GPT-3, we separate the input and the label with a newline (\n), and each demonstration example with three newlines. For fairseq models, we use a newline to separate the input and the label as well as each demonstration example.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Details in variants of the demonstrations For "demonstrations w/ a% accurate labels" (0 ≤ a ≤ 100), we use k × a/100 correct pairs and k × (1 − a/100) incorrect pairs in a random order, as described in Algorithm 1. For "OOD demonstrations", we use CC-News (Nagel, 2016) as an external corpus. We consider the length of the text during sampling, so that sampled sentences have similar length to the test input. For "demonstrations with random English words", we use pypi.org/ project/english-words for the set of En- 
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
No Demos Demos w/ gold labels Demos w/ random labels Figure 11: Results of No-demonstration, Gold demonstration and Random demonstration on 3 classification datasets (top) and 3 multi-choice datasets (bottom). Details in Section 4.1. This figure is for providing numbers that are comparable across models-full results with more datasets are reported in Figure 3.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Algorithm 1 Forming the demonstrations with an accuracy of a%.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
n ← k × a/100 // number of correct pairs 4:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
G ← Sample(Range(1, k), n) 5:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
for i ∈ Range(1, k) do 6:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
if i ∈ G then // add correct pair 7:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
D.append((xi, yi)) 8:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
else // add incorrect pair 9:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
D.append((xi, Sample(C − yi))) 10:
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
return D glish words, which consists of 61,569 words. Table 4 provides a list of example demonstrations for each method used in Section 5.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
C.1 Gold labels vs. random labels Figure 11 shares the same interface as Figure 3, but all models are evaluated on 3 classification and 3 multi-choice datasets and are thus comparable to each other.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
In Section 4, random labels are sampled from the label space from a uniform distribution. We experiment with another variant of demonstrations in the classification tasks, where labels are randomly sampled from the true distribution of labels on the training data. This may have large impact if labels are far from uniform on the training data. Results indicate that performance drop from using gold labels is further reduced compared to using uniformly random labels: with Channel MetaICL, the gap is reduced from 1.9% to 1.3% absolute, and with Channel GPT-J, the gap is reduced from 5.0% to 3.5% absolute. Figure 12 shows performance gap between using gold labels and using random labels per dataset. We find that the trend that the gap is smaller than previously thought is consistant across most datasets. Nonetheless, there are a few outlier datasets where performance gap is non-negligible, such as finan-cial_phrasebank and a few hate speech detection datasets. Future work may investigate on which tasks the model makes more use of the correctly paired training data.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We explored demonstrations with a constant label where all labels in the demonstrations are replaced with a constant text, "answer". Specifically, a prediction is made via argmax y∈C P (y|x 1 , answer...x k , answer, x). This can be viewed as another way to remove the impact of the label space while keeping the impact of the distribution of the input text. However, results are consistently worse than the results of demonstrations with random English labels. We think this is because constant labels actually change the format of the demonstrations, since they can be viewed as part of a separator between different demonstration examples.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
We also explored demonstrations with the test input where all inputs in the demonstrations are replaced with the test input, each paired with a ran-dom label. Specifically, a prediction is made via argmax y∈C P (y|x,ỹ 1 ...x,ỹ k , x), whereỹ i (1 ≤ i ≤ k) is randomly sampled at uniform from C. This variant is seemingly a reasonable choice given that it satisfies the condition that the inputs in the demonstrations come from the same distribution as the test input (since they are identical), and using random labels is as good as using gold labels. Nonetheless, we find that this variant is significantly worse than most other methods with demonstrations. We think this is because using the constant input for all demonstration example significantly changes the format of the sequence, since the input can be viewed as part of a separator between different demonstration examples.  Figure 12: Performance gap from using the demonstrations with gold labels to using the demonstrations with random labels. Datasets are sorted in descending order. The top two figures use random labels that are sampled at uniform, with Channel MetaICL and Channel GPT-J, respectively. The bottom two figures use random labels that are sampled from a true distribution of labels on the training data, with Channel MetaICL and Channel GPT-J, respectively.    Figure 6 for discussion in empirical results. The input and the label are in the red text and in the blue text, respectively. Note that | is used to separate different options for the labels.  Table 4: Example demonstrations when using methods in Section 5. The financial_phrasebank dataset with C = {"positive", "neutral", "negative"} is used. Red text indicates the text is sampled from an external corpus; blue text indicates the labels are randomly sampled from the label set; purple text indicates a random English word.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Large pretrained language models (PLMs, Devlin et al., 2019;Peters et al., 2018;Raffel et al., 2020;Liu et al., 2019;Yang et al., 2019;Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual context (Petroni et al., 2019(Petroni et al., , 2020Jiang et al., 2020;Shin et al., 2020;Davison et al., 2019). For example, when conditioned on a long document and a "TL;DR:" token, they can generate a summary of said document, and when provided a partial question ("The theory of relativity was developed by __"), they can generate the correct answer. Perhaps most strikingly, when primed with a context consisting of very few training examples, they produce text classification results that can match those of fully supervised models. This type of few shot setting, is commonly referred to as "In-context Learning" (Brown et al., 2020).
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
A core component of in-context learning is the text-based prompt that serves as the context. Composing a prompt requires: (i) text linearisation using a template; and (ii) training sample concatenation (See Table 1 for an example). It has been established that the structure of the template has a large impact on performance (Shin et al., 2020;Gao et al., 2020;Schick and Schütze, 2020;Jiang et al., 2020). However, to the best of our knowledge, no work has studied the effect of the sample ordering on In-context Learning performance.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Perhaps counter-intuitively, we find that the right sample order can make as much of a difference as Example training set (the greatest musicians, 1) (redundant concept, 0)  the right template. As can be seen in Figure 1, some permutations have comparable performance (over 85% accuracy) to supervised training for sentiment classification, while others perform close to random (around 50%). This order sensitivity is universal across models, and although increasing the model size somewhat addresses it, the problem is still present for some text classification tasks (Subj in Figure 1) for models with billions of parameters. In our analysis, we find no common denominator between performant sample orders and that they are not transferable across different model sizes and tasks. In a fully-supervised setting, we could rely on a development set to select among sample orders. However, this is not desirable in a few-shot setting where the size of the development set is very limited, even unavailable (Perez et al., 2021) . Instead, we use the generative nature of language models to construct an unlabelled artificial development set and refer to it as a probing set. As the probing set is unlabelled, we use the predicted label distribution statistics and propose entropy-based metrics to measure the quality of candidate prompts.Experimental results show that we can achieve on average 13% relative improvement across eleven different established text classification tasks across all different sizes (four orders of magnitude) of PLMs.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
To summarise, our contributions are as follows:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
1. We study order sensitivity for In-context Learning, which we show is crucial for the success of pretrained language models for fewshot learning.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
2. We propose a simple, generation-based probing method to identify performant prompts without requiring additional data.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
3. Our probing method is universally applicable and effective across different sizes of pretrained language models and for different types of datasets -achieving on average a Figure 2: Training sample permutations for the Incontext Learning setting. The concatenation of training samples as well as test data transforms the classification task into a sequence generation task.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
13% relative improvement over a wide range of tasks.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
In this section, we study the relationship between permutation performance and various factors. For the ease of visualisation, we use a fixed random subset of four samples with a balanced label distribution from the SST-2 dataset and consider all 24 possible sample order permutations. This setup is illustrated in Figure 2. We also test five randomlyselected sets of examples and summarised variance statistics in the experiment section (Section 5).
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Although beneficial, increasing model size does not guarantee low variance We evaluate the order permutations for four different sizes of GPT-2 (0.1B-1.5B) 1 and GPT-3 (2.7B-175B). As we can observe in Figure 1, models can obtain remarkable few-shot performance. We see that the GPT2-XL (1.5B) model can even surpass 90% accuracy given just four samples. This result is comparable to those of supervised models trained on more than 60,000 samples. However, the performance variation of different permutations remain a big issue, especially for "smaller" models. 2 The same model can exhibit nearly perfect behaviour given one sample order, but then fall back to be on par with a random baseline for another. While increasing the model size (by a few order of magnitudes) can sometimes alleviate the issue, it still cannot resolve it entirely (especially if we consider tasks other than SST-2). In contrast, different initialisations of supervised fine-tuning approaches typically result in less than 1% standard deviation for their test set performance (Gao et al., 2020). Adding training samples does not significantly reduce variance To further explore the order sensitivity of few-shot prompts, we increase the number of training samples and then sample a subset of at most 24 different orderings. 3 We use the GPT2 family models for this experiment. In Figure 3, we can observe that increasing the number of training samples leads to increases in performance. However, a high level of variance remains, even with a large number of samples and can even increase. Based on this, we draw the conclusion that order sensitivity is likely to be a fundamental issue of In-context Learning regardless of the number of training samples.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Performant prompts are not transferable across models We find that a specific permutation's performance may drop from 88.7% to 51.6% by changing the underlying model from GPT2-XL (1.5B) to GPT2-Large (0.8B). This suggests that a particular permutation working well for one model does not imply that it will provide good results for another model. To validate this hypothesis, we use all possible order permutations of the four samples as prompts -24 in total. We then perform prediction conditioned on each of these prompts for different models and calculate the pairwise Spearman's rank correlation coefficient between the scores. These results are shown in Figure 4.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
If there is a common pattern for performant prompts, we should then be able to observe high correlation across models. However, the behaviour of permutations is seemingly random even across 3 Bounded at the lower limit by the total number of samples given, and at the upper limit as there can be up to 64! possible orders. different sizes of the same model. For example, the 175B and 2.7B model only has a correlation of 0.05, this means a good permutation for the 2.7B model is in no way guaranteed that it will also yield good performance for the 175B model.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Performant label orderings are not consistent across models In addition to training example ordering, we also explore label ordering for training prompts. We use all patterns of the abovementioned full permutations -six different label patterns. 4 We then compute the pairwise Spearman correlation across different models as described in the previous paragraph. As shown in Figure 5, the behaviour of label orderings is once again seemingly random across different sizes of the same model. It is thus not possible to identify a label ordering that is performant across different models.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Degenerate behaviour of bad prompts We perform error analysis across performant and nonperformant prompts and observe that the majority of failing prompts suffer from highly unbalanced predicted label distributions ( Figure 6, left). An intuitive way to address this would be by calibrating the output distribution, along the lines of Zhao et al. (2021). However, we find that although calibration leads to much higher performance, the variance remains high (Figure 6, right).
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
The previous section demonstrates that prompt order can have a substantial effect on performance, with some orderings of the same prompts for the same model providing random performance, and other "better" orderings providing performance competitive with supervised approaches. This suggests that there could be various ways of selecting prompt orders to achieve better performance, but the challenge is to do so automatically and without the need for additional labels (e.g., a development set). Hence, in this section, we explore the question of: "How can we automatically generate a 'probing set' to find performant prompt orderings"? We approach this by: (i) for a randomly-selected set of training samples, we use every possible ordering permutation of this set as candidates; (ii) constructing a probing set by querying the language model using all candidate prompts as context; and (iii) use this probing set to identify the best ordering by ranking them using a probing metric.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We propose a simple methodology to automatically construct a "probing set", by directly sam-pling from the language model itself. This approach makes it possible to generate probing sets automatically, without access to any additional data. Concretely, given a set of training samples S = {(x i , y i )}, i = 1, · · · , n, where x i and y i denote the sentence and label of the i th training sample. We then define a transformation T , mapping each sample into natural language space, such that t i = T (x i , y i ). t i is therefore a text sequence of the i th training sample using the template defined by T . In this work, we use a simple transformation function T such that T (x i , y i ) = input:x i type:y i . This transforms each sample into a standard format sentence, which linearises each element in the set into natural language space defined as S = {t i }, i = 1, · · · , n. We then define a full permutation function group of n training samples, F = {f m }, m = 1, · · · , n!, where each function f m takes S as input and outputs c m : the concatenation of a unique permutation. In our case, sampling four training samples at random gives up to 24 possible ordering permutations of the transformed samples.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
For each prompt candidate c m , we then sample from the language model to obtain the probing sequence g m ∼ P (·|c m ; θ), where θ denotes the parameters of the pretrained language model. We stop decoding from the language model upon generating the special end-of-sentence token defined by a template, or reach the generation length limit. Our probing set construction method is illustrated in Figure 7, where the objective is to generate a probing set that shares a similar distribution to the training samples.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We run this sampling process for all possible prompt ordering permutations and extract probing samples from them (T −1 (g)). Then gather extracted samples together to form the probing set
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
. Although the probing set contains predicted label for each sentence, there is no guarantee on the validity of these labels. Therefore, we discard them from the probing set as we are only interested in sampling probes from the language model corresponding to the input distribution.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Once we have constructed a probing set for a given set of samples, we can now use that probing set to identify the best possible prompt ordering for that particular sample set. Here, we explore two Figure 7: Our probing set construction method, showing the various possible ordering permutations of the randomly selected training samples, the resulting generation for each permutation, and the concatenation of each into a probing set. Note that we discard the generated labels, as there is no guarantee that these generated labels are correct. methods for selecting the best ordering: Global Entropy (GlobalE), and Local Entropy (LocalE).
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Global Entropy (GlobalE) The motivation behind GlobalE is to identify prompts of specific sample orderings that avoid the issue of extremely unbalanced predictions (as we have previously established it as key problem for non-performant prompts). We compute the predicted labelŷ i for data point (x i , y i ) under context c m as follows:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
For each label v ∈ V (where V denotes the target label set), we compute the label probability over the probing set as:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We then use the predicted category label entropy as the GlobalE score for c m as follows:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Local Entropy (LocalE) The motivation behind LocalE is that if a model is overly confident for all probing inputs, then it is likely that the model is not behaving as desired. At the very least, it is poorly calibrated, which could also be an indication of a poor capability to appropriately differentiate between classes. Similar to the GlobalE computation, we calculate the prediction probability of a data point (x i , y i ) over the target labels v ∈ V under context c m , as follows:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We then calculate the average prediction entropy per data point as the LocalE score:
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
As we now have a way to score each prompt ordering, based on its effect against the probing set, we can rank each prompt ordering by performance as measured by GlobalE or LocalE respectively.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We use four different sizes of GPT-2 (Radford et al., 2019) (with 0.1B, 0.3B, 0.8B, and 1.5B parameteers) and two sizes of GPT-3 (Brown et al., 2020) (with 2.7B, and 175B parameters). Due to limited context window size (up to 1024 word-pieces for the GPT-2 series of models), we use a 4-shot setting for all datasets except AGNews and DBPedia. Our experiments are based on the open-source checkpoints of GPT-2 models and access to the OpenAI GPT-3 API. 5 For probing set generation, we restrict the maximum generation length to 128. We also use sampling with a temperature, t, of 2, and we also make use of block n-gram repetitions (Paulus et al., 2018) to encourage diverse generation. We use 24 different permutations for each set of randomly selected training samples and use 5 different sets (except for GPT-3 with 175B parameters, where we only do two sets with 12 different permutation due to the high monetary cost) for each experiment, giving a total of 120 runs. We report the mean and standard deviation of the corresponding evaluation metric over 5 different sets.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
For performant prompt selection, we rank candidate prompts using the LocalE and GlobalE prob-  ing metrics over the automatically generated probing set. We then select top k samples ranked by highest entropy values, where k = 4 in our experiments, of the available 24 permutations as performant prompts. Finally, we use these performant prompts to evaluate performance on various datasets and demonstrate both better performance and reduced variance. We also provide results for a majority baseline, which always predicts the majority label in the dataset, as a lower-bound of performance. We also provide an oracle to show the upper-bound of performance by selecting the top four performant orderings based on prompt performance on the validation set.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Similar to previous work (Gao et al., 2020;Zhao et al., 2021), we use eleven text classification datasets ranging from sentiment classification to textual entailment. Further details of the datasets are provided in the Appendix. For evaluation, we sub-sample 256 samples of the validation sets for all datasets to control for the GPT-3 inference costs as it requires the usage of a monetary paid-for API.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We report experimental results in Table 2 and observe consistent improvements for both LocalE and GlobalE across all tasks.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Entropy-based probing is effective for performant prompt selection regardless of model size We find that GlobalE achieves, on average, a 13% relative improvement across the eleven different sentence classification tasks in comparison to prompts that do not make use of probing. LocalE provides results slightly inferior to GlobalE, with an average 9.6% relative improvement over the baseline model. Our selected performant prompts also demonstrate considerably lower variance than using all candidate prompts.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
In Figure 8, we visualise the average performance when varying K for the top K prompt selection. K = 24 corresponds to using all sampled prompt orders, which is equivalent to the baseline model performance in Table 2. We can observe that the slope of curves are negative for all datasets, suggesting that our method can rank performant prompts effectively. Though K = 1 can provide good performance for most cases, in our experiments, we use K = 4 as preliminary experiments indicated that it yielded stable performance across datasets. Entropy-based probing is effective across templates We evaluate Entropy-based probing for four different templates similar to Gao et al. (2020) and Zhao et al. (2021) (Table 4) for the SST-2 dataset. Experimental results in Table 3 indicate that Entropy-based probing is valid for different templates. We also observe that the randomness across different templates is similar to Section 2. These findings suggest that Entropy-based probing is not sensitive to specific templates, as it consistently provides improvements for all cases.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Performant permutation selection is a safe option for In-context Learning We find that for models that suffer from high prompt variance, our prompt selection process can show large improvements -up to 30% relative improvement. Furthermore, for tasks with low initial prompt performance variance, our method does not negatively impact performance. Our prompt selection provides marginal improvement at worse and on average a 13% relative improvement in the most cases.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Sentence-pair tasks remain challenging for smaller-sized models even with performant permutation selection For the CB and RTE datasets,   the performance of GPT-2 models is not significantly different from that of a random baseline. Despite this, we find that our method for identifying performant prompts can still provide minimal performance gains, although these are still within the levels of a random guess or majority vote. One reason for this could be that, for these particular sizes of models on these tasks, no good prompt exists. As such, optimising the prompt is not particularly effective in this setting. This is further supported by the observation that prompt selection can considerably improve performance on both CB and RTE at larger model sizes (particularly so for the GPT-3 175B parameter model). In fact, we find that prompt selection using GlobalE improves performance by 4.9% for GPT-3 175B on CB. This indicates that our method is widely applicable to all model sizes, and across all tasks, as long as they already possess some existing classification ability that can be improved through prompt design.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Entropy-based probing outperforms using subsets of the training data for tuning If one was not to rely on generation, an alternative approach to prompt selection could be to split the (limited) training data to form a validation set. To compare against this approach, we split the 4-shot training samples (same setting as in Table 2) in half. We then select the top four performing prompts using validation set performance. As can be seen in Table 5, this approach consistently outperforms the baseline. However, both Entropy-based probing methods consistently provides better performance across all model sizes.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We have shown that few-shot prompts suffer from order sensitivity, in that for the same prompt the order in which samples are provided can make the difference between state-of-the-art and random performance. In our analysis of the problem, we established that it is present across tasks, model sizes, prompt templates, samples, and number of training samples. To alleviate this problem, we introduced a novel probing method that exploits the generative nature of language models to construct an artificial development set. We were able to identity performant permutations using entropy-based statistics over this set, leading to an on average 13% improvement across eleven text classification tasks.    (Pang and Lee, 2005) 2 25.7 Yes CR (Hu and Liu, 2004) 2 22.1 Yes MPQA (Wiebe et al., 2005) 2 3.9 Yes Subj (Pang and Lee, 2004) 2 28.9 Yes TREC (Voorhees and Tice, 2000) 6 11.6 No AGNews (Zhang et al., 2015) 4 53.8 Yes DBPedia (Zhang et al., 2015) 14 65.  
What learning algorithm is in-context learning? Investigations with linear models
One of the most surprising behaviors observed in large neural sequence models is in-context learning (ICL; Brown et al., 2020). When trained appropriately, models can map from sequences of (x, f (x)) pairs to accurate predictions f (x ) on novel inputs x . This behavior occurs both in models trained on collections of few-shot learning problems (Chen et al., 2022;Min et al., 2022) and surprisingly in large language models trained on open-domain text (Brown et al., 2020;Zhang et al., 2022;Chowdhery et al., 2022). ICL requires a model to implicitly construct a map from in-context examples to a predictor without any updates to the model's parameters themselves. How can a neural network with fixed parameters to learn a new function from a new dataset on the fly? This paper investigates the hypothesis that some instances of ICL can be understood as implicit implementation of known learning algorithms: in-context learners encode an implicit, contextdependent model in their hidden activations, and train this model on in-context examples in the course of computing these internal activations. As in recent investigations of empirical properties of ICL (Garg et al., 2022;Xie et al., 2022), we study the behavior of transformer-based predictors (Vaswani et al., 2017) on a restricted class of learning problems, here linear regression. Unlike in past work, our goal is not to understand what functions ICL can learn, but how it learns these functions: the specific inductive biases and algorithmic properties of transformer-based ICL.
What learning algorithm is in-context learning? Investigations with linear models
In Section 3, we investigate theoretically what learning algorithms transformer decoders can implement. We prove by construction that they require only a modest number of layers and hidden units to train linear models: for d-dimensional regression problems, with O(d) hidden size and constant depth, a transformer can implement a single step of gradient descent; and with O(d 2 ) hidden size a Correspondence to akyurek@mit.edu. Ekin is a student at MIT, and began this work while he was intern at Google Research. Code and reference implementations are released at this web page b The work is done when Tengyu Ma works as a visiting researcher at Google Research. and constant depth, a transformer can update a ridge regression solution to include a single new observation. Intuitively, n steps of these algorithms can be implemented with n times more layers.
What learning algorithm is in-context learning? Investigations with linear models
In Section 4, we investigate empirical properties of trained in-context learners. We begin by constructing linear regression problems in which learner behavior is under-determined by training data (so different valid learning rules will give different predictions on held-out data). We show that model predictions are closely matched by existing predictors (including those studied in Section 3), and that they transition between different predictors as model depth and training set noise vary, behaving like Bayesian predictors at large hidden sizes and depths. Finally, in Section 5, we present preliminary experiments showing how model predictions are computed algorithmically. We show that important intermediate quantities computed by learning algorithms for linear models, including parameter vectors and moment matrices, can be decoded from in-context learners' hidden activations.
What learning algorithm is in-context learning? Investigations with linear models
A complete characterization of which learning algorithms are (or could be) implemented by deep networks has the potential to improve both our theoretical understanding of their capabilities and limitations, and our empirical understanding of how best to train them. This paper offers first steps toward such a characterization: some in-context learning appears to involve familiar algorithms, discovered and implemented by transformers from sequence modeling tasks alone.
What learning algorithm is in-context learning? Investigations with linear models
Training a machine learning model involves many decisions, including the choice of model architecture, loss function and learning rule. Since the earliest days of the field, research has sought to understand whether these modeling decisions can be automated using the tools of machine learning itself. Such "meta-learning" approaches typically treat learning as a bi-level optimization problem (Schmidhuber et al., 1996;Andrychowicz et al., 2016;Finn et al., 2017): they define "inner" and "outer" models and learning procedures, then train an outer model to set parameters for an inner procedure (e.g. initializer or step size) to maximize inner model performance across tasks.
What learning algorithm is in-context learning? Investigations with linear models
Recently, a more flexible family of approaches has gained popularity. In in-context learning (ICL), meta-learning is reduced to ordinary supervised learning: a large sequence model (typically implemented as a transformer network) is trained to map from sequences [x 1 , f (x 1 ), x 2 , f (x 2 ), ..., x n ] to predictions f (x n ) (Brown et al., 2020;Olsson et al., 2022;Laskin et al., 2022;Kirsch & Schmidhuber, 2021). ICL does not specify an explicit inner learning procedure; instead, this procedure exists only implicitly through the parameters of the sequence model. ICL has shown impressive results on synthetic tasks and naturalistic language and vision problems (Garg et al., 2022;Min et al., 2022;Zhou et al., 2022;Hollmann et al., 2022).
What learning algorithm is in-context learning? Investigations with linear models
Past work has characterized what kinds of functions ICL can learn (Garg et al., 2022;Laskin et al., 2022;Müller et al., 2021) and the distributional properties of pretraining that can elicit in-context learning (Xie et al., 2021;Chan et al., 2022). But how ICL learns these functions has remained unclear. What learning algorithms (if any) are implementable by deep network models? Which algorithms are actually discovered in the course of training? This paper takes first steps toward answering these questions, focusing on a widely used model architecture (the transformer) and an extremely well-understood class of learning problems (linear regression).
What learning algorithm is in-context learning? Investigations with linear models
Transformers (Vaswani et al., 2017) are neural network models that map a sequence of input vectors x = [x 1 , . . . , x n ] to a sequence of output vectors y = [y 1 , . . . , y n ]. Each layer in a transformer maps a matrix H (l) (interpreted as a sequence of vectors) to a sequence H (l+1) . To do so, a transformer layer processes each column h (l)
What learning algorithm is in-context learning? Investigations with linear models
i of H (l) in parallel. Here, we are interested in autoregressive (or "decoder-only") transformer models in which each layer first computes a self-attention:
What learning algorithm is in-context learning? Investigations with linear models
where each b is the response of an "attention head" defined by:
What learning algorithm is in-context learning? Investigations with linear models
Published as a conference paper at ICLR 2023 then applies a feed-forward transformation:
What learning algorithm is in-context learning? Investigations with linear models
Here σ denotes a nonlinearity, e.g. a Gaussian error linear unit (GeLU; Hendrycks & Gimpel, 2016):
What learning algorithm is in-context learning? Investigations with linear models
and λ denotes layer normalization (Ba et al., 2016):
What learning algorithm is in-context learning? Investigations with linear models
where the expectation and variance are computed across the entries of x. To map from x to y, a transformer applies a sequence of such layers, each with its own parameters. We use θ to denote a model's full set of parameters (the complete collection of W matrices across layers). The three main factors governing the computational capacity of a transformer are its depth (the number of layers), its hidden size (the dimension of the vectors h), and the number of heads (denoted m above).
What learning algorithm is in-context learning? Investigations with linear models
We study transformer models directly trained on an ICL objective. (Some past work has found that ICL also "emerges" in models trained on general text datasets; Brown et al., 2020.) To train a transformer T with parameters θ to perform ICL, we first define a class of functions F, a distribution p(f ) supported on F, a distribution p(x) over the domain of functions in F, and a loss function L. We then choose θ to optimize the auto-regressive objective, where the resulting T θ is an in-context learner:
What learning algorithm is in-context learning? Investigations with linear models
Our experiments focus on linear regression problems. In these problems, F is the space of linear functions f (x) = w x where w, x ∈ R d , and the loss function is the squared error L(y, y ) = (y − y ) 2 . Linear regression is a model problem in machine learning and statistical estimation, with diverse algorithmic solutions. It thus offers an ideal test-bed for understanding ICL. Given a dataset with inputs X = [x 1 , . . . , x n ] and y = [y 1 , . . . , y n ], the (regularized) linear regression objective:
What learning algorithm is in-context learning? Investigations with linear models
With λ = 0, this objective is known as ordinary least squares regression (OLS); with λ > 0, it is known as ridge regression (Hoerl & Kennard, 1970). (As discussed further in Section 4, ridge regression can also be assigned a Bayesian interpretation.) To present a linear regression problem to a transformer, we encode both x and f (x) as d + 1-dimensional vectors:
What learning algorithm is in-context learning? Investigations with linear models
where 0 d denotes the d-dimensional zero vector.
What learning algorithm is in-context learning? Investigations with linear models
For a transformer-based model to solve Eq. (9) by implementing an explicit learning algorithm, that learning algorithm must be implementable via Eq. (1) and Eq. (4) with some fixed choice of transformer parameters θ. In this section, we prove constructively that such parameterizations exist, giving concrete implementations of two standard learning algorithms. These proofs yield upper bounds on how many layers and hidden units suffice to implement (though not necessarily learn) each algorithm. Proofs are given in Appendices A and B.
What learning algorithm is in-context learning? Investigations with linear models
It will be useful to first establish a few computational primitives with simple transformer implementations. Consider the following four functions from R H×T → R H×T : mov(H; s, t, i, j, i , j ): selects the entries of the s th column of H between rows i and j, and copies them into the t th column (t ≥ s) of H between rows i and j , yielding the matrix:
What learning algorithm is in-context learning? Investigations with linear models
mul(H; a, b, c, (i, j), (i , j ), (i , j )): in each column h of H, interprets the entries between i and j as an a × b matrix A 1 , and the entries between i and j as a b × c matrix A 2 , multiplies these matrices together, and stores the result between rows i and j , yielding a matrix in which each column has the form
What learning algorithm is in-context learning? Investigations with linear models
div(H; (i, j), i , (i , j )): in each column h of H, divides the entries between i and j by the absolute value of the entry at i , and stores the result between rows i and j , yielding a matrix in which every column has the form
What learning algorithm is in-context learning? Investigations with linear models
in each column h of H, applies an affine transformation to the entries between i and j and i and j , then stores the result between rows i and j , yielding a matrix in which every column has the form
What learning algorithm is in-context learning? Investigations with linear models
Lemma 1. Each of mov, mul, div and aff can be implemented by a single transformer decoder layer: in Eq.
What learning algorithm is in-context learning? Investigations with linear models
(1) and Eq. (4), there exist matrices W Q , W K , W V , W F , W 1 and W 2 such that, given a matrix H as input, the layer's output has the form of the corresponding function output above. 1
What learning algorithm is in-context learning? Investigations with linear models
With these operations, we can implement building blocks of two important learning algorithms.
What learning algorithm is in-context learning? Investigations with linear models
Rather than directly solving linear regression problems by evaluating Eq. (10), a standard approach to learning exploits a generic loss minimization framework, and optimizes the ridge-regression objective in Eq. (9) via gradient descent on parameters w. This involves repeatedly computing updates:
What learning algorithm is in-context learning? Investigations with linear models
for different examples (x i , y i ), and finally predicting w x n on a new input x n . A step of this gradient descent procedure can be implemented by a transformer: Theorem 1. A transformer can compute Eq. (11) (i.e. the prediction resulting from single step of gradient descent on an in-context example) with constant number of layers and O(d) hidden space, where d is the problem dimension of the input x. Specifically, there exist transformer parameters θ such that, given an input matrix of the form:
What learning algorithm is in-context learning? Investigations with linear models
the transformer's output matrix H (L) contains an entry equal to w x n (Eq. (11)) at the column index where x n is input.
What learning algorithm is in-context learning? Investigations with linear models
Another way to solve the linear regression problem is to directly compute the closed-form solution Eq. (10). This is somewhat challenging computationally, as it requires inverting the regularized covariance matrix X X + λI. However, one can exploit the Sherman-Morrison formula (Sherman & Morrison, 1950) to reduce the inverse to a sequence of rank-one updates performed example-byexample. For any invertible square A,
What learning algorithm is in-context learning? Investigations with linear models
Because the covariance matrix X X in Eq. (10) can be expressed as a sum of rank-one terms each involving a single training example x i , this can be used to construct an iterative algorithm for computing the closed-form ridge-regression solution.
What learning algorithm is in-context learning? Investigations with linear models
Theorem 2. A transformer can predict according to a single Sherman-Morrison update:
What learning algorithm is in-context learning? Investigations with linear models
with constant layers and O(d 2 ) hidden space. More precisely, there exists a set of transformer parameters θ such that, given an input matrix of the form in Eq. (12), the transformer's output matrix H (L) contains an entry equal to w x n (Eq. (14)) at the column index where x n is input.
What learning algorithm is in-context learning? Investigations with linear models
Discussion. There are various existing universality results for transformers (Yun et al., 2020;Wei et al., 2021), and for neural networks more generally (Hornik et al., 1989). These generally require very high precision, very deep models, or the use of an external "tape", none of which appear to be important for in-context learning in the real world. Results in this section establish sharper upper bounds on the necessary capacity required to implement learning algorithms specifically, bringing theory closer to the range where it can explain existing empirical findings. Different theoretical constructions, in the context of meta-learning, have been shown for linear self-attention models (Schlag et al., 2021), or for other neural architectures such as recurrent neural networks (Kirsch & Schmidhuber, 2021). We emphasize that Theorem 1 and Theorem 2 each show the implementation of a single step of an iterative algorithm; these results can be straightforwardly generalized to the multi-step case by "stacking" groups of transformer layers. As described next, it is these iterative algorithms that capture the behavior of real learners.
What learning algorithm is in-context learning? Investigations with linear models
The previous section showed that the building blocks for two specific procedures-gradient descent on the least-squares objective and closed-form computation of its minimizer-are implementable by transformer networks. These constructions show that, in principle, fixed transformer parameterizations are expressive enough to simulate these learning algorithms. When trained on real datasets, however, in-context learners might implement other learning algorithms. In this section, we investigate the empirical properties of trained in-context learners in terms of their behavior. In the framework of Marr's (2010) "levels of analysis", we aim to explain ICL at the computational level by identifying the kind of algorithms to regression problems that transformer-based ICL implements.
What learning algorithm is in-context learning? Investigations with linear models
Determining which learning algorithms best characterize ICL predictions requires first quantifying the degree to which two predictors agree. We use two metrics to do so:
What learning algorithm is in-context learning? Investigations with linear models
Squared prediction difference. Given any learning algorithm A that maps from a set of inputoutput pairs D = [x 1 , y 1 , . . . , x n , y n ] to a predictor f (x) = A(D)(x), we define the squared prediction difference (SPD):
What learning algorithm is in-context learning? Investigations with linear models
where D is sampled as in Eq. (8). SPD measures agreement at the output level, regardless of the algorithm used to compute this output.
What learning algorithm is in-context learning? Investigations with linear models
Implicit linear weight difference. When ground-truth predictors all belong to a known, parametric function class (as with the linear functions here), we may also investigate the extent to which different learners agree on the parameters themselves. Given an algorithm A, we sample a context dataset D as above, and an additional collection of unlabeled test inputs D X = {x i }. We then compute A's prediction on each x i , yielding a predictor-specific dataset   Figure 1: Fit between ICL and standard learning algorithms: We plot (dimension normalized) SPD and ILWD values between textbook algorithms and ICL on noiseless linear regression with d = 8. GD(α) denotes one step of batch gradient descent and SGD(α) denotes one pass of stochastic gradient descent with learning rate α. Ridge(λ) denotes Ridge regression with regularization parameter λ. Under both evaluations, in-context learners agree closely with ordinary least squares, and are significantly less well approximated by other solutions to the linear regression problem.
What learning algorithm is in-context learning? Investigations with linear models
We can then quantify agreement between two predictors A 1 and A 2 by computing the distance between their implied weights in expectation over datasets:
What learning algorithm is in-context learning? Investigations with linear models
When the predictors are not linear, ILWD measures the difference between the closest linear predictors (in the sense of Eq. (16)) to each algorithm. For algorithms that have linear hypothesis space (e.g. Ridge regression), we will use the actual value ofŵ A instead of the estimated value.
What learning algorithm is in-context learning? Investigations with linear models
We train a Transformer decoder autoregresively on the objective in Eq. 
What learning algorithm is in-context learning? Investigations with linear models
ICL matches ordinary least squares predictions on noiseless datasets. We begin by comparing a (L = 16, H = 512, M = 4) transformer against a variety of reference predictors:
What learning algorithm is in-context learning? Investigations with linear models
• k-nearest neighbors: In the uniform variant, models predictŷ i = 1 3 j y j , where j is the top-3 closest data point to x i where j < i. In the weighted variant, a weighted averagê y i ∝ 1 3 j |x i − x j | −2 y j is calculated, normalized by the total weights of the y j s. • One-pass stochastic gradient descent:ŷ i = w i x i where w i is obtained by stochastic gradient descent on the previous examples with batch-size equals to 1:
What learning algorithm is in-context learning? Investigations with linear models
• One-step batch gradient descent:ŷ i = w i x i where w i is obtained by one of step gradient descent on the batch of previous examples:
What learning algorithm is in-context learning? Investigations with linear models
We denote the case of λ = 0 as OLS.
What learning algorithm is in-context learning? Investigations with linear models
The agreement between the transformer-based ICL and these predictors is shown in Fig. 1. As can be seen, there are clear differences in fit to predictors: for almost any number of examples, normalized SPD and ILWD are small between the transformer and OLS predictor (with squared error less than 0.01), while other predictors (especially nearest neighbors) agree considerably less well.
What learning algorithm is in-context learning? Investigations with linear models
(0.0/1.0) 2 = 0 (0.25/1.0) 2 = 1/16 (0.5/1.5) 2 = 1/9 (0.5/1.0) 2 = 1/4 (0.5/0.75) 2 = 4/9 2 / 2 (Lstsq, ICL)  Figure 2: ICL under uncertainty: With problem dimension d = 8, and for different values of prior variance τ 2 and data noise σ 2 , we display (dimension-normalized) MSPD values for each predictor pair, where MSPD is the average SPD value over underdetermined region of the linear problem. Brightness is proportional with 1 MSPD . ICL most closely follows the minimum-Bayes-risk Ridge regression output for all σ 2 τ 2 values.
What learning algorithm is in-context learning? Investigations with linear models
When the number of examples is less than the input dimension d = 8, the linear regression problem is under-determined, in the sense that multiple linear models can exactly fit the in-context training dataset. In these cases, OLS regression selects the minimum-norm weight vector, and (as shown in Fig. 1), the in-context learner's predictions are reliably consistent with this minimum-norm predictor. Why, when presented with an ambiguous dataset, should ICL behave like this particular predictor? One possibility is that, because the weights used to generate the training data are sampled from a Gaussian centered at zero, ICL learns to output the minimum Bayes risk solution when predicting under uncertainty (see Müller et al. (2021)). Building on these initial findings, our next set of experiments investigates whether ICL is behaviorally equivalent to Bayesian inference more generally.
What learning algorithm is in-context learning? Investigations with linear models
ICL matches the minimum Bayes risk predictor on noisy datasets. To more closely examine the behavior of ICL algorithms under uncertainty, we add noise to the training data: now we present the in-context dataset as a sequence:
What learning algorithm is in-context learning? Investigations with linear models
Recall that ground-truth weight vectors are themselves sampled from a Gaussian distribution; together, this choice of prior and noise mean that the learner cannot be certain about the target function with any number of examples.
What learning algorithm is in-context learning? Investigations with linear models
Standard Bayesian statistics gives that the optimal predictor for minimizing the loss in Eq. (8) is: 
What learning algorithm is in-context learning? Investigations with linear models
For linear regression with Gaussian priors and Gaussian noise, the Bayesian estimator in Eq. (18) has a closed-form expression:
What learning algorithm is in-context learning? Investigations with linear models
Note that this predictor has the same form as the ridge predictor from Section 2.3, with the regularization parameter set to σ 2 τ 2 . In the presence of noisy labels, does ICL match this Bayesian predictor? We explore this by varying both the dataset noise σ 2 and the prior variance τ 2 (sampling w ∼ N (0, τ 2 )). For these experiments, the SPD values between the in-context learner and various regularized linear models is shown in Fig. 2. As predicted, as variance increases, the value of the ridge parameter that best explains ICL behavior also increases. For all values of σ 2 , τ 2 , the ridge parameter that gives the best fit to the transformer behavior is also the one that minimizes Bayes risk. These experiments clarify the finding above, showing that ICL in this setting behaviorally matches minimum-Bayes-risk predictor. We also note that when the noise level σ → 0 + , the Bayes predictor converges to the ordinary least square predictor. Therefore, the results on noiseless datasets studied in the beginning paragraph of this subsection can be viewed as corroborating the finding here in the setting with σ → 0 + .
What learning algorithm is in-context learning? Investigations with linear models
ICL exhibits algorithmic phase transitions as model depth increases. The two experiments above evaluated extremely high-capacity models in which (given findings in Section 3) computational constraints are not likely to play a role in the choice of algorithm implemented by ICL. But what about smaller models-does the size of an in-context learner play a role in determining the learning algorithm it implements? To answer this question, we run two final behavioral experiments: one in which we vary the hidden size (while optimizing the depth and number of heads as in Section 4.2), then vary the depth of the transformer (while optimizing the hidden size and number of heads). These experiments are conducted without dataset noise.
What learning algorithm is in-context learning? Investigations with linear models
Results are shown in Fig. 3. When we vary the depth, learners occupy three distinct regimes: very shallow models (1L) are best approximated by a single step of gradient descent (though not wellapproximated in an absolute sense). Slightly deeper models (2L-4L) are best approximated by ridge regression, while the deepest (+8L) models match OLS as observed in Fig. 3. Similar phase shift occurs when we vary hidden size in a 16D problem. Interestingly, we can read hidden size requirements to be close to ridge-regression-like solutions as H ≥ 16 and H ≥ 32 for 8D and 16D problems respectively, suggesting that ICL discovers more efficient ways to use available hidden state than our theoretical constructions requiring O(d 2 ). Together, these results show that ICL does not necessarily involve minimum-risk prediction. However, even in models too computationally constrained to perform Bayesian inference, alternative interpretable computations can emerge.
What learning algorithm is in-context learning? Investigations with linear models
Section 4 showed that transformers are a good fit to standard learning algorithms (including those constructed in Section 3) at the computational level. But these experiments leave open the question of how these computations are implemented at the algorithmic level. How do transformers arrive at the solutions in Section 4, and what quantities do they compute along the way? Research on extracting precise algorithmic descriptions of learned models is still in its infancy (Cammarata et al., 2020;Mu & Andreas, 2020). However, we can gain insight into ICL by inspecting learners' intermediate states: asking what information is encoded in these states, and where.
What learning algorithm is in-context learning? Investigations with linear models
To do so, we identify two intermediate quantities that we expect to be computed by gradient descent and ridge-regression variants: the moment vector X Y and the (min-norm) least-square estimated weight vector w OLS , each calculated after feeding n exemplars. We take a trained in-context learner, freeze its weights, then train an auxiliary probing model (Alain & Bengio, 2016) to attempt to recover the target quantities from the learner's hidden representations. Specifically, the probe model takes hidden states at a layer H (l) as input, then outputs the prediction for target variable. We define a probe with position-attention that computes (Appendix E):
What learning algorithm is in-context learning? Investigations with linear models
We train this probe to minimize the squared error between the predictions and targets v: L(v,v) = |v −v| 2 . The probe performs two functions simultaneously: its prediction error on held-out representations determines the extent to which the target quantity is encoded, while its attention mask, α identifies the location in which the target quantity is encoded. For the FF term, we can insert the function approximator of our choosing; by changing this term we can determine the manner in which the target quantity is encoded-e.g. if FF is a linear model and the probe achieves low error, then we may infer that the target is encoded linearly.
What learning algorithm is in-context learning? Investigations with linear models
For each target, we train a separate probe for the value of the target on each prefix of the dataset:
What learning algorithm is in-context learning? Investigations with linear models
i.e. one probe to decode the value of w computed from a single training example, a second probe to decode the value for two examples, etc. Results are shown in Fig. 4. For both targets, a 2layer MLP probe outperforms a linear probe, meaning that these targets are encoded nonlinearly (unlike the constructions in Section 3). However, probing also reveals similarities. Both targets are decoded accurately deep in the network (but inaccurately in the input layer, indicating that probe success is non-trivial.) Probes attend to the correct timestamps when decoding them. As in both constructions, X Y appears to be computed first, becoming predictable by the probe relatively early in the computation (layer 7); while w becomes predictable later (around layer 12). For comparison, we additionally report results on a control task in which the transformer predicts ys generated with a fixed weight vector w = 1 (so no ICL is required). Probes applied to these models perform significantly worse at recovering moment matrices (see Appendix E for details).
What learning algorithm is in-context learning? Investigations with linear models
We have presented a set of experiments characterizing the computations underlying in-context learning of linear functions in transformer sequence models. We showed that these models are capable in theory of implementing multiple linear regression algorithms, that they empirically implement this range of algorithms (transitioning between algorithms depending on model capacity and dataset noise), and finally that they can be probed for intermediate quantities computed by these algorithms.
What learning algorithm is in-context learning? Investigations with linear models
While our experiments have focused on the linear case, they can be extended to many learning problems over richer function classes-e.g. to a network whose initial layers perform a non-linear feature computation. Even more generally, the experimental methodology here could be applied to larger-scale examples of ICL, especially language models, to determine whether their behaviors are also described by interpretable learning algorithms. While much work remains to be done, our results offer initial evidence that the apparently mysterious phenomenon of in-context learning can be understood with the standard ML toolkit, and that the solutions to learning problems discovered by machine learning researchers may be discovered by gradient descent as well. 
What learning algorithm is in-context learning? Investigations with linear models
The operations for 1-step SGD with single exemplar can be expressed as following chain (please see proofs for the Transformer implementation of these operations (Lemma 1) in Appendix C):
What learning algorithm is in-context learning? Investigations with linear models
• mov(; 1, 0, (1, 1 + d),
What learning algorithm is in-context learning? Investigations with linear models
• aff(; (3 + 2d, 3 + 3d), (3 + 3d, 3 + 4d), (3 + 2d, 3 + 3d), W 1 = I, W 2 = −2α, ) (w ) • mov(; 2, 1, (3 + 2d, 3 + 3d), (3 + 2d, 3 + 3d)) (move w )
What learning algorithm is in-context learning? Investigations with linear models
• mul(; 1, d, 1, (3 + 2d, 3 + 3d), (1, 1 + d), (3 + 3d, 4 + 3d)) (w x 2 ) This will map:
What learning algorithm is in-context learning? Investigations with linear models
We can verify the chain of operator step-by-step. In each step, we show only the non-zero rows.
What learning algorithm is in-context learning? Investigations with linear models
• mov(; 1, 0, (1, 1 + d), (1, 1 + d)) (move x) 0 y 1 0
What learning algorithm is in-context learning? Investigations with linear models
• mov(; 2, 1, (3 + 2d, 3 + 3d), (3 + 2d, 3 + 3d)) (move w )
What learning algorithm is in-context learning? Investigations with linear models
• mul(; 1, d, 1, (3 + 2d, 3 + 3d), (1, 1 + d), (3 + 3d, 4 + 3d))
What learning algorithm is in-context learning? Investigations with linear models
We obtain the updated prediction in the last hidden unit of the third time-step.
What learning algorithm is in-context learning? Investigations with linear models
Generalizing to multiple steps of SGD. Since w is written in the hidden states, we may repeat this iteration to obtainŷ 3 = w x 3 where w is the one step update w − 2α(x 2 w x 2 − y 2 x 2 + λw, requiring a total of O(n) layers for a single pass through the dataset where n is the number of examplers.
What learning algorithm is in-context learning? Investigations with linear models
As an empirical demonstration of this procedure, the accompanying code release contains a reference implementation of SGD defined in terms of the base primitive provided in an anymous links https://icl1.s3.us-east-2.amazonaws.com/theory/{primitives,sgd,ridge}.py (to preserve the anonymity we did not provide the library dependencies). This implementation predictŝ y n = w n x n , where w n is the weight vector resulting from n − 1 consecutive SGD updates on previous examples. It can be verified there that the procedure requires O(n + d) hidden space. Note that, it is not O(nd) because we can reuse spaces for the next iteration for the intermediate variables, an example of this performed in (w ) step above highlighted with blue color.
What learning algorithm is in-context learning? Investigations with linear models
We provide a similar construction to Theorem 1 (please see proofs for the Transformer implementation of these operations in Appendix C, specifically for div see Appendix C.6)
What learning algorithm is in-context learning? Investigations with linear models
• mov(; 1, 0, (1, 1 + d), (1, 1 + d)) (move x 1 ) • mul(; d, 1, 1, (1, 1 + d), (0, 1), (1 + d, 1 + 2d)) (x 1 y)
What learning algorithm is in-context learning? Investigations with linear models
• div(; (1 + 4d + d 2 , 1 + 4d + 2d 2 ), 1 + 4d + 2d 2 , (2 + 4d + 2d 2 , 2 + 4d + 3d 2 )) (right term) • aff(; (1+2d, 1+2d+d 2 ), (2+4d+2d 2 , 2+4d+3d 2 ), (1+2d, 1+2d+d 2 ), W 1 = I, W 2 = −I) (A −1 1 ) • mul(; d, d, 1, (1 + 2d, 1 + 2d + d 2 ), (1, 1 + d), (2 + 4d + 3d 2 , 2 + 5d + 3d 2 )) (A −1 1 x 1 ) • mul(; d, 1, 1, (2 + 4d + 3d 2 , 2 + 5d + 3d 2 ), (0, 1), (2 + 4d + 3d 2 , 2 + 5d + 3d 2 )) (A −1 1 x 1 y 1 )
What learning algorithm is in-context learning? Investigations with linear models
• mov(; 2, 1, (2 + 4d + 3d 2 , 2 + 5d + 3d 2 ), (2 + 4d + 3d 2 , 2 + 5d + 3d 2 )) (move w )
What learning algorithm is in-context learning? Investigations with linear models
• mul(; d, 1, 1(2 + 4d + 3d 2 , 2 + 5d + 3d 2 ), (1, 1 + d), (2 + 5d + 3d 2 , 3 + 5d + 3d 2 )) (w x 2 ) Note that, in contrast to Appendix A, we need O(d 2 ) space to implement matrix multiplications. Therefore over-all required hidden size is O(d 2 )
What learning algorithm is in-context learning? Investigations with linear models
As Theorem 1, generalizing it to multiple iterations will at least require O(n) layers, as we repeat the process for the next examplar.
What learning algorithm is in-context learning? Investigations with linear models
C LEMMA 1
What learning algorithm is in-context learning? Investigations with linear models
All of the operators mentioned in this lemma share a common computational structure, and can in fact be implemented as special cases of a "base primitive" we call RAW (for Read-Arithmetic-Write). This operator may also be useful for future work aimed at implementing other algorithms.
What learning algorithm is in-context learning? Investigations with linear models
The structure of our proof of Lemma 1 is as follows:
What learning algorithm is in-context learning? Investigations with linear models
1. Motivation of the base primitive RAW.
What learning algorithm is in-context learning? Investigations with linear models
3. Definition of dot, aff, mov in terms of RAW.
What learning algorithm is in-context learning? Investigations with linear models
4. Implementation of RAW in terms of transformer parameters.
What learning algorithm is in-context learning? Investigations with linear models
5. Brief discussion of how to parallelize RAW, making it possible to implement mul.
What learning algorithm is in-context learning? Investigations with linear models
6. Seperate proof for div by utilizing layer norm.
What learning algorithm is in-context learning? Investigations with linear models
At a high level, all of the primitives in Lemma 1 involve a similar sequence of operations:
What learning algorithm is in-context learning? Investigations with linear models
1) Operators read some hidden units from the current or previous timestep: dot and aff read from two subsets of indices in the current hidden state h t 2 , while mov reads from a previous hidden state h t . This selection is straightforwardly implemented using the attention component of a transformer layer.
What learning algorithm is in-context learning? Investigations with linear models
We may notate this reading operation as follows:
What learning algorithm is in-context learning? Investigations with linear models
Here r denotes a list of indice to read from, and K denotes a map from current timesteps to target timesteps. For convenience, we use Numpy-like notation to denote indexing into a vector with another vector:
What learning algorithm is in-context learning? Investigations with linear models
is Python index notation where the resulting vector, x = x[r]:
What learning algorithm is in-context learning? Investigations with linear models
The first step of our proof below shows that the attention output a (l) can compute the expression above.
What learning algorithm is in-context learning? Investigations with linear models
2) Operators perform element-wise arithmetic between the quantity read in step 1 and another set of entries from the current timestep: This step takes different forms for aff and mul (mov ignores values at the current timestep altogether).
What learning algorithm is in-context learning? Investigations with linear models
The second step of the proof below computes these operations inside the MLP component of the transformer layer.
What learning algorithm is in-context learning? Investigations with linear models
3) Operators reduce, then write to the current hidden state Once the underlying element-wise operation calculated, the operator needs to write these values to the some indices in current hidden state, defined by a list of indices w. Writing might be preceded by a reduction state (e.g. for computing dot products), which can be expressed generically as a linear operator W o . The final form of the computation is thus:
What learning algorithm is in-context learning? Investigations with linear models
Here, ← means that the other indices i / ∈ w are copied from h l−1 .
What learning algorithm is in-context learning? Investigations with linear models
We denote this "master operator" as RAW:
What learning algorithm is in-context learning? Investigations with linear models
Definition C.2. RAW(h; , s, r, w, W o , W a , W, K) is a function R H×T → R H×T . It is parameterized by an elementwise operator ∈ {+, }, three matrices W ∈ R d×|s| , W a ∈ R d×|r| , W o ∈ R |w|×d , three index sets s, r, and w, and a timestep map K : Z + → (Z + * ). Given an input matrix h, it outputs a matrix with entries:
What learning algorithm is in-context learning? Investigations with linear models
We additionally require that j ∈ K(i) =⇒ j < i (since self-attention is causal.)
What learning algorithm is in-context learning? Investigations with linear models
(For simplicity, we did not include a possible bias term in linear projections W o , W a , W , we can always assume the accompanying bias parameters b 0 , b a , b when needed) C.3 REDUCING LEMMA 1 OPERATORS TO RAW OPERATOR Given this operator, we can define each primitive in Lemma 1 using a single RAW operator, except the mul and div. Instead of the matrix multiplication operator mul, we will first show the dot product dot (a special case of mul), then later in the proof, we will argue that we can parallelize these dot products in Appendix C.5 to obtain mul. We will show how to implement div separately in Appendix C.6.
What learning algorithm is in-context learning? Investigations with linear models
Lemma 2. We can define mov, aff operator, and the dot product case of mul in Lemma 1 by using a single RAW operator dot(h; (i, j), (i , j ), (i , j )) = mul(h; 1, |i − j|, 1, (i, j), (i , j ), (i , i + 1)) = RAW(h; , W = I, W a = I,
What learning algorithm is in-context learning? Investigations with linear models
Proof. Follows immediately by substituting parameters into Eq. (26).
What learning algorithm is in-context learning? Investigations with linear models
It remains only to show: Lemma 3. A single transformer layer can implement the RAW operator: there exist settings of transformer parameters such that, given an arbitrary hidden matrix h as input, the transformer computes h (Eq. (26)) as output.
What learning algorithm is in-context learning? Investigations with linear models
Our proof proceeds in stages. We begin by providing specifying initial embedding and positional embedding layers, constructing inputs to the main transformer layer with necessary positional information and scratch space. Next, we prove three useful procedures for bypassing (or exploiting) non-linearities in the feed-forward component of the transformer. Finally, we provide values for remaining parameters, showing that we can implement the Elementwise and Reduction steps described above.
What learning algorithm is in-context learning? Investigations with linear models
Embedding Layer for Initialization: Rather than inserting the input matrix h directly into the transformer layer, we assume (as is standard) the existence of a linear embedding layer. We can set this layer to pad the input, providing extra scratch space that will be used by later steps of our implementation.
What learning algorithm is in-context learning? Investigations with linear models
We define the embedding matrix W e as:
What learning algorithm is in-context learning? Investigations with linear models
Then, the embedded inputs will bex
What learning algorithm is in-context learning? Investigations with linear models
Position Embeddings for Attention Manipulation: Implementing RAW ultimately requires controlling which position attends to which position in each layer. For example, we may wish to have layers in which each position attends to the previous position only, or in which even positions attends to other even positions. We can utilize position embeddings, p i , to control attention weights. In a standard transformer, the position embedding matrix is a constant matrix that is added to the inputs of the transformer after embedding layer (before the first layer), so the actual input to to the transformer is:
What learning algorithm is in-context learning? Investigations with linear models
We will use these position embeddings to encode the timestep map K. To do this, we will use 2p units per layer (p will be defined momentarily). p units will be used to encode attention keys k i , and the other p will be used to encode queries q i .
What learning algorithm is in-context learning? Investigations with linear models
We define the position embedding matrix as follows:
What learning algorithm is in-context learning? Investigations with linear models
With K encoded in positional embeddings, the transformer matrices W Q and W K are easy to define: they just need to retrieve the corresponding embedding values:
What learning algorithm is in-context learning? Investigations with linear models
The constructions used in this paper rely on two specific timestep maps K, each of which can be implemented compactly in terms of k and q:
What learning algorithm is in-context learning? Investigations with linear models
Case 1: Attend to previous token. This can be constructed by setting:
What learning algorithm is in-context learning? Investigations with linear models
where N is a sufficiently large number. In this case, the output of the attention mechanism will be:
What learning algorithm is in-context learning? Investigations with linear models
, 0]
What learning algorithm is in-context learning? Investigations with linear models
Case 2: Attend to a single token. For simpler patterns, such as attention to a specific token:
What learning algorithm is in-context learning? Investigations with linear models
only 1 hidden unit is required. We set:
What learning algorithm is in-context learning? Investigations with linear models
from which it can be verified (using the same procedure as in Case 1) that the desired attention pattern is produced.
What learning algorithm is in-context learning? Investigations with linear models
Intricacy: How can K be empty? We can also cause K(i) to attend to an empty set by assuming the softmax has extra ("imaginary") timestep obtained by prepending a 0 to attention vector pot-hoc (Chen et al., 2021).
What learning algorithm is in-context learning? Investigations with linear models
Cumulatively, the parameter matrices defined in this subsection implement the Read with Attention component of the RAW operator.
What learning algorithm is in-context learning? Investigations with linear models
The mul operator requires elementwise multiplication of quantities stored in hidden states. While transformers are often thought of as only straightforwardly implementing affine transformations on hidden vectors, their nonlinearities in fact allow elementwise multiplication to a high degree of approximation. We begin by observing the following property of the GeLU activation function in the MLP layers of the Transformer network: Lemma 4. The GeLU nonlinearity can be used to perform multiplication: specifically,
What learning algorithm is in-context learning? Investigations with linear models
Proof. A standard implementation of the GeLU nonlinearity is defined as follows:
What learning algorithm is in-context learning? Investigations with linear models
Thus
What learning algorithm is in-context learning? Investigations with linear models
For small x and y, the third-order term vanishes. By scaling inputs down by a constant before the GeLU layer, and scaling them up afterwards, models may use the GeLU operator to perform elementwise multiplication.
What learning algorithm is in-context learning? Investigations with linear models
We can generalize this proof to other smooth functions as we discussed further in [TODO REF].
What learning algorithm is in-context learning? Investigations with linear models
Previous work also shows, in practice, Transformers with ReLU activation utilize non-linearities to get the multiplication in other settings.
What learning algorithm is in-context learning? Investigations with linear models
When implementing the aff operator, we have the opposite problem: we would like the output of addition to be transmitted without nonlinearities to the output of the transformer layer. Fortunately, for large inputs, the GeLU nonlinearity is very close to linear; to bypass it it suffices to add to inputs a large N : Lemma 5. The GeLU nonlinearity can be bypassed: specifically,
What learning algorithm is in-context learning? Investigations with linear models
Proof.
What learning algorithm is in-context learning? Investigations with linear models
For all verions of the RAW operator, it is additionally necessary to bypass the LayerNorm operation.
What learning algorithm is in-context learning? Investigations with linear models
The following formula will be helpful for this: Lemma 6. Let N be a large number and λ the LayerNorm function. Then the following approximation holds:
What learning algorithm is in-context learning? Investigations with linear models
Then,
What learning algorithm is in-context learning? Investigations with linear models
By adding a large number N to two padding locations and sum the part of the hidden state that we are interested to pass through LayerNorm, we make x to the output of LayerNorm pass through. This addition can be done in the transformer's feed-forward computation (with parameter W F ) prior to layer norm. This multiplication of 2 L N can be done in first layer of MLP back, then linear layer can output/use x. For convenience, we will henceforth omit the LayerNorm operation when it is not needed.
What learning algorithm is in-context learning? Investigations with linear models
We may make each of these operations as precise as desired (or allowed by system precision). With them defined, we are ready to specify the final components of the RAW operator.
What learning algorithm is in-context learning? Investigations with linear models
We want to show a layer of Transformer defined in above, hence parameterized by θ = {W f , W 1 , W 2 , (W Q , W K , W v ) m }, can well-approximate the RAW operator defined in Eq. (25). We will provide step by step constructions and define the parameters in θ. Begin by recalling the transformer layer definition:
What learning algorithm is in-context learning? Investigations with linear models
Attention Output We will only use m = 2 attention heads for this construction. We show in Eq. (32) that we can control attentions to uniformly attend with a pattern by setting key and query matrices. Assume that the first head parameters W Q 1 , W K 1 have been set in the described way to obtain the pattern function K. Now we will set remaining attention parameters W V 1 , W Q 2 , W K 2 , W V 2 and show hat we can make the a i + h (l) i term in Eq. (4) to contain the corresponding term in Eq. (25), in some unused indices t such that:
What learning algorithm is in-context learning? Investigations with linear models
Then the term on the RAW operator can be obtained by the first head's output. In order to achieve that, we will set W a as a part of actual attention value network such that W V 1 is sparse matrix 0 everywhere expect:
What learning algorithm is in-context learning? Investigations with linear models
Now our first heads stores the right term in Eq. (53) in the indicies t. However, when we add the residual term h (l)
What learning algorithm is in-context learning? Investigations with linear models
i , this will change. To remove the residual term, we will use another head to output h (l) i , by setting W Q 2 , W K 2 such that K(i) = i, and W V 2 (similar to Eq. (34)):
What learning algorithm is in-context learning? Investigations with linear models
Then, W f ∈ R H×2H is zero otherwise:
What learning algorithm is in-context learning? Investigations with linear models
We already defined (W Q , W K , W V ) 1,2 and W f and obtained the first term in the Eq. (25) 
What learning algorithm is in-context learning? Investigations with linear models
(122) We then set b 2 :
What learning algorithm is in-context learning? Investigations with linear models
(127) We have used 4|t| space for internal computation of this operation, and finally used |w| space to write the final result. We show RAW operator is implementable by setting the parameters of a Transformer.
What learning algorithm is in-context learning? Investigations with linear models
C.5 PARALLELIZING THE RAW OPERATOR Lemma 7. With the conditions that K is constant, the operators are independent (i.e (r i ∪ s i ∪ w i ) ∩ w j =i = ∅), and there is k (4|t k | + |w k |) available space in the hidden state, then a Transformer layer can apply k such RAW operation in parallel by setting different regions of W 1 , W 2 , W f and (W V ) k matrices.
What learning algorithm is in-context learning? Investigations with linear models
Proof. From the construction above, it is straightforward to modify the definition of the RAW operator to perform k operations as all the indices of matrices that we use in Appendix C.4.3 do not overlap with the given conditions in the lemma.
What learning algorithm is in-context learning? Investigations with linear models
This makes it possible to construct a Transformer layer not only to implement vector-vector dot products, but general matrix-matrix products, as required by mul. With this, we show that we can implement mul by using single layer of a Transformer.
What learning algorithm is in-context learning? Investigations with linear models
Let say we have the input [c, y, 0] calculated before the attention output in Eq. (53), and we want to divide y to c. This trick is very similar to the on in Lemma 6. We can use the following formula: Lemma 8. using LayerNorm for division. Let N, M to be large numbers, λ LayerNorm function, the following approximation holds:
What learning algorithm is in-context learning? Investigations with linear models
Proof.
What learning algorithm is in-context learning? Investigations with linear models
Published as a conference paper at ICLR 2023
What learning algorithm is in-context learning? Investigations with linear models
To get the input to the format used in this Lemma, we can easily use W f to convert the head outputs. Then, after the layer norm, we can use W 1 to pull the y c back and write it to the attention output. By this way, we can approximate scalar division in one layer.
What learning algorithm is in-context learning? Investigations with linear models
Lemma 1 By Lemmas 2, 3, 3, 7 and 8; we constructed the operators in Lemma 1 using single layer of a Transformer, thus proved Lemma 1 D DETAILS OF TRANSFORMER ARHITECTURE AND TRAINING We perform these experiments using the Jax framework on P100 GPUs. The major hyperparameters used in these experiments are presented in Table 1. The code repository used for reproducing these experiments will be open sourced at the time of publication. Most of the hyperparameters adapted from previous work Garg et al. (2022) to be compatible, and we adapted the Transformer architecture details. We use Adam optimizer with cosine learning rate scheduler with warmup where number of warmup steps set to be 1/5 of total iterations. We use larned absolute position embeddings.
What learning algorithm is in-context learning? Investigations with linear models
Search Range In the phase shift plots in Fig. 3, we keep the value in the x-axis constant and used the best setting over the parameters: {number of layers, hidden size, number of heads and learning rate}.
What learning algorithm is in-context learning? Investigations with linear models
We will use the terms probe model and task model to distinguish probe from ICL. Our probe is defined as:
What learning algorithm is in-context learning? Investigations with linear models
The position scores s v ∈ R T are learned parameters where T is the max input sequence length (T = 80 in our experiments). The softmax of position scores attention weights α for each position and for each target variable. This enables us to learn input-independent, optimal target locations for each target (displayed on the right side of Fig. 4). We then average hidden states by using these attention weights. A linear projection, W v ∈ R T ×H , is applied before averaging. FF is either a linear layer or a 2-layer MLP (hidden size=512) with a GeLU activation function. For each layer, we train a different probe with different parameters using stochastic gradient descent. H equals to the 512. The probe is trained using an Adam optimizer with a learning rate of 0.001 (chosen from among {0.01, 0.001, 0.0001} on validation data).  Fig. 4, dashed lines show probing results with a task model trained on a control task, in which w is always the all-ones 1. This problem structurally resembles our main experiment setup, but does not require in-context learning. During probing, we feed this model data generated by w sampled form normal distribution as in the original task model. We observe that the control probe has a significantly higher error rate, showing that the probing accuracy obtained with actual task model is non-trivial. We present detailed error values of the control probe in Fig. 5.
What learning algorithm is in-context learning? Investigations with linear models
In Fig. 1b, we compare implicit linear weight of the ICL against the linear algorithms using ILWD measure. Note that this measure do not assume predictors to be linear: when the predictors are not linear, ILWD measures the difference between closest linear predictors (in Eq. (16) sense) to each algorithm.
What learning algorithm is in-context learning? Investigations with linear models
To gain more insight to ICL's algorithm, we can measure how linear ICL in different regimes of the linear problem (underdetermined, determined) by using R 2 (coefficient of determination) measure. So, instead of asking what's the best linear fit in Eq. (16), we can ask how good is the linear fit, which is the R 2 of the estimator. Interestingly, even though our model matches min-norm least square solution in both metrics in Section 4.3, we show that ICL is becoming gradually linear in the under-determined regime Fig. 6. This is an important result, enables us to say the in-context learner's hypothesis class is not purely linear.  
What learning algorithm is in-context learning? Investigations with linear models
We can show that for a real-valued and smooth non-linearity f (x), we can apply the same trick in in the paper body. In particular, we can write Taylor expansion as:
What learning algorithm is in-context learning? Investigations with linear models
a i x i = a 0 + a 1 x + a 2 x 2 + . . .
What learning algorithm is in-context learning? Investigations with linear models
which converges for some sufficiently small neighborhood: X ∈ [− , ]. First, assume that the second order term a 2 dominates higher-order terms in this domain such that:
What learning algorithm is in-context learning? Investigations with linear models
It's is easy to verify that the following is true:
What learning algorithm is in-context learning? Investigations with linear models
So, given the expansion for GeLU in Eq. (37), we can use this generic formula to obtain the multiplication approximation:
What learning algorithm is in-context learning? Investigations with linear models
π 2 (GeLU (x + y) − GeLU (x) − GeLU (y)) ≈ xy
What learning algorithm is in-context learning? Investigations with linear models
We plot this approximation against x 2 for [0.1, −0.1] range in Fig. 7a.
What learning algorithm is in-context learning? Investigations with linear models
In the case of a 2 is zero, we cannot get any second order term, and in the case of a 2 is negligible O(x 3 + y 3 ) will dominate the Eq. (135), so we cannot obtain a good approximation of xy. In this case, we can resort to numerical derivatives and utilize the a 3 term:
What learning algorithm is in-context learning? Investigations with linear models
f (x) = a 1 + 2a 2 x + 3a 3 x 3 + . . .
What learning algorithm is in-context learning? Investigations with linear models
If a 3 is not negligible, a 3 x 2 a i>3 x i in the same domain, we can use numerical derivatives to get a multiplication term:
What learning algorithm is in-context learning? Investigations with linear models
(138) For example, tanh has no second order term in its Taylor expansion:
What learning algorithm is in-context learning? Investigations with linear models
Using above formula we can obtain the following expression:
What learning algorithm is in-context learning? Investigations with linear models
Similar to our construction in Eq. (110), we can construct a Transformer layer that calculates these quantities (noting that δ is a small, input-independent scalar).
What learning algorithm is in-context learning? Investigations with linear models
We plot this approximation against x 2 for [0.1, −0.1] range in Fig. 7b. Note that, if we use this approximation in our constructions we will need more hidden space as there are 6 different tanh term as opposed to 3 GeLU term in Eq. (110).
What learning algorithm is in-context learning? Investigations with linear models
Non-smooth non-linearities ReLU is another commonly used non-linearity that is not differentiable. With ReLU, we can only hope to get piece-wise linear approximations. For example, we can try to approximate x 2 with the following function:
What learning algorithm is in-context learning? Investigations with linear models
0.0375 * ReLU(x) + 0.0375 * ReLU(−x) + ReLU(0.05 * (x − 0.05))+ ReLU(−0.05 * (x + 0.05)) + ReLU(0.025 * (x − 0.025)) + ReLU(−0.025 * (x + 0.025)) ≈ x 2
What learning algorithm is in-context learning? Investigations with linear models
We plot this approximation against x 2 for [0.1, −0.1] range in Fig. 7c.
What learning algorithm is in-context learning? Investigations with linear models
In Figs. 3a and 3b, we showed that ICL needs different hidden sizes to enter the "Ridge regression phase" (orange background) or "OLS phase" (green background) depending on the dimensionality d of inputs x. However, we cannot reliably read the actual relations between size requirements and the dimension of the problem from only two dimensions. To better understand size requirements, we ask the following empirical question for each dimension: how many layer/hidden size/heads are needed to better fit the least-squares solution than the Ridge(λ = ) regression solution (the green phase in Figs. 3a and 3b)?
What learning algorithm is in-context learning? Investigations with linear models
To answer this important question, we experimented with d = {1, 2, 4, 8, 12, 16, 20} and run an experiment sweep for each dimension over: For each feature that affects computational capacity of transformer (L, H, M ), we optimize other features and find the minimum value for the feature that satisfies SPD(OLS, ICL) < SPD(Ridge(λ = ), ICL). We plot our experiment with = 0.1 in Appendix H. We find that single head is enough for all problem dimensions, while other parameters exhibit a step-functionlike dependence on input size.
What learning algorithm is in-context learning? Investigations with linear models
Please note that other hyperparameters discussed in Appendix D (e.g weight initialization) were not optimized for each dimension independently.  
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Pretrained masked language models (Devlin et al., 2019;Liu et al., 2019b) have benefitted a wide range of natural language processing (NLP) tasks (Liu, 2019;Wadden et al., 2019;Zhu et al., 2020). Despite recent progress in understanding what useful information is captured by MLMs (Liu et al., 2019a;Hewitt and Manning, 2019), it remains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
One popular justification of MLMs relies on viewing masking as a form of cloze reduction.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Cloze-like Masking I this movie [MASK] Dependency Learning I like this movie Figure 1: We study the inductive bias of MLM objectives and show that cloze-like masking (left) does not account for much of the downstream performance gains. Instead, we show that MLM objectives are biased towards extracting both statistical and syntactic dependencies using random masks (right).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Cloze reductions reformulate an NLP task into a prompt question and a blank and elicit answers by filling in the blank (Figure 1). When tested by cloze reductions pretrained MLMs and left-to-right language models (LMs) have been shown to possess abundant factual knowledge (Petroni et al., 2019) and display impressive few-shot ability (Brown et al., 2020). This success has inspired recent hypotheses that some word masks are cloze-like and provide indirect supervision to downstream tasks Lee et al., 2020). For example, a sentiment classification task (Pang et al., 2002) can be reformulated into filling in like or hate in the cloze I [MASK] this movie. Such cloze-like masks provide a clear way in which an MLM can implicitly learn to perform sentiment classification.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
While this hypothesis is appealing, MLMs in practice are trained with uniform masking that does not contain the special structure required by cloze-like masks most of the time. For example, predicting a generic word this in the cloze I like [MASK] movie would not offer task-specific supervision. We quantify the importance of cloze-like and generic masks by explicitly creating cloze-like masks using task-specific lexicons and comparing models pretrained on these masks. These experiments suggest that although cloze-like masks can be helpful, the success of uniform masking cannot be explained via cloze-like masks alone. In fact, we demonstrate that uniform masking performs as well as a negative control where we explicitly remove cloze-like masks from the mask distribution.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To address this mismatch between theory and practice, we offer a new hypothesis of how generic masks can help downstream learning. We propose a conceptual model for MLMs by drawing a correspondence between masking and graphical model neighborhood selection (Meinshausen and Bühlmann, 2006). Using this, we show that MLM objectives are designed to recover statistical dependencies in the presence of latent variables and propose an estimator that can recover these learned dependencies from MLMs. We hypothesize that statistical dependencies in the MLM objective capture useful linguistic dependencies and demonstrate this by using recovered statistical dependencies to perform unsupervised parsing, outperforming an actual unsupervised parsing baseline (58.74 vs 55.91 UUAS; Klein and Manning, 2004). We release our implementation on Github 1 .
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Theories inspired by Cloze Reductions. Cloze reductions are fill-in-the-blank tests that reformulate an NLP task into an LM problem. Existing work demonstrates that such reductions can be highly effective for zero/few-shot prediction (Radford et al., 2019;Brown et al., 2020) as well as relation extraction (Petroni et al., 2019;.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
These fill-in-the-blank tasks provide a clear way by which LMs can obtain supervision about downstream tasks, and recent work demonstrates how such implicit supervision can lead to useful representations . More general arguments by Lee et al. (2020) show these theories hold across a range of self-supervised settings. While these theories provide compelling arguments for the value of pre-training with cloze tasks, they do not provide a clear reason why uniformly random masks such as those used in BERT provide such strong gains. In our work, we quantify this gap using lexicon-based cloze-like masks and show that cloze-like masks alone are unlikely to account for the complete success of MLM since generic and non-cloze masks are responsible for a substantial part of the empirical performance of MLMs. 1 https://github.com/tatsu-lab/mlm_ inductive_bias Theories for vector representations. Our goal of understanding how masking can lead to useful inductive biases and linguistic structures is closely related to that of papers studying the theory of word embedding representations (Mikolov et al., 2013;Pennington et al., 2014;Arora et al., 2015). Existing work has drawn a correspondence between word embeddings and low-rank factorization of a pointwise mutual information (PMI) matrix (Levy and Goldberg, 2014) and others have shown that PMI is highly correlated with human semantic similarity judgements (Hashimoto et al., 2016).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
While existing theories for word embeddings cannot be applied to MLMs, we draw inspiration from them and derive an analogous set of results. Our work shows a correspondence between MLM objectives and graphical model learning through conditional mutual information, as well as evidence that the conditional independence structure learned by MLMs is closely related to syntactic structure.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Probing Pretrained Representations. Recent work has applied probing methods (Belinkov and Glass, 2019) to analyze what information is captured in the pretrained representations. This line of work shows that pretrained representations encode a diverse range of knowledge (Peters et al., 2018;Tenney et al., 2019;Liu et al., 2019a;Hewitt and Manning, 2019;. While probing provides intriguing evidence of linguistic structures encoded by MLMs, they do not address the goals of this work, which is how the pretraining objective encourages MLMs to extract such structures.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Masked Language Modeling asks the model to predict a token given its surrounding context. Formally, consider an input sequence X of L tokens x 1 , . . . , x L where each variable takes a value from a vocabulary V. Let X ∼ D be the data generating distribution of X. Let x i be the ith token in X, and let X \i denote the sequence after replacing the ith token with a special [MASK] token. In other words,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Similarly, define X \{i,j} as replacing both x i and  Figure 2: In our case study, we append the true label to each input and create ideal cloze-like masks. We study how deviations from the ideal mask distribution affect downstream performance by adding in generic masks. that minimizes
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In BERT pretraining, each input token is masked with a fixed, uniform probability, which is a hyperparameter to be chosen. We refer to this strategy as uniform masking.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Finetuning is the canonical method for using pretrained MLMs. Consider a prediction task where y ∈ Y is the target variable, e.g., the sentiment label of a review. Finetuning uses gradient descent to modify the pretrained parameters θ and learn a new set of parameters φ to minimize
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where p(y|x) is the ground-truth distribution and D is the data distribution of the downstream task. Our goals. We will study how the mask distribution M affects downstream performance. We define perfect cloze reductions as some partition of the vocabulary V y such that p(x i ∈ V y |X \i ) ≈ p(y|X). For a distribution M such that the masks we draw are perfect cloze-reductions, the MLM objective offers direct supervision to finetuning since L MLM ≈ L finetune . In contrast to cloze-like masking, in uniform masking we can think of p θ as implicitly learning a generative model of X (Wang and Cho, 2019). Therefore, as M moves away from the ideal distribution and becomes more uniform, we expect p θ to model more of the full data distribution D instead of focusing on cloze-like supervision for the downstream task. This mismatch between theory and practice raises questions about how MLM with uniform masking can learn useful inductive biases.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
When L MLM is not L finetune , what is L MLM learning? We analyze L MLM and show that it is similar to a form of conditional mutual information based graphical model structure learning.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To motivate our subsequent discussions, we perform a controlled study for the case when L MLM ≈  L finetune and analyze how deviations from the ideal mask distribution affect downstream performance. We perform analysis on the Stanford Sentiment Treebank (SST-2; Socher et al., 2013), which requires models to classify short movie reviews into positive or negative sentiment. We append the ground-truth label (as the word positive or negative) to each movie review ( Figure 2). Masking the last word in each review is, by definition, an ideal mask distribution. To study how the deviation from the ideal mask distribution degrades downstream performance, we vary the amount of cloze-like masks during training. We do this by masking out the last word for p% of the time and masking out a random word in the movie review for (100 − p)% of the time, and choose p ∈ {0, 20, 40, 60, 80, 100}. Experimental details. We split the SST-2 training set into two halves, use one for pretraining, and the other for finetuning. For the finetuning data, we do not append the ground-truth label. We pretrain small transformers with L MLM using different masking strategies and finetune them along with a baseline that is not pretrained (NOPRETRAIN). Further details are in Appendix A.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. We observe that while cloze-like masks can lead to successful transfer, even a small modification of the ideal mask distribution deteriorates performance. Figure 3 shows the development set accuracy of seven model variants averaged across ten random trials. We observe as p decreases, the performance of CLOZE-p% degrades. Notably, CLOZE-80% is already worse than CLOZE-100% and CLOZE-20% does not outperform NOPRE-TRAIN by much. We notice that CLOZE-0% in fact degrades finetuning performance, potentially because the pretrained model is over-specialized to the language modeling task (Zhang et al., 2020;Tamkin et al., 2020). While this is a toy example, we observe similar results for actual MLM models across three tasks (Section 5.1), and this motivates us to look for a framework that explains the success of generic masks in practice.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In the previous section, we saw that cloze-like masks do not necessarily explain the empirical success of MLMs with uniform masking strategies. Understanding uniform masking seems challenging at first, as uniform-mask MLMs seem to lack task-specific supervision and is distinct from existing unsupervised learning methods such as word embeddings (which rely upon linear dimensionality reduction) and autoencoders (which rely upon denoising). However, we show in this section that there is a correspondence between MLM objectives and classic methods for graphical model structure learning. As a consequence, we demonstrate that MLMs are implicitly trained to recover statistical dependencies among observed tokens.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our starting point is the observation that predicting a single feature (x i ) from all others (X \i ) is the core subroutine in the classic Gaussian graphical model structure learning algorithm of Meinshausen and Bühlmann (2006). In this approach, L different Lasso regression models are trained (Tibshirani, 1996) with each model predicting x i from X \i , and the nonzero coefficients of this regression correspond to the conditional dependence structure of the graphical model.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The MLM objective can be interpreted as a nonlinear extension of this approach, much like a classical algorithm that uses conditional mutual information (MI) estimators to recover a graphical model (Anandkumar et al., 2012). Despite the similarity, real world texts are better viewed as models with latent variables (e.g. topics; Blei et al., 2003) and many dependencies across tokens arise due to latent variables, which makes learning the direct dependencies difficult. We show that MLMs implicitly recover the latent variables and can capture the direct dependencies while accounting for the effect of latent variables. Finally, MLMs are only approximations to the true distribution and we show that the MLM objective can induce highquality approximations of conditional MI.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Analysis setup. To better understand MLMs as a way to recover graphical model structures, we show mask-based models can recover latent variables and the direct dependencies among variables in the Gaussian graphical model setting of Meinshausen and Bühlmann (2006). Let X = [x 1 , . . . , x L ] ∈ R L represent an input sequence where each of its coordinates x i represents a token, and Z ∈ R k be a latent variable that controls the sequence generation process. We assume that all coordinates of X are dependent on the latent variable Z, and there are sparse dependencies among the observed variables ( Figure 4). In other words, we can write Z ∼ N(0, Σ ZZ ) and X ∼ N(AZ, Σ XX ). Intuitively, we can imagine that Z represents shared semantic information, e.g. a topic, and Σ XX represents the syntactic dependencies. In this Gaussian graphical model, the MLM is analogous to regressing each coordinate of X from all other coordinates, which we refer to as masked regression.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM representations can recover latent variable. We now study the behavior of masked regression through the representation x mask,i that is obtained by applying masked regression on the ith coordinate of X and using the predicted values. Our result shows that masked regression is similar to the two-step process of first recovering the latent variable Z from X \i and then predicting x i from Z.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Let Σ XX,\i,i ∈ R d−1 be the vector formed by dropping the ith row and taking the ith column of Σ XX and β 2SLS,i be the linear map resulting from the two-stage regression X \i → Z → x i . Proposition 1. Assuming that Σ XX is full rank,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In other words, masked regression implicitly recovers the subspace that we would get if we first explicitly recovered the latent variables (β 2SLS,i ) with an error term that scales with the off-diagonal terms in Σ XX . The proof is presented in Appendix C.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To give additional context for this result, let us consider the behavior of a different representation learning algorithm: PCA. It is well-known that PCA can recover the latent variables as long as the Σ ZZ dominates the covariance Cov(X). We state this result in terms of X PCA , the observed data projected to the first k components of PCA. Proposition 2. Let λ k be the kth eigenvalue of AΣ ZZ A and λ XX,k+1 be the k+1th eigenvalue of Σ XX and V be the first k eigenvectors of Cov(X). Assuming λ k > λ XX,k+1 , we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where · op is the operator norm and tr(·) is the trace.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This shows that whenever Σ XX is sufficiently small and λ k is large (i.e., the covariance is dominated by Z), then PCA recovers the latent information in Z. The proof is based on the Davis-Kahan theorem (Stewart and Sun, 1990) and is presented in Appendix C.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Comparing the bound of PCA and masked regression, both bounds have errors that scales with Σ XX , but the key difference in the error bound is that the error term for masked regression does not scale with the per-coordinate noise (diag(Σ XX )) and thus can be thought of as focusing exclusively on interactions within X. Analyzing this more carefully, we find that Σ XX,\i,i corresponds to the statistical dependencies between x i and X \i , which we might hope captures useful, task-agnostic structures such as syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM log-probabilies can recover direct dependencies. Another effect of latent variables is that many tokens have indirect dependencies through the latent variables, which poses a challenge to recovering the direct dependencies among tokens. We now show that the MLMs can account for the effect of latent variable.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In the case where there are no latent variables, we can identify the direct dependencies via conditional MI (Anandkumar et al., 2012) because any x i and x j that are disconnected in the graphical model will have zero conditional MI, i.e., I(x i ; x j |X \{i,j} ) = 0. One valuable aspect of MLM is that we can identify direct dependencies even in the presence of latent variables.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
If we naively measure statistical dependency by mutual information, the coordinates of X would appear dependent on each other because they are all connected with Z. However, the MLM objective resolves this issue by conditioning on X \{i,j} . We show that latent variables (such as topics) that are easy to predict from X \{i,j} can be ignored when considering conditional MI. Proposition 3. The gap between conditional MI with and without latent variables is bounded by the conditional entropy H(Z|X \{i,j} ),
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This suggests that when the context X \{i,j} captures enough of the latent information, conditional MI can remove the confounding effect of the shared topic Z and extract the direct and sparse dependencies within X (see Appendix C for the proof).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM objective encourages capturing conditonal MI. We have now shown that conditional MI captures direct dependencies among tokens, even in the presence of latent variables. Next, we will show that the MLM objective ensures that a LM with low log-loss accurately captures the conditional MI. We now show that learning the MLM objective implies high-quality estimation of conditional MI. Denote X(i, v) as substituting x i with a new token v,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Conditional MI is defined as the expected pointwise mutual information (PMI) conditioned on the rest of the tokens,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where I p is the abbreviation of I p (x i ; x j |X \{i,j} ). Our main result is that the log-loss MLM objective directly bounds the gap between the true conditional mutual information from the data distribution and an estimator that uses the log-probabilities from the model. More formally, Proposition 4. Let
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
be an estimator constructed by the model distribution p θ . Then we can show,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where D kl represents the KL-divergence.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Here, the KL-divergence corresponds to the L MLM objective, up to a constant entropy term that depends on p. We present the proof in Appendix C. In other words, the MLM objective is implicitly encouraging the model to match its implied conditional MI to that of the data. We now use this result to create an estimator that extracts the conditional independence structures implied by MLM.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our earlier analysis in Proposition 4 suggests that an MLM with low loss has an accurate approximation of conditional mutual information. Using this result, we will now propose a procedure which estimatesÎ p θ . The definition ofÎ p θ shows that if we can access samples of x i and x j from the true distribution p, then we can directly estimate the conditional mutual information by using the log probabilities from the MLM. Unfortunately, we cannot draw new samples of x j | X \{i,j} , leading us to approximate this distribution using Gibbs sampling on the MLM distribution. Our Gibbs sampling procedure is similar to the one proposed in Wang and Cho (2019). We start with X 0 = X \{i,j} . For the tth iteration, we draw a sample x t i from p θ (x i |X t−1 \i ), and update by X t = X t−1 (i, x t i ). Then, we draw a sample x t j from p θ (x j |X t \j ) and set X t = X t (j, x t j ). We repeat and use the samples (x 1
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
i , x 1 j ), . . . , (x t i , x t j ) to compute the expectations for conditional MI.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This procedure relies upon an additional assumption that samples drawn from the MLM are faithful approximations of the data generating distribution. However, we show empirically that even this approximation is sufficient to test the hypothesis that the conditional independences learned by an MLM capture syntactic dependencies (Section 5.2).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We now test two predictions from our analyses. First, similar to our observation in the case study, we show that cloze-like masks do not explain the success of uniform masks on three real-world datasets. Second, our alternative view of relating MLM to graphical models suggests that statistical dependencies learned by MLMs may capture linguistic structures useful for downstream tasks. We demonstrate this by showing that MLMs' statistical dependencies reflect syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Setup. We now demonstrate that real-world tasks and MLMs show a gap between task-specific cloze masks and random masks. We compare the MLM with random masking to two different control groups. In the positive control (CLOZE), we pretrain with only cloze-like masks and in the negative control (NOCLOZE), we pretrain by explicitly excluding cloze-like masks. If the success of MLM can be mostly explained by implicit cloze reductions, then we should expect CLOZE to have strong downstream performance while NOCLOZE leads to a minimal performance gain. We compare pretraining with the uniform masking strategy used in BERT (UNIFORM) to these two control groups. If UNIFORM performs worse than the positive control and more similar to the negative control, then we know that uniform masking does not leverage cloze-like masks effectively.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Simulating Pretraining. Given computational constraints, we cannot retrain BERT from scratch. Instead, we approximate the pretraining process by continuing to update BERT with MLM (Gururangan et al., 2020), which we refer to as second-stage pretraining. Although this is an approximation to the actual pretraining process, the second-stage pretraining shares the same fundamental problem for pretraining: how can unsupervised training lead to downstream performance gains?
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study the effectiveness of different masking strategies by comparing to a BERT model without second-stage pretraining (VANILLA). We experiment with three text classification datasets: SST-2 (Socher et al., 2013), Hyperpartisan (Kiesel et al., 2019), and AGNews (Zhang et al., 2015). SST-2 classifies movie reviews by binary sentiment; Hyperpartisan is a binary classification task on whether a news article takes an extreme partisan standpoint; and AGNews classifies news articles into four different topics. On SST-2 and AGNews, we perform the second-stage pretraining on the training inputs (not using the labels). On Hyperpartisan, we use 100k unlabeled news articles that are released with the dataset. For SST-2 and AG-News, we study a low-resource setting and set the number of finetuning examples to be 20. For Hyperpartisan, we use the training set, which has 515 labeled examples. All evaluations are performed by fine-tuning a bert-base-uncased model (See Appendix A for full details).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Approximating Cloze-like Masking. We cannot identify the optimal set of cloze-like masks for an arbitrary downstream task, but these three tasks have associated lexicons which we can use to approximate the cloze-like masks. For SST-2, we take the sentiment lexicon selected by Hu and Liu (2004); for Hyperpartisan, we take the NRC word-emotion association lexicon (Mohammad and Turney, 2013); and for AGNews, we extract topic words by training a logistic regression classifier and AGNews 20-shot taking the top 1k features to be cloze-like masks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. Figure 5 plots the finetuning performance of different masking strategies. We observe that UNIFORM outperforms VANILLA, which indicates that second-stage pretraining is extracting useful information and our experiment setup is useful for studying how MLM leads to performance gains. As expected, CLOZE achieves the best accuracy, which confirms that cloze-like masks can be helpful and validates our cloze approximations.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The UNIFORM mask is much closer to NO-CLOZE than CLOZE. This suggests that uniform masking does not leverage cloze-like masks well and cloze reductions alone cannot account for the success of MLM. This view is further supported by the observation that NOCLOZE outperforms VANILLA suggesting that generic masks that are not cloze-like still contain useful inductive biases.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our results support our earlier view that there may be an alternative mechanism that allows generic masks that are not cloze-like to benefit downstream learning. Next, we will empirically examine BERT's learned conditional independence structure among tokens and show that the statistical dependencies relate to syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our analysis in section 4.1 shows that conditional MI (which is optimized by the MLM objective) can extract conditional independences. We will show that statistical dependencies estimated by conditional MI are related to syntactic dependencies by using conditional MI for unsupervised parsing.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Background. One might expect that the statistical dependencies among words are correlated with syntactic dependencies. Indeed, Futrell et al. (2019) show that heads and dependents in dependency parse trees have high pointwise mutual information (PMI) on average. However, previous at-tempts (Carroll and Charniak, 1992;Paskin, 2002) show that unsupervised parsing approaches based on PMI achieve close to random accuracy. Our analysis suggests that MLMs extract a more finegrained notion of statistical dependence (conditional MI) which does not suffer from the existence of latent variables (Proposition 3). We now show that the conditional MI captured by MLMs achieves far better performance, on par with classic unsupervised parsing baselines.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Baselines. We compare conditional MI to PMI as well as conditional PMI, an ablation in which we do not take expectation over possible words. For all statistical dependency based methods (cond. MI, PMI, and cond. PMI), we compute pairwise dependence for each word pair in a sentence and construct a minimum spanning tree on the negative values to generate parse trees. To contextualize our results, we compare against three simple baselines: RANDOM which draws a random tree on the input sentence, LINEARCHAIN which links adjacent words in a sentence, and a classic unsupervised parsing method (Klein and Manning, 2004).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental Setup. We conduct experiments on the English Penn Treebank using the WSJ corpus and convert the annotated constituency parses to Stanford Dependency Formalism (de Marneffe et al., 2006). Following Yang et al. (2020), we evaluate on sentences of length ≤ 10 in the test split, which contains 389 sentences (Appendix B.1 describes the same experiment on longer sentences, which have similar results). We experiment with the bert-base-cased model (more details in Appendix A) and evaluate by the undirected unlabeled attachment score (UUAS).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. Table 1 shows a much stronger-thanrandom association between conditional MI and dependency grammar. In fact, the parses extracted
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The above represents a triumph of either apathy or civility .  Figure 6: An example parse extracted from conditional MI. The black parse tree above the sentence represents the ground-truth parse and the red parse below is extracted from conditional MI. The correctly predicted edges are labeled with the annotated relations, and the incorrect ones are labeled as wrong.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Method UUAS
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
28.50 ± 0.73 LINEARCHAIN 54.13 Klein and Manning (2004)  from conditional MI has better quality than LIN-EARCHAIN and the classic method (Klein and Manning, 2004). Unlike conditional MI, PMI only has a close-to-random performance, which is consistent with prior work. We also see that conditional MI outperforms conditional PMI, which is consistent with our theoretical framework that suggests that conditional MI (and not PMI) recovers the graphical model structure.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We also perform a fine-grained analysis by investigating relations where conditional MI differs from LINEARCHAIN. Because the test split is small and conditional MI does not involve any training, we perform this analysis on 5,000 sentences from the training split. Table 2 presents the results and shows that conditional MI does not simply recover the linear chain bias. Meanwhile, we also observe a deviation between conditional MI and dependency grammar on relations like number and cc. This is reasonable because certain aspects of dependency grammar depend on human conventions that do not necessarily have a consensus (Popel et al., 2013). Figure 6 illustrates with an example parse extracted from conditional MI. We observe that conditional MI correctly captures dobj and conj. Knowing the verb, e.g. represents, limits the range of objects that can appear in a sentence so intuitively we expect a high conditional MI between the direct object and the verb. Similarly for phrases like "A and B", we would expect A and B to be statistically dependent. However, conditional MI fails  to capture cc (between apathy and or). Instead, it links or with either which certainly has statistical dependence. This once again suggests that the 'errors' incurred by the conditional PMI method are not simply failures to estimate dependence but natural differences in the definition of dependence.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study how MLM with uniform masking can learn useful linguistic structures and inductive biases for downstream tasks. Our work demonstrates that a substantial part of the performance gains of MLM pretraining cannot be attributed to taskspecific, cloze-like masks.  
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental details for Section 3.2 Our transformers have 2 layers and for each transformer block, the hidden size and the intermediate size are both 64. We finetune the models for 10 epochs and apply early stopping based on validation accuracy. We use Adam (Kingma and Ba, 2014) for optimization, using a learning rate of 1e −3 for pretraining and 1e −4 for finetuning. Experimental details for Section 5.1 Table 3 summarizes the dataset statistics of three real-world datasets we studied. For second stage pretraining, we update the BERT model for 10 epochs. Following the suggestion in Zhang et al. (2020), we finetune the pretrained BERT models for 400 steps, using a batch size of 16 and a learning rate of 1e −5 . We apply linear learning rate warmup for the first 10% of finetuning and linear learning rate decay for the rest. For SST-2 and AGNews, we average the results over 20 random trials. For Hyperpartisan, because the test set is small and the variation is larger, we average the results over 50 random trials and evaluate on the union the development set and the test set for more stable results.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental details for Section 5.2 We convert the annotated constituency parses using the Stanford CoreNLP package . We compute conditional MI and conditional PMI using the bert-base-cased model and run Gibbs sampling for 2000 steps. BERT's tokenization may split a word into multiple word pieces. We aggregate the dependencies between a word and multiple word pieces by taking the maximum value. We compute the PMI statistics and train the K&M model (Klein and Manning, 2004) on sentences of length ≤ 10 in the WSJ train split (section 2-21). For DMV, we train with the annotated POS tags using a public implementation released by (He et al., 2018). Results are averaged over three runs when applicable.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We conduct an additional experiment on the English Penn Treebank to verify that conditional MI can extract parses for sentences longer than ten words. To expedite experimentation, we subsample 200 out of 2416 sentences from the test split of English Penn Treebank and the average sentence length of our subsampled dataset is 24.1 words. When applicable, we average over three random seeds and report standard deviations. Table 4 presents the UUAS of conditional MI and other methods. We draw similar conclusions as in Section 5.2, observing that the parses drawn by conditional MI have higher quality than those of other baselines.   
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Proof of Proposition 2 We first recall our statement.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Proposition 2. Let λ k be the kth eigenvalue of AΣ ZZ A and λ XX,k+1 be the k+1th eigenvalue of Σ XX and V be the first k eigenvectors of Cov(X). Assuming λ k > λ XX,k+1 , we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where · op is the operator norm and tr(·) is the trace.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We will use the Davis-Kahan Theorem for our proof.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Theorem (Davis-Kahan (Stewart and Sun, 1990)). Let σ be the eigengap between the kth and the k+1th eigenvalue of two positive semidefinite symmetric matrices Σ and Σ . Also, let V and V be the first k eigenvectors of Σ and Σ respectively. Then we have,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
That is, we can bound the error in the subspace projection in terms of the matrix perturbation.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In our setting, we choose Σ = AΣ ZZ A + Σ XX and Σ = AΣ ZZ A . We know the eigengap of Σ is λ k because Σ only has k nonzero eigenvalues. By Weyl's inequality, the kth eigenvalue is at most perturbed by λ XX,k+1 , which is the k+1 eigenvalue of Σ XX . Let V be the top k eigenvectors of Σ and assuming λ k > λ XX,k+1 , we have,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Turning this operator norm bound into approximation bound, we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We use the fact that E X,Z AZ − X 2 2 = tr(Σ XX ) and Jensen's inequality to bound, E X AZ − X 2 ≤ tr(Σ XX ).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Few-shot learning-the ability to learn tasks with limited examples-is an important aspect of intelligence (Lake et al., 2015;Yogatama et al., 2019). Recent work shows that large neural language models can perform few-shot learning without finetuning (Radford et al., 2019;Brown et al., 2020). Specifically, GPT-3 (Brown et al., 2020) can perform numerous tasks when provided a few examples in a natural language prompt. For example, to perform sentiment analysis one can condition GPT-3 on a prompt such as: where the first two lines correspond to two training examples and the last line is a test example. To make predictions, the model predicts whether the subsequent token is more likely to be the word "Positive" or "Negative".
Calibrate Before Use: Improving Few-Shot Performance of Language Models
This style of few-shot "in-context" learning is interesting because it shows that the model can learn without parameter updates. And, more importantly, it has numerous practical advantages over the now-standard approach of finetuning (Radford et al., 2018;Devlin et al., 2019). First, it allows practitioners to "rapidly prototype" NLP models: changing the prompt immediately leads to a new model. Second, it provides a fully natural language interface to a machine learning model, which allows users-even those without technical expertise-to create NLP systems. Finally, since in-context learning reuses the same model for each task, it reduces memory requirements and system complexity when serving many different tasks.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
However, despite these promises, we show that GPT-3's accuracy can be highly unstable across different prompts (Section 3). A prompt contains three components: a format, a set of training examples, and a permutation (ordering) for those examples. We show that different choices for these factors can lead to highly different accuracies, e.g., changing the permutation of the training examples in a sentiment analysis prompt can change accuracy from near chance (54%) to near state-of-the-art (93%). This instability implies that GPT-3 users, who typically design prompts manually, cannot expect to consistently obtain good accuracy.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We next analyze what causes this instability. We identify three pitfalls of language models that lead them to be biased toward certain answers during few-shot learning. In particular, they suffer from majority label bias, recency bias, and common token bias (Section 4). The majority label and recency biases lead the model to predict training answers that appear frequently or near the end of the prompt. For example, a prompt that ends with a Negative training example may cause a bias towards the Negative class. On the other hand, the common token bias leads the model to prefer answers that are frequent in its pre-training data, e.g., it prefers "United States" over "Saint Lucia", which is likely suboptimal for the task of interest.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We identify that these biases typically result in a shift in the output distribution of the model. We can thus coun- . Few-shot learning can be highly unstable across different choices of the prompt. Above, we plot the mean accuracy (± one standard deviation) across different choices of the training examples for three different datasets and model sizes. We show that our method, contextual calibration, improves accuracy, reduces variance, and overall makes tools like GPT-3 more effective for end users. teract these biases by "calibrating" the output distribution. Concretely, we estimate the model's bias towards certain answers by feeding in a dummy test input that is content-free. In the prompt above for example, if we replace "Amazing." with the string "N/A", the model predicts 62% Positive. We then fit the calibration parameters so that the content-free input has uniform scores for each answer. This contextual calibration procedure provides a good setting of the calibration parameters without additional training data.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We test the effectiveness of contextual calibration on a range of tasks (Section 5). Contextual calibration consistently improves GPT-3 and GPT-2's accuracy (up to 30.0% absolute) across different choices of the prompt format and examples (e.g., Figure 1). It also makes the accuracy more stable across different prompts, thus mitigating the need for prompt engineering. Overall, contextual calibration is a simple method that makes language models better few-shot learners: it enables end users to obtain higher accuracy with considerably less effort.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Neural autoregressive language models (LMs) take as input a sequence of tokens and output a probability distribution over the next token. Large neural LMs can perform tasks in a zero-or few-shot manner using in-context learning (Radford et al., 2019;Brown et al., 2020). To do so, a natural language prompt is fed into the model. This prompt contains three components: a format, a set of training examples, and a permutation (ordering) of the training examples.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The prompt format is a template which consists of placeholders for the training and test example(s) and possibly a natural language description of the task. For example, the format of the prompt in Section 1 is a template with the style: "Input:" input "Sentiment:" label. Many alternate formats exist, e.g., one could frame the task as question answering.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The prompt's training examples are used to teach the LM how to solve the task at hand. The prompt from Section 1 consists of two training examples; we refer to this as "two-shot" learning. We also consider "zero-shot" learning, where no training examples are present.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
When training examples are used, they have a particular permutation, e.g., the "Subpar acting" example comes first in the prompt from Section 1. The permutation matters because neural language models update their hidden states in a left-to-right-fashion.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
To make predictions on an input, we slot it into the test placeholder and generate from the LM. For example, see the "Amazing." test example in the prompt from Section 1. For generation tasks, we generate greedily from the LM until it produces a newline character. For classification tasks, the probability for each class is given by the probability assigned to its associated label name, e.g., the words "Negative" and "Positive" for sentiment classification.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We use datasets for three tasks: text classification, fact retrieval, and information extraction. We use a fixed prompt format for each dataset unless otherwise specified. We show the format and examples from each dataset in Appendix B.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We study text classification using six datasets: sentiment analysis using SST-2 (Socher et al., 2013), 6-way question classification using TREC (Voorhees & Tice, 2000), textual entailment using 3-way CB  Fact Retrieval We evaluate fact retrieval with LAMA (Petroni et al., 2019). The dataset consists of knowledge base triples that are placed into templates with missing objects, e.g. "Obama was born in". We use these templates as our prompts, and remove the relations where the missing answer is not at the end of the template (left-to-right LMs cannot solve these). The answers are always single tokens, and we report average accuracy across all triples.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We consider information extraction using two slot filling datasets, ATIS (Hemphill et al., 1990) and MIT Movies trivia10k13 (Liu et al., 2012). We use two random slots for each dataset, airline and departure date for ATIS, and director name and movie genre for MIT Movies. The answer for both datasets is a span of text from the input, e.g., the ATIS airline task is to predict "american airlines" when given the sentence "list a flight on american airlines from toronto to san diego". We use Exact Match between the model's generated output and the ground-truth span as our evaluation metric.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We run our experiments on three sizes of GPT-3 (2.7B, 13B, and 175B parameters) as well as GPT-2 (1.5B parameters). We access GPT-3 using the OpenAI API. We release code to replicate our experiments. 1 1 https://www.github.com/tonyzhaozh/few-shot-learning
Calibrate Before Use: Improving Few-Shot Performance of Language Models
This section studies how GPT-3's accuracy changes as we vary each aspect of the prompt (training examples, permutation, format). We focus on a subset of the datasets to simplify our analysis; in Section 5 we show that our findings hold across all of the datasets we study.   Table 2 in Appendix A. This high importance on example order is in contrast to standard machine learning, where the ordering of examples during training is typically an afterthought.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The variance persists with more data and larger models. Adding more training examples into the prompt does not necessarily reduce the variance in accuracy. We sweep over the number of training examples for three different datasets in Figure 1 (red curves). The variance remains high even when we use 16 training examples. Moreover, adding more training examples can sometimes hurt accuracy (e.g., mean accuracy drops from 36.0% to 25.9% for DBPedia 0-shot to 1-shot). The variance in accuracy can also remain high when using larger models, e.g., the left of Figure 1. GPT-3's accuracy depends highly on prompt format. We next keep the set of training examples and permutations fixed but vary the prompt format. We focus on SST-2, and we manually design an additional 14 prompt formats. The formats include question-answer templates, conversationstyle templates, prompts that resemble Web pages, and variations on the label names (all formats available in Table 7 in Appendix B). The accuracy for ten of the formats is shown in Figure 3. We find that some of the formats are better than others on average. However, all of the formats still suffer from high variance across different training sets.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We next analyze why GPT-3's accuracy varies across different training examples, permutations, and prompt formats. Concretely, we show that the variance arises because LMs are biased towards outputting answers that are (1) frequent in the prompt (majority label bias), (2) towards the end of the prompt (recency bias), and (3) common in the pre-training data (common token bias).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We find that GPT-3 is biased towards answers that are frequent in the prompt. A trivial case is when a text classification prompt has a class imbalance, e.g., more Positive than Negative sentiment examples. This is demonstrated in the "unbalanced" region of Figure 4: when one class is more common, GPT-3 2.7B is heavily biased towards predicting that class. Since the SST-2 sentiment analysis dataset is balanced, this bias causes large accuracy degradations. The majority label bias also explains why we frequently observe a drop in accuracy when moving from 0-shot to 1-shot-we found that the drop is due to the model frequently repeating the class of the one training example.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The majority label bias also occurs for generation tasks. On the validation set for 4-shot LAMA with GPT-3 2.7B, 50.2% of the model predictions are a repeat of one of the four train-ing answers (the correct repeat rate is 24.7%). Overall, the majority label bias helps to explain why different choices for the training examples heavily influence GPT-3's accuracyit shifts the distribution of model predictions.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The model's majority label bias is aggravated by its recency bias: the tendency to repeat answers that appear towards the end of the prompt. The "balanced" region of Figure 4 demonstrates this. For instance, when two Negative examples appear at the end (P P N N), the model will heavily prefer the Negative class. Moreover, the recency bias can outweigh the majority label bias, e.g., the "P P P N" training set leads to nearly 90% of predictions being Negative, despite 3 4 of the training examples being Positive. Recency bias also affects generation tasks. For 4-shot LAMA, the training answers that are closer to the end of the prompt are more likely to be repeated by the model. Concretely, the model "overpredicts" the answer from the 1st, 2nd, 3rd, and 4th training example by 8.5%, 8.3%, 14.3%, and 16.1%, respectively. 2 Overall, recency bias helps to explain why the permutation of the training examples is important-the ordering of the examples heavily influences the distribution of the model predictions.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Common Token Bias Finally, we find that GPT-3 is biased towards outputting tokens that are common in its pretraining distribution, which is likely suboptimal for the distribution of answers on the downstream task. A simple case of this occurs for the LAMA fact retrieval dataset, where the model often predicts common entities such as "America" when the ground-truth answer is instead a rare entity.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
A more nuanced case of the common token bias occurs for text classification. Recall that the model makes predictions by generating the label name associated with each class. Because certain label names appear more frequently in the pre-training data, the model will be inherently biased towards predicting certain classes. For example, on DBPedia (a balanced 14-way topic classification dataset), GPT-3 predicts the "book" class 11× more often than the "artist" class. In fact, there is a moderate correlation (r = 0.67) between the frequency of a DBPedia label name and the rate at which GPT-3 predicts its class. 3 Overall, the common token bias helps to explain why the choice of label names is important, and why the model struggles on rare answers.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The Impact of Biases on Model Predictions We find that the end result of the above three biases is typically a simple shift in the model's output distribution. For example, Figure 5 visualizes this shift for a SST-2 sentiment prompt.  The prompt used in Figure 5 and the model's intrinsic biases cause it to frequently predict high confidence for the Positive class. Since the default 50% threshold is used to make predictions, this results in frequent false positives. Importantly, note that if we could optimally set the classification threshold (p(Positive) = 0.68 in this case), the classifier would be highly accurate (94% on the validation set).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Thus far, we have shown that GPT-3 is biased towards certain answers due to the prompt and the model's intrinsic biases. Here, we look to correct this by "calibrating" the model's output probabilities. 4 A common technique for adjusting output probabilities is to apply an affine transformation (Platt, 1999;Guo et al., 2017):
Calibrate Before Use: Improving Few-Shot Performance of Language Models
where a weight matrix W and a bias vector b are applied to the original probabilitiesp to get the new probabilities 3 The frequency of a token on the web is calculated using Google Ngrams https://books.google.com/ngrams. The predictions are from the 0-shot setting on the validation set. 4 The output of GPT-3 is biased (its outputs are shifted), similar to how measurement devices such as voltage meters or weighing scales are biased. Just like how these devices require "calibration before use", where the devices' outputs are scaled/zeroed-out, we hope to apply a similar calibration procedure to LMs. This goal is distinct from statistical calibration (Brier, 1950;Guo et al., 2017), i.e., aligning a model's confidence estimate with its true accuracy.q . 5 For classification tasks,p is the set of probabilities that are associated with each label name, renormalized to one. For generation tasks,p is the entire set of probabilities for the first token. 6 In this paper, we restrict the matrix W to be diagonal, known as vector scaling (Guo et al., 2017), to prevent the parameters from growing quadratically in the size ofp (which is ≈ 50, 000 for generation tasks).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
The main challenge in the zero-or few-shot setting is that we do not have data to learn W and b. We thus propose a novel data-free procedure to infer a good setting of these parameters. The key idea is that the model's bias towards certain answers can be estimated by feeding in a contentfree input such as the string "N/A". For example, consider the two-shot prompt:
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Input: Subpar acting. Sentiment: Negative Input: Beautiful film. Sentiment: Positive Input: N/A Sentiment:
Calibrate Before Use: Improving Few-Shot Performance of Language Models
where "N/A" serves as the test input. Ideally, GPT-3 would score this test input as 50% Positive and 50% Negative. However, the model's biases cause it to score this input as 61.8% Positive. Note that this error is contextual: a different choice of the training examples, permutation, and format will lead to different predictions for the content-free input.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We can correct this error by setting W and b so that the class scores for the content-free input are uniform. We first obtainp for the content-free input, denotedp cf . We then set W = diag(p cf ) −1 and b to the all-zero vector. 7 To make test predictions, we compute Wp + b and take the argmax.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Implementation Details This contextual calibration procedure adds trivial amounts of computational overhead and is implemented in a few lines of code (compute and savê p cf , adjust output probabilities). For the content-free input, many good choices exist, including "N/A", the empty string, and gibberish tokens. In all our experiments, we average the probabilities from three content-free inputs: "N/A", "[MASK]", and the empty string. 8 One could also craft the content-free input in a task-specific manner. We explore this for LAMA, where we replace the subject with the contentfree input, e.g., we use "N/A was born in" as the input. 5 This affine transformation is usually applied to the logits, i.e., prior to the softmax. However, we only have access to GPT-3's output probabilities in the OpenAI API. 6 We only calibrate the prediction of the first output token for generation tasks. This is reasonable because, for the tasks we consider, we found that the model's predictions are highly deterministic after generating the first token. 7 An alternate solution is to set b to −p cf and W to the identity. Empirically, this alternate solution yields higher accuracy for generation tasks (where the dimensionality ofp is large). The solution in the main text performs better for classification. 8 We found this simple ensemble to achieve the best results for AGNews, and we reuse it for all other datasets. See Section 5.2 for an ablation on the choice of content-free input.  Table 1. Contextual calibration improves accuracy across a range of tasks. We show the mean and standard deviation across different choices of the training examples (the prompt format is fixed). The LM column indicates the GPT-3 size (see Appendix A for GPT-2 results). The Baseline column shows the standard approach of greedy decoding (Brown et al., 2020) and Ours corresponds to greedy decoding after modifying the output probabilities using contextual calibration. We bold the better result of the baseline and ours. MIT-G, MIT-D, ATIS-A, and ATIS-D indicate the MIT Genre, MIT Director, ATIS Airline, and ATIS Departure Date datasets.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Here, we evaluate the effectiveness of contextual calibration across all of our datasets and LMs. We first use a fixed prompt format and select five different random sets of training examples, placing them in an arbitrary order in the prompt. We do not artificially balance the labels of the training examples for the classification tasks. We use the same sets of training examples for the baseline (standard decoding without calibration) and contextual calibration. We use labeling budgets of 0-8 examples; using more than 8shots causes the cost of querying the OpenAI API to become prohibitively expensive. Table 1 shows the results and Figure 1 in Section 1 plots the same data for a subset of the tasks.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Contextual calibration dramatically improves GPT-3's average and worst-case accuracy, by up to 30.0% absolute. These gains hold for both classification and generation tasks. Contextual calibration also sometimes allows GPT-3 2.7B to outperform the GPT-3 175B baseline-by up to 19.3%-despite being over 50x smaller.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Can Reduce Variance Across Training Sets Figure 6 plots the difference in the standard deviation between the baseline and contextual calibration for all tasks from Table 1. Contextual calibration reduces the variance considerably in a majority of cases, and it does not increase variance by much in the remaining cases.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Reduces Drop from 0-shot to 1-shot For the baseline, there are four cases where there is a drop in accuracy when 15 10 5 0 5
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Std Dev of Contextual Calibration Baseline Figure 6. Aside from improving mean accuracy, contextual calibration also reduces the standard deviation of accuracy across different choices of the training examples. We plot the difference in standard deviation between contextual calibration and the baseline from Table 1. moving from 0-shot to 1-shot (TREC, AGNews, DBpedia, SST-2). We attribute this drop to the majority label bias (see discussion in Section 4). Calibration removes this drop in three out of four cases.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Improves GPT-2 We also test GPT-2 1.5B (see Table 4 in Appendix A). We find that like GPT-3, GPT-2's accuracy also highly varies across different prompts. This suggests that the variance that we observe for few-shot in-context learning is a general problem for LMs. Second, contextual calibration works out-of-the-box for GPT-2-it improves the mean accuracy and reduces variance for most tasks.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Improves Accuracy Across Formats In our next set of experiments, we use a fixed set of training examples and vary the prompt format. We use the 15 prompt formats for SST-2 discussed in Section 3. We also create 15 prompt formats for each of three random relations in LAMA (P20, P159, P19) by using the paraphrases of the original LAMA templates generated by Jiang et al. (2020b). Figure 7 shows the results before and after calibration for SST-2, and Figure 9 in Appendix A show the results for LAMA. Contextual calibration improves the average and worst-case accuracy for both datasets, and reduces the variance for SST-2.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We finally conduct two analyses/ablations on contextual calibration. We first analyze how effective contextual calibration is at inferring a good setting of W. To do so, we compare its accuracy to an "oracle calibration" method that uses the validation set to find the best possible diagonal W. We evaluate this oracle on AGNews, and find that contextual calibration is surprisingly close to it (Figure 8).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We also study how the choice of content-free input affects accuracy. In Table 3 in Appendix A, we show the accuracy for SST-2 and AGNews for different choices of the content-free input. The choice of content-free input matters, however, many good choices exist. GPT-3 has high variance across different prompt formats; contextual calibration reduces this variance and improves mean accuracy. We show the mean accuracy (± standard deviation) over 15 different prompt formats for SST-2.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Does Calibration Eliminate the Need to Engineer Prompts? The motivation behind "prompt engineering" is that not all prompts lead to the same accuracy. Thus, one should tune the prompt's format and examples to achieve the best possible performance (Brown et al., 2020;Gao et al., 2020). Contextual calibration does not eliminate the need to engineer prompts, however, it does mitigate it: contextual calibration makes the accuracy of the best, average, and worst-case prompts more similar (and higher).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Should You Finetune in the Few-shot Setting? We use a fixed LM with no finetuning. As mentioned in Section 1, there are numerous reasons not to finetune: it enables rapid prototyping, provides a fully natural language interface, and is more efficient in terms of memory requirements and system complexity when serving many different tasks. Moreover, like in-context learning without contextual calibration, finetuning can be unstable in the few-shot setting (Schick & Schütze, 2021). Nevertheless, if these disadvantages are acceptable or avoidable, finetuning can improve accuracy over in-context learning in some cases (Schick & Schütze, 2020;Gao et al., 2020). An interesting direction for future work is to study the interplay between contextual calibration and finetuning, e.g., does contextual calibration alleviate the need to finetune, or vice versa?
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We identify failures when LMs are used for in-context learning (e.g., recency bias). Past work identifies similar failures when LMs are used for text generation. For example, neural LMs often repeat themselves (Holtzman et al., 2020), suffer from overconfidence (Braverman et al., 2020;Jiang et al., 2020a), suffer from recency bias (Khandelwal et al., 2018;Ravfogel et al., 2019), and prefer generic responses instead of rare text (Li et al., 2016;Logan et al., 2019). Past work mitigates these degeneracies by modifying the model's output probabilities or generation schemes, e.g., explicitly preventing repetitions (Paulus et al., 2018) or using sampling instead of greedy decoding (Holtzman et al., 2020).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We show that few-shot learning can be highly volatile across different choices of the prompt. Through a detailed analysis, we identify that this volatility arises from biases in LMs, e.g., their tendency to output recent or common tokens. We use these insights to develop contextual calibration-a simple procedure to adjust the model's output probabilities-which improves accuracy, reduces variance, and overall makes tools like GPT-3 more effective for end users.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Looking at the bigger picture, our results inspire two future research directions in few-shot learning for NLP. First, on the methods side, we show that good few-shot learning requires attention to detail: small but non-trivial decisions such as calibration can greatly influence results. This makes it difficult to correctly develop and compare new methods (e.g., pretraining schemes or model architectures). We thus hope to make other few-shot learning methods more robust, and also expand our techniques to cover a wider ranger of tasks (e.g., calibration for open-ended generation). Second, on the analysis side, our results highlight the need to understand what GPT-3 learns from the prompt. The model has an impressive ability to improve with more training examples, however, we show that the model learns some superficial patterns such as repetition of common answers. We hope to better understand and analyze the dynamics of in-context learning in future work. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and  Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. In NeurIPS, 2015.   Table 3. We show the accuracy for 1-shot SST-2 and 0-shot AG-News over different choices for the content-free input. The choice of content-free input matters, however, many good choices exist. The token '' indicates the empty string. Recall that in our experiments, we ensemble over N/A, [MASK], and the empty string. Figure 9 shows how GPT-3 accuracy changes as the prompt format is varied for LAMA, with and without calibration. Table 4 shows the effect of calibration for GPT-2.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Tables 5 and 6 show the default prompt format used for all tasks. Table 7 shows the 15 different formats used when studying the effect of prompt format for SST-2.  True, False Table 5. The prompts used for text classification. We show one training example per task for illustration purposes. The right column shows the label names (to make predictions, we check the LM's probability for these tokens).
Larger language models do in-context learning differently
Language models can perform a range of downstream NLP tasks via in-context learning (ICL), where models are given a few exemplars of input-label pairs as part of the prompt before performing the task on an unseen example (Brown et al., 2020, inter alia). To successfully perform ICL, models can (a) mostly use semantic prior knowledge to predict labels while following the format of in-context exemplars (e.g., seeing "positive sentiment" and "negative sentiment" as labels and performing sentiment analysis using prior knowledge) and/or (b) learn the input-label mappings from the presented exemplars (e.g., finding a pattern that positive reviews should be mapped to one label, and negative reviews should be mapped to a different label). Prior work on which of these factors drives performance is mixed. For instance, although Min et al. (2022b) showed that presenting random ground truth mappings in-context does not substantially affect performance (suggesting that models primarily rely on semantic prior knowledge), other work has shown that transformers in simple settings (without language modeling pretraining) implement learning algorithms such as ridge regression and gradient descent (Akyürek et al., 2023;von Oswald et al., 2022;Dai et al., 2022). Figure 1: An overview of flipped-label ICL and semantically-unrelated label ICL (SUL-ICL), compared with regular ICL. Flipped-label ICL uses flipped targets, forcing the model override semantic priors in order to follow the in-context exemplars. SUL-ICL uses targets that are not semantically related to the task, which means that models must learn input-label mappings in order to perform the task because they can no longer rely on the semantics of natural language targets.
Larger language models do in-context learning differently
In this paper, we study how these two factors-semantic priors and input-label mappings-interact in several experimental settings (see Figure 1 for an example of each setting):
Larger language models do in-context learning differently
1. In regular ICL, both semantic priors and input-label mappings can allow the model to perform in-context learning successfully.
Larger language models do in-context learning differently
2. In flipped-label ICL, all labels in the exemplars are flipped, which means that semantic prior knowledge and input-label mappings disagree. Labels for the evaluation set stay the same, so for binary classification tasks, performing better than 50% accuracy in this setting means that the model is unable to override semantic priors, and performing below 50% accuracy means that the model is able to learn input-label mappings and override semantic priors.
Larger language models do in-context learning differently
3. In semantically-unrelated label ICL (SUL-ICL), the labels are semantically unrelated to the task (e.g., for sentiment analysis, we use "foo/bar" instead of "negative/positive"). Since the semantic priors from labels are removed, the model can only perform ICL by using input-label mappings.
Larger language models do in-context learning differently
We run experiments in these settings spanning multiple model families with varying sizes, training data, and instruction tuning InstructGPT,Codex,PaLM, in order to analyze the interplay between semantic priors and input-label mappings, paying special attention to how results change with respect to model scale. First, we examine flipped-label ICL, where we find that small models do not change their predictions when seeing flipped labels, but large models can flip their predictions to follow flipped exemplars (Section 3). This means that the ability to override semantic priors with input-label mappings emerges with model scale, which should not be taken for granted because larger models presumably have stronger priors that are more challenging to override.
Larger language models do in-context learning differently
Second, we compare the SUL-ICL setting to regular ICL (Section 4). We find that small language models experience a large performance drop when semantic priors are removed, whereas large language models can perform the task well even without semantic priors from the labels. For some datasets, doing better than random in the SUL-ICL setting required substantial scaling (e.g., only PaLM-540B achieves above-random performance). We also found this to be true for high-dimensional linear classification tasks (Section 6). This means that learning input-label mappings without being given semantic priors is also an emergent ability of large language models for those tasks.
Larger language models do in-context learning differently
Finally, we study the effect of instruction tuning (Min et al., 2022a;Wei et al., 2022a; on ICL abilities (Section 5). We find that instruction-tuned models achieve better performance than pretraining-only models on SUL-ICL settings, which means that instruction tuning increases the model's ability to learn input-label mappings. On the other hand, we also see that instruction-tuned models are more reluctant to follow flipped labels, which means that instruction tuning decreases the model's ability to override semantic priors more than it increases its ability to learn input-label mappings. Overall, our work aims to shed light on the interaction between semantic prior knowledge and input-label mappings while considering the effects of scaling and instruction tuning.
Larger language models do in-context learning differently
We experiment on seven NLP tasks that have been widely used in the literature (Kim, 2014;Wang et al., 2018;2019). These evaluation tasks and an example prompt/target pair are shown in Figure 9 in the Appendix; additional dataset details are described in Appendix A. The seven tasks are: Sentiment Analysis (Socher et al., 2013, SST-2); Subjective/Objective Sentence Classification (Conneau & Kiela, 2018, SUBJ); Question Classification (Li & Roth, 2002, TREC); Duplicated-Question Recognition (Chen et al., 2017;Wang et al., 2018, QQP); Textual Entailment Recognition (Dagan et al., 2006; Table 1: Models used in this paper.
Larger language models do in-context learning differently
We perform experiments on five language model families as shown in Table 1. We use three families of OpenAI language models accessed via the OpenAI API: GPT-3 (Brown et al., 2020), In-structGPT (Ouyang et al., 2022), and Codex (Chen et al., 2021). For GPT-3 models, ada, babbage, curie, and davinci seem to correspond to the following model sizes: 350M, 1.3B, 6.7B, and 175B (Gao et al., 2021). For InstructGPT and Codex, however, it is not publicly known what the sizes of these language models are, but we assume that they are in increasing model scale for some scaling factor.
Larger language models do in-context learning differently
We also experiment on three different sizes of PaLM (Chowdhery et al., 2022) (8B, 62B, and 540B) and their instruction-tuned variants (Chung et al., 2022, Flan-PaLM). PaLM models have the same training data and protocol and only differ by model size (Chowdhery et al., 2022), which provides an additional data point for the effect of scaling model size specifically.
Larger language models do in-context learning differently
As additional experimental details, we follow the prior literature on in-context learning and use a different set of few-shot exemplars for each inference example (Brown et al., 2020;Chowdhery et al., 2022;Wang et al., 2023, inter alia). By default, we use k = 16 in-context exemplars per class, though we also experiment with varying number of exemplars in Section 4 and Appendix C.2. We also use the "Input/Output" template for prompts shown in Figure 9, with ablations for input format shown in Appendix B.4 and Appendix B.5, and the semantically-unrelated "Foo"/"Bar" targets as shown in Figure 9 (ablations for target type are shown in Appendix B.3). Finally, to reduce inference costs, we use 100 randomly sampled evaluation examples per dataset, as it is more beneficial to experiment with a more-diverse range of datasets and model families than it is to include more evaluation examples per dataset, and our research questions depend more on general behaviors than on small performance deltas (note that all y-axes in our plots go from 0-100).
Larger language models do in-context learning differently
To what extent are models able to override semantic priors from pretraining in favor of input-label mappings presented in-context? When presented in-context exemplars with flipped labels, models that are able to override priors and learn input-label mappings in-context should experience a decrease in performance to below random guessing (assuming ground-truth evaluation labels are not flipped).
Larger language models do in-context learning differently
To test this, we randomly flip an increasing proportion of labels for in-context exemplars. As shown in Figure 1, for example, 100% flipped labels for the SST-2 dataset would mean that all exemplars labeled as "positive" will now be labeled as "negative," and all exemplars that were labeled as "negative" will now be labeled as "positive." Similarly, 50% flipped labels is equivalent to random labels, as we use binary classification datasets (we exclude TREC from this experiment since it has six classes). We do not change the labels of the evaluation examples, so a perfect model that can override semantic priors should achieve 0% accuracy when presented with 100% flipped labels.
Larger language models do in-context learning differently
Figure 2 shows average model performance for each of the model families across all tasks with respect to the proportion of labels that are flipped (per-dataset results are shown in Figure 16). We see that there is a similar trend across all model families-at 0% flipped labels (i.e., no labels are changed), larger models have better performance than small models, which is expected since larger models should be more capable than smaller models. As more and more labels are flipped, however, the performance of small models remains relatively flat and often does not dip below random guessing, even when 100% of labels are flipped. Large models, on the other hand, experience performance drops to well-below random guessing (e.g,. text-davinci-002 performance drops from 90.3% with 0% flipped labels to just 22.5% with 100% flipped labels). Note that GPT-3 models can remove semantic priors (i.e., perform at guessing accuracy) but cannot override them (i.e., perform significantly worse than guessing), even when presented with 100% flipped labels. For this reason, we consider all GPT-3 models to be "small" models because they all behave similarly to each other this way.
Larger language models do in-context learning differently
These results indicate that large models can override prior knowledge from pretraining with inputlabel mappings presented in-context. Small models, on the other hand, do not flip their predictions and thus are unable to override semantic priors (consistent with Min et al. (2022b)). Because this ability to override prior knowledge with input-label mappings only appears in large models, we conclude that it is an emergent phenomena unlocked by model scaling (Wei et al., 2022b). The ability to override semantic priors when presented with flipped in-context exemplar labels emerges with model scale. Smaller models cannot flip predictions to follow flipped labels (performance only decreases slightly), while larger models can do so (performance decreases to well below 50%). Ground truth labels for evaluation examples are not flipped, so if a model learns to follow flipped labels, its accuracy should be below 50% when more than 50% of labels are flipped.
Larger language models do in-context learning differently
For example, a model with 80% accuracy at 0% flipped labels will have 20% accuracy at 100% flipped labels if it learns to perfectly flip its predictions. Accuracy is computed over 100 evaluation examples per dataset with k = 16 in-context exemplars per class and averaged across all datasets. : Small models rely more on semantic priors than large models do, as performance decreases more for small models than for large models when using semantically-unrelated targets instead of natural language targets. For each plot, models are shown in order of increasing model size (e.g., for GPT-3 models, a is smaller than b, which is smaller than c). We use k = 16 in-context exemplars per class, and accuracy is calculated over 100 evaluation examples per dataset and averaged across all datasets. A per-dataset version of this figure is shown in Figure 17 in the Appendix.
Larger language models do in-context learning differently
Another way to examine how much models use semantic priors from pretraining versus input-label mappings is to replace natural language targets with semantically-unrelated targets. If a model mostly relies on semantic priors for in-context learning, then its performance should significantly decrease after this change, since it will no longer be able to use the semantic meanings of targets to make predictions. A model that learns input-label mappings in-context, on the other hand, would be able to learn these semantically-unrelated mappings and should not experience a major drop in performance.
Larger language models do in-context learning differently
We use an experimental setup that we call Semantically-Unrelated Label In-Context Learning (SUL-ICL) to test model behavior in these scenarios. 2 In this setup, all natural language targets are swapped with semantically-unrelated targets (we use "Foo" and "Bar" by default, although we get similar results with other semantically-unrelated targets-see Appendix B.3). For example, SUL-ICL relabels examples labeled as "negative" as "foo" and examples labeled as "positive" as "bar" for the SST-2 dataset ( Figure 1). We then examine model performance in the SUL-ICL setup (in Appendix B, we investigate other aspects of the SUL-ICL setup such as remapping inputs, formatting prompts differently, changing target types, and using out-of-distribution datasets).
Larger language models do in-context learning differently
In Figure 3, we examine average model accuracy across all tasks on the SUL-ICL setup compared with a regular in-context learning setup (per-dataset results are shown in Figure 17). As expected, we see that increasing model scale improves performance for both regular in-context learning and SUL-ICL. The performance drop from regular ICL to SUL-ICL, however, is far more interesting. We find that using semantically-unrelated targets results in a greater performance drop from using natural language targets for small models compared with large models. Because small models are heavily affected when the semantic meaning of targets is removed, we conclude that they primarily rely on the semantic meaning of targets for in-context learning rather than learn the presented input-label mappings. Large models, on the other hand, experience very small performance drops after this change, indicating that they have the ability to learn input-label mappings in-context when the semantic nature of targets is removed. 3 Hence, the ability to learn input-label mappings in-context without being given semantic priors can also be seen as an emergent ability of model scale.
Larger language models do in-context learning differently
2 Rong (2021) previously evaluated a setup where they replaced natural language targets with nonalphanumeric characters; our paper uses a similar setup and investigates with more-extensive experimentation.
Larger language models do in-context learning differently
3 For the reasons stated in Section 3, we consider davinci to be a small model. We next analyze how models perform on a SUL-ICL setup when presented with an increasing number of in-context exemplars, and we show these data in Figure 4 (per-dataset results are shown in Figure 18). We find that for the three model families that we tested, 4 including more in-context exemplars results in a greater performance improvement for large models than it does for small models. This indicates that large models are better at learning from in-context exemplars than small models are, implying that large models are more capable of using the additional input-label mappings presented in context to better learn the correct relationships between inputs and labels.
Larger language models do in-context learning differently
Finally, looking at the per-dataset performance reveals how the ability to perform some benchmark tasks in the SUL-ICL setting emerges with scale. In Figure 5, we highlight two tasks (RTE and ETHOS) that seem particularly emergent in the SUL-ICL setting by plotting model performance at each model size for Codex and PaLM models ( Figure 18 shows how each model performs for each dataset). We see that performance on the RTE dataset is around random for PaLM-8B and PaLM-62B, yet increases to well above random for PaLM-540B. Similarly, the performance on both the RTE and ETHOS datasets is around random for code-cushman-001 and code-davinci-001, then jumps to 80%+ for code-davinci-002. PaLM models seem to emerge earlier on the ETHOS dataset, however, as the performance spikes when scaling from PaLM-8B to PaLM-62B. For many datasets that do not show emergence, even small models can outperform random guessing without many in-context exemplars (e.g., on SST-2, TREC, SUBJ, FP). These results show another example of how, for some tasks, the ability to learn input-label mappings in-context without being given semantic priors is only emergent in large-enough language models. Flan-PaLM-540B PaLM-540B Flan-PaLM-62B PaLM-62B Flan-PaLM-8B PaLM-8B Random Figure 6: Instruction-tuned language models are better at learning input-label mappings than pretraining-only language models are. Accuracy is calculated using 100 evaluation examples per dataset and averaged across six datasets. A per-dataset version of this figure is shown in Figure 19 in the Appendix.
Larger language models do in-context learning differently
A popular technique for improving the performance of pretrained language models is to finetune them on a collection of NLP tasks phrased as instructions, with few-shot exemplars as part of the finetuning inputs (Min et al., 2022a;Wei et al., 2022a;Longpre et al., 2023). Since instruction tuning uses natural language targets, however, an open question is whether it improves the ability to learn input-label mappings in-context or whether it strengthens the ability to recognize and apply semantic priors, as both would lead to an improvement in performance on standard ICL tasks.
Larger language models do in-context learning differently
To study this, we run the same experiments from Section 3 and Section 4, and we now compare PaLM models to their instruction-tuned versions (Chung et al., 2022, Flan-PaLM). We do not compare InstructGPT against GPT-3 models in this experiment because we cannot determine if the only difference between these model families is instruction tuning (e.g., we do not even know if the base models are the same). Figure 6 shows the average model performance across all datasets with respect to the number of in-context exemplars for PaLM and Flan-PaLM models. We see that Flan-PaLM performs better in the SUL-ICL setting than PaLM does, an effect that is most prominent in small models, as Flan-PaLM-8B outperforms PaLM-8B by 9.6%, almost catching up to PaLM-62B. This trend suggests that instruction tuning strengthens the ability to learn input-label mappings (an expected outcome).
Larger language models do in-context learning differently
In Figure 7, we show model performance with respect to the proportion of labels that are flipped for each PaLM and Flan-PaLM model. We find that, compared to pretraining-only models, instructiontuned models are worse at flipping their predictions-Flan-PaLM models were unable to override their semantics more than what could be achieved by random guessing, even with 100% flipped labels. Standard PaLM models, on the other hand, could achieve as low as 31% accuracy when presented with 100% flipped labels. These results indicate that instruction tuning either increases the extent to which models rely on semantic priors when they are available or gives models more semantic priors, as instruction-tuned models are less capable of flipping their natural language targets to follow the flipped labels that were presented. Combined with the result from Figure Figure 20 in the Appendix.
Larger language models do in-context learning differently
In addition to the natural language reasoning abilities that we studied throughout the rest of the paper, we also seek to learn about how model scale affects the ability to perform other tasks. Specifically, we look at the linear classification task, where large models should perform better than small models (especially at high dimensions) if their greater capacity to learn input-label mappings as shown in Section 4 also holds for non-natural-language tasks.  Figure 21 in the Appendix.
Larger language models do in-context learning differently
To analyze this, we create N -dimensional linear classification datasets and examine model behavior with respect to the number of dimensions in the SUL-ICL setup. In these datasets, we provide k N -dimensional points above a threshold and k N -dimensional points below that same threshold as in-context exemplars, and the model must determine whether an N -dimensional evaluation point is above or below the threshold (we do not tell the model the equation or the threshold). When selecting random N -dimensional points, we use random integers between 1 and 1000 for each coordinate value. Algorithm 1 in the Appendix shows the precise dataset generation procedure.
Larger language models do in-context learning differently
In Figure 8, we show Codex model performance on N = 16 dimensional linear classification (per-dimension results are shown in Figure 21 in the Appendix). We find that the largest model outperforms random guessing by 19% on this task, while smaller models cannot outperform random guessing by more than 9%. These results suggest that there exists some scaling factor that allows large-enough language models to perform high-dimensional linear classification.
Larger language models do in-context learning differently
There has been a growing body of work on in-context learning that suggests that good performance is primarily driven by semantic priors and other factors such formatting and inducing intermediate token generation. For instance, Min et al. (2022b) showed the surprising result that using random groundtruth labels in exemplars barely hurts performance, suggesting that performance is instead mainly driven by the label space, distribution of input text, and overall format of the sequence. Along the same lines, Madaan & Yazdanbakhsh (2022) and  show that for chain-of-thought prompting (Wei et al., 2022c), logically-incorrect prompts do not hurt performance on multi-step reasoning tasks. On a theoretical level, Xie et al. (2022) provide an explanation of in-context learning in which transformers infer tasks from exemplars because they are trained to infer latent concepts during pretraining, and prior knowledge obtained from pretraining data can then be applied to incontext examples. Finally, Reynolds & McDonell (2021) showed that clever zero-shot prompts can outperform few-shot prompts, which implies that some NLP tasks benefit more from leveraging the model's existing knowledge than from learning about the task from in-context exemplars. In this paper, we do not contest the claim that language models can benefit greatly from semantic prior knowledge-our results instead add nuance to the understanding of ICL by showing that, when semantic prior knowledge is not available, large-enough language models can still do ICL using input-label mappings. Our experiments are consistent with Min et al. (2022b) for models scaling up to davinci, and we show that learning input-label mappings only emerges with larger models (e.g., PaLM-540B, text-davinci-002, and code-davinci-002).
Larger language models do in-context learning differently
Other recent work has suggested to some degree that language models can actually learn input-label mappings from exemplars given in-context, which is a more-attractive ability than using semantic priors because it means that the model would be able to perform a wide range of tasks even if those tasks are not seen in or even contradict pretraining data. For instance, transformers trained from scratch can perform in-context learning on linear-regression datasets with performance that is comparable to the least-squares estimator (Garg et al., 2022), and recent work has shown that transformers can do so by implementing standard learning algorithms such as ridge regression and gradient descent (Akyürek et al., 2023;von Oswald et al., 2022;Dai et al., 2022). In the natural language setting, Webson & Pavlick (2022) showed that language models learn just as fast with irrelevant or misleading prompts during finetuning or prompt-tuning. Our work makes similar claims about the ability for language models to learn tasks via input-label mappings only, though it differs crucially in that we observe frozen pretrained transformers without any additional learning.
Larger language models do in-context learning differently
In this paper we have also focused on the effect of scaling on in-context learning, which relates to a nascent body of work showing that scaling language models leads to qualitatively-different behavior (Ganguli et al., 2022;Wei et al., 2022b;Srivastava et al., 2022). For instance, it has recently been shown that scaling up language models can allow them to perform a variety of challenging tasks that require reasoning (Wei et al., 2022c;Chowdhery et al., 2022;Kojima et al., 2022;. Our experimental findings on the flipped-label ICL setup show that language models can learn input-label mappings even when the input-label mapping contradicts the semantic meaning of the label, demonstrating another type of symbolic reasoning where language models can learn input-label mappings regardless of the actual identity of the labels. Although we have shown that this behavior is emergent with respect to model scale, the investigation of why scaling unlocks such behaviors (Xie et al., 2022;Chan et al., 2022) is still an open question that we leave for future work.
Larger language models do in-context learning differently
In this paper, we examined the extent to which language models learn in-context by utilizing prior knowledge learned during pretraining versus input-label mappings presented in-context. We first showed that large language models can learn to override semantic priors when presented with enough flipped labels (i.e., input-label mappings that contradict prior knowledge), and that this ability emerges with model scale. We then created an experimental setup that we call Semantically-Unrelated Label In-Context Learning (SUL-ICL) which removes semantic meaning from labels by replacing natural language targets with semantically-unrelated targets. Successfully doing ICL in the SUL-ICL setup is another emergent ability of model scale. Additionally, we analyzed instructiontuned language models and found that instruction tuning improves the capacity to learn input-label mappings but also strengthens semantic priors. Finally, we examined language model performance on linear classification tasks, finding that successfully performing high-dimensional linear classification emerges with model scale. These results underscore how the in-context learning behavior of language models can change depending on the scale of the language model, and that larger language models have an emergent ability to map inputs to many types of labels, a form of true symbolic reasoning in which input-label mappings can be learned for arbitrary symbols.   Figure 9: Prompt formatting for all datasets. We use varying number of in-context exemplars per class in our experiments, but we show one in-context exemplar per class in this figure for conciseness.
Larger language models do in-context learning differently
A DATASET CREATION Figure 9 shows example prompts with inputs and targets from each dataset that we tested (full prompt examples for the seven datasets used in the main paper are shown in Appendix D). For each natural language task, we use the version of the dataset that is available on HuggingFace (Lhoest et al., 2021), and we randomly choose in-context exemplars from the training set and evaluation examples from the validation set, following Min et al. (2022b). For datasets without existing train/validation splits, we use a random 80/20 train/validation split.
Larger language models do in-context learning differently
For the FP dataset, we use the sentences_allagree subset. We also use the binary subset of the ETHOS dataset. Additionally, we use the six coarse labels for the TREC dataset.
Larger language models do in-context learning differently
A natural question about the SUL-ICL setup is whether it is more difficult than the flipped labels setup. Intuitively, one would expect that the SUL-ICL setting is easier than the flipped-label setting because while the model needs to override contradiction labels in the flipped-label setting, it does not need to do so in the SUL-ICL setting.
Larger language models do in-context learning differently
We investigate this question by analyzing model outputs in the SUL-ICL and flipped-label settings. We use the same results from Section 4 to show model performance in the SUL-ICL setting (specifically, we use the per-dataset results from Figure 3). For the flipped-label setting, we use model outputs and In Figure 10, we compare model performance in the SUL-ICL setting with model performance in the flipped-label setting. We find that performance is almost always higher in the SUL-ICL setting than it is in the flipped-label setting. In particular, medium-sized models perform much worse in the flipped-label setting than they do in the SUL-ICL setting, with performance differing by up to 74% (text-curie-001 on SST-2). Small and large models, on the other hand, see smaller but still significant performance drops when using flipped-labels compared to SUL-ICL labels.
Larger language models do in-context learning differently
These results suggest that the SUL-ICL setting is indeed easier than the flipped-label setting, and that this trend is particularly true for medium-sized models. Small and large models are still affected by the setting, though perhaps to a lesser degree because small models often do not outperform guessing anyway and large models are more capable of overriding semantic priors (i.e., perform better in flipped-label settings). This may be an indication that the flipped-label setting's requirement of overriding priors is more difficult than learning mappings to semantically-unrelated labels.  
Larger language models do in-context learning differently
As a sanity check, we want to show that even large models cannot succeed in the SUL-ICL setup in all environments. For example, when presented with semantically-meaningless inputs, even the largest models should not be able to perform the task because there are no longer any semantics that can be used to learn what the task is (the SUL-ICL setup already removes semantics from labels).
Larger language models do in-context learning differently
To show this, we remap an increasing percentage of input words to other input words at a per-prompt level. We first compile the set of all words used in the inputs for a given prompt, and we then map a randomly selected proportion of those words to other randomly selected words, thereby reducing the semantic meaningfulness of inputs. In this setup, 0% remapped words means that no input words have been changed (i.e., regular SUL-ICL), and 100% remapped words means that every input word has been remapped (i.e., inputs are now a concatenation of random words from other inputs, making them essentially meaningless). An example of this procedure is shown in Figure 11.
Larger language models do in-context learning differently
In Figure 12, we show model performance with respect to the proportion of remapped words. We find that small models generally approach guessing performance at 25%-50% remapped words, while large models see linear performance drops, usually reaching guessing accuracy at 75%-100% remapped words. At 100% remapped input words, even the largest models (code-davinci-002 and PaLM-540B) are unable to beat random guessing on almost all datasets. 6
Larger language models do in-context learning differently
These results suggest that larger models are more robust to input noise, but only to some extent because they still cannot consistently learning the required mappings to unscramble the words when a large enough proportion of words have been remapped. Indeed, 100% remapped words is most likely too difficult of a task to learn for these models, as the only way to solve the task reliably would be to unscramble most mapped words back to their original words, which would be difficult for even a human to do given the large number of input words per prompt. Reversed targets such as (0/1) and (1/0) means that, for example, if (0/1) assigns 0 = negative and 1 = positive for sentiment analysis, then (1/0) assigns 1 = negative and 0 = positive. "Natural language" indicates that natural language targets are used (i.e., regular ICL). Accuracy is calculated over 250 evaluation examples inputted to code-davinci-002 with k = 16 in-context exemplars per class.
Larger language models do in-context learning differently
In Section 4, we showed that large language models can learn input-label mappings for one set of semantically-unrelated targets ("Foo" and "Bar"), but can they still learn these mappings for other types of semantically-unrelated targets? To test this, we evaluate models in the SUL-ICL setup using varying semantically-unrelated targets in addition to Foo/Bar targets: numerical targets, alphabetical targets, and fruit targets. 7 For each target format, we also reverse the targets (e.g., 0 → 1 and 1 → 0) to verify that labels can be interchanged, at least within each set of labels. We experiment using natural language targets (i.e., regular ICL) for comparison. Figure 13 shows model performance for each target type used. 8 We see that, in most cases, model performance stays relatively constant with respect to the target that is used. Additionally, there is no consistent difference between using natural language targets and using semantically-unrelated targets, 7 While numerical targets such as "0" and "1" may have some semantic meaning in that "0" is often correlated with "negative" and "1" is often correlated with positive, our experiments show that this is not significant since reversing the 0/1 labels does not always hurt performance to the extent that the flipped-labels setting does. 8  which may suggest that given a large enough model and enough in-context exemplars, input-label mappings alone are enough to drive model performance. These findings demonstrate that for many types of semantically-unrelated targets, large models can still learn input-label mappings.
Larger language models do in-context learning differently
We can also see that some tasks are too difficult for the model to learn, regardless of whether natural language targets or SUL-ICL targets were used. Specifically, the model cannot significantly outperform random guessing on the QNLI and WSC datasets for any target type, and for this reason, we remove the QNLI and WSC datasets from other experiments.
Larger language models do in-context learning differently
Can any prompt format be used for SUL-ICL as long as it clearly presents inputs and their respective labels? We explore this question by comparing the default Input/Output prompt template shown in Figure 9 with five additional formats, where [input] and [label] stand for the inputs and labels respectively (templates are shown in quotes). In Figure 14, we show model performance for each of the input formats that we tested. We find that no input format is significantly better than any other input format, as the mean accuracy across all NLP tasks for all input formats (which ranges from 77.9% to 87.7%) is within ±6.3% of the mean (84.2%). These findings suggest that SUL-ICL may work across many simple formats that present input-label mappings, which may indicate that a factor to succeed in a SUL-ICL setup is that prompt templates should show a clear mapping between an input and its respective label. 
Larger language models do in-context learning differently
In Appendix B.4, we did not test any prompt templates that include semantic information that is relevant to the task (e.g., using "Review: [input] \n Sentiment: [label]" for SST-2). We thus want to explore this setting in order to investigate whether models use semantic priors more or input-label mappings more they are given a semantically-relevant template.
Larger language models do in-context learning differently
We investigate this by using semantic prompt formats from Zhao et al. (2021) in the SUL-ICL setting and compare these results to the results from using our default "Input/Output" prompt template. We run these experiments on the SST-2, TREC, and RTE datasets-the datasets in our paper that intersect with those used in Zhao et al. (2021)-and we evaluate on the Codex model family.
Larger language models do in-context learning differently
As shown in Figure 15, we find that the smallest Codex model (code-cushman-001) sees performance drop across all tested datasets when switching to semantically-relevant prompt templates. The largest Codex model (code-davinci-002), on the other hand, is relatively unaffected by the change, while the middle Codex model (code-davinci-001) experiences performance changes that vary across datasets.
Larger language models do in-context learning differently
These results suggest that small models get worse at learning input-label mappings when presented with semantically-relevant prompts, perhaps because seeing semantically-charged words encourages the model to try to utilize semantic priors rather than learn input-label mappings in-context. We also see that large models may be more robust to these inputs-their performance being unaffected by the change indicates that despite seeing the semantic prompt templates, they are still able to learn the semantically-unrelated input-label mappings in-context. Tran et al. (2022) previously showed that model scale improves robustness to out-of-distribution (OOD) datasets where the input distribution of text for a given task changes. We aim to analyze whether this behavior is present in the SUL-ICL setting. In this experiment, we combine examples from SST-2 and the Rotten Tomatoes dataset (Pang & Lee, 2005, RT)-which is also a sentiment analysis dataset-and prompt the model with in-context exemplars from one dataset while evaluating it on examples from the other dataset. We then test InstructGPT models in a SUL-ICL environment using these varied input distributions.
Larger language models do in-context learning differently
As shown in Table 2, we see that small models (e.g., text-ada-001 and text-babbage-001) suffer from significant performance drops of up to 36% when OOD datasets are used. Large models (e.g., text-curie-001 and text-davinci-001), on the other hand, do not suffer from these drops, with text-curie-001 only seeing a 4% decrease in accuracy and text-davinci-001 seeing no significant change in accuracy. These results suggest that robustness to OOD datasets emerges with scale in the SUL-ICL setup, implying that this behavior could be related to the presentation of input-label mappings (something that both regular in-context learning and SUL-ICL share) and not necessarily the availability of semantic targets (which SUL-ICL lacks).  
Larger language models do in-context learning differently
Here, we present per-dataset results for each model family after flipping labels for in-context exemplars, as described in Section 3. In Figure 16, we plot model accuracy with respect to the proportion of labels that we flip for each dataset and for each model family. We exclude the RTE dataset for PaLM models because the prompts from this dataset at k = 16 in-context exemplars per class consistently exceed the maximum-allowable context length.
Larger language models do in-context learning differently
For many model families, we see that large models have better performance than small models do at 0% flipped labels, but that flipping more labels results in performance drops for large models but not for small models. This trend is especially true for the InstructGPT model family and, to a lesser extent, the Codex and PaLM model families. The base GPT-3 model family, on the other hand, does not see this trend happen for most tasks, which is likely due to the fact that even the large models in this model family have trouble outperforming random guessing for many tasks. For example, the largest GPT-3 model (davinci) only achieves guessing accuracy on the QQP and RTE datasets, while the largest InstructGPT and Codex models both achieve 80%+ accuracy on these two tasks.
Larger language models do in-context learning differently
We find that many model families exhibit this behavior on the FP, RTE, and ETHOS datasets. Conversely, the SUBJ dataset seems to show that model performance drops across all model families and for all models within each model family, a result that suggests that it is easier for models to flip their predictions to follow flipped labels for this task, even if the model is small. It is unclear why this task in particular encourages flipping predictions to follow flipped labels more than other tasks do.
Larger language models do in-context learning differently
In this section, we show per-dataset results for each model family after converting prompts to our SUL-ICL setup described in Section 4. Figure 17 gives a per-dataset overview of the performance differences between using SUL-ICL labels and using natural language labels as described in Section 4. We exclude the RTE dataset for PaLM models because the prompts from this dataset at k = 16 in-context exemplars per class consistently exceed the maximum allowable context length. We find that for InstructGPT, Codex, and PaLM models, large models see less of a performance drop than small models do when switching from natural language targets to semantically-unrelated targets, implying that they are more capable of learning input-label mappings when semantic priors are unavailable. Conversely, base GPT-3 models do not seem to follow the same trend, specifically in the case of davinci, which (on many tasks) sees the largest performance drops when using SUL-ICL targets despite being the largest model in the family. It is unclear why davinci seems to be the only large model that is not capable of learning input-label mappings in the SUL-ICL setup, though this behavior is consistent with davinci behaving similarly to small models as described in Section 3.
Larger language models do in-context learning differently
In Figure 18, we show per-dataset results for model accuracy with respect to the number of in-context exemplars provided. We do not run experiments on InstructGPT models and davinci in order to reduce cost. Lines do not always extend to k = 32 due to context-length constraints. These results indicate that for many datasets and model families, larger models are better at utilizing in-context exemplars in a SUL-ICL setup than small models are. This suggests that larger language models are more capable than small language models are at learning input-label mappings using the exemplars presented in-context rather than using prior knowledge from pretraining.  Figure 17: For many datasets and model families, performance decreases more for small models than it does for large models when using semantically-unrelated targets instead of natural language targets. Accuracy is calculated over 100 evaluation examples with k = 16 in-context exemplars per class.
Larger language models do in-context learning differently
We compare PaLM and Flan-PaLM model behaviors on a per-dataset level as an extension of Section 5. First, we show model behavior in the SUL-ICL setting in Figure 19, finding that for the SST-2, QQP, RTE, and ETHOS datasets, Flan-PaLM models achieve higher performance than their respective PaLM models. On the SST-2 dataset in particular, Flan-PaLM-8B outperforms PaLM-8B by 28% and even outperforms PaLM-62B by 2%. There are some datasets, however, for which instruction tuning seemed to decrease performance (e.g., PaLM-8B outperforms Flan-PaLM-8B on SUBJ by 23%). These results indicate that for many tasks, instruction tuning increases the model's capacity to  Figure 19: For many datasets, instruction-tuned language models are better at learning input-label mappings than pretraining-only language models are. Accuracy is calculated over 100 evaluation examples in the SUL-ICL setup.
Larger language models do in-context learning differently
learn input-label mappings in-context (though there are some exceptions), which follows the findings from Section 5. We also found that across most datasets, Flan-PaLM does worse than PaLM and scores close to 0% accuracy when given one in-context exemplar per class, yet this does not seem to be the case when two or more in-context exemplars per class are presented. Why this occurs is unknown, but it may indicate that Flan-PaLM does not give a response that is part of the target set of responses (e.g., does not output "Foo" or "Bar") in a 1-shot SUL-ICL setting.
Larger language models do in-context learning differently
In Figure 20, we show results for PaLM and Flan-PaLM in the flipped-label setting. For all datasets, 9 we find that every Flan-PaLM model achieves better performance than its respective PaLM model. PaLM models notably have lower accuracy when more labels are flipped, which means that PaLM models are better than Flan-PaLM models are at learning flipped input-label mappings presented in context, suggesting that it is harder for Flan-PaLM models to override semantic priors. This suggests that instruction tuning reinforces the model's semantic priors or gives it more semantic priors, making it more difficult for the model to override its prior knowledge.
Larger language models do in-context learning differently
Algorithm 1 Generating one evaluation example for N -dimensional linear classification (y = a 1 x 1 + ... + a N x N ) with k in-context exemplars per class. Random N -D vectors are generated using np.random.randint(). for i ← 1 to k do 2k in-context exemplars 7:
Larger language models do in-context learning differently
x+ ← random N -D vector conditioned on x+, a > t Positive example 8:
Larger language models do in-context learning differently
x− ← random N -D vector conditioned on x−, a ≤ t Negative example 9:
Larger language models do in-context learning differently
xtrain ← xtrain + [x+, x−] 10:
Larger language models do in-context learning differently
ytrain ← ytrain + [1, −1] 11: end for 12:
Larger language models do in-context learning differently
x eval ← random N -D vector 13:
Larger language models do in-context learning differently
y eval ← 1 if x eval , a > t, else −1 14:
Larger language models do in-context learning differently
return xtrain, ytrain, x eval , y eval 15: end procedure 
Larger language models do in-context learning differently
In Figure 21, we show model performance for Codex and PaLM models versus an exponentially increasing number of dimensions N (the data generation procedure is shown in Algorithm 1). We also include results from a standard polynomial SVM implemented via scikit-learn (svm.SVC(kernel='poly')) for comparison. We find that for the Codex model family, the largest model can successfully perform linear classification up to N = 64, while the smaller models reach guessing performance at approximately N = 16. For PaLM models, on the other hand, model scale does not seem to significantly correlate with the number of dimensions to which the model can perform linear classification, though all PaLM models can perform linear classification up to at least N = 8. 10 Neither PaLM models nor Codex models can outperform an SVM baseline.
Larger language models do in-context learning differently
These results suggest that model size alone does not necessarily unlock the ability to perform linear classification at high dimensionality (since PaLM-540B does not outperform PaLM-8B or PaLM-62B), but instead imply that there is another scaling factor seen in the Codex models that allows this ability to emerge. Because we do not know the particular scaling factors of the Codex model family, we leave exploration as to what factors unlock this ability to future work.
Larger language models do in-context learning differently
In Appendix D.1-Appendix D.7, we include an example of a full few-shot prompt for each of the seven datasets used in the main paper. We show prompts with k = 16 in-context exemplars per class and the Input/Output prompt template from Appendix B.4 (our default experimental setup) and natural language targets (i.e., regular ICL). Prompts in a SUL-ICL and flipped-label ICL setup can be obtained by swapping labels with the desired labels (e.g., replacing "Negative Sentiment" with "Foo" and "Positive Sentiment" with "Bar" to convert SST-2 in a regular ICL setup to SST-2 in a SUL-ICL setup). Prompts (especially from the ETHOS dataset) may contain offensive language-note that all examples are directly taken from the existing datasets as referenced in Appendix A.
Larger language models do in-context learning differently
In Appendix D.8, we provide an example of a full prompt for the linear classification task from Section 6 and Appendix C.4. This prompt uses the same default experimental setup as the prompts from Appendix D.1-Appendix D.7 but uses SUL-ICL targets since we only used this dataset in SUL-ICL settings. Input: a generic family comedy unlikely to be appreciated by anyone outside the under-10 set .
Larger language models do in-context learning differently
Input: , treasure planet is truly gorgeous to behold .
Larger language models do in-context learning differently
Input: the bai brothers have taken an small slice of history and opened it up for all of us to understand , and they 've told a nice little story in the process Input: an impossible romance , but we root for the patronized iranian lad .
Larger language models do in-context learning differently
Output: Subjective Sentence Input: . . . plays like a badly edited , 91-minute trailer ( and ) the director ca n't seem to get a coherent rhythm going . in fact , it does n't even seem like she tried .
Larger language models do in-context learning differently
Input: the stunt work is top-notch ; the dialogue and drama often food-spittingly funny .
Larger language models do in-context learning differently
Input: no such thing may be far from perfect , but those small , odd hartley touches help you warm to it .
Larger language models do in-context learning differently
Input: a positively thrilling combination of ethnography and all the intrigue , betrayal , deceit and murder of a shakespearean tragedy or a juicy soap opera .
Larger language models do in-context learning differently
Input: it trusts the story it sets out to tell .
Larger language models do in-context learning differently
Input: so , shaun goes to great lengths with a little help from his girlfriend ashley and his drugged-out loser brother lance to get into stanford any way they see fit .
Larger language models do in-context learning differently
Output: Objective Sentence Input: are they illusions , visions from the past , ghosts -or is it reality ?
Larger language models do in-context learning differently
Input: all the amped-up tony hawk-style stunts and thrashing rap-metal ca n't disguise the fact that , really , we 've been here , done that .
Larger language models do in-context learning differently
Input: a master at being everybody but himself he reveals to his friend and confidant saiid ( isa totah ) the truth behind his struggles .
Larger language models do in-context learning differently
Input: three families , living in a three storey building , leave for their summer vacations .
Larger language models do in-context learning differently
Input: the directing and story are disjointed , flaws that have to be laid squarely on taylor 's doorstep . but the actors make this worth a peek .
Larger language models do in-context learning differently
Input: together , they team up on an adventure that would take them to some very unexpected places and people .
Larger language models do in-context learning differently
Input: jacquot 's rendering of puccini 's tale of devotion and double-cross is more than just a filmed opera . in his first stab at the form , jacquot takes a slightly anarchic approach that works only sporadically .
Larger language models do in-context learning differently
Input: evil czar and his no-less-evil sidekick general with the help of the local witch yaga try to eliminate fedot by giving him more and more complex quests and to take marusya to tsar 's palace .
Larger language models do in-context learning differently
Input: the clues are few and time is running out for the students of rogers high school .
Larger language models do in-context learning differently
Input: seducing ben is only beginning ; she becomes his biggest " fan " and most unexpected nightmare , as her obsessions quickly spiral out of control into betrayal , madness and , ultimately , murder .
Larger language models do in-context learning differently
Output: Objective Sentence Input: but despite his looks of francis , he indeed is henry ( timothy bottoms ) , a man with a much better character than patricia ever could have dreamt of .
Larger language models do in-context learning differently
Input: the actors pull out all the stops in nearly every scene , but to diminishing effect . the characters never change .
Larger language models do in-context learning differently
Output: Subjective Sentence Input: in 1946 , tests began using nazi v-1 " buzz bombs " launched from the decks of american diesel submarines .
Larger language models do in-context learning differently
Input: a clichM-id and shallow cautionary tale about the hard-partying lives of gay men .
Larger language models do in-context learning differently
Input: the characters search for meaning in capricious , even dangerous sexual urges . the irony is that the only selfless expression of love may be the failure to consummate it .
Larger language models do in-context learning differently
Output: Subjective Sentence Input: meanwhile , chris 's radio horoscopes seem oddly personal , and the street musicians outside uwe 's restaurant keep getting more numerous .
Larger language models do in-context learning differently
Input: battling his own demons he realizes he is just like the rest of us : good and evil .
Larger language models do in-context learning differently
Output: Objective Sentence Input: or so he tells bobby ( alex feldman ) the eighteen year old male hustler smith employs for company .
Larger language models do in-context learning differently
Input: two brothers along with an ensemble of fresh talent made all this possible and were brought into the light .
Larger language models do in-context learning differently
Output: Objective Sentence Input: sandra bullock and hugh grant make a great team , but this predictable romantic comedy should get a pink slip .
Larger language models do in-context learning differently
Input: nora is not interested in foreign political smalltalk , she is after government secrets .
Larger language models do in-context learning differently
Input: godard has never made a more sheerly beautiful film than this unexpectedly moving meditation on love , history , memory , resistance and artistic transcendence .
Larger language models do in-context learning differently
Input: elmo touts his drug as being 51 times stronger than coke . if you 're looking for a tale of brits behaving badly , watch snatch again . it 's 51 times better than this .
Larger language models do in-context learning differently
Output: Subjective Sentence Input: culled from nearly two years of filming , the documentary 's candid interviews , lyric moments of grim beauty , and powerful verite footage takes us beyond the usual stereotypes of the rap world and into the life of tislam milliner , a struggling rapper who 's ambitious to make it out of the " hood " .
Larger language models do in-context learning differently
Input: i wish windtalkers had had more faith in the dramatic potential of this true story . this would have been better than the fiction it has concocted , and there still could have been room for the war scenes .
Larger language models do in-context learning differently
Input: has lost some of the dramatic conviction that underlies the best of comedies . . .
Larger language models do in-context learning differently
Subjective Sentence
Larger language models do in-context learning differently
Input: What is the real name of the singer , Madonna ? and 'depression' and 'mental illness' have almost become buzz words. It seems that a lot of people don't know how difficult it is to really suffer and feel out of control of it all. As you spoke about, there's a fine line these days between awareness and influence. People get ideas in their heads and see the memes and the relateable content and start linking everything to their own lives. Before you know it, you've got perfectly healthy people being tainted by the world and people around them, imposing problems upon themselves and making life more difficult than it needs to be. It desensitises the whole situation and now I have people coming to me with real problems who don't want to speak up because of the upsurge in people talking about it. They feel they wouldn't be taken seriously. And that's horrible. I do understand though that it's an impossible seesaw to balance since so many people are involved and so many minds with a million ideas and actions are impossible to control and have on the same wave length.
Larger language models do in-context learning differently
Answer:
Larger language models do in-context learning differently
Not Hate Speech D.7 RTE
Larger language models do in-context learning differently
Input: At least 19 people have been killed in central Florida in the city of Lady Lake and Paisley after severe storms and a tornado ripped through the cities in the middle of the night. Eleven of those killed were in Paisley and three were in Lady Lake. The death toll is expected to rise as rescue crews resume tomorrow morning. Volusia, Sumter, Lake and Seminole counties have all been declared a state of an emergency as dozens of houses, mobile homes and a church were destroyed. Clothes and furniture are scattered around the wrecked houses and pieces of trees are scattered about. Cars are reported to have been turned over or thrown around in the air. "Our priority today is search and rescue," said Gov. of Florida, Charlie Crist. Rescuers are still looking through the wreckage to find survivors of those who might have been killed.
Larger language models do in-context learning differently
Gov. of Florida, Charlie Crist, has visited the cities of Lady Lake and Paisley.
Larger language models do in-context learning differently
Output: Does not entail Input: Glue sniffing is most common among teenagers. They generally grow out of it once other drugs such as alcohol and cannabis become available to them. Seven-year-olds have been known to start "glue sniffing". Because of the social stigma attached to "glue sniffing" most snifters stop around 16 or 17 years, unless they are seriously addicted.
Larger language models do in-context learning differently
Glue-sniffing is common among youngsters.
Larger language models do in-context learning differently
Input: Neil Armstrong was an aviator in the Navy and was chosen with the second group of astronauts in 1962. Made seven flights in the X-15 program (1960 photo), reaching an altitude of 207,500 feet. Was backup command pilot for Gemini 5, command pilot for Gemini 8, backup command pilot for Gemini 11, backup commander for Apollo 8, and commander for Apollo 11: successfully completing the first moonwalk.
Larger language models do in-context learning differently
Neil Armstrong was the first man who landed on the Moon.
Larger language models do in-context learning differently
Input: Anna Politkovskaya was found shot dead on Saturday in a lift at her block of flats in the Russian capital, Moscow.
Larger language models do in-context learning differently
Anna Politkovskaya was murdered.
Larger language models do in-context learning differently
Input: Argentina sought help from Britain on its privatization program and encouraged British investment.
Larger language models do in-context learning differently
Argentina sought UK expertise on privatization and agriculture.
Larger language models do in-context learning differently
Output: Does not entail Input: The Security Council voted in 2002 to protect U.S. soldiers and personnel from other nations that haven't ratified the creation of the court through a treaty, and last June renewed the immunity for a year.
Larger language models do in-context learning differently
Immunity for soldiers renewed.
Larger language models do in-context learning differently
Input: World leaders expressed concern on Thursday that North Korea will quit six-party nuclear disarmament talks and will bolster its nuclear weapons arsenal.
Larger language models do in-context learning differently
North Korea says it has a stockpile of nuclear weapons and is building more.
Larger language models do in-context learning differently
Output: Does not entail Input: The Osaka World Trade Center is the tallest building in Western Japan.
Larger language models do in-context learning differently
The Osaka World Trade Center is the tallest building in Japan.
Larger language models do in-context learning differently
Output: Does not entail Input: He endeared himself to artists by helping them in lean years and following their careers, said Henry Hopkins, chairman of UCLA's art department, director of the UCLA/Armand Hammer Museum and Cultural Center and former director of the Weisman foundation.
Larger language models do in-context learning differently
The UCLA/Hammer Museum is directed by Henry Hopkins.
Larger language models do in-context learning differently
Input: Green cards are becoming more difficult to obtain.
Larger language models do in-context learning differently
Green card is now difficult to receive.
Larger language models do in-context learning differently
Input: Nor is it clear whether any US support to Germany, in favour of Bonn as the WTO headquarters, would necessarily tilt a decision in that direction.
Larger language models do in-context learning differently
The WTO headquarters is in Bonn.
Larger language models do in-context learning differently
Output: Does not entail Input: The Prime Minister's Office and the Foreign Office had earlier purposely asserted that the case is strictly in the jurisdiction of the police and the justice system.
Larger language models do in-context learning differently
The jurisdiction of the case was queried by the Prime Minister and the Ministry of Foreign Affairs.
Larger language models do in-context learning differently
Output: Does not entail Input: Only a few Mag-lev trains have been used commercially such as at the Birmingham airport in the UK.
Larger language models do in-context learning differently
Maglev is commercially used.
Larger language models do in-context learning differently
Input: Durham is the 'City of Medicine' and home of Duke University and North Carolina Central.
Larger language models do in-context learning differently
Duke University is in Durham.
Larger language models do in-context learning differently
Input: Babe Ruth's career total would have been 1 higher had that rule not been in effect in the early part of his career. The all-time career record for home runs in Major League Baseball is 755, held by Hank Aaron since 1974.
Larger language models do in-context learning differently
Babe Ruth hit 755 home runs in his lifetime.
Larger language models do in-context learning differently
Output: Does not entail for the first time and went on to become the most prolific tennis player.
Larger language models do in-context learning differently
Boris Becker is a Wimbledon champion.
Larger language models do in-context learning differently
Input: Rabies is a viral disease of mammals and is transmitted primarily through bites. Annually, 7,000 to 8,000 rabid animals are detected in the United States, with more than 90 percent of the cases in wild animals.
Larger language models do in-context learning differently
Rabies is fatal in humans.
Larger language models do in-context learning differently
Output: Does not entail Input: There are suppositions that the US Democratic Congress may re-establish the luxury taxes, which were already once introduced in the 1990s. The suppositions resulted in the National Association of Watch and Clock Collectors commissioning a report on various tax issues. Material goods such as jewelry, watches, expensive furs, jet planes, boats, yachts, and luxury cars had already been subjected to additional taxes back in 1990. After 3 years these taxes were repealed, though the luxury automobiles tax was still active for the next 13 years.
Larger language models do in-context learning differently
The US Congress may re-establish luxury taxes.
Larger language models do in-context learning differently
Input: The U.S. handed power on June 30 to Iraq's interim government chosen by the United Nations and Paul Bremer, former governor of Iraq.
Larger language models do in-context learning differently
The U.S. chose Paul Bremer as new governor of Iraq.
Larger language models do in-context learning differently
Output: Does not entail Input: FBI agent Denise Stemen said in an affidavit that Lowe's alerted the FBI recently that intruders had broken into its computer at company headquarters in North Carolina, altered its computer programs and illegally intercepted credit card transactions.
Larger language models do in-context learning differently
Non-authorized personnel illegally entered into computer networks.
Larger language models do in-context learning differently
Input: A man who died during the G20 protests was pushed back by a police line minutes earlier, independent investigators have said. Ian Tomlinson, 47, who died of a heart attack, was blocked from passing through a police cordon as he attempted to walk home from work at a newsagent, the Independent Police Complaints Commission (IPCC) said. He was caught on several CCTV cameras walking up King William Street where he was confronted by uniformed officers shortly before 7.30pm last Wednesday.
Larger language models do in-context learning differently
Ian Tomlinson was shot by a policeman.
Larger language models do in-context learning differently
Input: GUS on Friday disposed of its remaining home shopping business and last non-UK retail operation with the 390m (265m) sale of the Dutch home shopping company, Wehkamp, to Industri Kapital, a private equity firm.
Larger language models do in-context learning differently
Wehkamp was based in the UK.
Larger language models do in-context learning differently
Output: Does not entail Input: Shiite and Kurdish political leaders continued talks, on Monday, on forming a new government, saying they expected a full cabinet to be announced within a day or two.
Larger language models do in-context learning differently
US officials are concerned by the political vacuum and fear that it is feeding sectarian tensions, correspondents say.
Larger language models do in-context learning differently
Output: Does not entail
Transformers learn in-context by gradient descent
In recent years Transformers (TFs; Vaswani et al., 2017) have demonstrated their superiority in numerous benchmarks and various fields of modern machine learning, and have emerged as the de-facto neural network architecture used for modern AI (Dosovitskiy et al., 2021;Yun et al., 2019;Carion et al., 2020;Gulati et al., 2020). It has been hypothesised that their success is due in part to a phenomenon called in-context learning (Brown et al., 2020;Liu et al., 2021): an ability to flexibly adjust their prediction based on additional data given in context (i.e. in the input sequence itself). In-context learning offers a seemingly different approach to few-shot and meta-learning (Brown et al., 2020), but as of today the exact mechanisms of how it works are not fully understood. It is thus of great interest to understand what makes Transformers pay attention to their context, what the mechanisms are, and under which circumstances, they come into play (Chan et al., 2022b;Olsson et al., 2022).
Transformers learn in-context by gradient descent
In this paper, we aim to bridge the gap between in-context and meta-learning, and show that in-context learning in Transformers can be an emergent property approximating gradient-based few-shot learning within its forward pass, see Figure 1. For this to be realized, we show how Transformers (1) construct a loss function dependent on the data given in sequence and (2) learn based on gradients of that loss. We will first focus on the latter, the more elaborate learning task, in sections 2 and 3, after which we provide evidence for the former in section 4.
Transformers learn in-context by gradient descent
We summarize our contributions as follows 1 :
Transformers learn in-context by gradient descent
• We construct explicit weights for a linear self-attention layer that induces an update identical to a single step of gradient descent (GD) on a mean squared error loss. Additionally, we show how several self-attention layers can iteratively perform curvature correction improving on plain gradient descent.
Transformers learn in-context by gradient descent
• When optimized on linear regression datasets, we demonstrate that linear self-attention-only Transform-1 Main experiments can be reproduced with notebooks provided under the following link: https://github.com/ google-research/self-organising-systems/ tree/master/transformers_learn_icl_by_gd Learning a neural network output layer by gradient descent on a dataset D train . The task-shared meta-parameters θ are obtained by meta-learning with the goal that after adjusting the neural network output layer, the model generalizes well on unseen data. Center: Illustration of a Transformer that adjusts its query prediction on the data given in-context i.e. t θ (xquery; D context ). The weights of the Transformer are optimized to predict the next token yquery. Right: Our results confirm the hypothesis that learning with K steps of gradient descent on a dataset D train (green part of the left plot) matches trained Transformers with K linear self-attention layers (central plot) when given D train as in-context data D context . ers either converge to our weight construction and therefore implement gradient descent, or generate linear models that closely align with models trained by GD, both in in-and out-of-distribution validation tasks.
Transformers learn in-context by gradient descent
• By incorporating multi-layer-perceptrons (MLPs) into the Transformer architecture, we enable solving nonlinear regression tasks within Transformers by showing its equivalence to learning a linear model on deep representations. We discuss connections to kernel regression as well as nonparametric kernel smoothing methods. Empirically, we compare meta-learned MLPs and a single step of GD on its output layer with trained Transformers and demonstrate striking similarities between the identified solutions.
Transformers learn in-context by gradient descent
• We resolve the dependency on the specific token construction by providing evidence that learned Transformers first encode incoming tokens into a format amenable to the in-context gradient descent learning that occurs in the later layers of the Transformer.
Transformers learn in-context by gradient descent
These findings allow us to connect learning Transformer weights and the concept of meta-learning a learning algorithm (Schmidhuber, 1987;Hinton & Plaut, 1987;Bengio et al., 1990;Chalmers, 1991;Schmidhuber, 1992;Thrun & Pratt, 1998;Hochreiter et al., 2001;Andrychowicz et al., 2016;Ba et al., 2016;Kirsch & Schmidhuber, 2021). In this extensive research field, meta-learning is typically regarded as learning that takes place on various time scales namely fast and slow. The slowly changing parameters control and prepare for fast adaptation reacting to sudden changes in the incoming data by e.g. a context switch. Notably, we build heavily on the concept of fast weights (Schmidhuber, 1992) which has shown to be equivalent to linear self-attention (Schlag et al., 2021) and show how optimized Transformers implement interpretable learning algorithms within their weights.
Transformers learn in-context by gradient descent
Another related meta-learning concept, termed MAML (Finn et al., 2017), aims to meta-learn a deep neural network initialization which allows for fast adaptation on novel tasks. It has been shown that in many circumstances, the solution found can be approximated well when only adapting the output layer i.e. learning a linear model on a meta-learned deep data representations (Finn et al., 2017;Finn & Levine, 2018;Gordon et al., 2019;Lee et al., 2019;Rusu et al., 2019;Raghu et al., 2020;von Oswald et al., 2021). In section 3, we show the equivalence of this framework to in-context learning implemented in a common Transformer block i.e. when combining self-attention layers with a multi-layerperceptron.
Transformers learn in-context by gradient descent
In the light of meta-learning we show how optimizing Transformer weights can be regarded as learning on two time scales. More concretely, we find that solely through the pressure to predict correctly Transformers discover learning algorithms inside their forward computations, effectively meta-learning a learning algorithm. Recently, this concept of an emergent optimizer within a learned neural network, such as a Transformer, has been termed "mesa-optimization" (Hubinger et al., 2019). We find and describe one possible realization of this concept and hypothesize that the in-context learning capabilities of language models emerge through mechanisms similar to the ones we discuss here.
Transformers learn in-context by gradient descent
Transformers come in different "shapes and sizes", operate on vastly different domains, and exhibit varying forms of phase transitions of in-context learning (Kirsch et al., 2022;Chan et al., 2022a), suggesting variance and significant complexity of the underlying learning mechanisms. As a result, we expect our findings on linear self-attention-only Transformers to only explain a limited part of a complex process, and it may be one of many possible methods giving rise to in-context learning. Nevertheless, our approach provides an intriguing perspective on, and novel evidence for, an incontext learning mechanism that significantly differs from existing mechanisms based on associative memory (Ramsauer et al., 2020), or by the copying mechanism termed induction heads identified by (Olsson et al., 2022). We, therefore, state the following Hypothesis 1 (Transformers learn in-context by gradient descent). When training Transformers on auto-regressive tasks, in-context learning in the Transformer forward pass is implemented by gradient-based optimization of an implicit auto-regressive inner loss constructed from its in-context data.
Transformers learn in-context by gradient descent
We acknowledge work done in parallel, investigating the same hypothesis. Akyürek et al. (2023) puts forward a weight construction based on a chain of Transformer layers (including MLPs) that together implement a single step of gradient descent with weight decay. Similar to work done by Garg et al. (2022), they then show that trained Transformers match the performance of models obtained by gradient descent. Nevertheless, it is not clear that optimization finds Transformer weights that coincide with their construction.
Transformers learn in-context by gradient descent
Here, we present a much simpler construction that builds on Schlag et al. (2021) and only requires a single linear selfattention layer to implement a step of gradient descent. This allows us to (1) show that optimizing self-attention-only Transformers finds weights that match our weight construction (Proposition 1), demonstrating its practical relevance, and (2) explain in-context learning in shallow two layer Transformers intensively studied by Olsson et al. (2022). Therefore, although related work provides comprehensive empirical evidence that Transformers indeed seem to implement gradient descent based learning on the data given in-context, we will in the following present mechanistic verification of this hypothesis and provide compelling evidence that our construction, which implements GD in a Transformer forward pass, is found in practice.
Transformers learn in-context by gradient descent
We start by reviewing a standard multi-head self-attention (SA) layer with parameters θ. A SA layer updates each element e j of a set of tokens {e 1 , . . . , e N } according to
Transformers learn in-context by gradient descent
with P h , V h , K h the projection, value and key matrices, respectively, and q h,i the query, all for the h-th head. To simplify the presentation, we omit bias terms here and throughout. 
Transformers learn in-context by gradient descent
We next show that with some simple manipulations we can relate the update performed by an LSA layer to one step of gradient descent on a linear regression loss.
Transformers learn in-context by gradient descent
We now introduce a reference linear model y(x) = W x parameterized by the weight matrix W ∈ R Ny×Nx , and a training dataset
Transformers learn in-context by gradient descent
comprising of input samples x i ∈ R Nx and respective labels y i ∈ R Ny . The goal of learning is to minimize the squared-error loss:
Transformers learn in-context by gradient descent
One step of gradient descent on L with learning rate η yields the weight change
Transformers learn in-context by gradient descent
Considering the loss after changing the weights, we obtain
Transformers learn in-context by gradient descent
where we introduced the transformed targets y i − ∆y i with ∆y i = ∆W x i . Thus, we can view the outcome of a gradient descent step as an update to our regression loss (equation 2), where data, and not weights, are updated. Note that this formulation is closely linked to predicting based on nonparametric kernel smoothing, see Appendix A.8 for a discussion.
Transformers learn in-context by gradient descent
Returning to self-attention mechanisms and Transformers, we consider an in-context learning problem where we are given N context tokens together with an extra query token, indexed by N + 1. In terms of our linear regression problem, the N context tokens e j = (x j , y j ) ∈ R Nx+Ny correspond to the N training points in D, and the N +1-th token e N +1 = (x N +1 , y N +1 ) = (x test ,ŷ test ) = e test to the test input x test and the corresponding predictionŷ test . We use the terms training and in-context data interchangeably, as well as query and test token/data, as we establish their equivalence now.
Transformers learn in-context by gradient descent
We have re-cast the task of learning a linear model as directly modifying the data, instead of explicitly computing and returning the weights of the model (equation 4). We proceed to establish a connection between self-attention and gradient descent. We provide a construction where learning takes place simultaneously by directly updating all tokens, including the test token, through a linear self-attention layer.
Transformers learn in-context by gradient descent
In other words, the token produced in response to a query (test) token is transformed from its initial value W 0 x test , where W 0 is the initial value of W , to the post-learning predictionŷ = (W 0 + ∆W )x test obtained after one gradient descent step. Proposition 1. Given a 1-head linear attention layer and the tokens e j = (x j , y j ), for j = 1, . . . , N , one can construct key, query and value matrices W K , W Q , W V as well as the projection matrix P such that a Transformer step on every token e j is identical to the gradient-induced dynamics e j ← (x j , y j ) + (0, −∆W x j ) = (x j , y j ) + P V K T q j such that e j = (x j , y j − ∆y j ). For the test data token (x N +1 , y N +1 ) the dynamics are identical.
Transformers learn in-context by gradient descent
The simple construction can be found in Appendix A.1 and we denote the corresponding self-attention weights by θ GD .
Transformers learn in-context by gradient descent
Below, we provide some additional insights on what is needed to implement the provided LSA-layer weight construction, and further details on what it can achieve:
Transformers learn in-context by gradient descent
• Full self-attention. Our dynamics model training is based on in-context tokens only, i.e., only e 1 , . . . , e N are used for computing key and value matrices; the query token e N +1 (containing test data) is excluded. This leads to a linear function in x test as well as to the correct ∆W , induced by gradient descent on a loss consisting only of the training data. This is a minor deviation from full self-attention. In practice, this modification can be dropped, which corresponds to assuming that the underlying initial weight matrix is zero, W 0 ≈ 0, which makes ∆W in equation 8 independent of the test token even if incorporating it in the key and value matrices. In our experiments, we see that these assumptions are met when initializing the attention weights θ to small values.
Transformers learn in-context by gradient descent
• Reading out predictions. When initializing the yentry of the test-data token with −W 0 x N +1 , i.e. e test = (x test , −W 0 x test ), the test-data predictionŷ can be easily read out by simply multiplying again by −1 the updated token, since −y N +1 + ∆y N +1 = −(y N +1 − ∆y N +1 ) = y N +1 + ∆W x N +1 . This can easily be done by a final projection matrix, which incidentally is usually found in Transformer architectures. Importantly, we see that a single head of self-attention is sufficient to transform our training targets as well as the test prediction simultaneously.
Transformers learn in-context by gradient descent
• Uniqueness. We note that the construction is not unique; in particular, it is only required that the products P W V as well as W K W Q match the construction. Furthermore, since no nonlinearity is present, any rescaling s of the matrix products, i.e., P W V s and W K W Q /s, leads to an equivalent result. If we correct for these equivalent formulations, we can experimentally verify that weights of our learned Transformers indeed match the presented construction.
Transformers learn in-context by gradient descent
• Meta-learned task-shared learning rates. When training self-attention parameters θ across a family of in-context learning tasks τ , where the data (x τ,i , y τ,i ) follows a certain distribution, the learning rate can be implicitly (meta-)learned such that an optimal loss reduction (averaged over tasks) is achieved given a fixed number of update steps. In our experiments, we find this to be the case. This kind of meta-learning to improve upon plain gradient descent has been leveraged in numerous previous approaches for deep neural networks (Li et al., 2017;Lee & Choi, 2018;Park & Oliva, 2019;Zhao et al., 2020;Flennerhag et al., 2020).
Transformers learn in-context by gradient descent
• Task-specific data transformations. A self-attention layer is in principle further capable of exploiting statistics in the current training data samples, beyond modeling task-shared curvature information in θ. More concretely, a LSA layer updates an input sample according to a data transformation x j ← x j + ∆x j = (I + P (X)V (X)K(X) T W Q )x j = H θ (X)x j , with X the N x × N input training data matrix, when neglecting influences by target data y i . Through H θ (X), a LSA layer can encode in θ an algorithm for carrying out data transformations which depend on the actual input training samples in X. In our experiments, we see that trained self-attention learners employ a simple form of H(X) and that this leads to substantial speed ups in for GD and TF learning.
Transformers learn in-context by gradient descent
We now experimentally investigate whether trained attention-based models implement gradient-based incontext learning in their forward passes. We gradually build up from single linear self-attention layers to multi-layer nonlinear models, approaching full Transformers. In this section, we follow the assumption of Proposition 1 tightly and construct our tokens by concatenating input and target data, e j = (x j , y j ) for 1 ≤ j ≤ N , and our query token by concatenating the test input and a zero vector, e N +1 = (x test , 0). We show how to lift this assumption in the last section of the  Figure 2. Comparing one step of GD with a trained single linear self-attention layer. Outer left: Trained single LSA layer performance is identical to the one of gradient descent. Center left: Almost perfect alignment of GD and the model generated by the SA layer after training, measured by cosine similarity and the L2 distance between models as well as their predictions. Center right: Identical loss of GD, the LSA layer model as well as the model obtained by interpolating between the construction and the optimized LSA layer weights for different N = Nx. Outer right: The trained LSA layer, gradient descent and their interpolation show identically loss (in log-scale) when provided input data different than during training i.e. with scale of 1. We display the mean/std. or the single runs of 5 seeds.
Transformers learn in-context by gradient descent
paper. The predictionŷ θ ({e τ,1 , . . . , e τ,N }, e τ,N +1 ) of the attention-based model, which depends on all tokens and on the parameters θ, is read-out from the y-entry of the updated N + 1-th token as explained in the previous section.
Transformers learn in-context by gradient descent
The objective of training, visualized in Figure 1, is to minimize the expected squared prediction error, averaged over tasks min θ E τ [||ŷ θ ({e τ,1 , . . . , e τ,N }, e τ,N +1 ) − y τ,test || 2 ]. We achieve this by minibatch online minimization (by Adam (Kingma & Ba, 2014)): At every optimization step, we construct a batch of novel training tasks and take a step of stochastic gradient descent on the loss function:
Transformers learn in-context by gradient descent
where each task (context) τ consists of in-context training data D τ = {(x τ,i , y τ,i )} N i=1 and test point (x τ,N +1 , y τ,N +1 ), which we use to construct our tokens {e τ,i } N +1 i=1 as described above. We denote the optimal parameters found by this optimization process by θ * . In our setup, finding θ * may be thought of as meta-learning, while learning a particular task τ corresponds to simply evaluating the modelŷ θ ({e τ,1 , . . . , e τ,N }, e τ,N +1 ). Note that we therefore never see the exact same training task twice during training. See Appendix A.12, especially Figure 16 for an analyses when using a fixed dataset size which we cycle over during training.
Transformers learn in-context by gradient descent
We focus on solvable tasks and similarly to Garg et al. (2022) generate data for each task using a teacher model with parameters W τ ∼ N (0, I). We then sample x τ,i ∼ U (−1, 1) n I and construct targets using the task-specific teacher model, y τ,i = W τ x τ,i . In the majority of our experiments we set the dimensions to N = n I = 10 and n O = 1. Since we use a noiseless teacher for simplicity, we can expect our regression tasks to be well-posed and analytically solvable as we only compute a loss on the Transformers last token, which stands in contrast to usual autoregressive training and the training setup of Garg et al. (2022). Full details and results for training with a fixed training set size may be found in Appendix A.12.
Transformers learn in-context by gradient descent
One-step of gradient descent vs. a single trained self-attention layer Our first goal is to investigate whether a trained single, linear self-attention layer can be explained by the provided weight construction that implements GD. To that end, we compare the predictions made by a LSA layer with trained weights θ * (which minimize equation 5) and with constructed weights θ GD (which satisfy Proposition 1).
Transformers learn in-context by gradient descent
Recall that a LSA layer yields the predictionŷ θ (x test ) = e N +1 + LSA θ ({e 1 , . . . , e N }, e N +1 ) = ∆W θ,D x test , which is linear in x test . We denote by ∆W θ,D the matrix generated by the LSA layer following the construction provided in Proposition 1, with query token e N +1 set such that the initial prediction is set to zero,ŷ test = 0. We compareŷ θ (x test ) to the prediction of the control LSAŷ θGD (x test ), which under our token construction corresponds to a linear model trained by one step of gradient descent starting from W 0 = 0. For this control model, we determine the optimal learning rate η by minimizing L(η) over a training set of 10 4 tasks through line search, with L(η) defined analogously to equation 5.
Transformers learn in-context by gradient descent
More concretely, to compare trained and constructed LSA layers, we sample T val = 10 4 validation tasks and record the following quantities, averaged over validation tasks: (1) the difference in predictions measured with the L2 norm, ∥ŷ θ (x τ,test ) −ŷ θGD (x τ,test )∥, (2) the cosine similarity between the sensitivities ∂ŷ θ GD (xτ,test) ∂xtest and ∂ŷ θ (xτ,test) ∂xtest as well as
Transformers learn in-context by gradient descent
∥ again according to the L2 norm, which in both cases yields the explicit models computed by the algorithm. We show the results of these comparisons in Figure 2. We find an excellent agreement between the two models over a wide range of hyperparameters. We note that as we do not have direct access to the initialization of W in the attention-based learners (it is hidden in θ), we cannot expect the models to agree exactly.
Transformers learn in-context by gradient descent
Although the above metrics are important to show similarities between the resulting learned models (in-context vs. gradient-based), the underlying algorithms could still be different. We therefore carry out an extended set of analyses:
Transformers learn in-context by gradient descent
1. Interpolation. We take inspiration on recent work (Benzing et al., 2022;Entezari et al., 2021) that showed approximate equivalence of models found by SGD after permuting weights within the trained neural networks. Since our models are deep linear networks with respect to x test we only correct for scaling mismatches between the two models -in this case the construction that implements GD and the trained weights. As shown in Figure 2, we observe (and can actually inspect by eye, see Appendix Figure 9) that a simple scaling correction on the trained weights is enough to recover the weight construction implementing GD. This leads to an identical loss of GD, the trained Transformer and the linearly interpolated weights θ I = (θ + θ GD )/2. See details in Appendix A.3 on how our weight correction and interpolation is obtained.
Transformers learn in-context by gradient descent
To test if our in-context learner has found a generalizable update rule, we investigate how GD, the trained LSA layer and its interpolation behave when providing in-context data in regimes different to the ones used during training. We therefore visualize the loss increase when (1) sampling the input data from U (−α, α) Nx or (2) scaling the teacher weights by α as αW when sampling validation tasks. For both cases, we set α = 1 during training. We again observe that when training a single linear self-attention Transformer, for both interventions, the Transformer performs equally to gradient descent outside of this training setups, see Figure 2 as well Appendix Figure 6. Note that the loss obtained through gradient descent also starts degrading quickly outside the training regime. Since we tune the learning rate for the input range [−1, 1] and one gradient step, tasks with larger input range will have higher curvature and the optimal learning rate for smaller ranges will lead to divergence and a drastic increase in loss also for GD.
Transformers learn in-context by gradient descent
3. Repeating the LSA update. Since we claim that a single trained LSA layer implements a GD-like learning rule, we further test its behavior when applying it repeatedly, not only once as in training. After we correct the learning rate of both algorithms, i.e. for GD and the trained Transformer with a dampening parameter λ = 0.75 (details in Appendix A.6), we see an identical loss decrease of both GD and the Transformer, see Figure 1.
Transformers learn in-context by gradient descent
To conclude, we present evidence that optimizing a single LSA layer to solve linear regression tasks finds weights that (approximately) coincide with the LSA-layer weight construction of Proposition 1, hence implementing a step of gradient descent, leading to the same learning capabilities on in-and out-of-distribution tasks. We comment on the random seed dependent phase transition of the loss during training in Appendix A.11.
Transformers learn in-context by gradient descent
Multiple steps of gradient descent vs. multiple layers of self-attention
Transformers learn in-context by gradient descent
We now turn to deep linear self-attention-only Transformers. The construction we put forth in Proposition 1, can be immediately stacked up over K layers; in this case, the final prediction can be read out from the last layer as before by negating the y-entry of the last test token:
Transformers learn in-context by gradient descent
are the test token values at layer k, and ∆y k,N +1 the change in the y-entry of the test token after applying the k-th step of self-attention, and ∆W k the k-th implicit change in the underlying linear model parameters W . When optimizing such Transformers with K layers, we observe that these models generally outperform K steps of plain gradient descent, see Figure 3. Their behavior is however well described by a variant of gradient descent, for which we tune a single parameter γ defined through the transformation function H(X) which transforms the input data according to x j ← H(X)x j , with H(X) = (I − γXX T ). We term this gradient descent variant GD ++ which we explain and analyze in Appendix A.10.
Transformers learn in-context by gradient descent
To analyze the effect of adding more layers to the architecture, we first turn to the arguably simplest extension of a single SA layer and analyze a recurrent or looped 2-layer LSA model. Here, we simply repeatably apply the same layer (with the same weights) multiple times i.e. drawing the analogy to learning an iterative algorithm that applies the same logic multiple times.
Transformers learn in-context by gradient descent
Somewhat surprisingly, we find that the trained model surpasses plain gradient descent, which also results in decreasing alignment between the two models (see center left column), and the recurrent Transformer realigns perfectly with GD ++ while matching its performance on in-and out-of distribution tasks. Again, we can interpolate between the Transformer weights found by optimization and the LSAweight construction with learned η, γ, see Figure 3 & 6.
Transformers learn in-context by gradient descent
We next consider deeper, non-recurrent 5-layer LSA-only Transformers, with different parameters per layer (i.e. no weight tying). We see that a different GD learning rate as well as γ per step (layer) need to be tuned to match the Transformer performance. This slight modification leads again to almost perfect alignment between the trained TF and GD ++ with in this case 10 additional parameters and The trained TF performance surpasses standard GD but matches GD ++ , our GD variant with simple iterative data transformation. On both cases, we tuned the gradient descent learning rates as well as the scalar γ which governs the data transformation H(X). Center left & center right columns: We measure the alignment between the GD as well as the GD ++ models and the trained TF. In both cases the TF aligns well with GD in the beginning of training but aligns much better with GD ++ after training. Far right column: TF performance (in log-scale) mimics the one of GD ++ well when testing on OOD tasks (α ̸ = 1).
Transformers learn in-context by gradient descent
loss close to 0, see Figure 3. Nevertheless, we see that the naive correction necessary for model interpolation used in the aforementioned experiments is not enough to interpolate without a loss increase. We leave a search for better weight corrections to future work. We further study Transformers with different depths for recurrent as well as non-recurrent architectures with multiple heads and equipped with MLPs, and find qualitatively equivalent results, see Appendix Figure 7 and Figure 8. Additionally, in Appendix A.9, we provide results obtained when using softmax SA layers as well as LayerNorm, thus essentially retrieving the standard Transformer architecture. We again observe and are able to explain (after slight architectural modifications) good learning performance and as well as alignment with the construction of Proposition 1, though worse than when using linear self-attention. These findings suggest that the incontext learning abilities of the standard Transformer with these common architecture choices can be explained by the gradient-based learning hypothesis explored here. Our findings also question the ubiquitous use of softmax attention, and suggest further investigation is warranted into the performance of linear vs. softmax SA layers in real-world learning tasks, as initiated by Schlag et al. (2021).
Transformers learn in-context by gradient descent
It is unreasonable to assume that the astonishing in-context learning flexibility observed in large Transformers is ex-plained by gradient descent on linear models. We now show that this limitation can be resolved by incorporating one additional element of fully-fledged Transformers: preceding self-attention layers by MLPs enables learning linear models by gradient descent on deep representations which motivates our illustration in Figure 1. Empirically, we demonstrate this by solving non-linear sine-wave regression tasks, see Figure 4. Experimental details can be found in Appendix A.7. We state Proposition 2. Given a Transformer block i.e. a MLP m(e) which transforms the tokens e j = (x j , y j ) followed by an attention layer, we can construct weights that lead to gradient descent dynamics descending 1
Transformers learn in-context by gradient descent
Iteratively applying Transformer blocks therefore can solve kernelized least-squares regression problems with kernel function k(x, y) = m(x) ⊤ m(y) induced by the MLP m(·).
Transformers learn in-context by gradient descent
A detailed discussion on this form of kernel regression as well as kernel smoothing w/wo softmax nonlinearity through gradient descent on the data can be found in Appendix A.8. The way MLPs transform data in Transformers diverges from the standard meta-learning approach, where a task-shared input embedding network is optimized by backpropagation-through-training to improve the learning performance of a task-specific readout (e.g., Raghu et al., 2020;Lee et al., 2019;Bertinetto et al., 2019). On the other hand, given our token construction in Proposition 1, MLPs in Transformers intriguingly process both inputs and targets. The output of this transformation is then processed by a sin-gle linear self-attention layer, which, according to our theory, is capable of implementing gradient descent learning. We compare the performance of this Transformer model, where all weights are learned, to a control Transformer where the final LSA weights are set to the construction θ GD which is therefore identical to training an MLP by backpropagation through a GD updated output layer.
Transformers learn in-context by gradient descent
Intriguingly, both obtained functions show again surprising similarity on (1) the initial (meta-learned) prediction, read out after the MLP, and (2) the final prediction, after altering the output of the MLP through GD or the self-attention layer. This is again reflected in our alignment measures that now, since the obtained models are nonlinear w.r.t. x test , only represent the two first parts of the Taylor approximation of the obtained functions. Our results serve as a first demonstration of how MLPs and self-attention layers can interplay to support nonlinear in-context learning, allowing to fine-tune deep data representations by gradient descent. Investigating the interplay between MLPs and SA-layer in deep TFs is left for future work.
Transformers learn in-context by gradient descent
The construction provided in Proposition 1 and the previous experimental section relied on a token structure where both input and output data are concatenated into a single token. This design is different from the way tokens are typically built in most of the related work dealing with simple few-shot learning problems as well as in e.g. language modeling. We therefore ask: Can we overcome the assumption required in Proposition 1 and allow a Transformer to build the required token construction on its own? This motivates Proposition 3. Given a 1-head linear or softmax attention layer and the token construction e 2j = (x j ), e 2j+1 = (0, y j ) with a zero vector 0 of dim N x − N y and concatenated positional encodings, one can construct key, query and value matrix W K , W Q , W V as well as the projection matrix P such that all tokens e j are transformed into tokens equivalent to the ones required in Proposition 1.
Transformers learn in-context by gradient descent
The construction and its discussion can be found in Appendix A.5. To provide evidence that copying is performed in trained Transformers, we optimize a two-layer self-attention circuit on in-context data where alternating tokens include input or output data i.e. e 2j = (x j ) and e 2j+1 = (0, y j ). We again measure the loss as well as the mean of the norm of the partial derivative of the first layer's output w.r.t. the input tokens during training, see Figure 5. First, the training speeds are highly variant given different training seeds, also reported in Garg et al. (2022). Nevertheless, the Transformer is able to match the performance of a single (not two) step gradient descent. Interestingly, before the Transformer performance jumps to the one of GD, token e j transformed by the first self-attention layer becomes notably dependant on the neighboring token e j+1 while staying independent on the others which we denote as e other in Figure 5. We interpret this as evidence for a copying mechanism of the Transformer's first layer to merge input and output data into single tokens as required by Proposition 1. Then, in the second layer the Transformer performs a single step of GD. Notably, we were not able to train the Transformer with linear self-attention layers, but had to incorporate the softmax operation in the first layer. These preliminary findings support the study of Olsson et al. (2022) showing that softmax self-attention layers easily learn to copy; we confirm this claim, and further show that such copying allows the Transformer to proceed by emulating gradient-based learning in the second or deeper attention layers.
Transformers learn in-context by gradient descent
We conclude that copying through (softmax) attention layers is the second crucial mechanism for in-context learning in Transformers. This operation enables Transformers to merge data from different tokens and then to compute dot products of input and target data downstream, allowing for in-context learning by gradient descent to emerge.
Transformers learn in-context by gradient descent
Transformers show remarkable in-context learning behavior. Mechanisms based on attention, associative memory and copying by induction heads are currently the leading explanations for this remarkable feature of learning within the Transformer forward pass. In this paper, we put forward the hypothesis, similar to Garg et al. (2022)   pass to transform tokens by gradient-based optimization.
Transformers learn in-context by gradient descent
To provide evidence for this hypothesis, we build on Schlag et al. (2021) that already provide a linear self-attention layer variant with (fast-)inner loop learning by the error-correcting delta rule (Widrow & Hoff, 1960). We diverge from their setting and focus on (in-context) learning where we specifically construct a dataset by considering neighboring elements in the input sequence as input-and target training pairs, see assumptions of Proposition 1. This construction could be realized, for example, due to the model learning to implement a copying layer, see section 4 and proposition 3, and allows us to provide a simple and different construction to Schlag et al. (2021) that solely is built on the standard linear, and approximately softmax, self-attention layer but still implements gradient descent based learning dynamics. We, therefore, are able to explain gradient descent based learning in these standard architectures. Furthermore, we extend this construction based on a single self-attention layer and provide an explanation of how deeper K-layer Transformer models implement principled K-step gradient descent learning, which deviates again from Schlag et al. and allows us to identify that deep Transformers implement GD++, an accelerated version of gradient descent.
Transformers learn in-context by gradient descent
We highlight that our construction of gradient descent and GD++ is not suggestive but when training multi-layer selfattention-only Transformers on simple regression tasks, we provide strong evidence that the construction is actually found. This allows us, at least in our restricted problems settings, to explain mechanistically in-context learning in trained Transformers and its close resemblance to GD observed by related work. Further work is needed to incorporate regression problems with noisy data and weight regularization into our hypothesis. We speculate aspects of learning in these settings are meta-learned -e.g., the weight magnitudes to be encoded in the self-attention weights. Additionally, we did not analyze logistic regression for which one possible weight construction is already presented in Zhmoginov et al. (2022).  (Zhmoginov et al., 2022) that produce weights for target networks and already offer a different perspective on merging Transformers and meta-learning. There, Transformers transform weights instead of data and could potentially allow for gradient computations of weights deep inside the target network lifting the limitation of GD on linear models analyzed here.
Transformers learn in-context by gradient descent
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=0g0X4H8yN4I.   First, we highlight the dependency on the tokens e i of the linear self-attention operation
Transformers learn in-context by gradient descent
with ⊗ the outer product between two vectors. With this we can now easily draw connections to one step of gradient descent on L
Transformers learn in-context by gradient descent
We first restate Proposition 1. Given a 1-head linear attention layer and the tokens e j = (x j , y j ), for j = 1, . . . , N , one can construct key, query and value matrices W K , W Q , W V as well as the projection matrix P such that a Transformer step on every token e j is identical to the gradient-induced dynamics e j ← (x j , y j ) + (0, −∆W x j ) = (x i , y i ) + P V K T q j such that e j = (x j , y j − ∆y j ). For the test data token (x N +1 , y N +1 ) the dynamics are identical.
Transformers learn in-context by gradient descent
We provide the weight matrices in block form: W K = W Q = I x 0 0 0 with I x and I y the identity matrices of size N x and N y respectively. Furthermore, we set W V = 0 0 W 0 −I y with the weight matrix W 0 ∈ R Ny×Nx of the linear model we wish to train and P = η N I with identity matrix of size N x + N y . With this simple construction we obtain the following dynamics
Transformers learn in-context by gradient descent
for every token e j = (x j , y j ) including the query token e N +1 = e test = (x test , −W 0 x test ) which will give us the desired result.
Transformers learn in-context by gradient descent
We provide more experimental results when comparing GD with tuned learning rate η and data transformation scalar γ and the trained Transformer on other data distributions than provided during training, see Figure 6. We do so by changing the in-context data distribution and measure the loss of both methods averaged over 10.000 tasks when either changing α that 1) affects the input data range x ∼ U (−α, α) Nx or 2) the teacher by αW with W ∼ N (0, I). This setups leads to results shown in the main text, in the first two columns of Figure 6 and in the corresponding plots of Figure 7. Although the match for deeper architectures starts to become worse, overall the trained Transformers behaves remarkably similar to GD and GD ++ for layer depth greater than 1.
Transformers learn in-context by gradient descent
Furthermore, we try GD and the trained Transformer on input distributions that it never has seen during training. Here, we chose by chance of 1/3 either a normal, exponential or Laplace distribution (with JAX default parameters) and depict the average loss value over 10.000 tasks where the α value now simply scales the input values that are sampled from one of the distributions αx. The teacher scaling is identical to the one described above. See for results the two right columns of Figure 6, where we see almost identical behavior for recurrent architectures with less good match for deeper non-recurrent (a) Comparing one step of gradient descent with trained one layer Transformers on OOD data.  architectures far away from the training range of α = 1. Note that for deeper Transformers (K > 2) the corresponding GD and GD ++ version, see for more experimental details Appendix section A.12, we include a harsh clipping of the token values after every step of transformation between [−10, 10] (for the trained TF and GD) to improve training stability. Therefore, the loss increase is restricted to a certain value and plateaus.
Transformers learn in-context by gradient descent
In order to interpolate between the construction θ GD and the trained weights of the Transformer θ, we need to correct for some scaling ambiguity. For clarification, we restate here the linear self-attention operation for a single head
Transformers learn in-context by gradient descent
Now, to match the weight construction of Prop. 1 we have the aim for the matrix product W KQ to match an identify matrix (except for the last diagonal entry) after re-scaling. Therefore we compute the mean of the diagonal of the matrix product of the trained Transformer weights W KQ which we denote by β. After resealing both operations i.e. W KQ ← W KQ /β and W P V ← W P V β we interpolate linearly between the matrix products of GD as well as these rescaled trained matrix products i.e. W I,KQ = (W GD,KQ + W T F,KQ )/2 as well as W I,P V = (W GD,P V + W T F,P V )/2. We use these parameters to obtain results throughout the paper denote with Interpolated. We do so for GD as well as GD ++ when comparing to  Figure 7. Comparing ten steps of gradient descent with trained recurrent ten-layer Transformers. Results comparable to recurrent Transformer with two layers, see Figure 3, but now with 10 repeated layers. We again observe for deeper recurrent linear self-attention only Transformers that overall GD ++ and the trained Transformer align very well with one another and are again interpolatable leading to very similar behavior insight as well as outside training situations. Note the inferior performance to the non-recurrent five-layer Transformer which highlights the importance on specific learning rate as well γ parameter per layer/step. Figure 8. Comparing twelve steps of GD ++ with a trained twelve-layer Transformers with MLPs and 4 headed linear self-attention layer. Results comparable to the deep recurrent Transformer, see Figure 7, but now with 12 independent Transformer blocks including MLPs and 4-head linear self-attention. We omit LayerNorm. We again observe a close resemblance of the trained Transformers and GD ++ . We hypotheses that even when equipped with multiple heads and MLPs, Transformers approximate GD ++ .  Figure 9. Visualizing the weight matrices of trained Transformers. Left & outer left: Weight matrix products of a trained single linear self-attention layer. We see (after scalar correction) a perfect resemblance of our construction. Right & outer right: Weight matrix products of a trained 3-layer recurrent linear self-attention Transformer. Again, we see (after scalar correction) a perfect resemblance of our construction and an additional curvature correction i.e. diagonal values in P WV of the same magnitude except the last entry that functions as the learning rate. recurrent Transformers. Note that for non-recurrent Transformers, we face more ambiguity that we have to correct for since e.g. scalings influence each other across layer. We also see this in practice and are not able (only for some seeds) to interpolate between weights with our simple correction from above. We leave the search for more elaborate corrections for future work.
Transformers learn in-context by gradient descent
The simplicity of our construction enables us to visually compare trained Transformers and the construction put forward in Proposition A.1 in weight space. As discussed in the previous section A.3 there is redundancy in the way the trained Transformer can construct the matrix products leading to the weights corresponding to gradient descent. We therefore visualize W KQ = W T K W Q as well as W P V = P K W V in Figure 9.
Transformers learn in-context by gradient descent
A.5. Proof and discussion of Proposition 3
Transformers learn in-context by gradient descent
We state here again Proposition 3, provide the necessary construction and a short discussion. Proposition 3. Given a 1-head linear-or softmax attention layer and the token construction e 2j = (x j ), e 2j+1 = (0, y j ) with a zero vector 0 of dim N x − N y and concatenated positional encodings, one can construct key, query and value matrix W K , W Q , W V as well as the projection matrix P such that all tokens e j are transformed into tokens equivalent to the ones required in proposition 1.
Transformers learn in-context by gradient descent
To get a simple and clean construction, we choose wlog x j ∈ R 2N +1 and (0, y j ) ∈ R 2N +1 as well as model the positional encodings as unit vectors p j ∈ R 2N +1 and concatenate them to the tokens i.e. e j = (x j/2 , p j ). We wish for a construction that realizes
Transformers learn in-context by gradient descent
This means that a token replaces its own positional encoding by coping the target data of the next token to itself leading to e j = (x j/2 , 0, y j/2+1 ), with slight abusive of notation. This can simply be realized by (for example) setting P = I,
Transformers learn in-context by gradient descent
with I x,of f the lower diagonal identity matrix fo size N x . Note that then simply K T W Q e j = p j+1 i.e. it chooses the j + 1 element of V which stays p j+1 if we apply the softmax operation on K T q j . Since the j + 1 entry of V is (0, y j/2+1 − p j ) we obtain the desired result.
Transformers learn in-context by gradient descent
For the (toy-)regression problems considered in this manuscript, the provided result would give N/2 tokens for which we also copy (parts) of x j underneath y j . This is desired for modalities such as language where every two tokens could be considered an in-and output pair for the implicit autoregressive inner-loop loss. These tokens do not have be necessarily next to each other, see for this behavior experimental findings presented in (Olsson et al., 2022). For the experiments conducted here, one solution is to zero out these tokens which could be constructed by a two-head self-attention layer that given uneven j simply subtracts itself resulting in a zero token. For all even tokens, we use the construction from above which effectively coincides with the token construction required in Proposition 1.  Figure 10. Roll-out experiments: applying a trained single linear self-attention layer multiple times. We observe that different dampening strengths affect the generalization of both methods with slightly better robustness for GD which matching performance for 50 steps when λ = 0.75.
Transformers learn in-context by gradient descent
A.6. Dampening the self-attention layer
Transformers learn in-context by gradient descent
As an additional out-of-distribution experiment, we test the behavior when repeating a single LSA-layer trained to lower our objective, see equation 5, with the aim to repeat the learned learning/update rule. Note that GD as well as the selfattention layer were optimized to be optimal for one step. For GD we line search the otpimal learning rate η on 10.000 task. Interestingly, for both methods we observe quick divergence when applied multiple times, see left plot of Figure  10. Nevertheless, both of our update functions are described by a linear self-attention layer for which we can control the norm, post training, by a simple scale which we denote as λ. This results in the new update y test + λ∆W x test for GD and y test + λP V K T W Q x test for the trained self-attention layer which effectively re-tunes the learning rate for GD and the trained self-attention layer. Intriguingly, both methods do generalize similarly well (or poorly) on this out-of-distribution experiment when changing λ, see again Figure 10. We show in Figure 1 the behavior for λ = 0.75 for which we see both methods steadily decreasing the loss within 50 steps.
Transformers learn in-context by gradient descent
For the sine wave regression tasks, we follow (Finn et al., 2017) and other meta-learning literature and sample for each task an amplitude a ∼ U (0.1, 5) and a phase ρ ∼ U (0, π). Each tasks consist of N = 10 data points where inputs are sampled x ∼ U (−5, 5) and targets computed by y = a sin(ρ + x). We choose here for the first time, for GD as well as for the Transformer, an input embedding emb that maps tokens e i = (x i , y i ) into a 40 dimensional space emb(e i ) = W emb e i through an affine projection without bias. We skip the first self-attention layer but, as usually done in Transformers, then transform the embedded tokens through an MLP m with a single hidden layer, widening factor of 4 (160 hidden neuros) and GELU nonlinearity (Hendrycks & Gimpel, 2016) i.e. e j ← m(emb(e j )) + emb(e j ).
Transformers learn in-context by gradient descent
We interpret the last entry of the transformed tokens as the (transformed) targets and the rest as a higher-dimensional input data representation on which we train a model with a single gradient descent step. We compare the obtained meta-learned GD solution with training a Transformer on the same token embeddings but instead learn a self-attention layer. Note that the embeddings of the tokens, including the transformation through the MLP, are not dependent on an interplay between the tokens. Furthermore, the initial transformation is dependent on e i = (x i , y i ), i.e., input as well as on the target data except for the query token for which y test = 0. This means that this construction is, except for the additional dependency on targets, close to a large corpus of meta-learning literature that aims to find a deep representation optimized for (fast) fine tuning and few-shot learning. In order to compare the meta-training of the MLP and the Transformer, we choose the same seed to initialize the network weights for the MLPs and the input embedding trained by meta-learning i.e. backprop through training or the Transformer. This leads to the plots and almost identical learned initial function and updated functions shown in Figure 4.
Transformers learn in-context by gradient descent
Let's consider the data transformation induced by an MLPm(x) and a residual connection commonly used in Transformer blocks i.e. e j ← e j +m(e j ) = (x j , y j ) + (m(x j ), 0) = (m(x j ), y j ) with m(x j ) = x j +m(x j ) andm not changing the targets y. When simply applying Proposition 1, it is easy to see that given this new token construction, a linear self-attention layer can induce the token dynamics e j ← (m(x j ), y j ) + (0, −∆W m(x j )) with ∆W = −η∇L(W ) given the loss function
Transformers learn in-context by gradient descent
Interestingly, for the test token e test = (x test , 0) this induces, after a multiplication with −1, an initial prediction after a single Transformer block given bŷ
Transformers learn in-context by gradient descent
interpreted as a kernel function. Concluding, we see that the combination of MLPs and a single self-attention layer can lead to dynamics induced when descending a kernelized regression (squared error) loss with a single step of gradient-descent.
Transformers learn in-context by gradient descent
Interestingly, when choosing W 0 = 0, we furthermore see that a single self-attention layer or Transformer block can be regarded as doing nonparametric kernel smoothingŷ = N i=1 y i k(x i , x test ) based on the data given in-context (Nadaraya, 1964;Watson, 1964). Note that we made a particular choice of kernel function here and that this view still holds when m(x j ) = 1 i.e. consider Transformers without MLPs or leverage the well-known view of softmax self-attention layer as a kernel function used to measure similarity between tokens (e.g. Choromanski et al., 2021;Zhang et al., 2021). Thus, implementing one step of gradient descent through a self-attention layer (w/wo softmax nonlinearity) is equivalent to performing kernel smoothing estimation. We however argue that this nonparametric kernel smoothing view of in-context learning is limited, and arises from looking only at a single self-attention layer. When considering deeper Transformer architectures, we see that multiple Transformer blocks can iteratively transform the targets based on multiple steps of gradient descent leading to minimization of a kernelized squared error loss L(W ). One way to obtain a suitable construction is by neglecting MLPs everywhere except in the first Transformer block. We leave the study of the exact mechanics, especially how the Transformer makes use of possibility transforming the targets through the MLPs, and the possibility of iteratively changing the kernel function throughout depth for future study.
Transformers learn in-context by gradient descent
Although linear Transformers and their variants have been shown to be competitive with their softmax counterpart (Irie et al., 2021), the removal of this nonlinearity is still a major departure from classic Transformers and more importantly from the Transformers used in related studies analyzing in-context learning. In this section we investigate whether and when gradient-based learning emerges in trained softmax self-attention layers, and we provide an analytical argument to back our findings.
Transformers learn in-context by gradient descent
First, we show, see Figure 12, that a single layer of softmax self-attention is not able to match GD performance. We tuned the learning rate as well as the weight initialization but found no significant difference over the hyperparameters we used througout this study. In general, we hypothesize that GD is an optimal update given the limited capacity of a single layer of (single-head) self-attention. We therefore argue that the softmax induces (at best) a linear offset of the matrix product of training data and query vector
Transformers learn in-context by gradient descent
proportional to a factor dependent on all {x τ,i } N +1 i=1 . We speculate that the dependency on the specific task τ , for large N x vanishes or that the x-dependent value matrix could introduce a correcting effect. In this case the softmax operation introduces an additive error w.r.t. to the optimal GD update. To overcome this disadvantageous offset, the Transformer can (approximately) introduce a correction with a second self-attention head by a simple subtraction i.e.  Figure 11. Visualizing the correction to the softmax operation when training Transformers on regression tasks. The left and center plot show the matrix product WKQ = W T K WQ including its scaling by η induced through P WV of the two heads of the trained softmax self-attention layer. We observe that both of the matrices are approximate diagonal almost perfect sign reversed values on the off-diagonal terms. After adding the matrices (right plot), we observe a diagonal matrix and therefore to much improved approximation of our construction and therefore gradient descent dynamics.
Transformers learn in-context by gradient descent
Here we assume that P V 1) subsumes the dividing factor of the softmax and that 2) is the same (up to scaling) for each head. Note that if (W 1,KQ − W 2,KQ ) is diagonal, and P and V chosen as in the Proposition of Appendix A.1, we recover our gradient descent construction.
Transformers learn in-context by gradient descent
We base this derivation on empirical findings, see Figure 12, that, first of all, show the softmax self-attention performance increases drastically when using two heads instead of one. Nevertheless, the self-attention layer has difficulties to match the loss values of a model trained with GD. Furthermore, this architecture change leads to a very much improved alignment of the trained model and GD. Second, we can observe that when training a two-headed softmax self-attention layer on regression tasks the correction proposed above is actually observed in weight space, see Figure 11. Here, we visualize the matrix product within the softmax operation W h,KQ per head which we scale with the last diagonal entry of P h W h,V which we denote by η h = P h W h,V (−1, −1). Intriguingly, this results in an almost perfect cancellation (right plot) of the off-diagonal terms and therefore in sum to an improved approximation of our construction, see the derivation above.
Transformers learn in-context by gradient descent
We would like to reiterate that the stronger inductive bias for copying data of the softmax layer remains, and is not invalidated by the analysis above. Therefore, even for our shallow and simple constructions they indeed fulfill an important role in support for our hypotheses: The ability to merge or copy input and target data into single tokens allowing for their dot product computation necessary for the construction in Proposition 1, see Section 4 in the main text.
Transformers learn in-context by gradient descent
We end this section by analysing Transformers equipped with LayerNorm which we apply as usually done before the self-attention layer: Overall, we observe qualitatively similar results to Transformers with softmax self-attention layer i.e. a decrease in performance compared to GD accompanied with a decrease in alignment between models generated by the Transformer and models trained with GD, see Figure 14. Here, we test again a single linear self-attention layer succeeding LayerNorm as well as two layers where we skip the first LayerNorm and only include a LayerNorm between the two. Including more heads does not help substantially. We again assume the optimality of GD and argue that information of targets and inputs present in the tokens is lost by averaging when applying LayerNorm. This naturally leads to decreasing performance compared to GD, see first row of Figure 14. Although the alignment to GD and GD ++ , especially for two layers, is high, we overall see inferior performance to one or two steps of GD or two steps of GD ++ . Nevertheless, we speculate that LayerNorm might not only stabilize Transformer training but could also act as some form of data normalization procedure that implicitly enables better generalization for larger inputs as well as targets provided in-context, see OOD experiments in Figure 14.
Transformers learn in-context by gradient descent
Overall we conclude that common architecture choices like softmax and LayerNorm seem supoptimal for the constructed in-context learning settings when comparing to GD or linear self-attention. Nevertheless, we speculate that the potentially small performance drops of in-context learning are negligible when turning to deep and wide Transformers for which these architecture choices have empirically proven to be superior. Test on larger targets GD Trained TF Figure 12. Comparing trained two-headed and one-headed single-layer softmax self-attention with 1 step of gradient descent on linear regression tasks. Left column: Softmax self-attention is not able to match gradient descent performance with hand-tuned learning rate, but adding a second attention head significantly reduces the gap, as expected by our analytical argument. Center left: The alignment suffers significantly for single-head softmax SA. We observe good but not as precise alignment when compared to linear Transformers for the two-headed softmax SA layer. Center right & right: The two-headed self-attention compared to the single-head layer shows similar robust out-of-distribution behavior compared to gradient descent.
Transformers learn in-context by gradient descent
We give here a precise construction showing how to implement in a single head, a step of GD and the discussed data transformation, resulting in GD ++ . Recall again the linear self-attention operation with a single head
Transformers learn in-context by gradient descent
We provide again the weight matrices in block form of the construction of Prop. 1 but now enabling additionally our described data transformation: W K = W Q = I x 0 0 0 with I x the identity matrix of size N x , I y od size N y resp. Furthermore, we set W V = I x 0 W −I y with the weight matrix W ∈ R Ny×Nx of the linear model we wish to train and P = −γI x 0 0 η N . This leads to the following update
Transformers learn in-context by gradient descent
for every token e j = (x j , y j ) including the query token e N +1 = e test = (x test , 0) which will give us the desired result.
Transformers learn in-context by gradient descent
Why does GD ++ perform better? We give here one possible explanation of the superior performance of GD ++ compared to GD. Note that there is a close resemblance of the GD transformation and a heavily truncated Neuman series approximation of the inverse XX T . We provide here a more heuristic explanation for the observed acceleration.
Transformers learn in-context by gradient descent
Given γ ∈ R, GD ++ transforms every input according to x i ← x i − γXX T x i = (I − γXX T )x i . We can therefore look at the change of squared regression loss L(W ) = 1 2 N i=0 (W x i − y i ) 2 induced by this transformation i.e. L ++ (W ) = Figure 13. GD ++ analyses. Left: We visualize the change of the eigenspectrum induced by the input data transformation of GD ++ for different γ observed in practice. Center: Given we know the maximum and minimum of eigenvalues λ1, λn of the loss Hessian XX T with X = (x0, . . . , xN ) for different N , we compare the original condition number (depicted by *'s at γ = 0) and the condition number (in log scale) of the GD ++ altered loss Hessian when varying γ. We plot in dotted lines the γ values that we observe in practice which are close the optimal ones i.e. the local minimum derived through our analysis. Right: ForN = 25, we plot for different γ values the distribution of condition numbers κ = λ1/λn for 10000 tasks and observe favorable κ values close to 1 when approaching the γ = 0.099 value was found in practice. The κ values quickly explode for γ > 0.1.
Transformers learn in-context by gradient descent
in turn leads to a change of the loss Hessian from
Transformers learn in-context by gradient descent
Given the original Hessian H = XX T = U ΣU T with it's set of sorted eigenvalues {λ 1 , . . . , λ n } and λ i ≥ 0 on the diagonal matrix Σ we can express the new Hessian through U, Σ i.e.
Transformers learn in-context by gradient descent
We can simplify H ++ further as
Transformers learn in-context by gradient descent
Given the eigenspectrum {λ 1 , . . . , λ n } of H, we obtain an (unsorted) eigenspecturm for H ++ with {λ 1 − 2γλ 2 1 + γ 2 λ 3 1 , . . . , λ n − 2γλ 2 n + γ 2 λ 3 n } which we visualize in Figure 13 for different γ observed in practice. We hypotheses that the Transformer chooses γ in a way that on average, across the distribution of tasks, the data transformation (iteratively) decreases the condition number λ 1 /λ n leading to accelerated learning. This could be achieved, for example, by keeping the smallest eigenvalue λ n ≈ λ ++ n fixed and choosing γ such that the largest eigenvalue of the transformed data λ ++ 1 is reduced, while the original λ 1 stays within [λ ++ 1 , λ ++ n ]. To support our hypotheses empirically, we computed the minimum and maximum eigenvalues of XX T across 10000 tasks while changing the number of datapoints N ∈ [10, 25, 50, 100] i.e. X = (x 0 , . . . , x N ) leading to better conditioned loss Hessians i.e. [1e−10, 0.097, 0.666, 2.870] and [4.6, 7.712, 10.845, 17.196] as the minimum and maximum eigenvalues of XX T across all tasks where we cut the smallest eigenvalue for N = 10 at 1e−10. Furthermore, we extract the γ values from the weights of optimized recurrent 2-layer Transformers trained on different task distributions and obtain γ values of [0.179, 0.099, 0.056, 0.029], see again Figure 13. Note that the observed eigenvalues stay within [0, 1/γ] i.e. the two roots of f (λ, γ) = λ − 2γλ 2 + γ 2 λ 3 .
Transformers learn in-context by gradient descent
Given the derived function of eigenvalue change f (λ, γ), we compute the condition number of H ++ by dividing the novel maximum eigenvalues λ ++ 1 = f (1/(3γ), γ) where λ = 1/(3γ) as the local maximum of f (λ, γ), for fixed γ, and the novel minimum eigenvalue λ ++ n = min(f (λ 1 , γ), f (λ n , γ)). Note that with too small γ, we move the original λ n closer to the root of f (λ, γ) i.e. λ = 1/γ and therefore can change the smallest eigenvalue.
Transformers learn in-context by gradient descent
Given the task distribution and its corresponding eigenvalue distribution, we see that choosing γ reduces the new condition number κ ++ = λ ++ 1 /λ ++ n which leads to better conditioned learning, see center plot of Figure 13. Note that the optimal γ based on our derivation above is based on the maximum and minimum eigenvalue across all tasks and does not take the change of the eigenvalue distribution into account. We argue therefore that the simplicity of the arguments above does not capture the task statistics and distribution shifts entirely and therefore obtains a slightly larger γ as an optimal value. Test on larger inputs GD GD + + Trained TF Figure 14. Comparing trained 1-layer and 2-layer Transformers with LayerNorm and 1 step or 2 steps of gradient descent resp. Left column: The Transformers is not able to match the gradient descent performance with hand-tuned learning rate. Alignment plots: The alignment suffers significantly when comparing to linear self-attention layers although still reasonable alignment is obtained which decreases slightly when comparing to GD ++ for the two-layer Transformer.Center right & right: The LayerNorm Transformer outperforms when GD when providing training input data that is significantly larger than the data provided during training.
Transformers learn in-context by gradient descent
We furthermore visualize the condition number change for N = 25 and 10000 tasks in the right plot of Figure 13 and observe the distribution moving to desirable κ values close to 1. For γ values larger than 0.1 the distribution quickly exhibits exploding condition numbers.
Transformers learn in-context by gradient descent
We comment shorty on the curiously looking phase transitions of the training loss observed in many of our experiments, see Figure 2. Nevertheless, simply switching from a single-headed self-attention layer to a two-headed self-attention layer mitigates the random seed dependent training instabilities in our experiments presented in the main text, see left and center plot of Figure 15.
Transformers learn in-context by gradient descent
Furthermore, these transitions look reminiscent of the recently observed "grokking" behaviour (Power et al., 2022). Interestingly, when carefully tuning the learning rate and batchsize we can also make the Transformers trained in these linear regression tasks grokk. For this, we train a single Transformer block (self-attention layer and MLP) on a limited amount of data (8192 tasks), see right plot of Figure 15, and observe grokking like train and test loss phase transitions where test set first increases drastically before experiencing a sudden drop in loss almost matching the desired GD loss of 0.2. We leave a thorough investigation of these phenomena for future study.
Transformers learn in-context by gradient descent
We use for most experiments identical hyperparameters that were tuned by hand which we list here • Optimizer: Adam (Kingma & Ba, 2014) with default parameters and learning rate of 0.001 for Transformer with depth K < 3 and 0.0005 otherwise. We use a batchsize of 2048 and applied gradient clipping to obtain gradients with global norm of 10. We used the Optax library.
Transformers learn in-context by gradient descent
• Haiku weight initialisation (fan-in) with truncated normal and std 0.002/K where K the number of layers.
Transformers learn in-context by gradient descent
• We did not use any regularisation and observed for deeper Transformers with K > 2 instabilities when reaching GD performance. We speculate that this occurs since the GD performance is, for the given training tasks, already close to divergence as seen when providing tasks with larger input ranges. Therefore, training Transformers also becomes Figure 15. Phase transitions during training. Left: Loss based on 10 different random seeds when optimizing a single-headed selfattention layer. We observe for some seeds very long initial phases of virtually zero progress after which the loss drops suddenly to the desired GD loss. Center: The same experiment but optimizing a two-headed self-attention layer. We observe fast and robust convergence to the loss of GD. Right: Training a single Transformer block i.e. a self-attention layer with MLP and a reduced training set size of 8192 tasks. We observe grokking like train and test loss phase transitions where test set first increases drastically before experiencing a sudden drop in loss almost matching the desired GD loss of 0.2.
Transformers learn in-context by gradient descent
instable when we approach GD with an optimal learning rate. In order to stabilize training, we simply clipped the token values to be in the range of [−10, 10].
Transformers learn in-context by gradient descent
• When applicable we use standard positional encodings of size 20 which we concatenated to all tokens.
Transformers learn in-context by gradient descent
• For simplicity, and to follow the provided weight construction closely, we did use square key, value and query parameter matrix in all experiments.
Transformers learn in-context by gradient descent
• The training length varied throughout our experimental setups and can be read off our training plots in the article.
Transformers learn in-context by gradient descent
• When training meta-parameters for gradient descent i.e. η and γ we used an identical training setup but usually training required much less iterations.
Transformers learn in-context by gradient descent
• In all experiments we choose inital W 0 = 0 for gradient descent trained models.
Transformers learn in-context by gradient descent
Inspired by (Garg et al., 2022), we additionally provide results when training a single linear self-attention layer on a fixed number of training tasks. Therefore, we iterate over a single fixed batch of size B instead of drawing new batch of tasks at every iteration. Results can be found in Figure 16. Intriguingly, we find that (meta-)gradient descent finds Transformer weights that align remarkable well with the provided construction and therefore gradient descent even when provided with an arguably very small number of training tasks. We argue that this again highlights the strong inductive bias of the LSA-layer to match (approximately) gradient descent learning in its forward pass.  Figure 16. Comparing trained Transformers with GD and their weight interpolation when training the Transformer on a fixed training set size B. Across our alignment measures as well as our tests on out-of-training behaviour, trained Transformers fail to align with GD when provided with a very small amount of tasks. Nevertheless, we see already almost perfect alignment in our base setting N = Nx = 10 when provided with B > 2048 tasks. In all settings, we train the Transformer on (non-stochastic) gradient descent iterating over a single batch of tasks of size B equal to the number provided in the Figure titles ( 128,512,2048,8192).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The construction of increasingly powerful language models has revolutionized natural language processing (NLP). Using gigantic text corpora and a cross-entropy objective, such models learn to predict a distribution over the next word after a given context. The learned representations are useful for many other tasks, either as initializations [Ramachandran et al., 2017, Howard andRuder, 2018] or as a source of contextual word embeddings [McCann et al., 2017, Peters et al., 2018. Although representations previously needed fine-tuning to solve downstream tasks, recent models [Radford et al., 2019, Brown et al., 2020 have demonstrated strong performance even without fine-tuning.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The construction of increasingly powerful language models has revolutionized natural language processing (NLP). Using gigantic text corpora and a cross-entropy objective, such models learn to predict a distribution over the next word after a given context. The learned representations are useful for many other tasks, either as initializations [Ramachandran et al., 2017, Howard andRuder, 2018] or as a source of contextual word embeddings [McCann et al., 2017, Peters et al., 2018. Although representations previously needed fine-tuning to solve downstream tasks, recent models [Radford et al., 2019, Brown et al., 2020 have demonstrated strong performance even without fine-tuning.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Since next word prediction is a powerful test of language understanding, at some intuitive level it is believable that doing well on language modeling can help with many diverse downstream tasks. At the same time, it is quite intriguing how even small decreases in test perplexity can lead to improved downstream performance. Attempting to understand this phenomenon naturally raises the following questions: (a) why should learning the next-word prediction task result in representations useful for downstream tasks? (b) what is the role of inductive biases of the models and algorithms used in this empirical success? Given the nascency of deep learning theory, it is very challenging to say anything mathematically precise about (b) for even 2-layer networks, let alone deep networks. Instead this paper takes a stab at the mathematical study of (a). We do so by exploring if and how quantitative improvements on downstream NLP tasks can be mathematically guaranteed as the language model distribution approaches the true one in some metric. As a first cut analysis, we restrict attention to classification tasks and the striking observation that they can be solved fairly well using linear classifiers on language model features without fine-tuning. Although we are forced to treat models as black boxes, just first-order optimality conditions reveal interesting properties of the learned features, leading to an understanding of their success on interesting linear classification tasks. We also use theoretical insights to construct, and empirically test, a simple objective that we call Quad, that provably learns useful features for classification tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Since next word prediction is a powerful test of language understanding, at some intuitive level it is believable that doing well on language modeling can help with many diverse downstream tasks. At the same time, it is quite intriguing how even small decreases in test perplexity can lead to improved downstream performance. Attempting to understand this phenomenon naturally raises the following questions: (a) why should learning the next-word prediction task result in representations useful for downstream tasks? (b) what is the role of inductive biases of the models and algorithms used in this empirical success? Given the nascency of deep learning theory, it is very challenging to say anything mathematically precise about (b) for even 2-layer networks, let alone deep networks. Instead this paper takes a stab at the mathematical study of (a). We do so by exploring if and how quantitative improvements on downstream NLP tasks can be mathematically guaranteed as the language model distribution approaches the true one in some metric. As a first cut analysis, we restrict attention to classification tasks and the striking observation that they can be solved fairly well using linear classifiers on language model features without fine-tuning. Although we are forced to treat models as black boxes, just first-order optimality conditions reveal interesting properties of the learned features, leading to an understanding of their success on interesting linear classification tasks. We also use theoretical insights to construct, and empirically test, a simple objective that we call Quad, that provably learns useful features for classification tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now summarize our contributions. With the observation that classification tasks of interest can be phrased as sentence completion tasks, we define natural classification tasks, in Section 3, as those that can be solved as linear functions of the conditional distribution over words that can follow a given context. Section 4 presents our main results, theorems 4.1 and 4.2, that mathematically quantify the benefit of language model features for solving natural tasks. We show that a -optimal language model (in cross-entropy) will do O( √ )-well on natural tasks; Theorem 4.2 proves a stronger result for low dimensional softmax models using a new tool that we call conditional mean features (Definition 4.1), which we show in Section 6 are good features for the downstream task. The usefulness of the language model features themselves is demonstrated by arguing that there is a linear relationship between them and the conditional mean features. Finally, we extend our theory to construct a new mathematically motivated objective Quad with useful guarantees in Section 5.2, and we report its good performance on linear classification tasks in Section 6.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now summarize our contributions. With the observation that classification tasks of interest can be phrased as sentence completion tasks, we define natural classification tasks, in Section 3, as those that can be solved as linear functions of the conditional distribution over words that can follow a given context. Section 4 presents our main results, theorems 4.1 and 4.2, that mathematically quantify the benefit of language model features for solving natural tasks. We show that a -optimal language model (in cross-entropy) will do O( √ )-well on natural tasks; Theorem 4.2 proves a stronger result for low dimensional softmax models using a new tool that we call conditional mean features (Definition 4.1), which we show in Section 6 are good features for the downstream task. The usefulness of the language model features themselves is demonstrated by arguing that there is a linear relationship between them and the conditional mean features. Finally, we extend our theory to construct a new mathematically motivated objective Quad with useful guarantees in Section 5.2, and we report its good performance on linear classification tasks in Section 6.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Embedding methods: Prior to language models, large text corpora like Wikipedia [Merity et al., 2016] were used to learn low dimensional embeddings for words [Mikolov et al., 2013b,a, Pennington et al., 2014 and subsequently for sentences [Kiros et al., 2015, Arora et al., 2017, Pagliardini et al., 2018, Logeswaran and Lee, 2018 for downstream task usage. These methods were inspired by the distributional hypothesis [Firth, 1957, Harris, 1954, which posits that meaning of text is determined in part by the surrounding context. Recent methods like BERT [Devlin et al., 2018] and variants [Lan et al., 2019, Yang et al., 2019, Liu et al., 2019 learn from auxiliary tasks, such as sentence completion, and are among the top performers on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Embedding methods: Prior to language models, large text corpora like Wikipedia [Merity et al., 2016] were used to learn low dimensional embeddings for words [Mikolov et al., 2013b,a, Pennington et al., 2014 and subsequently for sentences [Kiros et al., 2015, Arora et al., 2017, Pagliardini et al., 2018, Logeswaran and Lee, 2018 for downstream task usage. These methods were inspired by the distributional hypothesis [Firth, 1957, Harris, 1954, which posits that meaning of text is determined in part by the surrounding context. Recent methods like BERT [Devlin et al., 2018] and variants [Lan et al., 2019, Yang et al., 2019, Liu et al., 2019 learn from auxiliary tasks, such as sentence completion, and are among the top performers on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Language models for downstream tasks: We are interested in language models, including n-gram models [Chen and Goodman, 1999], and more recent models [Xu andRudnicky, 2000, Bengio et al., 2003] that use neural networks to compute low dimensional features for contexts and parametrize the next word distribution using softmax. Language models have shown to be useful for downstream tasks as initializations [Ramachandran et al., 2017, Howard andRuder, 2018] or as learned feature maps [Radford et al., 2017, McCann et al., 2017, Peters et al., 2018. The idea of phrasing classification tasks as sentence completion problems is motivated by recent works [Radford et al., 2019, Puri and Catanzaro, 2019, Schick and Schütze, 2020 that show that many downstream tasks can be solved by next word prediction for an appropriately conditioned language model. This idea also shares similarities with work that phrase a suite of downstream tasks as question-answering tasks [McCann et al., 2018] or text-to-text tasks [Raffel et al., 2019] and symbolic reasoning as fill-in-the-blank tasks [Talmor et al., 2019]. Our work exploits this prevalent idea of task rephrasing to theoretically analyze why language models succeed on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Language models for downstream tasks: We are interested in language models, including n-gram models [Chen and Goodman, 1999], and more recent models [Xu andRudnicky, 2000, Bengio et al., 2003] that use neural networks to compute low dimensional features for contexts and parametrize the next word distribution using softmax. Language models have shown to be useful for downstream tasks as initializations [Ramachandran et al., 2017, Howard andRuder, 2018] or as learned feature maps [Radford et al., 2017, McCann et al., 2017, Peters et al., 2018. The idea of phrasing classification tasks as sentence completion problems is motivated by recent works [Radford et al., 2019, Puri and Catanzaro, 2019, Schick and Schütze, 2020 that show that many downstream tasks can be solved by next word prediction for an appropriately conditioned language model. This idea also shares similarities with work that phrase a suite of downstream tasks as question-answering tasks [McCann et al., 2018] or text-to-text tasks [Raffel et al., 2019] and symbolic reasoning as fill-in-the-blank tasks [Talmor et al., 2019]. Our work exploits this prevalent idea of task rephrasing to theoretically analyze why language models succeed on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theoretical analysis: Since the success of early word embedding algorithms like word2vec [Mikolov et al., 2013a] and GloVe [Pennington et al., 2014], there have been attempts to understand them theoretically. Levy and Goldberg [2014] show that, in some regimes, the word2vec algorithm implicitly factorizes the PMI matrix. The theory of Noise Contrastive Estimation (NCE) is used by Dyer [2014] to understand word embedding methods and by Ma and Collins [2018] to prove parameter recovery for negative sampling methods that learn conditional models. A latent variable log-linear model is proposed in Arora et al. [2016] to explain and unify various word embedding algorithms. Theoretical justification is provided for sentence embedding methods either by using a latent variable model [Arora et al., 2017] or through the lens of compressed sensing [Arora et al., 2018]. Also relevant is recent work on theory for contrastive learning [Arora et al., 2019, Tosh et al., 2020b,a, Wang and Isola, 2020 and reconstruction-based methods [Lee et al., 2020], which analyze the utility of representations learned in the self-supervised regime for downstream tasks. Our work is the first to analyze the efficacy language model features on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theoretical analysis: Since the success of early word embedding algorithms like word2vec [Mikolov et al., 2013a] and GloVe [Pennington et al., 2014], there have been attempts to understand them theoretically. Levy and Goldberg [2014] show that, in some regimes, the word2vec algorithm implicitly factorizes the PMI matrix. The theory of Noise Contrastive Estimation (NCE) is used by Dyer [2014] to understand word embedding methods and by Ma and Collins [2018] to prove parameter recovery for negative sampling methods that learn conditional models. A latent variable log-linear model is proposed in Arora et al. [2016] to explain and unify various word embedding algorithms. Theoretical justification is provided for sentence embedding methods either by using a latent variable model [Arora et al., 2017] or through the lens of compressed sensing [Arora et al., 2018]. Also relevant is recent work on theory for contrastive learning [Arora et al., 2019, Tosh et al., 2020b,a, Wang and Isola, 2020 and reconstruction-based methods [Lee et al., 2020], which analyze the utility of representations learned in the self-supervised regime for downstream tasks. Our work is the first to analyze the efficacy language model features on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We use S to denote the discrete set of all contexts, i.e. sentences and partial sentences in natural language, W to denote the vocabulary of words, with V = |W|. ∆ A denotes the set of all distributions on A. We use p, p L ∈ ∆ S to denote probability distributions over S, and p ·|s , p * ·|s ∈ ∆ W to denote conditional distributions given a context s. When clear from context, we use p ·|s , p * ·|s ∈ R V as vectors of probabilities. For v ∈ R V , we index the coordinate for w ∈ W by v[w]; thus p * ·|s [w] := p * ·|s (w), where p * ·|s is used as a vector on the left and distribution on the right. Let φ w ∈ R d denote a d-dimensional embedding for word w ∈ W; word embeddings are stacked into a matrix Φ ∈ R d×V with columns being {φ w } w∈W . We use f : S → R d for a feature map that maps contexts to d-dimensional embeddings, e.g. f (s) can be the output of a transformer or RNN for an input context s ∈ S. For embeddings {θ s } s∈S , we use {θ s } to denote a function g with g(s) = θ s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We use S to denote the discrete set of all contexts, i.e. sentences and partial sentences in natural language, W to denote the vocabulary of words, with V = |W|. ∆ A denotes the set of all distributions on A. We use p, p L ∈ ∆ S to denote probability distributions over S, and p ·|s , p * ·|s ∈ ∆ W to denote conditional distributions given a context s. When clear from context, we use p ·|s , p * ·|s ∈ R V as vectors of probabilities. For v ∈ R V , we index the coordinate for w ∈ W by v[w]; thus p * ·|s [w] := p * ·|s (w), where p * ·|s is used as a vector on the left and distribution on the right. Let φ w ∈ R d denote a d-dimensional embedding for word w ∈ W; word embeddings are stacked into a matrix Φ ∈ R d×V with columns being {φ w } w∈W . We use f : S → R d for a feature map that maps contexts to d-dimensional embeddings, e.g. f (s) can be the output of a transformer or RNN for an input context s ∈ S. For embeddings {θ s } s∈S , we use {θ s } to denote a function g with g(s) = θ s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
A language model is trained to learn the true distribution of a text corpus. Given a context (e.g., a partial sentence s ∈ S), it predicts a distribution p ·|s over the word to follow. For example, given the context "The food was ", the model could place high probabilities on words "delicious", "expensive", "bland", etc. We use p L to denote the true distribution over the context set S in the language modeling corpus and p * ·|s to denote the true conditional distribution on words for s. A standard approach is to minimize the expected cross-entropy loss between p * ·|s and the model prediction p ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
A language model is trained to learn the true distribution of a text corpus. Given a context (e.g., a partial sentence s ∈ S), it predicts a distribution p ·|s over the word to follow. For example, given the context "The food was ", the model could place high probabilities on words "delicious", "expensive", "bland", etc. We use p L to denote the true distribution over the context set S in the language modeling corpus and p * ·|s to denote the true conditional distribution on words for s. A standard approach is to minimize the expected cross-entropy loss between p * ·|s and the model prediction p ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Using the property of cross-entropy that xent,s (p ·|s ) − xent (p * ·|s ) = D KL (p * ·|s , p ·|s ), it is easy to show that the optimal unconstrained language modeling recovers the distributions p * ·|s exactly. Proposition 2.1. The unique minimizer of xent ({p ·|s }) is p ·|s = p * ·|s for every s ∼ p L .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Using the property of cross-entropy that xent,s (p ·|s ) − xent (p * ·|s ) = D KL (p * ·|s , p ·|s ), it is easy to show that the optimal unconstrained language modeling recovers the distributions p * ·|s exactly. Proposition 2.1. The unique minimizer of xent ({p ·|s }) is p ·|s = p * ·|s for every s ∼ p L .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Recent models parametrize the conditional distribution p ·|s as a softmax computed using low dimensional embeddings. Such models are often more sample efficient and achieve lower test perplexity than traditional n-gram models. For an embedding θ ∈ R d , the softmax distribution over W using word embeddings Φ ∈ R d×V is p θ (w) = e θ φw /Z θ , where Z θ = w ∈W e θ φ w is the partition function. Just like p * ·|s , we can interpret p θ ∈ R V as a vector of probabilities. The context embeddings are parametrized by a model architecture of choice (e.g. transformer [Vaswani et al., 2017]) as a feature map f : S → R d . These embeddings then induce the softmax distribution p ·|s = p f (s) . We can now write the cross-entropy for the softmax parametrization
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Recent models parametrize the conditional distribution p ·|s as a softmax computed using low dimensional embeddings. Such models are often more sample efficient and achieve lower test perplexity than traditional n-gram models. For an embedding θ ∈ R d , the softmax distribution over W using word embeddings Φ ∈ R d×V is p θ (w) = e θ φw /Z θ , where Z θ = w ∈W e θ φ w is the partition function. Just like p * ·|s , we can interpret p θ ∈ R V as a vector of probabilities. The context embeddings are parametrized by a model architecture of choice (e.g. transformer [Vaswani et al., 2017]) as a feature map f : S → R d . These embeddings then induce the softmax distribution p ·|s = p f (s) . We can now write the cross-entropy for the softmax parametrization
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
is the cross-entropy loss for a context s that uses embedding θ.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
is the cross-entropy loss for a context s that uses embedding θ.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Analogous to Proposition 2.1, we want to know the optimal d-dimensional feature map f * and the induced conditional distribution p f * (s) 1 . Since p f (s) are parametrized by softmax of d V dimensional embeddings, these cannot express all the distributions p * ·|s , owing to the low dimensionality. The following result formalizes what information about p * ·|s is retained.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Analogous to Proposition 2.1, we want to know the optimal d-dimensional feature map f * and the induced conditional distribution p f * (s) 1 . Since p f (s) are parametrized by softmax of d V dimensional embeddings, these cannot express all the distributions p * ·|s , owing to the low dimensionality. The following result formalizes what information about p * ·|s is retained.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Unlike Proposition 2.1, p f * (s) ∈ R V is only equal to p * ·|s ∈ R V in the subspace of the rows of Φ ∈ R d×V . Thus smaller values of d will guarantee learning p * ·|s on smaller subspaces. This is proved by observing that for s ∈ S, ∇ θ xent,s (θ) = −Φp * ·|s + ∇ θ Z θ Z θ = −Φp * ·|s + Φp θ . Applying the first-order optimality condition for f * (s), i.e. ∇ θ xent,s (f * (s)) = 0 gives the desired result.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Unlike Proposition 2.1, p f * (s) ∈ R V is only equal to p * ·|s ∈ R V in the subspace of the rows of Φ ∈ R d×V . Thus smaller values of d will guarantee learning p * ·|s on smaller subspaces. This is proved by observing that for s ∈ S, ∇ θ xent,s (θ) = −Φp * ·|s + ∇ θ Z θ Z θ = −Φp * ·|s + Φp θ . Applying the first-order optimality condition for f * (s), i.e. ∇ θ xent,s (f * (s)) = 0 gives the desired result.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We focus on developing theory only for classification tasks. A binary classification task 2 T is characterized by a distribution p T over S ×{±1}, where the input s is a sentence from S and the label y is {±1}. Given any feature map g : S → R D (for any D), we solve task T by fitting a linear classifier v ∈ R D on top of g(s). The classification loss is written as T (g, v) = E (s,y)∼p T (v g(s), y) , where is a 1-Lipschitz surrogate to the 0-1 loss, like the hinge loss (ŷ, y) = (1 − yŷ) + or the logistic loss (ŷ, y) = log(1 + e −yŷ ). The loss incurred by a representation function g is defined as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We focus on developing theory only for classification tasks. A binary classification task 2 T is characterized by a distribution p T over S ×{±1}, where the input s is a sentence from S and the label y is {±1}. Given any feature map g : S → R D (for any D), we solve task T by fitting a linear classifier v ∈ R D on top of g(s). The classification loss is written as T (g, v) = E (s,y)∼p T (v g(s), y) , where is a 1-Lipschitz surrogate to the 0-1 loss, like the hinge loss (ŷ, y) = (1 − yŷ) + or the logistic loss (ŷ, y) = log(1 + e −yŷ ). The loss incurred by a representation function g is defined as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For embeddings {θ s } s∈S , classification loss is thus
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For embeddings {θ s } s∈S , classification loss is thus
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
3 Using language models for classification tasks Sections 2.1 and 2.2 show that both unconstrained and softmax language models aim to learn either p * ·|s exactly or a projection Φp * ·|s . Thus we ask: why should learning p * ·|s be useful for solving downstream tasks? To answer this, we use a thought experiment where we have access to an oracle that provides p * ·|s for any s. We then demonstrate how to solve a classification task through reframing the task as a sentence completion problem and using p * ·|s to get completions to predict the label. Classification task as sentence completion: For an input s in a task, we propose to predict the label by using p * ·|s to predict words that could follow s. In particular, for a movie review sentiment analysis task, we can use the oracle to compare probabilities of ":)" and ":(" after an input and predict sentiment based on which is higher. However, p * ·|s will likely place higher probability on words that start sentences, like "The", instead of discriminative words useful for the task. To allow a larger set of grammatically correct completions, we can additionally append a prompt like "This movie is " at the end of a review and query probabilities of indicative adjectives like good, bad, interesting, boring etc. This approach also works for the AG news dataset [Zhang et al., 2015] which contains news articles from 4 categories: world, science/technology, sports, and business, where a prompt like "This article is about " can be added. We further theoretically motivate the usage of prompts in the discussion of Theorem 4.1, and we experimentally verify in Section 6 that probabilities of a small subset of words are sufficient to solve these tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
3 Using language models for classification tasks Sections 2.1 and 2.2 show that both unconstrained and softmax language models aim to learn either p * ·|s exactly or a projection Φp * ·|s . Thus we ask: why should learning p * ·|s be useful for solving downstream tasks? To answer this, we use a thought experiment where we have access to an oracle that provides p * ·|s for any s. We then demonstrate how to solve a classification task through reframing the task as a sentence completion problem and using p * ·|s to get completions to predict the label. Classification task as sentence completion: For an input s in a task, we propose to predict the label by using p * ·|s to predict words that could follow s. In particular, for a movie review sentiment analysis task, we can use the oracle to compare probabilities of ":)" and ":(" after an input and predict sentiment based on which is higher. However, p * ·|s will likely place higher probability on words that start sentences, like "The", instead of discriminative words useful for the task. To allow a larger set of grammatically correct completions, we can additionally append a prompt like "This movie is " at the end of a review and query probabilities of indicative adjectives like good, bad, interesting, boring etc. This approach also works for the AG news dataset [Zhang et al., 2015] which contains news articles from 4 categories: world, science/technology, sports, and business, where a prompt like "This article is about " can be added. We further theoretically motivate the usage of prompts in the discussion of Theorem 4.1, and we experimentally verify in Section 6 that probabilities of a small subset of words are sufficient to solve these tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Solving tasks using a linear function of p * ·|s : The above process can be seen as a sub-case of learning a linear function of p * ·|s ∈ R V . In the case of movie reviews, we can let w + = ":)" and w − = ":(" and compute the sign of p * ·|s (w + ) − p * ·|s (w − ) to predict the label. This strategy can be expressed as a linear separator on p *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Solving tasks using a linear function of p * ·|s : The above process can be seen as a sub-case of learning a linear function of p * ·|s ∈ R V . In the case of movie reviews, we can let w + = ":)" and w − = ":(" and compute the sign of p * ·|s (w + ) − p * ·|s (w − ) to predict the label. This strategy can be expressed as a linear separator on p *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The prediction function then is v p * ·|s > 0. This also works when we append a prompt; we can assign positive weights in v to adjectives like "good" and negative weights to adjectives like "boring". Two adjectives that convey the same sentiment may still have different strengths (e.g., "good" vs "amazing"), and we can capture this nuance by assigning different magnitude weights to them. To solve other tasks similarly, we can select a different set of words for each class. We verify experimentally that SST and AG news tasks can be solved by a linear function of probabilities of just a small subset of words in Section 6. We formalize this intuition below to define natural tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The prediction function then is v p * ·|s > 0. This also works when we append a prompt; we can assign positive weights in v to adjectives like "good" and negative weights to adjectives like "boring". Two adjectives that convey the same sentiment may still have different strengths (e.g., "good" vs "amazing"), and we can capture this nuance by assigning different magnitude weights to them. To solve other tasks similarly, we can select a different set of words for each class. We verify experimentally that SST and AG news tasks can be solved by a linear function of probabilities of just a small subset of words in Section 6. We formalize this intuition below to define natural tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus a natural task T is one that can achieve a small error of τ by learning an ∞ -norm bounded 3 linear classifier on top of features p * ·|s ∈ R V . Low dimensional softmax models, however, only learn p * ·|s in the subspace of Φ, as per Proposition 2.2. Thus we cannot ensure guarantees on all natural tasks and we are interested in tasks that this subspace can solve.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus a natural task T is one that can achieve a small error of τ by learning an ∞ -norm bounded 3 linear classifier on top of features p * ·|s ∈ R V . Low dimensional softmax models, however, only learn p * ·|s in the subspace of Φ, as per Proposition 2.2. Thus we cannot ensure guarantees on all natural tasks and we are interested in tasks that this subspace can solve.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This is a subset of natural tasks where the classifier v on p * ·|s is restricted to be in the span of
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This is a subset of natural tasks where the classifier v on p * ·|s is restricted to be in the span of
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. We argue that restricting v to be in the span of Φ will not preclude solving many tasks of interest if Φ satisfies some nice properties. For instance, if interesting tasks in practice tend to assign similar weights to pairs of words that are synonyms, then Φ just needs to assigns similar embeddings for those pairs. In fact, Section 5.2 describes a carefully designed objective that can learn word embeddings where synonyms have similar embeddings.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. We argue that restricting v to be in the span of Φ will not preclude solving many tasks of interest if Φ satisfies some nice properties. For instance, if interesting tasks in practice tend to assign similar weights to pairs of words that are synonyms, then Φ just needs to assigns similar embeddings for those pairs. In fact, Section 5.2 describes a carefully designed objective that can learn word embeddings where synonyms have similar embeddings.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now show guarantees for features from language models on natural tasks. For an unconstrained model, we use the learned p ·|s ∈ R V as features and for softmax model f , we show guarantees for Φp f (s) ∈ R d . Since we cannot practically hope to learn the optimal solutions described in Propositions 2.1 and 2.2 , we only assume that the language models are -optimal in cross-entropy. To define -optimality in the two settings, we first define optimal cross-entropies. *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now show guarantees for features from language models on natural tasks. For an unconstrained model, we use the learned p ·|s ∈ R V as features and for softmax model f , we show guarantees for Φp f (s) ∈ R d . Since we cannot practically hope to learn the optimal solutions described in Propositions 2.1 and 2.2 , we only assume that the language models are -optimal in cross-entropy. To define -optimality in the two settings, we first define optimal cross-entropies. *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where * xent is the absolute minimum achievable cross-entropy, while * xent (Φ) is the minimum achievable cross-entropy by a d-dimensional softmax language model using Φ; clearly * xent ≤ * xent (Φ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where * xent is the absolute minimum achievable cross-entropy, while * xent (Φ) is the minimum achievable cross-entropy by a d-dimensional softmax language model using Φ; clearly * xent ≤ * xent (Φ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We show guarantees for a language model that satisfies xent ({p ·|s }) − * xent ≤ . An important consideration is that the language model distribution p L of contexts is often a diverse superset of the downstream distribution p T (defined in Section 2.3), thus requiring us to show how guarantees of p ·|s ≈ p * ·|s on average over the distribution s ∼ p L transfer to guarantees on a subset p T . In the worst case, all of the error by {p ·|s } occurs on sentences from the subset p T , leading to pessimistic bounds 4 . In practice, however, the errors might be more evenly distributed across p L , thus bypassing this worst case bound. As a first step, we present the worst case bound here; stronger guarantees are in Section 5.1. The worst-case coefficient γ(p T ), defined below, captures that p T is a γ(p T )-fraction of p L .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We show guarantees for a language model that satisfies xent ({p ·|s }) − * xent ≤ . An important consideration is that the language model distribution p L of contexts is often a diverse superset of the downstream distribution p T (defined in Section 2.3), thus requiring us to show how guarantees of p ·|s ≈ p * ·|s on average over the distribution s ∼ p L transfer to guarantees on a subset p T . In the worst case, all of the error by {p ·|s } occurs on sentences from the subset p T , leading to pessimistic bounds 4 . In practice, however, the errors might be more evenly distributed across p L , thus bypassing this worst case bound. As a first step, we present the worst case bound here; stronger guarantees are in Section 5.1. The worst-case coefficient γ(p T ), defined below, captures that p T is a γ(p T )-fraction of p L .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now present our results that applies to any language model, regardless of the parametrization (e.g., n-gram models, softmax models). The result suggests that small test cross-entropy (hence test perplexity) is desirable to guarantee good classification performance, thus formalizing the intuition that better language models should do better on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now present our results that applies to any language model, regardless of the parametrization (e.g., n-gram models, softmax models). The result suggests that small test cross-entropy (hence test perplexity) is desirable to guarantee good classification performance, thus formalizing the intuition that better language models should do better on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.1. Let {p ·|s } be a language model that is -optimal, i.e. xent ({p ·|s }) − * xent ≤ , for some > 0. For a classification task T that is (τ, B)-natural, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.1. Let {p ·|s } be a language model that is -optimal, i.e. xent ({p ·|s }) − * xent ≤ , for some > 0. For a classification task T that is (τ, B)-natural, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Discussion: The suboptimality propagates gracefully as O( √ ) to a downstream task. While τ can be thought of as a small constant like 0.01, the norm bound B captures the margin of task T when solved linearly using {p * ·|s }. Intuitively, for T ({p * ·|s }, v) to be smaller than τ , B needs to be large enough so that v p * ·|s = Ω(1). Thus if the words of interest in the support of v, described in Section 3, have total probability mass of Ω(α) in p * ·|s , then B ∼ O(1/α). It is thus desirable for a task T to depend on a larger and more frequent set of words. A task that depends on probabilities of rare words will have a high value of B. Adding a prompt, as described in Section 3, can broaden the set of indicative words, thus potentially decreasing B. A key step in the proof is to bound the difference in prediction on s for classifier v as |v (p ·|s − p * ·|s )| ≤ v ∞ p ·|s − p * ·|s 1 ≤ v ∞ 2( xent,s (p ·|s ) − xent,s (p * ·|s )), using Holder's and Pinsker's inequalities respectively.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Discussion: The suboptimality propagates gracefully as O( √ ) to a downstream task. While τ can be thought of as a small constant like 0.01, the norm bound B captures the margin of task T when solved linearly using {p * ·|s }. Intuitively, for T ({p * ·|s }, v) to be smaller than τ , B needs to be large enough so that v p * ·|s = Ω(1). Thus if the words of interest in the support of v, described in Section 3, have total probability mass of Ω(α) in p * ·|s , then B ∼ O(1/α). It is thus desirable for a task T to depend on a larger and more frequent set of words. A task that depends on probabilities of rare words will have a high value of B. Adding a prompt, as described in Section 3, can broaden the set of indicative words, thus potentially decreasing B. A key step in the proof is to bound the difference in prediction on s for classifier v as |v (p ·|s − p * ·|s )| ≤ v ∞ p ·|s − p * ·|s 1 ≤ v ∞ 2( xent,s (p ·|s ) − xent,s (p * ·|s )), using Holder's and Pinsker's inequalities respectively.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now describe guarantees for a softmax language model with feature map f that satisfies xent (f, Φ) − * xent (Φ) ≤ ; suboptimality is measured w.r.t. the best d-dimensional model, unlike Theorem 4.1. Note that Theorem 4.1 can be applied here to give a bound of
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now describe guarantees for a softmax language model with feature map f that satisfies xent (f, Φ) − * xent (Φ) ≤ ; suboptimality is measured w.r.t. the best d-dimensional model, unlike Theorem 4.1. Note that Theorem 4.1 can be applied here to give a bound of
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
is the suboptimality of the best d-dimensional model. This fixed error of O(B * Φ ) (even when = 0), however, is undesirable. We improve on this by proving a stronger result specifically for softmax models. Inspired by Proposition 2.2 that shows Φp f * (s) = Φp * ·|s , our guarantees are for features Φp f (s) ∈ R d that we call conditional mean features.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
is the suboptimality of the best d-dimensional model. This fixed error of O(B * Φ ) (even when = 0), however, is undesirable. We improve on this by proving a stronger result specifically for softmax models. Inspired by Proposition 2.2 that shows Φp f * (s) = Φp * ·|s , our guarantees are for features Φp f (s) ∈ R d that we call conditional mean features.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Definition 4.1 (Conditional Mean Features). For a feature map f : S → R d and Φ ∈ R d×V , we define conditional mean features g f,Φ :
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Definition 4.1 (Conditional Mean Features). For a feature map f : S → R d and Φ ∈ R d×V , we define conditional mean features g f,Φ :
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The result below will show that conditional mean features g f,Φ are guaranteed to do well on natural tasks w.r.t. Φ, thereby suggesting a novel way to use softmax features f for downstream tasks. We also test g f,Φ on downstream tasks in Section 6 and find that they perform comparably to f . We now present the result for softmax language models that has the similar implication as Theorem 4.1, but with above-mentioned subtle differences. The proof (Section D.2) is similar to that of Theorem 4.1, but crucially requires showing a d-dimensional version of Pinkser's inequality.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The result below will show that conditional mean features g f,Φ are guaranteed to do well on natural tasks w.r.t. Φ, thereby suggesting a novel way to use softmax features f for downstream tasks. We also test g f,Φ on downstream tasks in Section 6 and find that they perform comparably to f . We now present the result for softmax language models that has the similar implication as Theorem 4.1, but with above-mentioned subtle differences. The proof (Section D.2) is similar to that of Theorem 4.1, but crucially requires showing a d-dimensional version of Pinkser's inequality.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.2 shows that g f,Φ is useful for linear classification. However, using feature map f directly is more standard and performs well in practice (see Section 6). Here we argue that there is a linear relation between f and g f,Φ if word embeddings Φ satisfy a certain property, which we show implies that tasks solvable linearly with g f,Φ are also solvable linearly using f . Our main assumption about word embeddings Φ is that the logarithm of the partition function Z f (s) is quadratic in f (s).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.2 shows that g f,Φ is useful for linear classification. However, using feature map f directly is more standard and performs well in practice (see Section 6). Here we argue that there is a linear relation between f and g f,Φ if word embeddings Φ satisfy a certain property, which we show implies that tasks solvable linearly with g f,Φ are also solvable linearly using f . Our main assumption about word embeddings Φ is that the logarithm of the partition function Z f (s) is quadratic in f (s).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. While some papers [Arora et al., 2016, Mu andViswanath, 2018] have noted that word embeddings are fairly random-like in the bulk to argue that the log partition function is constant for all θ 2 = 1, our quadratic assumption is a bit stronger. However, empirically we find the fit to be very good, as evident in Figure 1. Under the above assumption, we can show a linear relation between f and Φp f .  Thus we get that f itself is good for natural linear classification tasks. However, in practice, the linearity between f and g f,Φ holds only approximately when tested on features from the pretrained GPT-2 language model Radford et al. [2018]. The ratio of the residual norm of the best linear map to the
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. While some papers [Arora et al., 2016, Mu andViswanath, 2018] have noted that word embeddings are fairly random-like in the bulk to argue that the log partition function is constant for all θ 2 = 1, our quadratic assumption is a bit stronger. However, empirically we find the fit to be very good, as evident in Figure 1. Under the above assumption, we can show a linear relation between f and Φp f .  Thus we get that f itself is good for natural linear classification tasks. However, in practice, the linearity between f and g f,Φ holds only approximately when tested on features from the pretrained GPT-2 language model Radford et al. [2018]. The ratio of the residual norm of the best linear map to the
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, is measured for different distributions p (r = 0 means perfect fit). These ratios are 0.28 for SST, 0.39 for AG News, and 0.18 for IMDb contexts. This non-trivial linear relationship, although surprising, might not completely explain the success of f . In fact, f almost always performs better than g f,Φ ; we leave exploring this to future work.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, is measured for different distributions p (r = 0 means perfect fit). These ratios are 0.28 for SST, 0.39 for AG News, and 0.18 for IMDb contexts. This non-trivial linear relationship, although surprising, might not completely explain the success of f . In fact, f almost always performs better than g f,Φ ; we leave exploring this to future work.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The bounds in the previous section use the coefficient γ(p T ) to transfer guarantees from p L to p T and we define a more refined notion of transferability here. The coefficient γ(p T ) is independent of the learned model and assumes a worst case distribution of errors. For the refined coefficient, we first define the error made in predicted probabilities by a softmax language model f as any distribution p ∈ ∆ S , we define the covariance 5 of a function g :
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The bounds in the previous section use the coefficient γ(p T ) to transfer guarantees from p L to p T and we define a more refined notion of transferability here. The coefficient γ(p T ) is independent of the learned model and assumes a worst case distribution of errors. For the refined coefficient, we first define the error made in predicted probabilities by a softmax language model f as any distribution p ∈ ∆ S , we define the covariance 5 of a function g :
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The refined transferability coefficient is then defined as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The refined transferability coefficient is then defined as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now state the result for softmax language models; detailed results (including those for unconstrained language models) can be found in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now state the result for softmax language models; detailed results (including those for unconstrained language models) can be found in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Simplified). In the same setting as Theorem 4.2,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Simplified). In the same setting as Theorem 4.2,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
measures of how average error by a language model f on p L can propagate to p T . This can potentially be much smaller than γ(p T ) due to some inductive biases of f . For instance, if errors made by the model
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
measures of how average error by a language model f on p L can propagate to p T . This can potentially be much smaller than γ(p T ) due to some inductive biases of f . For instance, if errors made by the model
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, making γ(p; {p ·|s }) ≈ 1. The independence between ∆ {p f (s) } (s) and s prevents language modeling error from accumulating on contexts from p T , bypassing the worst case transfer of γ(p T ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, making γ(p; {p ·|s }) ≈ 1. The independence between ∆ {p f (s) } (s) and s prevents language modeling error from accumulating on contexts from p T , bypassing the worst case transfer of γ(p T ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Definition 3.2 we discuss how low dimensional softmax language models learn a linear projection of p * ·|s , only solving tasks that lie in the row span of word embeddings Φ. Although Φ determines the tasks that language model features can solve, the standard cross-entropy objective does not lend a simple closed form expression for optimal Φ. This motivates the construction of our Quad objective, that has two nice properties: (1) the optimal feature map f * is a linear function of p * ·|s and thus can solve some natural tasks, and (2) the optimal Φ * has an intuitively meaningful closed-form solution.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Definition 3.2 we discuss how low dimensional softmax language models learn a linear projection of p * ·|s , only solving tasks that lie in the row span of word embeddings Φ. Although Φ determines the tasks that language model features can solve, the standard cross-entropy objective does not lend a simple closed form expression for optimal Φ. This motivates the construction of our Quad objective, that has two nice properties: (1) the optimal feature map f * is a linear function of p * ·|s and thus can solve some natural tasks, and (2) the optimal Φ * has an intuitively meaningful closed-form solution.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The Quad objective is very similar to the cross-entropy objective from Equation (2), with the log partition function log(Z f (s) ) replaced with a quadratic function 1 2 Φ f (s) 2 , inspired in part by Assumption 4.1. We can derive the optimal solution Φ * that depends on the eigen-decomposition of a substitutability matrix. Table 1: Pretrained GPT-2 performance on linear classification tasks using features f (s), p f (s) and g f,Φ (s). An asterisk indicates that we added a task-specific prompt.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The Quad objective is very similar to the cross-entropy objective from Equation (2), with the log partition function log(Z f (s) ) replaced with a quadratic function 1 2 Φ f (s) 2 , inspired in part by Assumption 4.1. We can derive the optimal solution Φ * that depends on the eigen-decomposition of a substitutability matrix. Table 1: Pretrained GPT-2 performance on linear classification tasks using features f (s), p f (s) and g f,Φ (s). An asterisk indicates that we added a task-specific prompt.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Features 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Features 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The matrix Ω * captures substitutability between pairs of words. Words w and w are substitutable if they have identical conditional probabilities for every context s ∈ S and thus can replace occurrences of each other while still providing meaningful completions. By definition, these words satisfy Ω *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The matrix Ω * captures substitutability between pairs of words. Words w and w are substitutable if they have identical conditional probabilities for every context s ∈ S and thus can replace occurrences of each other while still providing meaningful completions. By definition, these words satisfy Ω *
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Such pairs of words were called "free variants" in the work on distributional semantics [Harris, 1954], and capture the notion of synonyms in the distributional hypothesis.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Such pairs of words were called "free variants" in the work on distributional semantics [Harris, 1954], and capture the notion of synonyms in the distributional hypothesis.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus f * excels on natural tasks w.r.t. Φ * , which in turn, is the best d-dimensional projection of Ω * . Thus words w, w ∈ W that are synonyms (hence substitutable) will satisfy φ * w = φ * w , fulfilling the desired property for word embeddings discussed in Definition 3.2. We train using the Quad objective and compare its performance to a similarly trained language model in Section 6, finding Quad to be reasonably effective. The goal of testing Quad is not to obtain state-of-the-art results, but to demonstrate that theoretical insights can aid the design of provably effective algorithms.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus f * excels on natural tasks w.r.t. Φ * , which in turn, is the best d-dimensional projection of Ω * . Thus words w, w ∈ W that are synonyms (hence substitutable) will satisfy φ * w = φ * w , fulfilling the desired property for word embeddings discussed in Definition 3.2. We train using the Quad objective and compare its performance to a similarly trained language model in Section 6, finding Quad to be reasonably effective. The goal of testing Quad is not to obtain state-of-the-art results, but to demonstrate that theoretical insights can aid the design of provably effective algorithms.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Tasks using linear function of p ·|s : We validate our claims from Section 3 that classification tasks can be solved by linear functions of p ·|s . Table 1 demonstrates that on the Stanford Sentiment Treebank (SST) [Socher et al., 2013] and AG News [Zhang et al., 2015] tasks we can use the conditional probabilities p ·|s = p f (s) from GPT-2 [Radford et al., 2019] of just 20 task-relevant tokens (see Section E.1) to solve tasks. Even just one token per class yields non-trivial performance. Furthermore, we validate the complete-the-sentence intuition in Section 3 by using the probabilities after adding a task specific prompt and observing improved performance. g f,Φ and f are good features: We validate Theorem 4.2 by verifying that the conditional mean features g f,Φ (s) = Φp f (s) also linearly solve downstream tasks fairly well. This performance is comparable to, but always worse than f (s), as seen in columns 2 and 3 of Table 1. We again find that adding a prompt improves performance. Section E.1 has results for a wider range of classification tasks. Evidence for Assumption 4.1 is provided by learning a quadratic function to fit the log partition function of features from pretrained GPT-2 model (see Section E.3). Figure 1 demonstrates that the fit holds for its training and unseen data (e.g., WebText [Radford et al., 2019]).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Tasks using linear function of p ·|s : We validate our claims from Section 3 that classification tasks can be solved by linear functions of p ·|s . Table 1 demonstrates that on the Stanford Sentiment Treebank (SST) [Socher et al., 2013] and AG News [Zhang et al., 2015] tasks we can use the conditional probabilities p ·|s = p f (s) from GPT-2 [Radford et al., 2019] of just 20 task-relevant tokens (see Section E.1) to solve tasks. Even just one token per class yields non-trivial performance. Furthermore, we validate the complete-the-sentence intuition in Section 3 by using the probabilities after adding a task specific prompt and observing improved performance. g f,Φ and f are good features: We validate Theorem 4.2 by verifying that the conditional mean features g f,Φ (s) = Φp f (s) also linearly solve downstream tasks fairly well. This performance is comparable to, but always worse than f (s), as seen in columns 2 and 3 of Table 1. We again find that adding a prompt improves performance. Section E.1 has results for a wider range of classification tasks. Evidence for Assumption 4.1 is provided by learning a quadratic function to fit the log partition function of features from pretrained GPT-2 model (see Section E.3). Figure 1 demonstrates that the fit holds for its training and unseen data (e.g., WebText [Radford et al., 2019]).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Quad objective: We compare downstream performance of the Quad objective to the cross-entropy objective by training GPT-2 on the IMDb dataset [Maas et al., 2011]. Table 2 shows that features learned by Quad perform comparably to g f,Φ for f learned by the cross-entropy objective, which fits  
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Quad objective: We compare downstream performance of the Quad objective to the cross-entropy objective by training GPT-2 on the IMDb dataset [Maas et al., 2011]. Table 2 shows that features learned by Quad perform comparably to g f,Φ for f learned by the cross-entropy objective, which fits  
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We provide intuitive and mathematical explanations for the success of language model features on linear classification tasks. The sentence completion intuition helps argue that p * ·|s can solve natural tasks linearly. Insights from our analysis help design the Quad objective that provably learns good features for these natural tasks. We hope our analysis will inspire other mathematical insights into language models. While Section 4.3 argues linearity between conditional mean features g f,Φ and f , it is insufficient to explain the observed superiority of f over g f,Φ . We leave exploring this limitation of our analysis to future work. Guarantees for softmax models are for natural tasks w.r.t. Φ, thus knowing the optimal d-dimensional word embeddings Φ * for xent (f, Φ) is also important. Other meaningful directions include providing guarantees for other successful models like BERT [Devlin et al., 2018] and more diverse downstream tasks. Although we would like to show stronger guarantees by exploiting model and algorithmic inductive biases, as well as study the setting of fine-tuning language model features, lack of a good theory of deep learning is the current bottleneck to doing so. 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We provide intuitive and mathematical explanations for the success of language model features on linear classification tasks. The sentence completion intuition helps argue that p * ·|s can solve natural tasks linearly. Insights from our analysis help design the Quad objective that provably learns good features for these natural tasks. We hope our analysis will inspire other mathematical insights into language models. While Section 4.3 argues linearity between conditional mean features g f,Φ and f , it is insufficient to explain the observed superiority of f over g f,Φ . We leave exploring this limitation of our analysis to future work. Guarantees for softmax models are for natural tasks w.r.t. Φ, thus knowing the optimal d-dimensional word embeddings Φ * for xent (f, Φ) is also important. Other meaningful directions include providing guarantees for other successful models like BERT [Devlin et al., 2018] and more diverse downstream tasks. Although we would like to show stronger guarantees by exploiting model and algorithmic inductive biases, as well as study the setting of fine-tuning language model features, lack of a good theory of deep learning is the current bottleneck to doing so. 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Section B is a more detailed version of Section 5.1 and Section C is a detailed version of Section 5.2. Section D contains proofs for all results. Section E contains many more experimental findings that consolidate many of our theoretical results. Section E.1 provides the information about subsets of words used for results in Table 1 and also additional experiments to test the performance of pretrained language model embeddings f on more downstream tasks and also verifying that conditional mean embeddings Φp f do well on these tasks. In Section E.2, we present additional results for our Quad objective trained on a larger corpus and tested on SST. Finally Section E.3 provides additional details on how A, b and c from Assumption 4.1 are learned and also further verification of the assumption on more datasets.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Section B is a more detailed version of Section 5.1 and Section C is a detailed version of Section 5.2. Section D contains proofs for all results. Section E contains many more experimental findings that consolidate many of our theoretical results. Section E.1 provides the information about subsets of words used for results in Table 1 and also additional experiments to test the performance of pretrained language model embeddings f on more downstream tasks and also verifying that conditional mean embeddings Φp f do well on these tasks. In Section E.2, we present additional results for our Quad objective trained on a larger corpus and tested on SST. Finally Section E.3 provides additional details on how A, b and c from Assumption 4.1 are learned and also further verification of the assumption on more datasets.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
While the bounds above used γ(p T ) to transfer from the distribution p L to p T , we define a more refined notion of transferability here. While γ(p T ) only depends on p L and p T , the more refined notions depend also on the learned language model, thus potentially exploiting some inductive biases. We first define the notion of error made in the predicted probabilities by any predictor p ·|s as ∆ {p ·|s } (s) = p ·|s − p * ·|s . Thus for any softmax language model f we have ∆ {p f (s) } (s) = p f (s) − p * ·|s . For any distribution p ∈ ∆ S , we define the covariance 6 of a function g : S → R D as Σ p (g) = E s∼p g(s)g(s) . We define 3 coefficients for the results to follow Definition B.1. For any distribution p ∈ ∆ S , we define the following
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
While the bounds above used γ(p T ) to transfer from the distribution p L to p T , we define a more refined notion of transferability here. While γ(p T ) only depends on p L and p T , the more refined notions depend also on the learned language model, thus potentially exploiting some inductive biases. We first define the notion of error made in the predicted probabilities by any predictor p ·|s as ∆ {p ·|s } (s) = p ·|s − p * ·|s . Thus for any softmax language model f we have ∆ {p f (s) } (s) = p f (s) − p * ·|s . For any distribution p ∈ ∆ S , we define the covariance 6 of a function g : S → R D as Σ p (g) = E s∼p g(s)g(s) . We define 3 coefficients for the results to follow Definition B.1. For any distribution p ∈ ∆ S , we define the following
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We notice that Σ p (∆ {p ·|s } ) = E s∼p (p ·|s − p * ·|s )(p ·|s − p * ·|s ) , Σ p (Φ∆ {p ·|s } ) = ΦΣ p (∆ {p ·|s } )Φ . We are now ready to state the most general results.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We notice that Σ p (∆ {p ·|s } ) = E s∼p (p ·|s − p * ·|s )(p ·|s − p * ·|s ) , Σ p (Φ∆ {p ·|s } ) = ΦΣ p (∆ {p ·|s } )Φ . We are now ready to state the most general results.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem B.1 (Strengthened Theorem 4.1). Let {p ·|s } be a language model that is -optimal, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem B.1 (Strengthened Theorem 4.1). Let {p ·|s } be a language model that is -optimal, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Strengthened Theorem 4.2). For a fixed Φ, let f be features from an -optimal ddimensional softmax language model, i.e. xent (f,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Strengthened Theorem 4.2). For a fixed Φ, let f be features from an -optimal ddimensional softmax language model, i.e. xent (f,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Discussions: It is not hard to show that the coefficients satisfy γ Φ (p T ; {p ·|s }) ≥ γ(p T ; {p ·|s }) ≥ γ(p T ) and γ(p T ; g f,Φ ) ≥ γ(p T ), thus showing that these results are strictly stronger than the ones from the previous section. The transferability coefficient is a measure of how guarantees on p L using a language model can be transferred to another distribution of contexts and it only depends on the distribution of contexts and not the labels. Unlike γ(p T ), the coefficients in Definition B.1 depend on the learned models, either {p ·|s } or {p f (s) }, and can be potentially much smaller due to the inductive bias of the learned models. For instance, if errors made by the model are random-like, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Discussions: It is not hard to show that the coefficients satisfy γ Φ (p T ; {p ·|s }) ≥ γ(p T ; {p ·|s }) ≥ γ(p T ) and γ(p T ; g f,Φ ) ≥ γ(p T ), thus showing that these results are strictly stronger than the ones from the previous section. The transferability coefficient is a measure of how guarantees on p L using a language model can be transferred to another distribution of contexts and it only depends on the distribution of contexts and not the labels. Unlike γ(p T ), the coefficients in Definition B.1 depend on the learned models, either {p ·|s } or {p f (s) }, and can be potentially much smaller due to the inductive bias of the learned models. For instance, if errors made by the model are random-like, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, making γ(p; {p ·|s }) ≈ 1. The independence between ∆ {p f (s) } (s) and s prevents language modeling error from accumulating on contexts from p T , bypassing the worst case transfer of γ(p T ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, making γ(p; {p ·|s }) ≈ 1. The independence between ∆ {p f (s) } (s) and s prevents language modeling error from accumulating on contexts from p T , bypassing the worst case transfer of γ(p T ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Definition 3.2 we discuss how low dimensional softmax language models learn a linear projection of p * ·|s , only solving tasks that lie in the row span of word embeddings Φ. Although Φ defines tasks that language model features can solve, the standard cross-entropy objective does not lend a simple closed form expression for optimal Φ. This motivates the construction of our Quad objective, that has two nice properties: (1) the optimal feature map f * is a linear function of p * ·|s and thus can solve some natural tasks, and (2) the optimal Φ * has an intuitively meaningful closed-form solution.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Definition 3.2 we discuss how low dimensional softmax language models learn a linear projection of p * ·|s , only solving tasks that lie in the row span of word embeddings Φ. Although Φ defines tasks that language model features can solve, the standard cross-entropy objective does not lend a simple closed form expression for optimal Φ. This motivates the construction of our Quad objective, that has two nice properties: (1) the optimal feature map f * is a linear function of p * ·|s and thus can solve some natural tasks, and (2) the optimal Φ * has an intuitively meaningful closed-form solution.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The Quad objective is very similar to the cross-entropy objective from Equation (2), with the log partition function replaced by a quadratic function, inspired in part by Assumption 4.1. We can derive the optimal solution Φ * that depends on the eigen-decomposition of a substitutability matrix.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The Quad objective is very similar to the cross-entropy objective from Equation (2), with the log partition function replaced by a quadratic function, inspired in part by Assumption 4.1. We can derive the optimal solution Φ * that depends on the eigen-decomposition of a substitutability matrix.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Definition 5.1. The substitutability matrix is defined to be Ω
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Definition 5.1. The substitutability matrix is defined to be Ω
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The matrix Ω * captures substitutability between pairs of words. Words w and w are substitutable if they have identical conditional probabilities for every context s ∈ S and thus can replace occurrences of each other while still providing meaningful completions. By definition, these words satisfy Ω * [w] = Ω * [w ]. Such pairs of words were called "free variants" in the work on distributional semantics [Harris, 1954], and capture the notion of synonyms in the distributional hypothesis. We now derive expressions for the optimal solution of the Quad objective described in Equation (11).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The matrix Ω * captures substitutability between pairs of words. Words w and w are substitutable if they have identical conditional probabilities for every context s ∈ S and thus can replace occurrences of each other while still providing meaningful completions. By definition, these words satisfy Ω * [w] = Ω * [w ]. Such pairs of words were called "free variants" in the work on distributional semantics [Harris, 1954], and capture the notion of synonyms in the distributional hypothesis. We now derive expressions for the optimal solution of the Quad objective described in Equation (11).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem C.1. The optimal solution f * , Φ * = arg min f,Φ quad (f, Φ) satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem C.1. The optimal solution f * , Φ * = arg min f,Φ quad (f, Φ) satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If Φ is fixed, then the optimal solution is f * (s) = (ΦΦ ) − 1 /2 Φp * ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If Φ is fixed, then the optimal solution is f * (s) = (ΦΦ ) − 1 /2 Φp * ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.2. Let f * , Φ * = arg min f,Φ quad (f, Φ). Then Φ * = BU d , for full rank B ∈ R d×d . Also, for a classification task T that is (τ, B)-natural w.r.t. Φ * , we have T (f * ) ≤ τ .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.2. Let f * , Φ * = arg min f,Φ quad (f, Φ). Then Φ * = BU d , for full rank B ∈ R d×d . Also, for a classification task T that is (τ, B)-natural w.r.t. Φ * , we have T (f * ) ≤ τ .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus f * excels on natural tasks w.r.t. Φ * , which in turn, is the best d-dimensional projection of Ω * . Thus words w, w ∈ W that are synonyms (hence substitutable) will satisfy φ * w = φ * w , fulfilling the desired property for word embeddings discussed in Definition 3.2. We train using the Quad objective and compare its performance to a similarly trained language model in Section 6, finding Quad to be reasonably effective. The goal of testing Quad is not to obtain state-of-the-art results, but to demonstrate that theoretical insights can aid the design of provably effective algorithms.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Thus f * excels on natural tasks w.r.t. Φ * , which in turn, is the best d-dimensional projection of Ω * . Thus words w, w ∈ W that are synonyms (hence substitutable) will satisfy φ * w = φ * w , fulfilling the desired property for word embeddings discussed in Definition 3.2. We train using the Quad objective and compare its performance to a similarly trained language model in Section 6, finding Quad to be reasonably effective. The goal of testing Quad is not to obtain state-of-the-art results, but to demonstrate that theoretical insights can aid the design of provably effective algorithms.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem B.1 (Strengthened Theorem 4.1). Let {p ·|s } be a language model that is -optimal, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem B.1 (Strengthened Theorem 4.1). Let {p ·|s } be a language model that is -optimal, i.e.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The proof has two main steps that we summarize by the following two lemmas. The first one upper bounds the downstream performance on natural tasks with the covariance of errors.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The proof has two main steps that we summarize by the following two lemmas. The first one upper bounds the downstream performance on natural tasks with the covariance of errors.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where γ(·) and γ Φ (·) are from Definition B.1.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where γ(·) and γ Φ (·) are from Definition B.1.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The second lemma upper bounds the covariance of error with the suboptimality of the language model. 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
The second lemma upper bounds the covariance of error with the suboptimality of the language model. 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
(p ·|s − p * ·|s )(p ·|s − p * ·|s ) as defined in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
(p ·|s − p * ·|s )(p ·|s − p * ·|s ) as defined in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Combining the two lemmas, we get the following inequality
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Combining the two lemmas, we get the following inequality
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses first part of Lemma D.1, (b) uses Lemma D.2 and (c) uses the -optimality of {p ·|s }. This proves the first part of the result. The second part can also be proved similarly.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses first part of Lemma D.1, (b) uses Lemma D.2 and (c) uses the -optimality of {p ·|s }. This proves the first part of the result. The second part can also be proved similarly.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses second part of Lemma D.1, (b) uses Lemma D.2 and (c) uses the -optimality of {p ·|s }. The proof of the lemmas can be found in Section D.5.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses second part of Lemma D.1, (b) uses Lemma D.2 and (c) uses the -optimality of {p ·|s }. The proof of the lemmas can be found in Section D.5.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.1. Let {p ·|s } be a language model that is -optimal, i.e. xent ({p ·|s }) − * xent ≤ , for some > 0. For a classification task T that is (τ, B)-natural, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.1. Let {p ·|s } be a language model that is -optimal, i.e. xent ({p ·|s }) − * xent ≤ , for some > 0. For a classification task T that is (τ, B)-natural, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. This follows from the first part of Theorem B.1 if we can also show that γ(p T ; {p ·|s }) −1 ≤ γ(p T ) −1 . For that we use the following lemma that we prove in Section D.5.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. This follows from the first part of Theorem B.1 if we can also show that γ(p T ; {p ·|s }) −1 ≤ γ(p T ) −1 . For that we use the following lemma that we prove in Section D.5.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Instantiating this for g = ∆ {p ·|s } and using Equation (7), we get γ(p T ; {p ·|s }) −1 ≤ γ(p T ) −1 , which completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Instantiating this for g = ∆ {p ·|s } and using Equation (7), we get γ(p T ; {p ·|s }) −1 ≤ γ(p T ) −1 , which completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Strengthened Theorem 4.2). For a fixed Φ, let f be features from an -optimal ddimensional softmax language model, i.e. xent (f, Φ) − * xent (Φ) ≤ , where * xent (Φ) is defined in Equation (4). For a classification task T that is (τ, B)-natural w.r.t. Φ, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 5.1 (Strengthened Theorem 4.2). For a fixed Φ, let f be features from an -optimal ddimensional softmax language model, i.e. xent (f, Φ) − * xent (Φ) ≤ , where * xent (Φ) is defined in Equation (4). For a classification task T that is (τ, B)-natural w.r.t. Φ, we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Instantiating Lemma D.1 for p ·|s = p f (s) , we get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Instantiating Lemma D.1 for p ·|s = p f (s) , we get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from Equation (9) 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from Equation (9) 
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. We now prove a similar result for the second term in the following lemma Lemma D.4. For a fixed Φ and a softmax language model with features f and λ ∈ R d ,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
. We now prove a similar result for the second term in the following lemma Lemma D.4. For a fixed Φ and a softmax language model with features f and λ ∈ R d ,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, and theoptimality almost completes the proof. The only thing remaining to show is that T ({p f (s) }) ≤ T (g f,Φ ) which follows from the following sequence.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
, and theoptimality almost completes the proof. The only thing remaining to show is that T ({p f (s) }) ≤ T (g f,Φ ) which follows from the following sequence.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.2. For a fixed Φ, let f be features from an -optimal d-dimensional softmax language model,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem 4.2. For a fixed Φ, let f be features from an -optimal d-dimensional softmax language model,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. This result follows directly from Theorem 5.1, if we can also show that γ(p T ; g f,Φ ) −1 ≤ γ(p T ) −1 just like in the proof of Theorem 4.1. For that we again use Lemma D.3 with g = Φ∆ {p f (s) } and Equation (9) and this completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. This result follows directly from Theorem 5.1, if we can also show that γ(p T ; g f,Φ ) −1 ≤ γ(p T ) −1 just like in the proof of Theorem 4.1. For that we again use Lemma D.3 with g = Φ∆ {p f (s) } and Equation (9) and this completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We first show why Assumption 4.1 is approximately true when word embeddings are gaussian like.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We first show why Assumption 4.1 is approximately true when word embeddings are gaussian like.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.5. Suppose word embeddings φ w are independent samples from the distribution N (µ, Σ). Then for any θ ∈ R d such that λ 2 = θ Σθ = O(1) we have that | log(Z θ ) − 1 2 θ Σθ − θ µ − log(V )| ≤ with probability 1 − δ for =Õ e λ 2 √ V and δ = 1 − exp(−Ω(log 2 (V ))).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.5. Suppose word embeddings φ w are independent samples from the distribution N (µ, Σ). Then for any θ ∈ R d such that λ 2 = θ Σθ = O(1) we have that | log(Z θ ) − 1 2 θ Σθ − θ µ − log(V )| ≤ with probability 1 − δ for =Õ e λ 2 √ V and δ = 1 − exp(−Ω(log 2 (V ))).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We first note that log(Z θ ) = log w e θ φw = θ µ + log w e θ (φw−µ) , thus we can simply deal with the case where φ w are sampled from N (0, Σ). Furthermore the only random variable of interest is X w = θ φ w which is a gaussian variable N (0, θ Σθ) = N (0, λ 2 ). Thus the problem reduces to showing that for V samples of X w ∼ N (0, λ 2 ), log(Z) is concentrated around λ 2 + log(V ) where Z = w exp(X w ). This can be proved similarly to the proof of Lemma 2.1 in Arora et al. [2016]. It is easy to see that E
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We first note that log(Z θ ) = log w e θ φw = θ µ + log w e θ (φw−µ) , thus we can simply deal with the case where φ w are sampled from N (0, Σ). Furthermore the only random variable of interest is X w = θ φ w which is a gaussian variable N (0, θ Σθ) = N (0, λ 2 ). Thus the problem reduces to showing that for V samples of X w ∼ N (0, λ 2 ), log(Z) is concentrated around λ 2 + log(V ) where Z = w exp(X w ). This can be proved similarly to the proof of Lemma 2.1 in Arora et al. [2016]. It is easy to see that E
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
[exp(X w )] = e λ 2 . However the variable exp(X w ) is neither sub-gaussian nor sub-exponential and thus standard inequalities cannot be used directly. We use the same technique as Arora et al.  Proof. Assumption 4.1 gives us that log(Z θ ) = 1 2 θ Aθ + θ b + c. We prove this lemma by matching the gradients of log(Z θ ) and the quadratic function on the R.H.S.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
[exp(X w )] = e λ 2 . However the variable exp(X w ) is neither sub-gaussian nor sub-exponential and thus standard inequalities cannot be used directly. We use the same technique as Arora et al.  Proof. Assumption 4.1 gives us that log(Z θ ) = 1 2 θ Aθ + θ b + c. We prove this lemma by matching the gradients of log(Z θ ) and the quadratic function on the R.H.S.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Whereas the gradient of the quadratic part is ∇ θ [ 1 2 θ Aθ + θ b + c] = Aθ + b. Matching the two gives us g f,Φ (s) = Φp f (s) = Af (s) + b.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Whereas the gradient of the quadratic part is ∇ θ [ 1 2 θ Aθ + θ b + c] = Aθ + b. Matching the two gives us g f,Φ (s) = Φp f (s) = Af (s) + b.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Corollary 4.1. Using Lemma 4.3, for any -optimal f , as defined in Theorem 4.2, for classification tasks that are (τ, B)-natural w.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Corollary 4.1. Using Lemma 4.3, for any -optimal f , as defined in Theorem 4.2, for classification tasks that are (τ, B)-natural w.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The main idea is that Lemma 4.3 gives us that g f,Φ (s) = Af (s) + b and thus any linear function of g f,Φ will also be a linear function of f (s). And from Theorem 5.1 (or Theorem 4.2), we know that g f,Φ will do well on T , i.e. T (g f,Φ ) ≤ τ + O(B √ ). We formalize the intuition as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The main idea is that Lemma 4.3 gives us that g f,Φ (s) = Af (s) + b and thus any linear function of g f,Φ will also be a linear function of f (s). And from Theorem 5.1 (or Theorem 4.2), we know that g f,Φ will do well on T , i.e. T (g f,Φ ) ≤ τ + O(B √ ). We formalize the intuition as
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
and completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
and completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem C.1. The optimal solution f * , Φ * = arg min f,Φ quad (f, Φ) satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Theorem C.1. The optimal solution f * , Φ * = arg min f,Φ quad (f, Φ) satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If Φ is fixed, then the optimal solution is f * (s) = (ΦΦ ) − 1 /2 Φp * ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
If Φ is fixed, then the optimal solution is f * (s) = (ΦΦ ) − 1 /2 Φp * ·|s .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Given that, quad,s (θ, Φ) = −θ Φp * ·|s + 1 2 Φ θ 2 , for a fixed Φ, we define f * Φ (s) = arg min θ∈R d quad,s (θ, Φ). We use the first order optimality condition to get f * Φ (s), by using the fact that ∇ θ quad,s (θ, Φ) = −Φp * ·|s + ΦΦ θ. Setting the gradient to zero, we f * Φ (s) = (ΦΦ ) −1 Φp * ·|s . To get the optimal Φ * for this objective, we plug in this expression for f * Φ (s) and observing that the optimal solution also satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Given that, quad,s (θ, Φ) = −θ Φp * ·|s + 1 2 Φ θ 2 , for a fixed Φ, we define f * Φ (s) = arg min θ∈R d quad,s (θ, Φ). We use the first order optimality condition to get f * Φ (s), by using the fact that ∇ θ quad,s (θ, Φ) = −Φp * ·|s + ΦΦ θ. Setting the gradient to zero, we f * Φ (s) = (ΦΦ ) −1 Φp * ·|s . To get the optimal Φ * for this objective, we plug in this expression for f * Φ (s) and observing that the optimal solution also satisfies
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Then the above objective reduces to quad (f * Φ , Φ) = − 1 2 V V , Ω * And hence learning the optimal Φ * reduces to learning an optimal V * such that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Then the above objective reduces to quad (f * Φ , Φ) = − 1 2 V V , Ω * And hence learning the optimal Φ * reduces to learning an optimal V * such that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We will now show that the best such matrix is the matrix of top d eigenvectors of Ω * , i.e. V * = U d .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We will now show that the best such matrix is the matrix of top d eigenvectors of Ω * , i.e. V * = U d .
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Here we will assume that the eigenvalues of Ω * are all distinct for simplicity of presentation.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Here we will assume that the eigenvalues of Ω * are all distinct for simplicity of presentation.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
First we note that V V , Ω * = V V Ω * 1 2 2 F , where Ω * 1 2 = U S 1 2 U , with U , U d and S define in Definition 5.1. This can be shown by the following sequence of steps
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
First we note that V V , Ω * = V V Ω * 1 2 2 F , where Ω * 1 2 = U S 1 2 U , with U , U d and S define in Definition 5.1. This can be shown by the following sequence of steps
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Note that V V Ω * 1 2 has columns that are columns of Ω * 1 2 projected on the space spanned by columns V . It is folklore that the best such subspace V * is the subspace spanned by the top d eigenvectors of Ω * 1 2 , which is the same as top d eigenvectors of Ω * , thus giving us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Note that V V Ω * 1 2 has columns that are columns of Ω * 1 2 projected on the space spanned by columns V . It is folklore that the best such subspace V * is the subspace spanned by the top d eigenvectors of Ω * 1 2 , which is the same as top d eigenvectors of Ω * , thus giving us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This tells us that the optimal solution Φ * will have SVD of the form Φ * = N * T * V * , thus giving us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This tells us that the optimal solution Φ * will have SVD of the form Φ * = N * T * V * , thus giving us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where γ(·) and γ Φ (·) are from Definition B.1.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where γ(·) and γ Φ (·) are from Definition B.1.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We note the following upper bounds on T ({p ·|s }) and T ({Φp ·|s }).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We note the following upper bounds on T ({p ·|s }) and T ({Φp ·|s }).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
When T is (τ, B)-natural, by Definition 3.1 we know that inf
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
When T is (τ, B)-natural, by Definition 3.1 we know that inf
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This, combined with Equation (12), gives us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This, combined with Equation (12), gives us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Using Lemma D.9 and the definition of γ(p T ; {p ·|s }) in Equation (7), we get that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Using Lemma D.9 and the definition of γ(p T ; {p ·|s }) in Equation (7), we get that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We have thus successfully transferred the bound from the distribution p T to p L . Combining this with Equation (14) completes the proof of the first part of the lemma.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We have thus successfully transferred the bound from the distribution p T to p L . Combining this with Equation (14) completes the proof of the first part of the lemma.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now prove the second part of the lemma where we only assume that T is (τ, B)-natural w.r.t. Φ.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We now prove the second part of the lemma where we only assume that T is (τ, B)-natural w.r.t. Φ.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Here we instead take the infimum over classifiers in the span of Φ in Lemma D.8 to get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Here we instead take the infimum over classifiers in the span of Φ in Lemma D.8 to get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This, combined with definition of (τ, B)-natural task w.r.t. Φ and Equation (13) gives us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This, combined with definition of (τ, B)-natural task w.r.t. Φ and Equation (13) gives us
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For the last term, for any
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For the last term, for any
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This combined with Equation (16), we get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This combined with Equation (16), we get
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We first note that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We first note that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses Holder's inequality, (b) uses Pinsker's inequality, (c) uses Equation (17).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) uses Holder's inequality, (b) uses Pinsker's inequality, (c) uses Equation (17).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. By definition of γ(p T ), we have that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. By definition of γ(p T ), we have that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This finishes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
This finishes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.4. For a fixed Φ, a softmax language model with features f and λ ∈ R d ,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.4. For a fixed Φ, a softmax language model with features f and λ ∈ R d ,
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We start by nothing that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. We start by nothing that
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We will use Lemma D.6 to bound each term on the right hand side, which essentially bounds the norm of the gradient of xent,s at f (s) when f (s) is an almost optimal for s ∈ S.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We will use Lemma D.6 to bound each term on the right hand side, which essentially bounds the norm of the gradient of xent,s at f (s) when f (s) is an almost optimal for s ∈ S.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from Lemma D.6. This completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from Lemma D.6. This completes the proof.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.6. For s ∈ S and embedding f (s) ∈ R d , we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.6. For s ∈ S and embedding f (s) ∈ R d , we have
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Since we are assuming Φ to be fixed, we will abuse notation and say xent,s (θ) := xent,s (θ, Φ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Since we are assuming Φ to be fixed, we will abuse notation and say xent,s (θ) := xent,s (θ, Φ).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
All gradients are w.r.t. θ. Before we get to the main proof, we compute the gradient and hessian of xent,s (θ) w.r.t. θ. The gradient is
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
All gradients are w.r.t. θ. Before we get to the main proof, we compute the gradient and hessian of xent,s (θ) w.r.t. θ. The gradient is
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Similarly the Hessian can be computed
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Similarly the Hessian can be computed
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
[g(s)g(s) ] and ∆ {p ·|s } (s) = p ·|s − p * ·|s are defined in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
[g(s)g(s) ] and ∆ {p ·|s } (s) = p ·|s − p * ·|s are defined in Section B.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The following sequence of inequalities proves it
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. The following sequence of inequalities proves it
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from 1-lipschitzness of , (b) follows from Jensen's inequality.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
where (a) follows from 1-lipschitzness of , (b) follows from Jensen's inequality.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.9. For matrices X, Y ∈ R D×D s.t. X, Y 0 and Y is full rank, we have that max
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Lemma D.9. For matrices X, Y ∈ R D×D s.t. X, Y 0 and Y is full rank, we have that max
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Note that a Xa a Y a is independent of the scaling of a. The following sequence of inequalities completes the proof
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Proof. Note that a Xa a Y a is independent of the scaling of a. The following sequence of inequalities completes the proof
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For all experiments, we use the 117M parameter "small" GPT-2 model proposed in  Table 3. We limited the training set size to 200K for larger datasets (i.e., DBPedia and Yahoo Answers). For CR and MPQA, we created train-test splits with 75-25 percentage random splits of the data.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For all experiments, we use the 117M parameter "small" GPT-2 model proposed in  Table 3. We limited the training set size to 200K for larger datasets (i.e., DBPedia and Yahoo Answers). For CR and MPQA, we created train-test splits with 75-25 percentage random splits of the data.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We find that g f,Φ consistently has comparable performance to f across non-sentiment and sentiment downstream classification tasks. We include results using a bag-of-n-grams (BoNG) and Sentiment Neuron (mLSTM) [Radford et al., 2017], and we note that using 768-dimensional features is more sample efficient than BoNG.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We find that g f,Φ consistently has comparable performance to f across non-sentiment and sentiment downstream classification tasks. We include results using a bag-of-n-grams (BoNG) and Sentiment Neuron (mLSTM) [Radford et al., 2017], and we note that using 768-dimensional features is more sample efficient than BoNG.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For sentiment tasks, adding a prompt always boosts performance. We also demonstrate that much of the performance can be recovered by only looking at "positive" and "negative" or ":)" and ":(" as class words. Using these 2-dimensional features is even more sample-efficient than the standard 768-dimensional ones.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
For sentiment tasks, adding a prompt always boosts performance. We also demonstrate that much of the performance can be recovered by only looking at "positive" and "negative" or ":)" and ":(" as class words. Using these 2-dimensional features is even more sample-efficient than the standard 768-dimensional ones.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We also include results using the pre-trained BERT base cased model [Devlin et al., 2018, Wolf et al., 2019, using the embedding at the first token as input to the downstream task. We also tried using the mean embedding and last token embedding and found that the first token embedding is best. Moreover, the first token embedding is what is extracted in the traditional usage of BERT on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We also include results using the pre-trained BERT base cased model [Devlin et al., 2018, Wolf et al., 2019, using the embedding at the first token as input to the downstream task. We also tried using the mean embedding and last token embedding and found that the first token embedding is best. Moreover, the first token embedding is what is extracted in the traditional usage of BERT on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We test two models with the same parametrization and initializations, one trained using our Quad objective and another trained with the standard language modeling objective using the Amazon product review dataset [McAuley et al., 2015] instead of IMDB. We slightly modify the standard architecture of GPT-2 to generate Tables 2 and 4. We add a single linear layer after the Transformer to add expressivity. Furthermore, instead of tying the input and output embeddings, we learn them separately so that f and Φ are independent functions. We fix the input embeddings and the positional embeddings to be the parameters from the pre-trained GPT-2. We initialize Φ, the output embeddings, using the singular vectors of the pre-trained word embeddings Φ. Given our parameterization, initializing with the singular vectors is as expressive as initializing with the pretrained embeddings Φ themselves; however it lends a better optimization landscape and speeds up training for our new objective.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We test two models with the same parametrization and initializations, one trained using our Quad objective and another trained with the standard language modeling objective using the Amazon product review dataset [McAuley et al., 2015] instead of IMDB. We slightly modify the standard architecture of GPT-2 to generate Tables 2 and 4. We add a single linear layer after the Transformer to add expressivity. Furthermore, instead of tying the input and output embeddings, we learn them separately so that f and Φ are independent functions. We fix the input embeddings and the positional embeddings to be the parameters from the pre-trained GPT-2. We initialize Φ, the output embeddings, using the singular vectors of the pre-trained word embeddings Φ. Given our parameterization, initializing with the singular vectors is as expressive as initializing with the pretrained embeddings Φ themselves; however it lends a better optimization landscape and speeds up training for our new objective.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We observe that even on a large dataset, training using Quad yields comparable performance to the language model on the SST task. Furthermore, adding a prompt consistently improves performance for both objectives.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We observe that even on a large dataset, training using Quad yields comparable performance to the language model on the SST task. Furthermore, adding a prompt consistently improves performance for both objectives.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Assumption 4.1, we assert that there is a quadratic fit for the log partition function, which allows us to show in Lemma 4.3 that a linear relation holds between f * and Φp f * . We validate these theoretical findings by fitting a quadratic function to the log partition function for a subset of embeddings from Table 3: GPT-2 performance without fine-tuning on downstream task test sets with k classes. We provide the performance of bag-of-n-grams as an approximate baseline for these tasks. DBPedia and Yahoo performances were reported in Zhang et al. [2015], and the other tasks were reported in Khodak et al. [2018]. We also include results from Sentiment Neuron [Radford et al., 2017] for the sentiment-related classification tasks: IMDb, CR, and MPQA. Furthermore, we include results from using BERT [Devlin et al., 2018] without fine-tuning, where we use the features produced for the first position as input to the linear classifier. An asterisk indicates we add a standard sentiment prompt "The sentiment is " to each input. We also tested the performance of the conditional probability distribution over "positive" and "negative" as well as " :)" and " :(" on the sentiment-related tasks with and without the prompt. the IMDb, SST, and AG News datasets ( Figure 1). Here, we describe how we learned A, b and c. To ensure A is symmetric and positive semi-definite as required, we parametrize A = U U T . Let µ θ = Φp θ . We minimize the following objective function:
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Assumption 4.1, we assert that there is a quadratic fit for the log partition function, which allows us to show in Lemma 4.3 that a linear relation holds between f * and Φp f * . We validate these theoretical findings by fitting a quadratic function to the log partition function for a subset of embeddings from Table 3: GPT-2 performance without fine-tuning on downstream task test sets with k classes. We provide the performance of bag-of-n-grams as an approximate baseline for these tasks. DBPedia and Yahoo performances were reported in Zhang et al. [2015], and the other tasks were reported in Khodak et al. [2018]. We also include results from Sentiment Neuron [Radford et al., 2017] for the sentiment-related classification tasks: IMDb, CR, and MPQA. Furthermore, we include results from using BERT [Devlin et al., 2018] without fine-tuning, where we use the features produced for the first position as input to the linear classifier. An asterisk indicates we add a standard sentiment prompt "The sentiment is " to each input. We also tested the performance of the conditional probability distribution over "positive" and "negative" as well as " :)" and " :(" on the sentiment-related tasks with and without the prompt. the IMDb, SST, and AG News datasets ( Figure 1). Here, we describe how we learned A, b and c. To ensure A is symmetric and positive semi-definite as required, we parametrize A = U U T . Let µ θ = Φp θ . We minimize the following objective function:
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In practice, we train only on the regression loss (i.e., λ 1 = 0, λ 2 = 1) for the most promising results. We use 20,000 examples from a mix of IMDb, SST, and AG News embeddings as the training set. We used the Adam [Kingma and Ba, 2014] optimizer with learning rate 1e-3 for U and learning rate 1e-4 for b and c. We decayed the learning rate every 50 steps by a factor of 0.1. We found the fit after 8 epochs of training.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In practice, we train only on the regression loss (i.e., λ 1 = 0, λ 2 = 1) for the most promising results. We use 20,000 examples from a mix of IMDb, SST, and AG News embeddings as the training set. We used the Adam [Kingma and Ba, 2014] optimizer with learning rate 1e-3 for U and learning rate 1e-4 for b and c. We decayed the learning rate every 50 steps by a factor of 0.1. We found the fit after 8 epochs of training.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We further demonstrate the quality of the learned fit by plotting the true log partition and estimated log partition function for embeddings from other datasets in Figure 2. is from an LM trained on the standard cross-entropy objective. All models were trained on the Amazon dataset. An asterisk indicates that we added the prompt "This movie is " to each input. Note that the validation loss was still decreasing at the time of measurement.  Figure 2: Fit of the learned quadratic function to the log partition function on various datasets for features computed by the full, pre-trained GPT-2. We also plot the y = x line for reference.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
We further demonstrate the quality of the learned fit by plotting the true log partition and estimated log partition function for embeddings from other datasets in Figure 2. is from an LM trained on the standard cross-entropy objective. All models were trained on the Amazon dataset. An asterisk indicates that we added the prompt "This movie is " to each input. Note that the validation loss was still decreasing at the time of measurement.  Figure 2: Fit of the learned quadratic function to the log partition function on various datasets for features computed by the full, pre-trained GPT-2. We also plot the y = x line for reference.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Natural language processing (NLP) has been revolutionized by large-scale pretrained language models such as BERT [4] and GPT [27], which are adapted to a variety of downstream NLP tasks. Although a large body of empirical work seeks to understand the effectiveness of pretrained models [7,5,12,37,36,11,29,15], theoretical understanding is scarce. Theoretically analyzing the relationship between the pretraining and downstream tasks is challenging because pretraining and downstream settings can greatly differ.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The key starting point for our analysis is to link the pretraining and downstream settings through an underlying generative model of the data. We model the data distribution as a latent variable model and the downstream task as a function of the latent variables. Assuming that pretraining on a large corpus allows us to learn the generative model, the conditional token probabilities predicted by the pretrained model carry information about the hidden variables. In downstream adaptation, we aim to recover this information to solve the downstream task.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Though full finetuning is the de facto empirical standard, analyzing it is challenging because it requires characterizing the weights of the pretrained model. In this paper, we focus on head tuning and prompt tuning, which both freeze all pretrained parameters and allow us to treat the pretrained model as a black box. Head tuning [24] trains task-specific heads on top of the pretrained model outputs. Prompt tuning [33,20,9,22] optimizes a task-specific "prompt" that is concatenated to the model input. Studying prompt tuning is particularly interesting since it can match the performance of full finetuning with less computation time [20,9,22].
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Our work contrasts with prior theoretical work [30], which assumes that downstream labels are recoverable via a linear head applied to the conditional token probabilities, and analyze how errors in pretraining or model misspecification propagate downstream. We consider specific generative distributions for which we can prove these assumptions, showing that head and prompt tuning can recover the downstream labels.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Our analysis considers two data-generating distributions with increasing realism. First, we consider data generated from a Hidden Markov Model (HMM), where the downstream task is to learn a linear classifier on the posterior distribution over the hidden states (Section 3). We prove that, under strong non-degeneracy conditions on token emission probabilities, a linear head applied to a pretrained model G which outputs exact conditional token probabilities (G i pxq " P rX i | x´is) can recover the downstream label (Theorem 3.3). Furthermore, we can prove better recovery guarantees with relaxed non-degeneracy assumptions (Assumption 3.1) by using continuous prompt tuning (Theorem 3.6), reflecting the strong empirical performance of prompt tuning [20,9,22]. Intuitively, prompt tuning conditions the latent variables so that nonessential information for the downstream task can be ignored during the tuning phase, making task-essential information easier to recover.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Second, we also strengthen our analysis by leveraging additional structure in the data. Motivated by longrange dependences in natural language, we analyze HMM variants with additional latent "memory" variables that can store long-term information more easily than vanilla HMMs (Section 4). Here, the downstream task is to learn a linear classifier on the posterior distribution of the memory variables. We show that, under weaker non-degeneracy conditions than the first setting, an attention-based classification head can recover ground-truth downstream labels from pretrained model outputs (Theorem 4.3). Intuitively, our recovery guarantees improve because the classification head can focus on the persistent, task-essential information in the memory while ignoring other transient and nonessential aspects of the latent variables. As with the vanilla HMM, we analyze prompt tuning for relaxing the non-degeneracy conditions even further (Theorem 4.6).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In summary, we relate the pretraining and downstream tasks by assuming that the downstream task is to learn a classifier on the posterior distributions of the latent variables defined by an underlying generative model of text. Our theoretical contributions are: 1) in this setting we analyze an HMM generative model show that simple classification heads can recover the true downstream labels under certain non-degeneracy assumptions, 2) we prove that soft prompt tuning can relax the non-degeneracy assumptions needed for downstream recovery making it easier to extract task-specific information, and 3) our recovery guarantees are stronger for memory-augmented HMMs in comparison to the vanilla HMM when tuning an attention-based classfication head.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We empirically evaluate our theoretical results with language models pretrained on synthetically generated data from HMMs. We find that prompt tuning obtains good downstream performance when our non-degeneracy conditions are relaxed, whereas head tuning performs poorly. Furthermore, we show that head tuning obtains better downstream performance when data is generated from a memory-augmented HMM, compared to a vanilla HMM, as is predicted by our theory. 1 
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The black box nature of BERT and related models has inspired a variety of empirical works which seek to understand them. Probing papers study whether a pretrained model computes various types of structured information (e.g., syntactic [37,11]) by evaluating the performance of simple classifiers, or probes, on the representations [7,12,36,29,15]. Other papers ablate various aspects of pretraining, such as changing the masking scheme [14,21,42] or permuting the word order [34].
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In comparison, theoretical analysis of pretrained language models is limited. Besides [30], which we discussed in Section 1, Zhang and Hashimoto [42] analyze using a linear classifier to approximately recover the latent variable in a Gaussian graphical model with sparse dependencies between observed variables. However, their analysis and setting are focused towards understanding syntactic dependencies between tokens, whereas we directly model and analyze downstream performance.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Prompt-based tuning [33,20,9,22,13,6,43,2,25], which has improved empirical downstream performance for lightweight adaptation methods beyond head tuning to approach full finetuning, is an important focus of our theoretical analysis. Shin et al. [33] employ task-specific prompts that are optimized over the discrete token space. Schick and Schütze [31,32] reformulate natural language tasks as cloze-style phrases to enable few-shot learning. Subsequent methods [20,9,22] optimize "soft" prompts, or continuous embedding vectors. Lester et al. [20] employ soft prompts on pretrained large-scale T5 [28] models and show that as the model size increases, prompt tuning performance can eventually match finetuning. Hambardzumyan et al. [9] applies a variant of soft prompt tuning to MLM models. Li and Liang [22] propose prefix tuning, which prepends a trainable prefix embedding sequence to all layers of the transformer.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
More broadly, Lee et al. [19] analyze reconstruction-based self-supervised learning methods in a general setting and show that under certain conditional independence assumptions, predicting one observed variable from another allows recovery of the latent with a linear head. Other theoretical works analyzing self-supervised or constrastive learning include [1,10,38,40,39,23], but they are not directly relevant for our particular setting.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We analyze models pretrained on masked language modeling (MLM) objectives. Let X denote a finite vocabulary of input tokens, X˚the set of variable-length sequences of tokens, and X " pX 1 , . . . , X T q P X˚a random sequence of T tokens. Let ∆ |X | denote the space of probability distributions over tokens.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Pretraining and downstream task. Let Gpxq " pG 1 pxq, G 2 pxq, . . .q denote the masked language model which predicts a probability vector for each timestep in the input x. Our theoretical abstraction is that G i perfectly computes the distribution of X i , the i-th token, conditioned on all other tokens: G i pxq " P rX i |X´i " x´is. Here P rX i | X´i " x´is P ∆ |X | is a probability vector. In particular, G i pxq does not depend on x i . The downstream task involves labeled examples px, F ‹ pxqq P X˚ˆY, where F ‹ : X˚Ñ Y provides ground-truth downstream labels and Y is a discrete set of labels for classification.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Head and prompt tuning. Head tuning trains a classification head f on top of fixed model outputs, resulting in the classifier F pxq " 1pfpGpxqq ě 0q. We expect f to be a simple function such as a linear or one layer attention model. We also analyze variants where f also takes the tokens x or embeddings of x as input, which provides additional information. Soft prompt tuning requires viewing the pretrained model G as a function of the token embeddings; we refer to this model by G. Letting epxq " epx 1 q, . . . , epx t q denote the token embeddings, we have Gpepxqq " Gpxq. Soft prompt tuning concatenates a trainable prompt u so that the model output is Gppu, epxqq. We consider simultaneously training the prompt parameter u and a classification head to fit the downstream task.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Notations. Let ∆ d denote the space of d-dimensional probability vectors. We work with discrete random variables V taking values in a finite set V. We use P rV s P ∆ |V| to denote the distribution of V and P rU | V " vs P R |U | the conditional distribution of U given V " v. PrpV " vq P r0, 1s will denote the probability that V takes values v. We also let P rU " u | V s P R |V| denote the vector with entries PrpU " u | V " vq. P rU | V s P R |U |ˆ|V| will describe the matrix with entries P rU | V s u,v " PrpU " u | V " vq.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
For a sequence v " pv 1 , . . . , v t q, we use the notation v i:j for i ď j to denote pv i , . . . , v j q, and v´i to denote pv 1:i´1 , v i`1:t q. We let 1 denote the indicator function. For set V, we let V˚" V 1 Y V 2 Y¨¨¨denote variablelength sequences of elements of V. Let d denote elementwise product. Let 1 d , 0 d denote the d-dimensional To abstractify soft prompt tuning, we note that every token has a natural embedding, the corresponding row of the emission probability matrix. We view prompt tuning as adding a fake token r z to the vocabulary, assigning it a row u in the emission matrix, and prepending it to the input embedding sequence. More details are provided in Section 3.1.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
all-1's and all-0's vector. We omit the subscript if the dimension is clear from context. For two vectors a, b P R d , we let a{b denote their element-wise division. We use supppaq to denote the set of indices where vector a is non-zero.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Defining a relation between pretraining and downstream tasks is the foremost challenge for analysis. We propose to link the two via latent variable generative assumptions on the input distribution. We model the downstream task as a function of the posterior distribution of the latent variables. Towards a first result, this section studies the case where inputs are generated by HMMs (see Figure 1 (left)), which have been well-studied in the context of language and speech processing (see e.g. [26,18,3]).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Let H denote the hidden state space of the HMM. We use H " pH 0 , H 1 , . . . , H T q P Ht o denote the sequence of hidden states. For all timesteps i ą 0, the transition probabilities are time-invariant, i.e. P rH i | H i´1 s " A for A P R |H|ˆ|H| . For each timestep i ě 1, tokens X i are emitted following some time-invariant probability: P rX i | H i s " W for W P R |X |ˆ|H| . The joint probability of X, H is
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Downstream tasks. We assume that H 0 has the meaningful information for the downstream task, which is a binary classification task where the ground-truth labeling F ‹ is assumed to be a linear classifier on the posterior P rH 0 | X 1:T " xs: for µ P R |H| . Our results are easily extended to the multiclass setting. We consider tuning a linear head for the downstream classifier, which formally computes 1pb J G 1 pxq ě 0q for b P R |X | . The following non-degeneracy condition is crucial for our recovery result in this setting. We also require the following regularity conditions on H 0 and the state transitions. We show that if W has linearly independent columns, a linear head fits downstream labels. Theorem 3.3. Assume that non-degeneracy (Assumption 3.1) and regularity (Assumption 3.2) hold. Then any downstream task F ‹ pxq of the form (3.1) can be computed by a linear head on G applied to a shifted sequence. That is, there exists linear head weights b P R |X | such that for all x P supppP rXsq,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where x 1 " p∅, x 1:t q is the concatenation of a special token ∅ with x. 2 The key for the proof is to leverage the following general statement about random variables U, V, Z such that U K V | Z, which decomposes the expression for P rU | V s. Proposition 3.4. Let U, V, Z be random variables such that U K V | Z. Then for any v, P rU | V " vs " P rU | Zs¨P rZ | V " vs. Thus, if P rU | Zs has a left inverse pP rU | Zsq : , then P rZ | V " vs " pP rU | Zsq : P rU | V " vs.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
By the conditional independence structure of the HMM, Proposition 3.4 immediately implies G 1 px 1 q " W P rH 1 |X 2:T`1 " xs ùñ P rH 1 |X 2:T`1 " xs " W : G 1 px 1 q where W : is the left inverse for W , guaranteed to exist by Assumption 3.1. This lets us recover P rH 1 |X 2:T`1 " xs by applying a linear function to G 1 px 1 q. Additional linear functions will be sufficient to obtain µ J P rH 0 |X 1:T " xs from P rH 1 |X 2:T`1 " xs. We provide the full proof in Section A. Proposition 3.4 is reminiscent of the arguments of [19], which leverages the independence structure in the same way. Subsequent sections will require more complicated analyses and recovery procedures.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
A drawback of Theorem 3.3 is that it relies heavily on assuming W has full column rank, which implies the necessary condition that |H| ď |X |. Without this assumption, it is unclear how to recover P rH 0 | X 1:T " xs from Gpxq alone. However, in realistic settings we would expect |H| ą |X |, as increasing the size of the hidden state space improves language modeling capabilities of HMMs [3].
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In this section, we study applying soft, or continuous, prompt tuning [20,9] to the setting above. We show that by using soft prompt tuning, we can recover F ‹ using a linear head on G for HMMs where the non-degeneracy assumptions on W are relaxed. Our analysis provides insight into the empirical successes of prompt-tuning: intuitively, prompt tuning enables better recovery of the downstream task by conditioning the output of G to only contain task-specific information.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Soft prompt tuning trains task-specific embedding vectors, but analyzing how the model processes embedding vectors is challenging because it requires opening up the black box of the pretrained model. Thus, we require additional abstractions about how the pretrained model processes the embedding vectors. We will extend the mask language model G to a model G that maps a sequence of embeddings e 1 , . . . , e t to conditional probabilities G 1 pxq, . . . , G t pxq as follows. We observe that each token z in the vocabulary X naturally corresponds to a |H|-dimensional vector: the z-th row of the emission probability matrix W , or equivalently, P rX i " z | H i s. We denote this embedding by epzq and call the family of embeddings tepzq : z P X u proper embeddings. A fundamental property of HMMs is that the conditional probability P rX i | X´i " x´is only depends on x 1 , . . . , x t through their embeddings epxq " pepx 1 q, . . . , epx t qq. In other words, there exists a function G i such that G i px 1 , . . . , x t q " G i pepx 1 q, . . . , epx t qq 2 We note that G 1 px 1 q does not depend on x 1 1 and therefore x 1 1 can be any token.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In particular, we let G i compute the standard message passing algorithm [16] that computes the conditional probability of HMMs. This ensures that G i is well defined on all sequences of nonnegative vectors in r0, 1s |H| , beyond sequences of proper embeddings.We assume that pretraining produces this G i , which we treat as a blackbox for prompt tuning.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In particular, for prompt tuning we can consider the case where we pass an arbitrary nonnegative vector u P r0, 1s |H| to G in the first argument and proper embeddings at positions i ą 1. We can interpret u as the embedding of a fake token r z. Concretely, consider adding a new token r z to the vocabulary X , and changing the emission probability at position 1 to satisfy P rX 1 " r z | H 1 s " u and for all z ‰ r z, P rX 1 " z | H 1 s9p1´uqdepzq. Then G i pu, epx 1 q, . . . , epx t qq precisely computes the conditional probability P rX i | X´i " pr z, x 1 , . . . , x t q´is under the modified HMM. We refer the readers to Section B for the formal definition of G i and formal proofs of the interpretation above.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We consider a downstream training algorithm which trains the prompt tuning parameter u described above and a linear classification head. Letting u denote the trainable prompt parameter and b P R |X | the trainable linear head weights, the model uses the embedding sequence
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
and outputs the prediction F pxq " 1pb J G 2 pp epxqq ě 0q. We can provide recovery guarantees for this model if the ground-truth classifier weights µ (defined in (3.1)) and columns of the HMM transition matrix A satisfy the following relaxation of the requirement in Theorem 3.3 that W is nondegenerate.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
There exists a set of essential hidden states H ‹ Ď H, so that the columns of W corresponding to H ‹ , tW :,h u hPH ‹ , are linearly independent. Furthermore, H ‹ covers all meaningful information for the downstream tasks: supppµq Ď H ‹ .
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In addition, a last technical requirement on H ‹ is as follows: there exists a set B Ď H such that H ‹ " Y hPB supppA :,h q. In other words, H ‹ must be the set of all states reachable by starting from some state in B and transitioning one step in the hidden Markov chain.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Compared to Assumption 3.1, which required that all columns of W are linearly independent, Assumption 3.5 only requires linear independence on a subset H ‹ of essential states. In the setting where |H| ą |X |, the condition for Theorem 3.3 can never hold. On the other hand, Assumption 3.5 could still hold, for example, if |supppµq| ă |X | and the set of columns of W corresponding to hidden states in supppµq is linearly independent. The last technical requirement in Assumption 3.5 is also required, which could be satisfied if columns of A are sparse. The following theorem shows that when Assumption 3.5 holds, we can recover F ‹ using soft prompt tuning with a linear head.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Theorem 3.6. In the above setting, assume that Assumptions 3.2 and 3.5 hold. Then F ‹ can be computed using soft prompt tuning with a linear head on G. Concretely, there is a continuous prompt parameter u P R |H| and weight vector b P R |X | , such that for all x P supppP rXsq,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where p e prepends u to the input embedding sequence, as defined in (3.2).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Theorem 3.6 provides a stronger recovery result than Theorem 3.3, which only used a linear head. This is also reflected in our synthetic experiments (Section 5), and prior work which shows that variants of prompt tuning can perform much better than only training the last few layers of the model [22]. Our theory suggests that prompt tuning could help by conditioning the hidden variables to remove nonessential information for the task from the output of G. This makes task-essential information easier to recover.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The key proof intuition is that although recovering P rH 0 | X 1:T " xs is impossible without strong nondegeneracy conditions (Assumption 3.1), we can aim to recover P rH 0 | X 1:T " xs on the subset of essential states H ‹ defined in Assumption 3.5, which suffices for computing µ J P rH 0 | X 1:T " xs, since H ‹ Ě supppµq. To recover P rH 0 | X 1:T " xs on H ‹ , we observe in Lemma B.2 that prepending the prompt u is equivalent to introducing a modified random sequence p X and fake token r z which influences the posterior of H 2 as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
for invertible diagonal matrix D and positive scalar r x . We choose u such that the vector P rH 2 | p X 1 " r zs d P rH 0 | X 1:T " xs is supported only on H ‹ . Because corresponding columns of W are linearly independent by Assumption 3.5, we can then recover PrpH 0 " h | X 1:T " xq for h P H ‹ by applying a linear function to G 2 pp epxqq. This suffices for computing µ J P rH 0 | X 1:T " xs. More details are in Section B.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We study a memory-augmented HMM which explicitly disentangles the evolution of hidden states from a persistent "memory" variable. Inspired by natural sentences, this model is intended to better capture the distinction between syntax, which constantly evolves, and semantics, which changes less. This additional structure in the generative model allows us to strengthen our results by relaxing the non-degeneracy conditions on W , the token emission probabilities. Thus, both head and prompt tuning are more powerful in this setting compared to Section 3 and can recover the downstream label with weaker non-degeneracy assumptions on W . In Section 4.2, we show that soft prompt tuning also provides an advantage over head tuning alone.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Data distribution. The memory-augmented HMM, depicted in Figure 2, can be viewed as a generative variant of memory networks [41,35] and is closely related to Hidden Topic Markov Models [8]. There are two sets of latent variables in the memory-augmented HMM: a Markov chain on hidden states H 0 , H 1 , . . ., meant to model the evolution of syntax, and a persistent "memory" M " pM 1 , . . . , M N q with N total cells, where each M i takes values in a finite set M. The full joint probability is as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The hidden state is modified to explicitly consist of a disentangled cell index J P rN s and syntax state S P S, such that H i " pJ i , S i q and H " rN sˆS. To sample the token at timestep i given the hidden state H i " pJ i , S i q, we first use J i to index the memory M , obtaining the random variable M Ji . X i is then sampled according to some time-invariant probability depending on M Ji , J i , S i :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Here W P R |X |ˆ|M||H| stores the emission probabilities for each choice of memory cell value and hidden state. Note that in particular, the conditional probabilities for X i only depend on a single memory cell for each timestep. We also note that memory-augmented HMMs can be viewed as vanilla HMMs with structured transitions because pH 0 , M q, pH 1 , M q, . . . can be viewed as a Markov chain where the memory component does not change.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Example 4.1 (Generating natural sentence with memory-augmented HMM). We consider how this model may generate the sentence "The cow in the pasture rolled on the grass' happily." M 1 could store the subject ("cow"), M 2 the location ("pasture"), M 3 the sentiment ("happily"), and S i could determine part-of-speech. For timesteps where "cow" and "rolled" are emitted J i " 1 because we emit information related to the sentence subject. Timesteps for "pasture" and "grass" would have J i " 2.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Downstream tasks. We consider downstream tasks where ground-truth labels are obtained via a linear classifier on the posterior distribution of a particular memory cell j ‹ P rN s:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
xs ě 0q, where µ P R |M| . Intuitively, this formulation models downstream tasks which depend on a particular aspect of the semantics but not on syntax (e.g. in the setting of Example 4.1, if j ‹ " 3, the task is sentiment analysis).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
To recover the downstream labeling, we require an attention-based classification head, which is a function of both the input embeddings and outputs of G. Formally, let q P R |H|`1 denote a query parameter and β 1 , . . . , β t P R |H|`1 denote trainable position embeddings. Given pretrained model outputs G i pxq and trainable token embeddings epx i q, the attention head Attnp¨q applies key and value functions K, V to compute the output as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where arg max refers to the set of indices achieving the maximum in (4.1). We note that standard attention heads in practice rely on the softmax function, but the expression based on arg max above captures the limiting behavior as }q} 2 Ñ 8. We consider linear key functions given by KpG i pxqq " Θ pKq G i pxq. The value function V : R |X |ˆR|M||H| Ñ R uses parameters Θ pV q P R |M||H|ˆ|X | and b P R |M||H| and computes
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Because our generative model disentangles H and M , we can relax the non-degeneracy assumption on the token emission probabilities W , compared to Theorem 3.3. The relaxed assumption only requires the columns tW :,pm,hq u mPM,hPH ‹ to be linearly independent in a subset H ‹ of "recoverable" hidden states, whereas Assumption 3.1 required all columns to be linearly independent.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Assumption 4.2 (Existence of "recoverable" hidden states). There exists a set of recoverable hidden states H ‹ " tj ‹ uˆS ‹ , such that the collection of token emission probabilities from MˆH ‹ , tW :,pm,hq u mPM,hPH ‹ , is a linearly independent set of vectors.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Furthermore, the span of these vectors must be disjoint from the span of token emission probabilities from MˆpHzH ‹ q: spanptW :,pm,hq u mPM,hPH ‹ q X spanptW :,pm,h 1 q u mPM,hPHzH ‹ q " t0 |X | u.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Note that the non-degeneracy condition of Theorem 3.3 would require tW :,pm,hq u mPM,hPH to be linearly independent, whereas Assumption 4.2 only requires linear independence for h P H ‹ . The second condition states that H ‹ and HzH ‹ are distinguishable by the token emission probabilities.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We explain Assumption 4.2 in the setting of Example 4.1. For natural language, there might be choices of h " pj i , s i q for which the set tW :,pm,hq u mPM of token emission probabilities is fundamentally not very diverse, and therefore not linearly independent. For example, if the syntax s i indicates "article", i.e. words such as "a", "an", and "the", the token emission probabilities would carry little information about M ji because the choice of article does not depend much on semantics, so columns corresponding to s i " "article" would not be linearly independent, violating Assumption 3.1. However, Assumption 4.2 allows us to avoid this issue by placing such h in HzH ‹ , a set of hidden states which we can ignore, and only including hidden states which carry a lot of information about M in H ‹ . In Example 4.1, when J i " 2 (location), S i " "noun", the position i should convey a lot about the location (in this case, "pasture"), so it is more reasonable to assume that tW :,m,h u mPM is linearly independent for this hidden state.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Thus, our aim is to focus on recovering information for the downstream task from positions i where H i P H ‹ . Formally, we define the following set of input sequences containing positions i where the posterior of H i given x´i concentrates on H ‹ :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
3)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The following theorem shows that under Assumption 4.2, we can recover F ‹ using the attention head described above, if x P R is nonempty. Note that R is nonempty if the posterior of H i concentrates on H ‹ for some i. For natural language, it is realistic to assume this can occur because syntactic aspects of a sentence are typically low-entropy when the full sentence is observed. . Then there exist an attention head on Gpxq and token embeddings epx i q such that the following holds for any x P R:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where the function Attn is in the form described in (4.2).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The idea is to use the attention mechanism to attend to positions i where supppP rH i | X´i " x´isq Ď H ‹ . The intuition of Assumption 4.2 is that such positions are more informative for recovering the latent posteriors; indeed, from the outputs G i pxq at such i, the value function in the attention will be able to recover P rM j ‹ | X 1:T " xs. A full proof is provided in Section C.1.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Though the generative modeling assumptions in this section already allowed us to relax the non-degeneracy assumptions, applying soft prompt tuning allows us to relax them even further. For simplicity, we consider the setting where there is a single memory cell, so M P M, and the downstream task is a linear classifier on the posterior of the memory: F ‹ pxq " 1pµ J P rM |X 1:T " xs ě 0q. This simplified setting also doesn't require the explicit disentanglement between J i and S i in H i . We analyze continuous prompt-tuning in a setting where the pretrained model G follows the same abstraction as in Section 3.1. We modify the model to take |M||H|-dimensional vectors, so the proper embedding for token z is given by epzq " P rX i " z|M, H i s " W J z,: . In Section C.3, we describe the formal construction and interpretation of G in the more general setting with more memories.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Letting u P R |M||H| denote the trainable prompt parameter, we define the input embeddings p epxq fi pu, epx 1 q, . . . , epx t qq (4.4)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The downstream model applies an attention head to the output of G: F pxq " 1pAttnppG i pp epxqq, p e i pxqq t`1 i"1 q ě 0q, where Attn is defined in (4.2). An additional stationarity assumption on P rH 0 s will simplify the recovery procedure (though it can be removed). As before, we assume sparsity of µ and some non-degeneracy of W , though the assumption is more relaxed and easier to state compared to the vanilla HMM setting. Furthermore, the span of these vectors must be disjoint from the span of token emission probabilities from M ‹ˆp HzH ‹ q: spanptW :,pm,hq u mPM ‹ ,hPH ‹ q X spanptW :,pm,h 1 q u mPM ‹ ,hPHzH ‹ q " t0 |X | u.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We note that Assumption 4.5, and Assumption C.5 for multiple memories, are relaxations of Assumption 4.2, as they only consider memory values in supppµq, whereas Assumption 4.2 considers all m P M. An additional advantage of the memory-augmented HMM is that Assumption 4.2 is simpler than Assumption 3.1 and does not require any conditions on the transition matrix A. We now state our result for recovering F ‹ with soft prompt tuning and an attention head.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Theorem 4.6. In the setting above, suppose that non-degeneracy Assumption 4.5 and stationarity Assumption 4.4 hold. Then there exists a prompt u and attention head on Gpp epxqq and the token embeddings which can compute the ground-truth F ‹ pxq for any x P R, defined in (4.3):
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where p e is the embedding in (4.4) and Attn is defined in (4.2).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The intuition for this proof is similar to Theorem 3.6: the soft prompt conditions the memory M to concentrate on supppµq. As a result, all irrelevant information to the task is removed from G i pp epxqq, making it easier to recover the task-specific information about the posterior of M . A more general theorem statement for the multiple memories setting, and the full proof, is provided in Section C.3
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We empirically evaluate our theoretical results by pretraining a BERT-like masked language model (MLM) [4] on synthetic data generated by an HMM. Our goal is to verify key implications of our theory in a more realistic setting where some assumptions, such as that G outputs exact conditional probabilities, may not hold. First, we compare head and prompt tuning and show that prompt tuning improves downstream performance, especially when the recovery problem is degenerate. Second, we compare the effect of changing the data distribution from vanilla HMMs to memory-augmented HMMs on head tuning with an attention layer. We find that the downstream performance improves when the data has a long-term memory component. These observations support our theory. Our code is available at the following URL: https://github.com/ sangmichaelxie/pretraining_analysis. Pretraining data and downstream task. We generate pretraining data from an HMM with randomly generated transition matrix, emission probabilities, and start distributions. In all experiments, the HMMs have 10 vocabulary symbols, while the hidden state size varies. The downstream task uses input sequences X 1:T of length 129, where the first token X 1 " [MASK]. We consider binary classifcation where labels are generated using linear functions of the analytically-computed posteriors in the HMMs. In all experiments, the ground truth linear weight is sparse with 6 nonzero entries at uniformly random locations with Gaussian values. More details are in Appendix D.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Head vs. prompt tuning. We compare head and prompt tuning as the hidden state size of the datagenerating HMM varies. The downstream label is generated by computing µ J P rH 1 | X´1 " x´1s, where µ is a random ground-truth linear weight. Head tuning learns a linear head on top of the softmax probabilities  Memory-augmented HMMs. We investigate the effect of augmenting the data-generating HMM with a long-term memory. We consider the single memory case with |H| " 4 and varying memory sizes |M| P t2, 3, 5, 7u. The downstream label is generated by computing µ J P rM | X´1 " x´1s, where µ denotes the ground-truth weights. Viewing the memory HMM as a HMM where the component on M never changes, we can compare against the vanilla HMMs from the previous setting. For the memory-augmented HMM, we use head tuning with a single-cell attention layer on the entire sequence of softmax probability outputs. For the vanilla HMM in the comparison, we use a linear head on the output at the first position, as an attention head would perform worse since the downstream task depends only on H 1 and not any other timesteps. Figure 3 (right) verifies that head tuning recovers the downstream task better when there is more structure in the data, as predicted by Theorem 4.3. Head tuning achieves near 100% downstream accuracy on all hidden state sizes.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We analyze how pretraining on generic language modeling tasks can improve performance on diverse downstream tasks. In our analysis framework, the downstream task requires predicting properties of the posterior distribution over latent variables in an underlying generative model. When the generative model is a standard HMM, downstream recovery is possible with a simple classification head under strong non-degeneracy assumptions. We also show that we can relax the non-degeneracy conditions by changing the generative model to a memory-augmented HMM or using prompt tuning. The generative distributions studied here are meant to provide a first-cut result -we also conjecture similar theorems to hold for other generative models, which we leave as an interesting direction for future work.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Another direction for future work is to analyze finetuning. Existing work analyzes finetuning for linear neural networks and obtains empirically useful insights [17], but analyzing neural networks with nonlinear activations is very challenging. Our analysis of head and prompt tuning treats the model as a black box. Analyzing finetuning requires understanding how to open up the black box, which is a major open question.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We provide the formal proof of Theorem 3.3 based on the sketch in Section 3. The following lemma will be useful in our analysis.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Claim A.1. In the setting of Section 3, suppose that Assumption 3.2 holds. Fix any timestep i ě 1. Then there exists a diagonal matrix D such that for all x P supppP rXsq, P rH i | X i`1:T`i " xs " r x DP rH 0 | X 1:T " xs where r x ą 0 is a positive scalar.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof. First, we note that by Assumption 3.2, P rH i s has full support. As a consequence, PrpX i`1:t`i " xq ą 0. By Bayes' rule,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
" P rH 0 | X 1:T " xs d P rH i s P rH 0 s¨P
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Note that the vector P rHis P rH0s has finite and positive entries. The same applies to the ratio r x fi PrpX 1:T "xq PrpX i`1:T`i "xq . Thus, we get the desired statement.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Theorem 3.3. By definition, G 1 px 1 q " P rX 1 | X 2:T`1 " xs. Therefore, our goal is to rewrite P rH 0 | X 1:T " xs as a linear function of P rX 1 |X 2:T`1 " xs (up to a scaling which won't affect the linear head prediction). Concretely, we will show P rH 0 | X 1:T " xs " r x BP rX 1 | X 2:T`1 " xs (A.1) for a scalar r x ě 0. With this equation, taking b " µ J B will give the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
First, observe that P rX 1 | X 2:T`1 " xs " W P rH 1 | X 2:T`1 " xs by Proposition 3.4. Next, we apply Claim A.1 to obtain an invertible matrix D such that for all x P supppP rXsq, P rH 1 |X 2:T`1 " xs " r x DP rH 0 |X 1:T " xs, where r x ą 0 is a scalar.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
If W has full row rank, it has a left inverse W : with W : W " I |H|ˆ|H| . Choosing b " µD´1W : , we obtain 1pb J G 1 px 1 q ě 0q " 1pµ J D´1W : W P rH 1 | X 2:T`1 " xs ě 0q
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Next, we complete the proof of Proposition 3.4.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
" P rU | ZsP rZ | V " vs B Formal abstraction for prompt tuning and proofs for Section 3.1
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We first formalize the definition of the model G described in Section 3.1. The model G takes a sequence of embedding vectors v " pv 1 , . . . , v t q as input and implements message passing to compute a sequence of t outputs. We first define left and right messages Ð Ý δ i`1Ñi pvq and Ý Ñ δ i´1Ñi pvq for i P rts, as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Next, we define the aggregated message at timestep i by
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Note that if Assumption 3.2 holds about the Markov chain H 0 , H 1 , . . ., τ i pvq is always well-defined because P rH i s will have full support. Note that for the proper embeddings epx i q " P rX i " x i | H i s, where for x " px 1 , . . . , x t q, we use epxq " pepx 1 q, . . . , epx t qq, we can check via classical results on message passing [16] that
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Finally, we let the model model G compute
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
There is an edge case where the demoninator is 0, i.e. }τ i pvq} 1 " 0. To make the behavior of G well-defined, in this case we set G i pvq " 0 |X | . We observe that if the input embedding are obtained by epxq, G i pvq indeed computes the desired conditional probability vector for x P supppP rXsq:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
First we formalize the observation that soft prompt tuning is equivalent to adding a fake token r z to the vocabulary with emission probabilities at timestep 1 given by u, and letting G compute conditional probabilities for this new distribution over sequences.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Lemma B.1. In the setting of Theorem 3.6, fix any prompt vector u P r0, 1s |H| . Define the random variable p X with the same emission probabilities as X for i ą 1: P r p X i | H i s " P rX i | H i s. For timestep 1, we define the emission probabilities of p X 1 as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In the above equations, r z is a fake token added to the vocabulary at timestep 1. It follows that for any i, defining τ i as in (B.1)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
As a consequence, it follows that for i ą 1 and any x such that pr z, ∅, xq´i P supppP r p X´isq,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
For any x with pr z, ∅, xq´i R supppP r p X´isq, G i pp epxqq " 0.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Next, the following lemma disentangles the influences of the fake token r z and the input sequence on the posterior distribution of the hidden variable.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Lemma B.2. In the setting above, there exists an invertible diagonal matrix D such that for all x such that pr z, xq P supppP r p X´2sq, the following equation holds:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Here r x ą 0 is a positive scalar.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We now complete the proof of Theorem 3.6.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Theorem 3.6. Let B be the set defined in Assumption 3.5 and define u such that u h " 1 if h P B and u h " 0 otherwise. First, we restrict our focus to x such that pr z, xq P supppP r p X´2sq. For these x, we can apply Lemma B.1 and Lemma B.2 in the manner described in the proof sketch. This gives G 2 pp epxqq " r x W Dv for v fi pApu d P rH 1 sqq d P rH 0 | X 1:T " xs. By definition of B, we have supppApu d P rH 1 sqq " H ‹ , so supppDvq Ď H ‹ . Thus, there is a matrix y W : such that
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The existence of y W : is due to the fact that tW :,h u hPH ‹ is a linearly independent set of vectors, and supppDvq Ď H ‹ whenever x satisfies pr z, xq P supppP r p X´2sq. Next, we note that a matrix B exists such that pBDvq h " PrpH 0 " h | X 1:T " xq for h P H ‹ and pBDvq h " 0 otherwise. This is because D is invertible, and supppApu d P rH 1 sqq " H ‹ , so we can recover P rH 0 | X 1:T " xs on coordinates in H ‹ by applying another coordinate-wise scaling. It follows that we can set b " µ J B y W : . With this choice of b, we compute
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where the last equality follows because supppµq Ď H ‹ . This completes the case where pr z, xq P supppP r p X´2sq.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Otherwise, for pr z, xq R supppP r p X´2sq, by the behavior of G in Lemma B.1, G 2 pp epxqq " 0, so any linear head must output b J G 2 pp epxqq " 0. Furthermore, by the conditional independence structure in p X, we must also have supppP rH 2 , p X 1 " r zsq X supppP rH 2 , p X 3:T`2 " xsq " H. As supppµq Ď supppP rH 2 , p X 1 " r zsq, this must also mean supppµqXsupppP rH 2 , p X 3:T`2 " xsq " H. However, we also have P rH 2 , p X 3:T`2 " xs " P rH 2 , X 3:T`2 " xs by the definition of p X, and this must have the same support as P rH 0 | X 1:T " xs by applying Claim A.1 and the fact that x P supppP rXsq. It follows that for this choice of x, µ J P rH 0 | X 1:T " xs " 0, so the desired statement still stands.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We fill in the proofs of the lemmas below.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Lemma B.1. First, we note that (B.2) follows directly from the derivation of τ , and well-known results about message passing [16]. Next, it suffices to consider the case where pr z, ∅, xq´i R supppP r p X´isq, as the other case follows directly from the definition of G in terms of τ . In this case, we observe that τ i pp epxqq " P rH i , p
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
X´i " pr z, ∅, xq´is " 0. It follows that }τ i pp epxqq} 1 " 0. Thus, from our definition of G, we must have G i pp epxqq " 0.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Lemma B.2. By the conditional independence relations in a HMM, p X 1 K p X 3:T`2 | H 2 . Using Bayes' rule, we obtain
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Prp p X 1 " r z, p X 3:T`2 " xq (by definition of p X and the Markovian property)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
" r x P r p X 1 " r z, H 2 s d P rH 0 | X 1:T " xs d 1 P rH 0 s Where we define r x fi
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
. We note that r x is positive and well-defined by the conditions of the lemma and Theorem 3.6. We can set D to be the matrix diagp 1 P rH0s q, which has finite positive entries on the diagonal by Assumption 3.2.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
First, we introduce a proposition which is generally useful for proving the theorems in Section 4.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proposition C.1. In the setting of Section 4, it holds that P rX i | X´i " x´is " P rX i | M Ji , J i , S i sP rM Ji , J i , S i , X´i " x´is Equivalently, we have the expansion
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof. An alternative interpretation of this statement is that X i is conditionally independent from everything else given M Ji , J i , S i . However, we will prove this statement algebraically. We compute P rXi | X´i " x´is " Throughout this section, we use M Ji to denote the random variable obtained by indexing M by J i , both of which are themselves random variables. Let p I denote the set of indices i where supppP rJ i | X´i " x´isq " tj ‹ u and supppP rS i | X´i " x´isq Ď S ‹ . We will first construct the key function K and query q such that the set of I of attended-to positions (4.2) is precisely p I. This construction does not require the position embeddings β 1 , . . . , β t , so we set them to 0.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The following lemma demonstrates the existence of K and q such that I " p I.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Lemma C.2. In the setting of Theorem 4.3, define p I fi ti : supppP rJ i | X´i " x´isq " tj ‹ u and supppP rS i | X´i " x´isq Ď S ‹ u. Then there exist query q P R |H| and key K parameterized by Θ pKq P R |H|ˆ|X | , such that when x P supppP rXsq and p I is nonempty, the set I of attended-to positions satisfies I " p I.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The proof of Lemma C.2 requires the following claim.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Claim C.3. In the setting of Theorem 4.3, there is a matrix Θ p1q P R |H|ˆ|X | such that for all x P supppP rXsq and s P S ‹ , pΘ p1q G i pxqq pj ‹ ,sq " P rH i " pj ‹ , sq | X´i " x´is. Furthermore, }Θ p1q G i pxq} 1 " 1. In addition, for s P S ‹ , there exists Θ p2,sq P R |M|ˆ|X | such that for all x P supppP rXsq,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof. We have, by Proposition C.1,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In the last equality, we defined ν phq to be the expression in the parentheses. Note that ν phq P V phq fi spanptW :,pm,hq u mPM q. Furthermore, for h R H ‹ , ν phq P s V fi spanptW :,pm,hq u mPM,hPHzH ‹ q. As the spans pV phq q hPH ‹ and s V are all pairwise disjoint, by Assumption 4.2, for each h P H ‹ , we can recover
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Likewise, we can obtain
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Now we have, for h P H ‹ ,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Likewise, the same reasoning gives 1 J ř hRH ‹ ν phq " ř hRH ‹ PrpH i " h | X´i " x´iq. Thus, we can choose Θ p1q to be the matrix with rows Θ We set all other rows to 0, and we can check that this satisfies the lemma requirements.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We now construct Θ p2,hq . We can express ν phq in a vectorized manner by writing ν phq " W :,pM,hq P rM j , H i " h | X´i " x´is where W :,pM,hq P R |X |ˆ|M| has columns tW :,pm,hq u mPM . Note that for j " j ‹ , s P S ‹ , the non-degeneracy assumptions imply that W :,pM,j ‹ ,sq has left inverse W :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
:,pM,j ‹ ,sq . Thus, we set Θ p2,sq " W : :,pM,j ‹ ,sq B pj ‹ ,sq to obtain for s P S ‹ , Θ p2,sq G i pxq " W :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
:,pM,j ‹ ,sq B pj ‹ ,sq P rX i | X´i " x´is " W :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
:,pM,j ‹ ,sq W :,pM,j ‹ ,sq P rM j ‹ , H i " pj ‹ , sq | X´i " x´is " P rM j ‹ , H i " pj ‹ , sq | X´i " x´is This gives the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Lemma C.2. We choose the first |H| entries of q such that q h " 1 if h " pj ‹ , sq for s P S ‹ , and q h " 0 otherwise. The last entry is 0. Next, we choose Θ pKq so that the first |H| rows are Θ p1q , and the last row is all zeros. where Θ p1q is defined in Claim C.3. With this choice of Θ pKq , KpG i pxqq h " PrpH i " h|X´i " x´iq for h P H ‹ . Furthermore, }KpG i pxqq} 1 " 1, by Claim C.3. Now we note that for all i, 1 " }KpG i pxqq} 1 ě q J KpG i pxqq, and for i P p I, q J KpG i pxqq " ř sPS ‹ PrpH i " pj ‹ , sq|X´i " x´iq " 1 by definition of q and p I. This implies that positions i P p I do indeed achieve the maximum attention scores.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Next, we also require a construction of the value function such that it computes the correct prediction for all i P p I.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Lemma C.4. In the setting of Theorem 4.3, let p I be defined as in Lemma C.2. We can choose the parameters of the value function V , Θ pV q P R |M||H|ˆ|X | , b P R |M||H| , such that when x P supppP rXsq and p I is nonempty, for all i P p I,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where r x,i ą 0 is a positive scalar.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof. We first choose Θ pV q such that the rows satisfy Θ pV q pm,j ‹ ,sq,: " Θ p2,sq m,: when s P S ‹ for Θ p2,sq constructed in Claim C.3, and Θ pV q pm,j,sq,: " 0 |X | otherwise for j ‰ j ‹ or s R S ‹ .
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We claim that for i P p I,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
This is because for s P S ‹ , Θ p2,sq G i pxq " P rM j ‹ , H i " pj ‹ , sq | X´i " x´is by Claim C.3, and for h " pj, sq for j ‰ j ‹ or s R S ‹ ,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Note that this last equality followed because PrpH i " h | X´i " x´iq " 0 for the choice of h and i P p I. By construction of Θ pV q , these computations imply that (C.2) does indeed hold. The embedding can be chosen such that epx i q " P rX i " x i | M Ji , J i , S i s. Thus, we have for i P p I:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The last equality followed from applying the same reasoning as in Proposition C.1.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Now we let B P R |M|ˆ|M||H| be the matrix such that
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Now we pick the last linear weight in the value function by b " B J µ. It follows that for i P p I,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We obtained the last equality by observing that Proof of Theorem 4.3. By applying Lemmas C.2 and C.4, we constructed key, query, and value functions for the attention head such that for all x P supppP rXsq with p I (defined in Lemma C.2) nonempty, the attended-to positions I satisfy I " p I, and V pG i pxq, epx i qq " r x,i µ J P rM j ‹ | X 1:T " xs. As the attention head computes the average of V pG i pxq, epx i qq over attended-to positions, and r x,i is positive for all i P p I, we obtain the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We note that this proof also works for the case where there is a single memory cell, as that is a special case where J i " j ‹ always, and we only need to consider the evolution of S i . Now we formalize the model G. G will take embedding vectors v " pv 1 , . . . , v t q with v i P R | r H| as follows. We define left and right messages Ð Ý δ i`1Ñi pvq and Ý Ñ δ i´1Ñi pvq for i P rts via:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We observe that this definition almost matches Section B, except it replaces H with r H. Next, we define the aggregated message at timestep i by
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In the edge case where P rM s does not have full support, the coordinate-wise division in the definition above would sometimes divide by 0. However, for all these cases both of the corresponding terms in the numerator must also be 0, so we can simply set the value of τ i in this coordinate to 0. We will see that this preserves the meaning of the message τ i , which for the proper embeddings epx i q " P rX i " x i | r H i s, with epxq " pepx 1 q, . . . , epx t qq, computes
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We can now define the reverse lifting function φ : R | r H|Ñ|M||H| as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We observe that φpτ i pepxqqq " P rM J i ,Ji,Si,X´i"x´is |M| N´1
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
. We now compute the model output as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In the edge case where }φpτ i pvqq} 1 " 0, we again define Gpvq " 0 |X | . We can observe that G i pepxqq " P rX i | X´i " x´is.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The downstream classifier uses the embedding p epxq defined as follows:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
p epxq " pu, epx 1 qq, . . . , epx t qqq with a tunable prompt embedding u P R | r H| . We also require a slightly modified attention head. The value function V in the attention head is slightly modified to accomodate the new embedding dimension. Letting
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The dimensions of the parameters b, Θ pV q remain unchanged. Note that when there is just a single memory, this reduces to the case in Section 4.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We will state and prove our result for the prompt tuning setting with multiple memories. For the multiple memory setting, the downstream classifier uses the following embedding function p e:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
p epxq " pu, ηpepx 1 qq, . . . , ηpepx t qqq with a tunable prompt embedding u P R | r H| . The attention head is changed so that the value function takes a larger dimensional embedding:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
where φ is defined in (C.4). The following assumption extends Assumption 4.5 to the multiple memory case.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Assumption C.5 (Multiple memories version of Assumption 4.5). Let M ‹ fi supppµq denote the set of non-zero coordinates in µ. There exists a set of recoverable hidden states H ‹ , such that the collection of token emission probabilities from M ‹ˆH‹ , tW :,pm,hq u mPM ‹ ,hPH ‹ , is a linearly independent set of vectors.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Furthermore, define the following span of vectors: s V fi spanptW :,pm,j ‹ ,sq u mPM ‹ ,sPSzS ‹ Y tW :,pm,j,sq u mPM,j‰j ‹ ,sPS q Then s V must be disjoint from the span of token emission probabilities from M ‹ˆH‹ :
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Note that Assumption C.5 reduces to Assumption 4.5 the case where N , the number of memory cells, is 1. In any case, it is a relaxation of Assumption 4.2.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We now state and prove the result for multiple memories.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Theorem C.6. In the setting above, suppose that non-degeneracy Assumption C.5 and holds. In addition, suppose that Assumption 4.4 (stationarity) holds. Then there exists a prompt u and attention head on Gpp epxqq and the token embeddings which can compute the ground-truth F ‹ pxq for any x P R, defined in (4.3):
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The proof of Lemma C.7 mirrors the proof of Lemma B.1, so we omit it here.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In particular, throughout the proof we will use the following prompt u:
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We will also use the notation p x fi pr z, x 1 , . . . , x t q. The following lemma considers behaviors in edge cases with this choice of u.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Towards our proofs, the following result is useful.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proposition C.8. In the setting of Theorem C.6, where P rH 0 s is the stationary distributions satisfying P rH 0 s " AP rH 0 s, it holds that P rM, H i , X i`1:i`t s " P rM, H 0 , X 1:t s for any t ě 1, i ě 1.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof. Because P rH 0 s is stationary, we observe that P rM, H i s " P rM, H 0 s for all i. We write
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We will now restrict our focus to the set of inputs Z fi tx : Prp p X´i " pr z, xq´iq ą 0 @i P rtsu (C.7)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We also define the set p I fi ti`1 : supppP rS i |X´i " x´isq Ď S ‹ , supppP rJ i |X´i " x´isq Ď tj ‹ u, i P rtsu (C.8)
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Here S ‹ is defined in the non-degeneracy assumption. We will first construct key and query parameters such that the set of attended-to positions is precisely p I, following the proof of Theorem 4.3.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Lemma C.9 (Analogue to Lemma C.2). In the setting of Theorem C.6 and above, define u as in (C.6).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
There are parameters Θ pKq P R p|H|`1qˆ|X | , q P R |H|`1 , and β 1 , β 2 , . . . P R |H|`1 such that for any x P Z where p I is nonempty, the set of attended-to positions I (defined in (4.1)) satisfies I " p I.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Towards proving Lemma C.9, the following construction will be useful.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Claim C.10 (Analogue of Claim C.3). In the setting of Theorem C.6, define H ‹ as in Assumption C.5. There is a matrix Θ p1q P R |H|ˆ|X | such that for all x P supppP rXsq, and i ą 1 with Prp p X´i " p x´iq ą 0,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
X´i " p x´is for any h P H ‹ . Furthermore, }Θ p1q G i pp epxqq} 1 " 1.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In addition, for s P S ‹ , there exists Θ p2,sq P R |M|ˆ|X | such that for all i ą 1 and x with Prp p X´i " p x´iq ą 0, Θ p2,sq G i pp epxqq " P rM j ‹ , H i " pj ‹ , sq | p X´i " p x´is Our proof will require the following result which shows that the distribution of M j ‹ has limited support.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proposition C.11. In the setting of Theorem C.6 and Lemma C.7, let u be defined as in (C.6). Then for all i ą 1, supppP rM j ‹ | p X´i " p x´isq Ď supppµq if Prp p X´i " p x´iq ą 0.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
First, if pr z, xq R supppP r p Xsq, by Claim C.15, we have µ J P rM j ‹ | X 1:T " xs " 0. The expression above must also equal 0, as pr z, xq R supppP r p Xsq. Otherwise, we have V pG i pp epxqq, p e i pxqq " µ J P rM j ‹ | p X " p xsPrp p X i " p x i | p X´i " p x´iq Now we apply Claim C.14 to get the desired result in this case. A additional case is when Prp p X´i " p x´iq " 0. In this case, Claim C.15 shows that G i pp epxqq " 0, so it follows that the value function also computes 0 in this case.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Finally, we need to check the case where p
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
x R supppP r p Xsq, and we want to show V pG i pp epxqq, p e i pxqq " 0 for all i ą 1. The case where Prp p X´i " p x´iq " 0 is already handled above. In the case where Prp p X´i " p x´iq ą 0, we can apply Claim C.10 to our construction for Θ pV q to get pΘ pV q G i pp epxqqq m,h " # P rM j ‹ " m, H i " pj ‹ , sq | p X´i " p x´is if h " pj ‹ , sq for s P S ‹ 0 otherwise Thus, taking the element-wise product with φpepx i qq " P r p X i " p x i | M Ji , J i , S i s, we must have, by Proposition C.1,
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Both of these terms must be 0 since p x R supppP r p Xsq, giving the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Now we are ready to prove Theorem C.6.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Proof of Theorem C.6. The first case we consider is when x P Z, defined in (C.7). By applying Lemmas C.9 and C.13, we constructed key, query, and value functions for the attention head such that when p I (C.8) is nonempty, the attended-to positions I satisfy I " p I. In addition, by applying Lemma C.13, we also obtain that for x P supppP rXsq, V pG i pp epxqq, p e i pxqq " r x,i µ J P rM j ‹ | X 1:T " xs. As the attention head averages V pG i pp epxqq, p e i pxqq over the attended-to positions, and r x,i is positive for all i P p I, we obtain the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
In the second case, x R Z, so pr z, xq R supppP r p Xsq. By Lemma C.13, for all i ą 1, the value function outputs 0. However, by the construction in Lemma C.9, the attention will only attend to i ą 1. Thus, the output of the attention head is 0. However, Claim C.15 also implies that µ J P rM j ‹ | X 1:T " xs " 0, giving the desired result.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Generating HMM parameters. For all experiments, we randomly generated the parameters of an HMM with 10 output symbols in its vocabulary. We generate a random transition matrix by taking a random convex combination of random permutation matrices. We mix as many permutation matrices as there are hidden states; i.e. if there are 4 hidden states, then we mix 4 random permutation matrices. The mixing weights are generated by sampling logits IID from a uniform distribution on r0, 1s and then taking a softmax with temperature 0.01. Although this is a small temperature, the transition probabilities can still be around 0.7 for some transitions. The start distribution is also sampled in the same way, but with softmax temperature 10.0. The rows of the emission probability matrix is also sampled the same way with temperature 0.01.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Pretrain model. The pretrained model follows the BERT-base architecture, except with 6 layers and a much smaller vocab size.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Pretrain data and task. The pretraining data consists of 5000 sequences (documents) generated from the HMM, each with length 10240. We pretrain on this data by doing 5% masked LM on chunks of length 512. Pretraining runs for 3 epochs and takes about 5 hours on a single NVIDIA Tesla K80 GPU on 16-bit precision. We use an internal cluster for all experiments. Pretraining uses batch size 8 and learning rate 1e-5 with a linear warmup of 500 steps and linear decay schedule after 500 steps. We generated 20 pretraining (and downstream) datasets for each problem instance and average over the 20 runs in the vanilla HMM comparison, while the memory-based distributions are run for 5 trials of pretraining and finetuning. Downstream. The downstream task samples a sparse ground truth linear weight µ with 6 nonzero elements. Positions for nonzero entries are sampled uniformly at random and values are sampled i.i.d. from a standard normal distribution. Although we do binary classification, we sample µ with 2 rows and take the label to be the argmax of the two scores, instead of having 1 row and taking the sign. We find that this results in less degenerate datasets (datasets where all labels are the same).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We generate 5000 training, 500 validation and 1000 test examples for the downstream tasks. Downstream training uses learning rate 0.01 for both prompt tuning and head tuning, with a linear warmup/decay schedule, for 5 epochs over the downstream data. We take the model returned at the last checkpoint as the result (no early stopping). We found that it was important to train prompt tuning with full precision, since the gradients are relatively small and become zero with discretization.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We used message passing in the HMM to compute the posterior distributions of the latent variables analytically.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Prompt tuning. We prepended a length 20 continuous prompt to each sequence of input word embeddings. We initialize elements of the prompt vectors IID from the uniform distribution on r´0.5, 0.5s. Our implementation for prompt tuning used the code of [20], available at https://github.com/kipgparker/ soft-prompt-tuning.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Large language models (LLMs) have demonstrated the ability to perform in-context learning (ICL), i.e., "learning" to perform a task purely from examples in the context without any parameter updates (Brown et al., 2020). This powerful and flexible phenomenon enables LLMs to be used as general-purpose models that can perform any task with a small set of labeled examples.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
However, there is still no consensus on how incontext learning works. Some previous work hy- 1 Our code is publicly available at https://github.com/ princeton-nlp/WhatICLLearns. pothesizes that during pre-training, LLMs implicitly learn tasks required for downstream applications, and the in-context demonstrations merely provide information that allow the model to recognize which task is required (Xie et al., 2022). Min et al. (2022) show empirical evidence of this hypothesis by demonstrating that ICL performance is insensitive to the usage of ground-truth labels.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
On the other hand, Akyürek et al. (2023);von Oswald et al. (2022) construct theories that Transformer-based models may perform implicit gradient descent to update an "inner-model", and Dai et al. (2023) demonstrate similarities between in-context learning and explicit fine-tuning through a series of metrics on real-world datasets. Such hypotheses assume the correct input-output mappings are important and ICL actually performs implicit learning over demonstrations.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
In this paper, we disentangle ICL into task recognition (TR), which recognizes the task from demonstrations and applies LLMs' pre-trained priors, and task learning (TL), which learns a new input-label mapping from demonstrations. In common ICL scenarios where ground-truth labels are provided, TR and TL take effect simultaneously. We propose two settings to tease them apart: 1) RANDOM, where the labels are uniformly sampled from the label space (Min et al., 2022), in order to restrict LLMs to only apply TR; 2) ABSTRACT, where the labels are replaced with abstract symbols (e.g., numbers or letters) that never co-occurred with the inputs in pre-training. We focus on how the two abilities in ICL evolve with two factorsmodel sizes and numbers of demonstrations, which have been neglected in related literature.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Through extensive experiments with a series of classification datasets on GPT-3 (Brown et al., 2020), LLaMA (Touvron et al., 2023), and OPT (Zhang et al., 2022), we find:
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
• The gap between GOLD and RANDOM is small with smaller models, corroborating with Min et al. (2022). However, with larger models and more examples, the gap becomes larger. This suggests TR plays a significant role in ICL, but it does not scale with increasing parameters or examples. • LLMs also perform TL, which emerges with larger models and more demonstrations. With the largest model and more than 16 examples, ABSTRACT outperforms RANDOM, pointing to a paradigm shift in in-context learning at scale. Together, our findings provide a better way to understand ICL behaviors. 2
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
An LLM (parameterized by θ) performs ICL by conditioning on the input-label pair demonstrations D demo = (x 1 , y 1 , x 2 , y 2 , . . . , x K , y K ) and the test input x test to predict the label y test ∼ p θ (y | D demo , x test ), where the demonstrations elicit a mapping f : X → Y, x ∈ X , y ∈ Y. We delineate two ways an LLM can leverage in-context demonstrations: task recognition and task learning.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Task recognition (TR) represents models' ability to recognize the mapping f purely by observing the input distribution {x i } K i=1 and the label distribution {y i } K i=1 , without the provided (x i , y i ) pairs.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
The LLM then applies its pre-trained priors to the recognized f . Formally, when only TR is enabled,
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
which suggests TR does not rely on the pair information. For example, an input distribution of movie reviews and a label distribution of "The sentiment is positive/negative" can be easily recognized as a sentiment classification task due to their prevalence during pre-training, and LLMs can make reasonable predictions without explicitly "learning" the task via ground-truth demonstrations. This leads to observations that the model can still perform well even when we provide wrong input-label mappings, e.g., "The movie is great. The sentiment is negative" (Min et al., 2022). Task learning (TL), on the other hand, characterizes how the model learns a new mapping from the input-label pairs through demonstrations. Unlike TR, TL allows models to learn novel mappings and thus correct input-label pairs will be crucial. We posit that the two mechanisms occur under separate conditions, as recognizing an already learned task is easier than learning a new mapping. Models are able to perform TR at a small scale, but this ability does not drastically improve with increasing model sizes and demonstrations; on the other hand, TL improves significantly when model sizes and numbers of demonstrations increase. To show the above phenomenon, we disentangle TR and TL through label space manipulation, including three different setups (examples in Figure 1):
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
• GOLD: the standard ICL setting where we use natural prompts and gold input-label pairs. This setup reflects both TR and TL abilities. • RANDOM: similar to Min et al. (2022), we use the same natural prompts as GOLD and sample demonstration labels uniformly at random from the label space. This setup reflects TR only. • ABSTRACT: we use minimal prompts (which provide no task information) and characters with no clear semantic meanings (e.g. numbers, letters, and random symbols) as the label for each class. We found that even abstract labels may have biases in pre-training, e.g., "0" is biased towards negative. Hence, for each prompt x 1 , y 1 , . . . , x K , y K , we randomly sample a 1-1 mapping φ : Y → Y * to avoid any bias, and no task-specific information is leaked in either the prompt template or the label space. To evaluate the model's ABSTRACT performance, we measure its accuracy using φ(y test ) as target labels. Since these input-label mappings are never seen in pre-training, it reflects the TL ability.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
In the following sections, we conduct comprehensive experiments with the above three different settings under two axes -model sizes and numbers of demonstrations -and show how TR and TL manifest under different conditions.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
We experiment on 16 classification datasets across 4 type of tasks: sentiment analysis, toxicity detection, natural language inference/paraphrase detection, and topic/stance classification. All datasets and references are in Appendix A. Our dataset selection largely follows Min et al. (2022), but we exclude multi-choice datasets since it is difficult to apply our ABSTRACT experiments on them.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
We use three state-of-the-art LLM families: GPT-3 (Brown et al., 2020) We design two kinds of prompts: natural language prompts (Table 1), which are similar to the manual prompts in Min et al. (2022), and minimal prompts (Table 3), which remove any natural language instructions for the task. For ABSTRACT, we tested three types of label choices: numbers (0, . . . , N − 1, where N is the number of classes), letters (N letters from A, B, C, . . . ), and symbols (first N symbols of "@", "#", "$", "%", "*", and "∧"). For each test example, we randomly sample a new mapping between labels and abstract characters. We report the number abstract labels in all the main results and compare the three forms in §4.2. Figure 2 shows our main results with GPT-3, LLaMA, and OPT with our 3 settings: GOLD, RANDOM, and ABSTRACT. Below we summarize the trends of TR and TL across different conditions.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Summary of overall trends. We first verify that GOLD consistently performs the best across model families and number of demonstrations, which is expected given that the GOLD setting provides the model with all information. Overall, the RANDOM curves do not increase with either model sizes or number of demonstrations, remaining largely flat; considering the scenario with small model sizes and few examples (K = 8), there is an insignificant gap between RANDOM and GOLD. Meanwhile, the ABSTRACT curves demonstrate an increasingly steep slope as the model sizes and the number of demonstrations grow; with small models or small K, ABSTRACT mostly underperforms RANDOM, whereas ABSTRACT with largest models and K = 32 performs well above RANDOM (and may even be competitive with GOLD). We note that the OPT curves demonstrate significant variance, which we hypothesize to be a result of the models potentially being under-trained. We elaborate the takeaways on TR and TL below.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Task recognition is a broader capability across scales. For all model families, the RANDOM setting shows similar performance at all sizes and numbers of demonstrations. Moreover, TR performance is significantly stronger than the random baseline, even with small models and few examples. For instance, even the smallest 350M parameter models are able to recognize the task using just 8 examples, drawing around 10 points of average performance lead against the random baseline for GPT-3 ada and 5 points for OPT-350M. This shows that task recognition from in-context examples does not drastically scale with model sizes or numbers of examples.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Task learning is enabled with scale. We observe that TL is dependent on model sizes: smaller models perform roughly the same across all numbers of demonstrations (see Figure 6). On the other hand, larger models can utilize the provided mapping information and perform TL, as ABSTRACT (TL) performance increases drastically with larger sizes (first row of Figure 2). When using a larger model, the results also improve as the number of demonstration increases (second row of Figure 2). With only 16 examples, OPT-66B and davinci are able to match the performance of GOLD while using a new label mapping. While LLaMA-65B's ABSTRACT is not as competitive as its GOLD, the trend of improving ABSTRACT performance with larger size s or larger K is clear. This suggests that TL is only enabled by scales and further improves with more demonstrations.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
The trends for task learning generalize across different types of abstract labels. In Figure 3, we show ABSTRACT results with number, letter, and symbol labels respectively. We observe that all three versions show a similar trend and coincide with our main results. Numbers and letters perform consistently better than symbols. This may be because letters and numbers appear more frequently in the pre-training corpus, and therefore make for a more "natural" label space.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Task difficulty affects the trends. We notice that ABSTRACT scales better with sizes and examples when the task is simpler. In Figure 4 we compare two types of tasks: sentiment analysis and natural language inference (NLI). Since NLI is more difficult, we observe that it produces a flatter AB-STRACT curve, suggesting that the model relies more on the natural prompts and pre-training priors to solve those tasks. We demonstrate the full task-type breakdown results in §C.  (2022) identify properties of the pre-training distribution that enable ICL behavior, including burstiness, label multiplicity, and a long-tailed class distribution -all of which are satisfied by natural language. Razeghi et al. (2022) show that the frequencies of terms in the pre-training corpora is positively correlated with model performance.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
While previous work often studies ICL as an umbrella term, regardless of model sizes and numbers of examples, we argue that there are two distinct characterizations of ICL -task recognition and task learning -and demonstrate that they emerge under different conditions. Even small models are capable of performing TR, but this ability does not scale. On the other hand, TL is an emergent ability of large models; small models are unable to perform TL even when provided with more demonstrations, whereas large models can leverage more demonstations to improve their TL performance. We suggest that future work on ICL should distinguish the two phenomena and clearly state the conditions under which the experiments are conducted.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Though LLMs with in-context learning are capable of all kinds of NLP tasks, this work is limited to classification tasks because they are easier to be adapted to our RANDOM and ABSTRACT setup. We leave other types of NLP tasks as future work.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
Another limitation of our work lies in the definition and discussion of task learning. Though we empirically show that large models are capable of acquiring a novel mapping to abstract labels like numbers or letters, how models "learn" mechanistically is still elusive. As suggested in previous work, LLMs may conduct implicit gradient descent over demonstrations, or they may alternatively map the patterns shown in the demonstrations back to concepts learned in pre-training. To some extent, these mechanisms could be considered an advanced form of "task recognition". This work only designs experiments to better observe and disentangle TR and TL, and we look forward to further studies that reveal more insights about the mechanistic innerworkings of these phenomena in ICL. 
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
For each task category (e.g. sentiment classification, topic detection), we manually design three natural language templates. Depending on exact specifications for the dataset, templates may be adjusted to better reflect the task (e.g. "Is this atheist?" for tweet_eval_atheist). We apply these templates to the natural language label sets (GOLD and RANDOM). All prompts are presented in Table 1.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
We also design two task-agnostic variations on three minimal templates for ABSTRACT: one for single-sentence tasks and one for multi-sequence tasks (e.g. NLI tasks). We use these minimal templates on the abstract language label sets in order to prevent the model from being exposed to any information regarding the task from the prompt design. All minimal templates are presented in Table 3 All prompts are designed to be answered with single-token responses (e.g.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
"Yes/No", "True/False", "positive/negative/neutral", "0/1/2", "A/B/C") so that we can directly check models' last token prediction results instead of applying decoding methods.
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
We demonstrate average model performance with respect to number of parameters in Figure 5. It is clear that small models struggle to perform ABSTRACT, regardless of how many examples, whereas the largest models (especially GPT-3 Davinci and OPT-66B) are able to perform AB-STRACT. Additionally, their performance improves even more when more demonstrations are provided. We demonstrate average model performance with respect to numbers of demonstrations in Figure 6. We can see a clear trend that RANDOM (TR) does not change much but ABSTRACT improves drastically with more examples, especially for GPT-3 Davinci and OPT-66B. Figure 7 shows all the ABSTRACT results and demonstrates a similar trend to what §4.2 describes. Figure 8, Figure 9, Figure 10, and Figure 11 show task-type breakdown results. Though individual task-type results are more noisy, we can make a similar observation compared to the main result -ABSTRACT (TL) scales better with sizes and numbers of examples compared to RANDOM (TR).                  
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Few-shot learning-the ability to learn tasks with limited examples-is an important aspect of intelligence (Lake et al., 2015;Yogatama et al., 2019). Recent work shows that large neural language models can perform few-shot learning without finetuning (Radford et al., 2019;Brown et al., 2020). Specifically, GPT-3 (Brown et al., 2020) can perform numerous tasks when provided a few examples in a natural language prompt. For example, to perform sentiment analysis one can condition GPT-3 on a prompt such as: where the first two lines correspond to two training examples and the last line is a test example. To make predictions, the model predicts whether the subsequent token is more likely to be the word "Positive" or "Negative".
Calibrate Before Use: Improving Few-Shot Performance of Language Models
This style of few-shot "in-context" learning is interesting because it shows that the model can learn without parameter updates. And, more importantly, it has numerous practical advantages over the now-standard approach of finetuning (Radford et al., 2018;Devlin et al., 2019). First, it allows practitioners to "rapidly prototype" NLP models: changing the prompt immediately leads to a new model. Second, it provides a fully natural language interface to a machine learning model, which allows users-even those without technical expertise-to create NLP systems. Finally, since in-context learning reuses the same model for each task, it reduces memory requirements and system complexity when serving many different tasks.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Neural autoregressive language models (LMs) take as input a sequence of tokens and output a probability distribution over the next token. Large neural LMs can perform tasks in a zero-or few-shot manner using in-context learning (Radford et al., 2019;Brown et al., 2020). To do so, a natural language prompt is fed into the model. This prompt contains three components: a format, a set of training examples, and a permutation (ordering) of the training examples.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We study text classification using six datasets: sentiment analysis using SST-2 (Socher et al., 2013), 6-way question classification using TREC (Voorhees & Tice, 2000), textual entailment using 3-way CB  Fact Retrieval We evaluate fact retrieval with LAMA (Petroni et al., 2019). The dataset consists of knowledge base triples that are placed into templates with missing objects, e.g. "Obama was born in". We use these templates as our prompts, and remove the relations where the missing answer is not at the end of the template (left-to-right LMs cannot solve these). The answers are always single tokens, and we report average accuracy across all triples.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We consider information extraction using two slot filling datasets, ATIS (Hemphill et al., 1990) and MIT Movies trivia10k13 (Liu et al., 2012). We use two random slots for each dataset, airline and departure date for ATIS, and director name and movie genre for MIT Movies. The answer for both datasets is a span of text from the input, e.g., the ATIS airline task is to predict "american airlines" when given the sentence "list a flight on american airlines from toronto to san diego". We use Exact Match between the model's generated output and the ground-truth span as our evaluation metric.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Thus far, we have shown that GPT-3 is biased towards certain answers due to the prompt and the model's intrinsic biases. Here, we look to correct this by "calibrating" the model's output probabilities. 4 A common technique for adjusting output probabilities is to apply an affine transformation (Platt, 1999;Guo et al., 2017):
Calibrate Before Use: Improving Few-Shot Performance of Language Models
where a weight matrix W and a bias vector b are applied to the original probabilitiesp to get the new probabilities 3 The frequency of a token on the web is calculated using Google Ngrams https://books.google.com/ngrams. The predictions are from the 0-shot setting on the validation set. 4 The output of GPT-3 is biased (its outputs are shifted), similar to how measurement devices such as voltage meters or weighing scales are biased. Just like how these devices require "calibration before use", where the devices' outputs are scaled/zeroed-out, we hope to apply a similar calibration procedure to LMs. This goal is distinct from statistical calibration (Brier, 1950;Guo et al., 2017), i.e., aligning a model's confidence estimate with its true accuracy.q . 5 For classification tasks,p is the set of probabilities that are associated with each label name, renormalized to one. For generation tasks,p is the entire set of probabilities for the first token. 6 In this paper, we restrict the matrix W to be diagonal, known as vector scaling (Guo et al., 2017), to prevent the parameters from growing quadratically in the size ofp (which is ≈ 50, 000 for generation tasks).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Implementation Details This contextual calibration procedure adds trivial amounts of computational overhead and is implemented in a few lines of code (compute and savê p cf , adjust output probabilities). For the content-free input, many good choices exist, including "N/A", the empty string, and gibberish tokens. In all our experiments, we average the probabilities from three content-free inputs: "N/A", "[MASK]", and the empty string. 8 One could also craft the content-free input in a task-specific manner. We explore this for LAMA, where we replace the subject with the contentfree input, e.g., we use "N/A was born in" as the input. 5 This affine transformation is usually applied to the logits, i.e., prior to the softmax. However, we only have access to GPT-3's output probabilities in the OpenAI API. 6 We only calibrate the prediction of the first output token for generation tasks. This is reasonable because, for the tasks we consider, we found that the model's predictions are highly deterministic after generating the first token. 7 An alternate solution is to set b to −p cf and W to the identity. Empirically, this alternate solution yields higher accuracy for generation tasks (where the dimensionality ofp is large). The solution in the main text performs better for classification. 8 We found this simple ensemble to achieve the best results for AGNews, and we reuse it for all other datasets. See Section 5.2 for an ablation on the choice of content-free input.  Table 1. Contextual calibration improves accuracy across a range of tasks. We show the mean and standard deviation across different choices of the training examples (the prompt format is fixed). The LM column indicates the GPT-3 size (see Appendix A for GPT-2 results). The Baseline column shows the standard approach of greedy decoding (Brown et al., 2020) and Ours corresponds to greedy decoding after modifying the output probabilities using contextual calibration. We bold the better result of the baseline and ours. MIT-G, MIT-D, ATIS-A, and ATIS-D indicate the MIT Genre, MIT Director, ATIS Airline, and ATIS Departure Date datasets.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Does Calibration Eliminate the Need to Engineer Prompts? The motivation behind "prompt engineering" is that not all prompts lead to the same accuracy. Thus, one should tune the prompt's format and examples to achieve the best possible performance (Brown et al., 2020;Gao et al., 2020). Contextual calibration does not eliminate the need to engineer prompts, however, it does mitigate it: contextual calibration makes the accuracy of the best, average, and worst-case prompts more similar (and higher).
Calibrate Before Use: Improving Few-Shot Performance of Language Models
We identify failures when LMs are used for in-context learning (e.g., recency bias). Past work identifies similar failures when LMs are used for text generation. For example, neural LMs often repeat themselves (Holtzman et al., 2020), suffer from overconfidence (Braverman et al., 2020;Jiang et al., 2020a), suffer from recency bias (Khandelwal et al., 2018;Ravfogel et al., 2019), and prefer generic responses instead of rare text (Li et al., 2016;Logan et al., 2019). Past work mitigates these degeneracies by modifying the model's output probabilities or generation schemes, e.g., explicitly preventing repetitions (Paulus et al., 2018) or using sampling instead of greedy decoding (Holtzman et al., 2020).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Natural language processing (NLP) has been revolutionized by large-scale pretrained language models such as BERT [4] and GPT [27], which are adapted to a variety of downstream NLP tasks. Although a large body of empirical work seeks to understand the effectiveness of pretrained models [7,5,12,37,36,11,29,15], theoretical understanding is scarce. Theoretically analyzing the relationship between the pretraining and downstream tasks is challenging because pretraining and downstream settings can greatly differ.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
We empirically evaluate our theoretical results with language models pretrained on synthetically generated data from HMMs. We find that prompt tuning obtains good downstream performance when our non-degeneracy conditions are relaxed, whereas head tuning performs poorly. Furthermore, we show that head tuning obtains better downstream performance when data is generated from a memory-augmented HMM, compared to a vanilla HMM, as is predicted by our theory. 1 
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
The black box nature of BERT and related models has inspired a variety of empirical works which seek to understand them. Probing papers study whether a pretrained model computes various types of structured information (e.g., syntactic [37,11]) by evaluating the performance of simple classifiers, or probes, on the representations [7,12,36,29,15]. Other papers ablate various aspects of pretraining, such as changing the masking scheme [14,21,42] or permuting the word order [34].
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Prompt-based tuning [33,20,9,22,13,6,43,2,25], which has improved empirical downstream performance for lightweight adaptation methods beyond head tuning to approach full finetuning, is an important focus of our theoretical analysis. Shin et al. [33] employ task-specific prompts that are optimized over the discrete token space. Schick and Schütze [31,32] reformulate natural language tasks as cloze-style phrases to enable few-shot learning. Subsequent methods [20,9,22] optimize "soft" prompts, or continuous embedding vectors. Lester et al. [20] employ soft prompts on pretrained large-scale T5 [28] models and show that as the model size increases, prompt tuning performance can eventually match finetuning. Hambardzumyan et al. [9] applies a variant of soft prompt tuning to MLM models. Li and Liang [22] propose prefix tuning, which prepends a trainable prefix embedding sequence to all layers of the transformer.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
More broadly, Lee et al. [19] analyze reconstruction-based self-supervised learning methods in a general setting and show that under certain conditional independence assumptions, predicting one observed variable from another allows recovery of the latent with a linear head. Other theoretical works analyzing self-supervised or constrastive learning include [1,10,38,40,39,23], but they are not directly relevant for our particular setting.
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Defining a relation between pretraining and downstream tasks is the foremost challenge for analysis. We propose to link the two via latent variable generative assumptions on the input distribution. We model the downstream task as a function of the posterior distribution of the latent variables. Towards a first result, this section studies the case where inputs are generated by HMMs (see Figure 1 (left)), which have been well-studied in the context of language and speech processing (see e.g. [26,18,3]).
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Data distribution. The memory-augmented HMM, depicted in Figure 2, can be viewed as a generative variant of memory networks [41,35] and is closely related to Hidden Topic Markov Models [8]. There are two sets of latent variables in the memory-augmented HMM: a Markov chain on hidden states H 0 , H 1 , . . ., meant to model the evolution of syntax, and a persistent "memory" M " pM 1 , . . . , M N q with N total cells, where each M i takes values in a finite set M. The full joint probability is as follows:
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Pretrained masked language models (Devlin et al., 2019;Liu et al., 2019b) have benefitted a wide range of natural language processing (NLP) tasks (Liu, 2019;Wadden et al., 2019;Zhu et al., 2020). Despite recent progress in understanding what useful information is captured by MLMs (Liu et al., 2019a;Hewitt and Manning, 2019), it remains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Cloze reductions reformulate an NLP task into a prompt question and a blank and elicit answers by filling in the blank (Figure 1). When tested by cloze reductions pretrained MLMs and left-to-right language models (LMs) have been shown to possess abundant factual knowledge (Petroni et al., 2019) and display impressive few-shot ability (Brown et al., 2020). This success has inspired recent hypotheses that some word masks are cloze-like and provide indirect supervision to downstream tasks Lee et al., 2020). For example, a sentiment classification task (Pang et al., 2002) can be reformulated into filling in like or hate in the cloze I [MASK] this movie. Such cloze-like masks provide a clear way in which an MLM can implicitly learn to perform sentiment classification.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To address this mismatch between theory and practice, we offer a new hypothesis of how generic masks can help downstream learning. We propose a conceptual model for MLMs by drawing a correspondence between masking and graphical model neighborhood selection (Meinshausen and Bühlmann, 2006). Using this, we show that MLM objectives are designed to recover statistical dependencies in the presence of latent variables and propose an estimator that can recover these learned dependencies from MLMs. We hypothesize that statistical dependencies in the MLM objective capture useful linguistic dependencies and demonstrate this by using recovered statistical dependencies to perform unsupervised parsing, outperforming an actual unsupervised parsing baseline (58.74 vs 55.91 UUAS; Klein and Manning, 2004). We release our implementation on Github 1 .
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Theories inspired by Cloze Reductions. Cloze reductions are fill-in-the-blank tests that reformulate an NLP task into an LM problem. Existing work demonstrates that such reductions can be highly effective for zero/few-shot prediction (Radford et al., 2019;Brown et al., 2020) as well as relation extraction (Petroni et al., 2019;.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
These fill-in-the-blank tasks provide a clear way by which LMs can obtain supervision about downstream tasks, and recent work demonstrates how such implicit supervision can lead to useful representations . More general arguments by Lee et al. (2020) show these theories hold across a range of self-supervised settings. While these theories provide compelling arguments for the value of pre-training with cloze tasks, they do not provide a clear reason why uniformly random masks such as those used in BERT provide such strong gains. In our work, we quantify this gap using lexicon-based cloze-like masks and show that cloze-like masks alone are unlikely to account for the complete success of MLM since generic and non-cloze masks are responsible for a substantial part of the empirical performance of MLMs. 1 https://github.com/tatsu-lab/mlm_ inductive_bias Theories for vector representations. Our goal of understanding how masking can lead to useful inductive biases and linguistic structures is closely related to that of papers studying the theory of word embedding representations (Mikolov et al., 2013;Pennington et al., 2014;Arora et al., 2015). Existing work has drawn a correspondence between word embeddings and low-rank factorization of a pointwise mutual information (PMI) matrix (Levy and Goldberg, 2014) and others have shown that PMI is highly correlated with human semantic similarity judgements (Hashimoto et al., 2016).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Probing Pretrained Representations. Recent work has applied probing methods (Belinkov and Glass, 2019) to analyze what information is captured in the pretrained representations. This line of work shows that pretrained representations encode a diverse range of knowledge (Peters et al., 2018;Tenney et al., 2019;Liu et al., 2019a;Hewitt and Manning, 2019;. While probing provides intriguing evidence of linguistic structures encoded by MLMs, they do not address the goals of this work, which is how the pretraining objective encourages MLMs to extract such structures.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To motivate our subsequent discussions, we perform a controlled study for the case when L MLM ≈  L finetune and analyze how deviations from the ideal mask distribution affect downstream performance. We perform analysis on the Stanford Sentiment Treebank (SST-2; Socher et al., 2013), which requires models to classify short movie reviews into positive or negative sentiment. We append the ground-truth label (as the word positive or negative) to each movie review ( Figure 2). Masking the last word in each review is, by definition, an ideal mask distribution. To study how the deviation from the ideal mask distribution degrades downstream performance, we vary the amount of cloze-like masks during training. We do this by masking out the last word for p% of the time and masking out a random word in the movie review for (100 − p)% of the time, and choose p ∈ {0, 20, 40, 60, 80, 100}. Experimental details. We split the SST-2 training set into two halves, use one for pretraining, and the other for finetuning. For the finetuning data, we do not append the ground-truth label. We pretrain small transformers with L MLM using different masking strategies and finetune them along with a baseline that is not pretrained (NOPRETRAIN). Further details are in Appendix A.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. We observe that while cloze-like masks can lead to successful transfer, even a small modification of the ideal mask distribution deteriorates performance. Figure 3 shows the development set accuracy of seven model variants averaged across ten random trials. We observe as p decreases, the performance of CLOZE-p% degrades. Notably, CLOZE-80% is already worse than CLOZE-100% and CLOZE-20% does not outperform NOPRE-TRAIN by much. We notice that CLOZE-0% in fact degrades finetuning performance, potentially because the pretrained model is over-specialized to the language modeling task (Zhang et al., 2020;Tamkin et al., 2020). While this is a toy example, we observe similar results for actual MLM models across three tasks (Section 5.1), and this motivates us to look for a framework that explains the success of generic masks in practice.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our starting point is the observation that predicting a single feature (x i ) from all others (X \i ) is the core subroutine in the classic Gaussian graphical model structure learning algorithm of Meinshausen and Bühlmann (2006). In this approach, L different Lasso regression models are trained (Tibshirani, 1996) with each model predicting x i from X \i , and the nonzero coefficients of this regression correspond to the conditional dependence structure of the graphical model.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The MLM objective can be interpreted as a nonlinear extension of this approach, much like a classical algorithm that uses conditional mutual information (MI) estimators to recover a graphical model (Anandkumar et al., 2012). Despite the similarity, real world texts are better viewed as models with latent variables (e.g. topics; Blei et al., 2003) and many dependencies across tokens arise due to latent variables, which makes learning the direct dependencies difficult. We show that MLMs implicitly recover the latent variables and can capture the direct dependencies while accounting for the effect of latent variables. Finally, MLMs are only approximations to the true distribution and we show that the MLM objective can induce highquality approximations of conditional MI.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Analysis setup. To better understand MLMs as a way to recover graphical model structures, we show mask-based models can recover latent variables and the direct dependencies among variables in the Gaussian graphical model setting of Meinshausen and Bühlmann (2006). Let X = [x 1 , . . . , x L ] ∈ R L represent an input sequence where each of its coordinates x i represents a token, and Z ∈ R k be a latent variable that controls the sequence generation process. We assume that all coordinates of X are dependent on the latent variable Z, and there are sparse dependencies among the observed variables ( Figure 4). In other words, we can write Z ∼ N(0, Σ ZZ ) and X ∼ N(AZ, Σ XX ). Intuitively, we can imagine that Z represents shared semantic information, e.g. a topic, and Σ XX represents the syntactic dependencies. In this Gaussian graphical model, the MLM is analogous to regressing each coordinate of X from all other coordinates, which we refer to as masked regression.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In the case where there are no latent variables, we can identify the direct dependencies via conditional MI (Anandkumar et al., 2012) because any x i and x j that are disconnected in the graphical model will have zero conditional MI, i.e., I(x i ; x j |X \{i,j} ) = 0. One valuable aspect of MLM is that we can identify direct dependencies even in the presence of latent variables.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study the effectiveness of different masking strategies by comparing to a BERT model without second-stage pretraining (VANILLA). We experiment with three text classification datasets: SST-2 (Socher et al., 2013), Hyperpartisan (Kiesel et al., 2019), and AGNews (Zhang et al., 2015). SST-2 classifies movie reviews by binary sentiment; Hyperpartisan is a binary classification task on whether a news article takes an extreme partisan standpoint; and AGNews classifies news articles into four different topics. On SST-2 and AGNews, we perform the second-stage pretraining on the training inputs (not using the labels). On Hyperpartisan, we use 100k unlabeled news articles that are released with the dataset. For SST-2 and AG-News, we study a low-resource setting and set the number of finetuning examples to be 20. For Hyperpartisan, we use the training set, which has 515 labeled examples. All evaluations are performed by fine-tuning a bert-base-uncased model (See Appendix A for full details).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Approximating Cloze-like Masking. We cannot identify the optimal set of cloze-like masks for an arbitrary downstream task, but these three tasks have associated lexicons which we can use to approximate the cloze-like masks. For SST-2, we take the sentiment lexicon selected by Hu and Liu (2004); for Hyperpartisan, we take the NRC word-emotion association lexicon (Mohammad and Turney, 2013); and for AGNews, we extract topic words by training a logistic regression classifier and AGNews 20-shot taking the top 1k features to be cloze-like masks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Background. One might expect that the statistical dependencies among words are correlated with syntactic dependencies. Indeed, Futrell et al. (2019) show that heads and dependents in dependency parse trees have high pointwise mutual information (PMI) on average. However, previous at-tempts (Carroll and Charniak, 1992;Paskin, 2002) show that unsupervised parsing approaches based on PMI achieve close to random accuracy. Our analysis suggests that MLMs extract a more finegrained notion of statistical dependence (conditional MI) which does not suffer from the existence of latent variables (Proposition 3). We now show that the conditional MI captured by MLMs achieves far better performance, on par with classic unsupervised parsing baselines.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Baselines. We compare conditional MI to PMI as well as conditional PMI, an ablation in which we do not take expectation over possible words. For all statistical dependency based methods (cond. MI, PMI, and cond. PMI), we compute pairwise dependence for each word pair in a sentence and construct a minimum spanning tree on the negative values to generate parse trees. To contextualize our results, we compare against three simple baselines: RANDOM which draws a random tree on the input sentence, LINEARCHAIN which links adjacent words in a sentence, and a classic unsupervised parsing method (Klein and Manning, 2004).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental Setup. We conduct experiments on the English Penn Treebank using the WSJ corpus and convert the annotated constituency parses to Stanford Dependency Formalism (de Marneffe et al., 2006). Following Yang et al. (2020), we evaluate on sentences of length ≤ 10 in the test split, which contains 389 sentences (Appendix B.1 describes the same experiment on longer sentences, which have similar results). We experiment with the bert-base-cased model (more details in Appendix A) and evaluate by the undirected unlabeled attachment score (UUAS).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
28.50 ± 0.73 LINEARCHAIN 54.13 Klein and Manning (2004)  from conditional MI has better quality than LIN-EARCHAIN and the classic method (Klein and Manning, 2004). Unlike conditional MI, PMI only has a close-to-random performance, which is consistent with prior work. We also see that conditional MI outperforms conditional PMI, which is consistent with our theoretical framework that suggests that conditional MI (and not PMI) recovers the graphical model structure.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We also perform a fine-grained analysis by investigating relations where conditional MI differs from LINEARCHAIN. Because the test split is small and conditional MI does not involve any training, we perform this analysis on 5,000 sentences from the training split. Table 2 presents the results and shows that conditional MI does not simply recover the linear chain bias. Meanwhile, we also observe a deviation between conditional MI and dependency grammar on relations like number and cc. This is reasonable because certain aspects of dependency grammar depend on human conventions that do not necessarily have a consensus (Popel et al., 2013). Figure 6 illustrates with an example parse extracted from conditional MI. We observe that conditional MI correctly captures dobj and conj. Knowing the verb, e.g. represents, limits the range of objects that can appear in a sentence so intuitively we expect a high conditional MI between the direct object and the verb. Similarly for phrases like "A and B", we would expect A and B to be statistically dependent. However, conditional MI fails  to capture cc (between apathy and or). Instead, it links or with either which certainly has statistical dependence. This once again suggests that the 'errors' incurred by the conditional PMI method are not simply failures to estimate dependence but natural differences in the definition of dependence.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental details for Section 5.2 We convert the annotated constituency parses using the Stanford CoreNLP package . We compute conditional MI and conditional PMI using the bert-base-cased model and run Gibbs sampling for 2000 steps. BERT's tokenization may split a word into multiple word pieces. We aggregate the dependencies between a word and multiple word pieces by taking the maximum value. We compute the PMI statistics and train the K&M model (Klein and Manning, 2004) on sentences of length ≤ 10 in the WSJ train split (section 2-21). For DMV, we train with the annotated POS tags using a public implementation released by (He et al., 2018). Results are averaged over three runs when applicable.
Transformers learn in-context by gradient descent
In recent years Transformers (TFs; Vaswani et al., 2017) have demonstrated their superiority in numerous benchmarks and various fields of modern machine learning, and have emerged as the de-facto neural network architecture used for modern AI (Dosovitskiy et al., 2021;Yun et al., 2019;Carion et al., 2020;Gulati et al., 2020). It has been hypothesised that their success is due in part to a phenomenon called in-context learning (Brown et al., 2020;Liu et al., 2021): an ability to flexibly adjust their prediction based on additional data given in context (i.e. in the input sequence itself). In-context learning offers a seemingly different approach to few-shot and meta-learning (Brown et al., 2020), but as of today the exact mechanisms of how it works are not fully understood. It is thus of great interest to understand what makes Transformers pay attention to their context, what the mechanisms are, and under which circumstances, they come into play (Chan et al., 2022b;Olsson et al., 2022).
Transformers learn in-context by gradient descent
These findings allow us to connect learning Transformer weights and the concept of meta-learning a learning algorithm (Schmidhuber, 1987;Hinton & Plaut, 1987;Bengio et al., 1990;Chalmers, 1991;Schmidhuber, 1992;Thrun & Pratt, 1998;Hochreiter et al., 2001;Andrychowicz et al., 2016;Ba et al., 2016;Kirsch & Schmidhuber, 2021). In this extensive research field, meta-learning is typically regarded as learning that takes place on various time scales namely fast and slow. The slowly changing parameters control and prepare for fast adaptation reacting to sudden changes in the incoming data by e.g. a context switch. Notably, we build heavily on the concept of fast weights (Schmidhuber, 1992) which has shown to be equivalent to linear self-attention (Schlag et al., 2021) and show how optimized Transformers implement interpretable learning algorithms within their weights.
Transformers learn in-context by gradient descent
Another related meta-learning concept, termed MAML (Finn et al., 2017), aims to meta-learn a deep neural network initialization which allows for fast adaptation on novel tasks. It has been shown that in many circumstances, the solution found can be approximated well when only adapting the output layer i.e. learning a linear model on a meta-learned deep data representations (Finn et al., 2017;Finn & Levine, 2018;Gordon et al., 2019;Lee et al., 2019;Rusu et al., 2019;Raghu et al., 2020;von Oswald et al., 2021). In section 3, we show the equivalence of this framework to in-context learning implemented in a common Transformer block i.e. when combining self-attention layers with a multi-layerperceptron.
Transformers learn in-context by gradient descent
• Meta-learned task-shared learning rates. When training self-attention parameters θ across a family of in-context learning tasks τ , where the data (x τ,i , y τ,i ) follows a certain distribution, the learning rate can be implicitly (meta-)learned such that an optimal loss reduction (averaged over tasks) is achieved given a fixed number of update steps. In our experiments, we find this to be the case. This kind of meta-learning to improve upon plain gradient descent has been leveraged in numerous previous approaches for deep neural networks (Li et al., 2017;Lee & Choi, 2018;Park & Oliva, 2019;Zhao et al., 2020;Flennerhag et al., 2020).
Transformers learn in-context by gradient descent
To provide evidence for this hypothesis, we build on Schlag et al. (2021) that already provide a linear self-attention layer variant with (fast-)inner loop learning by the error-correcting delta rule (Widrow & Hoff, 1960). We diverge from their setting and focus on (in-context) learning where we specifically construct a dataset by considering neighboring elements in the input sequence as input-and target training pairs, see assumptions of Proposition 1. This construction could be realized, for example, due to the model learning to implement a copying layer, see section 4 and proposition 3, and allows us to provide a simple and different construction to Schlag et al. (2021) that solely is built on the standard linear, and approximately softmax, self-attention layer but still implements gradient descent based learning dynamics. We, therefore, are able to explain gradient descent based learning in these standard architectures. Furthermore, we extend this construction based on a single self-attention layer and provide an explanation of how deeper K-layer Transformer models implement principled K-step gradient descent learning, which deviates again from Schlag et al. and allows us to identify that deep Transformers implement GD++, an accelerated version of gradient descent.
Transformers learn in-context by gradient descent
Interestingly, when choosing W 0 = 0, we furthermore see that a single self-attention layer or Transformer block can be regarded as doing nonparametric kernel smoothingŷ = N i=1 y i k(x i , x test ) based on the data given in-context (Nadaraya, 1964;Watson, 1964). Note that we made a particular choice of kernel function here and that this view still holds when m(x j ) = 1 i.e. consider Transformers without MLPs or leverage the well-known view of softmax self-attention layer as a kernel function used to measure similarity between tokens (e.g. Choromanski et al., 2021;Zhang et al., 2021). Thus, implementing one step of gradient descent through a self-attention layer (w/wo softmax nonlinearity) is equivalent to performing kernel smoothing estimation. We however argue that this nonparametric kernel smoothing view of in-context learning is limited, and arises from looking only at a single self-attention layer. When considering deeper Transformer architectures, we see that multiple Transformer blocks can iteratively transform the targets based on multiple steps of gradient descent leading to minimization of a kernelized squared error loss L(W ). One way to obtain a suitable construction is by neglecting MLPs everywhere except in the first Transformer block. We leave the study of the exact mechanics, especially how the Transformer makes use of possibility transforming the targets through the MLPs, and the possibility of iteratively changing the kernel function throughout depth for future study.
Larger language models do in-context learning differently
Language models can perform a range of downstream NLP tasks via in-context learning (ICL), where models are given a few exemplars of input-label pairs as part of the prompt before performing the task on an unseen example (Brown et al., 2020, inter alia). To successfully perform ICL, models can (a) mostly use semantic prior knowledge to predict labels while following the format of in-context exemplars (e.g., seeing "positive sentiment" and "negative sentiment" as labels and performing sentiment analysis using prior knowledge) and/or (b) learn the input-label mappings from the presented exemplars (e.g., finding a pattern that positive reviews should be mapped to one label, and negative reviews should be mapped to a different label). Prior work on which of these factors drives performance is mixed. For instance, although Min et al. (2022b) showed that presenting random ground truth mappings in-context does not substantially affect performance (suggesting that models primarily rely on semantic prior knowledge), other work has shown that transformers in simple settings (without language modeling pretraining) implement learning algorithms such as ridge regression and gradient descent (Akyürek et al., 2023;von Oswald et al., 2022;Dai et al., 2022). Figure 1: An overview of flipped-label ICL and semantically-unrelated label ICL (SUL-ICL), compared with regular ICL. Flipped-label ICL uses flipped targets, forcing the model override semantic priors in order to follow the in-context exemplars. SUL-ICL uses targets that are not semantically related to the task, which means that models must learn input-label mappings in order to perform the task because they can no longer rely on the semantics of natural language targets.
Larger language models do in-context learning differently
Finally, we study the effect of instruction tuning (Min et al., 2022a;Wei et al., 2022a; on ICL abilities (Section 5). We find that instruction-tuned models achieve better performance than pretraining-only models on SUL-ICL settings, which means that instruction tuning increases the model's ability to learn input-label mappings. On the other hand, we also see that instruction-tuned models are more reluctant to follow flipped labels, which means that instruction tuning decreases the model's ability to override semantic priors more than it increases its ability to learn input-label mappings. Overall, our work aims to shed light on the interaction between semantic prior knowledge and input-label mappings while considering the effects of scaling and instruction tuning.
Larger language models do in-context learning differently
We experiment on seven NLP tasks that have been widely used in the literature (Kim, 2014;Wang et al., 2018;2019). These evaluation tasks and an example prompt/target pair are shown in Figure 9 in the Appendix; additional dataset details are described in Appendix A. The seven tasks are: Sentiment Analysis (Socher et al., 2013, SST-2); Subjective/Objective Sentence Classification (Conneau & Kiela, 2018, SUBJ); Question Classification (Li & Roth, 2002, TREC); Duplicated-Question Recognition (Chen et al., 2017;Wang et al., 2018, QQP); Textual Entailment Recognition (Dagan et al., 2006; Table 1: Models used in this paper.
Larger language models do in-context learning differently
We perform experiments on five language model families as shown in Table 1. We use three families of OpenAI language models accessed via the OpenAI API: GPT-3 (Brown et al., 2020), In-structGPT (Ouyang et al., 2022), and Codex (Chen et al., 2021). For GPT-3 models, ada, babbage, curie, and davinci seem to correspond to the following model sizes: 350M, 1.3B, 6.7B, and 175B (Gao et al., 2021). For InstructGPT and Codex, however, it is not publicly known what the sizes of these language models are, but we assume that they are in increasing model scale for some scaling factor.
Larger language models do in-context learning differently
As additional experimental details, we follow the prior literature on in-context learning and use a different set of few-shot exemplars for each inference example (Brown et al., 2020;Chowdhery et al., 2022;Wang et al., 2023, inter alia). By default, we use k = 16 in-context exemplars per class, though we also experiment with varying number of exemplars in Section 4 and Appendix C.2. We also use the "Input/Output" template for prompts shown in Figure 9, with ablations for input format shown in Appendix B.4 and Appendix B.5, and the semantically-unrelated "Foo"/"Bar" targets as shown in Figure 9 (ablations for target type are shown in Appendix B.3). Finally, to reduce inference costs, we use 100 randomly sampled evaluation examples per dataset, as it is more beneficial to experiment with a more-diverse range of datasets and model families than it is to include more evaluation examples per dataset, and our research questions depend more on general behaviors than on small performance deltas (note that all y-axes in our plots go from 0-100).
Larger language models do in-context learning differently
A popular technique for improving the performance of pretrained language models is to finetune them on a collection of NLP tasks phrased as instructions, with few-shot exemplars as part of the finetuning inputs (Min et al., 2022a;Wei et al., 2022a;Longpre et al., 2023). Since instruction tuning uses natural language targets, however, an open question is whether it improves the ability to learn input-label mappings in-context or whether it strengthens the ability to recognize and apply semantic priors, as both would lead to an improvement in performance on standard ICL tasks.
Larger language models do in-context learning differently
There has been a growing body of work on in-context learning that suggests that good performance is primarily driven by semantic priors and other factors such formatting and inducing intermediate token generation. For instance, Min et al. (2022b) showed the surprising result that using random groundtruth labels in exemplars barely hurts performance, suggesting that performance is instead mainly driven by the label space, distribution of input text, and overall format of the sequence. Along the same lines, Madaan & Yazdanbakhsh (2022) and  show that for chain-of-thought prompting (Wei et al., 2022c), logically-incorrect prompts do not hurt performance on multi-step reasoning tasks. On a theoretical level, Xie et al. (2022) provide an explanation of in-context learning in which transformers infer tasks from exemplars because they are trained to infer latent concepts during pretraining, and prior knowledge obtained from pretraining data can then be applied to incontext examples. Finally, Reynolds & McDonell (2021) showed that clever zero-shot prompts can outperform few-shot prompts, which implies that some NLP tasks benefit more from leveraging the model's existing knowledge than from learning about the task from in-context exemplars. In this paper, we do not contest the claim that language models can benefit greatly from semantic prior knowledge-our results instead add nuance to the understanding of ICL by showing that, when semantic prior knowledge is not available, large-enough language models can still do ICL using input-label mappings. Our experiments are consistent with Min et al. (2022b) for models scaling up to davinci, and we show that learning input-label mappings only emerges with larger models (e.g., PaLM-540B, text-davinci-002, and code-davinci-002).
Larger language models do in-context learning differently
Other recent work has suggested to some degree that language models can actually learn input-label mappings from exemplars given in-context, which is a more-attractive ability than using semantic priors because it means that the model would be able to perform a wide range of tasks even if those tasks are not seen in or even contradict pretraining data. For instance, transformers trained from scratch can perform in-context learning on linear-regression datasets with performance that is comparable to the least-squares estimator (Garg et al., 2022), and recent work has shown that transformers can do so by implementing standard learning algorithms such as ridge regression and gradient descent (Akyürek et al., 2023;von Oswald et al., 2022;Dai et al., 2022). In the natural language setting, Webson & Pavlick (2022) showed that language models learn just as fast with irrelevant or misleading prompts during finetuning or prompt-tuning. Our work makes similar claims about the ability for language models to learn tasks via input-label mappings only, though it differs crucially in that we observe frozen pretrained transformers without any additional learning.
Larger language models do in-context learning differently
In this paper we have also focused on the effect of scaling on in-context learning, which relates to a nascent body of work showing that scaling language models leads to qualitatively-different behavior (Ganguli et al., 2022;Wei et al., 2022b;Srivastava et al., 2022). For instance, it has recently been shown that scaling up language models can allow them to perform a variety of challenging tasks that require reasoning (Wei et al., 2022c;Chowdhery et al., 2022;Kojima et al., 2022;. Our experimental findings on the flipped-label ICL setup show that language models can learn input-label mappings even when the input-label mapping contradicts the semantic meaning of the label, demonstrating another type of symbolic reasoning where language models can learn input-label mappings regardless of the actual identity of the labels. Although we have shown that this behavior is emergent with respect to model scale, the investigation of why scaling unlocks such behaviors (Xie et al., 2022;Chan et al., 2022) is still an open question that we leave for future work.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Large pretrained language models (PLMs, Devlin et al., 2019;Peters et al., 2018;Raffel et al., 2020;Liu et al., 2019;Yang et al., 2019;Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual context (Petroni et al., 2019(Petroni et al., , 2020Jiang et al., 2020;Shin et al., 2020;Davison et al., 2019). For example, when conditioned on a long document and a "TL;DR:" token, they can generate a summary of said document, and when provided a partial question ("The theory of relativity was developed by __"), they can generate the correct answer. Perhaps most strikingly, when primed with a context consisting of very few training examples, they produce text classification results that can match those of fully supervised models. This type of few shot setting, is commonly referred to as "In-context Learning" (Brown et al., 2020).
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
A core component of in-context learning is the text-based prompt that serves as the context. Composing a prompt requires: (i) text linearisation using a template; and (ii) training sample concatenation (See Table 1 for an example). It has been established that the structure of the template has a large impact on performance (Shin et al., 2020;Gao et al., 2020;Schick and Schütze, 2020;Jiang et al., 2020). However, to the best of our knowledge, no work has studied the effect of the sample ordering on In-context Learning performance.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We use four different sizes of GPT-2 (Radford et al., 2019) (with 0.1B, 0.3B, 0.8B, and 1.5B parameteers) and two sizes of GPT-3 (Brown et al., 2020) (with 2.7B, and 175B parameters). Due to limited context window size (up to 1024 word-pieces for the GPT-2 series of models), we use a 4-shot setting for all datasets except AGNews and DBPedia. Our experiments are based on the open-source checkpoints of GPT-2 models and access to the OpenAI GPT-3 API. 5 For probing set generation, we restrict the maximum generation length to 128. We also use sampling with a temperature, t, of 2, and we also make use of block n-gram repetitions (Paulus et al., 2018) to encourage diverse generation. We use 24 different permutations for each set of randomly selected training samples and use 5 different sets (except for GPT-3 with 175B parameters, where we only do two sets with 12 different permutation due to the high monetary cost) for each experiment, giving a total of 120 runs. We report the mean and standard deviation of the corresponding evaluation metric over 5 different sets.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
We have shown that few-shot prompts suffer from order sensitivity, in that for the same prompt the order in which samples are provided can make the difference between state-of-the-art and random performance. In our analysis of the problem, we established that it is present across tasks, model sizes, prompt templates, samples, and number of training samples. To alleviate this problem, we introduced a novel probing method that exploits the generative nature of language models to construct an artificial development set. We were able to identity performant permutations using entropy-based statistics over this set, leading to an on average 13% improvement across eleven text classification tasks.    (Pang and Lee, 2005) 2 25.7 Yes CR (Hu and Liu, 2004) 2 22.1 Yes MPQA (Wiebe et al., 2005) 2 3.9 Yes Subj (Pang and Lee, 2004) 2 28.9 Yes TREC (Voorhees and Tice, 2000) 6 11.6 No AGNews (Zhang et al., 2015) 4 53.8 Yes DBPedia (Zhang et al., 2015) 14 65.  
What learning algorithm is in-context learning? Investigations with linear models
One of the most surprising behaviors observed in large neural sequence models is in-context learning (ICL; Brown et al., 2020). When trained appropriately, models can map from sequences of (x, f (x)) pairs to accurate predictions f (x ) on novel inputs x . This behavior occurs both in models trained on collections of few-shot learning problems (Chen et al., 2022;Min et al., 2022) and surprisingly in large language models trained on open-domain text (Brown et al., 2020;Zhang et al., 2022;Chowdhery et al., 2022). ICL requires a model to implicitly construct a map from in-context examples to a predictor without any updates to the model's parameters themselves. How can a neural network with fixed parameters to learn a new function from a new dataset on the fly? This paper investigates the hypothesis that some instances of ICL can be understood as implicit implementation of known learning algorithms: in-context learners encode an implicit, contextdependent model in their hidden activations, and train this model on in-context examples in the course of computing these internal activations. As in recent investigations of empirical properties of ICL (Garg et al., 2022;Xie et al., 2022), we study the behavior of transformer-based predictors (Vaswani et al., 2017) on a restricted class of learning problems, here linear regression. Unlike in past work, our goal is not to understand what functions ICL can learn, but how it learns these functions: the specific inductive biases and algorithmic properties of transformer-based ICL.
What learning algorithm is in-context learning? Investigations with linear models
Training a machine learning model involves many decisions, including the choice of model architecture, loss function and learning rule. Since the earliest days of the field, research has sought to understand whether these modeling decisions can be automated using the tools of machine learning itself. Such "meta-learning" approaches typically treat learning as a bi-level optimization problem (Schmidhuber et al., 1996;Andrychowicz et al., 2016;Finn et al., 2017): they define "inner" and "outer" models and learning procedures, then train an outer model to set parameters for an inner procedure (e.g. initializer or step size) to maximize inner model performance across tasks.
What learning algorithm is in-context learning? Investigations with linear models
Recently, a more flexible family of approaches has gained popularity. In in-context learning (ICL), meta-learning is reduced to ordinary supervised learning: a large sequence model (typically implemented as a transformer network) is trained to map from sequences [x 1 , f (x 1 ), x 2 , f (x 2 ), ..., x n ] to predictions f (x n ) (Brown et al., 2020;Olsson et al., 2022;Laskin et al., 2022;Kirsch & Schmidhuber, 2021). ICL does not specify an explicit inner learning procedure; instead, this procedure exists only implicitly through the parameters of the sequence model. ICL has shown impressive results on synthetic tasks and naturalistic language and vision problems (Garg et al., 2022;Min et al., 2022;Zhou et al., 2022;Hollmann et al., 2022).
What learning algorithm is in-context learning? Investigations with linear models
We study transformer models directly trained on an ICL objective. (Some past work has found that ICL also "emerges" in models trained on general text datasets; Brown et al., 2020.) To train a transformer T with parameters θ to perform ICL, we first define a class of functions F, a distribution p(f ) supported on F, a distribution p(x) over the domain of functions in F, and a loss function L. We then choose θ to optimize the auto-regressive objective, where the resulting T θ is an in-context learner:
An Explanation of In-context Learning as Implicit Bayesian Inference
Large language models (LMs) such as GPT-3 (Brown et al., 2020, Lieber et al., 2021, Radford et al., 2019, Wang and Komatsuzaki, 2021 are pretrained on massive text corpora to predict the next word given previous words. They demonstrate the surprising ability to do in-context learning, where an LM "learns" to do a task simply by conditioning on a prompt containing input-output pairs, achieving SOTA results on LAMBADA (Paperno et al., 2016) and TriviaQA (Joshi et al., 2017) tasks (18% and 3% over previous SOTA (Brown et al., 2020)). For example, consider the task of predicting nationalities from names. A prompt (Figure 1) is constructed by concatenating independent "training" examples (e.g., "Albert Einstein was German") followed by a "test example" ("Marie Curie was"). Conditioning on this prompt, GPT-3 places the largest probability on the correct output p("Polish" | "Albert Einstein was German \n Mahatma Gandhi was Indian \n Marie Curie was") 1 The code, data, and experiments are located on GitHub and CodaLab. Figure 1: In-context learning can emerge from modeling long-range coherence in the pretraining data. During pretraining, the language model (LM) implicitly learns to infer a latent concept (e.g., wiki bios, which typically transition between name (Albert Einstein) → nationality (German) → occupation (physicist) → ...) shared across sentences in a document. Although prompts are unnatural sequences that concatenate independent examples, in-context learning occurs if the LM can still infer the shared concept across examples to do the task (name → nationality, which is part of wiki bios). by inferring the task from examples. Intruigingly, GPT-3 was not explicitly pretrained to learn from examples, and the distribution of prompts (which concatenate independent examples) is quite different from natural language. Our understanding of in-context learning is limited since (i) real pretraining data is messy and (ii) in-context learning has so far required large-scale datasets and models. In this paper, we introduce a simple pretraining distribution where in-context learning emerges. To generate a document, we first draw a latent concept θ, which parameterizes the transitions of a Hidden Markov Model (HMM) (Baum and Petrie, 1966), then sample a sequence of tokens from the HMM (Figure 9). This latent variable structure is common in topic models such as LDA (Blei et al., 2003, Gruber et al., 2007. During pretraining, the LM must infer the latent concept across multiple sentences to generate coherent continuations. When conditioning on a prompt, in-context learning occurs when the LM also infers a shared prompt concept across examples to make a prediction. We assume the LM fits the pretraining distribution p exactly with enough data and expressivity, so that the question of in-context learning becomes characterizing the conditional distribution of completions given prompts p(output|prompt) under the pretraining distribution, where the prompt is generated from a different distribution p prompt . This conditional distribution, which is the posterior predictive distribution, marginalizes out the latent concepts: p(output|prompt) = concept p(output|concept, prompt)p(concept|prompt)d(concept).
An Explanation of In-context Learning as Implicit Bayesian Inference
If p(concept|prompt) concentrates on the prompt concept with more examples, then the LM learns via marginalization by "selecting" the prompt concept. Thus, in-context learning can be viewed as the LM implicitly performing Bayesian inference. The main challenge is that prompts are sampled from a different distribution than the pretraining distribution. The canonical Bayesian asymptotic tool is the Bernstein-von Mises theorem (Gunst and Shcherbakova, 2008, Kleijn and van der Vaart, 2012, van der Vaart, 1998, which asserts (under regularity conditions) that the posterior distribution of a latent variable concentrates on the maximum likelihood estimate. However, Bernstein-von Mises typically assumes observations are independent and/or drawn from the same distribution as the model, both of which are not satisfied. We prove that despite the distribution mismatch, the asymptotic prediction error of in-context learning is optimal when the signal about the latent concept in each prompt example is larger than the error due to the distribution mismatch. Additionally, we prove that the in-context learning error decreases with the length of each example-thus, information in the inputs, not just the input-output mapping, can be useful for in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
Our first result relies on a continuity assumption between the concept parameter and its corresponding output distribution. Our assumption is based on prior works (Kleijn and van der Vaart, 2012), where the KL divergence is assumed to have a 2nd-order Taylor expansion.
An Explanation of In-context Learning as Implicit Bayesian Inference
Main result. We train GPT-2-based Transformers (Radford et al., 2019) and LSTMs on three versions of the GINC dataset with vocabulary sizes 50, 100, and 150, then evaluate the in-context accuracy (see Appendix F.2, F.3). We average all results over 5 pretraining runs. Figure 3 shows that for both Transformer and LSTMs, in-context accuracy improves as the number of prompt examples n and the example length k increase, verifying our theory.
An Explanation of In-context Learning as Implicit Bayesian Inference
(2020). Table 6 shows that even though larger Transformers may have the same pretraining loss (e.g., 12 and 16 layer Transformers both get 1.33 validation loss for vocab size 50), the in-context accuracy still improves (81% to 85% from 12 to 16 layers), suggesting that larger models can improve in-context learning beyond improving pretraining perplexity. This may be related to phenomena from overparameterization and overtraining (Power et al., 2021, Zhang et al., 2017. Finally, the model architecture also plays a role -LSTMs consistently outperform Transformers on GINC despite having fewer parameters, perhaps due to the similarity between HMMs and LSTMs. We leave analysis of the effect of model scaling and model architecture as open questions.
An Explanation of In-context Learning as Implicit Bayesian Inference
Learning via Bayesian inference and extrapolation. The canonical Bernstein-von Mises theorem (van der Vaart, 1998) does not apply for in-context learning since the prompt examples are not independent under the pretraining distribution. Gunst and Shcherbakova (2008) show a Bernsteinvon Mises-type result for observations from an HMM, but do not handle observations from a dif-ferent distribution. Future directions include more precise asymptotic results about the posterior distribution and results under misspecification/extrapolation (Kleijn and van der Vaart, 2012). A possible avenue for extrapolation to some types of unseen concepts is to factorize the latent concept into semantics and syntax. While the pretraining data may contain only some semantics-syntax pairs, the language model could generalize to unseen pairs if it learns generalizable syntactical operations such as copying or reordering.
An Explanation of In-context Learning as Implicit Bayesian Inference
Topic models and HMMs. Topic models such as LDA (Blei et al., 2003) also have document-level latent variables, but learning is typically relies on algorithms such as EM (Dempster et al., 1977), variational inference (Jordan et al., 1999), or MCMC (Hastings, 1970, Metropolis et al., 1953. We focus on learning as a natural result of Bayesian inference without an explicit inference algorithm. Wei et al. (2021a) also use an HMM model in their pretraining analysis. However, they analyze how pre-trained representations learned with masked LMs (Clark et al., 2020, Devlin et al., 2019, Lewis et al., 2020, Liu et al., 2019 can improve optimization-based downstream learning (Lester et al., 2021, Li andLiang, 2021) rather than in-context learning.
An Explanation of In-context Learning as Implicit Bayesian Inference
Bridging the mismatch between pretraining and prompting. Prior works support our theoretical intuitions that reducing the prompt distribution mismatch would improve in-context learning. Finetuning LMs on text with a prompting format improves its zero-shot performance (Sanh et al., 2021, Wei et al., 2021b  Meta-learning. Meta-learning methods can also train a sequence model to learn from examples (Ravi and Larochelle, 2017). However, meta-learning models are trained to learn, while incontext learning emerges from LM pretraining.
An Explanation of In-context Learning as Implicit Bayesian Inference
Our Transformer models are based on the GPT-2 architectures with 4, 12, and 16 layers respectively, with 12 attention heads, 768 dimensional embeddings, residual/embedding/attention dropout set to 0.1, and a context window of 1024. Other than the number of layers, the other parameters are the default settings from the HuggingFace library (Wolf et al., 2019). We train for 5 epochs using the AdamW optimizer (Kingma andBa, 2015, Loshchilov andHutter, 2019) with a batch size of 8 and a linear learning rate schedule (with 1000 step warmup) up to a learning rate of 8e-4 for the 4 layer and 12 layer model, while for the 16 layer model we start with a constant learning rate of 8e-4 and reduce by a factor of 0.25 whenever the best validation loss does not improve. We tried both learning rate strategies for all models and take the most consistent. We tuned these models so that the training loss curves between seeds have smaller variability between the runs in terms of the curve shape and when the loss decreases -we found that this is an important indication of stable results. The models took 50 minutes, 2 hours, 3 hours to train respectively. The hardware was mainly Titan Xp GPUs, trained and evaluated using 16-bit precision. All the results are reported with 5 pretraining runs (5 different seeds).
An Explanation of In-context Learning as Implicit Bayesian Inference
We conduct an additional experiment which shows that longer examples improve in-context learning in GPT-3 on the LAMBADA (Paperno et al., 2016) completion task.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Embedding methods: Prior to language models, large text corpora like Wikipedia [Merity et al., 2016] were used to learn low dimensional embeddings for words [Mikolov et al., 2013b,a, Pennington et al., 2014 and subsequently for sentences [Kiros et al., 2015, Arora et al., 2017, Pagliardini et al., 2018, Logeswaran and Lee, 2018 for downstream task usage. These methods were inspired by the distributional hypothesis [Firth, 1957, Harris, 1954, which posits that meaning of text is determined in part by the surrounding context. Recent methods like BERT [Devlin et al., 2018] and variants [Lan et al., 2019, Yang et al., 2019, Liu et al., 2019 learn from auxiliary tasks, such as sentence completion, and are among the top performers on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Embedding methods: Prior to language models, large text corpora like Wikipedia [Merity et al., 2016] were used to learn low dimensional embeddings for words [Mikolov et al., 2013b,a, Pennington et al., 2014 and subsequently for sentences [Kiros et al., 2015, Arora et al., 2017, Pagliardini et al., 2018, Logeswaran and Lee, 2018 for downstream task usage. These methods were inspired by the distributional hypothesis [Firth, 1957, Harris, 1954, which posits that meaning of text is determined in part by the surrounding context. Recent methods like BERT [Devlin et al., 2018] and variants [Lan et al., 2019, Yang et al., 2019, Liu et al., 2019 learn from auxiliary tasks, such as sentence completion, and are among the top performers on downstream tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
3 Using language models for classification tasks Sections 2.1 and 2.2 show that both unconstrained and softmax language models aim to learn either p * ·|s exactly or a projection Φp * ·|s . Thus we ask: why should learning p * ·|s be useful for solving downstream tasks? To answer this, we use a thought experiment where we have access to an oracle that provides p * ·|s for any s. We then demonstrate how to solve a classification task through reframing the task as a sentence completion problem and using p * ·|s to get completions to predict the label. Classification task as sentence completion: For an input s in a task, we propose to predict the label by using p * ·|s to predict words that could follow s. In particular, for a movie review sentiment analysis task, we can use the oracle to compare probabilities of ":)" and ":(" after an input and predict sentiment based on which is higher. However, p * ·|s will likely place higher probability on words that start sentences, like "The", instead of discriminative words useful for the task. To allow a larger set of grammatically correct completions, we can additionally append a prompt like "This movie is " at the end of a review and query probabilities of indicative adjectives like good, bad, interesting, boring etc. This approach also works for the AG news dataset [Zhang et al., 2015] which contains news articles from 4 categories: world, science/technology, sports, and business, where a prompt like "This article is about " can be added. We further theoretically motivate the usage of prompts in the discussion of Theorem 4.1, and we experimentally verify in Section 6 that probabilities of a small subset of words are sufficient to solve these tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
3 Using language models for classification tasks Sections 2.1 and 2.2 show that both unconstrained and softmax language models aim to learn either p * ·|s exactly or a projection Φp * ·|s . Thus we ask: why should learning p * ·|s be useful for solving downstream tasks? To answer this, we use a thought experiment where we have access to an oracle that provides p * ·|s for any s. We then demonstrate how to solve a classification task through reframing the task as a sentence completion problem and using p * ·|s to get completions to predict the label. Classification task as sentence completion: For an input s in a task, we propose to predict the label by using p * ·|s to predict words that could follow s. In particular, for a movie review sentiment analysis task, we can use the oracle to compare probabilities of ":)" and ":(" after an input and predict sentiment based on which is higher. However, p * ·|s will likely place higher probability on words that start sentences, like "The", instead of discriminative words useful for the task. To allow a larger set of grammatically correct completions, we can additionally append a prompt like "This movie is " at the end of a review and query probabilities of indicative adjectives like good, bad, interesting, boring etc. This approach also works for the AG news dataset [Zhang et al., 2015] which contains news articles from 4 categories: world, science/technology, sports, and business, where a prompt like "This article is about " can be added. We further theoretically motivate the usage of prompts in the discussion of Theorem 4.1, and we experimentally verify in Section 6 that probabilities of a small subset of words are sufficient to solve these tasks.
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Tasks using linear function of p ·|s : We validate our claims from Section 3 that classification tasks can be solved by linear functions of p ·|s . Table 1 demonstrates that on the Stanford Sentiment Treebank (SST) [Socher et al., 2013] and AG News [Zhang et al., 2015] tasks we can use the conditional probabilities p ·|s = p f (s) from GPT-2 [Radford et al., 2019] of just 20 task-relevant tokens (see Section E.1) to solve tasks. Even just one token per class yields non-trivial performance. Furthermore, we validate the complete-the-sentence intuition in Section 3 by using the probabilities after adding a task specific prompt and observing improved performance. g f,Φ and f are good features: We validate Theorem 4.2 by verifying that the conditional mean features g f,Φ (s) = Φp f (s) also linearly solve downstream tasks fairly well. This performance is comparable to, but always worse than f (s), as seen in columns 2 and 3 of Table 1. We again find that adding a prompt improves performance. Section E.1 has results for a wider range of classification tasks. Evidence for Assumption 4.1 is provided by learning a quadratic function to fit the log partition function of features from pretrained GPT-2 model (see Section E.3). Figure 1 demonstrates that the fit holds for its training and unseen data (e.g., WebText [Radford et al., 2019]).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Tasks using linear function of p ·|s : We validate our claims from Section 3 that classification tasks can be solved by linear functions of p ·|s . Table 1 demonstrates that on the Stanford Sentiment Treebank (SST) [Socher et al., 2013] and AG News [Zhang et al., 2015] tasks we can use the conditional probabilities p ·|s = p f (s) from GPT-2 [Radford et al., 2019] of just 20 task-relevant tokens (see Section E.1) to solve tasks. Even just one token per class yields non-trivial performance. Furthermore, we validate the complete-the-sentence intuition in Section 3 by using the probabilities after adding a task specific prompt and observing improved performance. g f,Φ and f are good features: We validate Theorem 4.2 by verifying that the conditional mean features g f,Φ (s) = Φp f (s) also linearly solve downstream tasks fairly well. This performance is comparable to, but always worse than f (s), as seen in columns 2 and 3 of Table 1. We again find that adding a prompt improves performance. Section E.1 has results for a wider range of classification tasks. Evidence for Assumption 4.1 is provided by learning a quadratic function to fit the log partition function of features from pretrained GPT-2 model (see Section E.3). Figure 1 demonstrates that the fit holds for its training and unseen data (e.g., WebText [Radford et al., 2019]).
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Assumption 4.1, we assert that there is a quadratic fit for the log partition function, which allows us to show in Lemma 4.3 that a linear relation holds between f * and Φp f * . We validate these theoretical findings by fitting a quadratic function to the log partition function for a subset of embeddings from Table 3: GPT-2 performance without fine-tuning on downstream task test sets with k classes. We provide the performance of bag-of-n-grams as an approximate baseline for these tasks. DBPedia and Yahoo performances were reported in Zhang et al. [2015], and the other tasks were reported in Khodak et al. [2018]. We also include results from Sentiment Neuron [Radford et al., 2017] for the sentiment-related classification tasks: IMDb, CR, and MPQA. Furthermore, we include results from using BERT [Devlin et al., 2018] without fine-tuning, where we use the features produced for the first position as input to the linear classifier. An asterisk indicates we add a standard sentiment prompt "The sentiment is " to each input. We also tested the performance of the conditional probability distribution over "positive" and "negative" as well as " :)" and " :(" on the sentiment-related tasks with and without the prompt. the IMDb, SST, and AG News datasets ( Figure 1). Here, we describe how we learned A, b and c. To ensure A is symmetric and positive semi-definite as required, we parametrize A = U U T . Let µ θ = Φp θ . We minimize the following objective function:
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
In Assumption 4.1, we assert that there is a quadratic fit for the log partition function, which allows us to show in Lemma 4.3 that a linear relation holds between f * and Φp f * . We validate these theoretical findings by fitting a quadratic function to the log partition function for a subset of embeddings from Table 3: GPT-2 performance without fine-tuning on downstream task test sets with k classes. We provide the performance of bag-of-n-grams as an approximate baseline for these tasks. DBPedia and Yahoo performances were reported in Zhang et al. [2015], and the other tasks were reported in Khodak et al. [2018]. We also include results from Sentiment Neuron [Radford et al., 2017] for the sentiment-related classification tasks: IMDb, CR, and MPQA. Furthermore, we include results from using BERT [Devlin et al., 2018] without fine-tuning, where we use the features produced for the first position as input to the linear classifier. An asterisk indicates we add a standard sentiment prompt "The sentiment is " to each input. We also tested the performance of the conditional probability distribution over "positive" and "negative" as well as " :)" and " :(" on the sentiment-related tasks with and without the prompt. the IMDb, SST, and AG News datasets ( Figure 1). Here, we describe how we learned A, b and c. To ensure A is symmetric and positive semi-definite as required, we parametrize A = U U T . Let µ θ = Φp θ . We minimize the following objective function:
What In-Context Learning"Learns"In-Context: Disentangling Task Recognition and Task Learning
On the other hand, Akyürek et al. (2023);von Oswald et al. (2022) construct theories that Transformer-based models may perform implicit gradient descent to update an "inner-model", and Dai et al. (2023) demonstrate similarities between in-context learning and explicit fine-tuning through a series of metrics on real-world datasets. Such hypotheses assume the correct input-output mappings are important and ICL actually performs implicit learning over demonstrations.