How to measure inductive biases of learning algorithms in NLP?
Measuring inductive biases. Our work builds
on existing research on measuring the inductive bi-
ases of learning algorithms in machine learning and
NLP. Dasgupta et al. (2022) introduce a method-
ology for measuring feature bias as well as rule-
vs. exemplar-based generalization, and Chan et al.
(2022) apply this approach to compare rule- vs.
exemplar-based learning in ICL. We use a simi-
lar framing as Dasgupta et al. (2022), but focus
on feature bias. In NLP, another line of work has
studied the inductive biases of neural networks in
the context of the poverty of the stimulus argu-
ment (Chomsky, 1980). These studies evaluate
whether neural networks generalize in a manner
consistent with structural or superficial linguistic
rules (McCoy et al., 2018, 2020). Several studies have found that models such as BERT acquire a preference for structural generalizations from large-
scale masked language model pretraining (Warstadt
and Bowman, 2020; Warstadt et al., 2020; Zhang
and Hashimoto, 2021; Mueller et al., 2022). We fol-
low a similar poverty-of-the-stimulus experimental
setup but focus on in-context learning and on fea-
tures arising in common NLP tasks. Lovering
et al. (2021) explore whether it is possible to pre-
dict the inductive biases of pre-trained models and
show that models tend to generalize on the basis
of features that are more “extractable”, measured
using probing techniques (Voita and Titov, 2020),
but it is not straightforward to extend the notion
of extrability to in-context learning. Tamkin et al.
(2023) also study how LLMs generalize to ambigu-
ous classification tasks, but focus on ambiguous
instructions and use template-generated datasets.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
One reason pretraining on self-supervised linguistic tasks is effective is that it teaches models features that are helpful for language understanding. However, we want pretrained models to learn not only to represent linguistic features, but also to use those features preferentially during fine-turning. With this goal in mind, we introduce a new English-language diagnostic set called MSGS (the Mixed Signals Generalization Set), which consists of 20 ambiguous binary classification tasks that we use to test whether a pretrained model prefers linguistic or surface generalizations during fine-tuning. We pretrain RoBERTa models from scratch on quantities of data ranging from 1M to 1B words and compare their performance on MSGS to the publicly available RoBERTa-base. We find that models can learn to represent linguistic features with little pretraining data, but require far more data to learn to prefer linguistic generalizations over surface ones. Eventually, with about 30B words of pretraining data, RoBERTa-base does demonstrate a linguistic bias with some regularity. We conclude that while self-supervised pretraining is an effective way to learn helpful inductive biases, there is likely room to improve the rate at which models learn which features matter.
[{"first":"Alex","last":"Warstadt","middle":[]},{"first":"Yian","last":"Zhang","middle":[]},{"first":"Haau-Sing","last":"Li","middle":[]},{"first":"Haokun","last":"Liu","middle":[]},{"first":"Samuel","last":"Bowman","middle":["R."]}]
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Most current NLP systems are based on a pre-train-then-fine-tune paradigm, in which a large neural network is first trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then finetuned for a specific task of interest. Recent work attempts to understand why this recipe works and explain when it fails. Currently, such analyses have produced two sets of apparently-contradictory results. Work that analyzes the representations that result from pre-training (via “probing classifiers”) finds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after fine-tuning (via “challenge sets”) indicates that decisions are often not based on such structure but rather on spurious heuristics specific to the training set. In this work, we test the hypothesis that the extent to which a feature influences a model’s decisions can be predicted using a combination of two factors: The feature’s extractability after pre-training (measured using information-theoretic probing techniques), and the evidence available during finetuning (defined as the feature’s co-occurrence rate with the label). In experiments with both synthetic and naturalistic data, we find strong evidence (statistically significant correlations) supporting this hypothesis.
[{"first":"Charles","last":"Lovering","middle":[]},{"first":"Rohan","last":"Jha","middle":[]},{"first":"Tal","last":"Linzen","middle":[]},{"first":"Ellie","last":"Pavlick","middle":[]}]
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applications, often fail to generalize in a hierarchy-sensitive manner when performing syntactic transformations—for example, transforming declarative sentences into questions. However, syntactic evaluations of seq2seq models have only observed models that were not pre-trained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syntactic capabilities of seq2seq models may have been greatly understated. We address this gap using the pre-trained seq2seq models T5 and BART, as well as their multilingual variants mT5 and mBART. We evaluate whether they generalize hierarchically on two transformations in two languages: question formation and passivization in English and German. We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not. This result presents evidence for the learnability of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization, though only after exposure to much more language data than human learners receive.
[{"first":"Aaron","last":"Mueller","middle":[]},{"first":"Robert","last":"Frank","middle":[]},{"first":"Tal","last":"Linzen","middle":[]},{"first":"Luheng","last":"Wang","middle":[]},{"first":"Sebastian","last":"Schuster","middle":[]}]
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).
[{"first":"Tianyi","last":"Zhang","middle":[]},{"first":"Tatsunori","last":"Hashimoto","middle":[]}]
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Syntactic rules in natural language typically need to make reference to hierarchical sentence structure. However, the simple examples that language learners receive are often equally compatible with linear rules. Children consistently ignore these linear explanations and settle instead on the correct hierarchical one. This fact has motivated the proposal that the learner's hypothesis space is constrained to include only hierarchical rules. We examine this proposal using recurrent neural networks (RNNs), which are not constrained in such a way. We simulate the acquisition of question formation, a hierarchical transformation, in a fragment of English. We find that some RNN architectures tend to learn the hierarchical rule, suggesting that hierarchical cues within the language, combined with the implicit architectural biases inherent in certain RNNs, may be sufficient to induce hierarchical generalizations. The likelihood of acquiring the hierarchical generalization increased when the language included an additional cue to hierarchy in the form of subject-verb agreement, underscoring the role of cues to hierarchy in the learner's input.
[{"first":"R.","last":"McCoy","middle":["Thomas"]},{"first":"Robert","last":"Frank","middle":[]},{"first":"Tal","last":"Linzen","middle":[]}]
Can neural networks acquire a structural bias from raw linguistic data?
We evaluate whether BERT, a widely used neural network for sentence processing, acquires an inductive bias towards forming structural generalizations through pretraining on raw data. We conduct four experiments testing its preference for structural vs. linear generalizations in different structure-dependent phenomena. We find that BERT makes a structural generalization in 3 out of 4 empirical domains---subject-auxiliary inversion, reflexive binding, and verb tense detection in embedded clauses---but makes a linear generalization when tested on NPI licensing. We argue that these results are the strongest evidence so far from artificial learners supporting the proposition that a structural bias can be acquired from raw data. If this conclusion is correct, it is tentative evidence that some linguistic universals can be acquired by learners without innate biases. However, the precise implications for human language acquisition are unclear, as humans learn language from significantly less data than BERT.
[{"first":"Alex","last":"Warstadt","middle":[]},{"first":"Samuel","last":"Bowman","middle":["R."]}]
Transformers generalize differently from information stored in context vs in weights
Transformer models can use two fundamentally different kinds of information: information stored in weights during training, and information provided ``in-context'' at inference time. In this work, we show that transformers exhibit different inductive biases in how they represent and generalize from the information in these two sources. In particular, we characterize whether they generalize via parsimonious rules (rule-based generalization) or via direct comparison with observed examples (exemplar-based generalization). This is of important practical consequence, as it informs whether to encode information in weights or in context, depending on how we want models to use that information. In transformers trained on controlled stimuli, we find that generalization from weights is more rule-based whereas generalization from context is largely exemplar-based. In contrast, we find that in transformers pre-trained on natural language, in-context learning is significantly rule-based, with larger models showing more rule-basedness. We hypothesise that rule-based generalization from in-context information might be an emergent consequence of large-scale training on language, which has sparse rule-like structure. Using controlled stimuli, we verify that transformers pretrained on data containing sparse rule-like structure exhibit more rule-based generalization.
[{"first":"Stephanie","last":"Chan","middle":["C.Y."]},{"first":"Ishita","last":"Dasgupta","middle":[]},{"first":"Junkyung","last":"Kim","middle":[]},{"first":"Dharshan","last":"Kumaran","middle":[]},{"first":"Andrew","last":"Lampinen","middle":["K."]},{"first":"Felix","last":"Hill","middle":[]}]
Information-Theoretic Probing with Minimum Description Length
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates"the amount of effort"needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.
[{"first":"Elena","last":"Voita","middle":[]},{"first":"Ivan","last":"Titov","middle":[]}]
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Learners that are exposed to the same training data might generalize differently due to differing inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order. All architectural factors that we investigated qualitatively affected how models generalized, including factors with no clear connection to hierarchical structure. For example, LSTMs and GRUs displayed qualitatively different inductive biases. However, the only factor that consistently contributed a hierarchical bias across tasks was the use of a tree-structured model rather than a model with sequential recurrence, suggesting that human-like syntactic generalization requires architectural syntactic structure.
[{"first":"R. Thomas","last":"McCoy","middle":[]},{"first":"Robert","last":"Frank","middle":[]},{"first":"Tal","last":"Linzen","middle":[]}]
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Self-supervised pretraining through language modeling on massive datasets has revolutionized NLP. One reason this method works is that pretraining shapes a model's hypothesis space, giving it inductive biases that help it learn linguistic tasks (Howard and Ruder, 2018). Numerous probing studies have provided support for this idea by showing that language models learn representations that encode linguistic features (Gulordava et al., 2019;Tenney et al., 2019;Hewitt and Manning, 2019).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
However, feature learning is just the first step to acquiring helpful inductive biases. Models must The rumor that a CEO is losing spread.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A boy who is hugging the cat sneezed.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A guest said that the boat is sinking.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A rumor that the CEO lost is spreading. Label=0,Prediction=0 The boy who hugged a cat is sneezing. 1 2 3 4 5 6 7 8
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A rumor that the CEO lost is spreading.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Test behavior: Linguistic bias observed Linguistic Generalization:
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Is the main verb in the "-ing" form?
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Surface Generalization:
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Does the word "the" precede "a"?
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
A boy who hugged the cat is sneezing. 1 2 3 4 5 6 7 8
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The boy who hugged a cat is sneezing.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The guest is saying that a boat sinks.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Label=0 Label=1,Prediction=1
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The rumor that a CEO is losing spread.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Figure 1: Example of an ambiguous experiment (without inoculation). A model is trained on ambiguous data whose labels are consistent with either a linguistic or a surface generalization, and tested on disambiguating data whose labels support only the linguistic generalization. Light green and dark red shading represents data or features associated with the positive and negative labels/predictions, respectively. also be able to learn which features matter. The NLU datasets these models are often fine-tuned on are ambiguous and contain artifacts, and often support multiple possible generalizations. Neural networks are not mind readers: Models that have been shown to represent linguistic features sometimes fail to use them during fine-tuning on NLU tasks, instead adopting shallow surface generalizations (Jia and Liang, 2017;. To this end, recent work in probing pretrained models advocates for shifting the focus of study away from whether they represent linguistic features and in favor of whether they learn useful representations of those features (Voita and Titov, 2020;Pimentel et al., 2020;Elazar et al., 2020).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We investigate how RoBERTa (Liu et al., 2019b) acquires language-specific inductive biases during self-supervised pretraining. We track separately how RoBERTa's representation of linguistic features and its preferences for linguistic generalizations over surface generalizations change as the amount of pretraining data increases. We pretrain RoBERTa from scratch on datasets ranging from 1M to 1B words and evaluate these models alongside RoBERTa BASE in a series of experiments to probe the inductive biases of a pretrained model at the time of fine-tuning on a downstream task.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We probe these models in three kinds of experiments: First, we conduct control experiments where we fine-tune models on unambiguous binary classification tasks to test whether they learn to represent simple linguistic and surface features. Second, we conduct ambiguous experiments following the poverty of the stimulus design (Wilson, 2006), as illustrated in Figure 1. In these experiments, we fine-tune a pretrained model on an ambiguous binary classification task in which the training set is consistent with both a linguistic generalization and a surface one. We then test the classifier on disambiguating data to reveal which generalization the model adopted, and by extension its preference among the two features. Third, we conduct inoculation experiments (following Liu et al., 2019a) to test how hard it is to sway a model with a surface bias to adopt a linguistic generalization. We do this by introducing small amounts of disambiguating data into an otherwise ambiguous training set. We automatically generate data for all these tasks, and call the resulting dataset MSGS (Mixed Signals Generalization Set), pronounced "messages".
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The results show that RoBERTa acquires a stronger linguistic bias as pretraining increases. RoBERTa BASE has the strongest linguistic bias, and requires little to no inoculating data to reliably make the linguistic generalization. In general, models with more pretraining data can generally be induced to adopt linguistic generalizations with less inoculating data. We also find a large gap between the amount of pretraining data that RoBERTa needs to learn the linguistic features necessary to generalize out-of-domain and the amount it needs to learns that it should prefer those features when generalizing. The control experiments on unambiguous data reveal that models with little pretraining do actually represent the linguistic features, but nonetheless show a strong surface bias. In other words, the main contribution of pretraining to linguistic bias learning is devoted not to extracting features, but to learning which features matter.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We conclude that helpful inductive biases can be learned through pretraining, but current models require abundant data to do so. The implications of this conclusion point in two directions: First, we can probably continue to pretrain on increasingly massive training sets to improve on the generalization and few-shot learning abilities of models like T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020). Second, since models learn useful features early, there is hope that future advances could accelerate by reducing the amount of data needed to learn which features matter. To aid in this effort, we release the MSGS dataset, our pretrained RoBERTas, and all our code: https://github.com/nyu-mll/msgs.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Background: Learning Inductive Bias Any finite set of training examples shown to a learning algorithm like a neural network is consistent with infinitely many generalizable decision functions. Inductive biases are a learner's preferences among these functions. An inductive bias can eliminate certain possible functions altogether, or result in a preference for some over others (Haussler, 1988). For instance, an RNN classifier is capable of representing any function, but prefers ones that focus mostly on local relationships within the input sequence (Dhingra et al., 2018;Ravfogel et al., 2019).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Some recent work seeks to design neural architectures that build in desirable inductive biases (Dyer et al., 2016;Battaglia et al., 2018), or compares the immutable biases of different architectures Hu et al., 2020). However, inductive biases can also be learned by biological (Harlow, 1949) and artificial systems alike (Lake et al., 2017). In the language model finetuning paradigm proposed by Howard and Ruder (2018) and popularized by models such as BERT (Devlin et al., 2019), a pretrained neural network plays the role of the learner. Pretraining adjusts a model's weights so that it will navigate the hypothesis space during training on a downstream task more effectively than a randomly initialized model.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
There is a difference between learning to extract a linguistic feature and acquiring a bias towards using it when generalizing. There is ample evidence that BERT encodes features such as syntactic category and constituency (Tenney et al., 2019;Clark et al., 2019;Hewitt and Manning, 2019). The acquisition of linguistic features is a prerequisite for a linguistic bias. However, these findings do not tell us if the model will make use of these features to form generalizations during target task training, or if it will fall back on surface features that account for most of the data.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Methods: Measuring Inductive Bias We conduct three kinds of experiments to probe a model's preference for linguistic or surface generalizations: unambiguous control experiments, fully ambiguous experiments, and partially ambiguous inoculation experiments. Figure 1 gives an overview of the ambiguous experiment design.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
First, it only makes sense to compare a model's preference between two features if it actually represents both features. This is the goal behind control experiments, in which we fine-tune RoBERTa to classify sentences based on a single linguistic or surface feature in a totally unambiguous setting.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Second, we conduct ambiguous experiments on models that pass the controls. We fine-tune a pretrained model on a binary sentence classification task using ambiguous data, which equally supports both a simple linguistic generalization and a simple surface one. For example, Figure 1 shows a linguistic task where sentences in the positive class are defined by having a main verb in the "ing" form. We make the training data ambiguous by introducing a surface feature that distinguishes the two classes: In all (and only) training examples with label 1, the word "the" precedes the word "a". Based on this training data, a model could reasonably adopt either a linguistic generalization or a surface one.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We then test the classifier on disambiguating data to observe which generalization it made. In this kind of data, the labels align with the linguistic generalization, and contradict the surface one: For example, in Figure 1, "a" now always precedes "the" in the positive test examples with label 1. We quantify a model's inductive bias using a metric we call the linguistic bias score (LBS). We define LBS as the Matthews correlation between the model predictions and the labels on the disambiguating test set (Matthews, 1975). If LBS is 1, the learner shows a systematic linguistic bias. If LBS is -1, it shows a systematic surface bias. If LBS is 0, it shows neither bias.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Finally, while the fully ambiguous experiments probe models' biases in an idealized setting, training data in more naturalistic contexts often does contain some evidence supporting a linguistic generalization over a simple surface one. To simulate this, we also conduct a series of inoculation experiments (following Liu et al., 2019a), in which we introduce small amounts of disambiguating data into an otherwise ambiguous training set. For each experiment, we replace 0.1%, 0.3%, or 1% of the training data with examples that support the linguistic generalization and contradict the surface one. These experiments allow us to compare the strength of linguistic bias in models that show an overall surface bias: If two models adopt the surface generalization in the fully ambiguous case, we can still say that one has a stronger linguistic bias than the other if it requires less inoculation data to be swayed towards the linguistic generalization.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We introduce MSGS (Mixed Signals Generalization Set), pronounced "messages", a dataset we design to be used in poverty of the stimulus and inoculation experiments. With the goal of contrasting inductive biases that are helpful and harmful in most NLP applications, the tasks in MSGS test a model's preferences for generalizations based on linguistic or surface features. Table 3 illustrates the 4 linguistic features and 5 surface features we con-Dom. Split L L L S Sentence In Train (Ambiguous) 1 1 These men weren't hating that this person who sang tunes destroyed the vase. 0 0 These men hated that this person who sang tunes was destroying some vase.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Inoc. (Disamb.) 1 0 These men weren't hating that this person who sang tunes destroyed some vase. 0
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
1 These men hated that this person who sang tunes was destroying the vase.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Out Test (Disamb.) 1 0 These reports that all students built that school were impressing some children. 0
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
1 These reports that all students were building the school had impressed some children.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Aux. (Ambiguous) 1 1 These reports that all students built the school were impressing some children. 0 0 These reports that all students were building that school had impressed some children. Table 2: A full paradigm from the SYNTACTIC POSITION × LEXICAL CONTENT task. L L and L S mark the presence of the linguistic feature (Is the main verb in the "ing" form?) and surface feature (Does S contain "the"?), respectively. Dom. is short for domain.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
sider. 1 Each feature is meant to be representative of a broad category of features (e.g. morphological features), though the precise implementation of each feature is necessarily much narrower (e.g. Does the sentence have an irregular past verb?).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Forming generalizations based on surface features entails knowledge of the identity of certain words (in our case, only "the" and "a"), the positional indices of words in the string, the total number of words in a string, or whether certain characters are lowercase or uppercase. 2 Forming generalizations based on linguistic features requires more abstract knowledge of tense and inflectional morphemes, parts of speech, the control construction, 3 and hierarchical syntactic structures, none of which are encoded in the surface string.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Dataset Structure MSGS contains 20 ambiguous binary classification tasks each gotten by pairing one of 4 linguistic features with one of 5 surface features. We write FEAT 1 × FEAT 2 to denote a task that combines features FEAT 1 and FEAT 2 . Each ambiguous dataset contains 50k sentences split into 1 We explored a slightly larger set of linguistic features and excluded several based on initial experiments showing our models did not encode them. For example, we constructed a task with the objective of identifying sentences that contain antonyms (e.g. The little girl likes the big dog.), but found that only RoBERTaBASE could solve the unambiguous control task.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
2 Although these are surface properties of the string, they are not all trivial for RoBERTa due to its subword tokenization. 3 The control construction is a syntactic construction in which a semantic argument of a predicate fills or controls an argument slot of an embedded verb. The raising construction is superficially similar, but the filler of the embedded argument slot is not a semantic argument of the main predicate (Sag et al., 2003). For instance, Sue is eager to sleep is an example of control because the NP Sue is the semantic subject of both eager and sleep. By contrast, Sue is likely to sleep is an example of raising because Sue is the semantic subject of sleep, but not of likely. These two phenomena have different syntactic derivations in some theories (Chomsky, 1981). training, evaluation, and inoculation sets. MSGS also includes 9 unambiguous control tasks-one for each feature. Each control dataset contains 30k sentences split into training and evaluation sets.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
For ambiguous tasks, we generate data in paradigms of 8 sentences following a 2 × 2 × 2 design, as shown in Table 2. We vary the following three features: a binary linguistic feature, a binary surface feature, and the domain from which the sentence is sampled. We generate in-domain and out-of-domain sentences from different templates (see §3:Data Generation for more detail).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
As shown in Table 2, we split the data into four contrasting pairs with different purposes: (1) Training data is ambiguous in-domain data makes up 99% to 100% of the training set.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
(2) Inoculating data is disambiguating in-domain data which makes up 0.1% to 1% of the training set in experiments with inoculation. We show the classifier only the linguistic label (L L ) to nudge it towards adopting a linguistic generalization. (3) Test data is disambiguating out-of-domain data used to test whether the model adopted the linguistic or surface generalization. (4) Auxiliary data is ambiguous out-of-domain data used to test how well the model adapts to the out-of-domain templates, regardless of which generalization it makes.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
For control tasks, we generate data in paradigms of 4 sentences following a 2 × 2 design by varying the feature and domain. We use control tasks to test whether each pretrained model represents each feature well enough to fine-tune an effective classifier in an unambiguous setting.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Data Generation The data is generated from templates using a generation toolkit from . This toolkit includes a vocabulary of over 3000 entries labeled with grammatical fea- tures that allow for lexical variation in the data while maintaining grammatical well-formedness. Although generated sentences often describe unlikely or implausible scenarios (e.g., The lawyer was sinking all canoes), semantic plausibility is independent of all the features we examine, so this should not affect a model that genuinely encodes these features. To prevent out-of-vocabulary tokens affecting our results, we ensure that every word stem in the vocabulary appears in the pretraining datasets for our RoBERTa models (see §4.1).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Our experimental logic only makes sense if we are reasonably confident that models can only achieve high test performance by genuinely adopting a linguistic generalization. However, training models on generated data can easily lead to overfitting, and classifiers trained and tested on data from the same domain can achieve perfect performance even on arbitrary tasks with random labels (Hewitt and Liang, 2019). For this reason, our primary evaluations test models' ability to generalize outof-domain. We manipulate domain in two ways:
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
First, we generate training data and test data for each dataset from separate in-domain and out-ofdomain templates. Thus a model cannot succeed at test time simply by recognizing a template or a key part of a template. For example, in the SYNTACTIC POSITION × LEXICAL CONTENT paradigm shown in Table 2, the in-domain data contrasts the main verb with a verb in a relative clause embedded in the complement clause of a verb; while the out-ofdomain data contrasts the main verb with a verb in the complement clause of a noun. In most tasks, each domain itself is generated from multiple templates as well to widen the domain and encourage better generalization during training.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Second, on tasks that test lexical knowledge (for instance, the knowledge that slept is an irregular past verb and meow is not), we divide the crucial lexical items into in-domain and out-of-domain sets. Thus, a model cannot succeed by memorizing the keywords associated with each class. See the Appendix for a more detailed description of the implementation details for each feature.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We test 13 RoBERTa models in our main experiments: We pretrain 12 from scratch, and also test RoBERTa BASE pretrained by Liu et al. (2019b).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Pretraining Data We pretrain RoBERTa using scaled-down recreations of the dataset used by Devlin et al. (2019) to train BERT, i.e English Wikipedia (2.5 billion tokens) and BookCorpus (800 million tokens). Both are included in the RoBERTa pretraining data. 4 We download the latest Wikipedia dump as of Feb 1, 2020. The original BookCorpus (Zhu et al., 2015) is no longer available, so we collect similar data from Smashwords, the original source of BookCorpus. 5 4 RoBERTa uses English Wikipedia, BookCorpus, CC-News, OpenWebText, and STORIES in pretraining. 5 We collect our data using the Wikipedia XML dump https://dumps.wikimedia.org/mirrors.html and dataprocessing code https://github.com/attardi/wikiextractor, and a Smashwords crawler https://github.com/soskek/bookcorpus.  Figure 3: Results measured in LBS for each pretraining and inoculating data amount, aggregated over the 20 tasks in MSGS. We exclude models that fail the corresponding controls, as described in Section 5. High density near LBS of 1 means many models in that group have a linguistic bias; high density near -1 means many models have a surface bias. Models with stronger linguistic bias achieve higher LBS with less inoculation data.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We pretrain RoBERTa on four training sets containing different numbers of words: 1M, 10M, 100M, and 1B. 6 To make these datasets, we sample entire Wikipedia articles and Smashwords books independently, keeping the proportions of Wikipedia and Smashwords text approximately constant.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Model Sizes Model size is the only hyperparameter we systematically search over during pretraining. We consider smaller model sizes to prevent overfitting on small training sets. The detailed configurations of the model sizes are summarized in the Appendix. We use RoBERTa BASE  Search Range For dropout, attention dropout, learning rate decay, weight decay and the Adam parameters, we adopt the same parameter values used in Liu et al. (2019b). We fix warm up steps to be 6% of max steps, peak learning rate to be 5e-4, early stopping patience to be 100M tokens, and heuristically define the search range of model size, max steps and batch size for each training set.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We randomly sample hyperparameters from the search range and train 25 models for each of the 1M, 10M, 100M datasets. We train 10 models on the largest (1B) dataset due to resource limitations. For each training set size, we choose three of the resulting models to evaluate. In order to avoid confounds caused by different model sizes, for each training set we choose three models 6 The publicly available RoBERTaBASE is trained on 160GB of data, which we estimate to be about 30B words. of the same size that have the lowest perplexity. The hyperparameters and validation perplexities of the selected models are listed in the Appendix.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We loosely follow the hyperparameter settings that Liu et al. (2019b) used for fine-tuning on GLUE tasks (Wang et al., 2018), and use the following learning rates: {1E-5, 2E-5, 3E-5}. We depart from Liu et al. in using a batch size of 16 and training for 5 epochs without early-stopping in all runs. These changes are based on pilots that showed that larger batch sizes and longer fine-tuning were no more effective for our tasks.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We conduct 3,471 fine-tuning runs: We fine-tune 13 RoBERTa models: (3 random initializations) × (4 pretraining data amounts) + (1 RoBERTa base ). We fine-tune each model 267 times: (3 learning rates) × ((9 control tasks) + (20 ambiguous tasks) × (4 inoculation amounts)). We evaluate model performance using LBS (see §2:Methods: Measuring Inductive Bias).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We have several main findings: (1) models learn to represent both surface features and linguistic features with relatively little data; (2) RoBERTa begins to acquire a linguistic bias with over 1B words of pretraining data; (3) increasing pretraining data strengthens linguistic bias; (4) there is considerable variation in models' preferences between specific pairs of linguistic and surface features. Figure 2 shows the results for the controls. Performance is near ceiling for most models and features. Because we evaluate all the models out-of-domain, this result cannot be explained by the models simply memorizing the features from the task training data. Thus, we conclude that most pretrained models we test encode both linguistic and surface features. The only exceptions are the syntactic category and syntactic construction features, for which models with less than 100M perform poorly. In subsequent plots, we filter out results where the controls are not passed. Specifically, if a particular combination of model checkpoint and learning rate achieves a Matthews correlation of less than 0.7 on the control task for feature F , we eliminate all results with this combination for any task involving F in Figure  3, or represent them as gray points in Figure 4. Figure 3 summarizes the main experiment results. For a given amount of pretraining and inoculation data, we consider all classifiers trained on all 20 tasks in MSGS and plot the density of their linguistic bias scores (LBSs).
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The results in the leftmost box (with 0% inoculation) show that only RoBERTa BASE demonstrates a consistent linguistic bias in the fully unambiguous setting. That said, it still adopts the surface bias much of the time. The other models show a clear surface bias overall. The results of experiments with inoculation data show that models with more pretraining data require less inoculation data to be swayed towards the linguistic generalization. We consistently observe, for each pretraining quantity, a phase transition where the linguistic generalization begins to overtake the surface generalization upon exposure to a certain amount of inoculating data. For example, the 1B model goes through this transition between 0.1% and 0.3% inoculating data. The 100M and 10M models go through this transition between 0.3% and 1% inoculating data. The phase transition comes earlier for models with more pretraining, indicating they have a stronger linguistic bias. We also notice distinctive behavior for the models at the extreme ends of pretraining data quantity: The 1M model never completes the transition, suggesting it has a strong surface bias, and RoBERTa BASE appears to be in the middle of this transition with 0% inoculating data, suggesting that even more pretraining data could produce a model with a more consistent linguistic bias.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
These findings are echoed in individual task results in Figure 4. 7 In each plot, models with the same amount of inoculation data (i.e. points with a given color) have higher LBS as the amount of pretraining data increases. Notably, on ambiguous tasks involving LEXICAL CONTENT, RoBERTa BASE usually favors generalizations based on linguistic features without any inoculating data, which no other pretrained model does. We find this result quite striking: Even if the labels are perfectly correlated with the presence or absence of the word "the", RoBERTa BASE overlooks that fact in favor of a deeper generalization based on an abstract feature like the inflectional form of a verb in a particular syntactic position. Furthermore, this preference is clearly acquired through additional pretraining. The results for MORPHOLOGY × ORTHOGRAPHY is a typical illustration of the differences between models. The 1M model never adopts the linguistic generalization based on the morphological feature, though it eventually rejects the surface generalization. The 100M and 1B models make robust linguistic generalizations only with 1.0% inoculating data. By contrast, RoBERTa BASE requires only 0.1% inoculating data (i.e. 10 out of 10k examples) to adopt the linguistic generalization.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Surface Biases of RoBERTa Our results also suggest some specific conclusions about which kinds of surface features RoBERTa pays attention to. 8 For instance, these models have little preference for sentence length. As shown in the second column of Figure 4, most of the models form linguistic generalizations rather than generalizations based on sentence length, even with no inoculating data. By contrast, the models strongly prefer generalizations based on orthography-and to a lesser extent lexical content and word order-over linguistic generalizations.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The Success of Pretrained Models Our findings provide insight into why pretraining on massive datasets is so successful. While linguistic feature learning is a major effect of pretraining, it is far from the end of the story: Pretraining also helps models learn which features are central to language. However, this second kind of learning seems to require far more exposure to data with current models and pretraining techniques. Therefore, massive datasets are needed to teach models which features are useful for generalizing.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
The data scale at which we observe RoBERTa beginning to show a linguistic bias (between 1B and 30B words) is similar to the amount of pretraining data used by the first pretrained LMs to achieve major successes at NLU tasks, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). This suggests a crucial data threshold below which language model pretraining is unlikely to be significantly helpful for most applications with current model architectures, and may explain the manyyear gap between the development of neural LMs and the first major applications of LM pretraining: The early LMs must have not have been trained sufficiently to cross that threshold, yielding consistently poor results.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
There is increasing interest in studying the inductive biases of neural networks. Much of this work has grown out of numerous findings that these models often fail to generalize in ways that task designers intend. For example, Jia and Liang (2017) and  demonstrate that ambiguity in widely used NLU datasets like SQuAD (Rajpurkar et al., 2016) and MultiNLI (Williams et al., 2018) leads models like BERT to adopt some surface generalizations, despite the fact that they represent linguistic features. This continues to be a problem for models like RoBERTa BASE which show an overall linguistic bias in our experiments. However, for tasks like NLI, the underlying linguistic feature depends on a combination of significant syntactic knowledge, semantic knowledge, and world knowledge. It stands to reason that representations and preferences for such high level features require more data to learn than the features we probe.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Other work has used the poverty of stimulus design to study inductive biases associated with particular neural architectures during syntactic gen-eralization. Ravfogel et al. (2019) train RNNs on a morphological prediction task using artificial languages derived from naturally occurring English text, finding that RNNs show a recency bias in acquiring agreement rules. McCoy et al. (2018 train a seq2seq models on generated data ambiguous between a surface and a structural generalization to learn the subject-auxiliary inversion rule in English question formation. They find that, while tree-structured models show a structural bias, sequence models do not.  conduct related experiments on subject-auxiliary inversion and other English structural rules, and find that BERT likely acquires a structural bias from pretraining.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
More abstract inductive biases have also been studied. Using zero-shot learning in an artificial language, Lake and Baroni (2018) show that RNNs lack a bias in favor of learning compositional meanings for new symbols. Gandhi and Lake (2019) and Gulordava et al. (2020) explore conditions under which neural networks exhibit a bias towards learning mutually exclusive meanings for new symbols.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Data augmentation and inoculation have also been explored previously as a way to influence how models generalize.  and Min et al. (2020) show that small amounts of inoculating data during training on textual entailment help BERT overlook certain surface generalizations. Jha et al. (2020) study inoculation using a constructed language of numerical sequences. Like us, they generate ambiguous datasets, though they only compare features that resemble our surface features. They find that it is relatively easy to nudge models away from shallow generalizations, but harder to nudge them towards deeper ones.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Finally, several earlier studies explored how increasing training data impacts linguistic knowledge in LMs. Unlike the present study, these studies evaluate LMs using an unsupervised acceptability judgment task on minimal pairs (i.e. not during fine-tuning), and do not attempt to separate feature learning from feature preferences. van Schijndel et al. (2019) find the greatest increase in sensitivity to acceptability contrasts occurs between training on 2M and 10M words.  find that while LMs learn agreement phenomena at a similarly early stage, other phenomena require more data to learn. Finally, Hu et al. (2020) find that adopting architectures that build in linguistic bias, such as RNNGs (Dyer et al., 2016), has a big-ger effect on the acceptability task than increasing training data from 1M to 40M words.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Our experiments shed light on the relationship between pretraining data and an inductive bias towards linguistic generalization. Our results indicate that, although some abstract linguistic features are learnable from relatively small amounts of pretraining data, models require significant pretraining after discovering these features to develop a bias towards using them when generalizing. This gives some insight into why extensive pretraining helps general-purpose neural networks adapt to downstream tasks with relative ease.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
We also introduce MSGS, a new diagnostic dataset for probing the inductive biases of learning algorithms using the poverty of the stimulus design and inoculation, and also introduce a set of 12 RoBERTa models we pretrain on smaller data quantities. These models could prove to be a helpful resource for future studies looking to study learning curves of various kinds with respect to the quantity of pretraining data.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Finally, while our results naturally lead to the conclusion that we should continue to pursue models with ever more pretraining, such as GPT-3 (Brown et al., 2020), we do not wish to suggest that this will be the only or best way to build models with stronger inductive biases. Future work might use MSGS as a diagnostic tool to measure how effectively new model architectures and selfsupervised pretraining tasks can more efficiently equip neural networks with better inductive biases. Table 3 (repeated from the main body of the paper, for convenience). Implementation details for the features are described below. The implementation of one feature sometimes depends on other feature it is paired with in an ambiguous dataset.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Absolute position This feature is 1 iff the sentence begins with the word "the". We generally ensure that sentences bearing a value for this feature contains two clauses and four determiners total. Some sentences in SYNTACTIC CATEGORY × AB-SOLUTE POSITION contain fewer than four NPs. The in-domain and out-of-domain sentences differ in the order or position of the clauses.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Length This feature is 1 iff the sentence exceeds some number of tokens. The exact threshold varies depending on the linguistic feature in an ambiguous task, since different linguistic features lead to sentences of different length, on average. In mixed tasks, we vary the length of sentences by adjoining subordinate clauses (e.g. If Sue wakes) of varying length to the clause in which the linguistic feature is varied.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Lexical content This feature is 1 iff the sentence contains the. The sentences generally contain at least two clauses and four determiners. The position of the varies between in-domain and out-ofdomain sentences.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Relative position This feature is 1 when the precedes a, and 0 when a precedes the. The sentences generally contain at least two clauses and four determiners. Thus, there are six different configurations in which the precedes a, and these are separated into in-domain and out-of-domain templates.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Orthography This feature is 1 iff the sentence appears in title case. In the control paradigm, the sentences generally contain two clauses, whose positions are varied between in-domain and out-ofdomain sentences.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Lexical semantics This feature is 1 iff the sentence contains a pair of antonyms. In sentences with label 0, there is a pair of words in a hypernym/hyponym or synonym relation. There are 21 pairs of adjective antonyms and 21 pairs of verb antonyms (not accounting for different inflectional forms). To prevent the task being solvable using lexical content, these pairs are divided into indomain and out-of-domain sets. There are different templates corresponding to whether the antonyms are adjectives, intransitive verbs, or transitive verbs. Each template appears in both in-domain and outof-domain sentences.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Morphology This feature is 1 when the sentence contains an irregular past tense verb, and 0 when it contains a 3rd person present plural verb (identical to the bare form). Sentences either contain an irregular past tense verb or a regular 3rd person present plural verb (identical to the bare form). We do this because other verb forms can be identified by inflectional morphemes such as -s or auxiliaries such as have, and so discrimination between them could in some cases reduce to a lexical content task. The verbs are divided into in-domain and out-of-domain sets.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Syntactic category This feature is 1 iff the sentence contains an adjective. To diversify the templates, we consider all grammatical combinations of a noun, an adjective, a locative PP, and a proper name (e.g., Sue is the tall actress in the park, or The actress is Sue). In out-of-domain sentences we also include single-word nominal predicates like president (see the example in Table 3 to control for the fact that predicative adjectives are always a single, lowercase word. This gives a total of 19 templates divided into in-domain and out-of-domain sets, some with adjectives and some without. The set of adjectives is also split between domains.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Syntactic construction This feature has value 1 iff the sentence contains the control construction. In the control construction a semantic argument of a predicate fills or controls an argument slot of an embedded verb (Sag et al., 2003). For instance, in Sue is eager to sleep, the NP Sue surfaces as the syntactic subject of eager, but Sue is also understood as the semantic subject of sleep. This contrasts with the raising construction in Sue is likely to sleep, where Sue is again surfaces as the syntactic subject of likely in the main clause, and is the semantic subject of sleep in the embedded position, but is not a semantic argument of likely. Different predicates are compatible with control and raising: eager is a control predicate and likely is a raising predicate. We include control and raising predicates of three kinds: subject control/raising verbs, object control/raising verbs, and control/raising adjectives. Specific predicates are divided into in-domain and  out-of-domain sets, but all three kinds of predicates appear in both domains.
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Syntactic position All sentences contain at one or two embedded clauses. We include six sentence types, divided into in-domain and out-of-domain. For example, some sentences contain a relative clause within a relative clause, or a verb phrase with a complement clause. Each sentence type is generated from multiple templates varying the position of the clauses. The set of -ing verbs is not split between domains.    
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Large pre-trained language models (LMs) (Devlin et al., 2019;Raffel et al., 2020;Brown et al., 2020) have demonstrated impressive empirical success on a range of benchmark NLP tasks. However, analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on, suggesting they are often "right for the wrong reasons" (McCoy et al., 2019). Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques: probing classifiers (Adi et al., 2017;Hupkes et al., 2018), which measure whether or not a given feature is encoded by a representation, and challenge sets (Cooper et al., 1996;Linzen et al., 2016;Rudinger et al., 2018), which measure whether model behavior in practice is consistent with use of a given feature. The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language. Work based on probing classifiers has consistently found evidence that models contain rich information about syntactic structure (Hewitt & Manning, 2019;Tenney et al., 2019a), while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features, rather they fail in trivial ways (Dasgupta et al., 2018;Glockner et al., 2018;Naik et al., 2018).
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
In this work, we attempt to link these two contrasting views of feature representations. We assume the standard recipe in NLP, in which linguistic representations are first derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features, and then are adapted for a task of interest via transfer learning, or fine-tuning, on a task-specific dataset. We test the 2.2 DEFINITIONS Let X be the set of all sentences and S be the space of all sentence-label pairs (x, y) ∈ X × {0, 1}. We use D ⊂ S to denote a particular training sample drawn from S. We define two types of binary features: target (t) and spurious (s). Each is a function from sentences x ∈ X to a binary label {0, 1} that indicates whether the feature holds.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Target and spurious features. The target feature t is such that there exists some function f : {0, 1} → {0, 1} such that ∀(x, y) ∈ S, f (t(x)) = y. In other words, the label can always be perfectly predicted given the value of t. 1 A feature s is spurious if it is not a target feature.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Partitions of S. To facilitate analysis, we partition S in four regions ( Figure 1). We define S s-only to be the set of examples in which the spurious feature occurs alone (without the target). Similarly, S t-only is the set of examples in which the target occurs without the spurious feature. S both and S neither are analogous. For clarity, we sometimes drop the S * notation (e.g., s-only in place of S s-only ). S both " tpx, yq | tpxq " 1^spxq " 1u S neither " tpx, yq | tpxq " 0^spxq " 0u S t-only " tpx, yq | tpxq " 1^spxq " 0u S s-only " tpx, yq | tpxq " 0^spxq " 1u < l a t e x i t s h a 1 _ b a s e 6 4 = " b S i q w Q A N S N 3 0 6 j L / e I 6 a F q d f p Q w = " > A A A 0 0 n i c n V t b c 9 z K c a Y c x 3 E Y x z l O H v 2 C C i X 7 n B T F I n W Y i l 9 c Z d 5 J c X l f X q S z O i o s t h c L E T d h Z r G k 4 E 1 V K i 9 + 8 E v y g / I / / F / 8 k J 4 Z D L o H C / L E Y Z V E 9 P f N N O b y 9 X Q v F h z m c S T k + v q f X v z o r 3 7 8 1 z / 5 m 5 / + 7 f L f / e z v f / 4 P X / 3 i H 2 9 E N i 0 C u A 6 y O C v u h r 6 A O E r h W k Y y h r u 8 A D 8 Z x n A 7 v N 9 R / G 0 J h Y i y t C 8 f c / i Q + G E a j a P A l w h 9 / O r P H v 5 c f a w G E h 5 k k V Q v h 5 m c v J z P f / X b Q f X 1 w + r j N 4 P V 3 w 9 W 5 d c P 3 / x 2 Y z C D U Q i e M M b c G w y W 2 7 1 T i O Q E i m 4 H 6 9 z B e r c D + f J 1 l s a P P z y A J / q L Z / q v t y a w / P G r l f W 1 d f 3 j L V 5 s 1 B c r S / X P + c d f v P m f w S g L p g m k M o h 9 I b 7 b W M / l h 8 o v Z B T E M F 8 e T A X k f n D v h / A d X q Z + A u J D p b d p 7 r 1 C Z O S N s w L / p d L T K O 9 R + Y k Q j 8 k Q W y a + n I g 2 p 8 A u 7 r u p H P / m Q x W l + V R C G p g b j a e x J z N P 7 b k 3 i g o I Z P y I F 3 5 Q R D h W L 5 j 4 h R 9 I V I Z z l 2 G W 3 U t / K N C H t 6 8 H W i Q + O v J R T 2 5 L N Q q Z Z b F u G q A C x f I C X 4 i x m H f O 0 A H F d D i O w r n n z C n V d 4 b k Q z X F / 7 s 6 r H r j O P P l f O G 2 / v B h N S z 8 f B I F D y 4 r s 1 G W Z h J a o 5 p h Y z W C p 2 b Q u R F t U O 3 p / 2 W 2 z 4 / w / + N 9 e f m V + v F O 9 2 6 9 k 6 3 + o b e 7 t 3 9 0 e t Q / O j u 9 8
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
S H x y F N 7 L 9 S O I W 8 8 7 l u P + 2 2 P m p a z z N 5 z 5 U 1 9 V + H Z R h 7 O q D b e 2 B 6 f p / 6 I u q x 8 u 7
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
g F u g u r c s W T w 2 c z Z N n l 6 0 o 6 f a Z 5 D o Z O G c b N X u 9 n r c r P l F f 6 M 1 r 3 l 7 P X r 1 3 6 Z R S N v K t T R F I 2 9 P B M i w n R l X O e x j 6 F T + 3 9 6 d C o f 5 h h J H X N U j O l e t / m L J 1 k 7 2 m k c 7 f y g I 5 x z i p W K O o N N W 2 F 8 a L g Z E U r F 0 t b V 6 9 d P y g R H 5 8 d h h v l / k n T M E z k z u q b R s x N l r h Z m u m V d b X W 4 s o K 3 9 8 N J N L 6 e P w z 6 T q e t H + y 0 s K g 5 H l r 1 z J n 6 F G q G q 6 6 e 2 x T T v 6 3 e 8 6 b / u d v f z r S 5 A Y 5 a X T 8 5 4 F p w E M V K r L G 6 w A M d G 6 i r 2 h / W P F m h a X 1 l e H 1 Z N 0 B q m F Q b 7 W w j C w y E e T V Q m T / w 4 2 q 3 3 a D 0 4 2 j E G 3 w 0 1 1 h V G 2 q + 4 B K E 7 O 6 g m X k z I 8 i F y n G 5 i O I s r f P T J b r I E q / 0 i 6 g u L r W + Q f q V q e Z N A V i 9 H C D 0 c m 6 X s 2 j R P j F D l x k S E 7 h M Q M z I Z U b E g M s A M W O X G R M T u k x I z M R l J s R E L h M R 8 8 l l P h F z 7 z L 3 x M Q u E 8 + 1 j I v E i w R G L H 4 w H D 2 q w 8 7 s 4 K r 3 a S q k h y X x r 6 W n P q q g H B / V y e N s j J f U v l P X d 0 p 3 z V w m I y Z 3 m Z y Y z y 7 z m Z j C Z Q p i h M s I Y q T L S G K m L j M l p n S Z k p i Z y 8 y I e X C Z B 2 I e X e a R m C 8 u 8 2 V u y j w
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
j 8 H u C 3 z 9 9 v L q i A 6 M 6 p t E t p l 8 t P c Z t c 2 7 H 5 X Y 4 t + t y u 5 z b c 7 k 9 z u 2 7 3 D 7 n D l z u g H O H L n f I u S O X O + L c W 5 d 7 y 7 l j l z v m X M / l e p w 7 c b k T z p 2 6 3 C n n z l z u j H P n L n f O u Q u X u + D c p c t d c u 7 K 5 a 4 4 1 3 e 5 P u e u X e 6 a c z c u d 8 O 5 W 5 e 7 5 d y d y 9 1 x 7 p 3 L v e P c e 5 e z s r / h J U T 5 B f T n C P z s u t 7 0  
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
O V e p x Z Z u r p v M q / 1 g N i q T S h k 5 8 y i s k e V R E m P K c / s 2 7 g 8 N H n e 7 0 a y D q J p g f W 7 7 t G y I T X + I n d f c W T s t z 3 v J 8 3 j W Y J B t B / N x E d I N m J s Z a u E / d 6 P y 5 R r m M 4 h H U L Q f a a E b f 9 M B j Q m b B x B f q 1 W d / K j P 9 C Q o K Z 4 S t F 1 h z 0 6 Y Z Y 9 1 l c Q A j c N o Z s 6 N d g Q Q e O r a d M V E L g X 6 G 5 j a O / T z 2 A 5 g 3 b 9 T 0 a k C 9 b V 1 f u 8 v r 9 t + b N x l v r z 2 Q n m A v 7 f T a 7 O W c 5 / b W q Z n k b I 1 b Z F z M 7 X M 3 l y g g n D d P 0 t p U I G m O y o r G E R R t 1 y I b y 8 R / o J Y W a L f D Z J H p l 5 j M Q 7 Z F L 3 k 8 V b P / o p 4 E u O x x b 8 7 f Y D r u L W z g j V / Q C J T R 9 i / x l 1 / g 3 h c Z a 3 m 1 s A E 7 W U m 0 M p R A b 7 N 4 X P i J e i A 1 m W U F F q j C f x T e y 9 7 3 b 1 6 q t 3 f 0 n w 1 M U / M W q s h x / 4 V + e + z l A O K Y t b E P R F 9 5 2 5 g A M e R T 9 d 8 j x j s k 6 i 0 2 V Q U b p 6 y 1 e o U 0 m 4 Y 6 Z + q i O J K w q t 2 L z B t l o N z N o v s o h 1 H k r 7 V e Q c 6 K J F Y P 7 + d V 7 / v 1 e Q e Z p a C 4 j S 5 O z n S / N 1 1 c r p i 8 g 9 F a 6 H 0 / i N K x f G y H T u 4 X 6 u E w H h u + C p Y r w L N W + C F 4 U e q l W V 3 Q S 3 h Y 8 3 Y m m V D L k 6 k C M J h 4 u / j Z N 4 V f C 0 / 9 F c T a s v M 4 5 y x X p 3 N W / A t q v A j 1 A P D 3 Y F V d P d d Q n Z O m I V 5 1 u 9 R q x W b 6 / y d a 9 F F Q f f W G X w x y 4 A 8 x z u J s N i z A v 2 / N X k 7 z G E x h E / t p G I N 5 Q V B f 4 v 0 / f r W y 0 f 7 7 l s W L m z d r G 5 t r / 3 q x u f K 7 7 f p v X 3 6 6 9 M u l f 1 7 6 e m l j 6 d + W f r d 0 u H S + d L 0 U v B i + + M O L / 3 r x 3 5 v 9 z S + b / 7 H 5 n 6 b p j 1 7 U f f 5 p y f n Z / O P / A s W X Y D g = < / l a t e x i t > Figure 1: We partition datasets into four sections, defined by the features (spurious and/or target) that hold. We sample training datasets D, which provide varying amounts of evidence against the spurious feature, in the form of s-only examples. In the illustration above, the s-only rate is 2 10 = 0.2, i.e., 20% of examples in D provide evidence that s alone should not be used to predict y.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Evidence from Spurious-Only Examples. We are interested in spurious features which are highly correlated with the target during training. Given a training sample D and features s and t, we define the s-only example rate as the evidence against the use of s as a predictor of y. Concretely, s-only rate = |D s-only | |D|, the proportion of training examples in which s occurs without t (and y = 0).
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Use of Spurious Feature. If a model has falsely learned that the spurious feature s alone is predictive of the label, it will have a high error rate when classifying examples for which s holds but t does not. We define the s-only error to be the classifier's error on examples from S s-only . When relevant, t-only error, both error, and neither error are defined analogously. In this work, "feature use" is a model's predictions consistency with that feature; we are not making a causal argument.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Extractability of a Feature. We want to compare features in terms of how extractable they are given a representation. For example, given a sentence embedding, it may be possible to predict multiple features with high accuracy, e.g., whether the word "dog" occurs, and also whether the word "dog" occurs as the subject of the verb "run". However, detecting the former will no doubt be an easier task than detecting the latter. We use the prequential minimum description length (MDL) Rissanen (1978)-first used by Voita & Titov (2020) for probing-to quantify this intuitive difference. 2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it. Formally, MDL measures the number of bits required to communicate the labels given the representations. Conceptually, MDL can be understood as a measure of the area under the loss curve: If a feature is highly extractable, a model trained to detect that feature will converge quickly to high accuracy, resulting in a low MDL. Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question. To compute MDL(s), we train a classifier (without freezing any parameters) to differentiate S s-only vs. S neither , and similarly compute MDL(t). See Voita & Titov (2020) for additional details on MDL. 3 2.3 HYPOTHESIS Stated using the above-defined terminology, our hypothesis is that a model's use of the target feature is modulated by two factors: The relative extractability of the target feature t (compared to the spurious feature s), and the evidence from s-only examples provided by the training data. In particular, we expect that higher extractability of t (relative to s), measured by MDL(s)/MDL(t), will yield models that achieve better performance despite less training evidence.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Since it is often difficult to fully decouple the target feature from competing spurious features in practice, we first use synthetic data in order to test our hypothesis in a clean setting. We use a simple classifier with an embedding layer, a 1-layer LSTM, and an MLP with 1 hidden layer with tanh activation. We use a synthetic sentence classification task with k-length sequences of numbers as input and binary labels as output. We use a symbolic vocabulary V with the integers 0 . . . |V | − 1. We fix k = 10 and |V | = 50K. We begin with an initial training set of 200K, evenly split between examples from S both and S neither . Then, varied across runs, we manipulate the evidence against the spurious feature (i.e., the s-only rate) by replacing a percentage p of the initial data with examples from S s-only for p ∈ {0%, 0.1%, 1%, 5%, 10%, 20%, 50%}. Test and validation sets consist of 1,000 examples each from S both , S neither , S t-only , S s-only . In all experiments, we set the spurious feature s to be the presence of the symbol 2. We consider several different target features t (Table 1), intended to vary in their extractability.  Figure 2 shows model performance as a function of s-only rate for each of the four features described above. Here, performance is reported using error rate (lower is better) on each partition (S s-only , S t-only , S both , S neither ) separately. We are primarily interested in whether the relative extractability of the target feature (compared to the spurious feature) predicts model performance. We indeed see a fairly clear relationship between the relative extractability (MDL(s) MDL(t)) and model performance, at every level of training evidence (s-only rate). For example, when t is no less extractable than s (i.e., contains-1), the model achieves zero error at an s-only rate of 0.001, meaning it learns that t alone predicts the label despite having only a handful of examples that support this inference. In contrast, when t is harder to extract than s (e.g., first-last), the model fails to make this inference, even when a large portion of training examples provide evidence supporting it.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We investigate whether the same trend holds for language models fine-tuned with naturalistic data, e.g., grammar-generated English sentences. To do this, we fine-tune models for the linguistic acceptability task, a simple sequence classification task as defined in Warstadt & Bowman (2019), longer blocks, using classifiers that are trained on previously transmitted data. The high MDL's are a result of overfitting by classifiers that are trained on limited data-and therefore, the classifiers have worse compression performance than the uniform baseline. 4 Note, all models are ultimately able to learn to detect t (achieve high test accuracy) on the both partition, but not on the t-only partition. Both Error Figure 2: Results on Synthetic Data. Error on each partition of the test set, as a function of s-only rate. A model that has learned to use the target feature alone to predict the label will achieve zero error across all partitions. s-only and t-only error reach 0 quickly when t is as easy to extract as s (i.e., the relative extractability is 1). However, when t is harder to extract than s (rel. extractability < 1), performance lags until evidence from s-only examples is quite strong.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
in which the goal is to differentiate grammatical sentences from ungrammatical ones. We focus on acceptability judgments since formal linguistic theory guides how we define the target features, and recent work in computational linguistics shows that neural language models can be sensitive to spurious features in this task (Marvin & Linzen, 2018;Warstadt et al., 2020a).
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We design a series of simple natural language grammars that generate a variety of feature pairs (s, t), which we expect will exhibit different levels of relative extractability (MDL(s) MDL(t)). We focus on three syntactic phenomena (described below). In each case, we consider the target feature t to be whether a given instance of the phenomenon obeys the expected syntactic rules. We then introduce several spurious features s which we deliberately correlate with the positive label during fine-tuning. The Subject-Verb Agreement (SVA) construction requires detecting whether the verb agrees in number with its subject, e.g., "the girls are playing" is acceptable while "the girls is playing" is not. In general, recognizing agreement requires some representation of hierarchical syntax, since subjects may be separated from their verbs by arbitrarily long clauses. We introduce four spurious features: 1) lexical, grammatical sentences begin with specific lexical items (e.g., "often"); 2) length, grammatical sentences are longer; 3) recent-noun, verbs in grammatical sentences agree with the immediately preceding noun (in addition to their subject); and 4) plural, verbs in grammatical sentences are preceded by singular nouns as opposed to plural ones.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The Negative Polarity Items (NPI) construction requires detecting whether a negative polarity item (e.g., "any", "ever") is grammatical in a given context, e.g., "no girl ever played" is acceptable while "a girl ever played" is not. In general, NPIs are only licensed in contexts that fall within the scope of a downward entailing operator (such as negation). We again consider four types of spurious features: 1) lexical, in which grammatical sentences always include one of a set of lexical items ("no" and "not"); 2) length (as above); 3) plural, in which each noun in a grammatical sentence is singular, as opposed to plural; and 4) tense, in which grammatical sentences are in present tense.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Some verbs (e.g. "recognize") require a direct object. However, in the right syntactic contexts (i.e., when in the correct syntactic relation with a wh-word), the object position can be empty, creating what is known as a "gap". E.g., "I know what you recognized " is acceptable while "I know that you recognized " is not. The Filler-Gap Dependencies (GAP) construction requires detecting whether a sentence containing a gap is grammatical. For our GAP tasks, we again consider four spurious features (lexical, length, plural, and tense), defined similarly to above.  Only s is highlighted since t is often defined over the structure of the sentence (see text) and thus difficult to localize to a few tokens. Table 9 in the Appendix has neither examples.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The templates above (and slight variants) result in 20 distinct fine-tuning datasets, over which we perform our analyses (see Appendix for details). Table 2 shows several examples. For the purposes of this paper, we are interested only in the relative extractability of t vs. s given the pre-trained representation; we don't intend to make general claims about the linguistic phenomena per se. Thus, we do not focus on the details of the features themselves, but rather consider each template as generating one data point, i.e., an (s, t) pair representing a particular level of relative extractability.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We evaluate T5, BERT, RoBERTa, GPT-2 and an LSTM with GloVe embeddings (Raffel et al., 2020;Devlin et al., 2019;Liu et al., 2019b;Radford et al., 2019;Pennington et al., 2014). 5 Both T5 and BERT learn to perform well over the whole test set, whereas the GloVe model struggles with many of the tasks. We expect that this is because contextualized pre-training encodes certain syntactic features which let the models better leverage small training sets (Warstadt & Bowman, 2020). Again, we begin with an initial training set of 2000 examples, evenly split between both and neither, and then introduce s-only examples at rates of 0%, 0.1%, 1%, 5%, 10%, 20%, and 50%, using three random seeds each. Test and validation sets consist of 1000 examples each from S both , S neither , S s-only . In the natural language setting, it is often difficult to generate t-only examples, and thus we cannot compute extractability of the target feature t by training a classifier to distinguish S t-only from a random subset of S neither , as we did in Section 3. Therefore, we estimate MDL by training a classifier to distinguish between examples from S s-only and examples from S both . Using the simulated data from Section 3, we confirm that both methods (S s-only vs. S both and S t-only vs. S neither ) produce similar estimates of MDL(t) (see Appendix). Per model, we filter out feature pairs for which the model could not achieve at least 90% accuracy on each probing task in isolation. 6
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
For each (s, t) feature pair, we plot the use of the spurious feature (s-only error) as a function of the evidence against the spurious feature seen in training (s-only example rate). 7 We expect to see the same trend we observed in our synthetic data, i.e., the more extractable the target feature t is relative to the spurious feature s, the less evidence the model will require before preferring t over s. To quantify this trend, we compute correlations between 1) the relative extractability of t compared to s and 2) the test F-score averaged across all rates and partitions of the data 8 , capturing how readily the model uses (i.e., makes predictions consistent with the use of) the target feature.    Figure 3 shows these correlations and associated scatter plots. We can see that relative extractability is strongly correlated with average test F-score (Figure 3a), showing high correlations for both BERT (ρ = 0.79) and T5 (ρ = 0.57). That is, the more extractable t is relative to s, the less evidence the model requires before preferring t, performing better across all partitions. This relationship holds regardless of whether relative extractability is computed using a ratio of MDL scores or an absolute difference. We also see that, in most cases, the relative extractability explains the model's behavior better than does the extractability of s or t alone. For GloVe there is little variation in model behavior: For most of the 11/20 pairs on which the model is able to learn the task, it requires an s-only example rate of 0.5. Thus, the correlations are weak, but qualitative results appear steady ( Figure 8 in Appendix A), following the pattern that when s is easier to extract than t, more evidence is required to stop using s. Figure 4 shows the performance curves for BERT and T5 (with others the in Appendix A), i.e., use of the spurious feature (s-only error) as a function of the evidence from s-only examples seen in training (s-only example rate). Each line corresponds to a different s, t feature pair, and each data point is the test performance on a dataset with a given s-only example rate (which varies along the x-axis.) For pairs with high MDL ratios (i.e., when t is actually easier to extract than s), the model learns to solve the task "the right way" even when the training data provides no incentive to do so: That is, in such cases, the models' decisions do not appear to depend on the spurious feature s even when s and the target feature t perfectly co-occur in the fine-tuning data. Figure 4 shows that T5 (compared to BERT) requires more data to perform well. This may be because we fine-tuned T5 with a linear classification head, rather than the text-only output on which it was pre-trained. We made this decision 1) because we had trouble training T5 in the original manner, and 2) using a linear classification head was consistent with the other model architectures.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Our experimental results provide support for our hypothesis: The relative extractability of features given an input representation (as measured by information-theoretic probing tech- and summarized in the bar chart). When t is much harder to extract relative to s (lower ratios), the classifier requires much more statistical evidence during training (higher s-only rate) in order to achieve low error. We find similar patterns GPT2 and RoBERTa; see Appendix A for all the results.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
niques) is predictive of the decisions a trained model will make in practice. In particular, we see evidence that models will tend to use imperfect features that are more readily extractable over perfectly predictive features that are harder to extract. This insight is highly related to prior work which has shown, e.g., that neural networks learn "easy" examples before they learn "hard" examples (Mangalam & Prabhu, 2019). Our findings additionally connect to new probing techniques which have received significant attention in NLP but have yet to be connected to explanations of or predictions about state-of-the-art models' decisions in practice.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Fine-tuning may not uncover new features. The models are capable of learning both the s and t features in isolation, so our experiments show that if the relative extractibility is highly skewed, one feature may hide the other -a fine-tuned model may not use the harder-to-extract feature. This suggests a pattern that seems intuitive but is in fact non-trivial: If one classifier does not pick up on a feature readily enough, another classifier (or, rather, the same classifier trained with different data) may not be sensitive to that feature at all. This has ramifications for how we view fine-tuning, which is generally considered to be beneficial because it allows models to learn new, task-relevant features. Our findings suggest that if the needed feature is not already extractable-enough after pretraining, fine-tuning may not have the desired effect.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Probing classifiers can be viewed as measures of a pre-trained representation's inductive biases. Analysis with probing classifiers has primarily focused on whether important linguistic features can be decoded from representations at better-than-baseline rates, but there has been little insight about what it would mean for a representations' encoding of a feature to be "sufficient". Based on these experiments, we argue that a feature is "sufficiently" encoded if it is as available to the model as are surface features of the text. For example, if a fine-tuned model can access features about a word's semantic role as easily as it can access features about that word's lexical identity, the model may need little (or no) explicit training signal to prefer a decision rule based on the former structural feature. The desire for models with such behavior motivates the development of architectures with explicit inductive biases (e.g., TreeRNNs). Evidence that similar generalization behavior can result from pre-trained representations has exciting implications for those interested in sample efficiency and cognitively-plausible language learning (Warstadt & Bowman, 2020;Linzen, 2020). We note that this work has not established that the relationship between extractability and feature use is causal. This could be explored using intermediate task training (Pruksachatkun et al., 2020) in order to influence the extractability of features prior to fine-tuning for the target task; e.g., Merchant et al. (2020) suggests fine-tuning on parsing might improve the extractability of syntactic features.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Significant prior work analyzes the representations and behavior of pre-trained LMs. Work using probing classifiers (Veldhoen et al., 2016;Adi et al., 2017;Conneau et al., 2018;Hupkes et al., 2018) suggests that such models capture a wide range of relevant linguistic phenomena (Hewitt & Manning, 2019;Tenney et al., 2019a;b). Similar techniques include attention maps/visualizations (Voita et al., 2019;Serrano & Smith, 2019), and relational similarity analyses (Chrupała & Alishahi, 2019). A parallel line of work uses challenge sets to understand model behavior in practice. Some works construct evaluation sets to analyze weaknesses in the decision procedures of neural NLP models (Jia & Liang, 2017b;Glockner et al., 2018;Dasgupta et al., 2018;Gururangan et al., 2018;Poliak et al., 2018b;Elkahky et al., 2018;Ettinger et al., 2016;Linzen et al., 2016;Isabelle et al., 2017;Naik et al., 2018;Jia & Liang, 2017a;Linzen et al., 2016;Goldberg, 2019, and others). Others use such datasets to improve models' handling of linguistic features (Min et al., 2020;Poliak et al., 2018a;Liu et al., 2019a), or to mitigate biases (Zmigrod et al., 2019;Zhao et al., 2018;Hall Maudslay et al., 2019;Lu et al., 2020). Nie et al. (2020) and Kaushik et al. (2020) explore augmenting training sets with human-in-the-loop methods.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Our work is related to work on generalization of neural NLP models. Geiger et al. (2019) discusses ways in which evaluation tasks should be sensitive to models' inductive biases and Warstadt & Bowman (2020) discusses the ability of language model pre-training to encode such inductive biases. Work on data augmentation (Elkahky et al., 2018;Min et al., 2020;Zmigrod et al., 2019) is relevant, as the approach relies on the assumption that altering the training data distribution (analogous to what we call s-only rate in our work) will improve model behavior in practice. Kodner (2019) show that neural networks learn "easy" examples (as defined by shallow machine learning model performance) before they learn "hard" examples. Zhang et al. (2016) and Arpit et al. (2017) show that neural networks which are capable of memorizing noise nonetheless acheive good generalization performance, suggesting that such models might have an inherent preference to learn more general features. Finally, ongoing theoretical work characterizes the ability of over-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc et al., 2020).
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Concurrent work (Warstadt et al., 2020b) also investigates the inductive biases of large pre-trained models (RoBERTa), in particular, they ask when (at what amount of pre-training data) such models shift from a surface feature (what we call spurious features) to a linguistic feature (what we call a target feature). In our work, we focus on how to predict which of these two biases characterize the model (via relative MDL).
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
This work bears on an open question in NLP, namely, the question of how models' internal representations (as measured by probing classifiers) influence model behavior (as measured by challenge sets). We find that the feature extractability can be viewed as an inductive bias: the more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during fine-tuning. Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models. of the spurious features ended up being very difficult for the models.) During the reviews, we did not control for the cases where the model did not solve the probing task. These 2 extraneous points accentuate the lineplot curves, but do not change the character of the results (nor much adjust the correlations). In the paper, now, we control for accuracy by filtering out these cases. With or without this control, the accuracy provides no predictive power about the inductive biases. We present the correlations without filtering for these cases for consistency with the reviews (Table 4 above); we believe it is important to control for these cases because they could have acted as giveaways, where even accuracy might have worked.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We initially used a different metric when computing the correlations to compact the lineplots. Rather than using the average test performance, we looked at the evidence required for the model to solve the test set. Both of these metrics conceptually capture what we are interested in, but the new one (simply averaging test performance) is much easier to understand, and captures the performance across all partitions. Here we report the correlations with this evidence required metric instead, which we called s-rate . Specifically, we defined it to be: s-rate is the lowest s-only example rate at which the fine-tuned model achieves essentially perfect performance (F-score > 0.99) (see Figure  5a). Intuitively, s-rate is the (observed) minimum amount of evidence from which the model infer that t alone is predictive of the label. See Table 5b for the results.  Table 5: AUC. Each column presents the Spearman correlation between the given measure and the s-only rate at which the model first acquired the target feature. * indicates a significant correlation. A feature is considered acquired when its test-performance is above 0.99 F-score. All (s, t) pairs (20 of 20 each) were eventually learned by T5, BERT, GPT2, and RoBERTa; 11 of 20 (s, t) pairs were learned by the GloVe-based LSTM.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
There are two major parts to this project in terms of reproducibility: (1) the data and (2) the model implementations. We describe the templates for the data below in Appendix D -the full details are in the project source. For the transformer models, we use Hugging Face for the implementations and access to the pre-trained embeddings (Wolf et al., 2020). We use PyTorch Lightning to organize the training code (Falcon, 2019). We fix all hyperparameters, which are reported in Table 6.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We want to call attention to BERT requiring much less data than T5 to capture our target features. At face value, it seems that BERT requires much less data than T5 to capture our target features. However, we are wary about making such strong claims. Something to consider here (noted in the Appendix B is that for T5 we used a linear model rather than formatting the task in text (which is how T5 is trained). We made this decision (1) because we had trouble training T5 in this purely textual manner, and (2) using a linear classification head over two classes is consistent with the other model architectures. Again, GPT2 and RoBERTa performed on par with BERT, so the difference between the performance of BERT and T5 may be due to how we trained T5.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
We measure the MDL for t with both and s-only examples. In the simulated setting we can compare this approach with measuring the MDL directly (t-only vs neither). See Table 7 for MDL results. The ordering of the feature's difficulty holds across the two methods.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Each template corresponds to a combination of target features, grammars, and spurious features (the target and spurious features are discussed in Section 4.1). See Table 8 for a complete list of templates. See Table 10 for further details about the templates that are used for each of the the target features. Complete details about implementation of these templates (and all data) will be released upon acceptance.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
Target features may be unavoidably linked to spurious ones. For example, for a Negative Polarity Item to be licensed (perhaps smoothing over some intricacies) the NPI ("any", "all", etc) must be a downward entailing context. These downward entailing contexts are created by triggers, e.g., if a negative word like "no" or "not" or a quantifier like "some". Linguists who study the problem have assembled a list of such triggers (see Hoeksema (2008)     Det N p V . Some student who no girl ever followed ran. NPI X
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The N p ever V . *The man who some lawyer hated ever traveled. NPI X
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The N p−ever V *The boy who ever smiled shouted.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The N x 1 of the (N i of the ) * N 2 V x 1 the N 3 .
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The piano teacher of the lawyers wounds the handyman. SVA X
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
The N x 1 of the (N i of the ) * N 2 V y 1 the N 3 .
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
*The piano teacher of the lawyers wound the handyman. Whether or not a noun and verb agree is given by whether their superscripts match.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
I know who he believed.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
*I know who he believed them.
P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
*I know who he knew the lawyers that she believed.      
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Human language is structured hierarchically. In NLP tasks like natural language inference, syntactic competence is a prerequisite for robust generalization (e.g., McCoy et al., 2019). Probing studies have found that masked language models (MLMs) contain hierarchical representations (Tenney et al., 2019;Hewitt and Manning, 2019;Clark Figure 1: The poverty of the stimulus experimental design. We fine-tune pre-trained seq2seq models and train small seq2seq models from scratch to perform syntactic transformations. The training set contains ambiguous examples consistent with hierarchical and linear transformation rules. The generalization set contains examples where only the hierarchical rule results in the correct output. Pre-trained models generalize using the hierarchical rule, while models trained from scratch generalize using the linear rule. et al., 2019), while behavioral studies of recurrent neural language models (Linzen et al., 2016;Marvin and Linzen, 2018;Wilcox et al., 2018;van Schijndel et al., 2019) and MLMs (Goldberg, 2019;Hu et al., 2020) have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Recent evidence suggests that MLMs like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) can learn to make hierarchical linguistic generalizations through exposure to text , though acquiring many of these linguistic generalizations requires large amounts of data . However, this evidence comes from binary acceptability judgment tasks, where a classifier head is attached to an MLM and the model is fine-tuned to classify which sentence in a given minimal pair is consistent with a hierarchical linguistic generalization, rather than a positional surface heuristic. Consider the following two transformations of Example (1):
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
(1) The yak that your unicorns have amused hasn't entertained a newt. a. Hasn't the yak that your unicorns have amused entertained a newt? b. *Have the yak that your unicorns amused hasn't entertained a newt?
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Example (1a) correctly forms the question by moving the main auxiliary verb to the front of the sentence, while (1b) relies on the incorrect positional heuristic that the first auxiliary in the declarative sentence should be moved to the front of the sentence. When differentiating grammatical and ungrammatical auxiliary movements, a model could rely on distributional information (Lewis and Elman, 2001) such as bigram heuristics (Reali and Christiansen, 2005;Kam et al., 2008) to make correct judgments in many cases, so high performance on binary classification tasks may overstate the syntactic competence of a model. By contrast, performing a syntactic transformation-e.g., given a declarative sentence like Example (1) as input, transforming it into a polar question like (1a)-is more difficult. It requires multiple complex but systematic operations that rely on hierarchical structure, including movement, number agreement, and-in languages that have grammatical case, such as German-case reinflection. Evaluations of syntactic transformational abilities can therefore act as more targeted behavioral indicators of syntactic structural representations in neural models. McCoy et al. (2018) evaluate non-pre-trained recurrent sequence-to-sequence (seq2seq) models (Sutskever et al., 2014) on the question formation task, finding that they rely on linear/positional surface heuristics rather than hierarchical structure to perform this syntactic transformation. More recent studies have also exclusively considered recurrent seq2seq models and Transformer models (Petty and Frank, 2021) trained from scratch on other transformations like tense reinflection (McCoy et al., 2020) and passivization (Mulligan et al., 2021), finding similar results. These studies were designed to understand the inductive biases of various seq2seq architectures, which is why they do not pre-train the models on non-annotated natural language data before training them to perform syntactic transformations.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
In this study, we create German datasets and modify English datasets for evaluating the induc-tive biases of pre-trained models. We use these datasets to analyze performance in monolingual and zero-shot cross-lingual settings. Further, we analyze how pre-trained models perform syntactic transformations. Our findings indicate that pretrained models generally perform syntactic transformations in a hierarchy-sensitive manner, while nonpre-trained models (including randomized-weight versions of pre-trained models) rely primarily on linear/positional heuristics to perform the transformations. This finding presents additional evidence to  and  for the learnability of hierarchical syntactic information from natural language text input. Our code and data are publicly available. 1 2 Syntactic Transformations
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We evaluate on syntactic transformations in English and German. We choose English to allow for comparisons to previous results (McCoy et al., 2018;Mulligan et al., 2021). We further extend our evaluations to German because it exhibits explicit case marking on determiners and nouns; this typological feature has been found to increase the sensitivity of language models to syntactic structure (Ravfogel et al., 2019). This allows us to compare transformational abilities for languages with different levels of surface cues for hierarchy.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We employ a poverty of the stimulus experimental design (Wilson, 2006), where we train the model on examples of a linguistic transformation that are compatible with either a hierarchical rule or a linear/positional rule, and then evaluate the model on sentences where only the hierarchical rule leads to the generalization pattern that is consistent with the grammar of the language (Figure 1). 2 In other words, we are interested in whether T5 and mT5 (henceforth, (m)T5), as well as BART and mBART (henceforth, (m)BART), demonstrate a hierarchical inductive bias, 3 unlike the linear inductive bias displayed in prior work by non-pre-trained models. 1 https://github.com/sebschu/ multilingual-transformations 2 There are other rules that could properly transform the stimuli we use, but we find that the models we test do learn one of these rules or the other.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
3 When multiple generalizations are consistent with the training data, "inductive bias" refers to a model's choice of one generalization over others. passiv: the zebra upon the yak confused your orangutans. → your orangutans were confused by the zebra upon the yak. We focus on two syntactic transformation tasks: question formation and passivization. See Table 1 for a breakdown of which structures we present to the model during training and which we hold out to evaluate hierarchical generalization. See Table 2 for examples of hierarchical and linear generalizations for each transformation.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Question formation. In this task, a declarative sentence is transformed into a polar question by moving the main (matrix) auxiliary verb to the start of the sentence; this hierarchical rule is called MOVE-MAIN. The linear rule, MOVE-FIRST, entails moving the linearly first auxiliary verb to the front of the sentence. Examples of both rules are provided in Figure 1 and Example (1). We train the model on sentences with no relative clauses (RCs) or with RCs on the object, where the first auxiliary verb is always the matrix verb. Disambiguating examples are those which place RCs on the subject, where the matrix auxiliary verb is the linearly second auxiliary in the sentence.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
In English, we use the auxiliaries "has", "hasn't", "have", and "haven't", with past participle main verbs (e.g., "have entertained", "has amused"). We use affirmative and negative forms to distinguish between the multiple auxiliaries: exactly one of the auxiliaries in such sentences is negative and the other is positive (counterbalanced across examples). As a result, we can determine whether the induced mapping is linear or hierarchical. In German, negation is realized as a separate word that is not fronted with the auxiliary. To distinguish the multiple auxiliaries, we therefore use the modal "können" (can) along with the auxiliary "haben" (have), together with infinitival or past participle main verbs as appropriate. This allows us to distinguish models with a hierarchical bias from those with a linear bias on the basis of the fronted auxiliary. Passivization. In this task, an active sentence is transformed into a passive sentence by moving the object noun phrase (NP) to the front of the sentence (MOVE-OBJECT). Our training examples are also compatible with a linear rule, MOVE-SECOND, in which the linearly second NP moves to the front of the sentence. We train on sentences with no prepositional phrases (PPs) or with PPs modifying the object, where the second NP is always the object. Disambiguating examples are those which place prepositional phrases (PPs) on the subject, where the object is the linearly third NP in the sentence.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Passivization additionally requires other movements, insertions, tense reinflection, and (for German) case reinflection. In Examples (2) and (3) below, the object (in blue) is fronted; 'be'/'werden' (in red) is inserted and inflected to agree with the fronted NP; the original subject NP (in brown) is moved to a 'by'/'von' phrase after the inserted verb; and the main verb (in orange) is reinflected to be a past participle or infinitive. In German, the case of the NPs (reflected largely in the determiners) must be reinflected, and the main verb needs to be moved to the end of the sentence.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
(2) English Passivization:
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
a. Your quails amused some vulture. b. Some vulture was amused by your quails.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
( Has my unicorn that hasn't amused the yaks eaten?
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Hasn't my unicorn that amused the yaks has eaten? quest: Die Hunde, die deine Löwen bewundern können, haben gewartet.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Haben die Hunde, die deine Löwen bewundern können, gewartet?
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Können die Hunde, die deine Löwen bewundern, haben gewartet? passiv: Her walruses above my unicorns annoyed her quail.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Her quail was annoyed by her walruses above my unicorns.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
My unicorns were annoyed by her walruses.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
passiv: Unsere Papageie bei meinen Dinosauriern bedauerten unsere Esel.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Unsere Esel wurden von unseren Papageien bei meinen Dinosauriern bedauert.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Meine Dinosaurier wurden von unseren Papageien bedauert. these are for evaluating in-distribution transformations on unseen sentences. For each transformation, we also generate a generalization set consisting of 10,000 transformed examples with RCs/PPs on the subject NP. For such examples, models relying on the linear rules will not generalize correctly.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We experiment with T5 (Raffel et al., 2020) and BART (Liu et al., 2020), two English pre-trained sequence-to-sequence models. We also experiment with their multilingual variants mT5 (Xue et al., 2021) and mBART . 5 These are 4 We generate our evaluation set such that it consists of grammatical but semantically improbable sentences which are unlikely to occur in a natural language corpus. This is to alleviate the confound of token collocations in the pre-training corpus. 5 We use HuggingFace implementations (Wolf et al., 2020).
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
12-layer Transformer-based (Vaswani et al., 2017) architectures with bidirectional encoders and autoregressive decoders. While we use the base sizes of (m)T5, we use the large sizes of (m)BART to keep the sizes of the models similar. When fine-tuning (m)T5 and (m)BART, we use task prefixes in the source sequence. We use "quest:" for question formation and "passiv:" for passivization. As in previous work, we also include identity transformation examples (prefixed with "decl:"), i.e., examples for which the model has to output the unchanged declarative or active sentence. When training seq2seq baselines from scratch, we follow McCoy et al. (2020) and append the task markers to the end of the input sequence.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
For fine-tuning on syntactic transformations, we use batch size 128 and initial learning rate 5×10 −5 . We fine-tune for 10 epochs and evaluate every 500 iterations. We find that the validation loss generally converges within 1-2 epochs.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
To confirm the finding of McCoy et al. (2020) and Petty and Frank (2021) that non-pre-trained models fail to generalize hierarchically, we also train baseline seq2seq models similar to the models used in those studies. We implement 1-and 2-layer LSTM-based seq2seq models, as well as 1-and 2-layer Transformer-based seq2seq models where the Transformers have 4 attention heads. 6 We find that the 1-layer models consistently achieve higher sequence accuracies on the dev sets, so we focus on the 1-layer baselines. We re-use all hyperparameters from McCoy et al. (2020). All baseline scores are averaged over 10 runs.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
For all transformations, we are primarily interested in sequence accuracy: is each token in the tar-   get sequence present in the proper order in the predicted sequence? However, it is possible that models could generalize hierarchically while making some other mistake, so we also use two more relaxed metrics. For question formation, we use main auxiliary accuracy, which evaluates whether the correct auxiliary was moved to the front of the sentence. The first word in the target sequence is always the main auxiliary verb, so we calculate main auxiliary accuracy by checking if the first word is the same in the predicted and target sequences. For passivization, we use object noun accuracy, which measures whether the correct object noun was moved to the subject position. The second word in the target sequence is always the original object noun, so we calculate object noun accuracy by checking if the second word is the same in the predicted and target sequences.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
All models learn the in-distribution transformations. We first present results on unseen sentences whose structures were seen in training, where both the hierarchical and the linear rules result in correct generalization (Table 3). All models perform well in this setting, including the LSTM-and Transformer-based models trained from scratch. However, (m)T5 converges to higher sequence accuracies than the non-pre-trained models. Additionally, while the non-pre-trained models require about 15-20 epochs of training to converge to a high score, (m)T5 and (m)BART converge to near-perfect sequence accuracy after only a fraction of an epoch of fine-tuning.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Only pre-trained models generalize hierarchically. Evaluations on the generalization-set examples (where the linear rule leads to incorrect generalization) reveal that none of the trained-fromscratch models have learned the hierarchical rule. These models consistently stay at or near 0% sequence accuracy on the generalization set throughout training, so we present main auxiliary/object noun accuracies (Table 4). Accuracy remains low even on these more forgiving metrics, indicating that the non-pre-trained models have not acquired the hierarchical rules. Low accuracies do not necessarily indicate reliance on the linear MOVE-FIRST or MOVE-SECOND rules. To test whether the non-pre-trained models have learned the linear rules, we implement metrics which calculate the proportion of generalization-set examples for which the MOVE-FIRST rule (for question formation) or MOVE-SECOND rule (for passivization) were used; we refer to these as the move-first frequency and movesecond frequency, respectively. For each model and language, the sum of the main auxiliary accuracy and move-first frequency for question formation is ≈ 1.0; the sum of the object noun accuracy and move-second frequency for passivization is also ≈ 1.0. Thus, where the model did not move the main auxiliary or object noun, it generally used the linear rule. In other words, the non-pre-trained models demonstrate linear inductive biases. This finding is in line with prior evaluations of non-pretrained seq2seq models (McCoy et al., 2020;Mulligan et al., 2021;Petty and Frank, 2021). 7 By contrast, (m)T5 and (m)BART achieve very high main auxiliary/object noun accuracies on the generalization sets. mBART struggles with En- Figure 2: Accuracies at every 500 fine-tuning iterations across 10 epochs of fine-tuning on each syntactic transformation. Xs indicate mean accuracies across epochs. T5 models are generally better at performing syntactic transformations than BART models. Monolingual models tend to achieve higher accuracies than multilingual models. We present full learning curves in Appendix B. glish question formation, achieving an average 59% main auxiliary accuracy throughout fine-tuning. However, it does achieve a maximum accuracy >90%, indicating that it is capable of hierarchical generalization after observing certain training examples. These accuracies are still well above the ≈0% accuracies of the non-pre-trained models.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Because sequence accuracy on the generalization set is often unstable for all pre-trained models, we present plots showing the distribution of accuracies sampled at every 500 fine-tuning iterations throughout 10 epochs of fine-tuning ( Figure 2). Each pretrained model learns the in-distribution transformation before the first 500 iterations of fine-tuning, so each plotted accuracy can be taken as indicative of model preferences after they have learned the transformations. (m)T5's sequence accuracies are generally close to 100% for all transformations except German passivization; this is far better than the non-pre-trained models' 0% sequence accuracies. (m)BART struggles more with syntactic transformations as indicated by its lower average accuracies, though it is still capable of detecting the correct auxiliaries and objects to move as indicated by the high maximum main auxiliary and object noun accuracies in Figure 2. This indicates that pre-trained seq2seq models demonstrate a hierarchical inductive bias, and that they can quickly learn syntactic transformations.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
There are two main differences between the two classes of models we test: (m)T5 and (m)BART are not only pre-trained, but are also much deeper and much more parameterized than our non-pre-trained models. Are hierarchical inductive biases a feature of deep architectures, then, or are they acquired during pre-training? To control for pre-training while keeping the model size consistent, we randomize the weights of mT5 (the better-performing model)  and fine-tune for up to 500 epochs using an initial LR 8 of 5 × 10 −4 . For all of the transformations, the maximum accuracies of the randomized models are much lower than the average accuracies of the pre-trained models (Table 5), which suggests that the deeper architecture on its own does not lead to structure-sensitive generalizations. This in return indicates that pre-trained models do not start with a hierarchical inductive bias; they acquire it through pre-training, extending the findings of  to generative sequence-to-sequence models. However, as indicated by the non-zero main auxiliary/object noun accuracies, the randomly initialized mT5 models do not exhibit a consistent linear generalization either-unlike the 1-layer non-pre-trained models. This may be due to the large number of parameters compared to the size of the transformations training corpus. A randomly initialized model of this size would likely need orders of magnitude more training data to learn stable generalizations.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Each pre-trained model almost always chooses the correct auxiliary/object to move; what errors account for their sub-perfect sequence accuracies, then? We perform a detailed error analysis, finding that pre-trained models drop PPs from the second noun phrase but otherwise perform many complex hierarchy-sensitive transformations properly. See Appendix C for details.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Our results indicate that pre-trained seq2seq models can consistently perform hierarchy-sensitive transformations. What strategy do they follow to do this? Because pre-training corpora include actives, passives, declaratives, and questions, model representations could encode these high-level sentence features. 9 Thus, one strategy could be to learn a mapping between abstract representations of different sentence structures (REPRESENTATION strategy). Alternatively, models could learn to correctly identify the relevant syntactic units in the input, and then learn a "recipe" of steps leading to the correct transformations (RECIPE strategy).
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
To distinguish which strategy models use to perform syntactic transformations, we observe crosslingual zero-shot transfer on syntactic transformations. We exploit that English and German use the same operations for question formation, whereas passivization in German involves the additional steps of case reinflection and moving the main verb. If structural representations are shared across English and German, 10 we do not expect divergent behaviors for question formation and passivization: if a model employs the REPRESENTATION strategy, then after fine-tuning on only English passivization, it should also correctly perform German passivization, including the additional steps of case reinflection and moving the main verb. Conversely, if it employs the RECIPE strategy, we expect a model trained on English passivization to only perform the steps that are required for English passivization, resulting in incorrect case marking and no main verb movement in German.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We first verify that mT5 and mBART are capable of cross-lingual transfer by training a model on the English question formation task and evaluating on German. In early experiments, we noticed the issue of "spontaneous translation" (Xue et al., 2021); we therefore also include German identity transformations in the training data to train the decoder to also output German sentences.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
As the top two panels of Figure 3 show, mT5 can correctly perform German question formation on in-domain structures (RCs on objects) after being exposed only to English transformations. For out-of-domain structures (RCs on subjects), mT5 almost always moves the main auxiliary but almost never deletes it from its original position, resulting in lower sequence accuracies. Apart from this error, the model is capable of cross-lingual transfer on the question formation task. By contrast, mBART achieves poor results on zero-shot German question formation, so we cannot make conclusive arguments using this approach; see Appendix D.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Given that cross-lingual transfer is possible for mT5, how does the model behave in the passivization task, which differs between English and German? We fine-tune mT5 on English passivization (as well as German identity transformations on active sentences). The results of this experiment (the lower two panels in Figure 3) show that the model is still able to move the main object to the subject position, but also that it never correctly performs German passivization in its entirety. This is because the model performs exactly the same steps for German sentences as for English sentences, resulting in outputs with English syntax: These behavioral patterns suggest that mT5 employs the RECIPE strategy: it succeeds if a transformation's required operations are the same across languages (as for question formation) but fails if the steps differ (as for passivization). Even in passivization, however, the model still learns to move the correct NPs, which provides additional evidence that mT5 makes use of structural features when performing transformations. Given the similarities between mT5's and mBART's architectures and training setups, one could reasonably presume that mBART may follow a similar strategy to perform syntactic transformations; nonetheless, mBART is less consistent in performing syntactic transformations, so this method cannot present strong evidence for use of the RECIPE strategy for that model.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Pre-trained models learn to use hierarchical features for performing syntactic transformations. Is this because there is explicit supervision for the hierarchical rules in the pre-training corpora? In other words, are there disambiguating examples in these models' training corpora that helps them memorize hierarchical transformation patterns? Here, we focus on English question formation examples in mT5's training corpus. 11 Disambiguating examples would be rare, as a single pre-training context window must contain a declarative sentence as well as the same sentence transformed into a question; humans would tend to replace at least some of the constituents with pronouns or delete them (e.g., Ariel, 2001). It would also require the MOVE-FIRST rule to not correctly transform the sentence-and for at least one of the auxiliaries to be noised in one sentence but not the other, such that the auxiliary has to be recovered from the other sentence. For example: We examine each pair of adjacent sentences in each document, manually inspecting any sentence pair meeting the following criteria: (1) the token Jaccard similarity of the sentences is > 0.7; (2) one sentence begins with an auxiliary verb and the other does not;
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
(3) there are at least two distinct auxiliaries in both sentences. There are 277 sentence pairs in our sample that met all criteria, of which 13 are adjacent declarative/question pairs that are equivalent except for the fronted auxiliary. Thus, the probability of an equivalent declarative/question pair with two auxiliaries in mC4 is ≈ 1.1 × 10 −7 . As T5's and mBART's training corpora consist of data from similar webtext distributions, it is likely that these structures exist in those corpora as well.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Crucially, however, none of the declarative/question pairs were disambiguating examples: each pair was consistent with the linear MOVE-FIRST rule. What is the probability of a disambiguating example, then? If we assume that the probability of a sentence containing an RC on the subject is independent from the probability of a declarative/question sentence pair, we can take the product of both probabilities to obtain an estimate. From the same sample of 118.3M sentences, we use spaCy's dependency parser to extract sentences containing an RC on the subject and where at least one auxiliary verb appears in the sentence. We obtain 526, 944 such sentences, meaning that the probability of an RC on a subject in an auxiliary-containing sentence in mC4's English corpus is ≈ 4.5 × 10 −3 . Thus, the probability of declarative/question pair with an RC on the subject and auxiliary in the RC is ≈ (4.5×10 −3 )·(1.1×10 −7 ) = 4.95×10 −10 . mT5 is trained on up to 1T tokens of data, and 5.67% of its documents are English; it therefore observes ≈ 56.7B English tokens. If we optimistically assume that English sentences contain an average of 15 tokens, it observes 3.78B English sentences. Then we would expect 3.78B×(4.95 × 10 −10 ) ≈ 2 disambiguating examples. This is not including the auxiliary masking criterion, which would make such examples even less likely.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Thus, while we cannot definitively rule out the 12 https://spacy.io possibility of disambiguating examples in mC4, they are rare if they exist in the corpus at all. Nonetheless, we have found evidence for supervision on question formation in the form of adjacent declarative/question sentence pairs, even if they do not explicitly support the hierarchical rule.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Our experiments provide evidence that pre-trained seq2seq models acquire a hierarchical inductive bias through exposure to non-annotated natural language text. This extends the findings of Warstadt and Bowman (2020) and  to a more challenging generative task, where models cannot rely on n-gram distributional heuristics (Kam et al., 2008). This also provides additional evidence that masking and reconstructing subsets of input sequences is a powerful training objective for inducing linguistic generalizations, whether in masked language models like RoBERTa  or sequence-tosequence models. Span denoising ((m)T5's objective) appears more effective for learning syntactic transformations than full sequence reconstruction ((m)BART's objective) given that (m)T5 is more consistently able to perform transformations, though there are too many other differences in training data and hyperparameters between (m)T5 and (m)BART for us to be able to directly implicate the training objective. This hypothesis can be tested explicitly in future work by training identical models that differ only in their pre-training objective. Counter to McCoy et al. (2020), our findings suggest that hierarchical architectural constraints (e.g., tree-structured networks) are not necessary for robust hierarchical generalization as long as the model has been exposed to large amounts of natural language text-possibly far more language than humans would be exposed to. However, one difference between the randomly initialized models employed by McCoy et al. (2020) and pre-trained models is that pre-trained models have likely seen the structures (but not sentences) present in the generalization set; thus, rather than relying on syntactic features, the model could choose the correct transformation because it is more similar to the grammatical examples it has already seen. We found declarative/question pairs in mT5's training corpus, but we did not find any examples that explicitly demonstrated the hierarchical rule for question formation. While we cannot fully rule out the possibil-ity of disambiguating examples, this strategy is still unlikely given that pre-trained models produce ungrammatical transformations, both in monolingual transformations (e.g., not deleting the main auxiliary after copying it to the start of the sentence) and in cross-lingual German passivization. Additionally, because we use greedy decoding, models are not able to take future words into account when predicting the fronted auxiliary: they must select the appropriate auxiliary to move solely based on the encoder's representations.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
More broadly, our findings counter the assumption that a hierarchical constraint is necessary in language learners to acquire hierarchical generalization (Chomsky, 1965). While the pre-trained models that we considered observe far more input than a child would receive , Huebner et al. (2021) recently demonstrated high performance on grammaticality judgments for models trained on much smaller child-directed speech corpora, suggesting that our findings may also hold when training models on more human-like input.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We have performed an analysis of the syntactic transformational ability of large pre-trained sequence-to-sequence models. We find that pretrained models acquire a hierarchical inductive bias during pre-training, and that the architecture does not yield this hierarchical bias by itself.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
It remains an open question whether such deep and highly parameterized models or such large pretraining datasets are necessary for hierarchical generalization. Future work could ablate over model depth and pre-training corpus size to observe the relative contribution of architecture and the training set to inducing hierarchical inductive biases in seq2seq models.   . Figure 6: Learning curves over 10 epochs of fine-tuning for T5 on both syntactic transformation tasks. in these curves are the same as those shown in Figure 2, but now associated with their respective fine-tuning iteration. All models except mBART immediately achieve near-perfect main auxiliary and object noun accuracies. Their loss on the validation sets converges almost immediately, so it's possible that reductions in generalization accuracies throughout fine-tuning are due to overfitting to the training distribution. For mBART, however, main auxiliary and object noun accuracies start high and then begin to vary dramatically throughout fine-tuning. This is perhaps due to quick overfitting on the training distribution. We analyze what errors cause mBART's deficiencies in §C.2.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Each pre-trained model almost always chooses the correct auxiliary/object to move; what other errors account for their sub-perfect sequence accuracies? We implement more specific metrics to observe more closely what mistakes (m)T5 and (m)BART are making. We show results for mT5 in §C.1 and mBART in §C.2, but the errors we discuss are generally consistent across models. We also present more detailed metrics for the most complex transformation, German passivization, in §C.3. achieve the lowest sequence accuracy. mT5 is almost always successful at the hierarchical transformation of moving the object NP to subject position (including its attached PP when present), and it correctly moves the original subject noun to a "by" phrase following the auxiliary. However, for both English and German passivization, the main error accounting for sub-perfect sequence accuracies is that the model fails to preserve the PP on the second NP (in the by-phrase):
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
(12) My yaks below the unicorns comforted the orangutans. → The orangutans were comforted by my yaks.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
As mT5 has not been fine-tuned on output sequences where PPs appear at the end of the sentence, the decoder could be assigning low probabilities to end-of-sentence PPs while otherwise encoding a hierarchical analysis of sentence structure.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Errors for question formation are more varied. Pre-trained models' sub-perfect main auxiliary accuracies on question formation are mainly due to improper negations on the main auxiliary: when the noun in the relative clause and the main noun agree in number, models will sometimes delete the main auxiliary (as expected) while copying the incorrect auxiliary to the beginning of the sentence. Additionally, the discrepancy between sequence and main auxiliary accuracies is almost always attributable to models not deleting the main auxiliary after moving it to the start of the sentence. These results (as with the passivization results) suggest that pre-trained seq2seq models are better at performing hierarchy-sensitive transformations than the sequence accuracies initially suggestbut also that they can fail to perform theoretically simpler operations, such as deletions and moving all parts of a constituent.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We present more detailed error analyses in Appendix C.3. We find that pre-trained models also consistently succeed in case reinflection, tense reinflection, and passive auxiliary insertion.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We have shown in §C.1 that mT5 achieves subperfect sequence accuracies on passivization due to its dropping the prepositional phrase on the second NP. Here, we present results for mBART ( Figure 9). The takeaways for mBART are similar to mT5's: the model succeeds in moving the proper nouns, but it often drops the prepositional phrase from the second NP during movement.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
As the model also fails to perform English question formation consistently, we also observe what errors it makes in that task. We find that deficiencies in main auxiliary accuracy are due to the model copying the incorrect auxiliary to the beginning of the sentence, while gaps between main auxiliary and sequence accuracy are due to the model dropping the relative clause on the second NP.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
We also present more detailed analyses of other required operations in passivization: namely, are mT5 and mBART capable of tense reinflection, case reinflection, and auxiliary insertions? And are they capable of this in zero-shot settings? Results for mT5 ( Figure 10) and mBART ( Figure 11) suggest that both models are generally capable of tense reinflection, case reinflection, and auxiliary insertion in supervised contexts.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Here, we present learning curves for mBART on zero-shot cross-lingual syntactic transformations ( Figure 12). While mBART is typically able to select the correct auxiliary verb or object noun to move, it never transforms the sequence fully correctly. Figure 9: Learning curves displaying alternative accuracy metrics for mBART on German passivization. We present the accuracy of the model in properly moving the object NP to the start of the sentence (top left), moving the subject NP after the auxiliary verb (top right), moving the subject NP after the auxiliary verb with or without its attached PP (bottom left), and the full sequence accuracy (bottom right).   
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Pretrained masked language models (Devlin et al., 2019;Liu et al., 2019b) have benefitted a wide range of natural language processing (NLP) tasks (Liu, 2019;Wadden et al., 2019;Zhu et al., 2020). Despite recent progress in understanding what useful information is captured by MLMs (Liu et al., 2019a;Hewitt and Manning, 2019), it remains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
One popular justification of MLMs relies on viewing masking as a form of cloze reduction.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Cloze-like Masking I this movie [MASK] Dependency Learning I like this movie Figure 1: We study the inductive bias of MLM objectives and show that cloze-like masking (left) does not account for much of the downstream performance gains. Instead, we show that MLM objectives are biased towards extracting both statistical and syntactic dependencies using random masks (right).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Cloze reductions reformulate an NLP task into a prompt question and a blank and elicit answers by filling in the blank (Figure 1). When tested by cloze reductions pretrained MLMs and left-to-right language models (LMs) have been shown to possess abundant factual knowledge (Petroni et al., 2019) and display impressive few-shot ability (Brown et al., 2020). This success has inspired recent hypotheses that some word masks are cloze-like and provide indirect supervision to downstream tasks Lee et al., 2020). For example, a sentiment classification task (Pang et al., 2002) can be reformulated into filling in like or hate in the cloze I [MASK] this movie. Such cloze-like masks provide a clear way in which an MLM can implicitly learn to perform sentiment classification.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
While this hypothesis is appealing, MLMs in practice are trained with uniform masking that does not contain the special structure required by cloze-like masks most of the time. For example, predicting a generic word this in the cloze I like [MASK] movie would not offer task-specific supervision. We quantify the importance of cloze-like and generic masks by explicitly creating cloze-like masks using task-specific lexicons and comparing models pretrained on these masks. These experiments suggest that although cloze-like masks can be helpful, the success of uniform masking cannot be explained via cloze-like masks alone. In fact, we demonstrate that uniform masking performs as well as a negative control where we explicitly remove cloze-like masks from the mask distribution.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To address this mismatch between theory and practice, we offer a new hypothesis of how generic masks can help downstream learning. We propose a conceptual model for MLMs by drawing a correspondence between masking and graphical model neighborhood selection (Meinshausen and Bühlmann, 2006). Using this, we show that MLM objectives are designed to recover statistical dependencies in the presence of latent variables and propose an estimator that can recover these learned dependencies from MLMs. We hypothesize that statistical dependencies in the MLM objective capture useful linguistic dependencies and demonstrate this by using recovered statistical dependencies to perform unsupervised parsing, outperforming an actual unsupervised parsing baseline (58.74 vs 55.91 UUAS; Klein and Manning, 2004). We release our implementation on Github 1 .
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Theories inspired by Cloze Reductions. Cloze reductions are fill-in-the-blank tests that reformulate an NLP task into an LM problem. Existing work demonstrates that such reductions can be highly effective for zero/few-shot prediction (Radford et al., 2019;Brown et al., 2020) as well as relation extraction (Petroni et al., 2019;.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
These fill-in-the-blank tasks provide a clear way by which LMs can obtain supervision about downstream tasks, and recent work demonstrates how such implicit supervision can lead to useful representations . More general arguments by Lee et al. (2020) show these theories hold across a range of self-supervised settings. While these theories provide compelling arguments for the value of pre-training with cloze tasks, they do not provide a clear reason why uniformly random masks such as those used in BERT provide such strong gains. In our work, we quantify this gap using lexicon-based cloze-like masks and show that cloze-like masks alone are unlikely to account for the complete success of MLM since generic and non-cloze masks are responsible for a substantial part of the empirical performance of MLMs. 1 https://github.com/tatsu-lab/mlm_ inductive_bias Theories for vector representations. Our goal of understanding how masking can lead to useful inductive biases and linguistic structures is closely related to that of papers studying the theory of word embedding representations (Mikolov et al., 2013;Pennington et al., 2014;Arora et al., 2015). Existing work has drawn a correspondence between word embeddings and low-rank factorization of a pointwise mutual information (PMI) matrix (Levy and Goldberg, 2014) and others have shown that PMI is highly correlated with human semantic similarity judgements (Hashimoto et al., 2016).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
While existing theories for word embeddings cannot be applied to MLMs, we draw inspiration from them and derive an analogous set of results. Our work shows a correspondence between MLM objectives and graphical model learning through conditional mutual information, as well as evidence that the conditional independence structure learned by MLMs is closely related to syntactic structure.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Probing Pretrained Representations. Recent work has applied probing methods (Belinkov and Glass, 2019) to analyze what information is captured in the pretrained representations. This line of work shows that pretrained representations encode a diverse range of knowledge (Peters et al., 2018;Tenney et al., 2019;Liu et al., 2019a;Hewitt and Manning, 2019;. While probing provides intriguing evidence of linguistic structures encoded by MLMs, they do not address the goals of this work, which is how the pretraining objective encourages MLMs to extract such structures.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Masked Language Modeling asks the model to predict a token given its surrounding context. Formally, consider an input sequence X of L tokens x 1 , . . . , x L where each variable takes a value from a vocabulary V. Let X ∼ D be the data generating distribution of X. Let x i be the ith token in X, and let X \i denote the sequence after replacing the ith token with a special [MASK] token. In other words,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Similarly, define X \{i,j} as replacing both x i and  Figure 2: In our case study, we append the true label to each input and create ideal cloze-like masks. We study how deviations from the ideal mask distribution affect downstream performance by adding in generic masks. that minimizes
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In BERT pretraining, each input token is masked with a fixed, uniform probability, which is a hyperparameter to be chosen. We refer to this strategy as uniform masking.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Finetuning is the canonical method for using pretrained MLMs. Consider a prediction task where y ∈ Y is the target variable, e.g., the sentiment label of a review. Finetuning uses gradient descent to modify the pretrained parameters θ and learn a new set of parameters φ to minimize
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where p(y|x) is the ground-truth distribution and D is the data distribution of the downstream task. Our goals. We will study how the mask distribution M affects downstream performance. We define perfect cloze reductions as some partition of the vocabulary V y such that p(x i ∈ V y |X \i ) ≈ p(y|X). For a distribution M such that the masks we draw are perfect cloze-reductions, the MLM objective offers direct supervision to finetuning since L MLM ≈ L finetune . In contrast to cloze-like masking, in uniform masking we can think of p θ as implicitly learning a generative model of X (Wang and Cho, 2019). Therefore, as M moves away from the ideal distribution and becomes more uniform, we expect p θ to model more of the full data distribution D instead of focusing on cloze-like supervision for the downstream task. This mismatch between theory and practice raises questions about how MLM with uniform masking can learn useful inductive biases.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
When L MLM is not L finetune , what is L MLM learning? We analyze L MLM and show that it is similar to a form of conditional mutual information based graphical model structure learning.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To motivate our subsequent discussions, we perform a controlled study for the case when L MLM ≈  L finetune and analyze how deviations from the ideal mask distribution affect downstream performance. We perform analysis on the Stanford Sentiment Treebank (SST-2; Socher et al., 2013), which requires models to classify short movie reviews into positive or negative sentiment. We append the ground-truth label (as the word positive or negative) to each movie review ( Figure 2). Masking the last word in each review is, by definition, an ideal mask distribution. To study how the deviation from the ideal mask distribution degrades downstream performance, we vary the amount of cloze-like masks during training. We do this by masking out the last word for p% of the time and masking out a random word in the movie review for (100 − p)% of the time, and choose p ∈ {0, 20, 40, 60, 80, 100}. Experimental details. We split the SST-2 training set into two halves, use one for pretraining, and the other for finetuning. For the finetuning data, we do not append the ground-truth label. We pretrain small transformers with L MLM using different masking strategies and finetune them along with a baseline that is not pretrained (NOPRETRAIN). Further details are in Appendix A.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. We observe that while cloze-like masks can lead to successful transfer, even a small modification of the ideal mask distribution deteriorates performance. Figure 3 shows the development set accuracy of seven model variants averaged across ten random trials. We observe as p decreases, the performance of CLOZE-p% degrades. Notably, CLOZE-80% is already worse than CLOZE-100% and CLOZE-20% does not outperform NOPRE-TRAIN by much. We notice that CLOZE-0% in fact degrades finetuning performance, potentially because the pretrained model is over-specialized to the language modeling task (Zhang et al., 2020;Tamkin et al., 2020). While this is a toy example, we observe similar results for actual MLM models across three tasks (Section 5.1), and this motivates us to look for a framework that explains the success of generic masks in practice.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In the previous section, we saw that cloze-like masks do not necessarily explain the empirical success of MLMs with uniform masking strategies. Understanding uniform masking seems challenging at first, as uniform-mask MLMs seem to lack task-specific supervision and is distinct from existing unsupervised learning methods such as word embeddings (which rely upon linear dimensionality reduction) and autoencoders (which rely upon denoising). However, we show in this section that there is a correspondence between MLM objectives and classic methods for graphical model structure learning. As a consequence, we demonstrate that MLMs are implicitly trained to recover statistical dependencies among observed tokens.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our starting point is the observation that predicting a single feature (x i ) from all others (X \i ) is the core subroutine in the classic Gaussian graphical model structure learning algorithm of Meinshausen and Bühlmann (2006). In this approach, L different Lasso regression models are trained (Tibshirani, 1996) with each model predicting x i from X \i , and the nonzero coefficients of this regression correspond to the conditional dependence structure of the graphical model.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The MLM objective can be interpreted as a nonlinear extension of this approach, much like a classical algorithm that uses conditional mutual information (MI) estimators to recover a graphical model (Anandkumar et al., 2012). Despite the similarity, real world texts are better viewed as models with latent variables (e.g. topics; Blei et al., 2003) and many dependencies across tokens arise due to latent variables, which makes learning the direct dependencies difficult. We show that MLMs implicitly recover the latent variables and can capture the direct dependencies while accounting for the effect of latent variables. Finally, MLMs are only approximations to the true distribution and we show that the MLM objective can induce highquality approximations of conditional MI.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Analysis setup. To better understand MLMs as a way to recover graphical model structures, we show mask-based models can recover latent variables and the direct dependencies among variables in the Gaussian graphical model setting of Meinshausen and Bühlmann (2006). Let X = [x 1 , . . . , x L ] ∈ R L represent an input sequence where each of its coordinates x i represents a token, and Z ∈ R k be a latent variable that controls the sequence generation process. We assume that all coordinates of X are dependent on the latent variable Z, and there are sparse dependencies among the observed variables ( Figure 4). In other words, we can write Z ∼ N(0, Σ ZZ ) and X ∼ N(AZ, Σ XX ). Intuitively, we can imagine that Z represents shared semantic information, e.g. a topic, and Σ XX represents the syntactic dependencies. In this Gaussian graphical model, the MLM is analogous to regressing each coordinate of X from all other coordinates, which we refer to as masked regression.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM representations can recover latent variable. We now study the behavior of masked regression through the representation x mask,i that is obtained by applying masked regression on the ith coordinate of X and using the predicted values. Our result shows that masked regression is similar to the two-step process of first recovering the latent variable Z from X \i and then predicting x i from Z.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Let Σ XX,\i,i ∈ R d−1 be the vector formed by dropping the ith row and taking the ith column of Σ XX and β 2SLS,i be the linear map resulting from the two-stage regression X \i → Z → x i . Proposition 1. Assuming that Σ XX is full rank,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In other words, masked regression implicitly recovers the subspace that we would get if we first explicitly recovered the latent variables (β 2SLS,i ) with an error term that scales with the off-diagonal terms in Σ XX . The proof is presented in Appendix C.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
To give additional context for this result, let us consider the behavior of a different representation learning algorithm: PCA. It is well-known that PCA can recover the latent variables as long as the Σ ZZ dominates the covariance Cov(X). We state this result in terms of X PCA , the observed data projected to the first k components of PCA. Proposition 2. Let λ k be the kth eigenvalue of AΣ ZZ A and λ XX,k+1 be the k+1th eigenvalue of Σ XX and V be the first k eigenvectors of Cov(X). Assuming λ k > λ XX,k+1 , we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where · op is the operator norm and tr(·) is the trace.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This shows that whenever Σ XX is sufficiently small and λ k is large (i.e., the covariance is dominated by Z), then PCA recovers the latent information in Z. The proof is based on the Davis-Kahan theorem (Stewart and Sun, 1990) and is presented in Appendix C.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Comparing the bound of PCA and masked regression, both bounds have errors that scales with Σ XX , but the key difference in the error bound is that the error term for masked regression does not scale with the per-coordinate noise (diag(Σ XX )) and thus can be thought of as focusing exclusively on interactions within X. Analyzing this more carefully, we find that Σ XX,\i,i corresponds to the statistical dependencies between x i and X \i , which we might hope captures useful, task-agnostic structures such as syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM log-probabilies can recover direct dependencies. Another effect of latent variables is that many tokens have indirect dependencies through the latent variables, which poses a challenge to recovering the direct dependencies among tokens. We now show that the MLMs can account for the effect of latent variable.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In the case where there are no latent variables, we can identify the direct dependencies via conditional MI (Anandkumar et al., 2012) because any x i and x j that are disconnected in the graphical model will have zero conditional MI, i.e., I(x i ; x j |X \{i,j} ) = 0. One valuable aspect of MLM is that we can identify direct dependencies even in the presence of latent variables.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
If we naively measure statistical dependency by mutual information, the coordinates of X would appear dependent on each other because they are all connected with Z. However, the MLM objective resolves this issue by conditioning on X \{i,j} . We show that latent variables (such as topics) that are easy to predict from X \{i,j} can be ignored when considering conditional MI. Proposition 3. The gap between conditional MI with and without latent variables is bounded by the conditional entropy H(Z|X \{i,j} ),
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This suggests that when the context X \{i,j} captures enough of the latent information, conditional MI can remove the confounding effect of the shared topic Z and extract the direct and sparse dependencies within X (see Appendix C for the proof).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
MLM objective encourages capturing conditonal MI. We have now shown that conditional MI captures direct dependencies among tokens, even in the presence of latent variables. Next, we will show that the MLM objective ensures that a LM with low log-loss accurately captures the conditional MI. We now show that learning the MLM objective implies high-quality estimation of conditional MI. Denote X(i, v) as substituting x i with a new token v,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Conditional MI is defined as the expected pointwise mutual information (PMI) conditioned on the rest of the tokens,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where I p is the abbreviation of I p (x i ; x j |X \{i,j} ). Our main result is that the log-loss MLM objective directly bounds the gap between the true conditional mutual information from the data distribution and an estimator that uses the log-probabilities from the model. More formally, Proposition 4. Let
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
be an estimator constructed by the model distribution p θ . Then we can show,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where D kl represents the KL-divergence.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Here, the KL-divergence corresponds to the L MLM objective, up to a constant entropy term that depends on p. We present the proof in Appendix C. In other words, the MLM objective is implicitly encouraging the model to match its implied conditional MI to that of the data. We now use this result to create an estimator that extracts the conditional independence structures implied by MLM.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our earlier analysis in Proposition 4 suggests that an MLM with low loss has an accurate approximation of conditional mutual information. Using this result, we will now propose a procedure which estimatesÎ p θ . The definition ofÎ p θ shows that if we can access samples of x i and x j from the true distribution p, then we can directly estimate the conditional mutual information by using the log probabilities from the MLM. Unfortunately, we cannot draw new samples of x j | X \{i,j} , leading us to approximate this distribution using Gibbs sampling on the MLM distribution. Our Gibbs sampling procedure is similar to the one proposed in Wang and Cho (2019). We start with X 0 = X \{i,j} . For the tth iteration, we draw a sample x t i from p θ (x i |X t−1 \i ), and update by X t = X t−1 (i, x t i ). Then, we draw a sample x t j from p θ (x j |X t \j ) and set X t = X t (j, x t j ). We repeat and use the samples (x 1
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
i , x 1 j ), . . . , (x t i , x t j ) to compute the expectations for conditional MI.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
This procedure relies upon an additional assumption that samples drawn from the MLM are faithful approximations of the data generating distribution. However, we show empirically that even this approximation is sufficient to test the hypothesis that the conditional independences learned by an MLM capture syntactic dependencies (Section 5.2).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We now test two predictions from our analyses. First, similar to our observation in the case study, we show that cloze-like masks do not explain the success of uniform masks on three real-world datasets. Second, our alternative view of relating MLM to graphical models suggests that statistical dependencies learned by MLMs may capture linguistic structures useful for downstream tasks. We demonstrate this by showing that MLMs' statistical dependencies reflect syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Setup. We now demonstrate that real-world tasks and MLMs show a gap between task-specific cloze masks and random masks. We compare the MLM with random masking to two different control groups. In the positive control (CLOZE), we pretrain with only cloze-like masks and in the negative control (NOCLOZE), we pretrain by explicitly excluding cloze-like masks. If the success of MLM can be mostly explained by implicit cloze reductions, then we should expect CLOZE to have strong downstream performance while NOCLOZE leads to a minimal performance gain. We compare pretraining with the uniform masking strategy used in BERT (UNIFORM) to these two control groups. If UNIFORM performs worse than the positive control and more similar to the negative control, then we know that uniform masking does not leverage cloze-like masks effectively.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Simulating Pretraining. Given computational constraints, we cannot retrain BERT from scratch. Instead, we approximate the pretraining process by continuing to update BERT with MLM (Gururangan et al., 2020), which we refer to as second-stage pretraining. Although this is an approximation to the actual pretraining process, the second-stage pretraining shares the same fundamental problem for pretraining: how can unsupervised training lead to downstream performance gains?
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study the effectiveness of different masking strategies by comparing to a BERT model without second-stage pretraining (VANILLA). We experiment with three text classification datasets: SST-2 (Socher et al., 2013), Hyperpartisan (Kiesel et al., 2019), and AGNews (Zhang et al., 2015). SST-2 classifies movie reviews by binary sentiment; Hyperpartisan is a binary classification task on whether a news article takes an extreme partisan standpoint; and AGNews classifies news articles into four different topics. On SST-2 and AGNews, we perform the second-stage pretraining on the training inputs (not using the labels). On Hyperpartisan, we use 100k unlabeled news articles that are released with the dataset. For SST-2 and AG-News, we study a low-resource setting and set the number of finetuning examples to be 20. For Hyperpartisan, we use the training set, which has 515 labeled examples. All evaluations are performed by fine-tuning a bert-base-uncased model (See Appendix A for full details).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Approximating Cloze-like Masking. We cannot identify the optimal set of cloze-like masks for an arbitrary downstream task, but these three tasks have associated lexicons which we can use to approximate the cloze-like masks. For SST-2, we take the sentiment lexicon selected by Hu and Liu (2004); for Hyperpartisan, we take the NRC word-emotion association lexicon (Mohammad and Turney, 2013); and for AGNews, we extract topic words by training a logistic regression classifier and AGNews 20-shot taking the top 1k features to be cloze-like masks.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. Figure 5 plots the finetuning performance of different masking strategies. We observe that UNIFORM outperforms VANILLA, which indicates that second-stage pretraining is extracting useful information and our experiment setup is useful for studying how MLM leads to performance gains. As expected, CLOZE achieves the best accuracy, which confirms that cloze-like masks can be helpful and validates our cloze approximations.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The UNIFORM mask is much closer to NO-CLOZE than CLOZE. This suggests that uniform masking does not leverage cloze-like masks well and cloze reductions alone cannot account for the success of MLM. This view is further supported by the observation that NOCLOZE outperforms VANILLA suggesting that generic masks that are not cloze-like still contain useful inductive biases.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our results support our earlier view that there may be an alternative mechanism that allows generic masks that are not cloze-like to benefit downstream learning. Next, we will empirically examine BERT's learned conditional independence structure among tokens and show that the statistical dependencies relate to syntactic dependencies.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Our analysis in section 4.1 shows that conditional MI (which is optimized by the MLM objective) can extract conditional independences. We will show that statistical dependencies estimated by conditional MI are related to syntactic dependencies by using conditional MI for unsupervised parsing.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Background. One might expect that the statistical dependencies among words are correlated with syntactic dependencies. Indeed, Futrell et al. (2019) show that heads and dependents in dependency parse trees have high pointwise mutual information (PMI) on average. However, previous at-tempts (Carroll and Charniak, 1992;Paskin, 2002) show that unsupervised parsing approaches based on PMI achieve close to random accuracy. Our analysis suggests that MLMs extract a more finegrained notion of statistical dependence (conditional MI) which does not suffer from the existence of latent variables (Proposition 3). We now show that the conditional MI captured by MLMs achieves far better performance, on par with classic unsupervised parsing baselines.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Baselines. We compare conditional MI to PMI as well as conditional PMI, an ablation in which we do not take expectation over possible words. For all statistical dependency based methods (cond. MI, PMI, and cond. PMI), we compute pairwise dependence for each word pair in a sentence and construct a minimum spanning tree on the negative values to generate parse trees. To contextualize our results, we compare against three simple baselines: RANDOM which draws a random tree on the input sentence, LINEARCHAIN which links adjacent words in a sentence, and a classic unsupervised parsing method (Klein and Manning, 2004).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental Setup. We conduct experiments on the English Penn Treebank using the WSJ corpus and convert the annotated constituency parses to Stanford Dependency Formalism (de Marneffe et al., 2006). Following Yang et al. (2020), we evaluate on sentences of length ≤ 10 in the test split, which contains 389 sentences (Appendix B.1 describes the same experiment on longer sentences, which have similar results). We experiment with the bert-base-cased model (more details in Appendix A) and evaluate by the undirected unlabeled attachment score (UUAS).
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Results. Table 1 shows a much stronger-thanrandom association between conditional MI and dependency grammar. In fact, the parses extracted
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
The above represents a triumph of either apathy or civility .  Figure 6: An example parse extracted from conditional MI. The black parse tree above the sentence represents the ground-truth parse and the red parse below is extracted from conditional MI. The correctly predicted edges are labeled with the annotated relations, and the incorrect ones are labeled as wrong.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Method UUAS
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
28.50 ± 0.73 LINEARCHAIN 54.13 Klein and Manning (2004)  from conditional MI has better quality than LIN-EARCHAIN and the classic method (Klein and Manning, 2004). Unlike conditional MI, PMI only has a close-to-random performance, which is consistent with prior work. We also see that conditional MI outperforms conditional PMI, which is consistent with our theoretical framework that suggests that conditional MI (and not PMI) recovers the graphical model structure.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We also perform a fine-grained analysis by investigating relations where conditional MI differs from LINEARCHAIN. Because the test split is small and conditional MI does not involve any training, we perform this analysis on 5,000 sentences from the training split. Table 2 presents the results and shows that conditional MI does not simply recover the linear chain bias. Meanwhile, we also observe a deviation between conditional MI and dependency grammar on relations like number and cc. This is reasonable because certain aspects of dependency grammar depend on human conventions that do not necessarily have a consensus (Popel et al., 2013). Figure 6 illustrates with an example parse extracted from conditional MI. We observe that conditional MI correctly captures dobj and conj. Knowing the verb, e.g. represents, limits the range of objects that can appear in a sentence so intuitively we expect a high conditional MI between the direct object and the verb. Similarly for phrases like "A and B", we would expect A and B to be statistically dependent. However, conditional MI fails  to capture cc (between apathy and or). Instead, it links or with either which certainly has statistical dependence. This once again suggests that the 'errors' incurred by the conditional PMI method are not simply failures to estimate dependence but natural differences in the definition of dependence.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We study how MLM with uniform masking can learn useful linguistic structures and inductive biases for downstream tasks. Our work demonstrates that a substantial part of the performance gains of MLM pretraining cannot be attributed to taskspecific, cloze-like masks.  
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental details for Section 3.2 Our transformers have 2 layers and for each transformer block, the hidden size and the intermediate size are both 64. We finetune the models for 10 epochs and apply early stopping based on validation accuracy. We use Adam (Kingma and Ba, 2014) for optimization, using a learning rate of 1e −3 for pretraining and 1e −4 for finetuning. Experimental details for Section 5.1 Table 3 summarizes the dataset statistics of three real-world datasets we studied. For second stage pretraining, we update the BERT model for 10 epochs. Following the suggestion in Zhang et al. (2020), we finetune the pretrained BERT models for 400 steps, using a batch size of 16 and a learning rate of 1e −5 . We apply linear learning rate warmup for the first 10% of finetuning and linear learning rate decay for the rest. For SST-2 and AGNews, we average the results over 20 random trials. For Hyperpartisan, because the test set is small and the variation is larger, we average the results over 50 random trials and evaluate on the union the development set and the test set for more stable results.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Experimental details for Section 5.2 We convert the annotated constituency parses using the Stanford CoreNLP package . We compute conditional MI and conditional PMI using the bert-base-cased model and run Gibbs sampling for 2000 steps. BERT's tokenization may split a word into multiple word pieces. We aggregate the dependencies between a word and multiple word pieces by taking the maximum value. We compute the PMI statistics and train the K&M model (Klein and Manning, 2004) on sentences of length ≤ 10 in the WSJ train split (section 2-21). For DMV, we train with the annotated POS tags using a public implementation released by (He et al., 2018). Results are averaged over three runs when applicable.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We conduct an additional experiment on the English Penn Treebank to verify that conditional MI can extract parses for sentences longer than ten words. To expedite experimentation, we subsample 200 out of 2416 sentences from the test split of English Penn Treebank and the average sentence length of our subsampled dataset is 24.1 words. When applicable, we average over three random seeds and report standard deviations. Table 4 presents the UUAS of conditional MI and other methods. We draw similar conclusions as in Section 5.2, observing that the parses drawn by conditional MI have higher quality than those of other baselines.   
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Proof of Proposition 2 We first recall our statement.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Proposition 2. Let λ k be the kth eigenvalue of AΣ ZZ A and λ XX,k+1 be the k+1th eigenvalue of Σ XX and V be the first k eigenvectors of Cov(X). Assuming λ k > λ XX,k+1 , we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
where · op is the operator norm and tr(·) is the trace.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We will use the Davis-Kahan Theorem for our proof.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Theorem (Davis-Kahan (Stewart and Sun, 1990)). Let σ be the eigengap between the kth and the k+1th eigenvalue of two positive semidefinite symmetric matrices Σ and Σ . Also, let V and V be the first k eigenvectors of Σ and Σ respectively. Then we have,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
That is, we can bound the error in the subspace projection in terms of the matrix perturbation.
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
In our setting, we choose Σ = AΣ ZZ A + Σ XX and Σ = AΣ ZZ A . We know the eigengap of Σ is λ k because Σ only has k nonzero eigenvalues. By Weyl's inequality, the kth eigenvalue is at most perturbed by λ XX,k+1 , which is the k+1 eigenvalue of Σ XX . Let V be the top k eigenvectors of Σ and assuming λ k > λ XX,k+1 , we have,
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Turning this operator norm bound into approximation bound, we have
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
We use the fact that E X,Z AZ − X 2 2 = tr(Σ XX ) and Jensen's inequality to bound, E X AZ − X 2 ≤ tr(Σ XX ).
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Speakers of a language can generalize from finite linguistic experience to sentences they have never heard or produced before. Although there are many possible ways to generalize from a set of sentences, language learners consistently choose certain generalizations over others. In the syntactic domain, learners typically learn generalizations that appeal to hierarchical structures rather than linear order. An influential explanation for this fact is that learners never entertain hypotheses based on linear order: they are innately constrained to assume that syntactic rules are structure-sensitive (Chomsky, 1980).
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
To test whether a structure-sensitivity constraint is necessary to account for the generalizations that human language learners make, we use recurrent neural networks (RNNs), which are not equipped with such an explicit pre-existing hierarchical constraint. 1 We simulate the acquisition of English subject-auxiliary inversion, the transformation that turns a declarative statement such as (1a) into a question such as (1b):
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(1) a. My walrus can giggle.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
b. Can my walrus giggle? 1 In fact, RNNs are not just capable of using non-hierarchical structures but in fact appear to be biased in favor of linear structures over hierarchical ones (Christiansen & Chater, 1999 Linear rule: Move the linearly first auxiliary to the front of the sentence.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
While both rules account for common cases such as (1), they make different predictions for complex sentences such as (2): (2) My walrus that will eat can giggle.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Specifically, the hierarchical rule predicts the correct question (3a), while the linear rule predicts the incorrect question (3b):
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(3) a. Can my walrus that will eat giggle? b. * Will my walrus that eat can giggle?
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Although such examples disambiguate the two hypotheses, Chomsky (1971) argues that they are highly infrequent, and thus children may never encounter them. Without these critical examples, according to Chomsky, children can only acquire the hierarchical rule by drawing on an innate constraint stipulating that syntactic rules must appeal to hierarchy. This argument, known as the argument from the poverty of the stimulus (Chomsky, 1980), has been challenged in a number of ways. Some have disputed the assumption that children never encounter critical cases such as (3a) (Pullum & Scholz, 2002). Others have questioned the assumption that an explicit hierarchical constraint is necessary for hierarchical generalization. One such approach has been to argue that the hierarchical rule can fall out of weaker or non-syntactic structural biases. For example, Perfors, Tenenbaum, and Regier (2011) showed that a learner whose task is to choose between an innately available hierarchical representation and an innately available linear representation will choose the hierarchical one; and Fitz and Chang (2017) argued that the hierarchical structure of questions is rooted in innately available structured semantic representations.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
T Training set, test set T Generalization set IDENT QUEST No RC Input: the newt can confuse my yak by the zebra .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Output: the newt can confuse my yak by the zebra .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Input: the newt can confuse my yak by the zebra . Output: can the newt confuse my yak by the zebra ? RC on object Input: the newt can confuse my yak who will sleep . Output: the newt can confuse my yak who will sleep .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Input: the newt can confuse my yak who will sleep . Output: can the newt confuse my yak who will sleep ? RC on subject Input: the newt who will sleep can confuse my yak .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Output: the newt who will sleep can confuse my yak .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Input: the newt who will sleep can confuse my yak .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Output: can the newt who will sleep confuse my yak ? Table 1: Examples for each combination of a sentence type and a task. RC stands for "relative clause."
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
A second approach has dispensed with pre-existing structural representations altogether. Lewis and Elman (2001) argued that an RNN trained to predict the next word can learn which questions are well formed, but this conclusion was convincingly called into question by Kam, Stoyneshka, Tornyova, Fodor, and Sakas (2008). The most immediate precursor to our work is Frank and Mathis (2007). Like Lewis and Elman, they used RNNs, but instead of modeling the wellformedness of the question alone, they followed the traditional framework of transformational grammar in modeling the generation of a question from a declarative sentence. 2 Their results were difficult to interpret because the network's generalization behavior depended heavily on the identity of the auxiliaries in the input sentence, and neither the linear hypothesis nor the hierarchical hypothesis predict such lexically dependent behavior. We significantly expand on their experiments, taking advantage of recent technological and architectural advances in RNNs that have shown promise in the acquisition of syntax (Linzen, Dupoux, & Goldberg, 2016).
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
To anticipate our results, of the six RNN architectures we explored, one of the architectures consistently learned a hierarchical generalization for question formation. This suggests that a learner's preference for hierarchy may arise from the hierarchical properties of the input, coupled with biases implicit in the network's computational architecture and learning procedure, without the need for pre-existing hierarchical constraints in the learner. We provide further evidence for the role of the hierarchical properties of the input by showing that adding syntactic agreement to the input increased the probability that a network would make hierarchical generalizations.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The networks were trained on two fragments of English, each consisting of a subset of all possible declarative sentences and questions. 3 We refer to the first fragment as the no-2 This is a simplification-a more psychologically plausible assumption would be that questions are generated from a semantic representation shared with the declarative sentence (Fitz & Chang, 2017). 3 The vocabulary of the fragments consisted of 66 words. The full context-free grammar characterizing the fragments, along with statistics about the generated sentences, can be found in the supplementary materials. agreement language. Examples of declarative sentences in this language are given in (4):
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(4) a. the walrus can giggle .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
b. the yak could amuse your quails by my raven .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
c. the walruses that the newt will confuse can high five your peacocks .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Each noun phrase in the language had at most one modifier, either a relative clause or a prepositional phrase. Relative clauses were never embedded inside other relative clauses. Every verb was associated with one of the auxiliary verbs can, could, will, and would. Since such modals do not show agreement, any noun, whether singular or plural, was allowed to appear with any auxiliary.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The second fragment, the agreement language, was identical to the no-agreement language, except that the auxiliaries in this language were do, don't, does, and doesn't. Subjects in this language agreed with the auxiliaries of their verbs: singular subjects appeared with does or doesn't, while plural subjects appeared with do or don't. Examples of declarative sentences in the agreement language are given in (5):
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(5) a. the walrus does giggle .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
b. the yak doesn't amuse your quails by my raven . c. the walruses that the newt does confuse do high five your peacocks .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Both languages reused structural units; for example, the same prepositional phrases could modify both subject and object nouns. Such shared structure served as a possible cue to hierarchy because it is more efficiently represented in a hierarchical grammar than a linear one. Subject-verb agreement in the agreement language provided an additional cue to hierarchy; in (5c), for example, do agrees with its hierarchicallydetermined plural subject of walruses even though the singular noun newt is linearly closer to it. We therefore predict that hierarchical generalizations will be more likely with the agreement language than the no-agreement language.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The networks were trained to perform two tasks: identity (returning the input sentence unchanged) and question formation. The task to be performed was indicated by a token at the   Figure 1: Basic sequence-to-sequence neural network without attention.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
end of the sentence-either IDENT for identity or QUEST for question formation. IDENT and QUEST served as endof-sequence tokens in both the input and output. Table 1 provides examples of these tasks on each of the three types of sentences in the languages: sentences without relative clauses, sentences with a relative clause on the object, and sentences with a relative clause on the subject. During training we withheld the question formation task for sentences with a relative clause on the subject (the shaded cell in Table 1); these are the only cases that directly disambiguate the linear and hierarchical hypotheses. The identity task was included in the training setup to familiarize the networks with the critical sentence type withheld from the question task; without such exposure, the networks could be justified in concluding that subjects cannot be modified by relative clauses, making it difficult to test such sentences.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We used two sets of sentences for evaluation, a test set and a generalization set. The test set consisted of novel sentences from the five non-withheld cases in Table 1. It was used to assess how well a network had learned the patterns in its training set. The generalization set consisted of sentences from the withheld case (the question formation task for sentences with relative clauses on their subjects). This set was used to assess how the networks generalized to sentence types from which they had not formed questions during training. The test and generalization set both contained 10,000 unique sentences and the training set contained 120,000 unique sentences.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Here we give a very brief bird's-eye view of our architectures.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
For a more precise description, including our hyperparameter values, see the supplementary materials.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
For all experiments we used the sequence-to-sequence model (Botvinick & Plaut, 2006;Sutskever, Vinyals, & Le, 2014) illustrated in Figure 1. This network has two subcomponents called the encoder and the decoder, both of which are RNNs. The encoder processes the input sentence one word at a time to create a single vector representing the entire input sentence. The decoder then receives this vector (called the encoding) and, based on it, outputs one word at a time until it generates a special end-of-sequence token.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The encoder and decoder each possess a component called a recurrent unit which governs how information flows from one time step to the next. We tested three types of recurrent units: a simple recurrent network (SRN) (Elman, 1990), a gated recurrent unit (GRU) (Cho et al., 2014), and long shortterm memory (LSTM) (Hochreiter & Schmidhuber, 1997). For each type of recurrent unit, we experimented with adding attention to the decoder (Bahdanau, Cho, & Bengio, 2015); attention is a mechanism which gives the decoder access to intermediate steps of the encoding process. For each pair of an architecture and a language, we trained 100 networks with different random initializations, for a total of 1200 networks.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
For the test set, all six architectures except the vanilla SRN (i.e., the SRN without attention) produced over 94% of the output sentences exactly correctly (accuracy was averaged across 100 trained networks for each architecture). The highest accuracy was 99.9% for the LSTM without attention. Using a more lenient evaluation criterion whereby the network was not penalized for replacing a word with another word of the same part of speech, the accuracy of the SRN without attention increased from 0.1% to 81%, suggesting that its main source of error was a tendency to replace words with other words of the same lexical category. This tendency is a known deficiency of SRNs (Frank & Mathis, 2007) and does not bear on our main concern of the networks' syntactic representations. Setting aside these lexical concerns, then, we conclude that all architectures were able to learn the language.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
On the generalization set, the networks were rarely able to correctly produce the full question -only about 13% of the questions were exactly correct in the best-performing architecture (LSTM with attention). However, getting the output exactly correct is a demanding metric; the full-question accuracy can be affected by a number of errors that are not directly related to the research question of whether the network preferred a linear or hierarchical rule. Such errors include repeating or omitting words or confusing similar words. To abstract away from such extraneous errors, for the generalization set we focus on accuracy at the first word of the output. Because all examples in the generalization set involve question formation, this word is always the auxiliary that is moved to form the question, and the identity of this auxiliary is enough to differentiate the hypotheses. For example, if the input is my yak who the seal can amuse will giggle . QUEST, a hierarchically-generalizing network would choose will as the first word of the output, while a linearly-generalizing network would choose can. This analysis only disambiguates the hypotheses if the two possible auxiliaries are different, so we only considered sentences where that was the case. For the agreement language, we made the further stipulation that both auxiliaries must agree with the subject so that the correct auxiliary could not be determined based on agreement alone. Figure 2 gives the accuracies on this metric across the six architectures for the two different languages (individual points represent different initializations). We draw three conclusions from this figure: 1. Agreement leads to more robust hierarchical generalization: All six architectures were significantly more likely (p < 0.01) to choose the main auxiliary when trained on the agreement language than the no-agreement language. In other words, adding hierarchical cues to the input increased the chance of learning the hierarchical generalization. 2. Initialization matters: For each architecture, accuracy often varied considerably across random initializations. This fact suggests that the architectural bias is not strong enough to reliably lead the networks to settle on the hierarchical generalization, even in GRUs with attention. From a methodological perspective, this observation highlights the importance of examining many initializations of the network before drawing qualitative conclusions about an architecture (in a particularly striking example, though the accuracy of most LSTMs with attention was low, there was one with near-perfect accuracy). 3. Different architectures perform qualitatively differently: Of the six architectures, only the GRU with attention showed a strong preference for choosing the main auxiliary instead of the linearly first auxiliary. By contrast, the vanilla GRU chose the first auxiliary nearly 100% of the time. In this case, then, attention made a qualitative difference for the generalization that was acquired. By contrast, for both LSTM architectures, most random initializations led to networks that chose the first auxiliary nearly 100% of the time. Both SRN architectures showed little preference for either the main auxiliary or the linearly first auxiliary; in fact the SRNs often chose an auxiliary that was not even in the input sentence, whereas the GRUs and LSTMs almost always chose one of the auxiliaries in the input. In the next section, we take some preliminary steps toward exploring why the architectures behaved in qualitatively different ways.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
A plausible hypothesis about the differences between networks is that linearly-generalizing networks used representations that contained linearly-relevant information whereas hierarchically-generalizing networks used representations that contained hierarchically-relevant information. To test this hypothesis, we analyzed the final hidden state of the encoder (E 6 in Figure 1), which we will refer to as the encoding of the sentence. In architectures without attention, this is the only information that the decoder has about the sentence; architectures with attention can use the intermediate encodings of sentence prefixes as well. We analyze the amount of information that these encodings contain about three properties of the input sentence: its main auxiliary, its fourth word, and the head noun of the subject (which, in the simple languages we used, was always the sentence's second word). Examples are shown in Table 2. Main auxiliary: The main auxiliary of a sentence can appear in many different linear positions but has a consistent hierarchical position. Therefore, a network whose encodings can be used to identify sentences' main auxiliaries must contain some hierarchical information. Fourth word: The fourth word of a sentence has a consistent role in a linear representation but not in a hierarchical one: the fourth word could be the main verb, the determiner on a prepositional object, or the auxiliary verb inside a subject relative clause. Therefore, a network whose encodings can be used to identify each sentence's fourth word must contain some information about linear order. Subject noun/second word: The head noun of the subject is always the second word of the sentence in our languages. Thus, this word can be reliably identified either from a linear representation (as the second word) or from a hierarchical representation (as the subject noun). Analysis: For each trained network, we trained three linear classifiers, one for each of these three properties of the sentence. Each classifier was trained to predict the word that filled the relevant role-main auxiliary, fourth word or subject noun/second word-from the final hidden state of the encoder. Each classifier's output layer had a dimensionality equal to the number of possible classes for that classifier's task: 4 for the main auxiliary, 28 for the fourth word, or 26 for the subject noun. The classifiers were trained on a training set   and tested on a withheld test set (see the supplementary materials for details). Figure 3 shows the classification results on the test set. Classifiers trained to predict the main auxiliary from the encodings produced by the SRNs with attention performed only slightly better than chance; this might explain why the SRNs with attention generalized poorly to the withheld sentence type in the question formation task. Similar classifiers trained on encodings from the other architectures did well at this task. Since the identity of the main auxiliary is the only information required to perform well on our evaluation of the networks' performance on the generalization set based on the first word produced, these results suggest that the differences in performance stem not from inability to identify the main auxiliary but rather from a misinterpretation of the task as requiring fronting of the linearly first auxiliary.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We now consider the fourth word and subject noun classifiers. The classifiers trained on the encodings from both types of LSTMs as well as the GRUs without attention performed well at both tasks. Crucially, the classifiers trained on the encodings from the GRU with attention did poorly on these tasks. Recall that the main auxiliary could be successfully decoded from the encodings of this architecture. The GRU with attention therefore appears to use its encoding only for information that could not be straightforwardly obtained from linear order, such as the main auxiliary, rather than information that could be obtained from linear order even if, like the subject head noun, that information was hierarchically relevant. On the other hand, the fact that the GRU without attention and both LSTM architectures performed very well at all three tasks suggests that they used their encodings for both linear and hierarchical information. Thus, perhaps the better generalization ability of the GRU with attention arises not from a better ability to encode relevant hierarchical informationall four LSTM and GRU architectures have that ability-but  Table 3: Analysis of output question types based on which auxiliary has been deleted (if any) and which auxiliary has been placed at the start of the sentence. Each number is the percent of GRU + attention outputs across all 100 random initializations that fit that category (the total sums to 65% because only 65% of the questions produced by the networks could be analyzed in that way). 1 st and 2 nd refer to the first and second auxiliaries in the input.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
rather from an ability to ignore linear information (Frank & Mathis, 2007).
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
We now return to the full questions produced by our networks and compare the networks' errors to the types of errors that humans make when acquiring English (Crain & Nakayama, 1987). We restrict ourselves to the GRU with attention networks as those were the networks that generally produced the correct auxiliary (see Figure 2). Subject-auxiliary inversion can be decomposed into two subtasks: placing an auxiliary at the start of the sentence and deleting an auxiliary within the sentence. Only 65% of the outputs that the 100 networks collectively produced could be interpreted as having been formed by inserting an auxiliary before the sentence and deleting zero or one of the auxiliaries in the sentence. Table 3 breaks down those results based on which auxiliary was preposed and which (if any) was deleted. 4 Two error types are by far the most common. In the first type, the network preposed the second auxiliary but did not delete either of the auxiliaries (could his newt who can giggle could swim from his newt who can giggle could swim). This error type is common among English-learning children (Crain & Nakayama, 1987) and is compatible with hierarchical generalization. In the other frequent error type, the network deleted the first auxiliary and preposed the second; for example, it might generate could his newt who giggle could swim from his newt who can giggle could swim. Such errors were never observed by Crain and Nakayama (1987) and are incompatible with a hierarchical generalization. In other words, though the networks' common error types overlapped with the common error types for humans, the networks also frequently made some mistakes that humans never would.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Learners of English acquire the correct hierarchical rule for forming questions even though there are few to no examples in their input that explicitly distinguish this rule from the linear one. This fact has been taken to suggest that learners must be innately constrained to consider only hierarchical syntactic rules. We have investigated whether a learner without such a constraint can learn the hierarchical generalization without the critical disambiguating examples. Based on the behavior of one of the architectures we examined (GRU with attention), the answer to this question appears to be yes. The hierarchical behavior of this non-hierarchically-constrained architecture plausibly arose from the influence of hierarchical cues in the input, a conclusion supported by the fact that the additional hierarchical cue of agreement increased the likelihood that a network would induce hierarchical generalizations.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Our argument has focused on a strong version of the poverty of the stimulus argument which claims that language learners require a hierarchical constraint. However, there remains a milder version which only claims that a hierarchical bias is necessary. This version of the argument is difficult to assess using RNNs because, while RNNs must possess some biases (Mitchell, 1980;Marcus, 2018), the nature of these biases-which likely arise both from the network architecture and from the learning algorithm-is currently poorly understood. However, given the linear way in which they process inputs, it is plausible that all six architectures we used had a bias toward linear order but that the GRU with attention was the only one that overcame this linear bias sufficiently to generalize hierarchically. It is not clear why it was the only architecture to do so; we intend to examine the differences in behavior between the recurrent units in future work.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Two caveats are in order. First, our results only cover restricted fragments of English and may not generalize to the linguistic input that human language learners encounter. In future work, we will replace our artificial languages with a corpus of child-directed speech. Second, even if our findings do generalize to realistic language, we would only be able to conclude that it is possible to solve the task without a hierarchical constraint; humans certainly could have such an innate constraint despite it being unnecessary for this particular task.  Figure 4 contains the context-free grammar used to generate the no-agreement language. 120,000 unique sentences were generated from this grammar as the training set, with each example randomly assigned either the identity task or the question formation task. If a sentence was assigned to the question formation task and contained a relative clause on the subject, it was not included in the training set.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The agreement language was generated from a similar grammar but with the auxiliaries changed to do, does, don't, and doesn't. In addition, to ensure proper agreement, the grammar for the agreement language had separate rules for sentences with singular subjects and sentences with plural subjects, as well as separate rules for relative clauses with singular subjects and relative clauses with plural subjects. Figure 5a shows how frequent each sentence type was based on the types of modifiers present in the sentence and which noun phrases those modifiers were modifying. Figure 5b shows the same statistics for the agreement language. In general, for a given left-hand side in the grammar in Figure 4, all rules with that left-hand side were equally probable; so, for example, one third of noun phrases were unmodified, one third were modified by a prepositional phrase, and one third were modified by a relative clause. The one exception to this generalization is that intransitive sentences with unmodified subjects were rare in both languages. This is because we did not allow any repeated items within or across data sets, and since there were relatively few possible intransitive sentences with unmodified subjects, this uniqueness constraint prevented the unmodified intransitive case from being as common as the modified cases. The no-agreement language has roughly twice as many intransitive sentences with unmodified subjects as the agreement language does because there are twice as many possible sentences of that type for the no-agreement language than the agreement language, but otherwise the two languages are essentially the same in the distributions of their constructions.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Neither language exhibited recursion. This is because relative clauses and prepositional phrases could only modify matrix noun phrases but not noun phrases within relative clauses or prepositional phrases. Thus, both languages contained a finite number of sentences, though this finite number is very large (greater than 10 15 ), orders of magnitude larger than the number of sentences present in the training set (120,000). Figure 1 (reproduced here as Figure 6) depicts the basic sequence-to-sequence architecture underlying all of our experiments. Here we elaborate on the different components of this architecture.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
The network consists of two components, the encoder and the decoder, both of which are RNNs. The encoder's hidden state is initialized at E 0 as a 256-dimensional vector of all zeros. The network is then fed the first word of the input sentence, represented in a distributed manner as a 256dimensional vector (i.e. an embedding) whose elements are learned during training. The encoder uses this distributed representation of the first word, along with the initial hidden state, to generate the next hidden state, E 1 . The component that performs this hidden state update is called the encoder's recurrent unit. Each subsequent word of the input sentence is then fed into the network, turned into its distributed representation learned by the network, and passed through the recurrent unit along with the previous hidden state to generate the next hidden state.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Once all of the input words have been passed through the encoder, the final hidden state of the encoder is used as the initial hidden state of the decoder, D 0 . This hidden state and a special start-of-sentence token (also represented by a 256dimensional distributed representation that is learned during training) are passed as inputs to the decoder's recurrent unit, which outputs a new 256-dimensional vector as the next decoder hidden state, D 1 . A copy of this new hidden state is  Figure 6: Basic sequence-to-sequence neural network without attention.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
also passed through a linear layer whose output is a vector with a length equal to the vocabulary size. The softmax function is then applied to this vector (so that its values sum to 1 and all fall between 0 and 1). Then, the element of this vector with the highest value is taken to correspond to the output word for that timestep; this correspondence is determined by a dictionary relating each index in the vector to a word in the vocabulary. For the next time step of decoding, this just-outputted word is converted to a distributed representation and is then taken as an input to the decoder's recurrent unit, along with the previous decoder hidden state, to generate the next decoder hidden state and the next output word. Once the outputted word is an end-of-sequence token (either IDENT or QUEST), decoding stops and the sequence of outputted words is taken as the output sentence. At all steps of this decoding process, whenever a distributed representation is used, dropout (Srivastava, Hinton, Krizhevsky, Sutskever, & Salakhutdinov, 2014) with a proportion of 0.1 is applied to the vector, meaning that each of its values will with 10% probability be turned to 0. This practice is meant to combat overfitting of the network's parameters.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
There are two main ways in which we varied this basic architecture. First was usage of an attention mechanism, depicted in Figure 7, which is a modification to the decoder's recurrent unit. The attention mechanism adds a third input (which we refer to as the attention-weighted sum) to the decoder recurrent unit. This attention-weighted sum is determined as follows: First, the previous hidden state and the distributed representation of the previous output word are passed through a linear layer whose output is a vector of length equal to the number of words in the input sentence. This vector is the vector of attention weights. Each of these weights is then multiplied by the hidden state of the encoder at the encoding time step equal to that weight's index. All of these products are then added together to give the attention-weighted sum, which is passed as an input to the decoder recurrent unit along with the previous output word and the previous hidden state.
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Second, we also vary the structure of the recurrent unit used for the encoder and decoder. The three types of recurrent units we experiment with are simple recurrent networks (SRNs) (Elman, 1990), gated recurrent units (GRUs) (Cho et al., 2014), and long short-term memory (LSTM) units   (Hochreiter & Schmidhuber, 1997). For all three of these types of recurrent units, we use the default PyTorch implementations, which are described in the next few paragraphs. The SRN concatenates its inputs, passes the result of the concatenation through a linear layer whose output consists of linear combinations of the elements of the input vector, and finally applies the hyperbolic tangent function to the result to create a vector whose values are mostly either very close to -1 or very close to +1. This hidden state update can be expressed with the following equation:
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
where D i is the i th hidden state of the decoder, w i indicates the i th output word, W is a matrix of learned weights, b is a learned vector called the bias term, and [v 1 , v 2 , ...] indicates the concatenation of vectors v 1 , v 2 , .... If attention is used, this equation then becomes
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
where A i is the i th attention-weighted sum. The GRU adds several internal vectors called gates to the basic SRN structure. Specifically, these gates are called the reset gate r t , the input gate z t , and the new gate n t , each of which has a corresponding matrix of weights (W r for r t , W z for z t , and two separate matrices W nw and W nD for n t ). The reset and input gates both take the previous hidden state and the previously outputted word (as a distributed representation) as inputs. The new gate also takes these two inputs as well as the reset gate as a third input. The next hidden state is then generated as the product of the input gate and the previous hidden state plus the product of one minus the input gate times the new gate. This can be thought of as the input gate determining which elements of the hidden state to preserve and which to change. The elements to be preserved are preserved through the term that is the product of the input gate times the previous hidden state, while the elements to be changed are determined through the term that is the product of one minus the input gate times the new gate; the new gate here determines what the updated values for these changed terms should be. Overall the GRU update can be expressed with the following equations (σ indicates the sigmoid function):
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Like the GRU, the LSTM also uses gates-specifically, the input gate i t , forget gate f t , cell gate g t , and output gate o t . Furthermore, while the other architectures all just use the hidden states as the memory of the network, the LSTM adds a second vector called the cell state c t that acts as another persistent state that is passed from time step to time step. These components interact according to the following equations to produce the next hidden state and cell state:
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
c t = f t * c t−1 + i t * g t (11)
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
For each pair of an architecture and a language, we trained 100 networks with different random initializations, for a total of 1200 trained networks. The networks were trained using stochastic gradient descent with the negative log likelihood objective function for 30,000 batches with a batch size of 5 (meaning that some training examples were seen more than once), a dropout rate of 0.1, and a learning rate of 0.01 (for the GRUs and LSTMs) or 0.001 (for the SRNs). All networks used 256-dimensional hidden states and trained 256dimensional vector representations of words. All parameter values were taken from a PyTorch tutorial on sequence-tosequence networks, 5 except that the learning rate for SRNs was lowered because these networks did not converge with the default learning rate. 
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
Each linear classifier consisted of a single linear layer which took as its input a 256-dimensional vector (specifically, the encoding of a sentence) and outputted a vector of dimension equal to the number of possible values for the feature used as the basis of classification (4 for the main auxiliary, 28 for the fourth word, or 26 for the subject noun). For example, since there are four auxiliaries, the main auxiliary classifier had an output of dimensionality 4. The chance baseline for each task is thus 1 n where n is the number of possible classes for that task. Each element in this output corresponded to a specific value for the feature being used as the basis for classification, and for a given input the element of the output with the highest value was taken as the classification for that input. The sentence encodings were randomly split into a training set (75% of the encodings), a development test set (5% of the encodings), and a test set (20% of the encodings), none of which overlapped. The weights of the classifier were trained on the training set using stochastic gradient descent, and training stopped when the cross entropy loss computed over the development test set ceased to improve. Classification accuracy was then determined based on the withheld test set. In addition to the information gleaned from the sentence encoding, the decoder may also access information about the input sentence through attention, but here we did not analyze the contribution of attention because it was not present in all architectures and because we wished to use an analysis method that could compare all six architectures. Table 3 categorized the 65% of GRU with attention outputs that could be viewed as taking the input, inserting an auxiliary at its start, and deleting either zero or one auxiliary from within the input. Here we show some examples of the remaining 35% of the outputs. Many of these outputs fall into certain common error types, including truncation of the out-put (6), repetition of words (7), and turning a relative clause into a prepositional phrase (8), but many outputs do not fall into one of those categories (9, 10, 11). (9) Other a. Input: the newt who some quails do accept doesn't comfort the ravens that do move . b. Output: do the newt who some quails doesn't accept doesn't comfort the ravens that do high five our yaks .
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(10) Other a. Input: the yak who does applaud your newts doesn't entertain our walruses . b. Output: does the yak who the yak newts doesn't entertain our walruses ?
Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
(11) Other a. Input: your vulture who does entertain our orangutan doesn't comfort my walruses above your newts . b. Output: does your vulture entertain some peacock upon our ravens my walruses ?
Can neural networks acquire a structural bias from raw linguistic data?
Humans appear to use structural biases to guide language acquisition. A classic example is the rule for subject-auxiliary inversion: Native English speakers uniformly acquire a rule like the structural generalization in Figure 1 that makes reference to hierarchical syntactic structures, despite the fact that the raw linguistic input often supports linear generalizations which are intuitively just as simple (Chomsky, 1965). Humans are not alone in possessing this inductive bias: Prior investigations have identified some artificial learners with a structural bias by virtue of having a significantly restricted the hypothesis space (Perfors, Tenenbaum, & Regier, 2011) or a hierarchically structured architecture that learns from preparsed data (McCoy, Frank, & Linzen, 2020).
Can neural networks acquire a structural bias from raw linguistic data?
However, these results cannot tell us whether a learner starting with very weak biases can acquire a structural bias merely from exposure to raw linguistic data. While inductive biases are often understood to be unchangeable properties of a learner, this need not be the case. For instance, in one dominant paradigm in natural language processing, pretraining on raw data is used to create a general purpose sentence processing model like BERT (Bidirectional Encoder Representations from Transformers; Devlin, Chang, Lee, & Toutanova, 2019), which can subsequently be fine-tuned to perform a downstream task. The model's inductive biases with respect to the Training Has the man who gone has seen the cat? Has the man who has gone seen the cat?
Can neural networks acquire a structural bias from raw linguistic data?
Has the man seen the cat who has gone?
Can neural networks acquire a structural bias from raw linguistic data?
Has the man has seen the cat who gone?
Can neural networks acquire a structural bias from raw linguistic data?
Has the man has seen the cat who gone? has the man who has gone has seen the cat?
Can neural networks acquire a structural bias from raw linguistic data?
Test behavior: Linear bias observed downstream task may be substantially influenced by the prior knowledge acquired during pretraining.
Can neural networks acquire a structural bias from raw linguistic data?
In this work, we present new experimental evidence that BERT may acquire an inductive bias towards structural generalizations from exposure to raw data alone. We conduct four experiments inspired by Linzen (2018, 2020) to evaluate whether BERT has a structural or linear bias when generalizing about structure-dependent English phenomena. We follow the poverty of the stimulus design (Wilson, 2006), outlined in Figure 1. First, we finetune BERT to perform a classification task using data intentionally ambiguous between structural and linear generalizations. Then, we probe the inductive biases of the learner by observing the behavior of the learner on held-out examples that disambiguate between the generalizations. The experimental datasets illustrate three structure dependent rules of English grammar regarding subject-auxiliary inversion, reflexive-antecedent agreement, and negative polarity item (NPI) licensing. A fourth dataset classifies sentences based on an arbitrary rule: whether a verb in an embedded clause is in the past tense. The data is generated from templates using the generation tools and lexicon created by Warstadt, Cao, et al. (2019) and Warstadt, Parrish, et al. (2019).
Can neural networks acquire a structural bias from raw linguistic data?
The results of these experiments suggest that BERT likely acquires a inductive bias towards structural rules from selfsupervised pretraining. BERT generalizes in a way consistent with a structural bias in 3 out of 4 experiments: those involving subject-auxiliary inversion, reflexive binding, and embedded verb tense detection. While these experiments leave open several alternative explanations for this generalization behavior, they add to mounting evidence that significant syntactic knowledge, including a structural biases, can be acquired from self-supervised learning on raw data.
Can neural networks acquire a structural bias from raw linguistic data?
Self-supervised Learning and BERT Recent advances in machine learning for natural language processing give new reason to believe that low-bias learners can acquire significant grammatical knowledge from raw data. The Transformer neural network architecture (Vaswani et al., 2017) used in models like BERT has very weak biases: It is a universal approximator of the class of sequence transduction functions (Yun, Bhojanapalli, Rawat, Reddi, & Kumar, 2019), and it has been applied effectively in nonlinguistic domains such computer vision (Parmar et al., 2018) and protein sequence modeling (Rives et al., 2019). However, rather than training a low bias model from scratch to perform a particular linguistic task, recent results show that it is far more more effective to pretrain a general purpose model on raw data and subsequently fine-tune it on a downstream task (e.g. Howard & Ruder, 2018). The implication is that they acquire helpful biases from pretraining.
Can neural networks acquire a structural bias from raw linguistic data?
Crucially, these models are usually pretrained with only raw data using the technique of self-supervised learning, which circumvents the need for labeled data by using the data itself as the labels. The most common self-supervised tasks for pretraining are the language modeling task, where the objective is predict the next word in a string (e.g. Peters et al., 2018;Radford et al., 2019), and, in the case of BERT, the cloze task where the objective is predict the identity of a masked token anywhere in a string.
Can neural networks acquire a structural bias from raw linguistic data?
Despite containing no explicit information about grammatical concepts, self-supervised tasks appear to teach neural models significant knowledge of grammar and hierarchical syntax. These models can perform human-like acceptability judgments, which are understood in linguistics as a probe on linguistics competence (Schütze, 1996). When fine-tuned to perform acceptability judgments, BERT approaches human performance on the Corpus of Linguistic Acceptability (CoLA; Warstadt, Singh, & Bowman, 2019), a dataset of over 10k example sentences from linguistics publications with Boolean acceptability judgments. Language models can also correctly discriminate minimal pairs for subjectverb agreement (Gulordava, Bojanowski, Grave, Linzen, & Baroni, 2019), wh-dependencies (Wilcox, Levy, Morita, & Futrell, 2018), and numerous other linguistic phenomena in English (Warstadt, Parrish, et al., 2019) without any supervised training on acceptability. BERT's internal representa-tions appear to attend to linguistic features such as syntactic category (Clark, Khandelwal, Levy, & Manning, 2019) and contain sufficient information from which to recover a dependency parse for an inputted sentence (Hewitt & Manning, 2019). However, it is not known whether BERT is biased towards forming generalizations based on structural features when fine-tuned on structure-dependent phenomena. Indeed, it is possible that BERT could acquire knowledge of hierarchical syntax but still preferentially use surface features to generalize. Our experiments are designed to address this question.
Can neural networks acquire a structural bias from raw linguistic data?
The learnability of structural bias has played a large role in debates about human language acquisition. Chomsky (1965Chomsky ( , 1971 proposes that humans have an innate bias towards learning structural grammatical rules. For example, children must learn a general rule for subject-auxiliary inversion primarily from input like (1). With this input, a learner could form a structural generalization (front the highest auxiliary in the corresponding declarative) or a linear one (e.g. front the first or last auxiliary), but human learners always choose the former. That is, no human learner of English acquires a linear rule that produces the form in (2a) over (2b). From such examples, Chomsky (1971) concludes that humans have an innate preference for structure-dependent rules. Otherwise it would be difficult to explain how we so consistently avoid deeply un-language-like hypotheses in lieu of significant disconfirming evidence.
Can neural networks acquire a structural bias from raw linguistic data?
(1) a. The cat has gone. b. Has the cat gone?
Can neural networks acquire a structural bias from raw linguistic data?
(2) a. *Is the man has seen the cat who going? b. Has the man seen the cat who is going?
Can neural networks acquire a structural bias from raw linguistic data?
These examples play a key role in Chomsky's influential argument from the poverty of the stimulus in support of this hypothesis. A version of the argument is given below: 1 Premise 1 Humans form grammatical hypotheses about their native language using either innate biases or data-driven learning.
Can neural networks acquire a structural bias from raw linguistic data?
Premise 2 Human language learners preferentially form structural hypotheses over equally simple linear hypotheses.
Can neural networks acquire a structural bias from raw linguistic data?
Premise 3 The raw linguistic input during language acquisition favors neither the structural nor the linear hypothesis.
Can neural networks acquire a structural bias from raw linguistic data?
Conclusion Humans' preference for structural generalization is not learned from the raw linguistic input (or any other part of the learner's environment), i.e., it is innate. This conclusion has spurred much fruitful research into the nature of linguistic universals (see e.g., Chomsky, 1981) and the argument has been fleshed out with evidence child language acquisition (Crain & Nakayama, 1987;Yang, 2000). Nonetheless the argument's validity has been at times been questioned (Pullum & Scholz, 2002;Reali & Christiansen, 2005;Perfors et al., 2011). If the premises are granted, the reasoning to the conclusion is sound. Premise 1 is a tautology given a suitable definition of data-driven learning. Premise 2 is an robust empirical result of generative linguistics.
Can neural networks acquire a structural bias from raw linguistic data?
Therefore, those arguing against this result have generally taken issue with Premise 3 of the argument, which I refer to henceforth as the impoverishment assumption. For instance, Pullum and Scholz (2002) conduct a corpus search to show that such sentences that would provide evidence for the structural rule such as (2b) are attested in naturalistic speech. However, Legate and Yang (2002) counter that such examples are insufficiently frequent to meaningfully impact learning. More importantly, whether or not crucial evidence is lacking for the acquisition of subject-auxiliary inversion, there are certainly numerous other structural rules which have been proposed for natural languages for which crucial examples are vanishingly rare. Reali and Christiansen (2005) articulate a broader criticism of the impoverishment assumption. It is not only sentences of the form in (2b) that count as evidence for the structural hypothesis. Rather, data from all domains of language provide indirect evidence that militates in favor of a structural understanding of grammar. Statistical regularities in language, they suggest, may be sufficient for "bootstrapping syntax". Indeed, Perfors et al. (2011) find that Bayesian grammar induction system given a choice between several grammar types will preferentially hypothesize phrase-structural rules for English over a flat grammar when presented with child directed speech. This results suggests that the raw data seen be children does favor the acquisition of structural rules in general, counter to the impoverishment assumption, at least if the hypothesis space is strictly limited. However, their models are not low-bias learners, nor do they discover syntactic representations on their own. Rather, they are presented with several hand-crafted candidate grammars of various types including a generally adequate phrase-structure grammar, as well as the syntactic categories of the input data.
Can neural networks acquire a structural bias from raw linguistic data?
There have been several prior efforts to test neural networks for a structural bias in the domain of subject-auxiliary inversion (Lewis & Elman, 2001;Frank & Mathis, 2007;McCoy et al., 2018McCoy et al., , 2020. These studies all adopt some form of the poverty of the stimulus design (Wilson, 2006), an experimental paradigm that probes the inductive biases of learners by training them on data that is ambiguous between several hypotheses, and evaluating them on examples that disambiguate between these hypotheses. For instance, in a series of papers McCoy et al. (2018McCoy et al. ( , 2020 train neural networks to output a polar question given the corresponding declarative sentence, using training and test data similar to our subjectauxiliary inversion paradigm shown in Table 1. They find that, while tree-structured models trained using parsed data make a structural generalization, low-bias models never do so consistently.
Can neural networks acquire a structural bias from raw linguistic data?
However, there is good reason to revisit this question with BERT. McCoy et al. do not evaluate BERT, but rather LSTMs that are pretrained on an auto-encoding task in which the model must reproduce the input sentence verbatim. Furthermore, the pretraining data is not naturally occurring, but generated from a restricted lexicon and small context-free grammar. Conneau, Kruszewski, Lample, Barrault, and Baroni (2018) have already been shown that auto-encoders are much weaker at learning syntactic features than surface features even with naturalistic training data. Thus, there is little reason to expect that BERT would perform similarly at this task.
Can neural networks acquire a structural bias from raw linguistic data?
We apply the poverty of the stimulus design to test whether unsupervised pretraining gives BERT a structural bias. We conduct four experiments using semi-automatically generated data illustrating different structural generalizations, including the subject-auxiliary inversion paradigm investigated by McCoy et al. (2018McCoy et al. ( , 2020. Each experiment consists of a training phase in which BERT is fine-tuned to classify sentences from an impoverished paradigm consistent with both a structural hypothesis and a linear hypothesis. Then, the classifier is evaluated on the full paradigm which disambiguates between the two hypotheses. If the classifier makes the structural generalization, this is evidence that BERT has learned a structural bias from pretraining on raw data.
Can neural networks acquire a structural bias from raw linguistic data?
Examples from the four experimental datasets are shown in Table 1. Each of these datasets is associated with a binary classification task. In the subject-auxiliary inversion, reflexive, and NPI datasets, the classes correspond to grammatical acceptability of the sentence. In the tense dataset, the classes correspond to whether or not the embedded verb appears in the past tense. The tense detection task enables us to draw additional conclusions not possible with the acceptability task alone. Since BERT appears to acquire some knowledge of acceptability from pretraining, it might converge on the structural generalization on the acceptability task not by forming an entirely new structural generalization from ambiguous training data, but merely accessing an implicit structural generalization acquired during pretraining. In the tense detection task there is no reason to expect BERT has implictly acquired the structural generalization during pretraining, and the structural and linear hypotheses are equally arbitrary. Thus, this paradigm tests whether BERT has a structural bias when forming completely novel generalizations.
Can neural networks acquire a structural bias from raw linguistic data?
In the subject-auxiliary inversion dataset, the structural and linear hypotheses are defined in terms of where the auxiliary at the front of the sentence has moved from. Each sentence contains two clausesa main clause and an embedded relative clause-each with an auxiliary verb. In the training examples the embedded auxiliary precedes the main auxiliary because the relative clause
Can neural networks acquire a structural bias from raw linguistic data?
Acceptable Unacceptable
Can neural networks acquire a structural bias from raw linguistic data?
Has the man who is going seen the cat? Is the man who going has seen the cat? Test
Can neural networks acquire a structural bias from raw linguistic data?
Has the man seen the cat who is going? Is the man has seen the cat who going?
Can neural networks acquire a structural bias from raw linguistic data?
The boy that loves himself talks to ladies. The boy that loves themselves talks to ladies? Test
Can neural networks acquire a structural bias from raw linguistic data?
The boy that loves ladies talks to himself. The boy that loves ladies talks to themselves.
Can neural networks acquire a structural bias from raw linguistic data?
Kids who saw the cats won't get any dogs. Kids who saw any cats won't get the dogs.
Can neural networks acquire a structural bias from raw linguistic data?
Kids who won't see any cats get the dogs. Kids who won't see the cats get any dogs.
Can neural networks acquire a structural bias from raw linguistic data?
Tense Training
Can neural networks acquire a structural bias from raw linguistic data?
The critic who sang arias praised a lady. The critic who sings arias praised a lady.
Can neural networks acquire a structural bias from raw linguistic data?
The critic praised a lady who sang arias. The critic praised a lady who sings arias. Table 1: Data from the subject-auxiliary inversion experiment. According to the relevant linear generalizations, sentences shaded in gray will belong to the positive class, and sentences in white belong to the negative class.
Can neural networks acquire a structural bias from raw linguistic data?
modifies the subject. In the test examples the embedded auxiliary follows the main auxiliary because the relative clause modifies the object. Therefore, a linear generalization which targets the last auxiliary will give the same result for the training as a structural generalization that targets the main auxiliary, but the two generalizations diverge for the test examples.
Can neural networks acquire a structural bias from raw linguistic data?
Reflexive Binding In the reflexive dataset, adapted from Marvin and Linzen (2018), the structural and linear hypotheses depend on the relation between the reflexive pronoun (e.g. himself ) and a binder noun phrase that agrees with it in person and number (e.g. the boy). To a first approximation, the structural c-command relation must hold between the binder and reflexive in order for the sentence to be acceptable (Chomsky, 1981). However, in each training examples, the binder also precedes the reflexive only in the acceptable sentences. These generalizations diverge in the test examples, in which there is an unacceptable sentence where the binder precedes, but does not c-command, the reflexive.
Can neural networks acquire a structural bias from raw linguistic data?
In the NPI dataset, also adapted from Marvin and Linzen (2018), the structural and linear hypotheses depend on the relation between a negative polarity item (NPI; e.g., any) and negation. Loosely speaking, negation must c-command the NPI in order for the sentence to be acceptable. The hypotheses are the same (mutatis mutandis) as those for the reflexive dataset.
Can neural networks acquire a structural bias from raw linguistic data?
Tense In the tense dataset, each sentence has a main verb and an embedded verb, and the classes correspond to whether the embedded verb has past tense inflection. A related objective is used to evaluate structural knowledge in pretrained models by Shi, Padhi, and Knight (2016) and Conneau et al. (2018). As with the subject-auxiliary inversion dataset, the embedded verb precedes the main verb in the training examples, and follows it in the test examples. Therefore, structural and linear generalizations about the position of the verb learned from the training data will diverge for the test data.
Can neural networks acquire a structural bias from raw linguistic data?
The data for these experiments are automatically generated using a method similar to that adopted by Ettinger, Elgohary, and Resnik (2016) and Marvin and Linzen (2018 
Can neural networks acquire a structural bias from raw linguistic data?
In each experiment, we train 20 random restarts of classifiers on top of BERT using the labeled training items. Following Devlin et al. (2019), we fine-tune BERT itself in addition to the classifier layer during training. We use Huggingface's (Wolf et al., 2019) implementation of BERT-Large in PyTorch (Paszke et al., 2017), and carry out fine-tuning in jiant (Wang et al., 2019). All 20 random restarts for each experiment use identical hyperparameters. The only difference between them is the random seed used to initialize the classifier weights. The models use a learning rate of 2e-5, a dropout rate of 0.2, and a batch size of 16. Training is carried out for 4 epochs, or until 5 evaluations occur without any improvement in development accuracy. During training, the model is evaluated on the development set after 10 batches. These hyperparameters are selected based on an exploratory grid search using the recommended hyperparamer ranges suggested by (Devlin et al., 2019). The hyperparameters selected for the experiments consistently led to the best or near-best development accuracy for each of the datasets.
Can neural networks acquire a structural bias from raw linguistic data?
The results for the experiments in Figure 2 show that BERT has an overwhelming tendency to generalize in a way con- Figure 2: Test results for 20 random restarts of the 4 experiments. "% Test Pairs Correct" is the percentage of minimal pairs from the test templates correctly classified. Individual runs are plotted as x's with random horizontal jitter.
Can neural networks acquire a structural bias from raw linguistic data?
sistent with structural generalization in the subject-auxiliary inversion, reflexive, and tense settings, but not in the NPI setting. This plot shows the proportion of test minimal pairs classified correctly, rather than the proportion of test items classified correctly. Because there are four ways to classify a pair of items, a totally random classifier would have an expected accuracy of 25% on this metric. In each experiment, minimal pair accuracy on held-out examples from the training templates is over 90%, indicating that the models robustly learned a generalization consistent with the training data. On the subject-auxiliary inversion task, BERT classifiers appear to make structural generalization with very high consistency and accuracy. The median minimal pair accuracy is over 99%, and the minimum is over 95%.
Can neural networks acquire a structural bias from raw linguistic data?
The reflexive classifiers also appear to make the structural generalization in most cases, but are less consistent. The median minimal pair accuracy is over 92% and the maximum exceeds 99%. However, there is a portion of classifiers that classify the test minimal pairs correctly only about half the time, despite achieving high performance on the training examples. The other half of the time, they tend to classify the test examples in a way consistent with the linear hypothesis. However none of these classifiers systematically makes predictions consistent with the linear generalization.
Can neural networks acquire a structural bias from raw linguistic data?
The tense classifiers make a structural generalization very consistently, with a median minimal pair accuracy over 99%. There are two outliers where the accuracy is below 90%, but still exceeds 77%.
Can neural networks acquire a structural bias from raw linguistic data?
Finally, the results from the NPI experiment are the exceptional case where we do not observe behavior consistent with a structural generalization. The median minimal pair accuracy is effectively 0%, and the maximum is barely above chance at 32%. While every model classifier is able to clas-sify the training pairs perfectly, only 6/20 consistently classify the test pairs in the same way. All of these classifiers learn a generalization that is different from the hypothesized linear generalization shown in Table 1. We find that their performance is consistent with grouping together all and only sentences with an NPI towards the end of the sentence.
Can neural networks acquire a structural bias from raw linguistic data?
These results suggest that it is likely that BERT does acquire some form of a structural inductive bias from self-supervised pretraining, at least outside of the NPI domain. These results point much more strongly in this direction than earlier results by McCoy et al. (2018McCoy et al. ( , 2020. If this interpretation turns out to be correct, it would cast further doubt on the impoverishment assumption from Chomsky's (1965) argument from the poverty of the stimulus by showing that raw data does contain overwhelming evidence that language is hierarchical. If some learner does not require innate bias to discover the utility of preferring structural rules over linear ones, it stands to reason humans may not either. On the other hand, our results are consistent with other interpretations, and so we caution against leaping to this strong conclusion, at least without further evidence.
Can neural networks acquire a structural bias from raw linguistic data?
The NPI Results First, in the NPI domain, BERT does not show a structural bias. However, it does not immediately follow that BERT does not have a structural bias at all. By virtue of the paradigm's design, the classes that the model converged on are consistent with both a linear and a structural position. As mentioned above, 6/20 classifiers classified the test items systematically, though not in the way predicted in Table 1. Instead, they grouped the top left and bottom right sentences in one class, and the top right and bottom left sentences in the other. The sentences in the first class might be characterized as all sentences with an NPI at the end of the sentence (a linear generalization), or all sentences with an NPI in the main clause (a structural generalization). Additional experiments are needed to determine which of these outcomes has occurred. Furthermore, it might be possible to explain this result while still allowing that BERT is demonstrating human-like knowledge. In fact, the unacceptable generalization items in the NPI paradigm, where the negation precedes the NPI, are known as NPI illusions, because they can spuriously appear to be acceptable to humans, and pattern with grammatical sentences in self-paced reading and event-related potential (ERP) experiments (Xiang, Dillon, & Phillips, 2009). Thus, NPIs may in retrospect not be the clearest example of humans' structural bias.
Can neural networks acquire a structural bias from raw linguistic data?
Limitations of the Poverty of the Stimulus Design Some serious doubts remain even in the domains where BERT appears to show a structural generalization. There may be some surface features that accidentally give the same predictions as the structural generalization on the test data. For instance, in the subject-auxiliary inversion experiment, if all the unacceptable examples resembled the ones found in Table 1, a classifier could coincidentally identify all the acceptable examples by learning to identify a string with a relativizer adjacent to an auxiliary (e.g. who is). In fact, we control for this particular confound by generating acceptable examples where a finite verb follows the relativizer (e.g. Has the man who went seen the cat?).
Can neural networks acquire a structural bias from raw linguistic data?
However, there are likely other surface generalizations that are consistent with the results. This problem can be addressed by training and evaluating on data that contradict these generalizations, but alternative convergent hypotheses cannot be eliminated entirely. This is a fundamental limitation of the poverty of the stimulus design: It is not possible to determine that BERT is adopting any particular generalization. The strongest conclusion that can be made is to rule out the possibility that BERT has adopted a particular generalization.
Can neural networks acquire a structural bias from raw linguistic data?
That said, if we continue to find convergent from multiple unrelated domains, Occam's Razor tells us that we should conclude BERT has a structural bias. Given the large number of conceivable surface generalizations, let us assume that an arbitrary generalization is equally likely to support any of the four classification behaviors for the test minimal pair. It follows that if the classifier does make some surface generalization, there is a 1 in 4 chance for each experiment that it would accidentally align with the structural generalization. Then the probability that this chance alignment would occur in at least 3 out of 4 domains is about 5%.
Can neural networks acquire a structural bias from raw linguistic data?
This worry could be alleviated further if it could be shown that baseline models without unsupervised pretraining tend to make the linear generalization on these datasets. These experiments will have to be included in future work. However, at present, the results of McCoy et al.'s (2018) experiments can be used as a proxy. As described in Section (2), these experiments test the ability of sentence encoders without substantial unsupervised pretraining to generalize from a paradigm resembling the polar question data in my experiments. In 5 out of 6 of the model architectures they tested, the linear generalization was preferred. While the task in their experiment is different the acceptability judgment task in the present work, based on this finding it seems that sequence models without substantial unsupervised pretraining are likely to prefer the linear generalization in the polar question domain.
Can neural networks acquire a structural bias from raw linguistic data?
Conclusion This work has presented new evidence that highlights the possibility that language learners could acquire a structural inductive bias from statistical regularities in raw linguistic data. In particular, we find the most comprehensive evidence to date (to our knowledge) of a low-bias learner demonstrating a structural bias acquired through unsupervised learning on raw data. However, more converging evidence would be necessary to strengthen our confidence in this conclusion.
Can neural networks acquire a structural bias from raw linguistic data?
Future work should also focus on drawing more direct connections between neural networks and human language learn-ers. BERT is trained on data from domains far outside the input to human learners, and in much greater quantities. Indeed, the issues of quantity and quality of data are two other pillars of Chomsky's (1965) argument from the poverty of the stimulus. Therefore, it is essential to replicate these experiments with models trained on less data bearing greater resemblance to the input to a child. Furthermore, (Chomsky, 1965) observes that human languages generally lack linear rules. If neural networks really do acquire human-like biases, they should not only form structural generalizations with ease, but also have a limited ability to form certain kinds linear generalizations. As techniques for machine language learning and self-supervised pretraining continue to advance, we expect to learn more about which linguistic universals are and are not learnable from data.
Transformers generalize differently from information stored in context vs in weights
Transformer-based architectures have an impressive ability to use both information stored in weights during training ("in-weights learning"), and information stored only in the inputs provided at inference time (without any gradient updates to the weights of the model; "in-context learning") [Chan et al., 2022]. In-context learning on pretrained models enables learning efficiently from a few examples ("few-shot learning") [Brown et al., 2020], or even efficiently compressing a large dataset ("prompt tuning") [Li and Liang, 2021, Lester et al., 2021, Sun et al., 2022. Given the evident current and future potential for this new learning paradigm, it is important and useful to understand its inductive biases, especially how it differs from in-weights learning.
Transformers generalize differently from information stored in context vs in weights
One way to understand inductive bias is by examining how models generalize to held-out data. In this work, we adapt the experimental paradigm in Dasgupta et al. [2022] that pose a classification task that distinguishes between two previously defined kinds of generalization behaviors (see 1). A "rulebased" decision is made on the basis of minimal features that support the category boundary [Ashby and Townsend, 1986], while an "exemplar-based" decision generalizes on the basis of similarity to examples from training data [Shepard and Chang, 1963], invoking many or all features available. Since BX is equally similar to AX and BW, it is equally likely to classify it as * or o.
Transformers generalize differently from information stored in context vs in weights
This distinction is particularly interesting when comparing in-weights vs in-context learning. Exemplarbased generalization (that uses all available features) is useful in a low-data regime where there is not enough information to form an abstract sparse rule [Feldman, 2020]. On the other hand, sparser rulebased generalization may help avoid sensitivity to spurious correlation when training with large, noisy, naturalistic datasets (that are commonly used to train in-weights learning).
Transformers generalize differently from information stored in context vs in weights
We find that transformers exhibit a striking difference in their generalization from in-context vs in-weights information. Transformers display a strong inductive bias towards exemplar-based generalization from incontext information. In contrast, transformers display a strong inductive bias towards rule-based generalization from in-weights information.
Transformers generalize differently from information stored in context vs in weights
However, when we pose a similar task to large transformer models pretrained on language, they exhibit stronger rule-based generalization from in-context information. One interpretation of these results is that the distribution of natural language is more compatible with rule-based generalization from context (rulebased generalization is in fact optimal in compositional domains like langauge [Arjovsky et al., 2019]), and such patterns might present strong enough learning pressure to overcome -and even reverse -transformers' inherent bias towards exemplar-based generalization from context.
Transformers generalize differently from information stored in context vs in weights
We adapted the "partial exposure" paradigm from Dasgupta et al. [2022] where each stimulus has two features; only one of the features predicts the label. We evaluate how the model generalizes to a held-out combination (using sparse rules or similarity to exemplars), see Fig 1. First, we explored generalization in transformers trained on controlled synthetic data, where we can examine generalization from both in-weights and in-context information and directly compare them. Second, we repeat this experiment on pretrained language models and characterize their in-context generalization. Finally, we compare the patterns observed and invesigate factors that explain the differences observed.
Transformers generalize differently from information stored in context vs in weights
For the trained-from-scratch transformers, we passed sequences of stimulus-label pairs as inputs to the transformer model [Vaswani et al., 2017]. The sequences consisted of two parts: a context (24 tokens; i.e. 12 stimulus-label pairs) and a query (stimulus). The model was trained to minimize a softmax cross-entropy loss on the prediction for the final (query) stimulus. Each stimulus consists of two subvectors concatenated together into a single token (Fig 4c) -these subvectors comprise the two features of the partial exposure paradigm. See Appendix A for further details.
Transformers generalize differently from information stored in context vs in weights
Generalization from in-weights information is rule-based. To investigate generalization from in-weights information, we trained the model on partial exposure data, and evaluated on the held-out combination. During training, the label for each stimulus class was fixed, so that the stimulus-label mappings were stored in weights. The context tokens are uninformative for the query. After training, Generalization from in-context information is exemplar-based. To investigate generalization from in-context information, we first pretrained the model for few-shot in-context learning (see A.2 for more details), i.e. to refer to information provided in-context when making a query prediction. Importantly, pretraining for few-shot learning imparts no bias towards either rule-based or exemplarbased generalization, because either is an equally valid strategy for solving the few-shot problems.
Transformers generalize differently from information stored in context vs in weights
Then we evaluated the model on partial exposure sequences. See Appendix Fig 4b for example sequences used for training and evaluation. When trained and evaluated in this way, transformers displayed totally exemplar-based generalization, in striking contrast to the rule-based generalization that was observed from weights. That is, when queried on the held-out combination BX, the models were equally likely to output the labels associated with either AX and BW (Fig 2b), indicating that they are comparing the query directly (and along all features) to examples that were shown in context.
Transformers generalize differently from information stored in context vs in weights
Do these patterns hold when evaluating on pretrained language models, arguably the most-used transformer-based model at the moment? We used a large (70B parameters) pre-trained language model (LM) trained for autoregressive language prediction on large-scale web data [Hoffmann et al., 2022]. To investigate in-context generalization in this model, we use the same "partial exposure" experimental paradigm, but instead using shape and color words as the two features comprising the stimuli, and nonsense words for the labels (see Fig 5 for an example). We take the completion generated by the LM as the predicted query label.
Transformers generalize differently from information stored in context vs in weights
With the synthetic data, we could ensure no a priori bias towards any feature; we have no such guarantee here. We account for any possible bias toward generalizing along a specific feature with a control condition from [Dasgupta et al., 2022] where neither feature is more predictive in the training data, so the model must use pure exemplar-based generalization. Performance in this condition is now used as the baseline for pure exemplar-based generalization, and deviation from this baseline is the measure for rule-based generalization. See Appendix B for further details. We don't investigate in-weights generalization in these models, since that would require manipulation and control of the training data and large-scale re-training.
Transformers generalize differently from information stored in context vs in weights
In language models, generalization from in-context information is partially rule-based. First, in the control condition where the model is forced to use exemplar-based generalization, we find that the language model prefers generalizing along the color dimension rather than the shape dimension (Fig 3a). Since bias towards shape gives better classification performance on naturalistic visual stimuli [Landau et al., 1988, Geirhos et al., 2018, it is interesting that we see the opposite bias in a system trained on naturalistic text data; future work should look into possible explanations (e.g. the "pragmatic" nature of language; Degen et al. [2019]). The LM predominantly produces one of the two labels provided in context, but does sometimes produce an unseen word ('other' in Fig 3). The LM produces an unseen word more frequently when the query is also unseen in-context (not reported), suggesting a mutual exclusivity bias [Gandhi and Lake, 2020], also worth future investigation.
Transformers generalize differently from information stored in context vs in weights
To measure the degree of rule-based generalization, we compared each partial exposure result to the respective control: for instance, we compared the probability of generalizing along color in the color predictive partial exposure evaluation condition (Fig 3c) vs in the control condition. If a model uses pure exemplar-based generalization, there will be no difference in the classification pattern between the partial exposure and the control conditions. Alternatively, rule-based generalization predicts an increased sensitivity to the predictive feature dimension (either shape or color) compared to control. Here, we found evidence of rule-based generalization for both color and shape. This is still in contrast with the purely exemplar-based in-context generalization we saw in our synthetic experiment.
Transformers generalize differently from information stored in context vs in weights
Smaller models are less rule-based. We investigate this further by evaluating language models of different sizes. We measure 'rule-ness' as how much more likely the model is the generalize along the predictive feature (as supported by a sparse rule) in the partial exposure condition, compared to the corresponding (model-specific) control condition. This corresponds to the difference between the purple bar and dotted control in Figs 2 and 3. We find that smaller models (1B and 7B parameters) are steadily less rule-based, with the 1B model effectively performing exact exemplar-based generalization -similar to those trained from scratch (Fig 3d, further details in Appendix 6).
Transformers generalize differently from information stored in context vs in weights
In-context generalization can be made more rule-based with pre-training.
Transformers generalize differently from information stored in context vs in weights
We did not observe the same effect of scale in transformers that were trained from scratch on synthetic data -increasing number of layers, number of attention heads, and number of classes did not lead to more rule-basedness. This may be because we were not able to achieve the necessary scale with those experiments or due to synergistic effects between scale and the type of training data.
Transformers generalize differently from information stored in context vs in weights
To evaluate the role of training data, we evaluated in-context generalization on a transformer trained on synthetic stimuli where the query explicitly required rule-based classification (more details in Appendix A.4). With this training data, the transformer learns rule-based generalization to held-out sequences 2c. Thus, while transformers exhibit inherent bias towards exemplar-based generalization from context, when trained on data that ecnourages rule-based generalization from context, they can learn to do so. This supports the interpretation that pretrained language model show rulebased generalization because natural language contains implicitly rule-based data, but crucially, this structure only seems to be picked up and used by larger models.
Transformers generalize differently from information stored in context vs in weights
We found distinct patterns of generalization when transformers generalize from information stored in weights vs in context. When trained on synthetic data, generalization from in-weights information is completely rule-based, whereas generalization from in-context information is almost entirely exemplar-based. However, a pre-trained language model is surprisingly rule-based when generalizing from in-context information, and this is increasingly true for larger models. We find that it is indeed possible to induce rule-based generalization from in-context information by pretraining a transformer on an explicitly rule-based classification problem. Together, these findings support the possibility that natural language data (perhaps because of its combinatorial nature) provides a strong learning pressure towards rule-like generalization, which works in concert with model scale. 
Transformers generalize differently from information stored in context vs in weights
context (repeat 2x and shuffle) query  Figure 4: (a) Sequences of alternating stimuli and labels are passed to a transformer. Each sequence consists of a "context" (12 stimulus-label pairs) and a "query" stimulus. The model is trained to minimize the loss on the query prediction. (b) To evaluate generalization from context, the model is first pretrained to perform in-context learning by training on few-shot sequences; stimulus classes and labels are randomly chosen for every sequence, so that the model must perform few-shot learning from context. Inductive biases are evaluated on "partial exposure" sequences, where one combination is held out for evaluation ("BX"). Consistent selection of the label associated with "B" indicates a rule-based bias, while equal selection of the labels associated with "A" and "B" indicates an exemplar-based bias (since "BX" is equally similar to "AX" and "BW"). (c) Each stimulus consists of two subvectors concatenated together into a single token. Each subvector belongs a particular class, and each class is characterized by a different centroid. The subvectors are sampled from a multivariate normal centered on that centroid. The subvectors have length 32, but here we only show 4 values. (d) To evaluate generalization from weights, the model is instead trained directly on partial exposure data, and inductive biases are evaluated on the held-out combination. (The context consisted of random samplings of the stimulus classes, and were irrelevant to the query prediction.)
Transformers generalize differently from information stored in context vs in weights
Each subvector belongs a particular "subvector class", and each subvector class is characterized by a different centroid. The subvectors are sampled from a multivariate normal centered on that centroid. A "stimulus class" is the concatenation of two subvector classes, and a stimulus is a sampling from a stimulus class. Example stimuli are shown in Fig 4. This design is a 64-dimensional generalization of the 2-D classification example from Dasgupta et al. [2022], and ensures that there is no a priori bias towards one feature or another. To pretrain the model for in-context learning, we pretrained the model on few-shot (4-shot 3-way) sequences, i.e. sequences for which the context consisted of 3 different stimulus classes each repeated 4 times, and where the query class was one of those 3 classes. The classes and labels were randomly assigned for each sequence.
Transformers generalize differently from information stored in context vs in weights
Models were trained and evaluated as shown in Figs 4b and 4d.
Transformers generalize differently from information stored in context vs in weights
An additional label ("2", in the example in Fig 4b) and corresponding examples were included in the partial exposure sequences, to ensure that if the model is equally likely to select the labels associated with "A" and "B", it is because of those examples' similarity to the query stimulus, rather than because the model is selecting labels at chance. This was similarly done for training on partial exposure data in weights (not shown in Fig 4d).
Transformers generalize differently from information stored in context vs in weights
Note also that the BW stimulus was shown twice as often as other stimuli in the partial exposure data, as was done in Dasgupta et al. [2022], in order to ensure that there is no bias induced by having one label more frequent than another. We also ran experiments where BW was not repeated, and the pattern of results was the same.
Transformers generalize differently from information stored in context vs in weights
To pretrain the model for rule-based generalization, we used the partial exposure sequences shown in 4b, but for training as well as for evaluation. In training, the label for the query BX was always the same as the label associated with BW in context (note that the stimuli and labels are randomly assigned, so that BX could be manifested as any of the stimulus classes, and the query label could be any of [0, 1, 2].)
Transformers generalize differently from information stored in context vs in weights
A.5 Architecture, training, and evaluation details Num layers: 12 | Embedding size: 64 | Optimizer: Adam | Batch size: 32 | Learning rate schedule: Linear warmup and square root decay, described as min(3e-4 / 4000 * global_step, power(4000, 0.5) * 3e-4 * power(global_step, -0.5))
Transformers generalize differently from information stored in context vs in weights
For each experiment, we ran 16 TPUv3 cores and 4 v100 GPU cores for 200,000 training steps.
Transformers generalize differently from information stored in context vs in weights
Models are evaluated on evaluation data throughout training, and bar plots in Fig 2 show  In order to account for potential low-level word-order effects, we evaluated on four different formats for the stimuli ('red circle', 'a circle that is red', 'an object that is circular and red', 'an object that is red and circular'). We found no significant differences in qualitative patterns across the different word orderings, so we report the average across all four formats in our results.
Transformers generalize differently from information stored in context vs in weights
We also include more detailed information about the experiments using different model sizes in Fig 6. These show the raw performances (including model-specific controls) on all model sizes and feature sets.
Transformers generalize differently from information stored in context vs in weights
a blue triangle is ?
Transformers generalize differently from information stored in context vs in weights
context (repeat 6x and shuffle) query Evaluation: Control a red triangle is hib, a blue square is fep, a red square is fep a blue triangle is ?
Transformers generalize differently from information stored in context vs in weights
context (repeat 6x and shuffle) query Evaluation: Partial exposure (shape predictive) a red triangle is hib, a blue square is fep a red triangle is hib, a blue square is fep, a red square is hib a blue triangle is ?
Transformers generalize differently from information stored in context vs in weights
context (repeat 6x and shuffle) query Evaluation: Partial exposure (color predictive) Figure 5: To evaluate generalization from context in a pretrained language model, the model is evaluated on partial exposure sequences where the features are instead text features (shape and color words). The control condition allows us to evaluate the model's baseline bias towards shape or color.
Transformers generalize differently from information stored in context vs in weights
(a) Control condition.    Figure 6: Results on the pretrained language models for different sizes.
Information-Theoretic Probing with Minimum Description Length
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates 'the amount of effort' needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes. 1
Information-Theoretic Probing with Minimum Description Length
To estimate to what extent representations (e.g., ELMo (Peters et al., 2018) or BERT (Devlin et al., 2019)) capture a linguistic property, most previous 1 We release code at https://github.com/ lena-voita/description-length-probing. work uses 'probing tasks' (aka 'probes' and 'diagnostic classifiers'); see Belinkov and Glass (2019) for a comprehensive review. These classifiers are trained to predict a linguistic property from 'frozen' representations, and accuracy of the classifier is used to measure how well these representations encode the property.
Information-Theoretic Probing with Minimum Description Length
Despite widespread adoption of such probes, they fail to adequately reflect differences in representations. This is clearly seen when using them to compare pretrained representations with randomly initialized ones (Zhang and Bowman, 2018). Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for tags randomly associated to word types ('control tasks', Hewitt and Liang (2019)). To see differences in the accuracy with respect to these random baselines, previous work had to reduce the amount of a probe training data (Zhang and Bowman, 2018) or use smaller models for probes (Hewitt and Liang, 2019).
Information-Theoretic Probing with Minimum Description Length
As an alternative to the standard probing, we take an information-theoretic view at the task of measuring relations between representations and labels. Any regularity in representations with respect to labels can be exploited both to make predictions and to compress these labels, i.e., reduce length of the code needed to transmit them. Formally, we recast learning a model of data (i.e., training a probing classifier) as training it to transmit the data (i.e., labels) in as few bits as possible. This naturally leads to a change of measure: instead of evaluating probe accuracy, we evaluate minimum description length (MDL) of labels given representations, i.e. the minimum number of bits needed to transmit the labels knowing the representations. Note that since labels are transmitted using a model, the model has to be transmitted as well (directly or indirectly). Thus, the overall codelength is a combination of the quality of fit of the model (compressed data length) with the cost of transmitting the model itself.
Information-Theoretic Probing with Minimum Description Length
Intuitively, codelength characterizes not only the final quality of a probe, but also the 'amount of effort' needed achieve this quality (Figure 1). If representations have some clear structure with respect to labels, the relation between the representations and the labels can be understood with less effort; for example, (i) the 'rule' predicting the label (i.e., the probing model) can be simple, and/or (ii) the amount of data needed to reveal this structure can be small. This is exactly how our vague (so far) notion of 'the amount of effort' is translated into codelength. We explain this more formally when describing the two methods for evaluating MDL we use: variational coding and online coding; they differ in a way they incorporate model cost: directly or indirectly.
Information-Theoretic Probing with Minimum Description Length
Variational code explicitly incorporates cost of transmitting the model (probe weights) in addition to the cost of transmitting the labels; this joint cost is exactly the loss function of a variational learning algorithm (Honkela and Valpola, 2004). As we will see in the experiments, close probe accuracies often come at a very different model cost: the 'rule' (the probing model) explaining regularity in the data can be either simple (i.e., easy to communicate) or complicated (i.e., hard to communicate) depending on the strength of this regularity.
Information-Theoretic Probing with Minimum Description Length
Online code provides a way to transmit data without directly transmitting the model. Intuitively, it measures the ability to learn from different amounts of data. In this setting, the data is transmitted in a sequence of portions; at each step, the data transmitted so far is used to understand the regularity in this data and compress the following portion. If the regularity in the data is strong, it can be revealed using a small subset of the data, i.e., early in the transmission process, and can be exploited to efficiently transmit the rest of the dataset. The online code is related to the area under the learning curve, which plots quality as a function of the number of training examples.
Information-Theoretic Probing with Minimum Description Length
If we now recall that, to get reasonable differ-ences with random baselines, previous work manually tuned (i) model size and/or (ii) the amount of data, we will see that these were indirect ways of accounting for the 'amount of effort' component of (i) variational and (ii) online codes, respectively. Interestingly, since variational and online codes are different methods to estimate the same quantity (and, as we will show, they agree in the results), we can conclude that the ability of a probe to achieve good quality using a small amount of data and its ability to achieve good quality using a small probe architecture reflect the same property: strength of the regularity in the data. In contrast to previous work, MDL incorporates this naturally in a theoretically justified way. Moreover, our experiments show that, differently from accuracy, conclusions made by MDL probes are not affected by an underlying probe setting, thus no manual search for settings is required.
Information-Theoretic Probing with Minimum Description Length
We illustrate the effectiveness of MDL for different kinds of random baselines. For example, when considering control tasks (Hewitt and Liang, 2019), while probes have similar accuracies, these accuracies are achieved with a small probe model for the linguistic task and a large model for the random baseline (control task); these architectures are obtained as a byproduct of MDL optimization and not by manual search.
Information-Theoretic Probing with Minimum Description Length
Our contributions are as follows:
Information-Theoretic Probing with Minimum Description Length
• we propose information-theoretic probing which measures MDL of labels given representations;
Information-Theoretic Probing with Minimum Description Length
• we show that MDL naturally characterizes not only probe quality, but also 'the amount of effort' needed to achieve it;
Information-Theoretic Probing with Minimum Description Length
• we explain how to easily measure MDL on top of standard probe-training pipelines;
Information-Theoretic Probing with Minimum Description Length
• we show that results of MDL probing are more informative and stable than those of standard probes.
Information-Theoretic Probing with Minimum Description Length
Let D = {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x n , y n )} be a dataset, where x 1:n = (x 1 , x 2 , . . . , x n ) are representations from a model and y 1:n = (y 1 , y 2 , . . . , y n ) are labels for some linguistic task (we assume that y i ∈ {1, 2, . . . , K}, i.e. we consider classification tasks). As in standard probing task, we want to measure to what extent x 1:n encode y 1:n . Differently from standard probes, we propose to look at this question from the information-theoretic perspective and define the goal of a probe as learning to effectively transmit the data.
Information-Theoretic Probing with Minimum Description Length
Setting. Following the standard information theory notation, let us imagine that Alice has all (x i , y i ) pairs in D, Bob has just the x i 's from D, and that Alice wants to communicate the y i 's to Bob. The task is to encode the labels y 1:n knowing the inputs x 1:n in an optimal way, i.e. with minimal codelength (in bits) needed to transmit y 1:n .
Information-Theoretic Probing with Minimum Description Length
Transmission: Data and Model. Alice can transmit the labels using some probabilistic model of data p(y|x) (e.g., it can be a trained probing classifier). Since Bob does not know the precise trained model that Alice is using, some explicit or implicit transmission of the model itself is also required. In Section 2.1, we explain how to transmit data using a model p. In Section 2.2, we show direct and indirect ways of transmitting the model.
Information-Theoretic Probing with Minimum Description Length
Interpretation: quality and 'amount of effort'. In Section 2.3, we show that total codelength characterizes both probe quality and the 'amount of effort' needed to achieve it. We draw connections between different interpretations of this 'amount of effort' part of the code and manual search for probe settings done in previous work. 2
Information-Theoretic Probing with Minimum Description Length
Suppose that Alice and Bob have agreed in advance on a model p, and both know the inputs x 1:n . Then there exists a code to transmit the labels y 1:n losslessly with codelength 3 L p (y 1:n |x 1:
Information-Theoretic Probing with Minimum Description Length
This is the Shannon-Huffman code, which gives an optimal bound on the codelength if the data are independent and come from a conditional probability distribution p(y|x).
Information-Theoretic Probing with Minimum Description Length
Learning is compression. The bound (1) is exactly the categorical cross-entropy loss evaluated on the model p. This shows that the task of compressing labels y 1:n is equivalent to learning a model p(y|x): quality of a learned model p(y|x) is the codelength needed to transmit the data.
Information-Theoretic Probing with Minimum Description Length
Compression is usually compared against uniform encoding which does not require any learning from data. It assumes p(y|x) = p unif (y|x) = 1 K , and yields codelength L unif (y 1:n |x 1:n ) = n log 2 K bits. Another trivial encoding ignores input x and relies on class priors p(y), resulting in codelength H(y).
Information-Theoretic Probing with Minimum Description Length
Relation to Mutual Information. If the inputs and the outputs come from a true joint distribution q(x, y), then, for any transmission method with codelength L, it holds E q [L(y|x)] ≥ H(y|x) (Grunwald, 2004). Therefore, the gain in codelength over the trivial codelength H(y) is
Information-Theoretic Probing with Minimum Description Length
In other words, the compression is limited by the mutual information (MI) between inputs (i.e. pretrained representations) and outputs (i.e. labels).
Information-Theoretic Probing with Minimum Description Length
Note that total codelength includes model codelength in addition to the data code. This means that while high MI is necessary for effective compression, a good representation is the one which also yields simple models predicting y from x, as we formalize in the next section.
Information-Theoretic Probing with Minimum Description Length
We consider two compression methods that can be used with deep learning models (probing classifiers):
Information-Theoretic Probing with Minimum Description Length
• variational code -an instance of two-part codes, where a model is transmitted explicitly and then used to encode the data;
Information-Theoretic Probing with Minimum Description Length
• online code -a way to encode both model and data without directly transmitting the model.
Information-Theoretic Probing with Minimum Description Length
We assume that Alice and Bob have agreed on a model class H = {p θ |θ ∈ Θ}. With two-part codes, for any model p θ * , Alice first transmits its parameters θ * and then encodes the data while relying on the model. The description length decomposes accordingly:
Information-Theoretic Probing with Minimum Description Length
To compute the description length of the parameters L param (θ * ), we can further assume that Alice and Bob have agreed on a prior distribution over the parameters α(θ * ). Now, we can rewrite the total description length as
Information-Theoretic Probing with Minimum Description Length
where m is the number of parameters and is a prearranged precision for each parameter. With deep learning models, such straightforward codes for parameters are highly inefficient. Instead, in the variational approach, weights are treated as random variables, and the description length is given by the expectation
Information-Theoretic Probing with Minimum Description Length
where
Information-Theoretic Probing with Minimum Description Length
is a distribution encoding uncertainty about the parameter values. The distribution β(θ) is chosen by minimizing the codelength given in Expression (3). The formal justification for the description length relies on the bits-back argument (Hinton and von Cramp, 1993;Honkela and Valpola, 2004;MacKay, 2003). However, the underlying intuition is straightforward: parameters we are uncertain about can be transmitted at a lower cost as the uncertainty can be used to determine the required precision. The entropy term in Equation (3), H(β) = E θ∼β log 2 β(θ), quantifies this discount.
Information-Theoretic Probing with Minimum Description Length
The negated codelength −L var β (y 1:n |x 1:n ) is known as the evidence-lower-bound (ELBO) and used as the objective in variational inference. The distribution β(θ) approximates the intractable posterior distribution p(θ|x 1:n , y 1:n ). Consequently, any variational method can in principle be used to estimate the codelength.
Information-Theoretic Probing with Minimum Description Length
In our experiments, we use the network compression method of Louizos et al. (2017). Similarly to variational dropout (Molchanov et al., 2017), it uses sparsity-inducing priors on the parameters, pruning neurons from the probing classifier as a byproduct of optimizing the ELBO. As a result we can assess the probe complexity both using its description length KL(β α) and by inspecting the discovered architecture.
Information-Theoretic Probing with Minimum Description Length
The online (or prequential) code (Rissanen, 1984) is a way to encode both the model and the labels without directly encoding the model weights. In the online setting, Alice and Bob agree on the form of the model p θ (y|x) with learnable parameters θ, its initial random seeds, and its learning algorithm. They also choose timesteps 1 = t 0 < t 1 < · · · < t S = n and encode data by blocks. 4 Alice starts by communicating y 1:t 1 with a uniform code, then both Alice and Bob learn a model p θ 1 (y|x) that predicts y from x using data {(x i , y i )} t 1 i=1 , and Alice uses that model to communicate the next data block y t 1 +1:t 2 . Then both Alice and Bob learn a
Information-Theoretic Probing with Minimum Description Length
and use it to encode y t 2 +1:t 3 . This process continues until the entire dataset has been transmitted. The resulting online codelength is
Information-Theoretic Probing with Minimum Description Length
In this sequential evaluation, a model that performs well with a limited number of training examples will be rewarded by having a shorter codelength (Alice will require fewer bits to transmit the subsequent y t i :t i+1 to Bob). The online code is related to the area under the learning curve, which plots quality (in case of probes, accuracy) as a function of the number of training examples. We will illustrate this in Section 3.2.
Information-Theoretic Probing with Minimum Description Length
Connection to previous work. To get larger differences in scores compared to random baselines, previous work tried to (i) reduce size of a probing model and (ii) reduce the amount of a probe training data. Now we can see that these were indirect ways to account for the 'amount of effort' component of (i) variational and (ii) online codes, respectively.
Information-Theoretic Probing with Minimum Description Length
Online code and model size. While the online code does not incorporate model cost explicitly, we can still evaluate model cost by interpreting the difference between the cross-entropy of the model trained on all data and online codelength as the cost of the model. The former is codelength of the data if one knows model parameters, the latter (online codelength) -if one does not know them. In Section 3.2 we will show that trends for model cost evaluated for the online code are similar to those for the variational code. It means that in terms of a code, the ability of a probe to achieve good quality using small amount of data or using a small probe architecture reflect the same property: the strength of the regularity in the data.
Information-Theoretic Probing with Minimum Description Length
Which code to choose? In terms of implementation, the online code uses a standard probe along with its training setting: it trains the probe on increasing subsets of the dataset. Using the variational code requires changing (i) a probing model to a Bayesian model and (ii) the loss function to the corresponding variational loss (3) (i.e. adding the model KL term to the standard data cross-entropy). As we will show later, these methods agree in results. Therefore, the choice of the method can be done depending on the preferences: the variational code can be used to inspect the induced probe architecture, but the online code is easier to implement.
Information-Theoretic Probing with Minimum Description Length
Hewitt and Liang (2019) noted that probe accuracy itself does not necessarily reveal if the representations encode the linguistic annotation or if the probe 'itself' learned to predict this annotation. They introduced control tasks which associate word types with random outputs, and each word token is assigned its type's output, regardless of context. By construction, such tasks can only be learned by the probe itself. They argue that selectivity, i.e. difference between linguistic task accuracy and control task accuracy, reveals how much the linguistic probe relies on the regularities encoded in the representations. They propose to tune probe hyperparameters so that to maximize selectivity. In contrast, we will show that MDL probes do not require such tuning.
Information-Theoretic Probing with Minimum Description Length
In all experiments, we use the data and follow the setting of Hewitt and Liang (2019); we build on top of their code and release our extended version to reproduce the experiments.
Information-Theoretic Probing with Minimum Description Length
In the main text, we use a probe with default hyperparameters which was a starting point in Hewitt and Liang (2019) and was shown to have low selectivity. In the appendix, we provide results for 10 different settings and show that, in contrast to accuracy, codelength is stable across settings.
Information-Theoretic Probing with Minimum Description Length
Task: part of speech. Control tasks were designed for two tasks: part-of-speech (PoS) tagging and dependency edge prediction. In this work, we focus only on the PoS tagging task, the task of assigning tags, such as noun, verb, and adjective, to individual word tokens. For the control task, for each word type, a PoS tag is independently sampled from the empirical distribution of the tags in the linguistic data.
Information-Theoretic Probing with Minimum Description Length
Data. The pretrained model is the 5.5 billionword pre-trained ELMo (Peters et al., 2018). The data comes from Penn Treebank (Marcus et al., 1993) with the traditional parsing training/development/testing splits 5 without extra preprocessing. Table 1 shows dataset statistics.
Information-Theoretic Probing with Minimum Description Length
Probes. The probe is MLP-2 of Hewitt and Liang (2019) with the default hyperparameters. Namely, it is a multi-layer perceptron with two hidden layers defined as:
Information-Theoretic Probing with Minimum Description Length
; hidden layer size h is 1000 and no dropout is used. Additionally, in the appendix, we provide results for both MLP-2 and MLP-1 for several h values: 1000, 500, 250, 100, 50.
Information-Theoretic Probing with Minimum Description Length
For the variational code, we replace dense layers with the Bayesian compression layers from Louizos et al. (2017); the loss function changes to Eq. (3).
Information-Theoretic Probing with Minimum Description Length
Optimizer. All of our probing models are trained with Adam (Kingma and Ba, 2015) with learning rate 0.001. With standard probes, we follow the original paper (Hewitt and Liang, 2019) and anneal the learning rate by a factor of 0.5 once the epoch does not lead to a new minimum loss on the development set; we stop training when 4 such epochs occur in a row. With variational probes, we do not anneal learning rate and train probes for 200 epochs; long training is recommended to enable pruning (Louizos et al., 2017).
Information-Theoretic Probing with Minimum Description Length
Results are shown in   Table 2: Experimental results; shown in pairs: linguistic task / control task. Codelength is measured in kbits (variational codelength is given in equation (3), online -in equation (4)). Accuracy is shown for the standard probe as in Hewitt and Liang (2019); for the variational probe, scores are similar (see Table 3). Different compression methods, similar results. First, we see that both compression methods show similar trends in codelength. For the linguistic task, the best layer is the first one. For the control task, codes become larger as we move up from the embedding layer; this is expected since the control task measures the ability to memorize word type. Note that codelengths for control tasks are substantially larger than for the linguistic task (at least twice larger). This again illustrates that description length is preferable to probe accuracy: in contrast to accuracy, codelength is able to distinguish these tasks without any search for settings.
Information-Theoretic Probing with Minimum Description Length
LAYER 0: MDL is correct, accuracy is not.
Information-Theoretic Probing with Minimum Description Length
What is even more surprising, codelength identifies the control task even when accuracy indicates the opposite: for LAYER 0, accuracy for the control task is higher, but the code is twice longer than for the linguistic task. This is because codelength characterizes how hard it is to achieve this accuracy: for the control task, accuracy is higher, but the cost of achieving this score is very big. We will illustrate this later in this section.
Information-Theoretic Probing with Minimum Description Length
Embedding vs contextual: drastic difference.
Information-Theoretic Probing with Minimum Description Length
For the linguistic task, note that codelength for the embedding layer is approximately twice larger than that for the first layer. Later in Section 4 we will see the same trends for several other tasks, and will show that even contextualized representations obtained with a randomly initialized model are a lot better than with the embedding layer alone.
Information-Theoretic Probing with Minimum Description Length
Model: small for linguistic, large for control. Figure 2(a) shows data and model components of the variational code. For control tasks, model size is several times larger than for the linguistic task. This is something that probe accuracy alone is not able to reflect: representations have structure with respect to the linguistic labels and this structure can be 'explained' with a small model. The same representations do not have structure with respect to random labels, therefore these labels can be predicted only using a larger model. Using interpretation from Section 2.3 to split  the online code into data and model codelength, we get Figure 2(b). The trends are similar to the ones with the variational code; but with the online code, the model component shows how easy it is to learn from small amount of data: if the representations have structure with respect to some labels, this structure can be revealed with a few training examples. Figure  Architecture: sparse for linguistic, dense for control. The method for the variational code we use, Bayesian compression of Louizos et al. (2017), lets us assess the induced probe complexity not only by using its description length (as we did above), but also by looking at the induced architecture (Table 3). Probes learned for linguistic tasks are much smaller than those for control tasks, with only 33-75 neurons at the second and third layers. This relates to previous work (Hewitt and Liang, 2019). The authors considered several predefined probe architectures and picked one of them based on a manually defined criterion. In contrast, the variational code gives probe architecture as a byproduct of training and does not need human guidance.
Information-Theoretic Probing with Minimum Description Length
Here we discuss stability of MDL results across compression methods, underlying probing classifier setting and random seeds.
Information-Theoretic Probing with Minimum Description Length
The two compression methods agree in results.
Information-Theoretic Probing with Minimum Description Length
Note that the observed agreement in codelengths Figure 3: Results for 10 probe settings: accuracy is wrong for 8 out of 10 settings, MDL is always correct (for accuracy higher is better, for codelength -lower).
Information-Theoretic Probing with Minimum Description Length
from different methods (Table 2) is rather surprising: this contrasts to Blier and Ollivier (2018), who experimented with images (MNIST, CIFAR-10) and argued that the variational code yields very poor compression bounds compared to online code. We can speculate that their results may be due to the particular variational approach they use. The agreement between different codes is desirable and suggests sensibility and reliability of the results.
Information-Theoretic Probing with Minimum Description Length
Hyperparameters: change results for accuracy, do not for MDL. While here we will discuss in detail results for the default settings, in the appendix we provide results for 10 different settings; for LAYER 0, results are given in Figure 3. We see that accuracy can change greatly with the settings. For example, difference in accuracy for linguistic and control tasks varies a lot; for LAYER 0 there are settings with contradictory results: accuracy can be higher either for the linguistic or for the control task depending on the settings (Figure 3). In striking contrast to accuracy, MDL results are stable across settings, thus MDL does not require search for probe settings.
Information-Theoretic Probing with Minimum Description Length
Random seed: affects accuracy but not MDL. We evaluated results from Table 2 for random seeds from 0 to 4; for the linguistic task, results are shown in Figure 2(d). We see that using accuracy can lead to different rankings of layers depending on a random seed, making it hard to draw conclusions about their relative qualities. For example, accuracy for LAYER 1 and LAYER 2 are 97.48 and 97.31 for seed 1, but 97.38 and 97.48 for seed 0. On the contrary, the MDL results are stable and the scores given to different layers are well separated. Note that for this 'real' task, where the true ranking of layers 1 and 2 is not known in advance, tuning a probe setting by maximizing difference with the synthetic control task (as done by Hewitt and Liang (2019)) does not help: in the tuned setting, scores for these layers remain very close (e.g., 97.3 and 97.0 (Hewitt and Liang, 2019)).   
Information-Theoretic Probing with Minimum Description Length
Now, from random labels for word types, we come to another type of random baselines: randomly initialized models. Probes using these representations show surprisingly strong performance for both token (Zhang and Bowman, 2018) and sentence (Wieting and Kiela, 2019) representations. This again confirms that accuracy alone does not reflect what a representation encodes. With MDL probes, we will see that codelength shows large difference between trained and randomly initialized representations.
Information-Theoretic Probing with Minimum Description Length
In this part, we also experiment with ELMo and compare it with a version of the ELMo model in which all weights above the lexical layer (LAYER 0) are replaced with random orthonormal matrices (but the embedding layer, LAYER 0, is retained from trained ELMo). We conduct a line of experiments using a suite of edge probing tasks (Tenney et al., 2019). In these tasks, a probing model (Figure 4) can access only representations within given spans, such as a predicate-argument pair, and must predict properties, such as semantic roles.
Information-Theoretic Probing with Minimum Description Length
Tasks and datasets. We focus on several core NLP tasks: PoS tagging, syntactic constituent and dependency labeling, named entity recognition, se-Figure 4: Probing model architecture for an edge probing task. The example is for semantic role labeling; for PoS, NER and constituents, only a single span is used. mantic role labeling, coreference resolution, and relation classification. Examples for each task are shown in Table 4, dataset statistics are in Table 5. See extra details in Tenney et al. (2019).
Information-Theoretic Probing with Minimum Description Length
We follow Tenney et al. (2019) and use ELMo (Peters et al., 2018) trained on the Billion Word Benchmark dataset (Chelba et al., 2014).
Information-Theoretic Probing with Minimum Description Length
Probes and optimization. Probing architecture is illustrated in Figure 4. It takes a list of contextual vectors [e 0 , e 1 , . . . , e n ] and integer spans s 1 = [i 1 , j 1 ) and (optionally) s 2 = [i 2 , j 2 ) as inputs, and uses a projection layer followed by the self-attention pooling operator of Lee et al. (2017) to compute fixed-length span representations. The span representations are concatenated and fed into a two-layer MLP followed by a softmax output 8.4 / 9.7 3.40 / 2.92 8.6 / 11.7 3.3 / 2.4 Table 6: Experimental results; shown in pairs: trained model / randomly initialized model. Codelength is measured in kbits (variational codelength is given in equation (3), online -in equation (4)), compression -with respect to the corresponding uniform code. layer. As in the original paper, we use the standard cross-entropy loss, hidden layer size of 256 and dropout of 0.3. For further details on training, we refer the reader to the original paper by Tenney et al. (2019). 7 For the variational code, the layers are replaced with that of Bayesian compression by Louizos et al. (2017); loss function changes to (3) and no dropout 7 The differences with the original implementation by Tenney et al. (2019) are: softmax with the cross-entropy loss instead of sigmoid with binary cross-entropy, using the loss instead of F1 in the early stopping criterion. is used. Similar to the experiments in the previous section, we do not anneal learning rate and train at least 200 epochs to enable pruning.
Information-Theoretic Probing with Minimum Description Length
We build our experiments on top of the original code by Tenney et al. (2019) and release our extended version.
Information-Theoretic Probing with Minimum Description Length
Results are shown in Table 6. LAYER 0 vs contextual. As we have already seen in the previous section, codelength shows dras-tic difference between the embedding layer (LAYER 0) and contextualized representations: codelengths differ about twice for most of the tasks. Both compression methods show that even for the randomly initialized model, contextualized representations are better than lexical representations. This is because context-agnostic embeddings do not contain enough information about the task, i.e., MI between labels and context-agnostic representations is smaller than between labels and contextualized representations. Since compression of the labels given model (i.e., data component of the code) is limited by the MI between the representations and the labels (Section 2.1), the data component of the codelength is much bigger for the embedding layer than for contextualized representations.
Information-Theoretic Probing with Minimum Description Length
Trained vs random. As expected, codelengths for the randomly initialized model are larger than for the trained one. This is more prominent when not just looking at the bare scores, but comparing compression against context-agnostic representations. For all tasks, compression bounds for the randomly initialized model are closer to those of context-agnostic LAYER 0 than representations from the trained model. This shows that gain from using context for the randomly initialized model is at least twice smaller than for the trained model.
Information-Theoretic Probing with Minimum Description Length
Note also that randomly initialized layers do not evolve: for all tasks, MDL for layers of the randomly initialized model is the same. Moreover, Table 7 shows that not only total codelength but data and model components of the code for random model layers are also identical. For the trained model, this is not the case: LAYER 2 is worse than LAYER 1 for all tasks. This is one more illustration of the general process explained in Voita et al. (2019a): the way representations evolve between layers is defined by the training objective. For the randomly initialized model, since no training objective has been optimized, no evolution happens.
Information-Theoretic Probing with Minimum Description Length
Probing classifiers are the most common approach for associating neural network representations with linguistic properties (see Belinkov and Glass (2019) for a survey). Among the works highlighting limitations of standard probes (not mentioned earlier) is the work by Saphra and Lopez (2019), who show that diagnostic classifiers are not suitable for understanding learning dynamics.
Information-Theoretic Probing with Minimum Description Length
In addition to task performance, learning curves have also been used before by Yogatama et al. (2019) to evaluate how quickly a model learns a new task, and by Talmor et al. (2019) to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data.
Information-Theoretic Probing with Minimum Description Length
Other methods for analyzing NLP models include (i) inspecting the mechanisms a model uses to encode information, such as attention weights (Voita et al., 2018;Raganato and Tiedemann, 2018;Voita et al., 2019b;Clark et al., 2019;Kovaleva et al., 2019) or individual neurons (Karpathy et al., 2015;Pham et al., 2016;Bau et al., 2019), (ii) looking at model predictions using manually defined templates, either evaluating sensitivity to specific grammatical errors (Linzen et al., 2016;Gulordava et al., 2018;Tran et al., 2018;Marvin and Linzen, 2018) or understanding what language models know when applying them as knowledge bases or in question answering settings (Radford et al., 2019;Petroni et al., 2019;Poerner et al., 2019;Jiang et al., 2019).
Information-Theoretic Probing with Minimum Description Length
An information-theoretic view on analysis of NLP models has been previously attempted in Voita et al. (2019a) when explaining how representations in the Transformer evolve between layers under different training objectives.
Information-Theoretic Probing with Minimum Description Length
We propose information-theoretic probing which measures minimum description length (MDL) of labels given representations. We show that MDL naturally characterizes not only probe quality, but also 'the amount of effort' needed to achieve it (or, intuitively, strength of the regularity in representations with respect to the labels); this is done in a theoretically justified way without manual search for settings. We explain how to easily measure MDL on top of standard probe-training pipelines. We show that results of MDL probing are more informative and stable compared to the standard probes.  Table 8: Experimental results; shown in pairs: linguistic task / control task. Codelength is measured in kbits (variational codelength is given in equation (3), online -in equation (4)). h is the probe hidden layer size.
Information-Theoretic Probing with Minimum Description Length
A.2 Random seeds: control task Results are shown in Figure 5.         
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Any finite training set is consistent with multiple generalizations. Therefore, the way that a learner generalizes to unseen examples depends not only on the training data but also on properties of the learner. Suppose a learner is told that a blue triangle is an example of a blick. A learner preferring shape-based generalizations would conclude that blick means ''triangle,'' while a learner preferring color-based generalizations would conclude that blick means ''blue object'' (Landau et al., 1988). Factors that guide a learner to choose one generalization over another are called inductive biases.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
What properties of a learner cause it to have a particular inductive bias? We investigate this question with respect to sequence-to-sequence neural networks (Botvinick and Plaut, 2006;Sutskever et al., 2014). As a test case for studying differences in how models generalize, we use the syntactic task of English question formation, such as transforming (1a) into (1b):
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(1) a. The zebra does chuckle.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
b. Does the zebra chuckle?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Following Chomsky's (1980) empirical claims about children's linguistic input, we constrain our training set to be consistent with two possible rules illustrated in Figure  (2) a. Your zebras that don't dance do chuckle. b. Do your zebras that don't dance chuckle? c. Don't your zebras that dance do chuckle?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Since no such examples appear in the training set, a model's behavior on them reveals which rule the model is biased toward. This task allows us to study a particular bias, namely, a bias for hierarchical generalization, which is important for models of 125 Figure 1: Two potential rules for English question formation.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
language because it has been argued to underlie human language acquisition (Chomsky, 1965).
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To test which models have a hierarchical bias, we use the question formation task and a second task: tense reinflection. For both tasks, our training set is ambiguous between a hierarchical generalization and a linear generalization. If a model chooses the hierarchical generalization for only one task, this preference is likely due to taskspecific factors rather than a general hierarchical bias. On the other hand, a consistent preference for hierarchical generalizations across tasks would provide converging evidence that a model has a hierarchical bias. We find that all the factors we tested can qualitatively affect how a model generalizes on the question formation task. These factors are the type of recurrent unit, the type of attention, and the choice of sequential vs. treebased model structure. Even though all these factors affected the model's decision between MOVE-MAIN and MOVE-FIRST, only the use of a treebased model can be said to impart a hierarchical bias, since this was the only model type that chose a hierarchical generalization across both of our tasks. Specific findings that support these general conclusions include:
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
• Generalization behavior is profoundly affected by the type of recurrent unit and the type of attention, and also by the interactions between these factors.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
• LSTMs and GRUs have qualitatively different inductive biases. The difference appears at least partly due to the fact that the values in GRU hidden states are bounded within a particular interval (Weiss et al., 2018).
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
• Only a model built around the correct tree structure displayed a robust hierarchical bias across tasks. Sequentially structured models failed to generalize hierarchically even when the input contained explicit marking of each sentence's hierarchical structure.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Overall, we conclude that many factors can qualitatively affect a model's inductive biases, but human-like syntactic generalization may require specific types of high-level structure, at least when learning from text alone.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
2 The Question Formation Task
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The classic discussion of the acquisition of English question formation begins with two empirical claims: (i) disambiguating examples such as Example (2) rarely occur in a child's linguistic input, but (ii) all learners of English nevertheless acquire MOVE-MAIN rather than MOVE-FIRST. Chomsky (1965Chomsky ( , 1980 uses these points to argue that humans must have an innate bias toward learning syntactic rules that are based on hierarchy rather than linear order (this argument is known as the argument from the poverty of the stimulus).
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
There has been a long debate about this line of argument. Though some have discussed the validity of Chomsky's empirical claims (Crain and Nakayama, 1987;Ambridge et al., 2008;Pullum and Scholz, 2002;Legate and Yang, 2002), most of the debate has been about which mechanisms could explain the preference for MOVE-MAIN. These mechanisms include an assumption of substitutability (Clark and Eyraud, 2007), a bias for simplicity (Perfors et al., 2011), exploitation of statistical patterns (Lewis and Elman, 2001;Reali and Christiansen, 2005), and semantic knowledge (Fitz and Chang, 2017); see Clark and Lappin (2010) for in-depth discussion.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
These past works focus on the content of the bias that favors MOVE-MAIN (i.e., which types of generalizations the bias supports), but we instead focus on the source of this bias (i.e., which factors of the learner give rise to the bias). In the book Rethinking Innateness, Elman et al. (1998) argue that innate biases in humans must arise from architectural constraints on the neural connections in the brain rather than from constraints stated at the symbolic level, under the assumption that symbolic constraints are unlikely to be specified in the genome. Here we use artificial neural networks to investigate whether syntactic inductive biases can emerge from architectural constraints.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Following Frank and Mathis (2007) and McCoy et al. (2018), we train models to take a declarative sentence as input and to either output the same sentence unchanged, or transform that sentence into a question. The sentences were generated from a context-free grammar containing only the sentence types shown in Figure 2 and using a 68-word vocabulary; the full grammar is at the project Web site. 1 The different types of sentences vary in the linear position of the main auxiliary, such that a model cannot identify the main auxiliary with a simple positional heuristic. The task to be performed is indicated by the final input token, as in Examples (3) and (4): (3) a. Input: your zebra does read . DECL b. Output: your zebra does read .
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(4) a. Input: your zebra does read . QUEST b. Output: does your zebra read ?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
During training, all question formation examples are consistent with both MOVE-FIRST and MOVE-MAIN, such that there is no direct evidence favoring one rule over the other (see Figure 2).
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To assess how models generalize, we evaluate them on a generalization set consisting of examples where MOVE-MAIN and MOVE-FIRST make different predictions due to the presence of a relative clause on the subject (see sentence (2a)). 1 Our code is at github.com/tommccoy1/rnnhierarchical-biases. Results for the over 3,500 models trained for this paper, with example outputs, are at rtmccoy.com/rnn hierarchical biases.html; only aggregate (median) results are reported here.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We focus on two metrics. The first is full-sentence accuracy on the test set. That is, for examples drawn from the same distribution as the training set, does the model get the output exactly right?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
For testing generalization to the withheld example type, a natural metric would be fullsentence accuracy on the generalization set. However, in preliminary experiments we found that most models rarely produced the exact output predicted by either MOVE-MAIN or MOVE-FIRST, as they tend to truncate the output, confuse similar words, and make other extraneous errors. To abstract away from such errors, we use first-word accuracy on the generalization set. With both MOVE-FIRST and MOVE-MAIN, the first word of the question is the auxiliary that has been moved from within the sentence. If the auxiliaries in the relative and main clauses are distinct, this word alone is sufficient to differentiate the two rules. For example, in the bottom right cell of Figure 2, MOVE-MAIN predicts having do at the start, whereas MOVE-FIRST predicts don't. 2 Models almost always produced either the main auxiliary or the first auxiliary as the first word of the output (over 98% of the time for most models 3 ), so a low first-word accuracy can be interpreted as high consistency with MOVE-FIRST.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We used the sequence-to-sequence architecture in Figure 3 (Sutskever et al., 2014). This model consists of two neural networks: the encoder and 2 We exclude from the generalization set cases where the two auxiliaries are the same. We also exclude cases where one auxiliary is singular and the other plural so that a model cannot succeed by using heuristics based on the grammatical number of the subject. 3 The one exception is noted in the caption to  the decoder. The encoder is fed the input sentence one word at a time; after each word, the encoder updates its hidden state, a vector representation of the information encountered so far. After the encoder has been fed the entire input, its final hidden state (E 6 in Figure 3) is fed to the decoder, which generates an output sequence one word at a time based on its own hidden state, which is updated after each output word. The weights that the encoder and decoder use to update their hidden states and generate outputs are learned via gradient descent; for more details, see Appendix A.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Holding the task constant, we first varied two aspects of the architecture that have no clear connection to question formation, namely, the recurrent unit and the type of attention; both of these aspects have been central to major advances in natural language processing (Sundermeyer et al., 2012;, so we investigate them here to see whether their contributions might be partially explained by linguistically relevant inductive biases that they impart. We also tested a more clearly task-relevant modification of the architecture, namely the use of tree-based models rather than the sequential structure in Figure 3.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The recurrent unit is the component that updates the hidden state after each word for the encoder and decoder. We used three types of recurrent units: simple recurrent networks (SRNs; Elman, 1990), gated recurrent units (GRUs; , and long short-term memory (LSTM) units (Hochreiter and Schmidhuber, 1997). In SRNs and GRUs, the hidden state is represented by a single vector, whereas LSTMs use two vectors (the hidden state and the cell state). In addition, GRUs and LSTMs both use gates, which control what information is retained across time steps, whereas SRNs do not; GRUs and LSTMs differ from each other in the number and types of gates they use.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
In the basic model in Figure 3, the final hidden state of the encoder is the decoder's only source of information about the input. To avoid having such a bottleneck, many contemporary sequence-tosequence models use attention , a feature that enables the decoder to consider all encoder hidden states (E 0 through E 6 in Figure 3) when generating hidden state D i . A model without attention has the only inputs to D i being D i−1 and y i−1 (the previous output); attention adds a third input, c i = j α i [j]E j , which is a weighted sum of the encoder's hidden states (E 0 through E n ) using a weight vector α i whose j th element is denoted by α i [j].
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Implementations of attention vary in how the weights α i [j] are derived (Graves et al., 2014;Chorowski et al., 2015;Luong et al., 2015). Attention can be solely location-based, where each α i is determined solely from D i−1 (and potentially also y i−1 ), so that the model chooses where to attend without first checking what it is attending to. Alternately, attention could be content-based, in which case each α i [j] is determined from both D i−1 and E j , such that the model does consider what it might attend to before attending to it. We test both location-based and content-based attention, and we also test models without attention.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We trained models with all nine possible combinations of recurrent unit and attention type, using the hyperparameters and training procedure described in Appendix A. The results are in Figure 4.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The SRN without attention failed on the test set, mainly because it often confused words that had the same part of speech, a known weakness = content-based attention. A grayed-out cell indicates that the architecture scored below 50% on the test set. In (b), the SRN produced the first auxiliary 45% of the time; for all other models, the proportion of first-auxiliary outputs is almost exactly one minus the first-word accuracy (i.e., the proportion of main-auxiliary outputs). of SRNs (Frank and Mathis, 2007). Therefore, its generalization set behavior is uninformative. The other architectures performed strongly on the test set (>50% full-sentence accuracy), so we now consider their generalization set performance. The GRU with location-based attention and the SRN with content-based attention both preferred MOVE-MAIN, while the remaining architectures preferred  These results suggest that both the recurrent unit and the type of attention can qualitatively affect a model's inductive biases. Moreover, the interactions of these factors can have drastic effects: with SRNs, content-based attention led to behavior consistent with MOVE-MAIN while location-based attention led to behavior consistent with MOVE-FIRST; these types of attention had opposite effects with GRUs.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
One striking result in Figure 4 is that LSTMs and GRUs display qualitative differences, even though the two architectures are often viewed as interchangeable and achieve similar performance in applied tasks (Chung et al., 2014). One difference between LSTMs and GRUs is that a squashing function is applied to the hidden state of a GRU to keep its values within the range (−1, 1), while the cell state of an LSTM is not 4 We say that a model preferred generalization A over generalization B if it behaved more consistently with A than B.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
bounded. Weiss et al. (2018) demonstrate that such squashing leads to a qualitative difference in how well these models generalize counting behavior. Such squashing may also explain the qualitative differences that we observe: Counting the input elements is equivalent to keeping track of their linear positions, so we might expect that a tendency to count would make the linear generalization more accessible.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To test whether squashing increases a model's preference for MOVE-MAIN, we created a modified LSTM that included squashing in the calculation of its cell state, and a modified GRU that did not have the squashing usually present in GRUs. See Appendix B for more details. Using the same training setup as before, we trained models with these modified recurrent units and with location-based attention. LSTMs and GRUs with squashing chose MOVE-MAIN more often than the corresponding models without squashing ( Figure 5), suggesting that such squashing is one factor that causes GRUs to behave differently than LSTMs.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
In addition to variation across architectures, we also observed considerable variation across multiple instances of the same architecture that differed only in random seed; the random seeds determined both the initial weights of each model and the order in which training examples were sampled. For example, the generalization set first-word accuracy for SRNs with contentbased attention ranged from 0.17 to 0.90. Based on our exploration of hyperparameters, it also appears that the learning rate and hidden size can qualitatively affect generalization. The effects of these details are difficult to interpret systematically, and we leave the characterization of their effects for future work. Results for all individual re-runs are at the project Web site.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
So far we have tested whether properties that are not interpretably related to hierarchical structure nevertheless affect how a model generalizes on a syntactic task. We now turn to a related but opposite question: when a model's design is meant to give it a hierarchical inductive bias, does this design succeed at giving the model this bias?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The first hierarchical model that we test is the Ordered Neurons LSTM (ON-LSTM; Shen et al., 2019). This model is not given the tree structure of each sentence as part of its input. Instead, its processing is structured in a way that leads to the implicit construction of a soft parse tree. This implicit tree structure is created by imposing a stack-like constraint on the updates to the values in the cell state of an LSTM: The degree to which the i th value is updated must always be less than or equal to the degree to which the j th value is updated for all j ≤ i. This hierarchy of cell-state values adds an implicit tree structure to the model, where each level in the tree is defined by a soft depth in the cell state to which that level extends. We re-implemented the ON-LSTM and trained 100 instances of it using the hyperparameters specified in Appendix A. This model achieved a test set full-sentence accuracy of 0.93 but a generalization set first-word accuracy of 0.05, showing a strong preference for MOVE-FIRST over MOVE-MAIN, contrary to what one would expect from a model with a hierarchical inductive bias. This lack of hierarchical behavior might be explained by the findings of Dyer et al. (2019) that ON-LSTMs do not perform much better than standard LSTMs at implicitly recovering hierarchical structure, even though ON-LSTMs (but not standard LSTMs) were designed in a way intended to impart a hierarchical bias. According to Dyer et al. (2019), the ON-LSTM's apparent success reported in Shen et al. (2019) was largely due to the method used to analyze the model rather than the model itself.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The ON-LSTM results show that hierarchically structured processing alone is not sufficient to induce a bias for MOVE-MAIN, suggesting that constraints on which trees are used may also be necessary. We therefore tested a second type of hierarchical model, namely, Tree-RNNs, that were explicitly fed the correct parse tree. Parse trees can be used to guide the encoder, the decoder, or both; Figure 6 shows a model where both the encoder and decoder are tree-based. For the tree-based encoder, we use the Tree-GRU from Chen et al. (2017). This model composes the vector representations for a pair of sister nodes to generate a vector representing their parent. It performs this composition bottom-up, starting with the word embeddings at the leaves and ending with a single vector representing the root (E 4 in Figure 6); this vector acts as the encoding of the input. For the tree-based decoder, we use a model based on the Tree-LSTM decoder from Chen et al. (2018), but using a GRU instead of an LSTM, for consistency with the tree encoder. Figure 7: Results with tree-based models (medians over 100 initializations). Model names indicate encoder/ decoder; e.g., Sequential/Tree has a sequential GRU encoder and a tree-GRU decoder.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
This tree decoder is the mirror image of the tree encoder: starting with the vector representation of the root node (D 0 in Figure 6), it takes the vector representation of a parent node and outputs two vectors, one for the left child and one for the right child, until it reaches a leaf node, where it outputs a word. We test models with a tree-based encoder and sequential decoder, a sequential encoder and tree-based decoder, or a tree-based encoder and tree-based decoder, all without attention; we investigate these variations to determine whether hierarchical generalization is determined by the encoder, the decoder, or both.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The results for these models are in Figure 7, along with the previous results of the fully sequential GRU (sequential encoder + sequential decoder) without attention for comparison. The model with a tree-based encoder and sequential decoder preferred MOVE-FIRST, like the fully sequential model. Only the models with a treebased decoder preferred MOVE-MAIN, consistent with the finding of McCoy et al. (2019) that it is the decoder that determines an encoderdecoder model's representations. However, the model with a sequential encoder and a tree decoder failed on the test set, so the only model that both succeeded on the test set and showed a bias toward a MOVE-MAIN generalization was the fully tree-based model (Tree/Tree). 5 The behavior of this Tree/Tree model was striking in another way as well: Its generalization set full-sentence accuracy was 69%, while all other models-even those that achieved high first-word accuracy on the generalization set-had close to 0% generalization set full-sentence accuracy. The 5 We do not have an explanation for the failure of the Sequential/Tree model on the test set; most of its errors involved confusion among words that had the same part of speech (e.g., generating my instead of your).
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
ON-LSTM and Tree-GRU results show that an architecture designed to have a certain inductive bias might, but will not necessarily, display the intended bias.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We have shown that several models reliably preferred MOVE-MAIN over MOVE-FIRST. However, this behavior alone does not necessarily mean that these models have a hierarchical bias, because a preference for MOVE-MAIN might arise not from a hierarchical bias but rather from some taskspecific factors such as the prevalence of certain n-grams (Kam et al., 2008;Berwick et al., 2011). A true hierarchical bias would lead a model to adopt hierarchical generalizations across training tasks; by contrast, we hypothesize that other factors (such as a bias for focusing on n-gram statistics) will be more sensitive to details of the task and will thus be unlikely to consistently produce hierarchical preferences. To test the robustness of the hierarchical preferences of our models, then, we introduce a second task, tense reinflection.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The reinflection task uses English subject-verb agreement to illuminate a model's syntactic generalizations (Linzen et al., 2016). The model is fed a past-tense English sentence as input. It must then output that sentence either unchanged or transformed to the present tense, with the final word of the input indicating the task to be performed: Because the past tense in English does not inflect for number (e.g., the past tense of swim is swam whether the subject is singular or plural), the model must determine from context whether each verb being turned to present tense should be singular or plural. Example (6) is consistent with two salient rules for determining which aspects of the context are relevant:
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(7) AGREE-SUBJECT: Each verb should agree with its hierarchically determined subject.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(8) AGREE-RECENT: Each verb should agree with the linearly most recent noun.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Though these rules make the same prediction for (6) , the ON-LSTM, and the model with a tree-based encoder and tree-based decoder that were provided the correct parse trees, using the hyperparameters in Appendix A. The example sentences were generated using the same contextfree grammar used for the question formation task, except with inflected verbs instead of auxiliary/ verb bigrams (e.g., reads instead of does read). We evaluated these models on the full-sentence accuracy on the test set and also main-verb accuracy for the generalization set-that is, the proportion of generalization set examples for which the main verb was correctly predicted, such as when swims rather than swim was chosen in the output for (9a). Models usually chose the correct lemma for the main verb (at least 87% of the time for all tense reinflection models), with most main verb errors involving the correct verb but with incorrect inflection (i.e., being singular instead of plural, or vice versa). Thus, a low main-verb accuracy can be interpreted as consistency with AGREE-RECENT. All sequential models, even the ones that generalized hierarchically with question formation, overwhelmingly chose AGREE-RECENT for this reinflection task (Figure 8), consistent with the results of a similar experiment done by Ravfogel et al. (2019). The ON-LSTM also preferred AGREE-RECENT. By contrast, the fully tree-based model preferred the hierarchical generalization AGREE-SUBJECT. Thus, although the question formation experiments showed qualitative differences in sequential models' inductive biases, this experiment shows that those differences cannot be explained by positing that there is a general hierarchical bias in some of our sequential models. What the relevant bias for these models is remains unclear; we only claim to show that it is not a hierarchical bias. Overall, the model with both a tree-based encoder and a tree-based decoder is the only model we tested that plausibly has a generic hierarchical bias, as it is the only one that behaved consistently with such a bias across both tasks.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Generalize Hierarchically?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
It may seem that the tree-based models are constrained by their structure to make only hierarchical generalizations, rendering their hierarchical generalization trivial. In this section, we test whether they are in fact constrained in this way, and similarly whether sequential models are con strained to make only linear generalizations. Earlier, the training sets for our two tasks were ambiguous between two generalizations, but we now used training sets that unambiguously supported either a linear transformation or a hierarchical transformation. 6  Similarly, for the tense reinflection task, we created an AGREE-SUBJECT training set and an AGREE-RECENT training set. For each of these four training sets, we trained 100 sequential GRUs and 100 Tree/Tree GRUs, all without attention. Each model learned to perform linear and hierarchical transformations with similar accuracy: On the MOVE-MAIN and MOVE-FIRST datasets, both the sequential and tree-based models achieved 100% first-word accuracy. On both the AGREE-SUBJECT and AGREE-RECENT datasets, the sequential model achieved 91% main-verb accuracy and the treebased model achieved 99% main-verb accuracy. Thus, the fact that the tree-based model preferred hierarchical generalizations when the training set was ambiguous arose not from any constraint imposed by the tree structure but rather from the model's inductive biases-biases that can be overridden given appropriate training data.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Our sequential and tree-based models differ not only in structure but also in the information they have been provided: The tree-based models have been given correct parse trees for their input and output sentences, while the sequential models have not been given parse information. Therefore, it is unclear whether the hierarchical generalization displayed by the tree-based models arose from the tree-based model structure, from the parse information provided to the models, or both.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To disentangle these factors, we ran two further experiments. First, we retrained the Tree/Tree GRU but using uniformly right-branching trees (as in (11b)) instead of correct parses (as in (11a)). Thus, these models make use of tree structure but not the kind of parse structure that captures linguistic information. Second, we retrained the sequential GRU without attention 7 but modified the input and output by adding brackets that indicate each sentence's parse; for example, (12a) would be changed to (12b). Thus, these models are provided with parse information in the input but such structure does not guide the neural network computation as it does with tree RNNs.  7 We chose this sequential model because the Tree/Tree model is also based on GRUs without attention. Figure 9: Disentangling tree structure and parse information. The GRU that is not provided the correct parse is the same as GRU in Figures 4 and  8. The Tree/Tree model that is provided the correct parse is the same as the Tree/Tree model in Figures 7  and 8. The other two conditions are new: The GRU that was provided the correct parses was given these parses via bracketing, while the Tree/Tree model that was not provided the correct parses was instead given right-branching trees. We ran 100 instances of each experiment using different random seeds. For the experiment with bracketed input, the brackets significantly increased the lengths of the sentences, making the learning task harder; we therefore found it necessary to use a patience of 6 instead of the patience of 3 we used elsewhere, but all other hyperparameters remained as described in Appendix A.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
For both tasks, neither the sequential GRU that was given brackets in its input nor the Tree/Tree model that was given right-branching trees displayed a hierarchical bias (Figure 9). 8 The lack of hierarchical bias in the sequential GRU with bracketed input indicates that simply providing parse information in the input and target output is insufficient to induce a model to favor hierarchical generalization; it appears that such parse information must be integrated into the model's structure to be effective. On the other hand, the lack of a hierarchical bias in the Tree/Tree model using right-branching trees shows that simply having tree structure is also insufficient; it is necessary to have the correct tree structure.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Transformations?
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Each experiment discussed so far involved a single linguistic transformation. By contrast, humans acquiring language are not exposed to phenomena in isolation but rather to a complete language encompassing many phenomena. This fact has been pointed to as a possible way to explain hierarchical generalization in humans without needing to postulate any innate preference for hierarchical structure. While one phenomenon, such as question formation, might be ambiguous in the input, there might be enough direct evidence among other phenomena to conclude that the language as a whole is hierarchical, a fact which learners can then extend to the ambiguous phenomenon (Pullum and Scholz, 2002;Perfors et al., 2011), under the nontrivial assumption that the learner will choose to treat the disparate phenomena in a unified fashion. While our training sets are ambiguous with respect to whether the phenomenon underlying the mapping is structurally driven, they do contain other cues that the language is more generally governed by hierarchical regularities. First, certain structural units are reused across positions in a sentence; for example, prepositional phrases can appear next to subjects or objects. Such reuse of structure can be represented more efficiently with a hierarchical grammar than a linear one. Second, in the question formation task, subject-verb agreement can also act as a cue to hierarchical structure: For example, in the sentence my walrus by the yaks does read, the inflection of does depends on the verb's hierarchically determined subject (walrus) rather than the linearly closest noun (yaks). 9 For the sequential RNNs we have investigated, it appears that these indirect cues to hierarchical structure were not sufficient to guide the models towards hierarchical generalizations. However, perhaps the inclusion of some more direct evidence for hierarchy would be more successful. Figure 10: Multi-task learning results for a GRU without attention. Single-task reports baselines from training on a single ambiguous task. Multi-task reports results from adding an unambiguous second task. Multi-task + auxiliaries reports results from adding an unambiguous second task and also adding overt auxiliaries to the tense reinflection sentences. The numbers give the generalization set performance on the ambiguous task.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To take a first step toward investigating this possibility, we use a multi-task learning setup, where we train a single model to perform both question formation and tense reinflection. We set up the training set such that one task was unambiguously hierarchical while the other was ambiguous between the hierarchical generalization and the linear generalization. This gave two settings: One where question formation was ambiguous, and one where tense reinflection was ambiguous. We trained 100 instances of a GRU without attention on each setting and assessed how each model generalized for the task that was ambiguous.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
For both cases, generalization behavior in the multi-task setting differed only minimally from the single-task setting ( Figure 10). One potential explanation for the lack of transfer across tasks is that the two tasks operated over different sentence structures: the question formation sentences always contained overt auxiliaries on their verbs (e.g., my walrus does giggle), while the tense reinflection sentences did not (e.g., my walrus giggles). To test this possibility, we reran the multi-task experiments but with overt auxiliaries added to the tense reinflection sentences (Figure 10, ''Multitask + auxiliaries'' row). In this setting, the model still generalized linearly when it was question formation that was ambiguous. However, when it was tense reinflection that was ambiguous, the model generalized hierarchically.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We hypothesize that the directionality of this transfer is due to the fact that the question formation training set includes unambiguous longdistance subject-verb agreement as in (13), which might help the model on generalization-set examples for tense reinflection such as Example (14):
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(13) my zebras by the yak do read . DECL → my zebras by the yak do read .
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
(14) my zebras by the yak did read . PRESENT → my zebras by the yak do read .
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
By contrast, the tense reinflection training set does not contain any outputs of the type withheld from the question formation training set. If this explanation is correct, it would mean that the improvement on the tense reinflection task derived not from the question formation transformation but rather from the subject-verb agreement incidentally present in the question formation dataset. Therefore, even the single potential case of generalization across transformations is likely spurious.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Recent NLP work has also found that neural networks do not readily transfer knowledge across tasks; e.g., pretrained models often perform worse than non-pretrained models (Wang et al., 2019). This lack of generalization across tasks might be due to the tendency of multi-task neural networks to create largely independent representations for different tasks even when a shared representation could be used (Kirov and Frank, 2012). Therefore, to make cross-phenomenon generalizations, neural networks may need to be given an explicit bias for sharing processing across phenomena.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
We have found that all factors we tested can qualitatively affect a model's inductive biases but that a hierarchical bias-which has been argued to underlie children's acquisition of syntax-only arose in a model whose inputs and computations were governed by syntactic structure.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Our experiments were motivated in part by the book Rethinking Innateness (Elman et al., 1998), which argued that humans' inductive biases must arise from constraints on the wiring patterns of the brain. Our results support two conclusions from this book. First, those authors argued that ''Dramatic effects can be produced by small changes'' (p. 359). This claim is supported by our observation that low-level factors, such as the size of the hidden state, qualitatively affect how models generalize (Section 3.5). Second, they argued that ''[w]hat appear to be single events or behaviors may have a multiplicity of underlying causes'' (p. 359); in our case, we found that a model's generalization behavior results from some combination of factors that interact in hardto-interpret ways; for example, changing the type of attention had different effects in SRNs than in GRUs.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The dramatic effects of these low-level factors offer some support for the claim that humans' inductive biases can arise from fine-grained architectural constraints in the brain. However, this support is only partial. Our only model that robustly displayed the kind of preference for hierarchical generalization that is necessary for language learning did not derive such a preference from low-level architectural properties but rather from the explicit encoding of linguistic structure.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Our experiments showed that some tree-based models displayed a hierarchical bias, although non-tree-based models never displayed such a bias, even when provided with strong cues to hierarchical structure in their input (through bracketing or multi-task learning). These findings suggest that the hierarchical preference displayed by humans when acquiring English requires making explicit reference to hierachical structure, and cannot be argued to emerge from more general biases applied to input containing cues to hierarchical structure. Moreover, because the only successful hierarchical model was one that took the correct parse trees as input, our results suggest that a child's set of biases includes biases governing which specific trees will be learned. Such biases could involve innate knowledge of likely tree structures, but they do not need to; they might instead involve innate tendencies to bootstrap parse trees from other sources, such as prosody (Morgan and Demuth, 1996) or semantics (Pinker, 1996). With such information, children might learn their language's basic syntax before beginning to acquire question formation, and this knowledge might then guide their acquisition of question formation. There are three important caveats for extending our conclusions to humans. First, humans may have a stronger bias to share processing across phenomena than neural networks do, in which case multi-task learning would be a viable explanation for the biases displayed by humans even though it had little effect on our models. Indeed, this sort of cross-phenomenon consistency is similar in spirit to the principle of systematicity, and it has long been argued that humans have a strong bias for systematicity whereas neural networks do not (e.g., Fodor and Pylyshyn, 1988;Lake and Baroni, 2018). Second, some have argued that children's input actually does contain utterances unambiguously supporting a hierarchical transformation (Pullum and Scholz, 2002), whereas we have assumed a complete lack of such examples. Finally, our training data omit many cues to hierarchical structure that are available to children, including prosody and realworld grounding. It is possible that, with data closer to a child's input, more general inductive biases might succeed.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
However, there is still significant value in studying what can be learned from strings alone, because we are unlikely to understand how the multiple components of a child's input interact without a better understanding of each component. Furthermore, during the acquisition of abstract aspects of language, real-world grounding is not always useful in the absence of linguistic biases (Gleitman and Gleitman, 1992). More generally, it is easily possible for learning to be harder when there is more information available than when there is less information available (Dupoux, 2018). Thus, our restricted experimental setup may actually make learning easier than in the more informationally-rich scenario faced by children.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
Our results leave room for three possible approaches to imparting a model with a hierarchical bias. First, one could search the space of hyperparameters and random seeds to find a setting that leads to the desired generalization. However, this may be ineffective: At least in our limited exploration of these factors, we did not find a hyperparameter setting that led to hierarchical generalization across tasks for any non-tree-based model.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
A second option is to add a pre-training task or use multi-task learning (Caruana, 1997;Collobert and Weston, 2008;Enguehard et al., 2017), where the additional task is designed to highlight hierarchical structure. Most of our multi-task experiments only achieved modest improvements over the single-task setting, suggesting that this approach is also not very viable. However, it is possible that further secondary tasks would bring further gains, making this approach more effective.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
A final option is to use more interpretable architectures with explicit hierachical structure. Our results suggest that this approach is the most viable, as it yielded models that reliably generalized hierarchically. However, this approach only worked when the architectural bias was augmented with rich assumptions about the input to the learner, namely that it provided correct hierarchical parses for all sentences. We leave for future work an investigation of how to effectively use tree-based models without providing correct parses.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
improvement over 3 consecutive evaluations on the validation set (the number 3 in this context is called the patience). The training set contained 100,000 examples, while the validation, test, and generalization sets contained 10,000 examples each. The datasets were held constant across experiments, but models sampled from the training set in different orders across experiments. During training, we used teacher forcing on 50% of examples.
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The equations governing a standard LSTM are:
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
To create a new LSTM whose cell state exhibits squashing, like the hidden state of the GRU, we modified the LSTM cell state update in (B.5) to (B.7), where the new coefficients now add to 1: 10
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The equations governing a standard GRU are:
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
The GRU's hidden state is squashed because its update gate z merges the functions of the input and forget gates (i and f ) of the LSTM (cf. Equations (B.5) and (B.11)). As a result, the input and forget weights are tied in the GRU but not the LSTM. To create a non-squashed GRU, we added an input gate i and changed the hidden state update (Equation (B.11)) to Equation (B.13) to make z act solely as a forget gate:
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks
h t = z t * h t−1 + i t * h (B.13) 10 We modified the structure of the gates rather than adding a squashing nonlinearity to avoid vanishing gradients.