How did prior work explore task specific pre-training adapation for language models?
Task-Specific Pretraining Adaptation Current
state-of-the-art methods in abstractive summariza-
tion apply a two-step approach where models are
first pretrained on large corpora of text with task-
agnostic variations of the text denoising objective
and next finetuned on labeled examples from the
target task (Lewis et al., 2020; Raffel et al., 2020).
However, in tasks where labeled data is scarce,
task-specific pretraining objectives have been
shown to provide significant benefits. Recent work
adapted language models to summarize multiple
documents (Xiao et al., 2022), produce more fac-
tual summaries (Wan and Bansal, 2022), or plan
with entity chains (Narayan et al., 2021). We build
on these methods, focusing on the downstream task
of controllable summarization. Other studies demonstrate the effect of contin-
ued pretraining (Gururangan et al., 2020) and pre-
finetuning (Aghajanyan et al., 2021; Wei et al.,
2021; Fabbri et al., 2021a) on downstream task
adaptation. These either continue training with the
same objective on data in the downstream task do-
main or perform multitask learning using labeled
data. In this work, we demonstrate the benefits
of language model adaptation with a task-specific
pretraining objective without additional supervised
data and show that these benefits are consistent and
statistically significant in low-resource settings like
query-focused summarization (QFS
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains -- ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of Rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.
[{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Joshua","last":"Maynez","middle":[]},{"first":"Gonccalo","last":"Simoes","middle":[]},{"first":"Vitaly","last":"Nikolaev","middle":[]},{"first":"Ryan","last":"McDonald","middle":[]}]
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present FactPEGASUS, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning: (1) We augment the sentence selection strategy of PEGASUS’s (Zhang et al., 2019) pre-training objective to create pseudo-summaries that are both important and factual; (2) We introduce three complementary components for fine-tuning. The corrector removes hallucinations present in the reference summary, the contrastor uses contrastive learning to better differentiate nonfactual summaries from factual ones, and the connector bridges the gap between the pre-training and fine-tuning for better transfer of knowledge. Experiments on three downstream tasks demonstrate that FactPEGASUS substantially improves factuality evaluated by multiple automatic metrics and humans. Our thorough analysis suggests that FactPEGASUS is more factual than using the original pre-training objective in zero-shot and few-shot settings, retains factual behavior more robustly than strong baselines, and does not rely entirely on becoming more extractive to improve factuality.
[{"first":"David","last":"Wan","middle":[]},{"first":"Mohit","last":"Bansal","middle":[]}]
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form the foundation of today’s NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task’s unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.
[{"first":"Suchin","last":"Gururangan","middle":[]},{"first":"Ana","last":"Marasović","middle":[]},{"first":"Swabha","last":"Swayamdipta","middle":[]},{"first":"Kyle","last":"Lo","middle":[]},{"first":"Iz","last":"Beltagy","middle":[]},{"first":"Doug","last":"Downey","middle":[]},{"first":"Noah A.","last":"Smith","middle":[]}]
Muppet: Massive Multi-task Representations with Pre-Finetuning
We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
[{"first":"Armen","last":"Aghajanyan","middle":[]},{"first":"Anchit","last":"Gupta","middle":[]},{"first":"Akshat","last":"Shrivastava","middle":[]},{"first":"Xilun","last":"Chen","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]},{"first":"Sonal","last":"Gupta","middle":[]}]
Finetuned Language Models Are Zero-Shot Learners
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.
[{"first":"Jason","last":"Wei","middle":[]},{"first":"Maarten","last":"Bosma","middle":[]},{"first":"Vincent","last":"Zhao","middle":["Y."]},{"first":"Kelvin","last":"Guu","middle":[]},{"first":"Adams","last":"Yu","middle":["Wei"]},{"first":"Brian","last":"Lester","middle":[]},{"first":"Nan","last":"Du","middle":[]},{"first":"Andrew","last":"Dai","middle":["M."]},{"first":"Quoc","last":"Le","middle":["V."]}]
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins.
[{"first":"Wen","last":"Xiao","middle":[]},{"first":"Iz","last":"Beltagy","middle":[]},{"first":"Giuseppe","last":"Carenini","middle":[]},{"first":"Arman","last":"Cohan","middle":[]}]
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. In this work, we introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction of the desired summaries. WikiTransfer models achieve state-of-the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our approach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training examples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset aspects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsupervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation.
[{"first":"Alexander","last":"Fabbri","middle":[]},{"first":"Simeng","last":"Han","middle":[]},{"first":"Haoyuan","last":"Li","middle":[]},{"first":"Haoran","last":"Li","middle":[]},{"first":"Marjan","last":"Ghazvininejad","middle":[]},{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Dragomir","last":"Radev","middle":[]},{"first":"Yashar","last":"Mehdad","middle":[]}]
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Planned and Grounded Abstractive Summarization We introduce a novel training objective to neural summarization models for content planning with entity chains, and study integrating it with supervised finetuning and self-supervised pretraining objectives without altering the models themselves. As the entity chains are extracted from the reference summaries during training, our models learn to ground the generation of summaries to the entity chains found in them. Hence, we refer to this objective by FROST for its ability to try to "FReeze entity-level infOrmation in abstractive SummarizaTion with planning."
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Abstractive Summarization with Entity Chains FROST provides a very effective knob for entity-level content modification in abstractive summaries. In this paper we empirically demonstrate how FROST is critical for faithfulness by enabling the drop-prompt mechanism where we drop out hallucinated entities from the predicted content plan and prompt the decoder with this modified plan to generate faithful summaries. We further qualitatively demonstrate that FROST enables generation of summaries (i) with topical diversity by choosing different sets of entities from the source to plan what we want to discuss in the summary, and (ii) with style diversity by reordering entities in the predicted plan to get an equivalent summary but with a different entity emphasis.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a new training objective for encoderdecoder generative models to do content planning while summarizing.
Planning with Learned Entity Prompts for Abstractive Summarization
Model Formulation. Let d be an input document, we aim to teach our model to first generate a content plan c for the summary s as p(c|d), and then generate the summary s as p(s|c, d). We define the ordered chain of entities observed in the summary s as its content plan. Instead of modeling p(c|d) and p(s|c, d) separately, we take a simpler approach, we train an encoder-decoder model to encode the document d and generate the concatenated content plan and summary sequences c; s, essentially the decoder first predicts the entity chain c and then continues predicting the summary s using both c and d. We prefix c and s with special markers "[ENTITYCHAIN]" and "[SUMMARY]", respectively, as shown in Figure 1. If s consists of multiple sentences, we use sentence markers "|||" to mark them in c. The model is trained with the standard maximum-likelihood objective generating the augmented target c; s.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Content Plans. We modified PE-GASUS (Zhang et al., 2019) to pretrain our models for entity-level content planning and summary generation. 2 In particular, we select a maximum of n most important sentences using self-ROUGE from an input document, the selected sentences work as a proxy for a human-authored abstractive summaries for the rest of the document. We construct a target by prepending the selected sentences dynamically with their entity chain. Our model is then trained to generate this target from the rest of the document. 3
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Generation with Entity Prompts. An advantage of training to generate the summary s following the generation of the plan c using the same decoder is that now during the inference time the decoder can be easily prompted with any desired content plan c to control the content in the output summary s with a probability of p(s |c , d).
Planning with Learned Entity Prompts for Abstractive Summarization
In Section 5, we prompt our decoder with modified content plans to mitigate hallucinations and to generate diverse summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Datasets. Following Zhang et al. (2019), our model pretraining also relied on two large web corpus which were processed to look like plain text: (i) C4 (Raffel et al., 2019) is composed of 350M Web-pages that were obtained from Common Crawl, and (ii) HugeNews (Zhang et al., 2019) is composed of 1.5B news and newslike articles from 2013-2019. This dataset includes articles from multiple allowlisted sources including news publishers, high-school newspapers and blogs.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
Entity Chain Annotation. We experimented with entity chains consisting of named entities,  Table 1: Abstractive summarization datasets studied in this work. We report on their train/validation/test sizes and how they were processed (cased/uncased). To better understand the effect of summary planning with entity chains, we report on average number of sentences (avg. sent.), average number of entities (avg. ent.) and average number of unique entities (avg. uniq. ent.), per target in validation sets. We also report on total number of named entities, date and number in target summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
dates and numbers. We annotate the whole document in the pretraining datasets to allow dynamic construction of summaries and their entity chains during pretraining. We only annotate the reference summaries for the finetuning summarization datasets. We use a BERT-based tagger trained on CoNLL-2003 data (Tjong Kim Sang andDe Meulder, 2003) to identify named entities, and regular expressions to identify dates and numbers (Guu et al., 2020). See Table 1 for the number of named entities, dates and numbers found in different datasets. Table 6 and 7 in Appendix A present hyperparameters used for pretraining and finetuning PE-GASUS and FROST base and large sized models. We used Cloud TPU v3 accelerators for training.
Planning with Learned Entity Prompts for Abstractive Summarization
We evaluate our FROST models on ROUGE, entity specificity, entity planning, faithfulness using automatic and human evaluations, and overall quality by humans. Our models predict a summary plan in the form of an entity chain, followed by a summary. All evaluations are done on the summary, the predicted entity chains are stripped out.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level ROUGE. We report ROUGE F1 scores (Lin and Hovy, 2003) to assess generated summaries. 7 Entity Planning. We evaluate the quality of content plans learned by our models by assessing the entities and the order in which they appear in the summary. We annotate both predicted and reference summaries with named entities, dates and numbers, and report on ROUGE F1 scores for entity chains found in the predicted summaries against corresponding reference entity chains. 7 We lowercased candidate and reference summaries and used pyrouge with parameters "-a -c 95 -m -n 4 -w 1.2." Entity Specificity. We compare entities in predicted and reference summaries, and report average entity F1-scores (ENTF1) in predicted summaries. We lower-case and remove duplicate entities; we report on the exact match with reference entities.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
We further assess summary faithfulness by humans. Our annotators, proficient in English, were tasked to read the document carefully and then grade its summary on a scale of 1-5 (fully unfaithful, somewhat unfaithful, 50-50, somewhat faithful and fully faithful); a summary is "fully faithful" if all of its content is fully supported or can be inferred from the document.
Planning with Learned Entity Prompts for Abstractive Summarization
Overall Summary Quality. Finally we assess the overall quality of summaries by humans. We ask our annotators to read the document carefully and then grade its summary on a scale of 1-5 (poor summary, better than poor, okay summary, better than okay and great summary). To improve the annotator agreement, we clearly define 4 features that a "great summary" should have: Relevant (summary should have most important information from the document), Accurate (summary "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell.
Planning with Learned Entity Prompts for Abstractive Summarization
Original Target: "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell. should be accurate with respect to the document or the expert knowledge), 8 Concise (summary should not have redundant or less-important content), and Fluent (summary should be grammatically correct and coherent). A great summary should pass on all 4 features, a better than okay should have 3 out of 4 features, and so on.
Planning with Learned Entity Prompts for Abstractive Summarization
For both human assessments, we collected 3 ratings for each (document, summary) pair and report the average of all assigned labels (1-5) to a system. We conducted 3 pilot studies for each setup to train our annotators with examples to improve their understanding of the task. Additionally, extra measures were taken to improve agreements among our annotators. For example, for the faithfulness assessment, when one of somewhat unfaithful, 50-50 and somewhat faithful were selected, annotators were asked to also specify what was faithful or unfaithful in the summary. Similarly for the overall quality assessment, when one of better than poor, okay summary and better than okay were selected, they were asked to list all features on which the candidate summary fails. 
Planning with Learned Entity Prompts for Abstractive Summarization
Sentence-Level vs Summary-Level Planning.
Planning with Learned Entity Prompts for Abstractive Summarization
Let the summary s consists of m sentences s 1 . . . s m and c i be the entity chain for the sentence s i , we consider generating the summary s in two ways. Sentence-level approach trains a model to generate s by consecutively generating the sentence-level content plan c i followed by its summary sentence s i with a probability
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
in Figure 3.
Planning with Learned Entity Prompts for Abstractive Summarization
We found that the summary-level planning is not only superior to the sentence-level planning, it helps with generating better quality summaries in terms of summary-level ROUGE (RL-Sum), entity planning (R2-EPlan) and entity specificity (ENTF1). For the rest of the pretraining or finetuning experiments, we focused on summary-level planning unless specified otherwise.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining for Planning from Scratch. We pretrained three base-sized models from scratch: PEGASUS-base pretrained with the original gapsentence objective (Zhang et al., 2019) for 1.5m steps, FROST(F)-base pretrained with the gapsentences prepended with their entity chain for 1.5m steps, and FROST(P+F)-base first pretrained with the PEGASUS objective for 1m steps and then with the FROST objective for another 500k steps. Maximum input and output lengths were set to 512 and 256 sentencepieces during pretraining, respectively. We finetuned these three models on the XSum dataset. Results are shown in Figure 4.
Planning with Learned Entity Prompts for Abstractive Summarization
First of all, our results confirm that the pretraining for planning is beneficial for summarization; both FROST(F) and FROST(P+F) learned better content plans (in terms of entity chainlevel ROUGE and ENTF1) than PEGASUS without sacrificing the summary quality (in terms of summary-level ROUGE) Figure 5: Finetuning results on the XSum (in blue) and CNN/DailyMail (in red) validation sets at various steps during pretraining FROST-Large. Instead of pretraining from scratch, we start with a PEGASUS-Large checkpoint, and continue pretraining for additional 1.5m steps with the planning objective. We report finetuning results for the PEGASUS finetuned baseline and our models at 0.1m, 1m and 1.5m steps. the pretraining for planning and summarization is more effective when started with summarization pretrained models such as PEGASUS than when trained jointly from scratch. Hence, for our future pretraining experiments we initialize our model with existing PEGASUS checkpoint and continue pretraining for content planning and refer to them as FROST for simplicity.
Planning with Learned Entity Prompts for Abstractive Summarization
Based on our findings, we pretrain a large FROST model for planning for summarization starting with an existing PEGASUS-Large checkpoint. Figure 5 presents results for finetuning these models on the XSum and CNN/DailyMail datasets at various steps during pretraining.
Planning with Learned Entity Prompts for Abstractive Summarization
Similar to our findings in Figure 4, our results with large models further confirm the advantages of the pretraining for planning; improvement over the PEGASUS baseline were larger for the XSum dataset than for the CNN/DailyMail dataset. We achieved the best performance on both datasets at the 1m pretraining step. We use the FROST model at 1m pretraining step for our finetuning experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Table 2 presents our final results from finetuning our large models on summarization datasets. For each dataset, we first report results from earlier work directly taken from corresponding papers; our results are in the bottom blocks for each dataset. We finetune our own PEGASUS using the standard approach (document to summary). We report results from FROST (ECP; Entity    (Dou et al., 2020) and PEGASUS (Zhang et al., 2019). We also include commonly reported LEAD-n baselines (selecting top n sentences from the document) and extractive oracle (EXT-ORACLE; selecting best set of sentences from the document with the most overlapping content with its reference summary), for the CNN/DailyMail and XSum datasets. EXT-ORACLEs are masked with * and are not directly comparable. All results are reported on the publicly available test sets.
Planning with Learned Entity Prompts for Abstractive Summarization
Chain Planning), i.e., PEGASUS finetuned with the FROST objective (document to content plan and summary). Finally, we report on FROST (ECPP; Entity Chain Planning with Pretraining), our models both pretrained and finetuned with the FROST objective.
Planning with Learned Entity Prompts for Abstractive Summarization
We find that even simply finetuning an existing PEGASUS pretrained model to do content planning and summarization (as in FROST (ECP)) leads to improvements in entity specificity (ENTF1) and the quality of entity plans in summaries (Entity Planning ROUGE), across all datasets. In fact in some cases, better content plans lead to large improvements in summary-level ROUGE as well, for example, FROST (ECP) improve on PEGASUS from 44.05/21.69/40.98 to 44.85/21.83/41.80 on ROUGE scores for CNN/DailyMail summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
We demonstrate in two ways the planning with entity chains can be useful in mitigating hallucinations in summarization: data filtering and dropprompt mechanism with entity chains. We focused on the XSum dataset for these experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Data Filtering using Entity Chains. During training FROST prepends reference summaries with their entity chains to better plan them. We leverage this to filter the dataset to only keep examples where summaries have fully extractive entity chains; an entity chain is fully extractive if all entities in it can be found in the input document. It's notable that this filtered dataset will not contain novel entities in the summary targets. FROST models trained on this data will ground the summary generation to extracted entity chains while allowing abstraction for non-entity related generations. The resulting XSum dataset has 62.7k/3.5k/3.5k train/validation/test instances. We finetune our models from Table 2 on this filtered dataset and report their performance on the filtered test set (3.5k). We also evaluate them on the rest of the test set (7.8k) where reference entity chains are not fully extractive to their input documents. 9 See Figure 6 for examples. 9 Such data divergence is not unique to XSum, around 30% of CNN/DailyMail summaries also have reference entity chains that are not fully extractive to their input documents. Writing these summaries requires either document-level inference or the background knowledge of the input documents to generate novel entities or numbers.  Table 3: Performance on XSum summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (3.5k) and the rest of the test set with novel entities in the targets (7.8k). Best results are bold faced.
Planning with Learned Entity Prompts for Abstractive Summarization
Drop-Prompt Mechanism. FROST decoders are trained to generate the summary s following the generation of its plan c. To improve faithfulness, we take the predictions from FROST (ECPP) 10 and modify the generated plan c to c drop by dropping entities (or parts of them) that are not found in the input document. We then prompt our decoder with c drop to generate a new summary. We conduct this with both models, one trained on the full dataset and another on the filtered subset. We also report results for the oracle entity chain prompts c oracle for a comparison.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Summary-level ROUGE, Entity Planning and Specificity. Results are presented in Table 3. First of all, similar to our results in Table 2 on the full XSum test set, our results with the models trained on the original dataset further validate that the pretraining and finetuning with the content planning objective is equally useful for both test sets: one where entities are simply copied from the input documents and another where novel entities were inferred, to generate corresponding targets. The data filtering using entity chains are particularly useful for test cases with extractive reference entity chains; models trained on the filtered data lead to much higher ENTF1 (e.g., 65.97 vs 63.03, for FROST (ECPP)) and higher quality entity plans (entitly-level ROUGE of 66.43/30.00/62.48 vs 63.73/30.08/60.10 for FROST (ECPP)), without sacrificing the summary quality substantially (summary-level ROUGE of 44.87/22.05/36.79 vs 10 The drop-prompt mechanism can be used with FROST (ECP) also, we have simply used the best among FROST (ECP) and FROST (ECPP) on the XSum set in terms of summarylevel ROUGE, entity planning and ENTF1.   Table 3 and 4. We highlight entities in orange that are not faithful to the input document. Entities in green are faithful to the input document. 45.08/22.01/36.80 for FROST (ECPP)). Unsurprisingly, these models don't do well on the test set with non-extractive reference entity chains, such examples were not seen during training.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Faithfulness. The planning objective in FROST itself does not ensure faithfulness; planning is done with the entities in the target, if the target is noisy, its plan may also be noisy (see Figure 6 for predicted entity chains with hallucinated entities). But the planning with the entity chains facilitates the data filtering and the drop-prompt mechanism, using entity chains. Table 4 presents our results assessing them for faithfulness. We randomly selected 50 documents for each of test sets (with extractive or non-extractive reference entity chains) and assessed their summaries from all 8 systems (except with c oracle ) from Table 3.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism is also very powerful in improving faithfulness. Irrespective of which training data were used to train the models and which test sets they were evaluated on, we see improvements in entailment scores, ENT-PREC and the human assessment of faithfulness across the board, except for a single case where the entailment score slightly drops from 0.465 to 0.453. Figure 6 demonstrates how we drop hallucinated entities 'Liam', 'two' and 'Conor' from entity chains to enforce models to generate faithful summaries grounded to the modified entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We achieve the best performance in terms of ENTPREC and the human assessment of faithfulness when both the data filtering and the dropprompt mechanism were used. We achieve 0.943 for ENTPREC and 4.41 for faithfulness for the test set with extractive entity chains only, and 0.826 for ENTPREC and 3.85 for faithfulness for the test set with non-extractive entity chains. Humans also found that the predictions from these models were the best in terms of overall summary quality.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, we carried out pairwise comparisons for human assessments for faithfulness and overall summary quality for all models (using a oneway ANOVA with post-hoc Tukey HSD tests; p < 0.01). Interestingly, differences among all model pairs for both faithfulness and overall summary quality are insignificant when evaluated on the test set with extractive reference entity chains only. On the more challenging test set with non-extractive entity chains, FROST (ECPP) trained on the fil-GOLD: [ENTITYCHAIN] BP | Lord Browne | North Sea [Summary] Former BP chief executive Lord Browne has warned North Sea oil operators' costs must fall in order for them to compete globally.
Planning with Learned Entity Prompts for Abstractive Summarization
Browne, the former boss of BP, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST (ECPP), c mod , Style diversity [ENTITYCHAIN] BP | Browne [Summary] The former boss of BP, Lord Browne, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
[ENTITYCHAIN] BBC | BP | Browne [Summary] In an interview with the BBC, former BP chief executive Lord Browne has warned the oil and gas industry is in danger of "going to the wall".  tered training set and with the drop-prompt mechanism is significantly better than all other models except for (i) FROST (ECPP) trained on the filtered training set but without the drop-prompt mechanism and (ii) FROST (ECPP) trained on the original training set and with the drop-prompt mechanism, for both faithfulness and overall summary quality. All other pairwise differences are insignificant.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST provides a handle to easily control or manipulate content plan for predicted summaries. In Figure 6, we saw how we can use this to control entity-level hallucinations, but the strength of FROST models go beyond this. Figure 7 shows how FROST models can be effectively used in generating summaries with different entity focus by simply modifying the entity prompt. Future work will focus on how to leverage FROST for synthetic data generation for training improvements.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
Planning with Learned Entity Prompts for Abstractive Summarization
There are several advantages of FROST-style decoding. Unlike encoder-guided control summarization systems, FROST models can be used in a usual way to generate summaries for input documents without relying on external systems to augment them during inference. Additionally, users can modify the generated entity prompts or provide their desired entity prompts to control the generated summaries (see Section 5.3 and 5.4).
Planning with Learned Entity Prompts for Abstractive Summarization
To the best of our knowledge, FROST is the first decoder-prompted control summarization model. Table 5 show that the prompting the decoder with entity prompts are more effective in generating high-quality and grounded summaries compared to when entity prompts are encoded with the input, especially for abstractive summaries; both PEGASUS versions (d; c oracle and c oracle ; d) perform inferior to FROST (d → c oracle ; s) in terms of summary-level ROUGE, entity-level ROUGE and ENTF1 on the XSum dataset. CTRLsum (k oracle ; d → s) achieves better ROUGE scores than FROST (d → c oracle ; s) on CNN/DailyMail (64.72/40.56/61.02 vs 61.85/38.95/59.00). This is not surprising as the oracle keywords k oracle in CTRLSum retain most words from the summary found in the source document; as such due to the extractive nature of CNN/DailyMail summaries, k oracle tend to be closer to the surface forms of the reference summaries (ROUGE scores between k oracle and reference summaries are 52.70/17.37/44.37). The difference in ROUGE scores narrows down between CTRLsum and FROST to 45.65/22.35/42.50 vs 45.11/22.11/42.01 with automatically extracted keywords. FROST outperforms CTRLsum on entity planning and specificity. It is not clear how well CTRLsum's keyword extraction system will generalize to more abstractive datasets where words in reference summaries are often not found in the source document.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, our FROST models work as a better microplanner and produce concise summaries than those generated by systems without doing content   Figure 10, Appendix C.
Planning with Learned Entity Prompts for Abstractive Summarization
In this work, we proposed the use of entity chains, both during pretraining and finetuning, to better plan and ground the generation of abstractive summaries. Our approach achieves state-of-theart ROUGE results in XSum and SAMSum, and competitive results in CNN/Dailymail. Compared to other guided summarization models, CTRL-Sum and GSum, which perform slightly better in CNN/Dailymail, our approach is drastically simpler to implement, trains by augmenting the targets only, utilizes no additional parameters than the baseline PEGASUS model, and does not rely on any external systems to augment the inputs during inference. We further demonstrate that by modifying the entity chain prompts, we can easily control hallucinations in summaries. Table 6 and 7 present hyperparameters used for pretraining and finetuning PEGASUS and FROST base and large sized models. 
Planning with Learned Entity Prompts for Abstractive Summarization
Figure 10 presents example predictions from FROST and CTRLSum along with their entity prompts and keywords used for guidance, respectively.    
Planning with Learned Entity Prompts for Abstractive Summarization
We experimented with entity chains consisting of named entities, dates and numbers. We finetuned PEGASUS-large models on the XSum dataset to predict the entity chain and then continue predicting the summary. Results are presented in Figure 11. We found that the entity chains with all named entities, dates and numbers performed best for entity level measures (entity chain-level ROUGE and ENTF1) without sacrificing the final summary quality (summary-level ROUGE) significantly. Following this, we always use entity chains with named entities, dates and numbers to have better control over them.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-Level Entity Chains Table 9 presents detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.
Planning with Learned Entity Prompts for Abstractive Summarization
Here, we present results from controlling hallucinations on CNN/DailyMail summaries (i) by training models on the dataset with extractive entity chains only and (ii) by modifying the predicted entity chains to keep only the supported entities. Table 10 and 11 present results from these mod- Figure 9: Instructions for human evaluations for overall quality of summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
GOLD: Theia, a bully breed mix, was apparently hit by a car, whacked with a hammer and buried in a field. "She's a true miracle dog and she deserves a good life," says Sara Mellado, who is looking for a home for Theia.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Oracle keywords) hit car apparently whacked hammer buried field | dog bully breed mix Theia | true miracle deserves good life (Summary) A bully breed mix named Theia was hit by a car, apparently whacked with a hammer and buried in a field. Four days after her apparent death, the dog managed to stagger to a nearby farm. "She's a true miracle dog and she deserves a good life," the dog's foster mother says. FROST: (Oracle Entity Prompt) Theia ||| Sara Mellado | Theia (Summary) The dog, now named Theia, was apparently hit by a car and buried. Sara Mellado is raising money to help pay for Theia's medical care.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Predicted keywords) stray pooch Washington State hit car head hammer mercy killing buried field survive | dog Theia | found (Summary) A stray pooch in Washington State was hit by a car and buried in a field. The dog was apparently hit on the head with a hammer in a mercy killing. She managed to survive and was found by a worker who took her to a vet. Theia is now being cared for at a veterinary hospital. FROST: (Predicted Entity Prompt) Washington State ||| Theia ||| (Summary) A stray dog in Washington State was apparently hit by a car and buried. The dog, now named Theia, survived but still needs surgery. A fundraising page has been set up to help pay for her care.    Table 9: Detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.  Table 10: Performance on CNN/DailyMail summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (7.8k) and the rest of the test set with novel entities in the targets (3.7k).  Figure 11: Ablations on entity chains with named entities, dates and numbers. We report summary-level ROUGE-L (RL-Sum), entity chain-level ROUGE-2 (R2-EPlan) and ENTF1 on the XSum validation set. Similar observations were made for other measures, see detailed results in Table 8.  Table 11: Faithfulness assessment using automatic and human evaluations, and overall quality assessment by humans for models presented in Table 10 for the CNN/DailyMail dataset (Test set with Non-Extractive Entity Chains Only). 
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Abstractive summarization aims at generating short summaries that capture the essentials of a long document. Research in this challenging task has made significant progress with the help of large pre-trained models (Lewis et al., 2020;Raffel et al., 2020;Zhang et al., 2020). However, current models suffer from the crucial problem of hallucinations (Maynez et al., 2020), where a summary contains facts or entities not present in the original document. Such unfaithful generation raises the question of whether the models can be trustworthy and used safely for real-world applications. To tackle this problem, many approaches propose postprocessing models (Chen et al., 2021;Dong et al., 2020;Liu and Liu, 2021), but such methods are often constrained by external resources to train additional correction or selection models. An alternative line of works focuses on learning factuality directly during fine-tuning by filtering nonfactual training data (Goyal and Durrett, 2021;Nan et al., 2021) or, most recently, incorporating contrastive learning (Cao and Wang, 2021) to encourage generating faithful summaries.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In this work, we propose FACTPEGASUS, a model that addresses the problem of hallucinations for abstractive summarization holistically, by incorporating factuality into the whole training pipeline: We tackle the lack of factuality objective in pre-training and the presence of hallucinations in the downstream dataset during finetuning. Current pre-training objectives focus on improving the quality of the generated output in the downstream tasks but often overlook the factuality aspect. Thus, we explore incorporating factuality into the pre-training objective of PEGASUS (Zhang et al., 2020) (a state-of-the-art abstractive summarization model). The original objective, gap sentence generation (GSG), transforms any text into a pseudo-summarization dataset by selecting important sentences using ROUGE (Lin, 2004) as output summaries. We explore strategies for combining ROUGE and the factuality metric FactCC (Kryscinski et al., 2020) as the selection criteria, so that the model learns to generate sentences that cover the most important information of the input document as well as remain faithful to it.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Next, we propose three complementary modules that further address factuality problems during fine-tuning: (1) Corrector that removes hallucinations existing in reference summaries, allowing training on the full training set without learning unfaithful behaviors; (2) Contrastor that encourages the model to better differentiate factual summaries from nonfactual ones by paying attention to the document using contrastive learning; (3) Connec-tor, a special mask-token fine-tuning technique enabled by the GSG-style objective, that simulates the pre-training task during fine-tuning by inserting the mask token into the input document so that the pre-trained model can adapt its knowledge of generating factual summaries directly to the downstream tasks. The connector, corrector, and contrastor address the input, output, and training objective of the downstream task, respectively, and the combination of the components reduces potential confounding problems that cannot be addressed by a single module. We show that the full model improves three factuality metrics, the token and sentence error of DEP Entail (Goyal and Durrett, 2021) and FactCC, on the downstream datasets of XSum (Narayan et al., 2018), WikiHow (Koupaee and Wang, 2018), and Gigaword (Rush et al., 2015). Most notably, FACTPEGASUS outperforms existing factualityaware summarization models by more than 40% and 34% on XSum for token error and FactCC, respectively. Ablation studies show the usefulness of each of our fine-tuning components as well as the additive gain of combining our complementary modules, and human evaluation confirms that FACTPEGASUS generates significantly more factual summaries over strong baselines.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Finally, we perform a detailed analysis of FACT-PEGASUS, which points to several important observations regarding learning and maintaining factuality: (1) Zero-shot setting demonstrates the utility of our factuality-aware pre-training objective, as our model outperforms PEGASUS (which uses the original objective) on all three factuality metrics when evaluated directly on the downstream task without any supervised training data. Few-shot experiment indicates that even a small number of nonfactual examples can have a strong negative impact on factuality and can nullify much of the gain from factuality pre-training, highlighting the importance of ensuring factuality during fine-tuning. (2) Factuality dynamics (Goyal et al., 2022) further shows that FACTPEGASUS exhibits a lesser degree of factuality degradation than what is observed for BART-base. (3) Factuality vs abstractiveness tradeoff curve reveals that FACTPEGASUS effectively improves factuality by not simply relying on the increase in extractiveness.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To summarize, our contributions are as follows:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
1. We propose a factuality-aware pre-training objective for abstractive summarization and study the effect of different sentence selection strate-gies on downstream factuality.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
2. We introduce three complementary components for improving factuality during fine-tuning that correct hallucinations present in the training set, discourage unfaithful generation during training, and bridge the gap between pre-training and finetuning. The full model consistently achieves better factuality scores than strong baselines on three downstream abstractive summarization tasks, confirmed by human evaluation.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
3. We conduct thorough factuality analysis and show that FACTPEGASUS generates more factual summaries with no or little supervision, slows down factuality degradation observed for current models, and improves factuality not by becoming more extractive.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We describe our training procedure consisting of pre-training with a factuality-aware objective (Section 3.1) and fine-tuning with three complementary modules for improving factuality (Section 3.2).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Recent exploration of good pre-training objectives for abstractive summarization aims at achieving high quality on downstream tasks, often in terms of ROUGE. However, few have analyzed the effect of pre-training objective on factuality. We focus on incorporating this aspect into the pre-training objective of PEGASUS, gap sentence generation (GSG), since PEGASUS achieves state-of-the-art performance on the downstream abstractive summarization tasks. The GSG objective transforms text documents into a pseudo-summarization dataset by selecting important sentences as the output summary, which are subsequently masked out in the original text. The best strategy determines the importance by calculating ROUGE-1 between each chosen sentence and the rest of the document. While the original strategy selects sentences that contain the most unigram overlap, there is no guarantee that the selected sentences are faithful to the rest of the document. We provide an illustrative example in Figure 1a, where the original objective selects sentence C due to its high ROUGE-1 score. However, this sentence is not a faithful summary to the rest of the document as the other sentences concern with the fire in Seattle while only sentence C talks about the fire in Denver.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To address this problem, we extend this objective, which we call factual GSG (factGSG), where we additionally measure the importance of the sentences according to factuality. We use FactCC (Kryscinski et al., 2020) as the factuality criteria when selecting the summary sentences, as it correlates highly with human factuality judgment (Pagnoni et al., 2021) and is relatively fast to compute. FactCC produces a binary prediction where a score of 1 indicates that the selected sentence is consistent with the rest of the document. Another change in factGSG is the choice of gap sentence ratio, which determines the percentage of sentences in the text that will be selected as the summary. Instead of selecting 30% of the text document as output summary, we only select one sentence, as selecting more sentences will inevitably increase the possibility of hallucinations.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Formally, given a document D of n sentences, D = {x 1 , x 2 , ..., x n }, we select the top-scoring sentence as the output summary, where the score of each sentence x i is calculated by:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Going back to the example in Figure 1a, FactCC assigns a score of 0 to the nonfactual sentence C because the fire in Denver is not entailed by the other sentences. This results in sentence A scoring higher than the nonfactual sentence, and thus overcomes the problem in the original objective.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Although the typical approach of updating all the model's parameters during fine-tuning adapts well to the downstream task, the model suffers from imitative falsehood (Lin et al., 2021): The model learns to generate similar hallucinations present in the downstream dataset, and even completely forgets its factual behaviors learned during pretraining. This is especially problematic for datasets (b) Fine-tuning Figure 1: Illustration of FACTPEGASUS. For pre-training (a), we use the factGSG objective introduced in Section 3.1 that transforms a text document into a pseudo-summarization dataset. We select the pseudo-summary using the combination of ROUGE and FactCC. Here, sentence A is selected as the pseudo-summary, and we mask this sentence in the original text to create the pseudo-document. During fine-tuning (b), the connector (i) simulates the factGSG task by appending the same mask token used in (a) to the input document, so that we have the same setup in both training stages. Then, corrector (ii) removes hallucinations (highlighted in red) from the summary. Finally, contrastive learning in (iii) encourages the model to prefer the corrected summary over the perturbed summary.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
like XSum that contains hallucinations on 70% of the summaries (Maynez et al., 2020).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To this end, we present three complementary fine-tuning modules, illustrated in Figure 1b. Each component addresses different parts of the downstream task and collaboratively ensures factuality throughout the fine-tuning stage.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The GSG objective enables faster and better adaptation during fine-tuning by simulating the downstream task (Zhang et al., 2020). However, there still exists a gap between pre-training and finetuning: GSG is a masked sentence prediction task, but downstream summarization does not make use of the mask token. Thus, we simply insert the mask token into the input document of the downstream dataset, so as to simulate what the model expects during pre-training. This can be seen as a form of prompting, which helps us to elicit the factuality knowledge of the pre-trained models. We insert the mask token between sentences, and the best position is determined by evaluating the summarization performance on the validation set. We report the best position of the mask token and discuss the similarity to prompting in Appendix C.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The corrector removes hallucinations in the reference summaries so that such examples can be used during training without contributing to the problem of imitative falsehood. We consider summary entities as hallucinating if the text cannot be matched to one of the document entities. We propose three approaches with varying degrees of aggressiveness w.r.t. the removal of hallucinations and the possibility of generating ungrammatical sentences.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Replace: Upon qualitative analysis, we discover that some hallucinated entities in the summary are partially present in the documents. The most prominent example is the use of names, where the summary contains the full name of the person while only the first or last name is mentioned in the document, as shown in Figure 2. Given such observation, we propose a method to find a similar entity with the same NER label in the document and use that to replace the original hallucinated entity in the summary. Although this approach cannot correct hallucinations where similar entities are missing in the document, grammaticality is ensured.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Remove: A more aggressive approach is to remove the hallucinated entities in the training examples. The intuition is that it is often better to not say anything than to say something wrong. We mitigate the problem of creating ungrammatical sentences by removing related words to the removed entities determined by dependency arcs.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Combined: As a middle ground that ensures no hallucinations are present in the reference summaries while being grammatical when possible, we first replace all possible entities and then apply the remove strategy on the remaining ones.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We refer the readers to Appendix-B.1 for the details about hallucination detection, as well as the algorithm and discussion of grammatically for the remove method. 
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Arteta has taken up a coaching role at Manchester City.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Mikel Arteta has taken up a coaching role at Manchester City. Combined Former Arsenal midfielder Arteta has taken up a coaching role at Manchester City.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Manuel Pellegrini has taken up a coaching role. Extrinsic
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Wenger has taken up a coaching role. Figure 2: Example output using different strategies of corrector and contrastor. The first two rows show the original document and summary with highlighted entities and their respective labels (date, number, ent). We mark hallucinated entities in the summaries with red, factual entities in document and summary with green and underlined, and removed entities by the corrector with a strikethrough. Perturbed entities by the contrastor are italicized.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To better distinguish factual summaries from nonfactual ones, we next introduce a contrastive learning objective that encourages the model to prefer factual summaries given the context of the document. We use the document D i as the anchor and only consider the reference summary S i as the positive sample. Then, we create a set of nonfactual summaries N i to form negative pairs following Kryscinski et al. (2020), where we replace factual entities with random entities of the same named entity labels. We experiment with two variants simulating either extrinsic and intrinsic hallucinations. As formulated in Maynez et al. (2020), extrinsic hallucinations refer to entities that are present in the summary but not in the document, whereas intrinsic hallucinations are those that are present in the document but contain inaccurate information or are misplaced. See Appendix B.2 for more details. We stress that we perform contrastive learning between the document and the summary, similar to Lee et al. (2021), instead of between summaries (Cao and Wang, 2021), as it follows closer to the definition of faithfulness -the summary should be generated within the context of the document.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use the NT-Xent loss (Chen et al., 2020):
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
and z S j are representation for D i , S i and S j , respectively. We generate z D and z S by performing mean pooling over the last hidden layer of the encoder and decoder output, respectively. sim(·, ·) is the cosine similarity between the representations, and τ is the temperature parameter.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The final loss is calculated by the sum of the cross-entropy loss L CE and the contrastive loss:   Table 10.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We describe our experimental setup, and refer to Appendix A for more details.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We pre-train on the C4 dataset (Raffel et al., 2020), and evaluate our pre-trained model on three downstream abstractive summarization datasets: XSum (Narayan et al., 2018), WikiHow (Koupaee and Wang, 2018), and Gigaword (Rush et al., 2015). XSum is the primary dataset for analysis unless otherwise stated, as most of the factuality works for abstractive summarization evaluate on this dataset. Dataset details are presented in Appendix A.1. We report ROUGE-L (Lin, 2004) to evaluate our generated summaries against the reference. However, we note that this method is not ideal given the presence of hallucinations in the reference summaries (Chen et al., 2021;Maynez et al., 2020): If a more factual model does not produce such hallucinations, the output is scored lower than those that contain the same hallucinations found in the reference.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To evaluate factuality, there have been many proposed automatic metrics (Durmus et al., 2020;Wang et al., 2020;Scialom et al., 2021). We report FactCC (Kryscinski et al., 2020) and DEP-Entail (Goyal and Durrett, 2021), as they are highly correlated with human judgment of factuality (Pagnoni et al., 2021). For DEP-Entail, we report the tokenlevel and sentence-level error. For FactCC, since the model has been trained to evaluate on single sentences, we calculate the average score across all sentences for each summary.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To confirm our observation, we conduct human evaluation asking Amazon Mechanical Turk 2 (AMT) to judge the factuality and informativeness of the summaries. We randomly select 100 documents and ask the annotators to check whether each of the generated summaries is factual and informative. Appendix E provides more details.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For pre-training, we use BART-base's architecture with PEGASUS's SentencePiece (Kudo, 2018) unigram model tokenizer. We first determine the best sentence selection criteria by experimenting with selection criteria that use ROUGE-1, ROUGE-2, and ROUGE-L, as well as combining each with FactCC. To save computation (Lewis et al., 2020;Zhang et al., 2020;Raffel et al., 2020), we pretrain these models on a smaller dataset and fewer training steps. We report the effect of the selection criteria using the normalized ROUGE score and factuality scores over the model that uses ROUGE-1 as the selection criteria. We take the complement of token error and sentence error as token accuracy and sentence accuracy, respectively, to present all metrics where higher is better. Details of pretraining are shown in Appendix A.4.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Finally, We evaluate our pre-trained model on the three downstream tasks. As baselines, we compare our model to BART-base and PEGASUS*, our variant of the PEGASUS-base as there is no publicly available checkpoint. We train PEGA-SUS* by using the original sentence selection metric (ROUGE-1), and observe higher ROUGE scores on XSum and WikiHow than the ones reported in the original paper. We also compare FACTPEGA-SUS to two summarization models optimized for factuality. DAE (Goyal and Durrett, 2021) uses 2 https://www.mturk.com/   DEP-Entail to mask out the nonfactual tokens during training, and CLIFF (Cao and Wang, 2021) uses contrastive learning between the reference summaries and automatically generated nonfactual summaries. We apply both methods to BART-base. Details are described in Appendix A.5. Figure 3 shows the effect of different sentence selection criteria. Adding FactCC to all three ROUGE-only criteria consistently improves all factuality metrics at the cost of a small decrease in quality. Overall, the selection strategy of combining ROUGE-1 and FactCC achieves the highest FactCC score out of all strategies while maintaining the smallest relative drop in ROUGE.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present our full result on the three downstream tasks in Table 1. While the two baseline models achieve similar factuality scores, FACTPEGASUS consistently improves factuality over the two baselines on all three datasets. The largest improvement can be seen for the XSum dataset, where FACT-PEGASUS, compared to BART-base, lowers the token error and sentence error by 51% and 36%, respectively, and increases FactCC by 43% 3 . The same trend but to a lesser degree can also be observed for WikiHow and Gigaword, most notably a 3-point decrease in sentence error for WikiHow and a 2-point increase in FactCC for Gigaword.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Compared to factuality-aware models, FACTPE-GASUS achieves the highest FactCC on all tasks. Notably, FACTPEGASUS outperforms DAE by 34% on XSum. In terms of DEP-Entail, FACTPE-GASUS outperforms CLIFF on XSum and Wiki-How. We note that DAE is trained using the DEP-Entail metric and thus is not a fair comparison.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We note that the ROUGE-L scores for FACT-PEGASUS are lower than both baseline models by about 2 points, but we stress that our increase in FactCC is substantially larger than the decrease in ROUGE-L for XSum and Gigaword. The negative relationship between factuality metrics and ROUGE is also reported in prior works (Chen et al., 2021;Kryscinski et al., 2019). For example, finetuning BART on a subset of XSum (Goyal and Durrett, 2021) improves factuality at the cost of a 6-point drop in ROUGE-L 4 , which is triple the amount of decrease observed for our model.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Human Evaluation results are shown in Table 2. The result agrees with our observation on automatic factuality metrics, as FACTPEGASUS produces significantly more factual summaries than the BART-base, and PEGASUS*, and CLIFF. We achieve moderately significantly better summaries (p = 0.055) than DAE. Although, FACTPEGA-SUS achieves low informativeness, we find no statistical significant difference between our model and other models (p > 0.15).  Table 3: Fine-tuning ablation on XSum. We present our pre-trained model factGSG fine-tuned without any of our proposed components, and adding different strategies of corrector and contrastor. We then combine the best of the two modules (corrector combined and contrastor intrinsic), and finally add the connector to form the final model, which we copy from  
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present ablation studies of our proposed methods in Table 3. We first compare the performance of different strategies for the corrector and contrastor. For corrector, the level of aggressiveness in correcting hallucinations has a positive relationship with factuality metrics but a negative relationship with ROUGE-L. Although the remove method achieves the highest FactCC score, the combined method further lowers the token and sentence error while achieving relatively high ROUGE-L and FactCC. For contrastor, simulating intrinsic errors, which creates more challenging negative samples, provides better factuality results than simulating extrinsic ones. Finally, we show the additive gain in combining the best corrector and contrastor, as well as adding the connector to form the final model. We report the same ablation studies for Gigaword and Wikihow in Appendix D.3, and that for PEGASUS* in Appendix D.4.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
With the help of connector proposed in Section 3.2.1, we can explore how knowledge about factuality is transferred to fine-tuning, especially in the zero-shot and few-shot settings 5 .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Zero-Shot. We apply the mask token to the best position and directly analyze the performance of the models on the test set. To better understand the effectiveness in transferring knowledge about summarization and factuality from the pre-training objective, we apply the connector to our pretrained model (factGSG+mask) and PEGASUS* (GSG+mask), so that the two models differ only in their pre-training objective. We report the result in Table 4. FactGSG+mask outperforms GSG+mask on all metrics, especially for factuality metrics. Specifically, factGSG+mask lowers the sentence error by 5 points and increases FactCC by about 10 points. This observation confirms that the factGSG objective is more effective at capturing factuality than the original GSG objective.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Few-Shot. We follow a similar setup in Zhang et al. (2020), where we limit the number of training data to 1, 10, 100, and 1,000, and then fine-tune the model up to 2,000 steps with the patience of 10 epochs for early stopping. We select the checkpoint with the best validation performance.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We conduct this experiment by comparing FACT-PEGASUS to PEGASUS*, which has been shown for its ability to transfer with as little as 100 training examples (Zhang et al., 2020). In addition, we report the performance of factGSG+mask to understand how the the model is affected without explicitly ensuring factuality (i.e. without corrector and contrastor). As shown in Figure 4, connector allows the model to better make use of the knowledge of pre-training and produces highquality summaries, as both FACTPEGASUS and factGSG+mask produces a ROUGE-L score comparable to PEGASUS* trained with 1000 examples.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In terms of factuality, we notice that with just 10 examples, PEGASUS* starts to degrade in factuality, which also applies to the factGSG+mask model. However, FACTPEGASUS demonstrates an opposite trajectory: Sentence error decreases with more training data, and FactCC remains about the same score. This indicates that factual behavior is prone to be overwritten when factuality is not ensured explicitly, and thus calls for the importance of the corrector and contrastor.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To see whether the factuality degradation observed in few-shot experiment also applies to the full fine- tuning process, we extend our analysis by studying the factuality dynamics, similar to Goyal et al. (2022). The authors observe an increase in sentence errors with the BART model during finetuning, and we analyze whether similar factuality degradation occurs for FACTPEGASUS. We save checkpoints of our models every 10% of the total training steps, and evaluate the models on all three factuality metrics. Figure 5 shows the factuality dynamics during fine-tuning. We notice that the degradation occurs for both models but at a different degree. The token and sentence error for BART-base increase by 2 and 8 points, respectively. However, factuality for FACTPEGASUS remains similar, with only an increase of 1 point for token error and 4.8 points for sentence error. The degradation is only about half of what is observed with BART-base, indicating that FACTPEGASUS is better at avoiding learning nonfactual behaviors.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Lastly, we wish to understand whether our proposed method is effectively improving factuality without relying on the increase in extractiveness. To this end, Ladhak et al. (2021) introduces a faithfulness-abstractiveness trade-off curve to measure the faithfulness given the model's extractiveness. The authors kindly provided the same set of examples for Gigaword and AMT template for calculating the faithfulness score.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We show our result on Gigaword in Figure 6. We include the result of their proposed Selector and previous works, including Loss Truncation (Kang and Hashimoto, 2020) and DAE (Goyal and Dur-rett, 2021). We note that the baseline models increase factuality but mostly due to an increase in extractiveness and thus fall below the curve. In contrast, FACTPEGASUS lies above the line, indicating that we are effectively increasing factuality without relying much on becoming more extractive.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In this work, we proposed FACTPEGASUS, a model for abstractive summarization consisting of factuality-aware pre-training and modules for ensuring factuality during fine-tuning. We demonstrated the effectiveness of our model at improving factuality on three downstream abstractive summarization datasets, confirmed by our human evaluation. Our analysis showed that our proposed factuality-aware pre-training objective is effective at capturing knowledge of factuality compared to the original objective and that our fine-tuning modules reduce the factuality degradation observed with current models. We finally showed that improvement in factuality is not solely explained by the increase of extractiveness.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Our work aims at reducing the risk of generating hallucinations, and even possibly misinformation, for abstractive summarization models so that such models can be used safely for real-world applications. While we demonstrate that we can alleviate this problem, we stress that there is still a long way to go for improving factuality. Thus, we stress that such models should be used with caution for real-world applications. 
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
A.1 Datasets
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Following PEGASUS, we pre-train on the C4 dataset, a large collection of documents from Common Crawl. We evaluate our pre-trained model on three downstream abstractive summarization datasets: XSum, WikiHow, and Gigaword. XSum is a collection of articles from the British Broadcasting Corporation, Gigaword is a large collection of news articles headlines, and WikiHow consists of how-to articles. We show the dataset statistics for pre-training and fine-tuning in Table 5, where we present the number of examples in the training, validation, and test splits. We also show the number of examples corrected using the replace and remove method. All datasets are from datasets (Lhoest et al., 2021).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use the ROUGE package provided by transformers (Wolf et al., 2020). We follow the instructions provided by the authors of the factuality metrics to set up and run their code. We report all scores of our models from single runs.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use transformers library for the training script and the checkpoints of the pre-trained models. We use the default setting, including the AdamW optimizer and the linear rate scheduler. We also use mixed precision for both pre-training and finetuning the models. We conduct our experiments on the RTX A6000 GPU with 48GB memory and the A100 GPU with 40GB memory. BART-base model has 139M parameters, and PEGASUS* and FACTPEGASUS have 175M parameters.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Model Architecture. We use the same architecture as BART-base. Specifically, the model has L = 6, H = 768, F = 3072, A = 12, where L is the number of layers, H is the hidden size, F is the dimension for feed-forward layer, and A is the number of self-attention heads. We use the Sen-tencePiece (Kudo, 2018) unigram model tokenizer from PEGASUS with a vocabulary size of 96,103.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Sentence Selection Criteria. Before pretraining the full model, we first determine the best sentence selection criteria that produces more factual summaries with comparable quality. We experiment with sentence selection criteria that use ROUGE-1, ROUGE-2, and ROUGE-L, as well as combining each with FactCC. To understand the effect of the pre-training objective on factuality directly, we evaluate the performance on the XSum dataset without applying any of our proposed fine-tuning modules. Following Zhang et al.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
(2020), we report the models' relative performance to the base model, which only uses ROUGE-1 as the selection criteria. We use the normalized ROUGE F1 scores 1 3 ( R1 R1 base + R2 R2 base + RL RL base ), where R1 base , R2 base , and RL base are the ROUGE F1 scores of the base model. We similarly report the factuality metrics by normalizing each score by that of the base model. We take the complement of token error and sentence error as token accuracy and sentence accuracy, respectively, to present all metrics where higher is better.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Similar to previous works (Lewis et al., 2020;Zhang et al., 2020;Raffel et al., 2020) that save computational resources when selecting strategies for pre-training, we pre-train these model on the realnewslike subset of the C4 dataset with less steps.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Pre-training Details. We use a learning rate of 1e-4, a weight decay of 0.01, and set the maximum number of input tokens to be 512 and a maximum number of output tokens to be 256. We use a batch size of 256. We pre-train the full model for 750,000 steps with a warm-up of 20,000 steps, and only pretrain the smaller models for the sentence selection criteria experiment for 250,000 steps. Pre-training the smaller models takes 30 hours, and pre-training the full model takes 90 hours.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Calculating FactCC Score. In practice, running FactCC on each sentence-document pair of the pretraining data is expensive. Thus, we opt to only calculate the FactCC score for the top 5 sentences  according to the ROUGE score between the sentence and the rest of the document.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For all datasets, we use a label smoothing of 0.1. For decoding, we use a beam size of 6 for all datasets. Task-specific hyper-parameters are shown in Table 6. Fine-tuning on XSum and WikiHow takes 8 hours, and fine-tuning on Gigaword takes 11 hours. Decoding on XSum and Gigaword takes half an hour, while decoding WikiHow takes an hour. We use 5 negative examples for the contrastor and set λ to 5 when calculating the combined loss. We set the temperature τ to 0.05. For fine-tuning DAE and CLIFF, we follow the authors' instructions and fine-tune BART-base with their respective code and hyper-parameters. For WikiHow and Gigaword, we use the same hyperparameters as above.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use spaCy's NER model 6 to find entities in the document and summary. Entities in the summary sentence are considered nonfactual if no matching document entities with the same string are found. We have previously experimented with the additional requirement of matching entity type similar to Kryscinski et al. (2020), but we find that this constraint unintentionally causes some correct entities to be considered hallucinating, leading to unnecessarily less informative summaries when removed. Given hallucinated entities, we can perform either replace or remove operations. For replace, we find document entities whose words are all contained in the selected entity.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For the remove method, we need to make sure to also remove any related words. We use spaCy's 6 We use the en_core_web_trf model. dependency parser to systematically remove those. The algorithm is as follows: We first add all the tokens in the selected hallucinated entity to the list of tokens to remove. Then, we recursively find all parents that contain the dependency relation of pobj and prep without any other children and add those to the tokens to remove. Finally, we add all children that do not have the label compound, relcl, and fixed. The final set of words will then be removed in the summary sentence.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We qualitatively observe that this approach can cover most of the edge cases that would otherwise result in ungrammatical sentences. Nevertheless, this method is not perfect. We include some sample output with the remove method in Figure 7. The algorithm is good at removing entities and related words, such as prepositions, as illustrated in example 1, 3, and 5. However, we observe that it will create ungrammatical sentences when the hallucinated entity is the subject (example 2), or the object of a transitive verb (example 6).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We leave exploration with the best systematic correction algorithm or models for future work.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Similar to Kryscinski et al. (2020), we generate hallucinated summaries by performing entity perturbation on the original summaries. We find entity candidates using the NER labels and sort them into three categories: We include MONEY, QUAN-TITY, and CARDINAL as number, DATE and TIME as date, and all other labels as named entities. We randomly select a factual entity in the summary and replace it with an entity belonging to the same category.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For extrinsic hallucinations, we sample candidates of the same category from the training corpus but exclude those present in the document. For the intrinsic case, we select to consider the entities from the document. The number of negative examples for all tasks is 5.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Learning rate Num Steps Warmup Batch size Max Input tokens Max Target tokens   XSum  3e-5  15k  500  256  512  64  WikiHow  3e-5  15k  500  256  512  256  Gigaword  3e-5  50k  2000  256 128 32  
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
This mask-token fine-tuning technique can be seen as a form of prompting, where we elicit our desired faithful abstractive summarization behavior from the pre-trained model directly. Specifically, we consider this as null-prompting (Logan et al., 2021), where using the mask token as the prompt can achieve competitive results with manually engineered prompts. Conveniently, since the mask token during pre-training already serves as a placeholder of where the summary sentence should be generated, it naturally serves as a valid prompt. Figure 1b shows an example of adding the mask token before the first sentence and thus creating a similar setup for pre-training.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We first need to determine the best position of mask token, as discussed in Section 3.2.1, where we insert the mask token before the ith sentence of the document, where i = 1, 2, ..., 6, and select the best position that achieves the highest ROUGE score on the dev collection. We report ROUGE score of all positions in Table 7 for the three datasets. Interestingly, we observe that the best mask token position for all datasets is before the first sentence. This agrees with the dataset generation of XSum: the summary is taken from the first sentence of the original article. For Gigaword, there is not a change after the first sentence, since the document only consists of a single sentence.   Table 9: Full Result of zero-shot and few-shot experiments.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We report the full result for the sentence selection criteria in Table 10. Surprisingly, each sentence selection criteria that uses FactCC excels in one specific factuality metric: R1+FactCC is best at FactCC, R2+FactCC is best at sentence error, and RL+FactCC is best for token error.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present additional results of the zero-shot and few-shot experiments here.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Zero-shot We first report the reference-based result of the two baseline models and FACTPEGA-  Table 10: Full result of pre-trained models with different sentence selection criteria shown in Figure 3. We denote the criteria with FactCC with (+FC). Table 8. Due to the mismatch of pretraining and fine-tuning, we observe that both baseline models perform much worse than their result when fully trained. However, with the help of the connector, we observe 11.5 ROUGE-1 points increase for our model compared to the baseline models, and almost four times and double the score for ROUGE-2 and ROUGE-L, respectively.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Few-shot We show FACTPEGASUS's full result of the few-shot experiment in Table 9.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We report ablation of each fine-tuning components on Gigaword and Wikihow. The result can be found in Table 11. We observe similar trend as Table 3, where each component improves the performance. For WikiHow and Gigaword, the extrinsic method for contrastive learning perform the best. We think that this is due to the fact that the two tasks do not contain rich entities in the document, and thus require introduction of additional entities from the training corpus.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We similarly perform the same ablation using the PEGASUS* model, which we present in Table 12.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We observe similar trend as Table 3. We note that using our pre-trained model factGSG achieves better factuality than PEGASUS* in each setting.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To ensure high-quality annotations, we select the workers from the United States and have more than 10,000 number of HITS approved as well as an approval rate greater than 98%. In addition, we also create a qualification test where we rate the factuality of the selected generated summaries. Such examples include cases where some summaries hallucinate the first name of a person, which the workers should mark them as not factual. Only workers with the correct annotation can perform the actual task.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To avoid giving too much text to the workers, we select the most important sentences and replace the less relevant sentences with an ellipsis. For each of the summaries, we select the ten most relevant sentences from the document by cosine similarity of the sentence embedding using SentenceTransformer 7 (Reimers and Gurevych, 2019). We combine and show all the selected relevant sentences from each summary. Since the summaries are similar, we see a large overlap of the relevant sentences.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We give the following prompt, which we modify from Dreyer et al. (2021): • consistency/factuality: Please avoid using general knowledge, and only consider it in the context of the provided document. Select not consistent if facts in the summary are not supported by the document, such as cases like these:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
1. The summary contradicts the information in the document. The summary might say "A fire broke out in Seattle", but a document says it broke out in Portland. Or the summary might say "the Republicans won the election", but the document indicates the Democrats won instead 2. The summary adds (hallucinates) a fact that is not mentioned anywhere in the document. For example, the summary might say that "A fire broke out at 2 am", but the document doesn't mention the time when the fire broke out.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
• Informativeness: Please select informative if the summary expresses the main points of the document. Summary should contain relevant and important information and few unimportant details. If you select the summary to be not consistent with the document, please only consider the consistent information when evaluating this category.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The order of the summary is randomly shuffled. Each task consists of three unique workers, where we take the mean as the scores for this document. The final score is the mean factuality score across all documents. The average time for each task is 7 We use the all-mpnet-base-v2 model.   Table 12: Fine-tuning ablation on XSum using PEGA-SUS*. We combine the modules by using the corrector combined and contrastor intrinsic. We name the model with all three components as PEGASUS* full.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
around 3 minutes and we pay 0.6 USD per task, hence an hourly rate of ≥ $12 per hour. We use boostrap test (Efron and Tibshirani, 1993) to determine statistical significance between two summaries with p < 0.05.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We include some sample outputs in Figure 8 and Figure 9.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Example 1 (XSum) Summary: Tap water in 80,000 homes in Lancashire has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water for three weeks. Remove: Tap water in homes has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Example 2 (XSum) Summary: Great Britain's Becky James won her second Olympic silver of Rio 2016 by finishing second in the women's sprint. Remove: Won her second Olympic silver of Rio 2016 by finishing second in the women's sprint.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic.Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye in Sharpies is usually weak against it. Remove: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic.Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye is usually weak against it.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: Call 911 if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your daily routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys. Remove: Call if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: xinhua summary of asia-pacific stocks news on tuesday feburary ## Remove: summary of stocks news on tuesday ## Example 6 (Gigaword) Summary: cuba urges eu to drop its common position Remove: cuba urges to drop its common position Figure 7: Summaries changed using the corrector. We mark hallucinated entities in the summaries with red.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Article: The Scots started their Six Nations campaign with a thrilling first win over Ireland in four years. They beat France for the first time in 10 years last season at home, but have lost on their past nine trips to Paris. "It is a long time ago since we won there but we understand there is a reason for that," said Laidlaw. "France are a good team and it is a hard place to win. We are under no illusions this week -we need to be on top form to get something out of the game. "We'll need accuracy and we need to be physical and aggressive because the French are big men. But we need to be subtle and clever in our attack. "If we do that, and move France about and play for 80 minutes, I would back our fitness against them. But we obviously need to be in the game right to the death and not let France get any momentum." Media playback is not supported on this device France have not finished in the top half of the Six Nations since 2011, but impressed in narrow defeats by New Zealand and Australia in the autumn, and came close to a shock victory against England at Twickenham in their Six Nations opener. "They have improved massively under [coach] Guy Noves and have not been far away from any team they have played," Laidlaw noted. "They have some really dangerous strike runners throughout the team. "As much as we need to understand why we beat them last year, we need to develop that, be clever and smart and be able to score points." We need to keep doing what we are doing well, keep our skills high and if we do that again this weekend, we are going to put ourselves in a strong position. "The start of the game is massively important, whether it is our kick-off, or or us receiving. It is very important psychologically for them as it is for us. "If we can start the game playing in the right areas of the field for the first 20 minutes and not muck about too much in our own half, hopefully we will put them under pressure and upset the crowd a little bit." Media playback is not supported on this device Reference: Captain Greg Laidlaw says Scotland must be "subtle and clever" as well as match France physically if they are to secure a first win in Paris since 1999. Bart-base: Scotland head coach Greig Laidlaw says his side are "under no illusions" when they face France at Twickenham on Saturday. PEGASUS*: Scotland coach Steve Laidlaw has warned France that they need to be "smart and smart" in their attack if they are to win their first Six Nations game of the season against the French in Paris on Saturday. CLIFF: Scotland head coach Mark Laidlaw says his side need to be " subtle and clever" in their attack against France in Paris onSaturday. DAE:Scotland must be " subtle and clever" in their attack when they face France at Twickenham on Saturday, says head coach Gordon Laidlaw. FACTPEGASUS: Scotland captain Laidlaw says his side need to be "subtle and clever" in their attack if they are to beat France.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Article: the investigating magistrate and prosecutor handling the case involving the french association , zoe ' s ark , arrived tuesday in the eastern town of abeche where they met the ### children the association was attempting to fly to france , according to reports .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Reference: chadian court officials go to UNK to investigation into children 's case Bart-base: zoe's ark investigating magistrate arrives in abeche PEGASUS*: prosecutor in zoe's ark case arrives in abeche CLIFF: prosecutor meets zoe 's ark children DAE: zoe's ark investigating magistrate prosecutor arrive in abeche FACTPEGASUS: prosecutor arrives in abeche to investigate french association 
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Today's pretrained language models are trained on massive, heterogeneous corpora (Raffel et al., 2019;Yang et al., 2019). For instance, ROBERTA  was trained on over 160GB of uncompressed text, with sources ranging from Englishlanguage encyclopedic and news articles, to literary works and web content. Representations learned by such models achieve strong performance across many tasks with datasets of varying sizes drawn from a variety of sources (e.g., Wang et al., 2018Wang et al., , 2019. This leads us to ask whether a task's textual domain-a term typically used to denote a distribution over language characterizing a given topic or genre (such as "science" or "mystery novels")-is still relevant. Do the latest large pretrained models work universally or is it still helpful to build Figure 1: An illustration of data distributions. Task data is comprised of an observable task distribution, usually non-randomly sampled from a wider distribution (light grey ellipsis) within an even larger target domain, which is not necessarily one of the domains included in the original LM pretraining domain -though overlap is possible. We explore the benefits of continued pretraining on data from the task distribution and the domain distribution. separate pretrained models for specific domains?
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
While some studies have shown the benefit of continued pretraining on domain-specific unlabeled data (e.g., , these studies only consider a single domain at a time and use a language model that is pretrained on a smaller and less diverse corpus than the most recent language models. Moreover, it is not known how the benefit of continued pretraining may vary with factors like the amount of available labeled task data, or the proximity of the target domain to the original pretraining corpus (see Figure 1).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
We address this question for one such highperforming model, ROBERTA  ( §2). We consider four domains (biomedical and computer science publications, news, and reviews; §3) and eight classification tasks (two in each domain). For targets that are not already in-domain for ROBERTA, our experiments show that contin-ued pretraining on the domain (which we refer to as domain-adaptive pretraining or DAPT) consistently improves performance on tasks from the target domain, in both high-and low-resource settings.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Above, we consider domains defined around genres and forums, but it is also possible to induce a domain from a given corpus used for a task, such as the one used in supervised training of a model. This raises the question of whether pretraining on a corpus more directly tied to the task can further improve performance. We study how domainadaptive pretraining compares to task-adaptive pretraining, or TAPT, on a smaller but directly taskrelevant corpus: the unlabeled task dataset ( §4), drawn from the task distribution. Task-adaptive pretraining has been shown effective (Howard and Ruder, 2018), but is not typically used with the most recent models. We find that TAPT provides a large performance boost for ROBERTA, with or without domain-adaptive pretraining.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Finally, we show that the benefits from taskadaptive pretraining increase when we have additional unlabeled data from the task distribution that has been manually curated by task designers or annotators. Inspired by this success, we propose ways to automatically select additional task-relevant unlabeled text, and show how this improves performance in certain low-resource cases ( §5). On all tasks, our results using adaptive pretraining techniques are competitive with the state of the art.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In summary, our contributions include:
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
• a thorough analysis of domain-and taskadaptive pretraining across four domains and eight tasks, spanning low-and high-resource settings; • an investigation into the transferability of adapted LMs across domains and tasks; and • a study highlighting the importance of pretraining on human-curated datasets, and a simple data selection strategy to automatically approach this performance. Our code as well as pretrained models for multiple domains and tasks are publicly available. 1
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Learning for most NLP research systems since 2018 consists of training in two stages. First, a neural language model (LM), often with millions of parameters, is trained on large unlabeled cor-pora. The word (or wordpiece; Wu et al. 2016) representations learned in the pretrained model are then reused in supervised training for a downstream task, with optional updates (fine-tuning) of the representations and network from the first stage.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
One such pretrained LM is ROBERTA , which uses the same transformerbased architecture (Vaswani et al., 2017) as its predecessor, BERT (Devlin et al., 2019). It is trained with a masked language modeling objective (i.e., cross-entropy loss on predicting randomly masked tokens). The unlabeled pretraining corpus for ROBERTA contains over 160 GB of uncompressed raw text from different English-language corpora (see Appendix §A.1). ROBERTA attains better performance on an assortment of tasks than its predecessors, making it our baseline of choice.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Although ROBERTA's pretraining corpus is derived from multiple sources, it has not yet been established if these sources are diverse enough to generalize to most of the variation in the English language. In other words, we would like to understand what is out of ROBERTA's domain. Towards this end, we explore further adaptation by continued pretraining of this large LM into two categories of unlabeled data: (i) large corpora of domain-specific text ( §3), and (ii) available unlabeled data associated with a given task ( §4).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Our approach to domain-adaptive pretraining (DAPT) is straightforward-we continue pretraining ROBERTA on a large corpus of unlabeled domain-specific text. The four domains we focus on are biomedical (BIOMED) papers, computer science (CS) papers, newstext from REALNEWS, and AMAZON reviews. We choose these domains because they have been popular in previous work, and datasets for text classification are available in each. Table 1 lists the specifics of the unlabeled datasets in all four domains, as well as ROBERTA's training corpus. 1
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Before performing DAPT, we attempt to quantify the similarity of the target domain to ROBERTA's pretraining domain. We consider domain vocabularies containing the top 10K most frequent unigrams (excluding stopwords) in comparably sized random samples of held-out documents in each domain's corpus. We use 50K held-out documents for each domain other than REVIEWS, and 150K held-out documents in REVIEWS, since they are much shorter. We also sample 50K documents from sources similar to ROBERTA's pretraining corpus (i.e., BOOKCORPUS, STORIES, WIKIPEDIA, and REALNEWS) to construct the pretraining domain vocabulary, since the original pretraining corpus is not released. Figure 2 shows the vocabulary overlap across these samples. We observe that ROBERTA's pretraining domain has strong vocabulary overlap with NEWS and REVIEWS, while CS and BIOMED are far more dissimilar to the other domains. This simple analysis suggests the degree of benefit to be expected by adaptation of ROBERTA to different domains-the more dissimilar the domain, the higher the potential for DAPT.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Our LM adaptation follows the settings prescribed for training ROBERTA. Under each domain, we consider two text classification tasks, as shown in Table 2. Our tasks represent both high-and low-resource (≤ 5K labeled training examples, and no additional unlabeled data) settings. For HYPERPARTISAN, we use the data splits from Beltagy et al. (2020). For RCT, we represent all sentences in one long sequence for simultaneous prediction.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Baseline As our baseline, we use an off-the-shelf ROBERTA-base model and perform supervised fine-tuning of its parameters for each classification task. On average, ROBERTA is not drastically behind the state of the art (details in Appendix §A.2), and serves as a good baseline since it provides a single LM to adapt to different domains.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Classification Architecture Following standard practice (Devlin et al., 2019) we pass the final layer [CLS] token representation to a task-specific feedforward layer for prediction (see Table 14 in Appendix for more hyperparameter details).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Test results are shown under the DAPT column of Table 3 (see Appendix §C for validation results). We observe that DAPT improves over ROBERTA in all domains. For BIOMED, CS, and REVIEWS, we see consistent improve-     
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Additionally, we compare DAPT against a setting where for each task, we adapt the LM to a domain outside the domain of interest. This controls for the case in which the improvements over ROBERTA might be attributed simply to exposure to more data, regardless of the domain. In this setting, for NEWS, we use a CS LM; for REVIEWS, a BIOMED LM; for CS, a NEWS LM; for BIOMED, a REVIEWS LM. We use the vocabulary overlap statistics in Figure 2 to guide these choices.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Our results are shown in Table 3, where the last column (¬DAPT) corresponds to this setting. For each task, DAPT significantly outperforms adapting to an irrelevant domain, suggesting the importance of pretraining on domain-relevant data. Furthermore, we generally observe that ¬DAPT results in worse performance than even ROBERTA on end-tasks. Taken together, these results indicate that in most settings, exposure to more data without considering domain relevance is detrimental to end-task performance. However, there are two tasks (SCIERC and ACL-ARC) in which ¬DAPT marginally improves performance over ROBERTA. This may suggest that in some cases, continued pretraining on any additional data is useful, as noted in Baevski et al. (2019).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Our analysis of DAPT is based on prior intuitions about how task data is assigned to specific domains. For instance, to perform DAPT for HELPFULNESS, we only adapt to AMAZON reviews, but not to any REALNEWS articles. However, the gradations in Figure 2 suggest that the boundaries between domains are in some sense fuzzy; for example, 40% of unigrams are shared between REVIEWS and NEWS. As further indication of this overlap, we also qualitatively identify documents that overlap cross-domain: in Table 4, we showcase reviews and REALNEWS articles that are similar to these reviews (other examples can be found in Appendix §D). In fact, we find that adapting ROBERTA to IMDB review REALNEWS article "The Shop Around the Corner" is one of the great films from director Ernst Lubitsch . In addition to the talents of James Stewart and Margaret Sullavan , it's filled with a terrific cast of top character actors such as Frank Morgan and Felix Bressart. [...] The makers of "You've Got Mail" claim their film to be a remake , but that's just nothing but a lot of inflated self praise. Anyway, if you have an affection for romantic comedies of the 1940 's, you'll find "The Shop Around the Corner" to be nothing short of wonderful. Just as good with repeat viewings.  NEWS not as harmful to its performance on RE-VIEWS tasks (DAPT on NEWS achieves 65.5 2.3 on HELPFULNESS and 95.0 0.1 on IMDB). Although this analysis is by no means comprehensive, it indicates that the factors that give rise to observable domain differences are likely not mutually exclusive. It is possible that pretraining beyond conventional domain boundaries could result in more effective DAPT; we leave this investigation to future work. In general, the provenance of data, including the processes by which corpora are curated, must be kept in mind when designing pretraining procedures and creating new benchmarks that test out-of-domain generalization abilities.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Datasets curated to capture specific tasks of interest tend to cover only a subset of the text available within the broader domain. For example, the CHEMPROT dataset for extracting relations between chemicals and proteins focuses on abstracts of recently-published, high-impact articles from hand-selected PubMed categories (Krallinger et al., 2017(Krallinger et al., , 2015. We hypothesize that such cases where the task data is a narrowly-defined subset of the broader domain, pretraining on the task dataset itself or data relevant to the task may be helpful.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Task-adaptive pretraining (TAPT) refers to pretraining on the unlabeled training set for a given task; prior work has shown its effectiveness (e.g. Howard and Ruder, 2018). Compared to domainadaptive pretraining (DAPT; §3), the task-adaptive approach strikes a different trade-off: it uses a far smaller pretraining corpus, but one that is much more task-relevant (under the assumption that the training set represents aspects of the task well). This makes TAPT much less expensive to run than DAPT, and as we show in our experiments, the performance of TAPT is often competitive with that of DAPT.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Similar to DAPT, task-adaptive pretraining consists of a second phase of pretraining ROBERTA, but only on the available task-specific training data. In contrast to DAPT, which we train for 12.5K steps, we perform TAPT for 100 epochs. We artificially augment each dataset by randomly masking different words (using the masking probability of 0.15) across epochs. As in our DAPT experiments, we pass the final layer [CLS] token representation to a task-specific feedforward layer for classification (see Table 14 in Appendix for more hyperparameter details).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Our results are shown in the TAPT column of Table 5. TAPT consistently improves the ROBERTA baseline for all tasks across domains. Even on the news domain, which was part of ROBERTA pretraining corpus, TAPT improves over ROBERTA, showcasing the advantage of task adaptation. Particularly remarkable are the relative differences between TAPT and DAPT. DAPT is more resource intensive (see Table 9 in §5.3), but TAPT manages to match its performance in some of the tasks, such as SCIERC. In RCT, HYPERPARTISAN, AGNEWS, HELPFULNESS, and IMDB, the results even exceed those of DAPT, highlighting the efficacy of this cheaper adaptation technique.    (Table 5), it is harmful when applied across tasks. These findings illustrate differences in task distributions within a domain.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Combined DAPT and TAPT We investigate the effect of using both adaptation techniques together. We begin with ROBERTA and apply DAPT then TAPT under this setting. The three phases of pretraining add up to make this the most computationally expensive of all our settings (see Table 9). As expected, combined domain-and task-adaptive pretraining achieves the best performance on all tasks (Table 5). 2
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Overall, our results show that DAPT followed by TAPT achieves the best of both worlds of domain and task awareness, yielding the best performance. While we speculate that TAPT followed by DAPT would be susceptible to catastrophic forgetting of the task-relevant corpus (Yogatama et al., 2019), alternate methods of combining the procedures may result in better downstream performance. Future work may explore pretraining with a more sophisticated curriculum of domain and task distributions.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
We complete the comparison between DAPT and TAPT by exploring whether adapting to one task transfers to other tasks in the same domain. For instance, we further pretrain the LM using the RCT unlabeled data, fine-tune it with the CHEMPROT labeled data, and observe the effect. We refer to this setting as Transfer-TAPT. Our results for tasks in all four domains are shown in Table 6. We see that TAPT optimizes for single task performance, to the detriment of cross-task transfer. These results demonstrate that data distributions of tasks within a given domain might differ. Further, this could also explain why adapting only to a broad domain is not sufficient, and why TAPT after DAPT is effective.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In §4, we continued pretraining the LM for task adaptation using only the training data for a supervised task. Inspired by the success of TAPT, we next investigate another setting where a larger pool of unlabeled data from the task distribution exists,  Table 7: Mean test set macro-F 1 (for HYP. and IMDB) and micro-F 1 (for RCT-500), with Curated-TAPT across five random seeds, with standard deviations as subscripts. † indicates high-resource settings.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
typically curated by humans. We explore two scenarios. First, for three tasks (RCT, HYPERPARTISAN, and IMDB) we use this larger pool of unlabeled data from an available human-curated corpus ( §5.1). Next, we explore retrieving related unlabeled data for TAPT, from a large unlabeled in-domain corpus, for tasks where extra human-curated data is unavailable ( §5.2).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Dataset creation often involves collection of a large unlabeled corpus from known sources. This corpus is then downsampled to collect annotations, based on the annotation budget. The larger unlabeled corpus is thus expected to have a similar distribution to the task's training data. Moreover, it is usually available. We explore the role of such corpora in task-adaptive pretraining.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Data We simulate a low-resource setting RCT-500, by downsampling the training data of the RCT dataset to 500 examples (out of 180K available), and treat the rest of the training data as unlabeled. The HYPERPARTISAN shared task (Kiesel et al., 2019) has two tracks: low-and high-resource. We use 5K documents from the high-resource setting as Curated-TAPT unlabeled data and the original lowresource training documents for task fine-tuning. For IMDB, we use the extra unlabeled data manually curated by task annotators, drawn from the same distribution as the labeled data (Maas et al., 2011).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
We compare Curated-TAPT to TAPT and DAPT + TAPT in Table 7. Curated-TAPT further improves our prior results from §4 across all three datasets. Applying Curated-TAPT after adapting to the domain results in the largest boost in performance on all tasks; in HYPERPARTISAN, DAPT + Curated-TAPT is within standard deviation of Curated-TAPT. Moreover, curated-TAPT achieves Figure 3: An illustration of automated data selection ( §5.2). We map unlabeled CHEMPROT and 1M BIOMED sentences to a shared vector space using the VAMPIRE model trained on these sentences. Then, for each CHEMPROT sentence, we identify k nearest neighbors, from the BIOMED domain.  Table 8: Mean test set micro-F 1 (for CHEMPROT and RCT) and macro-F 1 (for ACL-ARC), across five random seeds, with standard deviations as subscripts, comparing RAND-TAPT (with 50 candidates) and kNN-TAPT selection. Neighbors of the task data are selected from the domain data.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
95% of the performance of DAPT + TAPT with the fully labeled RCT corpus (Table 5) with only 0.3% of the labeled data. These results suggest that curating large amounts of data from the task distribution is extremely beneficial to end-task performance. We recommend that task designers release a large pool of unlabeled task data for their tasks to aid model adaptation through pretraining.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Consider a low-resource scenario without access to large amounts of unlabeled data to adequately benefit from TAPT, as well as absence of computational resources necessary for DAPT (see Table 9 for details of computational requirements for different pretraining phases). We propose simple unsuper-vised methods to retrieve unlabeled text that aligns with the task distribution, from a large in-domain corpus. Our approach finds task-relevant data from the domain by embedding text from both the task and domain in a shared space, then selects candidates from the domain based on queries using the task data. Importantly, the embedding method must be lightweight enough to embed possibly millions of sentences in a reasonable time.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Given these constraints, we employ VAMPIRE (Gururangan et al., 2019; Figure 3), a lightweight bag-of-words language model. We pretrain VAM-PIRE on a large deduplicated 3 sample of the domain (1M sentences) to obtain embeddings of the text from both the task and domain sample. We then select k candidates of each task sentence from the domain sample, in embeddings space. Candidates are selected (i) via nearest neighbors selection (kNN-TAPT) 4 , or (ii) randomly (RAND-TAPT). We continue pretraining ROBERTA on this augmented corpus with both the task data (as in TAPT) as well as the selected candidate pool. Table 8 show that kNN-TAPT outperforms TAPT for all cases. RAND-TAPT is generally worse than kNN-TAPT, but within a standard deviation arising from 5 seeds for RCT and ACL-ARC. As we increase k, kNN-TAPT performance steadily increases, and approaches that of DAPT. Appendix F shows examples of nearest neighbors of task data. Future work might consider a closer study of kNN-TAPT, more sophisticated data selection methods, and the tradeoff between the diversity and task relevance of selected examples.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The computational requirements for all our adaptation techniques on RCT-500 in the BIOMED domain in Table 9. TAPT is nearly 60 times faster to train than DAPT on a single v3-8 TPU and storage requirements for DAPT on this task are 5.8M times that of TAPT. Our best setting of DAPT + TAPT amounts to three phases of pretraining, and at first glance appears to be very expensive. However, once the LM has been adapted to a broad domain, it can be reused for multiple tasks within that domain, with only a single additional TAPT phase per task. While Curated-TAPT tends to achieve the best cost-  Table 9: Computational requirements for adapting to the RCT-500 task, comparing DAPT ( §3) and the various TAPT modifications described in §4 and §5.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
benefit ratio in this comparison, one must also take into account the cost of curating large in-domain data. Automatic methods such as kNN-TAPT are much cheaper than DAPT.  data (Swayamdipta et al., 2019). We compare DAPT and TAPT as well as their interplay with respect to dataset size for continued pretraining (hence, expense of more rounds of pretraining), relevance to a data sample of a given task, and transferability to other tasks and datasets. See Table 11 in Appendix §A for a summary of multi-phase pretraining strategies from related work.  (Remus and Biemann, 2016).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
What is a domain? Despite the popularity of domain adaptation techniques, most research and practice seems to use an intuitive understanding of domains. A small body of work has attempted to address this question (Lee, 2001;Eisenstein et al., 2014;van der Wees et al., 2015;Plank, 2016;Ruder et al., 2016, among others). For instance, Aharoni and Goldberg (2020) define domains by implicit clusters of sentence representations in pretrained LMs. Our results show that DAPT and TAPT complement each other, which suggests a spectra of domains defined around tasks at various levels of granularity (e.g., Amazon reviews for a specific product, all Amazon reviews, all reviews on the web, the web).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
We investigate several variations for adapting pretrained LMs to domains and tasks within those domains, summarized in Table 10. Our experiments reveal that even a model of hundreds of millions of parameters struggles to encode the complexity of a single textual domain, let alone all of language. We show that pretraining the model towards a specific task or small corpus can provide significant benefits. Our findings suggest it may be valuable to complement work on ever-larger LMs with parallel efforts to identify and use domain-and taskrelevant corpora to specialize models. While our results demonstrate how these approaches can improve ROBERTA, a powerful LM, the approaches we studied are general enough to be applied to any pretrained LM. Our work points to numerous future directions, such as better data selection for TAPT, efficient adaptation large pretrained language models to distant domains, and building reusable language models after adaptation. 
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In this supplementary material, we provide: (i) additional information for producing the results in the paper, and (ii) results that we could not fit into the main body of the paper.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Appendix A. A tabular overview of related work described in Section §6, a description of the corpus used to train ROBERTA in , and references to the state of the art on our tasks.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Appendix B. Details about the data preprocessing, training, and implementation of domain-and taskadaptive pretraining.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Appendix C. Development set results.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Appendix D. Examples of domain overlap.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Appendix E. The cross-domain masked LM loss and reproducibility challenges.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
A Related Work Table 11 shows which of the strategies for continued pretraining have already been explored in the prior work from the Related Work ( §6). As evident from the table, our work compares various strategies as well as their interplay using a pretrained language model trained on a much more heterogeneous pretraining corpus.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
ROBERTA was trained on data from BOOKCOR-PUS (Zhu et al., 2015), 6 WIKIPEDIA, 7 a portion of the CCNEWS dataset (Nagel, 2016), 8 OPENWEB-TEXT corpus of Web content extracted from URLs shared on Reddit (Gokaslan and Cohen, 2019), 9 and a subset of CommonCrawl that it is said to resemble the "story-like" style of WINOGRAD schemas (STORIES; Trinh and Le, 2018). 10
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In this section, we specify the models achieving state of the art on our tasks. See the caption of 6 https://github.com/soskek/bookcorpus 7 https://github.com/google-research/ bert 8 https://github.com/fhamborg/ news-please 9 https://github.com/jcpeterson/ openwebtext 10 https://github.com/tensorflow/models/ tree/master/research/lm_commonsense Table 5 for the reported performance of these models. For ACL-ARC, that is SCIBERT (Beltagy et al., 2019), a BERT-base model for trained from scratch on scientific text. For CHEMPROT and SCI-ERC, that is S2ORC-BERT (Lo et al., 2020), a similar model to SCIBERT. For AGNEWS and IMDB, XLNet-large, a much larger model. For RCT, Cohan et al. (2019). For HYPERPARTISAN, LONGFORMER, a modified Transformer language model for long documents (Beltagy et al., 2020). Thongtan and Phienthrakul (2019) report a higher number (97.42) on IMDB, but they train their word vectors on the test set. Our baseline establishes the first benchmark for the HELPFULNESS dataset.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Preprocessing for DAPT The unlabeled corpus in each domain was pre-processed prior to language model training. Abstracts and body paragraphs from biomedical and computer science articles were used after sentence splitting using scispaCy (Neumann et al., 2019). We used summaries and full text of each news article, and the entire body of review from Amazon reviews. For both news and reviews, we perform sentence splitting using spaCy (Honnibal and Montani, 2017).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Training details for DAPT We train ROBERTA on each domain for 12.5K steps. We focused on matching all the domain dataset sizes (see Table  1) such that each domain is exposed to the same amount of data as for 12.5K steps it is trained for. AMAZON reviews contain more documents, but each is shorter. We used an effective batch size of 2048 through gradient accumulation, as recommended in . See Table 13 for more hyperparameter details.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Training details for TAPT We use the same pretraining hyperparameters as DAPT, but we artificially augmented each dataset for TAPT by randomly masking different tokens across epochs, using the masking probability of 0.15. Each dataset was trained for 100 epochs. For tasks with less than 5K examples, we used a batch size of 256 through gradient accumulation. See Table 13 for more hyperparameter details.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Optimization We used the Adam optimizer (Kingma and Ba, 2015), a linear learning rate scheduler with 6% warm-up, a maximum learning rate of 0.0005. When we used a batch size of 256, we  used a maximum learning rate of 0.0001, as recommended in . We observe a high variance in performance between random seeds when fine-tuning ROBERTA to HYPERPARTISAN, because the dataset is extremely small. To produce final results on this task, we discard and resample degenerate seeds. We display the full hyperparameter settings in Table 13.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Implementation Our LM implementation uses the HuggingFace transformers library (Wolf et al., 2019) 
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In 
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In Section §3.2, we provide ROBERTA's masked LM loss before and after DAPT. We display crossdomain masked-LM loss in Table 12, where we evaluate masked LM loss on text samples in other domains after performing DAPT. We observe that the cross-domain masked-LM loss mostly follows our intuition and insights from the paper, i.e. ROBERTA's pretraining corpus and NEWS are closer, and BIOMED to CS (relative to other domains). However, our analysis in §3.1 illustrates that REVIEWS and NEWS also have some similarities. This is supported with the loss of ROBERTA that is adapted to NEWS, calculated on a sample of REVIEWS. However, ROBERTA that is adapted to REVIEWS results in the highest loss for a NEWS sample. This is the case for all domains. One of the properties that distinguishes REVIEWS from all other domains is that its documents are significantly shorter. In general, we find that cross-DAPT masked-LM loss can in some cases be a noisy predictor of domain similarity.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
In       Table 18: Mean development set macro-F 1 (for HYPERPARTISAN and IMDB) and micro-F 1 (for RCT-500), with Curated-TAPT across five random seeds, with standard deviations as subscripts. † indicates high-resource settings. Table 19: Mean development set macro-F 1 (for HYP. and IMDB) and micro-F 1 (for RCT), across five random seeds, with standard deviations as subscripts, comparing RAND-TAPT (with 50 candidates) and kNN-TAPT selection. Neighbors of the task data are selected from the domain data.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Spooks is enjoyable trash, featuring some well directed sequences, ridiculous plots and dialogue, and some third rate acting. Many have described this is a UK version of "24", and one can see the similarities. The American version shares the weak silly plots, but the execution is so much slicker, sexier and I suspect, expensive. Some people describe weak comedy as "gentle comedy". This is gentle spy story hour, the exact opposite of anything created by John Le Carre. Give me Smiley any day.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
[ The Sopranos is perhaps the most mind-opening series you could possibly ever want to watch. It's smart, it's quirky, it's funny -and it carries the mafia genre so well that most people can't resist watching.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The best aspect of this show is the overwhelming realism of the characters, set in the subterranean world of the New York crime families. For most of the time, you really don't know whether the wise guys will stab someone in the back, or buy them lunch. Further adding to the realistic approach of the characters in this show is the depth of their personalities -These are dangerous men, most of them murderers, but by God if you don't love them too. I've laughed at their wisecracks, been torn when they've made err in judgement, and felt scared at the sheer ruthlessness of a serious criminal. [...] The drumbeat regarding the "Breaking Bad" finale has led to the inevitable speculation on whether the final chapter in this serialized gem will live up to the hype or disappoint (thank you, "Dexter," for setting that bar pretty low), with debate, second-guessing and graduate-thesis-length analysis sure to follow. The Most Memorable TV Series Finales of All-Time [...] No ending in recent years has been more divisive than "The Sopranos" -for some, a brilliant flash (literally, in a way) of genius; for others (including yours truly), a too-cute copout, cryptically leaving its characters in perpetual limbo. The precedent to that would be "St.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Elsewhere," which irked many with its provocative, surreal notion that the whole series was, in fact, conjured in the mind of an autistic child.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
[...]
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The Wicker Man, starring Nicolas Cage, is by no means a good movie, but I can't really say it's one I regret watching. I could go on and on about the negative aspects of the movie, like the terrible acting and the lengthy scenes where Cage is looking for the girl, has a hallucination, followed by another hallucination, followed by a dream sequence-with a hallucination, etc., but it's just not worth dwelling on when it comes to a movie like this. Instead, here's five reasons why you SHOULD watch The Wicker Man, even though it's bad: 5. It's hard to deny that it has some genuinely creepy ideas to it, the only problem is in its cheesy, unintentionally funny execution. If nothing else, this is a movie that may inspire you to see the original 1973 film, or even read the short story on which it is based. 4. For a cheesy horror/thriller, it is really aesthetically pleasing. [...] NOTE: The Unrated version of the movie is the best to watch, and it's better to watch the Theatrical version just for its little added on epilogue, which features a cameo from James Franco.
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
[...] What did you ultimately feel about "The Wicker Man" movie when all was said and done? [...] I'm a fan of the original and I'm glad that I made the movie because they don't make movies like that anymore and probably the result of what "Wicker Man" did is the reason why they don't make movies like that anymore. Again, it's kind of that '70's sensibility, but I'm trying to do things that are outside the box. Sometimes that means it'll work and other times it won't. Again though I'm going to try and learn from anything that I do. I think that it was a great cast, and Neil La Bute is one of the easiest directors that I've ever worked with. He really loves actors and he really gives you a relaxed feeling on the set, that you can achieve whatever it is that you're trying to put together, but at the end of the day the frustration that I had with 'The Wicker Man,' which I think has been remedied on the DVD because I believe the DVD has the directors original cut, is that they cut the horror out of the horror film to try and get a PG-13 rating. I mean, I don't know how to stop something like that. So I'm not happy with the way that the picture ended, but I'm happy with the spirit with which it was made. [...] Dr. Seuss would sure be mad right now if he was alive. Cat in the Hat proves to show how movie productions can take a classic story and turn it into a mindless pile of goop. We have Mike Myers as the infamous Cat in the Hat, big mistake! Myers proves he can't act in this film. He acts like a prissy show girl with a thousand tricks up his sleeve. The kids in this movie are all right, somewhere in between the lines of dull and annoying. The story is just like the original with a couple of tweaks and like most movies based on other stories, never tweak with the original story! Bringing in the evil neighbor Quin was a bad idea. He is a stupid villain that would never get anywhere in life. [...] The Cat in the Hat, [...] Based on the book by Dr. Seuss [...] From the moment his tall, red-and-white-striped hat appears at their door, Sally and her brother know that the Cat in the Hat is the most mischievous cat they will ever meet. Suddenly the rainy afternoon is transformed by the Cat and his antics. Will their house ever be the same? Can the kids clean up before mom comes home? With some tricks (and a fish) and Thing Two and Thing One, with the Cat in The Hat, the fun's never done!Dr. Seuss is known worldwide as the imaginative master of children's literature. His books include a wonderful blend of invented and actual words, and his rhymes have helped many children and adults learn and better their understanding of the English language. [...]  Neighbor 0 Of this group, 26% died after discharge from hospital, and the median time to death was 11 days (interquartile range, 4.0-15.0 days) after discharge. Neighbor 1
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The median hospital stay was 17 days (range 8-26 days), and all the patients were discharged within 1 month. Neighbor 2
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The median hospital stay was 17 days (range 8-26 days). Neighbor 3
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The median time between discharge and death was 25 days (mean, 59.1 days) and no patient was alive after 193 days. Neighbor 4
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
The length of hospital stay after colostomy formation ranged from 3 days to 14 days with a median duration of 6 days (+IQR of 4 to 8 days).
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Randomized , controlled , parallel clinical trial .
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Neighbor 0 Design: Unblinded, randomised clinical controlled trial. Neighbor 1
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
These studies and others led to the phase III randomized trial RTOG 0617/NCCTG 0628/ CALGB 30609. Neighbor 2 -Definitive randomized controlled clinical trial (RCT): Neighbor 3 RCT 1 4 randomized controlled trial. Neighbor 4 randomized controlled trial [ Fig. 3(A)].
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Forty primary molar teeth in 40 healthy children aged 5-9 years were treated by direct pulp capping .
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Neighbor 0 In our study, we specifically determined the usefulness of the Er:YAG laser in caries removal and cavity preparation of primary and young permanent teeth in children ages 4 to 1 8 years. Neighbor 1
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Males watched more TV than females, although it was only in primary school-aged children and on weekdays. Neighbor 2 Assent was obtained from children and adolescents aged 7-17 years. Neighbor 3
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Cardiopulmonary resuscitation was not applied to children aged ¡5 years (Table 2). Neighbor 4
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
It measures HRQoL in children and adolescents aged 2 to 25 years. 
Muppet: Massive Multi-task Representations with Pre-Finetuning
The recent success of language model pre-training (Devlin et al., 2018;Liu et al., 2019b;Lewis et al., 2019;Raffel et al., 2019;Radford et al., 2019) is remarkable, at least in part, due to the exclusive use of self supervision, without any manually labeled data. For many tasks, however, we already have training examples for related problems, which we should be able to leverage. Recent work has shown gains from fine-tuning schemes that are multi-task (Raffel et al., 2019;Khashabi et al., 2020) and multi-stage (Liu et al., 2019a), but it can be difficult to know which intermediate tasks will best transfer (Raffel et al., 2019). In this paper, we show that multi-task supervised tuning, if done at a sufficiently large scale with many different tasks, can be an effective second stage of task-agnostic pre-training, removing the need to pre-select the best intermediate tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
More specifically, in addition to the standard pre-training/fine-tuning methodology of learning language tasks, we introduce a new intermediate stage, pre-finetuning. Pre-finetuning involves a massive multi-task learning step (4.8 million total training examples) performed on around 50 classification, summarization, question answering, and common sense reasoning tasks. We believe we are the first to investigate multi-task learning at this scale in terms of both number and types of tasks. We show, in particular, that standard multi-tasking schemes can be unstable and often fail to learn high quality representations. However, we introduce a new training scheme which uses loss scaling and task-heterogeneous batches so that gradient steps are more evenly balanced across multiple different competing tasks, greatly improving training stability and overall performance. We call our prefinetuned models MUPPET; Massive Multi-task RePresentation with PrE-fineTuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Through extensive experiments, we show that incorporating pre-finetuning to RoBERTa (Liu et al., 2019b) and BART (Lewis et al., 2019) models yields consistent improvements, including new state-of-the-art performance for RTE  and HellaSWAG , without having to specify specific intermediate transfer tasks. These gains are particularly strong in the low resource regime, where there is relatively little labeled data for fine-tuning. We also study why pre-finetuning outperforms previous multitasking schemes. We first compare different optimization techniques to stabilize training, and find it important to use task-heterogeneous batches with task-rebalancing loss scaling. We also show that scale is crucial for effective multi-task learning. We empirically see a critical point in terms of the number of tasks (usually over 15); having fewer tasks degrades representations, while having more seems to improve performance linearly as far as we were able to scale.
Muppet: Massive Multi-task Representations with Pre-Finetuning
To summarize, our contributions include:
Muppet: Massive Multi-task Representations with Pre-Finetuning
• We show that we can further improve pre-trained representations with an additional stage we call pre-finetuning, which utilizes massively multi-task learning. We show standard pre-trained representations, when further refined with pre-finetuning consistently improve performance on downstream tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
• We introduce a new multi-task training scheme for effective learning at scale, which uses loss scaling and task-heterogeneous batches.
Muppet: Massive Multi-task Representations with Pre-Finetuning
• We explore the effects of scale on multi-task learning and show the existence of critical points in multi-task training, beyond which increasing the number of tasks improves generalizable representations.
Muppet: Massive Multi-task Representations with Pre-Finetuning
• We conduct a study surrounding the data efficiency of standard pre-trained representations and their respective pre-finetuned counterparts. We show that the pre-finetuned models consistently require less data for fine-tuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Previous work has reported mixed results from experiments on multi-task learning (Liu et al., 2019a;Raffel et al., 2019). In general, it can be challenging to balance the losses from different tasks; upsampling can lead to overfitting low resource tasks, and downsampling can lead to improper learning of specific tasks. This difficulty is particularly pronounced when operating at the scale of experiments we show in Section 5.1, where there are more diverse tasks than previously considered. This section presents our pre-finetuning approach that leads to more stable and accurate multi-task training by introducing new optimization, loss scaling, and task sampling schemes to balance each minibatch's updates better.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Diverse Tasks To learn general language representations, we include a variety of tasks across many domains. We select language tasks across four different domains: classification, commonsense reasoning, machine reading comprehension, and summarization. In Table 1, we show the break down of each of the task types along with the number of samples used from each during prefinetuning. In total our multi-task set up learns over 4.8 supervised samples across 4 families of tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
A full list of all of the datasets we leverage for pre-finetuning is described in appendix §A.1.  Standard Losses To train on several datasets, our model contains task-specific heads, each optimizing for a task-specific loss. The loss functions are summarized in table 2. Each loss is scaled with loss scaling described in §3.3. After loss scaling, the gradients from each task are averaged before doing the model update step.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We show two strategies to learn multi-task representations at scale: Accumulating Gradients Across Tasks (Heterogeneous Batches) and Leveraging Better Finetuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Accumulating Gradients Across Tasks Our model is trying to optimize not a single objective but several potentially competing objectives to create a unified representation across several tasks during model training. During gradient descent, moving along the gradient of a single task may not be the optimal direction for the model to move to learn a single unified representation across tasks. To overcome this, we ensure each batch our model optimizes consists of several tasks. Each worker samples a random batch from our set of tasks and computes a gradient, accumulated for the final update. Empirically we use 64 GPUs for pre-finetuning, resulting in each batch consisting of gradients across 64 sampled tasks. In §5.2 we show how such a strategy allows for our model to arrive at a better representation for end task finetuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Better Finetuning Instead of starting from scratch, we initialize our model with representations learned from self-supervised pre-training in pre-finetuning. This can inherit the knowledge captured in the pre-trained representations and speed up training. Mosbach et al. (2020) show that standard fine-tuning of pre-trained models can be unstable, which may be aggravated in our case as we are training on a diverse set of tasks simultaneously. Therefore, we employ the R3F/R4F methods (Aghajanyan et al., 2020) to combat this issue.
Muppet: Massive Multi-task Representations with Pre-Finetuning
In particular, R3F/R4F consists of an additional loss term, ensuring that small perturbations to the input space result in similar representations, which can be used to learn more robust representations during pre-finetuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
In early experimentation, we found that R3F was pivotal in getting MUPPET to work for BART. All other fine-tuning and pre-finetuning was done using standard SGD.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Loss scaling methods introduce a multiplicative reweighting of individual losses per data-point. Various loss scaling techniques have been proposed, from dynamic scaling by inverse training loss to simple scaling by the number of data-points in respective datasets (Chen et al., 2018).
Muppet: Massive Multi-task Representations with Pre-Finetuning
As pre-finetuning optimizes several different types of tasks and datasets, each having its own output spaces, loss scaling becomes essential to ensure stable training. We attempted various forms of loss-scaling throughout initial experimentation, but the most effective was the novel method we describe below.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Let us denote L i (x i , y i ; θ) as the loss for datapoint i for a model parameterized by θ. Remember that the loss depends on the type of task (commonsense loss is different from binary classification). Furthermore let n : N → N be a function which for each data-point returns the number of predictions L operates over. For example, for binary classification, n would return two, while for generation, n would return the size of the vocabulary (since we average across loss per token generated). We scale data-point loss so that, if the class distribution were uniformly distributed along with our models predictions, all of our losses would have equivalent values.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We found that this static scaling worked incredibly well, outperforming other loss scaling methods in early experimentation.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Another approach to balancing various tasks in a multi-task set up is to up-sample smaller datasets and down-sample larger ones to achieve more uniformity between dataset sizes.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Existing results for dataset sampling methods in multi-task learning are conflicting, but recent work has shown that it does not work well for multitask learning of pre-trained representations. For example, T5 showed that all various forms of sampling did not improve overusing the natural size of datasets (Raffel et al., 2019).
Muppet: Massive Multi-task Representations with Pre-Finetuning
We also found that sampling datasets were consistently detrimental for multi-task learning over pre-trained representations during initial experimentation. Specifically, we saw unmanageable over-fitting and stability issues. Therefore we opt for maintaining the natural distribution of the datasets throughout all of our experiments.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We selected RoBERTa (Liu et al., 2019b) and BART (Lewis et al., 2019) as our initial pre-trained models to further pre-finetune. For each task type we use a different prediction scheme. Every Sentence Prediction dataset gets a separate classification head, for Commonsense and MRC we utilize a separate unified head for each task. For Summarization, we do not add any parameters and use the BART decoder and output layer as is. Experimentally we saw using a different head per individual Commonsense and MRC datasets lead to severe overfitting.
Muppet: Massive Multi-task Representations with Pre-Finetuning
For both models, we do the pre-finetuning procedure for both the Base and Large models. We trained each model configuration with 64 GPUs until convergence. Dependent on configuration, this ranged from a day to 4 days. We include the hyper-parameters used per pre-finetuning run in the Appendix in Section §A.2.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We first show that pre-finetuning improves the representations of pre-training models. To do so, we fine-tune our pre-finetuned models on a large set of tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
For each of the individual downstream tasks, we use a fixed hyper-parameter search to optimize over simple hyperparameters such as learning rate, Adam (Kingma and Ba, 2014) and dropout (Srivastava et al., 2014). We present our results in two tables. Table 3 shows our results on the GLUE benchmark (Wang et al., 2018) as well as two MRC tasks; SQuAD  and ReCoRD . Table 4 reports results on other Sentence Prediction tasks as well as Commonsense tasks. We also include results from MT-DNN (Liu et al., 2019a), ELECTRA , 1 and RoBERTa (Liu et al., 2019b) models. For Summarization tasks we show that our pre-finetuned BART model outperforms all other summarization baselines. Both of these tables report over data-sets available during the prefinetuning stage.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Given that our pre-finetuned models now have an understanding of the task at hand through the use of classification heads, we have a choice during finetuning on whether or not to use these heads. In general we found re-using heads to be beneficial for MRC, Commonsense and Sentence Prediction tasks with small dataset size.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Across the board, pre-trained representations that were further refined with pre-finetuning outperformed standard pre-trained representations. We see more modest gains on larger datasets, most likely because we do not need to refine representations beforehand if the fine-tuning dataset is large. On smaller datasets, we see substantial gains. For example, the pre-finetuned RoBERTa-BASE model on RTE improves by close to 9 points, rivaling the RoBERTa-Large accuracy, while the pre-finetuned RoBERTa-Large model gets new state-of-the-art on RTE rivaling models an order of magnitude larger than it.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We do not improve just over sentence prediction tasks but on every set of tasks that we measured. For example, we reach a new state of the art on the HellaSwag dataset previously achieved by utilizing a new fine-tuning approach. Our methods do not increase parameter count or any complexity measures but are quite successful at refining features and preparing them for downstream fine-tuning.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We also report the performance on tasks not included in the pre-finetuning data. To do so, we finetune our models on a set of tasks including  (3) Chunking, Constituency Parsing and Part-Of-Speech tagging for structured prediction from the Penn Treebank dataset (Marcus et al., 1993). We present these results in Table 5 and Table 6.   We see that the MUPPET variants of our models out-perform the baselines consistently across task type and dataset. As a special case we do an in depth analysis of the MUPPET variant of RoBERTa on the notoriously tough ANLI dataset and see the same pattern. Pre-finetuned models consistently outperform their base counterparts.
Muppet: Massive Multi-task Representations with Pre-Finetuning
The first axis we would like to explore is the scale on which multi-task learning is done. Previous work, such as T5 and MT-DNN, focused on the MTL scale of around a dozen datasets. To the best of our knowledge, our paper has the largest MTL set up to date. Accordingly, we are interested in empirically exploring the effects of scaling up the number of datasets to the representations learned during MTL.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We pre-finetune a collection of RoBERTa-Base models with varying numbers of datasets. We train seven models, six uniformly chosen between 10 and 40, ensuring that at each point, the selected datasets are a superset of the datasets from prior points. The last model is fully trained on all datasets. Concretely given two models trained with a different number of datasets a, b : a > b, model a will contain all datasets used to train model b and more.
Muppet: Massive Multi-task Representations with Pre-Finetuning
For each version of the model, we fine-tune five datasets and plot the results in Figure 1. Specifically we finetune STS-B (Cer et al., 2017) Table 6: We show the performance of the RoBERTa model and the pre-finetuned RoBERTa-MUPPET model on the ANLI benchmark. Bolded numbers signify MUPPET vs base model, underline signifies best number. 'S' refers to SNLI, 'M' to MNLI dev (-m=matched, -mm=mismatched), and 'F' to FEVER; 'A1-A3' refer to the rounds respectively and 'ANLI' refers to A1+A2+A3. , RACE , SQuAD , and MNLI . We include these five datasets in the first MTL run (10 datasets) to remove any bias from adding them in a later stage. We see a couple of interesting patterns. First, for individual tasks such as RTE , increasing the pre-finetuning scale monotonically improves performance. This is aligned with other papers that have seen benefits from first training on MNLI  and then fine-tuning on RTE (Liu et al., 2019b). For other datasets, we see that doing MTL in the < 15 datasets regime is detrimental for end-task finetuning. This is also aligned with other empirical observations, i.e., T5 reported that doing MTL did not improve over only fine-tuning. Nevertheless, it seems that as we increase the number of tasks past some critical point, our pre-trained representations become more generalizable. Furthermore, al-though dependent on the dataset, this critical point is roughly between 10 and 25 tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
This suggests that previously observed MTL limitations were not fundamental and can instead be attributed to the lack of sufficient scale.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Another critical factor to getting MTL to learn generalizable representations is the method through which MTL is implemented, specifically the selection of batches. To better quantify this trend, we experimented with three balancing schemes: dataset homogenous, batch homogenous and batch heterogenous.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We refer to dataset homogenous as selecting batches from datasets sequentially. So we first train on dataset A, then train on dataset B, etc. On the other hand, batch homogenous refers to selecting batches containing only data from the same task; therefore, all gradients are from the same dataset. This is implemented by selecting all datasets, batching on a dataset level, and selecting those same batches randomly during training. Finally, batch heterogeneous refers to a single update containing a batch from multiple different datasets spanning different tasks. We implemented this by first creating homogenous sub-batches, calculating loss per sub-batch per GPU, and then aggregating across GPUs manifesting in a gradient update that contains various datasets and, therefore, tasks.
Muppet: Massive Multi-task Representations with Pre-Finetuning
To dissect the importance of heterogeneous batches, we train a RoBERTa-Base model on 35 randomly selected tasks using the three data selection methodologies outlined above. We then finetune these three models on the same five data-sets mentioned in the previous section.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We present our results in Figure 2. We see the importance of properly defining a batching strategy for effective multi-task learning. Our findings are also consistent with (Aghajanyan et al., 2020) which saw that sequential training of data-sets degrades generalizable representations.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We noticed in Section §4 that data-sets with smaller data-set sizes tended to improve more from MTL training. To strengthen this hypothesis, we look at two factors: the scale of pre-finetuning and the scale of fine-tuning (size of fine-tuning data-set).
Muppet: Massive Multi-task Representations with Pre-Finetuning
We select three data-sets that were not used in pre-finetuning in Section §5.1. We also select nine partitions per fine-tuning data-set, which is sampled uniformly between 10% of the data-set and 100% of the data-set. Selecting the low-resource splits was done through random sampling.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We then fine-tune every low-resource split with every pre-finetuning checkpoint from Section §5.1. We plot the heatmaps generated from these runs in Figure 3. Multiple patterns emerge. First, we see a clear visualization of the critical point mentioned when doing pre-finetuning. As we increase the scale of MTL, better representations are available for downstream finetuning. Furthermore, we see that prefinetuned models at a larger scale are much more data-efficient than standard pre-trained models.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Specifically looking at the 34/40 pre-finetuning scale on Figure 3 we see that we reach higher evaluation accuracies much sooner than the base RoBERTa model (row 0).
Muppet: Massive Multi-task Representations with Pre-Finetuning
In this work, we propose pre-finetuning, a stage after pre-training to further refine representations before end-task finetuning. We show that we can effectively learn more robust representations through multi-task learning (MTL) at scale. Our MTL models outperform their vanilla pre-trained counterparts across several tasks. Our analysis shows that properly scaling MTL with heterogeneous batches and loss scaling is critical to leveraging better representations. We also show a critical point regarding the number of tasks when doing multi-task learning, where fewer tasks degrade representations compared to the pre-trained model, but more tasks than this point improve representations.
Muppet: Massive Multi-task Representations with Pre-Finetuning
We discussed a practical setting in which doing this massive multi-task learning is stable and effective through simple loss scaling and heterogeneous batches. With our method, we improve upon prior state of the art methods for RTE    Figure 3: We fine-tune every low-resource split with every pre-finetuning checkpoint from Section §5.1 for two datasets not available in any of the pre-finetuning MTL datasets; QNLI  and CoLA . The pre-finetuning scale is reported in terms of the number of datasets.
Muppet: Massive Multi-task Representations with Pre-Finetuning
2019), as well as improve upon vanilla pre-trained representations for MNLI , SQuAD , BoolQ , and Common Sense QA . We also our MTL model performance with low resource experiments. We show that on heldout datasets, leveraging representations from our pre-finetuned models with 34-40 tasks, we reach higher evaluation accuracies with much less data than the RoBERTa model.  
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
Finetuned Language Models Are Zero-Shot Learners
We take a 137B parameter pretrained language model and instruction tune it on over 60 NLP datasets verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 datasets that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.
Finetuned Language Models Are Zero-Shot Learners
keep stack of pillow cases in fridge Inference on unseen task type Finetune on many tasks ("instruction-tuning") … Translate this sentence to Spanish:
Finetuned Language Models Are Zero-Shot Learners
The new office building was built in less than three months.
Finetuned Language Models Are Zero-Shot Learners
El nuevo edificio de oficinas se construyó en tres meses.
Finetuned Language Models Are Zero-Shot Learners
It is not possible to tell Here is a goal: Get a cool sleep on summer days. How would you accomplish this goal? OPTIONS: -Keep stack of pillow cases in fridge.
Finetuned Language Models Are Zero-Shot Learners
-Keep stack of pillow cases in oven.
Finetuned Language Models Are Zero-Shot Learners
Premise: At my age you will probably have learnt one lesson.
Finetuned Language Models Are Zero-Shot Learners
Hypothesis: It's not certain how many lessons you'll learn by your thirties.
Finetuned Language Models Are Zero-Shot Learners
Does the premise entail the hypothesis?
Finetuned Language Models Are Zero-Shot Learners
OPTIONS: -yes -it is not possible to tell -no Figure 1: Top: overview of instruction tuning and FLAN. Instruction tuning finetunes a pretrained language model on a mixture of tasks phrased as instructions. At inference time, we evaluate on an unseen task type; for instance, we could evaluate the model on natural language inference (NLI) when no NLI tasks were seen during instruction tuning. Bottom: performance of zero-shot FLAN, compared with zero-shot and few-shot GPT-3, on three unseen task types where instruction tuning improved performance substantially out of ten we evaluate. NLI datasets: ANLI R1-R3, CB, RTE. Reading comprehension datasets: BoolQ, MultiRC, OBQA. Closed-book QA datasets: ARC-easy, ARC-challenge, NQ, TriviaQA.
Finetuned Language Models Are Zero-Shot Learners
Language models (LMs) at scale, such as GPT-3 (Brown et al., 2020), have been shown to perform few-shot learning remarkably well. They are less successful at zero-shot learning, however. For example, GPT-3's zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference. One potential reason is that, without few-shot exemplars, it is harder for models to perform well on prompts that are not similar to the format of the pretraining data.
Finetuned Language Models Are Zero-Shot Learners
In this paper, we explore a simple method to improve the zero-shot performance of large language models, which would expand their reach to a broader audience. We leverage the intuition that NLP tasks can be described via natural language instructions, such as "Is the sentiment of this movie review positive or negative?" or "Translate 'how are you' into Chinese." We take a pretrained language model of 137B parameters and perform instruction tuning-finetuning the model on a mixture of more than 60 NLP datasets expressed via natural language instructions. We refer to this resulting model as FLAN, for Finetuned Language Net.
Finetuned Language Models Are Zero-Shot Learners
To evaluate the zero-shot performance of FLAN on unseen tasks, we group NLP datasets into clusters based on their task types and hold out each cluster for evaluation while instruction tuning FLAN on all other clusters. For example, as shown in Figure 1, to evaluate FLAN's ability to perform natural language inference, we instruction tune the model on a range of other NLP tasks such as commonsense reasoning, translation, and sentiment analysis. As this setup ensures that FLAN has not seen any natural language inference tasks in instruction tuning, we then evaluate its ability to perform zero-shot natural language inference.
Finetuned Language Models Are Zero-Shot Learners
Our evaluations show that FLAN substantially improves the zero-shot performance of the base 137B-parameter model. FLAN's zero-shot also outperforms 175B-parameter GPT-3's zero-shot on 20 of 25 datasets that we evaluate, and even outperforms GPT-3's few-shot by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. In ablation studies, we find that increasing the number of task clusters in instruction tuning improves performance on unseen tasks and that the benefits of instruction tuning emerge only with sufficient model scale.
Finetuned Language Models Are Zero-Shot Learners
Instruction tuning is a simple method that, as depicted in Figure 2, combines appealing aspects of both the pretrain-finetune and prompting paradigms by using supervision via finetuning to improve language model's responses to inference-time text interactions. Our empirical results demonstrate promising abilities of language models to perform tasks described purely via instructions. Source code for loading the instruction tuning dataset used for FLAN is publicly available at https://github.com/google-research/flan. 
Finetuned Language Models Are Zero-Shot Learners
Improve performance via few-shot prompting or prompt engineering
Finetuned Language Models Are Zero-Shot Learners
Instruction-tune on many tasks: B, C, D, …
Finetuned Language Models Are Zero-Shot Learners
Model learns to perform many tasks via natural language instructions Figure 2: Comparing instruction tuning with pretrain-finetune and prompting.
Finetuned Language Models Are Zero-Shot Learners
The motivation of instruction tuning is to improve the ability of language models to respond to NLP instructions. The idea is that by using supervision to teach an LM to perform tasks described via instructions, the LM will learn to follow instructions and do so even for unseen tasks. To evaluate performance on unseen tasks, we group datasets into clusters by task type and hold out each task cluster for evaluation while instruction tuning on all remaining clusters.
Finetuned Language Models Are Zero-Shot Learners
As creating an instruction tuning dataset with many tasks from scratch would be resource-intensive, we transform existing datasets from the research community into an instructional format. We aggregate 62 text datasets that are publicly available on Tensorflow Datasets, including both language understanding and language generation tasks, into a single mixture. Figure 3 shows these datasetseach dataset is categorized into one of twelve task clusters, for which datasets in a given cluster are of the same task type. Descriptions, sizes, and examples of each dataset are shown in Appendix G.
Finetuned Language Models Are Zero-Shot Learners
Natural language inference ( For each dataset, we manually compose ten unique templates that use natural language instructions to describe the task for that dataset. While most of the ten templates describe the original task, to increase diversity, for each dataset we also include up to three templates that "turned the task around," (e.g., for sentiment classification we include templates asking to generate a movie review). We then instruction tune a pretrained language model on the mixture of all datasets, with examples in each dataset formatted via a randomly selected instruction template for that dataset. Figure 4 shows multiple instruction templates for a natural language inference dataset.
Finetuned Language Models Are Zero-Shot Learners
Russian cosmonaut Valery Polyakov set the record for the longest continuous amount of time spent in space, a staggering 438 days, between 1994 and 1995.
Finetuned Language Models Are Zero-Shot Learners
Russians hold the record for the longest stay in space.  Figure 4: Multiple instruction templates describing a natural language inference task.
Finetuned Language Models Are Zero-Shot Learners
We are interested in how FLAN performs on tasks not seen in instruction tuning, and so it is crucial to define what counts as an unseen task. Whereas some prior work defines unseen tasks by disallowing the same dataset to appear in training, we use a more conservative definition that leverages the task clusters from Figure 3. In this work, we only consider dataset D unseen at evaluation time if no datasets from any task clusters that D belongs to were seen during instruction tuning. For instance, if D is an entailment task, then no entailment datasets appeared in instruction tuning, and we instruction-tuned on all other clusters. 1 Hence, to evaluate zero-shot FLAN on c task clusters, we instruction tune c models, where each model holds out a different task cluster for evaluation.
Finetuned Language Models Are Zero-Shot Learners
The output space for a given task is either one of several classes (classification) or free text (generation).
Finetuned Language Models Are Zero-Shot Learners
As FLAN is an instruction-tuned version of a decoder-only language model, it naturally responds in free text, and so no further modifications are needed for generation tasks.
Finetuned Language Models Are Zero-Shot Learners
For classification tasks, prior work (Brown et al., 2020) used a rank classification approach where, for example, only two outputs ("yes" and "no") are considered and the higher probability one is taken as the model's prediction. Though this procedure is logically sound, it is imperfect in that the probability mass for answers may have an undesired distribution among ways of saying each answer (e.g., a large number of alternative ways of saying "yes" may lower the probability mass assigned to "yes"). Therefore, we include an options suffix, in which we append the token OPTIONS to the end of a classification task along with a list of the output classes for that task. This makes the model aware of which choices are desired when responding to classification tasks. Example use of options is shown in the NLI and commonsense examples in Figure 1.
Finetuned Language Models Are Zero-Shot Learners
Model architecture and pretraining. In our experiments, we use LaMDA-PT, a dense left-to-right, decoder-only transformer language model of 137B parameters (Thoppilan et al., 2022). This model is pretrained on a collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with a 32k vocabulary using the SentencePiece library (Kudo & Richardson, 2018). Around 10% of the pretraining data was non-English. Note that LaMDA-PT only has language model pretraining (c.f. LaMDA, which was finetuned for dialog).
Finetuned Language Models Are Zero-Shot Learners
Instruction tuning procedure. FLAN is the instruction-tuned version of LaMDA-PT. Our instruction tuning pipeline mixes all datasets and randomly samples from each dataset. To balance the different sizes of datasets, we limit the number of training examples per dataset to 30k and follow the examples-proportional mixing scheme (Raffel et al., 2020) with a mixing rate maximum of 3k. 2 We finetune all models for 30k gradient steps with a batch size of 8,192 tokens using the Adafactor Optimizer (Shazeer & Stern, 2018) with a learning rate of 3e-5. The input and target sequence lengths used in finetuning are 1024 and 256, respectively. We use packing (Raffel et al., 2020) to combine multiple training examples into a single sequence, separating inputs from targets using a special EOS token. This instruction tuning takes around 60 hours on a TPUv3 with 128 cores. For all evaluations, we report results on the final checkpoint trained for 30k steps.
Finetuned Language Models Are Zero-Shot Learners
We evaluate FLAN on natural language inference, reading comprehension, closed-book QA, translation, commonsense reasoning, coreference resolution, and struct-to-text. As described in §2.2, we evaluate on unseen tasks by grouping datasets into task clusters and holding out each cluster for evaluation while instruction tuning on all remaining clusters (i.e., each evaluation task cluster uses a different checkpoint). For each dataset, we evaluate the mean of performance on all templates, which proxies the expected performance given a typical natural language instruction. As a dev set is sometimes available for manual prompt engineering (Brown et al., 2020), for each dataset we also obtain the test set performance using the template with the best dev set performance.
Finetuned Language Models Are Zero-Shot Learners
For comparison, we report zero and few-shot results for LaMDA-PT using the same prompts as GPT-3 (as LaMDA-PT is not suitable for natural instructions without instruction tuning). This baseline provides the most direct ablation of how much instruction tuning helps. Instruction tuning significantly improves LaMDA-PT on most datasets.
Finetuned Language Models Are Zero-Shot Learners
We also show the zero-shot performances of GPT-3 175B (Brown et al., 2020) and GLaM 64B/64E (Du et al., 2021), as reported in their respective papers. With the best dev template, zero-shot FLAN outperforms zero-shot GPT-3 on 20 of 25 datasets and even surpasses GPT-3's few-shot performance on 10 datasets. With the best dev-template, zero-shot FLAN outperforms zero-shot GLaM on 13 of 19 available datasets and one-shot GLaM on 11 of 19 datasets.
Finetuned Language Models Are Zero-Shot Learners
Overall, we observe that instruction tuning is very effective on tasks naturally verbalized as instructions (e.g., NLI, QA, translation, struct-to-text) and is less effective on tasks directly formulated as language modeling, where instructions would be largely redundant (e.g., commonsense reasoning and coreference resolution tasks that are formatted as finishing an incomplete sentence or paragraph). Results on natural language inference, reading comprehension, closed-book QA, and translation are summarized in Figure 5 and described below.
Finetuned Language Models Are Zero-Shot Learners
Reading comprehension  Figure 5: Zero-shot performance of FLAN compared to LaMDA-PT 137B, GPT-3 175B, and GLaM 64B/64E on natural language inference, reading comprehension, closed-book QA, and translation. Performance of FLAN is the mean of up to 10 instructional templates per task. Supervised models were either T5, BERT, or translation models (specified in Table 2 and Table 1 in the Appendix).
Finetuned Language Models Are Zero-Shot Learners
Natural language inference (NLI). On five NLI datasets, where a model must determine whether a hypothesis is true given some premise, FLAN outperforms all baselines by a large margin. As noted by Brown et al. (2020), perhaps one reason why GPT-3 struggles with NLI is that NLI examples are unlikely to have appeared naturally in an unsupervised training set and are thus awkwardly phrased as a continuation of a sentence. For FLAN, we phrase NLI as the more natural question "Does <premise> mean that <hypothesis>?", achieving much higher performance.
Finetuned Language Models Are Zero-Shot Learners
Reading comprehension. On reading comprehension, where models are asked to answer a question about a provided passage, FLAN outperforms baselines for MultiRC (Khashabi et al., 2018) and OBQA (Mihaylov et al., 2018). On BoolQ , FLAN outperforms GPT-3 by a large margin, though LaMDA-PT already achieves high performance on BoolQ.
Finetuned Language Models Are Zero-Shot Learners
Closed-book QA. For closed-book QA, which asks models to answer questions about the world without access to specific information containing the answer, FLAN outperforms GPT-3 on all four datasets. Compared to GLaM, FLAN has better performance on ARC-e and ARC-c , and slightly lower performance on NQ (Lee et al., 2019;Kwiatkowski et al., 2019) and TQA (Joshi et al., 2017).
Finetuned Language Models Are Zero-Shot Learners
Translation. Similar to GPT-3, the training data for LaMDA-PT is around 90% English and includes some text in other languages that was not specifically used to train the model to perform machine translation. We also evaluate FLAN's performance on machine translation for the three datasets evaluated in the GPT-3 paper: French-English from WMT'14 , and German-English and Romanian-English from WMT'16 . Compared with GPT-3, FLAN outperforms zero-shot GPT-3 for all six evaluations, though it underperforms few-shot GPT-3 in most cases. Similar to GPT-3, FLAN shows strong results for translating into English and compares favorably against supervised translation baselines. Translating from English into other languages, however, was relatively weaker, as might be expected given that FLAN uses an English sentencepiece tokenizer and that the majority of pretraining data is English.
Finetuned Language Models Are Zero-Shot Learners
Additional tasks. Although we see strong results for the above task clusters, one limitation with instruction tuning is that it does not improve performance for many language modeling tasks (e.g., commonsense reasoning or coreference resolution tasks formulated as sentence completions). For seven commonsense reasoning and coreference resolution tasks (see Table 2 in the Appendix), FLAN only outperforms LaMDA-PT on three of the seven tasks. This negative result indicates that when the downstream task is the same as the original language modeling pre-training objective (i.e., in cases where instructions are largely redundant), instruction tuning is not useful. Finally, we report results for sentiment analysis, paraphrase detection, and struct-to-text, as well as additional datasets for which GPT-3 results are not available, in Table 2 and Table 1 in the Appendix. Generally, zero-shot FLAN outperforms zero-shot LaMDA-PT and is comparable with or better than few-shot LaMDA-PT.
Finetuned Language Models Are Zero-Shot Learners
As the core question of our paper asks how instruction tuning improves a model's zero-shot performance on unseen tasks, in this first ablation we examine how performance is affected by the number of clusters and tasks used in instruction tuning. For this setup, we hold out NLI, closed-book QA, and commonsense reasoning as evaluation clusters, and use the seven remaining clusters for instruction tuning. 3 We show results for one to seven instruction tuning clusters, where clusters are added in decreasing order of number of tasks per cluster. Figure 6 shows these results. As expected, we observe that average performance across the three held-out clusters improves as we add additional clusters and tasks to instruction tuning (with the exception of the sentiment analysis cluster), confirming the benefits of our proposed instruction tuning approach on zero-shot performance on novel tasks. It is further interesting to see that, for the seven clusters we test, the performance does not appear to saturate, implying that performance may further improve with even more clusters added to instruction tuning. Of note, this ablation does not allow us to draw conclusions about which instruction tuning cluster contributes the most to each evaluation cluster, although we see minimal added value from the sentiment analysis cluster.  Performance on held-out tasks Figure 7: Whereas instruction tuning helps large models generalize to new tasks, for small models it actually hurts generalization to unseen tasks, potentially because all model capacity is used to learn the mixture of instruction tuning tasks.
Finetuned Language Models Are Zero-Shot Learners
As Brown et al. (2020) shows that zero and few-shot capabilities of language models substantially improve for larger models, we next explore how the benefits of instruction tuning are affected by model scale. Using the same cluster split as in the previous ablation study, we evaluate the effect of instruction tuning on models of size 422M, 2B, 8B, 68B, and 137B parameters. Figure 7 shows these results. We see that for the two models on the order of 100B parameters, instruction tuning substantially improves performance on held-out tasks, as is expected given the prior results in our paper. The behavior on held-out tasks for the 8B and smaller models, however, is thoughtprovoking-instruction tuning actually hurts performance on held-out tasks. One potential explanation for this result could be that for small-scale models, learning the ∼40 tasks used during instruction tuning fills the entire model capacity, causing these models to perform worse on new tasks. Under this potential explanation, for the larger scale models, instruction tuning fills up some model capacity but also teaches these models how to follow instructions, allowing them to generalize to new tasks with the remaining capacity. Zero-shot performance (4 task cluster avg.) Figure 8: Ablation study result using models with instructions removed from finetuning (FT).
Finetuned Language Models Are Zero-Shot Learners
In a final ablation study, we explore the role of instructions during finetuning, as one possibility is that performance gains come entirely from multi-task finetuning and the model could perform just as well without instructions. We hence consider two finetuning setups without instructions. In a no template setup, only inputs and outputs were given to the model (e.g., for translation the input would be "The dog runs." and the output would be "Le chien court."). In a dataset name setup, each input is prepended with the name of the task and dataset (e.g., for translation to French, the input would be "[Translation: WMT'14 to French] The dog runs.").
Finetuned Language Models Are Zero-Shot Learners
We compare these two ablations to FLAN's finetuning procedure, which used natural instructions (e.g., "Please translate this sentence to French: 'The dog runs.'"). We perform evaluations for four held-out clusters from Figure 5. For the no template setup, we used the FLAN instructions during zero-shot inference (because if we used no template, the model would not know what task to perform). For models finetuned on dataset name only, we report zero-shot performance for FLAN instructions as well as using the dataset name. Figure 8 shows the results-both ablation configurations performed substantially worse than FLAN, indicating that training with instructions is crucial for zero-shot performance on unseen tasks.
Finetuned Language Models Are Zero-Shot Learners
So far, we have focused on instruction tuning in the zero-shot setting. Here, we study how instruction tuning can be used when few-shot exemplars are available at inference time. The format for the few-shot setting builds on the zero-shot format. For some input x and output y, let instruct(x) denote the zero-shot instructions. Then, given k few-shot exemplars (x i , y i ) k i=1 and a new input x, the instruction format for the few-shot setting is "instruct(x 1 ) ⊕ y 1 ⊕ instruct(x 2 ) ⊕ y 2 ⊕ . . . ⊕ instruct(x k )⊕y k ⊕instruct(x)", where ⊕ denotes string concatenation with a delimiter token inserted in between. At both training and inference time, exemplars are randomly drawn from the training set, and the number of exemplars is capped at 16 and such that the total sequence length is less than 960 tokens. Our experiment uses the same task splits and evaluation procedure as §3, such that few-shot exemplars for an unseen task are only used at inference time.
Finetuned Language Models Are Zero-Shot Learners
As shown in Figure 9, few-shot exemplars improve the performance on all task clusters, compared with zero-shot FLAN. Exemplars are especially effective for tasks with large/complex output spaces, such as struct to text, translation, and closed-book QA, potentially because exemplars help the model better understand the output format. In addition, for all task clusters, standard deviation among templates is lower for few-shot FLAN, indicating reduced sensitivity to prompt engineering.   Figure 9: Adding few-shot exemplars to FLAN is a complementary method for improving the performance of instruction-tuned models. The orange bars indicate standard deviation among templates, averaged at the dataset level for each task cluster.  Instruction-tuned models respond better to continuous inputs from prompt tuning. When prompt tuning on a given dataset, no tasks from the same cluster as that dataset were seen during instruction tuning. Performance shown is the average on the SuperGLUE dev set.
Finetuned Language Models Are Zero-Shot Learners
As we've seen that instruction tuning improves the ability of a model to respond to instructions, it follows that, if FLAN is indeed more amenable to performing NLP tasks, then it should also achieve better performance when performing inference using soft prompts, represented by prepended continuous variables optimized via prompt tuning (Li & Liang, 2021;Lester et al., 2021). As further analysis, we train continuous prompts for each of the SuperGLUE (Wang et al., 2019a) tasks in accordance with the cluster splits from §2.2 such that when prompt-tuning on task T , no tasks in the same cluster as T were seen during instruction tuning. Our prompt tuning setup follows the procedure of Lester et al. (2021) except that we use a prompt length of 10, weight decay of 1e-4, and did not use dropout on the attention scores; we found in preliminary experiments that these changes improved the performance of LaMDA-PT. Figure 10 shows the results of these prompt tuning experiments for both using a fully-supervised training set and in a low-resource setting with only 32 training examples. We see that in all scenarios, prompt tuning works better with FLAN than LaMDA-PT. In many cases, especially for the low-resource setting, prompt tuning on FLAN even achieves more than 10% improvement over prompt tuning on the LaMDA-PT. This result exemplifies in another way how instruction tuning can result in a checkpoint that is more desirable for performing NLP tasks.
Finetuned Language Models Are Zero-Shot Learners
Our work relates to several broad research areas including zero-shot learning, prompting, multi-task learning, and language models for NLP applications (Radford et al., 2019;Raffel et al., 2020;Brown et al., 2020;Efrat & Levy, 2020;Aghajanyan et al., 2021;Li & Liang, 2021, inter alia). We describe prior work for these broad areas in an extended related work section (Appendix D), and here we describe two subareas narrower in scope that perhaps relate most closely to our work.
Finetuned Language Models Are Zero-Shot Learners
The way we ask a model to respond to instructions is similar to QA-based task formulation (Kumar et al., 2016;McCann et al., 2018), which aims to unify NLP tasks by casting them as QA over a context. Though these methods are very similar to ours, they mostly focus on multi-task learning instead of zero-shot learning, and-as noted by Liu et al. (2021)-they are generally not motivated by using existing knowledge in pretrained LMs. Moreover, our work supercedes recent work such as Chai et al. (2020) and Zhong et al. (2021) in terms of both model scale and scope of tasks.
Finetuned Language Models Are Zero-Shot Learners
The success of language models has led to nascent research on the ability of models to follow instructions. Most recently, Mishra et al. (2021) finetune 140M parameter BART on instructions with few-shot exemplars, and evaluate its few-shot abilities on unseen tasks-this is similar to our few-shot instruction tuning result from §4.4. This promising result (as well as one from Ye et al. (2021), which does not emphasize instructions as much) suggests that finetuning on a collection of tasks improves few-shot performance on unseen tasks, even at a smaller model scale. Sanh et al. (2021) finetune T5 in a setup similar to ours, finding that zero-shot learning can be improved in a model of 11B parameters. At a model scale similar to ours, OpenAI's InstructGPT models are trained via both finetuning and reinforcement learning to produce outputs that are more preferred by human raters (Ouyang et al., 2022).
Finetuned Language Models Are Zero-Shot Learners
Our paper has explored a simple question in zero-shot prompting: does finetuning a model on a collection of tasks phrased as instructions improve its performance on unseen tasks? We operationalize this question via instruction tuning, a simple method that combines appealing aspects of both the pretrain-finetune and prompting paradigms. Our instruction-tuned model, FLAN, improves performance against an untuned model and surpasses zero-shot GPT-3 on the majority of tasks that we evaluate on. Ablation studies reveal that performance on unseen tasks improves with the number of instruction tuning task clusters, and, interestingly, that performance improvements from instruction tuning emerge only with sufficient model scale. Moreover, instruction tuning can be combined with other prompting methods such as few-shot prompting and prompt tuning.
Finetuned Language Models Are Zero-Shot Learners
The diverse capabilities of language models at scale have drawn attention to the tradeoffs between specialist models (one model per task) and generalist models (one model for many tasks; Arivazhagan et al., 2019;Pratap et al., 2020), for which our study has potential implications. Although one might expect labeled data to have the most natural role in improving specialist models, instruction tuning demonstrates how labeled data can be used to help large language models perform many, unseen tasks. In other words, the positive effect of instruction tuning on cross-task generalization shows that task-specific training is complementary to general language modeling and motivates further research on generalist models.
Finetuned Language Models Are Zero-Shot Learners
As for limitations of our study, there is a degree of subjectivity in assigning tasks to clusters (though we try to use accepted categorizations in the literature), and we only explore the use of relatively short instructions of typically a single sentence (c.f. detailed instructions given to crowd-workers). A limitation for our evaluation is that individual examples might have appeared in the models' pretraining data, which includes web documents, though in post-hoc analysis (Appendix C) we do not find any evidence that data overlap substantially impacted the results. Finally, the scale of FLAN 137B makes it costly to serve. Future work on instruction tuning could include gathering/generating even more task clusters for finetuning, cross-lingual experiments, using FLAN to generate data for training downstream classifiers, and using finetuning to improve model behavior with respect to bias and fairness (Solaiman & Dennison, 2021).
Finetuned Language Models Are Zero-Shot Learners
This paper has explored a simple method for improving the ability of language models at scale to perform zero-shot tasks based purely on instructions. Our instruction-tuned model, FLAN, compares favorably against GPT-3 and signals the potential ability for language models at scale to follow instructions. We hope that our paper will spur further research on instructions-based NLP, zero-shot learning, and using labeled data to improve large language models.
Finetuned Language Models Are Zero-Shot Learners
This work uses language models, for which the risks and potential harms are discussed in Bender & Koller (2020), Brown et al. (2020), Bender et al. (2021), Patterson et al., (2021, and others. As our contribution in this paper is not a pretrained language model itself but rather an empirical study of how instruction tuning affects the zero-shot performance of a language model on unseen tasks, we additionally highlight two relevant ethical considerations. First, labeled datasets such as those we use for finetuning can contain undesirable biases, and these biases can be propagated into zero-shot applications of the model on downstream tasks. And second, instruction-tuned models can potentially require less data and expertise to use; such lower barriers to access could increase both the benefits and associated risks of such models.
Finetuned Language Models Are Zero-Shot Learners
We use the same pretrained language models as Austin et al. (2021). The energy cost and carbon footprint for the pretrained models were 451 MWh and 26 tCO2e, respectively. The additional instruction tuning gradient-steps for finetuning FLAN is less than 2% of the number of pretraining steps, and so the estimated additional energy cost is comparatively smaller.
Finetuned Language Models Are Zero-Shot Learners
Maarten Bosma conceived the original idea and implemented the first version of FLAN. Vincent Zhao prototyped the training and evaluation pipelines, as well as rank classification. Kelvin Guu proposed and implemented the idea of task clusters and evaluation using inter-cluster splits. Jason Wei, Maarten Bosma, Vincent Zhao, and Adams Wei Yu implemented the NLP tasks. Jason Wei, Vincent Zhao, and Adams Wei Yu conducted and managed most of the experiments. Jason Wei designed and ran the ablation studies. Jason Wei, Maarten Bosma, and Quoc V. Le wrote most of the paper. Jason Wei, Maarten Bosma, and Nan Du obtained the zero and few-shot baselines. Vincent Zhao and Kelvin Guu designed, implemented, and conducted the few-shot FLAN experiments. Maarten Bosma and Jason Wei ran the data contamination analysis. Brian Lester ran the prompt tuning experiments. Quoc V. Le and Andrew M. Dai advised, provided high-level guidance, and helped edit the paper. 
Finetuned Language Models Are Zero-Shot Learners
This section shows the full results for all datasets we evaluate. Results for translation and struct to text are shown in Table 1, and the results for eight NLU task clusters are shown in Table 2.
Finetuned Language Models Are Zero-Shot Learners
We show FLAN's performance using the best of up to ten instruction templates as well as the template with the best performance on the dev set. For LaMDA-PT, we use the templates from Brown et al. (2020), which were optimized for GPT-3, without performing any prompt engineering to optimize them on our model. For simplicity, we use greedy search for all generative tasks (compared with beam search used in Brown et al. (2020)). Unlike GPT-3, which chooses the number of few-shot exemplars k via best dev set performance, for few-shot LaMDA-PT we choose the highest k that fits in the context length of 1024 tokens, from k ∈ {1, 3, 5, 10}.
Finetuned Language Models Are Zero-Shot Learners
For DROP (Dua et al., 2019) and SQuADv2 , based on email correspondence with Brown et al. (2020), their definition of zero-shot differs from ours in that they actually use exemplars, but only from the same passage as the inference question (each passage has more than one question). Hence, GPT-3 zero-shot results are not directly comparable with ours for DROP and SQuADv2. We mark these results using the † symbol. Moreover, it is unclear how to parse the end of an answer for these two datasets, and so we use curly bracket delimiters { and }, where we expect } to indicate the end of the answer.
Finetuned Language Models Are Zero-Shot Learners
For struct to text, reported T5/mT5 results are from the GEM benchmark paper (Gehrmann et al., 2021), though we do not report their results for DART (through correspondence with authors, we confirmed that their results for DART were incorrect). Though we use a summarization task cluster during instruction tuning, we leave evaluation of summarization for future work, as the mean input of most summarization datasets exceeds FLAN's input length of 1024 tokens.
Finetuned Language Models Are Zero-Shot Learners
LaMDA-PT GPT-3 175B zero-shot few-shot   
Finetuned Language Models Are Zero-Shot Learners
Our primary hypothesis is that instruction tuning on a diverse set of tasks improves performance on unseen tasks. §4.1 showed that adding more task clusters improves performance; here, we further explore whether adding additional datasets improves performance when the number of task clusters is held constant. We use the same split as in §4.1, where the NLI, commonsense reasoning, and closed-book QA clusters are held-out, and seven other task clusters remain for instruction tuning. For these seven task clusters, we instruction tune models using just one dataset per task cluster and using four datasets per task cluster (for task clusters that did not have four tasks, we just used all available tasks). In addition, we simultaneously explore the role of the number of instruction templates per dataset; as mentioned in §2.1, for each dataset we manually composed ten instructional templates for instruction tuning. Here, we instruction tune models using 1, 4, and 10 templates per dataset. Figure 11 shows these results. Using more datasets per cluster improved performance by almost 10% on average across the three held-out clusters. Using more templates per dataset, however, had a comparatively negligible effect on performance when there was one task per cluster, which disappeared when there were four tasks per cluster. The small effect of templates is striking given our original motivation that composing ten templates per task would mitigate overfitting to any particular template. This results serves to underscore, however, the unpredictability of finetuning large language models, as one hypothesis is that models at such scale do not easily overfit to a finetuning single task. Figure 11: Effect of datasets per task cluster and templates per dataset on performance on three held-out clusters: NLI, commonsense reasoning, and closed-book QA. Adding more datasets per task cluster substantially improves performance. Using more templates per dataset, however, only had a very small effect on performance, which disappeared when there were sufficient dataset per task cluster.
Finetuned Language Models Are Zero-Shot Learners
The per-cluster results for the ablation study from §4.3 are shown in Table 3.
Finetuned Language Models Are Zero-Shot Learners
The per-dataset results for the analysis in §4.5 are given in Table 4. As the above tasks are all classification, further work in this direction might include tasks such as summarization or question answering, or try to finetune the model using the supervised datasets.
Finetuned Language Models Are Zero-Shot Learners
One reasonable concern is that since the pretraining corpus of FLAN has more than 2 trillion tokens, it is possible that examples from a given evaluation dataset may have already been seen verbatim by the model during pre-training, hence inflating the performance of our purported zero-shot model. To this end, like GPT-3 (Brown et al., 2020), we perform post-hoc data contamination analysis to   Table 4: FLAN (instruction tuning) responds better to continuous inputs attained via prompt tuning than LaMDA-PT (no instruction tuning). When prompt tuning on a given dataset, no tasks from the same cluster as that dataset were seen during instruction tuning.
Finetuned Language Models Are Zero-Shot Learners
investigate whether the performance of the model is in fact inflated by evaluating on examples that occurred in the pretraining dataset.
Finetuned Language Models Are Zero-Shot Learners
Our data contamination procedure follows the setup of Brown et al. (2020), which, for each benchmark, produces a "clean" version that removes all potentially leaked examples, defined as examples for which any n-gram (n varies per dataset but is roughly 13) overlapped with anything in the pretraining corpus. We use the same n per dataset as Brown et al. (2020) and also split on spaces. We then evaluate our model on this clean subset, comparing against model performance on the original dataset (clean + dirty). Lower performance on the clean subset would suggest that data contamination leads to inflated results. Figure 12 summarizes these results, with the exact numbers given in Table 5. We see several trends very similar to those in the GPT-3 paper: (1) many datasets had a substantial number of examples that overlapped with the pretraining data, (2) across all datasets, we do not see a correlation that evaluating on clean data does worse than evaluating on the total dataset, and (3)  ANLI R1 and R2  also had almost complete data contamination, to a much higher degree than GPT-3. Upon further inspection, we see that most overlaps occur in example contexts and not hypotheses (97.3% for ANLI R1 and 98.2% for ANLI R2). As ANLI R1 and R2 are based entirely from Wikipedia examples (R3 is not), we posit that this higher degree of contamination in our pretraining dataset compared with GPT-3's is potentially due to using a more-recent version of Wikipedia that includes the contexts used in ANLI R1 and R2 (which were collected in 2019). Because seeing a particular context in pretraining does not help with the NLI task given a new, unseen sentence, we think it is unlikely that these overlaps affected performance on the two datasets.
Finetuned Language Models Are Zero-Shot Learners
Of the remaining datasets, only ReCoRD and PIQA had a clean subset performance that was lower than the overall evaluation set performance by more than 1%. These two datasets are language modeling (i.e., "what's the best continuation of this sentence?"), and so it is more likely compared with previous tasks that seeing a complete sentence in the pretraining data could help the model predict the right answer in downstream evaluations. For PIQA, both the goal and solution had overlaps in 93 of the 1,838 evaluation examples, and for ReCoRD, the query had overlaps in 2,320 of the 10,000 training examples. We hence mark these results with an asterisk * in Table 2. Brown et al. (2020) also reported substantial contamination rates for these two datasets (61% dirty for ReCoRD and 29% for PIQA), and also mark PIQA with an asterisk.
Finetuned Language Models Are Zero-Shot Learners
As this overlap analysis follows that performed in Brown et al. (2020), we reiterate the same caveats: the conservative nature of our n-gram matching procedure likely introduces additional false positives; there are no guarantees that the clean subset is drawn from the same distribution as the overall subset; and, accurately detecting test contamination is a relatively new research area without established best practices. Moreover, as our pretraining corpus is almost five times larger than that used for GPT-3 (which was 500B tokens), it is possible that there are more false positives in detecting dirty data.  Table 5: Overlap statistics for the subset of datasets that are also used in GPT-3, sorted from dirtiest to cleanest. An evaluation example was dirty if it had any n-gram collision with the pretraining corpus. We computed these results for FLAN's performance using only a single template for each dataset, so they differ slightly compared with the results for average performance over all templates.
Finetuned Language Models Are Zero-Shot Learners
Our work is broadly inspired by a long line of prior work on language models for NLP applications (Dai & Le, 2015;Peters et al., 2018;Howard & Ruder, 2018;Radford et al., 2018;2019, inter alia). Instruction tuning can be seen as a formulation of multitask learning (MTL), which is an established area within deep learning (Collobert et al., 2011;Luong et al., 2016;Ruder, 2017;Velay & Daniel, 2018;Clark et al., 2019b;Liu et al., 2019b, inter alia)-see Worsham & Kalita (2020) for a recent survey on MTL for NLP. Differing from prior MTL work which focuses on performance improvements across training tasks (Raffel et al., 2020;Aghajanyan et al., 2021) or to new domains (Axelrod et al., 2011), our work is motivated by improving zero-shot generalization to tasks that were not seen in training.
Finetuned Language Models Are Zero-Shot Learners
Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (  . Most recently, the emergent ability of language models (Brown et al., 2020) has led to increased interest in how models generalize to unseen tasks, the definition of zero-shot learning used in our paper. In addition, meta-learning (Finn et al., 2017;Vanschoren, 2018, inter alia) also broadly tries to train models that adapt quickly to unseen tasks, typically based on a few examples.
Finetuned Language Models Are Zero-Shot Learners
Instruction tuning leverages the intuition that language models at scale contain substantial world knowledge and can perform a range of NLP tasks (Brown et al., 2020, see also Bommasani et al. (2021)). Another line of work that shares this goal prompts models with continuous inputs optimized via backpropagation to substantially improve performance (Li & Liang, 2021;Lester et al., 2021;Qin & Eisner, 2021), as well as work that prompts models to produce specialized outputs (Wei et al., 2022). Although the success of these approaches depends heavily on model scale (Lester et al., 2021), for which large models can be costly to serve, the ability of a single large model to perform many tasks slightly eases this burden. As shown by our experiments in §4.5, prompt tuning is an orthogonal method for which instruction tuning can additionally improve performance. Reif et al. (2021) is similar to our work in that they also use related tasks to improve zero-shot learning, though they differ by only using related tasks in the context (and not finetuning), and focus on the application of text style transfer.
Finetuned Language Models Are Zero-Shot Learners
Our work shares similar motivations with prompting in that we use inference-time text interactions to prompt a single model, without creating separate checkpoints for each task. Whereas prompting work such as GPT-3 uses prompt engineering to write prompts that intentionally mimic text that is likely to be seen during pretraining (e.g., for MultiRC GPT-3 tries a prompt that mimics a test with an answer key), we hope that finetuning models to respond to natural language instructions instead of completing a sentence will make such large models more accessible to non-technical users.
Finetuned Language Models Are Zero-Shot Learners
Finetuning pretrained language models is a well-established method in NLP, with much of the work so far occurring on models in the range of 100M to 10B parameters (Dai & Le, 2015;Devlin et al., 2019;Raffel et al., 2020;Lewis et al., 2020, inter alia). For models of O(100B) parameters, recent work has finetuned task-specific models for program synthesis (Austin et al., 2021;, summarization , as well as improved bias and fairness behavior (Solaiman & Dennison, 2021). In addition to the traditional "dense" models, sparse mixture of experts (MoE) models of up to more than 1T parameters have been trained and finetuned (Lepikhin et al., 2020;Fedus et al., 2021). Compared with this prior work that finetunes and evaluates on the same downstream task, our setup studies the effect of instruction tuning on ability to perform unseen tasks.
Finetuned Language Models Are Zero-Shot Learners
The instructions we use for instruction tuning are similar to QA-based task formulation research, which aims to unify NLP tasks by casting them as question-answering over a context. answering, text classification, and text modification, and find that this few-shot finetuning with instructions improves performance on unseen tasks. Ye et al. (2021) introduce a setup for cross-task few-shot learning, finding that multi-task meta-learning using MAML (Finn et al., 2017) improves the few-shot capabilities of BART on unseen downstream tasks. Our work differs from these two papers in that we focus on zero-shot learning, for which we observe the crucial importance of model scale (FLAN is 1,000x larger than BART-base).
Finetuned Language Models Are Zero-Shot Learners
Perhaps the papers most related to ours are the recent Sanh et al. (2021) andMin et al. (2021), which were released after our initial preprint. Min et al. (2021) finetunes GPT-2 Large (770M parameters) to be a few-shot learner, which is the same approach as our experiment in Section 4.3. Similar to our conclusions, they also observe that including few-shot exemplars and instruction tuning are complementary ways to improve performance. Sanh et al. (2021) propose to finetune T5-11B to respond to prompts, and they also report performance improvements on zero-shot learning. These two papers and our work all study finetuning with instructions, but, as noted by Min et al. (2021), it is hard to directly compare results, due to differing model sizes, model types (decoder-only vs encoder-decoder), pretraining data, task mixtures, and type of instructions (Sanh et al. (2021) say that their instructions are more diverse).
Finetuned Language Models Are Zero-Shot Learners
Finally, OpenAI has a model called InstructGPT (Ouyang et al., 2022). InstructGPT uses human anntations to guide desired model behavior, both via finetuning and reinforcement learning, finding that InstructGPT is preferred by human rathers compared with unmodified GPT-3.
Finetuned Language Models Are Zero-Shot Learners
How do the FLAN instructions differ from GPT-3 or T5 prompts?
Finetuned Language Models Are Zero-Shot Learners
GPT-3 prompting is done in a way such that the prompt looks like data that the model has been pretrained on, and the model finishes the continuation. T5 prompts are mostly just a tag for the dataset, which would not work in the zero-shot setting. In contrast, the prompts that we use for FLAN are similar to what would be used to ask a human to perform the task.
Finetuned Language Models Are Zero-Shot Learners
For instance, given an input for an NLI task, these would be the prompts.
Finetuned Language Models Are Zero-Shot Learners
T5 prompt:
Finetuned Language Models Are Zero-Shot Learners
cb hypothesis: At my age you will probably have learnt one lesson. premise: It's not certain how many lessons you'll learn by your thirties.
Finetuned Language Models Are Zero-Shot Learners
GPT-3 prompt:
Finetuned Language Models Are Zero-Shot Learners
At my age you will probably have learnt one lesson. question: It's not certain how many lessons you'll learn by your thirties. true, false, or neither? answer:
Finetuned Language Models Are Zero-Shot Learners
FLAN prompt:
Finetuned Language Models Are Zero-Shot Learners
Premise: At my age you will probably have learnt one lesson. Hypothesis: It's not certain how many lessons you'll learn by your thirties. Does the premise entail the hypothesis?
Finetuned Language Models Are Zero-Shot Learners
So because FLAN prompts are formulated as responding to an instruction, they do not work well for pretrained language models without finetuning. Performance was near zero for most generation tasks. For instance, given the input "'The dog runs.' Translate this sentence to French.", LaMDA-PT continues with "The dog runs after the cat" instead of actually translating the sentence. Hence, we used the established GPT-3 prompts for our LaMDA-PT baselines.
Finetuned Language Models Are Zero-Shot Learners
What are some limitations/failure cases of FLAN?
Finetuned Language Models Are Zero-Shot Learners
While we qualitatively find that FLAN responds well to most tasks, it does fail on some simple tasks. For instance, as shown in Figure 22, FLAN fails at the very simple task of returning the second word in a sentence, and also incorrectly translates a question to Danish when asked to answer the question in Danish. Additional limitations include a context length of only 1024 tokens (which is not enough for most summarization tasks), and that the model was mostly trained on English data.
Finetuned Language Models Are Zero-Shot Learners
Can FLAN be used when large amounts of training data are available?
Finetuned Language Models Are Zero-Shot Learners
In this work, we focus on cross-task generalization to zero-shot tasks, but we also believe that instruction tuning could result in positive task transfer among seen tasks, depending on the mixture of tasks (though we leave this for future work). In §4.5, where we apply prompt tuning to the FLAN checkpoint, we see promising results that indicate positive task transfer in a supervised setting.
Finetuned Language Models Are Zero-Shot Learners
Are the ten unique templates per dataset or per task cluster?
Finetuned Language Models Are Zero-Shot Learners
The ten unique templates are for each dataset and not for a task cluster. This is because datasets in the same task cluster often differed slightly (e.g., "is this movie review positive" vs "is this yelp review positive").
Finetuned Language Models Are Zero-Shot Learners
In Figure 7A, why does the untuned LaMDA-PT model see worse performance with more parameters for reading comprehension and sentiment analysis?
Finetuned Language Models Are Zero-Shot Learners
For context, Figure 7A is a check of correctness for Figure 7B. Figure 7A confirms that scale improves performance for tasks that were seen during instruction tuning, as expected. The untuned LaMDA-PT model performance in Figure 7A is shown just for completeness.
Finetuned Language Models Are Zero-Shot Learners
Nonetheless, the fact that scale does not always improve zero-shot performance of untuned LaMDA-PT is an interesting artifact. Initially, we were surprised, because Brown et al. (2020) shows that scale improves performance across a large number of tasks in aggregate.
Finetuned Language Models Are Zero-Shot Learners
It turns out that scale does not improve performance for certain tasks. This is especially true for zero-shot learning, and we think that this happens to be the case for the reading comprehension and sentiment analysis tasks we evaluate. The GPT-3 paper itself similarly reports that zero-shot performance on BoolQ and DROP decreases from 13B to 175B parameters. The GPT-3 paper does not show results on sentiment analysis, but Holtzman et al. (2021) find that zero-shot performance on SST-2 also gets worse from 13B to 175B parameters. Hence, this artifact is consistent across both GPT-3 and the models we use.
Finetuned Language Models Are Zero-Shot Learners
This artifact is certainly worth further study, but is outside the scope of instruction tuning. Ideally, we would have performed the Figure 7 ablation with cross-validation instead of a single split, which likely would have smoothed out that artifact.  Figure 13: For sentiment analysis, FLAN changes the answer appropriately when the question is flipped.
Finetuned Language Models Are Zero-Shot Learners
What is Barack Obama's profession?
Finetuned Language Models Are Zero-Shot Learners
• 总统 • 美国总统 Input:
Finetuned Language Models Are Zero-Shot Learners
What is Barack Obama's profession?
Finetuned Language Models Are Zero-Shot Learners
• politician • lawyer • senator • us president FLAN output: Figure 14: For question answering, FLAN can answer a question in another language when instructed to do so. Multiple FLAN outputs are generated via random sampling with a temperature of 0.9 and top k of 40.
Finetuned Language Models Are Zero-Shot Learners
John is reading the paper "Finetuned Language Models are Zero-Shot Learners."
Finetuned Language Models Are Zero-Shot Learners
Change to passive voice.
Finetuned Language Models Are Zero-Shot Learners
The paper "Finetuned Language Models are Zero-Shot Learners" is being read by John.
Finetuned Language Models Are Zero-Shot Learners
John is reading the paper "Finetuned Language Models are Zero-Shot Learners."
Finetuned Language Models Are Zero-Shot Learners
Change to past tense.
Finetuned Language Models Are Zero-Shot Learners
John was reading the paper "Finetuned Language Models are Zero-Shot Learners."
Finetuned Language Models Are Zero-Shot Learners
John is reading the paper "Finetuned Language Models are Zero-Shot Learners."
Finetuned Language Models Are Zero-Shot Learners
Change the verb to eat and the subject to Noam Chomsky.
Finetuned Language Models Are Zero-Shot Learners
Noam Chomsky is eating the paper "Finetuned Language Models are Zero-Shot Learners."
Finetuned Language Models Are Zero-Shot Learners
Change to the verb to eat.
Finetuned Language Models Are Zero-Shot Learners
John is eating the paper "Finetuned Language Models are Zero-Shot Learners." FLAN output: Figure 15: FLAN can perform zero-shot rule-based manipulations.
Finetuned Language Models Are Zero-Shot Learners
Recommend activities to do on a sunny weekend in Copenhagen.
Finetuned Language Models Are Zero-Shot Learners
• Input:
Finetuned Language Models Are Zero-Shot Learners
Generate utterances with the intent "get COVID vaccine" Input:
Finetuned Language Models Are Zero-Shot Learners
• I would like to get the covid vaccine.
Finetuned Language Models Are Zero-Shot Learners
• I would like a COVID vaccine as soon as possible • Can I get a COVID vaccine as a preventative measure?
Finetuned Language Models Are Zero-Shot Learners
• Get COVID vaccine for a five year old • Where is a center for COVID vaccine testing?
Finetuned Language Models Are Zero-Shot Learners
FLAN output: Figure 17: FLAN can be used in a zero-shot fashion to generate data, such as utterances that are consistent with a given intent. Multiple FLAN outputs are generated via random sampling with a temperature of 0.9 and top k of 40.
Finetuned Language Models Are Zero-Shot Learners
Suggest a related search term to "Novak Djokovic" 
Finetuned Language Models Are Zero-Shot Learners
On the way to Yosemite, our front tire blew.
Finetuned Language Models Are Zero-Shot Learners
Finish writing the story, and include a happy ending.
Finetuned Language Models Are Zero-Shot Learners
• We got our tire replaced. The mechanic was honest, kind and helpful. Then we had a smooth ride for the rest of the trip. Our trip was filled with fun and excitement.
Finetuned Language Models Are Zero-Shot Learners
• We waited three hours for the tow truck, but had a great picnic. The tow truck driver helped us put the spare on, and we arrived in Yosemite for our reservations dinner, with time to spare. Figure 19: FLAN can perform zero-shot tasks relevant to assisted-writing applications. Multiple FLAN outputs are generated via random sampling with a temperature of 0.9 and top k of 40.
Finetuned Language Models Are Zero-Shot Learners
Make up a word that means "like a pipedream"
Finetuned Language Models Are Zero-Shot Learners
Input:
Finetuned Language Models Are Zero-Shot Learners
Make up a new word that means "when bananas shower from the sky" Input: Write a sad story about carrots.
Finetuned Language Models Are Zero-Shot Learners
The sun had barely risen when the carrots were taken from their beds and brought to market. They'd had no time to gather their strength. They smelled of earth, their greens still wet from the dew. At first no one was interested in buying them. But one man finally reached out and picked up one of the carrots. "I'll take this one!" he cried out. "It's for my mother! I'll pay you well for it." The man took the carrot to the cashier, eager to get it home to his mother. He was careful to wrap it carefully so that it wouldn't get bruised or dirty. He hurried home, and when he saw his mother, he gave her the carrot. "Is this one of the ones I grew?" she asked, her voice wavering with emotion. "Yes, mother," he replied. "It is, dear." The man left the carrot with his mother and hurried back to market. His mother was very sick, and he was afraid she wouldn't last much longer. He needed to buy all the carrots in market so that none would be left when she was gone.
Finetuned Language Models Are Zero-Shot Learners
John moved to mountain view. He spent friday night in San Francisco. He got a COVID vaccine. He voted in the state senate election.
Finetuned Language Models Are Zero-Shot Learners
Write a story that follows the above plot.
Finetuned Language Models Are Zero-Shot Learners
John arrived at the city with his vaccine, feeling prepared enough for the state senate election of San Francisco. He had voted for the first time last night and was feeling good about it. He felt more prepared than ever to win, and win big, because he was getting a COVID vaccine. He knew that his opponent had not gotten a COVID vaccine, and he knew he had already defeated him before the election began.
Finetuned Language Models Are Zero-Shot Learners
Figure 21: Open-ended generation tasks by FLAN. The carrot story was from sampling sixteen outputs with a minimum length of 150 and choosing the highest probability output.   
Finetuned Language Models Are Zero-Shot Learners
After years of study, the Vatican's doctrinal congregation has sent church leaders a confidential document concluding that "sex-change" procedures do not change a person's gender in the eyes of the church.
Finetuned Language Models Are Zero-Shot Learners
Based on the paragraph above can we conclude that "Sex-change operations become more common."?
Finetuned Language Models Are Zero-Shot Learners
OPTIONS: -yes -no TARGET no    OPTIONS: -I was hunting for a new apartment.
Finetuned Language Models Are Zero-Shot Learners
-I was moving out of my apartment.
Finetuned Language Models Are Zero-Shot Learners
TARGET I was moving out of my apartment. 
Finetuned Language Models Are Zero-Shot Learners
What happens next in this paragraph?
Finetuned Language Models Are Zero-Shot Learners
Once the rope is inside the hook, he begins moving up the wall but shortly after he stops and begins talking. The male then begins talking about the clip again and goes back up the wall. as he OPTIONS: -progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety.
Finetuned Language Models Are Zero-Shot Learners
-changes time, an instant replay of his initial move is shown a second time.
Finetuned Language Models Are Zero-Shot Learners
-continues to talk, another male speaks about the move and shows another closeup of the plex by the male.
Finetuned Language Models Are Zero-Shot Learners
-continues, other people start to arrive and begin to hang out with him as he makes a few parts of the rope.
Finetuned Language Models Are Zero-Shot Learners
TARGET progresses, there are hooks everywhere on the wall and when he gets near them, he puts his rope inside of it for support and safety. 
Finetuned Language Models Are Zero-Shot Learners
Here is a goal: Remove smell from garbage disposal.
Finetuned Language Models Are Zero-Shot Learners
How would you accomplish this goal?
Finetuned Language Models Are Zero-Shot Learners
OPTIONS: -Create soda ice cubes and grind through disposal.
Finetuned Language Models Are Zero-Shot Learners
-Create vinegar ice cubes and grind through disposal.
Finetuned Language Models Are Zero-Shot Learners
TARGET Create vinegar ice cubes and grind through disposal. 
Finetuned Language Models Are Zero-Shot Learners
Caroline never drinks carbonated beverages. Her friends pick on her because of it. One day they challenged her to drink a soda. Caroline wanted to win the challenge.
Finetuned Language Models Are Zero-Shot Learners
Predict the next sentence.   INPUT Question: who is the girl in more than you know?? Answer:
Finetuned Language Models Are Zero-Shot Learners
TARGET Romi Van Renterghem. 
Finetuned Language Models Are Zero-Shot Learners
Please answer this question: Henry Croft, an orphan street sweeper who collected money for charity, is associated with what organised charitable tradition of working class culture in London, England?
Finetuned Language Models Are Zero-Shot Learners
TARGET pearly kings and queens  (Brown et al., 2020). as our test set for reporting numbers.
Finetuned Language Models Are Zero-Shot Learners
INPUT How does the sentence end?
Finetuned Language Models Are Zero-Shot Learners
Elena wanted to move out of her parents fast but Victoria wanted to stay for a while, OPTIONS: -Elena went to school.
Finetuned Language Models Are Zero-Shot Learners
-Victoria went to school.
Finetuned Language Models Are Zero-Shot Learners
TARGET Victoria went to school. 
Finetuned Language Models Are Zero-Shot Learners
Jane knocked on Susan's door, but there was no answer. OPTIONS: -Jane was out.
Finetuned Language Models Are Zero-Shot Learners
-Susan was out.
Finetuned Language Models Are Zero-Shot Learners
TARGET Susan was out. (CNN) -At first glance, "The Flat" might seem like an episode of "Hoarders," Israeli-style. The documentary film opens after an elderly woman dies in Tel Aviv. Her grandchildren assemble to clean out her apartment, packed with dusty books, vintage clothing (dozens of pairs of fancy gloves, for instance), enough purses to stock a department store, jewelry, mementoes and closets full of knickknacks. But buried among the detritus they chance upon something remarkablemysterious papers linking the grandparents to an important Nazi figure. How could such ardent Zionists, who left their native Germany in the early 1930s, have been involved with an SS official like Leopold von Mildenstein?
Finetuned Language Models Are Zero-Shot Learners
What I found out was this journey, the Nazi ( OPTIONS: -Arnon Goldfinger) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-CNN) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Germany) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Israeli) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Leopold von Mildenstein) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Nazi) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-SS) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Tel Aviv) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-The Flat) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
-Zionists) and his wife were accompanied by my grandparents," Goldfinger told CNN.
Finetuned Language Models Are Zero-Shot Learners
TARGET Leopold von Mildenstein) and his wife were accompanied by my grandparents," Goldfinger told CNN. 
Finetuned Language Models Are Zero-Shot Learners
Here the largest town of the district is located: Nordenham , lying opposite to Bremerhaven at the Weser mouth.
Finetuned Language Models Are Zero-Shot Learners
TARGET An der B 211 befindet sich in Loyermoor der so genannte "Geest-Abbruch", der eine Höhendifferenz von gut 30 Meter überbrückt. 
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Multi-Document Summarization is the task of generating a summary from a cluster of related documents. State-of-the-art approaches to multi-document summarization are primarily either graph-based (Liao et al., 2018;Li et al., 2020;Pasunuru et al., 2021), leveraging graph neural networks to connect information between the documents, or hierarchical (Liu and Lapata, 2019a;Fabbri et al., 2019;Jin et al., 2020), building intermediate representations of individual documents and then aggregating information across. While effective, these models either require domain-specific additional information e.g. Abstract Meaning Representation (Liao et al., 2018), or discourse graphs (Christensen et al., 2013;Li et al., 2020), or use dataset-specific, customized architectures, making it difficult to leverage pretrained language models. Simultaneously, recent pretrained language models (typically encoder-decoder transformers) * Work mainly done during an internship at AI2. 1 The code and pre-trained models can be found at https: //github.com/allenai/PRIMER  have shown the advantages of pretraining and transfer learning for generation and summarization (Raffel et al., 2020;Lewis et al., 2020;Beltagy et al., 2020;Zaheer et al., 2020). Yet, existing pretrained models either use single-document pretraining objectives or use encoder-only models that do not work for generation tasks like summarization (e.g., CDLM, Caciularu et al., 2021).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Therefore, we argue that these pretrained models are not necessarily the best fit for multi-document summarization. Alternatively, we propose a simple pretraining approach for multi-document summarization, reducing the need for dataset-specific architectures and large fine-tuning labeled data (See Figure 1 to compare with other pretrained models). Our method is designed to teach the model to identify and aggregate salient information across a "cluster" of related documents during pretraining. Specifically, our approach uses the Gap Sentence Generation objective (GSG) (Zhang et al., 2020), i.e. masking out several sentences from the input document, and recovering them in order in the decoder. We propose a novel strategy for GSG sentence masking which we call, Entity Pyramid, inspired by the Pyramid Evaluation method (Nenkova and Passonneau, 2004). With Entity Pyramid, we mask salient sentences in the entire cluster then train the model to generate them, encouraging it to find important information across documents and aggregate it in one summary.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We conduct extensive experiments on 6 multidocument summarization datasets from 3 different domains. We show that despite its simplic-  ity, PRIMERA achieves superior performance compared with prior state-of-the-art pretrained models, as well as dataset-specific models in both few-shot and full fine-tuning settings. PRIMERA performs particularly strong in zero-and few-shot settings, significantly outperforming prior state-of-the-art up to 5 Rouge-1 points with as few as 10 examples. Our contributions are summarized below: 1. We release PRIMERA, the first pretrained generation model for multi-document inputs with focus on summarization. 2. We propose Entity Pyramid, a novel pretraining strategy that trains the model to select and aggregate salient information from documents. 3. We extensively evaluate PRIMERA on 6 datasets from 3 different domains for zero-shot, few-shot and fully-supervised settings. We show that PRIMERA outperforms current state-of-the-art on most of these evaluations with large margins.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In this section, we discuss our proposed model PRIMERA, a new pretrained general model for multi-document summarization. Unlike prior work, PRIMERA minimizes dataset-specific modeling by simply concatenating a set of documents and processing them with a general efficient encoderdecoder transformer model ( §2.1). The underlying transformer model is pretrained on an unlabeled multi-document dataset, with a new entity-based sentence masking objective to capture the salient information within a set of related documents ( §2.2).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Our goal is to minimize dataset-specific modeling to leverage general pretrained transformer models for the multi-document task and make it easy to use in practice. Therefore, to summarize a set of related documents, we simply concatenate all the documents in a single long sequence, and process them with an encoder-decoder transformer model. Since the concatenated sequence is long, instead of more standard encoder-decoder transformers like BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), we use the Longformer-Encoder-Decoder (LED) Model (Beltagy et al., 2020), an efficient transformer model with linear complexity with respect to the input length. 2 LED uses a sparse local+global attention mechanism in the encoder self-attention side while using the full attention on decoder and cross-attention.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
When concatenating, we add special document separator tokens (<doc-sep>) between the documents to make the model aware of the document boundaries ( Figure 2). We also assign global attention to these tokens which the model can use to share information across documents (Caciularu et al., 2021) (see §5 for ablations of the effectiveness of this input structure and global attention).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In summarization, task-inspired pretraining objectives have been shown to provide gains over general-purpose pretrained transformers (PEGASUS;Zhang et al., 2020). In particular, PE-GASUS introduces Gap Sentence Generation (GSG) as a pretraining objective where some sentences are masked in the input and the model is tasked to generate them. Following PEGASUS, we use the GSG objective, but introduce a new masking strategy designed for multi-document summarization. As in GSG, we select and mask out m summary-like sentences from the input documents we want to summarize, i.e. every selected sentence is replaced by a  Figure 3: An example on sentence selection by Principle vs our Entity Pyramid strategy. Italic text in red is the sentence with the highest Principle ROUGE scores, which is thereby chosen by the Principle Strategy. Most frequent entity 'Colorado' is shown with blue, followed by the Pyramid ROUGE scores in parenthesis. The final selected sentence by Entity Pyramid strategy is in italic. which is a better pseudo-summary than the ones selected by the Principle strategy.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
single token [sent-mask] in the input, and train the model to generate the concatenation of those sentences as a "pseudo-summary" (Figure 2). This is close to abstractive summarization because the model needs to reconstruct the masked sentences using the information in the rest of the documents.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The key idea is how to select sentences that best summarize or represent a set of related input documents (which we also call a "cluster"), not just a single document as in standard GSG. Zhang et al. (2020) use three strategies -Random, Lead (first m sentences), and "Principle". The "Principle" method computes sentence salience score based on ROUGE score of each sentence, s i , w.r.t the rest of the document (D/{s i }), i.e. Score(s i ) = ROUGE(s i , D/{s i }). Intuitively, this assigns a high score to the sentences that have a high overlap with the other sentences.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
However, we argue that a naive extension of such strategy to multi-document summarization would be sub-optimal since multi-document inputs typically include redundant information, and such strategy would prefer an exact match between sentences, resulting in a selection of less representative information.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
For instance, Figure 3 shows an example of sentences picked by the Principle strategy (Zhang et al., 2020) vs our Entity Pyramid approach. The figure shows a cluster containing three news articles discussing a wildfire happened in Corolado, and the pseudo-summary of this cluster should be related to the location, time and consequence of the wildfire, but with the Principle strategy, the non-salient sentences quoting the words from an officer are assigned the highest score, as the exact same sentence appeared in two out of the three articles. In comparison, instead of the quoted words, our strategy selects the most representative sentences in the cluster with high frequency entities.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
To address this limitation, we propose a new masking strategy inspired by the Pyramid Evaluation framework (Nenkova and Passonneau, 2004) which was originally developed for evaluating summaries with multiple human written references. Our strategy aims to select sentences that best represent the entire cluster of input documents.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Pyramid Evaluation The Pyramid Evaluation method (Nenkova and Passonneau, 2004) is based on the intuition that relevance of a unit of information can be determined by the number of references (i.e. gold standard) summaries that include it. The unit of information is called Summary Content Unit (SCU); words or phrases that represent single facts. These SCUs are first identified by human annotators in each reference summary, and they receive a score proportional to the number of reference summaries that contain them. A Pyramid Score for a candidate summary is then the normalized mean of the scores of the SCUs that it contains. One advantage of the Pyramid method is that it directly assesses the content quality.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Entity Pyramid Masking Inspired by how content saliency is measured in the Pyramid Evaluation, we hypothesize that a similar idea could be applied in multi-document summarization to identify salient sentences for masking. Specifically, for a cluster with multiple related documents, the more documents an SCU appears in, the more salient that information should be to the cluster. Therefore, it should be considered for inclusion in the pseudosummary in our masked sentence generation objective. However, SCUs in the original Pyramid Evaluation are human-annotated, which is not feasible for large scale pretraining. As a proxy, we explore leveraging information expressed as named entities, since they are key building blocks in extracting information from text about events/objects and the relationships between their participants/parts (Jurafsky and Martin, 2009 if |selected| == m then 8:
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Break 9: end if 10: end for 11: Return selected framework, we use the entity frequency in the cluster as a proxy for saliency. Concretely, as shown in Fig. 4, we have the following three steps to select salient sentences in our masking strategy: 1. Entity Extraction. We extract named entities using SpaCy (Honnibal et al., 2020). 3 2. Entity Pyramid Estimation. We then build an Entity Pyramid for estimating the salience of entities based on their document frequency, i.e. the number of documents each entity appears in. 3. Sentence Selection. Similar to the Pyramid evaluation framework, we identify salient sentences with respect to the cluster of related documents. Algorithm 1 shows the sentence selection procedure. As we aim to select the entities better representing the whole cluster instead of a single document, we first remove all entities from the Pyramid that appear only in one document. Next, we iteratively select entities from top of the pyramid to bottom (i.e., highest to lowest frequency), and then select sentences in the document that include the entity as the initial candidate set. Finally, within this candidate set, we find the most representative sentences to the cluster by measuring the content overlap of the sentence w.r.t documents other than the one it appears in. This final step supports the goal of our pretraining objective, namely to reconstruct sentences that can be recovered using information from other documents in the cluster, which encourages the model to better connect and aggregate information across multiple documents. Following Zhang et al.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
(2020) we use ROUGE scores (Lin, 2004) as a proxy for content overlap. For each sentence s i , we specifically define a Cluster ROUGE score as Score(s i ) = {doc j ∈C,s i ∈ doc j } ROUGE(s i , doc j ) Where C is the cluster of related documents. Note that different from the importance heuristic defined in PEGASUS (Zhang et al., 2020), Entity Pyramid strategy favors sentences that are representative of more documents in the cluster than the exact matching between fewer documents (See Figure 3 for a qualitative example.) . The benefit of our strategy is shown in an ablation study ( §5).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We aim to answer the following questions:   (R-1, -2, and -L), which are the standard evaluation metrics, to evaluate the downstream task of multi-document summarization. 4 For better readability, we use AVG ROUGE scores (R-1, -2, and -L) for evaluation in the few-shot setting.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Many existing works in adapting pretrained models for summarization require large amounts of finetuning data, which is often impractical for new domains. In contrast, since our pretraining strategy is mainly designed for multi-document summarization, we expect that our approach can quickly adapt to new datasets without the need for significant fine-tuning data. To test this hypothesis, we first provide evaluation results in zero and few-shot settings where the model is provided with no, or only a few (10 and 100)  Comparison To better show the utility of our pretrained models, we compare with three state-of-theart pretrained generation models: BART (Lewis et al., 2020) 5 , PEGASUS (Zhang et al., 2020) and Longformer-Encoder-Decoder(LED) (Beltagy et al., 2020). These pretrained models have been shown to outperform dataset-specific models in summarization (Lewis et al., 2020;Zhang et al., 2020), and because of pretraining, they are expected to also work well in the few-shot settings. As there is no prior work doing few-shot and zeroshot evaluations on all the datasets we consider, and also the results in the few-shot setting might be influenced by sampling variability (especially with only 10 examples) (Bragg et al., 2021), we run the same experiments for the compared models five times with different random seeds (shared with all the models), with the publicly available checkpoints . 6 Similar to Pasunuru et al. (2021), the inputs of all the models are the concatenations of the documents within the clusters (in the same order), each document is truncated based on the input length limit divided by the total number of documents so Models Multi-News (256) Multi-XSci (128) WCEP (50) WikiSum (128) arXiv(300) DUC2004 (128)  Table 2: Zero-shot results. The models in the first block use the full-length attention (O(n 2 )) and are pretrained on the single document datasets. The numbers in the parenthesis following each dataset indicate the output length limit set for inference. PEGASUS means results taken exactly from PEGASUS (Zhang et al., 2020), where available. that all documents are represented in the input. 7 To preserve the same format as the corresponding pretrained models, we set the length limit of output for BART and PEGASUS exactly as their pretrained settings on all of the datasets (except for the zero-shot experiments, the details can be found in Sec.4.3). Regarding length limit of inputs, we tune the baselines by experimenting with 512, 1024, 4096 on Multi-News dataset in few-shot setting (10 data examples), and the model with length limit 512(PEGASUS)/1024(BART) achieves the best performance, thus we use this setting (detailed experiment results for different input lengths can be found in Appx. §C.1). We use the same length limit as our model for the LED model, i.e. 4096/1024 for input and output respectively, for all the datasets.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
For zero-shot 8 abstractive summarization experiments, since the models have not been trained on the downstream datasets, the lengths of generated summaries mostly depend on the pretrained settings. Thus to better control the length of generated summaries and for a fair comparison between all models, following Zhu et al. (2021), we set the 7 Pilot experiments show simple truncation results in inferior performance, which is in line with Pasunuru et al. (2021). 8 For clarity, by zero-shot we mean using the pretrained model directly without any additional supervision. length limit of the output at inference time to the average length of gold summaries. 9 Exploring other approaches to controlling length at inference time (e.g., Wu et al., 2021) is an orthogonal direction, which we leave for future work. Table 2 shows the performance comparison among all the models. Results indicate that our model achieves substantial improvements compared with all the three baselines on most of the datasets. As our model is pretrained on clusters of documents with longer input and output, the benefit is stronger on the dataset with longer summaries, e.g. Multi-News and arXiv. Comparing PEGASUS and BART models, as the objective of PEGASUS is designed mainly for summarization tasks, not surprisingly it has relatively better performances across different datasets. Interestingly, LED underperforms other models, plausibly since part of the positional embeddings (1k to 4k) are not pretrained. Encouragingly, our model performs the best, demonstrating the benefits of our pretraining strategy for multi-document summarization.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Compared with the strict zero-shot scenario, fewshot experiments are closer to the practical scenarios, as it is arguably affordable to label dozens of examples for almost any application.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We fine-tune all of the four models on different subsets with 10 and 100 examples, and the results are shown in Figure 5. (hyperparameter settings in Appx. §D.1) Since R-1, -2, and -L show the same trend, we only present the average of the three metrics in the figure for brevity (full ROUGE scores can be found in Appx. Table 8) To show the generality, all the results of few-shot experiments are the average over 5 runs on different subsets (shared by all the models).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The result of each run is obtained by the 'best' model chosen based on the ROUGE scores on a randomly sampled few-shot validation set with the same number of examples as the training set, which is similar with Zhang et al. (2020). Note that their reported best models have been selected based on the whole validation set which may give PEGA-SUS some advantage. Nevertheless, we argue that sampling few-shot validation sets as we do here is closer to real few-shot scenarios (Bragg et al., 2021).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Our model outperforms all baselines on all of the datasets with 10 and 100 examples demonstrating the benefits of our pretraining strategy and input structure. Comparing the performances of our model with the different number of training data fed in, our model converges faster than other models with as few as 10 data examples.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
To show the advantage of our pretrained model when there is abundant training data, we also train the model with the full training set (hyperparameter settings can be found in Appx. §D.2). Table 3 shows the performance comparison with previous state-of-the-art 10 , along with the results of previous SOTA. We observe that PRIMERA achieves stateof-the-art results on Multi-News, WCEP, and arXiv, while slightly underperforming the prior work on Multi-XScience (R-1). One possible explanation is that in Multi-XScience clusters have less overlapping information than in the corpus on which PRIMERA was pretrained. In particular, the source documents in this dataset are the abstracts of all the publications cited in the related work paragraphs, which might be less similar to each other and the target related work(i.e., their summary) . PRIMERA    outperforms the LED model (State-of-the-art) on the arXiv dataset while using a sequence length 4x shorter (4K in PRIMERA v.s. 16K in LED), further showing that the pretraining and input structure of our model not only works for multi-document summarization, but can be also effective for summarizing single documents having multiple sections.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We conduct ablation studies on the Multi-News dataset in few-shot setting, to validate the contribution of each component in our pretrained models. Input structure: In Figure 6 (a) we observe the effectiveness of both pretraining and the input structure (<doc-sep> tokens between documents and global attention on them). Sentence masking strategy: To isolate the effect of our proposed pretraining approach, we compare with a model with exactly the same architecture when pretrained on the same amount of data but using the PEGASUS (Zhang et al., 2020) masking strategy instead of ours. In other words, we keep all the other settings the same (e.g., data, length limit of input and output, pretraining dataset, input structure, as well as the separators) and only modify the pretraining masking strategy. We run the same experiments under zero-/few-shot scenar-ios on the Multi-News dataset as in §4.2, and the results are shown in Figure 6 (b). The model pretrained with our Entity Pyramid strategy shows a clear improvement under few-shot scenarios.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We also conduct human evaluations to validate the effectiveness of PRIMERA on DUC2007 and TAC2008 (Dang and Owczarzak, 2008) datasets in the few-shot setting (10/10/20 examples for train/valid/test). Both datasets consist of clusters of news articles, and DUC2007 contains longer inputs (25 v.s. 10 documents/cluster) and summaries (250 v.s. 100 words). Since the goal of our method is to enable the model to better aggregate information across documents, we evaluate the content quality of the generated summaries following the original Pyramid human evaluation framework (Nenkova and Passonneau, 2004). In addition, we also evaluate the fluency of generated summaries following the DUC guidelines. 12
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Settings Three annotators 13 are hired to do both Pyramid Evaluation and Fluency evaluation, they harmonize the standards on one of the examples. Specifically, for each data example, we provide three anonymized system generated summaries, along with a list of SCUs. The annotators are asked to find all the covered SCUs for each summary, and score the fluency in terms of Grammaticality, Referential clarity and Structure & Coherence, according to DUC human evaluation guidelines, with a scale 1-5 (worst to best). They are also suggested to make comparison between three generated summaries into consideration when scoring the fluency. To control for the ordering effect of the given summaries, we re-order the three summaries for each data example, and ensure the chance of their appearance in different order is the same (e.g. BART appears as summary A for 7 times, B for 7 times and C for 6 times for both datasets). The instruction for human annotation can be found in Figure 7 and Figure 8 in the appendix. Annotators were aware that annotations will be used solely for computing aggregate human evaluation metrics and reporting in the scientific paper.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Compared Models We compare our model with LED and PEGASUS in human evaluations. Be-12 https://www-nlpir.nist.gov/projects/ duc/duc2007/quality-questions.txt 13 We recruited expert annotators with payment above average of the participants' demographics.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
DUC2007 (20) TAC2008(20) Sr R P F Sr R P F PEGASUS 6.0 2.5 2.4 2.4 8.7 9.1 9.4 9.1 LED 9.6 3.9 4.0 3.8 6.9 7.1 10.8 8.4 PRIMERA 12.5 5.1 5.0 5.0 8.5 8.9 10.0 9.3  cause PEGASUS is a task-specific model for abstractive summarization, and LED has the same architecture and length limits as our model with the parameters inherited from BART, which is more comparable with our model than vanilla BART.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Pyramid Evaluation Both TAC and DUC datasets include SCU (Summary Content Unit) annotations and weights identified by experienced annotators. We then ask 3 annotators to make a binary decision whether each SCU is covered in a candidate summary. Following Nenkova and Passonneau (2004), the raw score of each summary is then computed by the sum of weights of the covered SCUs, i.e. S r = SCU w i I(SCU i ), where I(SCU i ) is an indicator function on whether SCU i is covered by the current summary, and w i is the weight of SCU i . In the original pyramid evaluation, the final score is computed by the ratio of S r to the maximum possible weights with the same number of SCUs as in the generated summaries. However, the total number of SCUs of generated summaries is not available in the simplified annotations in our design. To take consideration of the length of generated summaries and make a fair comparison, instead, we compute Recall, Precision and F-1 score regarding lengths of both gold references and system generated summaries as R= S r len(gold)
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
; P= S r len(sys)
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
; F1= 2 · R · P (R + P )
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Fluency Evaluation Fluency results can be found in Table 5, and PRIMERA has the best performance on both datasets in terms of all aspects. Only for Grammaticality PRIMERA's top performance is matched by PEGASUS.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In this paper, we present PRIMERA a pre-trained model for multi-document summarization. Unlike prior work, PRIMERA minimizes dataset-specific modeling by using a Longformer model pretrained with a novel entity-based sentence masking objective. The pretraining objective is designed to help the model connect and aggregate information across input documents. PRIMERA outperforms prior state-of-the-art pre-trained and datasetspecific models on 6 summarization datasets from 3 different domains, on zero, few-shot, and full fine-tuning setting. PRIMERA's top performance is also revealed by human evaluation.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In zero-shot setting, we can only control the output length of generated summaries at inference time by specifying a length limit during decoding. Exploring a controllable generator in which the desired length can be injected as part of the input is a natural future direction. Besides the summarization task, we would like to explore using PRIMERA for other generation tasks with multiple documents as input, like multi-hop question answering.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
While there is limited risk associated with our work, similar to existing state-of-the-art generation models, there is no guarantee that our model will always generate factual content. Therefore, caution must be exercised when the model is deployed in practical settings. Factuality is an open problem in existing generation models. A Implementation details of pre-training
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
As the multi-document summarization task has a higher compression ratio, defined as len(Summary)/len(Input), (e.g. 12% for Multi-News dataset and 15% for Multi-Xscience dataset), we use 15% as the ratio of masked sentences for generation. In addition to this 15% masked sentences, following PEGASUS (Zhang et al., 2020), we also copy an additional 15% of the input sentences to the output without masking them in the input. This allows the model to also learn to copy information from the source directly and found to be useful by Zhang et al. (2020).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We pretrain the model for 100K steps, with early stopping, batch size of 16, Adam optimizer with a learning rate of 3e−5 following Beltagy et al. (2020), with 10K warmup steps and linear decay. The pretraining process takes likely 7 days on 4 A100 GPUs.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
As the backbone of PRIMERA is the Longformer Encoder Decoder model (LED), it has the same number of parameters with LED (447M).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The details of evaluation datasets can be found below.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Multi-News (Fabbri et al., 2019): A multidocument dataset with summaries written by professional editors from the newser.com.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Wikisum (Liu* et al., 2018) Each summary is a Wikipedia article, and the source documents are either citations in the reference section or the Web Search results of section titles. 14 In our experiments, we use the data crawled by Liu and Lapata (2019a).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
WCEP (Gholipour Ghalandari et al., 2020) is built based on news events from Wikipedia Current Events Portal and the references are obtained similar to Wikisum. There are at most 100 documents within each cluster in the original dataset, thus we remove all the duplicates and only keep up to 10 documents for each cluster based on the relevance score in the original dataset, which is similar to the WCEP-10 variant in the original paper.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Multi-X-Science (Lu et al., 2020) a multidocument summarization dataset created from scientific articles, the summaries are paragraphs of related work section, while source documents include the abstracts of the query and referred papers.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
DUC benchmarks (Dang, 2005) include multidocument summarization datasets in the news domain, with 10-30 documents and 3-4 humanwritten summaries per cluster. Since these datasets are small, we use them primarily for a few-shot evaluation. We use DUC2003 for training (only one of the reference summaries for each document is used for training) and DUC2004 as test.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
ArXiv (Cohan et al., 2018) is a single document summarization dataset in the scientific paper domain. Each document is a scientific paper, and the summary is the corresponding abstract. As each scientific paper consists of multiple sections, we treat each section as a separate document within a cluster in our experiments. This is to evaluate our model's effectiveness on summarizing single documents having multiple sections.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The details of compared models in the zero-/fewshot setting can be found below.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
BART (Lewis et al., 2020) an encoder-decoder transformer model pretrained on the objective of reconstructing the corrupted documents in multiple ways, e.g. Token Deletion, Text Infilling, Sentence Rotation and etc.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
PEGASUS (Zhang et al., 2020) a pretrained model designed for abstractive summarization as the downstream task, especially for the single document input. It is trained on the objective of Gap Sentence Generation on C4 (Raffel et al., 2020) and Hugenews datasets (Note that the pretraining data size in PEGASUS is magnitudes larger than ours). As it is only evaluated on one multi-document summarization dataset (Multi-news), we rerun the model on all the datasets. To verify the quality of our reproduction, the average ROUGE scores of our re-run model vs. 37.9 11.0 17.5 34.9 8.7 17.6 Table 6: The ROUGE score (R-1/R-2/R-3) for pretrained models (BART and PEGASUS) with different input length limit in few-shot setting (10 data example) on the multi-news dataset. The results are the average over 5 runs on different subsets (the same seeds shared with all the other models in this paper).
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Longformer Encoder-Decoder (LED) (Beltagy et al., 2020) is the initial state of our model before pretraining. The parameters of LED are inherited from the BART model, and to enable the model to deal with longer input, the position embeddings are repeatedly copied from BART's 1K position embeddings. It is different from our model with respect to both pretraining and input structure (document separators and global attentions), with global attention on the (<s>) token only and no document separators.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We run an experiment to select the proper length limit for compared pretrained models, i.e. BART and PEGASUS. Specifically, we train both models with different input length limits (512/1024/4096) in the few-shot setting (with 10 data examples) on the multi-news dataset. Similar as the few-shot experiments described in §4.2, we train each model with each specific input length limit for 5 times on different subsets, which are shared by all the models. As shown in Table 6, BART with length limit 1024 performs the best and PEGASUS with length limit 512 performs the best, thus in all our experiments, we use 1024 as the input length limit for BART and 512 for PEGASUS.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We use Adam as the optimizer with linear scheduled learning rate 3e − 5 for BART, LED and our model, and use the default optimization settings of the few-shot experiments from Zhang et al. (2020), i.e. AdaFactor optimizer with scheduled learning rate 5e − 4. For all the experiments with 10 examples, the batch size is 10, the models are trained for 200 steps, with warm-up as 20 steps. For the experiments with 100 examples, we use the same batch size, with the total step and warm-up step set to be 1000 and 100, respectively.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We use Adam as the optimizer with linear scheduled learning rate 3e − 5, and batch size as 16 for all the datasets in the full supervised experiments. The number of steps and warm-up steps are set based on the size of the datasets. The details can be found in Table 7 Dataset  
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The exact ROUGE scores in Figure 5 are shown in Table 8.  
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
To show the advantage of our pre-trained model when there is sufficient data, we also train the model with the full training set, and the results can be found in Table 9-12 15 , along with the results from previous works. Differently from the zero-/few-shot experiments, here we report the state-of-the-art results on different datasets, as they were presented in the corresponding original papers. Since we use the same train/valid/test set as in those prior works, we can perform a fair comparison , without re-running all those extremely time-consuming experiments . Overall, our model achieves state-of-the-art on Multi-News (see Table 9 , WCEP dataset (see Table 11) and arXiv dataset (see Table 12).  Multi-News The experiment results on Multi-News dataset can be found in Table 9. Specifically, the PEGASUS model (Zhang et al., 2020) is pretrained on a large-scale single-document dataset with the Gap Sentence Generation objective, which is the same as ours, but with a different masking strategy, BART-Long (Pasunuru et al., 2021) uses the same model structure as ours , and BART-Long-Graph (Pasunuru et al., 2021) additionally has discourse graph injected. Comparing the results with the BART-Long model, our model is around 1 ROUGE point higher, which may result from either better model structure or pre-training. Interestingly, in one of the ablation studies in Pasunuru et al. (2021), they find that the BART-Long model achieves its best performance with the length limit of 1000, and no further improvement is found when the length limit is greater than that. Thus we may conclude the gap between the performances is mainly from our design on the model, i.e. the document separators, proper global attention as well as the pre-training on a multi-document dataset.   WCEP As for the WCEP dataset, BERTREG (Gholipour Ghalandari et al., 2020) is a Regressionbased sentence ranking system with BERT embedding, which is used as extractive summarization method, while Submodular+Abs is a simple two-step abstractive summarization model with a submodular-based extractive summarizer followed by a bottom-up abstractive summarizer (Gehrmann et al., 2018). DynE is a BART-based abstractive approach, which is to ensemble multiple input, allowing single document summarization models to be directly leveraged on the multi-document summarization task. Our model outperforms all the models by a large margin, including the SOTA model DynE, and it may indicate that the plain structure is more effective than purely ensembling the output of single documents.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
arXiv In addition to the experiments on multidocument summarization datasets, we also compare our fully supervised model with previous works on the arXiv dataset, with each section treated as a single document. All the models to be compared with are based on pre-trained models, and Bigbird-PEGASUS and LED utilize the pre-training of PEGASUS (Zaheer et al., 2020) and BART (Lewis et al., 2020), respectively. However, both Bigbird and LED apply more efficient attentions, which make the models able to take longer  Table 12: ROUGE scores of the previous models and our fully supervised model on the arXiv dataset. The result of PEGASUS and BigBird-PEGASUS are from (Zaheer et al., 2020), and the results of LED are from (Beltagy et al., 2020). The number in the parenthesis indicates the length limit of the input.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
input (3k for BigBird, 4K and 16k for LED). Our model has a better performance than all the models, including LED(16K), which allows for the input 4 times longer than ours. It is worth mentioning that LED(4K) has the same structure as our model, with the same length limit of the input, and with the pre-training on multi-document datasets, our model is more than 3 ROUGE point better than it, which shows that the strategy not only works for multi-document summarization but can also effectively improve single-document summarization for long documents.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
We show an example (from Multi-News) of generated summaries by PRIMERA and compared models trained with different number of examples in Table 13. And we show an example from DUC2007 (which is one of the examples used for human evaluation) with generated summaries by PRIMERA and two compared models in Table 14, with all the models trained on 10 data examples from DUC2007.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Our code is licensed under Apache License 2.0. Our framework dependencies are:
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
• HuggingFace Datasets 16 , Apache 2.0 • NLTK 17 , Apache 2.0 • Numpy 18 , BSD 3-Clause "New" or "Revised" • Spacy 19 , MIT Model Summaries PRIMERA-0 BALTIMORE (AP) -The U.S. Coast Guard says it's looking for a man who has cost the service about $500,000 after responding to nearly 30 of his fake distress calls.. The Coast Guard also said such false distress alerts detract from their ability to respond to actual alerts.Anyone with information about the caller is asked to contact the Coast Guard's regional command center at 410-576-2525 or email investigators at CGIS-Baltimore@uscg.mil..
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The U.S. Coast Guard says it's looking for a man who has cost the service about $500,000 after responding to nearly 30 of his fake distress calls. In a press release published Friday, the Coast Guard says the 28 calls have originated from around the area of Annapolis, Maryland. Each call involved the same male voice and used an emergency radio channel. The two most recent calls were made on the night of July 21 and the early morning of July 22. The Coast Guard also says hoax calls distract rescuers from real emergencies, putting both the public and the responding crews at risk. "Calls like these not only put our crews at risk, but they put the lives of the public at risk," said Lt. Cmdr. Sara Wallace, who heads the response sector in Maryland. "Calls like these not only put our crews at risk, but they put the lives of the public at risk." Anyone with information about the caller is asked to contact the Coast Guard's regional command center at 410-576-2525 or email investigators at CGIS-Baltimore@uscg.mil.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The US Coast Guard is looking for a man who has cost the service about $500,000 after making 28 false distress calls from Maryland. The 28 calls have involved the same male voice and used an emergency radio channel, the AP reports. The Coast Guard says it has responded to nearly 30 of the fake calls and that the estimated cost of the responses is $500,000. The caller has been making the calls since July 2014. He could be facing a six-year prison sentence, a $10,000 civil fine, and a $250,000 criminal fine, the AP reports. "A hoax call is a deadly and serious offense," says Lt. Cmdr. Sara Wallace, head of the response sector in Maryland. "Calls like these not only put our crews at risk, but they put the lives of the public at risk."
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The US Coast Guard has responded to 28 false distress calls in two years-and it's a man who's been making the calls, not the waves. The unidentified caller has made the calls in Annapolis, Maryland, since July 2014, the Coast Guard reports. Each call involved the same male voice and used an emergency radio channel. The two most recent calls were made on the night of July 21 and early morning of July 22. The Coast Guard says the calls were made in the hope that the public would get into the water. Rather than responding to actual alerts, the calls were made to divert the attention of rescue crews from real emergencies, the AP reports. The Coast Guard says the cost of the responses to these false alarms is about $500,000. The caller made 28 false distress alerts from Annapolis, Maryland, beginning in July 2014, according to the Coast Guard. The estimated cost of the responses to these false alarms is $500,000. The Coast Guard says the calls were made in the hope that the public would get into the water. Rather than responding to actual alerts, the calls were made to divert the attention of rescue crews from real emergencies, the AP reports.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The US Coast Guard says it's looking for a man who has cost the service about $500,000 after responding to nearly 30 of his fake distress calls, the AP reports. In a press release published Friday, the Coast Guard says the 28 calls have originated from around the area of Annapolis, Maryland. Each call involved the same male voice and used an emergency radio channel. The two most recent calls were made on the night of July 21 and the early morning of July 22. The Coast Guard also says hoax calls distract rescuers from real emergencies, putting both the public and the responding crews at risk. "A hoax call is a deadly and serious offense," says Lt. Cmdr. Sara Wallace, who heads the response sector in Maryland. "Calls like these not only put our crews at risk, but they put the lives of the public at risk." Making false distress calls is a felony, punishable by a maximum of six years in prison, a $10,000 civil fine, $250,000 criminal fine, and a reimbursement to the Coast Guard for their efforts, the Coast Guard says.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
The US Coast Guard says it's looking for a man who has cost the service about $500,000 after responding to nearly 30 of his fake distress calls, reports the AP. In a press release published Friday, the Coast Guard says the 28 calls have originated from around the area of Annapolis, Maryland. Each call involved the same male voice and used an emergency radio channel. He's been making the calls since July 2014. The two most recent calls were made on the night of July 21 and the early morning of July 22. "A hoax call is a deadly and serious offense," a Coast Guard rep tells NBC4, which notes that such calls are a felony that carry six years in prison, $10,000 civil fine, $250,000 criminal fine, and reimbursement to the Coast Guard. "Calls like these not only put our crews at risk, but they put the lives of the public at risk." 
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In 1996, Congress passed the Line-Item Veto Act, which gave the president the power to cut individual projects from tax and spending bills without vetoing the entire legislation. The act was followed by the President's line-item veto, which he used to trim 144 million dollars from a 248 billion dollars defense spending bill. He also used the veto power to block a congressional rejection of his line-item veto on 38 military construction projects. The bill was passed by the House and the President signed it into law. The veto was challenged by members of both parties who said it was unconstitutional because it gave the president unchecked power to rewrite legislation. The Supreme Court agreed on Friday to hear argument and decide the constitutionality of the president line-item veto. In 1998 the President used his line-item veto to cut $38 million from a military construction bill. In 1999 the President used his line-item veto to cut $54 million from a military spending bill. In 2000 the President used his line-item veto to cut $54 million from a defense spending bill. In January the President vetoed a tax and spending bill, which gave him the power to cut individual projects from tax and spending bills without vetoing the entire legislation. In February the President vetoed a spending bill, which gave him the power to cut individual projects from tax and spending bills without vetoing the entire legislation. In September the President used his line-item veto to cut $54 million from a defense spending bill. The bill was rejected by the House and the President vetoed it. In November the President used his line-item veto to trim 144 million dollars from a defense spending bill.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In 1996, the Republican-led Congress passed the Line Item Veto Act, giving the president the power to delete individual items of spending and tax bills. Clinton used the power to cut individual projects from tax and spending bills. In February 
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In 1996, Congress gave the President the power to veto specific items of tax and spending bills. Before the law's enactment in 1996, the president could veto an entire spending measure but not individual items. The court ruled that such a specialized veto can be authorized only through a constitutional amendment. In January 1997, the line-item veto law was passed. It was passed under the Republican Party's "Contract with Congress". It was passed after President Clinton vetoed thirteen relatively obscure research and spending programs, almost all of the military spending increases approved by Congress. In October 1998, Clinton used his line-item veto authority to have trimmed 144 million U.S. dollars from a 248 billion defense spending bill. In November 1998, Clinton vetoed 38 military construction projects, worth 287 million U.S. dollars. In February 1999, the Justice Department appealed the line-item veto law to the Supreme Court, which agreed to hear argument and decide the constitutionality of the law. Earlier this month, a federal judge struck down the line-item veto law as unconstitutional. The highest court's review will yield a momentous balance of powers ruling. The case is scheduled to be argued before the justices on April 27. The line item veto, strongly supported by President Bill Clinton and a number of his predecessors, was passed in 1996 under the Republican Party's "Contract with Congress". It was passed in January 1997. Before the law's enactment, the only way presidents could reject spending laws was to veto whole budget bills. In 1996, Congress gave the president the power to cancel individual items in tax and spending bills. In January 1997, the line-item veto law was passed. It was passed under the Republican Party's "Contract with Congress". It was passed in January 1997. In 1998, President Clinton threatened to veto some items of the military construction bill because of the increased funding. In November 1998, Clinton used his line-item veto power to delete 38 projects in 24 states worth 287 million U.S. dollars. In February 1999, the Justice Department appealed the line-item veto law to the Supreme Court, which agreed to hear a case about its constitutionality.
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
In 1996 a Republican congress overwhelmingly passed a Line Item Veto Act allowing presidents (including the incumbent Democratic president), to strike individual tax or spending items within 5 days after signing a bill into law. Congress could restore those items in a new bill passed by majority vote. If the president vetoed that bill, Congress could override that veto with a two-thirds majority. Proponents argued that the law preserved the integrity of federal spending, saved billions of dollars, and that it did not repeal any portion of a law, but was simply a delegated spending authorization from Congress. In January 1997, the first year of the law, the president vetoed 163 line-items in six bills, and in 1998 82 line-items in 11 bills. In October 1997 Congress overrode the president's line-item veto against 36 of 38 military construction projects. Initial 1997 efforts by congressmen to challenge the law in the Supreme Court were rejected due to lack of standing. On June 25, 1998 after lower courts rejected the Line Item Veto Act as unconstitutional, on appeal by the White House the Supreme Court ruled 6-3 that Congress unconstitutionally violated the principle of separation of powers, because that procedure allows the president to create a law that was not voted on by either house of Congress in violation of the Constitution's Article I "presentment" clause. A constitutional amendment would be required to institute line item vetoes. Justices Breyer and Scalia argued similar dissenting opinions that separation of powers was not violated. 
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Automatic text summarization aims to distill the most salient content of a given text in a compact form. Recent advances in summarization have been driven by the availability of large-scale datasets such as the CNN-DailyMail (CNNDM) corpus (Nallapati et al., 2016) and the New York Times corpus (Sandhaus, 2008) as well as by the introduction of large pretrained models such as BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2019), in some cases resulting in summaries which are even favored over the human-written reference summaries. Creating data for every new domain, however, is infeasible and highly costly. Thus, the ability to transfer large pretrained models to new domains with little or no in-domain data is necessary, especially as such models make their way into real-world applications.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Unsupervised summarization approaches include autoencoders to mirror the information compression inherent in summarization (Baziotis et al., 2019;Chu and Liu, 2019;Bražinskas et al., 2020b) as well as large-scale pretraining for domainspecific adaptation (Yang et al., 2020). However, little work has focused on domain adaptation in summarization.  examine domain adaptation for extractive summarization. Hua and Wang (2017) showed that summarization models have difficulty generating text in the style of the target domain, while more recently, Zhang et al. (2019) report strong performance of pretrained models when trained in few-shot settings and (Bražinskas et al., 2020a) fine-tune datasetspecific components of a model for few-shot learning. We aim to build on recent work in pretrained models and improve zero-shot and few-shot summarization by encoding characteristics of the target summarization dataset in unsupervised, intermediate fine-tuning data.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Summarization can be seen as a function of subfunctions of the input, called subaspects, which determine the output form. Jung et al. (2019) define three subaspects for summarization: position, importance, and diversity, and study how these subaspects manifest themselves in summarization corpora and model outputs. For example, a common subaspect for the CNNDM dataset is position; earlier sentences tend to constitute a good sum-mary. Inspired by this view of summarization as subaspects, we aim to encode subaspects of a target dataset into unlabeled data to allow a model finetuned on this data to learn characteristics of the target dataset to improve zero-shot and few-shot transfer of the model. In our work, we focus on the subaspects of extractive diversity, as determined by how well an extractive model performs on the data, compression ratio between the source document and summary, and, in the case of CNNDM, the lead bias. We assume knowledge of the target dataset such as the size of input documents, the size of the desired summaries, and the extent to which the summary is abstractive, all of that can be treated as prior knowledge if the task is to be well-defined (Kryscinski et al., 2020). We encode this knowledge into Wikipedia article data by extracting summaries of the desired output length and filtering examples based on the desired level of abstraction.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Our contributions are the following: 1) We introduce a novel method, called WikiTransfer, which creates pseudo-summaries with subaspects of the target dataset which can be used as unlabeled data for intermediate fine-tuning. We show that this method improves zero-shot domain transfer over transfer from other domains, achieving state-of-theart unsupervised abstractive summarization performance on the CNNDM dataset while generalizing to other domains, and we perform extensive hyperparameter studies on the factors influencing zeroshot performance 2) We demonstrate the benefits of WikiTransfer in few-shot settings, and show additional improvements when applying WikiTransfer with data augmentation and a regularization term for training with potentially noisy augmented data. We show robustness in these settings and analyze differences in performance in both automatic and human assessments.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
WikiTransfer Intermediate Fine-tuning: We propose a method for fine-tuning pretrained models using unsupervised Wikipedia data. We create dataset-specific unsupervised data for this intermediate fine-tuning, by making use of characteristics of the target dataset such as the average length of input documents, the average summary length, and the general bin of whether the summaries desired are very abstractive or very extractive, as discussed above. Assume that we want a summary of M sentences from source documents of N sentences on average, and that we know approximately how extractive the summaries are in the target dataset, as defined as the upper bound ROUGE (Lin, 2004) performance of an extractive model, the extractive oracle, on that dataset. We bin the level of extraction of the target summaries into extremely abstractive (ROUGE oracle 10-30), more abstractive (ROUGE oracle 20-30), more extractive (ROUGE oracle 30-50), and extremely extractive (ROUGE oracle 40-60). We then iterate the following procedure on all Wikipedia articles available in a Wikipedia dump: We remove the first M sentences from the Wikipedia article for use as a summary and the following N sentences for use as a source document. Then, we want to check whether this pseudo data point matches the level of extraction of the target dataset. We select the M sentences in the pseudo source document with the highest individual ROUGE scores against the pseudo summary and calculate the ROUGE score between those M sentences concatenated and the pseudo summary, which amounts to a greedy upper bound of the performance of an extractive model on this example. The example will be kept if this ROUGE score falls into the general range of the extractive oracle of the target dataset defined previously and otherwise discarded. We use knowledge of how abstractive a dataset is as a type of summary style which an end-user would know ahead of time. We filter the Wikipedia data points so that only those which fall into the bin for a given dataset are used for fine-tuning. For datasets that are extremely abstractive, such examples may be hard to find, so we remove high-ROUGE sentences from the input until the desired ROUGE oracle score is reached. From here on we refer to data created through this process as WikiTransfer. We then fine-tune a pretrained model on this dataset-specific WikiTransfer data to transfer to a target domain.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Data Augmentation via Round-Trip Translation: In addition to fine-tuning on WikiTransfer data for zero-shot domain transfer, we test the ability of our model to transfer when we have few examples and whether data augmentation further improves these results. In few-shot fine-tuning, we conduct data augmentation to reduce brute-force memorization and introduce a regularization effect. Specifically, we perform round-trip translation (Yu et al., 2018) to generate paraphrases of both the source documents and summaries, as previous work has found this approach creates diverse paraphrase for augmentation while preserving semantic meaning (Yu et al., 2018;Xie et al., 2019). Our examination found that round-trip translation increased the number of novel n-grams while preserving semantic meaning. Given a dataset of N data points, we translate the source and target sentencewise into a non-English language and keep the top k beam hypotheses from beam search as output. We then do likewise for the backtranslation to English. This results in N * k 2 augmented data points in addition to the N original supervised data points. We align a single beam from the translation to non-English text to a single beam in the backtranslation to English; using all combinations of beams for augmented data did not result in an improvement in initial experiments. We refer to the training setting of N supervised data points with this additional augmented data as N-a. Data Augmentation Consistency: While data augmentation may introduce a regularization effect, naively training with augmented data does not necessarily account for noise introduced in the augmented examples. To balance learning from the examples while not overfitting to the small number of supervised samples, the model must learn to be robust to small changes in input examples. We thus investigate the effect of using a consistency loss (Xie et al., 2019;Athiwaratkun et al., 2019) for few-shot training which enforces consistency between the original and round-trip translated documents with respect to the original summary. Let x = {x 1 , x 2 , ..., x i , ..., x n } be a source document with n words and N sentences, where x i represents the i-th word in x. It could also be represented as {s 1 , s 2 , ..., s j , ..., s N }, where s t represents the j-th sentence in x. The corresponding target summary y contains m words and M sentences, and y i denotes the i-th token of y. Standard training, used in the above sections, minimizes the negative log-likelihood loss using supervised teacher forcing (Williams and Zipser, 1989), which we label L sup :
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
where f (·|·, θ) represents the distribution among the vocabulary predicted by our model with parameter θ. In our formulation, the output (summary) distribution given an augmented (round-trip translated) example should not diverge much from the distribution given the original document, with teacher forcing, so that the model learns to be resilient to small perturbations. Letx be a paraphrase of input document x generated via round-trip translation as described in the previous section. In addition to the supervised loss L sup (x, y), we introduce another loss L cons (x,x, y):
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
where KL is the KL divergence, which penalizes the model if the probability distribution of the output using the original input is far from the distribution using the round-trip translated input document. Following Xie et al. (2019), the gradient does not backpropagate through the model for the distribution of the original input while it does propagate through to the round-trip translated input. The total loss L for training with consistency then is:
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We note that the original formulation of Unsupervised Data Augmentation (UDA) (Xie et al., 2019) enforces consistency in a semi-supervised framework. We also experiment with this setup using unlabeled examples from the target dataset with pseudo labels (for teacher forcing) generated by a model trained on the associated few-shot subset, although this approach is very sensitive to the quality of the pseudo labels (see Appendix). We refer to the training setting of N supervised data points with consistency training as N-c.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Datasets: We experiment with four datasets, CN-NDM, XSum (Narayan et al., 2018), Reddit_tifu (Reddit) (Kim et al., 2019), and BigPatent (Sharma et al., 2019). The datasets were chosen as they all differ in their abstractiveness, output length (from one sentence in XSum to on average four in Big-Patent), and cover multiple domains from news (CNNDM and XSum) to social media (Reddit) to patent documents (BigPatent), to show the generalizability of our results. Each of the datasets falls into a different extractive bin, from the most extractive CNNDM to the more abstractive XSum; we discuss these settings further in the Appendix.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Model Selection and Metric: For the experiments which follow, we first choose the model with the best zero-shot performance on a given domain. We test the zero-shot performance from all four domains onto every other domain. For models from our WikiTransfer subset, we choose the best model based on performance on an unsupervised Wiki-Transfer validation subset. We find that fine-tuning the model longer does not result in performance gains in few-shot transfer, and the checkpoints chosen were typically fine-tuned from 2 to 5 epochs. Results from hyperparameter studies for zero-shot transfer from WikiTransfer data are shown on the validation set of that given target dataset. Unless otherwise stated, all results reported are ROUGE-1/2/L. We run all few-shot transfer experiments on five subsets of supervised data, and the reported numbers, unless zero-shot, are the average of the top three results of the five runs following previous work (Gunel et al., 2020). The 10 data point sets are subsets of the 100 data point sets.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Data Augmentation Parameters: For data augmentation via round-trip translation, we use a beam size of 10 and k of 10 on German and Russian translation models; fairseq provides bidirectional pretrained translation models (Edunov et al., 2018) from WMT19  for these language pairs. For both 10 and 100 data points, this resulted in 2010 and 20100 total data points. For consistency loss, we use the same augmented data.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Model Hyperparameters: We use the fairseq codebase  for our experiments. Our base abstractive text summarization model is BART-large (Lewis et al., 2020), a pretrained denoising autoencoder with 336M parameters that builds off of the sequence-to-sequence transformer of Vaswani et al. (2017). We fine-tune BART using a polynomial decay learning rate scheduler using the Adam optimizer (Kingma and Ba, 2015). We mainly vary the learning-rate scheduler, warmup updates, and total updates. As in the previ-ous few-shot summarization work (Zhang et al., 2019) and work in unsupervised machine translation (Conneau and Lample, 2019), we use a subset of the target-domain validation set for early stopping based on the validation loss. We used the following (warmup updates, total updates, learning rate) parameter tuples based on an examination of the validation curves in initial experiments: 10: (25, 100, 3e-5); 10-a: (20, 200, 3e-5); 100 (20, 200, 3e-5); 100-a: (200, 1000, 1e-5). For consistency loss experiments, we use the λ values of 0.1 and 0.5 for experiments with 10 and 100 data points, respectively, chosen manually based on Xie et al. (2019). See the Appendix for more details.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We compare the zero-shot performance of BART fine-tuned on WikiTransfer data to that of one transferred from other summarization datasets. We also show the effect of different choices for WikiTransfer fine-tuning data on CNNDM and XSum.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We aim to show that a model fine-tuned on Wik-iTransfer data has better zero-shot performance than models transferred from other summarization datasets. We fine-tune BART on WikiTransfer data for each of the four target datasets described above and also fine-tune a model on each of the fully-supervised datasets. We compare the zeroshot performance of transferring from WikiTransfer against the best zero-shot transfer performance from another dataset in Table 1. Zero-shot transfer from WikiTransfer notably outperforms transferring from other datasets on CNNDM, XSum, and BigPatent. On Reddit, we perform better on ROUGE-1 and comparably on ROUGE-2/L, which may be due to distinct writing style on Reddit data, as noted in Zhang et al. (2019). We also experimented with training a model on data combined from multiple datasets for zero-shot transfer, but this does not report improved results, so for the experiments which follow we use the best performing single-domain transfer model. Details of the fullysupervised BART models are in the Appendix. We compare our model to the state-of-the-art unsupervised abstractive model on CNNDM in Table 2. We outperform the recently-introduced TED model (Yang et al., 2020) which was specifically motivated for the news domain. We believe the creation of task-specific data from a generic corpus   such as Wikipedia allows for more control over the transfer process than relying on the autoencoder objective of TED, and more generalizable crossdomain results.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We study the effect the characteristics of our intermediate fine-tuning data have on downstream zero-shot performance on CNNDM and XSum to compare highly extractive and abstractive datasets. Effect of learning rate in intermediate finetuning: We examine the extent to which overfitting to the unsupervised WikiTransfer data occurs by examining the effect of the learning rate in intermediate fine-tuning on zero-shot transfer performance. We finetune the models on the CNNDM and XSum WikiTransfer data respectively each with a maximum learning rate of 3e-6 and 3e-5. Results are shown in Table 3  transfer results for XSum and a moderate effect on CNNDM. This is to be expected, as the model otherwise is missing information about XSum's distinctive output style.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We examine how the choice of M affected performance. We set M = 1 for CNNDM and M = 3 for XSum and filtered examples in a similar way based on the extractive bin of the target dataset. We see that the choice of M has a large impact on CN-NDM performance but no decrease on XSum. This result, combined with the effect of filtering examples based on the extractive bin, gives insight into the importance of the subaspect of abstractiveness over compression for XSum performance.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We examined the effect of the size of the Wiki-Transfer data on downstream performance. Results are shown in Table 4. We see a general increase with the addition of more data, although smaller increases after 100k data points and even a decrease in 250k on XSum, likely due to noise variation. The performance with 10k data points on CNNDM is already much closer to the best performance than the XSum case. We believe that this is due to the highly extractive nature of CNNDM, which is especially easy for a model such as BART to learn, as it is pretrained as a denoising autoencoder. For XSum, we see a noticeable improvement from 10k to 100k examples. We suspect that the abstractive objective is harder for the model to learn with small datasets. As we add more examples, we do not see a noticeable improvement. Such observations agree with our observation of the effect of learning rate and overfitting to the easier CNNDM objective. For the remaining experiments, we use 400k data points based on initial experiments. Effect of summary sentence choice: The first M sentences of a given Wikipedia article were chosen as this introduction intuitively form a coherent summary of the article. We examine the effect of choosing the first sentences compared to choosing based    (2019)). As in Zhang et al. (2019), we use ROUGE-1 F1. The sentences chosen under this heuristic consistently corresponded to those which were longest, and the resulting summaries were hence longer. Thus, we also experimented with choosing important sentences by using ROUGE-1 Precision, IND-ORIG-P. The comparison of these methods is shown in Table 5. The choice of the summary sentence has a noticeable impact on performance. We hypothesize that the coherence lost in the summaries is especially important for the longer CNNDM summaries. Using important sentences other than the first sentence likely adds more diversity in the data, and finding a balance between coherence and output style is an interesting direction for additional work (Christensen et al., 2013). The removal of the first sentences may remove too much information in the case of CNNDM, while for XSum, which already has an initial sentence headline removed as the summary, the first sentence may not constitute a very good summary of the remaining document. Wikipedia data often contains multi-paragraph introductions; thus the removal of the first few sentences may still leave a pyramid-structured document with coherent informative content placed at the front. This result supports the emphasis on learning the subaspects of the target domain over simply in-domain training. An analysis of the output of intermediate fine-tuning on CNNDM reveals that the output was more abstractive, due to information present in the summary not being directly stated in the source, than fine-tuning on Wikipedia. We also experiment with further in-domain pretraining of BART before zero-shot transfer, but this does not result in consistent improvements across datasets.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We examine whether zero-shot transfer improvements also carry over to the few-shot setting. Also, we explore the effect of data augmentation and consistency regularization techniques. The results of our experiments with varying training data sizes and augmentation methods for all 4 datasets are shown in Figure 1 and the Appendix.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
10 and 100-shot performance with round-trip translation augmentation: We see that in fewshot settings, without data augmentation or consistency training, our model outperforms transferring from another domain or vanilla BART. In the case of transfer to Reddit, we observe that despite similar zero-shot performance with transfer from CNNDM, there is a more sizeable gap with 10-shot transfer. This suggests that our intermediate finetuning does more closely align the BART model with the target domain. Furthermore, when training on augmented data from round-trip translation, we see the best performance in transfer from Wiki-Transfer in all cases except BART transfer to CN-NDM on 10-aug, which is likely due to the autoencoder pretraining objective of BART which biases it towards copying and lead bias, allowing it to perform well in applications to CNNDM. We see improvements when training with augmented data in 10-example cases and most 100-example cases for WikiTransfer. Less improvement is seen in the 100-aug setting when transferring from BART or another domain. We hypothesize that the noise present in the larger augmented dataset causes this occasional performance drop, while the WikiTransfer models appear more robust to potential noise. We also found model robustness as the standard deviation of top-performing WikiTransfer models was least among all models in the majority of cases. Interestingly, for transfer from BART and another domain 100-aug only improves on CNNDM, the most extractive dataset, while the largest drop in performance from augmented data occurs on XSum. This XSum performance drop may be caused by the high compression in the XSum summaries which leaves less room for noisy output when compared to the longer CNNDM and BigPatent summaries which may still preserve the main meaning of the original summary better despite backtranslation noise. In most cases, 100-aug with WikiTransfer results in the best performance, only several points from the state-of-the-art supervised performance.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Transfer with Consistency Training: We find contrasting trends with the added consistency loss compared to data augmentation via round-trip translation. We note the most sizeable improvements in the more abstractive cases of XSum and Reddit. We hypothesize that the consistency loss promotes better abstraction as the model learns to be invariant to noise which does not change the meaning of the text, and is thus equipped with a better notion of paraphrasing. The consistency loss allows for better training of vanilla BART as well as in general better transfer from other domains than without consistency loss. The loss likely provides a regularization factor which prevents the models from overfitting to the supervised examples. As the WikiTransfer model is already more closely tuned to the target domain, this regularization may not make as large of a difference. This aligns with our observation of WikiTransfer models being more robust to noisy backtranslated data on XSum and
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Reddit. Transfer to Reddit shows similar results across models for consistency loss with 100 examples (better ROUGE-L for WikiTransfer, better ROUGE-1/2 for Reddit); vanilla BART's strong performance at 100 examples suggests that the in- formation provided in this subset is sufficient for good performance, thus diminishing the gains from the head-start the WikiTransfer model provides in zero and 10-shot transfer. We leave aspects of the consistency training such as the role of the quality of the round-trip translation data and its relation to the transfer domain to future work.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We examine how the improved performance from WikiTransfer manifests itself in qualitative annotations when varying the amount of training data. We collect human judgment annotations for two of the four quality dimensions studied in Kryscinski et al. (2019) Table 6: Summary relevance and factual consistency across CNNDM and XSum datasets with varying amounts of training data. All results except those with an asterisks do not differ in a statistically significant way (p-value of 0.05) from the full supervision score. Bold results emphasize the least amount of data to achieve statistically indistinguishable results from the fully-supervised results.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
information should be included in the summary. We did not include fluency as a dimension as an initial inspection of the data found fluency to be of very high quality, and we did not include coherence due to our inclusion of single-sentence XSum summaries where coherence is not a factor. We randomly select 50 examples per dataset and collect the model output from the best-performing zeroshot, 10-aug, 100-aug, and fully supervised models on CNNDM and XSum. The annotator sees the source article and randomly-ordered output from the four models rates the summaries for relevance and consistency on a Likert from 1-5, with 5 being the best score. We averaged the score of two native English-speaking annotators on each example and then across examples, and found moderate and strong annotator correlations for relevance and consistency, respectively. Results are shown in Table 6. For CNNDM, we see an increase in consistency as more training data is added but not a statistically significant difference (using a Student's t-test with a p-value of 0.05) between 100 and full supervision for any of the relevance or consistency results. The relevance of the full model does not outperform the others, likely because the model output was more concise and was judged as not including source information, while the zero-shot output more closely resembles the lead-three bias, so was judged as more informative. For XSum, we see that relevance improves noticeably as more training data is used. We see varied results for consistency, although without statistically significant differences. This fluctuation in scores may be due to the transition of the model from using knowledge from pretraining in its output versus knowledge from the target dataset obtained during fine-tuning, which we discuss in the Appendix.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We introduced WikiTransfer, a novel and generalizable method for fine-tuning pretrained models on dataset-specific unsupervised data obtained from generic Wikipedia data. WikiTransfer models achieve state-of-the-art zero-shot abstractive summarization performance on the CNN-DailyMail dataset and generalize across three additional datasets. In few-shot settings, WikiTransfer models are robust to noise introduced through data augmentation and benefit from consistency loss on more abstractive datasets. Furthermore, human assessments of the resulting summaries do not show significant differences between the WikiTransfer fewshot summaries and fully-supervised summaries, demonstrating the efficiency of our approach.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We make use of existing datasets available through libraries such as huggingface's datasets library. Biases may exist in the datasets, such as political bias in the news datasets as well as gender bias in potentially all of the datasets. Thus, models trained on these datasets may propagate these biases. When used as intended, applying the summarization models described in this paper can save people much time. However, the current models are still prone to producing hallucinated summaries, and in such a case may contribute to misinformation on the internet. Further research is needed for ensuring the faithfulness of abstractive summaries to address this issue, as this issue is present among all current abstractive summarization models. The experiments make use of V100 GPUs. We used up to 8 GPUs per experiment (depending on the experiment; sometimes a single GPU was used to run the maximum number of experiments in parallel). The experiments may take from several minutes in the case of few-shot experiments without augmentation to a couple of hours for the larger augmented datasets, and up to one day for fulldataset training. Over 400 experiments were run due to our requirement of averaging across multiple experiments. Future work should experiment with distilled models for more light-weight training. We note that while our work required extensive experiments to draw sound conclusions, future work will be able to draw on these insights and need not run as many large-scale comparisons, and models in production may be trained once for use using the most promising settings. Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151-5169, Online. Association for Computational Linguistics.    that the model output is stylistically already much like that of the fully-supervised output and gold summary. This stylistic change is also reflected in the change in hallucination; the use of Rachel Jones is likely caused by the appearance of the name of a minister Rachel Haves in an article on Welsh politics found in the 100-aug subset. The model at this point is already fitting strongly to the target domain. For the fully supervised output, we see the use of Carwyn Jones, which does not match the gender of Ms. Jones but which is found 1090 times in the training source documents. Caroline Jones, the actual person in question, only appears 21 times in the training set. This phenomenon points to two interesting research directions for future work, how to properly preserve world knowledge from pretraining and improvement faithfulness to the source text in knowing when to insert world knowledge.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We provide additional details regarding the training and validation of models. We also provide the exact numbers for few-shot transfer in Table 9. WikiTransfer Data: We use the statistics from the original papers to determine the extractive bin of the dataset except for the case of Reddit; upon seeing the strong zero-shot performance of the CNNDM, we investigated the extractive oracle of the Reddit dataset and found it to be much higher (about 31 ROUGE-1) than that stated in the original paper. We select the first M sentences for the pseudo-summaries from Wikipedia except in the case of Reddit, where we choose the IND-ORIG setting; this did not result in a difference in zeroshot performance but upon a qualitative inspection of the output, we found the IND-ORIG to be less biased towards Wikipedia style with the coherence of the summaries not being an issue. We believe that the approximate level of extraction of desired summaries should be treated as prior knowledge. We also examine, however, how many data points are needed to accurately find the extractive oracle bin from target datasets. We found that using 10 data points sufficed to accurately estimate the bin of the extractive oracle.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
Using the first M sentences does not produce ideal summaries of the remaining Wikipedia article, but experiments comparing the WikiTransfer approach on Wikipedia data as opposed to using in-domain data, as well as manual inspection of the data showed the validity of using Wikipedia data for proxy summaries. While the extractive oracle provides some measure of overlap, this heuristic does not ensure deeper semantic overlap or faithfulness between the pseudo summary and the rest of the article. We believe a valuable direction for future work is improving the target-specific data as well as encoding additional semantics and stylebased subaspects into the pseudo summaries. Training and Validation Hyperparameters: We found that full-precision floating-point gave slightly better, and more stable, results, so we report full-precision floating-point numbers. We set a maximum tokens-per-batch of 1024 and use gradient accumulation with an update frequency of 8 for all experiments with 10 data points, and 32 for 10-aug as well as all experiments with 100 (+ augmented) data points. For CNNDM 10 examples, we found it necessary to use a smaller learning rate (3e-6) to avoid immediate overfitting. We per-form validation after each model update, as the models typically converge in under 50 iterations. For the 100-aug setting, we begin validation checking after 50 iterations as the models typically converged around 100 iterations. We train with labelsmoothed cross-entropy (Szegedy et al., 2016) loss for few-shot transfer. We found that models can be sensitive to the choice of hyperparameters in the few-shot settings, hence the averaging over 5 subsets to reduce variation.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We use the standard training and testing splits of each dataset (for Reddit, we use the same 80-10-10% split as in Zhang et al. (2019)), and thus refer the reader to the original papers for detailed statistics. For validation, we used a subset of the target-dataset validation set consisting of 4k examples. While this matches previous unsupervised and transfer settings, we understand that the use of a large validation set is not ideal. We experimented with smaller validation sets on Reddit transfer and found that the results did not change using a validation set of only 10 data points, although we leave a further examination of the effect of validation set size for future work.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We provide the range of the label-smoothed cross-entropy validation loss by taking the average validation loss (over five subsets) from the bestperforming and worst-performing transfer models on a given dataset. The range of validation losses for CNNDM is (4.49, 5.05), for XSum (4.63, 5.45), for Reddit (5.98, 6.65), and for BigPatent (4.88, 6.40).
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
For zero and few-shot transfer, we compare transfer from BART trained on WikiTransfer data to the best-transferring BART model trained on the datasets. The following numbers are ROUGE-1. Our application of BART on fully-supervised data achieves state-of-the-art performance on Reddit (32.74). We perform slightly worse on CNNDM (44.16 vs 45.94 from Dou et al. (2020)). Lower performance when compared to Pegasus-large (Zhang et al., 2019) on XSum (45.14 vs 47.21) and Big-Patent (43.34 vs 53.63) is likely due to differences in capacity and training batch size, as our performance is comparable to Pegasus-base. Our approach is not model-specific to BART, so we leave the application of other models such as Pegasus to future work and do not focus on achieving state-ofthe-art on the fully-supervised individual datasets.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We limit our primary few-shot experiments to  10 and 100 data points, as we are primarily interested in real-world few-shot applications where we likely do not have 1k data points. Initial experiments using 1k and 10k data points on CNNDM showed that WikiTransfer still outperforms transfer from other domains, although both remain below state-of-the-art performance. We leave a further examination of fine-tuning on larger training sets for future work.
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
We experimented with the original formulation of UDA in a semi-supervised setting. In this framework, the label (summary) outputted by the model for an augmented example should be the same as the label of the original document on unlabeled examples. Let x U be an unsupervised source document from the target dataset other than our supervised few-shot examples. Letx U be a paraphrase of input x U generated via round-trip translation as in our above data augmentation experiments. To apply teacher forcing, we require a label y U , which we obtain for each model by applying the model fine-tuned on the analogous few-shot subset. In addition to the supervised loss L sup (x, y), we thus introduce another loss L uda (x U ,x U , y U ) =: m t=1 KL(f (·|y U 0:t−1 , x U )||f (·|y U 0:t−1 ,x U )) (4)
Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
In practice, for an epoch, we iterate through the supervised examples with loss L sup followed by iterating over the unsupervised examples L uda . We sampled 1k unlabeled data points for 10-UDA experiments and 3k unlabeled data points for 100-UDA. Results of initial experiments are shown in Table 10. We find that the performance of the UDA models is very dependent on the quality of the pseudo-labels generated. We chose the model trained on the first data subset of the 5 runs to generate the pseudo-labels and if this model had higher performance then this model likely performed better in UDA (this occurred in our Reddit transfer to CNNDM with 100 data points. As a result, as the quality of the pseudo-labels improves with 100shot training the UDA performance improves and is more comparable to the unaugmented performance in Table 9.