What are the transformer-based methods proposed for dialogue summarizations?
Pre-trained Transformer-based (Vaswani et al.,
2017) language models (Devlin et al., 2019; Rad-
ford et al., 2019; Yang et al., 2019) have become
increasingly popular in natural language process-
ing tasks for tackling the data shortage problem.
However, many of these models have limitations
when it comes to dialogue summarization. Zhang
et al. (2019) propose PEGASUS, which masks mul-
tiple whole sentences and pre-trains sequence-to-
sequence models to reconstruct the original text.
Built on that, Wan and Bansal (2022) improve the
sentence selection strategy and add modules for
ensuring factuality during fine-tuning to address
the problem of factuality in summarization. Phang
et al. (2022) extend PEGASUS with a modified ar-
chitecture and long-sequence pre-training to tackle
long-input summarization. He et al. (2022) propose
ZCode++, a pre-trained language model optimized
for abstractive summarization with improved en-
coder. However, all these methods rely on the Gap
Sentence Selection method, which has limitations
for dialogue summarization. In contrast, our ap-
proach uses pseudo-summary construction as the
pre-training objective, making it possible for zero-
shot dialogue summarization.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present FactPEGASUS, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning: (1) We augment the sentence selection strategy of PEGASUS’s (Zhang et al., 2019) pre-training objective to create pseudo-summaries that are both important and factual; (2) We introduce three complementary components for fine-tuning. The corrector removes hallucinations present in the reference summary, the contrastor uses contrastive learning to better differentiate nonfactual summaries from factual ones, and the connector bridges the gap between the pre-training and fine-tuning for better transfer of knowledge. Experiments on three downstream tasks demonstrate that FactPEGASUS substantially improves factuality evaluated by multiple automatic metrics and humans. Our thorough analysis suggests that FactPEGASUS is more factual than using the original pre-training objective in zero-shot and few-shot settings, retains factual behavior more robustly than strong baselines, and does not rely entirely on becoming more extractive to improve factuality.
[{"first":"David","last":"Wan","middle":[]},{"first":"Mohit","last":"Bansal","middle":[]}]
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples.
[{"first":"Jingqing","last":"Zhang","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Mohammad","last":"Saleh","middle":[]},{"first":"Peter","last":"Liu","middle":["J."]}]
Investigating Efficiently Extending Transformers for Long Input Summarization
While large pretrained Transformer models have proven highly capable at tackling natural language tasks, handling long sequence inputs continues to be a significant challenge. One such task is long input summarization, where inputs are longer than the maximum input context of most pretrained models. Through an extensive set of experiments, we investigate what model architectural changes and pretraining paradigms can most efficiently adapt a pretrained Transformer for long input summarization. We find that a staggered, block-local Transformer with global encoder tokens strikes a good balance of performance and efficiency, and that an additional pretraining phase on long sequences meaningfully improves downstream summarization performance. Based on our findings, we introduce PEGASUS-X, an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. PEGASUS-X achieves strong performance on long input summarization tasks comparable with much larger models while adding few additional parameters and not requiring model parallelism to train.
[{"first":"Jason","last":"Phang","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Peter","last":"Liu","middle":["J."]}]
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state-of-the-art encoder-decoder model using three techniques. First, we use a two-phase pre-training to improve the model’s performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ createsa new state-of-the-art on 9 of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.
[{"first":"Pengcheng","last":"He","middle":[]},{"first":"Baolin","last":"Peng","middle":[]},{"first":"Song","last":"Wang","middle":[]},{"first":"Yang","last":"Liu","middle":[]},{"first":"Ruochen","last":"Xu","middle":[]},{"first":"Hany","last":"Hassan","middle":[]},{"first":"Yu","last":"Shi","middle":[]},{"first":"Chenguang","last":"Zhu","middle":[]},{"first":"Wayne","last":"Xiong","middle":[]},{"first":"Michael","last":"Zeng","middle":[]},{"first":"Jianfeng","last":"Gao","middle":[]},{"first":"Xuedong","last":"Huang","middle":[]}]
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations because they do not take into account the inherent structure of dialogue and rely heavily on labeled data, which can lead to poor performance in new domains. In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain. To pre-train DIONYSUS, we create two pseudo summaries for each dialogue example: one from a fine-tuned summarization model and the other from important dialogue turns. We then choose one of these pseudo summaries based on information distribution differences in different types of dialogues. This selected pseudo summary serves as the objective for pre-training DIONYSUS using a self-supervised approach on a large dialogue corpus. Our experiments show that DIONYSUS outperforms existing methods on six datasets, as demonstrated by its ROUGE scores in zero-shot and few-shot settings
[{"first":"Yu","last":"Li","middle":[]},{"first":"Baolin","last":"Peng","middle":[]},{"first":"Pengcheng","last":"He","middle":[]},{"first":"Michel","last":"Galley","middle":[]},{"first":"Zhou","last":"Yu","middle":[]},{"first":"Jianfeng","last":"Gao","middle":[]}]
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Abstractive summarization aims at generating short summaries that capture the essentials of a long document. Research in this challenging task has made significant progress with the help of large pre-trained models (Lewis et al., 2020;Raffel et al., 2020;Zhang et al., 2020). However, current models suffer from the crucial problem of hallucinations (Maynez et al., 2020), where a summary contains facts or entities not present in the original document. Such unfaithful generation raises the question of whether the models can be trustworthy and used safely for real-world applications. To tackle this problem, many approaches propose postprocessing models (Chen et al., 2021;Dong et al., 2020;Liu and Liu, 2021), but such methods are often constrained by external resources to train additional correction or selection models. An alternative line of works focuses on learning factuality directly during fine-tuning by filtering nonfactual training data (Goyal and Durrett, 2021;Nan et al., 2021) or, most recently, incorporating contrastive learning (Cao and Wang, 2021) to encourage generating faithful summaries.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In this work, we propose FACTPEGASUS, a model that addresses the problem of hallucinations for abstractive summarization holistically, by incorporating factuality into the whole training pipeline: We tackle the lack of factuality objective in pre-training and the presence of hallucinations in the downstream dataset during finetuning. Current pre-training objectives focus on improving the quality of the generated output in the downstream tasks but often overlook the factuality aspect. Thus, we explore incorporating factuality into the pre-training objective of PEGASUS (Zhang et al., 2020) (a state-of-the-art abstractive summarization model). The original objective, gap sentence generation (GSG), transforms any text into a pseudo-summarization dataset by selecting important sentences using ROUGE (Lin, 2004) as output summaries. We explore strategies for combining ROUGE and the factuality metric FactCC (Kryscinski et al., 2020) as the selection criteria, so that the model learns to generate sentences that cover the most important information of the input document as well as remain faithful to it.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Next, we propose three complementary modules that further address factuality problems during fine-tuning: (1) Corrector that removes hallucinations existing in reference summaries, allowing training on the full training set without learning unfaithful behaviors; (2) Contrastor that encourages the model to better differentiate factual summaries from nonfactual ones by paying attention to the document using contrastive learning; (3) Connec-tor, a special mask-token fine-tuning technique enabled by the GSG-style objective, that simulates the pre-training task during fine-tuning by inserting the mask token into the input document so that the pre-trained model can adapt its knowledge of generating factual summaries directly to the downstream tasks. The connector, corrector, and contrastor address the input, output, and training objective of the downstream task, respectively, and the combination of the components reduces potential confounding problems that cannot be addressed by a single module. We show that the full model improves three factuality metrics, the token and sentence error of DEP Entail (Goyal and Durrett, 2021) and FactCC, on the downstream datasets of XSum (Narayan et al., 2018), WikiHow (Koupaee and Wang, 2018), and Gigaword (Rush et al., 2015). Most notably, FACTPEGASUS outperforms existing factualityaware summarization models by more than 40% and 34% on XSum for token error and FactCC, respectively. Ablation studies show the usefulness of each of our fine-tuning components as well as the additive gain of combining our complementary modules, and human evaluation confirms that FACTPEGASUS generates significantly more factual summaries over strong baselines.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Finally, we perform a detailed analysis of FACT-PEGASUS, which points to several important observations regarding learning and maintaining factuality: (1) Zero-shot setting demonstrates the utility of our factuality-aware pre-training objective, as our model outperforms PEGASUS (which uses the original objective) on all three factuality metrics when evaluated directly on the downstream task without any supervised training data. Few-shot experiment indicates that even a small number of nonfactual examples can have a strong negative impact on factuality and can nullify much of the gain from factuality pre-training, highlighting the importance of ensuring factuality during fine-tuning. (2) Factuality dynamics (Goyal et al., 2022) further shows that FACTPEGASUS exhibits a lesser degree of factuality degradation than what is observed for BART-base. (3) Factuality vs abstractiveness tradeoff curve reveals that FACTPEGASUS effectively improves factuality by not simply relying on the increase in extractiveness.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To summarize, our contributions are as follows:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
1. We propose a factuality-aware pre-training objective for abstractive summarization and study the effect of different sentence selection strate-gies on downstream factuality.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
2. We introduce three complementary components for improving factuality during fine-tuning that correct hallucinations present in the training set, discourage unfaithful generation during training, and bridge the gap between pre-training and finetuning. The full model consistently achieves better factuality scores than strong baselines on three downstream abstractive summarization tasks, confirmed by human evaluation.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
3. We conduct thorough factuality analysis and show that FACTPEGASUS generates more factual summaries with no or little supervision, slows down factuality degradation observed for current models, and improves factuality not by becoming more extractive.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We describe our training procedure consisting of pre-training with a factuality-aware objective (Section 3.1) and fine-tuning with three complementary modules for improving factuality (Section 3.2).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Recent exploration of good pre-training objectives for abstractive summarization aims at achieving high quality on downstream tasks, often in terms of ROUGE. However, few have analyzed the effect of pre-training objective on factuality. We focus on incorporating this aspect into the pre-training objective of PEGASUS, gap sentence generation (GSG), since PEGASUS achieves state-of-the-art performance on the downstream abstractive summarization tasks. The GSG objective transforms text documents into a pseudo-summarization dataset by selecting important sentences as the output summary, which are subsequently masked out in the original text. The best strategy determines the importance by calculating ROUGE-1 between each chosen sentence and the rest of the document. While the original strategy selects sentences that contain the most unigram overlap, there is no guarantee that the selected sentences are faithful to the rest of the document. We provide an illustrative example in Figure 1a, where the original objective selects sentence C due to its high ROUGE-1 score. However, this sentence is not a faithful summary to the rest of the document as the other sentences concern with the fire in Seattle while only sentence C talks about the fire in Denver.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To address this problem, we extend this objective, which we call factual GSG (factGSG), where we additionally measure the importance of the sentences according to factuality. We use FactCC (Kryscinski et al., 2020) as the factuality criteria when selecting the summary sentences, as it correlates highly with human factuality judgment (Pagnoni et al., 2021) and is relatively fast to compute. FactCC produces a binary prediction where a score of 1 indicates that the selected sentence is consistent with the rest of the document. Another change in factGSG is the choice of gap sentence ratio, which determines the percentage of sentences in the text that will be selected as the summary. Instead of selecting 30% of the text document as output summary, we only select one sentence, as selecting more sentences will inevitably increase the possibility of hallucinations.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Formally, given a document D of n sentences, D = {x 1 , x 2 , ..., x n }, we select the top-scoring sentence as the output summary, where the score of each sentence x i is calculated by:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Going back to the example in Figure 1a, FactCC assigns a score of 0 to the nonfactual sentence C because the fire in Denver is not entailed by the other sentences. This results in sentence A scoring higher than the nonfactual sentence, and thus overcomes the problem in the original objective.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Although the typical approach of updating all the model's parameters during fine-tuning adapts well to the downstream task, the model suffers from imitative falsehood (Lin et al., 2021): The model learns to generate similar hallucinations present in the downstream dataset, and even completely forgets its factual behaviors learned during pretraining. This is especially problematic for datasets (b) Fine-tuning Figure 1: Illustration of FACTPEGASUS. For pre-training (a), we use the factGSG objective introduced in Section 3.1 that transforms a text document into a pseudo-summarization dataset. We select the pseudo-summary using the combination of ROUGE and FactCC. Here, sentence A is selected as the pseudo-summary, and we mask this sentence in the original text to create the pseudo-document. During fine-tuning (b), the connector (i) simulates the factGSG task by appending the same mask token used in (a) to the input document, so that we have the same setup in both training stages. Then, corrector (ii) removes hallucinations (highlighted in red) from the summary. Finally, contrastive learning in (iii) encourages the model to prefer the corrected summary over the perturbed summary.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
like XSum that contains hallucinations on 70% of the summaries (Maynez et al., 2020).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To this end, we present three complementary fine-tuning modules, illustrated in Figure 1b. Each component addresses different parts of the downstream task and collaboratively ensures factuality throughout the fine-tuning stage.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The GSG objective enables faster and better adaptation during fine-tuning by simulating the downstream task (Zhang et al., 2020). However, there still exists a gap between pre-training and finetuning: GSG is a masked sentence prediction task, but downstream summarization does not make use of the mask token. Thus, we simply insert the mask token into the input document of the downstream dataset, so as to simulate what the model expects during pre-training. This can be seen as a form of prompting, which helps us to elicit the factuality knowledge of the pre-trained models. We insert the mask token between sentences, and the best position is determined by evaluating the summarization performance on the validation set. We report the best position of the mask token and discuss the similarity to prompting in Appendix C.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The corrector removes hallucinations in the reference summaries so that such examples can be used during training without contributing to the problem of imitative falsehood. We consider summary entities as hallucinating if the text cannot be matched to one of the document entities. We propose three approaches with varying degrees of aggressiveness w.r.t. the removal of hallucinations and the possibility of generating ungrammatical sentences.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Replace: Upon qualitative analysis, we discover that some hallucinated entities in the summary are partially present in the documents. The most prominent example is the use of names, where the summary contains the full name of the person while only the first or last name is mentioned in the document, as shown in Figure 2. Given such observation, we propose a method to find a similar entity with the same NER label in the document and use that to replace the original hallucinated entity in the summary. Although this approach cannot correct hallucinations where similar entities are missing in the document, grammaticality is ensured.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Remove: A more aggressive approach is to remove the hallucinated entities in the training examples. The intuition is that it is often better to not say anything than to say something wrong. We mitigate the problem of creating ungrammatical sentences by removing related words to the removed entities determined by dependency arcs.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Combined: As a middle ground that ensures no hallucinations are present in the reference summaries while being grammatical when possible, we first replace all possible entities and then apply the remove strategy on the remaining ones.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We refer the readers to Appendix-B.1 for the details about hallucination detection, as well as the algorithm and discussion of grammatically for the remove method. 
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Arteta has taken up a coaching role at Manchester City.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Mikel Arteta has taken up a coaching role at Manchester City. Combined Former Arsenal midfielder Arteta has taken up a coaching role at Manchester City.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Manuel Pellegrini has taken up a coaching role. Extrinsic
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Former Arsenal midfielder Wenger has taken up a coaching role. Figure 2: Example output using different strategies of corrector and contrastor. The first two rows show the original document and summary with highlighted entities and their respective labels (date, number, ent). We mark hallucinated entities in the summaries with red, factual entities in document and summary with green and underlined, and removed entities by the corrector with a strikethrough. Perturbed entities by the contrastor are italicized.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To better distinguish factual summaries from nonfactual ones, we next introduce a contrastive learning objective that encourages the model to prefer factual summaries given the context of the document. We use the document D i as the anchor and only consider the reference summary S i as the positive sample. Then, we create a set of nonfactual summaries N i to form negative pairs following Kryscinski et al. (2020), where we replace factual entities with random entities of the same named entity labels. We experiment with two variants simulating either extrinsic and intrinsic hallucinations. As formulated in Maynez et al. (2020), extrinsic hallucinations refer to entities that are present in the summary but not in the document, whereas intrinsic hallucinations are those that are present in the document but contain inaccurate information or are misplaced. See Appendix B.2 for more details. We stress that we perform contrastive learning between the document and the summary, similar to Lee et al. (2021), instead of between summaries (Cao and Wang, 2021), as it follows closer to the definition of faithfulness -the summary should be generated within the context of the document.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use the NT-Xent loss (Chen et al., 2020):
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
and z S j are representation for D i , S i and S j , respectively. We generate z D and z S by performing mean pooling over the last hidden layer of the encoder and decoder output, respectively. sim(·, ·) is the cosine similarity between the representations, and τ is the temperature parameter.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The final loss is calculated by the sum of the cross-entropy loss L CE and the contrastive loss:   Table 10.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We describe our experimental setup, and refer to Appendix A for more details.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We pre-train on the C4 dataset (Raffel et al., 2020), and evaluate our pre-trained model on three downstream abstractive summarization datasets: XSum (Narayan et al., 2018), WikiHow (Koupaee and Wang, 2018), and Gigaword (Rush et al., 2015). XSum is the primary dataset for analysis unless otherwise stated, as most of the factuality works for abstractive summarization evaluate on this dataset. Dataset details are presented in Appendix A.1. We report ROUGE-L (Lin, 2004) to evaluate our generated summaries against the reference. However, we note that this method is not ideal given the presence of hallucinations in the reference summaries (Chen et al., 2021;Maynez et al., 2020): If a more factual model does not produce such hallucinations, the output is scored lower than those that contain the same hallucinations found in the reference.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To evaluate factuality, there have been many proposed automatic metrics (Durmus et al., 2020;Wang et al., 2020;Scialom et al., 2021). We report FactCC (Kryscinski et al., 2020) and DEP-Entail (Goyal and Durrett, 2021), as they are highly correlated with human judgment of factuality (Pagnoni et al., 2021). For DEP-Entail, we report the tokenlevel and sentence-level error. For FactCC, since the model has been trained to evaluate on single sentences, we calculate the average score across all sentences for each summary.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To confirm our observation, we conduct human evaluation asking Amazon Mechanical Turk 2 (AMT) to judge the factuality and informativeness of the summaries. We randomly select 100 documents and ask the annotators to check whether each of the generated summaries is factual and informative. Appendix E provides more details.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For pre-training, we use BART-base's architecture with PEGASUS's SentencePiece (Kudo, 2018) unigram model tokenizer. We first determine the best sentence selection criteria by experimenting with selection criteria that use ROUGE-1, ROUGE-2, and ROUGE-L, as well as combining each with FactCC. To save computation (Lewis et al., 2020;Zhang et al., 2020;Raffel et al., 2020), we pretrain these models on a smaller dataset and fewer training steps. We report the effect of the selection criteria using the normalized ROUGE score and factuality scores over the model that uses ROUGE-1 as the selection criteria. We take the complement of token error and sentence error as token accuracy and sentence accuracy, respectively, to present all metrics where higher is better. Details of pretraining are shown in Appendix A.4.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Finally, We evaluate our pre-trained model on the three downstream tasks. As baselines, we compare our model to BART-base and PEGASUS*, our variant of the PEGASUS-base as there is no publicly available checkpoint. We train PEGA-SUS* by using the original sentence selection metric (ROUGE-1), and observe higher ROUGE scores on XSum and WikiHow than the ones reported in the original paper. We also compare FACTPEGA-SUS to two summarization models optimized for factuality. DAE (Goyal and Durrett, 2021) uses 2 https://www.mturk.com/   DEP-Entail to mask out the nonfactual tokens during training, and CLIFF (Cao and Wang, 2021) uses contrastive learning between the reference summaries and automatically generated nonfactual summaries. We apply both methods to BART-base. Details are described in Appendix A.5. Figure 3 shows the effect of different sentence selection criteria. Adding FactCC to all three ROUGE-only criteria consistently improves all factuality metrics at the cost of a small decrease in quality. Overall, the selection strategy of combining ROUGE-1 and FactCC achieves the highest FactCC score out of all strategies while maintaining the smallest relative drop in ROUGE.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present our full result on the three downstream tasks in Table 1. While the two baseline models achieve similar factuality scores, FACTPEGASUS consistently improves factuality over the two baselines on all three datasets. The largest improvement can be seen for the XSum dataset, where FACT-PEGASUS, compared to BART-base, lowers the token error and sentence error by 51% and 36%, respectively, and increases FactCC by 43% 3 . The same trend but to a lesser degree can also be observed for WikiHow and Gigaword, most notably a 3-point decrease in sentence error for WikiHow and a 2-point increase in FactCC for Gigaword.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Compared to factuality-aware models, FACTPE-GASUS achieves the highest FactCC on all tasks. Notably, FACTPEGASUS outperforms DAE by 34% on XSum. In terms of DEP-Entail, FACTPE-GASUS outperforms CLIFF on XSum and Wiki-How. We note that DAE is trained using the DEP-Entail metric and thus is not a fair comparison.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We note that the ROUGE-L scores for FACT-PEGASUS are lower than both baseline models by about 2 points, but we stress that our increase in FactCC is substantially larger than the decrease in ROUGE-L for XSum and Gigaword. The negative relationship between factuality metrics and ROUGE is also reported in prior works (Chen et al., 2021;Kryscinski et al., 2019). For example, finetuning BART on a subset of XSum (Goyal and Durrett, 2021) improves factuality at the cost of a 6-point drop in ROUGE-L 4 , which is triple the amount of decrease observed for our model.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Human Evaluation results are shown in Table 2. The result agrees with our observation on automatic factuality metrics, as FACTPEGASUS produces significantly more factual summaries than the BART-base, and PEGASUS*, and CLIFF. We achieve moderately significantly better summaries (p = 0.055) than DAE. Although, FACTPEGA-SUS achieves low informativeness, we find no statistical significant difference between our model and other models (p > 0.15).  Table 3: Fine-tuning ablation on XSum. We present our pre-trained model factGSG fine-tuned without any of our proposed components, and adding different strategies of corrector and contrastor. We then combine the best of the two modules (corrector combined and contrastor intrinsic), and finally add the connector to form the final model, which we copy from  
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present ablation studies of our proposed methods in Table 3. We first compare the performance of different strategies for the corrector and contrastor. For corrector, the level of aggressiveness in correcting hallucinations has a positive relationship with factuality metrics but a negative relationship with ROUGE-L. Although the remove method achieves the highest FactCC score, the combined method further lowers the token and sentence error while achieving relatively high ROUGE-L and FactCC. For contrastor, simulating intrinsic errors, which creates more challenging negative samples, provides better factuality results than simulating extrinsic ones. Finally, we show the additive gain in combining the best corrector and contrastor, as well as adding the connector to form the final model. We report the same ablation studies for Gigaword and Wikihow in Appendix D.3, and that for PEGASUS* in Appendix D.4.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
With the help of connector proposed in Section 3.2.1, we can explore how knowledge about factuality is transferred to fine-tuning, especially in the zero-shot and few-shot settings 5 .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Zero-Shot. We apply the mask token to the best position and directly analyze the performance of the models on the test set. To better understand the effectiveness in transferring knowledge about summarization and factuality from the pre-training objective, we apply the connector to our pretrained model (factGSG+mask) and PEGASUS* (GSG+mask), so that the two models differ only in their pre-training objective. We report the result in Table 4. FactGSG+mask outperforms GSG+mask on all metrics, especially for factuality metrics. Specifically, factGSG+mask lowers the sentence error by 5 points and increases FactCC by about 10 points. This observation confirms that the factGSG objective is more effective at capturing factuality than the original GSG objective.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Few-Shot. We follow a similar setup in Zhang et al. (2020), where we limit the number of training data to 1, 10, 100, and 1,000, and then fine-tune the model up to 2,000 steps with the patience of 10 epochs for early stopping. We select the checkpoint with the best validation performance.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We conduct this experiment by comparing FACT-PEGASUS to PEGASUS*, which has been shown for its ability to transfer with as little as 100 training examples (Zhang et al., 2020). In addition, we report the performance of factGSG+mask to understand how the the model is affected without explicitly ensuring factuality (i.e. without corrector and contrastor). As shown in Figure 4, connector allows the model to better make use of the knowledge of pre-training and produces highquality summaries, as both FACTPEGASUS and factGSG+mask produces a ROUGE-L score comparable to PEGASUS* trained with 1000 examples.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In terms of factuality, we notice that with just 10 examples, PEGASUS* starts to degrade in factuality, which also applies to the factGSG+mask model. However, FACTPEGASUS demonstrates an opposite trajectory: Sentence error decreases with more training data, and FactCC remains about the same score. This indicates that factual behavior is prone to be overwritten when factuality is not ensured explicitly, and thus calls for the importance of the corrector and contrastor.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To see whether the factuality degradation observed in few-shot experiment also applies to the full fine- tuning process, we extend our analysis by studying the factuality dynamics, similar to Goyal et al. (2022). The authors observe an increase in sentence errors with the BART model during finetuning, and we analyze whether similar factuality degradation occurs for FACTPEGASUS. We save checkpoints of our models every 10% of the total training steps, and evaluate the models on all three factuality metrics. Figure 5 shows the factuality dynamics during fine-tuning. We notice that the degradation occurs for both models but at a different degree. The token and sentence error for BART-base increase by 2 and 8 points, respectively. However, factuality for FACTPEGASUS remains similar, with only an increase of 1 point for token error and 4.8 points for sentence error. The degradation is only about half of what is observed with BART-base, indicating that FACTPEGASUS is better at avoiding learning nonfactual behaviors.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Lastly, we wish to understand whether our proposed method is effectively improving factuality without relying on the increase in extractiveness. To this end, Ladhak et al. (2021) introduces a faithfulness-abstractiveness trade-off curve to measure the faithfulness given the model's extractiveness. The authors kindly provided the same set of examples for Gigaword and AMT template for calculating the faithfulness score.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We show our result on Gigaword in Figure 6. We include the result of their proposed Selector and previous works, including Loss Truncation (Kang and Hashimoto, 2020) and DAE (Goyal and Dur-rett, 2021). We note that the baseline models increase factuality but mostly due to an increase in extractiveness and thus fall below the curve. In contrast, FACTPEGASUS lies above the line, indicating that we are effectively increasing factuality without relying much on becoming more extractive.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
In this work, we proposed FACTPEGASUS, a model for abstractive summarization consisting of factuality-aware pre-training and modules for ensuring factuality during fine-tuning. We demonstrated the effectiveness of our model at improving factuality on three downstream abstractive summarization datasets, confirmed by our human evaluation. Our analysis showed that our proposed factuality-aware pre-training objective is effective at capturing knowledge of factuality compared to the original objective and that our fine-tuning modules reduce the factuality degradation observed with current models. We finally showed that improvement in factuality is not solely explained by the increase of extractiveness.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Our work aims at reducing the risk of generating hallucinations, and even possibly misinformation, for abstractive summarization models so that such models can be used safely for real-world applications. While we demonstrate that we can alleviate this problem, we stress that there is still a long way to go for improving factuality. Thus, we stress that such models should be used with caution for real-world applications. 
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
A.1 Datasets
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Following PEGASUS, we pre-train on the C4 dataset, a large collection of documents from Common Crawl. We evaluate our pre-trained model on three downstream abstractive summarization datasets: XSum, WikiHow, and Gigaword. XSum is a collection of articles from the British Broadcasting Corporation, Gigaword is a large collection of news articles headlines, and WikiHow consists of how-to articles. We show the dataset statistics for pre-training and fine-tuning in Table 5, where we present the number of examples in the training, validation, and test splits. We also show the number of examples corrected using the replace and remove method. All datasets are from datasets (Lhoest et al., 2021).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use the ROUGE package provided by transformers (Wolf et al., 2020). We follow the instructions provided by the authors of the factuality metrics to set up and run their code. We report all scores of our models from single runs.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use transformers library for the training script and the checkpoints of the pre-trained models. We use the default setting, including the AdamW optimizer and the linear rate scheduler. We also use mixed precision for both pre-training and finetuning the models. We conduct our experiments on the RTX A6000 GPU with 48GB memory and the A100 GPU with 40GB memory. BART-base model has 139M parameters, and PEGASUS* and FACTPEGASUS have 175M parameters.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Model Architecture. We use the same architecture as BART-base. Specifically, the model has L = 6, H = 768, F = 3072, A = 12, where L is the number of layers, H is the hidden size, F is the dimension for feed-forward layer, and A is the number of self-attention heads. We use the Sen-tencePiece (Kudo, 2018) unigram model tokenizer from PEGASUS with a vocabulary size of 96,103.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Sentence Selection Criteria. Before pretraining the full model, we first determine the best sentence selection criteria that produces more factual summaries with comparable quality. We experiment with sentence selection criteria that use ROUGE-1, ROUGE-2, and ROUGE-L, as well as combining each with FactCC. To understand the effect of the pre-training objective on factuality directly, we evaluate the performance on the XSum dataset without applying any of our proposed fine-tuning modules. Following Zhang et al.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
(2020), we report the models' relative performance to the base model, which only uses ROUGE-1 as the selection criteria. We use the normalized ROUGE F1 scores 1 3 ( R1 R1 base + R2 R2 base + RL RL base ), where R1 base , R2 base , and RL base are the ROUGE F1 scores of the base model. We similarly report the factuality metrics by normalizing each score by that of the base model. We take the complement of token error and sentence error as token accuracy and sentence accuracy, respectively, to present all metrics where higher is better.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Similar to previous works (Lewis et al., 2020;Zhang et al., 2020;Raffel et al., 2020) that save computational resources when selecting strategies for pre-training, we pre-train these model on the realnewslike subset of the C4 dataset with less steps.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Pre-training Details. We use a learning rate of 1e-4, a weight decay of 0.01, and set the maximum number of input tokens to be 512 and a maximum number of output tokens to be 256. We use a batch size of 256. We pre-train the full model for 750,000 steps with a warm-up of 20,000 steps, and only pretrain the smaller models for the sentence selection criteria experiment for 250,000 steps. Pre-training the smaller models takes 30 hours, and pre-training the full model takes 90 hours.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Calculating FactCC Score. In practice, running FactCC on each sentence-document pair of the pretraining data is expensive. Thus, we opt to only calculate the FactCC score for the top 5 sentences  according to the ROUGE score between the sentence and the rest of the document.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For all datasets, we use a label smoothing of 0.1. For decoding, we use a beam size of 6 for all datasets. Task-specific hyper-parameters are shown in Table 6. Fine-tuning on XSum and WikiHow takes 8 hours, and fine-tuning on Gigaword takes 11 hours. Decoding on XSum and Gigaword takes half an hour, while decoding WikiHow takes an hour. We use 5 negative examples for the contrastor and set λ to 5 when calculating the combined loss. We set the temperature τ to 0.05. For fine-tuning DAE and CLIFF, we follow the authors' instructions and fine-tune BART-base with their respective code and hyper-parameters. For WikiHow and Gigaword, we use the same hyperparameters as above.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We use spaCy's NER model 6 to find entities in the document and summary. Entities in the summary sentence are considered nonfactual if no matching document entities with the same string are found. We have previously experimented with the additional requirement of matching entity type similar to Kryscinski et al. (2020), but we find that this constraint unintentionally causes some correct entities to be considered hallucinating, leading to unnecessarily less informative summaries when removed. Given hallucinated entities, we can perform either replace or remove operations. For replace, we find document entities whose words are all contained in the selected entity.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For the remove method, we need to make sure to also remove any related words. We use spaCy's 6 We use the en_core_web_trf model. dependency parser to systematically remove those. The algorithm is as follows: We first add all the tokens in the selected hallucinated entity to the list of tokens to remove. Then, we recursively find all parents that contain the dependency relation of pobj and prep without any other children and add those to the tokens to remove. Finally, we add all children that do not have the label compound, relcl, and fixed. The final set of words will then be removed in the summary sentence.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We qualitatively observe that this approach can cover most of the edge cases that would otherwise result in ungrammatical sentences. Nevertheless, this method is not perfect. We include some sample output with the remove method in Figure 7. The algorithm is good at removing entities and related words, such as prepositions, as illustrated in example 1, 3, and 5. However, we observe that it will create ungrammatical sentences when the hallucinated entity is the subject (example 2), or the object of a transitive verb (example 6).
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We leave exploration with the best systematic correction algorithm or models for future work.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Similar to Kryscinski et al. (2020), we generate hallucinated summaries by performing entity perturbation on the original summaries. We find entity candidates using the NER labels and sort them into three categories: We include MONEY, QUAN-TITY, and CARDINAL as number, DATE and TIME as date, and all other labels as named entities. We randomly select a factual entity in the summary and replace it with an entity belonging to the same category.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
For extrinsic hallucinations, we sample candidates of the same category from the training corpus but exclude those present in the document. For the intrinsic case, we select to consider the entities from the document. The number of negative examples for all tasks is 5.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Learning rate Num Steps Warmup Batch size Max Input tokens Max Target tokens   XSum  3e-5  15k  500  256  512  64  WikiHow  3e-5  15k  500  256  512  256  Gigaword  3e-5  50k  2000  256 128 32  
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
This mask-token fine-tuning technique can be seen as a form of prompting, where we elicit our desired faithful abstractive summarization behavior from the pre-trained model directly. Specifically, we consider this as null-prompting (Logan et al., 2021), where using the mask token as the prompt can achieve competitive results with manually engineered prompts. Conveniently, since the mask token during pre-training already serves as a placeholder of where the summary sentence should be generated, it naturally serves as a valid prompt. Figure 1b shows an example of adding the mask token before the first sentence and thus creating a similar setup for pre-training.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We first need to determine the best position of mask token, as discussed in Section 3.2.1, where we insert the mask token before the ith sentence of the document, where i = 1, 2, ..., 6, and select the best position that achieves the highest ROUGE score on the dev collection. We report ROUGE score of all positions in Table 7 for the three datasets. Interestingly, we observe that the best mask token position for all datasets is before the first sentence. This agrees with the dataset generation of XSum: the summary is taken from the first sentence of the original article. For Gigaword, there is not a change after the first sentence, since the document only consists of a single sentence.   Table 9: Full Result of zero-shot and few-shot experiments.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We report the full result for the sentence selection criteria in Table 10. Surprisingly, each sentence selection criteria that uses FactCC excels in one specific factuality metric: R1+FactCC is best at FactCC, R2+FactCC is best at sentence error, and RL+FactCC is best for token error.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present additional results of the zero-shot and few-shot experiments here.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Zero-shot We first report the reference-based result of the two baseline models and FACTPEGA-  Table 10: Full result of pre-trained models with different sentence selection criteria shown in Figure 3. We denote the criteria with FactCC with (+FC). Table 8. Due to the mismatch of pretraining and fine-tuning, we observe that both baseline models perform much worse than their result when fully trained. However, with the help of the connector, we observe 11.5 ROUGE-1 points increase for our model compared to the baseline models, and almost four times and double the score for ROUGE-2 and ROUGE-L, respectively.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Few-shot We show FACTPEGASUS's full result of the few-shot experiment in Table 9.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We report ablation of each fine-tuning components on Gigaword and Wikihow. The result can be found in Table 11. We observe similar trend as Table 3, where each component improves the performance. For WikiHow and Gigaword, the extrinsic method for contrastive learning perform the best. We think that this is due to the fact that the two tasks do not contain rich entities in the document, and thus require introduction of additional entities from the training corpus.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We similarly perform the same ablation using the PEGASUS* model, which we present in Table 12.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We observe similar trend as Table 3. We note that using our pre-trained model factGSG achieves better factuality than PEGASUS* in each setting.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To ensure high-quality annotations, we select the workers from the United States and have more than 10,000 number of HITS approved as well as an approval rate greater than 98%. In addition, we also create a qualification test where we rate the factuality of the selected generated summaries. Such examples include cases where some summaries hallucinate the first name of a person, which the workers should mark them as not factual. Only workers with the correct annotation can perform the actual task.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
To avoid giving too much text to the workers, we select the most important sentences and replace the less relevant sentences with an ellipsis. For each of the summaries, we select the ten most relevant sentences from the document by cosine similarity of the sentence embedding using SentenceTransformer 7 (Reimers and Gurevych, 2019). We combine and show all the selected relevant sentences from each summary. Since the summaries are similar, we see a large overlap of the relevant sentences.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We give the following prompt, which we modify from Dreyer et al. (2021): • consistency/factuality: Please avoid using general knowledge, and only consider it in the context of the provided document. Select not consistent if facts in the summary are not supported by the document, such as cases like these:
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
1. The summary contradicts the information in the document. The summary might say "A fire broke out in Seattle", but a document says it broke out in Portland. Or the summary might say "the Republicans won the election", but the document indicates the Democrats won instead 2. The summary adds (hallucinates) a fact that is not mentioned anywhere in the document. For example, the summary might say that "A fire broke out at 2 am", but the document doesn't mention the time when the fire broke out.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
• Informativeness: Please select informative if the summary expresses the main points of the document. Summary should contain relevant and important information and few unimportant details. If you select the summary to be not consistent with the document, please only consider the consistent information when evaluating this category.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
The order of the summary is randomly shuffled. Each task consists of three unique workers, where we take the mean as the scores for this document. The final score is the mean factuality score across all documents. The average time for each task is 7 We use the all-mpnet-base-v2 model.   Table 12: Fine-tuning ablation on XSum using PEGA-SUS*. We combine the modules by using the corrector combined and contrastor intrinsic. We name the model with all three components as PEGASUS* full.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
around 3 minutes and we pay 0.6 USD per task, hence an hourly rate of ≥ $12 per hour. We use boostrap test (Efron and Tibshirani, 1993) to determine statistical significance between two summaries with p < 0.05.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We include some sample outputs in Figure 8 and Figure 9.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Example 1 (XSum) Summary: Tap water in 80,000 homes in Lancashire has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water for three weeks. Remove: Tap water in homes has been declared safe to drink, after the discovery of a parasite at a treatment works left residents boiling water .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Example 2 (XSum) Summary: Great Britain's Becky James won her second Olympic silver of Rio 2016 by finishing second in the women's sprint. Remove: Won her second Olympic silver of Rio 2016 by finishing second in the women's sprint.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic.Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye in Sharpies is usually weak against it. Remove: Use a "magic eraser.", Try a regular eraser. Make an alcohol solution. Apply a dry erase marker over the permanent marker. Try nail polish remover or acetone on plastic.Acetone and nail polish removers containing acetone can strip away most paints and dyes, and the dye is usually weak against it.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: Call 911 if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your daily routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys. Remove: Call if you think you are in danger. Cut through your denial. Do not confront your stalker. Tell other people. Keep a record of stalking activities. Get a restraining order. Be prepared to defend yourself. Change your routine. Change your passwords. Use anti-virus software. Clean up your social network. Make your online materials private. Notice signs of control in your relationship. Make your break up clear. Cut off contact. Get back your keys.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Summary: xinhua summary of asia-pacific stocks news on tuesday feburary ## Remove: summary of stocks news on tuesday ## Example 6 (Gigaword) Summary: cuba urges eu to drop its common position Remove: cuba urges to drop its common position Figure 7: Summaries changed using the corrector. We mark hallucinated entities in the summaries with red.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Article: The Scots started their Six Nations campaign with a thrilling first win over Ireland in four years. They beat France for the first time in 10 years last season at home, but have lost on their past nine trips to Paris. "It is a long time ago since we won there but we understand there is a reason for that," said Laidlaw. "France are a good team and it is a hard place to win. We are under no illusions this week -we need to be on top form to get something out of the game. "We'll need accuracy and we need to be physical and aggressive because the French are big men. But we need to be subtle and clever in our attack. "If we do that, and move France about and play for 80 minutes, I would back our fitness against them. But we obviously need to be in the game right to the death and not let France get any momentum." Media playback is not supported on this device France have not finished in the top half of the Six Nations since 2011, but impressed in narrow defeats by New Zealand and Australia in the autumn, and came close to a shock victory against England at Twickenham in their Six Nations opener. "They have improved massively under [coach] Guy Noves and have not been far away from any team they have played," Laidlaw noted. "They have some really dangerous strike runners throughout the team. "As much as we need to understand why we beat them last year, we need to develop that, be clever and smart and be able to score points." We need to keep doing what we are doing well, keep our skills high and if we do that again this weekend, we are going to put ourselves in a strong position. "The start of the game is massively important, whether it is our kick-off, or or us receiving. It is very important psychologically for them as it is for us. "If we can start the game playing in the right areas of the field for the first 20 minutes and not muck about too much in our own half, hopefully we will put them under pressure and upset the crowd a little bit." Media playback is not supported on this device Reference: Captain Greg Laidlaw says Scotland must be "subtle and clever" as well as match France physically if they are to secure a first win in Paris since 1999. Bart-base: Scotland head coach Greig Laidlaw says his side are "under no illusions" when they face France at Twickenham on Saturday. PEGASUS*: Scotland coach Steve Laidlaw has warned France that they need to be "smart and smart" in their attack if they are to win their first Six Nations game of the season against the French in Paris on Saturday. CLIFF: Scotland head coach Mark Laidlaw says his side need to be " subtle and clever" in their attack against France in Paris onSaturday. DAE:Scotland must be " subtle and clever" in their attack when they face France at Twickenham on Saturday, says head coach Gordon Laidlaw. FACTPEGASUS: Scotland captain Laidlaw says his side need to be "subtle and clever" in their attack if they are to beat France.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Article: the investigating magistrate and prosecutor handling the case involving the french association , zoe ' s ark , arrived tuesday in the eastern town of abeche where they met the ### children the association was attempting to fly to france , according to reports .
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
Reference: chadian court officials go to UNK to investigation into children 's case Bart-base: zoe's ark investigating magistrate arrives in abeche PEGASUS*: prosecutor in zoe's ark case arrives in abeche CLIFF: prosecutor meets zoe 's ark children DAE: zoe's ark investigating magistrate prosecutor arrive in abeche FACTPEGASUS: prosecutor arrives in abeche to investigate french association 
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Text summarization aims at generating accurate and concise summaries from input document(s). In contrast to extractive summarization which merely copies informative fragments from the input, abstractive summarization may generate novel words. A good abstractive summary covers principal information in the input and is linguistically fluent.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In abstractive summarization, sequence-to-sequence (Sutskever et al., 2014) has become a dominant framework using encoder-decoder architectures based on RNNs (Chung et al., 2014;Hochreiter & Schmidhuber, 1997) and more recently Transformers (Vaswani et al., 2017). Most prior work on neural abstractive summarization relied on large-scale, high-quality datasets of supervised document-summary pairs (Hermann et al., 2015) and achieved promising results (Rush et al., 2015;Nallapati et al., 2016;See et al., 2017). In recent years, there has been increased interest in collecting new summarization datasets that have more abstractive summaries (Narayan et al., 2018), have longer documents, (Cohan et al., 2018;Sharma et al., 2019), utilize multiple documents (Fabbri et al., 2019), and are sourced from diverse domains (Grusky et al., 2018;Koupaee & Wang, 2018;Kim et al., 2019;Kornilova & Eidelman, 2019;Zhang & Tetreault, 2019); however, there has been little work on systematic evaluation of models across these broad settings.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Contemporaneously, the adoption of Transformer models (Vaswani et al., 2017) pre-trained using self-supervised objectives on large text corpora (Radford et al., 2018a;Devlin et al., 2019) have improved performance on many NLP tasks Rajpurkar et al., 2016).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Recent work leveraging such pre-training for Transformerbased sequence-to-sequence models (Dong et al., 2019;Song et al., 2019;Rothe et al., 2019;Lewis et al., 2019;Raffel et al., 2019) has extended the success to text generation, including abstractive summarization.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this work, we study pre-training objectives specifically for abstractive text summarization and evaluate on 12 downstream datasets spanning news (Hermann et al., 2015;Narayan et al., 2018;Grusky et al., 2018;Rush et al., 2015;Fabbri et al., 2019), science (Cohan et al., 2018), short stories (Kim et al., 2019), instructions (Koupaee & Wang, 2018), emails (Zhang & Tetreault, 2019), patents (Sharma et al., 2019), and legislative bills (Kornilova & Eidelman, 2019). We find that masking whole sentences from a document and generating these gap-sentences from the rest of the document works well as a pre-training objective for downstream summarization tasks. In particular, choosing putatively important sentences outperforms lead or randomly selected ones. We hypothesize this objective is suitable for abstractive summarization as it closely resembles the downstream task, encouraging whole-document understanding and summary-like generation. We call this self-supervised objective Gap Sentences Generation (GSG). Using GSG to pre-train a Transformer encoder-decoder on large corpora of documents (Web and news articles) results in our method, Pre-training with Extracted Gap-sentences for Abstractive SUmmarization Sequence-to-sequence models, or PEGASUS.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
With our best 568M parameter model trained on the recently introduced C4 (Raffel et al., 2019) corpus we equal or exceed state-of-the-art on the 12 summarization tasks we consider. We further push forward the state-of-the-art using a newly collected text corpus comprised of news-like articles we call HugeNews, including the highly competitive XSum and CNN/DailyMail summarization datasets.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Large-scale document-summary datasets are rare and in practice there is a mismatch between research datasets and real-world use-cases where collecting summaries is expensive; the most common setting is that of low-resource summarization. We simulate this setting and show that our model is able to adapt very quickly when fine-tuning with small numbers of supervised pairs, obtaining state-of-the-art results in 6 datasets with only 1000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Qualitatively we observed high quality outputs from our best models and validated this in human evaluation studies. We found that PEGASUS summaries are at least as good as reference summaries for the datasets we assessed -XSum, CNN/DailyMail, and Reddit TIFU -even at low-levels of supervision.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To summarize our contributions:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We propose a new self-supervised pre-training objective for abstractive summarization, gap-sentences generation, and study strategies for selecting those sentences.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We evaluate the proposed pre-training objective on a broad range of downstream summarization tasks, with careful ablations to choose the best model settings, which we use to train a 568M parameter PEGASUS model that surpasses or is on-par with the state-of-theart on all 12 downstream datasets considered.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We show how good abstractive summarization performance can be achieved across broad domains with very little supervision by fine-tuning the PEGASUS model and surpassing previous state-of-the-art results on many tasks with as little as 1000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We conducted human evaluation studies to validate our experimental design and demonstrate human-level summarization performance on XSum, CNN/DailyMail, and Reddit TIFU.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We propose a new pre-training objective, GSG, in this work, but for comparison, we also evaluate BERT's maskedlanguage model objective, in isolation and in conjunction with GSG.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We hypothesize that using a pre-training objective that more closely resembles the downstream task leads to better and faster fine-tuning performance. Given our intended use for abstractive summarization, our proposed pre-training objective involves generating summary-like text from an input document. In order to leverage massive text corpora for pretraining, we design a sequence-to-sequence self-supervised objective in the absence of abstactive summaries. A naive option would be to pre-train as an extractive summarizer; however, such a procedure would only train a model to copy sentences, thus not suitable for abstractive summarization.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Inspired by recent success in masking words and contiguous spans (Joshi et al., 2019;Raffel et al., 2019), we select and mask whole sentences from documents, and concatenate the gap-sentences into a pseudo-summary. The corresponding position of each selected gap sentence is replaced by a mask token [MASK1] to inform the model. Gap sentences ratio, or GSR, refers to the number of selected gap sentences to the total number of sentences in the document, which is similar to mask rate in other works.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To even more closely approximate a summary, we select sentences that appear to be important/principal to the document. The resulting objective has both the empirically demonstrated benefits of masking, and anticipates the form of the downstream task.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We consider 3 primary strategies for selecting m gap sentences without replacement from a document, D = {x i } n , comprised of n sentences:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Random Uniformly select m sentences at random.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Lead Select the first m sentences.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Principal Select top-m scored sentences according to importance. As a proxy for importance we compute ROUGE1-F1 (Lin, 2004) between the sentence and the rest of the document,
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this formulation sentences are scored independently (Ind) and the top m selected. We also consider selecting them sequentially (Seq) as in Nallapati et al. (2017) by greedily maximizing the ROUGE1-F1 between selected sentences, S ∪ {x i }, and remaining sentences, D \ (S ∪ {x i }) as in Algorithm 1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Algorithm 1 Sequential Sentence Selection 1: S := ∅ 2: for j ← 1 to m do 3:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
When calculating ROUGE1-F1, we also consider n-grams as a set (Uniq) instead of double-counting identical n-grams as in the original implementation (Orig). This results in four variants of the principal sentence selection strategy, choosing Ind/Seq and Orig/Uniq options.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
An example containing lead, random and principal gap sentence selection strategies are shown in Figure 2.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
INVITATION ONLY We are very excited to be co-hosting a major drinks reception with our friends at Progress. This event will sell out, so make sure to register at the link above. Speakers include Rajesh Agrawal, the London Deputy Mayor for Business, Alison McGovern, the Chair of Progress, and Seema Malhotra MP. Huge thanks to the our friends at the ACCA, who have supported this event. The Labour Business Fringe at this year's Labour Annual Conference is being co-sponsored by Labour in the City and the Industry Forum. Speakers include John McDonnell, Shadow Chancellor, and Rebecca Long-Bailey, the Shadow Chief Secretary to the Treasury, and our own Chair, Kitty Ussher. Attendance is free, and refreshments will be provided. Figure 2: An example of sentences (from the C4 corpus) selected by Random, Lead and Ind-Orig respectively. Best viewed in color.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Following BERT, we select 15% tokens in the input text, and the selected tokens are (1) 80% of time replaced by a mask token [MASK2], or (2) 10% of time replaced by a random token, or (3) 10% of time unchanged. We apply MLM to train the Transformer encoder as the sole pre-training objective or along with GSG. When MLM is the sole pre-training objective, the Transformer decoder shares all parameters with encoder when fine-tuning on downstream tasks following Rothe et al. (2019). 
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
For pre-training we considered two large text corpora:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• C4, or the Colossal and Cleaned version of Common Crawl, introduced in Raffel et al. (2019); consists of text from 350M Web-pages (750GB).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• HugeNews, a dataset of 1.5B articles (3.8TB) collected from news and news-like websites from 2013-2019. A whitelist of domains ranging from highquality news publishers to lower-quality sites such as high-school newspapers, and blogs was curated and used to seed a web-crawler. Heuristics were used to identify news-like articles, and only the main article text was extracted as plain text.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
For downstream summarization, we only used public abstractive summarization datasets, and access them through TensorFlow Summarization Datasets 1 , which provides publicly reproducible code for dataset processing and train/validation/test splits. We used train/validation/test ratio of 80/10/10 if no split was provided, and 10% train split as validation if there was no validation split.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
XSum (Narayan et al., 2018) consists of 227k BBC articles from 2010 to 2017 covering a wide variety of subjects along with professionally written single-sentence summaries.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
CNN/DailyMail (Hermann et al., 2015) dataset contains 93k articles from the CNN, and 220k articles the Daily Mail newspapers. Both publishers supplement their articles with bullet point summaries. We use the non-anonymized variant used in See et al. (2017).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
NEWSROOM (Grusky et al., 2018) is a large dataset containing 1.3M article-summary pairs written by authors and editors in the newsrooms of 38 major publications between 1998 and 2017.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Multi-News (Fabbri et al., 2019) is a multi-document summarization dataset consisting of 56k pairs of news articles and their human-written summaries from the site newser.com.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Gigaword (Rush et al., 2015) contains 4M examples extracted from news articles (seven publishers) from the Gigaword corpus (Graff et al., 2003). The task is to generate the headline from the first sentence.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
arXiv, PubMed (Cohan et al., 2018) are two long document datasets of scientific publications from arXiv.org (113k) and PubMed (215k). The task is to generate the abstract from the paper body.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
BIGPATENT (Sharma et al., 2019) consists of 1.3 million U.S. patents along with human summaries under nine patent classification categories.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
WikiHow (Koupaee & Wang, 2018) is a large-scale dataset of instructions from the online WikiHow.com website. Each of 200k examples consists of multiple instruction-step paragraphs along with a summarizing sentence. The task is to generate the concatenated summary-sentences from the paragraphs.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Reddit TIFU (Kim et al., 2019) contains 120K posts of informal stories from the online discussion forum Reddit, more specifically the TIFU sub-reddit from 2013-Jan to 2018-Mar. The sub-reddit posts strictly follow the rule of writing a descriptive "TL;DR" summary and has higher qual-ity than (Völske et al., 2017) (which used more subreddits) based on our manual inspection. We uses the TIFU-long subset (using TLDR as summaries) in the work.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
AESLC (Zhang & Tetreault, 2019) consists of 18k email bodies and their subjects from the Enron corpus (Klimt & Yang, 2004), a collection of email messages of employees in the Enron Corporation.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
BillSum ( We used sinusoidal positional encoding following Vaswani et al. (2017). For optimization, both pre-training and finetuning used Adafactor (Shazeer & Stern, 2018) with square root learning rate decay and dropout rate of 0.1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We used greedy-decoding for studies in Section 6.1, and used beam-search with a length-penalty, α, as in Wu et al. (2016) for the final large model.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
All experiments' hyper parameters can be found in Appendix C and reported numbers are in Appendix D and E.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We used PEGASUS BASE to evaluate choices of pre-training corpus, pre-training objective, and vocabulary size. For reproducibility, we evaluated the latter two using the publicly available C4 corpus.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Note that the y-axis in Figures 3, 4 Figure 3: Effect of pre-training corpus. PEGASUS BASE pre-trained on C4 (350M Web-pages) and HugeNews (1.5B news-like documents). Figure 3 shows that pre-training on HugeNews was more effective than C4 on the two news downstream datasets, while the non-news informal datasets (WikiHow and Reddit TIFU) prefer the pre-training on C4. This suggests pretraining models transfer more effectively to downstream tasks when their domains are aligned better.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
GSG We compared six variants of GSG (Lead, Random, Ind-Orig, Ind-Uniq, Seq-Orig, Seq-Uniq) while choosing 30% sentences as gap sentences. As shown in Figure 4a, Ind-Orig achieved the best performance followed by Seq-Uniq. Ind-Orig and Seq-Uniq were consistently better (or similar) than Random and Lead across the four downstream datasets. Lead had decent performance on the two news datasets but was significantly worse on the two non-news datasets, which agrees findings of lead bias in news datasets (See et al., 2017;Zhong et al., 2019). The results suggest choosing principal sentences works best for downstream summarization tasks, and we chose Ind-Orig for the PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
A significant hyper-parameter in GSG is the gap-sentences ratio (GSR). A low GSR makes the pre-training less challenging and computationally efficient. On the other hand, choosing gap sentences at a high GSR loses contextual in-  formation necessary to guide the generation. We compared GSRs from 15% to 75%. For a fair comparison, the original documents were truncated to have up to 400 words. The maximum input length, L input in the encoder and the maximum target length, L target in the decoder were set as 512 tokens. Figure 4b shows that different downstream datasets had slightly different optima. The best performance always had GSR lower than 50%. The model with 15% gap sentences achieved the highest ROUGE scores on CNN/DailyMail, while XSum/Reddit TIFU and WikiHow did better with 30% and 45% respectively. When scaling up to PEGASUS LARGE (Section 6.2), we chose an effective GSR of 30%.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
MLM As mentioned, the MLM objective can either be applied solely or together with GSG. We jointly trained MLM with GSG Ind-Orig (MLM & Ind-Orig), which masks 30% sentences and extra 15% tokens in unselected sentences, as shown in Figure 1. Figure 4a shows that the model pretrained with MLM alone performed significantly worse and MLM & Ind-Orig had similar performance as Random. Interestingly, when comparing MLM & Ind-Orig to Ind-Orig, we empirically observed MLM improved fine-tuning performance at early pre-training checkpoints (100k -200k steps), but inhibited further gains with more pre-training steps (500k). Therefore, we chose not to include MLM in PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We compared two tokenization methods 2 : Byte-pairencoding algorithm (BPE) (Wu et al., 2016;Sennrich et al., 2016), and SentencePiece Unigram algorithm (Unigram) proposed in Kudo (2018). We evaluated Unigram with different vocabulary sizes ranging from 32k to 256k. In these experiments, models were pre-trained for 500k steps on the C4 corpus with the Ind-Orig objective and 15% GSR. As shown in Figure 5, BPE and Unigram were comparable on news datasets while Unigram outperformed BPE 2 Implemented in https://github.com/google/sentencepiece  on non-news datasets, especially WikiHow. On XSum and CNN/DailyMail, Unigram 96k achieved the highest ROUGE scores. On WikiHow and Reddit TIFU, the best configurations were Unigram 128k and 64k respectively. Therefore, we used the overall best vocabulary option Unigram 96k in PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Compared with PEGASUS BASE , the large model PEGASUS LARGE had increased capacity from larger hidden size (H : 768 → 1024, F : 3072 → 4096, A : 12 → 16), number of layers (L : 12 → 16) and traversed much more data, due to larger batch size (B : 256 → 8192) (same number of pre-training steps, 500k). We adopted the best practices found in the PEGASUS BASE ablation studies using the GSG (Ind-Orig) pre-training objective without MLM and Unigram vocabulary size of 96k. In total, PEGASUS LARGE had 568M parameters.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To encourage the model to copy, which is an important aspect of the more extractive datasets, we left 20% of selected sentences unchanged in the input instead of replacing with [MASK1]. We increased the GSR to 45% to achieve a similar number of "gaps" as the optimal 30% found above. We reported the performance of the models pre-trained on HugeNews and C4 separately. We conducted a simple hyper-parameter sweep of learning rate and length penalty,  (Lewis et al., 2019;Shi et al., 2019;Fabbri et al., 2019;Koupaee & Wang, 2018;Kim et al., 2019;Subramanian et al., 2019;Song et al., 2019;Zhang & Tetreault, 2019;Kornilova & Eidelman, 2019). We only compared with previous abstractive models except on BillSum which had extractive results only. BIGPATENT, arXiv, PubMed and Multi-News datasets contain very long summaries and we truncate them to 256 tokens, in similar range compared to (Sharma et al., 2019;Cohan et al., 2018;Fabbri et al., 2019;Goodman et al., 2019). Best ROUGE numbers on each dataset and numbers within 0.15 of the best numbers are bolded.  CNN/DailyMail, Multi-News, arXiv, PubMed, BIG-PATENT datasets contain input documents longer than the maximum input length (L input = 512 tokens) in pretraining. This would present a problem for position embeddings which would never be updated for longer input lengths, but we confirm the postulation that sinusoidal positional encodings (Vaswani et al., 2017) generalize well when fine-tuning PEGASUS LARGE beyond the input lengths observed in training up to L input = 1024 tokens. Since average input length in BIGPATENT, arXiv, PubMed and Multi-News are well beyond 1024 tokens, further scaling up L input or applying a two-stage approach (Liu et al., 2018) may improve performance even more, although this is outside the scope of this work.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Tables 1 and 2 show the performance improvements of PEGASUS BASE and PEGASUS LARGE on downstream datasets. While PEGASUS BASE exceeded current state-ofthe-art on many datasets, PEGASUS LARGE achieved better than state-of-the-art results on all downstream datasets using HugeNews, although C4 performed better on WikiHow.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The improvement from a Transformer model without pretraining (Transformer BASE ) to PEGASUS LARGE was more significant on smaller datasets. For example, the ROUGE2-F1 scores nearly tripled on AESLC and quintupled on Reddit TIFU. The large jumps in performance suggest that small text summarization datasets benefit the most from pre-training. We further investigate low resource summarization in Section 6.3.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In real-world practice, it is often difficult to collect a large number of supervised examples to train or fine-tune a summarization model. To simulate the low-resource summarization setting, we picked the first 10 k (k = 1, 2, 3, 4) training examples from each dataset to fine-tune PEGASUS LARGE (HugeNews) . We fine-tuned the models up to 2000 steps with batch size 256, learning rate 0.0005, and picked the checkpoint with best validation performance. In Fig Khandelwal et al. (2019) with 3000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Overall, we observed high-linguistic quality (in terms of fluency and coherence), closely emulating the style of groundtruth summaries. While some previous work suggested that maximum likelihood training results in repetitive text in model outputs (Welleck et al., 2019) we found this to be rare in our outputs and did not require additional countermeasures to mitigate dis-fluencies.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Although ROUGE clearly has its draw-backs (Kryscinski et al., 2019), over-penalizing abstractive approaches com-pared to extractive ones and having no sense of linguistic quality, we found that choosing perplexity-optimized models using aggregated ROUGE (rather than directly optimizing ROUGE as in Paulus et al. (2017)) resulted in qualitatively good models. Randomly sampled (by a program) model decodes across all datasets and a broad range of ROUGE scores can be found in Appendix I.We found that even low-ROUGE model summaries often were highquality, Figure G.1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To assess how close PEGASUS LARGE is to human performance we conducted human evaluation experiments on Amazon Mechanical Turk comparing model summaries with (human) reference summaries given the input document. The examples were drawn from the XSum, CNN/DailyMail, and Reddit TIFU datasets; the first two were chosen due to their popularity in past work, and the third was chosen for its significant difference in style. Workers were asked to rate the summaries on a 1-5 scale, with higher being better (full experiment details provided in Appendix F) and a paired t-test was used to assess whether scores were significantly different from human.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In the first experiment, PEGASUS LARGE (HugeNews), PEGASUS LARGE (C4), and Transformer BASE were compared with reference summaries; in the second experiment, PEGASUS LARGE (HugeNews) fine-tuned using 10, 100, 1000, and all supervised examples were compared with references; the results are shown in Table 3. According to the significance level of p < 0.01, both PEGASUS LARGE (HugeNews) and PEGASUS LARGE (C4) outputs were at least as good as the reference summaries in all cases. Even at low-levels of supervision PEGASUS LARGE (HugeNews) was not measurably worse than human summaries on XSum and CNN/DailyMail. In the Reddit TIFU case, however, perhaps due to its diverse writing styles, human performance required full supervision.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The pre-training corpora are a large collection of documents from the Internet and potentially have overlap with the downstream test sets. In this section, we measured the extent of overlap between the pre-training corpus and downstream datasets. We also studied if the pre-trained model was able to exploit memorization to achieve higher performance on the downstream datasets.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To measure the overlap, we calculated similarities between all pairs of downstream test set targets and pre-training documents. We use the ROUGE-2 recall as a similarity measure (common 2-grams / test set targets 2-grams). It is not necessarily exact match even if the similarity score is 1.0. We filtered all test set examples that have similarity to any pre-training example above a threshold, and recalculated the ROUGE scores on the remaining test set. In Figure 7  tences ratio uniformly between 15%-45%.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
(3) Importance sentences were stochastically sampled with 20% uniform noise on their scores. (4) The model was pre-trained for 1.5M steps instead of 500k steps, as we observed slower convergence of pre-training perplexity. (5) The SentencePiece tokenizer was updated to encode the newline character. The PEGASUS LARGE (mixed, stochastic) model achieved best results on almost all downstream tasks, as shown in Table 4.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this work, we proposed PEGASUS, a sequence-tosequence model with gap-sentences generation as a pretraining objective tailored for abstractive text summarization. We studied several gap-sentence selection methods and identified principle sentence selection as the optimal strategy. We demonstrated the effects of the pre-training corpora, gap-sentences ratios, vocabulary sizes and scaled up the best configuration to achieve state-of-the-art results on all 12 diverse downstream datasets considered. We also showed that our model was able to adapt to unseen summarization datasets very quickly, achieving strong results in as little as 1000 examples. We finally showed our model summaries achieved human performance on multiple datasets using human evaluation.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The training code and instructions for using model checkpoints can be found at https://github.com/google-research/ pegasus
Investigating Efficiently Extending Transformers for Long Input Summarization
Large pretrained Transformer models have proven to be extremely capable at tackling natural language tasks (Devlin et al., 2018;Brown et al., 2020). However, handling long textual sequences continues to be a significant challenge for these models. Training models to handle long sequences is expensive in both computation and memory, and moreover requires training and evaluating on long sequence data, which can be rarer and more costly to collect. Given the broad success of Transformer models on short-sequence language tasks, our goal is to investigate the best way to extend these models to handle longer sequences. * Work done while at Google. In this work, we focus on the task of long input summarization: summarizing long input documents into shorter textual sequences. The input documents of such tasks are often significantly longer than the maximum context lengths of most standard Transformer models, and hence warrant both specialized model architecture modifications as well as new training regimes to handle. For instance, to avoid the quadratic growth in memory consumption of the attention computation in Transformers, many memory-efficient Transformer variants have been proposed (Tay et al., 2020(Tay et al., , 2021. However, the manner in which these changes are incorporated into models has been inconsistent and ad-hoc, and there are few established best-practices. For instance, some works add an additional long input pretraining stage to adapt the model weights to the new architecture (Beltagy et al., 2020), while others directly fine-tune on the long-input summarization data without any pre-adaptation (Zaheer et al., 2020;Pang et al., 2022). Because of the high cost of training these models, there has yet to be a systematic study of how best to adapt models for long input sequences. Hence, it has been difficult to establish which model and training changes are necessary or complementary.
Investigating Efficiently Extending Transformers for Long Input Summarization
To answer these questions, we conduct an extensive empirical investigation into the architectural changes, model configurations and pretraining schemes to identify the better approaches to training Transformer models to tackle long input summarization. We evaluate a set of efficient Transformer variants, and propose a simpler blockwise local Transformer architecture with staggered blocks and global tokens that strikes a good balance of performance and memory efficiency. We also show that given a fixed token budget, pretraining on short sequences and then pre-adapting the model to an efficient Transformer architecture on long sequence for additional training steps leads to superior performance compared to only long input pretraining or no adaptation at all. We also investigate several other model design choices such as position encoding schemes, encoder-decoder layer distributions, and the impact of discrepancies between pretraining and fine-tuning architecture hyperparameters.
Investigating Efficiently Extending Transformers for Long Input Summarization
Based on the findings from our empirical investigation, we adapt the pretrained PEGASUS Large model (Zhang et al., 2020) to tackle long input summarization on up to 16K input tokens. The resulting model, which we call PEGASUS-X attains top scores on long summarization tasks, outperforming much larger models like LongT5 (Guo et al., 2021) in some cases, and sets the state of the art of two tasks: GovReport and PubMed. Morever, impact on short input summarization performance is minimal. A smaller version which we call PEGASUS-X Base attains similar scores with much fewer parameters. The code and weights for both models will be released at https://github.com/ google-research/pegasus and as well as in Hugging Face Transformers (Wolf et al., 2020). Beyond long input summarization, we believe that many of our findings will be useful to the community for efficiently adapting Transformer models to handle ever longer input sequences for other tasks.
Investigating Efficiently Extending Transformers for Long Input Summarization
In summary, our contributions are:
Investigating Efficiently Extending Transformers for Long Input Summarization
1. We evaluate a series of proposed efficient Transformer architectures as well as a host of other model tweaks, and report their efficacy as well as trade-offs on computational resources when applied to long input summarization tasks.
Investigating Efficiently Extending Transformers for Long Input Summarization
2. Based on our findings, we propose a recipe for adapting a short-context, pretrained Transformer encoder-decoder to longer inputs, and apply it to PEGASUS to greatly improve its long-document summarization performance, with comparable short-input performance.
Investigating Efficiently Extending Transformers for Long Input Summarization
3. We release model checkpoints for the resulting 568M-parameter model, which we call PEGASUS-X, and a smaller 272Mparameter model with most of the performance, PEGASUS-X Base .
Investigating Efficiently Extending Transformers for Long Input Summarization
2 Challenges of Long Input Summarization
Investigating Efficiently Extending Transformers for Long Input Summarization
While summarization is fundamentally about extracting and compressing information from longer to shorter sequences, most commonly studied summarization tasks have had inputs on average shorter than the input sequence lengths of Transformer language models-typically 512 to 2048 tokens. As the ability for models to handle language has improved, the field has pushed for more challenging summarization tasks with longer input lengths. The quadratic scaling of the memory requirements and computation for the attention mechanism in Transformers poses a challenge to tackling these longer summarization tasks. Many memory-and computeefficient variants of Transformers (Beltagy et al., 2020;Zaheer et al., 2020;Choromanski et al., 2021;Kitaev et al., 2020) have been proposed to address this constraint. However, even when incorporating efficient Transformer architectures that achieve approximately linear memory scaling with input sequences, it is still common for models to be pretrained on short sequence inputs and only be adapted to handle long sequences when fine-tuning on a downstream task, which may be suboptimal. While using decoder-only autoregressive language models for summarization has received some recent attention (Radford et al., 2019;Brown et al., 2020;Chowdhery et al., 2022), encoder-decoder models still generally perform better and remain the architecture of choice for the task (Wang et al., 2022b). The asymmetry between the input length and summary lengths requires new considerations for resource limitations of models. Consider a summarization model with 12 encoder and 12 decoder layers, pretrained on an input length of 512 and finetuned on a task with input sequence length 16384, using output length of 512 in both cases. Since pretraining is typically done with shorter sequences while fine-tuning uses long inputs for summaries, fine-tuning can now be more resource intensive and slower than pretraining, which is contrary to the conventional paradigm. Since the encoder inputs have increased 32×, the quadratic scaling in the memory consumption of the self-attention operation means that we expect the encoder self-attention to consume 1024× the amount of memory in finetuning relative to pretraining. Even if we use an efficient Transformer variant that achieves linear scaling in memory consumption and computation, both the encoder self-attention and decoder crossattention operations still consume 32× the memory compared to pretraining. Besides attention, expensive operations such as the FFN that scale linearly with the input length also greatly increase the computation required both at training and inference.
Investigating Efficiently Extending Transformers for Long Input Summarization
On the other hand, the unique characteristics of long-document summarization may also prompt new solutions to these issues. For instance, if encoder computations over long sequences pose a compute bottleneck, we may consider using fewer encoder layers and more decoder layers, exchanging decoding speed at inference for faster training. The higher relative cost of fine-tuning can also justify greater efforts to adapt the pretrained model to fine-tune more quickly, via mixing short-and long input training curricula, adapting the model to efficient Transformer architectures via additional pretraining, and so on.
Investigating Efficiently Extending Transformers for Long Input Summarization
To address these questions and challenges, we conduct a series of ablation experiments investigating which approaches can lead to improvements in downstream summarization results, as well as the computational trade-offs therein.
Investigating Efficiently Extending Transformers for Long Input Summarization
A challenge in building long-document summarization models is the relative scarcity of long-input summarization datasets with sufficient data to train and evaluate models on. Recent work introducing new long-document summarization datasets has alleviated this problem somewhat Shaham et al., 2022;Kryściński et al., 2021), although the relative scarcity of good datasets continue to make this a challenging problem to make progress on. The main issues in current datasets are: relative simplicity of summarization, lack of diverse inputs, potential leakage of data due to the data collection procedure, and low quantity of examples for training. We refer the reader to Wang et al. (2022a) for more discussion on the challenges of creating large, high-quality long-document summarization datasets.
Investigating Efficiently Extending Transformers for Long Input Summarization
Similar to Zhang et al. (2020), we perform the majority of our experiments with a PEGASUS Basesized model, before applying our findings to PEGASUS Large -sized model.
Investigating Efficiently Extending Transformers for Long Input Summarization
We generally follow the recipe from PEGASUS (Zhang et al., 2020) for pretraining PEGASUS Basesized models. All experiments in our ablation study performed pretraining with C4 (Raffel et al., 2020) for 500k steps with 512 input tokens and 256 output tokens and a masking ratio of 45%, unless otherwise stated. For long input pretraining we extend the input length to 4096 tokens, adjust the masking ratio from 45% to 5.625%, reducing the ratio by a factor of 8 to account for the 8x increase in input sequence length. We also filter for only documents longer than 10000 characters.
Investigating Efficiently Extending Transformers for Long Input Summarization
We evaluate our pretrained models by fine-tuning on the arXiv (Cohan et al., 2018) and GovReport (Huang et al., 2021) long-context summarization tasks. Where relevant, we also fine-tune on the shorter-context XSUM and CNN/DailyMail tasks. For each experiment, we report the best validation set scores based on the geometric average (RG) of ROUGE-1, ROUGE-2 and ROUGE-L scores (Lin, 2004) based on the rouge-score package. 1 For arXiv, we fine-tune with an input length of up to 16384 tokens and 256 output tokens, while for GovReport we use an input length of 10240 input tokens and 1024 output tokens given the longer summaries for the task. For XSUM and CNN/Daily Mail, with use an input length of 512, and output lengths of 64 and 128 respectively, following PEGASUS hyperparameters. The full set of hyperparameters for fine-tuning can be found in Appendix 7. Unless otherwise stated, we directly switch over to the efficient Transformer architectures between pretraining (on shorter context) and fine-tuning (on longer contexts), with no adaptation phase in between.
Investigating Efficiently Extending Transformers for Long Input Summarization
We begin by investigating the efficacy of swapping the encoder for an efficient Transformer encoder to allow our models to incorporate longer input sequences while consuming reasonable amounts of device memory. We first consider two efficient encoder architectures that exemplify two different approaches to memory-efficient attention. Big Bird (Zaheer et al., 2020) takes the approach of using sparse attention computation, combining slidingwindow attention, random attention and a set of global-attention tokens. Conversely, Performer (Choromanski et al., 2021) takes the approach of factorizing attention matrices via orthogonal random features. Both Big Bird and Performer have the benefit of requiring no new parameters to be introduced, and hence the weights from a pretrained Transformer can be ported directly to these architectures. Both model also performed well on the Long Range Arena tasks (Tay et al., 2021). However, for this experiment, we perform both pretraining and fine-tuning with the same encoder architecture to avoid the issue of mismatch between pretraining and fine-tuning architectures.
Investigating Efficiently Extending Transformers for Long Input Summarization
In addition, we also introduce two simple variants of local attention Transformer encoders. First, we use a simple block-local Transformer (Local), where encoder input tokens are divided into nonoverlapping blocks, tokens can only attend to other tokens within the block. Second, we extend this local Transformer by adding a set of global tokens with learnable embeddings, that can attend to and be attended from every encoder token (Global-Local). These components are similar in principle to the sliding window attention and global token attention of Big Bird, as well as similar constructs in other efficient Transformers such as ETC  and Longformer (Beltagy et al., 2020). However, we opt for the simpler block-local attention rather than sliding window attention, and compensate for the lack of overlapping blocks by staggering the local attention blocks, which we elaborate on in Section 4.2. As we show below, the performance is highly competitive despite its simplicity.
Investigating Efficiently Extending Transformers for Long Input Summarization
BigBird, Local and Global-Local all use a block size of 64, and 32 global tokens where relevant. Performer uses 256 random features.
Investigating Efficiently Extending Transformers for Long Input Summarization
Results on short and long summarization tasks are shown in Table 1, with the relative training steps per second and memory consumed per device for fine-tuning on arXiv shown in the right-most columns. Among the short tasks, the full-attention Transformer performs best, followed by BigBird. On the long tasks, Big Bird and Global-Local models perform best, but Big Bird consumes significantly more memory and trains much more slowly than the other architectures. Conversely, we find that although the Performer has relatively low memory consumption and trains efficiently, it performs the worst out of the architectures we tested by a noticeable margin.
Investigating Efficiently Extending Transformers for Long Input Summarization
On the other hand, we find that the Local and Global-Local encoders strike a good balance of both performance and efficiency. The simple local attention encoder, which uses a block-local attention mechanism, attains performance surprisingly close to that of Big Bird while being much faster and using much less memory. The Global-Local encoder trades off a small amount of speed and memory for better performance, outperforming Big Bird. While both Local and Global-Local models underperform Big Bird and Transformer for shorttasks, it appears that the model architectures make the right trade-offs for performance on long summarization tasks.  Table 2: Comparison of architectural tweaks to Local and GlobalLocal encoder. Staggering local blocks uses different blocks boundaries for different layers in block-local attention. Global information is incorporated in the decoder via an additional cross-attention before cross-attention over the encoded input. Takeaways: Local attention is a surprisingly strong baseline, while adding global tokens significantly improves performance, and both models are resource-efficient.
Investigating Efficiently Extending Transformers for Long Input Summarization
Given the good performance of both Local and Global-Local encoder variants, we next consider further architectural tweaks to these models. First, we introduce staggering of local attention blocks. Unlike in sliding window attention, in block-local attention tokens can only attend to other tokens within the same block. If the input tokens are divided up into the same blocks in every layer, this means that no information is exchanged across blocks through the entire encoder. To address this pitfall, we introduce a small architectural change wherein we stagger the block allocation across alternating layers. We show an example of this in Figure 2. Concretely, we stagger attention blocks by shifting the block boundaries by half a block every other layer: in practice, we implement this by padding the hidden representations on either side by half a block and masking accordingly.
Investigating Efficiently Extending Transformers for Long Input Summarization
Secondly, in the Global-Local model, the decoder only attends to the encoded token representations, and not the global token representations. We consider a variant where we supply the global token representations to the decoder, and in par-ticular introduce a second encoder-decoder crossattention that attends only to the global tokens, before performing cross-attention over the encoded tokens. Our goal is to allow the decoder to incorporate global information before performing crossattention over the encoded sequence.
Investigating Efficiently Extending Transformers for Long Input Summarization
We show the results of both of these changes in Table 2. We find that staggering local blocks improves performance in both Local and Global-Local models by a noticeable amount. We highlight that this improves performance even with the Global-Local models, which already has a channel for cross-block interactions via global tokens, indicating that both of these model improvements are complementary. Conversely, we did not find that incorporating global token information in the decoder led to much of a performance improvement, particular once staggered local blocks were used.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: Staggering local attention blocks significantly improves performance, and is complementary to global tokens.
Investigating Efficiently Extending Transformers for Long Input Summarization
Next, we vary the block size and number of global tokens for the Global-Local encoder, with results shown in Table 3  Broadly, we find that increasing either block size or global tokens leads to improved performance, with a corresponding increase in memory consumption and computation time. However, the effect size from going to larger block sizes is not large, and appears to saturate as we get to larger block sizes or number of global tokens. As such, increasing either of these hyperparameters is preferable if resources allow, but may not be a high priority compared to other potential model improvements. For the remainder of the ablation experiments, we stick to a block size of 64 and 32 global tokens for consistency.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: Larger block sizes and/or number of global tokens leads to improved performance, although the effect saturates.
Investigating Efficiently Extending Transformers for Long Input Summarization
New position encoding schemes encoding schemes such as RoPE (Su et al., 2021) and ALiBi (Press et al., 2022) have garnered recent attention, showway in which TPUs pad small dimensions of arrays to certain minimum lengths, leading to larger than expected memory consumption.
Investigating Efficiently Extending Transformers for Long Input Summarization
ing improved performance on downstream evaluations. As input sequence lengths have gotten much longer, and in particular longer than the dimensions of hidden representations, previous choices of position encoding may no longer be optimal. Moreover, relative position encodings such as RoPE, T5 and ALiBi may be better suited for adapting models to different input lengths between pretraining and fine-tuning. Hence, this is a good opportunity to revisit the choice of positioning encoding schemes in encoder models.
Investigating Efficiently Extending Transformers for Long Input Summarization
Because of the more complex interaction between local attention blocks and relative position encoding implementations, we conduct a preliminary investigation with a full-attention Transformer. We pretrain with an input length of 512, and finetune with an input length of 2048 for the long sequence tasks -this experiment also tests the propensity for position encodings to be adapted to longer sequences downstream. In addition to the sinusoidal position encoding used in PEGA-SUS and Vaswani et al. (2017), we also consider the bucket-based relative position encoding scheme of T5, RoPE, absolute position embeddings, and   no position encoding as a baseline. For absolute position embeddings, we follow the recipe of Beltagy et al. (2020) and duplicate the learned position embeddings to handle longer sequences before finetuning. The chosen position encoding scheme is applied to all parts of the model, including both the encoder and the decoder. We do not experiment with ALiBi, as we found no natural way to adapt ALiBi to cross-attention.
Investigating Efficiently Extending Transformers for Long Input Summarization
Our results are shown in Table 4. We find that although T5 performs the best, it is also almost twice as slow as the other position encoding schemes, which is consistent with the findings of Press et al. (2022). Sinusoidal position encodings and RoPE perform only slightly worse than T5 with much better efficiency, making them more desirable choices. Given the much simpler implementation of sinusoidal position encodings, we opt to stick with them for the remainder of the experiments.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: Sinusoidal position encodings still remain a good choice for long input Transformers.
Investigating Efficiently Extending Transformers for Long Input Summarization
Scaling laws (Kaplan et al., 2020;Ghorbani et al., 2021;Zhang et al., 2022) that describe the empirical relationship between model sizes and performance have proven surprisingly consistent and gotten significant attention in recent years. We present in this section a small set of scaling experiments, exploring the distribution of layers between encoder and decoder.
Investigating Efficiently Extending Transformers for Long Input Summarization
Our results are shown in Table 5. In the top half, we fix the total number of layers to 24, and consider both encoder-heavy and decoder-heavy distributions, for both Local and Global-Local models. We observe that impact of distribution of encoder and decoder layers on performance is relatively small. For Local models, we see a slight boost from decoder-heavy models. For Global-Local models, we observe that a balanced encoder-decoder outperforms encoder-and decoder-heavy models, both of which perform about comparably.
Investigating Efficiently Extending Transformers for Long Input Summarization
We also consider cases where we further increase the size of either the encoder or decoder to 18 layers, shown in the second half of Table 5. We observe no improvement in performance over the 12/12-layer encoder-decoder, and suspect that other hyperparameters (e.g. hidden size) might be the bottleneck rather than the number of layers.
Investigating Efficiently Extending Transformers for Long Input Summarization
We highlight here that because of the asymmetry of the input and output lengths, there are different computational trade-offs to different balances of encoder and decoder layers. Encoder-heavy models require more memory because of the long input sequences, whereas decoder-heavy models are relative slower at inference because of the autoregressive nature of decoding. Given the relatively small  difference in the margin of performance, memory or computational constraints may outweigh the performance differences in practical scenarios.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: A balanced Global-Local model outperforms other variants, but the difference in performance may be outweighed by other resource considerations.
Investigating Efficiently Extending Transformers for Long Input Summarization
Previous works using efficient Transformer encoders have generally taken the model weights of a full-attention Transformer pretrained on a shorter sequence, and adapted them to the efficient architecture either directly during fine-tuning (Zaheer et al., 2020), or with an intermediate stage of additional pretraining (Beltagy et al., 2020). In this section, we investigate if such an approach is optimal, or if the model would benefit from being pretrained with the efficient encoder from the beginning. Note that we are still performing pretraining on a short sequence (512 tokens), even with an efficient encoder. We consider both pretraining with a Transformer and pretraining with the efficient architecture for both Local and Global-Local models. We also vary the block size, as the main difference between a Transformer and Local Transformer is the block size (aside from staggering, a Local model with block size 512 is equivalent to a dense Transformer), and hence the difference in block size also corresponds to the extent to which the model needs to adapt between architectures. When adapting from a pretrained Transformer encoder to a Global-Local architecture, because the Global-Local model relies on newly introduced global token embeddings, we initialize them by randomly sampling tokens from the vocabulary embeddings.
Investigating Efficiently Extending Transformers for Long Input Summarization
Our results are shown in Table 6. For Local models, we find that pretraining with local attention using small block sizes tends to hurt performance, but at moderate block sizes (e.g. 64) there is little difference between the two approaches. In contrast, we find that for Global-Local, pretraining with the efficient architecture tends to perform better. We hypothesize that this difference arises because of the presence of the learned global embedding tokens, which are randomly initialized when adapting from a pretrained Transformer and hence may benefit from pretraining and being jointly trained with the local attention.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: For moderate block sizes, either pretraining or adapting to a Local encoder performs about equally well, but pretraining with a Global-Local encoder performs slightly better.
Investigating Efficiently Extending Transformers for Long Input Summarization
Up to this point, we have only considered pretraining with short sequences. We might expect that pretraining with longer sequences ought to improve performance of our model on downstream long input summarization. However, pretraining only on long sequences is computationally expensive and requires a large collection of long input documents, which are relatively rarer. Moreover, long documents may contain different information from short documents, hence limiting training to only Table 7: Comparison of different pretraining formats, given a input token budget of 131B tokens, which corresponds to 1M steps with 512 input tokens. Short pretraining uses 512 input tokens, whereas long pretraining uses 4096 input tokens. long inputs maybe reduce the diversity of training data. Different long context Transformers have taken different approaches to pretraining on long inputs. For instance, Longformer (Beltagy et al., 2020) performed several additional stages of increasingly longer-sequence pretraining to adapt the initial RoBERTa to long sequence inputs. On the other hand, LongT5 (Guo et al., 2021) is pretrained exclusively with long input sequences. Others (Zaheer et al., 2020;Ivgi et al., 2022) perform no long input pretraining at all. In this section, we investigate how the balance of short and long pretraining impact downstream performance, and try to find the best trade-off between pretraining cost and downstream performance.
Investigating Efficiently Extending Transformers for Long Input Summarization
We consider two setups for pretraining: shortinput pretraining, with 512 input tokens and 256 output tokens, and long-input pretraining, with 4096 input tokens and 256 output tokens. We describe the corresponding differences in data preprocessing in Section 3.1. We choose to fix the number of input tokens seen during training as the constraint, and vary configurations subject to this constraint. This constraint roughly proxies for the amount of compute consumed as well as corresponds to the number of input tokens seen during pretraining, in contrast to fixing the number of steps, where long-input pretraining would consume far more compute for the same number of steps.
Investigating Efficiently Extending Transformers for Long Input Summarization
In contrast to the above experiments where we generally performed short pretraining for 500k steps, we set our total input token budget at 131 billion tokens, which correponds to 1 million steps with 512 input tokens. This larger budget ensures that when we do only long-input pretraining, the model is still pretrained for a reasonable number of steps. Given this budget, we consider four configurations:
Investigating Efficiently Extending Transformers for Long Input Summarization
• Short-input pretraining for 100% of tokens (1M steps)
Investigating Efficiently Extending Transformers for Long Input Summarization
• Short-input for 75% of tokens (98.3B, 750k steps), then long-input for 25% of tokens (32.8B, 31.25k steps)
Investigating Efficiently Extending Transformers for Long Input Summarization
• Short-input for 50% of tokens (62.5B, 500k steps), then long-input for 50% of tokens (62.5B, 62.5k steps)
Investigating Efficiently Extending Transformers for Long Input Summarization
• Long-input pretraining for 100% of tokens (125k steps)
Investigating Efficiently Extending Transformers for Long Input Summarization
We compare the performance of the different pretraining scehemes in Table 7. We also include the short-input pretraining for 500k steps for comparison. First, comparing short-input pretraining for 500k and 1M steps, we find that more pretraining still improves performance, indicating that our base models may still be undertrained at 500k steps. Secondly, we observe that long-input pretraining performs consistently worse than the other variants, which we attribute to the fewer number of training steps taken, again highlighting the issue of potential under-training. Focusing our analysis on the middle three configurations, on the long tasks, we find that all three non-long-only variants atttain similar scores, with more long-input pretraining having slightly better performance, particularly on the ROUGE-2 and ROUGE-L scores. While the small absolute differences in scores make it hard to draw strong conclusions, we lean towards the conclusion that adding a short phase of long input pretraining can be beneficial can improve performance on long input summarization tasks.   Table 9: Comparison of models pretrained with cross-attention for a subset of layers, and adapting a pretrained model by dropping cross-attention layers only during fine-tuning Takeaways: Given a fixed compute budget, allocating some portion of training to long-input training can improve performance, although the precise optimal allocation is difficult to determine. Exclusively long pretraining results in worse performance.
Investigating Efficiently Extending Transformers for Long Input Summarization
Given the use of an efficient attention architecture, which has memory consumption scale linearly rather than quadratically in input sequence length, another major memory bottleneck is the encoder-decoder cross-attention. Because each decoder layer attends separately to the long encoder representations, and the attention is dense, this is a large contiguous chunk of memory that we could seek to reduce. Perceiver AR (Hawthorne et al., 2022) demonstrated strong performance by using only a single cross-attention at the bottom layer of an autoregressive language model. Based on these results, we investigate the impact of only having cross-attention on a subset of decoder layers. In Table 8, we show the results of pretraining and fine-tuning Global-Local models with cross-attention only on specific layers on a variety of configurations. We find that reducing the number of cross-attention layers leads dings to capture different position information. In contrast, because our models use sinusoidal position encodings which can naturally extrapolate to longer input lengths, we find that fine-tuning has been sufficient to adapt the model to reasonable performance. to a drop in performance, but the impact on performance is smaller than expected. For instance, with only cross-attention on the first and sixth layer, the Global-Local model still outperforms a Local model. The reduction of cross-attention layers also leads to a corresponding improvement in training step and reduction in memory consumption.
Investigating Efficiently Extending Transformers for Long Input Summarization
Given the small drop in performance from using fewer decoder layers with cross-attention, we consider the viability of dropping cross-attention layers after pretraining. In other words, we take a Global-Local model pretrained with full cross-attention, drop the cross-attention for a subset of layers, and fine-tune directly. Our results are shown in Table 9. We find that dropping the cross-attention after pretraining again only leads to a small (additional) dip in performance. This indicates that dropping cross-attention may be a viable strategy for further reducing memory requirements for an existing pretrained model with a small performance trade-off, and pretraining a separate model from scratch is not necessary.
Investigating Efficiently Extending Transformers for Long Input Summarization
Takeaways: Dropping cross-attention for a fraction of decoder layers can reduce memory consumption at the cost of slight performance regression. Cross-attention can be dropped after pretraining, with an associated performance trade-off.
Investigating Efficiently Extending Transformers for Long Input Summarization
Based on our findings above, we settle on the following recipe for adapting the PEGASUS models (Zhang et al., 2020) to long sequence summarization.
Investigating Efficiently Extending Transformers for Long Input Summarization
• We use a Global-Local architecture with block staggering, a large number of global tokens, and large block sizes during pretraining.
Investigating Efficiently Extending Transformers for Long Input Summarization
• We conduct an additional stage of long input pretraining on 4096 token inputs for 300,000 steps.
Investigating Efficiently Extending Transformers for Long Input Summarization
• We extend input sequences up to 16384 input tokens in fine-tuning, depending on the task.
Investigating Efficiently Extending Transformers for Long Input Summarization
We experiment with two model sizes PEGASUS-X (PEGASUS eXtended), based on PEGASUS Large ; and PEGASUS-X Base , based on a newly trained PEGASUS Base model which we call PEGASUS Base+ . In a similar finding as Hoffmann et al. (2022), we found that PEGASUS Base benefits from training on significantly more tokens, which we set to the same as PEGASUS Large .
Investigating Efficiently Extending Transformers for Long Input Summarization
We initialize the weights of PEGASUS-X and PEGASUS-X Base on the pretrained weights of PEGASUS Large and PEGASUS Base+ respectively. Only two new sets of parameters introduced: the global token embeddings, and a separate Layer-Norm for the global input representations in each Transformer layer. This is approximately 1M more parameters for PEGASUS-X Base and 2M more for PEGASUS-X. We initialize the global token embeddings by randomly sampling tokens from the input token embedding, and we initialize the Lay-erNorm weights with the regular input LayerNorm weights.
Investigating Efficiently Extending Transformers for Long Input Summarization
The task-and model-specific hyperparameters for fine-tuning can be found in Appendix 15. For this section, we report ROUGE-Lsum 4 rather than ROUGE-L for consistency with the metrics reported in other papers and leaderboards.  
Investigating Efficiently Extending Transformers for Long Input Summarization
Long summarization tasks In Table 11, we compare the performance of PEGASUS models to those of PEGASUS-X on three longinput summarization tasks: arXiv, Big Patent and PubMed. In all three tasks, we see significant improvements in performance of PEGASUS-X Base over PEGASUS Base+ , and PEGASUS-X over PEGASUS Large . To isolate the impact of additional long input pretraining compared to only switching the architecture to accomodate long input sequences, we also include evaluation on the PEGA-SUS models using the Global-Local architecture with no further pretraining, which we list in the table as PEGASUS Base+ + Global-Local.
Investigating Efficiently Extending Transformers for Long Input Summarization
We also compare to reported results of PEGASUS Large using the Big Bird architecture (Zaheer et al., 2020), Longformer encoder-ecoder (LED; Beltagy et al., 2020), the Top-Down Transformer (Pang et al., 2022) in both Average-Pool (AvgP) and Adaptive-Pool (AdaP) variants, the Large and XL sizes of LongT5, and the SLED (Ivgi et al., 2022). LED, Top-Down and SLED are all initialized with BART Large weights with no additional pretraining on long input sequences, although AdaP has a multi-step fine-tuning setup (see below).
Investigating Efficiently Extending Transformers for Long Input Summarization
We note that the Big Bird-PEGASUS uses only 3072 tokens context, which is likely due to the larger memory consumption of Big Bird. We find that PEGASUS-X outperforms Big Bird-PEGASUS on all tasks, and Top-Down-AvgP on both compared tasks. Top-Down-AdaP still outperforms PEGASUS-X, we highlight that Top-Down-AdaP uses a much more complex, multistep fine-tuning setup, involving using an importance tagger on reference summaries to construct weights for pooling tokens within segments. In contrast, PEGASUS-X is fine-tuned with the standard fine-tuning pipeline. Even so, PEGASUS-X still outperforms Top-Down with adaptive pooling on PubMed. PEGASUS-X also outperforms LongT5 on both arXiv and PubMed summarization, despite both compared LongT5 models having more parameters. However, we find that LongT5 performs much better on BigPatent, which is a largely extractive summarization task. We hypothesize that a much larger much may be better at extraction over   very long encoded sequences.
Investigating Efficiently Extending Transformers for Long Input Summarization
Short summarization tasks We show in Table 12 the performance of PEGASUS and PEGASUS-X models on shorter summarization tasks. We observe that there is a slight regression in performance of both PEGASUS-X models compared to their PEGASUS equivalents. We hypothesize that the long input pretraining might negatively impact the performance on shorter input tasks because of the difference data filtering for long documents, resulting in a potentially less diverse training data distribution.
Investigating Efficiently Extending Transformers for Long Input Summarization
We report the performance of the PEGASUS-X models on the summarization tasks in the recently introduced SCROLLS benchmark in Table 13. This includes GovReport (Huang et al., 2021), the ForeverDreaming subset of SummScreen , and QMSum (Zhong et al., 2021). We observe that PEGASUS-X outperforms all other models on GovReport, setting the state of the art on the dataset. PEGASUS-X performs comparably to both LongT5 Large and Top-Down-AvgP on SummScreen/FD, although it underperforms both LongT5 models on QMSum. Moreover, we find that PEGASUS-X Base also performs competitively, outperforming both LongT5 models on GovReport, and only a small margin behind PEGASUS-X on all three tasks. PEGASUS-X Base also outperforms BART Large -SLED, a larger model with a similar 16K token context length. A major difference between PEGASUS-X and BART Large -SLED, besides being based on PEGASUS and BART respectively, is that BART Large -SLED does not have additional pretraining on long documents. We also note that UL2 only uses a context length of 2K tokens.
Investigating Efficiently Extending Transformers for Long Input Summarization
In this work, we investigate a range of proposed improvements to allow Transformer models to effectively and economically handle long inputs in text summarization tasks. Through extensive ablation experiments, we find a simple but effective recipe for extending short input Transformers to tackle long-input summarization. Based on our findings, we introduce PEGASUS-X, an extended version of PEGASUS with a modified architecture and additional long-sequence pretraining. We show that PEGASUS-X sets the state of the art on two long input summarization tasks (GovReport and PubMed) and performs competitively on many others, even despite being much smaller than some compared models. Our findings can also be applied to extending models to handle long input sequences in other domains beyond summarization, both for pretraining long input models from scratch as well as extending already pretrained short sequence models.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Text summarization aims at producing a concise and fluent summary while preserving salient content and overall meaning of the source documents. It has been applied in a wide range of real-world applications, e.g., summarizing Web search results for interactive information retrieval (Gao et al., 2022) and generating medical summaries from doctor-patient conversation transcripts (Zhang et al., 2021).
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
While the extractive approach is the dominant approach in commercial systems due to its simplicity and effectiveness (Allahyari et al., 2017), the abstractive approach is getting more attention in the research community as neural language models are used (e.g., Rush et al., 2015;Nallapati et al., 2016;Chopra et al., 2016;Liu and Lapata, 2019b,a;Pasunuru et al., 2021). Compared to the extractive approach where a summary is constructed using extracted sentences, abstractive summarizers paraphrase the idea of the source documents in a new form, and have a potential of generating more concise and coherent summaries.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
However, good abstractive summarizers are harder to develop since we have to deal with problems like semantic representation, inference and low-resource text generation, which are more challenging than sentence extraction. Recently, largescale pre-trained language models (PLMs) such as PEGASUS (Zhang et al., 2020), GPT (Radford et al., 2019;Brown et al., 2020), T5 (Raffel et al., 2020), have been applied for abstractive summarization. While these models can produce surprisingly fluent text, the generated summaries often contain factual inconsistencies, caused by distorted or fabricated facts about the source documents, which is known as the hallucination problem (Kryściński et al., 2019;Celikyilmaz et al., 2020;Ji et al., 2022). In addition, since the amount of text in the source documents can be very large, it is expensive to train an end-to-end abstractive model (e.g., an encoder-decoder transformer model) given the memory constraints of current hardware and the latency constraints of applications such as online document summarization for interactive information retrieval. Therefore, a two-stage approach is widely used, where a subset of document sentences is coarsely selected using an extractive summarizer, and an abstractive summarizer generates the summary conditioning on the extraction (Liu and Lapata, 2019b). This approach is sub-optimal in that salient information might be missed in the extraction.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In this paper, we propose a new encoderdecoder PLM optimized for abstractive summarization, Z-Code++, which significantly extends Z-Code (Wang et al., 2020), a state-of-the-art PLM developed for machine translation, as follows.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
First, Z-Code++ is pre-trained on web text using two tasks, replaced token detection (RTD) and corrupted span prediction (CSP). RTD uses a generator to generate ambiguous corruptions and a discriminator to distinguish the ambiguous tokens from the original inputs . RTD is proved to be more sample-efficient than the classic mask language modeling (MLM) task in learning text representations for language understanding (Bajaj et al., 2022;Hao et al., 2021). In CSP, a consecutive segment of tokens are corrupted and the model is learned to predict the corrupted spans using all the uncorrupted tokens in the original input (Raffel et al., 2020;Joshi et al., 2020). CSP can be viewed as a generalized form of gap sentences generation (GSG), a pre-training task tailored to abstractive summarization (Zhang et al., 2020), where the spans are entire sentences. CSP outperforms GSG in our experiments. In the second phase of grounded pre-training (Peng et al., 2022), the model is continually trained on summarization corpora of documents-summary pairs to better support low-resource fine-tuning to downstream summarization tasks that require the model to produce summaries grounded in source documents. We find in our experiments that grounded pre-training significantly boosts the results on downstream tasks in low-resource settings.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
To handle the large input documents, we use fusion-in-encoder (FiE), a simple yet effective method of encoding long sequences in a hierarchical manner. It works by first splitting the input sequence into small chunks, applying attention on each chunk locally to get the chunk representation, and applying attention globally on the concatenated chunk representations to get the representation of the original input.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In addition, we replace the self-attention layer in the encoder with the disentangled attention (DA) layer (He et al., 2020(He et al., , 2021, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. DA is motivated by the observation that the attention weight of a word pair depends on not only their contents but their relative positions. For example, the dependency between the words "deep" and "learning" is much stronger when they occur next to each other than when they occur in different sentences. We show in our experiments that DA leads to a more effective abstractive summarizer.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
For evaluation, we have pre-trained two Z-Code++ models on English data and multi-lingual data, respectively. The English model is trained using 160G English text data and the vocabulary of DeBERTaV2 (He et al., 2020). The multilingual model is trained on mC4 corpus which is the same as mT5. These models are evaluated on 13 text summarization tasks across 5 languages, and create new state of the art on 9 tasks. As of May 6th, 2022, Z-Code++ sits atop of the XSum leaderboard, surpassing UL2 20B , T5 11B and PE-GASUS. It is worth noting that our models are very parameter-efficient. For example, Z-Code++ outperforms PaLM 540B , which is 600x larger in model parameters, on XSum, and outperforms a fine-tuned, 200x larger, GPT3 175B on SAMSum. In zero-shot and few-shot settings, our models outperform more substantially the competing models.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
This section describes three modeling techniques we have exploited to optimize Z-Code++ for abstractive summarization, including two-phase pretraining, disentangled attention, and long sequence encoding.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
The two-phase pre-training, which includes the language model pre-training and grounded pretraining phases, is inspired by the GODEL recipe (Peng et al., 2022) that has been proposed to pretrain language models for grounded text generation tasks, such as dialog response generation and abstractive question-answering.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In the language model pre-training phase, Z-Code++ is pre-trained using two language modeling tasks, replaced token detection (RTD)  and corrupted span prediction (CSP) (Raffel et al., 2020;Joshi et al., 2020). As illustrated in Figure1 (Left), RTD uses a generator trained with MLM to generate ambiguous tokens to replace tokens in the original input X, and a discriminator to determine whether a token is from X or generated by the generator. Let θ G and θ D be the parameters of the generator and the discriminator, respectively. The MLM loss of the generator is written as Text Span
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Figure 1: The two pre-training tasks, replaced token detection (RTD) and corrupted span prediction (CSP), used in the language model pre-training phase of Z-Code++. RTD task is to optimize the encoder, and CSP is to optimize the encoder-decoder. Encoders in the same color share parameters during training.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
whereX G is the input to the generator by randomly masking 15% tokens in original input X. The input sequence of the discriminator is constructed by replacing the masked tokens, x i , i P C, with the tokens,x i , sampled by the generator as
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Then the discriminator is trained using the loss
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
where 1p¨q is the indicator function andX D is the input to the discriminator constructed via (2). In ELECTRA , the discriminator and generator share token embeddings and their parameters are optimized via MLM and RTD jointly as L " L MLM`λ L RTD . However, as pointed out in (He et al., 2021), such embedding sharing makes training highly inefficient since MLM and RTD pull token embeddings into very different directions, creating the "tug-of-war" dynamics. MLM tries to map the tokens that are semantically similar to the embedding vectors that are close to each other. RTD, on the other hand, tries to discriminate semantically similar tokens, pulling their embeddings as far as possible to optimize the classification accuracy. Thus, we use the method of gradientdisentangled embedding sharing (He et al., 2021) by re-parameterizing the token embeddings of the discriminator as
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
where E D and E G are the embedding parameters of the discriminator and generator, respectively, sg is the stop gradient operator which only allows gradients propagation through E ∆ . E ∆ is initialized as a zero matrix. In each training pass, we first run a forward pass of the generator to generate inputs for the discriminator, and then a backward pass to update E G with respect to MLM. After that, we run a forward pass for the discriminator using the inputs produced by the generator and run a backward pass with respect to the RTD loss to update E D by propagating gradients only through E ∆ . After model training, E ∆ is added to E G and the sum is saved as E D in the discriminator, as Equation 4. The CSP is widely used to optimize the encoderdecoder PLMs such as T5 (Raffel et al., 2020). As illustrated in Figure 1 (Right), given input string X, we first select a continuous span Y i by first randomly selecting a start position in X and a span with an average length of 3. Then we replace the selected span Y i with a sentinel token [M i ]. We repeat the process until the replaced tokens amount to 15% of all tokens in X. Then, we feed the corrupted inputX CSR to the encoder. The encoderdecoder model is then trained to recover the Y i from the context. The CSP loss is written as
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
If we restrict the corrupted span Y i to a complete sentence, CSP is equivalent to the GSG task which simulates the process of extractive summarization and is shown to be effective for training abstractive summarizers (Zhang et al., 2020). In this study, we find the that CSP, as a more general form of GSG, works better across many natural language understanding and generation tasks, including summarization, as to be discussed in Section 3.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Combining the pre-training tasks of MLM, RTD and CSP, in the language model pre-training phase, Z-Code++ is optimized using the joint loss as L "
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In the second phase of grounded pre-training, Z-Code++ is continually pre-trained on a collection of summarization datasets, as shown in Table 1, which consist of documents-summary pairs pX, Y q, to better support low-resource finetuning for downstream summarization tasks that require the model to generate target summaries Y grounded in source documents X, as ppY |Xq " N ź n"1 ppy n |y 1 ,¨¨¨, y n´1 , Xq
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Following T0 , FLAN (Sanh et al., 2022), and GODEL (Peng et al., 2022), we add for each training pair pX, Y q a natural language instruction of the summarization task, as illustrated in the below example and in Table 1. In our experiment, we only apply grounded pre-training for low-resource summarizations. Unless specified, we apply the first phase Z-Code++ to downstream task adaptation.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Disentangled Attention (DA) is first used in De-BERTa (He et al., 2020(He et al., , 2021. DA is an extension of the classic self-attention (SA) mechanism in that Instruction: Summarize the following news article into a one sentence summary. Source: Officers searched properties in the Waterfront Park and Colonsay View areas of the city on Wednesday. Detectives said three firearms, ammunition and a five-figure sum of money were recovered. A 26-year-old man who was arrested and charged appeared at Edinburgh Sheriff Court on Thursday.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Target: A man has appeared in court after firearms, ammunition and cash were seized by police in Edinburgh Figure 2: Examples of instructions used for grounded pre-training.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
DA represents each input word using two separate vectors: one for the content and the other for the position. Meanwhile, its attention weights among words are computed via disentangled matrices on both their contents and relative positions. The experiments of DeBERTa shows that DA is more efficient than SA to encode the positional dependency in Transformer models. Z-Code++ adopts DA in modeling. Our experiments show that DA leads to a more effective abstractive summarizer.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
It is challenging to encode long sequence given the OpN 2 q memory and computation complexity of self-attention and DA. Various sparse attention mechanisms have been proposed to alleviate the problem. However, sparse attention often hurts performance on short sequences due to the decrease of attention precision. Inspired by fusion-indecoder (Izacard and Grave, 2020) and hierarchical transformer (Liu and Lapata, 2019a), we propose fusion-in-encoder (FiE), a simple but effective mechanism to encode long sequences while retaining high attention precision on short sequences. FiE works by separating the L encoder layers of Z-Code++ into m local layers and n global layers. In each local layer, the hidden states of input sequence are split into small chunk of size l (e.g. 256 or 512), and self-attention (or DA) is only applied to those small chunks locally with a complexity of Opl 2 q. After local layer, the hidden states of those small chunks are concatenated together to form the representation of the long sequence.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Global layers are the same as original self-attention (or DA) layers in encoder to fuse the local states of small chunks. With FiE, the complexity of encoder is reduced from OpLN 2 q to OpmN l`nN 2 q. Both the local layers and fusion layers are initialized with the corresponding weights of encoder  3 Experiment
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Datasets We validate the effectiveness of Z-Code++ on 11 representative summarization tasks, which are detailed in Table 2. Among these datasets, XSum (  , and Reddit TIFU  are conversation-like summarization tasks. Following LongT5, we use MultiNews, MediaSum, arXiv (Cohan et al., 2018) and PubMed (Cohan et al., 2018) to assess the long document summarization capability. In addition, WikiLingua (Ladhak et al., 2020) and MLsum (Scialom et al., 2020) are used to evaluate the capacity of Z-Code++ on multilingual summarization. We pre-train Z-Code++ LARGE for 1M steps with a batch size of 2048 in Azure Machine Learning cluster 3 with 128 A-100 GPUS for 20 days. AdamW is used as the optimizer in all experiments. For tasks with an input length of more than 10k words, i.e., arXiv and PubMed, Fusion-in-Encoder is used to encode the document as described in 2.3. For the other standard summarization tasks with moderate input length (i.e., less than 4k words) we directly feed the input document to the encoder. For multilingual summarization, we have built Z-Code++ LARGE with the same architecture but different training data and vocabulary. Specifically, Z-Code++ LARGE is trained with mC4 data and a vocabulary of size 250k, which are the same as mT5 (Xue et al., 2021). Following XLM (Lample and Conneau, 2019), CCMatrix (Schwenk et al., 2019) and CCAligned (El-Kishky et al., 2019), parallel data is used to enhance the cross-lingual summarization of Z-Code++ LARGE . Due to the limited computational resource, Z-Code++ LARGE is trained with only 500B tokens instead of 1T tokens as that for mT5 training.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We use grid search to choose the grounded training and fine-tuning hyper-parameters based on validation set, the parameter search range are listed in appendix A.1.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We first conduct experiments to compare the performance of Z-Code++ LARGE with SOTA and PEGASUS LARGE on 7 representative standard public English summarization datasets with moderate document length, including AESLC, SAMSum, XSUM, WikiHow, News-Room, CNN/DailyMail(CNNDM), and Reddit TIFU. Following (Chowdhery et al., 2022;, for each dataset we re-    (Zoph et al., 2022), b T5 11B (Rothe et al., 2021), c GPT3 175B+LoRA (Hu et al., 2021), d MAPPET+BART LARGE (Aghajanyan et al., 2021). port the average F-measure ROUGE-2 score of 5 runs. Detailed F-measure of ROUGE-1/ROUGE-2/ROUGE-L scores can be found in Appendix ??.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
As listed in Table 3 , Z-Code++ LARGE achieves substantial improvements over PEGASUS LARGE , which is a PLM optimized for abstractive summarization, on 6 out of 7 tasks in terms of ROUGE-2 F-measure score. 5 Specifically, on SAMSum, a critical dialog summarization task, Z-Code++ LARGE outperforms GPT-3 175B that is extensively fine-tuned with LoRA(Hu et al., 2021) even though Z-Code++ LARGE has less than 1/175 parameters of GPT-3 175B . Furthermore, Z-Code++ LARGE lifts SO-TAs by 0.36 points on average. These results demonstrate the effectiveness of Z-Code++ on English document summarization tasks. Additionally, we observe that Z-Code++ LARGE outperforms PEGASUS LARGE on WikiHow, SAMSum, Reddit TIFU, and AESLC by a much larger margin (ą 1%) than it does on XSum, CNNDM, and News-Room. We speculate that PEGASUS is biased to news-like tasks since it is heavily pre-trained on large amounts of news data. In contrast, Z-Code++ is pre-trained on diverse web data and thus is more adaptable for general-domain summarization tasks.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We compare Z-Code++ to PEGASUS and LongT5, which is optimized for long document summarization. Results in Table 4     layer of encoder as fusion layer based on the experiment results. Specifically, Z-Code++ LARGE outperforms LongT5 3B with less than 1/3 of parameters. These results demonstrate both the effectiveness and flexibility of Z-Code++ by using Disentangled-Attention to encode word dependencies.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
As human evaluation is the most reliable measurement of the quality of natural language generation models, we submit the test results of XSum to the leaderboard (Khashabi et al., 2021) which requires human raters to compare the generated summaries side by side with human written references. Please check the paper of the leaderboad (Khashabi et al., 2021) to get more details of human evaluation process including instructions, dataset preparing, payments and demographics of the raters. We list the human evaluation results in Table5. Z-Code++ outperforms all the other models, e.g., BART LARGE , PEGASUS LARGE , T5 11B , UL2 20B (Tay et al., 2022), on the leaderboard in terms of human-overall score. As the human evaluation score is an average of sideby-side preference comparison scores, a score of 0.51 indicates that the annotators prefer the output of Z-Code++ to the human written references. Further more, while hallucination is one of the most critical problems for abstractive summarization, Z-Code++ does not suffer much, i.e., 0.55, among the leaderbard. The human evaluation results validate that Z-Code++ produces higher quality summaries than other models.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Following GEM-benchmark (Gehrmann et al., 2021), we evaluate the performance of Z-Code++ LARGE 6 on multilingual summarization with WikiLingua and MLSum. We compare Z-Code++ LARGE with mT5 LARGE and mT5 XLARGE . The results of PaLM 540B , a state of the art PLM, are also listed in Table 6. Compared with mT5 XLARGE , Z-Code++ LARGE achieves substantially better performance across all the tasks with only 1/3 parameters and half training data. In addition, we observe a significant performance gap between Z-Code++ LARGE and PaLM 540B on WikiLingua, which is not surprising due to the sharp difference in model size and capacity. However, Z-Code++ LARGE surpasses PaLM 540B on MLSum by a large margin, i.e., 3.7% on MLSum(de), 2.8% on MLSum(es), albeit Z-Code++ LARGE has less than 1/500 parameters. We believe that by scaling up Z-Code++ to a moderate size (e.g., 10B), the performance gap on WikiLingua would be mitigated. We leave it to future work.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We explore how well knowledge learned in different pre-training stages can generalize to lowresource summarization scenarios, i.e. zero/fewshot evaluation. For the grounded pre-training phase, we choose to include MediaSum, Multi-News, NewsRoom, and WikiHow datasets. Corresponding instructions are listed in Table 1. We reckon that incorporating diverse datasets and instructions is beneficial, which we leave it to future work. For the fine-tuning stage, following the setting in Zhang et al. (2020), we randomly select the number of training data to 0, 10, 100, and 1000, and sample examples from XSUM, CNNDM, and SAMSum, and then fine-tune Z-Code++ until no significant improvement on the validation set is observed. Note that 0 denotes zero-shot evaluation. 
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We present Z-Code++, an efficient and effective pre-trained language model optimized for abstractive text summarization. The model extends the encoder-decoder model using three techniques. The first is a two-phase pre-training process, where the model is first pre-trained using text corpora for language understanding, and then is continually  Results marked with˚mean that unfine-tuned checkpoints perform the best, i.e., zeroshot performance is better than the fine-tuned one. Z-Code++ : LARGE refers to fine-tuning from phase 1 pretrained model. Z-Code++ LARGE fine-tuned from twophase pre-trained model. pre-trained on summarization corpora for grounded text generation. The second technique is the use of the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively. The third is the fusion-in-encoder method for encoding long sequence inputs. We present a comprehensive empirical study to validate the effectiveness of Z-Code++. The model creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. In addition, we show that our model is parameter-efficient in that it outperforms the 600x larger PaLM 540B on XSum, and the finetuned 200x larger GPT3 175B on SAMSum. Z-Code++ also generalizes well to low-resource downstream tasks. For example, in zero-shot and few-shot settings, our model outperforms more substantially the competing models.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
However, evaluation (Liang et al., 2022) and hallucinations are still two long-standing problems of summarizations that we do not touch with in this work, in the future we will 1) explore evaluation metrics that correlate well with human experience, 2) learn to summarize to better align with human preferences (Stiennon et al., 2020;Ouyang et al., 2022), and 3) ground summarization models on world knowledge to largely reduce hallucinations (LeCun, 2022;Hafner et al., 2023).
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In this paper, we introduce Z-Code++, a robust pre-trained model tailored for summarization tasks. However, it should be noted that there are certain limitations to our model. Firstly, the model is not versatile enough as it is specifically designed for summarization. It is unclear whether it performs well on other natural language tasks. Secondly, while FiE can handle document summarization, there are still significant potential for improving cost efficiency. Lastly, the evaluation of multilingual summarization is not thorough enough due to the limitations of available datasets. We intend to address these limitations in our future work.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
The same as all existing generative language models, the generated text of Z-Code++ raises various ethical considerations. One crucial consideration is the issue of potential hallucinations in the summaries generated by the model. The summaries produced by a generative model may not necessarily be faithful to the original article or entirely factual which may mislead the users to make incorrect decisions based on the summary without additional knowledge. In addition, another important consideration is the potential for bias in generated summaries, such as bias based on gender, race, and other factors.   
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We list the rouge scores of summarizaiton tasks in table10
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
A.3 Fusion-in-Encoder structure
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In figure 3, we show the architecture of FiE.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We conducted a comprehensive experiment to explore what is important for the encoder's language understanding ability. Specifically, we experiment on the natural language inference task, e.g., MNLI (Williams et al., 2018), the question answering task,   e.g., SQuAD (Rajpurkar et al., 2016), the summarization tasks, e.g., XSum (Narayan et al., 2018) and CNNDM (See et al., 2017). The results in Table 12 show that using disentangled attention improves MNLI-matched/mismatched accuracy by 0.9%/1.2%, indicating an improvement in the encoder's language understanding ability. This improvement is also reflected in the performance of two summarization tasks, which see an improvement in R2 scores by 0.39% and 0.22%. Removing RTD significantly decreased performance, indicating that it is essential for improving the model's NLU capability.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
In order to assess the model's effectiveness on natural language understanding (NLU) tasks, we conducted experiments using the eight NLU tasks from   the GLUE dataset (Wang et al., 2019). These tasks are commonly used to evaluate sentence classification performance in machine learning. Our model, Z-Code++, was tested using two approaches: adapting only the encoder and fine-tuning with a classification head, similar to BERT, or adapting the encoder-decoder and treating the task as a generation task, similar to T5. We compared Z-Code++ to other encoder-based PLMs with similar structures, including BERT, RoBERTa, ELECTRA, De-BERTa, and DeBERTaV3, as well as T5 for the encoder-decoder comparison.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
The results, shown in Table 13, demonstrate that Z-Code++ performs comparably or better than the other models on all tasks. In particular, Z-Code++ outperformed the other encoder PLMs by an average of more than 1% and outperformed T5 on all tasks with an average improvement of 1.98% in test scores. These results demonstrate Z-Code++ as a strong universal language model with excellent performance on generation tasks and superior performance on NLU tasks.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
We evaluated the language generation performance of Z-Code++ on a range of English tasks, including abstractive document summarization tasks (XSum, CNNDM, Wikilingual-en), a conversa-  tional summarization task (SAMSum), data-to-text tasks (WebNLG-en, E2ENLG) and a question answering task (SQuAD v1.1). We compared the performance of the Z-Code++ model to other stateof-the-art models with similar architectures and parameters, as shown in Table 14.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Results show that Z-Code++ outperforms all of the other models' scores by a large margin in terms of ROUGE and BLEU scores. For example, Z-Code++ significantly outperformed T5 XLARGE on CNNDM by 1% in terms of ROUGE-2 score, on the WebNLG-en task by 6.9%, and about 1% BLEU score on dialog response generation tasks. Even though it has less than 1/3 the parameters of T5 XLARGE , Z-Code++ outperformed PEGASUS on SAMSum task by 4% in terms of ROUGE-2 score. We conjecture that PEGASUS is a model specifically optimized for summarization using 1500GB of news data, which may have introduced a domain mismatch with the conversational summarization task. We also compared Z-Code++ to other stateof-the-art models with extremely large parameters, including PaLM, GPT3, and UL2. Z-Code++ outperformed PaLM on three out of four tasks by a large margin, even though it has less than 1/600 the parameters of PaLM. Z-Code++ also outperformed UL2 20B on four out of five tasks, even though it has less than 1/20 the parameters of UL2 20B . These results demonstrate the efficiency of the Z-Code++ model.  B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
No response.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Section 3, experiments C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.1, experiment setup
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Text summarization aims to produce concise and accurate summaries of long texts. Recent research on pre-trained neural language models has shown success in summarizing monologues (Lewis et al., 2020;Raffel et al., 2022;He et al., 2022), such as news articles Ravaut et al., 2022) and scientific publications (Ibrahim Altmami and El Bachir Menai, 2022;Dong et al., 2021). However, dialogue summarization presents additional challenges due to the different information distribution in dialogues.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Self-supervised text summarization models Wan and Bansal, 2022; Phang * Work was done when Yu Li was interning at MSR Figure 1: A summary of a dialogue in the SAMSum dataset, where the golden summary effectively compiles relevant information (in yellow) from the entire conversation. et al., 2022) are typically pre-trained on free-form text data, with selected sentences as the pre-training objective. While this approach can be effective for monologues such as news articles, it is less successful at summarizing semistructured and multiparticipant dialogues. As illustrated in Figure 1, in daily chats, dialogue information is often dispersed across various dialogue turns, making it difficult to extract all relevant information through a few selected turns. While a golden summary needs to accurately captures vital information throughout the entire conversation. Furthermore, real-world dialogue-summarization applications often have limited or even no labeled data, making it challenging to develop effective models. Therefore, it is crucial to develop dialogue summarization models that can perform well in zero-shot and few-shot settings for their practical use.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
To address these challenges, we propose DIONY-SUS, a pre-trained sequence-to-sequence model designed to summarize dialogues in any domain, even with a lack of labeled data. It uses pseudo summaries as its pre-training objective, which can be dynamically selected from two sources.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
First, for daily chats where multiple dialogue turns are not sufficient to summarize the dialogue, we train a summary helper using high-quality dialogue summarization datasets to generate pseudo summaries for these types of dialogues. On the other hand, for dialogues like meeting minutes, interviews, and debates, which can be summarized through a selection of essential turns, we use a method inspired by the gap sentence generation (GSG) technique in PEGASUS to select these turns as pseudo summaries for training. For instance, choosing the final few turns in a conversation can effectively summarize meeting minutes. We have improved upon the GSG method by using the generated summaries from the summary helper as references during gap sentence selection, as they tend to have less noise compared to the full dialogue context. We refer to this source of pseudo summaries as "Principal" and refer to our improved method as GSG+. We find that our improved method outperforms previous methods in low-resource settings across different domains, such as daily chats, emails, and customer service dialogues. Additionally, we study different objective strategies for selecting the pseudo summary as a pre-training objective from the generated summary and the "Principal. " We evaluate DIONYSUS on six dialogue summarization datasets. Our best model trained on 19 dialogue corpora surpasses PEGASUS LARGE in a zero-shot setting across all domains. We also found that the best performance is achieved by selecting the source with the highest ROUGE score as the objective strategy. Our main contributions are:
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
• The development of DIONYSUS, a pretrained sequence-to-sequence model for summarizing dialogues in any domain in a zeroshot or few-shot setting.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
• The introduction of new self-supervised pretraining objectives for dialogue summarization using a summary helper and GSG+.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
• The demonstration that DIONYSUS outperforms baselines on six domains in low-resource settings, and can be fine-tuned with only 10 training examples to outperform vanilla T5 (Raffel et al., 2022) fine-tuning with 1, 000 examples. Figure 2 outlines the steps for constructing DIONY-SUS: § 2.1 First, a summary helper is constructed using two high-quality dialogue summarization datasets. This helper generates a pseudo summary for each dialogue in our pre-training corpus. § 2.2 Next, the "Principal" is extracted using GSG+ as the other pseudo summary for the dialogue. § 2.3 Finally, various strategies are employed to select the best pseudo summaries from the first and second steps to serve as the objective for pre-training.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In certain types of dialogue, such as daily chats, it can be challenging to gather all necessary information from just a few dialogue turns due to the dispersed nature of dialogue information. To address this problem, we have created a summary helper model that generates pseudo summaries for each training example in our pre-training corpus. We build our summary helper upon the T5 (Raffel et al., 2022) model. To capture essential information in a dialogue, we have trained our helper on the MultiWoz dataset (Budzianowski et al., 2018;Eric et al., 2020) in DS2 (Shin et al., 2022), which contains summaries derived from dialogue states using templates. This allows us to capture essential information from each turn in the conversation. Additionally, we have continued training our helper on the DialogSum (Chen et al., 2021) dataset, a human-annotated dataset in the daily life domain. This allows us to overcome the fixed format of summaries introduced by templates in DS2 and produce more natural pseudo summaries.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Algorithm 1 GSG+ 1: P ← ∅ 2: for j ← 1 to m do 3:
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
k := argmax{s i } n
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
P := P ∪ {x k } 6: end for Dialogues in certain settings, such as meetings and medical dialogues, often include summary Figure 2: A diagram of pre-training in DIONYSUS: The summary helper ( § 2.1) generates a pseudo-summary (G) to select dialogue turns ( § 2.2) as the "Principal" (P) and using various strategies ( § 2.3) to choose between the generated summary and the principal as the pre-training objective.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
turns that summarize the entire conversation. For example, a participant may summarize a meeting, or a doctor may explain the outcome. These summary turns can be used as a pre-training objective because they highlight the main points of the dialogue and provide a concise overview of the topic discussed. In order to make DIONYSUS more adaptable to these scenarios, we have improved the independent principal method in the GSG method  by using it to select essential summary turns as pseudo summaries for training. Our new method, called Gap Sentence Selection Plus (GSG+), uses the ROUGE1-F1 score between each dialogue turn x i and the generated summary G from the helper in Section 2.1 rather than the remaining text D \ x i to determine the importance of each turn. The generated summary eliminates much of the extraneous information from the dialogue and thus tends to have less noise than the full dialogue context, resulting in a less cluttered summary. This enables us to select the top-m-scored summary turns as the "Principal," which will provide a more comprehensive overview of the vital information in the dialogue. For instance, Using the summary helper to identify key points increases the likelihood of selecting the most important dialogue turns as the "Principal" summary when creating pseudo summaries for meeting minutes instead of randomly selecting dialogue turns.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Specifically, given a dialogue D = {x i } n , we use Algorithm 1 to obtain the pseudo-summary "Principal" P . The input for our training example is the remainder of the dialogue D \ P . In Appendix C, we explore the impact of the dialogue turns order on the formation of the "Principal". Using GSG+ can effectively identify essential summary turns and generate more accurate pseudo-summaries than with the original GSG method. 
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
To generate the final pseudo summary S for each specific dialogue training example, we consider three strategies. These strategies are based on the generated pseudo summary G and the extracted "Principal" P . These strategies serve as the pretrain objective for the dialogue training example.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
All G S = G: We always select the generated summary from the summary helper as the pretraining objective.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
All P S = P : We always select the "Principal" as the pre-training objective.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Better ROUGE We use either G or P based on the recall of information from the dialogue to determine the pre-training objective. We utilize Algorithm 2 to get the pre-training objective by calculating the ROUGE1-F1 score for the pseudo summaries and the dialogue, excluding the "Principal" D \ P . It is important to note that we use the same reference to ensure a fair comparison.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
For pre-training with above strategies, if we choose G as the pseudo summary, we input the full dialogue. If we choose P , we input the dialogue, excluding the "Principal," D \ P to create an abstract summary. However, we also include the "Principal" with a probability, using a copying mechanism to create an extractive summary.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
More information about this copy mechanism can be found in Section 5.4. It is important to note that we do not combine these two pseudo summaries for a single training example. Each example in our pre-training corpus will have either G or P as its designated pseudo summary.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
To train DIONYSUS, we utilized 19 conversational corpora that do not come with pre-defined dialogue summaries. We employed a self-supervised approach by using pseudo-summaries as the pretraining objective.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Conversational Corpora We collect 19 available conversational corpora consisting of 1.7M examples after truncating for pre-training. Corpus information is listed in Table 1. We access these corpora through ConvoKit v2.5.3 1 . This helps us to ensure that DIONYSUS is well-equipped to handle a variety of conversational scenarios.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
CaSiNo (Chawla et al., 2021) 1,030 Chromium (Meyers et al., 2018) 163,675 Gone Awry (CMV) (Zhang et al., 2018) 6,842 Gone Awry (Wiki) (Zhang et al., 2018) 4,188 Diplomacy (Peskov et al., 2020) 246 Friends (Zhou and Choi, 2018) 1,301 GAP (Braley and Murray, 2018) 28 IQ2 (Zhang et al., 2016) 108 Cornell Movie Dialogs 2 83,097 Parliament (Zhang et al., 2017b) 216,894 PERSUASIONFORGOOD 3 1,017 Reddit Coarse (Zhang et al., 2017a) 9,483 Reddit Corpus (small) 4 8,286 Supreme Court 5 7,700 Switchboard (Stolcke et al., 2000) 1,155 Tennis (Fu et al., 2016) 81,974 Wiki Deletion (Mayfield and Black, 2019) 383,918 Wiki Talk Pages 6 125,292 Winning Arguments (Tan et al., 2016) 3,051 We train our objective summary helper with a rule-based dialogue summarization dataset (DS2) and an abstractive summarization dataset (Dialog-Sum).
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
DS2 This dataset (Shin et al., 2022) creates dialogue summaries for the MultiWOZ (Budzianowski et al., 2018;Eric et al., 2020) dataset by heuristic rules from the dialogue states. It includes 5 domains and 10, 000 dialogues.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
DialogSum This dataset (Chen et al., 2021) collects human annotated summaries for daily-life dialogues from three datasets: DailyDialog (Li et al., 2017), DREAM (Sun et al., 2019), andMuTual (Cui et al., 2020), as well as dialogues from an English-speaking practice website. It has 13,460 dialogues in total.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We evaluate our methods on three public dialogue summarization datasets or benchmarks: SAMSum ConvoSumm It is a benchmark of four domains: New York Times comment, StackExchange, W3C email, and Reddit. Dialogues are extracted from publicly available data, and each domain has 500 dialogues. They hire crowdsorce workers on Amazon Mechanical Turk to annotate dialogue summary.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
TweetSumm This dataset contains 1,100 reconstructed real-world customer support dialogues from Tweet. Each dialogue has human annotated abstractive summaries and extractive summaries. We only use abstractive summaries in the dataset as references in our experiments.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We report ROUGE-1, ROUGE-2, and ROUGE-L scores (Lin, 2004) to evaluate generated summaries against references.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We compare our methods with three competitive baselines.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
T5v1.1 It is an improved version of the original T5 model (Raffel et al., 2022). Since the original T5 model is pre-trained on downstream tasks in supervised learning, the test set of downstream tasks overlaps with the pre-training data. To make a fair comparison in a zero-shot setting, we choose T5v1.1 as it is pre-trained on C4 without mixing in the downstream tasks. PEGASUS Zhang et al. (2019) propose this pretrained model for abstractive summarization tasks. The pre-training objective is GSG, transforms any text into an abstractive summarization example by selecting important sentences as output summaries. We use the PEGASUS LARGE checkpoint 7 as there is no publicly available PEGASUS BASE checkpoint.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
GSG* We use the independent principal strategy of GSG training objective in PEGASUS  but pre-train DIONYSUS with our training corpora. We build this baseline to explore the performance gap between our pre-training objective and GSG.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We focus on low-resource dialogue summarization settings because it is difficult to collect enough training examples. We evaluate DIONYSUS with "All G", "All P", and "Better ROUGE" strategies in zero-shot and few-shot settings and compare it to the baselines.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In order to evaluate the effectiveness of DIONYSUS, we conduct a zero-shot test on DIONYSUS LARGE with all strategies and other baselines. We present the results in Table 2. The ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1 scores are used as the standard evaluation measures for summarization tasks. Our models show impressive performance improvements over the baselines on all downstream datasets. Specifically, DIONYSUS LARGE with the "Better ROUGE" strategy performs the best overall across all downstream datasets (Average: ROUGE-1/2/L: 29.7/8.0/20.2), indicating that it benefits from both generated and extractive pseudo summaries and can adapt to various domains. The "All P" strategy performs better than the GSG* baseline on most datasets, indicating that our Gap Sentence Selection Plus method can effectively select dialogue turns that provide an accurate dialogue summary. Additionally, the DIONYSUS LARGE with "All G" and "Better ROUGE" strategies demonstrate significant improvement compared to T5v1.1 LARGE (Average ROUGE2: +5.6/ + 6.1) and PEGASUS LARGE (Average ROUGE2: +2.2/ + 2.7), indicating that pre-training with our summary helper is 7 https://huggingface.co/google/pegasus-large highly beneficial. However, the "All G" strategy only performs as well as the "Better ROUGE" strategy on the SAMSum dataset (ROUGE-1/2/L/: 41.3/16.1/30.6 → 41.3/16.2/30.9), suggesting that the improvement from the summary helper is more pronounced on this particular dataset. This may be due to the similarity between the datasets used to train the helper and the SAMSum dataset, which we discuss further in Sections 5.5 and 5.6. Overall, our models outperform previous methods, such as PEGASUS, in a zero-shot setting, demonstrating their effectiveness and potential for further development.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We investigated reducing annotation labor in dialogue summarization tasks by using few-shot dialogue summarization. We report ROUGE1-F1, ROUGE2-F1, ROUGEL-F1, and ROUGELSum-F1 scores to evaluate model performance. Specifically, We fine-tune DIONYSUS LARGE , PEGASUS LARGE , and T5v1.1 LARGE with the first 1/10/100/1K/10K training examples from the SAMSum dataset. We show the results of our experiments with varying training data sizes in Figure 3. We found that all models improved with more examples. Among these models, DIONYSUS LARGE consistently outperformes both PEGASUS LARGE and T5v1.1 LARGE when trained with a dataset ranging from 0 to 10, 000 examples. This suggests that our pre-training process helps DIONYSUS adapt to downstream tasks more quickly. Additionally, we observed that PEGASUS LARGE outperformed T5v1.1 LARGE due to its pre-training on summarization tasks. Figure  3 shows the gap between DIONYSUS LARGE and PEGASUS LARGE is particularly significant when using fewer than 100 training examples, indicating better recall capabilities in dialogue summarization for DIONYSUS. Even with only 10 training examples, DIONYSUS LARGE achieves higher ROUGE scores than the T5v1.1 LARGE model trained with 1,000 examples, making it the best option for lowresource dialogue summarization.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In GSG+, we can choose a fixed number of turns in the dialogue as a training objective or select turns with a compression ratio. We investigate the compression ratio in a dialogue turn level as the number of selected turns over the number of totals turns in the dialogue (N principal /N dialogue ). A  Table 2: The ROUGE-1/ROUGE-2/ROUGE-L scores of the DIONYSUS LARGE with strategy P: "All P", G: "All G", and BR: "Better ROUGE" and compared to T5v1.1 LARGE and PEGASUS LARGE in a zero-shot setting on three datasets: SAMSum, ConvoSumm, and TweetSumm.  low compression ratio will select fewer turns in the dialogue as the objective, making pre-training less challenging. However, it tends to have a lower ROUGE1-F1 score with the remaining dialogue turns, meaning the "Better ROUGE" strategy selects more generated summaries as the objective.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
While choosing a high compression ratio will make the pre-training more challenging. Nevertheless, it has a higher ROUGE score compared to generated summaries, leading to more principal under the "Better ROUGE" strategy. We show the zero-shot performance on development sets of the SAMSum dataset and TweetSumm dataset with compression rates from 10% to 60% in Figure 4. It shows that the model with 15% compression ratio achieves the highest ROUGE-2 score.  Table 3: ROUGE-1/2/L scores of zero-shot setting for DIONYSUS BASE with "All P" strategy and "All P" without copying mechanism on SAMSum, ConvoSumm, and TweetSum.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
The copying mechanism is important for dialogues like meetings and medical dialogues because it allows for summarization of entire dialogue through several turns. As shown in Table 3, we compare the performance of the "All P" strategy to a scenario where 50% of the selected dialogue turns are retained in the input rather than being removed. In this case, the input for each pre-training example includes the entire dialogue D, rather than D \ P . This leads the model to focus on extractive summarization. We observed that adding a random copy mechanism significantly improved the overall performance. Additionally, we Figure 5: Comparing probabilities of copying selected sentences in the input of the "Principal" using the "Better ROUGE" strategy. Evaluating performance using the ROUGE2-F1 metric on SamSum and TweetSumm development datasets. also evaluate the "Better ROUGE" strategy with different copying probabilities ranging from 0.15 to 0.7. In these experiments, we choose top-2 dialogue turns as principal, which results in 51.9% of pre-training objectives being the principal, and the rest is the generated summary. Figure 5 shows that leaving 15% of dialogue turns in the principal best enhances the overall quality of dialogue summarization.  Table 4: ROUGE-1/2/L scores of zero-shot setting for DIONYSUS BASE with "All G" strategy and the summary helper on SAMSum, ConvoSumm, and TweetSum.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Since the summary helper model provides the generated summary as an objective candidate and has shown strong capabilities in zero-shot dialogue summarization. As shown in Table 4, we compare the helper model to our "All G" model in a zeroshot setting. The difference is that we train the "All G" model on the pre-training corpora annotated by the helper. We found that the helper model is not on par with our model. While the helper model may have performed well on a particular task (NYT), its overall performance is not as strong as our model. This is because DIONYSUS has been extensively trained on various dialogue datasets, which makes it consistently perform well in a wide range of tasks and scenarios.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Threshold ConvoKit DS2 DialogSum In order to ensure a fair comparison, we check for overlap between pre-training and downstream test datasets. This is done by calculating the similarity between all pairs of test set targets in the SAMSum dataset and pre-training documents using the ROUGE2-recall measure, which is calculated as the number of overlapping bigrams divided by the total number of bigrams in the test target. We then count the number of test set examples that have a similarity to any pre-training example above a certain threshold. As shown in Table 5, the overlap between the SAMSum dataset and the datasets used for training the helper and the pre-training datasets is low when the similarity threshold is set between 0.4 and 1.0. This suggests that there is not significant similarity between our test set and the pre-training datasets. It indicates that the improvement in DIONYSUS is due to the pre-training process rather than potential test data leakage.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Human-written 4.08 We evaluate the performance of DIONYSUS by conducting human evaluation experiments on Amazon Mechanical Turk. We randomly select 100 examples from the SAMSum dataset to compare summaries generated by our model with those written by humans in the dataset. We choose DIONY-SUS trained with the "Better ROUGE" strategy and generate summaries in a zero-shot setting. Participants are asked to rate the summaries on a scale of 1 to 5, with higher scores indicating better quality. We collect the scores from three participants for each example and report the average scores in Table 6. A paired t-test is conducted to determine if scores are significantly different between our model and other models. Our results show that DIONYSUS could generate summaries of similar quality to human-written summaries without any training data. DIONYSUS also gets better ratings than the vanilla T5 and PEGASUS models, which aligns with the results obtained from the automatic evaluation. More information on the human evaluation process can be found in Appendix F.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We present DIONYSUS, a pre-trained encoderdecoder model for zero-shot dialogue summarization in any new domain. We pre-train using a self-supervised approach that generates pseudosummaries for large dialogue corpora as the pretraining objective. We investigate the impact of various pre-training objective strategies and model sizes on dialogue summarization performance. Our experiments show that DIONYSUS outperforms state-of-the-art models on six datasets in a zeroshot setting. Furthermore, DIONYSUS can be fine-tuned with only 10 examples to outperform vanilla T5 fine-tuning with 1,000 examples. This makes dialogue summarization more practical and easier to use in various contexts with minimal effort. We plan to extend this method to abstractive summarization tasks to develop a general zero-shot summarization model.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Training Data Our pre-training data is sourced from 19 existing dialogue datasets. However, it's important to note that these datasets may contain noise, such as harmful content, irrelevant file names, and URL links. Despite utilizing multiple automatic tools to filter out this content during preprocessing, there is still a chance that some noise may be present in our pre-training data. This could potentially impact the performance of DIONYSUS, making it important to monitor and improve the pre-processing steps continuously.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We also know the potential drawbacks of constructing pseudo summaries using the GSG method, which may lead to unnatural summaries for dialogue data. To mitigate this, we introduced the Summary Helper in Section 2.1, which is specifically trained on two dialogue summarization datasets containing natural summaries. This approach enables more realistic pseudo-summaries and enhances zero-shot performance. Although we employ top-m turns as an additional source of pseudo summaries, Figure 4 illustrates that GSG+ contributes a minor portion of the pseudo summary, with a 0.7 to 0.3 ratio between generated and topm turns. Our method thus minimizes referent and pronoun confusion, ensuring better coherence than solely employing the standard GSG technique.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Training Resource To improve our model's performance, we employ the "Better ROUGE" strategy, which calculates the ROUGE score for both candidates and selects the best one as the final training objective. This data pre-processing process can be pretty time-consuming, taking approximately one day to complete for our pre-training data when utilizing 100 threads. Additionally, we utilize 16 Nvidia V100 GPUs to train our models, which may not be accessible or reproducible for all researchers. This could present a significant obstacle for those looking to replicate or build upon our work.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Test Data Another potential concern is the test datasets used to evaluate DIONYSUS. The test set size is relatively small, which may not fully represent the breadth of dialogue types that a general dialogue summarization model should be able to handle. This could lead to the model performing well on the test set but not generalizing to other unseen dialogue types. Further, our analysis did not include the assessment of long dialogue summarization, such as lengthy meetings (Carletta et al., 2005;Zhong et al., 2021;Janin et al., 2003) or screenplays (Chen et al., 2022). However, our study's approach has the potential to handle these scenarios, even though it was not specifically designed for them. By incorporating LongT5 (Guo et al., 2022) or DialogLM (Zhong et al., 2022), which are known for their ability to process extended input sequences, we expect that they could efficiently tackle this task.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Our gratitude goes out to Microsoft Research for providing us with computational resources. We would also like to thank Kun Qian for valuable discussions and the Columbia NLP and Microsoft Deep Learning Group members for their feedback and discussions. Additionally, we thank the Mechanical Turk workers for conducting the human evaluation. 
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Following Raffel et al. (2022) and  to save time and computation, we first conduct ablation experiments on a reduced-size T5v1.1 BASE model with 250M parameters. Then we scale up with the best settings to the final T5v1.1 LARGE model with 800M parameters. We use heuristics to clean up our pre-training corpora. First, we remove dialogues with less than two dialogue turns since they are too short to summarize. Then we remove URLs and emojis in the text. DIONYSUS is implemented with Huggingface Pytorch Transformers 8 (Wolf et al., 2020). We split dialogue turns with line breakers in pre-training input and add a "[Summary]" prefix. For pseudo summary creation, we use a compression ratio of 0.15 for the "Principal." This means that for a dialogue with l turns, we select 0.15l turns as "Principal." We explore the effect of different compression ratios in Section 5.3. We use Adam (Kingma and Ba, 2014) with weight decay for pre-training. We truncate dialogue training examples to ensure a maximum length of 512. Models are pre-trained with batch size 8 and learning rate 0.00001 on 16 Nvidia V100 GPUs until we observe no progress on validation data or up to 5 epochs. For few-shot experiments in Section 5.2, we fine-tune models up to 20 epochs with batch size 8 and learning rate 0.00005, and pick the checkpoint with the best validation performance. Table 7 presents the results of DIONYSUS BASE in a zero-shot setting, and Figure 6 compares the few-shot results of DIONYSUS BASE with those of the T5 base model. These initial results demonstrate the potential for further analysis and optimization of DIONYSUS. Upon comparison with other baselines, it is clear that DIONYSUS performs better under both zero-shot and few-shot conditions, outperforming the GSG* model. These results provide valuable insight into the capabilities of DIONYSUS and can inform the development of larger models.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
We could use two possible orders to align the dialogue turns in the principal. The first order is to align the text with the ROUGE1-F1 score. The second order is to align the principal with the order in the original dialogue. This means that the principal will be arranged in the same order as in the original dialogue, without rearrangement. This option helps preserve the original flow and structure of the dialogue. We compare these two orders of principal in the GSG* baseline. As shown in Table 8, the results suggest that keeping the order in the original dialogue helps improve zero-shot performance as it provides a more nuanced understanding of the dialogue. We choose this order for all our models.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
To evaluate the performance of DIONYSUS during pre-training, we measured the ROUGE1-F1, ROUGE2-F1, ROUGEL-F1, and ROUGELSum-F1 scores on the SAMSum dataset in Figure 7. We keep track of the model's progress by logging its performance every 1,000 training steps. This allows us to monitor the model's improvements over time and confirm that it is learning effectively.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In order to evaluate the performance of DIONY-SUS, we randomly selected model output examples   from both the SAMSum dataset and the Tweet-Summ dataset. We report these examples with their corresponding gold summaries in Tables 9 and 10. The gold summaries served as a benchmark for our model's output, allowing us to compare and estimate the quality of the generated summaries. We found that DIONYSUS could generate zero-shot summaries on par with those written by humans. However, we also identified factual errors in the generated summaries, such as misunderstandings of the subject matter. These errors suggest room for improvement in DIONYSUS, and we plan to address this issue in future work.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
In our human evaluation experiments, we utilized the task template shown in Figure 8. Mechanical workers were instructed to rate four summaries for a given dialogue on a scale of 1 (poor) to 5 (excellent). To minimize bias, we provided a di-alogue with its corresponding gold summary as an example of a high-quality summary. The summaries were presented in a randomized order for each task to prevent order bias. Three different workers independently completed each task, and the median score across all workers was retained for each summary. Participants were compensated with 0.3 USD per task, and we implemented the following qualifications for worker selection to ensure a high level of quality: (1) HIT approval rate for all requesters' HITs is greater than 90%.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
(2) Location is one of AU, NZ, GB, and US.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
(3) Number of HITs approved is greater than 100. Figure 7: Performance of DIONYSUS on the SAMSum dataset during pre-training process.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
The customer is complaining that he facing some uncomfortable vibe. The agent confronted the customer saying that they had a few problems recently and they are looking into improving security.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
The store is getting a more an more uncomfortable vibe. Nick asks Tesco to bring security back to the Hall Green store and confirms the location. Nick also tells Tesco the Express store has had some problems recently and is looking into improving security. A2. Did you discuss any potential risks of your work?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
A3. Do the abstract and introduction summarize the paper's main claims?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Abstract and Section 1 A4. Have you used AI writing assistants when working on this paper?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Left blank.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
B Did you use or create scientific artifacts?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Section 1 B1. Did you cite the creators of artifacts you used?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 1 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. It is discussed in the original artifacts I use.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. It is discussed in the original artifacts I use.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Section 5 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A D Did you use human annotators (e.g., crowdworkers) or research with human participants?
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Section 5.7
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix F
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? Appendix F D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix F D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. It is in the Amazon Mechanical Turk user agreement protocal.
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. It is in the Amazon Mechanical Turk user agreement protocal.