What are different methods proposed for unsupervised abstractive summarization?
Unsupervised abstractive summarization In
unsupervised abstractive summarization, SummAE
(Liu et al., 2019a) proposes to auto-encode para-
graphs with a sequence-to-sequence model and de-
code single-sentence summaries from the latent
embeddings. SEQ3 (Baziotis et al., 2019) also uses
an auto-encoder to compress the input then recon-
struct it into a differentiable manner, the encoder
output serving as a summary. However, both meth-
ods stick to unsupervised sentence summarization.
More recent approaches typically rely on language
models being pre-trained, then used in a zero-shot
fashion. PEGASUS (Zhang et al., 2020) treats
salient sentences as pseudo abstractive targets to
build a pre-training objective. TED (Yang et al.,
2020) exploits the lead bias in news articles and
takes out the first sentences of the document as
pseudo summary targets for pre-training. Due to
their pre-training objective built for summary gen-
eration, these pre-trained models can be directly
used for unsupervised summarization. The Sum-
mary Loop (Laban et al., 2020) uses reinforcement
learning to train a model to fill-in deleted important
words from the source document using the sum-
mary generated so far, then refines this summary.
Unsupervised Summarization Re-ranking
With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).
[{"first":"Mathieu","last":"Ravaut","middle":[]},{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Nancy","last":"Chen","middle":[]}]
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder (SEQ^3), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets.
[{"first":"Christos","last":"Baziotis","middle":[]},{"first":"Ion","last":"Androutsopoulos","middle":[]},{"first":"Ioannis","last":"Konstas","middle":[]},{"first":"Alexandros","last":"Potamianos","middle":[]}]
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples.
[{"first":"Jingqing","last":"Zhang","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Mohammad","last":"Saleh","middle":[]},{"first":"Peter","last":"Liu","middle":["J."]}]
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We propose an end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and introduce a benchmark task, ROCSumm, based on ROCStories, a subset for which we collected human summaries. In this task, five-sentence stories (paragraphs) are summarized with one sentence, using human summaries only for evaluation. We show results for extractive and human baselines to demonstrate a large abstractive gap in performance. Our model, SummAE, consists of a denoising auto-encoder that embeds sentences and paragraphs in a common space, from which either can be decoded. Summaries for paragraphs are generated by decoding a sentence from the paragraph representations. We find that traditional sequence-to-sequence auto-encoders fail to produce good summaries and describe how specific architectural choices and pre-training techniques can significantly improve performance, outperforming extractive baselines. The data, training, evaluation code, and best model weights are open-sourced.
[{"first":"Peter","last":"Liu","middle":["J."]},{"first":"Yu-An","last":"Chung","middle":[]},{"first":"Jie","last":"Ren","middle":[]}]
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Text summarization aims to extract essential information from a piece of text and transform the text into a concise version. Existing unsupervised abstractive summarization models leverage recurrent neural networks framework while the recently proposed transformer exhibits much more capability. Moreover, most of previous summarization models ignore abundant unlabeled corpora resources available for pretraining. In order to address these issues, we propose TED, a transformer-based unsupervised abstractive summarization system with pretraining on large-scale data. We first leverage the lead bias in news articles to pretrain the model on millions of unlabeled corpora. Next, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of generated summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are highly abstractive, and each component in the objective function of TED is highly effective.
[{"first":"Ziyi","last":"Yang","middle":[]},{"first":"Chenguang","last":"Zhu","middle":[]},{"first":"Robert","last":"Gmyr","middle":[]},{"first":"Michael","last":"Zeng","middle":[]},{"first":"Xuedong","last":"Huang","middle":[]},{"first":"Eric","last":"Darve","middle":[]}]
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. It introduces a novel method that encourages the inclusion of key terms from the original document into the summary: key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. A novel unsupervised training procedure leverages this coverage model along with a fluency model to generate and score summaries. When tested on popular news summarization datasets, the method outperforms previous unsupervised methods by more than 2 R-1 points, and approaches results of competitive supervised methods. Our model attains higher levels of abstraction with copied passages roughly two times shorter than prior work, and learns to compress and merge sentences without supervision.
[{"first":"Philippe","last":"Laban","middle":[]},{"first":"Andrew","last":"Hsi","middle":[]},{"first":"John","last":"Canny","middle":[]},{"first":"Marti A.","last":"Hearst","middle":[]}]
Unsupervised Summarization Re-ranking
Transformer-based encoder-decoder language models have achieved great success in abstractive summarization in the last few years, and produce fluent summaries which can be quite abstractive (Raffel et al., 2019;Lewis et al., 2020;Zhang et al., 2020). These models follow the pre-train then fine-tune paradigm: they are first pre-trained with a selfsupervised objective on a large text corpus; then they are fine-tuned on the downstream dataset of interest, using the available supervision, which may be very scarce. Finding a better pre-training objective remains an active research area. Some models like T5 (Raffel et al., 2019) and BART (Lewis et al., 2020) adopt a more general language modeling objective (e.g., masked span generation), while   al., 2020) or TED (Yang et al., 2020) are pre-trained specifically for the task of summarizing a document. PEGASUS uses salient sentences of the document as a proxy summary label, while TED leverages the lead bias to get the pseudo-summary target.
Unsupervised Summarization Re-ranking
Despite the impressive success on supervised abstractive summarization tasks, unsupervised summarization remains very challenging. The LEAD-3 (extractive) baseline which simply takes the first three sentences of a document as its summary, remains far ahead of unsupervised approaches on several news summarization datasets (See et al., 2017), especially the popular CNN/DM dataset (Hermann et al., 2015). In fact, it was only improved on by supervised abstractive models not more than five years ago (Narayan et al., 2018). It is expected that a model which has never seen any summarization example would struggle, as summarization is a task that is subjective and complex even for humans (Kryscinski et al., 2019). Since summarization labels are expensive to collect, it is essential to develop models with good zero-shot performance. Starting from instruction-tuned GPT-3, LLMs are offering promising performance in zero-shot summarization (Goyal et al., 2022), but remain an unscalable solution as these models are arXiv: 2212.09593v3 [cs.CL] 
Unsupervised Summarization Re-ranking
rarely open-source, and extremely computationally intensive.
Unsupervised Summarization Re-ranking
Recently, in the supervised setup, second-stage approaches have gathered interest in abstractive summarization research. While the base encoderdecoder model is trained with maximum-likelihood estimation (MLE) to predict each token of the ground-truth summary in an autoregressive manner, second-stage methods work with a global view at the whole sequence level. SimCLS (Liu and Liu, 2021) and SummaReranker (Ravaut et al., 2022a) propose to train another neural model to rank summary candidates generated by decoding methods like beam search (Reddy, 1977) or diverse beam search (Vijayakumar et al., 2016). BRIO (Liu et al., 2022a) bypasses the need for another model, and re-uses the fine-tuned model for another fine-tuning stage in which the model also learns to rank candidates in the correct order. SummaFusion (Ravaut et al., 2022b) encodes each summary candidate separately and decodes into a new, abstractive secondstage summary. Such second-stage methods have improved ROUGE-1 state-of-the-art on CNN/DM by more than 3 points (Liu et al., 2022a).
Unsupervised Summarization Re-ranking
In this paper, we propose to re-rank summary candidates in the unsupervised setup. Following observations made by second-stage summarization studies in the supervised setup (Liu et al., 2021;Ravaut et al., 2022a), we also observe large variance in performance among summary candidates in the unsupervised setup. In Table 1, the oracle for PEGASUS, which is the summary candidate maximizing the ROUGE score with the reference, reaches 42.62 when using beam search with 20 beams on CNN/DM (Hermann et al., 2015). This is in the same range (42-45 ROUGE-1) as the top beam of supervised leading models on this dataset (Lewis et al., 2020;Zhang et al., 2020). This observation implies strong potential motivating our work: with a perfect unsupervised summarization re-ranker, one could potentially by-pass supervised fine-tuning and just re-rank instead.
Unsupervised Summarization Re-ranking
The main challenge lies in the fact that the reranker must also not access any supervision. Our proposed model does not train any neural model, but simply computes features indicative of summary quality to score each summary candidate, some of them which also leverage the source document. A weighted average of these features is used for candidate re-ranking, and we explore several methods to estimate the feature weights. Our method, named SummScore, is lightweight, fast and easy to use as it does not rely on a neural network. Since it is purely unsupervised, the re-ranked results can provide more refined self-supervision to the pre-trained models, complementing the pretraining with rounds of self-training.
Unsupervised Summarization Re-ranking
Our contributions in this paper are threefold:
Unsupervised Summarization Re-ranking
• We propose SummScore, the first system to rerank summarization candidates in an unsupervised setup and in an unsupervised manner.
Unsupervised Summarization Re-ranking
• We demonstrate the strength of SummScore by consistent performance improvement: up to +7.27% with PEGASUS and +6.86% with Chat-GPT 2 mean ROUGE gains over four unsupervised summarization datasets, +7.51% mean ROUGE gains averaged over 30 zero-shot transfer setups.
Unsupervised Summarization Re-ranking
• Using the re-ranker, we derive an original and effective self-training method which continuously improves the base unsupervised summarization model, pushing PEGASUS from 35.47 to 39.76 ROUGE-1 (+12.09%).
Unsupervised Summarization Re-ranking
Second-stage or sequence-level methods are gaining traction recently in supervised summarization. Among such methods, re-ranking consists in selecting a better summary candidate out of several of them produced by a base model (which has already been fine-tuned). RefSum (Liu et al., 2021) uses a meta-learning approach to learn how to rank summaries coming from multiple systems. SimCLS (Liu and Liu, 2021) trains a RoBERTa (Liu et al., 2019b) model with a ranking loss to learn how to rank summary candidates generated by a base BART or PEGASUS in their target metric order. SummaReranker (Ravaut et al., 2022a) also trains a RoBERTa re-ranker, but this time in a multi-label binary classification manner to predict whether each summary candidate maximizes each of the metrics of interest. To avoid using another neural network for re-ranking, BRIO (Liu et al., 2022b) performs a second fine-tuning stage with the re-ranking loss built in the base summarization system. Each of the four models above improves the SOTA on the CNN/DM benchmark, reaching 47.78 ROUGE-1 for BRIO. To the best of our knowledge, there is no work on sequence-level unsupervised abstractive summarization. Concurrently to our work, MBRD (Suzgun et al., 2022) proposes to rank generated candidates in several generation tasks using majority voting based on BERTScore (Zhang et al., 2019).
Unsupervised Summarization Re-ranking
As an unsupervised summarization re-ranking approach, our method assumes access to a zero-shot self-supervised summarization model. We refer to it as the base model M base . Given a source document D, M base will generate k summary candidates using a generation method to transform model predictions into a natural language summary. A widely used such generation approach is beam search, which maintains k top summary candidates throughout decoding, ranking them with decreasing mean log-probability of the sequence. In the end, practitioners keep the candidate maximizing the log-probability and discard the remaining, whereas we propose to keep all k candidates and re-rank them, following (Ravaut et al., 2022a).
Unsupervised Summarization Re-ranking
Let C = {C 1 , . . . , C k } be the pool of candidates. Our goal in (re-)ranking the candidates is to assign to each of them a score S, such that S(C i ) > S(C j ) if C i is a better candidate than C j (for 1 ≤ i, j ≤ k) according to some summary quality measures. We can then select the candidate maximizing the score as the best output:
Unsupervised Summarization Re-ranking
Unlike re-ranking in a supervised setup, where one can compute such scores by comparing with the ground truth summary or build models to optimize them (Liu and Liu, 2021;Ravaut et al., 2022a;Liu et al., 2022a), in our unspervised setup, we cannot assume access to the ground truth, which thus excludes scoring the candidate with regards to it (e.g., using ROUGE). In the following, we describe how we build our unsupervised scoring method (named SummScore) following principles assessing the quality of a summary.
Unsupervised Summarization Re-ranking
We design our candidate-level SummScore as an aggregation of features, each representing desired properties for a summary. Features either come from the comparison between the summary candidate and the source, or from the candidate itself. Fig. 1 synthesizes the overall SummScore re-ranking process.
Unsupervised Summarization Re-ranking
Comparison with the source One evident property of a summary is that it should stick to the source content, and contain as much of the important content as possible. The most straightforward way to measure this consists in using n-gram overlap metrics between the source document and each candidate. We use ROUGE-1 (noted R-1) (Lin, 2004), ROUGE-2 (R-2), and BLEU (Papineni et al., 2002), which form our first set of features:
Unsupervised Summarization Re-ranking
The above metrics only evaluate n-gram overlap, which can be helpful penalizing summary candidates departing too much from the source, potentially hallucinating. However, they have been shown to not be well suited at evaluating semantic similarity, and might encourage too much copying.
Unsupervised Summarization Re-ranking
Thus, our next batch of SummScore features consists in model-based metrics designed to capture semantic similarity between two text items. We explore three such metrics: BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021) and BLEURT (Sellam et al., 2020). BERTScore (noted BS) computes token-level cosine similarity between the contextual embeddings of the pre-trained BERT (Devlin et al., 2019) of each text item to compare. BARTScore (noted BaS) uses BART (Lewis et al., 2020) token-level log-probabilities from the pretrained BART to score the generated text. BLEURT (noted BRT) also leverages BERT but extends its pre-training with an additional multi-task pretraining on synthetic data. Our next features are:
Unsupervised Summarization Re-ranking
When each of these metrics is referred to, it is implicit that they are used to compare a summary candidate with the source document (in contrast to the supervised case, comparing with the target).
Unsupervised Summarization Re-ranking
Summary quality A good summary should be diverse, meaning it should avoid repeated n-grams. We build a summary-level diversity score which measures the proportion of unique n-grams.
Unsupervised Summarization Re-ranking
unique n-grams total n-grams
Unsupervised Summarization Re-ranking
We take N = 3 in practice. The summary should not be too short, nor too long. We penalize summaries which deviate a lot from the average summary length on the given dataset. To build a score with increasing values being desirable, we use a smooth inverse of the absolute length difference between the summary candidate and the mean length of summaries µ len .
Unsupervised Summarization Re-ranking
Final Score Our final set of summary features is:
Unsupervised Summarization Re-ranking
where S quality = {F div , F len }. For data point x i , SummScore simply outputs the summary candidate among the set C i maximizing a weighted combination of all features above:
Unsupervised Summarization Re-ranking
where we enforce coefficients to be |S| j=1 θ j = 1.0
Unsupervised Summarization Re-ranking
SummScore is simply a linear combination of eight features in total. Yet a last crucial question remains: how to estimate the coefficients to assign to each feature? We propose to bootstrap a pseudosummary using sentences from the source document. Coefficients are then tuned to maximize the mean of ROUGE-1/2/L between the summary candidate with the highest SummScore (e.g., Summ-Score output candidate), and the pseudo-target. We compare three approaches to extract pseudotargets:
Unsupervised Summarization Re-ranking
• Random-3: As a baseline, we randomly select three sentences from the source document to form a pseudo-target.
Unsupervised Summarization Re-ranking
• LEAD-3: This consists in the first three sentences of the document. LEAD-3 is a strong baseline for lead-biased news summarization datasets (Hermann et al., 2015;See et al., 2017), and it has even been used as a pseudo-target for summarization pre-training in TED (Yang et al., 2020).
Unsupervised Summarization Re-ranking
• Salient Sentences: We follow the gap-sentences generation idea introduced by PEGASUS pretraining objective (Zhang et al., 2020), and also used by SUPERT (Gao et al., 2020)   top 30% such sentences are extracted to form a pseudo-summary. We experiment with all three standard versions ROUGE-1, ROUGE-2 and ROUGE-L for salient sentences definition, referred to as Salient-R1, Salient-R2 and Salient-RL, respectively.
Unsupervised Summarization Re-ranking
We emphasize that none of these pseudo-targets definition makes any access to human supervision. Training SummScore amounts to estimating the coefficients θ in Eq. (7) using the pseudo-targets:
Unsupervised Summarization Re-ranking
where R is the mean of ROUGE-1, ROUGE-2 and ROUGE-L, C i is the set of candidates predicted by the base model M base for data point x i , andỹ i is the pseudo-target. To optimize coefficients, we hill climb with randomness to maximize R between the SummScore selected summary candidate, and the pseudo-target. Specifically, we estimate coefficients with stochastic local search on the validation set in a hierarchical manner: we first tune coefficients for S overlap and S semantic separately, then estimate coefficients for S quality ∪ {F overlap , F semantic }, where F overlap (resp. F semantic ) is the set S overlap (resp. S semantic ) after reduction to a single feature. Such hierarchical estimation is natural given that S overlap (resp. S semantic ) is made of features capturing similar properties, and dramatically reduces the search space.
Unsupervised Summarization Re-ranking
We experiment on four popular abstractive summarization datasets, from three different domains (see Table 2 for basic statistics on each dataset):
Unsupervised Summarization Re-ranking
• CNN-DailyMail (Hermann et al., 2015;See et al., 2017) is made of 93k and 220k articles from the CNN and DailyMail newspapers, respectively. CNN/DM is the most extractive dataset among all the ones we consider and has the longest source documents.
Unsupervised Summarization Re-ranking
• XSum (Narayan et al., 2018) has 227k articles from the BBC from 2010 to 2017. This is an extreme summarization task, compressing each article into a single, very abstractive sentence.
Unsupervised Summarization Re-ranking
• WikiHow (Koupaee and Wang, 2018) contains 168k lists of short instructions from Wikipedia.
Unsupervised Summarization Re-ranking
• SAMSum (Gliwa et al., 2019) is a dialogue summarization dataset containing 17k conversations. In this dataset, source length is significantly shorter than in the other datasets.
Unsupervised Summarization Re-ranking
To estimate coefficients, we subsample randomly (on datasets other than SAMSum) 1,000 data points from the validation set. To avoid coefficients optimization to overfit, we cap each random search at 1,000 trials. Evaluation of summaries selected by SummScore is done with the standard ROUGE-1/2/L (Lin, 2004) (using summary-level ROUGE-LSUM variant for ROUGE-L) and BERTScore (Zhang et al., 2019). We use transformers (Wolf et al., 2020) and datasets (Lhoest et al., 2021) for pre-trained checkpoints and datasets, respectively.
Unsupervised Summarization Re-ranking
We first apply SummScore to unsupervised abstractive summarization, using as base model (M base ) two models of different capacity: the pretraind PEGASUS (Zhang et al., 2020) (loading the google/pegasus-large checkpoint from transformers), and the recently introduced, highlyperforming ChatGPT 3 , accessed through OpenAI API (calling the gpt-3.5-turbo checkpoint). Due to its pre-training objective of generating gapsentences, PEGASUS can directly be applied to the summarization task after pre-training. This is not the case of comparable sequence-to-sequence Transformer-based models T5 (Raffel et al., 2019) and BART (Lewis et al., 2020), which are pretrained with token spans generation and sequence de-noising, respectively. For ChatGPT, to lower costs, we subsample randomly 1,000 data points from the test set on datasets other than SAMSum.  We decode PEGASUS with beam search, and ChatGPT with top-p sampling with p = 0.9 and temperature 0.8 to enhance diversity, both models with 20 candidates. We report candidate selection baselines from Table 1: top beam or first, and random (a randomly sampled candidate).
Unsupervised Summarization Re-ranking
We show unsupervised summarization results with PEGASUS and ChatGPT with 20 summary candidates in Table 3. SummScore improves the base PEGASUS by 4.37% to 7.27% across the four datasets. Notably, SummScore fails with ChatGPT on XSum, which we hypothesize is due to the nature of XSum and the fact that pseudo-labels from XSum source documents are too different from the ground truth labels, an issue not affecting PEGA-SUS because its performance range is far lower than ChatGPT. However, SummScore improves ChatGPT by 2.74% to 6.86% on the other datasets. We point out that SummScore gains are achieved without using any human supervision.
Unsupervised Summarization Re-ranking
SummScore -LEAD-3 performs best for the news domain, which intuitively makes sense due to the lead bias and first sentences containing an overview of the article. On WikiHow, SummScore -Salient-R2 works the best, yet gains are more moderate and SummScore fails to improve the BERTScore on this dataset. SummScore -Random-3 is tied with SummScore -LEAD-3 on SAMSum: we attribute it to the fact that SAMSum source documents are very short (Table 2), and the LEAD-3, Random-3, and entire source document all overlap a lot. Appendix A confirms that SummScore re-ranking always finds a non-trivial (e.g., longest) candidate selection.
Unsupervised Summarization Re-ranking
Next, we investigate SummScore performance in the transfer setup, with standard-size models (discarding ChatGPT or similar models). We perform zero-shot summarization inference followed by SummScore on a target dataset where the base model M base was fine-tuned on another source dataset. As M base , we use three high-performing summarization models: PEGASUS (Zhang et al., 2020), BART (Lewis et al., 2020), and the recently introduced BRIO (Liu et al., 2022a), which achieves SOTA results on news summarization (CNN/DM & XSum). We use publicly available fine-tuned checkpoints on CNN/DM and XSum, and PEGASUS on WikiHow. We fine-tune ourselves PEGASUS on SAMSum, and BART on Wik-iHow and SAMSum. Generation and fine-tuning hyper-parameters and results are in Appendix B.
Unsupervised Summarization Re-ranking
Given the findings from §4.2, we use Summ-Score -LEAD-3 on CNN/DM, XSum, and SAM-Sum, and SummScore -Salient-R2 on WikiHow. We tune coefficients in the same process described in §4.1. To stick to a no supervision scenario, we do not apply SummScore on a dataset on the which the base model was fine-tuned, which would fall into the supervised learning use case. We compare SummScore zero-shot transfer performance on CNN/DM with that of SOTA WikiTransfer (Fabbri et al., 2021), which fine-tunes BART on external data retrieved from Wikipedia before applying the model in zero-shot summarization.
Unsupervised Summarization Re-ranking
Zero-shot transfer results are displayed in Table 4. SummScore consistently improves transfer performance, with ROUGE gains of 7.51% averaged over 30 setups: +9.43% on CNN/DM, +1.27% on XSum, +9.20% on WikiHow (up to +17.64% average when transferring from XSum) and +9.61% on SAMSum. Notably, on CNN/DM, BART transferred from SAMSum with SummScore improves on the ROUGE-1 and ROUGE-L of SOTA transfer model WikiTransfer (also using a BART backbone), despite WikiTransfer being fine-tuned on data specifically crafted to transfer better to the downstream task. We notice that SummScore helps  more when the base model transfers less well, such as from single-sentence summaries XSum. Appendix C evalutes re-ranking itself and shows that SummScore can also reach strong recall.
Unsupervised Summarization Re-ranking
Using the selected summary candidate as a pseudotarget, one can naturally extend SummScore into a self-training summarization objective. Indeed, if γ parametrizes M base , we can further train M base through the objective:
Unsupervised Summarization Re-ranking
This process can be repeated: if we denote new model weights by γ k , we can re-apply SummScore and perform another round of self-training, yielding new model weights γ k+1 .
Unsupervised Summarization Re-ranking
We notice that the unsupervised PEGASUS beam search summary candidates, including the one selected by SummScore, are quite extractive (see Appendix D). This could be because the selfsupervised gap-sentences are extracts from the source document. To make the pseudo-summaries more abstractive and diverse enough to mitigate the confirmation bias in self-training (Tarvainen and Valpola, 2017), we use the paraphrasing approach proposed in FAR- RW (Zhang et al., 2022). On each dataset, we train a paraphrase model to generate the top n sentences maximizing the mean ROUGE with the top n most salient sentences, conditioning on these salient sentences. This yields an unsupervised, in-domain paraphrase model which we apply to the SummScore pseudo-labels on the training set to make them more abstractive and diverse. We refer to Appendix E for details on the paraphrasing model training, its performance and resulting abstractiveness and diversity levels on pseudo-labels. As the unsupervised process of paraphrasing may harm the pseudo-summary quality, in practice, we apply it to the x% most extractive training data points, where x is among {12.5%, 25%, 50%, 100%}. We use 25% for CNN/DM, 100% for XSum, 50% for WikiHow, and 12.5% on SAMSum, as these provide an ideal ROUGE/abstractiveness trade-off (see Appendix D).
Unsupervised Summarization Re-ranking
For each dataset except SAMSum, we randomly subsample 50k data points from the training set and 1k from the validation set to self-train and validate the model, resulting in a self-training process much less computationally expensive than finetuning. We show self-training results on the test sets using PEGASUS as base model in Table 5. Self-training improves unsupervised summarization performance on all datasets, resulting in a selftrained model better than the base model although not as performing as SummScore. Notably, reapplying SummScore on the new model after selftraining further improves performance drastically. Besides, paraphrasing self-training pseudo-labels helps maintain some degree of abstractiveness, as seen in Appendix D. On CNN/DM, one round of self-training followed by SummScore brings PE-   Mean number of times out of 50 that each model or a tie is selected, with standard deviation in parenthesis, across two use cases and two attributes.
Unsupervised Summarization Re-ranking
GASUS performance above the Summary Loop, two rounds above TED, and three rounds to 39.76 ROUGE-1, within 1% of SOTA model FAR-RW.
Unsupervised Summarization Re-ranking
We conduct a human evaluation on 50 data points randomly sampled from CNN/DM test set. We show human participants the source news article, alongside the summary candidate from the base PEGASUS model, and the one re-ranked by Summ-Score. Participants are asked to pick which summary is more informative, and which is more factually consisteny, with the option of choosing a tie. We cover two use cases: unsupervised abstractive summarization, and zero-shot transfer from a model fine-tuned on XSum. In the former use case, both summaries are identical in 7/50 data points, and 4/50 data points in the latter. Human raters are three volunteer graduate students, with full professional command of English. Results are displayed  Table 7: Ablation study for unsupervised abstractive summarization with PEGASUS. We isolate each feature of Summ-Score and report its re-ranking performance (picking the candidate maximizing this feature), using the mean of ROUGE-1/2/L as reported metric. Best results within 0.1 are in bold.
Unsupervised Summarization Re-ranking
in Table 6. Although both summaries often overlap significantly (rightmost column), resulting in a high Tie, SummScore is strongly preferred over PEGASUS across both use cases and attributes.
Unsupervised Summarization Re-ranking
To better understand SummScore performance gains, we perform an ablation study where reranking is done with each feature taken individually. Results for PEGASUS in unsupervised summarization are shown in Table 7. N-gram overlap features are very strong re-ranking baselines on WikiHow and SAMSum. In fact, ROUGE-1 with the source is even slightly better than Summ-Score on WikiHow. On news datasets, semantic similarity features such as BERTScore are strong baselines. Interestingly, our hand-crafted feature diversity has a negative contribution when used as standalone re-ranker ; however it can help a lot when combined with the other features, acting as a regularizer by encouraging some diversity. On average, SummScore performs the best. We also report trivial feature aggregation baselines Plain average and Random coefficients, which SummScore outpeforms, confirming the efficiency of estimating coefficients through pseudo-labels.
Unsupervised Summarization Re-ranking
In Appendix F, we show that SummScore unsupervised re-ranking is also robust to other decoding methods diverse beam search (Vijayakumar et al., 2016) and nucleus sampling (Holtzman et al., 2019), and a different number of beams (5 to 20). We confirm that our default setup of beam search with 20 beams yields optimal ROUGE results. Echoing SummaReranker findings (Ravaut et al., 2022a), gains further increase when mixing in several decoding methods.
Unsupervised Summarization Re-ranking
Reports speak of at least four people injured. The city is at the heart of the conflict between the Turkish government and Kurdish separatists. Interior Minister Suleyman Soylu said the blast happened at a vehicle repair unit, and appeared to be an accident. He said "it seems there is no outside interference, and the explosion came from the vehicle under repair". Mr Soylu said one person was trapped under rubble, another was seriously injured, and others had minor injuries. The blast brought a roof down, left a huge crater and a pall of smoke drifted over part of the city. The cause remains unclear. The banned Kurdistan Workers' Party (PKK) is active in the area. Turkey is five days away from a key referendum on granting President Recep Tayyip Erdogan sweeping new powers [...] PEGASUS summary (ROUGE-1: 10.53): Interior Minister Suleyman Soylu said the blast happened at a vehicle repair unit, and appeared to be an accident. Self-training summary (ROUGE-1: 32.43):
Unsupervised Summarization Re-ranking
The blast happened at a vehicle repair unit in the city of Diyarbakir, near the border with Syria.
Unsupervised Summarization Re-ranking
A large explosion has struck a police headquarters in the mainly Kurdish city of Diyarbakir in south-eastern Turkey. Table 8: Qualitative sample with self-training PEGASUS from the XSum dataset, after a single round of self-training.
Unsupervised Summarization Re-ranking
We refer to Appendix H for full qualitative unsupervised re-ranking examples on all datasets, and to Table 8 for an example of summary generated by the self-trained PEGASUS model on XSum. As seen, both re-ranking and self-training can improve dramatically from the unsupervised PEGA-SUS baseline, capturing entirely new phrases.
Unsupervised Summarization Re-ranking
As noted in Table 6, SummScore summaries tend to be more factually consistent than the baseline. There is strong intuition to this result: since Summ-Score is built to maximize features of n-gram overlap and semantic similarity with the source, it should yield summaries closer to the source, and more factually consistent as a result. We investigate this further, and use two popular models to evaluate summarization factuality: the established factCC (Kryscinski et al., 2020) and the recently introduced state-of-the-art QAFactEval (Fabbri et al., 2022). factCC uses a BERT model to classify each summary sentence as consistent or inconsistent with regards to the source, and reports the average accuracy over 100%. QAFactEval improves each step of the QA evaluation pipeline (answer selection, question generation, etc) and combines entailment with QA-based metrics into a learned metric. In Table 9, we observe that SummScore QAFactEval is consistently above PEGASUS, and SummScore factCC is better on news datasets too.
Unsupervised Summarization Re-ranking
We analyze coefficients learned by SummScore from a high level perspective in Table 10, gathering features from a same group together. Semantic similarity features are dominating (except for Wik-iHow), encouraging further research using newer semantic similarity metrics for re-ranking.
Unsupervised Summarization Re-ranking
A finer-grain analysis, covering all SummScore   pseudo-labeling techniques, can be viewed in Tables 19 and 20 of Appendix G. SummScore -Salient-R1 and SummScore -Salient-RL place much more emphasis on n-gram overlap with the source. In contrast, SummScore -LEAD-3 (which we use for self-training on CNN/DM, XSum, SAM-Sum) uses relatively more semantic similarity features like BERTScore, suggesting that it is able to exploit key semantic content contained in initial sentences.
Unsupervised Summarization Re-ranking
We introduced SummScore, the first unsupervised abstractive summarization re-ranking system. SummScore does not rely on a neural network: instead, it builds features for each summary candidate, some of them using the source as well, and aggregates them into a final re-ranking score. Feature coefficients are estimated through tuning against a pseudo-label derived from the source document. It is a simple framework which easily supports the addition of new features. SummScore significantly improves the performance of the base summarization model, in terms of ROUGE, BERTScore, factual consistency, and human preference ; in both unsupervised and zeroshot transfer scenarios. Moreover, SummScore selected summary candidate naturally extends into a self-training objective for abstractive summarization, which improves unsupervised summarization.
Unsupervised Summarization Re-ranking
As a second-stage method, SummScore requires access to a base abstractive summarization model generating summary candidates. Generating up to 20 summary candidates per data point can take a long time, especially on training sets, which is needed for the self-training use case. Besides, even though SummScore does not need to train a new neural network, we also need to generate all eight features for each summary candidate once all candidates are generated. N-gram overlap features are very fast, but model-based semantic similarity features (e.g, BERTScore) can be time-consuming to extract, once again, especially on entire training sets.
Unsupervised Summarization Re-ranking
While SummScore will significantly improve the quality of the base model across base models and datasets, ultimately, the performance of the final selected summary is bounded by the capacity of this base model: SummScore improves more PE-GASUS than it does on ChatGPT ; but PEGASUS performance drags ChatGPT. Another limitation lays in the metric used to compare summary candidates with the pseudo-target. We used mean ROUGE, although a model-based semantic similarity metric would make sense too, but at a much greater computational cost.  We perform a sanity check counting the percentage of time that SummScore falls back to a trivial method of re-ranking summary candidates. For each feature described in §3.2, we report the overlap between SummScore and a re-ranking approach consisting in picking the summary candidate maximing this feature. We also report baselines consisting in picking the top beam, an oracle or a worst candidate, and the longest candidate. As seen in Tables 11 and 12, across both backbones PEGA-SUS and ChatGPT, SummScore never collapses  to a trivial candidate selection, and we see similar patterns on the same dataset (e.g., highest overlap with a single feature selection is with BERTScore with source feature on CNN/DM).
Unsupervised Summarization Re-ranking
In Table 13, we show generation hyper-parameters used for each dataset to generate beam search summary candidates used in Table 3. For the transfer setup shown in Table 4, we use as generation hyper-parameters on each target dataset the parameters used on that dataset for   
Unsupervised Summarization Re-ranking
Besides the quality of the selected summary, we also analyze re-ranking performance itself. In Fig. 2, Fig. 3, Fig. 4 and Fig. 5, we show recall curves on each dataset and for all unsupervised and zero-shot summarization setups. Recall@k is defined as the probability of outputting one of the oracle summary candidates (candidates maximizing the mean ROUGE with the target) among the first k candidates. We compare SummScore with the baseline beam search output, and a random candidate selection baseline.
Unsupervised Summarization Re-ranking
In most cases, SummScore (green curves) provides higher recall, with the notable exception of XSum, where both beam search and SummScore and XSum can fail to improve the random baseline.
Unsupervised Summarization Re-ranking
In Table 15, we show ROUGE results from Table 5 alongside abstractiveness results, as measured per the fraction of novel n-grams in output summaries, for re-ranking and self-training experiments. Maximizing both ROUGE and abstractiveness is notoriously difficult, as easy solutions for abstractiveness optimization can deviate a lot from the source, resulting in a harmed ROUGE score.
Unsupervised Summarization Re-ranking
The unsupervised PEGASUS (first row of each block) is very extractive and only produces a small fraction of novel n-grams. SummScore selected summaries, despite maximizing a score which maximizes the mean ROUGE with pseudo-labels extracted from the source document, both improve the ROUGE and the abstractiveness level. However, SummScore re-ranking applied to self-trained models tends to reduce their abstractiveness level, although it stays above the level of the baseline PEGASUS. Paraphrased summaries drastically increase abstractiveness, at the expense of ROUGE -  Table 3, and the next seven plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next seven plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next eight plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next eight plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 15: ROUGE and abstractiveness for several models: the unsupervised PEGASUS (first sub-block), re-ranking with SummScore (first sub-block), paraphrasing the resulting pseudo-labels (second sub-block), self-training with the pseudo-labels (third sub-block), then re-ranking self-training outputs with SummScore again (third sub-block). All results are on the test set, results of self-training pseudo-labels are underlined, and highest numbers within 0.1 are in bold.
Unsupervised Summarization Re-ranking
except on XSum where paraphrasing also improves ROUGE, motivating our choice to use 100% paraphrased summaries as pseudo-labels. We confirm that our pseudo-labels for self-training, made of a blend of SummScore selected summaries and selected summaries being paraphrased, maintains high ROUGE while being much more abstractive than the baseline PEGASUS.  We track the mean of ROUGE-1, ROUGE-2 and ROUGE-L between the generated paraphrase and target paraphrase on the validation set during training, and perform early stopping. Best mean ROUGE results are shown in Table 16.
Unsupervised Summarization Re-ranking
Next, we study the impact of the paraphrasing model on the SummScore pseudo-targets. In Ta Table 17: Impact of paraphrasing on the pseudo-targets. We report mean ROUGE and percentage of novel n-grams between the paraphrased pseudo-targets and the original pseudo-targets, on the training set of each dataset since this is the subset that paraphrasing is applied to.
Unsupervised Summarization Re-ranking
ble 17, we compute the mean ROUGE between pseudo-targets and their paraphrase, and analyze the novel n-grams. We point out that the paraphrasing is only applied to the training pseudo-labels as the goal of paraphrasing is to encourage the model to learn diversity during self-training, hence Table 17 reporting results on training sets. On each dataset, the mean ROUGE is in the 50-80 range, indicating that the paraphrased pseudo-labels do not deviate too much from the original pseudo-labels and yet is able to re-write some content. Besides, there is a high proportion of new n-grams: more than 10% new 1-grams (with the exception of Wik-iHow on the which the paraphrasing model seems to struggle more to rephrase the input), and more than 20% 2-grams.  In Table 18, we apply SummScore outside of the standard beam search with 20 beams setup. Results show that SummScore performance continuously improves with more summary candidates, whereas the top beam stays around the same level. Besides, SummScore relative gains are stronger with lower quality decoding methods diverse beam search and nucleus sampling. Lastly, combining 20 summary candidates from each of the three decoding methods yields a pool of 60 summary candidates, out of the which SummScore re-ranking can improve by an extra +0.21 mean ROUGE the performance compared to re-ranking 20 beam search candidates (28.59 mean ROUGE vs 28.38). Overall, we recommend our default setup of beam search with 20 beams to apply SummScore re-ranking. A greater number of beams becomes difficult to fit into a standard GPU with 16 GB memory.
Unsupervised Summarization Re-ranking
In Table 19 (PEGASUS backbone) and Table 20 (ChatGPT backbone), we show coefficients found by SummScore (for each of the five methods to select pseudo-labels which we studided), and on each dataset, including when applying SummScore again on top the self-trained models. For the sake of conciseness, we do not include SummScore coefficients obtained in zero-shot setups. BERTScore with source appears as the feature which consistently receives the highest weight for SummScore -Random-3 and SummScore -LEAD-3 ; while ROUGE-2 with source dominates for SummScore -Salient-R1/R2/RL. Diversity and Length features are significantly less used.
Unsupervised Summarization Re-ranking
In the following, beam search output (for PEGA-SUS) or the first candidate from top-p sampling (for ChatGPT) is in orange, SummScore selected summary candidate in blue, and oracle candidate(s) in teal. On each dataset, we show one re-ranking example on the unsupervised PEGASUS and/or ChatGPT (Table 3), one zero-shot re-ranking example selected from Table 4, and one re-ranking example applied on top of the self-trained PEGA-SUS (Table 5).    'We can confirm that protestors from Greenpeace have illegally boarded the 'Polar Pioneer,'under contract to Shell, jeopardizing not only the safety of the crew on board, but the protestors themselves,' she said on Monday. She added: 'Shell has met with organizations and individuals who oppose energy exploration offshore Alaska. We respect their views and value the dialogue. 'We will not, however, condone the illegal tactics employed by Greenpeace. Nor will we allow these stunts to distract from preparations underway to execute a safe and responsible exploration program,' she said in a statement.
Unsupervised Summarization Re-ranking
Summary 'We're here to highlight that in less than 100 days Shell is. Scores Mean ROUGE: 6.55 (rank 11) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Beam #2 Summary 'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists.
Unsupervised Summarization Re-ranking
Mean ROUGE: 43.17 (rank 3) || SummScore rank: 2
Unsupervised Summarization Re-ranking
Beam #3 Summary 'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the ship's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists.
Unsupervised Summarization Re-ranking
Mean ROUGE: 42.85 (rank 4) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists from Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 43.59 (rank 2) || SummScore rank: 12 ...
Unsupervised Summarization Re-ranking
'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove them from the Scores Mean ROUGE: 43.91 (rank 1) || SummScore rank: 6 ...
Unsupervised Summarization Re-ranking
Shell has filed a complaint in federal court in Alaska seeking an order to remove Greenpeace activists who climbed aboard an oil rig in the Pacific . The environmental group said in a statement its team would occupy the underside of the main deck of the Polar Pioneer . The six activists are camping on the 38,000-tonne Polar Pioneer platform, which they boarded using inflatable boats from the Greenpeace vessel 'Esperanza' 'We made it! We're on Shell's platform. And we're not alone. Everyone can help turn this into a platform for people power!' tweeted Aliyah Field . 
Unsupervised Summarization Re-ranking
Although Hillary Clinton boasts a robust 3.6 million Twitter followers, not even a vast right-wing conspiracy would be able to interact with 2 million of them. According to two popular online measuring tools, no more than 44 per cent of her Twitter fan base consists of real people who are active in using the social media platform. And at least 15 per cent -more than 544,000 -are completely fake. StatusPeople.com, the oldest publicly available Twitter-auditing tool, reports that 44 per cent of the former secretary of state's followers are 'good'; 15 per cent are 'fake'; and 41 per cent are 'inactive,' meaning that they never tweet or reply to any tweets. SCROLL DOWN FOR VIDEO . FAKERS: According to one popular online audit tool, only 44 per cent of Hillary Clinton's Twitter followers are real people who participate on the social media platform . 'I'M RUNNING FOR PRESIDENT': Clinton has cast herself as a champion of 'everyday Americans' Another Twitter sleuthing website sampled more than 320,000 of Clinton's followers and found that a much larger number of them were 'fake' Just 4 per cent of President Barack Obama's Twitter followers, by comparison, are considered fake. The White House worked overtime to purge most of them after a September 2013 report found that more than half of his followers didn't really exist. Michelle Obama's Twitter audience is 25 per cent fake, according to StatusPeople, along with 21 per cent of Vice President Joe Biden's. Another tool, TwitterAudit.com, sampled 320,000 of Mrs. Clinton's followers and found that 18 per cent were fake. The new measurements will add to the Clinton presidential campaign's embarrassment following news on Tuesday that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned. REALLY? Hillary Clinton's Twitter follower-count appears to be significantly inflated . FACEBOOK FAKERY: Clinton boasts two-thirds of a million Facebook 'likes,' but more than 46,000 of them list 'Baghdad' as their hometown . Vocativ reported that at least 7 per cent of them listed Baghdad, Iraq as their hometown, a larger number than any U.S. city. That would represent more than 46,000 people. Additional evidence of digital astroturfing is that while most of her U.S. Facebook fans are older than 55, most of the Baghdad contingent is in the 18-34 age range. While Clinton was America's top diplomat, her State Department was buying Facebook 'likes,'according to an Associated Press report from last year. 'In 2013, the State Department, which has more than 400,000 likes and was recently most popular in Cairo, said it would stop buying Facebook fans after its inspector general criticized the agency for spending $630,000 to boost the numbers,' the wire service reported. Clinton left the agency while the inspector general was investigating. ASTROTURF? A woman cast as a kindly, gardening grandma in Clinton's campaign launch video is actually a former abortion lobbyist who once helmed a campaign for Texas state Senator Wendy Davis . WHERE'S SCOOBY? Clinton's caravan was spotted in Maumee, Ohio on Monday when she lunched at Chipotle, but no one recognized her . Facebook says it has entire teams of employees tasked with rooting out schemes to artificially boost statistics. 'When we catch fraudulent activity, we work to counter and prevent it -including blocking accounts and removing fake likes,' the company reports on its website. Clinton is in Monticello, Iowa on Tuesday for the first stop on her nascent presidential 'listening' tour. She was driven from New York in a van that her campaign aides have dubbed 'Scooby,' after the 'Mystery Machine' van in the Scooby-Doo cartoon series. Daily Mail Online visited Kirkwood Community College on Monday, the site of the event, and heard from students that the former first lady is a 'control freak' who may be pursuing legal status for illegal immigrants because the Democratic Party needs a new pool of loyal voters.
Unsupervised Summarization Re-ranking
Online measuring tools show that only 44% of Hillary Clinton's Twitter followers are real people who actively use the platform, with 15% completely fake.
Unsupervised Summarization Re-ranking
Comparatively, just 4% of President Obama's Twitter followers are fake. Clinton's Facebook audience has also come under scrutiny, with reports of purchased likes and a large number of followers from Baghdad, Iraq.
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.88 (rank 9) || SummScore rank: 18
Unsupervised Summarization Re-ranking
At least 15% of Hillary Clinton's Twitter followers are fake, according to two online measuring tools, with more than 544,000 completely fake. In comparison, just 4% of Barack Obama's Twitter followers are fake. Clinton's recent embarrassment of having a large number of Facebook likes purchased rather than earned further adds to the digital astroturfing speculation.
Unsupervised Summarization Re-ranking
Mean ROUGE: 19.60 (rank 26) || SummScore rank: 15
Unsupervised Summarization Re-ranking
An investigation has revealed that at least 15% of Hillary Clinton's Twitter followers are completely fake. The same investigation also found that only 44% of her Twitter followers are real people who actively use the platform. This adds to the Clinton presidential campaign's embarrassment following news on Tuesday that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.88 (rank 8) || SummScore rank: 5
Unsupervised Summarization Re-ranking
According to two online measuring tools, only 44% of Hillary Clinton's 3.6 million Twitter followers are real people who actively use the platform, with at least 15% being fake. In comparison, only 4% of President Barack Obama's Twitter followers are considered fake. Clinton's Facebook page has also been called into question, with evidence of 'likes' being purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 27.09 (rank 3) || SummScore rank: 10
Unsupervised Summarization Re-ranking
According to online measuring tools, over half of Hillary Clinton's Twitter followers are either fake or inactive. One tool reported that only 44% of her followers are real people who are active on the platform. Additionally, a large number of Clinton's Facebook fans may have been purchased rather than earned, with over 46,000 of them listing Baghdad, Iraq as their hometown.
Unsupervised Summarization Re-ranking
Mean ROUGE: 24.82 (rank 7) || SummScore rank: 7 ...
Unsupervised Summarization Re-ranking
According to online measuring tools, only 44% of Hillary Clinton's 3.6 million Twitter followers consist of real people who are active on the social media platform. At least 15% of her followers, or more than 544,000, are completely fake. This comes after news that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 31.34 (rank 1) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Two different online audit tools say no more than 44 per cent of Hillary's 3.6 million Twitter fans are real people who participate in the platform . The newly minted presidential candidate is fending off accusations that her Facebook page is full of fake 'likes' Her Facebook fan base includes more people from Baghdad, Iraq than any US city . When she was secretary of state, her agency paid $630,000 to bulk up its Facebook likes, but pledged to stop after she left . 
Unsupervised Summarization Re-ranking
Assault: Dr Sahar Hussain attacked two Tube workers because she didn't want to miss the last train home . A GP attacked two Tube workers while screaming 'I'm a doctor' because she did not want to miss the last train home on a Friday night. Dr Sahar Hussain, 53, panicked when she was unable to get through the gates at Leicester Square station, and started ranting at staff. She denied assaulting the two workers, saying she was worried about being stranded on her own in central London because she is a Muslim woman. But Hussain has now been found guilty and ordered to pay a total of £2,250 in fines, compensation and court costs -and she could face disciplinary action from the General Medical Council. In video footage captured on her own mobile phone, Hussain could be heard to shout: 'I'm a doctor actually, I work for the NHS. I'm a doctor. Get me through the gate, I'm going to miss my train.' City of London Magistrates' Court heard Hussain arrived at the station around 11.30pm on June 20 last year, trying to get home to Woodford Green after socialising with friends in the West End. When she was refused entry by the automatic gates, she demanded that ticket seller Malcolm Shaw let her through before lashing out at his colleague Indira Ramsaroop, who was trying to help. Hussain, originally from Iraq, screamed and shouted at Mrs Ramsaroop as she thrust a camera phone into her face before grabbing her by the arm. The 24-year-old Transport for London worker was then chased by the doctor as she tried to flee to the control room, bumping her head on the way. In the video on Hussain's phone she was heard shouting: 'This woman is on something, she's not sober is she? You're in work and you're not sober. Get me through the gate.' During the scuffle Hussain, a mother of one who helps train GPs at two universities, also grabbed Mr Shaw by the arms, leaving him with scratches. Mrs Ramsaroop was close to tears in court as she told how she had to take almost two weeks off work following the incident, adding: 'I had a lot of sleepless nights. It had an impact on myself with customers when I came back to work. 'I have felt very let down to have been threatened and been running away in my place of work. It actually affected me for a very long time and I got quite ill just at the worrying and fear.' Row: The assault took place on a Friday night at Leicester Square station in central London . Hussain admitted losing her temper, telling the court: 'I'm very sorry about the way I expressed myself with my agitation and frustration.' District Judge Quentin Purdy found her guilty of two counts of assault by beating, saying: 'The evidence is overwhelming. You completely lost your self-control. 'Unusually for the sort of incident this court regularly deals with, there is no hint of intoxication in any way. 'But the explanation is there. You were anxious about missing the last train, concerned about being late for work or unable to work the next day, cultural embarrassment by being left in London as a married Muslim woman. 'It was completely out of character, but as we have all seen, you acted in an unpleasant and hysterical fashion towards numerous members of staff.' He said she had subjected Mrs Ramsaroop to 'completely unjustified violence', as well as 'humiliation' for falsely suggesting she was intoxicated at work. The judge added: 'You have lost a great deal by your lack of self-control. Your good name is now lost with these two convictions for assault.' Hussain was fined £500 for each assault and ordered her to pay £500 compensation to Mrs Ramsaroop, £200 to Mr Shaw, £500 prosecution costs and a £50 victim surcharge. Beam 
Unsupervised Summarization Re-ranking
Tell the court she was worried about being stranded on her own in central London because she is a Muslim woman. Describe to the court how she screamed: 'I'm a doctor, get me through the gate!' Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 40.97 (rank 2) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Tell the court she was worried about being stranded on her own in central London because she is a Muslim woman. Describe to the court how she screamed: 'I'm a doctor, get through the gate.' Scores Mean ROUGE: 42.04 (rank 1) || SummScore rank: 9 ...
Unsupervised Summarization Re-ranking
Shell has filed a complaint in federal court in Alaska seeking an order to remove Greenpeace activists who climbed aboard an oil rig in the Pacific . The environmental group said in a statement its team would occupy the underside of the main deck of the Polar Pioneer . The six activists are camping on the 38,000-tonne Polar Pioneer platform, which they boarded using inflatable boats from the Greenpeace vessel 'Esperanza' 'We made it! We're on Shell's platform. And we're not alone. Everyone can help turn this into a platform for people power!' tweeted Aliyah Field .  .' The 28-year-old's father Jalal Kassab said he was worried about his grandchildren living in a war zone, but said the threat of imprisonment made it difficult for his daughter to return to Australia. 'I know she wants to come back and we are trying everything we can to bring her back,' Mr Kassab told the Herald Sun. Another former Melbourne woman Zehra Duman last month shared a series of propaganda pictures she says shows her 'five star jihad' lifestyle . In photographs posted to a Twitter several women are pictured standing under an Islamic State flag, reclining against a clean white BMW M5 and wielding machine guns . In one tweet, Duman said: 'US + Australia, how does it feel that all 5 of us were born n raised in your lands, & now here thirsty for ur blood?'
Unsupervised Summarization Re-ranking
The children's paternal grandparents say they are worried Kassab, 28, is 'brainwashing' the children, after their father was killed near the Syria-Turkey border last year. 'We pray they come back but it does not look good.' Kassab's Twitter paints a picture of their life in the city the terrorist group IS have made their headquarters, where the children cannot go to school and airstrikes hit blocks away from their apartment. 
Unsupervised Summarization Re-ranking
Former Melbourne woman Dullel Kassab fled to Raqqa in Syria with her children last year, and she regularly boasts on Twitter that her four-year-old daughter and two-year-old son sleep with toy guns next to their beds and her daughter likes watching IS videos of 'Muslims killing bad ppl.' The children's paternal grandparents say they are worried Kassab, 28, is 'brainwashing' the children, after their father was killed near the Syria-Turkey border last year.
Unsupervised Summarization Re-ranking
Mean ROUGE: 57.48 (rank 1) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Grandparents have pleaded for the safe return of two children in Syria . Former Melbourne woman Dullel Kassab fled to Raqqa in Syria with her four-year-old daughter and two-year-old son last year . She said her daughter likes watching IS videos of 'Muslims killing bad ppl' Beam #20 Summary Mr Martin has said his party will facilitate a minority government, but will not support a programme for government. Fianna Fil want to see the immediate removal of water charges, but Fine Gael see a role for them. However, no party was able to form a majority government and TDs will so far failed Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 19.28 (rank 2) || SummScore rank: 5
Unsupervised Summarization Re-ranking
Reference Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government.  WikiHow: re-ranking from the unsupervised PEGASUS
Unsupervised Summarization Re-ranking
On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.; , Click on the program's shortcut icon on your desktop or programs list to launch it. Wait until the program starts. Any version will do, but the latest one is better because they have additional useful functions. , On the Menu bar located at the top of the screen, go to File and click "Open." Locate the image, select it, then click "Open." , To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image" on the Menu bar, into "Black and White." , On the Layer panel on the palette dock located at the bottom left of the screen, create a new layer by clicking a small paper-like icon beside the folder icon. , Change the layer name to "Skin" as this will be the first part of the image that you will add a color to. , Press "B" on your keyboard or get the Brush tool on the toolbar located at the left side of the screen. The icon is a brush. , Pick a nice skin tone color on the Swatches, which are also on the Palette dock. You may also set the foreground color to get more colors on the Toolbar, below the Zoom Tool, which is the two square that indicates as foreground and background color. A skin tone color is the lightest Orange or Yellow Orange. , Drag and move your mouse to the skin tone of your image. To avoid misplacing the colors, use the Zoom tool or simply press CTRL + Z. Use the Eraser tool, which is also located on the toolbar, to delete some misplaced part; but change the Brush Preset, which is on the Palette Dock, and choose "Soft Round" so it will create pixelated color when you erase them. , Once you're done with the skin, repeat Steps 1 to 6 with other parts of the body. For example, if you want to color the clothes, add a layer and rename the layer "Clothes." After renaming, proceed with selecting the right tool and color, and coloring the clothes. Continue with other parts, like lips, hair, background, etc. , To merge all layers, go to Layer dock, right-click, and choose "Flatten Image" or "Merge Visible" on the options. , Adjust the image tone, level, or brightness on the Image Menu to enhance the photo. , Press CTRL + S, choose JPEG as file type, add a file name, and click "Save" when you're done! Beam #1
Unsupervised Summarization Re-ranking
Summary You may also set the foreground color to get more colors on the Toolbar, below the Zoom Tool, which is the two square that indicates as foreground and background color. Scores Mean ROUGE: 13.01 (rank 7) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Beam #2 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-into " Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.18 (rank 1) || SummScore rank: 2
Unsupervised Summarization Re-ranking
, On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen, into " Scores Mean ROUGE: 23.18 (rank 1) || SummScore rank: 5
Unsupervised Summarization Re-ranking
Beam #4 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar, into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Beam #5 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar, into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 13
Unsupervised Summarization Re-ranking
Beam #6 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-to " Scores Mean ROUGE: 22.83 (rank 2) || SummScore rank: 4
Unsupervised Summarization Re-ranking
Beam #7 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar-into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 7
Unsupervised Summarization Re-ranking
On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-into " Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.18 (rank 1) || SummScore rank: 3 ...
Unsupervised Summarization Re-ranking
, On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-and choose Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 22.71 (rank 3) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. , the base of the tongue. Or, use your index finger and thumb to push the pill over the back of your cat's tongue. If you do it this way, you must do it quickly to prevent your fingers from getting bitten.Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue. This way you can avoid having to put your fingers in your cat's mouth. Coat the pill with butter to make swallowing it easier. , Once you have placed the pill at the base of its tongue, use your hands to keep its mouth closed by applying gentle pressure. Also re-position your cat's head so that it is level instead of tilted back; this will make it easier for your cat to swallow the pill. Gently rub your cat's throat or blow in its nose to encourage it to swallow the pill.Try not to let your cat go until the pil lis swallowed. Place a drop of water on its nose. If your cat licks the water off, then this means that it has swallowed the pill. , Tricyclic antidepressants should be given with food to decrease side effects such as vomiting. Therefore, feed your cat directly after giving it its medication. If your cat will not eat, then entice it to eat by feeding it its favorite food like cheese, or by giving it a treat.Make sure to provide your cat with water as well. You can also give your cat flavored water like juice from a tuna can to entice it to drink.
Unsupervised Summarization Re-ranking
Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.17 (rank 8) || SummScore rank: 12
Unsupervised Summarization Re-ranking
Beam #2 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in the towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 18.78 (rank 6) || SummScore rank: 8
Unsupervised Summarization Re-ranking
Beam #3 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.17 (rank 8) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Beam #4 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. gently stabilize it by holding it steady with one or both hands. Rub your cat's throat or blow in its nose to encourage it to swallow the pill.
Unsupervised Summarization Re-ranking
Mean ROUGE: 18.68 (rank 7) || SummScore rank: 7
Unsupervised Summarization Re-ranking
Beam #5 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. Wrap your fingers around the bottom of its cheekbones for leverage. Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 26.93 (rank 3) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
Beam #6 Summary Tricyclic antidepressants should be given with food to decrease side effects such as vomiting. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 15.09 (rank 10) || SummScore rank: 17 Beam #7 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. gently stabilize it by holding it steady with one or both hands. Scores Mean ROUGE: 8.72 (rank 12) || SummScore rank: 19
Unsupervised Summarization Re-ranking
Beam #8 Summary Gently stabilize it by holding it steady with one or both hands. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.18 (rank 7) || SummScore rank: 14 ...
Unsupervised Summarization Re-ranking
If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out. Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.74 (rank 1) || SummScore rank: 2 ...
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. WikiHow: re-ranking from the self-trained PEGASUS
Unsupervised Summarization Re-ranking
Talking in private will help you feel less nervous and more comfortable about asking for help.It will also give your teacher more time to help you because they aren't trying to teach at the same time. For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. The note could say, "Can we talk later about something personal? Thanks, Mark." You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue. , Sometimes teachers will try to offer advice on what you should do, when all you wanted was for them to listen. If you let your teacher know how they can help, it will make it easier for them to give you the help you want and need.Think about what kind of help you want. Ask yourself, "Do I want her to listen, to give me advice, or to do something about the problem?" Tell your teacher how they can help. For example, "Can you help me come up with ways to make more friends?" If you don't know how you want your teacher to help, you it's okay for you to say that, too. Try saying, "I need your help with a problem, but I don't know what kind of help I need." , Also, being honest will help your teacher figure out the right way to help you. The more truthful information your teacher has about what is going on, the more they will be able to help. If you are afraid you will get in trouble, then say that. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid."
Unsupervised Summarization Re-ranking
You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid." Scores Mean ROUGE: 22.54 (rank 14) || SummScore rank: 14
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you're afraid to approach your teacher, put a note on their chair.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.92 (rank 7) || SummScore rank: 18
Unsupervised Summarization Re-ranking
Beam #3 Summary For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. Try saying, "I need your help with a problem, but I don't know what kind of help I need.", Also, being honest will help your teacher figure out the right way to help you.
Unsupervised Summarization Re-ranking
Mean ROUGE: 29.41 (rank 9) || SummScore rank: 15
Unsupervised Summarization Re-ranking
Beam #4 Summary Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.
Unsupervised Summarization Re-ranking
Mean ROUGE: 28.40 (rank 10) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid." Scores Mean ROUGE: 27.36 (rank 12) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 31.47 (rank 6) || SummScore rank: 9
Unsupervised Summarization Re-ranking
Beam #7 Summary Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you are afraid to approach your teacher, put a note on their chair.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.92 (rank 7) || SummScore rank: 17
Unsupervised Summarization Re-ranking
Beam #8 Summary You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 28.06 (rank 11) || SummScore rank: 13 ...
Unsupervised Summarization Re-ranking
For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. Try saying, "I need your help with a problem, but I don't know what kind of help I need.", Also, being honest will help your teacher figure out the right way to help you. The more truthful information your teacher has about what is going on, the more they will be able to help Scores Mean ROUGE: 35.08 (rank 1) || SummScore rank: 2
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you're afraid to approach your teacher, put a note on their chair. It will also give your teacher more time to help you because they aren't trying to teach at the same time Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 34.51 (rank 2) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. 
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Neural sequence-to-sequence models (SEQ2SEQ) perform impressively well in several natural language processing tasks, such as machine translation (Sutskever et al., 2014;Bahdanau et al., 2015) or syntactic constituency parsing (Vinyals et al., 2015). However, they require massive parallel training datasets (Koehn and Knowles, 2017). Consequently there has been extensive work on utilizing non-parallel corpora to boost the performance of SEQ2SEQ models (Sennrich et al., 2016;Gülçehre et al., 2015), mostly in neural machine translation where models that require absolutely no parallel corpora have also been pro- posed (Artetxe et al., 2018;Lample et al., 2018b).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Unsupervised (or semi-supervised) SEQ2SEQ models have also been proposed for summarization tasks with no (or small) parallel text-summary sets, including unsupervised sentence compression. Current models, however, barely reach lead-N baselines (Fevry and Phang, 2018;Wang and Lee, 2018), and/or are non-differentiable (Wang and Lee, 2018;Miao and Blunsom, 2016), thus relying on reinforcement learning, which is unstable and inefficient. By contrast, we propose a sequence-to-sequence-to-sequence autoencoder, dubbed SEQ 3 , that can be trained end-to-end via gradient-based optimization. SEQ 3 employs differentiable approximations for sampling from categorical distributions (Maddison et al., 2017;Jang et al., 2017), which have been shown to outperform reinforcement learning (Havrylov and Titov, 2017). Therefore it is a generic framework which can be easily extended to other tasks, e.g., machine translation and semantic parsing via task-specific losses. In this work, as a first step, we apply SEQ 3 to unsupervised abstractive sentence compression. SEQ 3 ( §2) comprises two attentional encoderdecoder (Bahdanau et al., 2015) pairs (Fig. 1): a compressor C and a reconstructor R. C ( §2.1) receives an input text x = x 1 , . . . , x N of N words, and generates a summary y = y 1 , . . . , y M of M words (M<N), y being a latent variable. R and C communicate only through the discrete words of the summary y ( §2.2). R ( §2.3) produces a sequencex = x 1 , . . . ,x N of N words from y, try-  Figure 2: More detailed illustration of SEQ 3 . The compressor (C) produces a summary from the input text, and the reconstructor (R) tries to reproduce the input from the summary. R and C comprise an attentional encoder-decoder each, and communicate only through the (discrete) words of the summary. The LM prior incentivizes C to produce human-readable summaries, while topic loss rewards summaries with similar topicindicating words as the input text.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
ing to minimize a reconstruction loss L R = (x,x) ( §2.5). A pretrained language model acts as a prior on y, introducing an additional loss L P (x, y) that encourages SEQ 3 to produce human-readable summaries. A third loss L T (x, y) rewards summaries y with similar topic-indicating words as x. Experiments ( §3) on the Gigaword sentence compression dataset (Rush et al., 2015) and the DUC-2003 andDUC-2004 shared tasks (Over et al., 2007) produce promising results.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Our contributions are: (1) a fully differentiable sequence-to-sequence-to-sequence (SEQ 3 ) autoencoder that can be trained without parallel data via gradient optimization; (2) an application of SEQ 3 to unsupervised abstractive sentence compression, with additional task-specific loss functions; (3) state of the art performance in unsupervised abstractive sentence compression. This work is a step towards exploring the potential of SEQ 3 in other tasks, such as machine translation.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
The bottom left part of Fig. 2 illustrates the internals of the compressor C. An embedding layer projects the source sequence x to the word embeddings e s = e s 1 , . . . , e s N , which are then en-coded by a bidirectional RNN, producing h s = h s 1 , . . . , h s N . Each h s t is the concatenation of the corresponding left-to-right and right-to-left states (outputs in LSTMs) of the bi-RNN.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
To generate the summary y, we employ the attentional RNN decoder of Luong et al. (2015), with their global attention and input feeding. Concretely, at each timestep (t ∈ {1, . . . , M}) we compute a probability distribution a i over all the states h s 1 , . . . , h s N of the source encoder conditioned on the current state h c t of the compressor's decoder to produce a context vector c t .
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
The matrix W a is learned. We obtain a probability distribution for y t over the vocabulary V by combining c t and the current state h c t of the decoder.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
c t is also used when updating the state h c t of the decoder, along with the embedding e c t of y t and a countdown argument M − t (scaled by a learnable w d ) indicating the number of the remaining words of the summary (Fevry and Phang, 2018;Kikuchi et al., 2016).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
For each input x = x 1 , . . . , x N , we obtain a target length M for the summary y = y 1 , . . . , y M by sampling (and rounding) from a uniform distribution U (αN, βN); α, β are hyper-parameters (α < β < 1); we set M = 5, if the sampled M is smaller. Sampling M, instead of using a static compression ratio, allows us to train a model capable of producing summaries with varying (e.g., user-specified) compression ratios. Controlling the output length in encoder-decoder architectures has been explored in machine translation (Kikuchi et al., 2016) and summarization (Fan et al., 2018).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
To generate the summary, we need to sample its words y t from the categorical distributions p(y t |y <t , x), which is a non-differentiable process.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Soft-Argmax Instead of sampling y t , a simple workaround during training is to pass as input to the next timestep of C's decoder and to the corresponding timestep of R's encoder a weighted sum of all the vocabulary's (V ) word embeddings, using a peaked softmax function (Goyal et al., 2017):
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
where u c t is the unnormalized score in Eq. 2 (i.e., the logit) of each word w i and τ ∈ (0, ∞) is the temperature. As τ → 0 most of the probability mass in Eq. 5 goes to the most probable word, hence the operation approaches the arg max.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Gumbel-Softmax We still want to be able to perform sampling, though, as it has the benefit of adding stochasticity and facilitating exploration of the parameter space. Hence, we use the Gumbel-Softmax (GS) reparametrization trick (Maddison et al., 2017;Jang et al., 2017) as a low variance approximation of sampling from categorical distributions. Sampling a specific word y t from the softmax (Eq. 3) is equivalent to adding (element-wise) to the logits an independent noise sample ξ from the Gumbel distribution 1 and taking the arg max:
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Therefore, using the GS trick, Eq. 5 becomes:
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Straight-Through Both relaxations lead to mixtures of embeddings, which do not correspond to actual words. Even though this enables the compressor to communicate with the reconstructor using continuous values, thus fully utilizing the available embedding space, ultimately our aim is to constrain them to communicate using only natural language. In addition, an unwanted discrepancy is created between training (continuous embeddings) and test time (discrete embeddings).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
We alleviate these problems with the Straight-Through estimator (ST) (Bengio et al., 2013). Specifically, in the forward pass of training we discretizeẽ c t by using the arg max (Eq. 6), whereas in the backward pass we compute the gradients using the GS (Eq. 7). This is a biased estimator due 1 ξi = − log(− log(xi)), xi ∼ U (0, 1) to the mismatch between the forward and backward passes, but works well in practice. ST GS reportedly outperforms scheduled sampling (Goyal et al., 2017) and converges faster than reinforcement learning (Havrylov and Titov, 2017).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
The reconstructor (upper right of Fig. 2) works like the compressor, but its encoder operates on the embeddings e c 1 , . . . , e c M of the words y 1 , . . . , y M of the summary (exact embeddings of the sampled words y t in the forward pass, approximate differentiable embeddings in the backward pass).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
We initialize the hidden state of each decoder using a transformation of the concatenation [ − → h s N ;
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
← − h s 1 ] of the last hidden states (from the two directions) of its bidirectional encoder and a length vector, following Mallinson et al. (2018). The length vector for the decoder of the compressor C consists of the target summary length M, scaled by a learnable parameter w v , and the compression ratio M N .
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
W c is a trainable hidden layer. The decoder of the reconstructor R is initialized similarly.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Reconstruction Loss L R (x,x) is the (negative) log-likelihood assigned by the (decoder of) R to the input (correctly reconstructed) words
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
We do not expect L R (x,x) to decrease to zero, as there is information loss through the compression. However, we expect it to drive the compressor to produce such sentences that will increase the likelihood of the target words in the reconstruction. LM Prior Loss To ensure that the summaries y are readable, we pretrain an RNN language model (see Appendix) on the source texts of the full training set. We compute the Kullback-Leibler divergence D KL between the probability distributions of the (decoder of) the compressor (p(y t |y <t , x), Eq. 3) and the language model (p LM (y t |y <t , x)). Similar priors have been used in sentence compression (Miao and Blunsom, 2016) and agent communication (Havrylov and Titov, 2017).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
We also use the following task-specific losses. Topic Loss Words with high TF-IDF scores are indicative of the topic of a text (Ramos et al., 2003;Erkan and Radev, 2004). To encourage the compressor to preserve in the summary y the topicindicating words of the input x, we compute the TF-IDF-weighted average v x of the word embeddings of x and the average v y of the word embeddings of y and use their cosine distance as an additional loss
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
(Using TF-IDF in v y did not help.) All IDF scores are computed on the training set.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Length Penalty A fourth loss L L (not shown in Fig. 1) helps the (decoder of the) compressor to predict the end-of-sequence (EOS) token at the target summary length M. L L is the cross-entropy between the distributions p(y t |y <t , x) (Eq. 3) of the compressor at t = M + 1 and onward, with the one-hot distribution of the EOS token.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Parameter Sharing We tie the weights of layers encoding similar information, to reduce the number of trainable parameters. First, we use a shared embedding layer for the encoders and decoders, initialized with 100-dimensional GloVe embeddings (Pennington et al., 2014). Additionally, we tie the shared embedding layer with the output layers of both decoders (Press and Wolf, 2017;Inan et al., 2017). Finally, we tie the encoders of the compressor and reconstructor (see Appendix).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
OOVs Out-of-vocabulary words are handled as in Fevry and Phang (2018) (see Appendix).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Datasets We train SEQ 3 on the Gigaword sentence compression dataset (Rush et al., 2015). 2 It consists of pairs, each containing the first sentence of a news article (x) and the article's headline (y), a total of 3.8M/189k/1951 train/dev/test pairs. We also test (without retraining) SEQ 3 on DUC-2003 and DUC-2004 shared tasks (Over et al., 2007), containing 624/500 news articles each, paired with 4 reference summaries capped at 75 bytes. Methods compared We evaluated SEQ 3 and an ablated version of SEQ 3 . We only used the article 2 github.com/harvardnlp/sent-summary sentences (sources) of the training pairs from Gigaword to train SEQ 3 ; our model is never exposed to target headlines (summaries) during training or evaluation, i.e., it is completely unsupervised. Our code is publicly available. 3 We compare SEQ 3 to other unsupervised sentence compression models. We note that the extractive model of Miao and Blunsom (2016) relies on a pre-trained attention model using at least 500K parallel sentences, which is crucial to mitigate the inefficiency of sampling-based variational inference and REINFORCE. Therefore it is not comparable, as it is semi-supervised. The results of the extractive model of Fevry and Phang (2018) are also not comparable, as they were obtained on a different, not publicly available test set. We note, however, that they report that their system performs worse than the LEAD-8 baseline in ROUGE-2 and ROUGE-L on Gigaword. The only directly comparable unsupervised model is the abstractive 'Pretrained Generator' of Wang and Lee (2018). The version of 'Adversarial REINFORCE' that Wang and Lee (2018) consider unsupervised is actually weakly supervised, since its discriminator was exposed to the summaries of the same sources the rest of the model was trained on.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
As baselines, we use LEAD-8 for Gigaword, which simply selects the first 8 words of the source, and PREFIX for DUC, which includes the first 75 bytes of the source article. We also compare to supervised abstractive sentence compression methods (Tables 1-3). Following previous work, we report the average F1 of ROUGE-1, ROUGE-2, ROUGE-L (Lin, 2004). We implemented SEQ 3 with LSTMs (see Appendix) and during inference we perform greedy-sampling.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Results Table 1 reports the Gigaword results. SEQ 3 outperforms the unsupervised Pretrained Generator across all metrics by a large margin. It also surpasses LEAD-8. If we remove the LM prior, performance drops, esp. in ROUGE-2 and ROUGE-L. This makes sense, since the pretrained LM rewards correct word order. We also tried removing the topic loss, but the model failed to converge and results were extremely poor (Table 1). Topic loss acts as a bootstrap mechanism, biasing the compressor to generate words that maintain the topic of the input text. This greatly reduces variance due to sampling in early stages of training, alleviating the need to pretrain individual Type Supervision Methods R-1 R-2 R-L Supervised 3.8M
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
ABS (Rush et al., 2015) 29.55 11.32 26.42 SEASS (Zhou et al., 2017) 36.15 17.54 33.63 words-lvt5k-1sent (Nallapati et al., 2016) 36 Model R-1 R-2 R-L ABS (Rush et al., 2015) 28   (Rush et al., 2015) 28  components, unlike works that rely on reinforcement learning (Miao and Blunsom, 2016;Wang and Lee, 2018). Overall, both losses work in synergy, with the topic loss driving what and the LM prior loss driving how words should be included in the summary. SEQ 3 behaves similarly on DUC-2003 andDUC-2004 (Tables 2-3), although it was trained on Gigaword. In DUC-2003, however, it does not surpass the PREFIX baseline. Finally, Fig. 3 illustrates three randomly sampled outputs of SEQ 3 on Gigaword. In the first one, SEQ 3 copies several words esp. from the beginning of the input (hence the high ROUGE-L) exhibiting extractive capabilities, though still being adequately abstractive (bold words denote paraphrases). In the second one, SEQ 3 showcases its true abstractive power by paraphrasing and compressing multi-word expressions to single content words more heavily, still without losing the overall meaning. In the last example, SEQ 3 progressively becomes ungrammatical though interestingly retaining some content words from the input.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
The model tends to copy the first words of the input sentence in the compressed text (Fig. 3). We input: the american sailors who thwarted somali pirates flew home to the u.s. on wednesday but without their captain , who was still aboard a navy destroyer after being rescued from the hijackers . gold: us sailors who thwarted pirate hijackers fly home SEQ 3 : the american sailors who foiled somali pirates flew home after crew hijacked .
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
input: the central election commission -lrb-cec -rrb-on monday decided that taiwan will hold another election of national assembly members on may # . gold: national <unk> election scheduled for may SEQ 3 : the central election commission -lrb-cec UNK announced elections .
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
input: dave bassett resigned as manager of struggling english premier league side nottingham forest on saturday after they were knocked out of the f.a. cup in the third round , according to local reports on saturday . gold: forest manager bassett quits . hypothesize that since the reconstructor is autoregressive, i.e., each word is conditioned on the previous one, errors occurring early in the generated sequence have cascading effects. This inevitably encourages the compressor to select the first words of the input. A possible workaround might be to modify SEQ 3 so that the first encoder-decoder pair would turn the inputs to longer sequences, and the second encoder-decoder would compress them trying to reconstruct the original inputs. In future work, we plan to explore the potential of SEQ 3 in other tasks, such as unsupervised machine translation (Lample et al., 2018a;Artetxe et al., 2018) and caption generation (Xu et al., 2015).
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
where w τ is a trainable parameter and τ (h c t ) ∈ (0, 1). Havrylov and Titov (2017) add τ 0 as a hyper-parameter which controls the upper bound of the temperature. In our experiments, we had convergence problems with the learned temperature technique. We found that the compressor preferred values close to the upper bound, which led to unstable training, forcing us to set τ 0 > 1 to stabilize the training process. Our findings align with the behavior reported by Gu et al. (2018). Consequently, we follow their choice and fix τ = 0.5, which worked well in practice.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
The vocabulary of our experiments comprises the 15k most frequent words of Gigaword's training input texts (without looking at their summaries). To handle OOVs, we adopt the approach of Fevry and Phang (2018), which can be thought of as a simpler form of copying compared to pointer networks (See et al., 2017). We use a small set (10 in our experiments) of special OOV tokens OOV 1 , OOV 2 , . . . , OOV 10 , whose embeddings are updated during learning. Given an input text x = x 1 , . . . , x N , we replace (before feeding x to SEQ 3 ) each unknown word x i with the first unused (for the particular x) OOV token, taking care to use the same OOV token for all the occurrences of the same unknown word in x. For example, if 'John' and 'Rome' are not in the vocabulary, then "John arrived in Rome yesterday. While in Rome, John had fun." becomes "OOV 1 arrived in OOV 2 yesterday. While in OOV 2 , OOV 1 had fun." If a new unknown word x i is encountered in x and all the available OOV tokens have been used, x i is replaced by 'UNK', whose embedding is also updated during learning. The OOV tokens (and 'UNK') are included in the vocabulary, and SEQ 3 learns to predict them as summary words, in effect copying the corresponding unknown words of x. At test time, we replace the OOV tokens with the corresponding unknown words.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
Our model is an instance of Variational Auto-Encoders (VAE) (Kingma and Welling, 2014). A common problem in VAEs is that the reconstructor tends to disregard the latent variable. We weaken the reconstructor R, in order to force it to fully utilize the latent sequence y to generatex. To this end, we employ word dropout as in Bowman et al. (2016) and randomly drop a percentage of the input words, thus forcing R to rely solely on y to make good reconstructions.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
We implemented SEQ 3 in PyTorch (Paszke et al., 2017). All the RNNs are LSTMs (Hochreiter and Schmidhuber, 1997). We use a shared encoder for the compressor and the reconstructor, consisting of a two-layer bidirectional LSTM with size 300 per direction. We use separate decoders for the compressor and the reconstructor; each decoder is a two-layer unidirectional LSTM with size 300. The (shared) embedding layer of the compressor and the reconstructor is initialized with 100-dimensional GloVe embeddings (Pennington et al., 2014) and is tied with the output (projec-tion) layers of the decoders and jointly finetuned during training. We apply layer normalization (Ba et al., 2016) to the context vectors (Eq. 1) of the compressor and the reconstructor. We apply word dropout ( §A.3) to the reconstructor with p = 0.5.
SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
During training, the summary length M is sampled from U (0.4 N, 0.6 N); during testing, M = 0.5 N. The four losses are summed, λs being scalar hyper-parameters. L = λ R L R + λ P L P + λ T L T + λ L L L We set λ R = λ T = 1, λ L = λ P = 0.1. We use the Adam (Kingma and  optimizer, with batch size 128 and the default learning rate 0.001. The network is trained for 5 epochs. LM Prior The pretrained language model is a two-layer LSTM of size 1024 per layer. It uses its own embedding layer of size 256, which is randomly initialized and updated when training the language model. We apply dropout with p = 0.2 to the embedding layer and dropout with p = 0.5 to the LSTM layers. We use Adam (Kingma and  with batch size 128 and the network is trained for 30 epochs. The learning rate is set initially to 0.001 and is multiplied with γ = 0.5 every 10 epochs. Evaluation Following Chopra et al. (2016), we filter out pairs with empty headlines from the test set. We employ the PYROUGE package with "-m -n 2 -w 1.2" to compute ROUGE scores. We use the provided tokenizations of the Gigaword and DUC-2003, DUC-2004 datasets. All hyper-parameters were tuned on the development set.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Text summarization aims at generating accurate and concise summaries from input document(s). In contrast to extractive summarization which merely copies informative fragments from the input, abstractive summarization may generate novel words. A good abstractive summary covers principal information in the input and is linguistically fluent.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In abstractive summarization, sequence-to-sequence (Sutskever et al., 2014) has become a dominant framework using encoder-decoder architectures based on RNNs (Chung et al., 2014;Hochreiter & Schmidhuber, 1997) and more recently Transformers (Vaswani et al., 2017). Most prior work on neural abstractive summarization relied on large-scale, high-quality datasets of supervised document-summary pairs (Hermann et al., 2015) and achieved promising results (Rush et al., 2015;Nallapati et al., 2016;See et al., 2017). In recent years, there has been increased interest in collecting new summarization datasets that have more abstractive summaries (Narayan et al., 2018), have longer documents, (Cohan et al., 2018;Sharma et al., 2019), utilize multiple documents (Fabbri et al., 2019), and are sourced from diverse domains (Grusky et al., 2018;Koupaee & Wang, 2018;Kim et al., 2019;Kornilova & Eidelman, 2019;Zhang & Tetreault, 2019); however, there has been little work on systematic evaluation of models across these broad settings.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Contemporaneously, the adoption of Transformer models (Vaswani et al., 2017) pre-trained using self-supervised objectives on large text corpora (Radford et al., 2018a;Devlin et al., 2019) have improved performance on many NLP tasks Rajpurkar et al., 2016).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Recent work leveraging such pre-training for Transformerbased sequence-to-sequence models (Dong et al., 2019;Song et al., 2019;Rothe et al., 2019;Lewis et al., 2019;Raffel et al., 2019) has extended the success to text generation, including abstractive summarization.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this work, we study pre-training objectives specifically for abstractive text summarization and evaluate on 12 downstream datasets spanning news (Hermann et al., 2015;Narayan et al., 2018;Grusky et al., 2018;Rush et al., 2015;Fabbri et al., 2019), science (Cohan et al., 2018), short stories (Kim et al., 2019), instructions (Koupaee & Wang, 2018), emails (Zhang & Tetreault, 2019), patents (Sharma et al., 2019), and legislative bills (Kornilova & Eidelman, 2019). We find that masking whole sentences from a document and generating these gap-sentences from the rest of the document works well as a pre-training objective for downstream summarization tasks. In particular, choosing putatively important sentences outperforms lead or randomly selected ones. We hypothesize this objective is suitable for abstractive summarization as it closely resembles the downstream task, encouraging whole-document understanding and summary-like generation. We call this self-supervised objective Gap Sentences Generation (GSG). Using GSG to pre-train a Transformer encoder-decoder on large corpora of documents (Web and news articles) results in our method, Pre-training with Extracted Gap-sentences for Abstractive SUmmarization Sequence-to-sequence models, or PEGASUS.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
With our best 568M parameter model trained on the recently introduced C4 (Raffel et al., 2019) corpus we equal or exceed state-of-the-art on the 12 summarization tasks we consider. We further push forward the state-of-the-art using a newly collected text corpus comprised of news-like articles we call HugeNews, including the highly competitive XSum and CNN/DailyMail summarization datasets.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Large-scale document-summary datasets are rare and in practice there is a mismatch between research datasets and real-world use-cases where collecting summaries is expensive; the most common setting is that of low-resource summarization. We simulate this setting and show that our model is able to adapt very quickly when fine-tuning with small numbers of supervised pairs, obtaining state-of-the-art results in 6 datasets with only 1000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Qualitatively we observed high quality outputs from our best models and validated this in human evaluation studies. We found that PEGASUS summaries are at least as good as reference summaries for the datasets we assessed -XSum, CNN/DailyMail, and Reddit TIFU -even at low-levels of supervision.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To summarize our contributions:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We propose a new self-supervised pre-training objective for abstractive summarization, gap-sentences generation, and study strategies for selecting those sentences.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We evaluate the proposed pre-training objective on a broad range of downstream summarization tasks, with careful ablations to choose the best model settings, which we use to train a 568M parameter PEGASUS model that surpasses or is on-par with the state-of-theart on all 12 downstream datasets considered.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We show how good abstractive summarization performance can be achieved across broad domains with very little supervision by fine-tuning the PEGASUS model and surpassing previous state-of-the-art results on many tasks with as little as 1000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• We conducted human evaluation studies to validate our experimental design and demonstrate human-level summarization performance on XSum, CNN/DailyMail, and Reddit TIFU.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We propose a new pre-training objective, GSG, in this work, but for comparison, we also evaluate BERT's maskedlanguage model objective, in isolation and in conjunction with GSG.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We hypothesize that using a pre-training objective that more closely resembles the downstream task leads to better and faster fine-tuning performance. Given our intended use for abstractive summarization, our proposed pre-training objective involves generating summary-like text from an input document. In order to leverage massive text corpora for pretraining, we design a sequence-to-sequence self-supervised objective in the absence of abstactive summaries. A naive option would be to pre-train as an extractive summarizer; however, such a procedure would only train a model to copy sentences, thus not suitable for abstractive summarization.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Inspired by recent success in masking words and contiguous spans (Joshi et al., 2019;Raffel et al., 2019), we select and mask whole sentences from documents, and concatenate the gap-sentences into a pseudo-summary. The corresponding position of each selected gap sentence is replaced by a mask token [MASK1] to inform the model. Gap sentences ratio, or GSR, refers to the number of selected gap sentences to the total number of sentences in the document, which is similar to mask rate in other works.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To even more closely approximate a summary, we select sentences that appear to be important/principal to the document. The resulting objective has both the empirically demonstrated benefits of masking, and anticipates the form of the downstream task.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We consider 3 primary strategies for selecting m gap sentences without replacement from a document, D = {x i } n , comprised of n sentences:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Random Uniformly select m sentences at random.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Lead Select the first m sentences.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Principal Select top-m scored sentences according to importance. As a proxy for importance we compute ROUGE1-F1 (Lin, 2004) between the sentence and the rest of the document,
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this formulation sentences are scored independently (Ind) and the top m selected. We also consider selecting them sequentially (Seq) as in Nallapati et al. (2017) by greedily maximizing the ROUGE1-F1 between selected sentences, S ∪ {x i }, and remaining sentences, D \ (S ∪ {x i }) as in Algorithm 1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Algorithm 1 Sequential Sentence Selection 1: S := ∅ 2: for j ← 1 to m do 3:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
When calculating ROUGE1-F1, we also consider n-grams as a set (Uniq) instead of double-counting identical n-grams as in the original implementation (Orig). This results in four variants of the principal sentence selection strategy, choosing Ind/Seq and Orig/Uniq options.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
An example containing lead, random and principal gap sentence selection strategies are shown in Figure 2.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
INVITATION ONLY We are very excited to be co-hosting a major drinks reception with our friends at Progress. This event will sell out, so make sure to register at the link above. Speakers include Rajesh Agrawal, the London Deputy Mayor for Business, Alison McGovern, the Chair of Progress, and Seema Malhotra MP. Huge thanks to the our friends at the ACCA, who have supported this event. The Labour Business Fringe at this year's Labour Annual Conference is being co-sponsored by Labour in the City and the Industry Forum. Speakers include John McDonnell, Shadow Chancellor, and Rebecca Long-Bailey, the Shadow Chief Secretary to the Treasury, and our own Chair, Kitty Ussher. Attendance is free, and refreshments will be provided. Figure 2: An example of sentences (from the C4 corpus) selected by Random, Lead and Ind-Orig respectively. Best viewed in color.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Following BERT, we select 15% tokens in the input text, and the selected tokens are (1) 80% of time replaced by a mask token [MASK2], or (2) 10% of time replaced by a random token, or (3) 10% of time unchanged. We apply MLM to train the Transformer encoder as the sole pre-training objective or along with GSG. When MLM is the sole pre-training objective, the Transformer decoder shares all parameters with encoder when fine-tuning on downstream tasks following Rothe et al. (2019). 
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
For pre-training we considered two large text corpora:
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• C4, or the Colossal and Cleaned version of Common Crawl, introduced in Raffel et al. (2019); consists of text from 350M Web-pages (750GB).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• HugeNews, a dataset of 1.5B articles (3.8TB) collected from news and news-like websites from 2013-2019. A whitelist of domains ranging from highquality news publishers to lower-quality sites such as high-school newspapers, and blogs was curated and used to seed a web-crawler. Heuristics were used to identify news-like articles, and only the main article text was extracted as plain text.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
For downstream summarization, we only used public abstractive summarization datasets, and access them through TensorFlow Summarization Datasets 1 , which provides publicly reproducible code for dataset processing and train/validation/test splits. We used train/validation/test ratio of 80/10/10 if no split was provided, and 10% train split as validation if there was no validation split.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
XSum (Narayan et al., 2018) consists of 227k BBC articles from 2010 to 2017 covering a wide variety of subjects along with professionally written single-sentence summaries.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
CNN/DailyMail (Hermann et al., 2015) dataset contains 93k articles from the CNN, and 220k articles the Daily Mail newspapers. Both publishers supplement their articles with bullet point summaries. We use the non-anonymized variant used in See et al. (2017).
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
NEWSROOM (Grusky et al., 2018) is a large dataset containing 1.3M article-summary pairs written by authors and editors in the newsrooms of 38 major publications between 1998 and 2017.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Multi-News (Fabbri et al., 2019) is a multi-document summarization dataset consisting of 56k pairs of news articles and their human-written summaries from the site newser.com.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Gigaword (Rush et al., 2015) contains 4M examples extracted from news articles (seven publishers) from the Gigaword corpus (Graff et al., 2003). The task is to generate the headline from the first sentence.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
arXiv, PubMed (Cohan et al., 2018) are two long document datasets of scientific publications from arXiv.org (113k) and PubMed (215k). The task is to generate the abstract from the paper body.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
BIGPATENT (Sharma et al., 2019) consists of 1.3 million U.S. patents along with human summaries under nine patent classification categories.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
WikiHow (Koupaee & Wang, 2018) is a large-scale dataset of instructions from the online WikiHow.com website. Each of 200k examples consists of multiple instruction-step paragraphs along with a summarizing sentence. The task is to generate the concatenated summary-sentences from the paragraphs.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Reddit TIFU (Kim et al., 2019) contains 120K posts of informal stories from the online discussion forum Reddit, more specifically the TIFU sub-reddit from 2013-Jan to 2018-Mar. The sub-reddit posts strictly follow the rule of writing a descriptive "TL;DR" summary and has higher qual-ity than (Völske et al., 2017) (which used more subreddits) based on our manual inspection. We uses the TIFU-long subset (using TLDR as summaries) in the work.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
AESLC (Zhang & Tetreault, 2019) consists of 18k email bodies and their subjects from the Enron corpus (Klimt & Yang, 2004), a collection of email messages of employees in the Enron Corporation.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
BillSum ( We used sinusoidal positional encoding following Vaswani et al. (2017). For optimization, both pre-training and finetuning used Adafactor (Shazeer & Stern, 2018) with square root learning rate decay and dropout rate of 0.1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We used greedy-decoding for studies in Section 6.1, and used beam-search with a length-penalty, α, as in Wu et al. (2016) for the final large model.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
All experiments' hyper parameters can be found in Appendix C and reported numbers are in Appendix D and E.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We used PEGASUS BASE to evaluate choices of pre-training corpus, pre-training objective, and vocabulary size. For reproducibility, we evaluated the latter two using the publicly available C4 corpus.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Note that the y-axis in Figures 3, 4 Figure 3: Effect of pre-training corpus. PEGASUS BASE pre-trained on C4 (350M Web-pages) and HugeNews (1.5B news-like documents). Figure 3 shows that pre-training on HugeNews was more effective than C4 on the two news downstream datasets, while the non-news informal datasets (WikiHow and Reddit TIFU) prefer the pre-training on C4. This suggests pretraining models transfer more effectively to downstream tasks when their domains are aligned better.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
GSG We compared six variants of GSG (Lead, Random, Ind-Orig, Ind-Uniq, Seq-Orig, Seq-Uniq) while choosing 30% sentences as gap sentences. As shown in Figure 4a, Ind-Orig achieved the best performance followed by Seq-Uniq. Ind-Orig and Seq-Uniq were consistently better (or similar) than Random and Lead across the four downstream datasets. Lead had decent performance on the two news datasets but was significantly worse on the two non-news datasets, which agrees findings of lead bias in news datasets (See et al., 2017;Zhong et al., 2019). The results suggest choosing principal sentences works best for downstream summarization tasks, and we chose Ind-Orig for the PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
A significant hyper-parameter in GSG is the gap-sentences ratio (GSR). A low GSR makes the pre-training less challenging and computationally efficient. On the other hand, choosing gap sentences at a high GSR loses contextual in-  formation necessary to guide the generation. We compared GSRs from 15% to 75%. For a fair comparison, the original documents were truncated to have up to 400 words. The maximum input length, L input in the encoder and the maximum target length, L target in the decoder were set as 512 tokens. Figure 4b shows that different downstream datasets had slightly different optima. The best performance always had GSR lower than 50%. The model with 15% gap sentences achieved the highest ROUGE scores on CNN/DailyMail, while XSum/Reddit TIFU and WikiHow did better with 30% and 45% respectively. When scaling up to PEGASUS LARGE (Section 6.2), we chose an effective GSR of 30%.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
MLM As mentioned, the MLM objective can either be applied solely or together with GSG. We jointly trained MLM with GSG Ind-Orig (MLM & Ind-Orig), which masks 30% sentences and extra 15% tokens in unselected sentences, as shown in Figure 1. Figure 4a shows that the model pretrained with MLM alone performed significantly worse and MLM & Ind-Orig had similar performance as Random. Interestingly, when comparing MLM & Ind-Orig to Ind-Orig, we empirically observed MLM improved fine-tuning performance at early pre-training checkpoints (100k -200k steps), but inhibited further gains with more pre-training steps (500k). Therefore, we chose not to include MLM in PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
We compared two tokenization methods 2 : Byte-pairencoding algorithm (BPE) (Wu et al., 2016;Sennrich et al., 2016), and SentencePiece Unigram algorithm (Unigram) proposed in Kudo (2018). We evaluated Unigram with different vocabulary sizes ranging from 32k to 256k. In these experiments, models were pre-trained for 500k steps on the C4 corpus with the Ind-Orig objective and 15% GSR. As shown in Figure 5, BPE and Unigram were comparable on news datasets while Unigram outperformed BPE 2 Implemented in https://github.com/google/sentencepiece  on non-news datasets, especially WikiHow. On XSum and CNN/DailyMail, Unigram 96k achieved the highest ROUGE scores. On WikiHow and Reddit TIFU, the best configurations were Unigram 128k and 64k respectively. Therefore, we used the overall best vocabulary option Unigram 96k in PEGASUS LARGE .
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Compared with PEGASUS BASE , the large model PEGASUS LARGE had increased capacity from larger hidden size (H : 768 → 1024, F : 3072 → 4096, A : 12 → 16), number of layers (L : 12 → 16) and traversed much more data, due to larger batch size (B : 256 → 8192) (same number of pre-training steps, 500k). We adopted the best practices found in the PEGASUS BASE ablation studies using the GSG (Ind-Orig) pre-training objective without MLM and Unigram vocabulary size of 96k. In total, PEGASUS LARGE had 568M parameters.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To encourage the model to copy, which is an important aspect of the more extractive datasets, we left 20% of selected sentences unchanged in the input instead of replacing with [MASK1]. We increased the GSR to 45% to achieve a similar number of "gaps" as the optimal 30% found above. We reported the performance of the models pre-trained on HugeNews and C4 separately. We conducted a simple hyper-parameter sweep of learning rate and length penalty,  (Lewis et al., 2019;Shi et al., 2019;Fabbri et al., 2019;Koupaee & Wang, 2018;Kim et al., 2019;Subramanian et al., 2019;Song et al., 2019;Zhang & Tetreault, 2019;Kornilova & Eidelman, 2019). We only compared with previous abstractive models except on BillSum which had extractive results only. BIGPATENT, arXiv, PubMed and Multi-News datasets contain very long summaries and we truncate them to 256 tokens, in similar range compared to (Sharma et al., 2019;Cohan et al., 2018;Fabbri et al., 2019;Goodman et al., 2019). Best ROUGE numbers on each dataset and numbers within 0.15 of the best numbers are bolded.  CNN/DailyMail, Multi-News, arXiv, PubMed, BIG-PATENT datasets contain input documents longer than the maximum input length (L input = 512 tokens) in pretraining. This would present a problem for position embeddings which would never be updated for longer input lengths, but we confirm the postulation that sinusoidal positional encodings (Vaswani et al., 2017) generalize well when fine-tuning PEGASUS LARGE beyond the input lengths observed in training up to L input = 1024 tokens. Since average input length in BIGPATENT, arXiv, PubMed and Multi-News are well beyond 1024 tokens, further scaling up L input or applying a two-stage approach (Liu et al., 2018) may improve performance even more, although this is outside the scope of this work.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Tables 1 and 2 show the performance improvements of PEGASUS BASE and PEGASUS LARGE on downstream datasets. While PEGASUS BASE exceeded current state-ofthe-art on many datasets, PEGASUS LARGE achieved better than state-of-the-art results on all downstream datasets using HugeNews, although C4 performed better on WikiHow.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The improvement from a Transformer model without pretraining (Transformer BASE ) to PEGASUS LARGE was more significant on smaller datasets. For example, the ROUGE2-F1 scores nearly tripled on AESLC and quintupled on Reddit TIFU. The large jumps in performance suggest that small text summarization datasets benefit the most from pre-training. We further investigate low resource summarization in Section 6.3.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In real-world practice, it is often difficult to collect a large number of supervised examples to train or fine-tune a summarization model. To simulate the low-resource summarization setting, we picked the first 10 k (k = 1, 2, 3, 4) training examples from each dataset to fine-tune PEGASUS LARGE (HugeNews) . We fine-tuned the models up to 2000 steps with batch size 256, learning rate 0.0005, and picked the checkpoint with best validation performance. In Fig Khandelwal et al. (2019) with 3000 examples.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Overall, we observed high-linguistic quality (in terms of fluency and coherence), closely emulating the style of groundtruth summaries. While some previous work suggested that maximum likelihood training results in repetitive text in model outputs (Welleck et al., 2019) we found this to be rare in our outputs and did not require additional countermeasures to mitigate dis-fluencies.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Although ROUGE clearly has its draw-backs (Kryscinski et al., 2019), over-penalizing abstractive approaches com-pared to extractive ones and having no sense of linguistic quality, we found that choosing perplexity-optimized models using aggregated ROUGE (rather than directly optimizing ROUGE as in Paulus et al. (2017)) resulted in qualitatively good models. Randomly sampled (by a program) model decodes across all datasets and a broad range of ROUGE scores can be found in Appendix I.We found that even low-ROUGE model summaries often were highquality, Figure G.1.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To assess how close PEGASUS LARGE is to human performance we conducted human evaluation experiments on Amazon Mechanical Turk comparing model summaries with (human) reference summaries given the input document. The examples were drawn from the XSum, CNN/DailyMail, and Reddit TIFU datasets; the first two were chosen due to their popularity in past work, and the third was chosen for its significant difference in style. Workers were asked to rate the summaries on a 1-5 scale, with higher being better (full experiment details provided in Appendix F) and a paired t-test was used to assess whether scores were significantly different from human.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In the first experiment, PEGASUS LARGE (HugeNews), PEGASUS LARGE (C4), and Transformer BASE were compared with reference summaries; in the second experiment, PEGASUS LARGE (HugeNews) fine-tuned using 10, 100, 1000, and all supervised examples were compared with references; the results are shown in Table 3. According to the significance level of p < 0.01, both PEGASUS LARGE (HugeNews) and PEGASUS LARGE (C4) outputs were at least as good as the reference summaries in all cases. Even at low-levels of supervision PEGASUS LARGE (HugeNews) was not measurably worse than human summaries on XSum and CNN/DailyMail. In the Reddit TIFU case, however, perhaps due to its diverse writing styles, human performance required full supervision.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The pre-training corpora are a large collection of documents from the Internet and potentially have overlap with the downstream test sets. In this section, we measured the extent of overlap between the pre-training corpus and downstream datasets. We also studied if the pre-trained model was able to exploit memorization to achieve higher performance on the downstream datasets.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
To measure the overlap, we calculated similarities between all pairs of downstream test set targets and pre-training documents. We use the ROUGE-2 recall as a similarity measure (common 2-grams / test set targets 2-grams). It is not necessarily exact match even if the similarity score is 1.0. We filtered all test set examples that have similarity to any pre-training example above a threshold, and recalculated the ROUGE scores on the remaining test set. In Figure 7  tences ratio uniformly between 15%-45%.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
(3) Importance sentences were stochastically sampled with 20% uniform noise on their scores. (4) The model was pre-trained for 1.5M steps instead of 500k steps, as we observed slower convergence of pre-training perplexity. (5) The SentencePiece tokenizer was updated to encode the newline character. The PEGASUS LARGE (mixed, stochastic) model achieved best results on almost all downstream tasks, as shown in Table 4.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
In this work, we proposed PEGASUS, a sequence-tosequence model with gap-sentences generation as a pretraining objective tailored for abstractive text summarization. We studied several gap-sentence selection methods and identified principle sentence selection as the optimal strategy. We demonstrated the effects of the pre-training corpora, gap-sentences ratios, vocabulary sizes and scaled up the best configuration to achieve state-of-the-art results on all 12 diverse downstream datasets considered. We also showed that our model was able to adapt to unseen summarization datasets very quickly, achieving strong results in as little as 1000 examples. We finally showed our model summaries achieved human performance on multiple datasets using human evaluation.
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
The training code and instructions for using model checkpoints can be found at https://github.com/google-research/ pegasus
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Extractive summarization has been studied extensively over the past several decades (Gupta and Lehal, 2010;Ferreira et al., 2013). However, humans typically summarize abstractively, paraphrasing and performing non-trivial compression of details that are difficult to encode in classical summarization algorithms. Recent progress in neural language models (Sutskever et al., 2014;Jozefowicz et al., 2016;Radford et al., 2019) has enabled models to generate near fluent language that are not mere regurgitations of training data. With large datasets of (document, summary) pairs * Equal contribution † Work done while interning at Google Brain.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(a) Story (paragraph) to summarize -Jason listened to the weather and heard it was going to be sunny. He thought the kids might like to go swimming. He gathered up the swimsuits, towels and sunscreen. Jason and the kids got into the truck and drove to the beach. They spent the next 2 hours playing and splashing in the surf.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(b) Three human summaries -Jason saw a nice weather forecast and went to the beach with his kids for 2 hours.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
-Jason took the kids swimming at the beach on a sunny day.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
-Jason decided to take the kids to the beach since it was a sunny day.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(c) Best extractive sentence -Jason listened to the weather and heard it was going to be sunny.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(d) Unsupervised abstractive summary (ours) -Jason listened to the weather to be the coolest and fun day and he went to the beach and got ready. primarily from the news domain, abstractive summarization has been approached as a supervised neural sequence transduction problem (Rush et al., 2015;Nallapati et al., 2016;Narayan et al., 2018;Fabbri et al., 2019). Outside of news, however, such large parallel datasets are rare, due to the cost-prohibitive "labeling" process (i.e., reading long documents and writing summaries). Much more commonly available are large corpora of documents without summaries. It is therefore desirable to have models capable of automatically summarizing documents abstractively with little to no supervision. In contrast to abstractive methods, many ex-tractive approaches do not rely on example summaries. Inspired by that we study the extreme case of no exposure to summaries during training, or unsupervised (zero-shot) abstractive summarization (UAS). Recently there has been some, thus far limited, work on UAS for both the multi-document (Chu and Liu, 2019) and single-document (Isonuma et al., 2019) cases. In this work we focus on UAS of paragraphs with a sentence, which is perhaps the most basic form of multi-sentence singledocument summarization. In particular, we summarize the five-sentence stories from ROCStories (Mostafazadeh et al., 2016) and show that there is a non-trivial abstractive gap between human and extractive performance, making it a suitable benchmark for measuring progress in UAS.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Our approach is based on training a denoising auto-encoder (Vincent et al., 2008) that encodes sentences and paragraphs in a shared space. The decoder input is pre-pended with a special token to signal whether to decode a sentence or a paragraph, and a summarizing sentence is generated by decoding a sentence from an encoded paragraph. However, we found that traditional approaches to training such an auto-encoder resulted in non-overlapping sentence and paragraph latent sub-spaces -which we call segregation -resulting in long, multi-sentence summaries. We describe architectural modifications and selfsupervised pre-training objectives to prevent segregation and improve performance significantly beyond sentence-extractive baselines. While the goal of human performance is still far, we believe the techniques presented here are a major step in that direction.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In summary, our contributions are as follows.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
1. We introduce a new benchmark task, ROC-Summ, for measuring progress toward human performance on UAS.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
2. We propose a novel end-to-end, fullydifferentiable neural model for UAS of paragraphs.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
3. We describe novel self-supervised (pretraining) and denoising objectives that significantly improve performance beyond sentence-extractive baselines.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
4. We conduct ablation experiments showing the importance of architectural choices and model objectives.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Our new task re-purposes and augments an existing dataset, ROCStories (Mostafazadeh et al., 2016, originally designed for the "Story Cloze Test" (SCT), where a model must choose the correct fifth sentence of two candidates given the first four. The stories are self-contained, diverse, realistic, non-technical, high-quality, and have a coherent story arch. The human performance on the SCT task is close to 100%. Our proposed UAS task involves summarizing the five-sentence training ROCStories with a single sentence without summaries at training, i.e., perform zero-shot summarization. We found summaries by independent human raters have high similarity in this task, suggesting it is well-defined, and relatively unambiguous, in contrast to other summarization tasks where the desired length or the topic of the summary is unclear. The simplicity of the task is conducive to iterating quickly and making rapid progress in UAS. Having only five-sentences and a low-bound on total number of words avoids engineering issues that often arise with very long sequences. Due to the constraints, it is simple to calculate the maximum (sentence) extractive summarization performance, which is far from the human performance (see Table 1) suggesting a need for abstractive models. In contrast, it is unclear for example, what human performance is on the popular CNN/DailyMail (supervised) summarization task (See et al., 2017) and whether abstractive models provide much of a benefit over extractive ones on it (Kryściński et al., 2019).
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
To evaluate summarization models, we collected multiple summaries from independent, experienced, and highly-reputable Amazon Mechanical Turk (AMT) workers. The full worker selection criteria and AMT template can be found in Appendix A. Collecting multiple summaries allowed us to estimate human performance as well as treat multiple "right" answers more fairly by averaging metrics across the summaries for a given example. An ex-ample story with 3 human summaries, the best extractive sentence, and one of our model summaries can be found in Figure 1.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Our summarization model, SummAE, is depicted in Figure 2 and consists of a denoising autoencoder, (φ enc , φ dec ), capable of auto-encoding text sequences T that can be sentences or paragraphs. In particular, the encoder, φ enc : T → R z dim , is a deterministic function, parameterized by θ E , mapping text to a latent vector representation, φ enc (t) = z ∈ R z dim . For an input sequence t ∈ T, we add random noise to t (described in Section 4.2),t = η(t) ∈ T. We consider two encoder implementations:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
1. a bidirectional RNN (Schuster and Paliwal, 1997) where φ enc (t) = z is derived from the RNN's final hidden state, h = [h → , h ← ] ∈ R 2h dim , followed by an affine transformation:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
2. a Transformer stacked with N identical Transformer encoder blocks (denoted as TRF enc ) from Vaswani et al. (2017). φ enc (t) = z is derived from the output representation of the first token as in Devlin et al.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(2019), followed by an affine transformation:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
The decoder, parameterized by θ G , is an autoregressive generative model defining a probability distribution over T conditioned on z. We also condition the decoder on whether to decode a sentence or paragraph, using two different beginningof-sequence tokens indicated by β ∈ {0, 1}. The reconstructed input,t, is obtained by sampling one token at a time until a special end-of-sequence token is obtained:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
(1)
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We consider two decoder implementations:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
1. a unidirectional RNN that conditions on z by concatenating the decoder input embeddings with z at each time-step;
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
2. a Transformer with causal, masked attention that conditions by adding z to each input embedding. This is similar to the Transformer decoder in Vaswani et al. (2017) without decoder-encoder attention.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In both cases, we avoid decoder-encoder attention to encourage all semantic information to be encapsulated in z.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In our dataset, a single example is a paragraph consisting of n(p) sentences, p = (s 1 , ..., s n(p) ), and the auto-encoder contributes two reconstruction loss terms, one for the sentences and one for the paragraph, weighted by λ s and λ p .
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
where ce (x,x) is the standard cross-entropy loss between the input sequence and its reconstruction. We optimize it using gradient descent and teacherforcing (Williams, 1992).
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Our approach to summarize paragraphs with a sentence is to prompt the decoder to generate a sentence (using β = 0) conditioned on the latent vector z of a paragraph. However, simply training an RNN or Transformer auto-encoder as described generally fails (as we see in Section 5.4); we hypothesize that the encoder can learn to map sentences and paragraphs to separate regions in the latent space, and the decoder can recognize whether to decode a sentence or paragraph in reconstruction based solely on the location of z and ignore β. We find this can result in the decoder generating a paragraph even if prompted for a sentence. We call this phenomenon segregation.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Ideally, the auto-encoder learns a higher-level latent concept conveyed in paragraphs and sentences, disentangled from their original expression as paragraphs and sentences. To explicitly encourage this we investigated adding an adversarial discriminator/critic D, which is trained to classify whether a latent vector z is an encoded sentence or a paragraph. In other words, it learns p D (y(t)|φ enc (t)), where y(t) is 0 if t is a sentence and 1 if a paragraph, while the auto-encoding loss is augmented to encourage fooling the discriminator into classifying paragraphs as sentences. Similar approaches have been used in style transfer (Hu et al., 2017;Shen et al., 2017;Romanov et al., 2019) and unsupervised machine translation (Lample et al., 2018a), although not for abstractive summarization. Details of our implementation can be found in Appendix A.2.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In our experiments we found adding the critic was very effective for generating one-sentence short summaries. However, for some encoderdecoder configurations, the critic was found to be unnecessary and even harmed performance, which we discuss in Section 5.4.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We use a denoising auto-encoder rather than a standard auto-encoder by reconstructing a text sequence from a noisy version of it. Denoising can be seen as a useful self-supervised objective for improving representations as seen in Devlin et al.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
; it also serves as a form of data augmentation, effectively increasing the number of training examples; finally, it discourages merely learning the identity function without having to reduce the information bottleneck z dim to a very small value. We employ two techniques for adding noise:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Randomly masking tokens Similar to Devlin et al. (2019) and Song et al. (2019), we randomly mask the input sequence tokens at training before feeding it to the encoder. However, instead of only predicting masked tokens, we generate the full denoised sequence. We apply the masking with the following procedure:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
1. Select sequences to mask with probability p s < 1.0, so that some of the sequences are unpermuted as they are during test time.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
2. For selected sequences, replace each token with a <MASK> token with probability p m .
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Permuting order of sentences within paragraphs Even with token-masking we observed a failure mode where the latent representation of a paragraph overly focuses on the first sentence of the paragraph and memorizes it. Indeed, it is the best sentence to memorize for the purpose of reconstructing a paragraph. However, to encourage learning the structure and coherence of paragraphs beyond the first sentence, with probability p perm , we permute the order of sentences in a paragraph and train the auto-encoder to recover the original paragraph.  2). The strategies are applied jointly in the pre-training phase by adding the corresponding losses.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Although we adopt the paradigm of pre-training followed by fine-tuning, there are two significant differences with other work. In past work labeled data are available for the downstream tasks and fine-tuning is supervised; in our work, however, both pre-training and fine-tuning are unsupervised. Additionally, most previous work pre-trained their models on extremely large corpora different from their downstream datasets, whereas our model learns everything from the same dataset 1 .
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Encoder pre-training #1: Corrupted Paragraph Prediction Summarizing a paragraph requires understanding how its sentences follow each other to form a coherent narrative. To encourage good paragraph representations we propose a novel pre-training task classifying whether a paragraph has been corrupted by swapping the order of the i-th sentence with its next sentence. We implemented this by adding a logistic regression layer to the encoder paragraph output z, and minimizing cross-entropy error with 50% of paragraphs corrupted, and 50% unmodified. We refer to this task as Corrupted Paragraph Prediction (CPP).
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Encoder pre-training #2: Next Sentence Prediction We propose another pre-training objective encouraging the encoder to understand how sentences follow within a paragraph. The objective, referred to as Next Sentence or Same Paragraph (NSSP), shares an idea with BERT's Next Sentence Prediction (NSP) objective (Devlin et al., 2019)-to classify whether two sentences are adjacent-but is modified to be more difficult. As in BERT, we sample the sentence pairs A and B such that 50% of the time B follows A, otherwise it does not (negatives). However, negative pairs are sampled from within the same paragraph instead of from the whole corpus. This is more challenging as sentences from the same paragraph usually are more similar and about the same topic. We observed this harder negative sampling leads to better downstream summarization performance.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Further our implementation differs in the following way from BERT's NSP:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
• In BERT, each input sequence is constructed by concatenating A and B separated by a <SEP> token. To better differentiate A and B, BERT further employs segment embeddings. In contrast, we encode A and B independently, avoiding the need for the separator token and segment embeddings.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
• In BERT, a binary multi-layer perceptron is added during pre-training. Instead of introducing these extra parameters we directly take the dot product of the two sentence representations followed by a sigmoid function:
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
σ(φ enc (A) · φ enc (B)), A, B ∈ T. 1 We tried pre-training on large corpora, but it didn't help.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
This also restricts the capacity of the classifier, forcing all relevant features into z. This implementation is simpler, and more generic since it is not tied to a specific encoder (e.g., BERT/Transformer) architecture.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In contrast to CPP, which learns to encode the sentence relationships in paragraph representations, NSSP is taught to encode sentence relationships in individual sentence representations.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Decoder pre-training: Auto-regressive Language Modeling We pre-train the decoder with a standard auto-regressive language modeling (LM) objective similar to Ramachandran et al. (2017). We implement this by setting z = 0 in Equation 1 regardless of the input sequences in pre-training, so that the decoder receives no conditioning signal during teacher-forcing, making the process equivalent to a standard LM task.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We found human summaries, which are quite abstractive, to be unfairly punished on ROUGE-2 and ROUGE-X precision variants; and thus only report ROUGE-1 and -L recall scores (Lin, 2004). We truncated generated summaries to their first sentence and to a word limit of 20 to prevent preferring models with overly long summaries, similar to Rush et al. (2015), although they used a byte limit. The limitations of ROUGE are wellknown, and in particular favor extractive methods, since paraphrasing and synonyms are not rewarded fairly. However, at the time of writing, ROUGE remains a dominant metric in summarization research (Kryściński et al., 2019). We treated the 3 summaries per evaluation example equally and report average ROUGE scores over the 500 test examples, using the 500 validation examples for hyper-parameter tuning. We observed little difference in averaged validation and test scores.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Extractive Because the documents consist of only five sentences, for each example there are only five possible sentence-extractive summaries. Thus we computed the performance of selecting sentence i as the summary, for i ∈ {1, ..., 5}, and denote these as Extract i. We also computed maximum achievable sentence-extractive score, by selecting the best performing sentence per example evaluated based on ROUGE-1 (recall) against the 3 human summaries, which we refer to as Extract Oracle. Although this method cheats by looking at the test data and may not actually be achievable in practice, it is useful to have an estimate of the ceiling of extractive methods.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Human We estimated human performance by computing (a) the maximum and (b) the average ROUGE between all pairs of human summaries for each evaluation example, and took the mean across examples.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
MeanSum-single Although designed for multidocument summarization, we adapted Mean-Sum (Chu and Liu, 2019) for the single-document case by treating each sentence as a document similar to Isonuma et al. (2019), where it is called MeanSum-single. Since MeanSum generates summaries of the same shape as the input documents, this adaptation generates sentencesummaries.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Text Tokenization We used a data-driven subword tokenizer (Sennrich et al., 2016) with a vocabulary size of 2 15 to convert text into integer ID tokens, followed by a 128-dim embedding layer. We shared the input and output embedding layers as in Press and Wolf (2017).
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Architecture We experimented with three base encoder-decoder configurations for SummAE: RNN-RNN, TRF-TRF, and TRF-RNN, where TRF stands for Transformer. Both RNN enc and RNN dec were single-layer GRUs (Chung et al., 2014) with h dim = 512, while RNN enc was bidirectional and RNN dec was unidirectional. TRF enc was a stack of 2 Transformer encoder layers with each consisting of a self-attention sublayer with 8 attention heads and a point-wise feedforward network with 512 hidden units. TRF dec had the same hyper-parameter setting as TRF enc but was composed of Transformer decoder layers. z dim was set to 256. All model weights, including embeddings were initialized randomly. Decoding at each time-step was performed greedily using arg-max.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Optimization We performed gradient descent using the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 0.001 and batch size of 64. The models were trained with early-stopping, using maximum validation ROUGE-1 (recall). The  number of pre-training steps was set to 100,000, when pre-training was employed.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
When critic/discriminator was used, it was implemented as a multi-layer perceptron with 128 hidden units. When token mask was added, we set p s = 0.15 and p m = 0.8. For permuting sentences in paragraphs, we set p perm = 0.5. The critic and noise were only added during the finetuning phase.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We experimented with different hyperparameters for constructing SummAE but found the reported setting worked the best empirically. Table 1 shows ROUGE scores and summary lengths for human and extractive baselines, and for SummAE with enhancements described in Section 4. For the base encoder-decoder configurations (RNN-RNN, TRF-TRF, and TRF-RNN), token masking and paragraph shuffling were added as noise, but no pre-training was done and no critic was added.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
The best extractive sentences were unsurprisingly the first and last, as the sentences introducing the subject, and revealing the ending. Extractive Oracle is by definition the best and its performance was considerably higher than any fixed-index sentence to extract. Estimated human performance was much higher than even the Extractive Oracle, suggesting an abstractive gap.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Critic effectively restricts summary-length All three base encoder-decoder configurations tended to generate overly long/invalid summaries as shown by the number of words (close to 40) and sentences (exceeding 1), indicating the β in Equation 1 was likely ignored. Once the critic was added, the model was able to generate singlesentence short summaries.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
To validate our hypothesis that segregation in the latent space was the underlying problem causing long summaries, we visualized the latent space of SummAE RNN-RNN (similar plots for TRF-TRF and TRF-RNN) in 2-D using t-SNE (Maaten and Hinton, 2008), with and without the critic. In Figure 3a where there was no critic, sentence and paragraph representations indeed were mapped to completely separate regions; while in Figure 3b the adversarial feedback from the critic effectively merged the two clusters, supporting our hypothesis.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Effect of LM pre-training As can be seen from the table, LM pre-training always improved the model performance while keeping the summaries short: for all of three encoder-decoder configurations, it boosted their ROUGE-1 scores from 19.8 to 26.4, 22.9 to 25.8, and 23.2 to 26.4, respec-tively. Qualitatively, we found models with LM pre-training generated more fluent summaries as well.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Best encoder-decoder configuration Interestingly, we observed that TRF-RNN outperformed the other two configurations that use the same sequential architecture for both encoder and decoder. Similar results were reported in Chen et al. (2018), where they found their hybrid model composed of a Transformer encoder and RNN decoder worked the best in machine translation. Another possible reason is that the decoder here does not need to attend to long-term dependencies, which is where Transformers have major advantages over RNNs.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Surprisingly we found that the TRF-RNN variant enhanced with LM pre-training did not require a critic to generate one-sentence summaries Table 1: ROUGE recall scores and generated summary lengths (number of words and sentences) for estimated human performance, extractive baselines, MeanSum-single, and variants of SummAE. By default, all SummAE models incorporate token masking and paragraph shuffling without critic and any pre-training. TRF stands for Transformer. Numbers in bold denote the best performed models in each category based on ROUGE; numbers in italics denote the models that are not qualified for generating one sentence summaries. Superscript letters, a,b,c,d , denote model pairs that are compared in human evaluations in Table 2. and prevented segregation as the other encoderdecoder configurations did. Figure 3c, visualizes its corresponding latent space where the paragraph and sentence representations were mapped to the same region, suggesting that LM pre-training has the same effect of critic in this specific configuration. It also outperformed the model with both critic and LM pre-training (33.4 vs. 26.4).
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Effect of token masking and paragraph shuffling By default both token masking and paragraph shuffling introduced in Section 4.2 were added as noise to base SummAE models during auto-encoding. Removing either one degraded the performance: ROUGE-1 score decreased from 33.4 to 30.9 when token masking was removed and to 31.8 when paragraph shuffling was removed. During experimentation, we also found adding this noise produced more stable results across different runs. Our best SummAE model-constructed by a Transformer encoder and an RNN decoder-was pre-trained with NSSP and LM objectives, followed by denoising auto-encoding of masked and shuffled input sequences. It achieved (ROUGE-1, ROUGE-L) = (36.5, 29.3) that significantly outperformed the fixed-index extractive sentences and was comparable with Extract Oracle, which looks at the test data.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Human evaluation of model summaries To validate that making progress on the ROUGE metrics correlates to making real progress as judged by humans, we conducted several side-by-side model comparisons covering the range of ROUGE scores from low to high-end using Amazon Mechanical Turk workers. Workers were presented with the 5-sentence story/paragraph along with two model summaries and asked to rate each on two dimensions: fluency and information relevance. To minimize inter-rater noise, scores by 3  Table 2.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We observed that the fluency improved significantly from the MeanSum-single model through the RNN-RNN+critic+LM models, while the information aspect continued to improve through our best model, the TRF-RNN with NSSP and language model pre-training. The human performance was still far better on both dimensions when compared side-by-side. Additional model samples can be viewed in Appendix Figures 6 and 7.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
In addition to decoding a sentence from a paragraph representation, we found it informative to look at reconstructed paragraphs from the autoencoder which are also included in Figure 7. The paragraph reconstructions show some coherence, although with some disfluencies and factual inaccuracies that are common with neural generative models. Since the summaries are decoded from the same latent vector as the reconstructions, improving them could lead to more accurate summaries.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We introduce, ROCSumm, a new benchmark task for zero-shot, unsupervised abstractive summarization (UAS) that is useful for iterating and measuring progress on this challenging problem. As one of the very first works approaching single-document UAS, we propose a novel neural model-SummAE-based on a denoising auto-encoder along with several self-supervised pre-training techniques for enhancing the model. While performance is still far behind humans, SummAE outperforms extractive baselines, and is a major step toward UAS.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
The code to reproduce our experimental setup is provided at https://github.com/ google-research/google-research/ tree/master/summae.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
We include the code to process the ROCStories dataset into ROCSumm, our train/test splits, the 3000 human summaries used in validation and test evaluation, Amazon Mechanical Turk templates used for data collection and evaluation. We also include model training and evaluation code, and the model weights for our best model.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Input : A collection of paragraphs, P for p in P do step ← step +1 if step < N pretrain then Update (θ E , θ G ) by gradient descent on rec (p, θ E , θ G ) (Eq.2) else if step mod2 = 0 then Update θ D by gradient descent on D (p, θ D ) (Eq.3) else
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
Update (θ E , θ G ) by gradient descent ae (p, θ E , θ G ) (Eq.2) end end Output: Auto-encoder parameters: (θ E , θ G )  
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Summarization refers to the task of condensing a document into a shorter version without losing the key information. Summarization models can be categorized into two types: abstractive and extractive. Extractive models select sentences from the input article as the summary. Such process ensures a basic level of grammaticality and accuracy, but also limits the model ability to copying. In contrast, abstractive models summarize a document using newly generated tokens and phrases that may not be found in the original article, which involves ⇤ Equal contribution. Work was done during first author's internship at Microsoft. a process requiring an advanced ability to refine, paraphrase and re-organize language information (See et al., 2017;Narayan et al., 2018;Gunel et al., 2020).
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Like most machine learning algorithms, summarization models can also be divided into supervised and unsupervised categories. Supervised approaches require in-domain parallel data, i.e. both input articles and corresponding reference summaries must be present for the teacher-forcing training (Hermann et al., 2015;Liu and Lapata, 2019). Unfortunately, high-quality paired data are not always available across different text domains and styles. Moreover, considering the fact that summarization is not an easy task even for people, reliable human-labeled data are also difficult to obtain. Therefore, several unsupervised summarization approaches have been proposed, which do not require reference summaries for the target domain. We introduce these methods as follows.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Unsupervised extractive models. TextRank (Mihalcea and Tarau, 2004) encodes sentences in the article as nodes in an undirected graph. The weights of edges are measured by sentences similarity. The centrality of a node (sentence) is computed by PageRank (Brin and Page, 1998) to decide whether a sentence should be included in the final summary. Zheng and Lapata (2019) advances upon TextRank by encoding sentences with BERT representation (Devlin et al., 2018) to compute pairs similarity and build graphs with directed edges decided by the relative positions of sentences.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Unsupervised abstractive models. Baziotis et al. (2019) leverages differentiable sampling and optimizes by re-constructing the input article from the generated summary. Chu and Liu (2018) proposes a similar idea in the multi-document summarization setting. Wang and Lee (2018) uses adversarial training and reinforcement learning to make the summary human-readable. Févry and Phang (2018) adopts denoising autoencoders originally used in sentence compression. However, most of these models are only tested on datasets with considerably small article/summary length. Also, previous models usually utilize the recurrent neural networks (RNNs). However, transformers (Vaswani et al., 2017;Devlin et al., 2018) have shown superior performances over RNNs on various NLP tasks, including machine translation, reading comprehension, sentiment analysis, etc. Few Efforts have been made to leverage transformers in unsupervised abstractive summarizations.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Pretraining Language Model. In recent years, pretraining language models have proved to be quite powerful in solving numerous NLP tasks. The state-of-the-art pretrained models include CoVe (McCann et al., 2017), ELMo (Peters et al., 2018, GPT (Radford et al., 2018), BERT (Devlin et al., 2018) and UniLM (Dong et al., 2019). Taking advantage of corpora with billions of tokens, the pretrained language models learn universal and robust representations for various semantic structures and linguistic relationships. As a result, pretrained models have been widely used with considerable success in applications such as question answering (Zhu et al., 2018), sentiment analysis (Peters et al., 2018) and passage reranking (Nogueira and Cho, 2019). Furthermore, UniLM (Dong et al., 2019) leverages its sequence-to-sequence capability for abstractive summarization; the BERT model has been employed as an encoder in BERTSUM (Liu and Lapata, 2019) for supervised extractive and abstractive summarization.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In this paper, we present TED, a pretrained unsupervised abstractive summarization model which is finetuned with theme modeling and denoising on in-domain data. TED utilizes a transformer-based encoder-decoder structure and the pretraining leverages large-scale corpora containing millions of unlabeled articles. Our primary contributions are twofold as follows.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
First, we leverage the lead bias in news articles to pretrain TED. The lead bias is introduced by the journalistic convention of writing using an inverted pyramid structure, placing the most important information in the beginning of an article. We propose to use the leading sentences as the target summary and train the model to predict it during pretraining. In this way, we pretrain a summarization model on a large-scale corpus with 21.4M news articles. The model yields better performance than most existing unsupervised methods.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Second, to finetune on specific datasets, TED is further trained with a theme modeling loss and a denoising autoencoder. The role of the theme modeling module is to make the generated summary semantically close to the article. The module uses a semantic classifier trained using a discriminative objective function. Furthermore, to optimize on the generated summary tokens, we adopt the Gumbel-Softmax (Jang et al., 2016) estimator to replace the non-differentiable arg max. The denoising autoencoder has been previously used in unsupervised machine translation (Lample et al., 2017) and sentence compression (Févry and Phang, 2018), and we employ it to help the model extract salient information from corrupted text.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Instead of classical word tokenization, we adopt the SentencePiece tokenization (Kudo and Richardson, 2018) to alleviates the long-standing out-ofvocabulary (OOV) problem in language generation tasks (Luong et al., 2014;Sennrich et al., 2015). We test TED on several benchmark datasets. The experimental results show that TED outperforms all unsupervised abstractive baselines on all datasets. For example, on the CNN/DM dataset, it outperforms the state-of-the-art unsupervised abstractive model by more than 9 ROUGE-1 points and compares favorably with most unsupervised extractive models. We further show that TED is capable of generating novel words and phrases in summaries and is a highly abstractive system even compared with supervised systems.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In this section, we will go through the model structure of TED, i.e. the transformer encoder and decoder. Then we introduce the pretraining method and two in-domain finetuning objectives: theme modelling and the denoising autoencoder. The overall architecture of TED is illustrated in Fig. 1.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Previous unsupervised summarization methods are based on the sequence to sequence (seq2seq) model  that primarily uses the RNN model. As the transformer structure (Vaswani et al., 2017) has been successfully applied in a large number of NLP tasks, TED employs the multi-layer transformer encoder-decoder architecture. We follow the standard transformer design in TED networks and refer readers to Vaswani et al. (2017)   Denote the input article tokens sequence as
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
., x n }, and each token is first transferred to a vector by a trainable embeddings matrix V . The output from transformer encoder
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
The decoder can be viewed as a conditional language model to generate the summary depending on the generator outputs. Given k input summary tokens W = {w 1 , w 2 , ..., w k }, the cross attention layer in the decoder D attends with encoder outputs
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
The probability distribution over the vocabulary for w k+1 is given by:
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In traditional tokenization algorithms, efforts have been made to address the out-of-vocabulary (OOV) issue  at the cost of losing semantic information, such as mapping OOV words to a special "UNK" token. To mitigate the open vocabulary problem, we adopt Sentence-Piece (Kudo and Richardson, 2018), a data-driven method that trains tokenization models from sentences in large-scale corpora. The advantage of the SentencePiece model is that its subwords can cover all possible word forms and the subword vocabulary size is controllable. In the evaluation experiments, we train a SentencePiece subword vocabulary of size 32,000.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Note for supervised summarization models, during training, the inputs to the decoder are the groundtruths/reference summary tokens; for unsupervised learning, input tokens are generated in the previous pass, i.e. one new token is generated in one pass. More details are available in section 2.3.1.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Leveraging large scale unlabeled text corpora to pretrain models has been proven as an effective method in multiple NLP tasks (Devlin et al., 2018). However, such approach has not yet been utilized in text summarization.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
News articles follow an inverted pyramid structure, i.e. front loading the most salient information. This so-called "lead bias" for news summarization is so strong that See et al. (2017) have shown that using the first 3 sentences in a news article as a summary can score higher than many sophisticated deep learning models. Although this poses a great challenge to previous research, we take advantage of this property in our favor in the pretraining phase of TED.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
For a news article, we set the target summary to be the first three sentences. This allows the model to exploit the structural bias of the news domain and infer the most important information using the background materials in the remainder of the article. To collect data for pretraining, we obtain three years of online news articles from 2016 to 2019 via an industrial search engine. The search engine indexes major online news domain, for instance, New York Times and Bloomberg. Then we collect the parsed articles within the 2016-2019 time range as the raw data. Note that this time span does not overlap any of three test datasets we use in this paper, therefore the pretraining should not lead to data leakage in test. It is also worth noting that this idea of utilizing structural bias for largescale summarization pretraining is not limited to  specific types of models, and it can be applied to other types of text as well: academic papers with abstracts, novels with editor's notes, books with tables of contents.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
However, one should carefully examine and clean the source data to take advantage of lead bias, as the top three sentences may not always form a good summary. Therefore, we conduct strict data cleaning to remove irrelevant distracting content and filter out articles whose top three sentences do not form a good summary:
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
First, many news articles begin with media names, reporter names, dates or other irrelevant information for summarization, e.g. "New York (CNN) -", "Adam Smith, June 3rd 2018:". We automatically clean these using regular expressions.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Second, we only include articles whose top three sentences contain between 10 and 150 words, and remaining sentences contain between 150 and 1,200 words. The criterion on top three sentences is set to filter out articles with either extremely short leading sentences, e.g. phrases of one or two words, which contain too little information to be reasonable summaries, or exceedingly long leading sentences to reduce the pretraining time. The limit on total number of words in the article is to filter out very long articles to reduce memory consumption. Another purpose is to remove very short articles of which the information is too condensed and not suitable for summarization pretraining.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Third, we also remove articles in which the first three sentences may not contain the major information in the article. We use a simple and easy-tocompute metric: overlapping words. We compute the portion of non-stopping words in the top three sentences that also appear in the rest of an article. A higher ratio indicates that the rest of the article is likely to elaborate on the beginning part. We keep those articles with the ratio of overlapping words higher than 0.65. We pick this threshold based on observations in the CNN/DM dataset, where the median overlapping ratio of non-stopping words between golden summary and the article is 0.87, and the median ratio between the top three sentences and the rest of the article is 0.77. Setting the threshold at 0.65 makes the final training set size fit with the available computation resources and ensures that the leading sentences contain enough information.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Finally, we end up with 21.4M articles, out of which 12,000 articles are randomly sampled as the validation set. We conduct pretraining for 10 epochs and pick the model with the best ROUGE-L score on the validation set. The pretraining task is to predict to the first three sentences of an article using the rest of the article (so pretraining will not teach the model to simply copy the leading three sentences since they are removed from the input to the transformers). Note that TED does not start off from other pretrained models like Bert.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
After pretraining, in order to adapt TED to a specific target dataset (for evaluation), we finetune TED on the target dataset in an unsupervised manner. The finetuning objective functions includes the following: theme modeling and denoising autoencoder.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Theme modeling aims to make the generated summary semantically close to the input article. We employ differential sampling to enable optimization on generated summaries and train a classifier to improve the semantic relatedness between the output summary and article.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In order to optimize the transformers using output summaries, we need to make the generation of summary tokens differentiable. Recall the conditional probability distribution of token w k+1 is P (w k+1 |w 1:k , x 1:n ) = softmax(V u D k ). Let ⇡ denote P (w k+1 |w 1:k , x 1:n ). One can use arg max on ⇡ to obtain the token w k+1 in the forward pass, however, it is not differentiable in the gradient back-propagation. Although arg max can be avoided by obtaining the embedding of w k+1 as a weighted sum of the vocabulary embeddings V , this results in an undesirable gap between the training (weighted sum) and the inference (discrete sampling) on the forward pass generation. To solve this issue, we employ the straight-through Gumbel-Softmax estimator (Jang et al., 2016) as in Yang et al. (2018); Baziotis et al. (2019). Specifically, the forward pass in training still uses arg max sampling, but for gradient computation, the following Gumbel-Softmax distribution is used as a differentiable approximation for the arg max operation:
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
where g 1 , · · · , g k are i.i.d samples drawn from the Gumbel distribution G(0, 1) and ⌧ denotes the softmax temperature. As shown in Jang et al. (2016), as ⌧ ! 0, the Gumbel-Softmax distribution converges to the categorical (one-hot) distribution; as ⌧ ! inf, the Gumbel-Softmax distribution converges to the uniform distribution. Although this gradient estimator is biased, we find that this method works well in practice. We choose ⌧ = 0.1 based on the CNN/DM validation set and use this value in all the experiments. Denote the input article as d, the generated summary as s = {w 1 , w 2 , ..., w m }. The generation of s follows the recursive process that input w 1:k to the transformer decoder to obtain w k+1 , then input w 1:k+1 to compute w k+2 and so on. The first input token w 1 is always the special beginning token [START].
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Classifier As the generated summary may be off the article theme at the beginning of finetuning, we also optimize TED such that the generated summaries are semantically closed to the input articles. We frame the semantic similarity problem in a discriminative setting. To better adapt to the target-domain data, we add sentence pairs from training articles to facilitate similarity computation.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Concretely, during training, we pick two consecutive sequences of tokens a 1 and a 2 from an article to form a positive sequence pair {a 1 , a 2 }. Second, sequence b 1 is chosen from another random article in the dataset to form the negative sequence pair {a 1 , b 1 }. Following Devlin et al. (2018), each sequence pair is packed into one single sequence by inserting a special token [SEP] between them and adding trainable segment embeddings. A special classification token [CLS] is also added to the beginning of the packed sequence. As shown  [CLS] Figure 3: Theme modeling is essentially updating TED with a semantic classifier. The input sentence pair is first processed by adding a "class" token in the beginning and a "separation" token between the two sentences. Then the sentence pair is fed into the transformer encoder, and the first output vector is classified to "similar" or "distinct". in Fig. 3, the packed sequence is then fed as input into TED's transformer encoder. The output vector associated with the token [CLS], is then classified into similar/distinct categories by a two-layer fully connected network. We use the following crossentropy loss to optimize the encoder such that the a 1 is semantically similar to a 2 and s is also closed to d, while a 1 is semantically distinct from b 1 .
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
L theme = log(p(y = 1|a 1 , a 2 )) log(p(y = 1|s, d)) log(p(y = 0|a 1 , b 1 ))
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
(3)
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
The idea of denoising autoencoder (Vincent et al., 2008) has been used in unsupervised machine translation (Artetxe et al., 2017;Lample et al., 2017) to prevent the model from learning to merely copy every input word one by one. This denoising process imitates text simplification and helps to refine essential semantic information.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In detail, a sequence of n consecutive tokens x from the input article is injected with two types of noise. First, we insert noisy tokens sampled from other articles in the same dataset into the original sequence at random positions, obtaining a new sequence with length n 0 , where n 0 is 40%-50% larger than n. Next, similar to Lample et al. (2017), the sequence is slightly shuffled by applying a permutation such that 8i 2 [1, 2,· · · , n 0 ], | (i) i|  k, where the permutation distance k is set to be 20% of the length of x. The final corrupted sequence is denoted as x 0 . TED model is then trained to recover the original token sequence given the corrupted sequence:
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
where CE denotes the mean of token-level crossentropy loss. TED(x 0 ) denotes the sequence of probability distribution outputs {⇡} from the decoder with inputting x 0 to the encoder. The final objective function is the mean of Eq. (3) and Eq. (4) (we empirically find that equal weights between the two terms work well enough in practice):
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
It is worth pointing out that we do not conduct "pretraining" on target evaluation datasets. This is because for a target dataset, we do not know beforehand whether the Lead-X sentences will make a quality summary or not. We do have the option to do so on datasets where Lead-X are good summaries, however, it is potentially cherry-picking datasets. Also, we do not conduct supervised finetuning with ground-truths summaries in evaluation datasets because we want to have an entirely unsupervised summarization system with motivations stated in the introduction section.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
We evaluate our model on three benchmark summarization datasets: NYT, CNN/DM and English Gigaword, containing 110K, 300K and 3.8M news articles, respectively. The detailed statistic information on the datasets can be found in the appendix. In NYT, following Liu and Lapata (2019) (2019), input articles are truncated to 500 tokens. In English Gigaword, we filter out data examples with articles containing only"UNK" tokens.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
We compare TED with the following baselines. (2) Unsupervised extractive systems: TextRank (Mihalcea and Tarau, 2004), Lead-X. (3) Supervised abstractive and abstractive (models trained with groundtruths summaries): PACSUM (Zheng and Lapata, 2019), PGNet (See et al., 2017), REFRESH (Narayan et al., 2018) and SUMO (Liu et al., 2019b). TED is unsupervised abstractive and therefore not directly comparable with supervised baselines. The purpose of supervised systems here is for references. We describe the implementation details of our model in Appendix. We measure the quality of generated summaries by ROUGE F1 score (Lin, 2004), including unigram (ROUGE-1), bigram (ROUGE-2) and longest common subsequence (ROUGE-L).
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Results on English Gigaword dataset are shown in Table 2, TED outperforms all unsupervised baselines. Table 2 shows the experimental results on NYT and CNN/DM datasets. In NYT, the unsupervised fine-tuning of TED improves upon the pretrained model by 2.75%/1.06%/2.37% on ROUGE-1/ROUGE-2/ROUGE-L respectively. Note that ROUGE metric prefers extractive systems that preserve original phrasing (See et al., 2017). Considering this factor, TED achieves results that are competitive with unsupervised extractive baselines and surpasses all unsupervised abstractive models. In CNN/DM, TED with a larger model size (10L8H) outperforms all unsupervised abstractive methods and compares favorably with unsupervised extractive baselines. Note that TED outperforms GPT-2, a powerful transformer-based language generation model pretrained on large scale webpage textual data, by significant margins. Again, TED further improves upon pretrained models on both 10L8H and 4L4H configurations.  Article: after exposing potential security risks with airlines' in-flight entertainment systems, one of the top experts on counter-threat intelligence in the world was pulled off a flight by fbi agents. chris roberts, who featured in a string of fox news reports, was yanked off his plane after it landed in syracuse, new york, on wednesday night by two fbi agents and two uniformed officers. roberts, who works for security intelligence company one world labs, was questioned for the next four hours ...
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
chris roberts, who works for security intelligence company one world labs, was pulled off a plane in syracuse, new york, on wednesday night by two fbi agents and two uniformed officers. the incident occurred only a few hours after a report about roberts' research was released by the government accountability office earlier this week.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Reference: chris roberts of one world labs grabbed after plane landed in syracuse. two fbi agents spent four hours questioning him about cyberhacking. agents confiscated electronic devices and computer files from roberts. he flew in to give talk at aerospace conference about plane vulnerabilities. roberts featured on fox news' on the record with greta van susteren. regarded as one of the world's top experts on counter-threat intelligence." Figure 4: An example of a generated summary by TED. The reference summary and parts of the input article are also included.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
The ablation studies shown in Table 3 verify the effectiveness of each component in TED. Training the transformer encoder-decoder from scratch yields reasonable performance. Pretraining on large-scale data results in more than 10% improvement on all three metrics on training TED from scratch. Pretraining plus either theme modeling or denoising improves upon the pretrained model by more than 2%. The full TED model, pretraining with theme modeling and denoising, produces the best result overall.  
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Example. We showcase a sample summary from CNN/DM dataset along with the input article and the reference summary (Fig. 4). As shown, TED is able to capture and organize the essential information into fluent and highly readable language. We attribute the grammatical correctness to the pretraining process and the denoising autoencoder. However, we also note that although TED manages to recognize the temporal information related to reported event (a few hours after Fox news reports), it makes a mistake by summarizing as "a few hours after a report about roberts' research was released. . . ". It shows that fact cross-checking is a potential future research direction.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Abstractiveness. To examine how abstractive TED is, we compute the proportion of novel Ngrams in the summary output (Fig. 5). The reference summary and the output from PGNet are included for comparison. Although TED is unsupervised, it includes more novel grams than the supervised model PGNet. The reference summaries have the highest proportion of n-grams.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
TED is an innovative unsupervised summarization model with several distinctive features setting it apart from previous approaches such as MeanSum and SEQ 3 . First, TED leverages the structure of news articles for an effective large-scale pretraining. Second, although both MeanSum and SEQ 3 have a loss to make the summary similar to the input article, they leverage the classical cosine similarity on text embeddings. In contrast, TED innovatively encodes the similarity by a transformer encoder with much more modeling capability. Third, the denoising module in TED is completely distinct from the idea of reconstruction in SEQ 3 and Mean-Sum. In TED's denoising module, the corrupted texts are input to the transformer and the model is trained to filter the added noises. The original clean document is not used as input and thus unseen by TED in the forward pass. However, the reconstruction process in MeanSum and SEQ 3 employs the original document to generate a summary, which is then used to reconstruct the original document.
TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
In this paper, we propose TED, an unsupervised abstractive summarization model. First, we introduce an effective large-scale pretraining approach leveraging the lead bias in news articles. The pretraining employs automatic filtering mechanism and does require any human-labeled data. We then develop a finetuning scheme to induce the semantic similarity between summaries and input articles, together with a denoising autoencoder to improve the quality of generated summaries. Experiments across three datasets show that TED significantly outperforms unsupervised abstractive baselines.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summarization, or the task of condensing a document's main points into a shorter document, is important for many text domains, such as headlines for news and abstracts for research papers.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
This paper presents a novel unsupervised abstractive summarization method that generates summaries directly from source documents, without the aid of example summaries. This approach simultaneously optimizes for the following important properties of a good summary:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
• coverage of the keywords of the document,
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
• fluency of generated language, and • brevity of generated summaries. * Author emails: {phillab,canny,hearst}@berkeley.edu, ahsil@bloomberg.net Original Document: Chilean President announced Wednesday that his country, which has been paralyzed by protests over the last two weeks, will no longer host two major international summits. [...] The President has now canceled the hosting of the economic APEC forum and COP25 environmental summit, which were both due to take place later this year. [...] Masked Document:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
announced Wednesday that his country, which has been by over the last two weeks, will no longer two major international . [...] The has now the of the and , which were both due to take place later this . [...] Summary Loop [10 word constraint]: Pinera cancelled the APEC summit at Santiago. Coverage Score: 0.22
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summary Loop [24 word constraint]: Pinera said Chileans have been canceled the hosting of the APEC summit, which was scheduled to take place in November. Coverage score: 0.33
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summary Loop [45 word constraint]: Sebastian Pinera announced Wednesday that his country will not hold the APEC summit, which was scheduled to take place in Santiago. Pinera said that Chileans had been paralyzed by protests over the last two weeks.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Coverage score: 0.39 Figure 1: Motivating example. A document from CNN.com (keywords generated by masking procedure are bolded), the masked version of the article, and generated summaries by three Summary Loop models under different length constraints.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
One of the main contributions of this work is a novel method of inducing good coverage of important concepts from the original article. The coverage model we propose takes as input the original document with keywords masked out (see Figure 1). It uses the current best automatically generated summary to try to uncover the missing keywords. The more informative the current summary is, the more successful the coverage model is at guessing the blanked out keywords from the original document. A resulting coverage score is fed back into the training process of the summarization model with the objective of producing summaries with high coverage.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
A second contribution is our unsupervised training procedure for summarization, the Summary Loop, which leverages the coverage model as well as a simple fluency model to generate and score summaries. During training, the procedure is conditioned on a desired summary length, forcing the Summarizer model to adapt to a length budget. Figure 1 shows Summary Loop summaries obtained for the same document under three different length budgets.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
A third contribution is a set of specialized techniques employed during training to guide the model away from pathological behavior. These guard rails include a method for reducing repetition, for encouraging the model to complete sentences, and to avoid frame filling patterns.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The models trained through the Summary Loop outperform all prior unsupervised summarization methods by at least 2 ROUGE-1 points on common news summarization datasets (CNN/DM and Newsroom), and achieve within a few points of state-of-the-art supervised algorithms, without ever being exposed to any summaries. In addition, summaries generated by our method use 50% more summarization techniques (compression, merging, etc.) than prior automatic work and achieve higher levels of abstraction, reducing by almost half the gap between human-generated summaries and automatic summaries in terms of length of copied spans.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Abstractive Summarization. Sequence-to-sequence (seq2seq) (Sutskever et al., 2014) models trained using teacher-forcing are the most common approach to abstractive summarization (Nallapati et al., 2016). A common architecture is the Pointer-Generator (See et al., 2017). Performance can further be improved by constraining the attention (Gehrmann et al., 2018;Gui et al., 2019; and using pretrained Transformer-based language models (Lewis et al., 2019;Chi et al., 2019;Edunov et al., 2019). Through architectural changes, the training procedure remains constant: using a large corpus of document-summary pairs, the model is trained to reproduce target summaries.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Unsupervised Summarization. Most unsupervised summarization work is extractive: sentences deemed relevant are pulled out of the original document and stitched into a summary, based on a heuristic for a sentence's relevance (Mihalcea and Tarau, 2004;Barrios et al., 2015;West et al., 2019). Nikolov and Hahnloser (2019)'s abstractive approach is partially unsupervised, not requiring parallel data, but only a group of documents and a group of summaries. In contrast, our work does not require any summaries, and is trained using only documents. Radford et al. (2019) summarize documents using a language model (GPT2) in a Zeroshot learning setting. The model reads the document followed by a special token "TL/DR", and is tasked with continuing the document with a summary. Our work is an extension of this work: we initialize our Summarizer model with a GPT2 and specialize it with a second unsupervised method.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summarization and Q&A. Eyal et al. (2019) and Arumae and Liu (2018) turn reference summaries into fill-in-the-blank (FIB) questions, either as an evaluation metric or to train an extractive summarization model. In this work, we directly generate FIB questions on the document being summarized, bypassing the need for a reference summary. Scialom et al. (2019)'s work stays closer to a Q&A scenario, and uses a Question Generation module to generate actual questions about the document, answered by a Squad-based (Rajpurkar et al., 2018) model using the generated summary. We refrain from using actual questions because question generation remains a challenge, and it is unclear how many questions should be generated to assess the quality of a summary.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
RL in Summarization. Paulus et al. (2018) introduced Reinforcement Learning (RL) to neural summarization methods by optimizing for ROUGE scores, leading to unreadable summaries. Since then, Reinforcement Learning has been used to select sentences with high ROUGE potential (Chen and Bansal, 2018), or optimize modified versions of ROUGE that account for readability . In all cases, the reward being computed relies on a reference summary, making the methods supervised. We craft a reward that does not require a target summary allowing our training process to remain unsupervised.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
For this work, the definition of a summary is: "A summary is a brief, fluent text that covers the main points of an original document."
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Brevity, fluency and coverage are the three pillars of a good summary. Under a length constraint, a good quality summary should contain as much information about the original document as possible while retaining fluent and coherent English.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Subsection 3.1 lays out the steps in the Summary Loop. Subsections 3.2-3.5 specify how each component is represented by a neural network. Section 4 shows how to train a summarizer model using this architecture in an unsupervised manner. 1
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Numbers in Figure 2 correspond to the following steps:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
1. Summarizer receives a document D and length-constraint L, and produces a summary S fulfilling the length constraint. 2. Using a Masking Procedure, D is modified into a masked document M, where important words have been replaced with blanks. 3. Coverage receives S and M, and uses them to fill in each blank in M with a word, producing F. F and D are compared, and the resulting fill-in accuracy is called the Coverage Score. 4. Fluency receives S, and gives a Fluency Score based on its assessment of the quality of the Summary's writing. 5. The Fluency Score is added to the Coverage Score (as a weighed sum) into a Summary Score for the (D, S) pair. 6. Reinforcement Learning is used to train the Summarizer to produce summaries with high Summary Score. The Summary Loop does not rely on the use of a target/reference/human-written summary, but only the summaries produced by the Summarizer model. The process can therefore be iterated upon without supervision from Summarization datasets.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We use a Generative Transformer (Radford et al., 2019) as the model architecture of the summarizer. We make this choice for two reasons. First, Generative Transformers can produce text one word at a time, allowing the system to produce abstractive 1 The code, model checkpoints and other resources are available at https://github.com/CannyLab/ summary_loop . summaries. Second, we use the pretrained Generative Transformer to initialize the Summarizer.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Practically, the Summarizer first reads through the entire document, followed by a special START token, signaling summarization. The Summarizer produces a probability distribution over words in its vocabulary, and a word is picked from the distribution and fed back as an input into the model. This procedure is repeated and halts either when the summary reaches a length constraint, or when the Summarizer produces a special END token. See Appendix C for the model size and initialization used to train the summarization paper.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The Masking Procedure decides on a set of keywords that are important elements in the document that should be recoverable using a summary. The keywords are replaced with blanks, indirectly indicating which information should be present in the summary. We use a tf-idf-based approach to decide on the set of masked keywords, as it is both simple and has been shown to represent word relevance to a document (Ramos, 2003). Masking procedure implementation details are presented in Section A of the Appendix.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We select the k words with highest tf-idf score for the document to serve as the masked words. The k parameter represents a balance: if too many words are masked, the filling-in becomes impos- sible, but if too few are masked, the Summarizer model will not be encouraged to include sufficient content in its summary. Varying the value of k (10,12,15,20) yielded only small discernible difference in the Summarizers produced, and we use k = 15 in all our final experiments.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The masking procedure can be adapted to a specific domain. For instance, if summarizing financial documents, the masking procedure could systematically mask all numbers, encouraging the Summarizer model to add numbers to its summary.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The Coverage Model receives a computationally generated summary and the masked document and attempts to fill in each blank word. The task of filling in blanks is similar to masked language modeling (MLM), used to pretrain BERT-like (Devlin et al., 2019) models. In MLM, some of the words are replaced with a special M ASK token, and the model must use other information (unmasked words) to fill in the masked words. Because of the similarity to our task, we use a BERT-based neural network as the architecture for the coverage model. However, the coverage task differs from MLM in two ways. First, we modify the masking procedure: instead of masking a random percentage of the words (often 15% for BERT), we mask all appearances of the keywords selected by the masking procedure described in Section 3.3. Second, the input to the coverage model is a concatenation of the unmasked summary, a separator token and the masked document. The model can leverage un-masked information available in the summary to fill in the masked document. The Coverage Model is illustrated in Figure 3.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Using the masking procedure, we obtain M = f (D), the masked document. The coverage model produces the filled document F = g(M, S). Raw coverage score is the fraction of correctly filled in words in F. Let D i , F i and M i correspond to the ith word in their respective document, I M the set indices of words that have been masked. Then:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The model can use information in the unmasked (visible) words of M to predict the masked words. For instance, if the word "Chile" is visible, then "Santiago" would be a well-informed guess near the word "capital", which might not be masked out. This is undesirable, because coverage should account for what information the model can learn from the summary S, not what it can guess from the unmasked portion of D. To counteract this problem, we modify the raw coverage score by computing how much information the model can guess without the summary present, using an empty string summary: F ∅ = g(M, " "). We then normalize a summary's coverage by subtracting the empty string coverage from the raw coverage, leaving only filled-in words answerable using S, as shown in Equation 2.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
In a nutshell, raw coverage score answers the question: "What fraction of blanked words can be correctly filled in with this summary?" and normalized coverage score answers: "What is the increase in the fraction of blanks that can be correctly filled in with this summary, compared to having no summary?" In the rest of this paper, Coverage Score refers to Normalized Coverage Score.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We train the Coverage Model once, and its weights are then fixed during the training of the Summarizer. In order to train the Coverage Model, we need pairs of documents (D) and summaries (S). However, we operate under the assumption that we do not have access to summaries (to keep the procedure unsupervised). In order to remove this dependency, we use the first 50 words of the unmasked 
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We present properties of the raw and normalized coverage through the analysis of existing humanwritten summary datasets. We focus our analysis on three datasets in the news domain: (1) a headline dataset obtained from common US news websites (Laban and Hearst, 2017), (2) the Newsroom dataset (Grusky et al., 2018), and (3) the CNN/DM dataset (Nallapati et al., 2016). For each dataset, we take document/summary pairs and obtain raw and normalized coverage score through our Coverage model, reported in Table 1.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
First, longer summaries obtain higher coverage scores: a CNN/DM summary with an average of 45 words can be used to fill in 73% of the blanks correctly, compared to 48% for a 9 word headline. Across datasets, the correlation between summary length and raw coverage score is 0.56, confirming that longer summaries contain more information, according to coverage.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Second, we simulate the first k words 2 of the document as a summary. We use k = 10, 24, 46 to match average word length in the three datasets. For two of the three values (10 and 46), the coverage of human-written summaries is higher than the first-k word counterpart. This is remarkable: even though the summary is farther away lexically (i.e., 2 We choose the first k words due to the similarity to Lede 3 (first 3 sentences), a common baseline in news.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
is not a subset of the original words), it obtains higher coverage, demonstrating that the coverage model can account for reworded information.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
A model solely trained to optimize coverage has no incentive to write in good English, use punctuation, determinants or pronouns, as these are not words removed by the masking procedure. The objective of a Fluency Model is to judge the writing quality of the summary, independent of its coverage.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Given the right corpus, we argue that a language model's probability can be modified into a Fluency Score. Therefore, we adapt a language model into the Fluency Model.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We choose the generative Transformer (Radford et al., 2019) architecture for our Fluency model, as it can be trained into a powerful language model. Just as with the Summarizer, by using a standardized architecture and model size, we can make use of pretrained models. However, it is important for Fluency to fine tune the language model on the target domain, so that the Summarizer is rewarded for generating text similar to target content.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
To produce a uniform Fluency Score, we linearly scale the language model's log-probability of a given summary (LM (S)) between an ideal value LP low and a maximum value LP high :
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
This ensures that the Fluency(S) is usually in the range [0, 1]. LP low and LP high are picked specifically for a particular language model, and ensure that the log-probability magnitudes of a specific language model do not affect the overall scores.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The final Summary Score is a weighed sum of the Coverage and Fluency Scores:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
α, β are hyperparameters giving relative importance to Coverage and Fluency. We set α = 5, β = 1 in all our experiments. Model choice, size, and initialization are summarized in Figure A1.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We first outline the training procedure and then detail several guard-rail mechanisms used during training to prevent the Summarizer from learning pathological writing strategies. Figure A2 presents training plots of a Summary Loop model and interpretation of the different learning phases.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We use Reinforcement Learning to train the Summarizer component (agent), such that it achieves high summary score (reward). Note that the Coverage and Fluency models are frozen, and their weights are not trained. We make this choice as allowing Fluency and Coverage models to evolve could enable the models to coordinate and cheat. We use the Self-critical sequence training (SCST) method (Rennie et al., 2017), as it has been shown to perform well on similar text generation tasks optimizing BLEU for image captioning or ROUGE scores in summarization.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
In SCST, the Summarizer is used to produce two summaries of document D: a greedy summaryŜ, using a decoding strategy that always picks the most likely next word, and a sampled summary S s , picking the next word in the summary by sampling from the word distribution.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summaries are scored using the Summary Loop:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Then we minimize the following loss:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
log p(w s i |w s 1 , ..., w s i−1 , D)
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Where p(w s i |...) represent the probability of the ith word conditioned on previously generated word, according to the model.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Intuitively, if R s >R, minimizing L maximizes the likelihood of the sampled sequence -which is desired because it outperformed the greedy summary -and increases expected reward of the model.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
During training, the Summarizer model learns pathological summarization strategies. We build training guard rails to detect the pathological behavior and penalize the model during training.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
A guard rail has a binary effect: if a pathology is detected in a summary, its Summary Score is reduced by a penalty amount δ. We use δ = 2 for all experiments. We found three training guard rails to be useful: No-repetition, Finish-your-sentence, and No-frame-filling.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
A common problem in neural text generation is repetition of text. Based on the observation that 3-grams seldom repeat in common summarization datasets, the "No-repetition" training guard rail raises a penalty on a summary when it contains any repeated 3-gram.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
When generating a summary, the model can either produce the END token, or generate a number of words up to the length constraint. We observe that if the model does not produce the END token, it often generates partial sentences, which is undesirable. Because we want to encourage the model to generate an END token, the "Finish-your-sentence" raises a penalty if a summary has no END token.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
During training, the model sometimes learns to overly rely on sentence patterns that achieves high reward as a one size fits all summary. In one example the model learns to produce summaries solely of the form: "X talks with Y about the Z". The model uses this frame, filling in the X, Y and Z slots with relevant keywords and entities to achieve a small but positive coverage. This form of framefilling is undesirable, as the model often produces inaccurate information to fit the entities to the pattern.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We implement a guard rail to penalize the model when frame-filling patterns are observed. During training, we keep track of the last 100 summaries produced by the model. We then aggregate the frequency of words for each word position in the 100 summaries. If any word appears more than 50% of the time at a specific word position, we raise the "No-frame-filling" penalty. In the example given above, the word "talks" appeared in the second word position in more than 50% of the summaries, as well as the word "about" in the fifth position.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
These rule-based training guard rails are simple and effective. In our finalized trained models, very few summaries exhibit penalized behavior: 2% for no-repetition, 5% for finish-your-sentence, and 2.5% for no-frame-filling.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We present results for Summary Loop models trained in the news domain under three different length constraints: 10, 24, and 46 words, matching the distributions of the Headline, Newsroom    (Grusky et al., 2018) and CNN/DM (Nallapati et al., 2016) datasets. We compare our summaries using the standard ROUGE metric, and by analyzing summaries for the errors made, the technique used and the level of abstraction. Finally, we show the Summary Loop can be complemented with supervision, reducing the amount of data needed to achieve comparable ROUGE results. Recent breakthroughs in pretrained Transformer models have shown that using larger models in Summarization can lead to large improvements. For instance, a "large" version of the PEGASUS model (Zhang et al., 2019a) outperforms the "base" version by 2.3 ROUGE-1 points. Because Summary Loop experiments were performed using "base" models, we expect that using larger Transformer models could lead to similar gains. Table 2 confirms that human-written summaries obtain amongst the highest Fluency and Coverage scores. Human-written summaries are only outperformed by Summary Loop summaries, and the Lede-3 baseline. However, the Summary Loop summaries are obtained by directly optimizing for Fluency and Coverage, and Lede-3 baseline summaries achieve their higher Coverage at the expense of being much longer (i.e. 84 words on average compared to 58 in human-written summaries).
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
We perform a manual analysis of 200 randomlyselected summaries on the test set of CNN/DM from the Pointer-Generator with Coverage (PGC), Bottom-Up (BU) and the unsupervised Summary Loop (SL). We annotated each summary with two types of errors: Inaccurate (information in summary contradicts document), Ungrammatical (one sentence or more is not properly constructed), and  The analysis was performed by the first author of the paper, labeling article/summary pairs without knowledge of model origin. A summary can manifest any number of summarization Techniques, or none. Labeling is binary: if a summary exhibits more than one or instances of a Technique, it receives a 1, otherwise it receives a 0. Results of the analysis are summarized in Table 4. SL uses significantly more summarization techniques (425) than PGC (148) and BU (287) summaries. Beyond raw counts, SL is more successful at applying summarization techniques (59% success) than BU (50% success), but less successful than PGC (72%). Note however that PGC takes little risk: 19% of the summaries go beyond sentence compression, and 39% are extractive, using none of the summarization techniques.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
All methods generating summaries one word at a time have potential for abstraction. In Figure 4 we analyze human and system written summaries for abstraction level. We measure a summary's level of abstraction by looking at the length of spans Figure 4: Histogram and average copied span lengths for abstractive summaries. A summary is composed of novel words and word spans of various lengths copied from the document. Summary Loop summaries copy shorter spans than prior automatic systems, but do not reach abstraction levels of human-written summaries.  copied from the document. Summary Loop is the most abstractive automated method, although less so than human written summaries. SL cuts nearly in half the length of copied spans compared to other automated methods.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
If summaries are available, we show that they can complement the unsupervised Summary Loop. We run supervised experiments on CNN/DM using a generative Transformer architecture and varying the initialization. We compare initializing with (1) random weights, (2) the original GPT2 weights, and (3) the Summary Loop weights of target length 45. We train each model with teacher forcing, comparing using the entire CNN/DM training set to just 10% of it. The results are summarized in Table 5.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
First, initializing with the Summary Loop leads to higher ROUGE score both in the 10% and full dataset setting. As expected, results improve when using the entirety of the data, and the Summary Loop initialized model trained with the entirety of CNN/DM obtains a ROUGE-1 F1-score of 41.0, within the confidence interval of the supervised Bottom Up (Gehrmann et al., 2018) architecture. This is a strong result as the Transformer we use is a generic language model, and is not specialized for summarization.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Second, initializing with Summary Loop and training with 10% of CNN/DM yields comparable ROUGE scores to initializing with GPT2 and using the entire CNN/DM, showing that Summary Loop can be useful when fewer summaries are available.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Customizing summaries. In Figure 1, we illustrate the effect of the length constraint by summarizing the same document under three different length constraints. Each model adapts to its word budget. However, length is only one way to customize summaries. One might want to summarize based on point of view, chronology, theme, etc.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Fluency vs. Grammaticality. By choosing to represent the validity of summaries with a Language model, we encourage fluent summaries (i.e., with likely sequences of words) but not necessarily grammatical ones. Extending the scoring to include grammaticality, either by using a parsing model, or leveraging the Corpus of Linguistic Acceptability (Warstadt et al., 2019) could prove useful.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Summarization in the wild. Because our method is unsupervised, it can be applied to new domains and languages. In this work, we benefited from pretrained BERT and GPT2 models in English, which do not yet exist publicly for other languages. Once they become available in other languages, the Summary Loop can be ported over.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Abstraction dangers. Recent work around measuring factuality in generated text, using Natural Language Inference (Guo et al., 2018) or rule-based fact extraction (Zhang et al., 2019b) becomes increasingly important with summaries that are more abstractive. This work can be naturally included into the Summary Loop, with a fact-checker model generating an accuracy score.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
In this work we present a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. When tested on common news summarization datasets, our method significantly outperforms previous unsupervised methods, and gets within the range of competitive supervised methods. Our models attain levels of abstraction closer to human-written summaries, although with more abstraction, more potential for factual inaccuracies arise.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
The masking procedure follows these steps:
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
1. We randomly sample 5,000 documents in the domain being summarized (e.g. News) as a training corpus, 2. The training corpus is tokenized using the tokenizer of the Coverage model. In our case, we tokenize with the Word Piece model of the BERT Base model (Devlin et al., 2019), 3. We train a tf-idf transformation model using the tokenized training corpus using default parameters of scikit-learn's tf-idf implementation (Pedregosa et al., 2011), 4. Given a document to be masked, we use the trained tf-idf model to produce a tf-idf for the document, 5. The words present in the document are ranked in decreasing order of tf-idf score, and the k words with highest tf-idf form the masking set, 6. All occurrences of the words in the masking set are replaced by a mask in the document, creating the masked document.    Figure A2 presents the plots of key variables we obtain during the training of the length 10 Summary Loop model. The training occurred over 10 days using a single Titan X GPU. During a first phase which occurs in the first 2 days of training, the model learns to copy content from the news article, which helps it achieve high Fluency and Coverage. In a second phase starting around the second day, the Summarizer learns to gain Coverage which maintaining Fluency mostly constant, which makes the overall Summary Score rise. The Summarizer model quickly learns to use its word budget, and after 10 days of training, the model uses an average of 9.7 words in its summaries. 
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Document: He has long struggled to convince voters that he is a suitable choice for prime minister. Now Ed Miliband has hired a leadership coaching firm that helps people overcome anxiety and find their "inner voice". The consultants drafted in by the Labour leader claim to work with politicians to build "leadership skills" using "neuroscience" and "business psychology".  
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Document: For most of us, the dream of a holiday home is one that will probably never be realised. But for the lucky minority with a few extra million in the bank, its seems the world is quite literally your oyster when looking for property around the world. From a Lake Garda mansion with a pool overlooking the water to an Italian villa that looks like a castle and an Antigua retreat with Giorgio Armani as a neighbour, these are some of the most spectacular holiday homes on the market at the moment. On the Lombardy side of Lake Garda, this Lionard property is a luxurious villa with one serious waterfront view. Lake Garda. On the Lombardy side of Lake Garda, in northern Italy, lies a luxury villa with a view -just several miles north of Brescia. And for e 18 million ( about £13 million or $20 million ) it can all be yours. Not only is there a large swimming pool looking out on the water, but also a large deck with plenty of space for sun beds, gazebos and al fresco dining spots, overlooking a 4000 square metre garden. Inside, the house is just as breathtaking. For about 18 million Euros ( or $ 13 million ), the modern home, complete with pool, gazebo, and al fresco dining options, can be yours. [...] Summary: The Lake Garda home is a luxury villa with a view on the Lombardy side of Lake Garda. This villa with gazebo and al fresco dining options. Inside, the house is just as breathtaking. For about 18 million Euros. Figure A5: Summary Loop summary from the Error and Technique analysis (Section 5.2) illustrating the Novel Sentence technique. The first sentence of the summary uses pieces from the original document (in boldface blue) to form a sentence with an alternative but correct meaning.
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Document: Sipping a glass of glorious red wine which has been carefully aged in a hand-crafted oak barrel is my idea of heaven. [...] A $ 5 bottle has suddenly become $ 12 because the wine has lingered in an oak barrel before bottling. So when I read this week about a new gadget that claims to be able to "oak age" wine in hours rather than years, my curiosity was seriously roused. The Oak Bottle promises to impart an authentic aged flavour -a process that can take up to two years -in just a day or two. Who wouldn't drink to that ? Scroll down for video. TV wine expert Oz Clarke puts to the test this oak bottle that claims to "oak age" wine in hours rather than years. The product, which retails at $ 50, is the brainchild of 30-year-old entrepreneur Joel Paglione. [...] Summary: Joel Paglione said the Oak Bottle promises to be able to oak age wine in hours rather than years. The Oak Bottle promises an authentic aged flavour that can take up to two years. A bottle has been made in an oak barrel. 
The Summary Loop: Learning to Write Abstractive Summaries Without Examples
Document: The traditional cookie cutter wedding no longer exists -new reports suggest Brits are ditching tradition in favour of alternative practices when it comes to getting hitched. Two of the biggest changes are the fact that religious services have fallen out of favour and that brides are opting for bold colour schemes for their big day. A new study, which has tracked the decisions of brides and grooms over the past five years interviewed 1,893 newlyweds and compared them to answers they have collated since 2010. Scroll down for video. [...] Summary: The new study showed that British couples are opting for religious ceremonies when it comes to their big day with services falling from 40 per cent of the past five years. The study showed that couples are opting to holiday in the UK. Figure A7: Summary Loop summary from the Error and Technique analysis (Section 5.2) illustrating the Inaccurate error. The summary inaccurately claims religious ceremonies are increasing, when the document says they are in decline. Key phrases are highlighted in boldface blue.