How have reranking been applied on abstractive summarization models?
Re-ranking in abstractive summarization
Second-stage or sequence-level methods are gain-
ing traction recently in supervised summarization.
Among such methods, re-ranking consists in
selecting a better summary candidate out of several
of them produced by a base model (which has
already been fine-tuned). RefSum (Liu et al., 2021)
uses a meta-learning approach to learn how to
rank summaries coming from multiple systems.
SimCLS (Liu and Liu, 2021) trains a RoBERTa
(Liu et al., 2019b) model with a ranking loss to
learn how to rank summary candidates generated
by a base BART or PEGASUS in their target metric
order. SummaReranker (Ravaut et al., 2022a) also
trains a RoBERTa re-ranker, but this time in a
multi-label binary classification manner to predict
whether each summary candidate maximizes each
of the metrics of interest. To avoid using another
neural network for re-ranking, BRIO (Liu et al.,
2022b) performs a second fine-tuning stage with
the re-ranking loss built in the base summarization
system. Each of the four models above improves
the SOTA on the CNN/DM benchmark, reaching
47.78 ROUGE-1 for BRIO.
Unsupervised Summarization Re-ranking
With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).
[{"first":"Mathieu","last":"Ravaut","middle":[]},{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Nancy","last":"Chen","middle":[]}]
RefSum: Refactoring Neural Summarization
Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement. Our system can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/Refactoring-Summarization.
[{"first":"Yixin","last":"Liu","middle":[]},{"first":"Zi-Yi","last":"Dou","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]}]
BRIO: Bringing Order to Abstractive Summarization
Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality.
[{"first":"Yixin","last":"Liu","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]},{"first":"Dragomir","last":"Radev","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
In this paper, we present a conceptually simple while empirically powerful framework for abstractive summarization, SimCLS, which can bridge the gap between the learning objective and evaluation metrics resulting from the currently dominated sequence-to-sequence learning framework by formulating text generation as a reference-free evaluation problem (i.e., quality estimation) assisted by contrastive learning. Experimental results show that, with minor modification over existing top-scoring systems, SimCLS can improve the performance of existing top-performing models by a large margin. Particularly, 2.51 absolute improvement against BART and 2.50 over PEGASUS w.r.t ROUGE-1 on the CNN/DailyMail dataset, driving the state-of-the-art performance to a new level. We have open-sourced our codes and results: https://github.com/yixinL7/SimCLS. Results of our proposed models have been deployed into ExplainaBoard platform, which allows researchers to understand our systems in a more fine-grained way.
[{"first":"Yixin","last":"Liu","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]}]
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search to generate a unique summary. However, the search space is very large, and with the exposure bias, such decoding is not optimal. In this paper, we show that it is possible to directly train a second-stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candidate and consistently improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN- DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art. Our code and checkpoints will be available at https://github.com/ntunlp/SummaReranker.
[{"first":"Mathieu","last":"Ravaut","middle":[]},{"first":"Shafiq","last":"Joty","middle":[]},{"first":"Nancy","last":"Chen","middle":[]}]
Unsupervised Summarization Re-ranking
Transformer-based encoder-decoder language models have achieved great success in abstractive summarization in the last few years, and produce fluent summaries which can be quite abstractive (Raffel et al., 2019;Lewis et al., 2020;Zhang et al., 2020). These models follow the pre-train then fine-tune paradigm: they are first pre-trained with a selfsupervised objective on a large text corpus; then they are fine-tuned on the downstream dataset of interest, using the available supervision, which may be very scarce. Finding a better pre-training objective remains an active research area. Some models like T5 (Raffel et al., 2019) and BART (Lewis et al., 2020) adopt a more general language modeling objective (e.g., masked span generation), while   al., 2020) or TED (Yang et al., 2020) are pre-trained specifically for the task of summarizing a document. PEGASUS uses salient sentences of the document as a proxy summary label, while TED leverages the lead bias to get the pseudo-summary target.
Unsupervised Summarization Re-ranking
Despite the impressive success on supervised abstractive summarization tasks, unsupervised summarization remains very challenging. The LEAD-3 (extractive) baseline which simply takes the first three sentences of a document as its summary, remains far ahead of unsupervised approaches on several news summarization datasets (See et al., 2017), especially the popular CNN/DM dataset (Hermann et al., 2015). In fact, it was only improved on by supervised abstractive models not more than five years ago (Narayan et al., 2018). It is expected that a model which has never seen any summarization example would struggle, as summarization is a task that is subjective and complex even for humans (Kryscinski et al., 2019). Since summarization labels are expensive to collect, it is essential to develop models with good zero-shot performance. Starting from instruction-tuned GPT-3, LLMs are offering promising performance in zero-shot summarization (Goyal et al., 2022), but remain an unscalable solution as these models are arXiv: 2212.09593v3 [cs.CL] 
Unsupervised Summarization Re-ranking
rarely open-source, and extremely computationally intensive.
Unsupervised Summarization Re-ranking
Recently, in the supervised setup, second-stage approaches have gathered interest in abstractive summarization research. While the base encoderdecoder model is trained with maximum-likelihood estimation (MLE) to predict each token of the ground-truth summary in an autoregressive manner, second-stage methods work with a global view at the whole sequence level. SimCLS (Liu and Liu, 2021) and SummaReranker (Ravaut et al., 2022a) propose to train another neural model to rank summary candidates generated by decoding methods like beam search (Reddy, 1977) or diverse beam search (Vijayakumar et al., 2016). BRIO (Liu et al., 2022a) bypasses the need for another model, and re-uses the fine-tuned model for another fine-tuning stage in which the model also learns to rank candidates in the correct order. SummaFusion (Ravaut et al., 2022b) encodes each summary candidate separately and decodes into a new, abstractive secondstage summary. Such second-stage methods have improved ROUGE-1 state-of-the-art on CNN/DM by more than 3 points (Liu et al., 2022a).
Unsupervised Summarization Re-ranking
In this paper, we propose to re-rank summary candidates in the unsupervised setup. Following observations made by second-stage summarization studies in the supervised setup (Liu et al., 2021;Ravaut et al., 2022a), we also observe large variance in performance among summary candidates in the unsupervised setup. In Table 1, the oracle for PEGASUS, which is the summary candidate maximizing the ROUGE score with the reference, reaches 42.62 when using beam search with 20 beams on CNN/DM (Hermann et al., 2015). This is in the same range (42-45 ROUGE-1) as the top beam of supervised leading models on this dataset (Lewis et al., 2020;Zhang et al., 2020). This observation implies strong potential motivating our work: with a perfect unsupervised summarization re-ranker, one could potentially by-pass supervised fine-tuning and just re-rank instead.
Unsupervised Summarization Re-ranking
The main challenge lies in the fact that the reranker must also not access any supervision. Our proposed model does not train any neural model, but simply computes features indicative of summary quality to score each summary candidate, some of them which also leverage the source document. A weighted average of these features is used for candidate re-ranking, and we explore several methods to estimate the feature weights. Our method, named SummScore, is lightweight, fast and easy to use as it does not rely on a neural network. Since it is purely unsupervised, the re-ranked results can provide more refined self-supervision to the pre-trained models, complementing the pretraining with rounds of self-training.
Unsupervised Summarization Re-ranking
Our contributions in this paper are threefold:
Unsupervised Summarization Re-ranking
• We propose SummScore, the first system to rerank summarization candidates in an unsupervised setup and in an unsupervised manner.
Unsupervised Summarization Re-ranking
• We demonstrate the strength of SummScore by consistent performance improvement: up to +7.27% with PEGASUS and +6.86% with Chat-GPT 2 mean ROUGE gains over four unsupervised summarization datasets, +7.51% mean ROUGE gains averaged over 30 zero-shot transfer setups.
Unsupervised Summarization Re-ranking
• Using the re-ranker, we derive an original and effective self-training method which continuously improves the base unsupervised summarization model, pushing PEGASUS from 35.47 to 39.76 ROUGE-1 (+12.09%).
Unsupervised Summarization Re-ranking
Second-stage or sequence-level methods are gaining traction recently in supervised summarization. Among such methods, re-ranking consists in selecting a better summary candidate out of several of them produced by a base model (which has already been fine-tuned). RefSum (Liu et al., 2021) uses a meta-learning approach to learn how to rank summaries coming from multiple systems. SimCLS (Liu and Liu, 2021) trains a RoBERTa (Liu et al., 2019b) model with a ranking loss to learn how to rank summary candidates generated by a base BART or PEGASUS in their target metric order. SummaReranker (Ravaut et al., 2022a) also trains a RoBERTa re-ranker, but this time in a multi-label binary classification manner to predict whether each summary candidate maximizes each of the metrics of interest. To avoid using another neural network for re-ranking, BRIO (Liu et al., 2022b) performs a second fine-tuning stage with the re-ranking loss built in the base summarization system. Each of the four models above improves the SOTA on the CNN/DM benchmark, reaching 47.78 ROUGE-1 for BRIO. To the best of our knowledge, there is no work on sequence-level unsupervised abstractive summarization. Concurrently to our work, MBRD (Suzgun et al., 2022) proposes to rank generated candidates in several generation tasks using majority voting based on BERTScore (Zhang et al., 2019).
Unsupervised Summarization Re-ranking
As an unsupervised summarization re-ranking approach, our method assumes access to a zero-shot self-supervised summarization model. We refer to it as the base model M base . Given a source document D, M base will generate k summary candidates using a generation method to transform model predictions into a natural language summary. A widely used such generation approach is beam search, which maintains k top summary candidates throughout decoding, ranking them with decreasing mean log-probability of the sequence. In the end, practitioners keep the candidate maximizing the log-probability and discard the remaining, whereas we propose to keep all k candidates and re-rank them, following (Ravaut et al., 2022a).
Unsupervised Summarization Re-ranking
Let C = {C 1 , . . . , C k } be the pool of candidates. Our goal in (re-)ranking the candidates is to assign to each of them a score S, such that S(C i ) > S(C j ) if C i is a better candidate than C j (for 1 ≤ i, j ≤ k) according to some summary quality measures. We can then select the candidate maximizing the score as the best output:
Unsupervised Summarization Re-ranking
Unlike re-ranking in a supervised setup, where one can compute such scores by comparing with the ground truth summary or build models to optimize them (Liu and Liu, 2021;Ravaut et al., 2022a;Liu et al., 2022a), in our unspervised setup, we cannot assume access to the ground truth, which thus excludes scoring the candidate with regards to it (e.g., using ROUGE). In the following, we describe how we build our unsupervised scoring method (named SummScore) following principles assessing the quality of a summary.
Unsupervised Summarization Re-ranking
We design our candidate-level SummScore as an aggregation of features, each representing desired properties for a summary. Features either come from the comparison between the summary candidate and the source, or from the candidate itself. Fig. 1 synthesizes the overall SummScore re-ranking process.
Unsupervised Summarization Re-ranking
Comparison with the source One evident property of a summary is that it should stick to the source content, and contain as much of the important content as possible. The most straightforward way to measure this consists in using n-gram overlap metrics between the source document and each candidate. We use ROUGE-1 (noted R-1) (Lin, 2004), ROUGE-2 (R-2), and BLEU (Papineni et al., 2002), which form our first set of features:
Unsupervised Summarization Re-ranking
The above metrics only evaluate n-gram overlap, which can be helpful penalizing summary candidates departing too much from the source, potentially hallucinating. However, they have been shown to not be well suited at evaluating semantic similarity, and might encourage too much copying.
Unsupervised Summarization Re-ranking
Thus, our next batch of SummScore features consists in model-based metrics designed to capture semantic similarity between two text items. We explore three such metrics: BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021) and BLEURT (Sellam et al., 2020). BERTScore (noted BS) computes token-level cosine similarity between the contextual embeddings of the pre-trained BERT (Devlin et al., 2019) of each text item to compare. BARTScore (noted BaS) uses BART (Lewis et al., 2020) token-level log-probabilities from the pretrained BART to score the generated text. BLEURT (noted BRT) also leverages BERT but extends its pre-training with an additional multi-task pretraining on synthetic data. Our next features are:
Unsupervised Summarization Re-ranking
When each of these metrics is referred to, it is implicit that they are used to compare a summary candidate with the source document (in contrast to the supervised case, comparing with the target).
Unsupervised Summarization Re-ranking
Summary quality A good summary should be diverse, meaning it should avoid repeated n-grams. We build a summary-level diversity score which measures the proportion of unique n-grams.
Unsupervised Summarization Re-ranking
unique n-grams total n-grams
Unsupervised Summarization Re-ranking
We take N = 3 in practice. The summary should not be too short, nor too long. We penalize summaries which deviate a lot from the average summary length on the given dataset. To build a score with increasing values being desirable, we use a smooth inverse of the absolute length difference between the summary candidate and the mean length of summaries µ len .
Unsupervised Summarization Re-ranking
Final Score Our final set of summary features is:
Unsupervised Summarization Re-ranking
where S quality = {F div , F len }. For data point x i , SummScore simply outputs the summary candidate among the set C i maximizing a weighted combination of all features above:
Unsupervised Summarization Re-ranking
where we enforce coefficients to be |S| j=1 θ j = 1.0
Unsupervised Summarization Re-ranking
SummScore is simply a linear combination of eight features in total. Yet a last crucial question remains: how to estimate the coefficients to assign to each feature? We propose to bootstrap a pseudosummary using sentences from the source document. Coefficients are then tuned to maximize the mean of ROUGE-1/2/L between the summary candidate with the highest SummScore (e.g., Summ-Score output candidate), and the pseudo-target. We compare three approaches to extract pseudotargets:
Unsupervised Summarization Re-ranking
• Random-3: As a baseline, we randomly select three sentences from the source document to form a pseudo-target.
Unsupervised Summarization Re-ranking
• LEAD-3: This consists in the first three sentences of the document. LEAD-3 is a strong baseline for lead-biased news summarization datasets (Hermann et al., 2015;See et al., 2017), and it has even been used as a pseudo-target for summarization pre-training in TED (Yang et al., 2020).
Unsupervised Summarization Re-ranking
• Salient Sentences: We follow the gap-sentences generation idea introduced by PEGASUS pretraining objective (Zhang et al., 2020), and also used by SUPERT (Gao et al., 2020)   top 30% such sentences are extracted to form a pseudo-summary. We experiment with all three standard versions ROUGE-1, ROUGE-2 and ROUGE-L for salient sentences definition, referred to as Salient-R1, Salient-R2 and Salient-RL, respectively.
Unsupervised Summarization Re-ranking
We emphasize that none of these pseudo-targets definition makes any access to human supervision. Training SummScore amounts to estimating the coefficients θ in Eq. (7) using the pseudo-targets:
Unsupervised Summarization Re-ranking
where R is the mean of ROUGE-1, ROUGE-2 and ROUGE-L, C i is the set of candidates predicted by the base model M base for data point x i , andỹ i is the pseudo-target. To optimize coefficients, we hill climb with randomness to maximize R between the SummScore selected summary candidate, and the pseudo-target. Specifically, we estimate coefficients with stochastic local search on the validation set in a hierarchical manner: we first tune coefficients for S overlap and S semantic separately, then estimate coefficients for S quality ∪ {F overlap , F semantic }, where F overlap (resp. F semantic ) is the set S overlap (resp. S semantic ) after reduction to a single feature. Such hierarchical estimation is natural given that S overlap (resp. S semantic ) is made of features capturing similar properties, and dramatically reduces the search space.
Unsupervised Summarization Re-ranking
We experiment on four popular abstractive summarization datasets, from three different domains (see Table 2 for basic statistics on each dataset):
Unsupervised Summarization Re-ranking
• CNN-DailyMail (Hermann et al., 2015;See et al., 2017) is made of 93k and 220k articles from the CNN and DailyMail newspapers, respectively. CNN/DM is the most extractive dataset among all the ones we consider and has the longest source documents.
Unsupervised Summarization Re-ranking
• XSum (Narayan et al., 2018) has 227k articles from the BBC from 2010 to 2017. This is an extreme summarization task, compressing each article into a single, very abstractive sentence.
Unsupervised Summarization Re-ranking
• WikiHow (Koupaee and Wang, 2018) contains 168k lists of short instructions from Wikipedia.
Unsupervised Summarization Re-ranking
• SAMSum (Gliwa et al., 2019) is a dialogue summarization dataset containing 17k conversations. In this dataset, source length is significantly shorter than in the other datasets.
Unsupervised Summarization Re-ranking
To estimate coefficients, we subsample randomly (on datasets other than SAMSum) 1,000 data points from the validation set. To avoid coefficients optimization to overfit, we cap each random search at 1,000 trials. Evaluation of summaries selected by SummScore is done with the standard ROUGE-1/2/L (Lin, 2004) (using summary-level ROUGE-LSUM variant for ROUGE-L) and BERTScore (Zhang et al., 2019). We use transformers (Wolf et al., 2020) and datasets (Lhoest et al., 2021) for pre-trained checkpoints and datasets, respectively.
Unsupervised Summarization Re-ranking
We first apply SummScore to unsupervised abstractive summarization, using as base model (M base ) two models of different capacity: the pretraind PEGASUS (Zhang et al., 2020) (loading the google/pegasus-large checkpoint from transformers), and the recently introduced, highlyperforming ChatGPT 3 , accessed through OpenAI API (calling the gpt-3.5-turbo checkpoint). Due to its pre-training objective of generating gapsentences, PEGASUS can directly be applied to the summarization task after pre-training. This is not the case of comparable sequence-to-sequence Transformer-based models T5 (Raffel et al., 2019) and BART (Lewis et al., 2020), which are pretrained with token spans generation and sequence de-noising, respectively. For ChatGPT, to lower costs, we subsample randomly 1,000 data points from the test set on datasets other than SAMSum.  We decode PEGASUS with beam search, and ChatGPT with top-p sampling with p = 0.9 and temperature 0.8 to enhance diversity, both models with 20 candidates. We report candidate selection baselines from Table 1: top beam or first, and random (a randomly sampled candidate).
Unsupervised Summarization Re-ranking
We show unsupervised summarization results with PEGASUS and ChatGPT with 20 summary candidates in Table 3. SummScore improves the base PEGASUS by 4.37% to 7.27% across the four datasets. Notably, SummScore fails with ChatGPT on XSum, which we hypothesize is due to the nature of XSum and the fact that pseudo-labels from XSum source documents are too different from the ground truth labels, an issue not affecting PEGA-SUS because its performance range is far lower than ChatGPT. However, SummScore improves ChatGPT by 2.74% to 6.86% on the other datasets. We point out that SummScore gains are achieved without using any human supervision.
Unsupervised Summarization Re-ranking
SummScore -LEAD-3 performs best for the news domain, which intuitively makes sense due to the lead bias and first sentences containing an overview of the article. On WikiHow, SummScore -Salient-R2 works the best, yet gains are more moderate and SummScore fails to improve the BERTScore on this dataset. SummScore -Random-3 is tied with SummScore -LEAD-3 on SAMSum: we attribute it to the fact that SAMSum source documents are very short (Table 2), and the LEAD-3, Random-3, and entire source document all overlap a lot. Appendix A confirms that SummScore re-ranking always finds a non-trivial (e.g., longest) candidate selection.
Unsupervised Summarization Re-ranking
Next, we investigate SummScore performance in the transfer setup, with standard-size models (discarding ChatGPT or similar models). We perform zero-shot summarization inference followed by SummScore on a target dataset where the base model M base was fine-tuned on another source dataset. As M base , we use three high-performing summarization models: PEGASUS (Zhang et al., 2020), BART (Lewis et al., 2020), and the recently introduced BRIO (Liu et al., 2022a), which achieves SOTA results on news summarization (CNN/DM & XSum). We use publicly available fine-tuned checkpoints on CNN/DM and XSum, and PEGASUS on WikiHow. We fine-tune ourselves PEGASUS on SAMSum, and BART on Wik-iHow and SAMSum. Generation and fine-tuning hyper-parameters and results are in Appendix B.
Unsupervised Summarization Re-ranking
Given the findings from §4.2, we use Summ-Score -LEAD-3 on CNN/DM, XSum, and SAM-Sum, and SummScore -Salient-R2 on WikiHow. We tune coefficients in the same process described in §4.1. To stick to a no supervision scenario, we do not apply SummScore on a dataset on the which the base model was fine-tuned, which would fall into the supervised learning use case. We compare SummScore zero-shot transfer performance on CNN/DM with that of SOTA WikiTransfer (Fabbri et al., 2021), which fine-tunes BART on external data retrieved from Wikipedia before applying the model in zero-shot summarization.
Unsupervised Summarization Re-ranking
Zero-shot transfer results are displayed in Table 4. SummScore consistently improves transfer performance, with ROUGE gains of 7.51% averaged over 30 setups: +9.43% on CNN/DM, +1.27% on XSum, +9.20% on WikiHow (up to +17.64% average when transferring from XSum) and +9.61% on SAMSum. Notably, on CNN/DM, BART transferred from SAMSum with SummScore improves on the ROUGE-1 and ROUGE-L of SOTA transfer model WikiTransfer (also using a BART backbone), despite WikiTransfer being fine-tuned on data specifically crafted to transfer better to the downstream task. We notice that SummScore helps  more when the base model transfers less well, such as from single-sentence summaries XSum. Appendix C evalutes re-ranking itself and shows that SummScore can also reach strong recall.
Unsupervised Summarization Re-ranking
Using the selected summary candidate as a pseudotarget, one can naturally extend SummScore into a self-training summarization objective. Indeed, if γ parametrizes M base , we can further train M base through the objective:
Unsupervised Summarization Re-ranking
This process can be repeated: if we denote new model weights by γ k , we can re-apply SummScore and perform another round of self-training, yielding new model weights γ k+1 .
Unsupervised Summarization Re-ranking
We notice that the unsupervised PEGASUS beam search summary candidates, including the one selected by SummScore, are quite extractive (see Appendix D). This could be because the selfsupervised gap-sentences are extracts from the source document. To make the pseudo-summaries more abstractive and diverse enough to mitigate the confirmation bias in self-training (Tarvainen and Valpola, 2017), we use the paraphrasing approach proposed in FAR- RW (Zhang et al., 2022). On each dataset, we train a paraphrase model to generate the top n sentences maximizing the mean ROUGE with the top n most salient sentences, conditioning on these salient sentences. This yields an unsupervised, in-domain paraphrase model which we apply to the SummScore pseudo-labels on the training set to make them more abstractive and diverse. We refer to Appendix E for details on the paraphrasing model training, its performance and resulting abstractiveness and diversity levels on pseudo-labels. As the unsupervised process of paraphrasing may harm the pseudo-summary quality, in practice, we apply it to the x% most extractive training data points, where x is among {12.5%, 25%, 50%, 100%}. We use 25% for CNN/DM, 100% for XSum, 50% for WikiHow, and 12.5% on SAMSum, as these provide an ideal ROUGE/abstractiveness trade-off (see Appendix D).
Unsupervised Summarization Re-ranking
For each dataset except SAMSum, we randomly subsample 50k data points from the training set and 1k from the validation set to self-train and validate the model, resulting in a self-training process much less computationally expensive than finetuning. We show self-training results on the test sets using PEGASUS as base model in Table 5. Self-training improves unsupervised summarization performance on all datasets, resulting in a selftrained model better than the base model although not as performing as SummScore. Notably, reapplying SummScore on the new model after selftraining further improves performance drastically. Besides, paraphrasing self-training pseudo-labels helps maintain some degree of abstractiveness, as seen in Appendix D. On CNN/DM, one round of self-training followed by SummScore brings PE-   Mean number of times out of 50 that each model or a tie is selected, with standard deviation in parenthesis, across two use cases and two attributes.
Unsupervised Summarization Re-ranking
GASUS performance above the Summary Loop, two rounds above TED, and three rounds to 39.76 ROUGE-1, within 1% of SOTA model FAR-RW.
Unsupervised Summarization Re-ranking
We conduct a human evaluation on 50 data points randomly sampled from CNN/DM test set. We show human participants the source news article, alongside the summary candidate from the base PEGASUS model, and the one re-ranked by Summ-Score. Participants are asked to pick which summary is more informative, and which is more factually consisteny, with the option of choosing a tie. We cover two use cases: unsupervised abstractive summarization, and zero-shot transfer from a model fine-tuned on XSum. In the former use case, both summaries are identical in 7/50 data points, and 4/50 data points in the latter. Human raters are three volunteer graduate students, with full professional command of English. Results are displayed  Table 7: Ablation study for unsupervised abstractive summarization with PEGASUS. We isolate each feature of Summ-Score and report its re-ranking performance (picking the candidate maximizing this feature), using the mean of ROUGE-1/2/L as reported metric. Best results within 0.1 are in bold.
Unsupervised Summarization Re-ranking
in Table 6. Although both summaries often overlap significantly (rightmost column), resulting in a high Tie, SummScore is strongly preferred over PEGASUS across both use cases and attributes.
Unsupervised Summarization Re-ranking
To better understand SummScore performance gains, we perform an ablation study where reranking is done with each feature taken individually. Results for PEGASUS in unsupervised summarization are shown in Table 7. N-gram overlap features are very strong re-ranking baselines on WikiHow and SAMSum. In fact, ROUGE-1 with the source is even slightly better than Summ-Score on WikiHow. On news datasets, semantic similarity features such as BERTScore are strong baselines. Interestingly, our hand-crafted feature diversity has a negative contribution when used as standalone re-ranker ; however it can help a lot when combined with the other features, acting as a regularizer by encouraging some diversity. On average, SummScore performs the best. We also report trivial feature aggregation baselines Plain average and Random coefficients, which SummScore outpeforms, confirming the efficiency of estimating coefficients through pseudo-labels.
Unsupervised Summarization Re-ranking
In Appendix F, we show that SummScore unsupervised re-ranking is also robust to other decoding methods diverse beam search (Vijayakumar et al., 2016) and nucleus sampling (Holtzman et al., 2019), and a different number of beams (5 to 20). We confirm that our default setup of beam search with 20 beams yields optimal ROUGE results. Echoing SummaReranker findings (Ravaut et al., 2022a), gains further increase when mixing in several decoding methods.
Unsupervised Summarization Re-ranking
Reports speak of at least four people injured. The city is at the heart of the conflict between the Turkish government and Kurdish separatists. Interior Minister Suleyman Soylu said the blast happened at a vehicle repair unit, and appeared to be an accident. He said "it seems there is no outside interference, and the explosion came from the vehicle under repair". Mr Soylu said one person was trapped under rubble, another was seriously injured, and others had minor injuries. The blast brought a roof down, left a huge crater and a pall of smoke drifted over part of the city. The cause remains unclear. The banned Kurdistan Workers' Party (PKK) is active in the area. Turkey is five days away from a key referendum on granting President Recep Tayyip Erdogan sweeping new powers [...] PEGASUS summary (ROUGE-1: 10.53): Interior Minister Suleyman Soylu said the blast happened at a vehicle repair unit, and appeared to be an accident. Self-training summary (ROUGE-1: 32.43):
Unsupervised Summarization Re-ranking
The blast happened at a vehicle repair unit in the city of Diyarbakir, near the border with Syria.
Unsupervised Summarization Re-ranking
A large explosion has struck a police headquarters in the mainly Kurdish city of Diyarbakir in south-eastern Turkey. Table 8: Qualitative sample with self-training PEGASUS from the XSum dataset, after a single round of self-training.
Unsupervised Summarization Re-ranking
We refer to Appendix H for full qualitative unsupervised re-ranking examples on all datasets, and to Table 8 for an example of summary generated by the self-trained PEGASUS model on XSum. As seen, both re-ranking and self-training can improve dramatically from the unsupervised PEGA-SUS baseline, capturing entirely new phrases.
Unsupervised Summarization Re-ranking
As noted in Table 6, SummScore summaries tend to be more factually consistent than the baseline. There is strong intuition to this result: since Summ-Score is built to maximize features of n-gram overlap and semantic similarity with the source, it should yield summaries closer to the source, and more factually consistent as a result. We investigate this further, and use two popular models to evaluate summarization factuality: the established factCC (Kryscinski et al., 2020) and the recently introduced state-of-the-art QAFactEval (Fabbri et al., 2022). factCC uses a BERT model to classify each summary sentence as consistent or inconsistent with regards to the source, and reports the average accuracy over 100%. QAFactEval improves each step of the QA evaluation pipeline (answer selection, question generation, etc) and combines entailment with QA-based metrics into a learned metric. In Table 9, we observe that SummScore QAFactEval is consistently above PEGASUS, and SummScore factCC is better on news datasets too.
Unsupervised Summarization Re-ranking
We analyze coefficients learned by SummScore from a high level perspective in Table 10, gathering features from a same group together. Semantic similarity features are dominating (except for Wik-iHow), encouraging further research using newer semantic similarity metrics for re-ranking.
Unsupervised Summarization Re-ranking
A finer-grain analysis, covering all SummScore   pseudo-labeling techniques, can be viewed in Tables 19 and 20 of Appendix G. SummScore -Salient-R1 and SummScore -Salient-RL place much more emphasis on n-gram overlap with the source. In contrast, SummScore -LEAD-3 (which we use for self-training on CNN/DM, XSum, SAM-Sum) uses relatively more semantic similarity features like BERTScore, suggesting that it is able to exploit key semantic content contained in initial sentences.
Unsupervised Summarization Re-ranking
We introduced SummScore, the first unsupervised abstractive summarization re-ranking system. SummScore does not rely on a neural network: instead, it builds features for each summary candidate, some of them using the source as well, and aggregates them into a final re-ranking score. Feature coefficients are estimated through tuning against a pseudo-label derived from the source document. It is a simple framework which easily supports the addition of new features. SummScore significantly improves the performance of the base summarization model, in terms of ROUGE, BERTScore, factual consistency, and human preference ; in both unsupervised and zeroshot transfer scenarios. Moreover, SummScore selected summary candidate naturally extends into a self-training objective for abstractive summarization, which improves unsupervised summarization.
Unsupervised Summarization Re-ranking
As a second-stage method, SummScore requires access to a base abstractive summarization model generating summary candidates. Generating up to 20 summary candidates per data point can take a long time, especially on training sets, which is needed for the self-training use case. Besides, even though SummScore does not need to train a new neural network, we also need to generate all eight features for each summary candidate once all candidates are generated. N-gram overlap features are very fast, but model-based semantic similarity features (e.g, BERTScore) can be time-consuming to extract, once again, especially on entire training sets.
Unsupervised Summarization Re-ranking
While SummScore will significantly improve the quality of the base model across base models and datasets, ultimately, the performance of the final selected summary is bounded by the capacity of this base model: SummScore improves more PE-GASUS than it does on ChatGPT ; but PEGASUS performance drags ChatGPT. Another limitation lays in the metric used to compare summary candidates with the pseudo-target. We used mean ROUGE, although a model-based semantic similarity metric would make sense too, but at a much greater computational cost.  We perform a sanity check counting the percentage of time that SummScore falls back to a trivial method of re-ranking summary candidates. For each feature described in §3.2, we report the overlap between SummScore and a re-ranking approach consisting in picking the summary candidate maximing this feature. We also report baselines consisting in picking the top beam, an oracle or a worst candidate, and the longest candidate. As seen in Tables 11 and 12, across both backbones PEGA-SUS and ChatGPT, SummScore never collapses  to a trivial candidate selection, and we see similar patterns on the same dataset (e.g., highest overlap with a single feature selection is with BERTScore with source feature on CNN/DM).
Unsupervised Summarization Re-ranking
In Table 13, we show generation hyper-parameters used for each dataset to generate beam search summary candidates used in Table 3. For the transfer setup shown in Table 4, we use as generation hyper-parameters on each target dataset the parameters used on that dataset for   
Unsupervised Summarization Re-ranking
Besides the quality of the selected summary, we also analyze re-ranking performance itself. In Fig. 2, Fig. 3, Fig. 4 and Fig. 5, we show recall curves on each dataset and for all unsupervised and zero-shot summarization setups. Recall@k is defined as the probability of outputting one of the oracle summary candidates (candidates maximizing the mean ROUGE with the target) among the first k candidates. We compare SummScore with the baseline beam search output, and a random candidate selection baseline.
Unsupervised Summarization Re-ranking
In most cases, SummScore (green curves) provides higher recall, with the notable exception of XSum, where both beam search and SummScore and XSum can fail to improve the random baseline.
Unsupervised Summarization Re-ranking
In Table 15, we show ROUGE results from Table 5 alongside abstractiveness results, as measured per the fraction of novel n-grams in output summaries, for re-ranking and self-training experiments. Maximizing both ROUGE and abstractiveness is notoriously difficult, as easy solutions for abstractiveness optimization can deviate a lot from the source, resulting in a harmed ROUGE score.
Unsupervised Summarization Re-ranking
The unsupervised PEGASUS (first row of each block) is very extractive and only produces a small fraction of novel n-grams. SummScore selected summaries, despite maximizing a score which maximizes the mean ROUGE with pseudo-labels extracted from the source document, both improve the ROUGE and the abstractiveness level. However, SummScore re-ranking applied to self-trained models tends to reduce their abstractiveness level, although it stays above the level of the baseline PEGASUS. Paraphrased summaries drastically increase abstractiveness, at the expense of ROUGE -  Table 3, and the next seven plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next seven plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next eight plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 3, and the next eight plots to all zero-shot transfer summarization setups from Table 4. Each re-ranking setup has 20 summary candidates, and we show recall over any oracle candidate for several thresholds k ∈ {1, 2, 3, 4, 5, 7, 10}.  Table 15: ROUGE and abstractiveness for several models: the unsupervised PEGASUS (first sub-block), re-ranking with SummScore (first sub-block), paraphrasing the resulting pseudo-labels (second sub-block), self-training with the pseudo-labels (third sub-block), then re-ranking self-training outputs with SummScore again (third sub-block). All results are on the test set, results of self-training pseudo-labels are underlined, and highest numbers within 0.1 are in bold.
Unsupervised Summarization Re-ranking
except on XSum where paraphrasing also improves ROUGE, motivating our choice to use 100% paraphrased summaries as pseudo-labels. We confirm that our pseudo-labels for self-training, made of a blend of SummScore selected summaries and selected summaries being paraphrased, maintains high ROUGE while being much more abstractive than the baseline PEGASUS.  We track the mean of ROUGE-1, ROUGE-2 and ROUGE-L between the generated paraphrase and target paraphrase on the validation set during training, and perform early stopping. Best mean ROUGE results are shown in Table 16.
Unsupervised Summarization Re-ranking
Next, we study the impact of the paraphrasing model on the SummScore pseudo-targets. In Ta Table 17: Impact of paraphrasing on the pseudo-targets. We report mean ROUGE and percentage of novel n-grams between the paraphrased pseudo-targets and the original pseudo-targets, on the training set of each dataset since this is the subset that paraphrasing is applied to.
Unsupervised Summarization Re-ranking
ble 17, we compute the mean ROUGE between pseudo-targets and their paraphrase, and analyze the novel n-grams. We point out that the paraphrasing is only applied to the training pseudo-labels as the goal of paraphrasing is to encourage the model to learn diversity during self-training, hence Table 17 reporting results on training sets. On each dataset, the mean ROUGE is in the 50-80 range, indicating that the paraphrased pseudo-labels do not deviate too much from the original pseudo-labels and yet is able to re-write some content. Besides, there is a high proportion of new n-grams: more than 10% new 1-grams (with the exception of Wik-iHow on the which the paraphrasing model seems to struggle more to rephrase the input), and more than 20% 2-grams.  In Table 18, we apply SummScore outside of the standard beam search with 20 beams setup. Results show that SummScore performance continuously improves with more summary candidates, whereas the top beam stays around the same level. Besides, SummScore relative gains are stronger with lower quality decoding methods diverse beam search and nucleus sampling. Lastly, combining 20 summary candidates from each of the three decoding methods yields a pool of 60 summary candidates, out of the which SummScore re-ranking can improve by an extra +0.21 mean ROUGE the performance compared to re-ranking 20 beam search candidates (28.59 mean ROUGE vs 28.38). Overall, we recommend our default setup of beam search with 20 beams to apply SummScore re-ranking. A greater number of beams becomes difficult to fit into a standard GPU with 16 GB memory.
Unsupervised Summarization Re-ranking
In Table 19 (PEGASUS backbone) and Table 20 (ChatGPT backbone), we show coefficients found by SummScore (for each of the five methods to select pseudo-labels which we studided), and on each dataset, including when applying SummScore again on top the self-trained models. For the sake of conciseness, we do not include SummScore coefficients obtained in zero-shot setups. BERTScore with source appears as the feature which consistently receives the highest weight for SummScore -Random-3 and SummScore -LEAD-3 ; while ROUGE-2 with source dominates for SummScore -Salient-R1/R2/RL. Diversity and Length features are significantly less used.
Unsupervised Summarization Re-ranking
In the following, beam search output (for PEGA-SUS) or the first candidate from top-p sampling (for ChatGPT) is in orange, SummScore selected summary candidate in blue, and oracle candidate(s) in teal. On each dataset, we show one re-ranking example on the unsupervised PEGASUS and/or ChatGPT (Table 3), one zero-shot re-ranking example selected from Table 4, and one re-ranking example applied on top of the self-trained PEGA-SUS (Table 5).    'We can confirm that protestors from Greenpeace have illegally boarded the 'Polar Pioneer,'under contract to Shell, jeopardizing not only the safety of the crew on board, but the protestors themselves,' she said on Monday. She added: 'Shell has met with organizations and individuals who oppose energy exploration offshore Alaska. We respect their views and value the dialogue. 'We will not, however, condone the illegal tactics employed by Greenpeace. Nor will we allow these stunts to distract from preparations underway to execute a safe and responsible exploration program,' she said in a statement.
Unsupervised Summarization Re-ranking
Summary 'We're here to highlight that in less than 100 days Shell is. Scores Mean ROUGE: 6.55 (rank 11) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Beam #2 Summary 'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists.
Unsupervised Summarization Re-ranking
Mean ROUGE: 43.17 (rank 3) || SummScore rank: 2
Unsupervised Summarization Re-ranking
Beam #3 Summary 'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the ship's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists.
Unsupervised Summarization Re-ranking
Mean ROUGE: 42.85 (rank 4) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove the activists from Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 43.59 (rank 2) || SummScore rank: 12 ...
Unsupervised Summarization Re-ranking
'We're here to highlight that in less than 100 days Shell is.' Greenpeace activists boarded the Polar Pioneer drill rig in the Pacific Ocean to unfurl a banner with the names of millions of people opposed to oil drilling in the Arctic. Greenpeace activists climbed aboard an oil drilling rig off the coast of Alaska on Monday in a protest against Shell's plans to drill for oil in Arctic waters, the environmental group said in a statement. The group said the activists would not interfere with the vessel's navigation, but the company said it had filed a complaint in federal court in Alaska seeking an order to remove them from the Scores Mean ROUGE: 43.91 (rank 1) || SummScore rank: 6 ...
Unsupervised Summarization Re-ranking
Shell has filed a complaint in federal court in Alaska seeking an order to remove Greenpeace activists who climbed aboard an oil rig in the Pacific . The environmental group said in a statement its team would occupy the underside of the main deck of the Polar Pioneer . The six activists are camping on the 38,000-tonne Polar Pioneer platform, which they boarded using inflatable boats from the Greenpeace vessel 'Esperanza' 'We made it! We're on Shell's platform. And we're not alone. Everyone can help turn this into a platform for people power!' tweeted Aliyah Field . 
Unsupervised Summarization Re-ranking
Although Hillary Clinton boasts a robust 3.6 million Twitter followers, not even a vast right-wing conspiracy would be able to interact with 2 million of them. According to two popular online measuring tools, no more than 44 per cent of her Twitter fan base consists of real people who are active in using the social media platform. And at least 15 per cent -more than 544,000 -are completely fake. StatusPeople.com, the oldest publicly available Twitter-auditing tool, reports that 44 per cent of the former secretary of state's followers are 'good'; 15 per cent are 'fake'; and 41 per cent are 'inactive,' meaning that they never tweet or reply to any tweets. SCROLL DOWN FOR VIDEO . FAKERS: According to one popular online audit tool, only 44 per cent of Hillary Clinton's Twitter followers are real people who participate on the social media platform . 'I'M RUNNING FOR PRESIDENT': Clinton has cast herself as a champion of 'everyday Americans' Another Twitter sleuthing website sampled more than 320,000 of Clinton's followers and found that a much larger number of them were 'fake' Just 4 per cent of President Barack Obama's Twitter followers, by comparison, are considered fake. The White House worked overtime to purge most of them after a September 2013 report found that more than half of his followers didn't really exist. Michelle Obama's Twitter audience is 25 per cent fake, according to StatusPeople, along with 21 per cent of Vice President Joe Biden's. Another tool, TwitterAudit.com, sampled 320,000 of Mrs. Clinton's followers and found that 18 per cent were fake. The new measurements will add to the Clinton presidential campaign's embarrassment following news on Tuesday that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned. REALLY? Hillary Clinton's Twitter follower-count appears to be significantly inflated . FACEBOOK FAKERY: Clinton boasts two-thirds of a million Facebook 'likes,' but more than 46,000 of them list 'Baghdad' as their hometown . Vocativ reported that at least 7 per cent of them listed Baghdad, Iraq as their hometown, a larger number than any U.S. city. That would represent more than 46,000 people. Additional evidence of digital astroturfing is that while most of her U.S. Facebook fans are older than 55, most of the Baghdad contingent is in the 18-34 age range. While Clinton was America's top diplomat, her State Department was buying Facebook 'likes,'according to an Associated Press report from last year. 'In 2013, the State Department, which has more than 400,000 likes and was recently most popular in Cairo, said it would stop buying Facebook fans after its inspector general criticized the agency for spending $630,000 to boost the numbers,' the wire service reported. Clinton left the agency while the inspector general was investigating. ASTROTURF? A woman cast as a kindly, gardening grandma in Clinton's campaign launch video is actually a former abortion lobbyist who once helmed a campaign for Texas state Senator Wendy Davis . WHERE'S SCOOBY? Clinton's caravan was spotted in Maumee, Ohio on Monday when she lunched at Chipotle, but no one recognized her . Facebook says it has entire teams of employees tasked with rooting out schemes to artificially boost statistics. 'When we catch fraudulent activity, we work to counter and prevent it -including blocking accounts and removing fake likes,' the company reports on its website. Clinton is in Monticello, Iowa on Tuesday for the first stop on her nascent presidential 'listening' tour. She was driven from New York in a van that her campaign aides have dubbed 'Scooby,' after the 'Mystery Machine' van in the Scooby-Doo cartoon series. Daily Mail Online visited Kirkwood Community College on Monday, the site of the event, and heard from students that the former first lady is a 'control freak' who may be pursuing legal status for illegal immigrants because the Democratic Party needs a new pool of loyal voters.
Unsupervised Summarization Re-ranking
Online measuring tools show that only 44% of Hillary Clinton's Twitter followers are real people who actively use the platform, with 15% completely fake.
Unsupervised Summarization Re-ranking
Comparatively, just 4% of President Obama's Twitter followers are fake. Clinton's Facebook audience has also come under scrutiny, with reports of purchased likes and a large number of followers from Baghdad, Iraq.
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.88 (rank 9) || SummScore rank: 18
Unsupervised Summarization Re-ranking
At least 15% of Hillary Clinton's Twitter followers are fake, according to two online measuring tools, with more than 544,000 completely fake. In comparison, just 4% of Barack Obama's Twitter followers are fake. Clinton's recent embarrassment of having a large number of Facebook likes purchased rather than earned further adds to the digital astroturfing speculation.
Unsupervised Summarization Re-ranking
Mean ROUGE: 19.60 (rank 26) || SummScore rank: 15
Unsupervised Summarization Re-ranking
An investigation has revealed that at least 15% of Hillary Clinton's Twitter followers are completely fake. The same investigation also found that only 44% of her Twitter followers are real people who actively use the platform. This adds to the Clinton presidential campaign's embarrassment following news on Tuesday that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.88 (rank 8) || SummScore rank: 5
Unsupervised Summarization Re-ranking
According to two online measuring tools, only 44% of Hillary Clinton's 3.6 million Twitter followers are real people who actively use the platform, with at least 15% being fake. In comparison, only 4% of President Barack Obama's Twitter followers are considered fake. Clinton's Facebook page has also been called into question, with evidence of 'likes' being purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 27.09 (rank 3) || SummScore rank: 10
Unsupervised Summarization Re-ranking
According to online measuring tools, over half of Hillary Clinton's Twitter followers are either fake or inactive. One tool reported that only 44% of her followers are real people who are active on the platform. Additionally, a large number of Clinton's Facebook fans may have been purchased rather than earned, with over 46,000 of them listing Baghdad, Iraq as their hometown.
Unsupervised Summarization Re-ranking
Mean ROUGE: 24.82 (rank 7) || SummScore rank: 7 ...
Unsupervised Summarization Re-ranking
According to online measuring tools, only 44% of Hillary Clinton's 3.6 million Twitter followers consist of real people who are active on the social media platform. At least 15% of her followers, or more than 544,000, are completely fake. This comes after news that a large number of her Facebook fans may represent 'likes' that were purchased rather than earned.
Unsupervised Summarization Re-ranking
Mean ROUGE: 31.34 (rank 1) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Two different online audit tools say no more than 44 per cent of Hillary's 3.6 million Twitter fans are real people who participate in the platform . The newly minted presidential candidate is fending off accusations that her Facebook page is full of fake 'likes' Her Facebook fan base includes more people from Baghdad, Iraq than any US city . When she was secretary of state, her agency paid $630,000 to bulk up its Facebook likes, but pledged to stop after she left . 
Unsupervised Summarization Re-ranking
Assault: Dr Sahar Hussain attacked two Tube workers because she didn't want to miss the last train home . A GP attacked two Tube workers while screaming 'I'm a doctor' because she did not want to miss the last train home on a Friday night. Dr Sahar Hussain, 53, panicked when she was unable to get through the gates at Leicester Square station, and started ranting at staff. She denied assaulting the two workers, saying she was worried about being stranded on her own in central London because she is a Muslim woman. But Hussain has now been found guilty and ordered to pay a total of £2,250 in fines, compensation and court costs -and she could face disciplinary action from the General Medical Council. In video footage captured on her own mobile phone, Hussain could be heard to shout: 'I'm a doctor actually, I work for the NHS. I'm a doctor. Get me through the gate, I'm going to miss my train.' City of London Magistrates' Court heard Hussain arrived at the station around 11.30pm on June 20 last year, trying to get home to Woodford Green after socialising with friends in the West End. When she was refused entry by the automatic gates, she demanded that ticket seller Malcolm Shaw let her through before lashing out at his colleague Indira Ramsaroop, who was trying to help. Hussain, originally from Iraq, screamed and shouted at Mrs Ramsaroop as she thrust a camera phone into her face before grabbing her by the arm. The 24-year-old Transport for London worker was then chased by the doctor as she tried to flee to the control room, bumping her head on the way. In the video on Hussain's phone she was heard shouting: 'This woman is on something, she's not sober is she? You're in work and you're not sober. Get me through the gate.' During the scuffle Hussain, a mother of one who helps train GPs at two universities, also grabbed Mr Shaw by the arms, leaving him with scratches. Mrs Ramsaroop was close to tears in court as she told how she had to take almost two weeks off work following the incident, adding: 'I had a lot of sleepless nights. It had an impact on myself with customers when I came back to work. 'I have felt very let down to have been threatened and been running away in my place of work. It actually affected me for a very long time and I got quite ill just at the worrying and fear.' Row: The assault took place on a Friday night at Leicester Square station in central London . Hussain admitted losing her temper, telling the court: 'I'm very sorry about the way I expressed myself with my agitation and frustration.' District Judge Quentin Purdy found her guilty of two counts of assault by beating, saying: 'The evidence is overwhelming. You completely lost your self-control. 'Unusually for the sort of incident this court regularly deals with, there is no hint of intoxication in any way. 'But the explanation is there. You were anxious about missing the last train, concerned about being late for work or unable to work the next day, cultural embarrassment by being left in London as a married Muslim woman. 'It was completely out of character, but as we have all seen, you acted in an unpleasant and hysterical fashion towards numerous members of staff.' He said she had subjected Mrs Ramsaroop to 'completely unjustified violence', as well as 'humiliation' for falsely suggesting she was intoxicated at work. The judge added: 'You have lost a great deal by your lack of self-control. Your good name is now lost with these two convictions for assault.' Hussain was fined £500 for each assault and ordered her to pay £500 compensation to Mrs Ramsaroop, £200 to Mr Shaw, £500 prosecution costs and a £50 victim surcharge. Beam 
Unsupervised Summarization Re-ranking
Tell the court she was worried about being stranded on her own in central London because she is a Muslim woman. Describe to the court how she screamed: 'I'm a doctor, get me through the gate!' Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 40.97 (rank 2) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Tell the court she was worried about being stranded on her own in central London because she is a Muslim woman. Describe to the court how she screamed: 'I'm a doctor, get through the gate.' Scores Mean ROUGE: 42.04 (rank 1) || SummScore rank: 9 ...
Unsupervised Summarization Re-ranking
Shell has filed a complaint in federal court in Alaska seeking an order to remove Greenpeace activists who climbed aboard an oil rig in the Pacific . The environmental group said in a statement its team would occupy the underside of the main deck of the Polar Pioneer . The six activists are camping on the 38,000-tonne Polar Pioneer platform, which they boarded using inflatable boats from the Greenpeace vessel 'Esperanza' 'We made it! We're on Shell's platform. And we're not alone. Everyone can help turn this into a platform for people power!' tweeted Aliyah Field .  .' The 28-year-old's father Jalal Kassab said he was worried about his grandchildren living in a war zone, but said the threat of imprisonment made it difficult for his daughter to return to Australia. 'I know she wants to come back and we are trying everything we can to bring her back,' Mr Kassab told the Herald Sun. Another former Melbourne woman Zehra Duman last month shared a series of propaganda pictures she says shows her 'five star jihad' lifestyle . In photographs posted to a Twitter several women are pictured standing under an Islamic State flag, reclining against a clean white BMW M5 and wielding machine guns . In one tweet, Duman said: 'US + Australia, how does it feel that all 5 of us were born n raised in your lands, & now here thirsty for ur blood?'
Unsupervised Summarization Re-ranking
The children's paternal grandparents say they are worried Kassab, 28, is 'brainwashing' the children, after their father was killed near the Syria-Turkey border last year. 'We pray they come back but it does not look good.' Kassab's Twitter paints a picture of their life in the city the terrorist group IS have made their headquarters, where the children cannot go to school and airstrikes hit blocks away from their apartment. 
Unsupervised Summarization Re-ranking
Former Melbourne woman Dullel Kassab fled to Raqqa in Syria with her children last year, and she regularly boasts on Twitter that her four-year-old daughter and two-year-old son sleep with toy guns next to their beds and her daughter likes watching IS videos of 'Muslims killing bad ppl.' The children's paternal grandparents say they are worried Kassab, 28, is 'brainwashing' the children, after their father was killed near the Syria-Turkey border last year.
Unsupervised Summarization Re-ranking
Mean ROUGE: 57.48 (rank 1) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Grandparents have pleaded for the safe return of two children in Syria . Former Melbourne woman Dullel Kassab fled to Raqqa in Syria with her four-year-old daughter and two-year-old son last year . She said her daughter likes watching IS videos of 'Muslims killing bad ppl' Beam #20 Summary Mr Martin has said his party will facilitate a minority government, but will not support a programme for government. Fianna Fil want to see the immediate removal of water charges, but Fine Gael see a role for them. However, no party was able to form a majority government and TDs will so far failed Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 19.28 (rank 2) || SummScore rank: 5
Unsupervised Summarization Re-ranking
Reference Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government.  WikiHow: re-ranking from the unsupervised PEGASUS
Unsupervised Summarization Re-ranking
On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.; , Click on the program's shortcut icon on your desktop or programs list to launch it. Wait until the program starts. Any version will do, but the latest one is better because they have additional useful functions. , On the Menu bar located at the top of the screen, go to File and click "Open." Locate the image, select it, then click "Open." , To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image" on the Menu bar, into "Black and White." , On the Layer panel on the palette dock located at the bottom left of the screen, create a new layer by clicking a small paper-like icon beside the folder icon. , Change the layer name to "Skin" as this will be the first part of the image that you will add a color to. , Press "B" on your keyboard or get the Brush tool on the toolbar located at the left side of the screen. The icon is a brush. , Pick a nice skin tone color on the Swatches, which are also on the Palette dock. You may also set the foreground color to get more colors on the Toolbar, below the Zoom Tool, which is the two square that indicates as foreground and background color. A skin tone color is the lightest Orange or Yellow Orange. , Drag and move your mouse to the skin tone of your image. To avoid misplacing the colors, use the Zoom tool or simply press CTRL + Z. Use the Eraser tool, which is also located on the toolbar, to delete some misplaced part; but change the Brush Preset, which is on the Palette Dock, and choose "Soft Round" so it will create pixelated color when you erase them. , Once you're done with the skin, repeat Steps 1 to 6 with other parts of the body. For example, if you want to color the clothes, add a layer and rename the layer "Clothes." After renaming, proceed with selecting the right tool and color, and coloring the clothes. Continue with other parts, like lips, hair, background, etc. , To merge all layers, go to Layer dock, right-click, and choose "Flatten Image" or "Merge Visible" on the options. , Adjust the image tone, level, or brightness on the Image Menu to enhance the photo. , Press CTRL + S, choose JPEG as file type, add a file name, and click "Save" when you're done! Beam #1
Unsupervised Summarization Re-ranking
Summary You may also set the foreground color to get more colors on the Toolbar, below the Zoom Tool, which is the two square that indicates as foreground and background color. Scores Mean ROUGE: 13.01 (rank 7) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Beam #2 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-into " Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.18 (rank 1) || SummScore rank: 2
Unsupervised Summarization Re-ranking
, On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen, into " Scores Mean ROUGE: 23.18 (rank 1) || SummScore rank: 5
Unsupervised Summarization Re-ranking
Beam #4 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar, into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Beam #5 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar, into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 13
Unsupervised Summarization Re-ranking
Beam #6 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-to " Scores Mean ROUGE: 22.83 (rank 2) || SummScore rank: 4
Unsupervised Summarization Re-ranking
Beam #7 Summary , On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the menu bar-into "Black and White.",On the Scores Mean ROUGE: 21.05 (rank 5) || SummScore rank: 7
Unsupervised Summarization Re-ranking
On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-into " Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 23.18 (rank 1) || SummScore rank: 3 ...
Unsupervised Summarization Re-ranking
, On your scanner tool, set the PPI (Pixel per inch) to 350 so that it will create a high-quality image.;, Click on the program's shortcut icon on your desktop or programs list to launch it., To change it, go to "Image" on the Menu bar then click "Mode" and choose "RGB." To turn your scanned image fully grayscale, just in case you didn't change it on your scanner setting and the image has traces of colors, change the Adjustment, under "Image"-on the Palette dock located at the bottom left of the screen-and choose Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 22.71 (rank 3) || SummScore rank: 1 (SummScore output) ...
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. , the base of the tongue. Or, use your index finger and thumb to push the pill over the back of your cat's tongue. If you do it this way, you must do it quickly to prevent your fingers from getting bitten.Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue. This way you can avoid having to put your fingers in your cat's mouth. Coat the pill with butter to make swallowing it easier. , Once you have placed the pill at the base of its tongue, use your hands to keep its mouth closed by applying gentle pressure. Also re-position your cat's head so that it is level instead of tilted back; this will make it easier for your cat to swallow the pill. Gently rub your cat's throat or blow in its nose to encourage it to swallow the pill.Try not to let your cat go until the pil lis swallowed. Place a drop of water on its nose. If your cat licks the water off, then this means that it has swallowed the pill. , Tricyclic antidepressants should be given with food to decrease side effects such as vomiting. Therefore, feed your cat directly after giving it its medication. If your cat will not eat, then entice it to eat by feeding it its favorite food like cheese, or by giving it a treat.Make sure to provide your cat with water as well. You can also give your cat flavored water like juice from a tuna can to entice it to drink.
Unsupervised Summarization Re-ranking
Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.17 (rank 8) || SummScore rank: 12
Unsupervised Summarization Re-ranking
Beam #2 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in the towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 18.78 (rank 6) || SummScore rank: 8
Unsupervised Summarization Re-ranking
Beam #3 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.17 (rank 8) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Beam #4 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. gently stabilize it by holding it steady with one or both hands. Rub your cat's throat or blow in its nose to encourage it to swallow the pill.
Unsupervised Summarization Re-ranking
Mean ROUGE: 18.68 (rank 7) || SummScore rank: 7
Unsupervised Summarization Re-ranking
Beam #5 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. Wrap your fingers around the bottom of its cheekbones for leverage. Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 26.93 (rank 3) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
Beam #6 Summary Tricyclic antidepressants should be given with food to decrease side effects such as vomiting. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 15.09 (rank 10) || SummScore rank: 17 Beam #7 Summary Pet your cat and talk to it in a soothing voice to calm and reassure it. gently stabilize it by holding it steady with one or both hands. Scores Mean ROUGE: 8.72 (rank 12) || SummScore rank: 19
Unsupervised Summarization Re-ranking
Beam #8 Summary Gently stabilize it by holding it steady with one or both hands. If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out.
Unsupervised Summarization Re-ranking
Mean ROUGE: 17.18 (rank 7) || SummScore rank: 14 ...
Unsupervised Summarization Re-ranking
If your cat resists you or is prone to scratching, then wrap your cat in a towel so that just its head is sticking out. Alternatively, you can use a pill syringe to place the pill at the base of your cat's tongue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.74 (rank 1) || SummScore rank: 2 ...
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. WikiHow: re-ranking from the self-trained PEGASUS
Unsupervised Summarization Re-ranking
Talking in private will help you feel less nervous and more comfortable about asking for help.It will also give your teacher more time to help you because they aren't trying to teach at the same time. For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. The note could say, "Can we talk later about something personal? Thanks, Mark." You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue. , Sometimes teachers will try to offer advice on what you should do, when all you wanted was for them to listen. If you let your teacher know how they can help, it will make it easier for them to give you the help you want and need.Think about what kind of help you want. Ask yourself, "Do I want her to listen, to give me advice, or to do something about the problem?" Tell your teacher how they can help. For example, "Can you help me come up with ways to make more friends?" If you don't know how you want your teacher to help, you it's okay for you to say that, too. Try saying, "I need your help with a problem, but I don't know what kind of help I need." , Also, being honest will help your teacher figure out the right way to help you. The more truthful information your teacher has about what is going on, the more they will be able to help. If you are afraid you will get in trouble, then say that. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid."
Unsupervised Summarization Re-ranking
You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid." Scores Mean ROUGE: 22.54 (rank 14) || SummScore rank: 14
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you're afraid to approach your teacher, put a note on their chair.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.92 (rank 7) || SummScore rank: 18
Unsupervised Summarization Re-ranking
Beam #3 Summary For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. Try saying, "I need your help with a problem, but I don't know what kind of help I need.", Also, being honest will help your teacher figure out the right way to help you.
Unsupervised Summarization Re-ranking
Mean ROUGE: 29.41 (rank 9) || SummScore rank: 15
Unsupervised Summarization Re-ranking
Beam #4 Summary Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.
Unsupervised Summarization Re-ranking
Mean ROUGE: 28.40 (rank 10) || SummScore rank: 20
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. For example, "My friend is thinking about cheating on a test and I need advice on how to stop them from doing something so stupid." Scores Mean ROUGE: 27.36 (rank 12) || SummScore rank: 11
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 31.47 (rank 6) || SummScore rank: 9
Unsupervised Summarization Re-ranking
Beam #7 Summary Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you are afraid to approach your teacher, put a note on their chair.
Unsupervised Summarization Re-ranking
Mean ROUGE: 30.92 (rank 7) || SummScore rank: 17
Unsupervised Summarization Re-ranking
Beam #8 Summary You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on. You could also send your teacher an email or message letting them know you would like to ask their help with a personal issue.
Unsupervised Summarization Re-ranking
Mean ROUGE: 28.06 (rank 11) || SummScore rank: 13 ...
Unsupervised Summarization Re-ranking
For example, you could say, "When you have time later today, could we talk about a problem I'm having?" If you're afraid to approach your teacher, put a note on their chair. Try saying, "I need your help with a problem, but I don't know what kind of help I need.", Also, being honest will help your teacher figure out the right way to help you. The more truthful information your teacher has about what is going on, the more they will be able to help Scores Mean ROUGE: 35.08 (rank 1) || SummScore rank: 2
Unsupervised Summarization Re-ranking
Think about what kind of help you want. You could say, "I need your help with something but I'm scared I'll get in trouble." If you are asking for help, but also trying not to get someone else in trouble, you could leave out names, but still be honest about what is going on.If you're afraid to approach your teacher, put a note on their chair. It will also give your teacher more time to help you because they aren't trying to teach at the same time Scores
Unsupervised Summarization Re-ranking
Mean ROUGE: 34.51 (rank 2) || SummScore rank: 1 (SummScore output)
Unsupervised Summarization Re-ranking
Negotiators from Ireland's two biggest parties are to meet for a final effort to form a minority government. 
RefSum: Refactoring Neural Summarization
In neural text summarization, system designers commonly have flexible choices in model architectures (Rush et al., 2015;Kedzie et al., 2018), decoding strategies (Paulus et al., 2018) (e.g. beam search) and etc. As a result, even on the same dataset, different selection biases of these choices will lead to diverse system outputs (Kedzie et al., 2018;Hossain et al., 2020).
RefSum: Refactoring Neural Summarization
To combine complementarity of system's output under different setups, researchers have made some preliminary efforts on two-stage learning (Collins and Koo, 2005;Huang, 2008; González-Rubio * Corresponding author. "Doc, Hypo, Ref" represent "input document, generated hypothesis, gold reference" respectively. "Hypo'" represents texts generated during test phase. Θ Base and Θ Meta represent learnable parameters in two stages. Mizumoto and Matsumoto, 2016), consisting of (i) a base-stage: first generates different outputs under different setups, and (ii) a meta-stage: then aggregates them in diverse ways, exemplified by stacking that uses a high-level model to combine multiple low-level models (Ting and Witten, 1997), or reranking (Collins and Koo, 2005), which aims to rerank different outputs of one system. Although these methods each play a role in different scenarios, they suffer from following potential limitations:
RefSum: Refactoring Neural Summarization
(i) Ad-hoc Methods: most existing methods are designed for a specific scenario. For example, Li et al. (2015) and Narayan et al. (2018b) resort to reranking techniques to select summary-worthy sentences that are usually generated from one system. By contrast, Hong et al. (2015) focus on summaries generated from different systems and use a non-neural system combination method to make their complementary advantages. Few works explore if the complementarity existing in different scenarios could be utilized in a unified framework.
RefSum: Refactoring Neural Summarization
(ii) Base-Meta Learning Gap: parameterized models between two learning stages are relatively independent. For example, Zhou et al. (2017) and  adapt the seq2seq (Sutskever et al., 2014) framework as the meta model for combination, which takes the outputs of multiple base systems as a part of the inputs for machine translation. As a result, there is no parameter sharing between the meta model and base systems as shown in Fig. 1, which prevents the meta model from fully utilizing the knowledge encoded in the base systems.
RefSum: Refactoring Neural Summarization
(iii) Train-Test Distribution Gap: regarding the meta-learning stage, there is a distribution gap between the training and test distributions. Fig. 1 elucidates this phenomenon: the training distribution of Hypo differs from the test distribution of Hypo'. Although both two are outputs from the base stage, Hypo would be more accurate (closer to gold summaries) since it is the output during the training phase.
RefSum: Refactoring Neural Summarization
In this work, we aim to address these limitations by proposing a general framework, named Refactor, which can not only serve as a base system to construct a summary by selecting sentences from the source document but also act as a meta system to select the best system output from multiple candidates. The unification of base and meta systems allows them to share a set of parameters, thereby alleviating the "Base-Meta learning gap". Besides, we propose a pretrain-then-finetune paradigm for Refactor that mitigates the "Train-Test distribution gap". In practice, our proposed Refactor can be applied to different scenarios. For example, as a meta system, it can be used for multiple system combination or single system re-ranking.
RefSum: Refactoring Neural Summarization
Our contributions can be briefly summarized as:
RefSum: Refactoring Neural Summarization
(1) We dissect two major factors that influence the performance of two-stage learning when leveraging the complementarity among different systems: (i) Base-Meta Learning Gap (ii) Train-Test Distribution Gap;
RefSum: Refactoring Neural Summarization
(2) We show these two types of gaps can be alleviated by promoting communication between the two stages in §4 , and therefore present a new paradigm where the base and meta learners are parameterized with shared parameters;
RefSum: Refactoring Neural Summarization
(3) We have made comprehensive experiments (twenty-two top-scoring systems, four datasets). In addition to achieving state-of-the-art results on CNN/DailyMail dataset ( §5) by a significant margin, the efficacy of the proposed Refactor opens up a thought-provoking direction for performance improvement: instead of pursuing a purely end-toend system, a promising exploration is to incorporate different types of inductive biases stage-wisely with the same parameterized function. Our exper-imental results demonstrate that there exists complementarity introduced by decoding algorithms (e.g. beam search) §5.5 or system combination §5.6 among the current state-of-the-art summarization systems, which can be effectively utilized by our model for boosting the system performance.
RefSum: Refactoring Neural Summarization
Existing works commonly design systems in an end-to-end fashion (Sutskever et al., 2014;Sukhbaatar et al., 2015), which, though effective, also proves to be insufficient in some scenarios (Glasmachers, 2017;Webb et al., 2019). Instead of optimizing a system in an end-to-end fashion, one more flexible paradigm, stage-wise learning, is to break down the holistic process into different stages. The basic idea is to incorporate different types of inductive biases stage-wisely and two typical examples are: Stacking and Reranking.
RefSum: Refactoring Neural Summarization
Stacking Stacking (a.k.a, Stacked Generalization) is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy (Ting and Witten, 1997).
RefSum: Refactoring Neural Summarization
In NLP research, this method has been widely explored in machine translation (MT) task. Traditionally, it is used to improve the performance of statistical MT systems (González-Rubio et al., 2011;Watanabe and Sumita, 2011;Duh et al., 2011;Mizumoto and Matsumoto, 2016). Some recent work (Zhou et al., 2017; also extends this method to neural MT where the meta model and base systems are all neural models. There is a handful of works about system combination for summarization (Hong et al., 2015), in which a feature-based meta model is used for combining unsupervised text summarization systems.
RefSum: Refactoring Neural Summarization
Reranking Reranking is a technique to improve performance by reranking the output of an existing system, which has been widely used across different NLP tasks, such as constituency parsing (Collins and Koo, 2005;Huang, 2008), dependency parsing Do and Rehbein, 2020), semantic parsing (Ge and Mooney, 2006;Yin and Neubig, 2019), machine translation (Shen et al., 2004;Mizumoto and Matsumoto, 2016).
RefSum: Refactoring Neural Summarization
Comparing reranking and stacking, both of them involve two-stage learning and the first stage would provide multiple candidate outputs as the input for the second stage. However, they differ in the way how multiple candidate outputs are generated at the first stage. Specifically, reranking usually decodes k-most qualified results during inference, using one base system. By contrast, stacking generates multiple outputs that are usually from different base systems.
RefSum: Refactoring Neural Summarization
In what follows, we detail how to formulate summarization as a two-stage learning task.
RefSum: Refactoring Neural Summarization
The system in the base stage aims to generate a summary based on the input text. Specifically, given a document D = {s 1 , · · · , s n } with n sentences, we refer to C as a candidate summary of D generated by a summarization system, which can be parameterized in diverse forms:
RefSum: Refactoring Neural Summarization
where BASE(, Θ base ) represents a base system that can be instantiated either as an extractive model or abstractive model with a specific experimental setup: training method T , decoding strategy S.
RefSum: Refactoring Neural Summarization
In practice, different choices of parameterized function BASE(·), training method T and decoding strategy S commonly lead to different candidate summaries, C = {C 1 , · · · , C k }, where C represents a set of different candidate summaries. The goal of the meta system is to utilize complementarities among C by popular techniques, such as reranking and system combination. Specifically, given a set of candidate summaries C, a meta system is used to re-construct a new candidate summary C *
RefSum: Refactoring Neural Summarization
where Θ meta represents learnable parameters of the meta system.
RefSum: Refactoring Neural Summarization
Despite effectiveness of existing meta systems, they, as briefly mentioned in §1, suffer from two major problems: (i) Base-Meta Learning Gap and (ii) Train-Test Distribution Gap.
RefSum: Refactoring Neural Summarization
In this paper, we propose the model Refactor that unifies the goal of the base and meta systems by the view that a summary can be generated by selecting the best combination of document sentences. Therefore, both base and meta systems aim to select an optimal candidate summary, and they only differ in how the candidate summary set is constructed. For example, Refactor can be a base system when the candidate summary set C is formed by directly enumerating different combinations of document sentences and would be a meta system when C represents summaries from different systems. This formulation is advantageous in two points:
RefSum: Refactoring Neural Summarization
(1) No matter where a system selects (from document sentences or multiple system outputs), the chosen criteria that define a good summary are shared. Therefore, the learning process of base and meta systems can be parameterized using a set of parameters, maximizing the information-sharing across two stages and mitigating the Base-Meta Learning Gap.
RefSum: Refactoring Neural Summarization
where REFACTOR(·, Θ refactor ) is the Refactor model, and the candidate summaries C can be constructed in different ways.
RefSum: Refactoring Neural Summarization
(2) Additionally, learning to select candidate summaries from document sentences enables the system to see more diverse candidates with different distributions. This is effective for solving the Train-Test Distribution Gap, where the distribution of the meta system outputs in training samples deviates from the test one.
RefSum: Refactoring Neural Summarization
Specifically, our proposed Refactor first learns to select candidate summaries from document sentences (pre-trained Refactor) and then learns to select candidate summaries from different system outputs (fine-tuned Refactor).
RefSum: Refactoring Neural Summarization
Pre-trained Refactor takes as input a document D = {s 1 , · · · , s n } as well as a set of candidate summaries C = {C 1 , · · · , C m }, which can be constructed by enumerating possible combinations of source sentences with heuristic pruning. For example, an extractive system could be used to prune unlikely sentences to control the number of candidates. REFACTOR(·, Θ refactor ) is instantiated as a score function which quantifies the degree to which a candidate summary C i is matched with the source document D.
RefSum: Refactoring Neural Summarization
where D and C i denote document and summary representations respectively, which are calculated by a BERT (Devlin et al., 2019) model. SCORE(·) is a function that measures the similarity between a document and candidate summary.
RefSum: Refactoring Neural Summarization
Contextualized Similarity Function To instantiate SCORE(·), we follow the forms as mentioned in Zhang et al. (2019b); Zhao et al. (2019); Gao et al. (2020), which have shown superior performance on measuring semantic similarity between documents and summaries. Specifically, SCORE(·) is defined based on the greedy matching algorithm, which matches every word in one text sequence to the most similar word in another text sequence and vise versa. Given the document embedding matrix D = d 1 , · · · , d k and the candidate embedding matrix C = c 1 , · · · , c l encoded by BERT, SCORE(·) can be calculated as:
RefSum: Refactoring Neural Summarization
where the weighted recall R, precision P are defined as follows: 1
RefSum: Refactoring Neural Summarization
w i is the weight of the i-th token in the document. We use weighted recall R based on the assumption that for text summarization, tokens in the source document have different importance and the summary should capture the most important information of the source document. Therefore, we introduce a weighting module built by a twolayer Transformer (Vaswani et al., 2017) assigning weights w i :
RefSum: Refactoring Neural Summarization
represents the embedding of the "[CLS]" token which encodes the global information. d is the dimension of d i .
RefSum: Refactoring Neural Summarization
Learning Objective We use a ranking loss to learn the parameter Θ refactor , inspired by the assumption (Zhong et al., 2020) that a good candidate summary should be as close with the source 1 We found that adding 1 to the precision and recall helps to stabilize the training. 
RefSum: Refactoring Neural Summarization
where C i and C j denote the i-th and j-th sample of the candidate list which is descendingly sorted by the ROUGE (Lin, 2004) scores between the reference summaryĈ and candidates. That is,
RefSum: Refactoring Neural Summarization
λ c is the corresponding margin set to 0.01.
RefSum: Refactoring Neural Summarization
In order to fit the distributions of the specific types of input, we then fine-tune Refactor using the outputs generated by the base systems. Specifically, fine-tuning is also based on Eq. 9 where the candidate summaries C are generated by the base systems under different application scenarios.
RefSum: Refactoring Neural Summarization
Why does Pre-train and Fine-tune matter? We elaborate on the proposed two-step training using a real case. Fig. 2 depicts the distribution of ROUGE-1 scores regarding the candidate summaries in the pre-training stage training set, finetuning stage training set and test set on the XSum dataset, where we sample the same number of {document, candidate summaries} pairs. We can observe that: (i) there is a distribution gap between train and test samples in fine-tuning stage. (ii) in pre-training stage the pre-trained Refactor has seen a large number of candidate summaries with diverse performance (ROUGE value), which improves its generalization ability. In §5 we will show that the Pre-train and Fine-tune paradigm outperforms one-step training where the model is directly trained with data generated from the base systems.
RefSum: Refactoring Neural Summarization
Our Refactor can be used as different roles in different scenarios as follows.
RefSum: Refactoring Neural Summarization
The pre-trained Refactor can not only be fine-tuned for a better selection of candidate summaries, but also be regarded as a base system, providing one system output. This feature of Refactor maximizes parameter sharing across the two training stages.
RefSum: Refactoring Neural Summarization
Both pre-trained Refactor and fine-tuned Refactor can be used as a meta system to select the best candidate when we have multiple system summaries. In this work, we explore the following settings:
RefSum: Refactoring Neural Summarization
(1) Single System: It considers re-ranking candidate summaries generated from a single abstractive system using beam search.
RefSum: Refactoring Neural Summarization
(2) Multi-system Summary-level: It is tasked to select the best candidate summary from the results of different systems.
RefSum: Refactoring Neural Summarization
(3) Multi-system Sentence-level: We also take a step towards the fine-grained fusion of summaries from extractive and abstractive systems. Specifically, here candidate summaries are generated by combining the results of different systems at the sentence level.
RefSum: Refactoring Neural Summarization
We mainly experiment on four datasets, whose statistics are shown in Tab. 1. CNNDM 2 (Hermann et al., 2015) is a widely used dataset containing news articles and the associated highlights which are used as the reference summaries. We follow the work of Nallapati et al. (2016)  WikiHow 5 (Koupaee and Wang, 2018) is a largescale dataset constructed from the articles using online WikiHow knowledge base.
RefSum: Refactoring Neural Summarization
Below, we mainly use BART, GSum and PEGA-SUS as the base systems since they have achieved state-of-the-art performance on at least one dataset.
RefSum: Refactoring Neural Summarization
BART (Lewis et al., 2020) is a large pre-trained sequence-to-sequence model that achieves strong performance on the abstractive summarization.
RefSum: Refactoring Neural Summarization
GSum (Dou et al., 2020) enhances the performance of BART using additional guidance information, which achieves the current state-of-the-art performance on the CNNDM dataset. PEGASUS  achieves competitive performance on various summarization datasets and is the current state-of-the-art on the XSum dataset.
RefSum: Refactoring Neural Summarization
To make a comprehensive evaluation of our proposed model, we additionally collect 19 top-scoring systems as base systems on CNNDM. 6 In details, for §5.7 we use the following systems: pointer-generator+coverage (See et al., 2017) 
RefSum: Refactoring Neural Summarization
Neural system combinator:
RefSum: Refactoring Neural Summarization
We use BERTScore (Zhang et al., 2019b) as an unsupervised baseline with neural models, which is an automatic evaluation metric computing the similarity of text pairs based on the corresponding BERT-encoded representations. We use it to directly compute the similarity score between the source documents and candidate summaries. Non-Neural system combinator:
RefSum: Refactoring Neural Summarization
We use RankSVM 7 (Joachims, 2002) as a non-neural baseline. We perform cross-validation on the development set for hyper-parameter searching and train the model on the development set. The set of features is listed in Appendix A. Oracles: We compare our model with sample-wise Min, Max and Random oracles using ROUGE.
RefSum: Refactoring Neural Summarization
For the following experiments in §5.5, §5.6 and §5.7 on CNNDM, we pre-train the Refactor model with a candidate set generated by enumerating combinations of sentences in the source documents. To reduce the number of candidates, we prune the sentences assigned with lower scores by an extractive model, BERTSum (Liu and Lapata, 2019), following Zhong et al. (2020). The maximum number of candidates for one data sample is 20. The pretrained Refactor is also used a base system in §5.6, whose outputs are used together with other base systems as candidate summaries. For different experiments, we fine-tune pre-trained Refactor on the base system's output, and name the model as fine-tuned Refactor. To analyze the effectiveness of the proposed two-stage training, we additionally train the model without the pre-training step, which is named as supervised Refactor.
RefSum: Refactoring Neural Summarization
The pre-trained BERT model we used is from Transformers library (Wolf et al., 2020). 8 We use Adam optimizer (Kingma and Ba, 2015) with learning rate scheduling.
RefSum: Refactoring Neural Summarization
where the warmup_steps is 10000. The model performance on the validation set is used to select the checkpoint. Pre-training takes around 40 hours  on 4 GTX-1080-Ti GPUs while fine-tuning takes around 20 hours.
RefSum: Refactoring Neural Summarization
We use BART and GSum for this experiment, and use beam search to generate the candidate summaries where the beam size is set to 4. The results are listed in Tab. 2, which shows that (1) Refactor can boost the base system's performance by a significant margin, (2) the fine-tuned Refactor outperforms supervised Refactor directly trained on the base system's outputs, showing the effectiveness of the two-step training. Notably, we observe the fine-tuned Refactor can boost BART's performance from 44.26 to 45.15 on ROUGE-1, indicating that the top-1 output selected by beam search is not always the best one, and Refactor can effectively utilize the complementarity introduced by considering all the beam search results.
RefSum: Refactoring Neural Summarization
Summary-level For summary-level combination, we explore two-system combination (BART & pre-trained Refactor) and three-system combination (BART, GSum & pre-trained Refactor). The results are shown in Tab. 3.
RefSum: Refactoring Neural Summarization
Sentence-level For sentence-level combination, we use BART and pre-trained Refactor as the base  Table 3: Summary level combination on CNNDM. Two denotes two-system combination (BART and pre-trained Refactor).
RefSum: Refactoring Neural Summarization
Three denotes three-system combination (BART, pre-trained Refactor and GSum). R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L. †: significantly better than the best single system (p < 0.01).  Table 4: Sentence level combination on CNNDM. R-1, R-2 and R-L denote ROUGE-1, ROUGE-2 and ROUGE-L. †: significantly better than the best single system (p < 0.01).
RefSum: Refactoring Neural Summarization
systems. The sentences of each system's output are merged together to form the candidate sentence set, and all combinations of three sentences in the candidate set are generated as candidate summaries.
RefSum: Refactoring Neural Summarization
To prune the candidates, we use tri-gram blocking to filter out candidates of which there exists an identical tri-gram in two sentences.  tion, supervised Refactor has similar performance as fine-tuned Refactor. We hypothesis that this is because here the number of candidates in the finetuning data is relatively large, therefore directly training on the fine-tuning data is sufficient enough.
RefSum: Refactoring Neural Summarization
(ii) The pre-trained Refactor cannot outperform GSum model in the three-system combination setting in Tab. 3. The reason might be that GSum has much stronger performance than the other two systems, which intuitively makes the expected gain from system combination lower than other settings.
RefSum: Refactoring Neural Summarization
To evaluate the Refactor's generalization ability, we explore another setting where the pre-trained Refactor is directly used to select the outputs of multiple systems without fine-tuning.
RefSum: Refactoring Neural Summarization
To this end, we collect 19 top-performing summarization systems on CNNDM dataset. Here, we investigate if our Refactor can boost the performance of candidate systems with similar performance. In addition, we also aim to investigate how the range width of different systems' performance affects Refactor's performance. Therefore, we group the candidate systems into equal-width bins based on their average ROUGE-1 scores, and evaluate our Refactor on each bin separately.
RefSum: Refactoring Neural Summarization
In Tab. 5 we report the average ROUGE-1 scores of the oracles, Refactor, and the best candidate system in each bin whose width is 1. Refactor consistently outperforms the best candidate system, showing its generalization ability.
RefSum: Refactoring Neural Summarization
Next, in Fig. 3 we plot the change of Refactor's performance with different bin widths. We define the success rate of Refactor with a given bin width to be the number of bins where Refactor outperforms the single best base system normalized by the total number of bins. We observe that Refactor is more likely to improve the performance of base systems when the system-level performance of the  base systems is similar. Intuitively, if one base system is significantly better than the other systems, it is more difficult for Refactor to use other systems to complement the best base system.
RefSum: Refactoring Neural Summarization
Next, we move on to other text summarization datasets to evaluate our proposed method's strength beyond CNNDM dataset. Some of the datasets used here are not as well-studied as CNNDM dataset, so there are less top-performing systems on these datasets. Therefore, here we focus on the experiments of the single system setting. The results in Tab. 6 show that Refactor is able to bring stable improvement over the base systems. The average summary length of these datasets varies from 23.3 (XSum) to 210.3 (Pubmed). Therefore, the results here demonstrate the Refactor can be applied to datasets with different characteristics. On XSum dataset, the pre-trained Refactor outperforms the fine-tuned Refactor. This may result from the additional pre-training data we introduced using BART, which is effective enough to train the Refactor for reranking PEGASUS output.
RefSum: Refactoring Neural Summarization
We perform a fine-grained evaluation of Refactor to understand where improvement mainly comes.
RefSum: Refactoring Neural Summarization
Setup We choose the summary-level system combination setting on CNNDM test set in §5.6 as a case study, where the base systems are: BART and pre-trained Refactor, and then we use a fine-tuned Refactor 9 to combine them. Specifically, we first (i) define δ(C BART , C Pretrain ) as the performance (i.e., ROUGE) gap on the candidate summary C.
RefSum: Refactoring Neural Summarization
(ii) then partition test samples into different buckets S 1 , · · · , S n according to the performance gap δ. (iii) calculate selection accuracy for each bucket, which represents how accurately the Refactor can identify the best one from two candidate summaries.
RefSum: Refactoring Neural Summarization
The results are shown in Fig. 4. We observe that the selection accuracy is increasing as the gap δ becoming larger, indicating that Refactor performs better on the candidate summaries with diverse performance. Combining the results we get in §5.7, we conclude that Refactor has the largest potential gain when the base systems effectively complement each other -They have similar system-level performance but diverse summary-level performance. For example, each base system may perform significantly better than others on a subset of data with different characteristics but could not outperform others across the whole dataset.
RefSum: Refactoring Neural Summarization
We present a general framework for utilizing the complementarity of modern text summarization systems by formulating text summarization as a two-stage learning problem. Our proposed model, Refactor, can be used either as a base system or a meta system, effectively mitigating the learning gaps introduced in the two-stage learning. Experimental results show that Refactor is able to boost the performance of the base systems, and achieves the state-of-the-art performance on CNNDM and XSum datasets. We believe this work opens up a new direction for improving the performance of text summarization systems apart from an iterative process of searching for better model architectures -The gain of performance could be made by fully investigating and utilizing the complementarity of different systems with various architectures, problem formulations, decoding strategies, etc.
BRIO: Bringing Order to Abstractive Summarization
Neural methods for abstractive summarization (Rush et al., 2015;Nallapati et al., 2016;Lewis et al., 2020; formulate summarization as a sequenceto-sequence (Seq2Seq) problem (Sutskever et al., 2014), learning to generate the summary in an autoregressive manner. Such models are commonly trained with maximum likelihood estimation (MLE), maximizing predictive probability of the reference output given the gold sub-sequence before it. However, during inference the model must also generate the output based on possibly erroneous previous steps. This can hurt model performance, a phenomenon often called exposure bias (Bengio et al., 2015;Ranzato et al., 2016). To maintain reasonable performance even in the case of a sub-sequence with errors, we argue that the  Table 1: Accuracy of different abstractive summarization systems w.r.t ranking the quality of candidate summaries on CNNDM dataset. Acc. stands for the frequency of the model assigning higher probabilities to better candidate summaries. The candidate summaries are generated by a pre-trained model (BART), and we select the best and the worst candidates (w.r.t. ROUGE scores) for each of the samples. High and Low represent the average performance of the best and worst candidates respectively. R-1/2/L are the ROUGE-1/2/L scores. The original BART only achieves 54.80% accuracy.
BRIO: Bringing Order to Abstractive Summarization
model must accurately estimate relative quality of different generated outputs, since effective inference requires comparison among these candidates.
BRIO: Bringing Order to Abstractive Summarization
To understand whether existing models can accurately perform such relative comparisons, we conducted a preliminary study on pre-trained BART (Lewis et al., 2020), first generating two candidate summaries from the model and observing whether a higher probability is assigned to the candidate with a higher ROUGE (Lin, 2004) score. As Tab. 1 shows, the accuracy is far from ideal. This is likely due to the fact that MLE training only encourages the model to assign high probability to the reference summary, and is agnostic about any relative comparison between non-reference summaries. However, we argue that it is also important for the order of model scores to be coordinated with the actual quality metrics by which the summaries will be evaluated -higher model scores should indicate better quality summaries. In the following we will refer to models that have such scores as "coordinated" for conciseness.
BRIO: Bringing Order to Abstractive Summarization
We introduce a training paradigm which requires the abstractive model to be able to be accurate with respect to predicting the tokens in the reference summaries and coordinated with respect to  Figure 1: Comparison of MLE loss (LMLE) and the contrastive loss (LCtr) in our method. MLE assumes a deterministic (one-point) distribution, in which the reference summary receives all the probability mass. Our method assumes a nondeterministic distribution in which system-generated summaries also receive probability mass according to their quality. The contrastive loss encourages the order of model-predicted probabilities of candidate summaries to be coordinated with the actual quality metric M by which the summaries will be evaluated. We assign the abstractive model a dual role -a single model could be used both as a generation model and a reference-free evaluation model.
BRIO: Bringing Order to Abstractive Summarization
the candidate summaries. In other words, we give the abstractive model a dual role: as a generation model, it generates the output summaries in an autoregressive way; as an evaluation model, it can be used to score the quality of candidate summaries by estimating a probability distribution over candidate outputs. The generation model is trained using the standard MLE loss, but to train the evaluation model we introduce a contrastive loss (Hadsell et al., 2006) defined over different candidate summaries generated by pre-trained abstractive models ( Fig. 1), following previous work on ranking-based or contrastive learning (Hopkins and May, 2011;Zhong et al., 2020;Liu et al., 2021b). Our main contribution is to change the target distribution of abstractive models from a one-point deterministic distribution assumed by MLE training to a non-deterministic distribution in which candidate summaries are also assigned probability mass according to their quality. The new SOTA performance on CNN/DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets demonstrated the effectiveness of our method. Our in-depth analysis also found that the abstractive models trained using our method can estimate the candidate summary quality more accurately, in concert with the the objective of our training paradigm.
BRIO: Bringing Order to Abstractive Summarization
The goal of abstractive summarization is to create a function g that takes a source document D and generates an appropriate summary S S ← g(D)
BRIO: Bringing Order to Abstractive Summarization
(1)
BRIO: Bringing Order to Abstractive Summarization
Training Objective Neural abstractive summarization models aim to learn a neural model g that results in good summaries. Maximum likelihood estimation (MLE) is the standard training algorithm. It aims to maximize the likelihood of the reference summary S * , i.e.,
BRIO: Bringing Order to Abstractive Summarization
where θ denotes the parameters of g and p g θ denotes the probability distribution entailed by these parameters. The summation is over the training set and {D (i) , S * (i) } is the i-th training sample. For a specific sample {D (i) , S * (i) }, Eq. 2 is equivalent to minimizing the sum of negative loglikelihoods of the tokens {s * 1 , · · · , s * j , · · · , s * l } in the reference summary S * whose length is l, which is the cross-entropy loss:
BRIO: Bringing Order to Abstractive Summarization
where S * <j denotes the partial reference sequence {s * 0 , · · · , s * j−1 } and s * 0 is a pre-defined start token. p true is a one-hot distribution under the standard MLE framework:
BRIO: Bringing Order to Abstractive Summarization
In practice, label smoothing (Szegedy et al., 2016) is a widely used and effective technique that modifies the target distribution in Eq. 4 to a "soft" label by assigning probability mass β to other tokens:
BRIO: Bringing Order to Abstractive Summarization
where N is the size of the dictionary. Inference and Exposure Bias During inference, the abstractive model g is used to generate the candidate summary in an autoregressive manner. It is intractable to enumerate all the possible candidate outputs, so in practice methods such as beam search are used to reduce the search space.
BRIO: Bringing Order to Abstractive Summarization
One important step in search is estimating the probability of the next word s t given the previous predicted sequence S <t :
BRIO: Bringing Order to Abstractive Summarization
Comparing Eq. 6 with Eq. 3, the major difference is that during inference the model makes new predictions based on its own previous predictions S <t instead of the reference S * <t . As a result, even if the generation model g achieves very high accuracy w.r.t. Eq. 3, once S <t starts to deviate from S * , there is the risk that the performance of g will significantly degrade. This problem has been identified as the exposure bias (Bengio et al., 2015).
BRIO: Bringing Order to Abstractive Summarization
Eq. 6 implies that the abstractive model g should be able to assign higher estimated probability to the better candidate summary during inference. However, this intuition is not directly captured in the standard MLE objective used in training -a model obtaining zero MLE loss would assign zero probability to any candidate summary different from the reference. This is obviously improper for any task where multiple reasonable generations may exist (Khayrallah et al., 2020), and also does not say anything about the ordering of two imperfect references. We therefore advocate for making the alternative assumption that the probability of one candidate should be well-correlated with its quality as evaluated by an automatic metric M . Since it is intractable to enumerate all the possible candidate outputs, we only require our model to be able to accurately predict the ranking order of a set of the most probable candidate summariesŜ, which are its own beam search results. In order to achieve this objective, we slightly modify the conditions of Eq. 5, maintaining the general functional form, but instead specifying the marginal probability of the non-reference candidates S to be β, and encouraging coordination of probabilities and qualities among non-reference candidates as follows:
BRIO: Bringing Order to Abstractive Summarization
We next describe precisely how we encourage coordination through contrastive learning. Contrastive Learning for Coordination The candidate quality measure M can be defined in many ways. In this work we define it as the ROUGE (Lin, 2004) score of a candidate summary S i given the reference summary S * . To coordinate a pre-trained abstractive model, we 1) use it to generate different candidate summaries with various levels of quality, 2 then 2) encourage the model to assign higher estimated probabilities to better candidates by fine-tuning the model with a contrastive loss, following the previous work (Hopkins and May, 2011;Zhong et al., 2020):
BRIO: Bringing Order to Abstractive Summarization
where S i and S j are two different candidate summaries and ROUGE(S i , S * ) > ROUGE(S j , S * ), ∀i, j, i < j. λ ij is the margin multiplied by the difference in rank between the candidates, i.e.,
BRIO: Bringing Order to Abstractive Summarization
where α is the length penalty hyperparameter. This loss gives the abstractive model a dual purpose, first as a reference-free evaluation model, which can be used in a two-stage summarization pipeline, where it is used to score the candidates generated by a pre-trained generation model and select the final output from them. However, since the autoregressive generation depends on both the token-level prediction accuracy and sequencelevel coordination, the model fine-tuned with the contrastive loss alone can no longer be used as a generation model. Multi-task Fine-tuning Following Edunov et al. (2018), we combine the contrastive (Eq. 8) and cross-entropy (Eq. 3) losses to preserve the generation ability of the pre-trained abstractive model:
BRIO: Bringing Order to Abstractive Summarization
where γ is the weight of the contrastive loss. We note that the contrastive and the cross-entropy loss can effectively complement each other -since the contrastive loss is defined on the sequence level, the token-level cross-entropy loss serves as a normalization to ensure that the model could assign balanced probability mass across the whole sequence.
BRIO: Bringing Order to Abstractive Summarization
Datasets We mainly use three datasets in our experiments (statistics in Appendix A). CNNDM 4 (Hermann et al., 2015) is a large scale news dataset. Following Nallapati et al. (2016), we treat the news articles as the source documents and the associated highlights as the summaries. XSum 5 (Narayan et al., 2018) is a highly abstractive dataset of articles from the British Broadcasting Corporation (BBC). NYT 6 (Sandhaus, 2008) contains articles from the New York Times and the associated summaries. We follow Kedzie et al. (2018) for data preprocessing and splitting, and use the associated archival abstracts as the summaries. Baselines We choose a variety of related models with strong performance as baselines.
BRIO: Bringing Order to Abstractive Summarization
BART (Lewis et al., 2020) and PEGASUS  are both large pre-trained Seq2Seq LMs standard in the literature. GSum (Dou et al., 2021) is built on BART, and improves performance by using additional guidance from an extractive summarizer. SimCLS  introduces a two-stage framework where the pre-trained BART model is used to generate candidates and a pre-trained RoBERTa  model is fine-tuned as an evaluation model to score the candidate summaries and select from them. It achieves state-of-the-art performance on both CNNDM and XSum. GOLD (Pang and He, 2021) uses offline reinforcement learning to train the BART model by treating the reference summaries as the demonstrations, a different formulation that can also improve the performance of the original BART. SeqCo (Xu et al., 2021) and ConSum (Sun and Li, 2021) are two recent methods that aim to leverage contrastive learning to improve the performance of the abstractive summarization model (BART). Implementation Details In the following experiments, we use either BART or PEGASUS as a backbone. We label our proposed methods BRIO, with two variants: (1) BRIO-Ctr is fine-tuned with the contrastive loss (Eq. 8) only; (2) BRIO-Mul is fine-tuned with the multi-task loss (Eq. 10). We use BRIO-Ctr as an evaluation model that scores different candidate summaries generated by a Seq2Seq abstractive model and selects the final output from them, and BRIO-Mul as a standard Seq2Seq model that takes the source documents as input and generates the output in an autoregressive manner. Further details are in Appendix B.
BRIO: Bringing Order to Abstractive Summarization
The results are shown in Tab 2. For CNNDM and NYT we use BART as the backbone model while for XSum we use the pre-trained PEGASUS model as our base model since it achieves better performance than BART. We have the following observations:
BRIO: Bringing Order to Abstractive Summarization
(1) BRIO-Ctr outperforms SimCLS, its counterpart as an evaluation model in a two-stage summarization framework. Specifically, both BRIO-Ctr and SimCLS are used to score the candidate summaries generated by a Seq2Seq abstractive model (BART). The final outputs are selected based on those scores. We attribute BRIO-Ctr's superior performance to its use of the same model architecture (BART) for both candidate generation and scoring, while SimCLS uses RoBERTa as the evaluation model. As a result, BRIO-Ctr maximizes the parameter sharing between the two stages, and preserves the power of the Seq2Seq model pre-trained on the same dataset.  Table 2: Results on CNNDM, XSum and NYT. On NYT we only reported our own results due to different data pre-processing. †: significantly better than the baseline model (p < 0.01). *: results reported in the original papers. ‡: results from our own evaluation script. R-1/2/L are the ROUGE-1/2/L F1 scores.
BRIO: Bringing Order to Abstractive Summarization
(2) BRIO-Mul is able to establish the new stare-of-the-art performance on CNNDM. Notably, the previous state-of-the-art model, GSum, takes additional guidance as input and needs a separate encoder to encode the guidance information, while BRIO-Mul uses the same parameterization of BART. Compared to other methods (ConSum, SeqCo, GOLD) that aim to improve upon BART, BRIO-Mul performs much better, showing the effectiveness of our training method.
BRIO: Bringing Order to Abstractive Summarization
(3) Since on XSum we use PEGASUS instead of BART as the base model, the result shows that our method is not restricted to the specific choice of the base model.
BRIO: Bringing Order to Abstractive Summarization
We further perform some in-depth analyses from diverse perspectives on the CNNDM dataset to gain more insights into our proposed method.  Table 3: Model performance with different γ coefficients weighting the contrastive loss (Eq. 10) on CNNDM. BRIO-Ctr is trained with the contrastive loss only, which no longer preserves its generation ability. We report its performance when it is used as an evaluation model to select from candidate summaries. R-1/2/L are the ROUGE-1/2/L F1 scores.   Coefficients of the Multi-Task Loss The multitask loss (Eq. 10) used to train our model contains two parts: the cross-entropy loss and the contastive loss. As shown in Tab. 3, as the weight of the contrastive loss (γ) increases, the model's performance improves. However, the cross-entropy loss is still necessary to preserve the model's ability as a generation model. We argue that this is because the token level accuracy is still important during the autoregressive generation process, where the individual tokens are predicted sequentially. In addition, we also found that the model tends to achieve the best performance (w.r.t the ROUGE scores on the development set) faster with a higher γ. Specifically, it requires less than one entire epoch to achieve the best performance on CNNDM, making our approach an efficient fine-tuning method.   (Stahlberg and Byrne, 2019), and the generator may not be able to differentiate them from high-quality candidates. In Tab. 5, we compare the performance of the pre-trained BART and our model (BRIO-Mul) with different beam widths used during inference. We observe that the performance of BART goes down as the beam width increases. On the other hand, our model is able to achieve better performance with a larger number of beams, demonstrating that our training method can improve the coordination of the model by encouraging the model to assign estimated probabilities to candidate summaries wellcorrelated with their quality. Training with Different Evaluation Metrics In the previous experiments, we used ROUGE as the evaluation metric to define the target ordering of the candidate summaries (Eq.7). To evaluate our method's performance beyond ROUGE,   we use a model-based semantic similarity metric, BERTScore , 7 as the evaluation metric M in Eq.7 to compare the performance of different candidate summaries. Then, we trained another version of BRIO-Mul based on the order of candidate summaries calculated by BERTScore.
BRIO: Bringing Order to Abstractive Summarization
The results in Tab. 6 show that (1) Our model can significantly improve the model performance when either ROUGE or BERTScore is used as the target evaluation metric for ordering candidate summaries. This suggests that it is possible to use our method to optimize any specific target metric, making our method an alternative to reinforcement learning or minimum risk training.
BRIO: Bringing Order to Abstractive Summarization
(2) Our model that is trained on one evaluation metric (e.g. BERTScore) also achieves improvement on another metric (e.g. ROUGE) compared with the baseline model, which indicates that the improvement made by our model is not from exploiting the potential weaknesses of individual metrics. Besides, this result also demonstrates a non-trivial degree of agreement between ROUGE and BERTScore. Novel n-grams We compare the ratio of novel n-grams in reference, BRIO-Mul's, and BART's summaries. As Tab. 7 shows, our model is more "abstractive" compared to BART, although reference summaries still contain more novel n-grams. This is likely due to the fact that our model is optimized at the sequence-level, allowing more freedom for paraphrasing and compression.
BRIO: Bringing Order to Abstractive Summarization
We further investigate the relation of the "abstractiveness" and model performance by com-7 https://github.com/Tiiiger/bert_score. We use its default version for English texts.  paring our model (BRIO-Mul) with the baseline model (BART) on different buckets of test examples grouped by the "novelty" of the reference summaries, 8 i.e.,
BRIO: Bringing Order to Abstractive Summarization
where D and S * are the source document and reference summary respectively, G D and G S * are the sets of bigrams in D and S * , 1 is the indicator function. The results in Fig. 3 show that when novelty is higher, (1) all models' performance decreases;
BRIO: Bringing Order to Abstractive Summarization
(2) our model achieves larger improvement over the baseline model.
BRIO: Bringing Order to Abstractive Summarization
We computed the rank correlation between the estimated probabilities of the candidate summaries calculated by the generators and the quality scores of the candidate summaries. We use Eq. 9 to calculate the estimated probabilities 9 and we use ROUGE-1 as the quality score metric of the candidate summaries. We calculate   Spearman's rank correlation for each sample, and use the average score as the overall correlation, We investigated two specific settings: 1) ranking candidate summaries generated by a different model (PEGASUS); 2) ranking candidate summaries generated by themselves (BART & BRIO-Mul). We use 16 candidates in total for calculation. As Tab. 8 shows, our model achieves better rank correlation on the candidate summaries generated by both itself and the independent model. This suggests that our model can better estimate the quality of candidate summaries.
BRIO: Bringing Order to Abstractive Summarization
Calibration requires that a model's confidence on its predictions is equal to the accuracy of these predictions (Guo et al., 2017). Previous work (Müller et al., 2019;Kumar and Sarawagi, 2019; has found that a more calibrated text generation model tends to have better performance, and techniques like label smoothing can improve both the token-level calibration and sequence-level accuracy (i.e. the ability of generating better results). One intuitive explanation of this phenomenon is to interpret the model's estimated probability of a generated summary as the product of the model's confidences on a series of tokenlevel predictions. Then, since a more calibrated model's confidence estimates better the accuracy of its predictions, the model's estimated probability of one sequence should be more indicative of the quality of this sequence, which is essential for the beam search during inference. However, the relation of token-level calibration and sequencelevel performance remains inconclusive (Müller et al., 2019). 10 For example, a generator that always predicts a uniform distribution over all tokens would be perfectly calibrated, however, such a model would not generate high-quality outputs.
BRIO: Bringing Order to Abstractive Summarization
We investigate this relation from the opposite direction by evaluating whether our model (BRIO-Mul), which is trained to have better sequencelevel performance, would also be more calibrated at the token-level compared with the baseline models that are trained using MLE and label smoothing. We follow previous work by using the Expected Calibration Error (Naeini et al., 2015) (ECE) as the evaluation metric of calibration:
BRIO: Bringing Order to Abstractive Summarization
where the samples are grouped into M equal-width buckets by confidence (conf), B m denotes the m-th bucket, and n is the total number of samples. Following , we evaluate model calibration on the system-generated summaries during inference and use the tercom toolkit 11 to assign labels (correct/incorrect) to the system-generated summaries based on the reference summaries.
BRIO: Bringing Order to Abstractive Summarization
The results in Tab. 9 show that BRIO-Mul is better calibrated compared to BART, suggesting that our method helps to improve the token-level calibration by explicitly encouraging the model to have more accurate sequence-level probability estimations. The reliability graph is shown in Fig. 4. We found that (1) abstractive models are generally over-confident on their own predictions, (2) models are generally more calibrated on XSum than CNNDM. This is likely due to the fact that XSum has shorter summaries therefore it is less likely to be affected by the exposure bias.
BRIO: Bringing Order to Abstractive Summarization
The training paradigm proposed in this paper may be extended to any Seq2Seq model. However, it can be a non-trivial overhead to generate the candidate summaries using large neural models on the entire training set. On the other hand, recent work (Raffel et al., 2020;Schick and Schütze, System Summary Reference chelsea forward tammy abraham nets first-half double for chelsea. dominic solanke adds a third late on as chelsea look set to win trophy. manchester city struggle without injured star thierry ambrose. read: mourinho warns his young chelsea players he can not play them all. click here to read our match report from man city 's academy stadium.
BRIO: Bringing Order to Abstractive Summarization
BART tammy abraham scored twice in the first half to give chelsea the lead. isaac buckley-ricketts levelled the game for manchester city. dominic solanke scored late on to put a gloss on the scoreline. click here to read sportsmail's player ratings from the youth cup final.
BRIO: Bringing Order to Abstractive Summarization
BRIO-Mul chelsea beat manchester city 3-1 in the youth cup final at the etihad stadium. tammy abraham scored twice in the first half to give chelsea the lead. dominic solanke scored late on to seal the win for the home side.
BRIO: Bringing Order to Abstractive Summarization
Reference alejandro valverde won ahead of julian alaphilippe and michael albasini. chris froome finished 123rd after a crash during the final 12 kilometres. team sky's sports director gabriel rasch praised froome for finishing. rasch said froome was 'banged up' but expects to ride tour de romandie.
BRIO: Bringing Order to Abstractive Summarization
BART movistar rider alejandro valverde won fleche wallonne on wednesday. team sky's chris froome fell in the final 12km but finished the race. philippe gilbert pulled out of the race after a bad crash 50km from the end. click here for more cycling news.
BRIO: Bringing Order to Abstractive Summarization
BRIO-Mul alejandro valverde defended his fleche wallonne title in belgium on wednesday. movistar rider finished ahead of julian alaphilippe and michael albasini. team sky's chris froome fell in the final 12km of the race but finished in 123rd. froome was involved in a crash but finished the race despite being 'banged up'
BRIO: Bringing Order to Abstractive Summarization
Reference manuel pellegrini won the premier league and capital one cup last season. city currently sit fourth in the league table -12 points behind chelsea. pellegrini's contract expires at the end of the 2015-16 season. city players have been impressed with vieira's work with the youth team. pep guardiola is city's first-choice to succeed pellegrini at the etihad.
BRIO: Bringing Order to Abstractive Summarization
BART manuel pellegrini's future at manchester city is under scrutiny. patrick vieira is highly-respected among the city players. city's first-choice managerial option is bayern munich boss pep guardiola. click here for all the latest manchester city news. click here for more premier league news.
BRIO: Bringing Order to Abstractive Summarization
BRIO-Mul manchester city players have backed patrick vieira to replace manuel pellegrini as manager of the club. the frenchman is highly-respected among the players at the etihad stadium. pellegrini's future at the club is under scrutiny after a disappointing season. city's first-choice manager is current bayern munich boss pep guardiola.  2021; Fabbri et al., 2021) has shown that few-shot learning can be an effective fine-tuning method of pre-trained models for text generation tasks. Therefore, we investigate our model's performance in a few-shot setting. Specifically, we randomly sample 100/1000 examples from the training set of CNNDM/XSum, and fine-tune the models that are pre-trained using MLE loss on those examples. More training details can be found in Appendix C. The results are shown in Tab. 11. All experiments are repeated three times, and the reported results are the average performance. The results indicate that our model can achieve improvement over the baseline model under the few-shot learning setting with a small computational overhead.
BRIO: Bringing Order to Abstractive Summarization
Tab. 10 presents an interesting pattern we observed when comparing the results of BRIO-Mul and BART, which demonstrates that our method helps the abstractive model to filter out noise patterns in the original data. Specifically, some of the reference summaries (331/11490) in CNNDM contains the phrase "click here", pointing to a hyperlink, and 103 source documents also contain this phrase. BART picked up this pattern, and generates this phrase in 96 output summaries. On the contrary, our model learns to ignore this noise pattern and never generated it across the whole test set, likely because it identified that generated candidates with this pattern rarely achieve a high ROUGE score, and downweighted the probability accordingly.
BRIO: Bringing Order to Abstractive Summarization
In this work, we presented a new training paradigm that assigns candidate outputs probability mass according to their quality using contrastive learning. While our method has achieved significant improvement on abstractive summarization, we note several directions for the future work to explore. First, since our method makes no assumptions specifically about the summarization task, it can be extended to other conditional text generation tasks such as machine translation. Second, it is possible to apply our method in a reinforcement learning setting, where the candidate summaries are dynamically generated. Finally, in experiments we only used diverse beam search to generate the candidate summaries, but it is likely that other candidate generation methods could yield further improvements. 
BRIO: Bringing Order to Abstractive Summarization
We use diverse beam search (Vijayakumar et al., 2018) to generate 16 candidates for each data sample. On CNNDM and XSum, we use the pre-trained BART 12 and PEGASUS 13 models from the Transformers (Wolf et al., 2020) library as the base abstractive models for candidate summary generation and model finetuning respectively. On NYT, we first fine-tuned a BART model 14 with MLE training as the base abstractive model, since our data pre-processing is sightly different from the previous work and there are no available pre-trained checkpoints. We use 4 NVIDIA RTX 3090 GPUs for the model training, and the average running time for one epoch is around 20 hours. We use the Adam optimizer (Kingma and Ba, 2015) with learning rate scheduling for the model training:
BRIO: Bringing Order to Abstractive Summarization
lr = 2 × 10 −3 min(step −0.5 , step · warmup −1.5 )
BRIO: Bringing Order to Abstractive Summarization
where warmup denotes the warmup steps, which is set to 10000, step is the number of updating steps, lr is the learning rate. We set the length penalty factor α in the scoring function (Eq. 9) to the same value as used in the original beam search. We search the value of the margin λ in the contrastive loss (Eq. 8) within the range [1 × 10 −5 , 1], and decide the value based on the model performance on the validation set. We also performed extensive search for the coefficient γ in Eq. 10. The specific hyper-parameter setting is reported in Tab. 13.
BRIO: Bringing Order to Abstractive Summarization
We use the standard ROUGE (Lin, 2004) Perl package 15 for evaluation. The command line parameters are '-c 95 -r 1000 -n 2 -m'. Before the 12 The checkpoint is "facebook/bart-large-cnn", containing around 400M parameters. 13 The checkpoint is "google/pegasus-xsum"" containing around 568M parameters. 14 The checkpoint is "facebook/bart-large". 15 https://github.com/summanlp/evaluation/tree/master/ ROUGE-RELEASE-1.5.5
BRIO: Bringing Order to Abstractive Summarization
Datasets λ (Eq. 8) α (Eq. 9) γ (Eq. 10) CNNDM 0.001 2.0 100 XSum 0.1 0.6 100 NYT 0.001 2.0 100 
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Sequence-to-sequence (Seq2Seq) neural models (Sutskever et al., 2014) have been widely used for language generation tasks, such as abstractive summarization (Nallapati et al., 2016) and neural machine translation . While abstractive models (Lewis et al., 2020;Zhang et al., 2020a) have shown promising potentials in the summarization task, they share the widely acknowledged challenges of Seq2Seq model training. Specifically, Seq2Seq models are usually trained under the framework of Maximum Likelihood Estimation (MLE) and in practice they are commonly trained with the teacher-forcing (Williams and * Corresponding author.  Zipser, 1989) algorithm. This introduces a gap between the objective function and the evaluation metrics, as the objective function is based on local, token-level predictions while the evaluation metrics (e.g. ROUGE (Lin, 2004)) would compare the holistic similarity between the gold references and system outputs. Furthermore, during the test stage the model needs to generate outputs autoregressivelly, which means the errors made in the previous steps will accumulate. This gap between the training and test has been referred to as the exposure bias in the previous work (Bengio et al., 2015;Ranzato et al., 2016).
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
A main line of approaches (Paulus et al., 2018;Li et al., 2019) proposes to use the paradigm of Reinforcement Learning (RL) to mitigate the aforementioned gaps. While RL training makes it possible to train the model with rewards based on global predictions and closely related to the evaluation metrics, it introduces the common challenges of deep RL. Specifically, RL-based training suffers from the noise gradient estimation (Greensmith et al., 2004) problem, which often makes the training un-stable and sensitive to hyper-parameters. Minimum risk training, as an alternative, has also been used in the language generation tasks (Shen et al., 2016;Wieting et al., 2019). However, the accuracy of the estimated loss is restricted by the number of sampled outputs. Other methods (Wiseman and Rush, 2016;Edunov et al., 2018) aim to extend the framework of MLE to incorporate sentence-level scores into the objective functions. While these methods can mitigate the limitations of MLE training, the relation between the evaluation metrics and the objective functions used in their methods can be indirect and implicit.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Among this background, in this work we generalize the paradigm of contrastive learning (Chopra et al., 2005) to introduce an approach for abstractive summarization which achieves the goal of directly optimizing the model with the corresponding evaluation metrics, thereby mitigating the gaps between training and test stages in MLE training. While some related work (Lee et al., 2021;Pan et al., 2021) have proposed to introduce a contrastive loss as an augmentation of MLE training for conditional text generation tasks, we instead choose to disentangle the functions of contrastive loss and MLE loss by introducing them at different stages in our proposed framework.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Specifically, inspired by the recent work of Zhong et al. (2020); Liu et al. (2021b) on text summarization, we propose to use a two-stage model for abstractive summarization, where a Seq2Seq model is first trained to generate candidate summaries with MLE loss, and then a parameterized evaluation model is trained to rank the generated candidates with contrastive learning. By optimizing the generation model and evaluation model at separate stages, we are able to train these two modules with supervised learning, bypassing the challenging and intricate optimization process of the RL-based methods.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Our main contribution in this work is to approach metric-oriented training for abstractive summarization by proposing a generate-then-evaluate twostage framework with contrastive learning, which not only put the state-of-the-art performance on CNN/DailyMail to a new level (2.2 ROUGE-1 improvement against the baseline model), also demonstrates the great potentials of this two-stage framework, calling for future efforts on optimizing Seq2Seq models using methods beyond maximum likelihood estimation.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Given a source document D and a reference sum-maryŜ, the goal of an abstractive summarization model f is to generate the candidate summary S = f (D) such that it receives the highest score m = M (S,Ŝ) assigned by an evaluation metric M . In this work, we break down the holistic generation process into two stages which consist of a generation model g for generating candidate summaries and a evaluation model h for scoring and selecting the best candidate. Fig 1 illustrates the general framework.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Stage I: Candidate Generation The generation model g(·) is a Seq2Seq model trained to maximize the likelihood of reference summaryŜ given the source document D. The pre-trained g(·) is then used to produce multiple candidate summaries S 1 , · · · , S n with a sampling strategy such as Beam Search, where n is the number of sampled candidates.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Stage II: Reference-free Evaluation The highlevel idea is that a better candidate summary S i should obtain a higher quality score w.r.t the source document D. We approach the above idea by contrastive learning and define an evaluation function h(·) that aims to assign different scores r 1 , · · · , r n to the generated candidates solely based on the similarity between the source document and the candidate S i , i.e., r i = h(S i , D). The final output summary S is the candidate with the highest score:
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Here, we instantiate h(·) as a large pre-trained selfattention model, RoBERTa (Liu et al., 2019). It is used to encode S i and D separately, and the cosine similarity between the encoding of the first tokens is used as the similarity score r i .
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Contrastive Training Instead of explicitly constructing a positive or negative example as most existing work with contrastive learning have adopted (Chen et al., 2020;, here the "contrastiveness" is reflect in the diverse qualities of naturally generated summaries evaluated by a parameterized model h(·). Specifically, we introduce a ranking loss to h(·):
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
whereS 1 , · · · ,S n is descendingly sorted by M (S i ,Ŝ). Here, λ ij = (j −i) * λ is the corresponding margin that we defined following Zhong et al. (2020), and λ is a hyper-parameter. 1 M can be any automated evaluation metrics or human judgments and here we use ROUGE (Lin, 2004 
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
We use ROUGE-1/2/L (R-1/2/L) as the main evaluation metrics for our experiments. We also evaluate our model on the recently developed semantic similarity metrics, namely, BERTScore (Zhang et al., 2020b) and MoverScore (Zhao et al., 2019).
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
As the generation model and the evaluation model in our two-stage framework are trained separately, we use pre-trained state-of-the-art abstractive summarization systems as our generation model. Specifically, we use BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2020a) as they are popular and have been comprehensively evaluated.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
For baseline systems, we use the checkpoints provided by the 
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
The results on CNNDM dataset are shown in Tab. 1. We use the pretrained BART 5 as the base generation model (Origin). We use BART, Pegasus, GSum (Dou et al., 2021) and ProphetNet (Qi et al., 2020) for comparison. Notably, the Max oracle which always selects the best candidate has much better performance than the original outputs, suggesting that using a diverse sampling strategy can further exploit the potential power of the pretrained abstractive system. Apart from ROUGE, we also present the evaluation results on semantic similarity metrics. Our method is able to outperform the baseline model on all metrics, demonstrating its improvement is beyond exploiting the potential artifacts of ROUGE. While the scale of improvement is harder to interpret with these metrics, we note that the improvement is able to pass the significance test.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Ref.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
chris ramsey says he has no problem shaking hands with john terry . queens park rangers host chelsea in the premier league on sunday . terry was once banned and fined for racist comments at loftus road . rio ferdinand , brother of anton , will not be fit to play against chelsea .
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
queens park rangers manager chris ramsey has revealed he will have no problem shaking john terry's hand in light of the racist comments the former england captain directed at former rs defender anton ferdinand four years ago . terry , who will line up against ramsey's side , was banned for four games and fined # 220,000 for the remarks made in october 2011 during chelsea's 1-0 defeat at loftus road . but ramsey , the premier league's only black manager , thinks the issue has been dealt with . ... ' i don't know what his feelings are towards me . as long as there wasn't anything on the field that was unprofessional by him , i would shake his hand . . queens park rangers manager chris ramsey speaks to the media on friday ahead of the chelsea match . chelsea captain john terry controls the ball during last weekend's premier league match against stoke . ramsey arrives for friday's pre-match press conference as qpr prepare to host chelsea at loftus road . ' the whole episode for british society sat uncomfortably . it's not something we want to highlight in football . it happened and it's being dealt with . we have to move on . and hopefully everyone has learned something from it . ' . ramsey revealed that rio ferdinand , who labelled terry an idiot for the abuse aimed at his brother , won't be fit in time for a reunion with the chelsea skipper this weekend . but the 52-year-old suspects his player's one-time england colleague will be on the receiving end of a hostile welcome from the home fans on his return the scene of the unsavoury incident . ... ferdinand and terry argue during qpr's 1-0 victory against chelsea at loftus road in october 2011 . rio ferdinand , brother of anton , will not be fit for sunday's match against chelsea .
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
SimCLS queens park rangers host chelsea in the premier league on sunday . qpr boss chris ramsey says he will have no problem shaking john terry's hand . terry was banned for four games and fined # 220,000 for racist comments . rio ferdinand , brother of anton , will not be fit for the match at loftus road .
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Origin. john terry was banned for four games and fined # 220,000 for the remarks made in october 2011 during chelsea's 1-0 defeat at loftus road . terry will line up against chris ramsey's side on sunday . rio ferdinand , who labelled terry an idiot for the abuse aimed at his brother , won't be fit in time for a reunion with the chelsea skipper this weekend .  With the constraints of computation power, we try to use as many candidates as possible for the evaluation model training. However, we also notice that our method is robust to the specific number of candidates, as during test we found that our model is still able to outperform the baseline model with fewer candidates, which is illustrated in Fig. 2.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
To demonstrate that our method is able to make meaningful improvement w.r.t the summary quality, here we compare our method with the baseline model at different semantic levels on CNNDM. , we compare the model performance w.r.t the salient entities, which are entities in source documents that appear in the reference summaries. Specifically, (1) we extract the entities from the source documents, 6 (2) select the salient entities based on the entities in reference summaries,
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
(3) compare the salient entities with entities in candidate summaries. Results in Tab. 3 demonstrate that our method can better capture the important semantic information of the source documents.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Sentence Alignments Here we investigate if our method makes sentence-level differences compared to the baseline model. Specifically, (1) we match each sentence in the summaries to a sentence in the source documents based on their similarity (indicated by ROUGE scores), 7 (2) compute the sentence-level similarity between the reference and system-generated summaries based on the overlaps of their matched sentences in the source documents. The results in Tab. 3 demonstrate that the generated summaries of our method is more similar to the reference summaries at the sentence level.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Positional Bias In Tab. 2, we present a case study of the sentence alignment. We use the same matching approach to map the summary sentences to the sentences in source articles. In this example, the output of our method focuses on the same sentences as the reference summary does, while the baseline summary focuses on some different sentences.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Interestingly, the reference summary focuses on the very last sentence in the article, and our method can follow this pattern. Upon examining this pattern, we notice a positional bias of abstractive models when handling long source articles (more than   Fig. 3 shows that the baseline summaries are more likely to focus on the head sentences compared to the references, which may result from the autoregressive generation process of the Seq2Seq models. Our method is able to mitigate this bias, as the candidate sampling process (diverse beam search) generates candidates different from the original outputs, and our evaluation model can assess the holistic quality of the candidates.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
To evaluate our method's performance beyond CNNDM dataset, we also test our method on XSum dataset, and the results are shown in Tab. 4. Here, we use Pegasus 8 as the base system since it achieves better performance than BART on XSum. We follow the same sampling strategy to generate the training data. However, as this strategy generally results in lower ROUGE-2 score on XSum dataset, we use a different strategy to generate the validation and test data (4 candidates generated by 4 diverse groups). Our method is still able to outperform the baseline, but with a smaller margin compared to CNNDM. Summaries in XSum are shorter (one-sentence) and more abstractive, which restricts the semantic diversity of candidates and makes it harder to make meaningful improvement.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
In this work, we present a contrastive summarization framework that aims to optimize the quality of generated summaries at summary-level, which mitigates the discrepancy between the training and test 8 'google/pegasus-xsum'  stages in the MLE framework. Apart from the significant improvement over the baseline model on CNNDM dataset, we present a comprehensive evaluation at different semantic levels, explaining the sources of the improvement made by our method. Notably, our experimental results also indicate that the existing abstractive systems have the potential of generating candidate summaries much better than the original outputs. Therefore, our work opens up the possibility for future directions including (1) extending this two-stage strategy to other datasets for abstractive models;
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
(2) improving the training algorithms for abstractive models towards a more holistic optimization process.  The source documents and reference summaries are lower-cased. Due to the input length limitation, some source documents are truncated during training.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Candidate Generation We use diverse beam search to generate the candidate summaries. We use the same beam search configuration as the original work except those related to diverse beam search. In particular, the diversity penalty is set to 1, and we use 16 diversity groups with 16 beams, which results in 16 candidates. Model We use the pretrained RoBERTa with 'roberta-base' version provided by the Transformers library as our evaluation model, which contains 125M parameters. Optimizer We use Adam optimizer with learning rate scheduling: lr = 0.002 · min(step num −0.5 ,
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
step num · warmup steps −1.5 ),
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
where the warmup steps is 10000.
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
Training details The batch size in our experiments is 32. We evaluate the model performance on the validation set at every 1000 steps, using the averaged ROUGE-1/2/L score as the selecting criteria. The training is converged in 5 epochs, which takes around 40 hours on 4 GTX-1080-Ti GPUs on CNN/DailyMail dataset and 20 hours on XSum dataset.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
In recent years, sequence-to-sequence neural models have enabled great progress in abstractive summarization (See et al., 2017;Lin et al., 2021). In the news domain, they have surpassed the strong LEAD-3 extractive baseline. With the rise of transfer learning since BERT (Devlin et al., 2019), leading approaches typically fine-tune a base pretrained model that either follows a general text generation training objective like T5 (Raffel et al., 2019), BART (Lewis et al., 2020), ERNIE (Zhang et al., 2019b) and ProphetNet (Qi et al., 2021), or an objective specifically tailored for summarization like in PEGASUS (Zhang et al., 2020).
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Most of these sequence-to-sequence models are history-based, where an output sequence is represented as a sequence of decisions and the probabil-* Equal contribution.  ity of the sequence is computed as a product of decision probabilities. This is also known as the autoregressive factorization. To transform the sequence of probabilities into summaries, beam search is commonly used. While auto-regressive decoding with beam search is simple and has many advantages, it can be difficult to encode global constraints such as grammaticality, coherence and factual consistency within this framework, properties that are believed to be useful in discriminating among candidate outputs. If the model starts decoding in a bad direction, mistakes might propagate, carry over the mistake of previous tokens to the generation of new ones, and the model has no way to know that it should adjust the decoding. Furthermore, these models are typically trained with teacher forcing (Williams and Zipser, 1989), which leads to an inherent discrepancy between training time and inference time known as the exposure bias problem (Bengio et al., 2015;Sun and Li, 2021). Decoding methods such as beam search maintain a list of top-k best candidates, and output a single best one. In the case of beam search, candidates are sorted by decreasing log-probability, and the last (k − 1) hypotheses are discarded. However, these (k − 1) other hypotheses often contain considerably better sequences in terms of different evaluation measures. This observation holds over other decoding methods: diverse beam search (Vi-jayakumar et al., 2016), top-k sampling (Fan et al., 2018) and top-p sampling (Holtzman et al., 2019). In Table 1, we illustrate this phenomenon with the oracle scores (maximum scores over the pool of candidates) for four popular decoding methods and five metrics on the CNN-DailyMail (Hermann et al., 2015) dataset with a PEGASUS model. The oracle ROUGE-1 scores are up to 10 points higher (+22.8%) than the top beam baseline. Moreover, oracle gains significantly increase when mixing several generation methods together, reaching an improvement of more than 13 ROUGE-1 points (+30.5%). Such a gap is larger than the progress made by research in the whole field of neural abstractive summarization in the last five years (Nallapati et al., 2016;Dou et al., 2021). This suggests that current abstractive models are not exploited to their full capacity, calling for better methods to identify the best summary candidate.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Given this assessment, we investigate whether it is possible to train a second-stage summarization model which learns to select the best summary among a set of candidates obtained from a base model and with a decoding process, which itself can potentially involve a set of decoding methods (e.g., beam search variants). This way, the model would recover the gap that separates it with the oracle. This raises the question of what makes a summary candidate the optimal one? Admittedly, summarization has been an underconstrained task and its evaluation is complex and remains an active research area (Kryscinski et al., 2019;Fabbri et al., 2021;Koto et al., 2021). To build a flexible approach, we use a multi-task learning framework based on a mixture-of-experts architecture in order to optimize jointly over several measures.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
To design a robust re-ranker, we systematically explore the dimensions of summary re-ranking: base model, decoding process, and evaluation measure. Our system, named SummaReranker, is flexible and multi-task: it can be trained with any set of evaluation metrics. It is considerably less computationnally expensive to train than the singlestage summarization models that it is plugged on. We apply our system across three different datasets {CNN-DailyMail, XSum, Reddit TIFU} and two base models {PEGASUS, BART}. Optimizing ROUGE metrics leads to relative performance improvements from 1.31% to 9.34% depending on the dataset. It outperforms recently proposed second-stage summarization approaches RefSum (Liu et al., 2021) and SimCLS (Liu and Liu, 2021) and sets a new state-of-the-art on CNN-DailyMail and XSum (Narayan et al., 2018). We present extensive quantitative results coupled with a qualitative human evaluation.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Our approach follows the paradigm of second-stage models. Specifically, given a source document S, a base model B, and a set of decoding methods D, we get a pool of m summary candidates C = {C 1 , . . . , C m }. Given an evaluation metric µ in a set of metrics M, we get associated scores for each candidates S µ = {µ(C 1 ), . . . , µ(C m )}. Our goal is to train a model f θ parameterized by θ to explicitly identify the best summary candidate C * µ according to the metric, which is given by:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We frame this problem as a binary classification. C * µ is the positive candidate, while other candidates are treated as negative. For a metric µ, the re-ranker f θ is trained with a binary cross-entropy loss:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
µ , otherwise y i = 0. Binary classification has been successfully employed for re-ranking in prior work (Nallapati, 2004;Nogueira and Cho, 2019). While multi-way classification could be an alternative, we noticed that for each generation method, a significant fraction of candidates share the same score for one or several metrics, while it is rare that all candidates share the same score (Appendix C-D). Thus, there is not enough signal to distinguish m candidates into m different classes, but enough for two classes.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
To optimize for N different metrics M = {µ 1 , . . . , µ N } simultaneously, we use a separate prediction head (tower) for each and we minimize the average over metric losses defined as:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We first need to get a good representation of the summary candidate. To use contextual information, we concatenate the source with the candidate, Once we have a joint representation of the source with the candidate (noted x), we perform multi-task learning in order to optimize for the desired metrics. Since metrics are different, yet may be strongly correlated (e.g., ROUGE variants), we adopt a mixtureof-experts (MoE) architecture. In particular, we follow the sparse MoE approach (Shazeer et al., 2017), which introduces experts dropout. To adapt it to multi-task training, we use the multi-gate approach proposed in Zhao et al. (2019b). Given E experts E 1 , . . . , E E and N prediction towers T 1 , . . . , T N , the prediction for an input summary representation x for a metric µ indexed by k ∈ {1, . . . , N } is:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
where W k is the weight matrix associated with gate k. The corresponding prediction probability is:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Experts are shared across all tasks, and through the softmax gates the model learns how much weight to assign to each expert for each task. Our SummaReranker model architecture is shown in Fig. 1. In practice, the shared bottom MLP consists in two fully-connected layers with ReLU activation (Glorot et al., 2011). Each expert E i is also a two-layer MLP with ReLU, and each prediction tower T k is a single-layer MLP. We set the number E of experts to be equal to twice the number of tasks (N ), and the experts dropout to 50%, so that the effective number of experts being used during training matches N . Our model has 370.09 million trainable parameters, representing a slight 4.14% increase due to the mixture-of-experts compared to the off-the-shelf RoBERTa-large.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Second-stage learning approaches may suffer from an inherent distribution bias. Indeed, the base model has a different output distribution on the training set than on the validation and test sets. Thus, it is ineffective to train a second-stage model on the training set outputs of the base model.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
To resolve this distribution shift, we shuffle the training set and randomly split it into equal parts, then fine-tune a pre-trained model on each half. Then, to build a training set for the re-ranker, we infer with each model on the half that it was not trained on. At testing time, we face two options:
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
• Base setup: in this setup, we infer on the test set with one of the two base models trained on half the training set, then apply the re-ranker.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Since the base models are trained on less data, their performance on the test set worsens. However, we will show that SummaReranker brings improvements which more than compensate this performance drop.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
• Transfer setup: this setup consists in applying SummaReranker on top of a base model trained on the whole training set. Note that SummaReranker is still trained in the same fashion as before. There could be a distribution mismatch in this setting too, since SummaReranker needs to rank summary candidates of a potentially higher quality (generated by a model trained on the full data) than the summaries that it was trained on  (generated by a model trained on half the data). Nevertheless, SummaReranker still transfers well and considerably improves the performance of the base model in this transfer setup.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
If D is made of multiple decoding methods {δ 1 , ..., δ j }, each producing several candidates, the overall candidate set may be large, slowing down inference. Thus, to explore lower-resource inference setups, we separate the sets of decoding methods D train and D test used for training and inference, respectively, and enforce that D test ⊂ D train .
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Throughout our experiments, we vary all the three dimensions of our re-ranking framework: the base model B, the set of decoding methods D and the set of scoring metrics M.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
As base models, we use PEGASUS (Zhang et al., 2020) and BART (Lewis et al., 2020), each one in their large version, as they are leading summarization models with publicly available checkpoints. We obtain pre-trained and fine-tuned checkpoints from the HuggingFace transformers library (Wolf et al., 2020).
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
For decoding methods (D), we experiment with beam search (referred to as 1), diverse beam search (2), top-k sampling (3) and top-p sampling (4). For each decoding method, we set the number of candidates to 15, as it is close to the maximum which could fit in a standard 11GB RAM GPU when doing generation with PEGASUS-large.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
As set of metrics, we first use ROUGE (Lin and Hovy, 2003), in its commonly used three flavours of ROUGE-1 (noted R-1), ROUGE-2 (noted R-2)  and ROUGE-L (noted R-L) for summarization evaluation. We also leverage recently introduced model based evaluation methods BERTScore (noted BS) (Zhang et al., 2019a) and BARTScore (noted BaS) (Yuan et al., 2021), which both rely on contextual word embeddings from pre-trained language models. Thus, our total set of metrics is M = {R-1, R-2, R-L, BS, BaS}. As seen in Table 2, R-1 and R-L are strongly correlated (Pearson correlation score of 0.977). BARTScore is the least correlated to other metrics, suggesting that it captures aspects complementary to the other four. We train SummaReranker on the following datasets, covering multiple domains: As in other summarization works (Zhang et al., 2020), we use the TIFU-long subset, containing 37k posts. As there is no official split, we build a random 80:10:10 split for training:validation:test.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We refer to Table 3 for statistics on each dataset.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
To help the model better discriminate between candidates, we found that sampling was useful.  This means that at training time, the model only sees two candidates per data point. We scale the pool of candidates that these two are sampled from to four decoding methods, totalling 60 summary candidates per source document. We train SummaReranker for five epochs. We use the Adafactor optimizer (Shazeer and Stern, 2018), with maximum learning rate 1e-5, warming up the learning rate linearly over the first 5% training steps. Training on CNN/DM takes four days on a single RTX 2080 Ti GPU.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
For inference, we need to output a single candidate. After getting predicted probabilities across each metric µ ∈ M, we output the candidate maximizing the sum of predicted probabilities. Note that relaxing inference to allow for a different best candidate for each metric would improve performance, but is not practical. We perform inference with the model checkpoint maximizing the sum of the scores for the metrics on the validation set.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
First, we investigate how our model performs in the base setup described in §3. We apply SummaReranker on top of PEGASUS and BART models fine-tuned on each half. For each model, we decode using beam search (1) and diverse beam search (2). The latter performs better for PEGASUS, while the former is better for BART. We then apply SummaReranker optimized jointly for R-1, R-2, and R-L on  top of each of the two base models, for each decoding method, and finally when using both decoding methods. Results are shown in Table 4. SummaReranker improves a base PEGASUS by 4.57% to 7.21% with 15 candidates, and 8.70% to 9.36% with 30 candidates. With BART, Sum-maReranker improves by 3.94% to 11.65% with 15 candidates, and 7.98% with 30 candidates. When using several decoding methods, we compare the reranker performance with the best baseline among decoding methods. Notably, with SummaReranker, PEGASUS and BART models trained on 50% of the training set now surpass their counterparts trained on the whole training set, achieving 46.19 R-1 with PEGASUS and 45.96 R-1 with BART. This is better than GSum (Dou et al., 2021), the best reported summarization model on CNN/DM.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Next, we look at how SummaReranker performs in the transfer setup. That means, we apply it on top of PEGASUS and BART models fine-tuned on the entire dataset, using public checkpoints. We also include R3F (Aghajanyan et al., 2020) and GSum (Dou et al., 2021) in our single-stage model comparison. In terms of second-stage approaches, we compare SummaReranker with RefSum (Liu et al., 2021) and SimCLS (Liu and Liu, 2021). Note that SummaReranker is trained as usual, on the outputs of two base models each trained on 50%.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We first optimize for ROUGE metric {R-1, R-2, R-L} with multi-task training on CNN/DM (Table 5). With two decoding methods, PEGASUS + SummaReranker sets a new state of the art on CNN/DM with 47.16 R-1, 22.55 R-2 and 43.87 R-L, corresponding to gains of 2.60/1.65/2.29 R-1/2/L or +5.44% from our diverse beam search baseline. As expected, the relative gains in transfer setup are lower than in base setup. Next, we optimize modelbased metrics, and note the difficulty in improving BERTScore, compared to BARTScore. Optimizing jointly ROUGE and model-based metrics improves all metrics, but does not match the results when training only ROUGE. Interestingly, performance gains saturate when adding two extra decoding methods (top-k and top-p sampling), despite gains in the oracle scores observed in Table 1.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
To assert statistical significance of performance gains, we perform a t-test between SummaReranker scores and scores from the base model with each of the decoding methods being used, and mark with † results where the p-value is smaller than 0.05 for all these decoding methods.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We also show experts utilization (obtained with  softmax weights from the gates) for the model optimized on all five metrics in Fig. 2. Notably, some experts specialize in certain metrics (for instance, expert 0 on R-2 and expert 4 on R-L).
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Then, we apply SummaReranker on XSum and Reddit TIFU, as shown in Table 6. We train SummaReranker using the three ROUGE metrics {R-1, R-2, R-L} as objective, and decoding methods {beam search, diverse beam search} to generate the candidates. On XSum, SummaReranker improves a base PEGASUS with beam search candidates by 1.31%, setting a new state-ofthe-art of 48.12/24.95/40.00 R-1/2/L. On Reddit TIFU, we improve a base PEGASUS with beam search and diverse beam search (30 candidates) by 9.34%, reaching 29.83/9.50/23.47 R-1/2/L, and a base BART with beam search by 4.22%, reaching 28.99/9.82/22.96 R-1/2/L. Across datasets, training on a combination of beam search and diverse beam search candidates is consistently effective.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Beyond summary properties, we investigate the performance of re-ranking itself with rank-based evaluation measures. A perfect re-ranker should always single out the best summary from the rest, yielding oracle results. To evaluate how SummaReranker ranks the best summary, we compute the best summary candidate recall at different thresholds. Since several candidates might get the same metric scores (Appendix C), the best candidate recall at threshold k for the random uniform ranking baseline is not the standard R@k = k m anymore    (6) where m best is the number of best candidates. Following Fig. 3, a PEGASUS with diverse beam search ranking of summary candidates (dashed lines) is not significantly better than the corresponding random baseline from eq. (6) (dotted lines) on CNN/DM and Reddit TIFU. However, it improves on it on XSum, confirming the observation made in Table 6 that it is harder to train a re-ranker on this dataset. On all three datasets, SummaReranker (solid lines) significantly pushes the recall at all thresholds. We note +14.90 absolute recall@5 improvement on CNN/DM (50.84 versus 35.94, indicated by the black arrow), +9.54 on XSum and +5.23 on Reddit TIFU.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Lastly, we demonstrate that re-ranking improvements in quantitative metrics also translate to qualitatively better summaries. Fig. 4 shows an example of summary selected by SummaReranker, alongside its source document, ground-truth (reference) summary and output from the base model. Sum-maReranker is able to include a whole sentence which was missed by the base summary. We refer to Appendix K for full re-ranking demonstrations on each of the three datasets.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We also conduct a human evaluation. We asked three different humans to evaluate 50 randomly sampled test summaries for each dataset. Human raters were graduate students with professional English proficiency (TOEFL scores above 100 out of 120). Humans were shown the source document alongside the top beam search summary from PEGASUS, and the corresponding summary candidate selected by SummaReranker. They were asked to choose which one they believe is more faithful. They could choose a tie, because in some cases the base summary and the re-ranked one are very similar, or even identical (Appendix I). In Fig. 5, we see that on average, humans are more likely to pick the SummaReranker candidate.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Abstractiveness Given that we are not modifying the base model nor its training procedure, we analyze whether our re-ranking system favors more abstractive candidates. In Fig. 6, we display the percentage of novel n-grams for n in {1,2,3,4}, for a base PEGASUS with beam search (blue) and diverse beam search (purple) decoding, and when adding SummaReranker in both cases (green and red, respectively). As first raised in (See et al., 2017), summary candidates are much less abstractive than ground truth summaries on CNN/DM. Yet, our re-ranker selects more abstractive candidates according to all n-grams metrics, even more so with diverse beam search, which is already more abstractive than beam search. This observation also holds on Reddit TIFU and XSum (other than 1-grams). XSum summary candidates are already almost as abstractive as the ground truth and it is harder to obtain significant abstractiveness gains through our re-ranking.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Speed/Performance trade-off On top of base model training and candidate generation, SummaReranker inference cost is linear in the number of candidates. A single candidate takes on average 38ms to be scored. As seen in Table 5 and Table 6, the performance gains from mixing several decoding methods to generate summary candidates are not scaling consistently (all four decoding methods are not better than just beam search and diverse beam search). To provide more insights on the speed/performance trade-off, we show in Appendix J SummaReranker performance when randomly sub-sampling k ∈ {1, . . . , 15} candidates. On CNN/DM, re-ranking as few as two candidates is sufficient to improve on the baseline PEGASUS. On XSum, it needs three to eight, and on Reddit TIFU three to four. As a rule of thumb, it is better to score all candidates when possible, but six to eight candidates provide a good trade-off between speed and performance across datasets.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Further Work To encode the source jointly with the summary candidate, we need to truncate the source to a fixed number of tokens. Thus, we are limited by the maximum context window of the language model encoder (512 in the case of RoBERTa-large). Applying SummaReranker to long-document summarization, such as scientific articles summarization (Cohan et al., 2018) would need better long-range modeling. In §3, we weighted metric-dependent losses uniformly. We leave to further work the exploration of more complex weight balancing or multi-task learning objectives (Lin et al., 2019).
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
We introduced SummaReranker, the first multi-task re-ranking framework for abstractive summarization. Encoding the source with the candidate, our model predicts whether the summary candidate maximizes each of the metrics optimized for. Sum-maReranker works well across diverse datasets, models, decoding methods and summarization evaluation metrics. Summaries selected by SummaReranker improve the ROUGE state-of-the-art on CNN/DM and XSum. In addition, we also show that they are more abstractive and more likely to be preferred by human evaluators over base model outputs.     Observations from Table 9 and Table 10 are consistent with the ones made in Table 1: oracle scores are widely above the top beam baseline, and keep increasing when mixing several decoding methods.  In Table 11, BERTScore (BS) and BARTScore (BaS) have results closer to 15, indicating that it is unlikely that two summary candidates share the exact metric score. This is understandable given that both these metrics are based on embeddings from pre-trained language models (BERT and BART, respectively), and embeddings values will vary whenever the input text is different, making it unlikely to have two candidates collude on the same score. In contrast, ROUGE measures n-gram overlaps, and two different summary candidates might get the same ROUGE score with the target summary (for instance if they only differ by n-grams not present in the target).  We note that cases where all scores are identical are a small minority. ROUGE-2 is more likely than other metrics to lead to such a scenario of all identical scores.   Metrics correlation from Table 13 and Table 14 follow the same pattern as in Table 2 Tables Table 15 and Table 16 complement the base setup results exposed in Table 4.    Table 19: Re-ranking overlap with base and best candidates. Fraction of time that the re-ranked summary coincides with the base model one (left), and one of the best ones (oracle scores) among generated candidates (right). SR is SummaReranker.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
In Table 19, we observe that SummaReranker is more likely to stick to the base model candidate with diverse beam search. Results in bold represent the most ideal scenario: SummaReranker differs the most from the base setup (lowest scores of the left column), and matches the most one of the best candidates (highest scores of the right column). In Fig. 8, we observe a failure mode of SummaReranker: on XSum and with PEGASUS when training the re-ranking with beam search candidates, performance decreases. However, the problem vanishes when SummaReranker is trained on a mixture of beam search and diverse beam search candidates. Fig. 9 top left (PEGASUS with beam search) represents a curious case: re-ranking a single candidate is better than the top beam baseline. Since re-ranking a single candidate is equivalent to randomly sampling one candidate, this means that the top beam baseline is on average lower than sampling a random candidate. We observed that such cases are rare and usually the top beam baseline is better than the random baseline. When the top beam baseline is lower, it is of utmost importance to keep all candidate and use a second-stage method to identify a better one.  
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Source Is this confirmation that Angel Di Maria is happy as a Manchester United player? The 27-year-old has endured a mixed start to his United career on-and-off the pitch since joining the club last summer -which has included an attempted burglary at his family home in Cheshire back in February. The midfielder has been linked with a move away from Old Trafford as a result, but speculation about his future could be squashed following his latest tattoo. Angel Di Maria (left) has a new No 7 tattoo which stands out among others on his left arm . Di Maria wears the No 7 shirt at Manchester United following his £60million from Real Madrid last summer. A new picture has been revealed on Twitter of Di Maria's latest piece of body art -the number seven which stands out strongly among others on his left arm. United's club record £60million signing of course adorns the No 7 shirt at the Red Devils -so could his latest tattoo suggest he's committed to Louis van Gaal's side for the long haul? However, before United fans get too carried away it must be noted that the former Real Madrid star does also wear the No 7 jersey for Argentina too. As well as adorning the No 7 shirt at United, 27-year-old (right) also wears that number for Argentina too.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Summary Angel Di Maria has revealed his latest tattoo on Twitter. The 27-year-old has the No 7 shirt at Manchester United on his left arm. The Argentine has endured a mixed start to his United career. He has been linked with a move away from Old Trafford as a result. 