What are different methods proposed for controllable summarization?
Controllable Summarization Controllable text
generation (Hu et al., 2017) aims to control prop-
erties of the generated text including style (Kumar
et al., 2021), length, or content (Fan et al., 2018;
He et al., 2020). Approaches for content control
vary according to the type of control: keywords (He
et al., 2020), entities (Narayan et al., 2021), ques-
tions (Vig et al., 2022), factoid question-answer
pairs (also called QA blueprints) (Narayan et al.,
2022). As opposed to methods like GSum (Dou
et al., 2021), which insert control tokens on the
encoder side, we focus on decoder-based meth-
ods which do not require re-encoding the docu-
ment when the control sequences are updated. In
summarization, these controls can broadly indicate
the information to summarize, like the questions
in query-focused summarization, or provide a de-
tailed plan of the text to be generated, like the entity
chains. While these types of control are not typi-
cally studied together we show that our SOCRATIC
pretraining provides benefits across the board for
both high-level and finegrained queries and plans.
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains -- ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of Rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.
[{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Joshua","last":"Maynez","middle":[]},{"first":"Gonccalo","last":"Simoes","middle":[]},{"first":"Vitaly","last":"Nikolaev","middle":[]},{"first":"Ryan","last":"McDonald","middle":[]}]
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.
[{"first":"Artidoro","last":"Pagnoni","middle":[]},{"first":"Alex","last":"Fabbri","middle":[]},{"first":"Wojciech","last":"Kryscinski","middle":[]},{"first":"Chien-Sheng","last":"Wu","middle":[]}]
Controllable Abstractive Summarization
Current models for document summarization disregard user preferences such as the desired length, style, the entities that the user might be interested in, or how much of the document the user has already read. We present a neural summarization model with a simple but effective mechanism to enable users to specify these high level attributes in order to control the shape of the final summaries to better suit their needs. With user input, our system can produce high quality summaries that follow user preferences. Without user input, we set the control variables automatically – on the full text CNN-Dailymail dataset, we outperform state of the art abstractive systems (both in terms of F1-ROUGE1 40.38 vs. 39.53 F1-ROUGE and human evaluation.
[{"first":"Angela","last":"Fan","middle":[]},{"first":"David","last":"Grangier","middle":[]},{"first":"Michael","last":"Auli","middle":[]}]
GSum: A General Framework for Guided Neural Abstractive Summarization
Neural abstractive summarization models are flexible and can produce coherent summaries, but they are sometimes unfaithful and can be difficult to control. While previous studies attempt to provide different types of guidance to control the output and increase faithfulness, it is not clear how these strategies compare and contrast to each other. In this paper, we propose a general and extensible guided summarization framework (GSum) that can effectively take different kinds of external guidance as input, and we perform experiments across several different varieties. Experiments demonstrate that this model is effective, achieving state-of-the-art performance according to ROUGE on 4 popular summarization datasets when using highlighted sentences as guidance. In addition, we show that our guided model can generate more faithful summaries and demonstrate how different types of guidance generate qualitatively different summaries, lending a degree of controllability to the learned models.
[{"first":"Zi-Yi","last":"Dou","middle":[]},{"first":"Pengfei","last":"Liu","middle":[]},{"first":"Hiroaki","last":"Hayashi","middle":[]},{"first":"Zhengbao","last":"Jiang","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
CTRLsum: Towards Generic Controllable Text Summarization
Current summarization systems yield generic summaries that are disconnected from users’ preferences and expectations. To address this limitation, we present CTRLsum, a generic framework to control generated summaries through a set of keywords. During training keywords are extracted automatically without requiring additional human annotations. At test time CTRLsum features a control function to map control signal to keywords; through engineering the control function, the same trained model is able to be applied to control summaries on various dimensions, while neither affecting the model training process nor the pretrained models. We additionally explore the combination of keywords and text prompts for more control tasks. Experiments demonstrate the effectiveness of CTRLsum on three domains of summarization datasets and five control tasks: (1) entity-centric and (2) length-controllable summarization, (3) contribution summarization on scientific papers, (4) invention purpose summarization on patent filings, and (5) question-guided summarization on news articles. Moreover, when used in a standard, unconstrained summarization setting, CTRLsum is comparable or better than strong pretrained systems.
[{"first":"Junxian","last":"He","middle":[]},{"first":"Wojciech","last":"Kryscinski","middle":[]},{"first":"Bryan","last":"McCann","middle":[]},{"first":"Nazneen","last":"Rajani","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]}]
Exploring Neural Models for Query-Focused Summarization
Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. While recently released datasets, such as QMSum or AQuaMuSe, facilitate research efforts in QFS, the field lacks a comprehensive study of the broad space of applicable modeling methods. In this paper we conduct a systematic exploration of neural approaches to QFS, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models. Within those categories, we investigate existing models and explore strategies for transfer learning. We also present two modeling extensions that achieve state-of-the-art performance on the QMSum dataset, up to a margin of 3.38 ROUGE-1, 3.72 ROUGE2, and 3.28 ROUGE-L when combined with transfer learning strategies. Results from human evaluation suggest that the best models produce more comprehensive and factually consistent summaries compared to a baseline model. Code and checkpoints are made publicly available: https://github.com/salesforce/query-focused-sum.
[{"first":"Jesse","last":"Vig","middle":[]},{"first":"Alexander","last":"Fabbri","middle":["R."]},{"first":"Wojciech","last":"Kry'sci'nski","middle":[]},{"first":"Chien-Sheng","last":"Wu","middle":[]},{"first":"Wenhao","last":"Liu","middle":[]}]
Conditional Generation with a Question-Answering Blueprint
The ability to convey relevant and faithful information is critical for many tasks in conditional generation and yet remains elusive for neural seq-to-seq models whose outputs often reveal hallucinations and fail to correctly cover important details. In this work, we advocate planning as a useful intermediate representation for rendering conditional generation less opaque and more grounded. Our work proposes a new conceptualization of text plans as a sequence of question-answer (QA) pairs. We enhance existing datasets (e.g., for summarization) with a QA blueprint operating as a proxy for both content selection (i.e.,~what to say) and planning (i.e.,~in what order). We obtain blueprints automatically by exploiting state-of-the-art question generation technology and convert input-output pairs into input-blueprint-output tuples. We develop Transformer-based models, each varying in how they incorporate the blueprint in the generated output (e.g., as a global plan or iteratively). Evaluation across metrics and datasets demonstrates that blueprint models are more factual than alternatives which do not resort to planning and allow tighter control of the generation output.
[{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Joshua","last":"Maynez","middle":[]},{"first":"Reinald","last":"Amplayo","middle":["Kim"]},{"first":"Kuzman","last":"Ganchev","middle":[]},{"first":"Annie","last":"Louis","middle":[]},{"first":"Fantine","last":"Huot","middle":[]},{"first":"Anders","last":"Sandholm","middle":[]},{"first":"Dipanjan","last":"Das","middle":[]},{"first":"Mirella","last":"Lapata","middle":[]}]
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Planned and Grounded Abstractive Summarization We introduce a novel training objective to neural summarization models for content planning with entity chains, and study integrating it with supervised finetuning and self-supervised pretraining objectives without altering the models themselves. As the entity chains are extracted from the reference summaries during training, our models learn to ground the generation of summaries to the entity chains found in them. Hence, we refer to this objective by FROST for its ability to try to "FReeze entity-level infOrmation in abstractive SummarizaTion with planning."
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Abstractive Summarization with Entity Chains FROST provides a very effective knob for entity-level content modification in abstractive summaries. In this paper we empirically demonstrate how FROST is critical for faithfulness by enabling the drop-prompt mechanism where we drop out hallucinated entities from the predicted content plan and prompt the decoder with this modified plan to generate faithful summaries. We further qualitatively demonstrate that FROST enables generation of summaries (i) with topical diversity by choosing different sets of entities from the source to plan what we want to discuss in the summary, and (ii) with style diversity by reordering entities in the predicted plan to get an equivalent summary but with a different entity emphasis.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a new training objective for encoderdecoder generative models to do content planning while summarizing.
Planning with Learned Entity Prompts for Abstractive Summarization
Model Formulation. Let d be an input document, we aim to teach our model to first generate a content plan c for the summary s as p(c|d), and then generate the summary s as p(s|c, d). We define the ordered chain of entities observed in the summary s as its content plan. Instead of modeling p(c|d) and p(s|c, d) separately, we take a simpler approach, we train an encoder-decoder model to encode the document d and generate the concatenated content plan and summary sequences c; s, essentially the decoder first predicts the entity chain c and then continues predicting the summary s using both c and d. We prefix c and s with special markers "[ENTITYCHAIN]" and "[SUMMARY]", respectively, as shown in Figure 1. If s consists of multiple sentences, we use sentence markers "|||" to mark them in c. The model is trained with the standard maximum-likelihood objective generating the augmented target c; s.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Content Plans. We modified PE-GASUS (Zhang et al., 2019) to pretrain our models for entity-level content planning and summary generation. 2 In particular, we select a maximum of n most important sentences using self-ROUGE from an input document, the selected sentences work as a proxy for a human-authored abstractive summaries for the rest of the document. We construct a target by prepending the selected sentences dynamically with their entity chain. Our model is then trained to generate this target from the rest of the document. 3
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Generation with Entity Prompts. An advantage of training to generate the summary s following the generation of the plan c using the same decoder is that now during the inference time the decoder can be easily prompted with any desired content plan c to control the content in the output summary s with a probability of p(s |c , d).
Planning with Learned Entity Prompts for Abstractive Summarization
In Section 5, we prompt our decoder with modified content plans to mitigate hallucinations and to generate diverse summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Datasets. Following Zhang et al. (2019), our model pretraining also relied on two large web corpus which were processed to look like plain text: (i) C4 (Raffel et al., 2019) is composed of 350M Web-pages that were obtained from Common Crawl, and (ii) HugeNews (Zhang et al., 2019) is composed of 1.5B news and newslike articles from 2013-2019. This dataset includes articles from multiple allowlisted sources including news publishers, high-school newspapers and blogs.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
Entity Chain Annotation. We experimented with entity chains consisting of named entities,  Table 1: Abstractive summarization datasets studied in this work. We report on their train/validation/test sizes and how they were processed (cased/uncased). To better understand the effect of summary planning with entity chains, we report on average number of sentences (avg. sent.), average number of entities (avg. ent.) and average number of unique entities (avg. uniq. ent.), per target in validation sets. We also report on total number of named entities, date and number in target summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
dates and numbers. We annotate the whole document in the pretraining datasets to allow dynamic construction of summaries and their entity chains during pretraining. We only annotate the reference summaries for the finetuning summarization datasets. We use a BERT-based tagger trained on CoNLL-2003 data (Tjong Kim Sang andDe Meulder, 2003) to identify named entities, and regular expressions to identify dates and numbers (Guu et al., 2020). See Table 1 for the number of named entities, dates and numbers found in different datasets. Table 6 and 7 in Appendix A present hyperparameters used for pretraining and finetuning PE-GASUS and FROST base and large sized models. We used Cloud TPU v3 accelerators for training.
Planning with Learned Entity Prompts for Abstractive Summarization
We evaluate our FROST models on ROUGE, entity specificity, entity planning, faithfulness using automatic and human evaluations, and overall quality by humans. Our models predict a summary plan in the form of an entity chain, followed by a summary. All evaluations are done on the summary, the predicted entity chains are stripped out.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level ROUGE. We report ROUGE F1 scores (Lin and Hovy, 2003) to assess generated summaries. 7 Entity Planning. We evaluate the quality of content plans learned by our models by assessing the entities and the order in which they appear in the summary. We annotate both predicted and reference summaries with named entities, dates and numbers, and report on ROUGE F1 scores for entity chains found in the predicted summaries against corresponding reference entity chains. 7 We lowercased candidate and reference summaries and used pyrouge with parameters "-a -c 95 -m -n 4 -w 1.2." Entity Specificity. We compare entities in predicted and reference summaries, and report average entity F1-scores (ENTF1) in predicted summaries. We lower-case and remove duplicate entities; we report on the exact match with reference entities.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
We further assess summary faithfulness by humans. Our annotators, proficient in English, were tasked to read the document carefully and then grade its summary on a scale of 1-5 (fully unfaithful, somewhat unfaithful, 50-50, somewhat faithful and fully faithful); a summary is "fully faithful" if all of its content is fully supported or can be inferred from the document.
Planning with Learned Entity Prompts for Abstractive Summarization
Overall Summary Quality. Finally we assess the overall quality of summaries by humans. We ask our annotators to read the document carefully and then grade its summary on a scale of 1-5 (poor summary, better than poor, okay summary, better than okay and great summary). To improve the annotator agreement, we clearly define 4 features that a "great summary" should have: Relevant (summary should have most important information from the document), Accurate (summary "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell.
Planning with Learned Entity Prompts for Abstractive Summarization
Original Target: "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell. should be accurate with respect to the document or the expert knowledge), 8 Concise (summary should not have redundant or less-important content), and Fluent (summary should be grammatically correct and coherent). A great summary should pass on all 4 features, a better than okay should have 3 out of 4 features, and so on.
Planning with Learned Entity Prompts for Abstractive Summarization
For both human assessments, we collected 3 ratings for each (document, summary) pair and report the average of all assigned labels (1-5) to a system. We conducted 3 pilot studies for each setup to train our annotators with examples to improve their understanding of the task. Additionally, extra measures were taken to improve agreements among our annotators. For example, for the faithfulness assessment, when one of somewhat unfaithful, 50-50 and somewhat faithful were selected, annotators were asked to also specify what was faithful or unfaithful in the summary. Similarly for the overall quality assessment, when one of better than poor, okay summary and better than okay were selected, they were asked to list all features on which the candidate summary fails. 
Planning with Learned Entity Prompts for Abstractive Summarization
Sentence-Level vs Summary-Level Planning.
Planning with Learned Entity Prompts for Abstractive Summarization
Let the summary s consists of m sentences s 1 . . . s m and c i be the entity chain for the sentence s i , we consider generating the summary s in two ways. Sentence-level approach trains a model to generate s by consecutively generating the sentence-level content plan c i followed by its summary sentence s i with a probability
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
in Figure 3.
Planning with Learned Entity Prompts for Abstractive Summarization
We found that the summary-level planning is not only superior to the sentence-level planning, it helps with generating better quality summaries in terms of summary-level ROUGE (RL-Sum), entity planning (R2-EPlan) and entity specificity (ENTF1). For the rest of the pretraining or finetuning experiments, we focused on summary-level planning unless specified otherwise.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining for Planning from Scratch. We pretrained three base-sized models from scratch: PEGASUS-base pretrained with the original gapsentence objective (Zhang et al., 2019) for 1.5m steps, FROST(F)-base pretrained with the gapsentences prepended with their entity chain for 1.5m steps, and FROST(P+F)-base first pretrained with the PEGASUS objective for 1m steps and then with the FROST objective for another 500k steps. Maximum input and output lengths were set to 512 and 256 sentencepieces during pretraining, respectively. We finetuned these three models on the XSum dataset. Results are shown in Figure 4.
Planning with Learned Entity Prompts for Abstractive Summarization
First of all, our results confirm that the pretraining for planning is beneficial for summarization; both FROST(F) and FROST(P+F) learned better content plans (in terms of entity chainlevel ROUGE and ENTF1) than PEGASUS without sacrificing the summary quality (in terms of summary-level ROUGE) Figure 5: Finetuning results on the XSum (in blue) and CNN/DailyMail (in red) validation sets at various steps during pretraining FROST-Large. Instead of pretraining from scratch, we start with a PEGASUS-Large checkpoint, and continue pretraining for additional 1.5m steps with the planning objective. We report finetuning results for the PEGASUS finetuned baseline and our models at 0.1m, 1m and 1.5m steps. the pretraining for planning and summarization is more effective when started with summarization pretrained models such as PEGASUS than when trained jointly from scratch. Hence, for our future pretraining experiments we initialize our model with existing PEGASUS checkpoint and continue pretraining for content planning and refer to them as FROST for simplicity.
Planning with Learned Entity Prompts for Abstractive Summarization
Based on our findings, we pretrain a large FROST model for planning for summarization starting with an existing PEGASUS-Large checkpoint. Figure 5 presents results for finetuning these models on the XSum and CNN/DailyMail datasets at various steps during pretraining.
Planning with Learned Entity Prompts for Abstractive Summarization
Similar to our findings in Figure 4, our results with large models further confirm the advantages of the pretraining for planning; improvement over the PEGASUS baseline were larger for the XSum dataset than for the CNN/DailyMail dataset. We achieved the best performance on both datasets at the 1m pretraining step. We use the FROST model at 1m pretraining step for our finetuning experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Table 2 presents our final results from finetuning our large models on summarization datasets. For each dataset, we first report results from earlier work directly taken from corresponding papers; our results are in the bottom blocks for each dataset. We finetune our own PEGASUS using the standard approach (document to summary). We report results from FROST (ECP; Entity    (Dou et al., 2020) and PEGASUS (Zhang et al., 2019). We also include commonly reported LEAD-n baselines (selecting top n sentences from the document) and extractive oracle (EXT-ORACLE; selecting best set of sentences from the document with the most overlapping content with its reference summary), for the CNN/DailyMail and XSum datasets. EXT-ORACLEs are masked with * and are not directly comparable. All results are reported on the publicly available test sets.
Planning with Learned Entity Prompts for Abstractive Summarization
Chain Planning), i.e., PEGASUS finetuned with the FROST objective (document to content plan and summary). Finally, we report on FROST (ECPP; Entity Chain Planning with Pretraining), our models both pretrained and finetuned with the FROST objective.
Planning with Learned Entity Prompts for Abstractive Summarization
We find that even simply finetuning an existing PEGASUS pretrained model to do content planning and summarization (as in FROST (ECP)) leads to improvements in entity specificity (ENTF1) and the quality of entity plans in summaries (Entity Planning ROUGE), across all datasets. In fact in some cases, better content plans lead to large improvements in summary-level ROUGE as well, for example, FROST (ECP) improve on PEGASUS from 44.05/21.69/40.98 to 44.85/21.83/41.80 on ROUGE scores for CNN/DailyMail summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
We demonstrate in two ways the planning with entity chains can be useful in mitigating hallucinations in summarization: data filtering and dropprompt mechanism with entity chains. We focused on the XSum dataset for these experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Data Filtering using Entity Chains. During training FROST prepends reference summaries with their entity chains to better plan them. We leverage this to filter the dataset to only keep examples where summaries have fully extractive entity chains; an entity chain is fully extractive if all entities in it can be found in the input document. It's notable that this filtered dataset will not contain novel entities in the summary targets. FROST models trained on this data will ground the summary generation to extracted entity chains while allowing abstraction for non-entity related generations. The resulting XSum dataset has 62.7k/3.5k/3.5k train/validation/test instances. We finetune our models from Table 2 on this filtered dataset and report their performance on the filtered test set (3.5k). We also evaluate them on the rest of the test set (7.8k) where reference entity chains are not fully extractive to their input documents. 9 See Figure 6 for examples. 9 Such data divergence is not unique to XSum, around 30% of CNN/DailyMail summaries also have reference entity chains that are not fully extractive to their input documents. Writing these summaries requires either document-level inference or the background knowledge of the input documents to generate novel entities or numbers.  Table 3: Performance on XSum summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (3.5k) and the rest of the test set with novel entities in the targets (7.8k). Best results are bold faced.
Planning with Learned Entity Prompts for Abstractive Summarization
Drop-Prompt Mechanism. FROST decoders are trained to generate the summary s following the generation of its plan c. To improve faithfulness, we take the predictions from FROST (ECPP) 10 and modify the generated plan c to c drop by dropping entities (or parts of them) that are not found in the input document. We then prompt our decoder with c drop to generate a new summary. We conduct this with both models, one trained on the full dataset and another on the filtered subset. We also report results for the oracle entity chain prompts c oracle for a comparison.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Summary-level ROUGE, Entity Planning and Specificity. Results are presented in Table 3. First of all, similar to our results in Table 2 on the full XSum test set, our results with the models trained on the original dataset further validate that the pretraining and finetuning with the content planning objective is equally useful for both test sets: one where entities are simply copied from the input documents and another where novel entities were inferred, to generate corresponding targets. The data filtering using entity chains are particularly useful for test cases with extractive reference entity chains; models trained on the filtered data lead to much higher ENTF1 (e.g., 65.97 vs 63.03, for FROST (ECPP)) and higher quality entity plans (entitly-level ROUGE of 66.43/30.00/62.48 vs 63.73/30.08/60.10 for FROST (ECPP)), without sacrificing the summary quality substantially (summary-level ROUGE of 44.87/22.05/36.79 vs 10 The drop-prompt mechanism can be used with FROST (ECP) also, we have simply used the best among FROST (ECP) and FROST (ECPP) on the XSum set in terms of summarylevel ROUGE, entity planning and ENTF1.   Table 3 and 4. We highlight entities in orange that are not faithful to the input document. Entities in green are faithful to the input document. 45.08/22.01/36.80 for FROST (ECPP)). Unsurprisingly, these models don't do well on the test set with non-extractive reference entity chains, such examples were not seen during training.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Faithfulness. The planning objective in FROST itself does not ensure faithfulness; planning is done with the entities in the target, if the target is noisy, its plan may also be noisy (see Figure 6 for predicted entity chains with hallucinated entities). But the planning with the entity chains facilitates the data filtering and the drop-prompt mechanism, using entity chains. Table 4 presents our results assessing them for faithfulness. We randomly selected 50 documents for each of test sets (with extractive or non-extractive reference entity chains) and assessed their summaries from all 8 systems (except with c oracle ) from Table 3.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism is also very powerful in improving faithfulness. Irrespective of which training data were used to train the models and which test sets they were evaluated on, we see improvements in entailment scores, ENT-PREC and the human assessment of faithfulness across the board, except for a single case where the entailment score slightly drops from 0.465 to 0.453. Figure 6 demonstrates how we drop hallucinated entities 'Liam', 'two' and 'Conor' from entity chains to enforce models to generate faithful summaries grounded to the modified entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We achieve the best performance in terms of ENTPREC and the human assessment of faithfulness when both the data filtering and the dropprompt mechanism were used. We achieve 0.943 for ENTPREC and 4.41 for faithfulness for the test set with extractive entity chains only, and 0.826 for ENTPREC and 3.85 for faithfulness for the test set with non-extractive entity chains. Humans also found that the predictions from these models were the best in terms of overall summary quality.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, we carried out pairwise comparisons for human assessments for faithfulness and overall summary quality for all models (using a oneway ANOVA with post-hoc Tukey HSD tests; p < 0.01). Interestingly, differences among all model pairs for both faithfulness and overall summary quality are insignificant when evaluated on the test set with extractive reference entity chains only. On the more challenging test set with non-extractive entity chains, FROST (ECPP) trained on the fil-GOLD: [ENTITYCHAIN] BP | Lord Browne | North Sea [Summary] Former BP chief executive Lord Browne has warned North Sea oil operators' costs must fall in order for them to compete globally.
Planning with Learned Entity Prompts for Abstractive Summarization
Browne, the former boss of BP, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST (ECPP), c mod , Style diversity [ENTITYCHAIN] BP | Browne [Summary] The former boss of BP, Lord Browne, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
[ENTITYCHAIN] BBC | BP | Browne [Summary] In an interview with the BBC, former BP chief executive Lord Browne has warned the oil and gas industry is in danger of "going to the wall".  tered training set and with the drop-prompt mechanism is significantly better than all other models except for (i) FROST (ECPP) trained on the filtered training set but without the drop-prompt mechanism and (ii) FROST (ECPP) trained on the original training set and with the drop-prompt mechanism, for both faithfulness and overall summary quality. All other pairwise differences are insignificant.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST provides a handle to easily control or manipulate content plan for predicted summaries. In Figure 6, we saw how we can use this to control entity-level hallucinations, but the strength of FROST models go beyond this. Figure 7 shows how FROST models can be effectively used in generating summaries with different entity focus by simply modifying the entity prompt. Future work will focus on how to leverage FROST for synthetic data generation for training improvements.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
Planning with Learned Entity Prompts for Abstractive Summarization
There are several advantages of FROST-style decoding. Unlike encoder-guided control summarization systems, FROST models can be used in a usual way to generate summaries for input documents without relying on external systems to augment them during inference. Additionally, users can modify the generated entity prompts or provide their desired entity prompts to control the generated summaries (see Section 5.3 and 5.4).
Planning with Learned Entity Prompts for Abstractive Summarization
To the best of our knowledge, FROST is the first decoder-prompted control summarization model. Table 5 show that the prompting the decoder with entity prompts are more effective in generating high-quality and grounded summaries compared to when entity prompts are encoded with the input, especially for abstractive summaries; both PEGASUS versions (d; c oracle and c oracle ; d) perform inferior to FROST (d → c oracle ; s) in terms of summary-level ROUGE, entity-level ROUGE and ENTF1 on the XSum dataset. CTRLsum (k oracle ; d → s) achieves better ROUGE scores than FROST (d → c oracle ; s) on CNN/DailyMail (64.72/40.56/61.02 vs 61.85/38.95/59.00). This is not surprising as the oracle keywords k oracle in CTRLSum retain most words from the summary found in the source document; as such due to the extractive nature of CNN/DailyMail summaries, k oracle tend to be closer to the surface forms of the reference summaries (ROUGE scores between k oracle and reference summaries are 52.70/17.37/44.37). The difference in ROUGE scores narrows down between CTRLsum and FROST to 45.65/22.35/42.50 vs 45.11/22.11/42.01 with automatically extracted keywords. FROST outperforms CTRLsum on entity planning and specificity. It is not clear how well CTRLsum's keyword extraction system will generalize to more abstractive datasets where words in reference summaries are often not found in the source document.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, our FROST models work as a better microplanner and produce concise summaries than those generated by systems without doing content   Figure 10, Appendix C.
Planning with Learned Entity Prompts for Abstractive Summarization
In this work, we proposed the use of entity chains, both during pretraining and finetuning, to better plan and ground the generation of abstractive summaries. Our approach achieves state-of-theart ROUGE results in XSum and SAMSum, and competitive results in CNN/Dailymail. Compared to other guided summarization models, CTRL-Sum and GSum, which perform slightly better in CNN/Dailymail, our approach is drastically simpler to implement, trains by augmenting the targets only, utilizes no additional parameters than the baseline PEGASUS model, and does not rely on any external systems to augment the inputs during inference. We further demonstrate that by modifying the entity chain prompts, we can easily control hallucinations in summaries. Table 6 and 7 present hyperparameters used for pretraining and finetuning PEGASUS and FROST base and large sized models. 
Planning with Learned Entity Prompts for Abstractive Summarization
Figure 10 presents example predictions from FROST and CTRLSum along with their entity prompts and keywords used for guidance, respectively.    
Planning with Learned Entity Prompts for Abstractive Summarization
We experimented with entity chains consisting of named entities, dates and numbers. We finetuned PEGASUS-large models on the XSum dataset to predict the entity chain and then continue predicting the summary. Results are presented in Figure 11. We found that the entity chains with all named entities, dates and numbers performed best for entity level measures (entity chain-level ROUGE and ENTF1) without sacrificing the final summary quality (summary-level ROUGE) significantly. Following this, we always use entity chains with named entities, dates and numbers to have better control over them.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-Level Entity Chains Table 9 presents detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.
Planning with Learned Entity Prompts for Abstractive Summarization
Here, we present results from controlling hallucinations on CNN/DailyMail summaries (i) by training models on the dataset with extractive entity chains only and (ii) by modifying the predicted entity chains to keep only the supported entities. Table 10 and 11 present results from these mod- Figure 9: Instructions for human evaluations for overall quality of summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
GOLD: Theia, a bully breed mix, was apparently hit by a car, whacked with a hammer and buried in a field. "She's a true miracle dog and she deserves a good life," says Sara Mellado, who is looking for a home for Theia.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Oracle keywords) hit car apparently whacked hammer buried field | dog bully breed mix Theia | true miracle deserves good life (Summary) A bully breed mix named Theia was hit by a car, apparently whacked with a hammer and buried in a field. Four days after her apparent death, the dog managed to stagger to a nearby farm. "She's a true miracle dog and she deserves a good life," the dog's foster mother says. FROST: (Oracle Entity Prompt) Theia ||| Sara Mellado | Theia (Summary) The dog, now named Theia, was apparently hit by a car and buried. Sara Mellado is raising money to help pay for Theia's medical care.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Predicted keywords) stray pooch Washington State hit car head hammer mercy killing buried field survive | dog Theia | found (Summary) A stray pooch in Washington State was hit by a car and buried in a field. The dog was apparently hit on the head with a hammer in a mercy killing. She managed to survive and was found by a worker who took her to a vet. Theia is now being cared for at a veterinary hospital. FROST: (Predicted Entity Prompt) Washington State ||| Theia ||| (Summary) A stray dog in Washington State was apparently hit by a car and buried. The dog, now named Theia, survived but still needs surgery. A fundraising page has been set up to help pay for her care.    Table 9: Detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.  Table 10: Performance on CNN/DailyMail summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (7.8k) and the rest of the test set with novel entities in the targets (3.7k).  Figure 11: Ablations on entity chains with named entities, dates and numbers. We report summary-level ROUGE-L (RL-Sum), entity chain-level ROUGE-2 (R2-EPlan) and ENTF1 on the XSum validation set. Similar observations were made for other measures, see detailed results in Table 8.  Table 11: Faithfulness assessment using automatic and human evaluations, and overall quality assessment by humans for models presented in Table 10 for the CNN/DailyMail dataset (Test set with Non-Extractive Entity Chains Only). 
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Summarization systems are designed to help users navigate large amounts of information (Edmunds and Morris, 2000), but often fail to meet the unique needs of different users, especially for long documents. Recent research has explored ways to make summarization systems more controllable (Bornstein et al., 1999;Leuski et al., 2003) by allowing users to input queries or control sequences such as keywords (He et al., 2020), questions (Zhong et al., 2021), entity chains (Narayan et al., 2021), or question-answer pairs (Narayan et al., 2022).
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
A challenge shared by all of the mentioned approaches is the absence of abundant labeled data. * Work done during internship at Salesforce
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In a group discussion about a philosophical concept, Sarah used the Socratic method by asking and answering questions to stimulate critical thinking and clarify underlying assumptions. The method helped her and her classmates achieve a deeper understanding of the concept and address areas of disagreement.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In a group [Mask] philosophical concept, Sarah [Mask] method by asking [Mask] questions to stimulate critical thinking [Mask] clarify [Mask]. The method helped her and [Mask] understanding of the concept and address areas of disagreement.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
How did Sarah use the Socratic method in a group discussion? [QSep] In a group discussion about a philosophical concept, Sarah used the Socratic method by asking and answering questions to stimulate critical thinking and clarify underlying assumptions.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
[Ask&Answer] [Mask] The method helped her and her classmates achieve a deeper understanding of the concept and address areas of disagreement.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Figure 1: Our SOCRATIC pretraining compared to denoising. We mask important sentences in unlabeled input documents and train the model to generate both questions and pseudo-summaries as their answers.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Currently available datasets for training these systems are the result of expensive annotation efforts (Zhong et al., 2021;Kulkarni et al., 2020;Wang et al., 2022) with only hundreds to a few thousand query-document pairs, with the same document often being repeated. This translates into poor adherence of generated summaries to user-provided queries, particularly when these are finegrained plans. Recent work demonstrates the benefits of tailoring the pretraining objective to downstream task characteristics, especially where training data is difficult to obtain in large quantities like factualityfocused and multi-document summarization (Wan and Bansal, 2022;Xiao et al., 2022). In controllable summarization, summaries are grounded by queries, so designing an objective for the task requires introducing realistic queries in unlabeled data in a scalable manner. This work introduces SOCRATIC pretraining, an unsupervised pretraining objective for language models that is specifically designed for controllable summarization. It is inspired by the Socratic method and aims to facilitate the identification of relevant content and ensure that the generated summary faithfully responds to the user query. During SOCRATIC pretraining (see Figure 1) the language model is trained to generate relevant questions based on an input document and then answer them, bringing finegrained controllability to model pretraining which translates to better adherence to user queries.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
SOCRATIC pretraining only relies on unlabeled data and a question generation system and outperforms pre-finetuning approaches relying on additional supervised data (Aghajanyan et al., 2021;Wei et al., 2021;Fabbri et al., 2021a). In this work, we demonstrate the effectiveness of the SOCRATIC objective through pretraining adaptation, where a language model is further pretrained with the SO-CRATIC objective before finetuning on task-specific labeled data.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In summary, our contributions are as follows 1 :
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• We introduce the SOCRATIC pretraining objective for controllable summarization to improve adherence to user-specified queries or plans, both high-level and finegrained.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• We show that SOCRATIC pretraining performs well across domains, control strategies, and achieves state-of-the-art performance on two datasets.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• We perform ablations on our approach showing that SOCRATIC pretraining cuts labeled data requirements in half.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
During SOCRATIC pretraining, the model takes as input a document with important sentences masked and is trained to generate questions about the masked content and produce the mask itself. As seen in Figure 1, SOCRATIC pretraining is formulated as a sequence-to-sequence task and consists of two steps 1) important content is selected from unlabeled documents to be masked, and 2) a question-generation system is applied to produce questions about the selected content. The question augmentation component trains the model to produce summaries grounded to questions and allows for controllability as the end-user can prompt the model decoder with new questions during inference. We describe both steps below.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Selecting important content is essential for the model to learn to generate salient questions and summaries. In SOCRATIC pretraining, this content selection is done using the PEGASUS-style Gap Sentence Generation (GSG) objective (Zhang et al., 2020a), which we now briefly describe. Sentences with the highest self-Rouge with the document are selected for masking, ensuring that there is high information overlap with the rest of the document. The selected sentences, concatenated, produce a pseudo-summary of the document. As in PEGA-SUS, a Gap Sentence Ratio (GSR) of 45% is used, meaning that 45% of the sentences in the document are selected to appear in the target pseudo-summary. To help the model learn to copy, 80% of these sentences are masked and 20% are kept unmasked in the input document. Documents and summaries are truncated to 512 and 256 tokens.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
After selecting the pseudo-summary, a question generation (QG) system is applied to obtain a question from each sentence of the pseudo-summary. The QG system takes as input one of the selected sentences at a time and the unmasked document as  context. We apply MixQG (Murakhovs'ka et al., 2022), a state-of-the-art QG system. The choice to generate a question for each selected sentence, as opposed to each entity or the entire summary, is driven by three reasons. First, sentences in the pseudo-summary are selected from across the document and generally lack coherence, so there is no single query they collectively answer. Second, current QG systems are not trained to produce paragraph-level questions. Third, entity-level questions are often simple paraphrases of the answer sentence and are uncommon in QFS datasets.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Questions whose answers are full sentences, therefore, offer a compromise in terms of the complexity of the question and the coherence of the answer. We refer to these sentence-level questions as content-questions as they tend to ask about the content of the document instead of specific entities.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
After obtaining the questions, there are multiple ways to introduce them in the training objective either in the input or in the target text. As seen in Figure 2, we experiment with three modes on top of the base GSG objective:
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Reconstruct. The reconstruct mode is the default GSG mode where no questions are introduced. The masked document is the input and the pseudo-summary is the target text. We provide this mode as a baseline for our approach.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Ask. Given the masked document as input, the model is trained to only predict the questions about the masked sentences. This is the only mode where the target text does not include the pseudo-summary. With this mode, the model is trained to predict which questions can be asked in a given context.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Answer. Here, the questions are prepended to the masked input document while the target text remains the pseudo-summary. This mode is similar to how queries are introduced to the model during query-focused summarization and should help the model learn to respond to user-provided queries. However, this mode forgoes content planning as each generated sentence corresponds to one of the questions prepended to the input.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Ask&Answer. This mode combines benefits from both Ask and Answer modes. The model is tasked to first generate questions about the document and then, conditioning on both the document and the generated questions, the pseudo-summary. The model conditions on the generated questions in the decoder. This mode can be seen as first generating a finegrained plan for the pseudo-summary and then the pseudo-summary itself.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Like Tay et al. (2022), we prepend special tokens <ask>, <answer>, and <ask&answer> to the input document to specify the augmentation mode, and the <qsep> token to separate the generated questions from the target pseudo-summary.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We describe the experimental setup that we use to study SOCRATIC pretraining along with empirical studies justifying our design decisions.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
The SOCRATIC objective can be applied to any sequence-to-sequence language model irrespective of its specific architecture. In our experiments, we choose BART-large (Lewis et al., 2020), as the starting point for SOCRATIC pretraining adaptation. Following previous work on pretraining adaption for summarization, we pick BART over PEGASUS for its smaller size without performance compromises on summarization benchmarks and its more general-purpose pretraining objective. BART is also the underlying model in the SegEnc (Vig et al., 2022) architecture, which achieved state-of-the-art performance on QMSum, outperforming models such as LongT5 (Guo et al., 2022).
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Instead of pretraining the language model from scratch, we demonstrate the effectiveness of the proposed objective through what we call pretraining adaptation, where a generic language model is further pretrained with the SOCRATIC objective before being finetuned on task-specific labeled data. Although we introduce a new term for this training phase, pretraining adaptation was recently employed to evaluate task-specific pretraining objectives for factuality and multi-document summarization (Wan and Bansal, 2022;Xiao et al., 2022).
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
After SOCRATIC pretraining adaptation, the resulting model is used to initialize the SegEnc architecture, which is then finetuned on labeled data from downstream tasks. Pretraining and finetuning hyperparameter details are available in A.2.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We experiment with three different corpora, two of which are part of the Pile (Gao et al., 2021).
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• OpenWebText2 is a web-scraped dataset inspired by WebText (Radford et al., 2019) that uses Reddit upvotes of outgoing links as a proxy for page quality. Raffel et al. (2020) found this dataset to work well for summarization pretraining.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Books3 is a collection of both fiction and nonfiction books. We explore this data because our downstream tasks involve the short story and dialogue domains, and Csaky and Recski (2021) show books can be a good source of dialogue data.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• UnDial (He et al., 2022) We also explore using a dialogue corpus. As there are only two speakers in each dialogue in UnDial, we use a simple rule-based system to convert dialogues to third person. The pseudo-summary and related questions are then expressed in the third person while the input remains in the original dialogue format.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
To determine whether SOCRATIC pretraining improves model initialization for finetuning on controllable summarization, we test on two downstream datasets for query-focused, long-document summarization: QMSum and SQuALITY (dataset statistics can be found in A.1). We focus on long  document datasets as a challenging and practical testbed for controllable summarization methods.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
QMSum. QMSum is a benchmark for querybased, multi-domain meeting summarization (Zhong et al., 2021). The dataset consists of 1,808 query-summary pairs over 232 meetings, including product, academic, and parliamentary meetings.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
SQuALITY. SQuALITY is a dataset for querybased short stories summarization (Wang et al., 2022). The dataset is composed of 625 examples over 100 stories with four long reference summaries per document-question pair.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We apply the standard Rouge (Lin, 2004) and BERTScore (Zhang et al., 2020b) metrics to compare model generations with reference summaries on downstream finetuning tasks. In SQuALITY, we use the same procedure as the dataset authors to incorporate multiple references by taking the maximum score over the reference summaries. We also conduct a human evaluation study to ensure the variations between models are meaningful to users. Details on the setup can be found in A.4.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In this section, we corroborate our design choices with ablation studies of the components of SO-CRATIC pretraining. Similar to Zhang et al. (2020a) and Raffel et al. (2020), to save time and resources, we conduct the ablations of the objective on a small scale by restricting the pretraining adaptation to 1M documents from the OpenWebText2 corpus and then finetuning it on the full downstream task datasets. We report the mean over five randomly initialized finetuning runs on the validation set.  Question Augmentation Modes In Figure 3, we compare the performance of the three approaches for incorporating generated questions in the SO-CRATIC objective. The Ask and Ask&Answer perform similarly while Answer lags behind. This is in line with our hypothesis that learning which questions are relevant in a given context is a useful training signal for the model. The Ask&Answer mode also grounds the pseudo-summary generation in a sequence of finegrained questions. Therefore, it is chosen to be used in SOCRATIC pretraining.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Question Augmentation Proportion Incorporating questions with the Ask&Answer mode in each pretraining example could bias the model to always start by generating questions. We hypothesize that combining the Reconstruct mode with the Ask&Answer mode could alleviate this bias. In Figure 4, we find that introducing questions in 25% of pretraining examples leads to the best performance and use this proportion when scaling the pretraining adaptation.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Pretraining Corpus Selection In Figure 5, we find that the choice of pretraining corpus has a small but consistent effect on the performance of the SOCRATIC pretrained model on downstream tasks. The Books3 corpus performs best both on QMSum and SQuALITY. The dialogue corpus offers a slight advantage over OpenWebText2 on QM-Sum, a dialogue summarization task, while the opposite is true for SQuALITY. As a result, the full Books3 corpus, consisting of 30M training instances, is used in further experiments.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We scale the SOCRATIC pretraining adaptation based on the findings of the previous ablation and evaluate its downstream effects on query-focused summarization. Unless specified, the results in this section are averaged over five randomly initialized finetuning runs on the downstream tasks.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In Table 1, we compare the effect of SOCRATIC pretraining to other pretraining strategies on QM-Sum and SQuALITY. We obtain an improvement  of +1.01 and +0.53 Rouge-1, respectively, surpassing even the use of additional supervision from the related dataset WikiSum in Vig et al. (2022) and achieving new state-of-the-art results. These improvements are validated by a human study reported in Figure 6 and showing that SOCRATIC SegEnc performs better than the baselines in 59-65% of instances. Details of the human evaluation are found in A.4.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
The main baseline for SOCRATIC pretraining is the PEGASUS style GSG pretraining. We therefore perform a pretraining adaptation of BART-large with the GSG objective on the full Books3 corpus.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In Table 1  We also compare to BART-LS, an orthogonal approach that tailors BART's architecture, pretraining corpus, and objective to long documents . While our approaches are complementary, we outperform BART-LS on QMSum by +1.64 Rouge-2. This confirms our hypothesis that grounding generations in control queries in SOCRATIC pretraining is beneficial in controllable summarization, even more so than better long document modeling.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Gururangan et al. (2020) show that language models can be successfully adapted to the task domain by continuing to pretrain them in the new domain. This raises the question of whether improvements due to SOCRATIC pretraining are simply due to a better affinity of the pretraining corpus to the task domain. To answer this question, we perform continued pretraining 2 on a 1M subset of the Books3 corpus and next finetune the model on QMSum. Table 1 shows that continued pretraining slightly hurts Rouge-1 performance. In comparison, performing SOCRATIC pretraining on the same corpus improves performance by +0.41 Rouge-1. This observation rules out that improvements achieved through SOCRATIC pretraining are simply due to improved domain adaptation.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Transferring information from related tasks is another approach to adapt generic models to specific tasks (Aghajanyan et al., 2021). We show in Table 1 that SOCRATIC pretraining outperforms even the best pre-finetuned BART SegEnc model, which uses additional supervision from the Wik-iSum dataset (Liu et al., 2018). This transfer dataset was selected from a wide range of relevant summarization datasets tested by Vig et al. (2022). Crucially, we note that transfer learning, like prefinetuning, is orthogonal to our line of work which operates on the pretraining side. We believe that SOCRATIC can therefore be used in combination with pre-finetuning to further boost performance. 
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Both QMSum and Squality datasets contain a substantial portion of general summaries (12.5-20%) that aim to summarize the entire document in addition to those answering more specific queries. We find that our approach improves in both cases (+0.98 and +0.28 ROUGE-1 on QMSum in general and specific queries respectively). This shows that SOCRATIC pretraining improves models intended to perform a combination of general-purpose and query-focused summarization. In addition, with users increasingly interacting with language models through prompts to perform different tasks, the query-focused datasets we evaluate on become realistic testbeds for NLP systems that aim to perform well across tasks.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
To show that SOCRATIC pretraining alleviates the need for labeled downstream task data, we study the few-shot learning performance of SOCRATIC and BART SegEnc models. We perform one finetuning run for each model on each subset of the task data. In Figure 7, we show that with half the QMSum examples, SOCRATIC SegEnc achieves the same performance as finetuning BART SegEnc on all of QMSum. We believe that bringing SO-CRATIC pretraining closer to the downstream task of query-focused summarization lets the models learn from fewer downstream task examples.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In this section, we evaluate the effect of SOCRATIC pretraining on the adherence to user-provided finegrained control sequences. In these experiments, the same SOCRATIC pretrained model is finetuned on task-specific data with various control strategies.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
The queries found in QMSum and SQuALITY are only one format to encode user intent. Previous research explored other control strategies like keywords (He et al., 2020), entity chains (Narayan et al., 2021), or factoid question-answer pairs (Narayan et al., 2022). As seen in Figure 8, these strategies offer a more finegrained level of control over the summaries as they operate at the sentence level. Reference control sequences are not available for QMSum and SQuALITY so we generate them automatically from reference summaries.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In the summarization literature, such control sequences are often modeled as intermediate plans generated before the summaries (Narayan et al., 2022;He et al., 2020). In these cases, given the input X, the model first generates the detailed plan for the summary B from P (B|X), then generates the summary Y conditioning on the plan and the input x from P (Y |B, X). Even if the plan B is initially generated by the model, a user can control the summary by altering the plan. In practice, we experiment with three different planning strategies.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Content questions. For each sentence in the reference summary, we generate a question using the MixQG system while giving the full summary as context. These are similar to the questions that we use in our SOCRATIC pretraining. The sentence-level questions are then concatenated into a single plan for the summary. To our knowledge, we are the first to propose using content questions as finegrained plans for summaries.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• QA blueprint. We reimplement the recently proposed text plan in the form of a sequence of question-answer (QA) pairs (Narayan et al., 2022). First, all noun phrase answers are extracted from the reference. Then, a QG system generates questions answered by each noun phrase. The QA pairs are then filtered using round-trip consistency, rheme, and coverage criteria. The final plan consists of the concatenation of the remaining QA pairs.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
• Keywords. We use keywords extracted from each sentence of the reference summary. We take the noun-phrase answers from the QA blueprint as keywords and concatenate them with sentence separators into a plan.   
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In Table 2, we report evaluation metrics for both the model-generated summaries and plans. We find that with all three control strategies, SOCRATIC pretraining provides a consistent improvement over the vanilla BART model and the PEGASUS pretraining on both the generated finegrained plan and summary. On the planning side, there is a small but consistent improvement, up to +0.9 Rouge-1 with keyword chain control, indicating that the model has improved planning abilities. On the summarization side, we find a more significant improvement with up to +1.9 Rouge-1 with blueprint QA control. We attribute this to a combination of improved planning and execution ability of the model from SOCRATIC pretraining.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
With respect to control strategy performance, we find that our content questions obtain the highest Rouge scores (42.4 Rouge-1), outperforming keyword chains with only 25.0 Rouge-1. Despite the keyword plan having low overlap with the reference, it results in good summarization performance, so it is unclear whether the model using keyword chains learns the right correspondence between plan and summary. Moreover, the generated keyword chain would need heavier editing to obtain the reference plan compared to the content question plan (0.88 Levenstein distance compared to 0.77),   making them less useful in practice. Previous work has focused on keyword controls (He et al., 2020) and fact-oriented questions for text generation (Narayan et al., 2022), but there are inherent limitations with these approaches, which we discuss in detail in A.5.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Ideally, users can tailor generated summaries with an intervention limited to editing the generated plans. However, this requires strong adherence of generations to the finegrained plans, which we test here with oracle plans. Instead of generating both plan and summary, the system is given the oracle plans automatically extracted from the reference summaries (see 7.1). In Table 3, we observe a large improvement of +3.1 Rouge-1 over the BART SegEnc baseline. Human annotators confirm that SOCRATIC SegEnc follows oracle finegrained plans better or similarly to the baseline in 74% of instances, shown in Figure 9 and described further in A.4. This confirms our hypothesis that SOCRATIC pretraining helps ground the generations to userprovided queries. We attribute these gains to using the Ask&Answer mode, which introduces structure in the pretraining data by using as target text a question plan followed by its pseudo-summary answer.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We hypothesize that this structure in pretraining is what helps the model adhere to the planning step more effectively regardless of the control strategy.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In this work, we introduce SOCRATIC pretraining, a question-driven, unsupervised pretraining objective to adapt generic language models to the task of controllable summarization. SOCRATIC pretraining trains the model to generate relevant questions in a given context and then to answer them. Our experiments demonstrate the generality of our approach both on query-focused summarization and finegrained controllable summarization. We show that SOCRATIC pretraining outperforms other pretraining and prefinetuning objectives, that it cuts downstream task data requirements in half, and that it works across control strategies and domains.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Downstream Tasks In this work, we focused on long-document summarization as we believe it is the task where controllable summarization is most needed. Future work could investigate the effect of SOCRATIC pretraining on other downstream applications beyond those studied here. To handle long document input we could not use the BART model with SOCRATIC pretraining adaptation directly. Instead, we applied the SegEnc architecture on top of BART. This adaptation of the pretrained model may have dampened some of the few-shot performance of SOCRATIC pretraining. We thus believe that tasks with shorter input documents for which the SegEnc architecture is not necessary would see even greater benefits in the low-resource setting.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Base Model Throughout this work, we restricted our analysis to one model architecture the SegEnc architecture with the BART base model. Previous work extensively studied the impact of different architectures for long-document query-focused summarization (Vig et al., 2022). These primarily differ in how they model long documents. The authors found SegEnc, a simple sliding window adaptation of BART, to perform best on QMSum. While the results presented here are specific to SegEnc and BART, our approach is agnostic to the underlying model architecture and is orthogonal to longdocument modeling. We leave it to future work to investigate the effect SOCRATIC pretraining has on other architectures.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Evaluation Metrics As discussed in prior work (Fabbri et al., 2021b;Pagnoni et al., 2021;Gehrmann et al., 2021), there are limitations with the current automated evaluation metrics which do not strongly correlate with human judgments. Our results from these metrics should therefore be interpreted with caution and in combination with the human evaluation we performed to support them. One area in which automated metrics have been reported to perform poorly is factuality. Moreover, current factuality metrics have been designed and tested in the news domain and their performance in the out-of-domain setting (long documents and dialog data) was not systematically evaluated and is hard to interpret (Agarwal et al., 2022). In this work, we therefore choose not to report any factuality metric results.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We did not optimize the efficiency of the QG component of SOCRATIC pretraining and, consequently, it is computationally expensive. Currently, given equal amounts of resources for QG and pretraining, it takes us about the same time to perform the QG phase and pretraining phase on the same amount of data. We note, however, that in low-resource scenarios, the additional compute can lead to significant benefits, as shown in our results. In addition, we did not experiment with efficient sampling strategies, and believe that improving the efficiency of the QG model inference, for example through model distillation (Hinton et al., 2015), could lead to significant efficiency gains.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
The datasets for pretraining and finetuning used in this work are in English and thus mainly represent the culture of the Englishspeaking populace. Political or gender biases may also exist in the dataset, and models trained on these datasets may propagate these biases. Additionally, the pretrained BART model carries biases from the data it was pretrained on. We did not stress test these models for biases and request that the users be aware of these potential issues in applying the models presented.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Misuse Potential and Failure Mode When properly used, the summarization models described in this paper can be time-saving. However, the current model outputs may be factually inconsistent with the input documents, and in such a case could contribute to misinformation on the internet. This issue is present among all current abstractive summarization models and is an area of active research.  
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We describe here the training details for SOCRATIC pretraining as well as downstream task finetuning. Our experiments rely on the Huggingface Transformers library (Wolf et al., 2020). Our code includes sample pretraining and finetuning scripts to facilitate the reproduction of our results. We use 8 Nvidia A100 GPUs to run the experiments described in this paper. We will release our code under BSD 3-Clause license.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In the Books3 corpus, documents are longer than the desired input and target texts, we therefore segment the documents to obtain roughly the desired lengths. In the UnDial dataset, the opposite is true and therefore we concatenate dialogues to obtain the desired lengths. Following this segmentation or concatenation, we mask the input text and construct the target as described in section 3 depending on the desired mode. We then truncate the input and target texts to 256 and 512 tokens respectively.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We introduce mode tokens and a new separator tokens to the tokenizer of the BARTlarge model before the pretraining adaptation step.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Training Hyperparameters We train the BARTlarge model for 100k steps with batch size 512, checkpointing every 10k steps. For the ablations, we use batch size of 64 and the same number of steps. In all our experiments, we use AdamW optimizer with 5k warmup steps, learning rate 3e-5, weight decay of 0.01, max grad norm of 0.1, and bfloat16. Our choice of hyperparameters is based on best practices from previous work performing pretraining adaptations of BART-large (Xiao et al., 2022;Wan and Bansal, 2022). We also performed grid-search on the learning rate on the small-scale pretraining dataset testing the values {3e-6, 3e-5, 1e-4} but finding the initial value to perform best. We use the same hyperparameters on all three pretraining corpora in our ablations.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We evaluate the checkpoints on the validation dataset of the target downstream tasks and pick the best performing checkpoint.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
SegEnc Implementation We use the SegEnc implementation from the original authors. Instead of using vanilla BART-large to initialize the SegEnc model, we use one of our pretrainined models.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Finetuning Hyperparameters We use the same hyperparameters for both QMSum and SQuAL-ITY datasets and for QFS and finegrained planning experiments. We train the SegEnc model for 10 epochs with batch size 1 and bfloat16. We use the AdamW optimizer with learning rate 5e-6. We tested the following learning rate values {5e-7, 5e-6, 5e-5, 5e-4}. We use beam search decoding with beam size of 4. Our hyperparameters follow the best performing hyperparameters found by the original authors of the SegEnc model (Vig et al., 2022). Annotations will be made available ensuring the identity of the workers remains anonymous. We will only report the answers to the questions for each example and anonymize the worker ID.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Mode While the SOCRATIC pretraining consists of both Reconstruct and Ask&Answer modes, we found that the latter performed best on the downstream tasks.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We perform an automated evaluation using Rouge and BERTScore metrics following best practices from previous work. Specifically, we use the evaluation setup from Vig et al. (2022) for QMSum and the evaluation setup from Wang et al. (2022) for SQuALITY. More details and the relevant scripts can be found the in the supporting code supporting their papers. We also provide scripts to reproduce our evaluation. For BERTScore, we report recall following recommendations from .
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We perform a human evaluation study to confirm that variations between models are perceptible and meaningful to human users. The study separately assesses the QFS and the finegrained planning models finetuned on the QMSum dataset. In both cases, we use 100 of the 281 examples from the QMSum test set, and three independent annotators from the Amazon Mechanical Turk platform. We restrict the study to the specific questions of the QMSum dataset as these also provide relevant text spans in the original dialogue. We measure inter-annotator agreement with Fleiss Kappa κ (Fleiss, 1971) and obtain fair to moderate agreement in our tasks. Other studies that also rely on untrained crowd-sourced workers report similar, or sometimes even lower, agreement (Goyal et al., 2022).
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In this task, we compare the SegEnc model with SOCRATIC pretraining to Pegasus and BART pretraining. We ask annotators to select the best answer to the given query between two candidate summaries or mark if they are equally good. We provide both the reference summary and the relevant text span as supporting information. Annotator agreement on this task is κ = 0.33. The results are summarized in Figure 6 and the annotation instructions can be found in Figure 10.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Finegrained Planning Task In this task, we compare the SOCRATIC SegEnc model to the baseline BART SegEnc model in terms of their adherence to a finegrained plan. Both models are finetuned to the finegrained planning task on QMSum with the content question control strategy. Here we test how well they follow oracle plans automatically generated from the reference summary. The task is structured in two parts. First, for each question of the oracle plan, we ask annotators whether a sentence of the summary answers the question. We repeat for both SOCRATIC and BART summaries. On this task, we obtain moderate agreement of κ = 0.49. Next, we ask the annotators to select the best summary between the two candidates in terms of how closely it follows the plan. For the second task, the agreement is κ = 0.34. The results are summarized in Figure 9 and the annotation instructions can be found in Figure 11.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Worker Selection and Considerations An ethics review board did not review this particular protocol, but we followed prior protocols and internally-approved guidelines, such as carefully calibrating the time/HIT to ensure a pay-rate of $12/hour and letting workers know that their an-notations will be used as part of a research project to evaluate the performance of summarization systems.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
We selected workers according to the following criteria: HIT approval rate greater than or equal to 98%, number of HITs approved greater than or equal to 10000, and located in either the United Kingdom or the United States. The workers also passed a qualification test for a related summarization task from a prior project, ensuring that the annotators were familiar with the task of judging model-generated summaries.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Using content questions for QG augmentation in SOCRATIC pretraining improves performance across control strategies, including on nonquestion-based finegrained controls like keyword chains (see Table 2). While most previous work has focused on keyword controls (He et al., 2020) and fact-oriented questions for text generation (Narayan et al., 2022), there are inherent limitations with these approaches. We identify important qualitative properties of queries for controllable generation below that informed our choice of content questions for SOCRATIC pretraining.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Natural To facilitate the use of controllable summarization, one overarching objective is to make the user interaction with the system as natural as possible. When evaluating how "natural" a query strategy is, we consider whether such a strategy is used by humans when they interact with one another. According to this perspective, using keywords is an unnatural query strategy. Users generally express themselves through natural language, and when inquiring about information, they use questions. Our query systems in controllable summarization should strive to reflect this and support natural queries from the users.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Unambiguous To ensure that summaries contain the intended information, it is necessary that queries refer with minimal ambiguity to the information of interest in the document. When dealing with long documents, where the same entities occur repeatedly, keywords often imprecisely describe the intended query. But it is precisely with such long documents that query-focused summarization is particularly useful. In Table 5 Table 5: Properties of finegrained control strategies for the QMSum dataset. We measure lexical overall between the control sequence and the reference summary. We also calculate the average and maximum lexical overlap of two control sequences from the same QMSum document but answering two different high-level queries.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
comparison, content questions have a word overlap of 36% on average and no more than 67%. When formulating queries in natural language, they more richly encode the entities and their relations making them less ambiguous.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Concise Fact-oriented question-answer pairs (blueprint QA) (Narayan et al., 2022) tend to be less ambiguous than keywords (with the least lexical overlap across the three query strategies) but often end up requiring more text than the summary itself. On average, blueprint QA uses 50% more words than the summary (see Table 5). This makes this query strategy impractical for controllable summarization where the concision of the query is a desirable property.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
In this task, you will compare two candidate summaries and pick the one that provides the most informative and correct answer to the given question.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
To correctly solve this task, follow these steps:
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
1. Carefully read the question, reference summary, and candidate summaries. If needed refer to the dialogue.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
2. Compare the two candidate summaries according to the following criteria:
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
a. Informativeness of the summary. How much information does the answer contain? The more informative the better.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
b. Relevance of the information. How relevant to the question is the information in the answer? The more relevant the information the better.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
c. Correctness of the information. How correct is the information in the answer? The fewer errors in the answer the better. 3. Pick the summary that provides the most informative and correct answer to the given question.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Note that the reference summary is provided as a guide for a satisfactory answer and that candidate summaries might be more informative than the reference.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Reference: Product Manager said that it is important to think of the weight of the device. User Interface agreed and mentioned that the current design was too heavy compared to the competition. Product manager concluded that reducing the weight of the device should be the focus of the team's work for the following week.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Warning: Annotations will be checked for quality against control labels, low quality work will be rejected.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Question: What what said about the weight of the device? Summary 1: Product manager said weight is important. User Interface agreed the team should focus on reducing the device's weight. Summary 2: Product manager noted that the device felt heavy and that the team might have overlooked this aspect. User interface mentioned the current design was heavier than the competition. Product manager concluded that the team should focus on a lighter design and present it during next week's meeting.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Explanation: Summary 2 is correct. Summary 2 is more informative than Summary 1. Summary 1 also contains a mistake: It was Product Manager that told the team to focus on reducing the weight of the device, not User Interface. 
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
This message is only visible to you and will not be shown to Workers. You can test completing the task below and click "Submit" in order to preview the data and format of the submitted results. In this task, you will be given a list of questions and a summary. For each question, you will determine whether it is correctly answered by one of the sentences of the summary. Each sentence of the summary can only correspond to one specific question. If the question says BLANK, please mark "Not Answered."
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
You will judge two summaries in this fashon.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
After judging two summaries in Part 1, you will pick the summary that most closely answers the list of questions. You will pick the best summary based on the number of questions it correctly answers and if the answers appear in the same order as the questions. We do provide an option to rate the summaries as equal, but note that this is very rare/may not occur.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Part 1
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Part 2
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Warning: Annotations will be checked for quality against control labels, low quality work will be rejected.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Task: Which one of the specific questions is answered by a sentence of the summary?
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Summary 1: Product manager noted that the device felt heavy and that the team might have overlooked this aspect. User interface mentioned the current design was heavier than the competition. Product manager concluded that the team should focus on a lighter design and present it during next week's meeting.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Question 1: What did Product manager note about the device?
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Question 2: What did User interface mention about the competition? Question 3: What was the conclusion?
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Each one of the three questions is answered by a different sentence of summary 1.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Task: Which one of the specific questions is answered by a sentence of the summary?
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Summary 2: Product manager noted that the device felt heavy and that the team might have overlooked this aspect. Product manager concluded that the team should focus on a lighter design and present it during next week's meeting.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Question 1 and 3 are answered by summary 2. Question 2 was not answered by any sentence in the summary 2.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Task: Which summary better answers the following questions in their order?
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Summary 1: Product manager noted that the device felt heavy and that the team might have overlooked this aspect. User interface mentioned the current design was heavier than the competition. Product manager concluded that the team should focus on a lighter design and present it during next week's meeting. Summary 2: Product manager noted that the device felt heavy and that the team might have overlooked this aspect. Product manager concluded that the team should focus on a lighter design and present it during next week's meeting.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Summary 1. Summary 1 answers 3/3 of the questions while Summary 2 only 2/3.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Which of the following questions are answered by a sentence of Summary 1? One sentence in the summary can only answer one question.
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
This message is only visible to you and will not be shown to Workers. You can test completing the task below and click "Submit" in order to preview the data and format of the submitted results. Figure 11: Finegrained planning human annotation instructions.
Controllable Abstractive Summarization
Summarization condenses a document into a short paragraph or a single sentence while retaining core information. Summarization algorithms are either extractive or abstractive. Extractive algorithms form summaries by pasting together relevant portions of the input, while abstractive algorithms may generate new text that is not present in the initial document (Das and Martins, 2007;Nenkova et al., 2011).
Controllable Abstractive Summarization
This work focuses on abstractive summarization and, in contrast to previous work, describes mechanisms that enable the reader to control important aspects of the generated summary. The reader can select the desired length of the summary depending on how detailed they would like the summary to be. The reader can require the text to focus on entities they have a particular interest in. We let the reader choose the style of the summary based on their favorite source of information, e.g., a particular news source. Finally, we allow the reader to specify that they only want to summarize a portion of the article, for example the remaining paragraphs they haven't read.
Controllable Abstractive Summarization
Our work builds upon sequence-to-sequence models (Sutskever et al., 2014;Bahdanau et al., 2015), which have been extensively applied to abstractive summarization (Rush et al., 2015;Chopra et al., 2016;Nallapati et al., 2016;See et al., 2017;Paulus et al., 2017). These conditional language models use an encoder network to build a representation of the input document and a decoder network to generate a summary by attending to the source representation (Bahdanau et al., 2015).
Controllable Abstractive Summarization
We introduce a straightforward and extensible controllable summarization model to enable personalized generation and fully leverage that automatic summaries are generated at the reader's request. We show that (1) our generated summaries follow the specified preferences and (2) these control variables guide the learning process and improve generation even when they are set automatically during inference. Our comparison with state-of-the-art models on the standard CNN/DailyMail benchmark (Nallapati et al., 2016), a multi-sentence summarization news corpus, highlights the advantage of our approach. On both the entity-anonymized (+0.76 F1-ROUGE1) and full text versions (+0.85 F1-ROUGE1) of the dataset, we outperform previous pointer-based models trained with maximum likelihood despite the relative simplicity of our model. Further, we demonstrate in a blind human evaluation study that our model generates summaries preferred by human readers. are deep convolutional networks (LeCun et al., 1990). Both start with a word embedding layer followed by alternating convolutions with Gated Linear Units (GLU) . The decoder is connected to the encoder through attention modules (Bahdanau et al., 2015) that performs a weighted sum of the encoder outputs. The weights are predicted from the current decoder states, allowing the decoder to emphasize the parts of the input document which are the most relevant for generating the next token. We use multi-hop attention, i.e. attention is applied at each layer of the decoder.
Controllable Abstractive Summarization
In addition to attending over encoder states (Bahdanau et al., 2015), we also use intra-attention in the decoder to enable the model to refer back to previously generated words. This allows the decoder to keep track of its progress and reduces the generation of repeated information (Vaswani et al., 2017;Paulus et al., 2017). To combine encoder and decoder attention, we alternate between each type of attention at every layer.
Controllable Abstractive Summarization
Much prior work on the CNN-Dailymail benchmark employed pointer networks to copy rare entities from the input (Nallapati et al., 2016), which introduces additional complexity to the model. Instead, we rely on sub-word tokenization and weight sharing. We show this simple approach is very effective. Specifically, we use byte-pair-encoding (BPE) for tokenization, a proven strategy that has been shown to improve the generation of proper nouns in translation (Sennrich et al., 2016b). We share the representation of the tokens in the encoder and decoder embeddings and in the last decoder layer.
Controllable Abstractive Summarization
Summarization allows a reader with limited time to quickly comprehend the essence of a document. Controlling summary length enables reading with different time budgets: a document might be summarized as a five-word headline, a single sentence or a paragraph, each providing more and more detail.
Controllable Abstractive Summarization
To enable the user to control length, we first quantize summary length into discrete bins, each representing a size range. Length bins are chosen so that they each contain roughly an equal number of training documents. We then expand the input vocabulary with special word types to indicate the length bin of the desired summary, which allows generation to be conditioned upon this discrete length variable. For training, we prepend the input of our summarizer with a marker that indicates the length of the ground-truth summary.
Controllable Abstractive Summarization
At test time, we control the length of generated text by prepending a particular length marker token. Our experiments ( §5.2) provide quantitative and qualitative evidence that the model effectively uses this variable: output length is easily controlled by changing the length marker and supplying ground truth markers drastically improves summary quality. We compare our method to Kikuchi et al. (2016) and demonstrate that our straightforward length control strategy is more effective.
Controllable Abstractive Summarization
The reader might be interested in a document to learn about specific entities, such as people or locations. For example, a sports fan reading about a recent game might want to focus the summary on the performance of their favorite player. To enable entity-centric summaries, we first anonymize entities by replacing all occurrences of a given entity in a document by the same token. For training, we also anonymize the corresponding reference summary. For a (document, summary) pair, each entity is replaced with a token from the set (@entity0, . . . , @entityN). This abstracts away the surface form, allowing our approach to scale to many entities and generalize to unseen ones. We then express that an entity should be present in the generated summary by prepending the entity token to the input -prepending @entity3 expresses that the model should generate a summary where @entity3 is present. In effect, this instructs the model to focus on sentences that mention the marked entities.
Controllable Abstractive Summarization
At training time, we prepend each document with markers referring to an entity from the ground-truth summary. To ensure the entity request is informative, we provide an entity that is present in the ground-truth but not present in the summary generated by the baseline model. At test time, we may specify any entity marker that we wish the summary to contain. Our experiments ( §5.2) evaluate the effect of prepending different markers to the input. We show that higher accuracy is achieved when we specify entities from the first few sentences of a document or if we supply markers taken from the reference summary to illustrate specific user preferences. We extend this approach to multiple entity markers and experiment with appending all ground-truth entities for training and provide all entities from Lead-3 at test time. We show that providing more entities improves summarization quality.
Controllable Abstractive Summarization
Text sources such as newspapers and magazines often have specific style guidelines to provide a consistent experience. Readers are accustomed to the styles of their favorite sources. Therefore, we enable users to specify a preferred source style for a summary. Similar to length and entities, we introduce special marker tokens (@genSource0, . . . , @genSourceN) to express source desiderata. For training, we preprend the input with the marker corresponding to the ground-truth source. At inference, we control the style of generated summary by prepending different markers. Our experiments ( §4) evaluate whether providing the true source-style produces summaries that are closer to the reference summary. We additionally provide examples of distinct summaries resulting from changing source-style conditioning.
Controllable Abstractive Summarization
Beyond reading summaries of full documents, readers may want the flexibility of only summarizing certain portions of a document. For example, a reader who has read the first few paragraphs would want a summary of the remaining text to cover what they missed.
Controllable Abstractive Summarization
Training and evaluating remainder summarization requires specific data, namely a dataset of full documents with position markers separating the already read portion from the remainder part along with the corresponding summaries. Such a dataset is not readily available and would be challenging to collect. To enable remainder summarization without such data, we align summaries to full documents. Our procedure matches each reference summary sentence to its best matching document sentence based on ROUGE-L. For any position in the document, we remove sentences aligned before this point from the full summary and consider this shorter summary as the summary of the remainder. In our experiment, we consider as read portions all article positions located at the middle of two alignment points, except for alignment points separated by less than 2 sentences.
Controllable Abstractive Summarization
We consider the following methods: (1) full summary baseline: the baseline model predicts a full summary, disregarding the separation of the read portion from the remainder.
Controllable Abstractive Summarization
(2) post-inference alignment: a full summary is generated from the baseline model and the summary is shortened with our alignment procedure. The decoded summary sentences that align to the remainder portion compose the summary of the remainder.
Controllable Abstractive Summarization
(3) remainder only: the model is trained to map the document remainders to the remainder summaries on pre-aligned training data. This model is not given the read portion of the article.
Controllable Abstractive Summarization
(4) read and remainder: the model receives both read portion of the article and the remainder separated by a special token. It is trained to predict the remainder summary. We distinguish the read and remainder part of the article by using distinct sets of position embeddings.
Controllable Abstractive Summarization
We compare these methods in Section 4 and show the advantage of the model that receives both the user-read portion and the remainder of the document.
Controllable Abstractive Summarization
Automatic summarization has been an active research field for 60 years (Luhn, 1958). Extractive and abstractive methods have benefited from advances in natural language processing, pattern recognition, and machine learning (Nenkova et al., 2011). Recently, sequenceto-sequence neural networks (Sutskever et al., 2014) have been applied to abstractive summarization (Nallapati et al., 2016;See et al., 2017;Paulus et al., 2017) following their success in translation (Bahdanau et al., 2015;Luong et al., 2015b), parsing (Luong et al., 2015a) and image captioning (Vinyals et al., 2015b). Neural abstractive summarization has built upon advances from machine translation and related fields: attention (Bahdanau et al., 2015) enables generation to focus on parts of the source document while pointers  help abstractive summarization to copy entities from the input (See et al., 2017;Paulus et al., 2017;Nallapati et al., 2016).
Controllable Abstractive Summarization
However, summarization also has distinct challenges. The generation of multi-sentence summaries differs from single sentence translation: left-to-right decoders need to be aware of their previous generation at a larger time scale, otherwise models tend to produce repeated text. To address this impediment, (See et al., 2017) introduce coverage modeling, (Paulus et al., 2017) propose intra-decoder attention, and (Suzuki and Nagata, 2017) equip the decoder with an estimator of unigram frequency. Previous work has also explored learning objectives: (Paulus et al., 2017) investigates replacing maximum likelihood training with Reinforcement Learning (RL) to optimize ROUGE, the most common automatic metric to assess summarization. Combining both strategies is found to perform best in human evaluations, as training with RL alone often produces non-grammatical text.
Controllable Abstractive Summarization
Our work builds upon prior research: like (Gehring et al., 2017), we rely on convolutional networks, which enable faster training. This contrasts with prior work using recurrent networks (Nallapati et al., 2016;See et al., 2017;Paulus et al., 2017). We borrow intraattention from (Paulus et al., 2017) and expand it to multi-hop intra-attention inspired by multi-hop source attention from (Gehring et al., 2017). To facilitate copying input entities, we share the word representations between encoder and decoder (Paulus et al., 2017), and also rely on BPE tokenization (Sennrich et al., 2016b). This combination allows us to forgo an additional pointer mechanism unlike (Paulus et al., 2017;See et al., 2017;Nallapati et al., 2016). Unlike (Paulus et al., 2017), we did not explore training objectives and maximized the likelihood of the training summaries given the source document. Our model is amenable to RL, but this aspect is largely orthogonal to our main goal, i.e. controllable summarization.
Controllable Abstractive Summarization
Text generation is an established research area (McKeown, 1992). The field follows recent advances in generative models, such as the introduction of variational auto-encoders (Kingma and Welling, 2013) and adversarial networks (Goodfellow et al., 2014). This is exemplified by work focusing on natural language generation such as (Bowman et al., 2016;Yu et al., 2017;Zhao et al., 2017;Rajeswar et al., 2017).
Controllable Abstractive Summarization
Building upon unconditioned generation, controllable generation is an emerging research field. Research in computer vision includes style transfer (Gatys et al., 2015) or controllable image generation (Lample et al., 2017). Text generation work focuses on controlling tense or sentiment with variational auto-encoders (Hu et al., 2017). Shen et al. (2017) relies on adversarial training for manipulating sentence sentiment and Sennrich et al. (2016a) propose using side constraints for polite neural machine translation models. Takeno et al. (2017) extend the side constraints to control further aspects of translation output, such as length. Others have worked on style, for example Ficler and Goldberg (2017) propose using a conditional language model to generate text with stylistic requirements and Kobus et al. (2017) propose using tokens and additional features to translate text in different domains. Filippova (2017) proposes controlling length for generating answers in a question answering task. Kikuchi et al. (2016) explores length control for sentence compression using decoding-time restrictions and training-time length token embeddings.
Controllable Abstractive Summarization
Motivated by simplicity, our work relies on conditional language modeling and does not require adversarial training, latent variable models such as variational auto-encoders, or pointer networks. While latent variable models are popular for the generation of continuous outputs such as images, (conditional) language models are flexible enough to capture the multimodal nature of the data. We leave the assessment of how additional latent variables might improve upon our results to future work.
Controllable Abstractive Summarization
Dataset: We use the CNN-Dailymail dataset (Hermann et al., 2015;Nallapati et al., 2016). It consists of news articles along with multi-sentence summaries, with a total of 287k train, 13k valid and 11k test articles. On average, the articles are 758 token long, and the summaries are 55 token long. Most of our experiments are performed with articles truncated at 400 tokens, as suggested by (See et al., 2017). We evaluate on two versions of the data: the entity anonymized version (Hermann et al., 2015;Nallapati et al., 2016;Paulus et al., 2017) and the full text version (See et al., 2017). We use BPE with 30K types (Sennrich et al., 2016b) for most experiments. For non-BPE models, input and output vocabularies have resp. 47k and 21k word types, corresponding to types with more than 20 train occurrences.
Controllable Abstractive Summarization
Further, we compare length control with (Kikuchi et al., 2016) on DUC-2004 single-sentence summarization task. We train on English Gigaword following the protocol of Rush et al. (2015). The data consist of 3.6 million pairs (first sentence, headline of news articles). Following (Kikuchi et al., 2016), we evaluate on the 500 documents in the DUC2004 task-1. We use a source and target vocabulary of 30k words.
Controllable Abstractive Summarization
Architecture, Training, and Generation: We implement models with the fairseq library 1 . For CNN-Dailymail, our model has 8 layers in the encoder and decoder, each with kernel width 3. We use 512 hidden units for each layer, embeddings of size 340, and dropout 0.2. For DUC, we have 6 layers in the encoder and decoder with 256 hidden units.
Controllable Abstractive Summarization
Similar to Gehring et al. (2017), we train using Nesterov accelerated gradient method (Sutskever et al., 2013) with gradient clipping 0.1 (Pascanu et al., 2013), momentum 0.99, and learning rate 0.2. We reduce the learning rate by an order of magnitude when the validation perplexity ceases to improve, and end training when the learning rate drops below 10 −5 . Summaries are generated using beam search with beam size 5. To avoid repetition, we prevent the decoder from generating the same trigram more than once, following Paulus et al. (2017).
Controllable Abstractive Summarization
Evaluation: On the CNN-Dailymail benchmark, our automatic evaluation reports F1-ROUGE scores for ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004). We compare to existing abstractive baselines (Nallapati et al., 2016;See et al., 2017;Paulus et al., 2017). We also compare with Lead-3 which selects the first three sentences of the article as the summary. Note that, although simple, this baseline is not outperformed by all models.
Controllable Abstractive Summarization
For human evaluation, we conduct a human evaluation study using Amazon Mechanical Turk and the test set generation output of See et al. (2017). 500 articles from the test set were randomly selected and evaluated by 5 raters. The raters were presented with the first 400 words of each news article and asked to select the summarization output they preferred.
Controllable Abstractive Summarization
For the DUC-2004, we report recall ROUGE for ROUGE-1, ROUGE-2, and ROUGE-L at 30, 50, and 75 byte lengths following Kikuchi et al. (2016).
Controllable Abstractive Summarization
We evaluate the design choices of our model and the impact of manipulating the control variables. We analyze the performance of the remainder summarization task and demonstrate the advantage of modeling both the read and remainder portions of the document. Table 1 details the effect of our design choices for our baseline. Adding a constraint to avoid repeated trigrams at generation time improves F1-ROUGE1 by +2.86. Adding intra-attention to enable the model to examine past generations over long distances improves the accuracy obtained with the trigram constraint by a further 0.51 F1-ROUGE1. The modest improvement is likely because the two features address a similar problem of avoiding repeated generations. Switching tokenization from word to BPE gives another +0.79 F1-ROUGE1. BPE improves the ability to copy proper nouns and rare inflections, both of which are difficult to model in word-based vocabularies. This agrees with translation results (Sennrich et al., 2016b). Lastly, we find tuning the min/max length on the validation set and applying the constraints to the test set improves F1-ROUGE1 by 0.25.   
Controllable Abstractive Summarization
Our summarizer lets users control the length of the generated summary, entities on which it focuses on, and source style it imitates (see §2). We first evaluate the effect of providing the oracle reference variables at decoding time. This simulates a user setting their preferences to specific values. We then assess the effect of providing non-reference control variables. Table 2 reports our results for each variable and their combined effect. All control variables improve the summary quality, but length control has the most  impact, followed by entity control and source style. Further, the advantages of each control variable cumulatively produce an even stronger summary: we obtain +2.2 F1-ROUGE1 when combining control variables. Length control improves accuracy by 1.68 F1-ROUGE1 (Table 2). This improvement is due to two effects: length mismatch is heavily penalized by F1-ROUGE. Moreover, the baseline struggles at predicting correct lengths. The latter is due to large uncertainty in summary length, i.e. even humans have difficulty predicting the correct length. Figure 1 reports the average summary length when decoding all examples in the test set using each of the 10 possible length markers. The model is shown to respect length markers. Table 8 demonstrates the effect of the length marker on a specific example.
Controllable Abstractive Summarization
Entity control has less impact on ROUGE compared to length control at +0.69 vs. +1.68 F1-ROUGE1 (Table 2). This is mainly because our summaries often already contain most entities from the ground-truth without the need for additional instruction. Table 6 further analyzes entity control for 100 test documents. We decode repeatedly requiring each entity from lead-3. We then repeat the experiment with each entity from the full article. We report how often the entitycentric model generates a summary that actually contains the requested entity. For Lead-3 entities, the model mentions the requested entity 61% of the time, while for all entities from the input, the model mentions required entities 34% of the time. In both settings, these rates are much higher than the baseline. The model has difficulty generating summaries with entities which are unlikely to appear in the human references, e.g. unimportant entities far from the beginning of the article.
Controllable Abstractive Summarization
Source-style control is the least impactful control in terms of ROUGE, we report +0.2 F1-ROUGE1 in Table 2. Changing the source style variable changes the summary as shown in Table 8. Generally, we observe that generated summaries in the Dailymailstyle are more repetitive and slightly longer than the CNN-style summaries. This matches the differences between the two sources in the reference text. The impact of style requests might be greater with a richer set of styles -in future work, we plan to evaluate on datasets where varied styles are available.
Controllable Abstractive Summarization
Our primary objective is to allow readers to control the attributes of generated summaries. However, we can also set the control variables automatically in absence of reader desiderata. For length and source-style, we set the variable to a constant value that maximizes ROUGE on the validation set. For entity control, we randomly sample an entity that appears in lead-3 and provide it as the entity of interest. Table 3 reports results on the entity-anonymized version of the dataset like (Nallapati et al., 2016;Paulus et al., 2017) and Table 4 reports results on the full text data like (See et al., 2017). In both cases, our method is advantageous over alternatives. Further, providing all of the entities at training time and only lead-3 entities at test time improves quality. On the original text, we report 40.38 F1-ROUGE1 as opposed to 39.53 for (See et al., 2017). On the entity-anonymized text, we report 39.06 F1-ROUGE1 as opposed to 38.30 for the best maximum likelihood setting of (Paulus et al., 2017). We hypothesize that providing all lead-3 entities encourages copying from lead-3. Our model does not outperform the reinforcement learning model of (Paulus et al., 2017) which optimizes ROUGE. However, training objectives are orthogonal to our work on control variables and we expect reinforcement learning to equally benefit our model. Table 5 compares results on DUC-2004 to the best method presented by (Kikuchi et al., 2016). We find that adding length embedding improves the ROUGE-1 and ROUGE-L scores for 30, 50, and 75 byte evaluation. Notably, ROUGE improves more for shorter text evaluation, likely because requesting a shorter document allows the model to plan its generation. Comparing to Kikuchi et al. (2016), our results are stronger while our method is very simple - (Kikuchi et al., 2016) explore embedding the remaining length at each timestep during decoding and creating a separate memory cell to control length. In contrast, we simply provide the desired length as a special token and show this simple approach is effective. Lastly, we note that length-control has less effect on DUC-2004 compared   to CNN-Dailymail since truncated recall-ROUGE evaluation does not penalize length mismatch strongly. Overall, the improvements from automatic control show that a better model can be obtained by providing additional information during training -we present the first model trained with maximum likelihood to match the strong Lead-3 baseline. When the model is not required to predict the summary length or the entities of interest, it can assign more capacity to generating text conditioned on these variables. This is particularly useful for variables which are hard to predict from the input due to intrinsic uncertainty like length. In subsequent work, we plan to explore architectures to explicitly divide the prediction of control variables and sequence-to-sequence mapping.  
Controllable Abstractive Summarization
Summarizing the remainder of an article helps the reader to quickly grasp what they have not yet read. Table 7 presents our results relying on aligned data introduced in §2.5. Generally, this task is more difficult than summarizing the entire article. First, the length of both read portions and summaries varies greatly. It is difficult for the model to distinguish information specific to the remaining portion of the document from the general point of the article. Despite this, when models trained on summarizing the remainder are tasked with summarizing only full documents, the performance is not much worse (37.02 F1-ROUGE1 compared to 37.73 F1-ROUGE1 of the baseline in Table 1).
Controllable Abstractive Summarization
Our baseline always presents the full summary, regardless of the portion of the article presented as input. It achieves an F1-ROUGE1 score of 28.12. Among our three proposed methods, forming the remainder summaries post-inference performs poorly as it depends largely on alignment quality. The news articles are repetitive, so one summary sentence can align to multiple locations in the source. Training the model to perform remainder summarization significantly improves our results. Models that receive only the remainder and produce a summary achieve F1-ROUGE1 of 30.30, while models that receive both the read and remainder portions are slightly better (F1-ROUGE1 30.54). We hypothesize that presenting the read portion of the article improves the quality as the model can focus on the new information in the remainder. An explicit method for eliminating redundancy between the read and the remainder is relevant future work.
Controllable Abstractive Summarization
Remainder summary length is particularly difficult to predict. We therefore rely on length control: we split the validation dataset into 10 partitions based on how far in the article the remainder begins and determine the best length setting for each partition. We decode the test data with this setting which provides an additional improvement, 30.70 F1-ROUGE1. However, partitioning is not an accurate length model and we hypothesize that length control could provide a greater improvement with a better model.  (See et al., 2017) 39.53 41.04% fixed ctrl+Lead-3 ent.
Controllable Abstractive Summarization
40.38 58.99% Table 9: Human evaluation: 59% of ratings prefer our summaries (500 CNN-DM test articles, 5 raters each).
Controllable Abstractive Summarization
Our study compares summarization with fixed value control variables on full text CNN-Dailymail with (See et al., 2017). Table 9 shows that human raters prefer our model about 59% of the time based 2.5k judgments. Our model can therefore improve summary quality in a discernible way. As an aside, we find that ROUGE and ratings agree in two-thirds of the cases, where at least four out of five humans agree.
Controllable Abstractive Summarization
We proposed a controllable summarization model to allow users to define high-level attributes of generated summaries, such as length, source-style, entities of interest, and summarizing only remaining portions of a document. We simulate user preferences for these variables by setting them to oracle values and show large ROUGE gains. The control variables are effective without user input which we demonstrate by assigning them fixed values tuned on a held-out set. This outperforms comparable state of the art summarization models for both ROUGE and human evaluation.
GSum: A General Framework for Guided Neural Abstractive Summarization
Modern techniques for text summarization generally can be categorized as either extractive methods (Nallapati et al., 2017;Narayan et al., 2018b;Zhou et al., 2018), which identify the most suitable words or sentences from the input document and concatenate them to form a summary, or abstractive methods (Rush et al., 2015;Chopra et al., 2016;Nallapati et al., 2016;Paulus et al., 2018), which generate summaries freely and are able to produce novel words and sentences. Compared with extractive algorithms, abstractive algorithms are more flexible, making them more likely to produce fluent and coherent summaries. However, the unconstrained nature of abstractive summarization can also result in problems. First, it can result 1 Code is available at https://github.com/ neulab/guided_summarization. : Our framework generates summaries using both the source document and separate guidance signals. We use an oracle to select guidance during training and use automatically extracted or user-specified guidance at test time.
GSum: A General Framework for Guided Neural Abstractive Summarization
in unfaithful summaries (Kryściński et al., 2019), containing factual errors as well as hallucinated content. Second, it can be difficult to control the content of summaries; it is hard to pick in advance which aspects of the original content an abstractive system may touch upon. To address the issues, we propose methods for guided neural abstractive summarization: methods that provide various types of guidance signals that 1) constrain the summary so that the output content will deviate less from the source document; 2) allow for controllability through provision of user-specified inputs. There have been some previous methods for guiding neural abstractive summarization models. For example, Kikuchi et al. (2016) specify the length of abstractive summaries,  provide models with keywords to prevent the model from missing key information, and Cao et al. (2018) propose models that retrieve and reference relevant summaries from the training set. While these methods have demonstrated improvements in summarization quality and controllability, each focuses on one particular type of guidance -it remains unclear which is better and whether they are complementary to each other.
GSum: A General Framework for Guided Neural Abstractive Summarization
In this paper, we propose a general and extensible guided summarization framework that can take different kinds of external guidance as in-Work Guidance Form
GSum: A General Framework for Guided Neural Abstractive Summarization
Triples Sentences Summaries Kikuchi et al. (2016) (length tokens) Cao et al. (2018) (retrieved sums.)  (keywords) Liu et al. (2018a) (highlighted sents.) Liu et al. (2018b) (length tokens) Fan et al. (2018) (length, entity, style tokens) Zhu et al. (2020) (relations) Jin et al. (2020) (relations) Saito et al. (2020) (keywords) (highlighted sents.)
GSum: A General Framework for Guided Neural Abstractive Summarization
Ours (keywords) (relations) (highlighted sents.) (retrieved sums.) put. Like most recent summarization models, our model is based on neural encoder-decoders, instantiated with contextualized pretrained language models, including BERT (Devlin et al., 2019) and BART (Lewis et al., 2020). With this as a strong starting point, we make modifications allowing the model to attend to both the source documents and the guidance signals when generating outputs. As shown in Figure 1, we can provide automatically extracted or user-specified guidance to the model during test time to constrain the model output. At training time, to encourage the model to pay close attention to the guidance, we propose to use an oracle to select informative guidance signals -a simple modification that nonetheless proved essential in effective learning of our guided summarization models. Using this framework, we investigate four types of guidance signals: (1) highlighted sentences in the source document, (2) keywords, (3) salient relational triples in the form of (subject, relation, object), and (4) retrieved summaries.
GSum: A General Framework for Guided Neural Abstractive Summarization
We evaluate our methods on 6 popular summarization benchmarks. Our best model, using highlighted sentences as guidance, can achieve stateof-the-art performance on 4 out of the 6 datasets, including 1.28/0.79/1.13 ROUGE-1/2/L improvements over previous state-of-the-art model on the widely-used CNN/DM dataset. In addition, we perform in-depth analyses of different guidance signals and demonstrate that they are complementary to each other in that there is potential to aggregate their outputs together and obtain further improvements. An analysis of the results also reveals that our guided models can generate more faithful summaries and more novel words. Finally, we demonstrate that we can control the output by providing user-specified guidance signals, with different provided signals resulting in qualitatively different summaries.
GSum: A General Framework for Guided Neural Abstractive Summarization
Neural abstractive summarization typically takes a source document x consisting of multiple sentences x 1 , · · · , x |x| , runs them through an encoder to generate representations, and passes them to a decoder that outputs the summary y one target word at a time. Model parameters θ are trained to maximize the conditional likelihood of the outputs in a parallel training corpus X , Y :
GSum: A General Framework for Guided Neural Abstractive Summarization
Several techniques have been proposed to improve the model architecture. For example, models of copying (Gu et al., 2016;See et al., 2017;Gehrmann et al., 2018) allow words to be copied directly from the input to the output, and models of coverage discourage the model from generating repetitive words (See et al., 2017).
GSum: A General Framework for Guided Neural Abstractive Summarization
Guidance can be defined as some variety of signal g that is fed into the model in addition to the source document x:
GSum: A General Framework for Guided Neural Abstractive Summarization
Within this overall framework, the types of information that go into g and the method for incorporating this information into the model may vary. While there are early attempts at non-neural guided models (Owczarzak and Dang, 2010;Genest and Lapalme, 2012), here we focus on neural approaches  Figure 2: General framework of our model. The two encoders encode the source document and guidance signal, which are attended to by the decoder. and summarize recent work in Table 1. For example,  first generate a set of keywords, which are then incorporated into the generation process by an attention mechanism. Cao et al. (2018) propose to search the training corpus and retrieve datapoint x j , y j whose input document x j is most relevant to the current input x, and treat y j as a candidate template to guide the summarization process. Besides, Jin et al. (2020) and Zhu et al. (2020) extract relational triples in the form of (subject, relation, object) from source documents and represent them by graph neural networks. The decoders then attend to the extracted relations to generate faithful summaries. A concurrent work by Saito et al. (2020) propose to extract keywords or highlighted sentences using saliency models and feed them to summarization models.
GSum: A General Framework for Guided Neural Abstractive Summarization
There are also works on controlling the summary length (Kikuchi et al., 2016;Liu et al., 2018b) and styles (Fan et al., 2018) by explicitly feeding the desired features to the model. In addition, Liu et al. (2018a) and Chen and Bansal (2018) follow a twostage paradigm, in which a subset of the source document {x i 1 , · · · , x in } will first be selected by a pretrained extractor as highlighted sentences and then be fed into the model encoder in the second stage with the rest of the text discarded. Figure 2 illustrates the general framework of our proposed method. We feed both the source docu-ments and various types of guidance signals to the model. Specifically, we experiment with guidance signals including highlighted sentences, keywords, relations, and retrieved summaries, although the framework is general and could be expanded to other varieties of guidance as well.
GSum: A General Framework for Guided Neural Abstractive Summarization
We adopt the Transformer model (Vaswani et al., 2017) as our backbone architecture, instantiated with BERT or BART, which can be separated into the encoder and decoder components.
GSum: A General Framework for Guided Neural Abstractive Summarization
Our model has two encoders, encoding the input source document and guidance signals respectively.
GSum: A General Framework for Guided Neural Abstractive Summarization
Similar to the Transformer model, each of our encoders is composed of N enc + 1 layers, with each encoding layer containing both a self-attention block and a feed-forward block:
GSum: A General Framework for Guided Neural Abstractive Summarization
where LN denotes layer normalization. Note the source document and guidance signal do not interact with each other during encoding.
GSum: A General Framework for Guided Neural Abstractive Summarization
We share the parameters of the bottom N enc layers and the word embedding layers between the two encoders, because 1) this can reduce the computation and memory requirements; 2) we conjecture that the differences between source documents and guidance signals should be high-level, which are captured at top layers of the encoders.
GSum: A General Framework for Guided Neural Abstractive Summarization
Different from the standard Transformer, our decoder has to attend to both the source document and guidance signal instead of just one input.
GSum: A General Framework for Guided Neural Abstractive Summarization
Concretely, our decoder is composed of N dec identical layers, with each layer containing four blocks. After the self-attention block, the decoder will first attend to the guidance signals and generate the corresponding representations, and hence the guidance signal will inform the decoder which part of the source documents should be focused on. Then, the decoder will attend to the whole source document based on the guidance-aware representations. Finally, the output representation will be fed into the feed-forward block: y = LN(y + SELFATTN(y)), y = LN(y + CROSSATTN(y, g)), y = LN(y + CROSSATTN(y, x)), y = LN(y + FEEDFORWARD(y)).
GSum: A General Framework for Guided Neural Abstractive Summarization
Ideally, the second cross-attention block allows the model to fill in the details of the input guidance signal, such as finding the name of an entity by searching through co-reference chains.
GSum: A General Framework for Guided Neural Abstractive Summarization
Before delving into the specifics of the types of guidance signal we used, we first note an important detail in training our model. At test time, there are two ways we can define the guidance signal: 1) manual definition where an interested user defines the guidance signal g by hand, and 2) automatic prediction where an automated system is used to infer the guidance signal g from input x. We demonstrate results for both in experiments.
GSum: A General Framework for Guided Neural Abstractive Summarization
At training time, it is often prohibitively expensive to obtain manual guidance. Hence, we focus on two varieties of generating them: 1) automatic prediction using x as detailed above, and 2) oracle extraction where we use both x and y to deduce a value g that is most likely useful in generating y.
GSum: A General Framework for Guided Neural Abstractive Summarization
Theoretically, automatic prediction has the advantage of matching the training and testing conditions of a system that will also receive automatic predictions at test time. However, as we will show in experiments, the use of oracle guidance has a large advantage of generating guidance signals that are highly informative, thus encouraging the model to pay more attention to them at test time.
GSum: A General Framework for Guided Neural Abstractive Summarization
With this in mind, we describe the four varieties of guidance signal we experiment with, along with their automatic and oracle extraction methods. Highlighted Sentences. The success of extractive approaches have demonstrated that we can extract a subset of sentences {x i 1 , · · · , x in } from the source document and concatenate them to form a summary. Inspired by this, we explicitly inform our model which subset of source sentences should be highlighted using extractive models.
GSum: A General Framework for Guided Neural Abstractive Summarization
We perform oracle extraction using a greedy search algorithm (Nallapati et al., 2017;Liu and Lapata, 2019) to find a set of sentences in the source document that have the highest ROUGE scores with the reference (detailed in Appendix) and treat these as our guidance g. At test time, we use pretrained extractive summarization models (BertExt (Liu and Lapata, 2019) or Match-Sum (Zhong et al., 2020) in our experiments) to perform automatic prediction.
GSum: A General Framework for Guided Neural Abstractive Summarization
Keywords. If we select full sentences, they may contain unnecessary information that does not oc-cur in an actual summary, which could distract the model from focusing on the desired aspects of the input. Therefore, we also try to feed our model with a set of individual keywords {w 1 , . . . , w n } from the source document.
GSum: A General Framework for Guided Neural Abstractive Summarization
For oracle extraction, we first use the greedy search algorithm mentioned above to select a subset of input sentences, then use TextRank (Mihalcea and Tarau, 2004) to extract keywords from these sentences. We also filter the keywords that are not in the target summary. The remaining keywords are then fed to our models. For automatic prediction, we use another neural model (BertAbs (Liu and Lapata, 2019) in the experiments) to predict the keywords in the target summary.
GSum: A General Framework for Guided Neural Abstractive Summarization
Relations. Relations are typically represented in the form of relational triples, with each triple containing a subject, a relation, and an object. For example, Barack Obama was born in Hawaii will create a triple (Barack Obama, was born in, Hawaii).
GSum: A General Framework for Guided Neural Abstractive Summarization
For oracle extraction, we first use Stanford Ope-nIE (Angeli et al., 2015) to extract relational triples from the source document. Similar to how we select highlighted sentences, we then greedily select a set of relations that have the highest ROUGE score with the reference, which are then flattened and treated as guidance. For automatic prediction, we use another neural model (similarly, BertAbs) to predict the relation triples on the target side.
GSum: A General Framework for Guided Neural Abstractive Summarization
Retrieved Summaries. Intuitively, gold summaries of similar documents with the input can provide a reference point to guide the summarization. Therefore, we also try to retrieve relevant summaries from the training data X , Y .
GSum: A General Framework for Guided Neural Abstractive Summarization
For oracle extraction, we directly retrieve five datapoints { x 1 , y 1 , . . . , x 5 , y 5 } from training data whose summaries y i are most similar to the target summary y using Elastic Search. 2 For automatic prediction at test time, we retrieve five datapoints whose source documents x i are most similar to each input source document x instead.
GSum: A General Framework for Guided Neural Abstractive Summarization
We experiment on 6 datasets (statistics in Table 2):
GSum: A General Framework for Guided Neural Abstractive Summarization
Reddit (Kim et al., 2019) is a highly abstractive dataset and we use its TIFU-long version.  XSum (Narayan et al., 2018a) is an abstractive dataset that contains one-sentence summaries of online articles from BBC. CNN/DM (Hermann et al., 2015;Nallapati et al., 2016) is a widely-used summarization dataset consisting of news articles and associated highlights as summaries. We use its non-anonymized version.
GSum: A General Framework for Guided Neural Abstractive Summarization
WikiHow (Koupaee and Wang, 2018) is extracted from an online knowledge base and requires high level of abstraction. New York Times (NYT) (Sandhaus, 2008) is a dataset that consists of news articles and their associated summaries. 3 We follow Kedzie et al. (2018) to preprocess and split the dataset.
GSum: A General Framework for Guided Neural Abstractive Summarization
PubMed (Cohan et al., 2018) is relatively extractive and is collected from scientific papers.
GSum: A General Framework for Guided Neural Abstractive Summarization
Our baselines include the following models: BertExt (Liu and Lapata, 2019) is an extractive model whose parameters are initialized with BERT (Devlin et al., 2019).
GSum: A General Framework for Guided Neural Abstractive Summarization
BertAbs (Liu and Lapata, 2019) is an abstractive model with encoder initialized with BERT and trained with a different optimizer than its decoder.
GSum: A General Framework for Guided Neural Abstractive Summarization
MatchSum (Zhong et al., 2020) is an extractive model that reranks the candidate summaries produced by BertExt and achieves state-of-the-art extractive results on various summarization datasets.
GSum: A General Framework for Guided Neural Abstractive Summarization
BART (Lewis et al., 2020) is an state-of-the-art abstractive summarization model pretrained with a denoising autoencoding objective.
GSum: A General Framework for Guided Neural Abstractive Summarization
We build our models based on both BertAbs and BART, and follow their hyperparameter settings to train our summarizers.   (2004)) on CNN/DM. "Auto" and "oracle" denote using automatically predicted and oracle-extracted guidance at test time respectively. Results with * are from Liu and Lapata (2019).
GSum: A General Framework for Guided Neural Abstractive Summarization
between the two encoders. For our model built on BART, there are 24 encoding layers, with the top layer initialized with pretrained parameters yet separately trained between the two encoders. The first cross-attention block of the decoder is randomly initialized whereas the second cross-attention block is initialized with pretrained parameters. BertAbs is used to predict guidance signals of relations and keywords during test time. Unless otherwise stated, we use oracle extractions at training time.
GSum: A General Framework for Guided Neural Abstractive Summarization
We first compare different kinds of guidance signals on the CNN/DM dataset using BertAbs, then evaluate the best guidance on the other five datasets using both BertAbs and BART.
GSum: A General Framework for Guided Neural Abstractive Summarization
Performance of Different Guidance Signals. As shown in Table 3, if we feed the model with automatically constructed signals, feeding either highlighted sentences or keywords can outperform the abstractive summarization baseline by a large margin. Especially, feeding highlighted sentences can outperform the best baseline by more than 1   ROUGE-L point. Using relations or retrieved summaries as guidance will not improve the baseline performance, likely because it is hard to predict these signals during test time.
GSum: A General Framework for Guided Neural Abstractive Summarization
If we use an oracle to select the guidance signals, all varieties of guidance can improve the baseline performance significantly, with the best-performing model achieving a ROUGE-1 score of 55.18. The results indicate that 1) the model performance has the potential to be further improved given a better guidance prediction model; 2) the model does learn to depend on the guidance signals.
GSum: A General Framework for Guided Neural Abstractive Summarization
Comparisons with State of the Art. We then try to build our model on the state-of-the-art model, using highlighted sentences as guidance as it achieves the best performance on CNN/DM. First, we build our model on BART and train it with oracleextracted highlighted sentences as guidance. Then, we use MatchSum to predict the guidance at test time. From Table 4, we can see that our model can achieve over 1 ROUGE-1/L point improvements compared with the state-of-the-art models, indicating the effectiveness of the proposed methods.
GSum: A General Framework for Guided Neural Abstractive Summarization
Performance on Other Datasets. We report the performance of the highlighted sentence model on all the other five datasets in Table 5. Generally, the model works better when the dataset is more extractive. For abstractive datasets such as Reddit and XSum, our model cannot achieve performance increases when the abstractive summarization base- line is already rather strong. For extractive datasets such as PubMed and NYT, on the other hand, our model can achieve some improvements over the baselines even though the abstractive baseline outperforms the extractive oracle model in some cases.
GSum: A General Framework for Guided Neural Abstractive Summarization
We perform extensive analyses on CNN/DM to gain insights into our (BERT-based) models. Unless otherwise stated, we use oracle extractions at training time and automatic prediction at test time.
GSum: A General Framework for Guided Neural Abstractive Summarization
Novel n-grams. While we sometimes provide information extracted from the source document as guidance signals, it is unclear whether the model will over-fit to and regurgitate this guidance, or still generate novel expressions. To measure this, we count the number of novel n-grams in the output summaries, namely n-grams that do not appear in the source document. As shown in Figure 3, all of our guided models in fact generate more novel ngrams than the baseline, likely because at training time the model is trained to compress and paraphrase the extracted information from the source document into the gold summary. In addition, our models cover more novel n-grams that are in the gold reference than baseline. The results indicate that our guided models can indeed generate novel expressions, and are not referencing the input guidance too strongly.
GSum: A General Framework for Guided Neural Abstractive Summarization
Complementarity of Different Guidance Signals. While some guidance signals achieve worse performance than others, it is still possible to aggregate their outputs and obtain better performance if their outputs are diverse and they complement eachother. To verify this hypothesis, we try to select the best output of the four guidance signals for each test datapoint and investigate if we can aggregate their best outputs and achieve better performance. Concretely, for each test input, we perform an oracle experiment where we compute the ROUGE
GSum: A General Framework for Guided Neural Abstractive Summarization
Reddit XSum WikiHow PubMed NYT R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L    score of each output of the four guidance signals and pick the best one. As shown in Table 6, despite the fact that the highlighted sentence signal achieves the best overall performance, it still underperforms one of the other three varieties of guidance more than 60% of the time. In addition, by aggregating their best outputs together, we can achieve a ROUGE-1/L point of 48.30/45.15, which significantly outperforms any single guided model. Further, we try to aggregate these guidance signals in a pairwise manner, and Table 7 demonstrates that each guidance signal is complementary to each other to some extent. Thus, we can safely conclude that each type of guidance signal has its own merits and one promising direction is to utilize a system combination method such as Hong et al. (2015) to aggregate the results together.
GSum: A General Framework for Guided Neural Abstractive Summarization
Controllability. It is also of interest what effect this guidance has on the model outputs qualitatively. We sample several generated outputs ( Faithfulness of Generated Summaries. We also evaluate whether our generated summaries are faithful to the source document. We randomly sample 100 datapoints from the test set and ask 3 people from Amazon Mechanical Turk to evaluate their factual correctness. Each person gives a score between 1 and 3, with 3 being perfectly faithful to the source document. Table 9 shows that our guided model can generate more faithful summaries compared with the baseline.
GSum: A General Framework for Guided Neural Abstractive Summarization
Necessity of Using Oracles During Training.
GSum: A General Framework for Guided Neural Abstractive Summarization
As mentioned previously, we use an oracle to select guidance signals during training. In this part, we investigate if we can provide automatically constructed guidance to the model during training as well. Table 10 shows that this methodology will lead to significantly worse performance. We con-
GSum: A General Framework for Guided Neural Abstractive Summarization
Ref.
GSum: A General Framework for Guided Neural Abstractive Summarization
nathan dailo has found a way to get his son to sleep in 42 seconds. in a youtube video he demonstrates how stroking his 3-month-old son's face with a white piece of tissue paper sends him to sleep. the video has received almost 26,000 views in just two weeks.
GSum: A General Framework for Guided Neural Abstractive Summarization
Sentence nathan dailo, from sydney, uploaded a video to his youtube channel demonstrating how he gets his three-month-old son seth to drift off in just 42 seconds. the clip that has now received almost 26,000 views sees the father gliding a piece of white tissue paper over his son's face repeatedly until he nods off. in his youtube video, nathan dailo shows that by running tissue paper over his son seth makes him sleep.
GSum: A General Framework for Guided Neural Abstractive Summarization
nathan dailo, from sydney, uploaded a video to his youtube channel demonstrating how he gets his three-month-old son seth to sleep in just 42 seconds. the clip has now received almost 26,000 views. by running tissue paper over his son seth makes him sleep.
GSum: A General Framework for Guided Neural Abstractive Summarization
the clip that has now received almost 26,000 views sees the father gliding a piece of white tissue paper over his son's face repeatedly until he nods off. in his youtube video, nathan dailo shows that by running tissue paper over his son seth makes him sleep. one grateful viewer of the video commented:'this worked for my baby 5 times already so thanks!' nathan dailo, from sydney, shows that by running tissue paper over his son seth makes him sleep in 42 seconds. the clip has now received almost 26,000 views. one viewer commented:'this worked for my baby 5 times already so thanks!'
GSum: A General Framework for Guided Neural Abstractive Summarization
Keyword video sleep dailo youtube way seconds demonstrates weeks piece received nathan dailo, from sydney, uploaded a video to his youtube channel. he demonstrates how he gets his three-month-old son seth to sleep in just 42 seconds. the clip has now received almost 26,000 views since being uploaded two weeks ago.
GSum: A General Framework for Guided Neural Abstractive Summarization
son paper sends found demonstrates way views face dailo piece nathan dailo, from sydney, posted a video to his youtube channel. he demonstrates how he gets his three-month-old son seth to sleep in just 42 seconds. the father glides a piece of white tissue paper over his son's face repeatedly until he nods off.
GSum: A General Framework for Guided Neural Abstractive Summarization
(father, gliding, piece of tissue paper), (clip, sees father over, his son's face)
GSum: A General Framework for Guided Neural Abstractive Summarization
a father from sydney has worked out a way of getting his threemonth-old son to sleep. in his youtube video, nathan dailo shows that by running tissue paper over his son seth makes him sleep in just 42 seconds. the clip has now received almost 26,000 views since being uploaded two weeks ago.
GSum: A General Framework for Guided Neural Abstractive Summarization
(nathan dailo, uploaded, video to his youtube channel), (has drifted, has received, 26,000 views) nathan dailo, from sydney, uploaded a video to his youtube channel. the clip has received 26,000 views since being uploaded two weeks ago. it shows the father gliding a piece of white tissue paper over his son's face repeatedly until he nodded off.
GSum: A General Framework for Guided Neural Abstractive Summarization
Retrieve phil smith posted the video of him. his son noah's tooth to his youtube channel which went viral with over 100,000 views. smith tied noah's tooth to dental attached to a golf ball. smith hits the ball and the tooth is gone in just seconds, made by the expression in noah's face.
GSum: A General Framework for Guided Neural Abstractive Summarization
nathan dailo, from sydney, posted a video to his youtube channel. he demonstrates how he gets his three-month-old son to sleep in just 42 seconds. the clip has now received 26,000 views since being uploaded two weeks ago.
GSum: A General Framework for Guided Neural Abstractive Summarization
karen wiltshire, 46, sends the babies to sleep by their and playing them the sound of a. it can take anything from a few minutes to half an hour for the tiny newborns to off to sleep. mother-of-two from poole, dorset, has won an award in studio children photography for her work.
GSum: A General Framework for Guided Neural Abstractive Summarization
mother-of-two from sydney, has won an award in studio children photography for her work. in a youtube video, nathan dailo shows how he gets his three-month-old son seth to sleep in just 42 seconds. the clip has now received 26,000 views since being uploaded two weeks ago.  jecture that this is because when the relevancy between guidance and reference is weakened, the model will not learn to depend on the guidance signals and thus the model will be reduced to the original abstractive summarization baseline.
GSum: A General Framework for Guided Neural Abstractive Summarization
We propose a general framework for guided neural summarization, using which we investigate four types of guidance signals and achieve state-of-theart performance on various popular datasets. We demonstrate the complementarity of the four guid-  ance signals, and find that our models can generate more novel words and more faithful summaries. We also show that we can control the output by providing user-specified guidance signals.
GSum: A General Framework for Guided Neural Abstractive Summarization
Given the generality of our framework, this opens the possibility for several future research directions including 1) developing strategies to ensemble models under different guidance signals; 2) incorporating sophisticated techniques such as copy or coverage over the source document, the guidance signal, or both; and 3) experimenting with other kinds of guidance signals such as salient 4838 elementary discourse units.
GSum: A General Framework for Guided Neural Abstractive Summarization
Input: A source document x consisting of multiple sentences {x 1 , · · · , x |x| }, its reference summary y, and a pre-defined integer N Output:
GSum: A General Framework for Guided Neural Abstractive Summarization
the highest ROUGE scores with the reference summary. We use a similar algorithm to select the relation triples as well. Concretely, we flatten each relational triple (s, r, o) by concatenating its elements together and treat each concatenated text as a source sentence, then use Algorithm 1 to select the relation triples greedily.
GSum: A General Framework for Guided Neural Abstractive Summarization
We perform more analysis on CNN/DM in this section. Unless otherwise stated, we use oracle extractions at training time and BertAbs as our base model.
GSum: A General Framework for Guided Neural Abstractive Summarization
In addition to the qualitative results in the main paper, we also perform a quantitative analysis to demonstrate the controllability of our models.
GSum: A General Framework for Guided Neural Abstractive Summarization
The quantitative results in Table 3 of the main text already demonstrate to some extent that we can control the model with guidance signals, as guidance signals of better quality can lead to better summaries. To further demonstrate this, we randomly sample guidance signals multiple times and plot the correlation between guidance quality and output quality in Figure 4. We can clearly see that there is a strong correlation between these two variables, indicating the controllability of our model.   Table 11: We divide each summary reference into two halves and deduce the oracle guidance from them separately. Feeding incompatible guidance signals can lead to degraded performance.
GSum: A General Framework for Guided Neural Abstractive Summarization
In addition, we try to divide each test reference summary into two halves, then use oracle extraction to obtain guidance signals for both of these two halves and feed them to the model. Table 11 shows that feeding incompatible guidance signals can lead to degraded performance, which further demonstrates that we can control the summary through provision of user-specified inputs.
GSum: A General Framework for Guided Neural Abstractive Summarization
To evaluate the semantic similarities between our model outputs and the reference, we also compute the METEOR scores (Banerjee and Lavie, 2005). As shown in Table 12, all of our guided models can outperform BertAbs in temrs of both of METEOR. However, it is surprising that BertExt achieves the Table 12: Semantic similarity evaluation. We report results both in exact match mode (rewarding exact matches between words) and full mode (rewarding matching stems, synonyms and paraphrases as well).  best performance, possibly because METEOR has a tendency to favor long summaries.
GSum: A General Framework for Guided Neural Abstractive Summarization
Besides human evaluation, we have also tried to use factCC (Kryściński et al., 2019) 4 to evaluate the factual correctness of our model outputs automatically. However, as shown in Figure 5, the factCC tool will give the gold reference an accuracy of about 10%. Considering our model is optimized towards the gold reference, the factCC score might not be a good indicator of whether there are factual errors in a generated summary. 
GSum: A General Framework for Guided Neural Abstractive Summarization
We have demonstrated in the main paper that it is necessary to use an oracle to select guidance signals during training for highlighted sentence models. In this part, we investigate if this is true for all the three guidance signals as well. Table 13 shows that this methodology will lead to significantly worse performance for other guidance signals as well, which further verifies our hypothesis that when the relevancy between guidance and reference is weakened, the model will not learn to depend on the guidance signals and thus the model will be reduced to the original abstractive summarization baseline.
GSum: A General Framework for Guided Neural Abstractive Summarization
We also evaluate the performance of our highlighted sentence-guided models under domain adaptation settings, namely train a summarization model on one dataset and test it on some other datasets. As shown in Table 14, generally, extractive models can outperform abstractive ones under domain adaptations settings and our model can achieve better performance than abstractive baselines. However, while our model is given the extracted sentences by the extractive model, we still cannot outperform extractive baselines. These negative results indicate that our model may still fail to fully depend on guidance signals when doing adaptation. Possible future directions include dropping out the input documents occasionally during training so that the model can learn to better condition on the guidance.  
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
CTRLsum: Towards Generic Controllable Text Summarization
Neural summarization systems aim to compress a document into a short paragraph or sentence while preserving key information. There are two common categories of summarization systems: extractive summarization where models find and copy important portions of a documents (Cheng and Lapata, 2016;Nallapati et al., 2017;Narayan et al., 2018), and abstractive summarization where models freely generates novel sentences (Rush et al., 2015;See et al., 2017;Paulus et al., 2018). The focus of this work is on abstractive summarization.
CTRLsum: Towards Generic Controllable Text Summarization
In a common setting, summarization methods take a source document as input and yield a generic summary which covers content selected arbitrarily by the model. However, to be useful, automatically generated summaries should cover content considered important by the readers. For example, Figure 1 shows an NBA basketball news article, and the reference summary aggregates the results of several matches. Yet fans of certain players or teams might only be interested in the matches and statistics which include their entities of interest.
CTRLsum: Towards Generic Controllable Text Summarization
Motivated by this observation, in this work we focus on controllable summarization which allows the users to control the generated summaries. We propose CTRLSUM, a framework to control summaries through a set of keywords. At training time, the model learns to predict summaries conditioned on both the source document and keywords, which are easily identified from training summaries. During inference, a control function is designed depending on the specific control aspect to map user preference to keywords to control the summary.
CTRLsum: Towards Generic Controllable Text Summarization
While simple, keywords provide a clean separation of test-time user control and the training process -different dimensions of the generated summary could be controlled through engineering the test-time control function, while the training process and pretrained model remain unchanged. In contrast, typical controllable summarization methods (Fan et al., 2018;Liu et al., 2018;Tan et al., 2020;Suhara et al., 2020;Chan et al., 2021) predefine the specific control aspect (e.g. entity, length, topic, etc.) and rely on the corresponding control annotations during training. Consequently, they require to train a separate model for each control aspect and cannot generalize to new control aspects at test time. Figure 1 demonstrates such distinction. The training process in CTRLSUM is agnostic to the specific control aspect, leading to possibly  generic controllable summarization with a single model.
CTRLsum: Towards Generic Controllable Text Summarization
CTRLSUM is closely related to recent keywordguided summarization methods. While they mainly focus on improving the summary quality in traditional, unconstrained summarization tasks (Li et al., 2018;Elsahar et al., 2021;Saito et al., 2020b;Dou et al., 2021), or only study a specific control aspect like length control (Saito et al., 2020a), we generalize keyword-guided summarization as a generic controllable summarization framework, and explore its novel applications to a wide range of control tasks in this paper. In addition, inspired by the multi-task ability of language models through prompting (Radford et al., 2019;Brown et al., 2020), we further study the possible combination of keywords and prompts in CTRLSUM for more generic control purposes.
CTRLsum: Towards Generic Controllable Text Summarization
We use pretrained BART (Lewis et al., 2020) as the underlying architecture and perform experiments on three datasets: CNN/Dailymail news articles (Hermann et al., 2015), arXiv scientific papers (Cohan et al., 2018), and BIGPATENT patent documents (Sharma et al., 2019). We quantitatively evaluate CTRLSUM on five control aspects:
CTRLsum: Towards Generic Controllable Text Summarization
(1) entity-centric ( §4.2) and (2) length-controllable summarization ( §4.3), (3) summarizing the contributions of scientific papers, (4) summarizing the purpose of an invention ( §4.4), and (5) summarizing answers to given questions in a zero-shot reading comprehension setting ( §4.5). Notably, our approach also achieves comparable or superior performance to the strong BART baseline on all datasets in a standard, unconstrained setting ( §4.6).
CTRLsum: Towards Generic Controllable Text Summarization
We released our trained CTRLSUM checkpoints to encourage others to try CTRLSUM for their own control of interest. On a higher level, we expect the exploration in this paper to attract attentions to the controllable summarization task, and provide a springboard for future research on personalized summarization systems.
CTRLsum: Towards Generic Controllable Text Summarization
Unconstrained neural summarization models are trained to learn the conditional distribution p(y|x), where x and y represent the source document and summary respectively. In this setting, generated summaries depend solely on the document x without requiring human involvement. To control the content of generated summaries, we propose using additional keywords z to represent user preferences and training a model that predicts the conditional distribution p(y|x, z). As shown in Figure 1, keywords act as an interface between users and an otherwise black-box neural model.
CTRLsum: Towards Generic Controllable Text Summarization
Formally, at training time keywords are identified using source documents and ground-truth summaries while at test time they are a function of source documents and user control signal:
CTRLsum: Towards Generic Controllable Text Summarization
where f denotes the automatic keyword extraction algorithm, c is the user control signal, and g control represents the control function. c can be instantiated as different forms of control signals (e.g., entity words or length value) as users or control tasks need during test stage while not requiring re-training the model. The control function admits different designs depending on the control task. We detail g control next, but describe the automatic keyword extraction later in §2.3.
CTRLsum: Towards Generic Controllable Text Summarization
g control (x, c) maps the user control signal c to specific keywords. Given a control task (e.g. entity or length control), g control can be designed specifically to perform the task, for example, in entity control c = e is the entity of interest and g control may directly output e as the keyword; in length control c = l is the desired summary length value, in this case g control may automatically extract a certain number (which depends on l) of keywords from the source document. This way, g control provides clean separation of test-time user control and the training process -through only engineering the control function, the same trained model can be adapted to new control tasks without changing model parameters or the training process. Such a framework is generic and broadly applicable to many control aspects, which stands in contrast to previous work on controllable summarization (Fan et al., 2018;Liu et al., 2018) which pre-define one control aspect and design a specific training procedure for it. In §3 we will show how the same model is applied to five different controllable summarization tasks. A note on the scope of focus: Mathematically, g control (x, c) covers a use case where the user may directly input arbitrary keyword sequences at test time. 2 However, such an open-ended setting is poorly defined as a control task -the specific control aspect is unconstrained and unclear, which brings challenges to the evaluation that is already difficult in controllable generation due to lack of references. Therefore, we focus on several specific, well-defined control tasks as to be shown in §3.
CTRLsum: Towards Generic Controllable Text Summarization
In this paper we do not explore CTRLSUM as an open-ended system, but leave it as future work.
CTRLsum: Towards Generic Controllable Text Summarization
As shown in Eq. 1, the training keywords z train depend on both the source x and reference y, while z test are manipulated according to the control signal.
CTRLsum: Towards Generic Controllable Text Summarization
We further provide g control (x, c) with an optional automatic keyword extraction method at inference time, which allows CTRLSUM to perform unconstrained summarization when the control signal c = ∅. It may also be useful in control tasks where automatic keywords from x need to be first extracted. Below we describe them respectively: At Training Time: We first greedily select sentences from the document that maximize the ROUGE scores (Lin, 2004) with the reference summary. 3 This step constrains keywords to those found in important sentences. Next, we identify all of the longest sub-sequences in the extracted sentences that have matched sub-sequences in the ground-truth summary. Finally, we remove duplicate words and stop words and keep the remaining tokens as keywords. Compared to typical keywords extraction methods (Riloff and Lehnert, 1994;Mihalcea and Tarau, 2004) which output only a few salient words, our extraction retains most content words found in the summary. This is to build a reliable correlation between their presence in the input and the target, ensuring that keywords are not ignored by the model at test time. At Inference Time: We formulate keyword extraction as a sequence labeling task. Concretely, we train a BERT-based sequence tagger (Devlin et al., 2019) on the keywords and documents from training dataset. This tagger then computes the selection probability q j for each token in the test document. Similar to training time extraction, we first select n s sentences with the highest average token selection probability. Within these sentences words with q j > ϵ are selected as keywords up to a maximum number of m max . n s , ϵ, m max are hyperparameters selected based on the unconstrained summarization performance on validation datasets, and the results are reasonably robust to different values, as shown in Appendix G.
CTRLsum: Towards Generic Controllable Text Summarization
We train the CTRLSUM model to maximize p(y|x, z). Specifically, we prepend the keyword sequence z to the source document x separated with a special token =>, then the resulted sequence is taken as the input to the model to predict the summary. Our training procedure is model-agnostic so that CTRLSUM can employ any sequence-tosequence architecture as the backbone without Source or Reference
CTRLsum: Towards Generic Controllable Text Summarization
Source: Hundreds of additional Iraqi troops are being sent to reinforce colleagues who are trying to fend off ISIS' attempt to overrun Iraq's largest oil refinery, a key paramilitary force said Tuesday. The reinforcements come four days after ISIS began attacking northern Iraq's Baiji oil refinery, a key strategic resource that has long been a target because the facility refines much of the fuel used by Iraqis domestically. Reference: multiple sclerosis is a degenerative condition of the central nervous system that affects nearly 2.5 million of individuals in terms of their [ignoring 60 tokens]. in this work we resort to regularized machine learning methods for binary classification and multiple output regression we propose a pipeline that can be used to predict the disease progression from patient reported measures . the obtained model is tested on a data set collected from an ongoing clinical research project .
CTRLsum: Towards Generic Controllable Text Summarization
[the main contributions of this paper are : ( 1 )] -we propose a regularization -based data analysis pipeline for the prediction of the transition point between relapsing -remitting and secondary -progressive forms of multiple sclerosis ;
CTRLsum: Towards Generic Controllable Text Summarization
( 2 ) we test the performance of the proposed pipeline on a data set collected from patients currently enrolled in an ongoing funded project ;
CTRLsum: Towards Generic Controllable Text Summarization
( 3 ) we discuss the potential role of patient reported outcome measures ( _ proms _ ) in the study of the disease evolution .
CTRLsum: Towards Generic Controllable Text Summarization
Reference: an implantable intraocular pressure sensor system has a sealed geometric shape with an internal pressure at a first value . the sealed geometric shape has a first light permitting surface and a second flexible surface . a pair of photocells is located in the sealed geometric shape . a light shield is coupled to the second flexible surface . when the second flexible surface is deflected , a light measurement by the pair of photocells indicates an intraocular pressure condition .
CTRLsum: Towards Generic Controllable Text Summarization
[the purpose of the present invention is] -to provide an intraocular pressure measuring system that is small enough to be implanted in the eye of a patient modifications. We note that this is much simpler than recent keyword-based summarization systems (Li et al., 2018;Saito et al., 2020a;Dou et al., 2021) where they tweak the specific model architecture to incorporate the keywords signal.
CTRLsum: Towards Generic Controllable Text Summarization
Keyword Dropout: As in §2.3, our training keyword extraction retains most content words from the summary. However, we found that such keywords are empirically too informative so that the model learns to ignore the source document while generating the summary. To remedy this, we randomly drop keywords at training time so that the model learns to rely on keywords, while also learning to still carry over key information from the source document. Note that keyword dropout is applied at training time only.
CTRLsum: Towards Generic Controllable Text Summarization
In this section we define five controllable summarization tasks that we study in this paper and the respective control function g control . Qualitative examples of each task are shown in Table 1.
CTRLsum: Towards Generic Controllable Text Summarization
Entity Control: The goal is to produce summaries that focus on specified entities of interest. We directly use the entity word as keyword, i.e. g control (x, c = e) = e where e is the entity. Length Control: We allow manipulation of the summary length through a user-specified length parameter. Specifically, we first separate the training data into 5 buckets by summary length so that each bucket has the same number of examples. Then we compute the average number of keywords K l for each bucket on the training data. At test time, a user can specify a length value c = l ∈ {0, 1, 2, 3, 4} and g control returns the top K l keywords with the highest selection probability computed by the tagger. This is similar to (Saito et al., 2020a); they use the number of "guiding words" to control summary length, which can also be viewed as an instantiation of the CTRLSUM framework. Beyond above, more complicated logic may be incorporated into g control to perform other control tasks -for instance, to achieve topic control, it is possible to select keywords that are related to a given topic word using external knowledge sources. Similar approach has been used in (Tan et al., 2020) for aspect-controlled summarization, whereas they require a specifically tailored training procedure. We leave such exploration as future work.
CTRLsum: Towards Generic Controllable Text Summarization
Inspired by recent advances on prompting pretrained models (Liu et al., 2021), we explore to combine keywords and prompts to perform several different controllable summarization tasks. 4 Specifically, we design a guiding token sequence for each task, which is used as both the keyword input and the decoder prompt. In this case the control signal c takes the task as a value, and g control (x, c) outputs the guiding sequence directly given the task c for any x. While traditional summarization systems are able to utilize prompts to constrain and control the generation as well, we emphasize that the combination with keywords turns out to be importantkeywords help focus on control-related content on the encoder side, compared to prompt alone which only impacts the decoder, thus "keyword+prompt" yields the best performance as we will show in §4.4. Next we define and describe three control tasks that we study in this paper.
CTRLsum: Towards Generic Controllable Text Summarization
Summarizing paper contributions: Existing datasets about scientific papers such as arXiv (Cohan et al., 2018) collect paper abstracts as the summaries, which often include extra background context and lack detailed contribution descriptions for the associated paper. In many cases, readers would benefit from an explicit list of contributions to understand the novelty of the paper. For these cases, we propose g control ="the main contributions of this paper are:(1)" as the keyword and prompt to generate summary focused on contributions.
CTRLsum: Towards Generic Controllable Text Summarization
Summarizing invention purpose: Patent article summaries in existing datasets such as BIG-PATENT (Sharma et al., 2019) can be overcomplicated, often covering core method details. Yet for a non-technical reader it would be preferred to provide a one-sentence summary on the purpose of the invention while ignoring technical details. Thus we set g control ="the purpose of the present invention is" to yield a concise summary on patent purpose.
CTRLsum: Towards Generic Controllable Text Summarization
summarization: Human summarization can be constrained by questions (Kryscinski et al., 2019) that require answers to be found in the summary. This points to a connection between summarization and reading comprehension that we further explore. We hypothesize that a summarization model may directly answer some questions about the article if guided properly. This suggests the possibility of subsuming reading comprehension as a form of summarization. To verify this, we set g control ="Q: question text? A:" to produce reading comprehension behaviour.
CTRLsum: Towards Generic Controllable Text Summarization
The guiding sequences we chose in this section are randomly-picked without any tuning, while in Appendix F we show that the performance is relatively robust across several human-written guiding sequences. Recent work on prompt engineering suggests that the guiding sequences may be further optimized automatically (Shin et al., 2020;Li and Liang, 2021), whereas they are not directly applicable without training data in the control tasks.
CTRLsum: Towards Generic Controllable Text Summarization
We note that keyword-and prompt-based control are complementary in practice -while prompts could ideally achieve any type of control, empirically they often do not work well for many aspects. For example, we found that using prompts alone (e.g. "a summary focused on [entity] is:") work poorly to control entities or length. Similarly, keyword alone struggles to perform paper contribution or patent purpose summarization. These empirical observations validate the importance of combining keyword and prompt for more generic controllable summarization. Please find quantitative analysis on this in Appendix E.
CTRLsum: Towards Generic Controllable Text Summarization
In this section we test CTRLSUM quantitatively, and an extensive list of qualitative output examples in all the control tasks can be found in Appendix H.
CTRLsum: Towards Generic Controllable Text Summarization
We perform experiments on three distinct-domain summarization datasets: CNN/Dailymail (CN-NDM) news articles (Hermann et al., 2015), arXiv scientific papers (Cohan et al., 2018), and BIG-PATENT patent articles (Sharma et al., 2019). The distribution p(y|x, z) in CTRLSUM is our fine-tuned version of the pretrained BART LARGE model (Lewis et al., 2020). Complete setup and training details can be found in Appendix A.1.   Evaluation: As emphasized in §2.2, we focus our evaluation on the five well-defined tasks. We measure ROUGE scores (Lin, 2004) and the recently proposed BERTScore (Zhang et al., 2020b) when ground-truth is available. In control task where we often do not have reference summaries, we (1) collect ground-truth summaries when possible, (2) examine whether summaries respect the control signal, or (3) resort to human evaluation.
CTRLsum: Towards Generic Controllable Text Summarization
Oracle entity setup: We first follow the evaluation in (Fan et al., 2018) to simulate a possible user preference by providing the model with oracle entities extracted from the ground-truth target. We note that this setting "cheats" by revealing the reference information and should be viewed as a sanity check experiment only to verify whether the model can utilize the input entities. Any entity setup: We assess whether the model can deal with diverse preferences where every entity in the document is a possible input. To this end, we sample 100 test documents and repeatedly acquire every entity in the document to generate summaries, following Fan et al. (2018). Then we compute Success Rate, the fraction of requested entity actually occurring in the output. The results are reported in separation of whether the entity is from leading 3 sentences or the full article. To test if the summaries are factually consistent with the document, for each of the 100 documents we randomly sample one "important" entity that appears in the reference, and one "unimportant" entity that occurs neither in the reference nor the leading three source sentences. For each (article, summary) pair we ask 3 annotators from Amazon Mechanical Turk to make a binary decision as to whether the summary can be entailed from the article. We then take the majority vote and report the fraction of factually correct summaries. We evaluate on CNNDM only since many examples in arXiv and BIGPATENT do not have identifiable entities.
CTRLsum: Towards Generic Controllable Text Summarization
Results: In Table 2 we observe that the use of oracle entities helps boost the ROUGE-2 score 5 compared with using automatic keywords, which indicates that CTRLSUM is able to take advantage of the given entities. Table 3 shows the Success Rate and factual correctness evaluations. We include the numbers from (Fan et al., 2018) (Entity-Code) as a reference point which requires entity annotations during training time and is based on a convolutional seq2seq model. 6 Remarkably, our model achieves a high success rate for both lead-3 and full-article entities reaching around 95%. Factual correctness scores from human annotators suggest that CTRLSUM is able to generate factually consistent summaries regardless of the importance of input entities, comparable to the BART baseline.
CTRLsum: Towards Generic Controllable Text Summarization
Oracle length setup: Similar to entity control, we examine the effect of oracle length signal from the reference for a sanity check.
CTRLsum: Towards Generic Controllable Text Summarization
Any length setup: To account for any possible length control signal (any length), we sample 1000 test documents and iterate every length bucket to decode several summaries for each document, and we measure whether the generated summaries follow the specified length signal. Specifically, we compute the mean of absolute deviation (MAD) of the actual length bucket code l sys of the decoded summary from the given length l ref ,  
CTRLsum: Towards Generic Controllable Text Summarization
We also report the Pearson Correlation Coefficient (PCC) between the input bucket code and actual bucket code. We compare with LengthCode (Fan et al., 2018), where the ground-truth length bucket code is prepended to the article at both training and test time. Experiments are conducted on CNNDM and arXiv. Results: In Table 2 CTRLSUM with oracle length signals only presents relatively small gains over the automatic CTRLSUM baseline. This implies that oracle lengths only convey limited extra information to help generate the reference. Table 4 includes results for the any length setup. Length-Code fails to consistently improve over BART with oracle length signals. Moreover, we find that LengthCode almost ignores the length signal with PCC close to 0, as shown in Table 4. This is possibly because that length code would be less helpful and thus more likely to be ignored when the model grows stronger through pretraining. 7 In contrast, CTRLSUM with length-guided keywords achieves high PCC between control signal and the actual output length, and is able to reduce the length deviation MAD compared to unconstrained baselines.
CTRLsum: Towards Generic Controllable Text Summarization
Contribution Summarization Setup: No existing dataset evaluates contribution summarization of scientific papers. However, researchers often summarize the bullet contributions of their paper in the Introduction section, which inspires us to extract them as the reference summary. Therefore, we resort to the entire arXiv database, 8 and download all the papers whose first submission date is within 7 We verified that LengthCode using the BART model but randomly initialized weights obtained a high PCC score. 8 https://www.kaggle.com/Cornell-University/arxiv  Results: Table 5 shows the results. Compared with BART which uses the prompt alone, CTRL-SUM achieves superior performance in most cases through using the guiding text as both the keywords and prompt. We further report the precision (P) and recall (R) scores in BERTScore besides F1. We observe that BART with prompt alone over-generates a full summary with low precision scores especially when a concise summary is desired in patent purpose summarization. CTRLSUM is able to mitigate this issue through keywords which guide the encoding process of encoder which prompts do not.
CTRLsum: Towards Generic Controllable Text Summarization
Setup: We directly test question-guided summarization on reading comprehension benchmarks in Model NewsQA SQuAD v1.1 Supervised SpanBERT (Joshi et al., 2020) 73.0 94.6 MatchLSTM (Wang and Jiang, 2017) 49  We report F1 scores on the dev set.
CTRLsum: Towards Generic Controllable Text Summarization
Results: BART is pretrained with a denoising task to denoise the source, and unsurprisingly obtains poor results in the zero-shot setting, as shown in Table 6. Interestingly, however, BART fine-tuned on a summarization task -without seeing any question-answer pairs -is able to improve the F1 scores by 24.4 and 25.9 points on NewsQA and SQuAD respectively. Moreover, CTRLSUM with keywords and prompts further boost F1 by 15.6 and 17.9 points, approaching the supervised MatchLSTM score on NewsQA. Such results suggest that summarization might be a suitable transfer task for abstractive reading comprehension, which we leave for future work to explore. Table 7 shows the unconstrained summarization performance without any user input, where CTRL-SUM uses the automatically extracted keywords from the tagger as described in §2.3. On CNNDM and arXiv datasets CTRLSUM (Automatic Keywords) decently outperforms the strong BART and PEGASUS baselines. It also performs comparably to BART on BIGPATENT in terms of BERTScore, though with an inferior ROUGE-2 score. There is a performance gap between BART-based models and PEGASUS on BIGPATENT, possibly due to the 10 Some NewsQA test articles are in the CNNDM training dataset, yet we still view it as a unsupervised setting since our model never sees questions or answers during training. inherent difference between BART and PEGASUS.
CTRLsum: Towards Generic Controllable Text Summarization
We present human evaluation results on controllable summarization tasks, but we refer the readers to Appendix B for human evaluation on unconstrained summarization. Full setup details are in Appendix A.2. We focus on entity and purpose control. Specifically, we inform the annotators our intent (which is to obtain summaries focused on a specific entity or purpose of patent), then we ask them to provide scores over two dimensions: (1) Control Accuracy (CA): whether the summary contains accurate main information of the intent, and (2) Control Relevance (CR): how the summary is relevant to the control intent overall -a summary that contains redundant contents that are unrelated to the intent will be penalized. Results including significance tests are shown in Table 8. The control accuracy for important entity control and purpose control are comparable between BART and CTRLSUM without significant difference (p-value > 0.05), while CTRLSUM shows better control relevance overall by focusing on the desired information. Also, BART fails to generate unimportantentity-related summaries with poor scores.
CTRLsum: Towards Generic Controllable Text Summarization
In this paper we propose a generic framework, CTRLSUM, to perform generic controllable summarization through a set of keywords. Experiments on five different control aspects demonstrate the efficacy of our method.
CTRLsum: Towards Generic Controllable Text Summarization
We thank Pengfei Liu for helpful discussion and advice on this paper.
CTRLsum: Towards Generic Controllable Text Summarization
(1) The ability of CTRLSUM to deal with arbitrary input keywords (especially when the keywords are not present in the source) is not sufficiently explored and largely limited by the underlying pretrained model weights (i.e. BART in our experiments).
CTRLsum: Towards Generic Controllable Text Summarization
(2) CTRLSUM is not taught to handle "unrelated user input" which may raise unwanted model behaviors. For instance, if a user inputs "war gun kill" as the keywords by mistake to an NBA news article, then the model may make up factually incorrect content that includes the keywords. (3) CTRLSUM requires an additional tagger to perform unconstrained summarization in a pipeline fashion which is more complicated than typical summarization models. This may be resolved by training CTRLsum with a multi-task objective, using special keyword symbols or empty keywords to trigger the unconstrained summarization task. The training of CTRLsum leads to strong dependence of the summaries on the input keywords, thus the summaries would almost always include the keywords in the generated summary from our observation, even when the keywords do not appear in the article or represent toxic information. This allows humans to deliberately create malicious content from CTRLsum, for example, through inputting racism or aggression words. A potential remedy is to train a toxicity classifier, and let the model replace the toxic keywords with automatic keywords.
CTRLsum: Towards Generic Controllable Text Summarization
The training of CTRLSUM leads to strong dependence of the summaries on the input keywords, thus the summaries would almost always include the keywords in the generated summary from our observation, even when the keywords do not appear in the article or represent toxic information. This allows humans to deliberately create malicious content from CTRLSUM, for example, through inputting racism or aggression words. A potential remedy is to train a toxicity classifier, and let the model replace the toxic keywords with other automatic keywords.
CTRLsum: Towards Generic Controllable Text Summarization
ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. . For all BARTbased summarization models, we fine-tune with learning rate 3e-5 and a polynomial learning rate decay schedule, the optimizer is Adam (Kingma and Ba, 2015) and batch size is 64. Our optimization scheme and hyperparameters follow the BART fine-tuning instructions in fairseq examples. We train the summarization models with 20k steps on CNNDM, 50k steps on arXiv, and 300k steps on BIGPATENT. We train the BERT tagger with learning rate 5e-5, Adam optimizer, and batch size of 128 on all datasets. Similar to summarization models, the tagger is trained with 20k, 50k, and 300k steps on CNNDM, arXiv, and BIGPATENT respectively. Also, we adopt a sliding window approach so that the BERT-based tagger is able to handle sequences that are longer than 512 tokens. 11 For both ROUGE and BERTScore evaluation, we report the F1 measure. We report the rescaled BERTScore, and the hash code is "roberta − large_L17_no − idf _version = 0.3.6(hug_trans = 3.0.2) − rescaled". As mentioned in §2.3, we need three hyperparameters for automatic keywords extraction during inference when applicable -the number of pre-selected sentences n s , the selection probability threshold ϵ, and the maximum number of keywords m max . We select these hyperparameters for each dataset based on the unconstrained summarization ROUGE-2 score on validation dataset. The summarization performance is robust to these hyperparameters in a reasonable range, as shown in Appendix G. Specifically, we use {n s = 10, ϵ = 0.25, m max = 30} for CNNDM, {n s = 10, ϵ = 0.15, m max = 40} for arXiv, and {n s = 5, ϵ = 0.15, m max = 30}. The order and sentence boundary of automatic keywords: for the automatic keywords extracted as in §2.3, the keyword sequence maintains the order of the keywords as they were in the source document, but we observe that the model often ignores this ordering information as it frequently differs between source and target summary. 12 We also separate keywords from different source sentences with the special token ("|"), though in preliminary experiments we find that the model generation is not sensitive to the sentence boundary information at test time. At inference time in applications where the sentence boundary is unknown, as in most of our control tasks, the "|" token is ignored in the experiments. Invention Purpose Summarization: in the experiment of summarizing invention purpose on patent articles ( §4.4), we examined whether the model would possibly copy source sentences through matching the prompts, we search strings in the form of "the purpose of [some words or phrases] is" among 763 test examples, and only 3 test articles are identified. This means the models are not generating by exactly matching prompts most of the time. Question-Guided Summarization: In Table 6, the GPT2-Large has 774M parameters while the BART architecture (including CTRLSUM) has 406M parameters. A note on the unconstrained summarization experiment: in our unconstrained summarization experiment ( §4.6), we note that the BERT tagger gives CTRLSUM parameter advantages over the baselines, leading to a strictly unfair comparison. Such a (unfair) parameter advantage setting is also adopted in recent two-stage summarization work (Dou et al., 2021). We constrain the seq2seq model in CTRLSUM to be the same as the baseline to better directly observe the effect of keywords. Also, given that unconstrained summarization is not the main focus of this paper, we leave a strict parameter-control experiment for future work. GPU resources. All our models including the summarization model and tagger are trained on 8 NVIDIA Tesla 16GB V100 GPUs using a dataparallel mechanism. The GPUs are provided by Google Cloud.
CTRLsum: Towards Generic Controllable Text Summarization
Here we include details about human evaluation experiments in §4.7. Unconstrained Summarization. For unconstrained summarization, we sample 100 examples for each dataset, and hire workers from Amazon Mechanical Turk to conduct evaluation. Each example is scored by 3 independent workers and the resulted total number of participants are 15, 20, 22 for CNNDM, arXiv, and BIGPATENT respectively. We set the price of tasks in a way that will guarantee each worker an hourly wage of at least $12. The median score of 3 workers is taken for each example, and average over all examples is reported. For CNNDM we provide article and summaries, while for arXiv and BIGPATENT we provide reference and summaries using the reference summary as a surrogate for the source article. This is because the source patent documents or scientific papers are very long and hard to be read by non-expert humans. For each aspect to be scored (i.e. factual consistency, relevance, fluency, or coherence), we provide specific definitions of the aspect. Then for each example we present summaries from different systems together in random order to be evaluated. Note that different aspects are presented and scored separately since we noticed that the MTurkers tend to mix different aspect concepts and fail to distinguish them during evaluation if the four aspects are scored together. 13 An example screenshot of the instructions shown to the workers is demonstrated in Figure 2. Controlled Summarization. For controlled summarization, we sample 100 examples for each task, and summaries of each example from different systems are presented together in random order to the human annotator to be scored. For CNNDM we provide the source article, the control entities, and summaries, while for BIGPATENT we provide reference and summaries using the reference summary as a surrogate for the source article. This is because the source patent documents are very long and hard to be read by non-expert humans. We did not evaluate contribution summarization since it is unrealistic to ask non-expert humans to judge contributions of many scientific papers from various domains. We tried to hire workers from Amazon Mechanical Turk first, but unfortunately we failed to obtain reliable results from them -they often ignored the given control intent and tended to score the text as standard unconstrained summaries, which is reflected by very poor scores on unimportant-entityfocused summaries because these summaries do not focus on the important information of the article, even though we instructed them that the control intent is critical. We note that obtaining reliable human judgement for summarization from non-expert crowdsourcing workers remains an active research direction (Shapira et al., 2019), and it has been shown before that non-expert human judgement of summarization can exhibit poor correlation with expert judgement (Fabbri et al., 2021).
CTRLsum: Towards Generic Controllable Text Summarization
To obtain more trustable human evaluation results for controlled summarization, we ask three our colleagues to evaluate through personal correspondence from the authors of this paper. They are able to follow the given control intent from our manual investigation and considered more reliable than the MTurkers. While we report important and unimportant entity control results separately, we abstract away the "important/unimportant" entity identities from human annotators and mix them for the annotators to score. Similar to the unconstrained setting, each example is evaluated by 3 annotators independently. Then we take the median score for each example and average over all examples to report the final score. Example screenshots of the instructions shown to the human annotators are demonstrated in Figure 3.
CTRLsum: Towards Generic Controllable Text Summarization
We follow (Grusky et al., 2018;Fabbri et al., 2021) to ask human annotators from Amazon Mechanical Turk to score summaries over four dimensions: (1  "${}" denotes placeholders of the source article or different system outputs (they are replaced by actual text when presented to the MTurkers). This figure shows the aspect of "consistency" and "fluency" for an example, while we have additional two aspects of "relevance" and "coherence" scored. Different aspects are scored separately so that the MTurkers can better distinguish them. As shown in this figure, the annotators are provided with specific definitions of the aspect. The "fluency" dimension is scored without providing the source article to remove the potential source confounder since the fluency of the generated text piece is unrelated to the source article. Different system outputs are presented in random order.     shows the entity control evaluation example on CNNDM and (b) demonstrates the purpose control instructions on BIGPATENT. We present different system outputs in random order to remove order biases, which means that the generations in the same column ("summary1" or "summary2") may not be from the same system.
CTRLsum: Towards Generic Controllable Text Summarization
In   Table 12: Ablation analysis on the role of keyword and prompt respectively. Entity success rate refers to the full article entity success rate. K denotes keyword while P denotes prompt.
CTRLsum: Towards Generic Controllable Text Summarization
In this paper CTRLsum uses control tokens either as keywords alone (entity and length), or as both keywords and prompts (contribution, purpose, QA).
CTRLsum: Towards Generic Controllable Text Summarization
Here we present further results when control tokens are used as prompts, keywords, or both for entity control, contribution control, and NewsQA tasks. Specifically for entity control, we use the control tokens "a summary focused on [entity] is:" for "prompt" and "prompt + keyword" variants. 15 In this case success rate is computed excluding the prompt text. The control tokens for other settings are the same as previous experiments. Results are shown in Table 12, where keywords and prompts are of different importance for different tasks and are complementary in general. For example, using prompts to control entities turns out to be difficult with a very low success rate -we find that the system fails to understand the prompt and produce summaries appropriately in most cases. However, prompts contribute the most to contribution summarization with comparable performance with using prompts and keywords together, while removing prompts and using keywords alone suffers from drastic performance drop to trigger the contribution. For NewsQA task, prompts and keywords demonstrate mixing effectiveness -using either of them alone experiences over 20 F1 points loss compared to using them together.
CTRLsum: Towards Generic Controllable Text Summarization
In §4.4 and §4.5 we used randomly picked prompts without any tuning process, which represents a practical setting where we often do not have supervision for the specific control tasks at hand. This stands in contrast with recent prompt optimiza- 15 We tried several prompts variants, for example, QA style ones "Q: What happened to [entity]? A:" or "Q: What do we know about [entity]? A:". None of them lead to meaningful entity control. tion methods which require a non-trivial amount of training examples to work well (Shin et al., 2020;Li and Liang, 2021;Perez et al., 2021). To analyze the model's sensitivity to different random prompts, we study contribution and purpose summarization with three randomly-picked prompts for each, and report the mean and standard deviation of ROUGE scores. Table 13 shows the three specific prompts we use, and Table 14 demonstrates the results. While we cannot iterate over all possible prompts, these results indicate that CTRLsum can work reasonably well with a reasonable set of simple, randomly-picked, human-written prompts.
CTRLsum: Towards Generic Controllable Text Summarization
Contribution the main contributions of this paper are: (1) the contributions of this paper can be summarized as: (1) the contributions of this paper are: (1) Purpose the purpose of the present invention is the goal of the invention is the invention is designed with the goal of Table 13: The prompts we use for contribution and purpose summarization respectively. We use three different prompts for each to analyze model's sensitivity to different prompts. Table 15 shows the ROUGE-2 scores of unconstrained summarization on the validation set with different keywords extraction hyperparameters. We use more fine-grained stride size to iterate the m max hyperparameter for CNNDM since its source articles are usually shorter than arXiv and BIG-PATENT. As observed, the automatic summarization performance is relatively robust to these hyperparameters in a reasonable range.  Table 14: Summarization performance on contributions of papers and purpose of inventions. The BART baseline uses prompts (P) while CTRLsum uses both keywords (K) and prompts (P). We run with three different randomlypicked prompts and report the mean and standard deviation (in the bracket) of scores.  
CTRLsum: Towards Generic Controllable Text Summarization
In this section, we randomly sample test examples and show the source aticle, reference summary, and the model output from CTRLsum for each control aspect.
CTRLsum: Towards Generic Controllable Text Summarization
For entity control, we randomly sample 3 articles from CNNDM and for each article we randomly select 5 entites as keywords to show the model output.
CTRLsum: Towards Generic Controllable Text Summarization
Article Americans on the United States' no-fly list will now be privy to information about why they have been banned from commercial flights and be given the opportunity to dispute their status, according to court documents filed by the Justice Department this week. The revised policy comes in response to a June ruling by a federal judge that said the old process was in violation of the Fifth Amendment's guarantee of due process. The decision was part of an American Civil Liberties Union lawsuit brought on behalf of 13 Americans on the list. But the ACLU isn't satisfied with the government's new policy, outlined in documents filed Monday in federal courts in Oregon (PDF) and Virginia (PDF). "After years of fighting in court for complete secrecy and losing, it's good that the government is finally now going to tell people of their status on the No Fly List," said Hina Shamsi, director of the ACLU National Security Project and the lead attorney on the case, in a statement. "Unfortunately, we've found that the government's new redress process falls far short of constitutional requirements because it denies our clients meaningful notice, evidence, and a hearing. The government had an opportunity to come up with a fair process but failed, so we're challenging it in court again." People on the no-fly list, managed by the FBI's Terrorist Screening Center, are prohibited from boarding a commercial flight for travel into or out of the United States. The number of people on the list is classified. An official with knowledge of the government's figures told CNN in 2012 that the list contained about 21,000 names, including about 500 Americans. Before the change, American citizens and permanent residents who inquired with the government about being denied aircraft boarding received a letter that neither confirmed nor denied their inclusion on the no-fly list. Now, they'll be made aware of their status if they apply for redress, with an option to request further information. "The U.S. government is making enhancements to the Department of Homeland Security Traveler Redress Inquiry Program (DHS TRIP) to provide additional transparency and process for U.S. citizens and lawful permanent residents who have been denied boarding on a commercial aircraft because they are on the No Fly List," the Department of Homeland Security said in a statement. In cases in which travelers included on the list request to receive or submit more information about their status, the government will provide a second, more detailed response, identifying "specific criterion under which the individual has been placed on the No Fly List," according to the court documents. An unclassified summary of that information will be provided "to the extent feasible, consistent with the national security and law enforcement interests at stake," court papers said. Those who appear on the no-fly list will then have further opportunity to dispute their status in writing, with supporting materials or exhibits, and will receive a final written decision from the Transportation Security Administration. The 2014 ruling that prompted the policy changes had called for passengers on the list to be given the opportunity to dispute their status before a judge.
CTRLsum: Towards Generic Controllable Text Summarization
Americans on the no-fly list will now get info about why they've been banned from flights. ACLU says the policy still denies "meaningful notice, evidence, and a hearing"
CTRLsum: Towards Generic Controllable Text Summarization
CTRLsum the Fifth Amendment's: The new policy is in response to a June ruling that said the old process was in violation of the Fifth Amendment's guarantee of due process.
CTRLsum: Towards Generic Controllable Text Summarization
Civil Liberties Union: The American Civil Liberties Union says it's not satisfied with the government's new policy.
CTRLsum: Towards Generic Controllable Text Summarization
the Transportation Security Administration: The Transportation Security Administration will provide travelers with more information about their status. A June ruling had called for passengers on the list to be given the opportunity to dispute their status before a judge.
CTRLsum: Towards Generic Controllable Text Summarization
Virginia: The ACLU is challenging the new policy in federal courts in Oregon and Virginia.   It's the ultimate treat for Benedict Cumberbatch fans and stands an imposing 6ft tall -just like the man himself. But shoppers at London's Westfield Stratford City shopping centre looked more than a little surprised to discover a chocolate sculpture of Benedict Cumberbatch in their midst. One lady was spotted cautiously approaching the edible artwork before quickly backing off, while another couldn't quite hide their smile of surprise. Scroll down for video . Finishing touches: The sculpture is readied for its big unveiling at Westfield Stratford City shopping centre. Oh dear: Reaction to the sculpture was mixed, with some shoppers bursting into laughter. Even less impressed was the shopper who stood stony-faced in front of the creation for several moments, while another burst into laughter as soon as she spotted it. It did, however, prove an immediate hit with a pair of police sniffer dogs who wagged their tails as they gave it a thorough sniffing down. . The artwork, which has been given pride of place in the shopping mall's atrium, was commissioned by UKTV to mark celebrate its screening of the third series of Sherlock. It took a crew of eight people to complete the sculpture, which took over 250 man hours to create and weighs 40kg . Does it look like me? Benedict Cumberbatch strikes a pose with James Corden during an Oscars party. Mixed reaction: A pair of police sniffer dogs loved the sculpture but shoppers looked baffled. Hilarious: A lady bursts into laughter after spotting the 6ft homage to Mr Cumberbatch. Not amused: A shopper looks thoroughly unimpressed as she contemplates the artwork. Luckily for Cumberbatch, who usually enjoys a considerably more complimentary response to projects he's involved in, the piece will only be in residence temporarily. The 38-year-old actor, who is currently expecting his first child with wife Sophie Hunter, 37, isn't the only famous face to have found himself the subject of an edible artwork. . In the run up to the release of 50 Shades of Grey, bakers created not one but two 6ft gateaux paying homage to Jamie Dornan. One depicted the actor in the grey suit beloved of his 50 Shades character Christian Grey, while the other showed him topless and came complete with an edible six-pack. Award-winning: Both Jennifer Lawrence and her cake alter-ego have won awards. Homage: The cake, which triumphed at a show last November, was inspired by the Hunger Games . Actress Jennifer Lawrence has also been immortalised in cake, with baker Lara Clarke creating a sweet treat designed to resemble the 24-year-old's Hunger Games alter-ego, Katniss Everdeen. The confection, which was baked ahead of the release of Mockingjay Part One in November, met with the approval of Lawrence herself, who, when asked about it, said Ms Clarke was 'incredibly talented'. Other A-listers to get the culinary treatment include Kevin Bacon, whose likeness was made from bacon, and Taylor Swift who featured on a cake. The Duke and Duchess of Cambridge were turned into a tasty-looking pizza, while President Barack Obama had his likeness sculpted from butter. Immortalised: The Duke and Duchess of Cambridge have been turned into pizza in the past. ribute: Kevin Bacon has a bacon sculpture to his name while Jamie Dornan has been made into a cake.
CTRLsum: Towards Generic Controllable Text Summarization
A 6ft chocolate sculpture of Benedict Cumberbatch has been unveiled. Toothsome statue has been placed inside a London shopping centre. But shoppers reactions to the creations were decidedly unenthusiastic. One woman glared at it while others just looked thoroughly baffled . It did manage to win the approval of pair of police sniffer dogs. It weighs 40kg and took eight people 250 man hours to create . Other celebrities to get culinary tributes include Jennifer Lawrence. Her 6ft cake won an award -and the 24-year-old's approval. Actor Kevin Bacon has also been immortalised -in bacon .
CTRLsum: Towards Generic Controllable Text Summarization
Westfield: The sculpture was unveiled at London's Westfield Stratford City shopping centre.
CTRLsum: Towards Generic Controllable Text Summarization
James Corden: Benedict Cumberbatch is not the only famous face to be immortalised in chocolate. Jennifer Lawrence and James Corden have also been turned into cakes .
CTRLsum: Towards Generic Controllable Text Summarization
Sophie Hunter: Actor, 38, is expecting his first child with wife Sophie Hunter.
CTRLsum: Towards Generic Controllable Text Summarization
Jamie Dornan: Shoppers at London's Westfield Stratford City shopping centre looked baffled by the sculpture. Other famous faces to be immortalised in edible art include Jamie Dornan and Jennifer Lawrence.
CTRLsum: Towards Generic Controllable Text Summarization
Hunger Games: The sculpture was commissioned by UKTV to mark the screening of Sherlock. It follows in the footsteps of other A-listers such as Jamie Dornan and Jennifer Lawrence, who have been immortalised in cakes. Lawrence's Hunger Games cake won an award at a show last year . 
CTRLsum: Towards Generic Controllable Text Summarization
The entity control experiments in this paper only consider one entity as the control signal, here we show examples inputting paired entities. Specifically, we are interested in the output when one of the paired entities is important and the other is unimportant. Therefore, we sample 3 articles from CNNDM and for each article we randomly select one important entity and one unimportant entity as paired keywords to show the model output. We repeat this sampling five times for each article to obtain five different summaries.
CTRLsum: Towards Generic Controllable Text Summarization
Article A former U.S. Army enlistee who posted on Facebook about "the adrenaline rush" of dying in jihad was arrested Friday and charged with trying to detonate a car bomb at Fort Riley military base in Kansas, authorities said. A second man, who allegedly knew about the bomb plot but didn't call authorities, was charged with failing to report a felony. John T. Booker Jr. of Topeka, an American citizen also known as Mohammed Abdullah Hassan, was taken into custody near Manhattan, Kansas, in a van that contained what he thought was a bomb, the criminal complaint said. The "bomb" had actually been put together by two confidential informants with nonexplosive materials, the complaint said. Fort Riley's security was never breached and no people were in danger, the U.S. Justice Department said in a press release. Booker enlisted in the Army last year and was due to ship out to basic training April 7, 2014, said Army spokesman Wayne Hall.    
CTRLsum: Towards Generic Controllable Text Summarization
For length control, we randomly sample 3 test articles from CNNDM and for each article we generate five different-length summaries.
CTRLsum: Towards Generic Controllable Text Summarization
Film director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" -a cult 1990s television show that was set to return in 2016. The offbeat TV series, created by Lynch and Mark Frost, featured a quirky FBI agent who went to the Pacific Northwest town of Twin Peaks to investigate the mysterious murder of a high school girl named Laura Palmer. The groundbreaking series is considered one of the most influential shows in television history. Lynch broke the news about his departure in a series of tweets, saying that the show's third season will continue without him. He said he felt the network was not offering enough money to produce the show "the way it needed to be done." Lynch also wrote that he had personally called the actors over the weekend to let them know he would no longer be directing. Showtime Network, which will air the nine-episode comeback, released a statement saying they were "saddened" by Lynch's decision. "We were saddened to read David Lynch's statement today since we believed we were working towards solutions with David and his reps on the few remaining deal points," read the statement. "Showtime also loves the world of Twin Peaks and we continue to hold out hope that we can bring it back in all its glory with both of its extraordinary creators, David Lynch and Mark Frost, at its helm." Showtime announced they would produce a third season in October last year. Actor Kyle MacLachlan, who played the coffee-obsessed FBI agent Dale Cooper in the original series, had confirmed he would reprise the lead role for the new season.
CTRLsum: Towards Generic Controllable Text Summarization
David Lynch says he won't be directing new episodes of Twin Peaks. Showtime "saddened" over decision, which involved a dispute over money.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 0) director David Lynch longer direct revival Twin Peaks return 2016 | season | Showtime: Film director David Lynch will no longer direct the revival of "Twin Peaks" Showtime announced they would produce a third season in October last year. The show was set to return in 2016.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 1) director David Lynch longer direct revival Twin Peaks set return 2016 | season | Showtime Network saddened: Film director David Lynch will no longer direct the revival of "Twin Peaks" Showtime Network say they are "saddened" by the decision. The show was set to return for a third season in 2016.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 2) director David Lynch confirmed longer direct revival Twin Peaks set return 2016 | series | season continue | Showtime Network saddened: Film director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" The series was set to return in 2016. Lynch broke the news about his departure in a series of tweets. He said the show's third season will continue without him. Showtime Network said they were "saddened" by Lynch's decision.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 3) director David Lynch confirmed longer direct revival Twin Peaks 1990s set return 2016 | series | season continue | Showtime Network saddened decision: SFilm director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks"The 1990s TV series was set to return in 2016. Lynch broke the news about his departure in a series of tweets. The show's third season will continue without him. Showtime Network said they were "saddened" by Lynch's decision.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 4) director David Lynch confirmed longer direct revival Twin Peaks cult 1990s set return 2016 | series Mark Frost FBI agent murder | tweets season continue | Showtime Network statement saddened decision | Kyle: Film director David Lynch has confirmed he will no longer direct the revival of "Twin Peaks" The cult 1990s TV series, created by Lynch and Mark Frost, was set to return in 2016. Lynch broke the news about his departure in a series of tweets, saying that the show's third season will continue without him. Showtime Network, which will air the nine-episode comeback, released a statement saying they were "saddened" by Lynch's decision. Actor Kyle MacLachlan, who played the coffee-obsessed FBI agent Dale Cooper in the original series, had confirmed he would reprise the lead role for the new season. 
CTRLsum: Towards Generic Controllable Text Summarization
Article Washington (CNN)An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf this month, sparking concern that top Iranian commanders might not be in full control of local forces, CNN has learned. The incident, which has not been publicly disclosed, troubled U.S. military officials because the unsafe maneuver could have triggered a serious incident. It also surprised U.S. commanders because in recent months Iranian forces have conducted exercises and operations in the region in a professional manner, one U.S. military official told CNN. "We think this might have been locally ordered," the official said. The incident took place as the U.S. and other world powers meet with Iran in Switzerland to negotiate a deal limiting Tehran's nuclear program. At the same time, Iran has been active in supporting proxies in several hotspots in the Persian Gulf and neighboring regions. The Navy MH-60R armed helicopter was flying from the deck of the USS Carl Vinson on a routine patrol in international airspace, the official said. An unarmed Iranian observation Y-12 aircraft approached. The Iranian aircraft made two passes at the helicopter, coming within 50 yards, before the helicopter moved off, according to the official. The official said the helicopter deliberately broke off and flew away in a 'predictable' manner so the Iranians could not misinterpret any U.S. intentions. The Navy helicopter was in radio contact with the ship during the encounter, but there was no contact between the two aircraft and no shots were fired. The Navy crew took photos of the incident but the military is not releasing them. The U.S. administration is considering a potential demarche protest against Iran, the official said. CNN has reached out to Iranian officials but has not received a response. This type of Iranian observation aircraft generally operates over the Gulf several times a month. But after the recent incident, U.S. naval intelligence did not see it again for two weeks, leading to the conclusion that the incident may have been ordered by a local commander who was then reprimanded by higher-ups. The Pentagon has noted for the last several years that most encounters with the Iranian military at sea or in air are conducted professionally, but that some missions run by Iranian Revolutionary Guard Corps forces have been too aggressive against U.S. forces in the area. The U.S. military's concern has been that one of these incidents could escalate into a military encounter. This incident "might have been buffoonery" the official said, but there is always a risk from such actions. The incident comes as the Navy patrols the Gulf of Aden to watch for Iranian ships the U.S. believes are trying to bring weapons to resupply the Houthi rebels in Yemen. The Navy would share such intelligence with Saudi Arabia, a second U.S. official told CNN.
CTRLsum: Towards Generic Controllable Text Summarization
Iranian plane came within 50 yards of U.S. Navy Sea Hawk copter. Navy copter was on patrol in international airspace. U.S. official think Iranian plane may have been under orders of local commander.
CTRLsum: Towards Generic Controllable Text Summarization
CTRLsum (Length bucket 0) Iranian military aircraft flew 50 yards armed U.S. Navy helicopter Gulf commanders: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf this month. The incident, which has not been publicly disclosed, has sparked concern that top Iranian commanders might not be in full control of local forces.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 1) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Gulf Iranian commanders forces: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The incident sparked concern that top Iranian commanders might not be in full control of local forces.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 2) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Persian Gulf Iranian commanders control forces | incident: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The incident sparked concern that top Iranian commanders might not be in full control of local forces.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 3) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Persian Gulf Iranian commanders control forces | incident | Iran | protest: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The incident sparked concern that top Iranian commanders might not be in full control of local forces. Iran has been active in supporting proxies in several hotspots in the region. The U.s. administration is considering a potential demarche protest against Iran.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 4) Iranian military observation aircraft flew 50 yards armed U.S. Navy helicopter Persian Gulf Iranian commanders control local forces | incident officials | official | ordered | Iran | USS | considering demarche protest | Navy Aden: An Iranian military observation aircraft flew within 50 yards of an armed U.S. Navy helicopter over the Persian Gulf. The Navy MH-60R armed helicopter was flying from the deck of the USS Carl Vinson on a routine patrol in international airspace. The incident sparked concern that top Iranian commanders might not be in full control of local forces. "We think this might have been locally ordered," the official said. CNN has reached out to Iranian officials but has not received a response. Incident comes as the Navy patrols the Gulf of Aden to watch for Iranian ships. The administration is considering a potential demarche protest against Iran. 
CTRLsum: Towards Generic Controllable Text Summarization
Article It's a sight that draws giggles and curious stares from tourists and other first-timers -an unusual festival where revellers carry gigantic phalluses through the streets of a Japanese city. But for the residents of Kawasaki, who lug erotic shapes of all different sizes, this odd tradition is not a joke. Shinto Kanamara Matsuri started as a small tradition but has grown into a popular a tourist attraction, with participants praying to a god of fertility, child birth and protection from sexually transmitted infections. Participants carry a gigantic phallus through the streets of Kawasaki, Japan during the Shinto Kanamara Matsuri festival. The sight of three large phalluses being paraded through neighbourhoods in the city south of Tokyo draws giggles from tourists. Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition but has grown into a popular a tourist attraction. Known as the Festival of the Steel Phallus, it is held every spring at the phallus-shaped Kanayama Shrine. Festivalgoers parade through the streets with three giant phalluses, while spectators lick lollies or snack on sausages or vegetables shaped as male and female genitalia. Rainy weather didn't ruin the mood at this year's festival, which had a massive collection of foreigners, according to Japanese website RocketNews24. They watched as groups of locals carried three heavy phalluses modelled after a mikoshi portable shrine, which is commonly used in Shinto festivals. Residents of Kawasaki carry phalluses of all different sizes while participating in a tradition that began nearly 40 years ago. Participants pray to a god who is said to help with fertility, child birth and protection from sexually transmitted infections. Rainy weather didn't ruin the mood at this year's festival, which attracted thousands of spectators from Japan and abroad. Spectators can buy lollies, sausages and vegetables shaped as male and female genitalia while watching the unusual parade. The three mikoshi carried through the streets for hours are the Kanamara Boat Mikoshi (tall and black), the Elizabeth Mikoshi (big and pink) and the Kanamara Big Mikoshi (squat and wooden), RocketNews24 reported. In addition to the parade there are performances and music, and the festival itself raises money and awareness for HIV research. The festival started nearly 40 years ago, and the Kanayama Shrine was once a popular place for prostitutes to pray for protection from sexually transmitted infections. Today, people visit the shrine to pray for everything from fertility and business prosperity to wedded bliss and good health. In addition to the parade there are performances and music, and the festival itself raises money and awareness for HIV research. The Kanayama Shrine was once a popular place for prostitutes to pray for protection from sexually transmitted infections. Today, people visit the shrine to pray for everything from fertility and business prosperity to wedded bliss and good health. Groups of locals carried three heavy phalluses modelled after a mikoshi portable shrine, which is commonly used in Shinto festivals.
CTRLsum: Towards Generic Controllable Text Summarization
Shinto Kanamara Matsuri started as a small tradition but has grown into a popular tourist attraction. Known as the Festival of the Steel Phallus, participants pray to a god of fertility and protection from infections. Rainy weather didn't ruin the mood at this year's festival, which attracted a large crowd of holidaymakers .
CTRLsum: Towards Generic Controllable Text Summarization
CTRLsum (Length bucket 0) Shinto Kanamara Matsuri started tradition god fertility child birth protection infections | Steel: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 1) Shinto Kanamara Matsuri started tradition praying god fertility child birth protection sexually infections | Festival Steel: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 2) Shinto Kanamara Matsuri started tradition popular tourist praying god fertility child birth protection sexually infections | Festival Steel Phallus: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition but has grown into a popular a tourist attraction. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 3) Shinto Kanamara Matsuri started tradition popular tourist attraction praying god fertility child birth protection sexually transmitted infections | Festival Steel Phallus: Shinto Kanamara Matsuri, the Festival of the Steel Phallus, started as a small tradition but has grown into a popular a tourist attraction. Participants pray to a god of fertility, child birth and protection from sexually transmitted infections.
CTRLsum: Towards Generic Controllable Text Summarization
(Length bucket 4) phalluses | Shinto Kanamara Matsuri started small tradition grown popular tourist attraction praying god fertility child birth protection sexually transmitted infections | Kawasaki Japan Matsuri festival | Tokyo | Festival Steel Phallus | 40: Shinto Kanamara Matsuri started as a small tradition 40 years ago. It has grown into a popular tourist attraction in Kawasaki, Japan. The Festival of the Steel Phallus is held every spring at the Kanayama Shrine. Participants carry three phalluses through neighbourhoods south of Tokyo. They are praying to a god of fertility, child birth and protection from sexually transmitted infections. Thousands of tourists attended this year's festival, which raised money for HIV research. 
CTRLsum: Towards Generic Controllable Text Summarization
Here we show three random examples from the arXiv test set. Note that this is the test set from (Cohan et al., 2018) instead of the contribution test data collected by us, because we want to show the difference between reference summaries (i.e. the paper abstract) in existing standard paper summarization dataset and our output contribution summaries. We truncate the source articles since they are too long to display.
CTRLsum: Towards Generic Controllable Text Summarization
Article synchronization of neural activity appears in different parts of the mammalian cerebral cortex @xcite , and underlies different neural processes in both normal and anomalous brain functions @xcite . it has been suggested that synchronization plays a vital role in information processing in the brain , e.g. , processing information from different sensory systems to form a coherent and unified perception of the external world @xcite . on the other hand , synchronization has been detected in pathological conditions such as parkinson s disease @xcite . and epileptic seizures have long been considered resulting from excessive synchronized brain activity @xcite , though some recent studies suggest that this picture may be an over -simplification @xcite . therefore understanding the mechanisms of synchronization may be a critical step in elucidating how neural systems work @xcite . it has stimulated a great deal of theoretical and numerical works , such as the studies on the effects of the topological properties of underlying networks @xcite and the dynamical properties of synaptic coupling @xcite . it was recently shown that the response time of synaptic couplings influences the stability of synchronized oscillation in the nonlocally coupled hodgkin -huxley ( hh ) equations @xcite . if the response time of synaptic coupling is slower , synchronized activity of the systems is instable for excitatory coupling . however , the underlying dynamical mechanism of the influence is not clear . in experimental studies @xcite , it has been suggested that the generation of prolonged epileptiform neuronal synchronization is favored by lower efficacy of synaptic transmission . the numerical studies @xcite in a detailed computational model revealed that seizure -like activity occurs when the excitatory synapses are weakened , and the results were confirmed experimentally in mouse neocortical slices . according to the common accepted assumption that synchronization of neuronal activity underlies seizures , the dynamical mechanism of synchronization may be useful for understanding the way the biological neural system works . in this work , we numerically investigated the dynamical mechanism underlying the influence of synaptic efficacy on firing synchronization in hh neuron networks . to do this , we first studied the dynamics of the response of hh neuron to excitatory synaptic current . when the efficacy of synapse is low , namely , strength is weak and duration is short , the limit cycle is stable to the perturbation of the synaptic current . when synaptic efficacy is high , synaptic current can induce the transition of the neurons from limit cycle to fixed point or transient state . the transition is determined by dynamics of neuron s ionic channel . the decrease of synaptic current depresses the feedback of sodium ionic current which is responsible for the initiation of the spike . for simplicity we will refer to the transitions as spike death . in neuronal networks , synaptic input of a neuron is the accumulation of the currents received from all presynaptic neurons . when the coherence of firing time of neurons is enhanced by the excitatory interaction , the synaptic input of neurons transforms from the fluctuating waveform into the pulse shape like the signal produced by one synapse . if synaptic efficacy is high , the input signal can induce spike death of the neuron . then spike death disorders the adjustment of the rhythm of neurons and prevents neurons from firing spikes synchronously . in contrast , for synapses of lower efficacy , the duration of synaptic current is too short to induce spike death of neurons . additionally , the firing synchronization is different from synchronous activity of oscillators for the existence of the transitions of neuron s state . the paper is organized as follows . the hh neuron model and the synaptic coupling were introduced in sec . the response of a hh neuron to synaptic current was investigated in sec . the influence of the dynamics of neurons on firing synchrony was shown in sec .
CTRLsum: Towards Generic Controllable Text Summarization
we investigated the influence of efficacy of synaptic interaction on firing synchronization in excitatory neuronal networks . we found spike death phenomena , namely , the state of neurons transits from limit cycle to fixed point or transient state . the phenomena occur under the perturbation of excitatory synaptic interaction that has a high efficacy . we showed that the decrease of synaptic current results in spike death through depressing the feedback of sodium ionic current . in the networks with spike death property the degree of synchronization is lower and unsensitive to the heterogeneity of neurons . the mechanism of the influence is that the transition of neuron state disrupts the adjustment of the rhythm of neuron oscillation and prevents further increase of firing synchronization .
CTRLsum: Towards Generic Controllable Text Summarization
[the main contributions of this paper are : ( 1 )]: we investigated the dynamical mechanism underlying the influence of synaptic efficacy on firing synchrony in hodgkin -huxley neuron networks ;
CTRLsum: Towards Generic Controllable Text Summarization
( 2 ) we found that the dynamics of synaptic current plays an important role in determining the stability of firing synchronization . Article for the understanding of surface reactions and the characterization of materials it is desirable to measure local forces close to a sample surface . the most common method to measure these surface forces is atomic force microscopy ( afm)@xcite . historically , the first force measurements were static measurements for which the force is presented as a scalar function of the static tipsample separation , the so -called force curve@xcite . this representation is sufficient for conservative forces but the total tipsurface force may also contain contributions from dissipative forces . since dissipative forces depend on probe velocity and past trajectory , dynamic force spectroscopy methods are required for their measurement . moreover , the visualization of dissipative forces as a function of position is valid only for a specific probe trajectory and simple force curves can not capture the full character of the interaction . despite the development of several dynamic methods@xcite surface forces are still usually treated as functions of the probe position only and represented by simple force curves . here , we present a comprehensive framework for the representation and analysis of complex surface forces as they are measured by dynamic afm . we concentrate on the most common modes of dynamic afm : amplitude -modulated afm ( am -afm ) and frequency -modulated afm ( fm -afm ) , which can be considered as narrow frequency band methods@xcite . we explore the fundamental limit of force reconstruction with narrow band dynamic afm at fixed probe height and show how minimal assumptions allow for a quantitative reconstruction of the tip -surface interaction . at the heart of the afm apparatus is a micro -cantilever with a sharp tip . the cantilever is firmly clamped at one end and the tip is located at the other end which can move freely . it is assumed that surface forces only act on the tip whereas the rest of the cantilever does not experience significant surface forces . in dynamic afm an additional external drive force is applied to maintain an oscillatory motion . thus , the dynamics are governed by the force between tip and surface , the external drive force and the properties of the cantilever beam . since the cantilever is a three dimensional continuum object its motion is usually described by the amplitudes of different oscillation eigenmodes . in general , these modes can cause the cantilever to bend in all directions in space . however , the cantilever is positioned such that the softest flexural modes bend the beam in a plane orthogonal to the surface plane . we restrict ourselves to the case where only these flexural modes are excited by the drive force . due to this experimental configuration the cantilever is much more susceptible to the component of the tipsurface force which is orthogonal to the surface plane . this component of the force is typically the most dominant component and the influence of lateral force components is considered negligible . in this case the cantilever acts as a mechanical projector which reacts only to one component of a three dimensional force vector field . the deflection @xmath0 of a cantilever of length @xmath1 orthogonal to surface is described by a one dimensional euler -bernoulli equation@xcite @xmath2 where @xmath3 is the young s modulus , @xmath4 is the second moment of area , @xmath5 is the mass per unit length of the cantilever , @xmath6 is the position coordinate along the cantilever beam and @xmath7 is the time variable . the force term @xmath8 includes the surface forces acting as a point -like load at position @xmath9 , the external drive force and the hydrodynamic damping due to the surrounding medium@xcite .
CTRLsum: Towards Generic Controllable Text Summarization
Reference Summary in atomic force microscopy ( afm ) tip -surface interactions are usually considered as functions of the tip position only , so -called force curves . however , tip -surface interactions often depend on the tip velocity and the past tip trajectory . here , we introduce a compact and general description of these interactions appropriate to dynamic afm where the measurement of force is restricted to a narrow frequency band . we represent the tip -surface interaction in terms of a force disk in the phase space of position and velocity . determination of the amplitude dependence of tip -surface forces at a fixed static probe height allows for a comprehensive treatment of conservative and dissipative interactions . we illuminate the fundamental limitations of force reconstruction with narrow band dynamic afm and we show how the amplitude dependence of the fourier component of the force at the tip oscillation frequency , gives qualitative insight into the detailed nature of the tip -surface interaction . with minimal assumptions this amplitude dependence force spectroscopy allows for a quantitative reconstruction of the effective conservative tip -surface force as well as a position -dependent damping factor . we demonstrate this reconstruction on simulated intermodulation afm data . _ keywords _ : atomic force microscopy , measurement of force , mechanical resonators , mems / nems , dissipation , intermodulation
CTRLsum: Towards Generic Controllable Text Summarization
[the main contributions of this paper are : ( 1 )]: a comprehensive framework for the representation and analysis of complex surface forces as they are measured by dynamic atomic force microscopy ( afm ) ; ( 2 ) a study of the fundamental limit of force reconstruction with narrow band dynamic afm at fixed probe height and show how minimal assumptions allow for a quantitative reconstruction of the tip -surface interaction . 5908 Article in this paper we discuss the mathematical aspects of the problems originating in the solution of nonlinear systems of hyperbolic partial differential equations . these equations describe a large variety of physical phenomena , such as , gasdynamics , magnetohydrodynamics ( mhd ) , shallow water equations , elasticity equations , etc . being nonlinear , these systems usually require numerical methods for their solution . presence of discontinuous solutions motivates the necessity of the development of reliable numerical methods based on the fundamental mathematical properties of hyperbolic systems . although such methods are rather well developed for the euler gasdynamic equations in the conservation law form , their extension to more complicated hyperbolic systems is not straightforward . it requires a mathematical justification of the solution uniqueness , a formulation of the selection principles for relevant solutions , and , finally , an investigation of their physical validity . most of highresolution methods for gasdynamic equations use the exact or some of the approximate self -similar riemann problem solutions to determine fluxes through the computational cell surfaces . similar methods are expected to be developed for various types of hyperbolic systems . in this case we must construct the elementary self -similar solution using only admissible discontinuities ( entropy consistent , evolutionary , etc . ) . basically the choice of the solution must be made on the basis of the structure of the solution of the extended problem @xcite . all mentioned above makes very important the study of discontinuous solutions behavior under vanishing viscosity and dispersion to create a proper background for the development of high -resolution numerical methods for hyperbolic systems more complicated than the euler equations of gasdynamics . we discuss several analytical and numerical solutions in the mentioned fields which illustrate the complexity of the selection problem and outline the methods of its solution . tvd upwind and symmetric differencing schemes have recently become very efficient tool for solving complex multi -shocked gasdynamic flows . this is due to their robustness for strong shock wave calculations . the extension of these schemes to the equations of the ideal magnetohydrodynamics is not simple . first , the exact solution @xcite of the mhd riemann problem is too multivariant to be used in regular calculations . second , several different approximate solvers @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , and @xcite applied to mhd equations are now at the stage of investigation and comparison . this investigation requires i ) determination of a proper slope limiting method in the parameter interpolation procedure necessary to obtain nonoscillatory schemes of the order of accuracy higher than one ; ii ) development of an efficient entropy correction method necessary to exclude rarefaction shocks ; and , finally , iii ) solution of the problem of excluding the origin of nonevolutionary solutions in ideal mhd calculations . the system of governing equations for a mhd flow of an ideal , infinitely conducting , perfect plasma in the cartesian coordinate system @xmath0 , @xmath1 , @xmath2 with the use of the conventional notations reads ( one fluid approximation ) : @xmath3 where @xmath4 is the vector of conservative variables and @xmath5 , @xmath6 , and @xmath7 are the flux vectors . we introduced here the source term @xmath8 in the form @xmath9 this form of the system can be used to satisfy the divergence -free condition by convecting away the magnetic charge from the computational region @xcite . otherwise , any other well -known method can be used to eliminate the magnetic charge . to determine a numerical flux @xmath10 normal to the computational cell boundary ( @xmath11 is a unit outward vector normal to the cell surface ) one can use the formulas based on the solution of the linearized problem @xmath12. ] ] here @xmath13 and @xmath14 are the matrices formed by the right and by the left eigenvectors , respectively , of the frozen jacobian matrix @xmath15 the matrix @xmath16 is a diagonal matrix consisting of the frozen jacobian matrix eigenvalue moduli . the superscripts @xmath17 and @xmath18 denote the values at the right-and at the left -hand side of the cell boundary .
CTRLsum: Towards Generic Controllable Text Summarization
Reference Summary a number of physical phenomena are described by nonlinear hyperbolic equations . presence of discontinuous solutions motivates the necessity of development of reliable numerical methods based on the fundamental mathematical properties of hyperbolic systems . construction of such methods for systems more complicated than the euler gas dynamic equations requires the investigation of existence and uniqueness of the self -similar solutions to be used in the development of discontinuity -capturing high -resolution numerical methods . this frequently necessitates the study of the behavior of discontinuities under vanishing viscosity and dispersion . we discuss these problems in the application to the magnetohydrodynamic equations , nonlinear waves in elastic media , and electromagnetic wave propagation in magnetics .
CTRLsum: Towards Generic Controllable Text Summarization
[the main contributions of this paper are : ( 1 )]: the mathematical aspects of the problems originating in the solution of nonlinear systems of hyperbolic partial differential equations ;
CTRLsum: Towards Generic Controllable Text Summarization
( 2 ) the study of discontinuous solutions behavior under vanishing viscosity and dispersion to create a proper background for the development of high -resolution numerical methods for hyperbola systems more complicated than the euler equations of gasdynamics ; and ( 3 ) solution of the problem of excluding the origin of nonevolutionary solutions in ideal magnetohydrodynamics calculations . 
CTRLsum: Towards Generic Controllable Text Summarization
Here we show three random examples from the BIGPATENT test set. Note that this is the test set from origial BIGPATENT, because we want to show the difference between reference summaries in existing standard dataset and our output purpose summaries. We truncate the source articles since they are too long to display.
CTRLsum: Towards Generic Controllable Text Summarization
Article referring to the drawings and , in particular to fig1 therein illustrated is a prior art surgical support mesh 10 . mesh 10 may be manufactured from monofilament or multifilament yarns . prior art mesh 10 , as shown , includes multifilament horizontally -extending yarns 12 and multifilament vertically -extending yarns 14 woven together to form a support trellis . the use of multifilament yarns , such as yarns 12 and 14 , provides a mesh having greater pliability and suppleness than the use of monofilament yarns . these characteristics result from both the smaller diameter of the individual filaments and the interstitial spaces or voids that are located between such filaments . in particular , the flexibility of a filament ( or fiber ) generally increases as its diameter decreases . because the solid cross -sectional area of the filaments of a multifilament yarn is less than the crosssectional area of a monofilament yarn of equivalent diameter , the multifilament yarn will have a greater degree of flexibility and pliability than that of the monofilament yarn . as shown in fig1 a , each of multifilament yarns 12 and 14 is composed of a plurality of filaments 16 that are intermingled or bundled together to form the yarn . interstitial spaces 18 , which are pockets of air , are formed between adjacent filaments of the yarn . although these voids contribute to the softness and pliability of the formed mesh , they also provide a natural breeding ground for bacteria or other infectious material . surgical mesh is , of course , thoroughly sterilized prior to implantation . nevertheless , surgeons typically prefer the use of monofilament -designed mesh to minimize any risk of infection . as a result , the advantages associated with multifilament -designed mesh ( i . e ., softness and pliability which result in better assimilation of the mesh into the body ) are typically sacrificed . it has been discovered herein that a surgical support mesh having both the softness and pliability of a multifilament -designed mesh and the infection resistance of a monofilament -designed mesh may be produced . particularly , it has been discovered that a support trellis formed of multifilament yarn wherein the interstitial voids located between adjacent filaments are enclosed within an infection -impervious matrix exhibits the desired resistance to harboring of infectious matter without significant loss of flexibility . particularly , the matrix , which completely encloses the interstitial voids between the filaments of the yarn , provides an effective barrier to the passage of infectious matter between the interior and exterior of the yarn . accordingly , any voids remaining in the yarn after encapsulation of such yarn are enclosed ( and thereby sealed ) within the resultant matrix . a first embodiment of the present invention is shown in fig2 . particularly , this first embodiment includes a support trellis 20 formed of multifilament yarns 22 and 24 which overlap at cross -over junctions 25 . subsequent to forming of the trellis , such trellis is encapsulated within a matrix 26 , which is preferably a flexible material that continuously surrounds the exterior of the yarns thereby enclosing interstitial voids 27 located between filaments 28 ( see fig2 a ). in one embodiment , the matrix is formed from a polymeric resin . as shown in fig2 a , the resin can be applied to the yarn in such a manner as to not allow the resin to substantially penetrate into the yarn . particularly , the penetration of the resin can be controlled through the application procedure , e . g ., quantity of resin applied and / or encapsulating time . in such an embodiment , the interstitial spaces are enclosed ( rather than filled ) within the continuous matrix . however , it is contemplated that the resin can be allowed to penetrate into the yarn , thereby substantially filling the void space located therein . in another embodiment of the present invention , individual yarns 29 , as shown in fig3 are encapsulated within matrix 30 prior to forming of the support trellis . fig3 a shows a compressed yarn 29 which provides a trellis having a reduced thickness . as a result of the encapsulation , interstitial voids 32 remaining in the yarn are enclosed ( and thereby sealed ) within the matrix .
CTRLsum: Towards Generic Controllable Text Summarization
Reference Summary a soft and pliable surgical support mesh exhibiting increased resistance to inhabitation of infectious matter . the mesh includes a support trellis formed of multifilament yarns wherein the interstitial voids located between the filaments of said yarns are enclosed within an infection -impervious matrix . the meshes may be designed to be extremely thin yet retain the requisite strength for repairing soft tissue , which allows for a low profile when folded for delivery .
CTRLsum: Towards Generic Controllable Text Summarization
[the purpose of the present invention is]: to provide a surgical mesh that is resistant to the growth of bacteria and other infectious matter . this is accomplished by encapsulating the interstitial spaces located between the filaments of the yarn within a matrix . Article fig1 shows a multicolor web fed rotary printing press 1 in accordance with the invention . the press 1 includes four tower arrangements 2 a , 2 b , 2 c and 2 d for printing a single color or a multicolor image on the webs 4 a , 4 b , 4 c and 4 d . the webs 4 a , 4 b , 4 c and 4 d travel in a substantially linear direction through each of the towers 2 a -2 d . for example , the web can travel along a substantially vertical path , as shown in fig1 . alternatively , as those skilled in the art will appreciate , the web path can be in a substantially horizontal direction , or in a substantially linear path at any desired angle relative to the vertical direction shown . the towers 2 a -2 d each include four printing units 6 c , 6 m , 6 y and 6 b for respectively printing an image in cyan , magenta , yellow and black on both sides of each web 4 a -4 d . other colors besides cyan , magenta , yellow and black can be used . the webs 4 can be , for example , between 1200 and 1600 millimeters wide . each of the printing units 6 c , 6 m , 6 y and 6 b in a tower can be moved along a respective web 4 by a lifting and positioning system 8 shown in fig2 . the lifting and positioning system 8 includes a spindle drive 10 , which has a fixed spindle 12 spanning a range 14 over which the printing units 6 c , 6 m , 6 y and 6 b can be moved . each of the printing units 6 c , 6 m , 6 y and 6 b includes a ball screw 16 , which is rotatably supported in a housing 18 . the ball screw 16 can be rotated by a motor 20 as shown in fig2 . fig2 shows one set of a spindle drive 10 , fixed spindle 12 , ball screws 16 , and motors 20 , but preferably each tower 2 is provided with several sets , one set for each corner of the print unit housing 18 . the motors 20 are controlled by a motor control unit 22 , which receives commands from a remote control 24 . by pressing a button on the remote control 24 , an operator can control the rotation of the motors 20 and thereby the movement direction and position of each printing unit 6 b , 6 y , 6 m and 6 c in a tower 2 . rail systems ( not shown ) fixed to a side frame of each tower 2 can also be used to precisely guide movements of the printing units 6 in the tower . as shown in fig1 and 2 , the position of each of the printing units 6 along the webs 4 and fixed spindles 12 can be controlled by the operator to allow access to a desired part of a printing unit 6 . for example , in fig1 the operator has moved the print unit 6 b of tower 2 b into a position where a printing plate of the print unit 6 b can be most easily accessed . after the printing plate has been accessed , the operator can move the print unit 6 b into a different position so that inker units in an upper part of the printing unit 6 b can be easily accessed . two or more printing units 6 in tower 2 can also be moved as a group . for example , if the operator wants to access the plate cylinder of the printing unit 6 m of tower 2 b shown in fig1 he can simply move the two printing units 6 y , 6 m upwardly together until the top surface 28 of the printing unit 6 y contacts the bottom surface 30 of the printing unit 6 b . thereafter , the operator can move the group of printing units 6 b , 6 y and 6 m upwards into the position shown in tower 2 c of fig1 where the plate cylinder of the printing unit 6 m can be easily accessed . although fig1 and 2 show four printing units 6 for each tower 2 , different numbers of printing units can be used for each tower . fig1 shows that the lower three printing units 6 of the tower 2 b are beneath an operating floor 26 . preferably at least two of the printing units can be lowered beneath the operating floor 26 , and a printing press in accordance with the invention can be configured so that all of the printing units in a tower can be lowered beneath an operating floor . the operating floor can be a floor of a print shop , or can be an elevated platform .
CTRLsum: Towards Generic Controllable Text Summarization
Reference Summary the present invention is directed to a multicolor web fed rotary printing press having printing units that can be moved along a linear section of a web by a positioning mechanism to allow easy access to each of the printing units . the easy access to the printing units significantly reduces maintenance costs and downtime of the press . in addition , the invention provides a printing press that has a reduced overall size and allows the printing units to be arranged in a nested formation during printing operations . this nested formation reduces fanout and paper waste . the linear web section can be oriented vertically or horizontally .
CTRLsum: Towards Generic Controllable Text Summarization
[the purpose of the present invention is]: to provide a web fed rotary printing press that allows an operator to access a desired part of a printing unit more easily than in the past . 
CTRLsum: Towards Generic Controllable Text Summarization
Article referring now to the drawings and more particularly to fig1 -3 , a fluid system 10 is illustrated and includes a variable displacement axial piston pump 12 that receives fluid from a tank 14 via a conduit 16 and delivers pressurized fluid via a supply conduit 18 to a fluid control valve 20 and selectively through work conduits 22 , 24 to a fluid actuator 26 . in the subject arrangement , the variable displacement axial piston pump 12 is a unidirectional pump that rotates in a counterclockwise direction as driven by a power input shaft 27 . the fluid system 10 also includes first and second pressure sensors 28 , 30 respectively connected to the tank conduit 16 and the supply conduit 18 . the pressure sensors 28 , 30 are operative to sense the pressure in the respective lines and deliver an electrical signal to a controller 32 through electrical lines 34 , 36 . a position sensor 40 is mounted on the variable displacement axial piston pump 12 and operative to sense the displacement of the pump and deliver a signal representative thereof to the controller 32 via an electrical line 42 . various other components could be used in the subject fluid system 10 without departing from the essence of the subject invention . for example , several control valves 20 and associated fluid actuators 26 could be used . likewise , other sensors of various types and styles could be used . the variable displacement axial piston pump 12 includes a housing 44 having a head portion 46 and a body portion 48 . the head portion 46 defines an inlet port passage 50 that is connected to the conduit 16 and an outlet port passage 52 that is connected to the supply conduit 18 . in the subject arrangement , a port plate 54 is disposed between the head portion 46 and the body portion 48 . the construction of the porting within the port plate 54 is more clearly illustrated in fig3 and will be discussed more fully below . it is recognized that the porting illustrated in fig3 could be made within the head portion 46 without departing from the essence of the subject invention . a rotating group 56 is disposed within the body portion 48 and includes a barrel 58 having a plurality of cylinder bores 59 defined therein spaced from one another around an axis of rotation 60 of the barrel 58 . each of the cylinder bores 59 is oriented within the barrel 58 parallel with the axis of rotation 60 . a plurality of piston assemblies 62 are operatively associated with the barrel 58 and each one of the plurality of piston assemblies 62 includes a piston 64 slideably disposed in the respective ones of the plurality of cylinder bores 59 . each one of the plurality of piston assemblies 62 also has a shoe 66 pivotably attached to one end of each piston 64 in a conventional manner . the barrel 58 has an end surface 68 that is in mating , sealing contact with the port plate 54 to provide communication between the cylinder bores 58 and the respective inlet and outlet port passages 50 , 52 of the head portion 46 . a closed chamber 70 is defined in each cylinder bore 59 of the barrel 58 between the end of the piston 64 and the end surface 68 thereof . referring to fig3 the porting between the barrel 58 and inlet and outlet port passages 50 , 52 of the head portion 46 is more clearly illustrated . for explanation purposes only , the " 270 " degree position illustrated in fig3 relates to a position on the right side of the drawing of fig1 and the " 0 " degree position illustrated in fig3 relates to a position on the right side of the drawing of fig2 . an arcuate slot 72 is defined in the port plate 54 and provides communication between the plurality of closed chambers 70 and the inlet port passage 50 . a plurality of slots 74 are defined in the port plate 54 circumferentially spaced from the arcuate slot 72 and provides communication between the plurality of closed chambers 70 and the outlet port passage 52 .
CTRLsum: Towards Generic Controllable Text Summarization
Reference Summary a variable displacement axial piston pump is typically used to receive fluid from a tank and supply pressurized fluid through a control valve to move an actuator . the present variable displacement axial piston pump has a swashplate arrangement that is capable of being angled in two different directions to control the pressure transitions between the low pressure inlet port passage and the higher pressure outlet port passage as cylinder bores in a barrel of a rotating group rotate through trapped volume regions situated between inlet and outlet port passages of the axial piston pump . movement of the swashplate arrangement in two different directions provides smooth pressure transitions and increases the operating efficiency of the variable displacement axial piston pump .
CTRLsum: Towards Generic Controllable Text Summarization
[the purpose of the present invention is]: to provide a variable displacement axial piston pump that is capable of delivering a variable amount of pressurized fluid in response to a change in the displacement of the pump . 
CTRLsum: Towards Generic Controllable Text Summarization
We randomly sample 3 articles from NewsQA and show five questions and answers from CTRLsum for each article. We also show the gold answers to these questions.
Exploring Neural Models for Query-Focused Summarization
Text summarization aims at transforming long documents into short snippets that contain only the most important information from the source document. The field has seen substantial progress driven by the availability of large-scale models pre-trained on vast amounts of data (Devlin et al., 2019;, the development of summarizationspecific pre-training strategies (Zhang et al., 2020;, and computationally efficient neural architectures (Zaheer et al., 2020).
Exploring Neural Models for Query-Focused Summarization
The majority of recent research efforts in text summarization assume an unconstrained setting * Equal contribution in which models are given only a source document as input and are expected to generate a general summary covering the salient aspects from the source. The performance of such models has been evaluated on benchmark datasets spanning various domains: news articles (Nallapati et al., 2016;Narayan et al., 2018;Fabbri et al., 2019a), legal documents (Sharma et al., 2019), scientific writing (Cohan et al., 2018), or creative writing (Kryściński et al., 2021;Chen et al., 2021). However, it has been shown that summarization in an unconstrained setting is an ill-defined task where multiple generated summaries are equally relevant (Kryscinski et al., 2019). This in turn hinders the ability to evaluate and understand the models' content selection capacity. In addition, such generic summarization models lack control mechanisms that would allow end users to customize summaries to their particular needs and expectations.
Exploring Neural Models for Query-Focused Summarization
Query-focused summarization (QFS) is a subtask within text summarization that focuses on generating summaries where the summary content is tailored to a user-specified query that is passed alongside the source document as input to the model. Each source document can be associated with multiple unique queries inquiring about different information from that document. In this setting, end users are enabled to explicitly specify their preferences for the summary, and the relevance of the output summary may be evaluated more precisely with respect to the input query. Research on this task has been accelerated by the recently introduced highquality datasets, such as QMSum (Zhong et al., 2021b) and AQuaMuSe (Kulkarni et al., 2020).
Exploring Neural Models for Query-Focused Summarization
In this work we conduct a systematic, exploratory study of different approaches to queryfocused text summarization, considering both twostep and end-to-end neural methods. We present two models, RELREG and SEGENC, which achieve state-of-the-art ROUGE scores on the QMSum dataset, up to a margin of 3.38 R-1, 3.72 R-2, and 3.28 R-L when combined with transfer learning methods. The RELREG model uses a two-step approach to solving the problem, where the first step extracts content relevant to the given query and the next step synthesizes the extracted fragments into a coherent summary. The SEGENC method follows an end-to-end framework in which individual document segments are separately encoded to avoid the computational bottleneck of long input documents, and the decoder jointly attends to all encoded segments when producing the summary. Through quantitative studies, we compare our models with other baselines and discuss the trade-offs of the end-to-end methods and pipelined approaches. We also perform human evaluation to understand the qualitative differences between the models. Together with this manuscript, we share the code base and model checkpoints to enable future research in this area.
Exploring Neural Models for Query-Focused Summarization
Query-focused summarization aims to generate a summary of a given text conditioned upon a query. Initial work in this area centered around unsupervised extractive approaches (Wan et al., 2007;Litvak and Vanetik, 2017) due to the limited availability of task-specific training data (Dang, 2005). More recent work has taken advantage of the relationship between query-focused summarization and the more data-rich task of question answering for extractive summarization (Egonmwan et al., 2019), reranking documents within a retrieval pipeline (Su et al., 2020), and abstractive summarization (Su et al., 2021;Baumel et al., 2018;Xie et al., 2020). Xu and Lapata (2020) introduce a pipeline consisting of a relevance estimator filter followed by query-focused evidence and centrality estimators, while other work converts generic summarization dataset to query-focused training data (Xu and Lapata, 2021a) or performs latent query modeling (Xu and Lapata, 2021b).
Exploring Neural Models for Query-Focused Summarization
Recently, several query-focused summarization datasets have been introduced, which can be further divided into short-document datasets, whose source document length does not exceed the input limits of standard pretrained models, and longdocument datasets. Within short-document, queryfocused summarization, AnswerSumm (Fabbri et al., 2021c) is composed of summaries of answers to queries from StackExchange forums, while Wik-iHowQA (Deng et al., 2020) proposes the task of answer selection followed by the summarization of individual response articles to queries from the how-to site WikiHow. Within long-document summarization, WikiSum (Liu et al., 2018a) consists of Wikipedia article titles as queries, the first paragraph of the article as the summary, and documents referenced by the article as the input. AQuaMuSe (Kulkarni et al., 2020) is a queryfocused multi-document summarization dataset with user-written queries and human-verified longanswer summaries from the Natural Questions dataset (Kwiatkowski et al., 2019), and QMSum (Zhong et al., 2021b) is a manually-curated dataset for query-focused dialog summarization. QMSum and AQuaMuSe are of particular interest to our study due to the combined challenges of queryfocused and long-document summarization and the presence of high-quality, curated query-summary pairs.
Exploring Neural Models for Query-Focused Summarization
Recent work on QMSum has introduced taskspecific denoising objectives for meeting summarization (Zhong et al., 2021a), generated final fine-grained summaries based on multiple coarsegrained steps (Zhang et al., 2021a), and treated the extractive text of an extractive-abstractive model as a latent variable (Mao et al., 2021). Zhang et al. (2021b) analyze the challenges of long dialogue summarization such as the input length, the role of queries, and domain adaptation. Our work builds on QA-motivated methods and presents two approaches yet to be applied in query-focused summarization that each achieve state-of-the-art results, including a two-step model and an end-to-end model.
Exploring Neural Models for Query-Focused Summarization
Long document summarization addresses the setting where source document length exceeds the input limits of standard pre-trained models. Approaches to this task can largely be divided into two categories: two-step extractive-abstractive frameworks, which first extract a subset of the text as input to an abstractive model, and end-to-end models, which process the input within a single model. The two-step pipeline has been applied to topic-focused Wikipedia summarization (Liu et al., 2018b;Perez-Beltrachini et al., 2019), low-resource summarization (Bajaj et al., 2021), and single-document summarization Chen and Bansal (2018). End-to-end approaches address the input-length problem using sparse-attention models. Beltagy et al. (2020) introduce the Longformer, consisting of local attention as well as global attention between select input tokens. Other approaches make use of dynamic attention mechanisms Manakul and Gales, 2021;Cui and Hu, 2021), sliding window strategies (Liu and Chen, 2021), and other mechanisms to introduce sparsity into the model (Huang et al., 2021;. Izacard and Grave (2021) concatenate the outputs of multiple encoders as input to a generator component for the task of open domain question answering. In our work we build on these models for query-focused summarization and perform extensive hyperparameter ablations, achieving state-of-the-art results over other twostep and end-to-end models.
Exploring Neural Models for Query-Focused Summarization
We present existing methods and propose modeling extensions to address the challenges of queryfocused summarization.
Exploring Neural Models for Query-Focused Summarization
Two-step approaches consist of an extractor model, which extracts parts of the source document relevant to the input query, and an abstractor model, which synthesizes the extracted segments into a final summary. We consider score-and-rank extractor models, which first score each source passage for relevance to the query and then rank the passages in descending order of relevance, with the concatenated and truncated results passed to the abstractor. In this work we present two types of scoring models: single-encoder models and dualencoder models, which we describe below. All two-step approaches share the same abstractor, a BART-large model.
Exploring Neural Models for Query-Focused Summarization
Single encoder models concatenate a query and source passage as input to the scoring function that produces the similarity score. Those models benefit from full cross-attention between query and passage, resulting in richer data representations.
Exploring Neural Models for Query-Focused Summarization
MARGE (Xu and Lapata, 2021a) is a singleencoder, Masked ROUGE extractor that aims to improve upon low-resource query-focused summarization by synthesizing query-focused data from more resource rich, generic summarization datasets. This model is trained to predict the relevance of each passage in the source document with respect to a query, where the proxy for relevance is the ROUGE overlap between the passage and the reference summary. For training on generic summarization datasets, MARGE uses pseudo-queries that are created by masking content words in the reference summaries.
Exploring Neural Models for Query-Focused Summarization
When performing inference using real queries, certain query words (e.g., wh-words) are masked to better align the queries to the pseudo-queries from the training process. Following Xu and Lapata (2021a), we apply MARGE trained for masked relevance prediction on Multi-News (Fabbri et al., 2019b) without training on our target dataset.
Exploring Neural Models for Query-Focused Summarization
RELREG Motivated by the retrieval component of MARGE, we propose the RELREG (RELevance REGression) model, which trains a relevance prediction model directly on QFS data using the original, non-masked query. Like MARGE, this model is trained to predict the ROUGE overlap between a source passage and the reference summary, using only the passage and query as input. A single-encoder model jointly encodes the delimiterseparated query and passage, and the final layer of the model outputs the predicted relevance value.
Exploring Neural Models for Query-Focused Summarization
Dual-encoder models separately encode a query and source passage before calculating the cosine similarity between the embeddings to compute the relevance score. This class of models offers computational benefits, as passage embeddings may be precomputed and stored for a given input, while the single-encoder model must be run over all passages should a new query be introduced.
Exploring Neural Models for Query-Focused Summarization
DPR (Karpukhin et al., 2020) is a dual-encoder model that separately encodes queries and passages into an embedding space optimized for calculating semantic similarity between the two, showing improved results over traditional vector-space models. We fine-tune a DPR extractor model directly on the target dataset. As opposed to other locators that optimize with respect to the continuous ROUGE overlap, DPR uses the ROUGE score between the passage and reference summary to identify binary positive and negative passages and optimizes the negative log likelihood of the positive passages.
Exploring Neural Models for Query-Focused Summarization
RELREGTT (RELevance REGression Two Tower) is a more computationally-efficient version of RELREG that uses a dual-encoder architecture to predict ROUGE-based relevance scores. This model is implemented with a backbone architecture of Sentence-BERT (Reimers and Gurevych, 2019), using a shared-parameter encoder for each of the query and passage and a special token appended to each input that identifies it is as either query or passage, following the suggested best practices of Reimers and Gurevych (2019). The final output for the model is based on the inner product of the pooled embeddings for the query and passage.
Exploring Neural Models for Query-Focused Summarization
Two-step pipelines depend on the strength of the retrieval component, and may still fail to capture all relevant content despite an ideal retriever, due to length limitations of the generation component. This motivates our experiments on end-to-end models that can incorporate longer input texts.
Exploring Neural Models for Query-Focused Summarization
BART (Lewis et al., 2020) As a baseline endto-end model, we consider BART, an encoderdecoder Transformer model pre-trained using a denoising objective. BART is composed of a bidirectional encoder module and an autoregressive decoder model that attends to the encoder's final layer outputs. Due to the quadratic memory complexity of the encoder's full self-attention mechanism, the model input size is limited to 1024 tokens. In our experiments, we prepare the input to BART by concatenating the query, a delimiter token, and the source document, and then truncating the combined text to the model's input size.
Exploring Neural Models for Query-Focused Summarization
LED To circumvent the input size limitations of the BART model, we include the Longformer Encoder-Decoder (Beltagy et al., 2020) (LED) in our study LED replaces the quadratic selfattention mechanism of traditional Transformers with a memory-efficient version that combines local attention with sparse global attention. The architecture allowed us to run experiments with input sizes up to 16384 tokens. Based on insights from the original work on tuning the model to the QA task, we configure the global attention mechanism to span the entire query.
Exploring Neural Models for Query-Focused Summarization
SEGENC We also consider a simpler form of sparse attention in the encoder based solely on windowed local attention, combining elements of LED with Fusion-in-Decoder (FiD) (Izacard and Grave, 2021), a model for open-domain question answering. In our Segment Encoder (SEGENC) model, the source document is split into fixed-length overlapping 1 segments, each of which is separately appended to the query and encoded using a standard Transformer model. Similar to FiD, these encodings are then concatenated into a single embedding sequence and passed to a decoder model that generates the summary. Since there is no cross-attention between the encoded segments, the attention mechanism scales linearly in the number of segments and hence the length of the source document. Nonetheless, the decoder can attend to all encoded segments jointly, enabling the encoderdecoder architecture to operate in an end-to-end fashion. This model is motivated by two hypotheses: 1) query-relevant sections within a source document are often small enough to be processed by standard Transformer models (e.g. 1024 tokens), and 2) each query-relevant section may be understood independently of other sections, removing the need for cross-attention between the segments.
Exploring Neural Models for Query-Focused Summarization
We analyze our methods on two high-quality queryfocused, long-document datasets. QMSum (Zhong et al., 2021b) is a query-focused dialogue summarization dataset consisting of 1,808 query-summary pairs over 232 meetings from product design, academic, and political committee meetings, all conducted in English. QMSum also includes additional annotations such as topic segmentations and highlighted text spans associated with reference summaries. We leverage the provided span annotations to run oracle experiments. We focus our analysis on QMSum due to the availability of prior work as points of comparison. AQuaMuSe (Kulkarni et al., 2020) is a queryfocused multi-document summarization dataset consisting of 5,519 query-long answer summary pairs from the Natural Questions questionanswering dataset (Kwiatkowski et al., 2019) and associated input documents from the Common Crawl 2 . Input documents for the original dataset were selected based on embedding similarity with respect to the summary, and hyperparameters can be chosen to control the level of semantic overlap between the input document set and the summary. Data replication details are found in the Appendix. We use AQuaMuSe to examine the generalizability of our QMSum results.
Exploring Neural Models for Query-Focused Summarization
Implementation Models were implemented using the PyTorch  and Huggingface (Wolf et al., 2019) libraries. Model weights were initialized from pre-trained checkpoints available through the Huggingface Model Hub 3 . All BART models were based on the facebook/bart-large checkpoint, the LED-model was based on the allenai/led-large-16384 checkpoint, which itself is based on BART-large.
Exploring Neural Models for Query-Focused Summarization
Training & Inference Models were trained for 10 epochs with final checkpoints selected based on the average of ROUGE-{1, 2, L} (R-1, R-2, R-L) scores achieved on the validation set. Gradient checkpointing (Chen et al., 2016) was used for the LED and SEGENC models to reduce the memory footprint. Model outputs were decoded using beam search with 4 beams. To ensure high consistency of results, all experiments in §4 were repeated 5 times with results averaged across runs.
Exploring Neural Models for Query-Focused Summarization
Evaluation Models were automatically evaluated using the ROUGE-{1, 2, L} metrics (Lin, 2004) included in the SummEval toolkit (Fabbri et al., 2021b). Models were also manually evaluated by hired human annotators. Annotators were hired through the Amazon Mechanical Turk platform. Workers were selected from English speaking countries and offered an hourly rate of approximately 12 USD. The study was conducted on 50 model generated examples chosen at random from the test set of QMSum.
Exploring Neural Models for Query-Focused Summarization
In this section, we first analyze the effects of modelspecific architectural and hyperparameter choices on the performance of two-stage ( §4.1) and end-toend models ( §4.2). Next, we study the task-specific knowledge transfer capabilities of different pretraining strategies in §4.3. Lastly, we conduct a final evaluation and comparison of all discussed models in §4.4. All experiments and analyses presented in this section were conducted on QMSum.
Exploring Neural Models for Query-Focused Summarization
For two-stage models, we first focus on evaluating the extractor component and comparing performance to baseline heuristics. We quantify extractor performance using two metrics: 1) lexical overlap between the extracted utterances and reference summaries, computed using R-1, R-2, and R-L metrics, 2) span overlap between the extracted and golden spans included with QMSum represented by Precision and Recall scores, with results shown in Table 1. In both cases, we first order utterances of the conversation according to the scores assigned by the extractor models, then concatenate the utterances and finally truncate the result to 1024 tokens (excluding the space reserved for the query) to mimic the input length limits of downstream abstractor models; we present those numbers as the All columns in the table. For the lexical overlap, we also show the scores for the best 1 (Top-1), 5 (Top-5), and 15 (Top-15) utterances.
Exploring Neural Models for Query-Focused Summarization
The results show that the best-performing extractor model is RELREG closely followed by REL-REGTT in the Top-1 evaluation and DPR in the Top-5, Top-15, and All cases. We note that both the RELREG and RELREGTT models tend to select longer utterances than the other extractors; the regression-based training mirrors the ROUGE overlap score which favors longer, more informative utterances. However, despite their strong performance in extracting top-matching utterances, the results also expose a considerable gap between model-based approaches and human annotations when considering the entirety of extracted spans. This shows a promising topic for future work in this matter. We also notice that despite the simplicity of the LEAD heuristic, which extracts the first k utterances in their original order, it remains competitive with the data-driven extractor models when we consider the All case. An extended version of this study, which includes the lexical overlap between extracted spans and input queries is presented in Table 8 in the Appendix.
Exploring Neural Models for Query-Focused Summarization
Next, we analyze how the performance of the extractor components carries over to the final summarization task. For the best-performing model, we additionally test the effect of varying the input segment size used during training and inference between 256 and 512 tokens. Validation-set results for all models are reported in Table 2.
Exploring Neural Models for Query-Focused Summarization
We find that DPR slightly outperforms REL-REGTT for dual-encoder models.
Exploring Neural Models for Query-Focused Summarization
Among single-encoder models, RELREG outperforms MARGE by over a full R-1 point, which may explained by RELREG using more direct supervision based on an in-domain query, rather than creating  Table 1: Performance of extractor models on the QMSum validation set. The left section presents the lexical overlap between the utterances retrieved by extractor models and the reference summaries, evaluated by means of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) metrics. Segments of the section focus on the lexical overlap between the highest ranked 1 (Top-1), 5 (Top-5), 15 (Top-15) utterances, and all utterances truncated to a 1024 token limit (All). The table also includes the average word counts of all extracted utterances, denoted asx. The right section shows the span overlap between the utterance spans retrieved by the extractor models and those collected from human annotators by the authors of QMSum. The performance is evaluated by means of Precision and Recall scores and uses the highest ranked utterances truncated to the limit of 1024 tokens.  Table 2: Performance of two-step models on the QM-Sum validation set, divided into dual-encoder and single-encoder extractors. Input segment lengths are indicated in parentheses, and otherwise the model operates on utterance-level input. synthetic queries from an external dataset using masking. We find that the single-encoder REL-REG outperforms the best dual-encoder model; the cross-attention term in the single-encoder REL-REG model allows it to better attend to the query when determining relevance. Intuitively, the ordering of results corresponds to the span overlap recall with the gold spans; the ability of the extractor to select produce high-recall rankings directly affects abstractor performance. We see that increasing the input segment length used in training and inference for RELREG improves at 256 tokens but decreases at 512 tokens, suggesting that a balance is found between including additional context for ranking versus enabling a greater number of shorter segments that may capture more diverse content from the source.
Exploring Neural Models for Query-Focused Summarization
We explore hyperparameter choices for two endto-end architectures described in §3.2: the Longformer Encoder-Decoder (LED) and Segment Encoder (SEGENC). For both models, we consider different choices for input size (4096, 8192, Table 3: Performance of end-to-end models on the QM-Sum validation set, across varying input and attention window sizes (in number of tokens). SEGENC-D is a variant of SEGENC in which the segments are disjoint rather than overlapping; this ablation was evaluated on the best-performing SEGENC hyperparameters. tokens) and attention window size 4 (256, 512, or 1024 tokens). For SEGENC, we also consider two different segmentation strategies: overlapping segments (50% overlap) and disjoint segments. Validation set results for both models and a baseline BART model are reported in Table 3. We notice that both the LED and SEGENC benefit from increasing the input size and perform best with the input limit set to 16,384 tokens. The optimal attention window for LED is 1024, while SEGENC performs best with an attention window  Table 4: QMSum validation-set performance of the end-to-end BART models first fine-tuned on related summarization tasks and then further fine-tuned on QMSum data. The model indicates the task first finetuned on, and input is truncated to 1024 tokens.
Exploring Neural Models for Query-Focused Summarization
of 512 tokens. For SEGENC, using overlapping segments improves performance compared to using disjoint segments, suggesting that the additional context provided by the former approach is helpful for locating relevant content. The SEGENC model achieves the highest performance out of the endto-end architectures with ROUGE scores of 37.47 R-1, 12.47 R-2, and 32.95 R-L on the validation set.
Exploring Neural Models for Query-Focused Summarization
The results also highlight that while the LED model matches or slightly outperforms the BART baseline for higher maximum input and window sizes, it performs substantially worse than SEGENC. This observation is consistent with prior findings on the QMSum dataset (Zhang et al., 2021b). One possible explanation for the lower performance of LED relative to SEGENC is that LED must adapt its parameters for a global attention mechanism that is absent from the backbone BART encoder model, whereas SEGENC relies solely on local self-attention that is aligned with the backbone model. This may be particularly relevant to QMSum given its relatively small size.
Exploring Neural Models for Query-Focused Summarization
Practitioners may wish to consider the computational cost and efficiency of various hyperparameter settings. Computational complexity increases with both input length and attention window size (since attention grows quadratically in attention-window size). Complexity is also greater with the overlapping segment strategy compared to the disjoint segment strategy for the SEGENC model, due to the greater number of resulting segments that are passed through the encoder and decoder modules.
Exploring Neural Models for Query-Focused Summarization
Having determined the best-performing models, we examine whether performance can be further improved by fine-tuning a model that has already been fine-tuned for a different summarization task. We conduct this study using the end-to-end BART on 1024 tokens, as this model is the backbone, al-  Table 5: QMSum test-set performance of two-stage and end-to-end models that performed best on the validation set (Tables 2 and 3), including versions fine-tuned from the WikiSum-finetuned checkpoint (denoted by -W). Results reported in prior work are italicized. Also included is an extractive-oracle model that takes the gold spans ( §3.3) as input.
Exploring Neural Models for Query-Focused Summarization
beit in varying ways, of both our two-step and end-to-end models. We test the transferring capabilities of models trained on the news summarization task from CNN/DailyMail (Nallapati et al., 2016), which performed best among non queryfocused datasets in Zhang et al. (2021b). We also explore transferring from the previously-mentioned query-and topic-focused summarization tasks: An-swerSumm, AQuaMuSe, WikiHowQA, and Wik-iSum. We compare to fine-tuning from the original BART checkpoint, with results shown in Table 4. We find that transferring from any of the tasks improves over no transfer in R-1 and R-L. Transferring from any of the constrained, query-focused tasks outperforms transferring from unconstrained news summarization. Furthermore, transferring from WikiSum outperforms transfer from other datasets, which aligns with other work that shows the generalizability of Wikipedia as a source of data for task transfer (Fabbri et al., 2021a).
Exploring Neural Models for Query-Focused Summarization
We now measure the test set performance of the best-performing architectures from §4.1 and §4.2 in combination with the optimal transfer-learning approach from §4.3. Results are presented in Table 5 along with baseline models.
Exploring Neural Models for Query-Focused Summarization
We find that RELREG and SEGENC outperform existing state-of-the-art models by a substantial margin, and that initializing the model from the  Table 6: Human evaluation of two best-performing models from Section 4, along with a baseline BART model. Summaries were evaluates across four dimensions: fluency (Flu.), relevance (Rel.), completeness (Comp.), and factuality (Fact.).
Exploring Neural Models for Query-Focused Summarization
Wikisum-fine-tuned checkpoint further improves performance, with the best model exceeding current state-of-the-art performance by a difference of 3.38 R-1, 3.72 R-2, and 3.28 R-L. Comparing the best models from each category, we find that the end-to-end approach outperforms the two-stage. Within the two-stage dual-encoder models, REL-REGTT outperforms DPR on the test set despite the slightly worse performance on the validation set. We attribute this variation to the small size of the validation set, and our other findings remain consistent across validation and test sets. The single-encoder RELREG outperforms the best dualencoder model, with RELREG-W improving upon the current state-of-the-art performance by a difference of 2.03 R-1, 3.10 R-2, and 2.18 R-L.
Exploring Neural Models for Query-Focused Summarization
In this section we conduct further analysis of the best performing models from Section 4. First, we offer additional insights into the performance of those models on the QMSum dataset through a human-based study. Next, we discuss the generalization abilities of those models by running experiments on the AQuaMuSe dataset.
Exploring Neural Models for Query-Focused Summarization
To gain a better understanding of the performance of the models on the QMSum dataset, human judges were hired and asked to assess the quality of generated summaries. Summaries were evaluated across four dimensions: 1) fluency, measuring their grammatical quality, 2) relevance, assessing their relevance to the input query, 3) completeness, evaluating their comprehensiveness considering the input conversation and query, and 4) factuality, measuring their factual consistency with respect to the conversation. Scores were assigned on a Likert scale from 1 to 5 (best), where each example was evaluated by 3 judges with the final score averaged. Results are presented in Table 6.  Table 7: AQuaMuSe test-set performance of two bestperforming models from §4, along with a baseline BART model and previously reported results (in italics) for Hi-MAP (Fabbri et al., 2019a) from Kulkarni et al. (2020). Note that the version of the dataset used for previous results would have been slightly different due to variations in document selection parameters and Common Crawl indices (see Appendix).
Exploring Neural Models for Query-Focused Summarization
We find that the RELREG-W and SEGENC-W models achieved comparable performance across all of the evaluated dimensions, with summaries generated by SEGENC-W rated as slightly more complete. The BART baseline was rated highest in the fluency dimension, however, it was substantially outperformed by both of the introduced models on completeness and factuality. One possible explanation for the slightly lower fluency scores for the RELREG-W and SEGENC-W models is that they are better able to retrieve content from the source, which itself may have low fluency due to its conversational nature. The results also highlight a gap between the performance of existing models and perfect scores, which shows that there is potential for improvement in future work.
Exploring Neural Models for Query-Focused Summarization
To test that the automated evaluation results generalize beyond the QMSum dataset, we trained and evaluated the best-performing models on AQua-MuSe, another high-quality dataset for QFS that includes long documents ( §2.1, §3.3). Test-set performance for the best-performing two-stage and endto-end models, along with a baseline BART model, are shown in Table 7. Results are consistent with those for the QMSum dataset (Table 5), with the best end-to-end model (SEGENC-W) outperforming the best two-stage model (RELREG-W), and both outperforming the baseline (BART) model.
Exploring Neural Models for Query-Focused Summarization
In this work, we conducted an exploratory study of neural models for query-focused summarization. We studied two categories of models: two-stage and end-to-end, and presented two architectures, RELREG and SEGENC, both of which improve ROUGE performance over prior state of the art by a substantial margin. We also explored taskspecific transfer learning, which further improved model performance. Besides model performance, we discussed issues of computational efficiency that practitioners may factor into their modeling choices. Finally, we conducted a human study suggesting that the summaries produced by the bestperforming models are more factually correct and complete than a baseline model by a substantial margin. We hope that the analysis and modeling contributions of this paper will be a resource for future research on query-focused summarization.
Exploring Neural Models for Query-Focused Summarization
Dataset Biases QMSum and AQuaMuSe contain meeting transcripts and documents in English and thus mainly represent the culture of the Englishspeaking populace. Political or gender biases may also exist in the dataset, and models trained on these datasets may propagate these biases Additionally, the pretrained BART model carries biases from the data it was pretrained on. We did not stress test these models for biases and request that the users be aware of these potential issues in applying the models presented.
Exploring Neural Models for Query-Focused Summarization
Crowdsourcing Protocols Workers were compensated $1 per example, calibrated to equal a $12/hour payrate. We use the following qualifications to recruit MTurk workers with good track records: HIT approval rate greater than or equal to 97%, number of HITs approved greater than or equal to 10000, and located in one of the following English native-speaking countries: Australia, Canada, New Zealand, United Kingdom, United States.
Exploring Neural Models for Query-Focused Summarization
Misuse Potential and Failure Mode When properly used, the summarization models described in this paper can be time-saving. However, the current model outputs may be factually inconsistent with the input documents, and in such a case could contribute to misinformation on the internet. This issue is present among all current abstractive summarization models and is an area of active research.
Exploring Neural Models for Query-Focused Summarization
The experiments described in the paper primarily make use of A100 GPUs. We typically used a single GPU per experiment, and the experiments may take up to a day when repeating across random seeds. The largest backbone model used, BART-Large, has 400 million parameters. While our work required extensive experiments, future work and applications can draw upon our insights and need not repeat these comparisons.
Exploring Neural Models for Query-Focused Summarization
We thank Divyansh Agarwal for his insights on existing model architectures and assistance with training DPR. We thank Semih Yavuz for providing the initial implementation of the Fusion-in-Decoder model. 
Conditional Generation with a Question-Answering Blueprint
Neural generation models are often prone to hallucination (Song et al., 2018;Kryscinski et al., 2020;Gabriel et al., 2021), repetition and redundancy (Li et al., 2018;Suzuki and Nagata, 2017), and struggle to identify which content units are salient (Tan et al., 2017a). These phenomena are amplified when generating longform text, i.e., documents with multiple paragraphs (Wiseman et al., 2017), when dealing with non-linguistic data (e.g., database tables), or very long input which is common when summarizing multiple documents Perez-Beltrachini et al., 2019), books , or dialogue (Chen et al., 2022;Zhong et al., 2021). An additional challenge concerns the blackbox nature of deep learning systems which hides the inherent complexity of modeling multiple interconnected linguistic phenomena in text generation, and makes it difficult to examine model decisions and attribute errors to specific components. The lack of modularity further affects controllability as these systems cannot be easily tailored to individual needs.
Conditional Generation with a Question-Answering Blueprint
Attempts to remedy some of these issues focus on changing the way entities are represented (Puduppully et al., 2019b;Iso et al., 2019), allowing the decoder to skip low-confidence tokens to enhance faithful generation (Tian et al., 2019), modeling graph connections between document elements to better capture salience (Tan et al., 2017b;, encoding documents hierarchically (Celikyilmaz et al., 2018;Rohde et al., 2021), learning latent alignments between the input and the target text (Xu et al., 2021) adopting sparse attention mechanisms (Child et al., 2019;Beltagy et al., 2020), and introducing content selection (Gehrmann et al., 2018;Dou et al., 2021) and planning components (Puduppully et al., 2019a;Moryossef et al., 2019b;Narayan et al., 2021;Wiseman et al., 2018).
Conditional Generation with a Question-Answering Blueprint
In this paper we also aim to render conditional generation more modular via an intermediate, plan-based representation. While autoregressive models of language predict one token at a time, there is evidence that in humans some degree of planning occurs at a higher level than individual words (Levelt, 1993;Guhe, 2007). A long tradition in natural language generation views planning as a central component to identifying important content and structuring it appropriately (Reiter and Dale, 2000), however, there is less agreement  Table 1: Question-answering (QA) blueprint for AQuaMuSe summary. QA pairs were obtained from a state-of-the-art question generation and answer identification system .
Conditional Generation with a Question-Answering Blueprint
on how plans should be represented. Common examples include discourse trees (Mellish et al., 1998), entity transitions (Kibble and Power, 2004;Barzilay and Lapata, 2008), sequences of propositions (Karamanis, 2004), and schemas (McKeown, 1985). Our work proposes a new conceptualization of text plans as a sequence of question-answer pairs. Specifically, we draw inspiration from the "Questions under Discussion" (QUD) theory of discourse structure which posits that one way of articulating the structure of a text is to identify the questions and sub-questions that are raised and answered by subsequent spans of text (Carlson, 1983;Ginzburg, 1994;Van Kuppevelt, 1995;Larson, 2002;Roberts, 2012;Riester, 2019). Theoretical models of Questions under Discussion assume that discourse contains implicit questions for each of the assertions made which are thereby turned into answers. These questions and answers can be understood in terms of their use in moving a discourse forward to achieve communicative goals. We propose to make QUDs explicit by exploiting state-of-the-art question generation technology Lu and Lu, 2021) and use them as an intermediate representation layer for conditional generation, i.e., a questionanswering (QA) blueprint operating as a proxy for both content selection (i.e., what to say) and planning (i.e., in what order). Table 1 illustrates a plan for generating a Wikipedia abstract from the AQuaMuSe dataset (Kulkarni et al., 2020).
Conditional Generation with a Question-Answering Blueprint
We enhance existing datasets (e.g., for summarization) with similar blueprints which we obtain automatically. We then convert input-output pairs into inputblueprint-output tuples and propose to learn encoder-decoder models from these augmented annotations. We develop three models which vary in how they integrate blueprints in the generation process and their ability to handle long outputs. Aside from generating blueprints and their correspond-ing text in one go, we propose a new architecture which iteratively plans and generates a sentence at a time, conditioning on the input and the output sentences generated so far. We do not generate a global blueprint, rather our planning process is incremental and informed by generation, which we argue affords greater control over the output and its fluency. Moreover, the model is better equipped for long-form generation, since it does not have to (autoregressively) decode the blueprint and its summary in one go, avoiding the risk of exceeding the maximum decoder length.
Conditional Generation with a Question-Answering Blueprint
We instantiate our models with a Transformer (Vaswani et al., 2017) encoder-decoder architecture and perform experiments on summarization datasets representing different information seeking tasks, application domains, and user requirements. 1 In all cases, we empirically demonstrate that blueprint models are more factual than alternatives which do not resort to planning; we also observe that QA blueprints are a better representation compared to plans based on entity chains (Narayan et al., 2021), allowing tighter control of the output, and providing a comprehensive explanation for model predictions (if the plan is erroneous, then the summary will be too).
Conditional Generation with a Question-Answering Blueprint
Question-answer pairs have been previously used as a proxy for expressing semantic content. QA-SRL (He et al., 2015) is a representation based on QA pairs which has been shown to capture the vast majority of arguments and modifiers in Prop-Bank (Palmer et al., 2005) and NomBank (Meyers et al., 2004). Instead of using a pre-defined role lexicon, QA-SRL labels semantic roles with questions, whose answers denote the argument bearing the role. Follow-on work uses QA pairs to represent discourse relations (Pyatkin et al., 2020) and to capture overlap or redundancy at the propositional level (Brook Weiss et al., 2021). We also employ QA pairs as an abstraction of propositional content, however, we do not target specific relation types, or make any linguistic assumptions about them (e.g., discourse relations vs semantic roles).
Conditional Generation with a Question-Answering Blueprint
Question-Answering in Summarization QA pairs have been used for evaluating summaries (Deutsch and Roth, 2021b;Eyal et al., 2019;Durmus et al., 2020;Wang et al., 2020), specifically as a means of estimating the information overlap between a reference summary and a systemgenerated one. QA-based signals have also been incorporated in the training of summarization models, using reinforcement learning (Arumae and Liu, 2018Scialom et al., 2019) or as a way of identifying salient content in the input document (Deutsch and Roth, 2021a). Cao and Wang (2022) introduce the task of hierarchical questionsummary generation, where a source document is condensed into multiple summaries each answering a different question. Questions are organized hierarchically into broad questions and more specific sub-questions which are learned from manual annotations. Our model outputs a QA-based plan and a single summary for a given document, although it is possible to generate different summaries from different plans for the same document. Our QA pairs are obtained automatically and they are not stuctured.
Conditional Generation with a Question-Answering Blueprint
Planning in Encoder-Decoder Models Various recent efforts have developed planning modules in the context of data-to-text generation. In most cases, the plans are specific to the input which varies from tables and records to RDF tuples. For instance, Puduppully et al. (2019a) learn a plan corresponding to a sequence of records, and generate a summary conditioned on it.  treat content selection as a task similar to extractive summarization, they first extract sentence plans and then verbalize them one-by-one. Moryossef et al. (2019a,b) propose a symbolic planning stage followed by a neural realization stage. Other work (Puduppully and Lapata, 2021;Puduppully et al., 2022) advocates macro planning, where document content is organized into a sequence of paragraph plans which are verbalizations of tabular input. Our work is closest to Narayan et al. (2021) who also target summarization applications and learn an intermediate plan to guide generation. We adopt a more elaborate plan representation based on QA blueprints, and interface decoding with plan generation similarly to .
Conditional Generation with a Question-Answering Blueprint
Let d denote the input to the model which could be a document (or multiple documents), a dialogue history, or even database tables. The model will learn to generate blueprint b for output s (e.g., a summary) and the output itself. The blueprint b is an ordered set of question-answer pairs {(q 1 , a 1 ), (q 2 , a 2 ), . . . , (q m , a m )}. Unsurprisingly, such blueprints are not naturally occurring in existing datasets which typically consist of (d, s) pairs. In the following we explain how we automatically augment training examples (d, s) into tuples (d, b, s) with blueprints (Section 3.2) and then describe how we devise blueprint models based on them (Section 3.3).
Conditional Generation with a Question-Answering Blueprint
We first explain how question-answer pairs are automatically (over-)generated for output s, and sub-   Figure 1 and blueprint annotation. We split the summary into propositions P and select no more than one QA pair per proposition. RT, RH, and CO are shorthands for Round Trip, Rheme, and Coverage. Questions that pass/fail each filter are marked with ✓/✗. sequently filtered to create blueprint b. We illustrate the different filtering stages via the example in Table 2.
Conditional Generation with a Question-Answering Blueprint
Question-Answer Generation We generate QA pairs following an approach similar to Honovich et al. (2021Honovich et al. ( , 2022. We convert the SQuAD reading comprehension dataset (Rajpurkar et al., 2018b) to a question generation dataset by concatenating the answer and context (with separators) and fine-tuning a sequence-to-sequence transformer model to predict the question. Specifically, we fine-tune the T5-11B checkpoint from Raffel et al. (2020); questions are decoded with a beam size of 4. During training, answer candidates are the answers provided in the SQuAD annotation. At inference time, answer candidates (i.e., base noun phrases and named entities) are identified in the output s using SpaCy 2 and questions are generated with the SQuAD trained system. This procedure yields a large list of QA pairs (see in Table 2 the questions generated for the summary at the bottom), which we reduce using the filtering explained below.
Conditional Generation with a Question-Answering Blueprint
Question-Answer Blueprints Initially, we apply a Round-trip Consistency check  which discards questions if they yield answers different from those used to generate them. In Table 2, Q 11 is discarded as the answer it is paired with is wrong ( Table 2) into propositions -a proposition is a sub-sentential unit which represents a single claim or fact (Stanovsky et al., 2018;Ernst et al., 2022), We use propositions instead of sentences since the latter can be too long and contain multiple facts. We split text into propositions based on punctuation (period, comma, and semicolon), coordination (e.g., and, but), relative pronouns (e.g., that, who), and prepositions (e.g., at, by). Following this simple approach, the summary in Table 2 is split into six propositions, shown within square brackets. We next match each proposition to a single QA pair heuristically, following a two-stage approach.
Conditional Generation with a Question-Answering Blueprint
We first find the question whose answer is at the rightmost position within a proposition. If there are multiple such questions, we select the one with the longest answer. This first stage, which we call Rheme, is motivated by the theme-rheme structure (Vallduví and Vilkuna, 1998) of natural language sentences: already known information (i.e., the theme) is usually placed first while new information (i.e., the rheme) is placed later in a sentence or phrase (Kruijff-Korbayová and Steedman, 2003). Following this idea, Rheme selection prioritizes new-information seeking questions. As can be seen in Table 2, it eliminates several questions (e.g., Q 1 -Q 4 ) as their answers are not the right most element in the obtained propositions. Questions Q 5 and Q 6 are identical, however we retain Q 5 as it yields the longest answer.
Conditional Generation with a Question-Answering Blueprint
The second stage, which we call Coverage, prioritizes the selection of informative QA pairs by selecting non-overlapping ones. Specifically, we first convert s to a bag of tokens and select the QA pair with the highest lexical overlap. We then remove the overlapping tokens from s, and repeat this greedy selection process until the bag is empty or the overlap is zero. Table 2 shows how Coverage further eliminates QA pairs Q 5 and Q 8 . The remaining four QA pairs constitute the final blueprint b. Rather than defaulting to a random order, we sort these based on the location of the answer spans in s (see the final order in Table 1).
Conditional Generation with a Question-Answering Blueprint
We devised three seq-to-seq models, which differ in the way the output and its blueprint are generated.
Conditional Generation with a Question-Answering Blueprint
End-to-End Model A straightforward approach would be to take d as input and learn to first predict blueprint b as p(b|d), and then generate output s as p(s|b). However, this approach crucially relies on the blueprint being accurate and capturing all required information, which might be overly optimistic, given that blueprints (for training) are generated automatically. Moreover, pipeline architectures are known to suffer from error propagation, which in our case would undoubtedly affect generation performance, the final stage of the pipeline.
Conditional Generation with a Question-Answering Blueprint
Rather than modeling the blueprint and output generation stages separately, we train an encoderdecoder model to encode d and generate b; s (i.e., the concatenation of the blueprint and output sequence) in one go. Essentially, the decoder first predicts blueprint b and then continues to generate output s, using both b and d. We prefix b and s with special markers "Plan:" and "Summary:", respectively. In particular, we predict b as a 1 ; q 1 ; . . . ; a m ; q m , namely a (concatenated) sequence of answer-question pairs. 3 The model is trained with the standard maximum-likelihood objective to generate the augmented target b; s. Interestingly, in this end-to-end model the blueprint functions as a macro-plan, i.e., a global sketch of the content and organization of the output.
Conditional Generation with a Question-Answering Blueprint
Multi-task Model It is generally challenging for encoder-decoder models to generate long output sequences Tan et al., 2021). The end-to-end model sketched above further amplifies this problem because it ultimately aims to generate sequence b; s rather than just s, increasing the sequence length by 220% (see Table 3).
Conditional Generation with a Question-Answering Blueprint
To mitigate this problem, we propose a multitask model optimized to perform two separate tasks. Let a and q denote an ordered sequence of answers (a 1 , . . . , a m ) and corresponding questions (q 1 , . . . , q m ), in blueprint b. The model is trained to generate (a) the answer plan concatenated with output sequence a; s, and (b) the answer plan concatenated with questions a; q. In particular, we train a single encoder-decoder model to encode input d, while the decoder first predicts answer plan a (as p(a|d)) and then continues to generate output s (as p(s|a, d)) or corresponding questions q (as p(q|a, d)), depending on the task. We prefix a, q, and s with special markers "Plan:", "Questions:", and "Summary:", respectively. We further prefix input d with "Generate Summary:" or "Generate Questions:" to instruct our model to generate output s or questions q, respectively. We sample data points from these two tasks with equal probability and train the model with the standard maximum-likelihood objective.
Conditional Generation with a Question-Answering Blueprint
During inference, we use a two-step process to generate output s ′ and its blueprint b ′ for input d. We first prefix d with "Generate Summary:" and generate a ′ ; s ′ , i.e., answer plan a ′ followed by output sequence s ′ . We then prefix d with "Generate Questions:", prompt our decoder with the predicted answer plan a ′ and generate corresponding questions q ′ for blueprint b ′ . The multi-task model alleviates the length issue discussed above by learning to generate a; s instead of b; s. However, this comes at the expense of generation quality, since the model now conditions on the answers only, not question-answer pairs. As such, it can be viewed as an extension of FROST (Narayan et al., 2021) with the plan being a sequence of answer spans rather than entity chains. This model also creates a macro-plan of the output, however, less detailed compared to the end-to-end model.
Conditional Generation with a Question-Answering Blueprint
Rather than predicting a global plan (i.e., answer plan a or blueprint b) prior to generating output s, we employ an incremental approach which interleaves planning with text generation. Let output s consist of n sentences {s 1 , s 2 , . . . , s n }; then, the corresponding blueprint b can be represented as
Conditional Generation with a Question-Answering Blueprint
. . , (a i j+k , q i j+k )} consists of k question-answer pairs for sentence s i . We train our model to iteratively plan and generate one sentence at a time, conditioning on the input and the output sentences generated so far. In particular, we train an encoder-decoder model where the encoder first encodes input d, while the decoder takes summary {s 1 , . . . , s i } generated so far as a prompt and generates blueprint b i+1 for next sentence s i+1 , followed by sentence s i+1 itself.
Conditional Generation with a Question-Answering Blueprint
The iterative model is trained on quadru-
Conditional Generation with a Question-Answering Blueprint
is an empty context placeholder used to predict the first blueprint b 1 and corresponding first sentence s 1 , (n + 1) is the blueprint length, and s 1,i = {s 1 , . . . , s i } are the output sentences generated so far; b end and s end are special tokens marking the end of the output prediction. We prefix s 1,i , b i , and s i with special markers "Context:", "Plan:", and "Next Sentence:", respectively. We train the model with the standard maximumlikelihood objective to predict s 1,i ; b i ; s i , however, we do not compute the loss for predicting context s 1,i to avoid over-optimizing for sentences that appear at the beginning of the output.
Conditional Generation with a Question-Answering Blueprint
The iterative approach does not create a global macro plan. Rather, it learns micro content plans and verbalizes them one-by-one, conditioning on previously generated sentences but not on previously generated QA pairs. Athough it does not have a global document view like the end-to-end model, the iterative decoder cannot exceed the output sequence length as it plans and predicts one sentence at a time as b i ; s i , instead of generating b; s in one go. And unlike the multi-task model, each sentence s i is generated by conditioning on the full blueprint b i (consisting of questions and answers).
Conditional Generation with a Question-Answering Blueprint
We evaluated our model on benchmarks representative of long-form question answering and summarization. Our datasets vary in terms of the input given to the generation model (e.g., multiple  Table 3: Summary statistics for the datasets used in this work (AQuM, WCSum, and SS-FD are shorthands for AQuaMuse, WikiCatSum and ScreenSumm-FD, respectively). We report on the number of queries, size of training, development, and test set, and average source and target length (in terms of documents, words, sentences, and words per document). We quantify the abstractiveness of the target by measuring the proportion of n-grams unseen in the source. We also report statistics on the target length augmented with the blueprint (number of QA pairs and words in total). documents or one, web pages, or dialogue transcripts), the user's information need (e.g., answering a question or aggregating information), and summary style (e.g., genuinely abstractive vs extractive). Common features among them are very long inputs and multi-sentence output summaries. We summarize various dataset statistics in Table 3.
Conditional Generation with a Question-Answering Blueprint
AQuaMuSe (Kulkarni et al., 2020(Kulkarni et al., , 2021) is a query-focused multi-document summarization dataset; it was created with the intent of simulating how a search engine might synthesize documents of high relevance to a user query. It consists of Google Natural Questions (Kwiatkowski et al., 2019) paired with web documents extracted from Common Crawl and long-form answers from Wikipedia. We approach this task as a generative QA problem where we take the query and associated web documents and generate a long-form answer to the query. We work on the split from Kulkarni et al. (2021); on average, each instance has 6.46 web documents (2,008 tokens per docu-ment), leading to very long input (12,987 tokens).  (Shaham et al., 2022) which incorporates episodes from 88 different shows. SummScreen-FD is a challenging testbed for several reasons. Plot details are often expressed indirectly in conversations between characters and are scattered across the entire transcript. The summarization task is highly compressive, a transcript the size of a book (on average 8,000 tokens; see Table 3) is condensed into a few sentences, and the evaluation of such summaries comes with its own challenges (e.g., it is not realistic to expect humans to read the transcript to be able to assess their quality). We further analyze the characteristics of these datasets in Table 3. Long-form answers in AQua-MuSe are mostly extractive with only 2%, 13%, 24%, and 31% novel unigrams, bigrams, trigrams, and 4-grams, respectively. In comparison, summaries in WikiCatSum and SummScreen-FD are more abstractive; WikiCatSum abstracts have 13% novel unigrams, 54% bigrams, 78% trigrams, and 86% 4-grams, whereas in SummScreen-FD summaries 17% unigrams, 66% bigrams, 92% trigrams, and 98% 4-grams were not seen in the training. Interestingly, SummScreen-FD summaries have far more propositions than AQuaMuSe or Wi-kiCatSum targets, leading to a much higher number for QA pairs in their blueprints (28.10 vs 8.16 or 9.56). This in turn makes the generation task for end-to-end models very challenging. The average summary length together with the blueprint annotations (i.e., b; s) for SummScreen-FD is almost twice the size of WikiCatSum and AQua-MuSe (597.90 vs 291.28 and 272.68). The majority of questions in AQuaMuSe and WikiCatSum are what questions (76.0% and 74.2%, respectively), followed by who, where, when, and how questions. For SummScreen-FD, what and who questions are most popular (50.1% and 42.9%, respectively).
Conditional Generation with a Question-Answering Blueprint
All our experiments used LONGT5 (Guo et al., 2021), an extension of the original T5 encoder (Raffel et al., 2020) with global-local attention sparsity patterns to handle long inputs. We compared a vanilla LONGT5 4 model (xl, 3B parameters) fine-tuned on our datasets (with a maximum input sequence length of 4,096 tokens and a maximum output length of 512 tokens) against several blueprint variants. These include an end-toend LONGT5 model (E2E) which first decodes blueprint b and then continues to decode output s; a LONGT5 multitask model (MULTITASK) which jointly learns to predict the answer plan followed by either the output s or the questions in b; and a LONGT5 iterative model (ITERATIVE) which plans and generates one sentence at a time.
Conditional Generation with a Question-Answering Blueprint
In addition, we implemented a two-stage model (2-STAGE) which first creates blueprint b given input d and then generates output s given b and d as input. Finally, we also fine-tuned T5 (xl, 3B parameters) on our datasets with a maximum input sequence length of 1,024 tokens and a maximum output length of 256 tokens, as a baseline. We present these comparisons in Table 4 together with the performance of various state-of-the-art systems.
Conditional Generation with a Question-Answering Blueprint
We finetuned all our models with a leaning rate of 0.001 and a batch size of 128, for 50K steps. We select best checkpoints using average Rouge performance on validation sets. During inference, we use beam search with size 5 and alpha 0.8.
Conditional Generation with a Question-Answering Blueprint
In this section we present experimental results using automatic evaluation metrics which assess overall summary (and blueprint) quality. Moreover, we quantify the extent to which automati-cally generated output is grounded to the blueprint and faithful to the input document/s.
Conditional Generation with a Question-Answering Blueprint
Summary and Blueprint Quality We evaluate summary quality automatically using (summarylevel) Rouge F1 (Lin and Hovy, 2003). We report only RougeLSum 5 in Table 4 for the sake of brevity. We also use RougeLSum to evaluate the quality of the automatically generated blueprint, i.e., the QA pairs and their order against the reference blueprint.
Conditional Generation with a Question-Answering Blueprint
Informativeness and Grounding We evaluate informativeness using QA-based metrics. Specifically, following the reading comprehension literature (Rajpurkar et al., 2016(Rajpurkar et al., , 2018a, we quantify the extent to which the generated text can answer all questions from its reference (Informativeness) and predicted blueprint (Grounding). Following Stelmakh et al. (2022), we use a RoBERTa model  finetuned on SQuAD-V2 for question-answering in both cases. 6 Given generated text s ′ and question-answer pair (q i , a i ) from the (reference or predicted) blueprint, we apply our question-answering model to s ′ to predict answer a ′ i to question q i . We then compute the tokenlevel F1 score between predicted answer a ′ i and ground truth answer a i , and report the average.
Conditional Generation with a Question-Answering Blueprint
Hallucinations are a widely known issue with neural abstractive summarization (Song et al., 2018;Kryscinski et al., 2020;Gabriel et al., 2021), especially when a sentence combines content from multiple sources (Lebanoff et al., 2019).
Conditional Generation with a Question-Answering Blueprint
Following previous work Falke et al., 2019;Narayan et al., 2022;Honovich et al., 2022;Dušek and Kasner, 2020), we quantify the extent to which generated summaries are faithful to their input using textual entailment. We resort to textual entailment for two reasons; firstly, it is a relatively intuitive metric, all information in a summary should be entailed by the source or at least not conflict with it; secondly recent studies Fischer et al., 2022), have shown that it correlates with human judgments of faithfulness across summarization datasets and tasks.
Conditional Generation with a Question-Answering Blueprint
Following Honovich et al. (2022), we trained an entailment model by fine-tuning T5-11B (Raffel et al., 2020) on the Adversarial NLI dataset (ANLI; Nie et al. 2020). For each sentence (hypothesis) in the summary, we compute its entailment probability given the input (premise) and report the average across all sentences to obtain an overall score .
Conditional Generation with a Question-Answering Blueprint
More formally, let E denote a textual entailment model that predicts E(a, b), namely that text b is entailed by text a. The faithfulness score F of summary s containing sentences s 1 , . . . , s 2 with respect to input D is computed as:
Conditional Generation with a Question-Answering Blueprint
where n is the number of sentences in the summary. If the input is longer than the T5 maximum encode length, we split it, calculate the entailment probability per split, and take the maximum. We convert probabilities to binary labels using a threshold (1 if > 0.5, and 0, otherwise). We further validated our ANLI entailment scores against human judgements of faithfulness elicited as part of SummEval (Fabbri et al., 2021), a recently released dataset for assessing automated summarization metrics. Our entailment predictions correlate well with human ratings, achieving a Spearman's rank correlation of ρ = 0.774.
Conditional Generation with a Question-Answering Blueprint
Why LONGT5 for Blueprint Models All the tasks we are dealing with require modeling input of highly complex nature which is often very long (see Table 3). Our results in Table 4 (see Rouge/summary column) demonstrate that T5 models always fall behind LONGT5, underscoring the importance of sparse attention mechanisms for modeling long inputs. In fact, LONGT5 sets a new state of the art on AQuaMuSe and SummScreen-FD. On WikiCatSum, it is slightly worse than RE-FLECT (Song et al., 2022), an extract-then-abstract model which has a dedicated content selection module. Similar content selection techniques could also benefit LONGT5, however, we leave this to future work. We henceforth use LONGT5 as a base model for finetuning our blueprint models.   (2022). Best results for each task are boldfaced. Scores that are not significantly different (using paired bootstrap resampling; p < 0.05) from the best score in each column are marked with a dagger ( †).
Conditional Generation with a Question-Answering Blueprint
Blueprint Models and Rouge Compared to LONGT5, blueprint variants slightly underperform on AQuaMuse, but score better on Wiki-CatSum and SummScreen-FD (see MULTITASK model). All differences between LONGT5and blueprint models are statistically significant using paired bootstrap resampling; p < 0.05). For a fair comparison, we always use a maximum decoder length of 512 tokens. With the exception of AQua-Muse, E2E is inferior to other blueprint models, which is not surprising since it has to generate much longer text (recall it predicts b; s rather than simply s). Overall, MULTITASK is significantly better than other blueprint models on WikiCatSum but on par with ITERATIVE on SummScreen-FD.
Conditional Generation with a Question-Answering Blueprint
Similar patterns emerge when evaluating the predicted blueprints against reference QA pairs, with ITERATIVE significantly outperforming the other two variants on SummScreen-FD. This could be due to the fact that SummScreen-FD summaries have far more propositions than AQua-MuSe or WikiCatSum targets; it is better to predict them one sentence at a time, rather than all together. With regard to WikiCatSum, the difference between MULTITASK and ITERATIVE is not significant (although MULTITASK has a slight numerical advantage) and both systems are significantly better than E2E. On AQuaMuSe MULTITASK is significantly better than E2E and ITERATIVE.
Conditional Generation with a Question-Answering Blueprint
Note that all 2-STAGE models are significantly worse in comparison to blueprint variants, when evaluating either their blueprints or summaries (in terms of Rouge). While our models learn to optimize blueprints and summaries together, 2-STAGE models are faced with the harder task of predicting the blueprint solely based on the input (text-to-data). Since the blueprints learned by the first stage are of poor quality, the summaries generated in the second stage are also inferior.
Conditional Generation with a Question-Answering Blueprint
Our blueprint annotation of reference summaries naturally provides a more principled alternative to Rouge. We can now use QA pairs in reference blueprints to evaluate the informativeness of predicted summaries. Results follow a pattern overall similar to Rouge, however, this approach reveals the complexity of the different generation tasks better than Rouge. While we were able to achieve reasonably high Rouge across datasets, we are far from generating informative summaries. On SummScreen-FD, in particular, we achieve a maximum Rouge score of 31.88, but are able to answer correctly only 7.59% of reference questions using the predicted summaries.
Conditional Generation with a Question-Answering Blueprint
Across datasets, LONGT5 performs on par with MULTITASK, the difference between the two models is not statistically significant, and the same is true of ITERATIVE on SummScreen.
Conditional Generation with a Question-Answering Blueprint
Reference Summary Grissom and Catherine investigate the death of a man found in a dumpster, who is found to have had a severe eating disorder. Meanwhile, Nick and Sara investigate the death of a couple at the brink of a bitter divorce, in which evidence seems to point to the couple's dog.
Conditional Generation with a Question-Answering Blueprint
Catherine Catherine, Sara and Grissom investigate the death of a man who was found dead in a garbage bin. The victim's wife, Lori Tinsley, was a poker player. The man's brother, Jesse Menyel, was also found dead. The case is complicated by the fact that the victim had a gun in his apartment. Meanwhile, Nick and Warrick investigate the murder of a woman who died in a car crash. The woman's husband, Greg Colletti, is the suspect. LONGT5 Grissom and Catherine investigate when a man is found dead in a dumpster. They soon discover a lot more went on in the kitchen than cooking. Meanwhile Nick and Sara are called to the scene of a double homicide. The victims are a husband and his wife who were both in the process of selling off their rare records. Suspicion quickly falls on the wife's ex-boyfriend, but the evidence increasingly points to the husband. E2E Grissom, Nick and Catherine investigate when a man is found dead in a dumpster. Meanwhile a couple are found dead in their home. Sara and Nick investigate a case involving a record collection. MULTITASK Grissom and Catherine investigate when a man is found dead in the garbage. Their investigation leads to Jackpot's pretzels as the killer. Meanwhile Sara and Nick are called to a double murder when a husband and his wife are found dead in their home. When the husband and wife were found dead, their extensive record collection was also missing. ITERATIVE Grissom, Catherine and David investigate when a man is found dead in a dumpster. The man had been eating at a restaurant called Aunt Jackpot's Pretzels. They discover that he ate himself to death. Meanwhile Nick and Sara look into the disappearance of a husband and wife who are found dead in their home. Also missing is a record collection that the husband had been collecting. They discover that the wife's neck was slashed in the attack. CSIs track down Missy Halter, a woman who helped them find the records. Table 5: System output and reference summary for SummScreen-FD (CSI S6.E9, "Dog Eat Dog"). Propositions which are not grounded to the input are in red. Generated questions from blueprint models are not shown due to space constraints.
Conditional Generation with a Question-Answering Blueprint
The E2E and ITERATIVE variants are significantly better than MULTITASK in generating texts grounded to their predicted blueprints (see ground. column in Table 4). This is because both models generate text conditioned on their blueprints; E2E first predicts blueprint b and then continues to generate output s using both b and the input, whereas, IT-ERATIVE plans and generates one sentence at a time as b i ; s i . This is not the case with MULTI-TASK which generates s conditioned on answer spans only. E2E performs slightly better than IT-ERATIVE on AQuaMuSe and WikiCatSum (differences are not statistically significant) but struggles on SummScreen-FD, where summaries are longer with more facts/propositions, requiring inference over long-range dependencies, and common sense reasoning. ITERATIVE seems the best option for grounded generation without sacrificing informativeness (ITERATIVE is most informative amongst blueprint models on SummScreen-FD, second best on AQuaMuSe, and third best on WikiCatSum).
Conditional Generation with a Question-Answering Blueprint
As far as faithfulness is concerned, ITERATIVE performs consistently better than E2E and MULTITASK, as well as T5 and LONGT5 models where text is generated from scratch without any planning (pairwise differences between ITERATIVE and comparison systems are all significant with the exception of E2E on AQuaMuse). On SummScreen-FD, IT-ERATIVE brings large gains on faithfulness without sacrificing informativeness (both in terms of Rouge and QA-F1). The ANLI score for ITER-ATIVE is 20.84, whereas it is below 10 for E2E and MULTITASK. E2E outperforms LONGT5 on AQuaMuSe and WikiCatSum, but gains are smaller compared to ITERATIVE.
Conditional Generation with a Question-Answering Blueprint
We show examples of system output in Table 5, highlighting propositions which are not grounded to the input in red. E2E summaries are shorter which is somewhat expected; the model has to decode both the plan and the summary and in cases where the blueprint is large (e.g., in SummScreen-FD), there is no more room to decode the summary. MULTITASK is more verbose, however, the plan (a sequence of answer spans) is less detailed and as a result the summary less accurate (Jackpot's pretzels is a restaurant, not a killer). ITERATIVE contains many details in the summary, more than the reference, which are not hallucinations. Both R2T-BART and LONGT5 are rather loose with the facts and generate multiple hallucinations.
Conditional Generation with a Question-Answering Blueprint
Blueprint Models are Controllable Our conceptualization of text plans as QA pairs brings inherent controllability to the generation process. By changing the blueprint, we can control content selection (i.e., what to say) and planning (i.e., in what order) without retraining the model or introducing additional control mechanisms. We provide an example in Table 6 where the plan predicted by the E2E model has been edited to render it more coherent and factual. As can be seen, the model is able to change its output according to the modified plan. Another example is shown in Table 7 where the output is rendered shorter by removing QA pairs from the predicted plan. Old English Bulldogs refers to a breed of dog that once existed but is no extinct. At the end of the 19th century, three breeds -the Old English Bulldog, Toy Bulldog, and Bullenbeisser -were considered extinct.
Conditional Generation with a Question-Answering Blueprint
Old English Bulldogs refers to a breed of dog that existed but is now extinct. It was bred for fighting in public arenas. Table 6: Example of plan/summary generated by our E2E blueprint model as answer to the question "What is the difference between an Old English Bulldog and an English Bulldog?" (AQua-Muse test set); user edits to the plan and updated summary are shown in red.
Conditional Generation with a Question-Answering Blueprint
We are also able to control the faithfulness of predicted summaries as follows. We take the predicted plan and remove question-answer pairs (E2E, ITERATIVE) or answer spans (MULTITASK) that cannot be answered based on the input. We then prompt our decoder with the modified plan and generate a new summary (or sentence for IT-ERATIVE). In Table 8, we quantitatively evaluate +drop variants which are controlled for faithfulness against vanilla blueprint models. We observe improvements in entailment scores across the board (see column entail. in the table), with the ITERATIVE+drop performing best. Improvements on abstractive datasets (WikiCatSum and SummScreen-FD) are larger compared to AQua-MuSe which is mostly extractive (see Table 3). The minor drop in Rouge and informativeness is somewhat expected as the models now zoom in on information they can reliably talk about, improving the consistency of the output.
Conditional Generation with a Question-Answering Blueprint
Finally, we also experiment with creating simple summaries, by forcing the ITERATIVE model to generate from a single question-answer pair on each iteration (see +Q1 variant in Table 8). In the example shown in Table 9, ITERATIVE+Q1 produces simple summary sentences, each focusing on a single information element. Interestingly, as far as the ITERATIVE model is concerned, +Q1 variants are as faithful as +drop ones even if they do not explicitly control for faithfulness (across datasets the differences between the two models are not statistically significant). This suggests that controlling for simplicity might be sufficiently to reduce hallucinations, however, at the expense of A 7 : diverse roots Q 8 : What started to develop between 500 BCE and 300 CE?
Conditional Generation with a Question-Answering Blueprint
A 8 : This Hindu synthesis Q 9 : When did the Vedic period end?
Conditional Generation with a Question-Answering Blueprint
A 9 : 500 BCE Hinduism is an Indian religion and dharma, or a way of life, widely practiced in the Indian subcontinent. Hinduism has been called the oldest religion in the world, and some practitioners and scholars refer to it as Santāna Dharma, "the eternal tradition", or the "eternal way", beyond human history. Scholars regard Hinduism as a fusion or synthesis of various Indian cultures and traditions, with diverse roots and no founder. This "Hindu synthesis" started to develop between 500 BCE and 300 CE, following the Vedic period (1500 BCE to 500 BCE).
Conditional Generation with a Question-Answering Blueprint
Hinduism is an Indian religion and dharma, or a way of life, widely practiced in the Indian subcontinent. Table 7: Example of plan/summary generated by the E2E blueprint model as answer to the question "What section of the world or country is hinduism usually found in? (AQuaMuse test set); the part of the plan which is removed by the user is highlighted in ; the shorter summary generated from the elided plan is shown in red.
Conditional Generation with a Question-Answering Blueprint
informativeness (Rouge scores for +Q1 variants tend to be significantly worse compared to +drop counterparts).
Conditional Generation with a Question-Answering Blueprint
Most of the controllability cases we illustrate here are fully automatic and could be conceptualized as system flags which users select according to requirements (e.g., low tolerance for hallucinations, shorter summaries for small screen displays). Another potential use case would be to generate summaries for a set of questions provided by the user. Their input might be articles retrieved as an answer to a query, or in an educational context several chapters on a topic (e.g., cell biology). However, we leave this to future work.
Conditional Generation with a Question-Answering Blueprint
As described in Section 3.2, we construct blueprint annotations using the Rheme-and Coverage-based selection strategies. Table 10 presents various ablations which provide rationales for these annotation choices. For the sake of brevity, we report experiments with the E2E model trained (for 50,000 steps) on AQuaMuSe. We observe very similar trends on the other two datasets. As can  be seen, it is empirically better to form blueprints from answer-question pairs rather than predicting the questions first and then their answers which is more natural (at least to humans). We further assessed whether sorting the QA pairs based on how they appear in the summary matters by defaulting to a random ordering (see −Sorted in the table). Removing either Rheme or Coverage has a small negative impact on the summaries but not their blueprints, while removing them both is detrimental to summary quality, while the absence of Sorting mostly affects the quality of the blueprint. It is not surprising that sorting is most important to generating a blueprint with correctly ordered propositions.
Conditional Generation with a Question-Answering Blueprint
In addition to automatic evaluation, we conducted three human-based studies which assessed different dimensions of output quality. Wishing to avoid well-documented issues 7 with automated bots on Amazon Mechanical Turk and crowdworkers run-7 https://stanforddaily.com/2020/06/21/    ning through HITs as quickly as possible without paying attention to the tasks, we used a few trained annotators. They were given task-specific instructions and went through several pilots to iron out disagreements on edge cases. 8
Conditional Generation with a Question-Answering Blueprint
Our first study assessed overall summary quality. Specifically, we asked our annotators to select the best among three system summaries taking into account how much they deviated from the reference in terms of informativeness (are the summaries on topic or emphasize irrelevant details?) and overall fluency. We adapted the definition of fluency provided in Howcroft et al. (2020): does the text 'flow well' or is it a sequence of unconnected parts?
Conditional Generation with a Question-Answering Blueprint
We conducted our annotation study on 100 instances, each randomly sampled from AQuaMuse, WikiCatSum, and SumScreen. We collected ratings from three annotators (after two rounds of pilot studies to improve agreement) for the output of seven systems. Overall, we obtained 100 (instances) x 3 (datasets) x 6 (systems) x 3 (annotators) = 5,400 annotations. Annotator agreement was 97.11%. Our results are presented in Table 11. We report on percentage of times each system was ranked best.
Conditional Generation with a Question-Answering Blueprint
In general, we observe that LONGT5 and blueprint models based on it are perceived as significantly better than previous state-of-the-art models (i.e., SIBERT and R2T-BART). On AQua-Muse, LONGT5 is rated overall best, followed by E2E and MULTITASK(however, differences between them are not statistically significant). On WikiCatSum, E2E is rated best bus is not significantly different compared to the other models. On SummScreen, our ITERATIVE variant is rated best followed by LONGT5. These results mirror the difficulty of the task (see Table 3), the longer the input/output, the better ITERATIVE performs.
Conditional Generation with a Question-Answering Blueprint
We further evaluated the predicted plans more directly. Participants were shown QA blueprints and asked to assess whether they tell a coherent story (are they all relevant and ordered comprehensively?) using a 3-point scale (where 3 is best and 1 is worst). They were also asked to evaluate whether the plans have redundant QA pairs; a QA  pair is redundant if it does not add new information to the plan. We collected judgments for the same instances used in our summary quality evaluation from three annotators whose overall agreement was 97.87%. Obtained a total of 100 (instances) x 3 (datasets) x 5 (systems) x 3 (raters) = 4,500 annotations. Table 12 shows the results of this study. We report mean scores per dataset for all blueprint models. As an upper bound, we further elicited annotations for blueprints automatically created from gold standard reference summaries (see row Gold in the table). E2E generates the most coherent blueprints: differences between E2E and all comparison systems are statistically significant with the exception of the gold standard. This is not surprising, since all QA pairs in E2E are generated together, whereas in MULTITASK the spans and their corresponding questions are generated separately. ITERATIVE only generates QA pairs for a sentence at a time and thus we would not expect it to be more coherent than models which generate a global document plan. With regard to redundancy, ITERATIVE blueprints are generally most redundant, which is again down to not having a global view of previously generated QA pairs. ITERA-TIVE further underscores issues with our question generation technology which is far from perfect, for example, several QA pairs are different on the surface but actually semantically equivalent, however, we have no means of detecting this without robust coreference resolution.
Conditional Generation with a Question-Answering Blueprint
We next examine whether model summaries are grounded to their blueprints. Specifically, we asked our annotators to decide whether each QA pair in the blueprint is mentioned in the summary, and report the number of times it isn't. Ideally, we would like the summary to follow the blueprint as closely as possible. For QA pairs mentioned in the summary, we further asked our annotators to highlight whether the intent of the question was preserved or contradicted (we report the number of contradictions). Finally, we also asked participants to decide whether the summary has additional information which cannot be found in its blueprint, using a 3-point scale (where 3 is for summaries with lots of new information and 1 is for summaries with no new information). We elicited annotations for blueprint models, and, as an upper bound, for gold summaries and blueprints extrapolated from them. We obtained 100 (instances) x 3 (datasets) x 5 (systems) x 3 (raters) = 4,500 judgments.
Conditional Generation with a Question-Answering Blueprint
The results of our grounding experiments are summarized in Table 13. Across datasets, we observe that ITERATIVE summaries are most grounded. ITERATIVE blueprints have the least number of questions that are absent from or contradict their generated texts. ITERATIVE summaries also display the least amount of new information in relation to their blueprints. ITERATIVE+drop is slightly less grounded compared to ITERA-TIVE, however, this is not entirely surprising since we prompt the ITERATIVE model with externally modified blueprints (see ITERATIVE+drop in Table 13). Note that ITERATIVE+drop summaries are deemed more faithful than ITERATIVE summaries in automatic evaluation. The entailment scores improve for all three datasets (see Table 4).
Conditional Generation with a Question-Answering Blueprint
In this work we proposed a novel plan-based approach to conditional generation. We conceptualized text plans as a sequence of QA pairs operating as a proxy for what to say and in what order. We developed Transformer-based models which generate by conditioning on a global QA blueprint plan (E2E, MULTITASK) or iteratively by planning and generating one sentence at a time (ITERATIVE). Experimental results across three challenging datasets demonstrate that blueprint models are inherently more informative than vanilla sequence-to-sequence approaches without a planning component. Amongst the three presented here (E2E, MULTITASK, ITERATIVE), we find that ITERATIVE is the best choice for grounded generation and a promising direction for long-form generation.
Conditional Generation with a Question-Answering Blueprint
Blueprint models offer several advantages compared to blackbox generation. Model predictions can be examined, and errors can be traced back to the blueprint which in turn can reveal whether the output is informative and faithful to its input. The formulation of the blueprint plan as question-answer pairs makes it intuitive and userfriendly. We have discussed how blueprint models might be used in a human-in-the-loop setting, where users interact with and influence model predictions directly, e.g., by editing the blueprint length and content (as different blueprints lead to different outputs). In the future, we would like to use blueprints more directly to advance methods for training language models using reward learning (Sutton and Barto, 2018), e.g., based on whether the output answers the blueprint questions. Rather than eliciting expensive human feedback (Stiennon et al., 2020), blueprints could provide a cheaper automatic alternative. Finally, although we focused primarily on the generation problem in this work, we believe blueprints might also be useful as a general-purpose approach to retrieving and organizing important content, especially when faced with many and very long inputs.  Table 13: Human evaluation results for blueprint grounded generation on AQuaMuse, WikiCatSum, and SummScreen-FD test sets. Proportion of QA pairs not mentioned in the summary (Absent; lower is better); proportion of QA pairs with information contradictory to the summary (Contra; lower is better), and mean scores for new information present in the summary (NewInfo; lower is better). The best results for each task are boldfaced. Systems in each column are marked with † when they are not statistically significant from the best system; unmarked pariwise differences from the best system are significant ( 