How have planning approaches been used to improve generation quality in neural data-to-text generation?
Planning in Encoder-Decoder Models Various
recent efforts have developed planning modules
in the context of data-to-text generation. In most
cases, the plans are specific to the input which
varies from tables and records to RDF tuples. For
instance, Puduppully et al. (2019a) learn a plan
corresponding to a sequence of records, and gen-
erate a summary conditioned on it. Narayan et al.
(2020) treat content selection as a task similar to
extractive summarization, they first extract sen-
tence plans and then verbalize them one-by-one.
Moryossef et al. (2019a,b) propose a symbolic
planning stage followed by a neural realization
stage. Other work (Puduppully and Lapata, 2021;
Puduppully et al., 2022) advocates macro plan-
ning, where document content is organized into
a sequence of paragraph plans which are verbal-
izations of tabular input. Our work is closest to
Narayan et al. (2021) who also target summariza-
tion applications and learn an intermediate plan
to guide generation. We adopt a more elaborate
plan representation based on QA blueprints, and
interface decoding with plan generation similarly
to Narayan et al. (2020).
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Data-to-text generation can be conceptually divided into two parts: ordering and structuring the information (planning), and generating fluent language describing the information (realization). Modern neural generation systems conflate these two steps into a single end-to-end differentiable system. We propose to split the generation process into a symbolic text-planning stage that is faithful to the input, followed by a neural generation stage that focuses only on realization. For training a plan-to-text generator, we present a method for matching reference texts to their corresponding text plans. For inference time, we describe a method for selecting high-quality text plans for new inputs. We implement and evaluate our approach on the WebNLG benchmark. Our results demonstrate that decoupling text planning from neural realization indeed improves the system’s reliability and adequacy while maintaining fluent output. We observe improvements both in BLEU scores and in manual evaluations. Another benefit of our approach is the ability to output diverse realizations of the same input, paving the way to explicit control over the generated text structure.
[{"first":"Amit","last":"Moryossef","middle":[]},{"first":"Yoav","last":"Goldberg","middle":[]},{"first":"Ido","last":"Dagan","middle":[]}]
Stepwise Extractive Summarization and Planning with Structured Transformers
We propose encoder-centric stepwise models for extractive summarization using structured transformers -- HiBERT and Extended Transformers. We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure. Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive content planner for different tasks. When evaluated on CNN/DailyMail extractive summarization, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filtering. This also holds true for Rotowire table-to-text generation, where our models surpass previously reported metrics for content selection, planning and ordering, highlighting the strength of stepwise modeling. Amongst the two structured transformers we test, stepwise Extended Transformers provides the best performance across both datasets and sets a new standard for these challenges.
[{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Joshua","last":"Maynez","middle":[]},{"first":"Jakub","last":"Adamek","middle":[]},{"first":"Daniele","last":"Pighin","middle":[]},{"first":"Blavz","last":"Bratanivc","middle":[]},{"first":"Ryan","last":"McDonald","middle":[]}]
Data-to-Text Generation with Content Selection and Planning
Recent advances in data-to-text generation have led to the use of large-scale datasets and neural network models which are trained end-to-end, without explicitly modeling what to say and in what order. In this work, we present a neural network architecture which incorporates content selection and planning without sacrificing end-to-end training. We decompose the generation task into two stages. Given a corpus of data records (paired with descriptive documents), we first generate a content plan highlighting which information should be mentioned and in which order and then generate the document while taking the content plan into account. Automatic and human-based evaluation experiments show that our model outperforms strong baselines improving the state-of-the-art on the recently released RotoWire dataset.
[{"first":"Ratish","last":"Puduppully","middle":[]},{"first":"Li","last":"Dong","middle":[]},{"first":"Mirella","last":"Lapata","middle":[]}]
Data-to-text Generation with Macro Planning
Recent approaches to data-to-text generation have adopted the very successful encoder-decoder architecture or variants thereof. These models generate text which is fluent (but often imprecise) and perform quite poorly at selecting appropriate content and ordering it coherently. To overcome some of these issues, we propose a neural model with a macro planning stage followed by a generation stage reminiscent of traditional methods which embrace separate modules for planning and surface realization. Macro plans represent high level organization of important content such as entities, events and their interactions; they are learnt from data and given as input to the generator. Extensive experiments on two data-to-text benchmarks (RotoWire and MLB) show that our approach outperforms competitive baselines in terms of automatic and human evaluation.
[{"first":"Ratish","last":"Puduppully","middle":[]},{"first":"Mirella","last":"Lapata","middle":[]}]
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains -- ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of Rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.
[{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Yao","last":"Zhao","middle":[]},{"first":"Joshua","last":"Maynez","middle":[]},{"first":"Gonccalo","last":"Simoes","middle":[]},{"first":"Vitaly","last":"Nikolaev","middle":[]},{"first":"Ryan","last":"McDonald","middle":[]}]
Data-to-text Generation with Variational Sequential Planning
We consider the task of data-to-text generation, which aims to create textual output from non-linguistic input. We focus on generating long-form text, i.e., documents with multiple paragraphs, and propose a neural model enhanced with a planning component responsible for organizing high-level information in a coherent and meaningful way. We infer latent plans sequentially with a structured variational model, while interleaving the steps of planning and generation. Text is generated by conditioning on previous variational decisions and previously generated text. Experiments on two data-to-text benchmarks (RotoWire and MLB) show that our model outperforms strong baselines and is sample efficient in the face of limited training data (e.g., a few hundred instances).
[{"first":"Ratish","last":"Puduppully","middle":[]},{"first":"Yao","last":"Fu","middle":[]},{"first":"Mirella","last":"Lapata","middle":[]}]
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
1. John, who was born in London, works for IBM.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Other outputs are also possible: 2. John, who works for IBM, was born in London.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
3. London is the birthplace of John, who works for IBM.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
These variations result from different ways of structuring the information: choosing which fact to mention first, and in which direction to express each fact. Another choice is to split the text into two different sentences, e.g.,
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Overall, the choice of fact ordering, entity ordering, and sentence splits for these facts give rise to 12 different structures, each of them putting the focus on somewhat different aspect of the information. Realistic inputs include more than two facts, greatly increasing the number of possibilities.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Another axis of variation is in how to verbalize the information for a given structure. For example, (2) can also be verbalized as 2a. John works for IBM and was born in London.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
5a. John is employed by IBM. He was born in London.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We refer to the first set of choices (how to structure the information) as text planning and to the second (how to verbalize a plan) as plan realization. 1 The distinction between planning and realization is at the core of classic natural language generation (NLG) works (Reiter and Dale, 2000;Gatt and Krahmer, 2017). However, a recent wave of neural NLG systems ignores this distinction and treat the problem as a single end-to-end task of learning to map facts from the input to the output text (Gardent et al., 2017;Dušek et al., 2018). These neural systems encode the input facts into an intermediary vector-based representation, which is then decoded into text. While not stated in these terms, the neural system designers hope for the network to take care of both the planning and realization aspect of text generation. A notable exception is the work of Puduppully et al. (2018), who introduce a neural content-planning module in the end-to-end architecture.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
While the neural methods achieve impressive levels of output fluency, they also struggle to maintain coherency on longer texts (Wiseman et al., 2017), struggle to produce a coherent order of facts, and are often not faithful to the input facts, either omitting, repeating, hallucinating or changing facts (the NLG community refers to such errors as errors in adequacy or correctness of the generated text). When compared to templatebased methods, the neural systems win in fluency but fall short regarding content selection and faithfulness to the input (Puzikov and Gurevych, 2018). Also, they do not allow control over the output's structure. We speculate that this is due to demanding too much of the network: while the neural system excels at capturing the language details required for fluent realization, they are less well equipped to deal with the higher levels text structuring in a consistent and verifiable manner.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Proposal we propose an explicit, symbolic, text planning stage, whose output is fed into a neural generation system. The text planner determines the information structure and expresses it unambiguously-in our case as a sequence of ordered trees. This stage is performed symbolically and is guaranteed to remain faithful and complete with regards to the input facts. Once the plan is determined, 2 a neural generation system is used to transform it into fluent, natural language text. By being able to follow the plan structure closely, the network is alleviated from the need to determine higher-level structural decisions and can track what was already covered more easily. This allows the network to perform the task it excels in, producing fluent, natural language outputs.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We demonstrate our approach on the WebNLG corpus and show it results in outputs which are as fluent as neural systems, but more faithful to the input facts. The method also allows explicit control of the output structure and the generation of diverse outputs (some diversity examples are available in the Appendix). We release our code and the corpus extended with matching plans in https://github.com/ AmitMY/chimera.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Task Description Our method is concerned with the task of generating texts from inputs in the form of RDF sets. Each input can be considered as a graph, where the entities are nodes, and the RDF relations are directed labeled edges. Each input is paired with one or more reference texts describing these triplets. The reference can be either a single sentence or a sequence of sentences. Formally, each input G consists of a set of triplets of the form (s i , r i , o i ), where s i , o i ∈ V ("subject" and "object") correspond to entities from DBPedia, and r i ∈ R is a labeled DBPedia relation (V and R are the sets of entities and relations, respectively). For example, Figure 1a shows a triplet set G and Figure 1d shows a reference text. We consider the data set as a set of input-output pairs (G, ref), where the same G may appear in several pairs, each time with a different reference.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We split the generation process into two parts: text planning and sentence realization. Given an input G, we first generate a text plan plan(G) specifying the division of facts to sentences, the order in which the facts are expressed in each sentence, and the ordering of the sentences. This data-to-plan step is non-neural (Section 3). Then, we generate each sentence according to the plan. This plan-to-sentence step is achieved through an NMT system (Section 4). Figure 1 demonstrates the entire process. To facilitate our plan-based architecture, we devise a method to annotate (G, ref ) pairs with the corresponding plans (Section 3.1), and use it to construct a dataset which is used to train the planto-text translation. The same dataset is also used to devise a plan selection method (Section 3.2).
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
General Applicability It is worth considering the dataset-specific vs. general applicability aspects of our method. On the low-level details, this  work is very much dataset dependent. We show how to represent plans for specific datasets, and, importantly for this work, how to automatically construct plans for this dataset given inputs and expected natural language outputs. The method of plan construction will likely not generalize "as is" to other datasets, and the plan structure itself may also be found to be lacking for more demanding generation tasks. However, on a higher level, our proposal is very general: intermediary plan structures can be helpful, and one should consider ways of obtaining them, and of using them. In the short term, this will likely take the form of ad-hoc explorations of plan structures for specific tasks, as we do here, to establish their utility. In the longer term, research may evolve to looking into how general-purpose plan are structured. Our main message is that the separation of planning from realization, even in the context of neural generation, is a useful one to be considered.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Plan structure Our text plans capture the division of facts to sentences and the ordering of the sentences. Additionally, for each sentence, the plan captures (1) the ordering of facts within the sentence; (2) The ordering of entities within a fact, which we call the direction of the relation. For example, the {A, location, B} relation can be expressed as either A is located in B or B is the location of A; (3) the structure between facts that share an entity, namely chains and sibling struc-tures as described below.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
London England  A text plan is modeled as a sequence of sentence plans, to be realized in order. Each sentence plan is modeled as an ordered tree, specifying the structure in which the information should be realized. Structuring each sentence as a tree enables a clear succession between different facts through shared entities. Our text-plan design assumes that each entity is mentioned only once in a sentence, which holds in the WebNLG corpus. The ordering of the entities and relations within a sentence is determined by a pre-order traversal of the tree. Figure 1b shows an example of a text plan. Formally, given the input G, a text plan T is a sequences of sentence plans T = s 1 , ..., s N T . A sen-tence plan s is a labeled, ordered tree, with arcs of the form (h, , m), where h, m ∈ V are head and modifier nodes, each corresponding to an input entity, and = (r, d) is the relation between nodes, where r ∈ R is the RDF relation, and d ∈ {→, ←} denotes the direction in which the relation is ex-
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Chains (h, 1 , m), (m, 2 , x) represent a succession of facts that share a middle entity ( Figure  2a), while siblings -nodes with the same parent -(h, 1 , m 1 ), (h, 2 , m 2 ) represents a succession of facts about the same entity ( Figure 2b). Sibling and chain structures can be combined ( Figure  2c). An example of an input we addressed in the WebNLG corpus, and matching text plan is given in Figure 1b.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Exhaustive generation For small-ish input graphs G-such as those in the WebNLG task we consider here-it is trivial to generate all possible plans by first considering all the ways of grouping the input into sets, then from each set generating all possible trees by arranging it as an undirected graph and performing several DFS traversals starting from each node, where each DFS traversal follows a different order of children. 
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
While the input RDFs and references are present in the training dataset, the plans are not. We devise a method to recover the latent plans for most of the input-reference pairs in the training set, constructing a new dataset of (G, ref, T ) triplets of inputs, reference texts, and corresponding plans.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We define the reference ref , and the text-plan T to be consistent with each other iff (a) they exhibit the same splitting into sentences-the facts in every sentence in ref are grouped as a sentence plan in T , and (b) for each corresponding sentence and sentence-plan, the order of the entities is identical.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The matching of plans to references is based on the observations that (a) it is relatively easy to identify entities in the reference texts, and a pair of entities in an input is unique to a fact; (b) it is relatively easy to identify sentence splits; (c) a reference text and its matching plan must share the same entities in the same order, and with the same sentence splits. Sentence split consistency We define a set of triplets to be potentially consistent with a sentence iff each triplet contains at least one entity from the sentence (either its subject or object appear in the sentence), and each entity in the sentence is covered by at least one triplet. Given a reference text, we split it into sentences using NLTK (Bird and Loper, 2004), and look for divisions of G into disjoint sets such that each set is consistent with a corresponding sentence. For each such division, we consider the exhaustive set of all induced plans. Facts order consistency A natural criterion would be to consider a reference sentence and a sentence-plan originating from the corresponding RDF as matching iff the sets of entities in the sentence and the plan are identical, and all entities appear in the same order. 4 Based on this, we could represent each sentence and each plan as a sequence of entities, and verify the sequences match. However, using this criterion is complicated by the fact that it is not trivial to map between the entities in the plan (that originate from the RDF triplets) and the entities in the text. In particular, due to language variability, the same plan entity may appear in several forms in the textual sentences. Some of these variations (i.e. "A.F.C Fylde" vs. "AFC Fylde") can be recognized heuristically, while others require external knowledge ("UK conservative party" vs. "the Tories"), and some are ambiguous and require full-fledged co-reference resolution ("them", "he", "the former"). Hence, we relax our matching criterion to allow for possible unrecognized entities in the text.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Concretely, we represent each sentence plan as a sequence of its entities (pe 1 , ..., pe k ), and each sentence as the sequence of its entities which we managed to recognize and to match with an input entity (se 1 , ..., se m ), m ≤ k. 5 We then consider a sentence and a sentenceplan to be consistent if the following two condi-4 An additional constraint is that no two triplets in the RDFs set share the same entities. This is to ensure that if two entities appeared in a structure, only one relation could have been expressed there. This almost always holds in the WebNLG corpus, failing on only 15 out of 6,940 input sets. 5 We match plan entities to sentence entities using greedy string matching with Levenshtein distance (Levenshtein, 1966) for each token and a manually tuned threshold for a match. While this approach results in occasional false positives, most cases are detected correctly. We match dates by using the chrono-python package that parses dates from natural language texts. tions hold: (1) The sentence entities (se 1 , ..., se m ) are a proper sub-sequence of the plan entities (pe 1 , ..., pe k ); and (2) each of the remaining entities in the plan already appeared previously in the plan. The second condition accounts for the fact that most un-identified entities are due to pronouns and similar non-lexicalized referring expressions, and that these only appear after a previous occurrence of the same entity in the text. 
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
To select the plan to be realized, we propose a mechanism for ranking the possible plans. Our plan scoring method is a product-of-experts model, where each expert is a conditional probability estimate for some property of the plan. The conditional probabilities are MLE estimates based on the plans in the training set constructed in section 3.1. Estimates involving relation names are smoothed using Lidstone smoothing to account for unseen relations. We use the following experts: Relation direction For every relation r ∈ R, we compute its probability to be expressed in the plan in its original order (d =→) or in the reverse order (d =←): p dir (d =→ |R). This captures the tendency of certain relations to be realized in the reversed order to how they are defined in the knowledge base. For example, in the WebNLG corpus the relation "manager" is expressed as a variation of "is managed by" instead of one of "is the manager of" in 68% of its occurrences (p dir (d =← |manager) = 0.68). Global direction We find that while the probability of each relation to be realized in a reversed order is usually below 0.5, still in most plans of longer texts there are one or two relations that appear in the reversed order. We capture this tendency using an expert that considers the conditional probability p gd (nr = n| |G|) of observing n reversed edges in an input with |G| triplets. Splitting tendencies For each input size, we keep track of the possible ways in which the set of facts can be split to subsets of particular sizes. That is, we keep track of probabilities such as p s (s = [3, 2, 2] | 7) of realizing an input of 7 RDF triplets as three sentences, each realizing the corresponding number of facts. Relation transitions We consider each sentence plan as a sequence of the relation types expressed in it r 1 , . . . , r k followed by an EOS symbol, and compute the markov transition probabilities over this sequence: p trans (r 1 , r 2 , . . . , r k , EOS) = ∏ i=1,k p t (r i+1 |r i ). The expert is the product of the transition probabilities of the individual sentence plans in the text plan. This captures the tendencies of relations to follow each other and in particular, the tendencies of related relations such as birth-place and birth-date to group, allowing their aggregation in the generated text (John was born in London on Dec 12th, 1980).
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Each of the possible plans are then scored based on the product of the above quantities. The scores work well for separating good from lousy text plans, and we observe a threshold above which most generated plans result in adequate texts. We demonstrate in Section 6 that realizing highly-ranked plans manages to obtain good automatic realization scores. We note that the plan in Figure 1b is the one our ranking algorithm ranked first for the input in Figure 1a.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Possible Alternatives In addition to the single plan selection, the explicit planning stage opens up additional possibilities. Instead of choosing and realizing a single plan, we can realize a diverse set of high-scoring plans, or realizing a random high-scoring plan, resulting in a diverse and less templatic set of texts across runs. This relies on the combination of two factors: the ability of the scoring component to select plans that correspond to plausible human-authored texts, and the ability of the neural realizer to faithfully realize the plan into fluent text. While it is challenging to directly evaluate the plans adequacy, we later show an evaluation of the plan realization component. Figure 3 shows three random plans for the same graph and their realizations. Further examples of the diversity of generation are given in the appendix.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The explicit and symbolic planning stage also allows for user control over the generated text, either by supplying constraints on the possible plans (e.g., number of sentences, entities to focus on, the order of entities/relations, or others) or by supplying complete plans. We leave these options for future work. 7 We note that for an input of n triplets, there are O(2 2n + n * n!) possible plans, making this method prohibitive for even moderately sized input graphs. However, it is sufficient for the WebNLG dataset in which n ≤ 7. For larger graphs, better plan scoring and more efficient search algorithms should be devised. We leave this for future work.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The Dessert Bionico requires Granola as one of its ingredients and originates from the Jalisco region of Mexico .
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Bionico is a food found in the Mexico region Jalisco. The Dessert Bionico requires Granola as an ingredient.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Bionico contains Granola and is served as a Dessert. Bionico is a food found in the region of Jalisco, Mexico Figure 3: Three random linearized plans for the same input graph, and their text realizations. All taken from the top 10% scoring plans. (a) structures the output as a single sentence, while (b) and (c) as two sentences. The second sentence in (b) puts emphasis on Bionico being a dessert, while in (c) the emphasis is on the ingredients.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
For plan realization, we use an off-the-shelf vanilla neural machine translation (NMT) system to translate plans to texts. The explicit division to sentences in the text plan allows us to realize each sentence plan individually which allows the realizer to follow the plan structure within each (rather short) sentence, reducing the amount of information that the model needs to remember. As a result, we expect a significant reduction in over-and under-generation of facts, which are common when generating longer texts. Currently, this comes at the expense of not modeling discourse structure (i.e., referring expressions). This deficiency may be handled by integrating the discourse into the text plan, or as a post-processing step. 8 . We leave this for future work.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
To use text plans as inputs to the NMT, we linearize each sentence plan by performing a preorder traversal of the tree, while indicating the tree structure with brackets ( Figure 1c). The directed relations (r, d) are expressed as a sequence of two or more tokens, the first indicating the direction and the rest expressing the relation. 9 Entities that are identified in the reference text are replaced with single, entity-unique tokens. This allows the NMT system to copy such entities from the input rather than generating them. Figure 1d is an example of possible text resulting from such linearization.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Training details We use a standard NMT setup with a copy-attention mechanism (Gulcehre et al., 2016) 10 and the pre-trained GloVe.6B word em-8 Minimally, each entity occurrence can keep track of the number of times it was already mentioned in the plan. Other alternatives include using a full-fledged referring expression generation system such as NeuralREG (Ferreira et al., 2018) 9 We map DBPedia relations to sequences of tokens by splitting on underscores and CamelCase. 10 Concretely, we use the OpenNMT toolkit (Klein et al., 2017) with the copy attn flag. Exact parameter values are beddings 11 (Pennington et al., 2014). The pretrained embeddings are used to initialize the relation tokens in the plans, as well as the tokens in the reference texts.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Generation details We translate each sentence plan individually. Once the text is generated, we replace the entity tokens with the full entity string as it appears in the input graph, and lexicalize all dates as Month DAY+ordinal, YEAR (i.e., July 4th, 1776) and for numbers with units (i.e., "5"(minutes)) we remove the parenthesis and quotation marks (5 minutes).
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The WebNLG challenge (Colin et al., 2016) consists of mapping sets of RDF triplets to text including referring expression generation, aggregation, lexicalization, surface realization, and sentence segmentation. It contains sets with up to 7 triplets each along with one or more reference texts for each set. The test set is split into two parts: seen, containing inputs created for entities and relations belonging to DBpedia categories that were seen in the training data, and unseen, containing inputs extracted for entities and relations belonging to 5 unseen categories. While the unseen category is conceptually appealing, we view the seen category as the more relevant setup: generating fluent, adequate and diverse text for a mix of known relation types is enough of a challenge also without requiring the system to invent verbalizations for unknown relation types. Any realistic generation system could afford to provide at least a few verbalizations for each relation of interest. We thus focus our attention mostly on the seen case (though our system does also perform well on the unseen case). Following Section 3.1, we manage to match a detailed in the appendix. 11 nlp.stanford.edu/data/glove.6B.zip consistent plan for 76% of the reference texts and use these plan-text pairs to train the plan realization NMT component. Overall, the WebNLG training set contains 18, 102 RDF-text pairs while our plan-enhanced corpus contains 13, 828 plantext pairs. 12 Compared Systems We compare to the best submissions in the WebNLG challenge (Gardent et al., 2017): Melbourne, an end-to-end system that scored best on all categories in the automatic evaluation, and UPF-FORGe (Mille et al., 2017), a classic grammar-based NLG system that scored best in the human evaluation. Additionally, we developed an end-to-end neural baseline which outperforms the WebNLG neural systems. It uses a set encoder, an LSTM (Hochreiter and Schmidhuber, 1997) decoder with attention (Bahdanau et al., 2014), a copy-attention mechanism (Gulcehre et al., 2016) and a neural checklist model (Kiddon et al., 2016), as well as applying entity dropout. The entity-dropout and checklist component are the key differentiators from previous systems. We refer to this system as StrongNeural.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
6 Experiments and Results
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We begin by comparing our plan-based system (BestPlan) to the state-of-the-art using the common automatic metrics: BLEU (Papineni et al., 2002), Meteor (Banerjee and Lavie, 2005), ROUGE L (Lin, 2004) and CIDEr (Vedantam et al., 2015), using the nlg-eval 13 tool (Sharma et al., 2017) on the entire test set and on each part separately (seen and unseen).
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
In the original challenge, the best performing system in automatic metric was based on end-toend NMT (Melbourne). Both the StrongNeural and BestPlan systems outperform all the WebNLG participating systems on all automatic metrics (Table 1). BestPlan is competitive with StrongNeural in all metrics, with small differences either way per metric. 14 12 Note that this only affects the training stage. At test time, we do not require gold plans, and evaluate on all sentences. 13 https://github.com/Maluuba/nlg-eval 14 At least part of the stronger results for StrongNeural can be attributed to its ability to generate referring expressions, which we currently do not support. 
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Next, we turn to manually evaluate our system's performance regarding faithfulness to the input on the one hand and fluency on the other. We describe here the main points of the manual evaluation setup, with finer details in the appendix.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Faithfulness As explained in Section 3, the first benefit we expect of our plan-based architecture is to make the neural systems task simpler, helping it to remain faithful to the semantics expressed in the plan which in turn is guaranteed to be faithful to the original RDF input (by faithfulness, we mean expressing all facts in the graph and only facts from the graph: not dropping, repeating or hallucinating facts). We conduct a manual evaluation over the seen portion of the WebNLG human evaluated test set (139 input sets). We compare Best-Plan and StrongNeural. 15 For each output text, we manually mark which relations are expressed in it, which are omitted, and which relations exist with the wrong lexicalization. We also count the number of relations the system over generated, either repeating facts or inventing new facts.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
16 Table 2 shows the results. BestPlan reduces all error types compared to StrongNeural, by 85%, 56% and 90% respectively. While on-par regarding automatic metrics, BestPlan substantially outperforms the new state-of-the-art end-to-end neural system in semantic faithfulness.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
For example, Figure 4 compares the output of He was selected by nasa in 1963 and became a crew member on the Apollo 8 flight mission.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
He retired on September 1st, 1969.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
William Anders was a fighter pilot who joined nasa in 1963 and served as a crew member of Apollo 8.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
William Anders retired on September 1st, 1969 and spent 8820.0 minutes in space.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
William Anders was born in British Hong Kong on october October 17th, 1933.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
(c) Output from BestPlan Figure 4: Comparing end-to-end neural generation with our plan based system.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
BestPlan (4c) on the last input in the seen test set (4b). While both systems chose three sentences split and aggregated details about birth in one sentence and details about the occupation in another, StrongNeural also expressed the information in chronological order. However, StrongNeural failed to generate facts 3 and 5. BestPlan made a lexicalization mistake in the third sentence by expressing "October" before the actual date, which is probably caused by faulty entity matching for one of the references, and (by design) did not generate any referring expression, which we leave for future work. Expressed  417  360  Omitted  6  41  Wrong-lexicalization 17 39 Over-generation 3 29 Table 2: Semantic faithfulness of each system regarding 440 RDF triplets from 139 input sets in the seen part of the manually evaluated test set.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Fluency Next, we assess whether our systems succeed at maintaining the high-quality fluency of the neural systems. We perform pairwise evaluation via Amazon Mechanical Turk wherein each task the worker is presented with an RDF set (both in a graph form, and textually), and two texts in random order, one from BestPlan, the other from a competing system. We compare Best-Plan against a strong end-to-end neural system (StrongNeural), a grammar-based system which StrongNeural Reference UPF-FORGe BestPlan -0.6% -5.4% +5.1% Table 3: MTurk average worker score for BestPlan compared to each system. It is a worse than the reference texts, on-par with the neural end-to-end system, and a better than the previous state-of-the-art.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
is the state-of-the-art in human evaluation (UPF-FORGe), and the human-supplied WebNLG references (Reference). The workers were presented with three possible answers: BestPlan text is better (scored as 1), the other text is better (scored as -1), and both texts are equally fluent (scored as 0). Table 3 shows the average worker score given to each pair divided by the number of texts compared. BestPlan performed on-par with StrongNeural, and surpassed the previous state-of-the-art UPF-FORGe. It, however, scored worse than the reference texts, which is expected given that it does not produce referring expressions. Our approach manages to keep the same fluency level typical to end-to-end neural systems, thanks to the NMT realization component.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We test the extent to which the realizer generates texts that are consistent with the plans. For several subsets of ranked plans (best plan, top 1%, and top 10%) for the seen and unseen test sets separately, we realize up to 100 randomly selected text-plans per input. We realize each sentence plan and evaluate using two criteria: (1) Do all entities from the plan appear in the realization; (2) Like the consis-  Table 4: Surface realizer performance. Entities: Percent of sentence plans that were realized with all the requested entities. Order: of the sentences that were realized with all requested entities, percentage of realizations that followed the requested entity order. tency we defined above, do all entities appear in the same order in the plan and the realization. Table 4 indicates that for decreasingly probable plans our realizer does worse in the first criterion. However, for both parts of the test set, if the realizer managed to express all of the entities, it expressed them in the requested order, meaning the outputs are consistent with plans. This opens up a potential for user control and diverse outputs, by choosing different plans for realization.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Finally, we verify that the realization of potentially diverse plans is not only consistent with each given plan but also preserves output quality. For each input, we realize a random plan from the top 10%. We repeat this process three times with different random seeds to generate different outputs, and mark these systems as RandomPlan-1/2/3. Table 1 shows that these random plans maintain decent quality on the automatic metrics, with a limited performance drop, and the automatic score is stable across random seeds.
Stepwise Extractive Summarization and Planning with Structured Transformers
Extractive document summarization is the task of creating a summary by identifying (and subsequently concatenating) the most important sentences in a document (Erkan and Radev, 2004;Nenkova and McKeown, 2011). In recent years this task has matured significantly, mostly thanks to advances in deep neural networks. Cheng and Lapata (2016) conceptualize extractive summarization as a sequence labeling task in which first a hierarchical long short-term memory network (LSTM;
Stepwise Extractive Summarization and Planning with Structured Transformers
Hochreiter and Schmidhuber, 1997) is used to encode a document and then another LSTM is used to predict for each sentence whether it should be included in the summary. This architecture was later adopted by Nallapati et al. (2016a), Nallapati et al. (2017), Narayan et al. (2018b), Zhang et al. (2018) and Dong et al. (2018).
Stepwise Extractive Summarization and Planning with Structured Transformers
Following the success of pre-trained transformerbased architectures for many tasks (Vaswani et al., 2017;Devlin et al., 2019), the current state-of-theart approach to extractive summarization uses transformers to learn sentence representations and to rank sentences by their saliency (Liu, 2019;Liu and Lapata, 2019b;Zhang et al., 2019;Zhong et al., 2019a;Bi et al., 2020). The top scoring sentences are then assembled to produce an extract of the document. Summaries built in this fashion (Cheng and Lapata, 2016;Narayan et al., 2018a;Zhang et al., 2018;Dong et al., 2018) are prone to contain redundant information. Several recent approaches have explored mechanisms to better handle redundancy, such as heuristic-based Trigram Blocking (TriBlk; Liu and Lapata, 2019b;Wang et al., 2020), handcrafted feature-driven models (Ren et al., 2017) and redundancy aware neural sequence models (Zhou et al., 2018;Bi et al., 2020). One common problem with these models is that their focus is limited to content overlap and to respecting length budgets. However, these are but a small subset of the dimensions necessary to produce informative and coherent summaries. Ideally, models would utilize enriched document and summary representations in order to implicitly learn better extractive plans for producing summaries Mendes et al., 2019). One such method is stepwise summarization , where a summary is constructed incrementally by choosing new content conditioned on previously planned content.
Stepwise Extractive Summarization and Planning with Structured Transformers
In this paper, we propose encoder-centric stepwise models for extractive summarization using structured transformers. Structured transformers are transformer-based architectures that have the flexibility to model some form of structure of the input, e.g., hierarchical document structure. In this paper, we specifically study two such architectures -HiBERT (Zhang et al., 2019) and Extended Transformers Construction (ETC; Ainslie et al., 2020). Details of these are given in Sections 4 and 5. We enable stepwise summarization by injecting the previously planned summary content into the structured transformer as an auxiliary sub-structure. The model then can holistically learn any documentlevel coherence properties, such as saliency, redundancy, and ordering, embodied in the gold summaries. This differs from other methods which are either task specific (e.g., redundancy aware modeling in Bi et al., 2020) or not holistic (e.g., manually curated features in ). An added advantage of structured encoders is that they break the quadratic attention mechanism of transformers (Devlin et al., 2019), making them more efficient and able to process longer inputs, instead of truncating the inputs to 512 tokens (Liu and Lapata, 2019b;Bi et al., 2020), which is critical for long inputs and outputs which require non-trivial planning. When evaluated on the CNN/DailyMail summarization dataset (Hermann et al., 2015), we achieve stateof-the-art performance in terms of Rouge (Lin and Hovy, 2003) without any redundancy (Zhou et al., 2018;Bi et al., 2020) or sentence selection mechanisms (Liu and Lapata, 2019b). Our model's task-agnostic approach allows it to implicitly learn and leverage content plans directly from the data. Moreover, structured transformers form the basis of our model, which are flexible in terms of content type (e.g., text or tables) that can be modeled. We demonstrate this by learning intricate extractive content plan for the Rotowire table-to-text generation task (Wiseman et al., 2017). This task requires the generation of long summaries from large score tables detailing the the specifics of a sports match, which often necessitates dedicated content selection and planning models to generate a high-quality summary (Wiseman et al., 2017;Puduppully et al., 2019a). We show that our stepwise framework achieves higher content selection, planning and ordering scores relative to prior work with task-specific planning mechanisms.
Stepwise Extractive Summarization and Planning with Structured Transformers
The contributions of the paper are as follows: 1) this is first study to use ETC (Ainslie et al., 2020) for summarization for its ability and flexibility to better model long and structured inputs; 2) we pro-pose augmentions of two structured transformers, HiBERT and ETC, in order to enable stepwise models for extractive planning; 3) we demonstrate empirically that our models are general purpose and can be adapted as an extractive document summarizer or as a content planner for table-to-text generation; 4) Our experiments highlight the effectiveness of stepwise modeling, specifically stepwise ETC, which sets a new standard for both tasks.
Stepwise Extractive Summarization and Planning with Structured Transformers
We define a general paradigm for stepwise content extraction that can be easily tailored to both extractive summarization and table-to-text generation. Given an input D = {s 1 , s 2 , . . . , s n } with n content units, the goal is to learn an extractive content plan, i.e., S m = {s j |1 ≤ j ≤ m, s j ∈ (D∪{Ø})}, of length m; s m is an empty unit (Ø) denoting the end of the plan. We formulate this as an iterative ranking problem Bi et al., 2020) where at each k-th step (1 ≤ k ≤ m) given the input D and the previously selected plan S k−1 , we select s k ∈ (D ∪ {Ø}) with a probability p(s k |S k−1 , D; θ) with model parameters θ. The selected content is then added to S k−1 to construct S k . The best planŜ can be defined as:
Stepwise Extractive Summarization and Planning with Structured Transformers
For extractive document summarization, let D = {s 1 , s 2 , . . . , s n } be a document with n sentences. Our goal is to learn an extractive plan (or summary in this case)Ŝ which best summarizes D. For table-to-text generation, we represent a table with n records as D = {s 1 , s 2 , . . . , s n }. We aim to generate a plan S m that can be used by a text generator to generate a meaningful and coherent summary.
Stepwise Extractive Summarization and Planning with Structured Transformers
For exposition, we use the extractive document summarization setup to introduce our stepwise models with HiBERT (Zhang et al., 2019) and ETC (Ainslie et al., 2020) in the following sections. Specifically, we use 'sentence' as a content unit and 'previously' or 'partially generated summary' for a previously selected content plan.
Stepwise Extractive Summarization and Planning with Structured Transformers
Hierarchical encodings have been used to model input structure with LSTMs (Nallapati et al., 2016b;Cheng and Lapata, 2016;Narayan et al., 2018b). Zhang et al. (2019) proposed HiBERT with two stacked Transformer encoders (Vaswani et al., 2017) for extractive summarization (see the middle diagram in Figure 1): a sentence encoder that independently builds representations for each sentence in the document; and a document encoder that operates over sentence encodings to build contextual representations for all sentences. These contextual sentence representations are then ingested by a classifier to predict the salience score of each sentence in the document. As in standard transformers, both encoders have multiple layers with each layer composed of a multi-head self-attention layer followed by a feed-forward sub-layer with residual connections (He et al., 2015) and layer normalizations (Ba et al., 2016). For Stepwise HiBERT, at time step k, we modify the document encoder with the content plan S k−1 , which is the previously selected sentences in the summary. This is depicted in Figure 2 (left) and allows the model to implicitly select new sentences relative to the previously generated summary.
Stepwise Extractive Summarization and Planning with Structured Transformers
Sentence and Document Encoders. Let D = {s 1 , s 2 , . . . , s n } be a document, where s i = {w i 1 , w i 2 , . . . , w i |s i | } is a sentence in D and w i j is a token in s i . s i is first mapped to a continuous space  Stepwise Modeling. At step k, let S k−1 = {s 1 , s 2 , . . . , s k−1 } be the partial summary with (k − 1) previously extracted sentences. In addition toĤ D , our document encoder takes the summary representationĤ S k−1 = {x 1 ,x 2 , . . . ,x k−1 }, wherex i = h i 1 + p sum i . h i 1 is the representation from the sentence encoder for sentence s i and p sum i is the positional embedding for sentence s i in S k−1 . At each layer, the document encoder employs three levels of nested multi-headed attentions (Vaswani et al., 2017) to build summary-informed contextual sentence representations {d 1 , d 2 , . . . , d n }: document self-attention, summary self-attention and document-summary attention (see Figure 2, left). The first two operate in parallel, followed by the document-summary attention.
Stepwise Extractive Summarization and Planning with Structured Transformers
While document self-attention learns the contextual hidden representation h doc→doc   Roy et al., 2020). HiBERT alleviates this problem by modeling each sentence independently; the memory usage in HiBERT scales with the square of the number of sentences, and the square of the maximum length of any sentence. However, the main disadvantage of this approach is that token-level attention across sentences is prohibited and long range attention only happens indirectly at the second-stage encoder (see the middle diagram in Figure 1). Recently, Extended Transformer Construction (ETC; Ainslie et al., 2020) provides an alternative. It alleviates the quadratic memory growth by introducing sparsity to the attention mechanism via its novel global-local attention mechanism (see the rightmost diagram in Figure 1). This not only permits encoding of long inputs, 2 but also enables a mechanism to model structure directly through nodes in the global attention layer. Global-Local Attention. The ETC model architecture receives two inputs: a long input, which in most cases corresponds to the text to be encoded; and an auxiliary global input, which serves as inductive bias features. First, the model builds an attention map, called long-to-long, across the long input with a sparse local attention of fixed length, this bypasses the quadratic memory complexity and allows to scale input lengths to the thousands of tokens, but limits the attention span of tokens to their nearest neighbors.
Stepwise Extractive Summarization and Planning with Structured Transformers
To overcome this limitation, the global-local attention defines three other attention parts: globalto-global, global-to-long and long-to-global, all with unrestricted attention. This allows tokens arbitrarily far apart to attend to each other with at most one hop through the global input tokens. We refer the reader to Ainslie et al. (2020) for more details. The right parts of Figures 1 and 2 illustrate these four types of attentions and the sparsity diagrams where each cell in a row i and column j is different than white input token w i can attend to input token w j , same relative position embeddings are indicated by using the same color.
Stepwise Extractive Summarization and Planning with Structured Transformers
Stepwise Modeling. Given the document D and its partial summary S k−1 at step k, we construct an input I = D S k−1 = {w 1 , . . . , w |D S k−1 | } by concatenating the document D and the partial summary S k−1 . ETC replaces absolute position encodings with relative position encodings (Shaw et al., 2018) to easily adapt to greater input lengths than seen during pretraining. In addition to modeling relative positions in an input sequence, relative position encodings in ETC are also used to model arbitrary pairwise token relations useful for structured inputs.
Stepwise Extractive Summarization and Planning with Structured Transformers
We used the auxiliary global input to represent sentence structure. Specifically, following (Ainslie et al., 2020), we placed one auxiliary token in the global input per each sentence in the input I. We linked the global tokens with the input tokens by using relative position labels to represent whether each token belongs to that sentence. Global-toglobal attention is left unrestricted, allowing all sentences to attend to each other. This result is summary-informed contextualized input token representations via attention through the global nodes. In the rest of the paper we refer to this summarizer by Stepwise ETCSum. Similar to HiBERT, we take the first token hidden representation h i 1 as the representation for the sentence s i . Finally, sentence embeddings are passed to the softmax layer for salience scoring. Both HiBERT and ETCSum are then trained with the cross entropy loss.
Stepwise Extractive Summarization and Planning with Structured Transformers
6 Extractive Document Summarization 6.1 Experimental Setup Dataset. We evaluate our models on the CNN and DailyMail news highlights datasets (Hermann et al., 2015). We used standard splits (287,227/13,368/11,490 documents) for training, validation, and testing. We did not anonymize entities or lower case tokens as in (Narayan et al., 2018b;Zhou et al., 2018;Zhang et al., 2019;Liu and Lapata, 2019b). The documents in the CNN/DailyMail dataset are long; the average lengths are 760.5 words (34 sentences) for CNN and 653.3 words (29.3 sentences), for DailyMail. The human written abstracts have 46 and 55 words for CNN and DailyMail, respectively. We evaluated summarization quality using F 1 Rouge. 3
Stepwise Extractive Summarization and Planning with Structured Transformers
Baselines. We compared our Stepwise HiBERT and ETCSum models to Lead and Oracle baselines. Lead selects the first 3 sentences to form the summary, while Oracle baselines creates a summary by selecting the best possible set of sentences in the document that gives the highest average of Rouge-1, Rouge-2 and Rouge-L F1 scores with respect to the human written summary. The Oracle (512) truncates the input document to 512 tokens. We further compared our models against several redundancyaware models (NeuSum; Zhou et al., 2018 and ARedSum; Bi et al., 2020) and models that uses Trigram Blocking (TriBlk; Liu and Lapata, 2019b) for redundancy elimination during sentence selection (see the second block in Table 1).
Stepwise Extractive Summarization and Planning with Structured Transformers
To understand the importance of modeling long documents for extractive summarization, we also trained BERTSum, similar to Liu and Lapata (2019b), with a receptive capacity of 512 tokens initialized with the BERT checkpoint. Our BERT-Sum differs slightly from Liu and Lapata (2019b), in that we don't use segment embeddings. We also report on RoBERTa  initialized version of BERTSum (RoBERTaSum).
Stepwise Extractive Summarization and Planning with Structured Transformers
We also trained non-stepwise variants of HiB-ERT and ETCSum models (the third block in Table 1). In this setting, HiBERT and ETC do not take partial summaries as input. Instead, they simply take the input document and generate salient scores (using a sigmoid layer) for each sentence in the document; the top three sentences are then assembled to generate the summary. Our implementation of HiBERT differs from Zhang et al. (2019). For example, we don't pretrain HiBERT from scratch for document modeling as in Zhang et al. (2019). Instead, we initialize our HiBERT models with publicly available RoBERTa  checkpoints following the superior performance of 3 We lowercased candidate and reference summaries and used pyrouge with parameters "-a -c 95 -m -n 4 -w 1.2."  (Zhang et al., 2018) 41.05 18.77 37.54 Refresh (Narayan et al., 2018b) 41.00 18.80 37.70 BanditSum (Dong et al., 2018) 41.50 18.70 37.60 NeuSUM (Zhou et al., 2018) 41.59 19.01 37.98 ExConSum (Mendes et al., 2019) 41.70 18.60 37.80 JECS (Xu and Durrett, 2019) 41.70 18.50 37.90 LSTM+PN (Zhong et al., 2019b) 41.85 18.93 38.13 HER (Luo et al., 2019) 42.30 18.90 37.60 HiBERT (Zhang et al., 2019) 42.37 19.95 38.83 PNBERT (Zhong et al., 2019a) 42.69 19.60 38.85 BERTSum (Liu and Lapata, 2019b) 42.61 19.99 39.09 BERTSum+TriBlk 43.25 20.24 39.63 ARedSum-CTX (Bi et al., 2020) 43.43 20.44 39.83 HSG (Wang et al., 2020) 42 RoBERTaSum over BERTSum. We use different number of layers in the document encoder (L doc = 3) and in the sentence encoder (L sent = 9), as opposed to equal number of layers (L = 6) in both encoders of Zhang et al. (2019). The layers in the document and sentence encoders were initialized with the top and the bottom layers of RoBERTa, respectively. All ETCSum models were initialized with the uncased version of ETC pretrained checkpoints (Ainslie et al., 2020) pretrained using the standard masked language model task and the contrastive predictive coding (van den Oord et al., 2018). 4 We also report on the effect of TriBLK with all our models. We only experiment with the basesized models and therefore have 12 layers, a hidden size of 768, filter size of 3072, and 12 attention heads. For comparison, we report results from BERTSum Large (Liu and Lapata, 2019b) which uses 24 layers. Finally, we employ a beam decoding to predict summaries using our stepwise models; we use a beam size of 3 for a maximum of 4 steps. We don't allow repeated sentences, though this is not a requirement. We refer the reader to the supplementary material for implementation and reproducibility details.
Stepwise Extractive Summarization and Planning with Structured Transformers
Generating Extractive Oracles. Following Narayan et al. (2018b), we train models to predict all sentences in Oracle (Full) for non-stepwise training. Stepwise training learns to do this gradually: at each step, we train model to predict the next sentence in Oracle (Full) using the earlier predicted sentences and the document. During testing, human written abstracts are used as reference summaries to evaluate our models.
Stepwise Extractive Summarization and Planning with Structured Transformers
Long form Summarization. In our experiments, ETCSum appears to be far more superior than HiB-ERT when modeling long documents for extractive summarization; ETCSum outperformed HiBERT in all cases including stepwise or non-stepwise predictions, and, with or without trigram blocking. The downside of HiBERT where token-level attention across sentences is not possible, is not optimal for modeling documents. Both ETCSum and ETCSum+TriBlk performed better than BERTSum and BERTSum+TriBlk, respectively. These results suggest the importance of modeling the whole document with ETCSum, rather than truncating it to only 512 tokens to fit BERTSum. However, the improvement may not be attributed solely to ETC-Sum's ability to model long inputs, but also to its better initialization with ETC checkpoints (Ainslie et al., 2020), specially when the improvement diminishes when compared against RoBERTaSum. 5 Stepwise vs Non-stepwise models. First of all, trigram filtering seems to be the key in addressing redundancy in generated summaries in nonstepwise models. It helps almost all models including our HiBERT and ETCSum (except for the single case of RoBERTaSum on Rouge-2). Interestingly, we don't observe the same pattern for our stepwise models. We observe that our stepwise models (both HiBERT and ETCSum, without TriBlk) consistently improve over their nonstepwise counterparts. But when stepwise is applied with TriBlk, we don't always see improvements. We conjecture that our stepwise models themselves are inherently better at avoiding redundancy in generated summaries due to the knowledge of previously generated summary at each prediction step, and improvements with TriBlk are not always complementary. The same is also demonstrated in Figure 3;  We also report on Stepwise RoBERTaSum baselines and performance dropped compared to corresponding non-stepwise models. Perhaps without any structure in the transformer, simple summary concatenation is not a good method for Stepwise RoBERTaSum to distinguish the document from the summary. There might be better ways (than the vanilla concatenation), but with Stepwise ETCSum or HiBERT, it is very natural. Stepwise RoBER-TaSum also loses access to the end of the input as the partial summary grows for documents that are already close to 512 tokens in length.
Stepwise Extractive Summarization and Planning with Structured Transformers
Finally, our Stepwise ETCSum model without any explicit redundancy or sentence selection mechanisms, achieved comparable performance to the state of the art on the CNN/DailyMail 5 One may consider to access the modeling of long inputs in ETCSum against the truncated inputs in BERTSum and RoBERTaSum, by initializing ETCSum with BERT or RoBERTa checkpoints, and not ETC checkpoint. However, this is not fair to ETCSum as BERT or RoBERTa uses absolute position embeddings (Devlin et al., 2019), whereas, ETC uses relative position embeddings (Shaw et al., 2018).   Table 1 are computed with a confidence interval of 95%. As such, Stepwise ETCSum(+TriBlk) is significantly better than BERTSum(+TriBlk), all variants of HierBERT, ETCSum and Stepwise RoBERTaSum(+TriBlk). For other models, such as RoBERTaSum(+TriBlk) and ETCSum+TriBlk, this confidence interval is not a deciding factor, hence we performed One-way ANOVA with posthoc Tukey-HSD tests (p < 0.01). Our best model Stepwise ETCSum performs significantly better than RoBERTaSum(+TriBlk), ETC-Sum+TriBlk and Stepwise ETCSum+TriBlk, on the average of ROUGE scores.
Stepwise Extractive Summarization and Planning with Structured Transformers
7 Table-to-Text Generation Task. We further explore our model's ability to learn content plans for the Rotowire data-to-text generation task (Wiseman et al., 2017). 6 The task is to generate a summary of an NBA game from its box score (a table of 2019a) we decompose the problem into two sub-problems, which we solve independently: content planning, which consists of selecting which records in the table should be mentioned in the summary, in what order, and how they should be organized into sentences; and realization, which uses the content plan to create a human-readable summary. We refer the reader to the supplementary material for an example. Our main focus in this paper is to demonstrate our models' ability to model long and structured Rotowire input tables, and generate long meaningful content plans. For realization, we simply use a RoBERTa  initialized sequence-to-sequence transformer model (Rothe et al., 2020), trained to emit the realization sentence by sentence.
Stepwise Extractive Summarization and Planning with Structured Transformers
We train our stepwise models to take a score table and the partially generated content plan, and predict the next element in the content plan. This can be either one of the entries in the score table, a sentence break or a token marking the end of the plan. Unlike extractive summarization, here an optimal extractive content plan can have repeated entries from the input table (e.g. team names) to better preserve and generate discourse relations among sentences in the target summary (Puduppully et al., 2019b), making it a challenging task for other iterative models that prohibit redundancy, e.g., (Bi et al., 2020). For details about model implementation, realization, and the induction of oracle content plans for training, we refer the reader to the supplementary material.
Stepwise Extractive Summarization and Planning with Structured Transformers
We report typical Rotowire metrics (Wiseman et al., 2017), using the standard information extraction system described by Puduppully et al. (2019a) to extract the box score table relations mentioned in the generated (G) and in the target (T) summary. The metrics measure: text quality (BLEU score between G and T); relation generation quality (the precision of the relations extracted from G against the box score table); content selection quality (the precision and recall of the relations extracted from G against those extracted from T); and content ordering quality (the complement of the normalized Damerau-Levenshtein distance on the sequences of relations extracted from G and T). We also conducted human evaluation of Rotowire summaries.
Stepwise Extractive Summarization and Planning with Structured Transformers
Results. We focus on evaluating our Stepwise HiBERT and ETCSum models. 7 Our results are presented in Table 2. The "realized" scores assess the quality of our realized summaries and are comparable to systems in the first block in Table 2 . It's possible that a higher BLEU score could be achieved by improving our simple sentence-bysentence realization method.
Stepwise Extractive Summarization and Planning with Structured Transformers
We also report content selection scores for the output of the content planning modules (see "planning only" models in Table 2). We drop name, city and date entries from our content plans before computing the metrics in order to make them comparable with others in Table 2. We see the roundtrip of realization and subsequent information extraction decreases CS quality slightly for both models (the absolute drop of F1 score is 1.68% for Stepwise HiBERT, and 1.74% for Stepwise ETCSum).
Stepwise Extractive Summarization and Planning with Structured Transformers
Human Evaluation. Participants were shown two summaries of an NBA game and asked to compare them with respect to informativeness (Does a summary present a better selection of the rele- 7 We don't reproduce BERTSum or RoBERTaSum baselines here for two reasons: i) these sequential models are not optimal for tabular data, and ii) they are also bounded by an input length of 512 tokens, the average length of linearized score tables is 7184 tokens per game. We also don't report on our non-stepwise models as they are not suitable to generate ordered content plans as required for this task.  vant facts about the game?) and readability (Which summary has a better narrative flow and is easier to read?). We randomly selected 50 NBA tables and evaluated summaries from Baseline (Wiseman et al., 2017), Stepwise HiBERT, Stepwise ETC and Gold. The average(max;min) number of sentences were 8(8;8), 12.7(17;9), 16.7(25;10) and 12.0(20;6), for Baseline, Stepwise HiBERT, Stepwise ETC, and Gold, respectively. We also included truncated summaries from Stepwise Hi-BERT and Stepwise ETC to match the number of sentences in corresponding Gold summaries. We elicited judgements from three different annotators for each pair. We report the Best(1)-Worst(-1) Scaling scores (Louviere and Woodworth, 1991;Louviere et al., 2015). Results are presented in Table 3. Overall, Stepwise ETC summaries were ranked most informative, but they performed worst on readability. The off-the-shelf sentence-level realizer (see the supplementary material) favors the statistics-dense sentences of the baseline summaries, as it tends to hallucinate on less dense plans. Future work will aim to address this limitation. For infromativeness, Stepwise ETC summaries are significantly better than Gold, Stepwise ETC truncated and Stepwise HiBERT truncated summaries. Stepwise HiBERT summaries are significantly better than both truncated variants. All other differences are not significant (p < 0.05). For readability, baseline summaries are significantly better than both ETC variants and Stepwise HiB-ERT. All other differences are not significant.
Stepwise Extractive Summarization and Planning with Structured Transformers
The stepwise structured transformer paradigm, exemplified by HiBERT and ETCSum, can be easily adapted both to extractive document summarization or content planning for table-to-text generation.
Stepwise Extractive Summarization and Planning with Structured Transformers
Stepwise ETCSum, in particular, sets a new standard for both tasks. Future work will focus on extending our models to generate extractive plans for better abstractive summarization of long or multiple documents (Liu et al., 2018). 
Stepwise Extractive Summarization and Planning with Structured Transformers
We did a wide range of hyperparameter search for HiBERT. We experimented with the number of layers in the document encoder (1 < L doc < 12); the number of layers in the sentence encoder (1 < L sent < 12, L doc < L sent ); the initialization and sharing of position embeddings, p token j , p doc j and p sum j ; the initialization and sharing of document and sentence encoder parameters with BERT and RoBERTa checkpoints; and the representation of sentence ("first token embedding" or "average of all token embeddings") from the sentence encoder.
Stepwise Extractive Summarization and Planning with Structured Transformers
For extractive summarization, we used HiBERT with a 8 transformer layer sentence encoder, and a 4 transformer layer document encoder. The model has 133,784,833 parameters. The word position embedding in the sentence encoder is initialized using the RoBERTa checkpoint, but the document and summary sentence position embeddings are learned from scratch. The document self attention and summary self attentions are shared and initialized using the RoBERTa checkpoint, the document-summary attention is also initialized using the RoBERTa checkpoint. We truncate each document to 128 sentences and each sentence to 32 words. We trained all HiBERT models for 100k steps saving checkpoints every 1000 steps, with a batch size of 32. Following Liu and Lapata (2019b), we choose the best model based on the MLE loss on the whole validation set.
Stepwise Extractive Summarization and Planning with Structured Transformers
For Rotowire, we use HiBERT with a 2 transformer layer sentence encoder, and a 4 transformer layer document encoder. The model has 91,448,065 trainable parameters. We don't use the document sentence position embeddings for Rotowire as the input consists of a set of entries in a table. We use the summary sentence position embedding to capture the order in the content plan. We use the ROBERTA vocabulary, but as discussed in B.3 we don't use ROBERTA pretraining, instead initializing with random weights. We trained the model with a batch size of 128 until the AUC score for predicting the next content plan entry on the validation dataset flattened out, which came after 766K steps. Since the dataset has 246290 examples (one for each element in the target content plan for each Rotowire example), the model saw the entire dataset approximately 398 times.
Stepwise Extractive Summarization and Planning with Structured Transformers
For all HiBERT models, we used Cloud TPU v3 accelerators for training and the Adam optimizer with a learning rate of 0.01.
Stepwise Extractive Summarization and Planning with Structured Transformers
The ETCSum model for both extractive summarization and table-to-text generation uses a 12 layer transformer as described in (Ainslie et al., 2020). The model is pretrained with MLM and CPC objectives as described in (Ainslie et al., 2020). In total, the model has 165,825,793 trainable parameters which mostly comes from the long input of 8192 tokens and the full attention of 512 of the global tokens. We trained our model with a batch size of 512 for 5,000 steps approximately equivalent to 10 epochs.
Stepwise Extractive Summarization and Planning with Structured Transformers
We used Cloud TPU v3 accelerators for training and inference was done on a V100GPU taking 10 hours to get predictions for the test set.
Stepwise Extractive Summarization and Planning with Structured Transformers
Model selection was done over models Rouge-1 performance in the validation set for all models except stepwise models where a subset of the validation set was used instead, consisting of the first 1000 examples, given the longer inference times.
Stepwise Extractive Summarization and Planning with Structured Transformers
We did a wide range of hyperparameter search where we experimented with learning rate (0.000025, 0.00005, 0.0001), relative position encoding vocabulary size (12, 24), the representation of sentences ("first token embedding" or "average of all token embeddings") from the sentence encoder and in additionally non-stepwise models we experimented with positive label weight used to for loss calculation. Finally, we used an Adam optimizer with learning rate of 0.000025.
Stepwise Extractive Summarization and Planning with Structured Transformers
We use a ROBERTASHARE model following (Rothe et al., 2020). The model has 152,491,008 trainable parameters. We trained the model until we reached the maximum BLEU score on validation data. We trained our model with a batch size of 512 for 36K steps. Since the dataset has 45533 examples (one for each element in the target content plan in each Rotowire example), the model saw the entire dataset approximately 405 times. We used Cloud TPU v3 accelerators for training. We used the Adam optimizer with a learning rate of 0.05.
Stepwise Extractive Summarization and Planning with Structured Transformers
B Table-to-Text Generation B.1 Task   Table 4 shows a prototypical input table from the Rotowire dataset 8 , along with a possible content plan and its realization. As shown in the example, a well-formed content plan can repeat some of the entries from the input table.
Stepwise Extractive Summarization and Planning with Structured Transformers
The Rotowire dataset does not contain ground truth content plans for its summaries. Instead, we infer them following a similar approach to (Puduppully et al., 2019a), but with a few minor modifications: 1) we use just a single convolutional model, instead of an ensemble of convolutional models and LSTMs, 2) our plans maintain the within-sentence order of information, and may include repetitions if a piece of information is repeated within a sentence in the target summary, 3) our plans include sentence breaks, though we remove sentences with no table entries, 4) our content plans can include the match date, if it's mentioned in the text (e.g. "on Saturday"), 5) when we resolve a pronoun, we emit the corresponding player or team name to the content plan. With respect to Table 4, if the realization at the bottom was a reference summary, then by applying this process we would obtain the content plan shown in the middle of the table. On average, the plans inferred in this fashion have 59.24 table entries and 12.72 sentences. 8 We are not presenting an actual example for legal reasons.
Stepwise Extractive Summarization and Planning with Structured Transformers
HiBERT. Conceptually, the input to HiBERT is a sequence of strings. We use three special strings, i.e., <BEG>, <EOS>, <EOT>, to explicitly mark the beginning of the content plan, the end of a sentence, and the end of the plan (text), respectively. The other strings are the values from the table, e.g., Chicago_Bulls Points, in the same order in which they appear in the text. In practice, in an attempt to leverage ROBERTA pre-training, we replace value strings with natural language sentences that we generate from each value using the templates listed in Table 5. For numeric values, such as the number of points of a team or player, similarly to Puduppully et al. (2019c) we compute the rank of the value among the instances of the same table entry type, and include that in the templated sentence in the form of a "which is [1st, 2nd, 3rd, ..., Nth] best" suffix 9 . With respect to the example in Table 4, the value Chicago_Bulls Points would then be represented as the natural language sentence: "team points scored of Chicago Bulls is 100 which is 1st best".
Stepwise Extractive Summarization and Planning with Structured Transformers
As we did not observe a significant benefit in terms of AUC when predicting the next content plan entry on validation data, we eventually initialized our model with random weights but retained the natural language representation of the value strings.
Stepwise Extractive Summarization and Planning with Structured Transformers
Because HiBERT has a sentence limit of 512, we do a pre-filtering step by discarding the table entries that are less likely to be mentioned in the summary, i.e., all player entries valued "N/A" and as many entries valued "0" as needed. Since the table entries aren't naturally ordered we don't feed a positional embedding p sent i in the document encoder, but we still feed it for the summary encoder.
Stepwise Extractive Summarization and Planning with Structured Transformers
Given the table entries and partial summary, Hi-BERT computes a distribution over the input sentences, where <EOS> corresponds to emitting a sentence break, <EOT> corresponds to ending the content plan, and <BEG> is not used.
Stepwise Extractive Summarization and Planning with Structured Transformers
We sample content plans from a trained model by greedy decoding with one modification: entries are not allowed to repeat in the content plan, except for sentence breaks, team names and team cities. If the highest probability sentence would have been a repeat, we instead emit the second highest, etc.  ETCSum. ETC models used the same filtered set of table entries used in HiBERT as input. We concatenated these entries into a flat input sequence. Similarly, we used special strings <EOS>, <EOT> and <BEG> which correspond to the same concepts as in HiBERT, end of sentence, end of text and beginning of text respectively. These special strings are appended at the beginning of the flat input sequence.
Stepwise Extractive Summarization and Planning with Structured Transformers
The partial summary input is constructed by concatenating the special string <BEG> and the entries that have been predicted so far, in order of prediction, with <EOS> indicating sentences breaks.
Stepwise Extractive Summarization and Planning with Structured Transformers
The full input sequence is then constructed by concatenating: a [CLS] delimiter, the flat input sequence, a special separator [SEP], the partial summary and finally a separator [SEP]. Both the input sequence and the partial summary are padded to 6141 and 2048 respectively, adding up in total to 8192 strings for the full input, including the special delimiters.
Stepwise Extractive Summarization and Planning with Structured Transformers
The model uses additional inputs to construct the global-local attention. One global token is assigned to each segment in the full input, each special delimiter gets assigned a global token, as well as every sentence in the input and partial summary. The model has a maximum global token id of 512, this has to be taken into account for examples where the number of segments, input sequence sentences, special delimiters and partial inputs is larger than 512. For those examples, we don't assign global tokens to the tail of the input sequence.
Stepwise Extractive Summarization and Planning with Structured Transformers
To be consistent we use the same decoding strategy where we sample content plans greedily but without repeated entries allowed in the content plan except for sentence breaks, team names and team cities.
Stepwise Extractive Summarization and Planning with Structured Transformers
The generated content plans are realized via a sequence-to-sequence transformer model initialized with ROBERTA  following (Rothe et al., 2020), trained to emit the realization sentence by sentence. The input to the model is the concatenation of the following: 1. The text of the previous sentence, or the empty string (for the first sentence). (The model can use this to pronominalize team and player names if they were already introduced.) 2. The literal string " <BEG> " as a separator. 3. The templated realizations (cfr. Table 5) of the entries in the sentence's content plan, space separated. 4. The literal string " <CONTEXT> " as a separator. 5. The templated representation of the match date. 6. For both teams, the templated representations of a) the team name, b) the team city, c) TEAM-PTS, d) TEAM-WINS, e) TEAM-LOSSES, f) whether the team was playing at home or away. These are space separated. 7. For each player in the sentence's content plan:
Stepwise Extractive Summarization and Planning with Structured Transformers
the templated representations of a) PLAYER-START POSITION, and b) which team the player was on. These are space separated. The input after the " <CONTEXT> " separator is provided because we noticed that sometimes the content plan doesn't provide all the necessary information for realizing a sentence. For example, sometimes the target text may refer to a player team 1st quarter points of T is V TEAM-PTS QTR2 team 2nd quarter points of T is V TEAM-PTS QTR3 team 3rd quarter points of T is V TEAM-PTS QTR4 team 4th quarter points of T is V TEAM-FT PCT team free throw percentage of T is V TEAM-PTS team points scored of T is V TEAM-AST team assists of T is V TEAM-LOSSES team losses of T is V TEAM-WINS team wins of T is V TEAM-REB team rebounds of T is V TEAM-TOV team turnovers of T is V TEAM-FG3 PCT team 3-point field goal percentage of T is V TEAM-FG PCT team field goal percentage of T is V team playing at home or away? T is home/away team of match player first name player first name of P is V player second name player second name of P is V PLAYER-PTS player points scored of P is V PLAYER-FGM player field goals made of P is V PLAYER-FGA player field goals attempted of P is V PLAYER-MIN player minutes played of P is V PLAYER-FG3M player 3-point field goals made of P is V PLAYER-FG3A player 3-point field goals attempted of P is V PLAYER-STL player steals of P is V PLAYER-FTM player free throws made of P is V PLAYER-FTA player free throws attempted of P is V PLAYER-BLK player blocks of P is V PLAYER-AST player assists of P is V PLAYER-TO player turnovers of P is V PLAYER-PF player fouls of P is V PLAYER-REB player rebounds of P is V PLAYER-START POSITION player starting position of P is V PLAYER-OREB player offensive rebounds of P is V PLAYER-DREB player defensive rebounds of P is V PLAYER-FG PCT player field goals percentage of P is V PLAYER-FG3 PCT player 3-point field goals percentage of P is V PLAYER-FT PCT player free throws percentage of P is V the team a player belongs to P is player of T  by their starting position and team, which is information that wouldn't otherwise be provided to the realizer. We create training data from the rotowire summaries and their inferred content plans by splitting them into sentences together with our inferred content plans. We realize content plans by autoregressively feeding the sentence produced in the previous step as input to the next step.
Stepwise Extractive Summarization and Planning with Structured Transformers
We report performance of our best models on the Rotowire validation data in Table 6.
Data-to-Text Generation with Content Selection and Planning
Data-to-text generation broadly refers to the task of automatically producing text from non-linguistic input (Reiter and Dale 2000;Gatt and Krahmer 2018). The input may be in various forms including databases of records, spreadsheets, expert system knowledge bases, simulations of physical systems, and so on. Table 1 shows an example in the form of a database containing statistics on NBA basketball games, and a corresponding game summary.
Data-to-Text Generation with Content Selection and Planning
Traditional methods for data-to-text generation (Kukich 1983;McKeown 1992) implement a pipeline of modules including content planning (selecting specific content from some input and determining the structure of the output text), sentence planning (determining the structure and lexical content of each sentence) and surface realization (converting the sentence plan to a surface string). Recent neural generation systems (Lebret et al. 2016;Mei et al. 2016;Wiseman et al. 2017) do not explicitly model any of these stages, rather they are trained in an end-to-end fashion using the very successful encoder-decoder architecture (Bahdanau et al. 2015) as their backbone.
Data-to-Text Generation with Content Selection and Planning
Despite producing overall fluent text, neural systems have difficulty capturing long-term structure and generating documents more than a few sentences long. Wiseman et al. (2017) show that neural text generation techniques perform poorly at content selection, they struggle to maintain inter-sentential coherence, and more generally a reasonable ordering of the selected facts in the output text. Additional challenges include avoiding redundancy and being faithful to the input. Interestingly, comparisons against templatebased methods show that neural techniques do not fare well on metrics of content selection recall and factual output generation (i.e., they often hallucinate statements which are not supported by the facts in the database).
Data-to-Text Generation with Content Selection and Planning
In this paper, we address these shortcomings by explicitly modeling content selection and planning within a neural data-to-text architecture. Our model learns a content plan from the input and conditions on the content plan in order to generate the output document (see Figure 1 for an illustration). An explicit content planning mechanism has at least three advantages for multi-sentence document generation: it represents a high-level organization of the document structure allowing the decoder to concentrate on the easier tasks of sentence planning and surface realization; it makes the process of data-to-document generation more interpretable by generating an intermediate representation; and reduces redundancy in the output, since it is less likely for the content plan to contain the same information in multiple places. We train our model end-to-end using neural networks and evaluate its performance on ROTOWIRE (Wiseman et al. 2017), a recently released dataset which contains statistics of NBA basketball games paired with human-written summaries (see Table 1). Automatic and human evaluation shows that modeling content selection and planning improves generation considerably over competitive baselines.
Data-to-Text Generation with Content Selection and Planning
The Boston Celtics defeated the host Indiana Pacers 105-99 at Bankers Life Fieldhouse on Saturday. In a battle between two injury-riddled teams, the Celtics were able to prevail with a much needed road victory. The key was shooting and defense, as the Celtics outshot the Pacers from the field, from three-point range and from the free-throw line. Boston also held Indiana to 42 percent from the field and 22 percent from long distance. The Celtics also won the rebounding and assisting differentials, while tying the Pacers in turnovers. There were 10 ties and 10 lead changes, as this game went down to the final seconds. Boston (5-4) has had to deal with a gluttony of injuries, but they had the fortunate task of playing a team just as injured here. Isaiah Thomas led the team in scoring, totaling 23 points and five assists on 4-of-13 shooting. He got most of those points by going 14-of-15 from the free-throw line. Kelly Olynyk got a rare start and finished second on the team with his 16 points, six rebounds and four assists. 
Data-to-Text Generation with Content Selection and Planning
The input to our model is a table of records (see Table 1 left hand-side). Each record r j has four features including its type (r j,1 ; e.g., LOSS, CITY), entity (r j,2 ; e.g., Pacers, Miles Turner), value (r j,3 ; e.g., 11, Indiana), and whether a player is on the home-or away-team (r j,4 ; see column H/V in Table 1), represented as {r j,k } 4 k=1 . The output y is a document containing words y = y 1 · · · y |y| where |y| is the document length. Figure 2 shows the overall architecture of our model which consists of two stages: (a) content selection and planning operates on the input records of a database and produces a content plan specifying which records are to be verbalized in the document and in which order (see Table 2) and (b) text generation produces the output text given the content plan as input; at each decoding step, the generation model attends over vector representations of the records in the content plan.
Data-to-Text Generation with Content Selection and Planning
Let r = {r j } |r| j=1 denote a table of input records and y the output text. We model p(y|r) as the joint probability of text y and content plan z, given input r. We further decompose p(y, z|r) into p(z|r), a content selection and planning   Figure 3.
Data-to-Text Generation with Content Selection and Planning
phase, and p(y|r, z), a text generation phase:
Data-to-Text Generation with Content Selection and Planning
In the following we explain how the components p(z|r) and p(y|r, z) are estimated.
Data-to-Text Generation with Content Selection and Planning
The input to our model is a table of unordered records, each represented as features {r j,k } 4 k=1 . Following previous work (Yang et al. 2017;Wiseman et al. 2017), we embed features into vectors, and then use a multilayer perceptron to obtain a vector representation r j for each record: r j = ReLU(W r [r j,1 ; r j,2 ; r j,3 ; r j,4 ] + b r ) where [; ] indicates vector concatenation, W r ∈ R n×4n , b r ∈ R n are parameters, and ReLU is the rectifier activation function.
Data-to-Text Generation with Content Selection and Planning
The context of a record can be useful in determining its importance vis-a-vis other records in the table. For example, if a player scores many points, it is likely that other meaningfully related records such as field goals, three-pointers, or rebounds will be mentioned in the output summary. To better capture such dependencies among records, we make use of the content selection gate mechanism as shown in Figure 3.
Data-to-Text Generation with Content Selection and Planning
We first compute the attention scores α j,k over the input table and use them to obtain an attentional vector r att j for each record r j :
Data-to-Text Generation with Content Selection and Planning
where W a ∈ R n×n , W g ∈ R n×2n are parameter matrices, and k =j α j,k = 1.  Figure 3: Content selection mechanism.
Data-to-Text Generation with Content Selection and Planning
We next apply the content selection gating mechanism to r j , and obtain the new record representation r cs j via: g j = sigmoid r att j r cs j = g j r j where denotes element-wise multiplication, and gate g j ∈ [0, 1] n controls the amount of information flowing from r j . In other words, each element in r j is weighed by the corresponding element of the content selection gate g j .
Data-to-Text Generation with Content Selection and Planning
In our generation task, the output text is long but follows a canonical structure. Game summaries typically begin by discussing which team won/lost, following with various statistics involving individual players and their teams (e.g., who performed exceptionally well or under-performed), and finishing with any upcoming games. We hypothesize that generation would benefit from an explicit plan specifying both what to say and in which order. Our model learns such content plans from training data. However, notice that RO-TOWIRE (see Table 1) and most similar data-to-text datasets do not naturally contain content plans. Fortunately, we can obtain these relatively straightforwardly following an information extraction approach (which we explain in Section 4).
Data-to-Text Generation with Content Selection and Planning
Suffice it to say that plans are extracted by mapping the text in the summaries onto entities in the input table, their values, and types (i.e., relations). A plan is a sequence of pointers with each entry pointing to an input record {r j } |r| j=1 . An excerpt of a plan is shown in Table 2. The order in the plan corresponds to the sequence in which entities appear in the game summary. Let z = z 1 . . . z |z| denote the content planning sequence. Each z k points to an input record, i.e., z k ∈ {r j } |r| j=1 . Given the input records, the probability p(z|r) is decomposed as:
Data-to-Text Generation with Content Selection and Planning
Since the output tokens of the content planning stage correspond to positions in the input sequence, we make use of Pointer Networks (Vinyals et al. 2015). The latter use attention to point to the tokens of the input sequence rather than creating a weighted representation of source encodings. As shown in Figure 2, given {r j } |r| j=1 , we use an LSTM decoder to generate tokens corresponding to positions in the  input. The first hidden state of the decoder is initialized by avg({r cs j } |r| j=1 ), i.e., the average of record vectors. At decoding step k, let h k be the hidden state of the LSTM. We model p(z k = r j |z <k , r) as the attention over input records:
Data-to-Text Generation with Content Selection and Planning
where the probability is normalized to 1, and W c are parameters. Once z k points to record r j , we use the corresponding vector r cs j as the input of the next LSTM unit in the decoder.
Data-to-Text Generation with Content Selection and Planning
The probability of output text y conditioned on content plan z and input table r is modeled as:
Data-to-Text Generation with Content Selection and Planning
where y <t = y 1 . . . y t−1 . We use the encoder-decoder architecture with an attention mechanism to compute p(y|r, z).
Data-to-Text Generation with Content Selection and Planning
We first encode the content plan z into {e k } |z| k=1 using a bidirectional LSTM. Because the content plan is a sequence of input records, we directly feed the corresponding record vectors {r cs j } |r| j=1 as input to the LSTM units, which share the record encoder with the first stage.
Data-to-Text Generation with Content Selection and Planning
The text decoder is also based on a recurrent neural network with LSTM units. The decoder is initialized with the hidden states of the final step in the encoder. At decoding step t, the input of the LSTM unit is the embedding of the previously predicted word y t−1 . Let d t be the hidden state of the t-th LSTM unit. The probability of predicting y t from the output vocabulary is computed via:
Data-to-Text Generation with Content Selection and Planning
are parameters, and |V y | is the output vocabulary size. We further augment the decoder with a copy mechanism, i.e., the ability to copy words directly from the value portions of records in the content plan (i.e., {z k } |z| k=1 ). We experimented with joint (Gu et al. 2016) and conditional copy methods (Gulcehre et al. 2016). Specifically, we introduce a variable u t ∈ {0, 1} for each time step to indicate whether the predicted token y t is copied (u t = 1) or not (u t = 0). The probability of generating y t is computed by:
Data-to-Text Generation with Content Selection and Planning
where u t is marginalized out.
Data-to-Text Generation with Content Selection and Planning
Joint Copy The probability of copying from record values and generating from the vocabulary is globally normalized:
Data-to-Text Generation with Content Selection and Planning
where y t ← z k indicates that y t can be copied from z k , W b is shared as in Equation (1), and W y , b y are shared as in Equation (2).
Data-to-Text Generation with Content Selection and Planning
The variable u t is first computed as a switch gate, and then is used to obtain the output probability:
Data-to-Text Generation with Content Selection and Planning
where β t,k and p gen (y t |y <t , z, r) are computed as in Equations (1)-(2), and w u ∈ R n , b u ∈ R are parameters. Following Gulcehre et al. (2016) and Wiseman et al. (2017), if y t appears in the content plan during training, we assume that y t is copied (i.e., u t = 1). 2
Data-to-Text Generation with Content Selection and Planning
Our model is trained to maximize the log-likelihood of the gold 3 content plan given table records r and the gold output text given the content plan and table records:
Data-to-Text Generation with Content Selection and Planning
where D represents training examples (input records, plans, and game summaries). During inference, the output for input r is predicted by:
Data-to-Text Generation with Content Selection and Planning
where z and y represent content plan and output text candidates, respectively. For each stage, we utilize beam search to approximately obtain the best results.
Data-to-Text Generation with Content Selection and Planning
Data We trained and evaluated our model on ROTOWIRE Wiseman et al. (2017), a dataset of basketball game summaries, paired with corresponding box-and line-score tables. The summaries are professionally written, relatively well structured and long (337 words on average). The number of record types is 39, the average number of records is 628, the vocabulary size is 11.3K words and token count is 1.6M. The dataset is ideally suited for document-scale generation. We followed the data partitions introduced in Wiseman et al. (2017): we trained on 3,398 summaries, tested on 728, and used 727 for validation.
Data-to-Text Generation with Content Selection and Planning
We extracted content plans from the ROTOWIRE game summaries following an information extraction (IE) approach. Specifically, we used the IE system introduced in Wiseman et al. (2017) which identifies candidate entity (i.e., player, team, and city) and value (i.e., number or string) pairs that appear in the text, and then predicts the type (aka relation) of each candidate pair. For instance, in the document in Table 1, the IE system might identify the pair "Jeff Teague, 20" and then predict that that their relation is "PTS", extracting the record (Jeff Teague, 20, PTS). Wiseman et al. (2017) train an IE system on RO-TOWIRE by determining word spans which could represent entities (i.e., by matching them against players, teams or cities in the database) and numbers. They then consider each entity-number pair in the same sentence, and if there is a record in the database with matching entities and values, the pair is assigned the corresponding record type or otherwise given the label "none" to indicate unrelated pairs. We adopted their IE system architecture which predicts relations by ensembling 3 convolutional models and 3 bidirectional LSTM models. We trained this system on the train-3 Strictly speaking, the content plan is not gold since it was not created by an expert but is the output of a fairly accurate IE system. ing portion of the ROTOWIRE corpus. 4 On held-out data it achieved 94% accuracy, and recalled approximately 80% of the relations licensed by the records. Given the output of the IE system, a content plan simply consists of (entity, value, record type, h/v) tuples in their order of appearance in a game summary (the content plan for the summary in Table 1 is shown in Table 2). Player names are pre-processed to indicate the individual's first name and surname (see Isaiah and Thomas in Table 2); team records are also pre-processed to indicate the name of team's city and the team itself (see Boston and Celtics in Table 2).
Data-to-Text Generation with Content Selection and Planning
Training Configuration We validated model hyperparameters on the development set. We did not tune the dimensions of word embeddings and LSTM hidden layers; we used the same value of 600 reported in Wiseman et al. (2017). We used one-layer pointer networks during content planning, and two-layer LSTMs during text generation. Input feeding (Luong et al. 2015) was employed for the text decoder. We applied dropout (Zaremba et al. 2014) at a rate of 0.3. Models were trained for 25 epochs with the Adagrad optimizer (Duchi et al. 2011); the initial learning rate was 0.15, learning rate decay was selected from {0.5, 0.97}, and batch size was 5. For text decoding, we made use of BPTT (Mikolov et al. 2010) and set the truncation size to 100. We set the beam size to 5 during inference. All models are implemented in OpenNMT-py (Klein et al. 2017).
Data-to-Text Generation with Content Selection and Planning
Automatic Evaluation We evaluated model output using the metrics defined in Wiseman et al. (2017). The idea is to employ a fairly accurate IE system (see the description in Section 4) on the gold and automatic summaries and compare whether the identified relations align or diverge.
Data-to-Text Generation with Content Selection and Planning
Letŷ be the gold output, and y the system output. Content selection (CS) measures how well (in terms of precision and recall) the records extracted from y match those found inŷ. Relation generation (RG) measures the factuality of the generation system as the proportion of records extracted from y which are also found in r (in terms of precision and number of unique relations). Content ordering (CO) measures how well the system orders the records it has chosen and is computed as the normalized Damerau-Levenshtein Distance between the sequence of records extracted from y andŷ. In addition to these metrics, we report BLEU (Papineni et al. 2002), with human-written game summaries as reference.
Data-to-Text Generation with Content Selection and Planning
Our results on the development set are summarized in Table 3. We compare our Neural Content Planning model (NCP for short) against the two encoder-decoder (ED) models presented in Wiseman et al. (2017) with joint copy (JC) and conditional copy (CC), respectively. In addition to our own re-implementation of these models, we include the best scores reported in Wiseman et al. (2017) which were obtained with an encoder-decoder model enhanced with con-   Table 4: Ablation results on ROTOWIRE development set using relation generation (RG) count (#) and precision (P%), content selection (CS) precision (P%) and recall (R%), content ordering (CO) in normalized Damerau-Levenshtein distance (DLD%), and BLEU. ditional copy. Table 3 also shows results when NCP uses oracle content plans (OR) as input. In addition, we report the performance of a template-based generator (Wiseman et al. 2017) which creates a document consisting of eight template sentences: an introductory sentence (who won/lost), six player-specific sentences (based on the six highest-scoring players in the game), and a conclusion sentence. As can be seen, NCP improves upon vanilla encoderdecoder models (ED+JC, ED+CC), irrespective of the copy mechanism being employed. In fact, NCP achieves comparable scores with either joint or conditional copy mechanism which indicates that it is the content planner which brings performance improvements. Overall, NCP+CC achieves best content selection and content ordering scores in terms of BLEU. Compared to the best reported system in Wiseman et al. (2017), we achieve an absolute improvement of approximately 12% in terms of relation generation; content selection precision also improves by 5% and recall by 15%, content ordering increases by 3%, and BLEU by 1.5 points. The results of the oracle system (NCP+OR) show that content selection and ordering do indeed correlate with the quality of the content plan and that any improvements in our planning component would result in better output. As far as the template-based system is concerned, we observe that it obtains low BLEU and CS precision but scores high on CS recall and RG metrics. This is not surprising as the template system is provided with domain knowledge which our model does not have, and thus represents an upper-bound on content selection and relation generation. We also measured  Table 5: Automatic evaluation on ROTOWIRE test set using relation generation (RG) count (#) and precision (P%), content selection (CS) precision (R%) and recall (R%), content ordering (CO) in normalized Damerau-Levenshtein distance (DLD%), and BLEU.
Data-to-Text Generation with Content Selection and Planning
the degree to which the game summaries generated by our model contain redundant information as the proportion of non-duplicate records extracted from the summary by the IE system. 84.5% of the records in NCP+CC are non-duplicates compared to Wiseman et al. (2017) who obtain 72.9% showing that our model is less repetitive. We further conducted an ablation study with the conditional copy variant of our model (NCP+CC) to establish whether improvements are due to better content selection (CS) and/or content planning (CP). We see in Table 4 that content selection and planning individually contribute to performance improvements over the baseline (ED+CC), and accuracy further increases when both components are taken into account. In addition we evaluated these components on their own (independently of text generation) by comparing the output of the planner (see p(z|r) block in Figure 2) against gold content plans obtained using the IE system (see row NCP in Table 4. Compared to the full system (NCP+CC), content selection precision and recall are higher (by 4.5% and 2%, respectively) as well as content ordering (by 1.8%). In another study, we used the CS and CO metrics to measure how well the generated text follows the content plan produced by the planner (instead of arbitrarily adding or removing information). We found out that NCP+CC generates game summaries which follow the content plan closely: CS precision is higher than 85%, CS recall is higher than 93%, and CO higher than 84%. This reinforces our claim that higher accuracies in the content selection and planning phases will result in further improvements in text generation.
Data-to-Text Generation with Content Selection and Planning
The test set results in Table 5 follow a pattern similar to the development set. NCP achieves higher accuracy in all metrics including relation generation, content selection, content ordering, and BLEU compared to Wiseman et al. (2017). We provide examples of system output in Table 6 and the supplementary material.
Data-to-Text Generation with Content Selection and Planning
Human-Based Evaluation We conducted two human evaluation experiments using the Amazon Mechanical Turk (AMT) crowdsourcing platform. The first study assessed relation generation by examining whether improvements in relation generation attested by automatic evaluation metrics are indeed corroborated by human judgments. We compared our best performing model (NCP+CC), with gold reference summaries, a template system and the best model of Wiseman et al. (2017). AMT workers were presented with a spe-The Golden State Warriors defeated the Boston Celtics 104-88 at TD Garden on Friday. The Warriors (10-2) came into this game winners of five of their last six games, but the Warriors (6-6) were able to pull away in the second half. Klay Thompson led the way for the Warriors with 28 points on 12-of-21 shooting, while Kevin Durant added 23 points, 10 rebounds, seven assists and two steals. Stephen Curry added 16 points and eight assists, while Draymond Green rounded out the box score with 11 points, eight rebounds and eight assists. For the Celtics, it was Isaiah Thomas who shot just 4-of-12 from the field and finished with 18 points. Avery Bradley added 17 points and 10 rebounds, while the rest of the Celtics combined to score just seven points. Boston will look to get back on track as they play host to the 76ers on Friday.  (bottom). Text that accurately reflects a record in the associated box or line score is in blue, erroneous text is in red.
Data-to-Text Generation with Content Selection and Planning
cific NBA game's box score and line score, and four (randomly selected) sentences from the summary. They were asked to identify supporting and contradicting facts mentioned in each sentence. We randomly selected 30 games from the test set. Each sentence was rated by three workers.
Data-to-Text Generation with Content Selection and Planning
The left two columns in Table 7 contain the average number of supporting and contradicting facts per sentence as determined by the crowdworkers, for each model. The template-based system has the highest number of supporting facts, even compared to the human gold standard. TEMPL does not perform any content selection, it includes a large number of facts from the database and since it does not perform any generation either, it exhibits a few contradictions. Compared to WS-2017 and the Gold summaries, NCP+CC displays a larger number of supporting facts. All models are significantly 5 different in the number of supporting facts (#Supp) from TEMPL (using a one-way ANOVA with posthoc Tukey HSD tests). NCP+CC is significantly different from WS-2017 and Gold. With respect to contradicting facts (#Cont), Gold and TEMPL are not significantly different from each other but are significantly different from the neural systems (WS-2017, NCP+CC).
Data-to-Text Generation with Content Selection and Planning
In the second experiment, we assessed the generation quality of our model. We elicited judgments for the same 30 games used in the first study. For each game, participants were asked to compare a human-written summary, NCP with conditional copy (NCP+CC), Wiseman et al.'s (2017) best model, and the template system. Our study used Best-Worst Scaling (BWS; Louviere, Flynn, and Marley 2015), a technique shown to be less labor-intensive and providing more reliable results as compared to rating scales (Kiritchenko and Mohammad 2017). We arranged every 4-tuple of com-5 All significance differences reported throughout this paper are with a level less than 0.05.  Table 7: Average number of supporting (#Support) and contradicting (#Contra) facts in game summaries and bestworst scaling evaluation (higher is better) for grammaticality (Gram), Coherence (Cohere), and Conciseness (Concise).
Data-to-Text Generation with Content Selection and Planning
peting summaries into 6 pairs. Every pair was shown to three crowdworkers, who were asked to choose which summary was best and which was worst according to three criteria: Grammaticality (is the summary fluent and grammatical?), Coherence (is the summary easy to read? does it follow a natural ordering of facts?), and Conciseness (does the summary avoid redundant information and repetitions?). The score of a system for each criterion is computed as the difference between the percentage of times the system was selected as the best and the percentage of times it was selected as the worst (Orme 2009). The scores range from −100 (absolutely worst) to +100 (absolutely best).
Data-to-Text Generation with Content Selection and Planning
The results of the second study are summarized in Table 7. Gold summaries were perceived as significantly better compared to the automatic systems across all criteria (again using a one-way ANOVA with post-hoc Tukey HSD tests). NCP+CC was perceived as significantly more grammatical than WS-2017 but not compared to TEMPL which does not suffer from fluency errors since it does not perform any generation. NCP+CC was perceived as significantly more coherent than TEMPL and WS-2017. The template fairs poorly on coherence, its output is stilted and exhibits no variability (see top block in Table 6). With regard to conciseness, the neural systems are significantly worse than TEMPL, while NCP+CC is significantly better than WS-2017. By design the template cannot repeat information since there is no redundancy in the sentences chosen to verbalize the summary.
Data-to-Text Generation with Content Selection and Planning
Taken together, our results show that content planning improves data-to-text generation across metrics and systems. We find that NCP+CC overall performs best, however there is a significant gap between automatically generated summaries and human-authored ones.
Data-to-Text Generation with Content Selection and Planning
We presented a data-to-text generation model which is enhanced with content selection and planning modules. Experimental results (based on automatic metrics and judgment elicitation studies) demonstrate that generation quality improves both in terms of the number of relevant facts contained in the output text, and the order according to which these are presented. Positive side-effects of content planning are additional improvements in the grammaticality, and conciseness of the generated text. In the future, we would like to learn more detail-oriented plans involving inference over multiple facts and entities. We would also like to verify our approach across domains and languages.
Data-to-Text Generation with Content Selection and Planning
There was a bug in the dataset creation of Wiseman et al. (2017) which they identified and corrected. They also posted updated scores on their webpage. 6 We have used this corrected dataset in our experiments. We then discovered a bug in their code which computes the automatic metrics. The scores reported in this paper are using the corrected automatic metrics. To make the scores on our paper comparable to the numbers published on the webpage of Wiseman et al. (2017), we recompute here our scores with older IE metrics (without the bug fix) in Table 8 (development set) and Table 9 (test set). Table 10 shows two sample documents generated using the template system, Wiseman et al. (2017) (WS-2017 and our neural content planning model with conditional copy (NCP+CC). The text is highlighted in blue if it agrees with respective box/line scores and red if the text contradicts box/line scores. We also use the orange color to highlight repetitions.
Data-to-Text Generation with Content Selection and Planning
The template documents are gold standard in relation generation accuracy and they appear all in blue. The documents of Wiseman et al. (2017) show instances of contradictions and tend to be verbose containing duplicate text too. In contrast, our neural content planning model generates more factual text with fewer contradictions to box/line scores and less duplicate information.  Table 8: Automatic system evaluation on the ROTOWIRE development set using automatic evaluation metrics; relation generation (RG) count and precision, content selection (CS) precision and recall, content ordering (CO) in normalized Damerau-Levenshtein distance, and BLEU. TEMPL is template system, WS-2017 is the best system of Wiseman et al. (2017), ED+JC, ED+CC are encoder-decoder with joint copy and conditional copy, respectively, NCP+JC, NCP+CC are our Neural Content Planning models with joint copy and conditional copy, respectively. NCP+OR is the system with oracle content plan as input.  3PT, 6-6 FT) to go with 3 rebounds. Nikola Jokic scored 17 points (6-10 FG, 0-0 3PT, 5-7 FT) to go with 11 rebounds. Markieff Morris scored 15 points (5-12 FG, 0-0 3PT, 5-5 FT) to go with 3 rebounds. John Wall scored 15 points (5-14 FG, 0-4 3PT, 5-6 FT) to go with 7 rebounds. Danilo Gallinari scored 14 points (3-11 FG, 1-8 3PT, 7-9 FT) to go with 4 rebounds. Jusuf Nurkic scored 13 points (6-6 FG, 0-0 3PT, 1-2 FT) to go with 7 rebounds. The Washington Wizards' next game will be at home against the Dallas Mavericks, while the Denver Nuggets will travel to play the Bulls. WS-2017
Data-to-Text Generation with Content Selection and Planning
The Washington Wizards (8-13) defeated the Denver Nuggets (8-15) 92-85 on Wednesday at the Verizon Center in Washington. The Wizards got off to a hot start in this one, out-scoring the Nuggets 29-18 in the first quarter alone. The Nuggets were able to out-score the Nuggets 28-12 in the third quarter, while the Wizards were able to coast to a victory in front of their home crowd. The Nuggets were the superior shooters in this game, going 45 percent from the field and 32 percent from the three-point line, while the Wizards went just 41 percent from the floor and 32 percent from beyond the arc. The Nuggets were also able to shoot 41 percent from the field and 32 percent from the three-point line , while the Nuggets went just 41 percent from the floor and 32 percent from beyond the arc. The Nuggets were led by the duo of John Wall and Bradley Beal. Wall went 5-for-14 from the field and 0-for-4 from the three-point line to score a game-high of 15 points, while also adding seven rebounds and five assists. The Golden State Warriors (10-2) defeated the Boston Celtics (6-6) 104-88. Klay Thompson scored 28 points (12-21 FG, 3-6 3PT, 1-1 FT) to go with 4 rebounds. Kevin Durant scored 23 points (10-13 FG, 1-2 3PT, 2-4 FT) to go with 10 rebounds. Isaiah Thomas scored 18 points (4-12 FG, 1-6 3PT, 9-9 FT) to go with 2 rebounds. Avery Bradley scored 17 points (7-15 FG, 2-4 3PT, 1-2 FT) to go with 10 rebounds. Stephen Curry scored 16 points (7-20 FG, 2-10 3PT, 0-0 FT) to go with 3 rebounds. Terry Rozier scored 11 points (3-5 FG, 2-3 3PT, 3-4 FT) to go with 7 rebounds. The Golden State Warriors' next game will be at home against the Dallas Mavericks, while the Boston Celtics will travel to play the Bulls.
Data-to-Text Generation with Content Selection and Planning
The Golden State Warriors defeated the Boston Celtics, 104-88, at TD Garden on Wednesday. The Warriors (10-2) checked in to Saturday's contest with only two road wins in their last 11 games, but they were able to come away with a win against the Celtics (6-6) on Friday. The Warriors (10-2) were able to pull away in the second half, outscoring the Celtics (6-6) by a 31-9 margin over the final 12 minutes. However, Golden State was able to pull away in the second half, outscoring the Celtics by a 31-9 margin over the final 12 minutes. The Warriors were led by Kevin Durant's 23 points, which he supplemented with seven rebounds, seven assists, two steals and a block. Stephen Curry was next with 16 points, eight assists, three rebounds and four steals. Klay Thompson was next with a 28-point, 10-rebound double-double that also included three assists, two steals and a block. Draymond Green was next with 11 points, eight rebounds, eight assists and two blocks. Draymond Green was next with 11 points, eight rebounds, eight assists and two blocks. Draymond Green supplied 11 points, eight rebounds, eight assists, two blocks and a steal. David West paced the reserves with 4 points, two rebounds, a block and a block. The Celtics were paced by Thomas' 18 points, which he supplemented with four assists, two rebounds and four steals. Avery Bradley posted a 17-point, 10-rebound double-double that also included two assists, two steals and a block. Avery Bradley posted a 17-point, 10-rebound double-double that also included two assists, two steals and a block. Kelly Olynyk led the second unit with 11 points, three rebounds, two assists and a pair of steals. The Warriors head back home to face off with the Detroit Pistons on Friday night, while the Celtics remain home and await the Toronto Raptors for a Wednesday night showdown. NCP+CC
Data-to-Text Generation with Content Selection and Planning
The Golden State Warriors defeated the Boston Celtics 104-88 at TD Garden on Friday. The Warriors (10-2) came into this game winners of five of their last six games, but the Warriors (6-6) were able to pull away in the second half. Klay Thompson led the way for the Warriors with 28 points on 12-of-21 shooting, while Kevin Durant added 23 points, 10 rebounds, seven assists and two steals. Stephen Curry added 16 points and eight assists, while Draymond Green rounded out the box score with 11 points, eight rebounds and eight assists. For the Celtics, it was Isaiah Thomas who shot just 4-of-12 from the field and finished with 18 points. Avery Bradley added 17 points and 10 rebounds, while the rest of the Celtics combined to score just seven points. Boston will look to get back on track as they play host to the 76ers on Friday.
Data-to-Text Generation with Content Selection and Planning
Table 10: Example documents from the template-based system, WS-2017, the best system of Wiseman et al. (2017), and our Neural Content Planning model with conditional copy (NCP+CC). Text that accurately reflects a record in the associated box or line score is recorded in blue, erroneous text is marked in red, duplicate text is marked in orange.
Data-to-text Generation with Macro Planning
Data-to-text generation refers to the task of generating textual output from non-linguistic input (Reiter andDale, 1997, 2000;Gatt and Krahmer, 2018) such as databases of records, simulations of physical systems, accounting spreadsheets, or expert system knowledge bases. As an example, Figure 1 shows various statistics describing a major league baseball (MLB) game, including extracts from the box score (i.e., the performance of the two teams and individual team members who played as batters, pitchers or fielders; Tables (A)), play-by-play (i.e., the detailed sequence of each play of the game as it occurred; Table (B)), and a human written game summary (Table (C)).
Data-to-text Generation with Macro Planning
Traditional methods for data-to-text generation (Kukich, 1983;McKeown, 1992;Reiter and Dale, 1997) follow a pipeline architecture, adopting separate stages for text planning (determining which content to talk about and how it might be organized in discourse), sentence planning (aggregating content into sentences, deciding specific words to describe concepts and relations, and generating referring expressions), and linguistic realisation (applying the rules of syntax, morphology and orthographic processing to generate surface forms). Recent neural network-based approaches (Lebret et al., 2016;Mei et al., 2016;Wiseman et al., 2017) make use of the encoder-decoder architecture (Sutskever et al., 2014), are trained endto-end, and have no special-purpose modules for how to best generate a text, aside from generic mechanisms such as attention and copy (Bahdanau et al., 2015;Gu et al., 2016). The popularity of end-to-end models has been further boosted by the release of new datasets with thousands of inputdocument training pairs. The example shown in Figure 1 is taken from the MLB dataset (Puduppully et al., 2019b) which contains baseball game statistics and human written summaries (~25K instances). ROTOWIRE (Wiseman et al., 2017) is another widely used benchmark, which contains NBA basketball game statistics and their descriptions (~5K instances).
Data-to-text Generation with Macro Planning
Despite being able to generate fluent text, neural data-to-text generation models are often imprecise, prone to hallucination (i.e., generate text that is not supported by the input), and poor at content selection and document structuring (Wiseman et al., 2017). Attempts to remedy some of these issues focus on changing the way entities are represented (Puduppully et al., 2019b;Iso et al., 2019), allowing the decoder to skip low-confidence tokens to enhance faithful generation (Tian et al., 2019), and making the encoder-decoder architecture more modular by introducing micro planning (Puduppully et al., 2019a;Moryossef et al., 2019). Micro planning operates at the record level (see Ta- <P> Keller gave up a home run to the first batter of the game -Cedric Mullins -but quickly settled in to pitch eight strong innings in the Kansas City Royals' 9-2 win over the Baltimore Orioles in a matchup of the teams with the worst records in the majors. <P> Keller (7-5) gave up two runs and four hits with two walks and four strikeouts to improve to 3-0 with a 2.16 ERA in his last four starts. <P> Ryan O'Hearn homered among his three hits and drove in four runs, Whit Merrifield scored three runs, and Hunter Dozier and Cam Gallagher also went deep to help the Royals win for the fifth time in six games on their current homestand. <P> With the score tied 1-1 in the fourth, Andrew Cashner (4-13) gave up a sacrifice fly to Merrifield after loading the bases on two walks and a single. Dozier led off the fifth inning with a 423-foot home run to left field to make it 3-1. <P> The Orioles pulled within a run in the sixth when Mullins led off with a double just beyond the reach of Dozier at third, advanced to third on a fly ball and scored on Trey Mancini's sacrifice fly to the wall in right. <P> . . .   (Tables (A) and (B)). Macro plan for the game summary is shown at the bottom (Table (E)). <P> indicates paragraph delimiters. There is a plan for every paragraph in the game summary (correspondence shown in same color); <V(entity)> verbalizes entities, while <V(inning-T/B)> verbalizes events related to the top/bottom side of an inning (see Section 3.1). Set of candidate paragraph plans are shown above macro plan (Table (D)) and grouped into two types: plans describing a single entity/event or their combinations. Best viewed in color.
Data-to-text Generation with Macro Planning
tioned within a textual unit (e.g., a sentence) and how these should be structured (e.g., the sequence of records). An explicit content planner essentially makes the job of the neural network less onerous allowing to concentrate on producing fluent natural language output, without expending too much effort on content organization. In this work, we focus on macro planning, the high-level organization of information and how it should be presented which we argue is important for the generation of long, multi-paragraph documents (see text (C) in Figure 1). Problematically, modern datasets like MLB (Puduppully et al. 2019b; and also Figure 1) and ROTOWIRE (Wiseman et al., 2017) do not naturally lend them-selves to document planning as there is no explicit link between the summary and the content of the game (which is encoded in tabular form). In other words, the underlying plans are latent, and it is not clear how they might be best represented, i.e., as sequences of records from a table, or simply words. Nevertheless, game summaries through their segmentation into paragraphs (and lexical overlap with the input) give clues as to how content might be organized. Paragraphs are a central element of discourse (Chafe, 1979;Longacre, 1979;Halliday and Hasan, 1976), the smallest domain where coherence and topic are defined and anaphora resolution is possible (Zadrozny and Jensen, 1991). We therefore operationalize the macro plan for a game summary as a sequence of paragraph plans.
Data-to-text Generation with Macro Planning
Although resorting to paragraphs describes the summary plan at a coarse level, we still need to specify individual paragraph plans. In the sports domain, paragraphs typically mention entities (e.g, players important in the game), key events (e.g., scoring a run), and their interaction. And most of this information is encapsulated in the statistics accompanying game summaries (see Tables (A) and (B) in Figure 1). We thus define paragraph plans such that they contain verbalizations of entity and event records (see plan (E) in Figure 1). Given a set of paragraph plans and their corresponding game summary (see Tables (D) and summary (C) in Figure 1), our task is twofold. At training time, we must learn how content was selected in order to give rise to specific game summaries (e.g., how input (D) led to plan (E) for summary (C) in Figure 1), while at test time, given input for a new game, we first predict a macro plan for the summary and then generate the corresponding document.
Data-to-text Generation with Macro Planning
We present a two-stage approach where macro plans are induced from training data (by taking the table and corresponding summaries into account) and then fed to the text generation stage. Aside from making data-to-text generation more interpretable, the task of generating a document from a macro plan (rather than a table) affords greater control over the output text and plays to the advantage of encoder-decoder architectures which excel at modeling sequences. We evaluate model performance on the ROTOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) benchmarks. Experimental results show that our plan-and-generate approach produces output which is more factual, coherent, and fluent compared to existing state-of-the-art models. Our code, trained models and dataset with macro plans can be found at https://github.com/ratishsp/ data2text-macro-plan-py.
Data-to-text Generation with Macro Planning
We hypothesize that generation based on plans should fare better compared to generating from a set of records, since macro plans offer a bird'seye view, a high-level organization of the document content and structure. We also believe that macro planning will work well for long-form text generation, i.e., for datasets which have multi-paragraph target texts, a large vocabulary space, and require content selection.
Data-to-text Generation with Macro Planning
We assume the input to our model is a set of para-
Data-to-text Generation with Macro Planning
where e i is a paragraph plan. We model the process of generating output summary y given E as a two step process, namely the construction of a macro plan x based on the set of paragraph plans, followed by the generation of a summary given a macro plan as input. We now explain how E is obtained and each step is realized. We discuss our model considering mainly an example from the MLB dataset (Puduppully et al., 2019b) but also touch on how the approach can be straightforwardly adapted to ROTOWIRE (Wiseman et al., 2017).
Data-to-text Generation with Macro Planning
A macro plan consists of a sequence of paragraph plans separated by a paragraph discourse marker <P>, i.e., x = e i <P> e j . . . <P> e k where e i , e j , e k ∈ E. A paragraph plan in turn is a sequence of entities and events describing the game. By entities we mean individual players or teams and the information provided about them in box score statistics (see rows and column headings in Figure 1 Table (A)), while events refer to information described in play-by-play (see Table (B)). In baseball, plays are grouped in half-innings. During each half of an inning, a team takes its turn to bat (the visiting team bats in the top half and the home team in the bottom half). An example macro plan is shown at the bottom of Figure 1. Within a paragraph plan, entities and events are verbalized into a text sequence along the lines of Saleh et al. (2019). We make use of special tokens for the <TYPE> of record followed by the value of record from the table. We retain the same position for each record type and value. For example, batter C.Mullins from Figure 1 would be verbalized as <PLAYER>C.Mullins <H/V>H <AB>4 <BR>2 <BH>2 <RBI>1 <TEAM>Orioles . . . . For the sake of brevity we use shorthand <V(C.Mullins)> for the full entity.
Data-to-text Generation with Macro Planning
Paragraph Plan for Entities For a paragraph containing entities, the corresponding plan will be a verbalization of the entities in sequence. For paragraphs with multiple mentions of the same entity, the plan will verbalize an entity only once and at its first position of mention. Paragraph "Keller gave up a home run . . . the teams with the worst records in the majors" from the summary in Fig Paragraph Plan for Events A paragraph may also describe one or more events. For example, the paragraph "With the score tied 1-1 in the fourth . . . 423-foot home run to left field to make it 3-1 " discusses what happened in the bottom halves of the fourth and fifth innings. We verbalize an event by first describing the participating entities followed by the plays in the event. Entities are described in the order in which they appear in a play, and within the same play we list the batter followed by the pitcher, fielder, scorer, and basemen. The paragraph plan corresponding to the bottom halves of the fourth and fifth inning is <V(4-B  Figure 1) and abbreviates the following detailed plan: <INN>5 <HALF>B <BATTING>Royals <PITCHING>Orioles <PL-ID>1 <BATTER>H.Dozier <PITCHER>A. Cashner> <ACTION>Home-run <SCORES> Royals-3-Orioles-1, etc.
Data-to-text Generation with Macro Planning
The procedure described above is not specific to MLB and can be ported to other datasets with similar characteristics such as ROTOWIRE. However, ROTOWIRE does not provide play-by-play information, and as a result there is no event verbalization for this dataset.
Data-to-text Generation with Macro Planning
We provided our definition for macro plans in the previous sections, however, it is important to note that such macro plans are not readily available in data-to-text benchmarks like MLB (Puduppully et al., 2019b) and ROTOWIRE (Wiseman et al., 2017) which consist of tables of records r paired with a gold summary y (see Tables (A)-(C) in Figure 1). We now describe our method for obtaining macro plans x from r and y.
Data-to-text Generation with Macro Planning
Similar to Moryossef et al. (2019), we define macro plans to be conformant with gold summaries such that (1) they have the same splits into paragraphs -entities and events within a paragraph in y are grouped into a paragraph plan in x; and (2) the order of events and entities in a paragraph and its corresponding plan are identical. We con-struct macro plans by matching entities and events in the summary to records in the tables. Furthermore, paragraph delimiters within summaries form natural units which taken together give rise to a high-level document plan.
Data-to-text Generation with Macro Planning
We match entities in summaries with entities in tables using exact string match, allowing for some degree of variation in the expression of team names (e.g., A's for Athletics and D-backs for Diamondbacks). Information pertaining to innings appears in the summaries in the form of ordinal numbers (e.g., first, ninth ) modifying the noun inning and can be relatively easily identified via pattern matching (e.g., in sentences like "Dozier led off the fifth inning"). However, there are instances where the mention of innings is more ambiguous (e.g., "With the scored tied 1-1 in the fourth, Andrew Cashner (4-13) gave up a sacrifice fly"). We could disambiguate such mentions manually and then train a classifier to learn to predict whether an inning is mentioned. Instead, we explore a novel annotationfree method which makes use of the pretrained language model GPT2 (Radford et al., 2019). Specifically, we feed the context preceding the ordinal number to GPT2 (i.e., the current paragraph up to the ordinal number and the paragraph preceding it) and if inning appears in the top 10 next word predictions, we consider it a positive match. On a held out dataset, this method achieves 98% precision and 98% recall at disambiguating inning mentions.
Data-to-text Generation with Macro Planning
To resolve whether the summary discusses the top or bottom side of an inning, we compare the entities in the paragraph with the entities in each half-inning (play-by-play Table (B) in Figure 1) and choose the side with the greater number of entity matches. For instance, Andrew Cashner, Merrifield and fourth inning uniquely resolves to the bottom half of the fourth inning. Figure 1 shows the macro plan we obtain for game summary (C). Importantly, macro plan (E) is the outcome of a content selection process after considering several candidate paragraph plans as input. So, what are the candidate paragraph plans which give rise to macro plan (E)? To answer this question, we examined the empirical distribution of paragraph plans in MLB and ROTOWIRE (training portion). Interestingly, we found that~79% of the paragraph plans in MLB refer to a single event or a single player (and team(s) (1), (2), e att 3 in Equation (3), and e c 3 in Equation (4).
Data-to-text Generation with Macro Planning
92% of paragraphs are about a singleton player (and team(s)) or a pair of players.
Data-to-text Generation with Macro Planning
Based on this analysis, we assume that paragraph plans can be either one (verbalized) entity/event or a combination of at most two. Under this assumption, we explicitly enumerate the set of candidate paragraph plans in a game. For the game in Figure 1, candidate paragraph plans are shown in Tables (D). The first table groups plans based on individual verbalizations describing the team(s), players, and events taking place in specific innings. The second table groups pairwise combinations thereof. In MLB, such combinations are between team(s) and players. In ROTOWIRE, we also create combinations between players. Such paragraph plans form set E based on which macro plan x is constructed to give rise to game summary y.
Data-to-text Generation with Macro Planning
The input to our model is a set of paragraph plans each of which is a sequence of tokens. We first compute paragraph plan representations ∈ R n , and then apply a contextualization and content planning mechanism similar to planning modules introduced in earlier work (Puduppully et al., 2019a;Chen and Bansal, 2018). Predicted macro plans serve as input to our text generation model which adopts an encoder-decoder architecture (Bahdanau et al., 2015;Luong et al., 2015).
Data-to-text Generation with Macro Planning
Paragraph Plan Representation We encode tokens in a verbalized paragraph plan e i as {e i, j } |e i | j=1 with a BiLSTM (Figure 2 bottom part). To reflect the fact that some records will be more important than others, we compute an attention weighted sum of {e i, j } |e i | j=1 following Yang et al. (2016). Let d ∈ R n denote a randomly initialized query vector learnt jointly with the rest of parameters. We compute attention values α i, j over d and paragraph plan token representation e i, j :
Data-to-text Generation with Macro Planning
Paragraph plan vector e i is the attention weighted sum of e i, j (with ∑ j α i, j = 1):
Data-to-text Generation with Macro Planning
Next, we contextualize each paragraph plan representation vis-a-vis other paragraph plans (Figure 2 top left part). First, we compute attention scores β i,k over paragraph plan representations to obtain an attentional vector e att i for each:
Data-to-text Generation with Macro Planning
where W a ∈ R n×n , W g ∈ R n×2n are parameter matrices, and ∑ k =i β i,k = 1. Then, we compute a content selection gate, and apply this gate to e i to obtain new paragraph plan representation e c i :
Data-to-text Generation with Macro Planning
where denotes element-wise multiplication. Thus, each element in e i is weighted by corresponding element of g i ∈ [0, 1] n to obtain a contextualized paragraph plan representation e c i .
Data-to-text Generation with Macro Planning
Our model learns to predict macro plans, after having been trained on pairs of sets of paragraph plans and corresponding macro plans (Sections 3.2 and 3.3 explain how we obtain these for data-to-text datasets like RO-TOWIRE and MLB). More formally, we model macro plan z = z 1 . . . z |z| as a sequence of pointers, with each z k pointing to an input paragraph plan,   Figure 2. The output points to e 3 , e |E| , and e 1 (see Equations (5) and (6)). EOM is end of macro plan token.
Data-to-text Generation with Macro Planning
i.e., z k ∈ {e i } |E| i=1 . We decompose p(z|E), the probability of macro plan z given paragraph plans E, as:
Data-to-text Generation with Macro Planning
where z <k = z 1 . . . z k−1 . We use Pointer Networks (Vinyals et al., 2015) to model p(z k |z <k , E) as:
Data-to-text Generation with Macro Planning
where p(z k |z <k , E) is normalized to 1 and W b ∈ R n×n . Rather than computing a weighted representation, Pointer Networks make use of attention to point to specific elements in the input (see Figure 3). We use a decoder LSTM to compute hidden representation h k at time step k. We initialize h 0 with the mean paragraph plan representation, avg({e c i } |E| i=1 ). Once the output points to e i , its representation e c i is used as input to the next step of the LSTM decoder. The process stops when the model points to EOM, a token indicating end of the macro plan.
Data-to-text Generation with Macro Planning
Recall that z is a sequence of pointers with each entry z k pointing to a paragraph plan i.e., z k ∈ {e i } |E| i=1 . We can deterministically obtain macro plan x from z by retrieving the paragraph plans being pointed to, adding <P> separators in between. The conditional output probability p(y|x) is modeled as:
Data-to-text Generation with Macro Planning
where y <t = y 1 . . . y t−1 .
Data-to-text Generation with Macro Planning
To compute p(y|x), we use an encoder-decoder architecture enhanced with an attention mechanism (Bahdanau et al., 2015;Luong et al., 2015). We encode macro plan x with a bidirectional LSTM (Hochreiter and Schmidhuber, 1997). At time step t, we lookup the embedding of the previously predicted word y t−1 and feed it as input to the decoder which is another LSTM unit. The decoder attends over hidden states of the macro plan to predict y t . We further incorporate a copy mechanism (Gulcehre et al., 2016) in the decoder to enable copying values directly from the macro plan.
Data-to-text Generation with Macro Planning
We expect the text generation model to learn to generate summary tokens while focusing on the corresponding macro plan and that the output summary will indeed follow the plan in terms of the entities and events being described and their order. At the same time, we believe that text generation is relatively easier as the encoder-decoder model is relieved from the tasks of document structuring and information selection.
Data-to-text Generation with Macro Planning
We train two independent models for macro planning and text generation. Our training objective for macro planning aims to maximize the log likelihood of the macro plan given the paragraph plans:
Data-to-text Generation with Macro Planning
where D is the training set consisting of pairs of (sets of) paragraph plans and macro plans, and θ are model parameters.
Data-to-text Generation with Macro Planning
Our training objective for text generation aims to maximize the log likelihood of the output text given the macro plan:
Data-to-text Generation with Macro Planning
where F is the training set consisting of pairs of macro plans and game summaries, and φ are model parameters.
Data-to-text Generation with Macro Planning
During inference, we employ beam search to find the most likely macro planẑ among candidate macro plans z given paragraph plans as input.
Data-to-text Generation with Macro Planning
We deterministically obtainx fromẑ, and output summaryŷ among candidate outputs y given macro planx as input: We reconstructed the MLB dataset, as the version released by Puduppully et al. (2019b) had removed all paragraph delimiters from game summaries. Specifically, we followed their methodology and downloaded the same summaries from the ESPN website 2 and added the <P> delimiter to paragraphs in the summaries. 3 ROTOWIRE does not have paragraph delimiters in game summaries either. We reverse engineered these as follows:
Data-to-text Generation with Macro Planning
(1) we split summaries into sentences using the NLTK (Bird et al., 2009) sentence tokenizer; (2) initialized each paragraph with a separate sentence;
Data-to-text Generation with Macro Planning
(3) merged two paragraphs into one if the entities in the former were a superset of entities in the latter; (4) repeated Step 3 until no merges were possible.
Data-to-text Generation with Macro Planning
Training Configuration We tuned the model hyperparameters on the development set. For training the macro planning and the text generation stages, we used the Adagrad (Duchi et al., 2011) optimizer. Furthermore, the text generation stage made use of truncated BPTT (Williams and Peng, 1990) with truncation length 100. We learn subword vocabulary (Sennrich et al., 2016) for paragraph plans in the macro planning stage. We used 2.5K merge operations for ROTOWIRE and 8K merge operations for MLB. In text generation, we learn a joint subword vocabulary for the macro plan and game summaries. We used 6K merge operations for RO-TOWIRE and 16K merge operations for MLB. All models were implemented on OpenNMT-py (Klein et al., 2017). We add to set E the paragraph plans corresponding to the output summary paragraphs, to ensure full coverage during training of the macro planner. During inference for predicting macro plans, we employ length normalization (Bahdanau et al., 2015) to avoid penalizing longer outputs; specifically, we divide the scores of beam search by the length of the output. In addition, we adopt bigram blocking (Paulus et al., 2018). For MLB, we further block beams containing more than two repetitions of a unigram. This helps improve the diversity of the predicted macro plans.
Data-to-text Generation with Macro Planning
System Comparisons We compared our model against the following systems: (1) the Templatebased generators from Wiseman et al. (2017) for ROTOWIRE and Puduppully et al. (2019b) for MLB. Both systems apply the same principle, they emit a sentence about the teams playing in the game, followed by player-specific sentences, and a closing sentence. MLB additionally contains a description of play-by-play; (2) ED+CC, the best performing system in Wiseman et al. (2017), is a vanilla encoder-decoder model equipped with an attention and copy mechanism; (3) NCP+CC, the micro planning model of Puduppully et al. (2019a), generates content plans from the table by making use of Pointer networks (Vinyals et al., 2015) to point to records; content plans are encoded with a BiLSTM and the game summary is decoded using another LSTM with attention and copy; (4) ENT, the entity-based model of Puduppully et al. (2019b), creates dynamically updated entity-specific representations; the text is generated conditioned on the data input and entity memory representations using hierarchical attention at each time step.
Data-to-text Generation with Macro Planning
Automatic Evaluation For automatic evaluation, following earlier work (Wiseman et al. 2017;Puduppully et al. 2019a,b, inter alia) we report BLEU (Papineni et al., 2002) with the gold summary as reference but also make use of the Infor-  mation Extraction (IE) metrics from Wiseman et al. (2017) which are defined over the output of an IE system; the latter extracts entity (players, teams) and value (numbers) pairs in a summary, and then predicts the type of relation. For instance, given the pair Kansas City Royals, 9, it would predict their relation as TR (i.e., Team Runs). Training data for the IE system is obtained by checking for matches between entity, value pairs in the gold summary and entity, value, record type triplets in the table.
Data-to-text Generation with Macro Planning
Letŷ be the gold summary and y the model output. Relation Generation (RG) measures the precision and count of relations extracted from y that also appear in records r. Content Selection (CS) measures the precision and recall of relations extracted from y that are also extracted fromŷ. Content Ordering (CO) measures the normalized Damerau-Levenshtein distance between the sequences of relations extracted from y andŷ.
Data-to-text Generation with Macro Planning
We reused the IE model from Puduppully et al. (2019a) for ROTOWIRE but retrained it for MLB to improve its precision and recall. Furthermore, the implementation of Wiseman et al. (2017) computes RG, CS, and CO excluding duplicate relations. This artificially inflates the performance of models whose outputs contain repetition. We include duplicates in the computation of the IE metrics (and recreate them for all comparison systems).  (Vaswani et al., 2017) with a hierarchical attention mechanism over entities and records within entities. The models of Saleh et al. make use of additional information not present in the input (e.g., previous/next games, summary writer) and are not directly comparable to the systems in Table 2. Results for the MLB test set are in the bottom portion of Table 2. Templ has the highest RG precision and count on both datasets. This is not surprising, by design Templ is always faithful to the input. However, notice that it achieves the lowest BLEU amongst comparison systems indicating that it mostly regurgitates facts with low fluency. Macro achieves the highest RG precision amongst all neural models for ROTOWIRE and MLB. We obtain an absolute improvement of 5.9% over ENT for ROTOWIRE and 13.3% for MLB. In addition, Macro achieves the highest CS F-measure for both datasets. On ROTOWIRE, Macro achieves the highest CO score, and the highest BLEU on MLB. On ROTOWIRE, in terms of BLEU, Macro is worse than comparison models (e.g., NCP+CC or ENT). Inspection of the output showed that the opening paragraph, which mostly describes how the two teams fared, is generally shorter in Macro, leading to shorter summaries and thus lower BLEU. There is high variance in the length of the opening paragraph in the training data and Macro verbalizes the corresponding plan conservatively. Ideas such as length normalisation (Wu et al., 2016) or length control (Kikuchi et al., 2016;Takeno et al., 2017;Fan et al., 2018) could help alleviate this; however, we do not pursue them further for fair comparison with the other models.
Data-to-text Generation with Macro Planning
To study the effect of macro planning in more detail, we further compared Macro against text generation models (see Section 4.2) which are trained on verbalizations of the tabular data (and gold summaries) but do not make use of document plans or a document planning mechanism. On ROTOWIRE, the model was trained on verbalizations of players and teams, with the input arranged such that the ver-  Table 3: Evaluation of macro planning stage; content selection precision (CS-P), recall (CS-R), F-measure (CS-F) and content ordering (CO) between the inferred plans and gold plans in terms of entities and events for ROTOWIRE (RW) and MLB test sets.
Data-to-text Generation with Macro Planning
balization of the home team was followed by the visiting team, the home team players and the visiting team players. Mention of players was limited to the four best ones, following Saleh et al. (2019) (see −Plan(4) in Table 2). For MLB, we additionally include verbalizations of innings focusing on scoring plays which are likely to be discussed in game summaries (see −Plan(SP,4) in Table 2). Note that by preprocessing the input in such a way some simple form of content selection takes place simply by removing extraneous information which the model does not need to consider. Across both datasets, −Plan variants appear competitive. On ROTOWIRE −Plan(4) is better than ED+CC in terms of content selection but worse compared to ENT. On MLB, −Plan(SP,4) is again superior to ED+CC in terms of content selection but not ENT whose performance lags behind when considering RG precision. Taken together, these results confirm that verbalizing entities and events into a text sequence is effective. At the same time, we see that −Plan variants are worse than Macro across most metrics which underlines the importance of an explicit planning component. Table 3 presents intrinsic evaluation of the macro planning stage. Here, we compare the inferred macro plan with the gold macro plans, CS and CO metrics with regard to entities and events instead of relations. We see that our macro planning model (Macro) achieves high scores for CS and CO for both ROTOWIRE and MLB. We further used the CS and CO metrics to check how well the generated summary follows the (predicted) plan. We followed the steps in Section 3.2 and reverse engineered macro plans from the model summaries and compared these extracted plans with the original macro plans with regard to entities and events. We found that Macro creates summaries which follow the plan closely: for ROTOWIRE, the CS F-score and CO are greater than 98%; for MLB, the CS F-score is greater than 94% and CO is greater than 89%. We show an output summary for Macro in <P> Akinori Iwamura hit a two-run homer in the eighth inning and the Rays beat the Boston Red Sox 2-1 on Sunday to complete a three-game sweep. <P> The Rays, who have the best record in the majors, have won six of their last seven games. <P> The Rays have won four of their last five series, including three in a row against the Red Sox, who have won six of their last seven overall. <P> Dioner Navarro singled with one out in the eighth off Clay Buchholz (1-2) and moved to third on Jason Bartlett's flyout to center. Iwamura then drove a 1-1 pitch into the left-field stands for his second homer of the season. <P> Scott Dohmann (2-0) got the win in relief , striking out Manny Ramirez with runners on first and third to end the eighth. <P> Troy Percival worked the ninth for his fifth save in five opportunities. <P> Clay Buchholz (1-2) gave up two runs and three hits in eight innings. He struck out nine and walked two. <P> The Red Sox loaded the bases with one out in the fifth on a single by Coco Crisp, a wild pitch and a walk to Jed Lowrie. Jacoby Ellsbury drove in Crisp with a two-out single to center. <P> Jackson struck out four and walked three. <P> The Red Sox loaded the bases with one out in the fifth on a single by Coco Crisp, a walk to Jed Lowrie and a one-out walk to Jed Lowrie. Jackson struck out Julio Lugo, but Jacoby Ellsbury singled to center to put the Red Sox up 1-0. <P> The Red Sox threatened in the eighth when J. D. Drew drew a two-out walk against Trever Miller, but Ramirez struck out to end the inning.  
Data-to-text Generation with Macro Planning
We also asked participants to assess model output in terms of relation generation, grammaticality, coherence, and conciseness (Wiseman et al., 2017;Puduppully et al., 2019a,b), For ROTOWIRE, we compared Macro against RBF-2020 4 , ED+CC, Gold, and Templ. For MLB, we compared Macro against ENT, ED+CC, Gold, and Templ.
Data-to-text Generation with Macro Planning
We conducted our study on the Amazon Mechanical Turk (AMT) crowdsourcing platform, following best practices for human evaluation in NLG . Specifically, to ensure consistent ratings, we required crowdworkers to have an approval rating greater than 98% and a minimum of 1,000 previously completed tasks. Raters were restricted to English speaking countries (i.e., US, UK, Canada, Ireland, Australia, or NZ). Participants were allowed to provide feedback on the task or field questions (our interface accepts free text).
Data-to-text Generation with Macro Planning
In our first study, we presented crowdworkers with sentences randomly selected from summaries along with their corresponding box score (and play-  Table 5: Average number of supported (#Supp) and contradicting (#Contra) facts in game summaries and best-worst scaling evaluation (higher is better). Systems significantly different from Macro are marked with an asterisk * (using a one-way ANOVA with posthoc Tukey HSD tests; p ≤ 0.05).
Data-to-text Generation with Macro Planning
. by-play in case of MLB) and asked them to count supported and contradicting facts (ignoring hallucinations, i.e., unsupported facts). We did not require crowdworkers to be familiar with NBA or MLB. Instead, we provided a cheat sheet explaining the semantics of box score tables. In addition, we provided examples of sentences with supported/contradicting facts. We evaluated 40 summaries from the test set (20 per dataset), 4 sentences from each summary and elicited 3 responses per summary. This resulted in 40 summaries × 5 systems × 3 raters for a total of 600 tasks. Altogether 131 crowdworkers participated in this study (agreement using Krippendorff's α was 0.44 for supported and 0.42 for contradicting facts).
Data-to-text Generation with Macro Planning
As shown in Table 5, Macro yields the smallest number of contradicting facts among neural models on both datasets. On ROTOWIRE the number of contradicting facts for Macro is comparable to Gold and Templ (the difference is not statistically significant) and significantly smaller compared to RBF-2020 and ED+CC. The count of supported facts for Macro is comparable to Gold, and ED+CC, and significantly lower than Templ and RBF-2020. On MLB, Macro has significantly fewer contradicting facts than ENT and ED+CC and is comparable to Templ, and Gold (the difference is not statistically significant). The count of supported facts for Macro is comparable to Gold, ENT, ED+CC and Templ. For both datasets, Templ has the lowest number of contradicting facts. This is expected as Templ essentially parrots facts (aka records) from the table.
Data-to-text Generation with Macro Planning
We also conducted a second study to evaluate the quality of the generated summaries. We presented crowdworkers with a pair of summaries and asked them to choose the better one in terms of Grammaticality (is the summary written in well-formed English?), Coherence (is the summary well structured and well organized and does it have a natural ordering of the facts?) and Conciseness (does the summary avoid unnecessary repetition including whole sentences, facts or phrases?). We provided example summaries showcasing good and bad output. For this task, we required that the crowdworkers be able to comfortably comprehend NBA/MLB game summaries. We elicited preferences with Best-Worst Scaling (Louviere and Woodworth, 1991;Louviere et al., 2015), a method shown to be more reliable than rating scales. The score of a system is computed as the number of times it is rated best minus the number of times it is rated worst (Orme, 2009). The scores range from −100 (absolutely worst) to +100 (absolutely best). We divided the five competing systems into ten pairs of summaries and elicited ratings for 40 summaries (20 per dataset). Each summary pair was rated by 3 raters. This resulted in 40 summaries × 10 system pairs × 3 evaluation criteria × 3 raters for a total of 3,600 tasks. 206 crowdworkers participated in this task (agreement using Krippendorff's α was 0.47). Table 5, on ROTOWIRE, Macro is comparable to Gold, RBF-2020, and ED+CC in terms of Grammaticality but significantly better than Templ. In terms of Coherence, Macro is comparable to RBF-2020 and ED+CC but significantly better than Templ and significantly worse than Gold. With regard to Conciseness, Macro is comparable to Gold, RBF-2020, and ED+CC, and significantly better than Templ. On MLB, Macro is comparable to Gold in terms of Grammaticality and significantly better than ED+CC, ENT and Templ. Macro is comparable to Gold in terms of Coherence and significantly better than ED+CC, ENT and Templ. In terms of Conciseness, raters found Macro comparable to Gold and Templ and significantly better than ED+CC, and ENT. Taken together, our results show that macro planning leads to improvement in data-to-text generation in comparison to other systems for both ROTOWIRE and MLB datasets.
Data-to-text Generation with Macro Planning
In this work we presented a plan-and-generate approach for data-to-text generation which consists of a macro planning stage representing high-level document organization in terms of structure and content, followed by a text generation stage. Extensive automatic and human evaluation shows that our approach achieves better results than existing state-of-the-art models and generates summaries which are factual, coherent, and concise.
Data-to-text Generation with Macro Planning
Our results show that macro planning is more advantageous for generation tasks expected to produce longer texts with multiple discourse units, and could be easily extended to other sports domains such as cricket (Kelly et al., 2009) or American football (Barzilay and Lapata, 2005). Other approaches focusing on micro planning (Puduppully et al., 2019a;Moryossef et al., 2019) might be better tailored for generating shorter texts. There has been a surge of datasets recently focusing on singleparagraph outputs and the task of content selection such as E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and WikiBio (Lebret et al., 2016;Perez-Beltrachini and Lapata, 2018). We note that in our model content selection takes place during macro planning and text generation. The results in Table 2 show that Macro achieves the highest CS F-measure on both datasets indicating that the document as a whole and individual sentences discuss appropriate content.
Data-to-text Generation with Macro Planning
Throughout our experiments we observed that template-based systems score poorly in terms of CS (but also CO and BLEU). This is primarily due to the inflexibility of the template approach which is limited to the discussion of a fixed number of (high-scoring) players. Yet, human writers (and neural models to a certain extent), synthesize summaries taking into account the particulars of a specific game (where some players might be more important than others even if they scored less) and are able to override global defaults. Template sentences are fluent on their own, but since it is not possible to perform aggregation (Reiter, 1995), the whole summary appears stilted, it lacks coherence and variability, contributing to low BLEU scores. The template baseline is worse for MLB than RO-TOWIRE which reflects the greater difficulty to manually create a good template for MLB. Overall, we observe that neural models are more fluent and coherent, being able to learn a better ordering of facts which is in turn reflected in better CO scores.
Data-to-text Generation with Macro Planning
Despite promising results, there is ample room to improve macro planning, especially in terms of the precision of RG (see Table 2, P% column of RG). We should not underestimate that Macro must handle relatively long inputs (the average input length in the MLB development set is~3100 tokens) which are challenging for the attention mechanism. Consider the following output of our model on the MLB dataset: Ramirez's two-run double off Joe Blanton tied it in the sixth, and Brandon Moss added a two-out RBI single off Alan Embree to give Boston a 3-2 lead. Here, the name of the pitcher should have been Joe Blanton instead of Alan Embree. In fact, Alan Embree is the pitcher for the following play in the half inning. In this case, attention diffuses over the relatively long MLB macro plan, leading to inaccurate content selection. We could alleviate this problem by adopting a noisy channel decomposition (Yee et al., 2019;Yu et al., 2020), i.e., by learning two different distributions: a conditional model which provides the probability of translating a paragraph plan to text and a language model which provides an unconditional estimate of the output (i.e., the whole game summary). However, we leave this to future work.
Data-to-text Generation with Macro Planning
For ROTOWIRE, the main source of errors is the model's inability to understand numbers. For example, Macro generates the following output The Lakers were the superior shooters in this game, going 48 percent from the field and 30 percent from the three-point line, while the Jazz went 47 percent from the floor and 30 percent from beyond the arc.. Here, 30 percent should have been 24 percent for the Lakers but the language model expects a higher score for the three-point line, and since 24 is low (especially compared to 30 scored by the Jazz), it simply copies 30 scored by the Jazz instead. A mechanism for learning better representations for numbers (Wallace et al., 2019) or executing operations such as argmax or minus (Nie et al., 2018) should help alleviate this problem.
Data-to-text Generation with Macro Planning
Finally, although our focus so far has been on learning document plans from data, the decoupling of planning from generation allows to flexibly generate output according to specification. For example, we could feed the model with manually constructed macro plans, consequently controlling the information content and structure of the output summary (e.g., for generating short or long texts, or focusing on specific aspects of the game).
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Planned and Grounded Abstractive Summarization We introduce a novel training objective to neural summarization models for content planning with entity chains, and study integrating it with supervised finetuning and self-supervised pretraining objectives without altering the models themselves. As the entity chains are extracted from the reference summaries during training, our models learn to ground the generation of summaries to the entity chains found in them. Hence, we refer to this objective by FROST for its ability to try to "FReeze entity-level infOrmation in abstractive SummarizaTion with planning."
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Abstractive Summarization with Entity Chains FROST provides a very effective knob for entity-level content modification in abstractive summaries. In this paper we empirically demonstrate how FROST is critical for faithfulness by enabling the drop-prompt mechanism where we drop out hallucinated entities from the predicted content plan and prompt the decoder with this modified plan to generate faithful summaries. We further qualitatively demonstrate that FROST enables generation of summaries (i) with topical diversity by choosing different sets of entities from the source to plan what we want to discuss in the summary, and (ii) with style diversity by reordering entities in the predicted plan to get an equivalent summary but with a different entity emphasis.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
We introduce a new training objective for encoderdecoder generative models to do content planning while summarizing.
Planning with Learned Entity Prompts for Abstractive Summarization
Model Formulation. Let d be an input document, we aim to teach our model to first generate a content plan c for the summary s as p(c|d), and then generate the summary s as p(s|c, d). We define the ordered chain of entities observed in the summary s as its content plan. Instead of modeling p(c|d) and p(s|c, d) separately, we take a simpler approach, we train an encoder-decoder model to encode the document d and generate the concatenated content plan and summary sequences c; s, essentially the decoder first predicts the entity chain c and then continues predicting the summary s using both c and d. We prefix c and s with special markers "[ENTITYCHAIN]" and "[SUMMARY]", respectively, as shown in Figure 1. If s consists of multiple sentences, we use sentence markers "|||" to mark them in c. The model is trained with the standard maximum-likelihood objective generating the augmented target c; s.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Content Plans. We modified PE-GASUS (Zhang et al., 2019) to pretrain our models for entity-level content planning and summary generation. 2 In particular, we select a maximum of n most important sentences using self-ROUGE from an input document, the selected sentences work as a proxy for a human-authored abstractive summaries for the rest of the document. We construct a target by prepending the selected sentences dynamically with their entity chain. Our model is then trained to generate this target from the rest of the document. 3
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
Controlled Generation with Entity Prompts. An advantage of training to generate the summary s following the generation of the plan c using the same decoder is that now during the inference time the decoder can be easily prompted with any desired content plan c to control the content in the output summary s with a probability of p(s |c , d).
Planning with Learned Entity Prompts for Abstractive Summarization
In Section 5, we prompt our decoder with modified content plans to mitigate hallucinations and to generate diverse summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining Datasets. Following Zhang et al. (2019), our model pretraining also relied on two large web corpus which were processed to look like plain text: (i) C4 (Raffel et al., 2019) is composed of 350M Web-pages that were obtained from Common Crawl, and (ii) HugeNews (Zhang et al., 2019) is composed of 1.5B news and newslike articles from 2013-2019. This dataset includes articles from multiple allowlisted sources including news publishers, high-school newspapers and blogs.
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
Entity Chain Annotation. We experimented with entity chains consisting of named entities,  Table 1: Abstractive summarization datasets studied in this work. We report on their train/validation/test sizes and how they were processed (cased/uncased). To better understand the effect of summary planning with entity chains, we report on average number of sentences (avg. sent.), average number of entities (avg. ent.) and average number of unique entities (avg. uniq. ent.), per target in validation sets. We also report on total number of named entities, date and number in target summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
dates and numbers. We annotate the whole document in the pretraining datasets to allow dynamic construction of summaries and their entity chains during pretraining. We only annotate the reference summaries for the finetuning summarization datasets. We use a BERT-based tagger trained on CoNLL-2003 data (Tjong Kim Sang andDe Meulder, 2003) to identify named entities, and regular expressions to identify dates and numbers (Guu et al., 2020). See Table 1 for the number of named entities, dates and numbers found in different datasets. Table 6 and 7 in Appendix A present hyperparameters used for pretraining and finetuning PE-GASUS and FROST base and large sized models. We used Cloud TPU v3 accelerators for training.
Planning with Learned Entity Prompts for Abstractive Summarization
We evaluate our FROST models on ROUGE, entity specificity, entity planning, faithfulness using automatic and human evaluations, and overall quality by humans. Our models predict a summary plan in the form of an entity chain, followed by a summary. All evaluations are done on the summary, the predicted entity chains are stripped out.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level ROUGE. We report ROUGE F1 scores (Lin and Hovy, 2003) to assess generated summaries. 7 Entity Planning. We evaluate the quality of content plans learned by our models by assessing the entities and the order in which they appear in the summary. We annotate both predicted and reference summaries with named entities, dates and numbers, and report on ROUGE F1 scores for entity chains found in the predicted summaries against corresponding reference entity chains. 7 We lowercased candidate and reference summaries and used pyrouge with parameters "-a -c 95 -m -n 4 -w 1.2." Entity Specificity. We compare entities in predicted and reference summaries, and report average entity F1-scores (ENTF1) in predicted summaries. We lower-case and remove duplicate entities; we report on the exact match with reference entities.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
We further assess summary faithfulness by humans. Our annotators, proficient in English, were tasked to read the document carefully and then grade its summary on a scale of 1-5 (fully unfaithful, somewhat unfaithful, 50-50, somewhat faithful and fully faithful); a summary is "fully faithful" if all of its content is fully supported or can be inferred from the document.
Planning with Learned Entity Prompts for Abstractive Summarization
Overall Summary Quality. Finally we assess the overall quality of summaries by humans. We ask our annotators to read the document carefully and then grade its summary on a scale of 1-5 (poor summary, better than poor, okay summary, better than okay and great summary). To improve the annotator agreement, we clearly define 4 features that a "great summary" should have: Relevant (summary should have most important information from the document), Accurate (summary "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell.
Planning with Learned Entity Prompts for Abstractive Summarization
Original Target: "Frozen," the latest Disney musical, preaches the importance of embracing your true nature. It depicts fearless Princess Anna who joins forces with mountaineer Kristoff and his reindeer sidekick to find estranged sister, Snow Queen Elsa, and break her icy spell. should be accurate with respect to the document or the expert knowledge), 8 Concise (summary should not have redundant or less-important content), and Fluent (summary should be grammatically correct and coherent). A great summary should pass on all 4 features, a better than okay should have 3 out of 4 features, and so on.
Planning with Learned Entity Prompts for Abstractive Summarization
For both human assessments, we collected 3 ratings for each (document, summary) pair and report the average of all assigned labels (1-5) to a system. We conducted 3 pilot studies for each setup to train our annotators with examples to improve their understanding of the task. Additionally, extra measures were taken to improve agreements among our annotators. For example, for the faithfulness assessment, when one of somewhat unfaithful, 50-50 and somewhat faithful were selected, annotators were asked to also specify what was faithful or unfaithful in the summary. Similarly for the overall quality assessment, when one of better than poor, okay summary and better than okay were selected, they were asked to list all features on which the candidate summary fails. 
Planning with Learned Entity Prompts for Abstractive Summarization
Sentence-Level vs Summary-Level Planning.
Planning with Learned Entity Prompts for Abstractive Summarization
Let the summary s consists of m sentences s 1 . . . s m and c i be the entity chain for the sentence s i , we consider generating the summary s in two ways. Sentence-level approach trains a model to generate s by consecutively generating the sentence-level content plan c i followed by its summary sentence s i with a probability
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
in Figure 3.
Planning with Learned Entity Prompts for Abstractive Summarization
We found that the summary-level planning is not only superior to the sentence-level planning, it helps with generating better quality summaries in terms of summary-level ROUGE (RL-Sum), entity planning (R2-EPlan) and entity specificity (ENTF1). For the rest of the pretraining or finetuning experiments, we focused on summary-level planning unless specified otherwise.
Planning with Learned Entity Prompts for Abstractive Summarization
Pretraining for Planning from Scratch. We pretrained three base-sized models from scratch: PEGASUS-base pretrained with the original gapsentence objective (Zhang et al., 2019) for 1.5m steps, FROST(F)-base pretrained with the gapsentences prepended with their entity chain for 1.5m steps, and FROST(P+F)-base first pretrained with the PEGASUS objective for 1m steps and then with the FROST objective for another 500k steps. Maximum input and output lengths were set to 512 and 256 sentencepieces during pretraining, respectively. We finetuned these three models on the XSum dataset. Results are shown in Figure 4.
Planning with Learned Entity Prompts for Abstractive Summarization
First of all, our results confirm that the pretraining for planning is beneficial for summarization; both FROST(F) and FROST(P+F) learned better content plans (in terms of entity chainlevel ROUGE and ENTF1) than PEGASUS without sacrificing the summary quality (in terms of summary-level ROUGE) Figure 5: Finetuning results on the XSum (in blue) and CNN/DailyMail (in red) validation sets at various steps during pretraining FROST-Large. Instead of pretraining from scratch, we start with a PEGASUS-Large checkpoint, and continue pretraining for additional 1.5m steps with the planning objective. We report finetuning results for the PEGASUS finetuned baseline and our models at 0.1m, 1m and 1.5m steps. the pretraining for planning and summarization is more effective when started with summarization pretrained models such as PEGASUS than when trained jointly from scratch. Hence, for our future pretraining experiments we initialize our model with existing PEGASUS checkpoint and continue pretraining for content planning and refer to them as FROST for simplicity.
Planning with Learned Entity Prompts for Abstractive Summarization
Based on our findings, we pretrain a large FROST model for planning for summarization starting with an existing PEGASUS-Large checkpoint. Figure 5 presents results for finetuning these models on the XSum and CNN/DailyMail datasets at various steps during pretraining.
Planning with Learned Entity Prompts for Abstractive Summarization
Similar to our findings in Figure 4, our results with large models further confirm the advantages of the pretraining for planning; improvement over the PEGASUS baseline were larger for the XSum dataset than for the CNN/DailyMail dataset. We achieved the best performance on both datasets at the 1m pretraining step. We use the FROST model at 1m pretraining step for our finetuning experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Table 2 presents our final results from finetuning our large models on summarization datasets. For each dataset, we first report results from earlier work directly taken from corresponding papers; our results are in the bottom blocks for each dataset. We finetune our own PEGASUS using the standard approach (document to summary). We report results from FROST (ECP; Entity    (Dou et al., 2020) and PEGASUS (Zhang et al., 2019). We also include commonly reported LEAD-n baselines (selecting top n sentences from the document) and extractive oracle (EXT-ORACLE; selecting best set of sentences from the document with the most overlapping content with its reference summary), for the CNN/DailyMail and XSum datasets. EXT-ORACLEs are masked with * and are not directly comparable. All results are reported on the publicly available test sets.
Planning with Learned Entity Prompts for Abstractive Summarization
Chain Planning), i.e., PEGASUS finetuned with the FROST objective (document to content plan and summary). Finally, we report on FROST (ECPP; Entity Chain Planning with Pretraining), our models both pretrained and finetuned with the FROST objective.
Planning with Learned Entity Prompts for Abstractive Summarization
We find that even simply finetuning an existing PEGASUS pretrained model to do content planning and summarization (as in FROST (ECP)) leads to improvements in entity specificity (ENTF1) and the quality of entity plans in summaries (Entity Planning ROUGE), across all datasets. In fact in some cases, better content plans lead to large improvements in summary-level ROUGE as well, for example, FROST (ECP) improve on PEGASUS from 44.05/21.69/40.98 to 44.85/21.83/41.80 on ROUGE scores for CNN/DailyMail summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
We demonstrate in two ways the planning with entity chains can be useful in mitigating hallucinations in summarization: data filtering and dropprompt mechanism with entity chains. We focused on the XSum dataset for these experiments.
Planning with Learned Entity Prompts for Abstractive Summarization
Data Filtering using Entity Chains. During training FROST prepends reference summaries with their entity chains to better plan them. We leverage this to filter the dataset to only keep examples where summaries have fully extractive entity chains; an entity chain is fully extractive if all entities in it can be found in the input document. It's notable that this filtered dataset will not contain novel entities in the summary targets. FROST models trained on this data will ground the summary generation to extracted entity chains while allowing abstraction for non-entity related generations. The resulting XSum dataset has 62.7k/3.5k/3.5k train/validation/test instances. We finetune our models from Table 2 on this filtered dataset and report their performance on the filtered test set (3.5k). We also evaluate them on the rest of the test set (7.8k) where reference entity chains are not fully extractive to their input documents. 9 See Figure 6 for examples. 9 Such data divergence is not unique to XSum, around 30% of CNN/DailyMail summaries also have reference entity chains that are not fully extractive to their input documents. Writing these summaries requires either document-level inference or the background knowledge of the input documents to generate novel entities or numbers.  Table 3: Performance on XSum summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (3.5k) and the rest of the test set with novel entities in the targets (7.8k). Best results are bold faced.
Planning with Learned Entity Prompts for Abstractive Summarization
Drop-Prompt Mechanism. FROST decoders are trained to generate the summary s following the generation of its plan c. To improve faithfulness, we take the predictions from FROST (ECPP) 10 and modify the generated plan c to c drop by dropping entities (or parts of them) that are not found in the input document. We then prompt our decoder with c drop to generate a new summary. We conduct this with both models, one trained on the full dataset and another on the filtered subset. We also report results for the oracle entity chain prompts c oracle for a comparison.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Summary-level ROUGE, Entity Planning and Specificity. Results are presented in Table 3. First of all, similar to our results in Table 2 on the full XSum test set, our results with the models trained on the original dataset further validate that the pretraining and finetuning with the content planning objective is equally useful for both test sets: one where entities are simply copied from the input documents and another where novel entities were inferred, to generate corresponding targets. The data filtering using entity chains are particularly useful for test cases with extractive reference entity chains; models trained on the filtered data lead to much higher ENTF1 (e.g., 65.97 vs 63.03, for FROST (ECPP)) and higher quality entity plans (entitly-level ROUGE of 66.43/30.00/62.48 vs 63.73/30.08/60.10 for FROST (ECPP)), without sacrificing the summary quality substantially (summary-level ROUGE of 44.87/22.05/36.79 vs 10 The drop-prompt mechanism can be used with FROST (ECP) also, we have simply used the best among FROST (ECP) and FROST (ECPP) on the XSum set in terms of summarylevel ROUGE, entity planning and ENTF1.   Table 3 and 4. We highlight entities in orange that are not faithful to the input document. Entities in green are faithful to the input document. 45.08/22.01/36.80 for FROST (ECPP)). Unsurprisingly, these models don't do well on the test set with non-extractive reference entity chains, such examples were not seen during training.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
Effect on Faithfulness. The planning objective in FROST itself does not ensure faithfulness; planning is done with the entities in the target, if the target is noisy, its plan may also be noisy (see Figure 6 for predicted entity chains with hallucinated entities). But the planning with the entity chains facilitates the data filtering and the drop-prompt mechanism, using entity chains. Table 4 presents our results assessing them for faithfulness. We randomly selected 50 documents for each of test sets (with extractive or non-extractive reference entity chains) and assessed their summaries from all 8 systems (except with c oracle ) from Table 3.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism is also very powerful in improving faithfulness. Irrespective of which training data were used to train the models and which test sets they were evaluated on, we see improvements in entailment scores, ENT-PREC and the human assessment of faithfulness across the board, except for a single case where the entailment score slightly drops from 0.465 to 0.453. Figure 6 demonstrates how we drop hallucinated entities 'Liam', 'two' and 'Conor' from entity chains to enforce models to generate faithful summaries grounded to the modified entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We achieve the best performance in terms of ENTPREC and the human assessment of faithfulness when both the data filtering and the dropprompt mechanism were used. We achieve 0.943 for ENTPREC and 4.41 for faithfulness for the test set with extractive entity chains only, and 0.826 for ENTPREC and 3.85 for faithfulness for the test set with non-extractive entity chains. Humans also found that the predictions from these models were the best in terms of overall summary quality.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, we carried out pairwise comparisons for human assessments for faithfulness and overall summary quality for all models (using a oneway ANOVA with post-hoc Tukey HSD tests; p < 0.01). Interestingly, differences among all model pairs for both faithfulness and overall summary quality are insignificant when evaluated on the test set with extractive reference entity chains only. On the more challenging test set with non-extractive entity chains, FROST (ECPP) trained on the fil-GOLD: [ENTITYCHAIN] BP | Lord Browne | North Sea [Summary] Former BP chief executive Lord Browne has warned North Sea oil operators' costs must fall in order for them to compete globally.
Planning with Learned Entity Prompts for Abstractive Summarization
Browne, the former boss of BP, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST (ECPP), c mod , Style diversity [ENTITYCHAIN] BP | Browne [Summary] The former boss of BP, Lord Browne, has warned that some oil companies could go out of business if costs continue to rise.
Planning with Learned Entity Prompts for Abstractive Summarization
[ENTITYCHAIN] BBC | BP | Browne [Summary] In an interview with the BBC, former BP chief executive Lord Browne has warned the oil and gas industry is in danger of "going to the wall".  tered training set and with the drop-prompt mechanism is significantly better than all other models except for (i) FROST (ECPP) trained on the filtered training set but without the drop-prompt mechanism and (ii) FROST (ECPP) trained on the original training set and with the drop-prompt mechanism, for both faithfulness and overall summary quality. All other pairwise differences are insignificant.
Planning with Learned Entity Prompts for Abstractive Summarization
FROST provides a handle to easily control or manipulate content plan for predicted summaries. In Figure 6, we saw how we can use this to control entity-level hallucinations, but the strength of FROST models go beyond this. Figure 7 shows how FROST models can be effectively used in generating summaries with different entity focus by simply modifying the entity prompt. Future work will focus on how to leverage FROST for synthetic data generation for training improvements.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
Planning with Learned Entity Prompts for Abstractive Summarization
There are several advantages of FROST-style decoding. Unlike encoder-guided control summarization systems, FROST models can be used in a usual way to generate summaries for input documents without relying on external systems to augment them during inference. Additionally, users can modify the generated entity prompts or provide their desired entity prompts to control the generated summaries (see Section 5.3 and 5.4).
Planning with Learned Entity Prompts for Abstractive Summarization
To the best of our knowledge, FROST is the first decoder-prompted control summarization model. Table 5 show that the prompting the decoder with entity prompts are more effective in generating high-quality and grounded summaries compared to when entity prompts are encoded with the input, especially for abstractive summaries; both PEGASUS versions (d; c oracle and c oracle ; d) perform inferior to FROST (d → c oracle ; s) in terms of summary-level ROUGE, entity-level ROUGE and ENTF1 on the XSum dataset. CTRLsum (k oracle ; d → s) achieves better ROUGE scores than FROST (d → c oracle ; s) on CNN/DailyMail (64.72/40.56/61.02 vs 61.85/38.95/59.00). This is not surprising as the oracle keywords k oracle in CTRLSum retain most words from the summary found in the source document; as such due to the extractive nature of CNN/DailyMail summaries, k oracle tend to be closer to the surface forms of the reference summaries (ROUGE scores between k oracle and reference summaries are 52.70/17.37/44.37). The difference in ROUGE scores narrows down between CTRLsum and FROST to 45.65/22.35/42.50 vs 45.11/22.11/42.01 with automatically extracted keywords. FROST outperforms CTRLsum on entity planning and specificity. It is not clear how well CTRLsum's keyword extraction system will generalize to more abstractive datasets where words in reference summaries are often not found in the source document.
Planning with Learned Entity Prompts for Abstractive Summarization
Finally, our FROST models work as a better microplanner and produce concise summaries than those generated by systems without doing content   Figure 10, Appendix C.
Planning with Learned Entity Prompts for Abstractive Summarization
In this work, we proposed the use of entity chains, both during pretraining and finetuning, to better plan and ground the generation of abstractive summaries. Our approach achieves state-of-theart ROUGE results in XSum and SAMSum, and competitive results in CNN/Dailymail. Compared to other guided summarization models, CTRL-Sum and GSum, which perform slightly better in CNN/Dailymail, our approach is drastically simpler to implement, trains by augmenting the targets only, utilizes no additional parameters than the baseline PEGASUS model, and does not rely on any external systems to augment the inputs during inference. We further demonstrate that by modifying the entity chain prompts, we can easily control hallucinations in summaries. Table 6 and 7 present hyperparameters used for pretraining and finetuning PEGASUS and FROST base and large sized models. 
Planning with Learned Entity Prompts for Abstractive Summarization
Figure 10 presents example predictions from FROST and CTRLSum along with their entity prompts and keywords used for guidance, respectively.    
Planning with Learned Entity Prompts for Abstractive Summarization
We experimented with entity chains consisting of named entities, dates and numbers. We finetuned PEGASUS-large models on the XSum dataset to predict the entity chain and then continue predicting the summary. Results are presented in Figure 11. We found that the entity chains with all named entities, dates and numbers performed best for entity level measures (entity chain-level ROUGE and ENTF1) without sacrificing the final summary quality (summary-level ROUGE) significantly. Following this, we always use entity chains with named entities, dates and numbers to have better control over them.
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-Level Entity Chains Table 9 presents detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.
Planning with Learned Entity Prompts for Abstractive Summarization
Here, we present results from controlling hallucinations on CNN/DailyMail summaries (i) by training models on the dataset with extractive entity chains only and (ii) by modifying the predicted entity chains to keep only the supported entities. Table 10 and 11 present results from these mod- Figure 9: Instructions for human evaluations for overall quality of summaries.
Planning with Learned Entity Prompts for Abstractive Summarization
GOLD: Theia, a bully breed mix, was apparently hit by a car, whacked with a hammer and buried in a field. "She's a true miracle dog and she deserves a good life," says Sara Mellado, who is looking for a home for Theia.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Oracle keywords) hit car apparently whacked hammer buried field | dog bully breed mix Theia | true miracle deserves good life (Summary) A bully breed mix named Theia was hit by a car, apparently whacked with a hammer and buried in a field. Four days after her apparent death, the dog managed to stagger to a nearby farm. "She's a true miracle dog and she deserves a good life," the dog's foster mother says. FROST: (Oracle Entity Prompt) Theia ||| Sara Mellado | Theia (Summary) The dog, now named Theia, was apparently hit by a car and buried. Sara Mellado is raising money to help pay for Theia's medical care.
Planning with Learned Entity Prompts for Abstractive Summarization
CTRLSum: (Predicted keywords) stray pooch Washington State hit car head hammer mercy killing buried field survive | dog Theia | found (Summary) A stray pooch in Washington State was hit by a car and buried in a field. The dog was apparently hit on the head with a hammer in a mercy killing. She managed to survive and was found by a worker who took her to a vet. Theia is now being cared for at a veterinary hospital. FROST: (Predicted Entity Prompt) Washington State ||| Theia ||| (Summary) A stray dog in Washington State was apparently hit by a car and buried. The dog, now named Theia, survived but still needs surgery. A fundraising page has been set up to help pay for her care.    Table 9: Detailed results for ablations on sentence-level vs summary-level entity chains in Figure 3. We report results on CNN/DailyMail validation set.  Table 10: Performance on CNN/DailyMail summaries when models are trained on the dataset with extractive entity chains only (data filtering, the bottom block) and when novel entities are dropped from the predicted entity chains using the drop-prompt mechanism (c drop ). Results with * are with oracle entity chain prompts. We report results on the filtered test set with extractive entity chains only (7.8k) and the rest of the test set with novel entities in the targets (3.7k).  Figure 11: Ablations on entity chains with named entities, dates and numbers. We report summary-level ROUGE-L (RL-Sum), entity chain-level ROUGE-2 (R2-EPlan) and ENTF1 on the XSum validation set. Similar observations were made for other measures, see detailed results in Table 8.  Table 11: Faithfulness assessment using automatic and human evaluations, and overall quality assessment by humans for models presented in Table 10 for the CNN/DailyMail dataset (Test set with Non-Extractive Entity Chains Only). 
Data-to-text Generation with Variational Sequential Planning
Data-to-text generation refers to the task of generating textual output from non-linguistic input such as database tables, spreadsheets, or simulations of physical systems Dale, 1997, 2000;Gatt and Krahmer, 2018). Recent progress in this area (Mei et al., 2016;Lebret et al., 2016;Wiseman et al., 2017) has been greatly facilitated by the very successful encoder-decoder neural architecture (Sutskever et al., 2014) and the development of large scale datasets. ROTOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) constitute such examples. They both focus on the sports domain which has historically drawn attention in the generation community (Barzilay and Lapata, 2005;Tanaka-Ishii et al., 1998;Robin, 1994) and consider the problem of generating long target texts from database records. Figure 1 (reproduced from Puduppully and Lapata, 2021) provides a sample from the MLB dataset which pairs human written summaries (Table C) with major league baseball game statistics. These are mostly scores (collectively referred to as box score) which summarize the performance of teams and players, e.g., batters, pitchers, or fielders (Table A) and a play-by-play description of the most important events in the game (Table B). Game summaries in MLB are relatively long (540 tokens on average) with multiple paragraphs (15 on average). The complexity of the input and the length of the game summaries pose various challenges to neural models which, despite producing fluent output, are often imprecise, prone to hallucinations, and display poor content selection (Wiseman et al., 2017). Attempts to address these issues have seen the development of special-purpose modules which keep track of salient entities (Iso et al., 2019;Puduppully et al., 2019b), determine which records (see the rows in Tables A and B) should be mentioned in a sentence and in which order (Puduppully et al., 2019a;, and reconceptualize the input in terms of paragraph plans (Puduppully and Lapata, 2021) to facilitate document-level planning (see Table D in Figure 1). Specifically, Puduppully and Lapata (2021) advocate the use of macro plans for improving the organization of document content and structure. A macro plan is a sequence of paragraph plans, and each paragraph plan corresponds to a document paragraph. A macro plan is shown in Table E (Figure 1). Examples of paragraph plans are given in Table D where <V(entity)> verbalizes records pertaining to entities and <V(inning-T/B)> verbalizes records for the Top/Bottom side of an inning. Verbalizations are sequences of record types followed by their values. Document paragraphs are shown in Table C and have the same color as their corresponding plans in Table E. During training, Puduppully and Lapata (2021) learn to predict a macro plan from a pool of paragraph plans, and produce a game summary based on it. Continuing with our example in Figure 1, 
Data-to-text Generation with Variational Sequential Planning
KANSAS CITY, Mo. -Brad Keller kept up his recent pitching surge with another strong outing. <P> Keller gave up a home run to the first batter of the game -Cedric Mullins -but quickly settled in to pitch eight strong innings in the Kansas City Royals' 9-2 win over the Baltimore Orioles in a matchup of the teams with the worst records in the majors. <P> Keller (7-5) gave up two runs and four hits with two walks and four strikeouts to improve to 3-0 with a 2.16 ERA in his last four starts. <P> Ryan O'Hearn homered among his three hits and drove in four runs, Whit Merrifield scored three runs, and Hunter Dozier and Cam Gallagher also went deep to help the Royals win for the fifth time in six games on their current homestand. <P> With the score tied 1-1 in the fourth, Andrew Cashner (4-13) gave up a sacrifice fly to Merrifield after loading the bases on two walks and a single. Dozier led off the fifth inning with a 423-foot home run to left field to make it 3-1. <P> The Orioles pulled within a run in the sixth when Mullins led off with a double just beyond the reach of Dozier at third, advanced to third on a fly ball and scored on Trey Mancini's sacrifice fly to the wall in right. <P> . . .   Table A is typically referred to as box score. It summarizes the data of the game per team and player. Table B reports statistics pertaining to innings or play-by-play scores. Table C contains the game summary. Paragraphs in Table C are separated with <P> delimiters. Table D contains paragraph plans obtained from Tables A  and B. Paragraph plans in the first column correspond to a single entity or event. Paragraph plans in the second column describe combinations of entities or events. <V(entity)> verbalizes records pertaining to entities and <V(inning-T/B)> verbalizes records for the Top/Bottom side of an inning. Paragraph plans correspond to paragraphs in Table C. Table E contains the macro plan for the document in Table C. A macro plan is a sequence of paragraph plans. Plan-document correspondences are highlighted using the same color.
Data-to-text Generation with Variational Sequential Planning
from paragraph plans (D), to give rise to game summary (C).
Data-to-text Generation with Variational Sequential Planning
The intermediate macro plan renders generation more interpretable (differences in the output can be explained by differences in macro planning). It also makes modeling easier, the input is no longer a complicated table but a sequence of paragraph plans which in turn allows us to treat data-to-text generation as a sequence-to-sequence learning problem. Nevertheless, decoding to a long document remains challenging for at least two rea-sons. Firstly, the macro plan may be encoded as a sequence but a very long one (more than 3,000 tokens) which the decoder has to attend to at each time step in order to generate a summary tokenby-token. Secondly, the prediction of the macro plan is conditioned solely on the input (i.e., pool of paragraph plans (D) in Figure 1) and does not make use of information present in the summaries. We hypothesize that planning would be more accurate were it to consider information available in the table (and corresponding paragraph plans) and the generated summary, more so because the plans are coarse-grained and there is a one-to-many relationship between a paragraph plan and its realization. For example, we can see that the plan for <V(B.Keller)> results in two very different realizations in the summary in Figure 1 (see first and third paragraph).
Data-to-text Generation with Variational Sequential Planning
In this work, we present a model which interleaves macro planning with text generation (see Figure 2 for a sketch of the approach). We begin by selecting a plan from a pool of paragraph plans (see Table D in Figure 1), and generate the first paragraph by conditioning on it. We select the next plan by conditioning on the previous plan and the previously generated paragraph. We generate the next paragraph by conditioning on the currently selected plan, the previously predicted plan, and generated paragraph. We repeat this process until the final paragraph plan is predicted. We model the selection of paragraph plans as a sequential latent variable process which we argue is intuitive since content planing is inherently latent. Contrary to Puduppully and Lapata (2021), we do not a priori decide on a global macro plan. Rather our planning process is incremental and as a result less rigid. Planning is informed by generation and vice versa, which we argue should be mutually beneficial (they are conditioned on each other).
Data-to-text Generation with Variational Sequential Planning
During training, the sequential latent model can better leverage the summary to render paragraph plan selection more accurate and take previous decisions into account. We hypothesize that the interdependence between planning and generation allows the model to cope with diversity. In general, there can be many ways in which the input table can be described in the output summary, i.e., different plans give rise to equally valid game summaries. The summary in Figure 1 (Table C) focuses on the performance of Brad Keller who is a high scoring pitcher (first three paragraphs). An equally plausible summary might have discussed a high scoring batter first (e.g., Ryan O'Hearn). Also notice that the summary describes innings in chronological order. However, another ordering might have been equally plausible, for example, describing innings where the highest runs are scored first or innings which are important in flipping the outcome of the match. In the face of such diversity, there may never be enough data to learn an accurate global plan. It is easier to select a paragraph plan from the pool once some of the summary is known, and different plans can be predicted for the same input. In addition, the proposed model is end-to-end differentiable and gradients for summary prediction also inform plan prediction.
Data-to-text Generation with Variational Sequential Planning
Our contributions can be summarized as follows: (1) we decompose data-to-text generation into sequential plan selection and paragraph generation. The two processes are interleaved and generation proceeds incrementally. We look at what has been already generated, make a plan on what to discuss next, realize the plan, and repeat; (2) in contrast to previous models (Puduppully et al., 2019a;Puduppully and Lapata, 2021) where content plans are monolithic and determined in advance, our approach is more flexible, it simplifies modeling (we do not need to learn alignments between paragraph plans and summary paragraphs), and leads to sample efficiency in low resource scenarios; (3) our approach scales better for tasks involving generation of long multi-paragraph texts, as we do not need to specify the document plan in advance; (4) experimental results on English and German ROTOWIRE (Wiseman et al., 2017;Hayashi et al., 2019), and MLB (Puduppully et al., 2019b) show that our model is well-suited to long-form generation and generates more factual, coherent, and less repetitive output compared to strong baselines.
Data-to-text Generation with Variational Sequential Planning
We share our code and models in the hope of being useful for other tasks (e.g., story generation, summarization) 1 .
Data-to-text Generation with Variational Sequential Planning
Following Puduppully and Lapata (2021), we assume that at training time our model has access to a pool of paragraph plans E (see Table D in Figure 1) which represent a clustering of records. We explain how paragraph plans are created from tabular input in Section 4. Given E, we aim to generate a sequence of paragraphs y = [y 1 , ..., y T ] that describe the data following a sequence of chosen plans z = [z 1 , ..., z T ]. Let y t denote a paragraph, which can consist of multiple sentences, and T the count of paragraphs in a summary. With a slight abuse of notation, superscripts denote indices rather than exponentiation. So, y t i refers to the i-th word in the t-th paragraph. A plan z = [z 1 , ..., z T ] is a
Data-to-text Generation with Variational Sequential Planning
Step 1 to t-1
Data-to-text Generation with Variational Sequential Planning
Step t
Data-to-text Generation with Variational Sequential Planning
Step t+1 and after list of discrete variables where z t = j means that we choose the j-th item from pool E of candidate plans to guide the generation of paragraph y t .
Data-to-text Generation with Variational Sequential Planning
The core technique of our model is learning the sequence of latent plans that guides long document generation. We consider a conditional generation setting where the input E is a set of paragraph plans and the output y 1:T are textual paragraphs verbalizing the selected sequence z = z 1:T . Our goal is to induce variables z that indicate which paragraphs are being talked about and in which order. Similar to previous work (Li and Rush, 2020;Fu et al., 2020), we model this process as a conditional generative model that produces both y and z and factorizes as:
Data-to-text Generation with Variational Sequential Planning
where θ denotes the model parameters and < t all indices smaller than t. We believe this formulation is intuitive, simulating incremental document generation: inspect y <t (what has been already said), make a plan z t about what to say next, realize this plan by generating a new paragraph y t , and so on.
Data-to-text Generation with Variational Sequential Planning
Inference Model We are interested in the posterior distribution p θ (z|y, E), i.e., the probability over plan sequences z for a known text y and input E. This distribution is intractable to compute in general as the summation of all possible plan sequences z is exponentially complex:
Data-to-text Generation with Variational Sequential Planning
We use variational inference (Kingma and Welling, 2014;Rezende et al., 2014) to approximate the posterior with a parametrized distribution q φ (z|y, E) from which we sample values of z that are likely to produce y (see Doersch 2016 for a tutorial on this topic). Specifically, we employ an autoregressive inference model factorized as:
Data-to-text Generation with Variational Sequential Planning
Note that a major difference between q above and p in Equation (1) is that p generates y t under the guidance of z t (conceptually z t → y t ) while q infers z t given observed y t (conceptually y t → z t ).
Data-to-text Generation with Variational Sequential Planning
Neural Parametrization At step t, we start with the encoding of previous paragraphs y <t and plans z <t (see Figure 3 left). Following Yang et al. (2016), we use a Bi-directional LSTM (BiLSTM) with a self-attention layer to encode paragraph y t as a vector r t y at step t:
Data-to-text Generation with Variational Sequential Planning
where q text is a trainable query vector, which is randomly initialized and learnt along with the rest of the parameters. Attn(·) returns the attention probability and output vector over BiLSTM representation y t with query vector q text . 2 Our model uses the output vector. Next, we encode r <t y with LSTM text as:
Data-to-text Generation with Variational Sequential Planning
We encode candidate plans in pool E = [e 1 , .., e N ] with a BiLSTM, similar to the paragraph encoding shown in Equation (4), and select one of them at each step. Let r t z denote a plan embedding at step t. We encode r <t z using LSTM plan as:
Data-to-text Generation with Variational Sequential Planning
The currently selected plan is parametrized as:
Data-to-text Generation with Variational Sequential Planning
where h t−1 summarizes information in y <t and z <t , FF plan (·) denotes a feed-forward layer, and Attn(·) returns the attention probability (and output vector) of choosing a plan from E with current state h t−1 . Here, we use the attention distribution, which serves essentially as a copy mechanism. Then, a plan z t is sampled from p (we use greedy decoding in our experiments), and its representation r t z is used to update LSTM plan (Figure 3 right):
Data-to-text Generation with Variational Sequential Planning
We guide the generation of y t with current plan z t and decode each word y t i sequentially with an LSTM gen decoder which makes use of beam search. Let s i denote the i-th decoder state (initialized with the plan encoding). We update it as:
Data-to-text Generation with Variational Sequential Planning
Note that we feed h t−1 y , representing the context of previous paragraphs, as additional input similar to Serban et al. (2017). Let r t z,1 , ..., r t z,l denote the encoding of tokens of the current plan where r t z,k is the output of the BiLSTM plan encoder and l the length of the chosen plan. We generate the next word as:
Data-to-text Generation with Variational Sequential Planning
where c denotes the context vector. In Equation 11, we use the output vector from Attn(·). FF gen (·) represents a feed-forward layer. In addition, we equip the decoder with copy attention (See et al., 2017) to enable copying tokens from z t . As part of this, we learn a probability for copy based on s i (Gehrmann et al., 2018). Once paragraph y t has been generated, we obtain its encoding r t y with
Data-to-text Generation with Variational Sequential Planning
Equation (4), and update LSTM text (Figure 3 middle):
Data-to-text Generation with Variational Sequential Planning
We parametrize the variational model so that it shares the LSTMs for encoding y and E with the generative model:
Data-to-text Generation with Variational Sequential Planning
where FF v (·) represents a feed-forward layer. Note that Equation (14) differs from Equation (7) in that it uses the updated h t y instead of the previous h t−1 y because now y t is observed. The variational distribution is again parametrized by the attention probability. Essentially, p and q are strongly tied to each other with the shared LSTM encoders.
Data-to-text Generation with Variational Sequential Planning
Although we primarily focus on the inference, and how the latent plan can improve the generation of long documents, we note that the model sketched above could be parametrized differently, e.g., by replacing the encoder and decoder with pretrained language models like BART (Lewis et al., 2020). However, we leave this to future work.
Data-to-text Generation with Variational Sequential Planning
Training We optimize the standard evidence lower bound (ELBO) loss:
Data-to-text Generation with Variational Sequential Planning
where log p θ (y|E) is the log-evidence from the data, and D(q φ (z|y, E) p θ (z|y, E)) is the Kullback-Leibler divergence between q φ and the true posterior p θ . The objective eventually decomposes to a summation of the reconstruction probability p θ (y t |·) and the ratio between p θ (z t |·) and q φ (z t |·) at each step.
Data-to-text Generation with Variational Sequential Planning
Advantageously, we can exploit oracle plans (see Table E in Figure 1 and the description in Section 4 for how these were created) to obtain weak labels z * which we use as distant supervision to the inference model: Such distant supervision is essential for stabilizing training (it would be extremely challenging to optimize the model in a fully unsupervised way) and for mitigating posterior collapse. We use Gumbel-Softmax (Maddison et al., 2017;Jang et al., 2017) for differentiable sampling (reparameterization) from q. The model is trained with scheduled sampling , and follows the curriculum learning strategy using linear decay scheduling. During earlier stages of training predicted plans are less accurate, and we thus sample from oracle plans at a rate which decays linearly with training:
Data-to-text Generation with Variational Sequential Planning
where c is the slope of the decay at training step k.
Data-to-text Generation with Variational Sequential Planning
Data We performed experiments on the RO-TOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) datasets and the German RO-TOWIRE provided as part of the WNGT 2020 DGT shared task on "Document-Level Generation and Translation" (Hayashi et al., 2019). Statistics on these datasets are shown in All three datasets were preprocessed following the method of Puduppully and Lapata (2021). A paragraph plan for an entity is constructed by verbalizing its records in a fixed sequence of record type followed by its value. For example, pitcher B.Keller from Figure 1 would be verbalized as <PLAYER> B.Keller <H/V> V <W> 7 <L> 5 <IP> 8 <PH> 4 . . . . We denote this using the shorthand <V(B.Keller)>. The paragraph plan for an event is the verbalization of the players in the event followed by the verbalization of play-byplays. Candidate paragraph plans E are obtained by enumerating entities and events and their combinations (see Table D in Figure 1). Oracle macro plans are obtained by matching the mentions of entities and events in the gold summary with the input table. We make use of these oracle macro plans during training. The versions of MLB and ROTOWIRE released by Puduppully and Lapata (2021) contain paragraph delimiters for gold summaries; we preprocessed the German ROTOWIRE in a similar fashion. Table 1 also shows the average length of the macro plan in terms of the number of paragraph plans it contains. This is 10.6 for ROTOWIRE, 15.1 for MLB, and 9.5 for German RotoWire.
Data-to-text Generation with Variational Sequential Planning
Training Configuration We train our model with the AdaGrad optimizer (Duchi et al., 2011) and tune parameters on the development set. We use a learning rate of 0.15. We learn a joint subword vocabulary (Sennrich et al., 2016) for paragraph plans and summaries with 6K merge operations for ROTOWIRE, 16K merge operations for MLB, and 2K merge operations for German RO-TOWIRE. The model is implemented on a fork of OpenNMT-py (Klein et al., 2017). For efficiency, we batch using summaries instead of individual paragraphs. Batch sizes for MLB, ROTOWIRE, and German-ROTOWIRE are 8, 5, and 1 respectively. We set λ to 2 in Equation (18). In Equation (19), c is 1/100000 for MLB, 1/50000 for ROTOWIRE, and 1/30000 for German-ROTOWIRE. We set the temperature of Gumbel-Softmax to 0.1.
Data-to-text Generation with Variational Sequential Planning
During inference in MLB, similar to Puduppully and Lapata (2021), we block the repetition of paragraph plan bigrams (i.e., we disallow the repetition of (z t , z t+1 )) and select the paragraph plan with the next higher probability in Equation (8). In addition, we block consecutive repetitions, and more than two repetitions of a unigram. During training we observed high variance in the length of paragraphs y t since the same plan can result in a shorter or longer paragraph. For example, <V(B.Keller)> corresponds to two paragraphs (first and third paragraph) with different lengths in Figure 1. We found that this encourages the model to be conservative and generate relatively short output. We control the paragraph length (Fan et al., 2018) by creating discrete bins, each containing approximately an equal number of paragraphs. During training, we prepend the embedding of the bin to the current plan r t z (see Equation (11)). For inference, bins are tuned on the validation set.
Data-to-text Generation with Variational Sequential Planning
We run inference for 15 paragraphs on RO-TOWIRE and German ROTOWIRE, and for 20 paragraphs on MLB; we stop when the model predicts the end of paragraph plan token EOP. Unlike previous work (Wiseman et al., 2017;Puduppully et al., 2019a,b, inter alia), we do not make use of truncated Back Propagation Through Time (BPTT; Williams and Peng, 1990), as we incrementally generate paragraphs instead of long documents.
Data-to-text Generation with Variational Sequential Planning
Comparisons We compared our model with: (1) a Template-based generator which creates a document consisting of template sentences. We used Wiseman et al.'s (2017) system on RO-TOWIRE and Puduppully et al.'s (2019b) system on MLB. They are both similar in that they describe team scores followed by player specific statistics and a concluding statement. In MLB, the template additionally describes play-by-play details. We also created a template system for German ROTOWIRE following a similar approach.
Data-to-text Generation with Variational Sequential Planning
(2) ED+CC, the best performing model of Wiseman et al. (2017). It consists of an encoder-decoder model equipped with attention and copy mechanisms. (3) NCP+CC, the micro planning model of Puduppully et al. (2019a). It first creates a content plan by pointing to input records through the use of Pointer Networks . The content plan is then encoded with a BiLSTM and decoded using another LSTM with an attention and copy mechanism. (4) ENT, the entity model of Puduppully et al. (2019b). It creates entity-specific representations which are updated dynamically. At each time step during decoding, their model makes use of hierarchical attention by attending over entity representations and the records corresponding to these. (5) MACRO, the two-stage planning model of Puduppully and Lapata (2021), which first makes use of Pointer Networks  to predict a macro plan from a set of candidate paragraph plans. The second stage takes the predicted plan as input and generates the game summary with a sequence-to-sequence model enhanced with attention and copy mechanisms. In addition, we compare with a variant of Macro enhanced with length control (+Bin).
Data-to-text Generation with Variational Sequential Planning
Our experiments were designed to explore how the proposed model compares to related approaches which are either not enhanced with planning modules or non-incremental. We also investigated the sample efficiency of these models and the quality of the predicted plans when these are available. The majority of our results focus on automatic evaluation metrics. We also follow previous work (Wiseman et al., 2017;Puduppully et al., 2019a,b;Puduppully and Lapata, 2021) in eliciting judgments to evaluate system output.
Data-to-text Generation with Variational Sequential Planning
We evaluate model output using BLEU (Papineni et al., 2002) with the gold summary as a reference. We also report model performance against the Information Extraction (IE) metrics of Wiseman et al. (2017) which are defined based on the output of an IE model which extracts entity (team and player names) and value (numbers) pairs from the summary and predicts the type of relation between them.
Data-to-text Generation with Variational Sequential Planning
Letŷ be the gold summary and y be the model output. Relation Generation (RG) measures the precision and count of relations obtained from y that are found in the input table. Content Selection (CS) measures the precision, recall, and F-measure of relations extracted from y also found inŷ. And Content Ordering (CO) measures the complement of the Damerau-Levenshtein distance between relations extracted from y andŷ. Higher values are better for RG Precision, CS F-measure, CO, and BLEU. We reuse the IE model from Puduppully et al. (2019a) for ROTOWIRE, Puduppully and Lapata (2021) for MLB, and Hayashi et al. (2019) for German ROTOWIRE. Our computation of IE metrics for all systems includes duplicate records (Puduppully and Lapata, 2021).
Data-to-text Generation with Variational Sequential Planning
In addition to IE-based metrics, we report the number of errors made by systems according to Number (incorrect number in digits, number spelled in words, etc.), Name (incorrect names of teams, players, days of week, etc.), and Word (errors in usage of words) following the classification of Thomson and Reiter (2020). We detect such errors automatically using the system of Kasner et al. (2021) which scored best against gold standard human annotations of the same type (Thomson and Table 2 summarizes our results on MLB. Our sequential planning model (SeqPlan) has the highest RG P among neural models and performs best in terms of CS F, CO, and BLEU. The variant of Macro with length control (+Bin) performs comparably or worse than Macro.
Data-to-text Generation with Variational Sequential Planning
To examine the importance of latent sequential planning, we also present a variant of our model which uniformly samples a plan from the pool E instead of Equation (8) (see row w(ith) Uniform in Table 2). This version obtains lower values compared to SeqPlan across all metrics underscoring the importance of sequential planning. We also present two variants of SeqPlan (a) one which makes use of oracle (instead of predicted) plans during training to generate y t ; essentially, it replaces z t with z * in Equation (12) (row w(ith) Oracle in Table 2) and (b) a two stage model which trains the planner (Equation (15)) and generator (Equation (12)) separately (row 2-stage in Table 2); in this case, we use greedy decoding to sample z t from Equation (15) instead of Gumbel-Softmax and replace z t with z * in Equation (12). Both variants are comparable to SeqPlan in terms of RG P but worse in terms of CS F, CO, and BLEU.
Data-to-text Generation with Variational Sequential Planning
Furthermore, we evaluate the accuracy of the inferred plans by comparing them against oracle plans, using the CS and CO metrics (computed over the entities and events in the plan) 4 . Table 4 shows that SeqPlan achieves higher CS F and CO scores than Macro. Again, this indicates planning is beneficial, particularly when taking the table and the generated summary into account.
Data-to-text Generation with Variational Sequential Planning
English and German ROTOWIRE Results on ROTOWIRE are presented in Table 3 (top). In addition to Templ, ED+CC, NCP+CC, and ENT, we compare with the models of Wiseman et al. (2017) (WS-2017) and Rebuffel et al. (2020) (RBF-2020. WS-2017 is the best performing model of Wiseman et al. (2017). Note that ED+CC is an improved re-implementation of WS-2017. RBF-2020 represents the current state-of-the-art on ROTOWIRE, and comprises of a Transformer encoder-decoder architecture (Vaswani et al., 2017) with hierarchical attention on entities and their records. The models of Saleh et al. (2019), Iso et al. (2019), andGong et al. (2019) are not comparable as they make use of information additional to the table such as previous/next games or the author of the game summary. The model of  is also not comparable as it relies on a pretrained language model (Rothe et al., 2020) to generate the summary sentences.   We find that SeqPlan achieves highest RG P amongst neural models, and performs on par with Macro (it obtains higher BLEU but lower CS F and CO scores). The +Bin variant of Macro performs better on BLEU but worse on other metrics. As in Table 2, w Uniform struggles across metrics corroborating our hypothesis that latent sequential planning improves generation performance. The other two variants (w Oracle and 2-Stage) are worse than SeqPlan in RG P and CS F, comparable in CO, 5 We thank Hiroaki Hayashi for providing us with the output of the NCP+CC system.  and slightly higher in terms of BLEU.
Data-to-text Generation with Variational Sequential Planning
On German, our model is best across metrics achieving an RG P of 91.8% which is higher by 42% (absolute) compared to of Macro. In fact, the RG P of SeqPlan is superior to Saleh et al. (2019) whose model is pretrained with additional data and is considered state of the art (Hayashi et al., 2019). RG# is lower mainly because of a bug in the German IE which excludes number records. RG# for NCP+CC and Macro is too high because the summaries contain a lot of repetition. The same record will repeat at least once with NCP+CC and three times with Macro, whereas only 7% of the records are repeated with SeqPlan. Table 4 evaluates the quality of the plans inferred by our model on the ROTOWIRE dataset. As can be seen, SeqPlan is slightly worse than Macro in terms of CS F and CO. We believe this is because summaries in ROTOWIRE are somewhat formulaic, with a plan similar to Templ: an opening statement is followed by a description of the top scoring players, and a conclusion describing the next match. Such plans can be learnt well by Macro without access to the summary. MLB texts show a lot more diversity in terms of length, and the sequencing of entities and events. The learning problem is also more challenging, supported by the fact that the template system does not do very well in this domain (i.e., it is worse in BLEU, CS F, and CO compared to ROTOWIRE). In German ROTOWIRE, SeqPlan plans achieve higher CS F and CO than Macro. Table 5 reports complementary automatic metrics on English ROTOWIRE aiming to assess the factuality of generated output. We find that Templ has the least Number, Name, and double-double errors. This is expected as it simply reproduces   facts from the table. SeqPlan and Macro have similar Number errors, and both are significantly better than other neural models. SeqPlan has significantly more Name errors than Macro, and significantly fewer than other neural models. Inspection of Name errors revealed that these are mostly due to incorrect information about next games. Such information is not part of the input and models are prone to hallucinate. SeqPlan fares worse as it attempts to discuss next games for both teams while Macro focuses on one team only. In terms of double-double errors, SeqPlan is comparable to Macro, ENT and NCP+CC, and significantly better than WS-2017, ED+CC, and RBF-2020.
Data-to-text Generation with Variational Sequential Planning
We also evaluated whether SeqPlan is more sample efficient in comparison to Macro, by examining how RG P varies with (training) data size. As shown in Figure 4, the difference between SeqPlan and Macro is more pronounced when relatively little data is available.  Table 6: Average number of supported (#Supp) and contradicting (#Contra) facts in game summaries and bestworst scaling evaluation for Coherence (Coher), Conciseness (Concis), and Grammaticality (Gram). Lower is better for contradicting facts; higher is better for Coherence, Conciseness, and Grammaticality. Systems significantly different from SeqPlan are marked with an asterisk * (using a one-way ANOVA with posthoc Tukey HSD tests; p ≤ 0.05).
Data-to-text Generation with Variational Sequential Planning
. ference in RG P decreases. The slope of increase in RG P for Macro is higher for ROTOWIRE than MLB. We hypothesize this is because MLB has longer summaries with more paragraphs, and is thus more difficult for Macro to learn alignments between paragraph plans and text paragraphs in the game summary.
Data-to-text Generation with Variational Sequential Planning
We used the Amazon Mechanical Turk (AMT) crowdsourcing platform for our judgment elicitation study. To ensure consistent ratings (van der Lee et al., 2019) we required that raters have completed at least 1,000 tasks, and have at least 98% approval rate. Participants were restricted to English speaking countries (USA, UK, Canada, Australia, Ireland, or New Zealand) and were allowed to provide feedback or ask questions. Raters were paid an average of 0.35$ for each task, ensuring that the remuneration is higher than the minimum wage per hour in the US. We compared SeqPlan with Gold, Templ, ED+CC, and Macro; we did not compare against ENT as previous work (Puduppully and Lapata, 2021) has shown that it performs poorly against Macro. For ROTOWIRE, we additionally compared against RBF-2020.
Data-to-text Generation with Variational Sequential Planning
Supported and Contradicted Facts Our first eliciation study provided raters with box scores (and play-by-plays in the case of MLB), along with sentences randomly extracted from game summaries. We asked them to count supported and contradicting facts (ignoring hallucinations). Participants were given a cheatsheet to help them understand box score and play-by-play statistics as well as examples of sentences with the correct count of supported and contradicting facts. This evaluation was conducted on 40 summaries (20 for each dataset), with four sentences per summary, each rated by three participants. For MLB, this resulted in 300 tasks (5 systems × 20 summaries × 3 raters) and for ROTOWIRE in 360 (6 systems × 20 summaries × 3 raters). Altogether, we had 177 participants. The agreement between raters using Krippendorff's α for supported facts and contradicting facts was 0.43. Table 6 (columns #Supp and #Contra) presents our results. Lower is better for contradicting facts. In case of supporting facts, the count should neither be too high nor too low. A high count of supporting facts indicates indicates poor content selection. A low count of supporting facts with a high count of contradicting facts indicates low accuracy of generation.
Data-to-text Generation with Variational Sequential Planning
Templ achieves the lowest count of contradicting facts and the highest count of supported facts for both the datasets. This is no surprise as it essentially regurgitates facts (i.e., records) from the table. On MLB, all systems display a comparable count of supported facts (differences are not statistically significant), with the exception of Templ which contains significantly more. In terms of contradicting facts, SeqPlan performs on par with Macro, Gold and Templ, and is significantly better than ED+CC. On ROTOWIRE, in terms of supported facts, Seq-Plan performs on par with the other neural models, is significantly higher than Gold, and significantly lower than Templ. In terms of contradicting facts, SeqPlan performs on par with Macro, Gold and Templ, and significantly better than ED+CC and RBF-2020.
Data-to-text Generation with Variational Sequential Planning
In our second study, raters were asked to choose the better summary from a pair of summaries based on Coherence (is the summary well structured and well organized and does it have a natural ordering of the facts?), Conciseness (does the summary avoid unnecessary repetition including whole sentences, facts or phrases?), and Grammaticality (is the summary written in well-formed English?). For this study, we required that the raters be able to comfortably comprehend summaries of NBA/MLB games. We obtained ratings using Best-Worst scaling (Louviere and Woodworth, 1991;Louviere et al., 2015), an elicitation paradigm shown to be more accurate than Likert scales. The score for a system is obtained by the number of times it is rated best minus the number of times it is rated worst (Orme, 2009). Scores range between −100 (absolutely worst) and +100 (absolutely best); higher is better. We assessed 40 summaries from the test set (20 for each dataset). Each summary pair was rated by three participants. For MLB, we created 1,800 tasks (10 system pairs × 20 summaries × 3 raters × 3 dimensions) and 2,700 for ROTOWIRE (15 pairs of systems × 20 summaries × 3 raters × 3 dimensions). Altogether, 377 raters participated in this task. The agreement between the raters using Krippendorff's α was 0.49.
Data-to-text Generation with Variational Sequential Planning
On MLB, SeqPlan is significantly more coherent than ED+CC and Templ, and is comparable with Gold and Macro. A similar picture emerges with grammaticality. SeqPlan is as concise as Gold, Macro and Templ, and significantly better than ED+CC. On ROTOWIRE, SeqPlan is significantly more coherent than Templ and ED+CC, but on par with Macro, RBF-2020 and Gold. In terms of conciseness, SeqPlan is comparable with Gold, Macro, RBF-2020, and ED+CC, and significantly better than Templ. In terms of grammaticality, SeqPlan is comparable with Macro, RBF-2020, and ED+CC, significantly better than Templ, and significantly worse than Gold.
Data-to-text Generation with Variational Sequential Planning
In this work, we proposed a novel sequential latent variable model for joint macro planning and generation. Key in our approach is the creation of a latent plan in a sequential manner, while interleaving the prediction of plans and the generation of corresponding paragraphs. We proposed to deconstruct monolithic long document generation into smaller units (paragraphs in our case) which affords flexibility and better communication between planning and generation. Taken together, the results of automatic and human evaluation suggest that SeqPlan performs best in terms of factuality and coherence, it generates diverse, and overall fluent summaries and is less data-hungry compared to strong systems like Macro and NCP+CC. As SeqPlan does not have to learn alignments between the macro plan and the output text, it is better suited <V(Cardinals)> → <V(9-T)> → <V(Cardinals)> → <V(Cardinals)> <V(Brewers)>→ <V(9-T)> → <V(8-B)> → <V(8-T)> → <V(8-B)> → <V(9-B)> → <V(Brewers)>→ <V(Adam Wainwright)>→ <V(Brewers)>→ <V(3-T)> → <V(3-B)> → <V(Carlos Villanueva)> ST. LOUIS -The St. Louis Cardinals have been waiting for their starting rotation. <P> Skip Schumaker drove in the go-ahead run with a double in the ninth inning, and the Cardinals beat the Milwaukee Brewers 4-3 on Wednesday night to avoid a three-game sweep. <P> The Cardinals have won four of five, and have won four in a row. <P> The Cardinals have won four of five, including a threegame sweep by the Brewers. <P> Brian Barton led off the ninth with a pinch-hit double off Derrick Turnbow (0-1) and moved to third on Cesar Izturis' sacrifice bunt. Schumaker drove in Barton with a double down the left-field line. <P> Ryan Braun, who had two hits, led off the eighth with a double off Ryan Franklin (1-1). Braun went to third on a wild pitch and scored on Corey Hart's triple into the right-field corner. <P> Albert Pujols was intentionally walked to load the bases with one out in the eighth, and Guillermo Ankiel flied out. Troy Glaus walked to load the bases for Kennedy, who hit a sacrifice fly off Guillermo Mota. <P> Ryan Franklin (1-1) got the win despite giving up a run in the eighth. Ryan Braun led off with a double and scored on Corey Hart's one-out triple. <P> Jason Isringhausen pitched a perfect ninth for his seventh save in nine chances. He has converted his last six save opportunities and has n't allowed a run in his last three appearances. <P> The Brewers lost for the seventh time in eight games. <P> Wainwright allowed two runs and four hits in seven innings. He walked four and struck out six. <P> Brewers manager Ron Roenicke was ejected by home plate umpire Bill Miller for arguing a called third strike. <P> The Cardinals took a 2-0 lead in the third. Albert Pujols walked with two outs and Rick Ankiel walked. Glaus then lined a two-run double into the left-field corner. <P> The Brewers tied it in the third. Jason Kendall led off with a double and scored on Rickie Weeks' double. Ryan Braun's RBI single tied it at 2. <P> Villanueva allowed two runs and three hits in seven innings. He walked four and struck out one. Table 7: Predicted macro plan (top) and generated output from our model. Transitions between paragraph plans are shown using →. Paragraphs are separated with <P> delimiters. Entities and events in the summary corresponding to the macro plan are boldfaced.
Data-to-text Generation with Variational Sequential Planning
for long-form generation. Potential applications include summarizing books (Kryściński et al., 2021) where the output can be longer than 1,000 tokens or generating financial reports (Kogan et al., 2009;Händschke et al., 2018) where the output exceeds 9,000 tokens. Existing approaches for long-form generation summarize individual paragraphs independently (Kryściński et al., 2021) or adopt a hierarchical approach (Wu et al., 2021) where summaries of paragraphs form the basis of chapter summaries which in turn are composed into a book summary. Table 7 gives an example of SeqPlan output. We see that the game summary follows the macro plan closely. In addition, the paragraph plans and the paragraphs exhibit coherent ordering. Manual inspection of SeqPlan summaries reveals that a major source of errors in MLB relate to attention diffusing over long paragraph plans. As an example, consider the following paragraph produced by SeqPlan "Casey Kotchman had three hits and three RBIs , including a two-run double in the second inning that put the Angels up 2-0. Torii Hunter had three hits and drove in a run ." In reality, Torii Hunter had two hits but the model incorrectly generates hits for Casey Kotchman. The corresponding paragraph plan is 360 tokens long and attention fails to discern important tokens. A more sophisticated encoder, e.g., based on Transformers (Vaswani et al., 2017), could make attention more focused. In RO-TOWIRE, the majority of errors involve numbers (e.g., team attributes) and numerical comparisons. Incorporating pre-executed operations such as min, max (Nie et al., 2018) could help alleviate these errors.
Data-to-text Generation with Variational Sequential Planning
Finally, it is worth mentioning that although the template models achieve highest RG precision for both MLB and ROTOWIRE (Tables 2 and 3), this is mainly because they repeat facts from the table. Template models score low against CS F, CO, and BLEU metrics. In addition, they obtain lowest scores in Grammaticality and Coherence (Table 6) which indicates that they are poor at selecting records from the table and ordering them correctly in a fluent manner.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We refer to the first set of choices (how to structure the information) as text planning and to the second (how to verbalize a plan) as plan realization. 1 The distinction between planning and realization is at the core of classic natural language generation (NLG) works (Reiter and Dale, 2000;Gatt and Krahmer, 2017). However, a recent wave of neural NLG systems ignores this distinction and treat the problem as a single end-to-end task of learning to map facts from the input to the output text (Gardent et al., 2017;Dušek et al., 2018). These neural systems encode the input facts into an intermediary vector-based representation, which is then decoded into text. While not stated in these terms, the neural system designers hope for the network to take care of both the planning and realization aspect of text generation. A notable exception is the work of Puduppully et al. (2018), who introduce a neural content-planning module in the end-to-end architecture.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
While the neural methods achieve impressive levels of output fluency, they also struggle to maintain coherency on longer texts (Wiseman et al., 2017), struggle to produce a coherent order of facts, and are often not faithful to the input facts, either omitting, repeating, hallucinating or changing facts (the NLG community refers to such errors as errors in adequacy or correctness of the generated text). When compared to templatebased methods, the neural systems win in fluency but fall short regarding content selection and faithfulness to the input (Puzikov and Gurevych, 2018). Also, they do not allow control over the output's structure. We speculate that this is due to demanding too much of the network: while the neural system excels at capturing the language details required for fluent realization, they are less well equipped to deal with the higher levels text structuring in a consistent and verifiable manner.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The matching of plans to references is based on the observations that (a) it is relatively easy to identify entities in the reference texts, and a pair of entities in an input is unique to a fact; (b) it is relatively easy to identify sentence splits; (c) a reference text and its matching plan must share the same entities in the same order, and with the same sentence splits. Sentence split consistency We define a set of triplets to be potentially consistent with a sentence iff each triplet contains at least one entity from the sentence (either its subject or object appear in the sentence), and each entity in the sentence is covered by at least one triplet. Given a reference text, we split it into sentences using NLTK (Bird and Loper, 2004), and look for divisions of G into disjoint sets such that each set is consistent with a corresponding sentence. For each such division, we consider the exhaustive set of all induced plans. Facts order consistency A natural criterion would be to consider a reference sentence and a sentence-plan originating from the corresponding RDF as matching iff the sets of entities in the sentence and the plan are identical, and all entities appear in the same order. 4 Based on this, we could represent each sentence and each plan as a sequence of entities, and verify the sequences match. However, using this criterion is complicated by the fact that it is not trivial to map between the entities in the plan (that originate from the RDF triplets) and the entities in the text. In particular, due to language variability, the same plan entity may appear in several forms in the textual sentences. Some of these variations (i.e. "A.F.C Fylde" vs. "AFC Fylde") can be recognized heuristically, while others require external knowledge ("UK conservative party" vs. "the Tories"), and some are ambiguous and require full-fledged co-reference resolution ("them", "he", "the former"). Hence, we relax our matching criterion to allow for possible unrecognized entities in the text.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Concretely, we represent each sentence plan as a sequence of its entities (pe 1 , ..., pe k ), and each sentence as the sequence of its entities which we managed to recognize and to match with an input entity (se 1 , ..., se m ), m ≤ k. 5 We then consider a sentence and a sentenceplan to be consistent if the following two condi-4 An additional constraint is that no two triplets in the RDFs set share the same entities. This is to ensure that if two entities appeared in a structure, only one relation could have been expressed there. This almost always holds in the WebNLG corpus, failing on only 15 out of 6,940 input sets. 5 We match plan entities to sentence entities using greedy string matching with Levenshtein distance (Levenshtein, 1966) for each token and a manually tuned threshold for a match. While this approach results in occasional false positives, most cases are detected correctly. We match dates by using the chrono-python package that parses dates from natural language texts. tions hold: (1) The sentence entities (se 1 , ..., se m ) are a proper sub-sequence of the plan entities (pe 1 , ..., pe k ); and (2) each of the remaining entities in the plan already appeared previously in the plan. The second condition accounts for the fact that most un-identified entities are due to pronouns and similar non-lexicalized referring expressions, and that these only appear after a previous occurrence of the same entity in the text. 
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
Training details We use a standard NMT setup with a copy-attention mechanism (Gulcehre et al., 2016) 10 and the pre-trained GloVe.6B word em-8 Minimally, each entity occurrence can keep track of the number of times it was already mentioned in the plan. Other alternatives include using a full-fledged referring expression generation system such as NeuralREG (Ferreira et al., 2018) 9 We map DBPedia relations to sequences of tokens by splitting on underscores and CamelCase. 10 Concretely, we use the OpenNMT toolkit (Klein et al., 2017) with the copy attn flag. Exact parameter values are beddings 11 (Pennington et al., 2014). The pretrained embeddings are used to initialize the relation tokens in the plans, as well as the tokens in the reference texts.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
The WebNLG challenge (Colin et al., 2016) consists of mapping sets of RDF triplets to text including referring expression generation, aggregation, lexicalization, surface realization, and sentence segmentation. It contains sets with up to 7 triplets each along with one or more reference texts for each set. The test set is split into two parts: seen, containing inputs created for entities and relations belonging to DBpedia categories that were seen in the training data, and unseen, containing inputs extracted for entities and relations belonging to 5 unseen categories. While the unseen category is conceptually appealing, we view the seen category as the more relevant setup: generating fluent, adequate and diverse text for a mix of known relation types is enough of a challenge also without requiring the system to invent verbalizations for unknown relation types. Any realistic generation system could afford to provide at least a few verbalizations for each relation of interest. We thus focus our attention mostly on the seen case (though our system does also perform well on the unseen case). Following Section 3.1, we manage to match a detailed in the appendix. 11 nlp.stanford.edu/data/glove.6B.zip consistent plan for 76% of the reference texts and use these plan-text pairs to train the plan realization NMT component. Overall, the WebNLG training set contains 18, 102 RDF-text pairs while our plan-enhanced corpus contains 13, 828 plantext pairs. 12 Compared Systems We compare to the best submissions in the WebNLG challenge (Gardent et al., 2017): Melbourne, an end-to-end system that scored best on all categories in the automatic evaluation, and UPF-FORGe (Mille et al., 2017), a classic grammar-based NLG system that scored best in the human evaluation. Additionally, we developed an end-to-end neural baseline which outperforms the WebNLG neural systems. It uses a set encoder, an LSTM (Hochreiter and Schmidhuber, 1997) decoder with attention (Bahdanau et al., 2014), a copy-attention mechanism (Gulcehre et al., 2016) and a neural checklist model (Kiddon et al., 2016), as well as applying entity dropout. The entity-dropout and checklist component are the key differentiators from previous systems. We refer to this system as StrongNeural.
Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation
We begin by comparing our plan-based system (BestPlan) to the state-of-the-art using the common automatic metrics: BLEU (Papineni et al., 2002), Meteor (Banerjee and Lavie, 2005), ROUGE L (Lin, 2004) and CIDEr (Vedantam et al., 2015), using the nlg-eval 13 tool (Sharma et al., 2017) on the entire test set and on each part separately (seen and unseen).
Stepwise Extractive Summarization and Planning with Structured Transformers
Hochreiter and Schmidhuber, 1997) is used to encode a document and then another LSTM is used to predict for each sentence whether it should be included in the summary. This architecture was later adopted by Nallapati et al. (2016a), Nallapati et al. (2017), Narayan et al. (2018b), Zhang et al. (2018) and Dong et al. (2018).
Stepwise Extractive Summarization and Planning with Structured Transformers
Following the success of pre-trained transformerbased architectures for many tasks (Vaswani et al., 2017;Devlin et al., 2019), the current state-of-theart approach to extractive summarization uses transformers to learn sentence representations and to rank sentences by their saliency (Liu, 2019;Liu and Lapata, 2019b;Zhang et al., 2019;Zhong et al., 2019a;Bi et al., 2020). The top scoring sentences are then assembled to produce an extract of the document. Summaries built in this fashion (Cheng and Lapata, 2016;Narayan et al., 2018a;Zhang et al., 2018;Dong et al., 2018) are prone to contain redundant information. Several recent approaches have explored mechanisms to better handle redundancy, such as heuristic-based Trigram Blocking (TriBlk; Liu and Lapata, 2019b;Wang et al., 2020), handcrafted feature-driven models (Ren et al., 2017) and redundancy aware neural sequence models (Zhou et al., 2018;Bi et al., 2020). One common problem with these models is that their focus is limited to content overlap and to respecting length budgets. However, these are but a small subset of the dimensions necessary to produce informative and coherent summaries. Ideally, models would utilize enriched document and summary representations in order to implicitly learn better extractive plans for producing summaries Mendes et al., 2019). One such method is stepwise summarization , where a summary is constructed incrementally by choosing new content conditioned on previously planned content.
Stepwise Extractive Summarization and Planning with Structured Transformers
In this paper, we propose encoder-centric stepwise models for extractive summarization using structured transformers. Structured transformers are transformer-based architectures that have the flexibility to model some form of structure of the input, e.g., hierarchical document structure. In this paper, we specifically study two such architectures -HiBERT (Zhang et al., 2019) and Extended Transformers Construction (ETC; Ainslie et al., 2020). Details of these are given in Sections 4 and 5. We enable stepwise summarization by injecting the previously planned summary content into the structured transformer as an auxiliary sub-structure. The model then can holistically learn any documentlevel coherence properties, such as saliency, redundancy, and ordering, embodied in the gold summaries. This differs from other methods which are either task specific (e.g., redundancy aware modeling in Bi et al., 2020) or not holistic (e.g., manually curated features in ). An added advantage of structured encoders is that they break the quadratic attention mechanism of transformers (Devlin et al., 2019), making them more efficient and able to process longer inputs, instead of truncating the inputs to 512 tokens (Liu and Lapata, 2019b;Bi et al., 2020), which is critical for long inputs and outputs which require non-trivial planning. When evaluated on the CNN/DailyMail summarization dataset (Hermann et al., 2015), we achieve stateof-the-art performance in terms of Rouge (Lin and Hovy, 2003) without any redundancy (Zhou et al., 2018;Bi et al., 2020) or sentence selection mechanisms (Liu and Lapata, 2019b). Our model's task-agnostic approach allows it to implicitly learn and leverage content plans directly from the data. Moreover, structured transformers form the basis of our model, which are flexible in terms of content type (e.g., text or tables) that can be modeled. We demonstrate this by learning intricate extractive content plan for the Rotowire table-to-text generation task (Wiseman et al., 2017). This task requires the generation of long summaries from large score tables detailing the the specifics of a sports match, which often necessitates dedicated content selection and planning models to generate a high-quality summary (Wiseman et al., 2017;Puduppully et al., 2019a). We show that our stepwise framework achieves higher content selection, planning and ordering scores relative to prior work with task-specific planning mechanisms.
Stepwise Extractive Summarization and Planning with Structured Transformers
The contributions of the paper are as follows: 1) this is first study to use ETC (Ainslie et al., 2020) for summarization for its ability and flexibility to better model long and structured inputs; 2) we pro-pose augmentions of two structured transformers, HiBERT and ETC, in order to enable stepwise models for extractive planning; 3) we demonstrate empirically that our models are general purpose and can be adapted as an extractive document summarizer or as a content planner for table-to-text generation; 4) Our experiments highlight the effectiveness of stepwise modeling, specifically stepwise ETC, which sets a new standard for both tasks.
Stepwise Extractive Summarization and Planning with Structured Transformers
For exposition, we use the extractive document summarization setup to introduce our stepwise models with HiBERT (Zhang et al., 2019) and ETC (Ainslie et al., 2020) in the following sections. Specifically, we use 'sentence' as a content unit and 'previously' or 'partially generated summary' for a previously selected content plan.
Stepwise Extractive Summarization and Planning with Structured Transformers
Hierarchical encodings have been used to model input structure with LSTMs (Nallapati et al., 2016b;Cheng and Lapata, 2016;Narayan et al., 2018b). Zhang et al. (2019) proposed HiBERT with two stacked Transformer encoders (Vaswani et al., 2017) for extractive summarization (see the middle diagram in Figure 1): a sentence encoder that independently builds representations for each sentence in the document; and a document encoder that operates over sentence encodings to build contextual representations for all sentences. These contextual sentence representations are then ingested by a classifier to predict the salience score of each sentence in the document. As in standard transformers, both encoders have multiple layers with each layer composed of a multi-head self-attention layer followed by a feed-forward sub-layer with residual connections (He et al., 2015) and layer normalizations (Ba et al., 2016). For Stepwise HiBERT, at time step k, we modify the document encoder with the content plan S k−1 , which is the previously selected sentences in the summary. This is depicted in Figure 2 (left) and allows the model to implicitly select new sentences relative to the previously generated summary.
Stepwise Extractive Summarization and Planning with Structured Transformers
While document self-attention learns the contextual hidden representation h doc→doc   Roy et al., 2020). HiBERT alleviates this problem by modeling each sentence independently; the memory usage in HiBERT scales with the square of the number of sentences, and the square of the maximum length of any sentence. However, the main disadvantage of this approach is that token-level attention across sentences is prohibited and long range attention only happens indirectly at the second-stage encoder (see the middle diagram in Figure 1). Recently, Extended Transformer Construction (ETC; Ainslie et al., 2020) provides an alternative. It alleviates the quadratic memory growth by introducing sparsity to the attention mechanism via its novel global-local attention mechanism (see the rightmost diagram in Figure 1). This not only permits encoding of long inputs, 2 but also enables a mechanism to model structure directly through nodes in the global attention layer. Global-Local Attention. The ETC model architecture receives two inputs: a long input, which in most cases corresponds to the text to be encoded; and an auxiliary global input, which serves as inductive bias features. First, the model builds an attention map, called long-to-long, across the long input with a sparse local attention of fixed length, this bypasses the quadratic memory complexity and allows to scale input lengths to the thousands of tokens, but limits the attention span of tokens to their nearest neighbors.
Stepwise Extractive Summarization and Planning with Structured Transformers
To overcome this limitation, the global-local attention defines three other attention parts: globalto-global, global-to-long and long-to-global, all with unrestricted attention. This allows tokens arbitrarily far apart to attend to each other with at most one hop through the global input tokens. We refer the reader to Ainslie et al. (2020) for more details. The right parts of Figures 1 and 2 illustrate these four types of attentions and the sparsity diagrams where each cell in a row i and column j is different than white input token w i can attend to input token w j , same relative position embeddings are indicated by using the same color.
Stepwise Extractive Summarization and Planning with Structured Transformers
We used the auxiliary global input to represent sentence structure. Specifically, following (Ainslie et al., 2020), we placed one auxiliary token in the global input per each sentence in the input I. We linked the global tokens with the input tokens by using relative position labels to represent whether each token belongs to that sentence. Global-toglobal attention is left unrestricted, allowing all sentences to attend to each other. This result is summary-informed contextualized input token representations via attention through the global nodes. In the rest of the paper we refer to this summarizer by Stepwise ETCSum. Similar to HiBERT, we take the first token hidden representation h i 1 as the representation for the sentence s i . Finally, sentence embeddings are passed to the softmax layer for salience scoring. Both HiBERT and ETCSum are then trained with the cross entropy loss.
Stepwise Extractive Summarization and Planning with Structured Transformers
6 Extractive Document Summarization 6.1 Experimental Setup Dataset. We evaluate our models on the CNN and DailyMail news highlights datasets (Hermann et al., 2015). We used standard splits (287,227/13,368/11,490 documents) for training, validation, and testing. We did not anonymize entities or lower case tokens as in (Narayan et al., 2018b;Zhou et al., 2018;Zhang et al., 2019;Liu and Lapata, 2019b). The documents in the CNN/DailyMail dataset are long; the average lengths are 760.5 words (34 sentences) for CNN and 653.3 words (29.3 sentences), for DailyMail. The human written abstracts have 46 and 55 words for CNN and DailyMail, respectively. We evaluated summarization quality using F 1 Rouge. 3
Stepwise Extractive Summarization and Planning with Structured Transformers
Baselines. We compared our Stepwise HiBERT and ETCSum models to Lead and Oracle baselines. Lead selects the first 3 sentences to form the summary, while Oracle baselines creates a summary by selecting the best possible set of sentences in the document that gives the highest average of Rouge-1, Rouge-2 and Rouge-L F1 scores with respect to the human written summary. The Oracle (512) truncates the input document to 512 tokens. We further compared our models against several redundancyaware models (NeuSum; Zhou et al., 2018 and ARedSum; Bi et al., 2020) and models that uses Trigram Blocking (TriBlk; Liu and Lapata, 2019b) for redundancy elimination during sentence selection (see the second block in Table 1).
Stepwise Extractive Summarization and Planning with Structured Transformers
To understand the importance of modeling long documents for extractive summarization, we also trained BERTSum, similar to Liu and Lapata (2019b), with a receptive capacity of 512 tokens initialized with the BERT checkpoint. Our BERT-Sum differs slightly from Liu and Lapata (2019b), in that we don't use segment embeddings. We also report on RoBERTa  initialized version of BERTSum (RoBERTaSum).
Stepwise Extractive Summarization and Planning with Structured Transformers
We also trained non-stepwise variants of HiB-ERT and ETCSum models (the third block in Table 1). In this setting, HiBERT and ETC do not take partial summaries as input. Instead, they simply take the input document and generate salient scores (using a sigmoid layer) for each sentence in the document; the top three sentences are then assembled to generate the summary. Our implementation of HiBERT differs from Zhang et al. (2019). For example, we don't pretrain HiBERT from scratch for document modeling as in Zhang et al. (2019). Instead, we initialize our HiBERT models with publicly available RoBERTa  checkpoints following the superior performance of 3 We lowercased candidate and reference summaries and used pyrouge with parameters "-a -c 95 -m -n 4 -w 1.2."  (Zhang et al., 2018) 41.05 18.77 37.54 Refresh (Narayan et al., 2018b) 41.00 18.80 37.70 BanditSum (Dong et al., 2018) 41.50 18.70 37.60 NeuSUM (Zhou et al., 2018) 41.59 19.01 37.98 ExConSum (Mendes et al., 2019) 41.70 18.60 37.80 JECS (Xu and Durrett, 2019) 41.70 18.50 37.90 LSTM+PN (Zhong et al., 2019b) 41.85 18.93 38.13 HER (Luo et al., 2019) 42.30 18.90 37.60 HiBERT (Zhang et al., 2019) 42.37 19.95 38.83 PNBERT (Zhong et al., 2019a) 42.69 19.60 38.85 BERTSum (Liu and Lapata, 2019b) 42.61 19.99 39.09 BERTSum+TriBlk 43.25 20.24 39.63 ARedSum-CTX (Bi et al., 2020) 43.43 20.44 39.83 HSG (Wang et al., 2020) 42 RoBERTaSum over BERTSum. We use different number of layers in the document encoder (L doc = 3) and in the sentence encoder (L sent = 9), as opposed to equal number of layers (L = 6) in both encoders of Zhang et al. (2019). The layers in the document and sentence encoders were initialized with the top and the bottom layers of RoBERTa, respectively. All ETCSum models were initialized with the uncased version of ETC pretrained checkpoints (Ainslie et al., 2020) pretrained using the standard masked language model task and the contrastive predictive coding (van den Oord et al., 2018). 4 We also report on the effect of TriBLK with all our models. We only experiment with the basesized models and therefore have 12 layers, a hidden size of 768, filter size of 3072, and 12 attention heads. For comparison, we report results from BERTSum Large (Liu and Lapata, 2019b) which uses 24 layers. Finally, we employ a beam decoding to predict summaries using our stepwise models; we use a beam size of 3 for a maximum of 4 steps. We don't allow repeated sentences, though this is not a requirement. We refer the reader to the supplementary material for implementation and reproducibility details.
Stepwise Extractive Summarization and Planning with Structured Transformers
Generating Extractive Oracles. Following Narayan et al. (2018b), we train models to predict all sentences in Oracle (Full) for non-stepwise training. Stepwise training learns to do this gradually: at each step, we train model to predict the next sentence in Oracle (Full) using the earlier predicted sentences and the document. During testing, human written abstracts are used as reference summaries to evaluate our models.
Stepwise Extractive Summarization and Planning with Structured Transformers
Long form Summarization. In our experiments, ETCSum appears to be far more superior than HiB-ERT when modeling long documents for extractive summarization; ETCSum outperformed HiBERT in all cases including stepwise or non-stepwise predictions, and, with or without trigram blocking. The downside of HiBERT where token-level attention across sentences is not possible, is not optimal for modeling documents. Both ETCSum and ETCSum+TriBlk performed better than BERTSum and BERTSum+TriBlk, respectively. These results suggest the importance of modeling the whole document with ETCSum, rather than truncating it to only 512 tokens to fit BERTSum. However, the improvement may not be attributed solely to ETC-Sum's ability to model long inputs, but also to its better initialization with ETC checkpoints (Ainslie et al., 2020), specially when the improvement diminishes when compared against RoBERTaSum. 5 Stepwise vs Non-stepwise models. First of all, trigram filtering seems to be the key in addressing redundancy in generated summaries in nonstepwise models. It helps almost all models including our HiBERT and ETCSum (except for the single case of RoBERTaSum on Rouge-2). Interestingly, we don't observe the same pattern for our stepwise models. We observe that our stepwise models (both HiBERT and ETCSum, without TriBlk) consistently improve over their nonstepwise counterparts. But when stepwise is applied with TriBlk, we don't always see improvements. We conjecture that our stepwise models themselves are inherently better at avoiding redundancy in generated summaries due to the knowledge of previously generated summary at each prediction step, and improvements with TriBlk are not always complementary. The same is also demonstrated in Figure 3;  We also report on Stepwise RoBERTaSum baselines and performance dropped compared to corresponding non-stepwise models. Perhaps without any structure in the transformer, simple summary concatenation is not a good method for Stepwise RoBERTaSum to distinguish the document from the summary. There might be better ways (than the vanilla concatenation), but with Stepwise ETCSum or HiBERT, it is very natural. Stepwise RoBER-TaSum also loses access to the end of the input as the partial summary grows for documents that are already close to 512 tokens in length.
Stepwise Extractive Summarization and Planning with Structured Transformers
Stepwise ETCSum, in particular, sets a new standard for both tasks. Future work will focus on extending our models to generate extractive plans for better abstractive summarization of long or multiple documents (Liu et al., 2018). 
Stepwise Extractive Summarization and Planning with Structured Transformers
For extractive summarization, we used HiBERT with a 8 transformer layer sentence encoder, and a 4 transformer layer document encoder. The model has 133,784,833 parameters. The word position embedding in the sentence encoder is initialized using the RoBERTa checkpoint, but the document and summary sentence position embeddings are learned from scratch. The document self attention and summary self attentions are shared and initialized using the RoBERTa checkpoint, the document-summary attention is also initialized using the RoBERTa checkpoint. We truncate each document to 128 sentences and each sentence to 32 words. We trained all HiBERT models for 100k steps saving checkpoints every 1000 steps, with a batch size of 32. Following Liu and Lapata (2019b), we choose the best model based on the MLE loss on the whole validation set.
Stepwise Extractive Summarization and Planning with Structured Transformers
The ETCSum model for both extractive summarization and table-to-text generation uses a 12 layer transformer as described in (Ainslie et al., 2020). The model is pretrained with MLM and CPC objectives as described in (Ainslie et al., 2020). In total, the model has 165,825,793 trainable parameters which mostly comes from the long input of 8192 tokens and the full attention of 512 of the global tokens. We trained our model with a batch size of 512 for 5,000 steps approximately equivalent to 10 epochs.
Data-to-text Generation with Variational Sequential Planning
Data-to-text generation refers to the task of generating textual output from non-linguistic input such as database tables, spreadsheets, or simulations of physical systems Dale, 1997, 2000;Gatt and Krahmer, 2018). Recent progress in this area (Mei et al., 2016;Lebret et al., 2016;Wiseman et al., 2017) has been greatly facilitated by the very successful encoder-decoder neural architecture (Sutskever et al., 2014) and the development of large scale datasets. ROTOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) constitute such examples. They both focus on the sports domain which has historically drawn attention in the generation community (Barzilay and Lapata, 2005;Tanaka-Ishii et al., 1998;Robin, 1994) and consider the problem of generating long target texts from database records. Figure 1 (reproduced from Puduppully and Lapata, 2021) provides a sample from the MLB dataset which pairs human written summaries (Table C) with major league baseball game statistics. These are mostly scores (collectively referred to as box score) which summarize the performance of teams and players, e.g., batters, pitchers, or fielders (Table A) and a play-by-play description of the most important events in the game (Table B). Game summaries in MLB are relatively long (540 tokens on average) with multiple paragraphs (15 on average). The complexity of the input and the length of the game summaries pose various challenges to neural models which, despite producing fluent output, are often imprecise, prone to hallucinations, and display poor content selection (Wiseman et al., 2017). Attempts to address these issues have seen the development of special-purpose modules which keep track of salient entities (Iso et al., 2019;Puduppully et al., 2019b), determine which records (see the rows in Tables A and B) should be mentioned in a sentence and in which order (Puduppully et al., 2019a;, and reconceptualize the input in terms of paragraph plans (Puduppully and Lapata, 2021) to facilitate document-level planning (see Table D in Figure 1). Specifically, Puduppully and Lapata (2021) advocate the use of macro plans for improving the organization of document content and structure. A macro plan is a sequence of paragraph plans, and each paragraph plan corresponds to a document paragraph. A macro plan is shown in Table E (Figure 1). Examples of paragraph plans are given in Table D where <V(entity)> verbalizes records pertaining to entities and <V(inning-T/B)> verbalizes records for the Top/Bottom side of an inning. Verbalizations are sequences of record types followed by their values. Document paragraphs are shown in Table C and have the same color as their corresponding plans in Table E. During training, Puduppully and Lapata (2021) learn to predict a macro plan from a pool of paragraph plans, and produce a game summary based on it. Continuing with our example in Figure 1, 
Data-to-text Generation with Variational Sequential Planning
In this work, we present a model which interleaves macro planning with text generation (see Figure 2 for a sketch of the approach). We begin by selecting a plan from a pool of paragraph plans (see Table D in Figure 1), and generate the first paragraph by conditioning on it. We select the next plan by conditioning on the previous plan and the previously generated paragraph. We generate the next paragraph by conditioning on the currently selected plan, the previously predicted plan, and generated paragraph. We repeat this process until the final paragraph plan is predicted. We model the selection of paragraph plans as a sequential latent variable process which we argue is intuitive since content planing is inherently latent. Contrary to Puduppully and Lapata (2021), we do not a priori decide on a global macro plan. Rather our planning process is incremental and as a result less rigid. Planning is informed by generation and vice versa, which we argue should be mutually beneficial (they are conditioned on each other).
Data-to-text Generation with Variational Sequential Planning
Our contributions can be summarized as follows: (1) we decompose data-to-text generation into sequential plan selection and paragraph generation. The two processes are interleaved and generation proceeds incrementally. We look at what has been already generated, make a plan on what to discuss next, realize the plan, and repeat; (2) in contrast to previous models (Puduppully et al., 2019a;Puduppully and Lapata, 2021) where content plans are monolithic and determined in advance, our approach is more flexible, it simplifies modeling (we do not need to learn alignments between paragraph plans and summary paragraphs), and leads to sample efficiency in low resource scenarios; (3) our approach scales better for tasks involving generation of long multi-paragraph texts, as we do not need to specify the document plan in advance; (4) experimental results on English and German ROTOWIRE (Wiseman et al., 2017;Hayashi et al., 2019), and MLB (Puduppully et al., 2019b) show that our model is well-suited to long-form generation and generates more factual, coherent, and less repetitive output compared to strong baselines.
Data-to-text Generation with Variational Sequential Planning
The core technique of our model is learning the sequence of latent plans that guides long document generation. We consider a conditional generation setting where the input E is a set of paragraph plans and the output y 1:T are textual paragraphs verbalizing the selected sequence z = z 1:T . Our goal is to induce variables z that indicate which paragraphs are being talked about and in which order. Similar to previous work (Li and Rush, 2020;Fu et al., 2020), we model this process as a conditional generative model that produces both y and z and factorizes as:
Data-to-text Generation with Variational Sequential Planning
We use variational inference (Kingma and Welling, 2014;Rezende et al., 2014) to approximate the posterior with a parametrized distribution q φ (z|y, E) from which we sample values of z that are likely to produce y (see Doersch 2016 for a tutorial on this topic). Specifically, we employ an autoregressive inference model factorized as:
Data-to-text Generation with Variational Sequential Planning
Neural Parametrization At step t, we start with the encoding of previous paragraphs y <t and plans z <t (see Figure 3 left). Following Yang et al. (2016), we use a Bi-directional LSTM (BiLSTM) with a self-attention layer to encode paragraph y t as a vector r t y at step t:
Data-to-text Generation with Variational Sequential Planning
Note that we feed h t−1 y , representing the context of previous paragraphs, as additional input similar to Serban et al. (2017). Let r t z,1 , ..., r t z,l denote the encoding of tokens of the current plan where r t z,k is the output of the BiLSTM plan encoder and l the length of the chosen plan. We generate the next word as:
Data-to-text Generation with Variational Sequential Planning
where c denotes the context vector. In Equation 11, we use the output vector from Attn(·). FF gen (·) represents a feed-forward layer. In addition, we equip the decoder with copy attention (See et al., 2017) to enable copying tokens from z t . As part of this, we learn a probability for copy based on s i (Gehrmann et al., 2018). Once paragraph y t has been generated, we obtain its encoding r t y with
Data-to-text Generation with Variational Sequential Planning
Although we primarily focus on the inference, and how the latent plan can improve the generation of long documents, we note that the model sketched above could be parametrized differently, e.g., by replacing the encoder and decoder with pretrained language models like BART (Lewis et al., 2020). However, we leave this to future work.
Data-to-text Generation with Variational Sequential Planning
Advantageously, we can exploit oracle plans (see Table E in Figure 1 and the description in Section 4 for how these were created) to obtain weak labels z * which we use as distant supervision to the inference model: Such distant supervision is essential for stabilizing training (it would be extremely challenging to optimize the model in a fully unsupervised way) and for mitigating posterior collapse. We use Gumbel-Softmax (Maddison et al., 2017;Jang et al., 2017) for differentiable sampling (reparameterization) from q. The model is trained with scheduled sampling , and follows the curriculum learning strategy using linear decay scheduling. During earlier stages of training predicted plans are less accurate, and we thus sample from oracle plans at a rate which decays linearly with training:
Data-to-text Generation with Variational Sequential Planning
Data We performed experiments on the RO-TOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) datasets and the German RO-TOWIRE provided as part of the WNGT 2020 DGT shared task on "Document-Level Generation and Translation" (Hayashi et al., 2019). Statistics on these datasets are shown in All three datasets were preprocessed following the method of Puduppully and Lapata (2021). A paragraph plan for an entity is constructed by verbalizing its records in a fixed sequence of record type followed by its value. For example, pitcher B.Keller from Figure 1 would be verbalized as <PLAYER> B.Keller <H/V> V <W> 7 <L> 5 <IP> 8 <PH> 4 . . . . We denote this using the shorthand <V(B.Keller)>. The paragraph plan for an event is the verbalization of the players in the event followed by the verbalization of play-byplays. Candidate paragraph plans E are obtained by enumerating entities and events and their combinations (see Table D in Figure 1). Oracle macro plans are obtained by matching the mentions of entities and events in the gold summary with the input table. We make use of these oracle macro plans during training. The versions of MLB and ROTOWIRE released by Puduppully and Lapata (2021) contain paragraph delimiters for gold summaries; we preprocessed the German ROTOWIRE in a similar fashion. Table 1 also shows the average length of the macro plan in terms of the number of paragraph plans it contains. This is 10.6 for ROTOWIRE, 15.1 for MLB, and 9.5 for German RotoWire.
Data-to-text Generation with Variational Sequential Planning
Training Configuration We train our model with the AdaGrad optimizer (Duchi et al., 2011) and tune parameters on the development set. We use a learning rate of 0.15. We learn a joint subword vocabulary (Sennrich et al., 2016) for paragraph plans and summaries with 6K merge operations for ROTOWIRE, 16K merge operations for MLB, and 2K merge operations for German RO-TOWIRE. The model is implemented on a fork of OpenNMT-py (Klein et al., 2017). For efficiency, we batch using summaries instead of individual paragraphs. Batch sizes for MLB, ROTOWIRE, and German-ROTOWIRE are 8, 5, and 1 respectively. We set λ to 2 in Equation (18). In Equation (19), c is 1/100000 for MLB, 1/50000 for ROTOWIRE, and 1/30000 for German-ROTOWIRE. We set the temperature of Gumbel-Softmax to 0.1.
Data-to-text Generation with Variational Sequential Planning
During inference in MLB, similar to Puduppully and Lapata (2021), we block the repetition of paragraph plan bigrams (i.e., we disallow the repetition of (z t , z t+1 )) and select the paragraph plan with the next higher probability in Equation (8). In addition, we block consecutive repetitions, and more than two repetitions of a unigram. During training we observed high variance in the length of paragraphs y t since the same plan can result in a shorter or longer paragraph. For example, <V(B.Keller)> corresponds to two paragraphs (first and third paragraph) with different lengths in Figure 1. We found that this encourages the model to be conservative and generate relatively short output. We control the paragraph length (Fan et al., 2018) by creating discrete bins, each containing approximately an equal number of paragraphs. During training, we prepend the embedding of the bin to the current plan r t z (see Equation (11)). For inference, bins are tuned on the validation set.
Data-to-text Generation with Variational Sequential Planning
We run inference for 15 paragraphs on RO-TOWIRE and German ROTOWIRE, and for 20 paragraphs on MLB; we stop when the model predicts the end of paragraph plan token EOP. Unlike previous work (Wiseman et al., 2017;Puduppully et al., 2019a,b, inter alia), we do not make use of truncated Back Propagation Through Time (BPTT; Williams and Peng, 1990), as we incrementally generate paragraphs instead of long documents.
Data-to-text Generation with Variational Sequential Planning
(2) ED+CC, the best performing model of Wiseman et al. (2017). It consists of an encoder-decoder model equipped with attention and copy mechanisms. (3) NCP+CC, the micro planning model of Puduppully et al. (2019a). It first creates a content plan by pointing to input records through the use of Pointer Networks . The content plan is then encoded with a BiLSTM and decoded using another LSTM with an attention and copy mechanism. (4) ENT, the entity model of Puduppully et al. (2019b). It creates entity-specific representations which are updated dynamically. At each time step during decoding, their model makes use of hierarchical attention by attending over entity representations and the records corresponding to these. (5) MACRO, the two-stage planning model of Puduppully and Lapata (2021), which first makes use of Pointer Networks  to predict a macro plan from a set of candidate paragraph plans. The second stage takes the predicted plan as input and generates the game summary with a sequence-to-sequence model enhanced with attention and copy mechanisms. In addition, we compare with a variant of Macro enhanced with length control (+Bin).
Data-to-text Generation with Variational Sequential Planning
Our experiments were designed to explore how the proposed model compares to related approaches which are either not enhanced with planning modules or non-incremental. We also investigated the sample efficiency of these models and the quality of the predicted plans when these are available. The majority of our results focus on automatic evaluation metrics. We also follow previous work (Wiseman et al., 2017;Puduppully et al., 2019a,b;Puduppully and Lapata, 2021) in eliciting judgments to evaluate system output.
Data-to-text Generation with Variational Sequential Planning
We evaluate model output using BLEU (Papineni et al., 2002) with the gold summary as a reference. We also report model performance against the Information Extraction (IE) metrics of Wiseman et al. (2017) which are defined based on the output of an IE model which extracts entity (team and player names) and value (numbers) pairs from the summary and predicts the type of relation between them.
Data-to-text Generation with Variational Sequential Planning
Letŷ be the gold summary and y be the model output. Relation Generation (RG) measures the precision and count of relations obtained from y that are found in the input table. Content Selection (CS) measures the precision, recall, and F-measure of relations extracted from y also found inŷ. And Content Ordering (CO) measures the complement of the Damerau-Levenshtein distance between relations extracted from y andŷ. Higher values are better for RG Precision, CS F-measure, CO, and BLEU. We reuse the IE model from Puduppully et al. (2019a) for ROTOWIRE, Puduppully and Lapata (2021) for MLB, and Hayashi et al. (2019) for German ROTOWIRE. Our computation of IE metrics for all systems includes duplicate records (Puduppully and Lapata, 2021).
Data-to-text Generation with Variational Sequential Planning
In addition to IE-based metrics, we report the number of errors made by systems according to Number (incorrect number in digits, number spelled in words, etc.), Name (incorrect names of teams, players, days of week, etc.), and Word (errors in usage of words) following the classification of Thomson and Reiter (2020). We detect such errors automatically using the system of Kasner et al. (2021) which scored best against gold standard human annotations of the same type (Thomson and Table 2 summarizes our results on MLB. Our sequential planning model (SeqPlan) has the highest RG P among neural models and performs best in terms of CS F, CO, and BLEU. The variant of Macro with length control (+Bin) performs comparably or worse than Macro.
Data-to-text Generation with Variational Sequential Planning
English and German ROTOWIRE Results on ROTOWIRE are presented in Table 3 (top). In addition to Templ, ED+CC, NCP+CC, and ENT, we compare with the models of Wiseman et al. (2017) (WS-2017) and Rebuffel et al. (2020) (RBF-2020. WS-2017 is the best performing model of Wiseman et al. (2017). Note that ED+CC is an improved re-implementation of WS-2017. RBF-2020 represents the current state-of-the-art on ROTOWIRE, and comprises of a Transformer encoder-decoder architecture (Vaswani et al., 2017) with hierarchical attention on entities and their records. The models of Saleh et al. (2019), Iso et al. (2019), andGong et al. (2019) are not comparable as they make use of information additional to the table such as previous/next games or the author of the game summary. The model of  is also not comparable as it relies on a pretrained language model (Rothe et al., 2020) to generate the summary sentences.   We find that SeqPlan achieves highest RG P amongst neural models, and performs on par with Macro (it obtains higher BLEU but lower CS F and CO scores). The +Bin variant of Macro performs better on BLEU but worse on other metrics. As in Table 2, w Uniform struggles across metrics corroborating our hypothesis that latent sequential planning improves generation performance. The other two variants (w Oracle and 2-Stage) are worse than SeqPlan in RG P and CS F, comparable in CO, 5 We thank Hiroaki Hayashi for providing us with the output of the NCP+CC system.  and slightly higher in terms of BLEU.
Data-to-text Generation with Variational Sequential Planning
On German, our model is best across metrics achieving an RG P of 91.8% which is higher by 42% (absolute) compared to of Macro. In fact, the RG P of SeqPlan is superior to Saleh et al. (2019) whose model is pretrained with additional data and is considered state of the art (Hayashi et al., 2019). RG# is lower mainly because of a bug in the German IE which excludes number records. RG# for NCP+CC and Macro is too high because the summaries contain a lot of repetition. The same record will repeat at least once with NCP+CC and three times with Macro, whereas only 7% of the records are repeated with SeqPlan. Table 4 evaluates the quality of the plans inferred by our model on the ROTOWIRE dataset. As can be seen, SeqPlan is slightly worse than Macro in terms of CS F and CO. We believe this is because summaries in ROTOWIRE are somewhat formulaic, with a plan similar to Templ: an opening statement is followed by a description of the top scoring players, and a conclusion describing the next match. Such plans can be learnt well by Macro without access to the summary. MLB texts show a lot more diversity in terms of length, and the sequencing of entities and events. The learning problem is also more challenging, supported by the fact that the template system does not do very well in this domain (i.e., it is worse in BLEU, CS F, and CO compared to ROTOWIRE). In German ROTOWIRE, SeqPlan plans achieve higher CS F and CO than Macro. Table 5 reports complementary automatic metrics on English ROTOWIRE aiming to assess the factuality of generated output. We find that Templ has the least Number, Name, and double-double errors. This is expected as it simply reproduces   facts from the table. SeqPlan and Macro have similar Number errors, and both are significantly better than other neural models. SeqPlan has significantly more Name errors than Macro, and significantly fewer than other neural models. Inspection of Name errors revealed that these are mostly due to incorrect information about next games. Such information is not part of the input and models are prone to hallucinate. SeqPlan fares worse as it attempts to discuss next games for both teams while Macro focuses on one team only. In terms of double-double errors, SeqPlan is comparable to Macro, ENT and NCP+CC, and significantly better than WS-2017, ED+CC, and RBF-2020.
Data-to-text Generation with Variational Sequential Planning
We used the Amazon Mechanical Turk (AMT) crowdsourcing platform for our judgment elicitation study. To ensure consistent ratings (van der Lee et al., 2019) we required that raters have completed at least 1,000 tasks, and have at least 98% approval rate. Participants were restricted to English speaking countries (USA, UK, Canada, Australia, Ireland, or New Zealand) and were allowed to provide feedback or ask questions. Raters were paid an average of 0.35$ for each task, ensuring that the remuneration is higher than the minimum wage per hour in the US. We compared SeqPlan with Gold, Templ, ED+CC, and Macro; we did not compare against ENT as previous work (Puduppully and Lapata, 2021) has shown that it performs poorly against Macro. For ROTOWIRE, we additionally compared against RBF-2020.
Data-to-text Generation with Variational Sequential Planning
for long-form generation. Potential applications include summarizing books (Kryściński et al., 2021) where the output can be longer than 1,000 tokens or generating financial reports (Kogan et al., 2009;Händschke et al., 2018) where the output exceeds 9,000 tokens. Existing approaches for long-form generation summarize individual paragraphs independently (Kryściński et al., 2021) or adopt a hierarchical approach (Wu et al., 2021) where summaries of paragraphs form the basis of chapter summaries which in turn are composed into a book summary. Table 7 gives an example of SeqPlan output. We see that the game summary follows the macro plan closely. In addition, the paragraph plans and the paragraphs exhibit coherent ordering. Manual inspection of SeqPlan summaries reveals that a major source of errors in MLB relate to attention diffusing over long paragraph plans. As an example, consider the following paragraph produced by SeqPlan "Casey Kotchman had three hits and three RBIs , including a two-run double in the second inning that put the Angels up 2-0. Torii Hunter had three hits and drove in a run ." In reality, Torii Hunter had two hits but the model incorrectly generates hits for Casey Kotchman. The corresponding paragraph plan is 360 tokens long and attention fails to discern important tokens. A more sophisticated encoder, e.g., based on Transformers (Vaswani et al., 2017), could make attention more focused. In RO-TOWIRE, the majority of errors involve numbers (e.g., team attributes) and numerical comparisons. Incorporating pre-executed operations such as min, max (Nie et al., 2018) could help alleviate these errors.
Data-to-text Generation with Macro Planning
Data-to-text generation refers to the task of generating textual output from non-linguistic input (Reiter andDale, 1997, 2000;Gatt and Krahmer, 2018) such as databases of records, simulations of physical systems, accounting spreadsheets, or expert system knowledge bases. As an example, Figure 1 shows various statistics describing a major league baseball (MLB) game, including extracts from the box score (i.e., the performance of the two teams and individual team members who played as batters, pitchers or fielders; Tables (A)), play-by-play (i.e., the detailed sequence of each play of the game as it occurred; Table (B)), and a human written game summary (Table (C)).
Data-to-text Generation with Macro Planning
Traditional methods for data-to-text generation (Kukich, 1983;McKeown, 1992;Reiter and Dale, 1997) follow a pipeline architecture, adopting separate stages for text planning (determining which content to talk about and how it might be organized in discourse), sentence planning (aggregating content into sentences, deciding specific words to describe concepts and relations, and generating referring expressions), and linguistic realisation (applying the rules of syntax, morphology and orthographic processing to generate surface forms). Recent neural network-based approaches (Lebret et al., 2016;Mei et al., 2016;Wiseman et al., 2017) make use of the encoder-decoder architecture (Sutskever et al., 2014), are trained endto-end, and have no special-purpose modules for how to best generate a text, aside from generic mechanisms such as attention and copy (Bahdanau et al., 2015;Gu et al., 2016). The popularity of end-to-end models has been further boosted by the release of new datasets with thousands of inputdocument training pairs. The example shown in Figure 1 is taken from the MLB dataset (Puduppully et al., 2019b) which contains baseball game statistics and human written summaries (~25K instances). ROTOWIRE (Wiseman et al., 2017) is another widely used benchmark, which contains NBA basketball game statistics and their descriptions (~5K instances).
Data-to-text Generation with Macro Planning
Despite being able to generate fluent text, neural data-to-text generation models are often imprecise, prone to hallucination (i.e., generate text that is not supported by the input), and poor at content selection and document structuring (Wiseman et al., 2017). Attempts to remedy some of these issues focus on changing the way entities are represented (Puduppully et al., 2019b;Iso et al., 2019), allowing the decoder to skip low-confidence tokens to enhance faithful generation (Tian et al., 2019), and making the encoder-decoder architecture more modular by introducing micro planning (Puduppully et al., 2019a;Moryossef et al., 2019). Micro planning operates at the record level (see Ta- <P> Keller gave up a home run to the first batter of the game -Cedric Mullins -but quickly settled in to pitch eight strong innings in the Kansas City Royals' 9-2 win over the Baltimore Orioles in a matchup of the teams with the worst records in the majors. <P> Keller (7-5) gave up two runs and four hits with two walks and four strikeouts to improve to 3-0 with a 2.16 ERA in his last four starts. <P> Ryan O'Hearn homered among his three hits and drove in four runs, Whit Merrifield scored three runs, and Hunter Dozier and Cam Gallagher also went deep to help the Royals win for the fifth time in six games on their current homestand. <P> With the score tied 1-1 in the fourth, Andrew Cashner (4-13) gave up a sacrifice fly to Merrifield after loading the bases on two walks and a single. Dozier led off the fifth inning with a 423-foot home run to left field to make it 3-1. <P> The Orioles pulled within a run in the sixth when Mullins led off with a double just beyond the reach of Dozier at third, advanced to third on a fly ball and scored on Trey Mancini's sacrifice fly to the wall in right. <P> . . .   (Tables (A) and (B)). Macro plan for the game summary is shown at the bottom (Table (E)). <P> indicates paragraph delimiters. There is a plan for every paragraph in the game summary (correspondence shown in same color); <V(entity)> verbalizes entities, while <V(inning-T/B)> verbalizes events related to the top/bottom side of an inning (see Section 3.1). Set of candidate paragraph plans are shown above macro plan (Table (D)) and grouped into two types: plans describing a single entity/event or their combinations. Best viewed in color.
Data-to-text Generation with Macro Planning
tioned within a textual unit (e.g., a sentence) and how these should be structured (e.g., the sequence of records). An explicit content planner essentially makes the job of the neural network less onerous allowing to concentrate on producing fluent natural language output, without expending too much effort on content organization. In this work, we focus on macro planning, the high-level organization of information and how it should be presented which we argue is important for the generation of long, multi-paragraph documents (see text (C) in Figure 1). Problematically, modern datasets like MLB (Puduppully et al. 2019b; and also Figure 1) and ROTOWIRE (Wiseman et al., 2017) do not naturally lend them-selves to document planning as there is no explicit link between the summary and the content of the game (which is encoded in tabular form). In other words, the underlying plans are latent, and it is not clear how they might be best represented, i.e., as sequences of records from a table, or simply words. Nevertheless, game summaries through their segmentation into paragraphs (and lexical overlap with the input) give clues as to how content might be organized. Paragraphs are a central element of discourse (Chafe, 1979;Longacre, 1979;Halliday and Hasan, 1976), the smallest domain where coherence and topic are defined and anaphora resolution is possible (Zadrozny and Jensen, 1991). We therefore operationalize the macro plan for a game summary as a sequence of paragraph plans.
Data-to-text Generation with Macro Planning
We present a two-stage approach where macro plans are induced from training data (by taking the table and corresponding summaries into account) and then fed to the text generation stage. Aside from making data-to-text generation more interpretable, the task of generating a document from a macro plan (rather than a table) affords greater control over the output text and plays to the advantage of encoder-decoder architectures which excel at modeling sequences. We evaluate model performance on the ROTOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019b) benchmarks. Experimental results show that our plan-and-generate approach produces output which is more factual, coherent, and fluent compared to existing state-of-the-art models. Our code, trained models and dataset with macro plans can be found at https://github.com/ratishsp/ data2text-macro-plan-py.
Data-to-text Generation with Macro Planning
where e i is a paragraph plan. We model the process of generating output summary y given E as a two step process, namely the construction of a macro plan x based on the set of paragraph plans, followed by the generation of a summary given a macro plan as input. We now explain how E is obtained and each step is realized. We discuss our model considering mainly an example from the MLB dataset (Puduppully et al., 2019b) but also touch on how the approach can be straightforwardly adapted to ROTOWIRE (Wiseman et al., 2017).
Data-to-text Generation with Macro Planning
A macro plan consists of a sequence of paragraph plans separated by a paragraph discourse marker <P>, i.e., x = e i <P> e j . . . <P> e k where e i , e j , e k ∈ E. A paragraph plan in turn is a sequence of entities and events describing the game. By entities we mean individual players or teams and the information provided about them in box score statistics (see rows and column headings in Figure 1 Table (A)), while events refer to information described in play-by-play (see Table (B)). In baseball, plays are grouped in half-innings. During each half of an inning, a team takes its turn to bat (the visiting team bats in the top half and the home team in the bottom half). An example macro plan is shown at the bottom of Figure 1. Within a paragraph plan, entities and events are verbalized into a text sequence along the lines of Saleh et al. (2019). We make use of special tokens for the <TYPE> of record followed by the value of record from the table. We retain the same position for each record type and value. For example, batter C.Mullins from Figure 1 would be verbalized as <PLAYER>C.Mullins <H/V>H <AB>4 <BR>2 <BH>2 <RBI>1 <TEAM>Orioles . . . . For the sake of brevity we use shorthand <V(C.Mullins)> for the full entity.
Data-to-text Generation with Macro Planning
We provided our definition for macro plans in the previous sections, however, it is important to note that such macro plans are not readily available in data-to-text benchmarks like MLB (Puduppully et al., 2019b) and ROTOWIRE (Wiseman et al., 2017) which consist of tables of records r paired with a gold summary y (see Tables (A)-(C) in Figure 1). We now describe our method for obtaining macro plans x from r and y.
Data-to-text Generation with Macro Planning
Similar to Moryossef et al. (2019), we define macro plans to be conformant with gold summaries such that (1) they have the same splits into paragraphs -entities and events within a paragraph in y are grouped into a paragraph plan in x; and (2) the order of events and entities in a paragraph and its corresponding plan are identical. We con-struct macro plans by matching entities and events in the summary to records in the tables. Furthermore, paragraph delimiters within summaries form natural units which taken together give rise to a high-level document plan.
Data-to-text Generation with Macro Planning
We match entities in summaries with entities in tables using exact string match, allowing for some degree of variation in the expression of team names (e.g., A's for Athletics and D-backs for Diamondbacks). Information pertaining to innings appears in the summaries in the form of ordinal numbers (e.g., first, ninth ) modifying the noun inning and can be relatively easily identified via pattern matching (e.g., in sentences like "Dozier led off the fifth inning"). However, there are instances where the mention of innings is more ambiguous (e.g., "With the scored tied 1-1 in the fourth, Andrew Cashner (4-13) gave up a sacrifice fly"). We could disambiguate such mentions manually and then train a classifier to learn to predict whether an inning is mentioned. Instead, we explore a novel annotationfree method which makes use of the pretrained language model GPT2 (Radford et al., 2019). Specifically, we feed the context preceding the ordinal number to GPT2 (i.e., the current paragraph up to the ordinal number and the paragraph preceding it) and if inning appears in the top 10 next word predictions, we consider it a positive match. On a held out dataset, this method achieves 98% precision and 98% recall at disambiguating inning mentions.
Data-to-text Generation with Macro Planning
The input to our model is a set of paragraph plans each of which is a sequence of tokens. We first compute paragraph plan representations ∈ R n , and then apply a contextualization and content planning mechanism similar to planning modules introduced in earlier work (Puduppully et al., 2019a;Chen and Bansal, 2018). Predicted macro plans serve as input to our text generation model which adopts an encoder-decoder architecture (Bahdanau et al., 2015;Luong et al., 2015).
Data-to-text Generation with Macro Planning
Paragraph Plan Representation We encode tokens in a verbalized paragraph plan e i as {e i, j } |e i | j=1 with a BiLSTM (Figure 2 bottom part). To reflect the fact that some records will be more important than others, we compute an attention weighted sum of {e i, j } |e i | j=1 following Yang et al. (2016). Let d ∈ R n denote a randomly initialized query vector learnt jointly with the rest of parameters. We compute attention values α i, j over d and paragraph plan token representation e i, j :
Data-to-text Generation with Macro Planning
where z <k = z 1 . . . z k−1 . We use Pointer Networks (Vinyals et al., 2015) to model p(z k |z <k , E) as:
Data-to-text Generation with Macro Planning
To compute p(y|x), we use an encoder-decoder architecture enhanced with an attention mechanism (Bahdanau et al., 2015;Luong et al., 2015). We encode macro plan x with a bidirectional LSTM (Hochreiter and Schmidhuber, 1997). At time step t, we lookup the embedding of the previously predicted word y t−1 and feed it as input to the decoder which is another LSTM unit. The decoder attends over hidden states of the macro plan to predict y t . We further incorporate a copy mechanism (Gulcehre et al., 2016) in the decoder to enable copying values directly from the macro plan.
Data-to-text Generation with Macro Planning
We deterministically obtainx fromẑ, and output summaryŷ among candidate outputs y given macro planx as input: We reconstructed the MLB dataset, as the version released by Puduppully et al. (2019b) had removed all paragraph delimiters from game summaries. Specifically, we followed their methodology and downloaded the same summaries from the ESPN website 2 and added the <P> delimiter to paragraphs in the summaries. 3 ROTOWIRE does not have paragraph delimiters in game summaries either. We reverse engineered these as follows:
Data-to-text Generation with Macro Planning
Training Configuration We tuned the model hyperparameters on the development set. For training the macro planning and the text generation stages, we used the Adagrad (Duchi et al., 2011) optimizer. Furthermore, the text generation stage made use of truncated BPTT (Williams and Peng, 1990) with truncation length 100. We learn subword vocabulary (Sennrich et al., 2016) for paragraph plans in the macro planning stage. We used 2.5K merge operations for ROTOWIRE and 8K merge operations for MLB. In text generation, we learn a joint subword vocabulary for the macro plan and game summaries. We used 6K merge operations for RO-TOWIRE and 16K merge operations for MLB. All models were implemented on OpenNMT-py (Klein et al., 2017). We add to set E the paragraph plans corresponding to the output summary paragraphs, to ensure full coverage during training of the macro planner. During inference for predicting macro plans, we employ length normalization (Bahdanau et al., 2015) to avoid penalizing longer outputs; specifically, we divide the scores of beam search by the length of the output. In addition, we adopt bigram blocking (Paulus et al., 2018). For MLB, we further block beams containing more than two repetitions of a unigram. This helps improve the diversity of the predicted macro plans.
Data-to-text Generation with Macro Planning
System Comparisons We compared our model against the following systems: (1) the Templatebased generators from Wiseman et al. (2017) for ROTOWIRE and Puduppully et al. (2019b) for MLB. Both systems apply the same principle, they emit a sentence about the teams playing in the game, followed by player-specific sentences, and a closing sentence. MLB additionally contains a description of play-by-play; (2) ED+CC, the best performing system in Wiseman et al. (2017), is a vanilla encoder-decoder model equipped with an attention and copy mechanism; (3) NCP+CC, the micro planning model of Puduppully et al. (2019a), generates content plans from the table by making use of Pointer networks (Vinyals et al., 2015) to point to records; content plans are encoded with a BiLSTM and the game summary is decoded using another LSTM with attention and copy; (4) ENT, the entity-based model of Puduppully et al. (2019b), creates dynamically updated entity-specific representations; the text is generated conditioned on the data input and entity memory representations using hierarchical attention at each time step.
Data-to-text Generation with Macro Planning
Automatic Evaluation For automatic evaluation, following earlier work (Wiseman et al. 2017;Puduppully et al. 2019a,b, inter alia) we report BLEU (Papineni et al., 2002) with the gold summary as reference but also make use of the Infor-  mation Extraction (IE) metrics from Wiseman et al. (2017) which are defined over the output of an IE system; the latter extracts entity (players, teams) and value (numbers) pairs in a summary, and then predicts the type of relation. For instance, given the pair Kansas City Royals, 9, it would predict their relation as TR (i.e., Team Runs). Training data for the IE system is obtained by checking for matches between entity, value pairs in the gold summary and entity, value, record type triplets in the table.
Data-to-text Generation with Macro Planning
We reused the IE model from Puduppully et al. (2019a) for ROTOWIRE but retrained it for MLB to improve its precision and recall. Furthermore, the implementation of Wiseman et al. (2017) computes RG, CS, and CO excluding duplicate relations. This artificially inflates the performance of models whose outputs contain repetition. We include duplicates in the computation of the IE metrics (and recreate them for all comparison systems).  (Vaswani et al., 2017) with a hierarchical attention mechanism over entities and records within entities. The models of Saleh et al. make use of additional information not present in the input (e.g., previous/next games, summary writer) and are not directly comparable to the systems in Table 2. Results for the MLB test set are in the bottom portion of Table 2. Templ has the highest RG precision and count on both datasets. This is not surprising, by design Templ is always faithful to the input. However, notice that it achieves the lowest BLEU amongst comparison systems indicating that it mostly regurgitates facts with low fluency. Macro achieves the highest RG precision amongst all neural models for ROTOWIRE and MLB. We obtain an absolute improvement of 5.9% over ENT for ROTOWIRE and 13.3% for MLB. In addition, Macro achieves the highest CS F-measure for both datasets. On ROTOWIRE, Macro achieves the highest CO score, and the highest BLEU on MLB. On ROTOWIRE, in terms of BLEU, Macro is worse than comparison models (e.g., NCP+CC or ENT). Inspection of the output showed that the opening paragraph, which mostly describes how the two teams fared, is generally shorter in Macro, leading to shorter summaries and thus lower BLEU. There is high variance in the length of the opening paragraph in the training data and Macro verbalizes the corresponding plan conservatively. Ideas such as length normalisation (Wu et al., 2016) or length control (Kikuchi et al., 2016;Takeno et al., 2017;Fan et al., 2018) could help alleviate this; however, we do not pursue them further for fair comparison with the other models.
Data-to-text Generation with Macro Planning
We also asked participants to assess model output in terms of relation generation, grammaticality, coherence, and conciseness (Wiseman et al., 2017;Puduppully et al., 2019a,b), For ROTOWIRE, we compared Macro against RBF-2020 4 , ED+CC, Gold, and Templ. For MLB, we compared Macro against ENT, ED+CC, Gold, and Templ.
Data-to-text Generation with Macro Planning
Our results show that macro planning is more advantageous for generation tasks expected to produce longer texts with multiple discourse units, and could be easily extended to other sports domains such as cricket (Kelly et al., 2009) or American football (Barzilay and Lapata, 2005). Other approaches focusing on micro planning (Puduppully et al., 2019a;Moryossef et al., 2019) might be better tailored for generating shorter texts. There has been a surge of datasets recently focusing on singleparagraph outputs and the task of content selection such as E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and WikiBio (Lebret et al., 2016;Perez-Beltrachini and Lapata, 2018). We note that in our model content selection takes place during macro planning and text generation. The results in Table 2 show that Macro achieves the highest CS F-measure on both datasets indicating that the document as a whole and individual sentences discuss appropriate content.
Data-to-text Generation with Macro Planning
Despite promising results, there is ample room to improve macro planning, especially in terms of the precision of RG (see Table 2, P% column of RG). We should not underestimate that Macro must handle relatively long inputs (the average input length in the MLB development set is~3100 tokens) which are challenging for the attention mechanism. Consider the following output of our model on the MLB dataset: Ramirez's two-run double off Joe Blanton tied it in the sixth, and Brandon Moss added a two-out RBI single off Alan Embree to give Boston a 3-2 lead. Here, the name of the pitcher should have been Joe Blanton instead of Alan Embree. In fact, Alan Embree is the pitcher for the following play in the half inning. In this case, attention diffuses over the relatively long MLB macro plan, leading to inaccurate content selection. We could alleviate this problem by adopting a noisy channel decomposition (Yee et al., 2019;Yu et al., 2020), i.e., by learning two different distributions: a conditional model which provides the probability of translating a paragraph plan to text and a language model which provides an unconditional estimate of the output (i.e., the whole game summary). However, we leave this to future work.
Data-to-text Generation with Macro Planning
For ROTOWIRE, the main source of errors is the model's inability to understand numbers. For example, Macro generates the following output The Lakers were the superior shooters in this game, going 48 percent from the field and 30 percent from the three-point line, while the Jazz went 47 percent from the floor and 30 percent from beyond the arc.. Here, 30 percent should have been 24 percent for the Lakers but the language model expects a higher score for the three-point line, and since 24 is low (especially compared to 30 scored by the Jazz), it simply copies 30 scored by the Jazz instead. A mechanism for learning better representations for numbers (Wallace et al., 2019) or executing operations such as argmax or minus (Nie et al., 2018) should help alleviate this problem.
Planning with Learned Entity Prompts for Abstractive Summarization
Jones (1993) described text summarization -the task of generating accurate and concise summaries from source document(s) -as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing * Work done while Ryan was at Google. the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation (Luhn, 1958;McKeown and Radev, 1995;Barzilay and Elhadad, 1997;Mihalcea and Tarau, 2004).
Planning with Learned Entity Prompts for Abstractive Summarization
State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanism (Hochreiter and Schmidhuber, 1997;Bahdanau et al., 2015;See et al., 2017), Transformer architectures with multi-headed selfattention (Vaswani et al., 2017), and large pretrained conditional language models Song et al., 2019;Lewis et al., 2019;Rothe et al., 2020a;Raffel et al., 2019;Zhang et al., 2019). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2019;Raffel et al., 2019;Zhang et al., 2019), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020;Kryscinski et al., 2020;Gabriel et al., 2020).
Planning with Learned Entity Prompts for Abstractive Summarization
In this paper, we investigate Entity Chainsordered sequences of entities 1 in the summaryas an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1,right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder-decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and Bill-Sum (Kornilova and Eidelman, 2019), and show that the state-of-the-art PEGASUS (Zhang et al., 2019) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to PEGASUS pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gapsentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of ROUGE. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state-of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization. There is a growing interest in enabling users to specify high level characteristics such as length, keywords, topic, in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018;He et al., 2020;Dou et al., 2020) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.
Planning with Learned Entity Prompts for Abstractive Summarization
Modeling Entity-Level Lexical Cohesion and Coherence. As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976;Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in FROST) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c, 4 our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.
Planning with Learned Entity Prompts for Abstractive Summarization
We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large 3 The PEGASUS objective uses a summary-document length ratio to select n. This could lead to an undesirably long summary when the input document is very long. Modeling such summaries prepended with long entity chains effectively is beyond the limit of our decoders (256 sentencepieces). Hence, we set n = 5. 4 Here, s is not strictly constrained to the entity chain c. We hope that this will happen given c is extracted from s during the training time. Future work will focus on constraining s to c, e.g., using a checklist model (Kiddon et al., 2016) or entity-chain constrained decoding (Mao et al., 2020). architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8. 5
Planning with Learned Entity Prompts for Abstractive Summarization
Abstractive Summarization Datasets. We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019) and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets 6 . We use the original train/validation/test splits for them. For BillSum where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and, 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
For entity-level faithfulness, we report on ENTPREC, measuring the precision of entities in predicted summaries against their input documents. ENTF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018;Falke et al., 2019;Kryscinski et al., 2019). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).
Planning with Learned Entity Prompts for Abstractive Summarization
Summary-level approach first generates a summary-level content plan c = c 1 ||| . . . |||c m and then continue generating the summary s with a probability p(s|c; d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentencelevel planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentencelevel and summary-level entity chains in Figure 2. We finetuned our large models initialized with the PEGASUS checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented  Figure 4: Finetuning results on the XSum validation set using one of the base-sized pretrained models: PEGASUS, FROST(F) and FROST(P+F). All pretrained models were trained for 1.5m steps. See text for more details. We only report on a subset of measures, similar observations were made for other measures.
Planning with Learned Entity Prompts for Abstractive Summarization
The pretraining for content planning and generation in FROST (ECPP) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level ROUGE for CNN/DailyMail and XSum. Our FROST models establish new state-of-the-art ROUGE results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2020) on ROUGE scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.
Planning with Learned Entity Prompts for Abstractive Summarization
The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning ROUGE Table 4: Faithfulness assessment using automatic (Entailment and ENTPREC) and human evaluations, and overall quality assessment by humans for models presented in Table 3 Maynez et al., 2020)), the dropprompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents. The drop-prompt mechanism doesn't help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.
Planning with Learned Entity Prompts for Abstractive Summarization
The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using ENT-PREC and by humans for faithfulness. We don't observe similar improvements in entailment. Contrary to findings in (Maynez et al., 2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.
Planning with Learned Entity Prompts for Abstractive Summarization
We compare the decoding strategy in FROST to encoder-guided control summarization systems (He et al., 2020;Dou et al., 2020). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k; d) to generate the summary. We also finetune PEGASUS where the entity chain (c) is encoded along with the input document d (as d; c or c; d). We only report the oracle results (with c oracle ) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.
Data-to-Text Generation with Content Selection and Planning
Data-to-text generation broadly refers to the task of automatically producing text from non-linguistic input (Reiter and Dale 2000;Gatt and Krahmer 2018). The input may be in various forms including databases of records, spreadsheets, expert system knowledge bases, simulations of physical systems, and so on. Table 1 shows an example in the form of a database containing statistics on NBA basketball games, and a corresponding game summary.
Data-to-Text Generation with Content Selection and Planning
Traditional methods for data-to-text generation (Kukich 1983;McKeown 1992) implement a pipeline of modules including content planning (selecting specific content from some input and determining the structure of the output text), sentence planning (determining the structure and lexical content of each sentence) and surface realization (converting the sentence plan to a surface string). Recent neural generation systems (Lebret et al. 2016;Mei et al. 2016;Wiseman et al. 2017) do not explicitly model any of these stages, rather they are trained in an end-to-end fashion using the very successful encoder-decoder architecture (Bahdanau et al. 2015) as their backbone.
Data-to-Text Generation with Content Selection and Planning
are parameters, and |V y | is the output vocabulary size. We further augment the decoder with a copy mechanism, i.e., the ability to copy words directly from the value portions of records in the content plan (i.e., {z k } |z| k=1 ). We experimented with joint (Gu et al. 2016) and conditional copy methods (Gulcehre et al. 2016). Specifically, we introduce a variable u t ∈ {0, 1} for each time step to indicate whether the predicted token y t is copied (u t = 1) or not (u t = 0). The probability of generating y t is computed by:
Data-to-Text Generation with Content Selection and Planning
where β t,k and p gen (y t |y <t , z, r) are computed as in Equations (1)-(2), and w u ∈ R n , b u ∈ R are parameters. Following Gulcehre et al. (2016) and Wiseman et al. (2017), if y t appears in the content plan during training, we assume that y t is copied (i.e., u t = 1). 2
Data-to-Text Generation with Content Selection and Planning
Training Configuration We validated model hyperparameters on the development set. We did not tune the dimensions of word embeddings and LSTM hidden layers; we used the same value of 600 reported in Wiseman et al. (2017). We used one-layer pointer networks during content planning, and two-layer LSTMs during text generation. Input feeding (Luong et al. 2015) was employed for the text decoder. We applied dropout (Zaremba et al. 2014) at a rate of 0.3. Models were trained for 25 epochs with the Adagrad optimizer (Duchi et al. 2011); the initial learning rate was 0.15, learning rate decay was selected from {0.5, 0.97}, and batch size was 5. For text decoding, we made use of BPTT (Mikolov et al. 2010) and set the truncation size to 100. We set the beam size to 5 during inference. All models are implemented in OpenNMT-py (Klein et al. 2017).
Data-to-Text Generation with Content Selection and Planning
Letŷ be the gold output, and y the system output. Content selection (CS) measures how well (in terms of precision and recall) the records extracted from y match those found inŷ. Relation generation (RG) measures the factuality of the generation system as the proportion of records extracted from y which are also found in r (in terms of precision and number of unique relations). Content ordering (CO) measures how well the system orders the records it has chosen and is computed as the normalized Damerau-Levenshtein Distance between the sequence of records extracted from y andŷ. In addition to these metrics, we report BLEU (Papineni et al. 2002), with human-written game summaries as reference.
Data-to-Text Generation with Content Selection and Planning
In the second experiment, we assessed the generation quality of our model. We elicited judgments for the same 30 games used in the first study. For each game, participants were asked to compare a human-written summary, NCP with conditional copy (NCP+CC), Wiseman et al.'s (2017) best model, and the template system. Our study used Best-Worst Scaling (BWS; Louviere, Flynn, and Marley 2015), a technique shown to be less labor-intensive and providing more reliable results as compared to rating scales (Kiritchenko and Mohammad 2017). We arranged every 4-tuple of com-5 All significance differences reported throughout this paper are with a level less than 0.05.  Table 7: Average number of supporting (#Support) and contradicting (#Contra) facts in game summaries and bestworst scaling evaluation (higher is better) for grammaticality (Gram), Coherence (Cohere), and Conciseness (Concise).