Are there any datasets available fro table-to-text generation tasks in NLP?
Multiple data-to-text and table-to-text tasks have
been presented in the literature, such as WebNLG
(Gardent et al., 2017a; Colin et al., 2016; Gar-
dent et al., 2017b), DART (Nan et al., 2020),
ToTTo (Parikh et al., 2020), and WikiTableT (Chen
et al., 2021), which primarily consist of data from
general-purpose sources like Wikipedia.
DART: Open-Domain Structured Data Record to Text Generation
We present DART, an open domain structured DAta Record to Text generation dataset with over 82k instances (DARTs). Data-to-text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and contain nontrivial structures. To this end, we propose a procedure of extracting semantic triples from tables that encodes their structures by exploiting the semantic dependencies among table headers and the table title. Our dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing. We present systematic evaluation on DART as well as new state-of-the-art results on WebNLG 2017 to show that DART (1) poses new challenges to existing data-to-text datasets and (2) facilitates out-of-domain generalization. Our data and code can be found at https://github.com/Yale-LILY/dart.
[{"first":"Linyong","last":"Nan","middle":[]},{"first":"Dragomir","last":"Radev","middle":[]},{"first":"Rui","last":"Zhang","middle":[]},{"first":"Amrit","last":"Rau","middle":[]},{"first":"Abhinand","last":"Sivaprasad","middle":[]},{"first":"Chiachun","last":"Hsieh","middle":[]},{"first":"Xiangru","last":"Tang","middle":[]},{"first":"Aadit","last":"Vyas","middle":[]},{"first":"Neha","last":"Verma","middle":[]},{"first":"Pranav","last":"Krishna","middle":[]},{"first":"Yangxiaokang","last":"Liu","middle":[]},{"first":"Nadia","last":"Irwanto","middle":[]},{"first":"Jessica","last":"Pan","middle":[]},{"first":"Faiaz","last":"Rahman","middle":[]},{"first":"Ahmad","last":"Zaidi","middle":[]},{"first":"Mutethia","last":"Mutuma","middle":[]},{"first":"Yasin","last":"Tarabar","middle":[]},{"first":"Ankit","last":"Gupta","middle":[]},{"first":"Tao","last":"Yu","middle":[]},{"first":"Yi Chern","last":"Tan","middle":[]},{"first":"Xi Victoria","last":"Lin","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]},{"first":"Richard","last":"Socher","middle":[]},{"first":"Nazneen Fatema","last":"Rajani","middle":[]}]
The WebNLG Challenge: Generating Text from RDF Data
The WebNLG challenge consists in mapping sets of RDF triples to text. It provides a common benchmark on which to train, evaluate and compare “microplanners”, i.e. generation systems that verbalise a given content by making a range of complex interacting choices including referring expression generation, aggregation, lexicalisation, surface realisation and sentence segmentation. In this paper, we introduce the microplanning task, describe data preparation, introduce our evaluation methodology, analyse participant results and provide a brief description of the participating systems.
[{"first":"Claire","last":"Gardent","middle":[]},{"first":"Anastasia","last":"Shimorina","middle":[]},{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Laura","last":"Perez-Beltrachini","middle":[]}]
The WebNLG Challenge: Generating Text from DBPedia Data
Citation for published version: Colin, E, Gardent, C, M’rabet, Y, Narayan, S & Perez, L 2016, The WebNLG Challenge: Generating Text from DBPedia Data. in Proceedings of The 9th International Natural Language Generation conference. Association for Computational Linguistics, pp. 163-167, 9th International Natural Language Generation conference , Edinburgh, United Kingdom, 5/09/16. https://doi.org/10.18653/v1/W16-6626
[{"first":"Emilie","last":"Colin","middle":[]},{"first":"Claire","last":"Gardent","middle":[]},{"first":"Yassine","last":"M’rabet","middle":[]},{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Laura","last":"Perez-Beltrachini","middle":[]}]
Creating Training Corpora for NLG Micro-Planners
In this paper, we present a novel framework for semi-automatically creating linguistically challenging micro-planning data-to-text corpora from existing Knowledge Bases. Because our method pairs data of varying size and shape with texts ranging from simple clauses to short texts, a dataset created using this framework provides a challenging benchmark for microplanning. Another feature of this framework is that it can be applied to any large scale knowledge base and can therefore be used to train and learn KB verbalisers. We apply our framework to DBpedia data and compare the resulting dataset with Wen et al. 2016’s. We show that while Wen et al.’s dataset is more than twice larger than ours, it is less diverse both in terms of input and in terms of text. We thus propose our corpus generation framework as a novel method for creating challenging data sets from which NLG models can be learned which are capable of handling the complex interactions occurring during in micro-planning between lexicalisation, aggregation, surface realisation, referring expression generation and sentence segmentation. To encourage researchers to take up this challenge, we made available a dataset of 21,855 data/text pairs created using this framework in the context of the WebNLG shared task.
[{"first":"Claire","last":"Gardent","middle":[]},{"first":"Anastasia","last":"Shimorina","middle":[]},{"first":"Shashi","last":"Narayan","middle":[]},{"first":"Laura","last":"Perez-Beltrachini","middle":[]}]
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Datasets for data-to-text generation typically focus either on multi-domain, single-sentence generation or on single-domain, long-form generation. In this work, we cast generating Wikipedia sections as a data-to-text generation task and create a large-scale dataset, WikiTableT, that pairs Wikipedia sections with their corresponding tabular data and various metadata. WikiTableT contains millions of instances, covering a broad range of topics, as well as a variety of flavors of generation tasks with different levels of flexibility. We benchmark several training and decoding strategies on WikiTableT. Our qualitative analysis shows that the best approaches can generate fluent and high quality texts but they struggle with coherence and factuality, showing the potential for our dataset to inspire future work on long-form generation.
[{"first":"Mingda","last":"Chen","middle":[]},{"first":"Sam","last":"Wiseman","middle":[]},{"first":"Kevin","last":"Gimpel","middle":[]}]
ToTTo: A Controlled Table-To-Text Generation Dataset
We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.
[{"first":"Ankur","last":"Parikh","middle":["P."]},{"first":"Xuezhi","last":"Wang","middle":[]},{"first":"Sebastian","last":"Gehrmann","middle":[]},{"first":"Manaal","last":"Faruqui","middle":[]},{"first":"Bhuwan","last":"Dhingra","middle":[]},{"first":"Diyi","last":"Yang","middle":[]},{"first":"Dipanjan","last":"Das","middle":[]}]
DART: Open-Domain Structured Data Record to Text Generation
Automatically generating textual descriptions from structured data improves the accessibility of knowledge bases to lay users. Such applications include explaining data records to non-experts (Cawsey et al., 1997), writing sports news (Chen and Mooney, 2008), summarizing information in multiple documents (Fan et al., 2019), and generating dialogue responses (Wen et al., 2015).
DART: Open-Domain Structured Data Record to Text Generation
While significant progress has been made in this field, there are still several issues with existing Data-to-Text datasets. First, they adopt a flat ontology structure of the data, such as slot-value pairs for data records (Lebret et al., 2016;Novikova et al., 2017b) or flat schema for tables (Wiseman et al., * Now at Facebook AI. 2017;Chen et al., 2020a;Parikh et al., 2020). This flat structure is not powerful enough to encode rich semantic relationships in the ontology of the structured data, especially tables, whose representation can be further improved with these semantic knowledge. Second, some of the datasets only focus on a small number of domains or knowledge graphs, therefore providing limited number of predicates and data ontologies. For example, E2E (Novikova et al., 2017b) on restaurants and WebNLG (Gardent et al., 2017) on 15 categories from DBPedia. Furthermore, some of them only have loose alignments between data input and sentence due to the nature of the task (Wiseman et al., 2017) and the automatic generation procedure (Vougiouklis et al., 2018;Elsahar et al., 2018).
DART: Open-Domain Structured Data Record to Text Generation
To address some of these issues and to encourage further research in natural language generation from structured data, we introduce DART, a large and open-domain structured DAta-Record-to-Text generation corpus. The goal of DART is to harvest the diverse predicates occurred in Wikipedia tables, which is significantly richer than those defined in the domain specific ontologies E2E and WebNLG were built on (Table 2). We also introduce a novel tree ontology annotation approach on tables, which converts a flat table schema into a tree structured semantic frame. The tree ontology reflects the core and auxiliary relations in the table schema, and naturally occurs across many domains. As a result, DART provides high-quality sentence annotations to tree structured semantic frames extracted from various data sources, including Wik-iSQL (Zhong et al., 2017) and WikiTableQuestions (Pasupat and Liang, 2015), two open-domain question answering datasets, as well as E2E (Novikova et al., 2017b) and WebNLG (Gardent et al., 2017) ( Figure 1). We evaluated several state-of-the-art data-to-text models on DART, and found that while these models achieve impressive performance on domain-specific datasets, their performance suffers on DART due to its open-domain nature and richer semantic structures.
DART: Open-Domain Structured Data Record to Text Generation
Our contributions are as follows.
DART: Open-Domain Structured Data Record to Text Generation
(1) We present a large and open-domain corpus for structured data record to text generation, annotated with tree ontologies converted from the table. This hierarchical input differentiates our corpus from existing datato-text corpora.
DART: Open-Domain Structured Data Record to Text Generation
(2) We benchmark several stateof-the-art data-to-text models to show that DART introduces new generalization challenges. (3) We demonstrate that using DART for data augmentation improves the performance of existing models on the WebNLG 2017 dataset. We expect the results to generalize to other data-to-text datasets given the open-domain nature of DART.
DART: Open-Domain Structured Data Record to Text Generation
As shown in Figure 1, DART is constructed from three different sources: (1) human annotation on Wikipedia tables from two table semantic parsing and question answering datasets WikiSQL and Wik-iTableQuestions ( § 2.1), (2) automatic conversion of questions in WikiSQL to declarative sentences ( § 2.2), and (3) incorporation of existing datasets including WebNLG 2017 and Cleaned E2E ( § 2.3). After collecting the triple-set, sentence pairs from various data sources, we manually canonicalized the predicates and show that DART covers a broad range of topics ( § 2.4). Finally, we discuss the data split in § 2.5.
DART: Open-Domain Structured Data Record to Text Generation
Tables are a major source of structured data that contain a wealth of information complementary to text and knowledge graphs. We aim to collect triple-set, sentence pairs from open-domain Wikipedia tables. However, table schema are flat, making them not directly usable for building subject-predicate-object triples to capture rich relationships in the data. As shown in Figure 2, we propose a two-stage annotation process that involves two groups of annotators: internal annotators and Amazon Mechanical Turk 1 workers. In the first stage, skilled internal annotators specify the parent of every column header to construct a tree-structured ontology for each table. In the second stage, both internal and external annotators provide a sentential description of the 1 https://www.mturk.com/ highlighted cells in a row that are automaticallychosen based on the ontology.
DART: Open-Domain Structured Data Record to Text Generation
Tree Ontology Annotation For each column in a given table, our internal annotators labeled its ontological parent. In Figure 2, for example, the annotator would provide the sequence {NULL, TEAM, STADIUM, STADIUM, TEAM} as the parent of each column -column TEAM has no parent, STADIUM has parent TEAM, and so on. In many cases, the relationship between a parent column and its child column can be conceptualized as a "has-a" relationship. For tables that are malformed or have duplicate or missing column names (as shown in Figure  5 of the Appendix), annotators either changed or added appropriate column names in order to fit these patterns. For each table we generate an ontology tree whose root is always [TABLECONTEXT]. This root node either has (1) one child node [TI-TLE] in the cases where the table title is the subject of entire table, or (2) column header node(s) and a [TITLE] node as children, as shown in Figure 2. This is because in some tables, the table title itself is more appropriate to be the root of the ontology tree (example shown in Figure 6 of the Appendix). In these cases, annotators assigned the special token [TITLE] as the parent of the relevant column nodes. For other tables, title usually provides important context for understanding the table's rows (example shown in Figure 7 of the Appendix). In such cases, [TITLE] is made a child of [TABLE-CONTEXT] together with the column headers that are appropriate.
DART: Open-Domain Structured Data Record to Text Generation
We evaluate the quality of the initial tree ontology annotation and made corrections with the following procedure: (1) reject and request corrections from the original annotators if the provided ontology is disconnected or contains a cycle, (2) verify that all column headers appear as a node in the tree. For many tables, the determination of an ontology is a subjective process with many "correct" answers -for example, swapping the positions of TEAM and CITY in the tree in Figure 2 produces an equally valid ontology for the referenced table.
DART: Open-Domain Structured Data Record to Text Generation
If there are multiple ways to construct an ontology based on annotators' decisions of attribute relationships among column headers, we manually unify the annotations for similar tables (for examples, tables about athletes in different sports). The ontologies exhibit a great deal of structural variety. Relevant statistics are summarized in Table 7 and Figure 3 of the Appendix.   Connected Component Extraction After we annotated the ontology, we automatically choose a subset of cells for a selected table row to form the triple set. Randomly selecting cells leads to poor quality annotation as the selected data could lack a subject, lack cohesion, or would require information not encoded in the ontology to form a coherent sentence. For example, in Figure 2, if only two nodes CITY and CAPACITY were highlighted then a coherent sentence cannot be produced as there is no direct logical relationship (functional dependency) between them. To solve these issues, instead of randomly selecting cells in a row, we extract connected components from the ontology.
DART: Open-Domain Structured Data Record to Text Generation
The extracted components have two controllable properties: size and shape. To create variation in size, we randomly sampled between [2, 5]. The shape is determined by two numbers: the number of sibling node pairs and parent-child node pairs. Increasing the number of sibling node pairs creates a wider tree, while increasing the latter creates a deeper tree. We created a sliding scale between width and depth using an expansion parameter, p. We recursively visit a node if it has children with probability p and otherwise move to a sibling if it exists. If p = 1, the search becomes a DFS and if p = 0, it becomes BFS. We found that randomly selecting p from 0.5 to 0.7 created a reasonable variation in extracted component shapes. This ensures the balance between breadth and depth of ontology coverage of the selected cells, therefore ensuring the quality of the sentence annotation.
DART: Open-Domain Structured Data Record to Text Generation
Sentence Annotation Given the table, title, and connected highlighted cells of a row, annotators We collect the parent-child relations between columns from internal annotators (yellow is parent, green is child). Then, we collect a surface realization of the cells highlighted in orange. Middle panel: We use the provided parent-child relations to construct an ontology tree on the columns, then select the nodes corresponding to the highlighted cells. We gather a connected subtree by collecting all nodes leading up to the highlighted cells' lowest common ancestor. Bottom panel: We extract a set of triples from the subtree as shown. This triple-set is paired with the provided realization to form a DART instance.
DART: Open-Domain Structured Data Record to Text Generation
were asked to write a description of the highlighted cells. We encouraged the annotators to use diverse vocabulary and syntactic structures. To ensure quality, internal annotators reviewed every crowd sourced sentence for correctness. They either rewrote or discarded the sentences that were nonsensical or incorrect. In some cases, they also changed cell highlighting patterns to match the sentence provided.
DART: Open-Domain Structured Data Record to Text Generation
Build Tripleset-Sentence Pairs Finally, we convert the highlighted cells to triplesets. For a row R, we start with the table's column ontology T . We first place the cell values in R in their corresponding slots in T , e.g. in Figure 2 we fill TEAM with "Amsterdam Admirals". We then check that the nodes of T corresponding to the highlighted cells in R form a connected subtree. If not, we walk up the tree and highlight each traversed node up until the lowest common ancestor of the highlighted nodes (inclusive) to form a connected subtree. For each node N in the tree except the root node, we can extract the triple (parent (N ), title (N ), N ). For example, since STADIUM is highlighted in Figure 2, we extract the triple (Amsterdam Admirals, STADIUM, Olympisch Stadion). A small number of triple-sets contained more than 10 triples. We discarded these because their associated surface realizations were of poor quality. The numbers of tripleset-sentence pairs annotated by different annotators are shown in Table 2.
DART: Open-Domain Structured Data Record to Text Generation
High quality natural language questions in open domain semantic parsing datasets such as Wik-iSQL and QA2D techniques found in automatically constructing NLI datasets (Demszky et al., 2018) present themselves as an attractive opportunity to semi-automatically construct an abundance of declarative sentences and align to table cells. We leveraged rule-based QA2D technique 2 together with manual screening to combine WikiSQL questions and SQL-retrieved-answers into declarative sentences and manually filtered out bad sentences. We only execute SQL queries without aggregate commands 3 to retrieve answers corresponding to questions answerable by single rows. An example of such conversion is as follows: Alignment with table cells is done at two stages. We first align sentences with corresponding rows by changing SQL commands to SELECT * and use string matching to obtain columns and column headers relevant to the answer and WHERE condition. After manually filtering out bad sentences, bad alignments, or tables without ontology annotations, we were able to get 4,204 sentences. Finally, the corresponding table cells are then converted into triples in the same way as we described in Section 2.1.
DART: Open-Domain Structured Data Record to Text Generation
Examples of produced declarative sentences can be found in Figure 10 of the Appendix.
DART: Open-Domain Structured Data Record to Text Generation
Since they provide a large amount of strictly aligned data-text pairs with high quality sentences, we incorporate the following existing datasets in the same triple-set, sentence pair format with some modifications.
DART: Open-Domain Structured Data Record to Text Generation
WebNLG 2017 An instance of the WebNLG dataset contains a set of triples extracted from DBpedia and the target text written by human. We include the WebNLG 2017 dataset 4 consisting of 27731 triple-set sentence pairs with up to 7 RDF triples in a triple set covering 15 domains. 
DART: Open-Domain Structured Data Record to Text Generation
We canonicalized the predicates in our triple sets such that those of the same meaning are also represented the same. We manually constructed a predicate mapping table to achieve this. As an example, our predicate mapping maps "Hometown," "Home Town," and "Home Town/City" to the unified predicate "HOMETOWN."
DART: Open-Domain Structured Data Record to Text Generation
After unifying predicates, we evaluated the diversity of DART by counting the number of unique predicates in its partitions. As shown in Table 2, we see that the Wikipedia partition of DART contains much more unique predicates than the WebNLG and Cleaned E2E partitions combined, despite having smaller number of triple-set, sentence pairs. This contributes significantly to the domain diversity of DART. In addition, we can see that DART exhibits a great deal of topical variety in terms of number of unique triples and vocabulary size.
DART: Open-Domain Structured Data Record to Text Generation
For WebNLG 2017 and Cleaned E2E, we use their original data splits. For our annotation on Wik-iTableQuestions and WikiSQL, random splitting will make train, dev, and test splits contain similar tables and similar triple-set, sentence examples. Therefore, to increase the generalization challenge, we compare the table title and the table header to find similar tables, and make sure the model is evaluated on test split tables that are least similar to those used for training. We first sample some tables as a seed test set, and then compute Jaccard similarity 6 with remaining tables based on the titles and the headers. If a table has a Jaccard similarity greater than 0.5 with any of the tables in the test set, we add it into the test set. A similar process is repeated to create the dev set, and the remaining tables form the training set. This results in 62,659/6,980/12,552 sentences in the train/dev/test sets, respectively.
DART: Open-Domain Structured Data Record to Text Generation
We conduct experiments on DART and the WebNLG 2017 dataset, with an ablation study on WebNLG to show the benefits of using DART for data augmentation.
DART: Open-Domain Structured Data Record to Text Generation
We investigate several state-of-the-art Data-to-Text generation models. We report results of the following models on DART-testset: (1) Bidirectional-LSTM with attention, for which we use 2-layer bi-LSTM for encoder, with 300 dimensional word embeddings (without using pretrained word vectors), 512 hidden units and 0.3 dropout rate for the decoder. 
DART: Open-Domain Structured Data Record to Text Generation
We use a variety of automatic metrics and human evaluation (Section 4) to evaluate the quality of the generated text. We report BLEU, METEOR, and TER which are used in the official WebNLG challenge. However, these measures have limitations in considering the semantic meanings of words or phrases (Novikova et al., 2017a), therefore we also report MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020), and BLEURT (Sellam et al., 2020) that incorporate semantics rather than surface forms using contextual embeddings. Furthermore, we include PARENT (Dhingra et al., 2019) which explicitly aligns n-grams from the reference and generated text to the data contents.
DART: Open-Domain Structured Data Record to Text Generation
DART Our experimental results on DART are summarized in Table 3. The T5-large model has the highest performance among all models with a BLEU score of 50.66. We attribute this to T5's generalization and transfer learning ability due to pretraining on multi-tasks. We can see that in general, pretrained models outperform others by a large margin, and increasing the model size seems to further boost the performance on DART. However, language models such as BART and T5 are pretrained by reconstructing text and, as a result, we found that their output on DART often contains hallucinated words (Parikh et al., 2020;Harkous et al., 2020;Reiter, 2020), as shown in Figure 11. In addition, while the pretrained model shows better text generation quality due to its generalization ability from pretraining, it does not fully capture the hierarchical ontology nature of the triple sets in their linearized input, therefore making DART more challenging. We suspect that models that are better at exploiting the ontology structure preserved in the input tripleset will achieve better performance on DART.
DART: Open-Domain Structured Data Record to Text Generation
WebNLG Furthermore, we investigate if DART can improve pretrained models' performance on other Data-to-Text generation tasks. To this end, we finetune the baseline transformer model, BART-[base, large] and T5-[small, base, large] on the WebNLG 2017 dataset, and augment the training by adding instances in the DART training set. The experimental results can be found in Table 4. We report performances of some competitive models that are not pretrained, as well as the state-of-the-art performances of pretrained models on the WebNLG 2017 dataset by Ribeiro et al. (2020). On the bottom panel, we include results of experiments augmented with DART instances whose triplesets are generated with table ontology annotation, paired with human written sentences. We are able to achieve new state-of-the-art results on all WebNLG 2017 test set splits (seen, unseen and all) by finetuning T5-large on DART. We observe that using DART for data augmentation consistently improves the performance across all models, including the baseline transformer model that is not pretrained. Furthermore, we observe that more improvement is shown on unseen split of the test set, due to DART's open-domain nature. See Figure 12 of the Appendix for example model outputs aligned with their human references.
DART: Open-Domain Structured Data Record to Text Generation
We also conduct an ablation study on the WebNLG dataset to investigate what part of DART contributes most to improving the Data-to-Text tasks in general. We report results of the study in Table 6 of the Appendix. We divide DART into 4 partitions, where declarative sentence (auto-generated) partition and human annotated sentence partition contain instances whose triplesets are extracted from Wikipedia tables based on ontology. E2E partition contains instances converted from the E2E    dataset, and WebNLG partition keeps the original data format. In general, we observe that adding DART instances that contain human written sentences brings most improvement, especially on unseen split. While adding E2E partition boosts the scores on seen test split and deteriorates the performance on unseen test split. This trend is consistent across all models. Comparing results of declarative sentence partition and human written sentence partition, we see that for most of the models, DART instances with human written sentences have better quality as it brings more improvement to the task.
DART: Open-Domain Structured Data Record to Text Generation
In Table 5, we perform human evaluation on DART based on two criteria: (1) fluency if a sentence is natural and grammatical, and (2) semantic faithfulness if a sentence is supported by the input triples. We defined three levels of fluency: fluent, mostly fluent, and not fluent, and the same for semantic faithfulness. We ask 5 internal annotators to evaluate on 100 triplesets sampled from declarative sentence partition and another 100 triplesets sampled from human written sentence partition. Each tripleset is paired with 3 sentences, one of them is the reference sentence, and the other two are outputs of BART-base and T5-base models. The results in Table 5 attest to the high quality of our annotations since the human written references achieve highest fluency and faithfulness comparing to outputs of two strong baseline models. The evaluation on faithfulness also demonstrates that there is a considerable gap between the DART reference and the outputs of the state-of-the-art pretrained model, showing that there is a large room for improvement. We also noticed that the auto-generated declarative sentences are not as fluent or faithful as the model outputs because they are generated with a rule-based system. However, we decided to release this partition, along with other partitions of DART because it demonstrates an economic way to obtain large amounts of DART instances and it also shows benefits for generalization due to the diverse topics it contains.
DART: Open-Domain Structured Data Record to Text Generation
In this paper, we introduce DART, an open-domain corpus for structured data record to text generation. DART's ontology-preserving representation of data inputs differentiates itself from other open-domain Data-to-Text corpora. We found that DART introduces new challenges to several state-of-the-art Data-to-Text models due to its open-domain nature and its ontology structure of the semantic triple input. Furthermore, we found that using it for data augmentation improves other Data-to-Text tasks. For future work, we will explore more controlled, high-fidelity generation that better incorporates the ontology hierarchy of data.
DART: Open-Domain Structured Data Record to Text Generation
Our dataset is constructed by accumulating and processing resources from various existing datasets that are open to the public. In addition, we collect annotations on structure of tabular data and human written sentences that describe data records.
DART: Open-Domain Structured Data Record to Text Generation
The existing resources that we utilize mainly consist of (1) tabular data from Wikipedia, (2) information of restaurants presented with dialogueact meaning representation and its textual description (E2E), and (3) information of various entities and their relationship that are in 15 different categories of DBPedia, which is a knowledge base built on contents created in various Wikimedia projects (WebNLG). It is possible that there are biases in these resources, either in the tabular data or the textual description written by humans.
DART: Open-Domain Structured Data Record to Text Generation
For additional annotations we collected, we have two groups of annotators participating: internal annotators who are the authors of this work, and external annotators recruited from the Amazon Mechanical Turk platform. On MTurk, we use a pay rate of $15 per hour approximately based on our estimation of the time it takes to complete our annotation tasks. In total, it took 125 hours to complete all tasks on the Amazon Mechanical Turk platform. There are three annotation tasks: (1) Annotators are asked to specify ontological structure of the table by indicating relationship between table column headers, (2) Annotators are asked to write descriptions that are fluent and semantically faithful to the data records presented to them, and (3) Annotators are asked to evaluate sentences that are either references or model generated outputs. We acknowledge that it is also possible to have biases in the sentences written by the annotators, or in the data records that are presented to them.
DART: Open-Domain Structured Data Record to Text Generation
We conducted experiments on our own dataset and the WebNLG dataset using BART and T5, two large-scale pretrained models. Both models are trained on large amounts of textual data such as news, books, and web text, which may contain any kinds of biases. As a result, it is possible to insert those biases into the models.
DART: Open-Domain Structured Data Record to Text Generation
In total, we conducted 43 experiments: 7 on DART and 36 for our ablation study on the WebNLG dataset. We use a single NVIDIA V100 GPU for all experiments and each experiment took from 5 to 40 hours depending on the model size.    <entry category="MISC" eid="Id5" size="3"> <modifiedtripleset> <mtriple>Apertura 2006 | JORNADA_OR_OTHER | Semifinals Ida</mtriple> <mtriple>Semifinals Ida | AWAY_TEAM | América</mtriple> <mtriple>Semifinals Ida | HOME_TEAM | Chivas</mtriple> </modifiedtripleset> <lex comment="WikiTableQuestions" lid="Id1"> Chivas and América will compete in the semifinals of the Apertura 2006 tournament. </lex> </entry> <entry category="MISC" eid="Id76" size="6"> <modifiedtripleset> <mtriple>Terry Jenkins | ROUND |      table from WikiTableQuestions. Annotators created a table ontology, and they wrote sentences encapsulating the information in the orange cells for a given row. Whenever a sentence referenced the table title, that sentence was also highlighted green. Figure 9: An example of collected MTurk-generated sentences for WikiTableQuestions. Internal annotators went through the generated sentences and checked for both sentence coherence and title usage. Below the generated sentences, 'y' meant the sentence references the table title, 'n' meant the sentence did not use the table title, 'x' meant the sentence was nonsensical. Figure 10: Automatically generated declarative sentences from WikiSQL with human validation. Annotators went through the generated sentences and checked for both sentence coherence and title use. Below the generated sentences, 'y' meant the sentence references the table title, 'n' meant the sentence did not use the table title, 'x' meant the sentence was nonsensical.  
The WebNLG Challenge: Generating Text from RDF Data
Previous Natural Language Generation (NLG) challenges have focused on surface realisation (Banik et al., 2013;Belz et al., 2011), referring expression generation (Belz and Gatt, 2007;Gatt et al., 2009;Belz et al., 2010) and content selection (Bouayad-Agha et al., 2013).
The WebNLG Challenge: Generating Text from RDF Data
In contrast, the WebNLG challenge focuses on microplanning, that subtask of NLG which consists in mapping a given content to a text verbalising this content. Microplanning is a complex choice problem involving several subtasks referred to in the literature as referring expression generation, aggregation, lexicalisation, surface realisation and sentence segmentation. For instance, given the WebNLG data unit shown in (1a), generating the text in (1b) involves choosing to lexicalise the JOHN E BLAHA entity only once (referring expression generation), lexicalising the OCCUPATION property as the phrase worked as (lexicalisation), using PP coordination to avoid repeating the word born (aggregation) and verbalising the three triples by a single complex sentence including an apposition, a PP coordination and a transitive verb construction (sentence segmentation and surface realisation).
The WebNLG Challenge: Generating Text from RDF Data
(1) a. Data: ( As illustrated by the above example, the WebNLG dataset was designed to exercise the ability of NLG systems to handle the whole range of microplanning operations and their interactions. It was created using a content selection procedure specifically designed to enhance data and text variety (Perez-Beltrachini et al., 2016). In , we compared a dataset created using the WebNLG process with existing benchmarks in particular, (Wen et al., 2016)'s dataset (RNNLG) which was produced using a similar process. In what follows, we give various statistics about the WebNLG dataset using the RNNLG dataset as a reference point.
The WebNLG Challenge: Generating Text from RDF Data
Size. The WebNLG dataset consists of 25,298 (data,text) pairs and 9,674 distinct data units. The data units are sets of RDF triples extracted from DB-Pedia and the texts are sequences of one or more sentences verbalising these data units.
The WebNLG Challenge: Generating Text from RDF Data
(2), different properties can induce different lexical forms (a property might be lexicalised as a verb, a relational noun, a preposition or an adjective). Therefore, the larger the number of properties, the more likely the data is to allow for a wider range of lexicalisation patterns.
The WebNLG Challenge: Generating Text from RDF Data
(
The WebNLG Challenge: Generating Text from RDF Data
To promote diverse lexicalisation patterns, we extracted data from 15 DBPedia categories (Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam, WrittenWork, Athlete, Artist, City, MeanOfTransportation, Celestial-Body, Politician) resulting in a set of 373 distinct RDF properties (more than three times the number of properties contained in the RNNLG dataset). The corrected type token ratio (CTTR 1 ) and the number of word types is roughly twice as large in theWebNLG dataset than in RNNLG.
The WebNLG Challenge: Generating Text from RDF Data
Surface Realisation. To increase syntactic variety, we use a content selection procedure which extracts data units of various shapes. The intuition is that different input shapes may induce distinct linguistic constructions. This is illustrated in Figure 2. Typically, while triples sharing a subject (SIBLING configuration) are likely to induce a VP or a sentence coordination, a CHAIN configuration (where the object of one triple is the subject of the other) will more naturally give rise to object relative clauses or participials.
The WebNLG Challenge: Generating Text from RDF Data
Another factor impacting syntactic variation is the set of properties (input patterns) cooccuring in a given input. This is illustrated by the examples in (3) where two inputs of the same length (3 triples hence 3 properties) result in text with different syntax. That is, a larger number of input patterns is more likely to induce texts with greater syntactic variety. By extracting data units from a large number of distinct domains (DBPedia categories), we seeked to produce a large number of distinct input patterns.
The WebNLG Challenge: Generating Text from RDF Data
1 Following , we use (Lu, 2008)'s system to compute the CTTR (Carroll, 1964 Table 3, the WebNLG dataset contains twice as many distinct input patterns and ten times more input shapes than the RNNLG dataset. It is also less redundant with a ratio between number of inputs and number of input patterns of 2.34 against 10.31 for RNNLG.
The WebNLG Challenge: Generating Text from RDF Data
Aggregation, Sentence Segmentation and Referring Expression Generation. Finally, the need for aggregation, sentence segmentation and referring expression generation mainly arise when texts contains more than one sentence. As Table 3 shows, although data units are overall smaller in the WebNLG dataset than in RNNLG, the WebNLG dataset has a higher number of texts containing more than one sentence and contains texts of longer length.
The WebNLG Challenge: Generating Text from RDF Data
The WebNLG challenge received eight submissions from six participating teams: the ADAPT Centre, Ireland (ADAPTCENTRE), the University of Melbourne, Australia (UMELBOURNE), Peking University, China (PKUWRITER), Tilburg University, The Netherlands (UTILBURG), University of Information Technology, VNU-HCM, Vietnam (UIT-VNU-  . Each team submitted outputs from a single system except UTILBURG who submitted outputs from three different systems. As a result, there were nine systems in total: eight participating systems and our baseline (BASELINE) system. These can be grouped into three categories: pipeline systems, statistical machine translation (SMT) and neural machine translation (NMT) systems. Table 3 shows the system categorisations.
The WebNLG Challenge: Generating Text from RDF Data
Pipeline Systems. Three submissions used a template or grammar-based pipeline framework with some NLG module: UTILBURG-PIPELINE, UIT-VNU-HCM and UPF-FORGE. The first two systems, UTILBURG-PIPELINE and UIT-VNU-HCM, extracted rules or templates from the training data for surface realisation, whereas the third system, UPF-FORGE, used the FORGe grammar (Mille et al., 2017).
The WebNLG Challenge: Generating Text from RDF Data
UTILBURG-PIPELINE extracted rules mapping a triple (or a triple set) to a text observed in the training data; both the triple and the associated text were delexicalised. Given a RDF triple set to generate  from, UTILBURG-PIPELINE first ordered triples to maintain discourse order. Extracted rules were then applied to generate a delexicalised text. Missing entities were added using a referring expression generation module (Castro Ferreira et al., 2016). Finally, a 6-gram language model trained on the Gigaword corpus was used to rank the system output.
The WebNLG Challenge: Generating Text from RDF Data
UIT-VNU-HCM did not resort to delexicalisation in their rules. Instead of using the text to extract templates, it used the typed-dependency structure of the text to facilitate rule extraction from the training data. In addition, at run time, WordNet was used to estimate similarity between predicates in the test and train sets.
The WebNLG Challenge: Generating Text from RDF Data
UPF-FORGE mostly focused on sentence planning with predicate-argument (PredArg) templates. For each of the DBPedia properties found in the training and evaluation data, they manually defined PredArg templates encoding various DBPediaspecific and linguistic features. Given a RDF triple set to generate from, PredArg templates were used to convert these triples to PredArg structures and to further aggregate them to form a PredArg graph structure. The FORGe generator took this linguistic PredArg structure as input and generated a text.
The WebNLG Challenge: Generating Text from RDF Data
UTILBURG-SMT was the only system which used the statistical machine translation framework. It was trained on the WebNLG dataset using the Moses toolkit (Koehn et al., 2007). The dataset was pre-processed whereby each entity in the input and each corresponding referring expression in the output were delexicalised and annotated with the entity Wikipedia ID. The alignments from the training set were obtained using MGIZA and model weights were tuned using 60batch MIRA with BLEU as the evaluation metric. Similar to UTILBURG-PIPELINE, the system used a 6-gram language model trained on the Gigaword corpus using KenLM.
The WebNLG Challenge: Generating Text from RDF Data
NMT Systems. Four systems (ADAPTCENTRE, UMELBOURNE, UTILBURG-NMT and PKUWRITER) build upon the attention-based encoder-decoder architecture proposed in (Bahdanau et al., 2014). Most of them make use of existing NMT frameworks. There are however important differences among systems with respect to both the concrete architecture and the sequence representations they use.
The WebNLG Challenge: Generating Text from RDF Data
ADAPTCENTRE makes use of the Nematus (Sennrich et al., 2017) system. They opt for subword representations rather than delexicalisation to deal with rare words and sparsity. They linearise the input sequence and insert tuple separation special tokens.
The WebNLG Challenge: Generating Text from RDF Data
does a combined delexicalisation procedure and enrichment of the input sequence. Entities are delexicalised using an entity identifier (ENTITY-ID). When available, the DBPedia type of the entity is appended. An n-gram search is used to assure the most accurate target sequence delexicalisation. They use a standard encoder-decoder with attention model.
The WebNLG Challenge: Generating Text from RDF Data
UTILBURG-NMT is based on the Edinburgh Neural Machine Translation submission for the 2016 machine translation shared task (WMT 2016). The target sequences are the delexicalised texts (cf. UTILBURG-PIPELINE) and the input sequences are the linearisation of the delexicalised input set of triples. The REG module from their pipeline system is used to post-process the decoder outputs.
The WebNLG Challenge: Generating Text from RDF Data
The PKUWRITER system relies upon two extra mechanisms, namely a ranking module and an extra Reinforcement Learning (RL) training objective. It uses an ensemble of attention-based encoderdecoder models based on the TensorFlow seq2seq API in addition to the baseline (7 models in total). They propose an output ranking module to choose the best verbalisation among those output by the generation models. The ranker is trained on supervised data generated automatically. Input triple sets are paired with verbalisations produced by each of the generation models. Then, each pair is associated with a quality score, i.e. the BLEU score of the verbalisation and the reference. Word and sentence level features are extracted to train the ranker. The generation models and ranker are trained on different data partitions. The RL objective encourages the generation of output texts which include subjects occurring in the input RDF triples. In addition, PKUWRITER uses a set of hand-crafted rules to handle input cases where the model fails.
The WebNLG Challenge: Generating Text from RDF Data
The WebNLG challenge includes both an automatic and a human-based evaluation. Due to time constraints, only the results of the automatic evaluation are presented in this paper. The results of the humanbased evaluation will be provided on the WebNLG website 2 in October 2017.
The WebNLG Challenge: Generating Text from RDF Data
Three automatic metrics were used to evaluate the participating systems:
The WebNLG Challenge: Generating Text from RDF Data
• BLEU-4 3 (Papineni et al., 2002). BLEU scores were computed using up to three references.
The WebNLG Challenge: Generating Text from RDF Data
• METEOR (v1.5) 4 (Denkowski and Lavie, 2014);
The WebNLG Challenge: Generating Text from RDF Data
• TER 5 (Snover et al., 2006).
The WebNLG Challenge: Generating Text from RDF Data
For statistical significance testing, we followed the bootstrapping algorithm described in (Koehn and Monz, 2006). To assess the ability of the participating systems to generalise to out of domain data, the test dataset consists of two sets of roughly equal size: a test set containing inputs created for entities belonging to DBpedia categories that were seen in the training data (Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam, City, and WrittenWork), and a test set containing inputs extracted for entities belonging to 5 unseen categories (Athlete, Artist, MeanOfTransportation, Ce-lestialBody, Politician). We call the first type of data seen categories, the second, unseen categories. Correspondingly, we report results for 3 datasets: the seen category dataset, the unseen category dataset and the total test set made of both the seen and the unseen category datasets. Table 3 gives more detailed statistics about the number of properties, objects and subject entities occurring in each test set.
The WebNLG Challenge: Generating Text from RDF Data
• |T est| is the number of distinct properties, subjects and objects in the test set;
The WebNLG Challenge: Generating Text from RDF Data
• |T est ∩ T nDv| is the number of distinct properties, subjects and objects which are in the test set and were seen in the training or the development set;
The WebNLG Challenge: Generating Text from RDF Data
• |T est \ T nDv| is the number of distinct properties, subjects and objects which occur in the 3 https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/generic/ multi-bleu.perl 4 http://www.cs.cmu.edu/˜alavie/METEOR/ 5 http://www.cs.umd.edu/˜snover/tercom/ test set, but not in the training and development set.
The WebNLG Challenge: Generating Text from RDF Data
Prop.  While in the seen test data (first column) almost all triple elements are present in the training and development sets, in the unseen test data (second column) the vast majority of subjects, objects, and, more importantly, properties (which need to be lexicalised) has not been seen in the training and development data.
The WebNLG Challenge: Generating Text from RDF Data
Participants were requested to submit tokenised and lowercased texts. To ensure consistency between submissions, we pre-processed the submitted results one more time to double-check that those requirements were fullfilled. As teams used different strategies of tokenisation, we had to modify submissions using our own scripts. In particular, all punctuation signs were separated from alphanumeric sequences (e.g. a two-token group 65.6 feet was modified to a four-token 65 . 6 feet). Moreover, we converted both references and submission outputs to the ASCII character set.
The WebNLG Challenge: Generating Text from RDF Data
We developed a baseline system using neural networks and delexicalisation. Before training, we preprocess the data by linearising triples, performing tokenisation and delexicalisation using exact matching.
The WebNLG Challenge: Generating Text from RDF Data
While delexicalising, we make the following replacements:
The WebNLG Challenge: Generating Text from RDF Data
• given a triple of the form (s p o) where s is of the category C for which the triple set has been produced (e.g., Alan Bean for the category Astronaut), we replace s by C.
The WebNLG Challenge: Generating Text from RDF Data
• given a triple of the form s p o, we replace o by p. E.g., (s country Indonesia) becomes (s country COUNTRY). The replacements were made using the exact match and as a result not all the entities were replaced.
The WebNLG Challenge: Generating Text from RDF Data
Examples 4 and 5 show a (data,text) pair before and after delexicalisation. Note that noodles was not substituted by the corresponding entity category in the target text (because there is no exact match with the NOODLE object in the input). Table 4 shows the number of distinct tokens occurring in the original and delexicalised data.  On this delexicalised data-to-text corpus, we trained a vanilla sequence-to-sequence model with attention mechanism using the OpenNMT toolkit (Klein et al., 2017) with default parameters for training and decoding. The network consists of a twolayered bidirectional encoder-decoder model with LSTM units. We use a batch size of 64 and a starting learning rate of 1.0. The size of the hidden layers is 500. The network was trained for 13 epochs with a stochastic gradient descent optimisation method and a dropout probability of 0.3. We used the entire vocabulary for the baseline due to its rather small size. Source  2703  1300  Target  5374  5013  Total  8077  6313   Table 4: Vocabulary size in tokens.
The WebNLG Challenge: Generating Text from RDF Data
After training we relexicalised sentences with corresponding entities if of course their counterparts are present in generated output. The performance of the baseline is shown in Tables 5, 6, 7 along with other teams' results.
The WebNLG Challenge: Generating Text from RDF Data
We briefly discuss the automatic scores distinguishing between results on the whole dataset, on data extracted from previously unseen categories and on data extracted from seen categories.
The WebNLG Challenge: Generating Text from RDF Data
Global Scores. Table 5 shows the global results that is, results on the whole test set. Horizontal lines group together systems for which the difference in scores is not statistically significant. The names of the teams are coloured according to system type: neural-based systems are in red, pipeline systems in blue, and SMT systems in light grey.
The WebNLG Challenge: Generating Text from RDF Data
Most systems (6 out of 8) outperform the baseline, four of them obtaining scores well above it. In terms of BLEU and TER scores, the first four systems include systems of each type (neural, SMT-based and pipelines).
The WebNLG Challenge: Generating Text from RDF Data
While BLEU and METEOR yield almost identical rankings, METEOR does not, suggesting that the systems handle synonyms and morphological variation differently. In particular, the fact that UPF-FORGE ranks first under the METEOR score suggests that it often generates text that differs from the references because of synonymic or morphological variation.
The WebNLG Challenge: Generating Text from RDF Data
Scores on Seen Categories. For data extracted from DBPedia categories that were seen in the training data, machine learning based systems (neural and SMT) mostly outperform rule-based systems. In particular, in terms of BLEU and TER scores, the three pipeline systems are at the low end of the ranking. Again though, the METEOR scores show a much higher ranking (3rd rather than 6th) for the UPF-FORGE systems.
The WebNLG Challenge: Generating Text from RDF Data
Scores on Unseen Categories. On unseen categories, the UPF-FORGE systems ranks first as the system could quickly be adapted to handle properties that had not been seen in the training data. The ranking of the other systems is more or less unchanged with the exception of the ADAPTCENTRE system. This neural system does not use delexicalisation and the subword approach that was adopted UIT-VNU 0.09    form a text of three simple sentences whose joint meaning MT 1 ∪ MT 2 ∪ MT 3 is the same as the meaning MS of the corresponding single complex sentence S. to handle unseen data does not seem to work well.
The WebNLG Challenge: Generating Text from RDF Data
The WebNLG challenge was novel in that it was the first challenge to provide a benchmark on which to evaluate and compare microplanners. Despite a tight schedule (we released the training data in April for a submission in August), it generated a high level of interest among the NLG community: 62 groups from 18 countries 6 downloaded the data, 6 groups submitted 8 systems and 3 groups developped a system but did not submit.
The WebNLG Challenge: Generating Text from RDF Data
The training data for the WebNLG 2017 challenge is available on the WebNLG website 7 and evaluation on the test data can be run by the organisers on demand. A larger dataset consisting of 40,049 (data, text) pairs, 15,095 distinct data input and 15 DBpedia categories is also available. Both datasets are under the creative common licence "CC Attribution-Noncommercial-Share Alike 4.0 International license". We hope that these resources will enable a long and fruitful strand of research on microplanning.
The WebNLG Challenge: Generating Text from RDF Data
The usefulness of the WebNLG dataset reaches far beyond the WebNLG challenge. It can be used for instance to train a semantic parser which would convert a sentence into a set of RDF triples. It can also be used to derive new datasets for related tasks. Thus in , we show how to derive from the WebNLG dataset, a dataset for sentence simplification which we call the Split-and-Rephrase dataset. In this dataset, each pair consists of (i) a single, complex sentence with its meaning representation in terms of RDF triples and (ii) a sequence of at least two sentences and their corresponding sets of RDF triples whereby these sets form a partition on the set of RDF triples associated with the input complex sentence. In other words, the Split-and-Rephrase dataset associates a complex sentence with a sequence of at least two sentences whose meaning is the same as that of the complex sentence. As explained in , this dataset was created using the meaning represen-tations (sets of RDF triples) as pivot. The Splitand-Rephrase dataset consists of 1,100,166 pairs of the form (M C , T C ), {(M 1 , T 1 ) . . . (M n , T n )} where T C is a complex sentence and T 1 . . . T n is a sequence of texts with semantics M 1 , . . . M n expressing the same content M C as T C . Figure 1 shows an example pair. It was used to train four neural systems and the associated meaning representations were shown to improve performance.
The WebNLG Challenge: Generating Text from RDF Data
In the future, we are planning to build a multilingual resource in which the English text present in the WebNLG dataset will be translated into French, Russian and Maltese. In this way, morphological variation can be explored which is an interesting avenue of research in particular for neural systems which have a limited ability to handle unseen input: how well will these systems be able to handle the generation of morphologically rich languages ?
The WebNLG Challenge: Generating Text from RDF Data
The analysis of the participants results presented in this paper will be complemented in an arxiv report by the results of a human-based evaluation. Using human judgements obtained through crowdsourcing, this human evaluation will assess the system results on three criteria, namely fluency, grammaticality and appropriateness (does the text correctly verbalise the input data?). We will also provide a more in depth analysis of the participant results on data extracted from different categories and data of various length.
The WebNLG Challenge: Generating Text from DBPedia Data
With the emergence of the linked data initiative and the rapid development of RDF (Resource Description Format) datasets, several approaches have recently been proposed for generating text from RDF data (Sun and Mellish, 2006;Duma and Klein, 2013;Bontcheva and Wilks, 2004;Lebret et al., 2016). To support the evaluation and comparison of such systems, we propose a shared task on generating text from DBPedia data. The training data will consist of Data/Text pairs where the data is a set of triples extracted from DBPedia and the text is a verbalisation of these triples. In essence, the task consists in mapping data to text. Specific subtasks include sentence segmentation (how to chunk the input data into sentences), lexicalisation (of the DBPedia properties), aggregation (how to avoid repetitions) and surface realisation (how to build a syntactically correct and natural sounding text).
The WebNLG Challenge: Generating Text from DBPedia Data
DBPedia is a multilingual knowledge base that was built from various kinds of structured information contained in Wikipedia (Mendes et al., 2012). This data is stored as RDF triples of the form (SUBJECT, PROPERTY, OBJECT) where the subject is a URI (Uniform Resource Identifier), the property is a binary relation and the object is either a URI or a literal value such as a string, a date or a number. The English version of the DBpedia knowledge base currently encompasses 6.2M entities, 739 classes, 1,099 properties with reference values and 1,596 proper-ties with typed literal values. 1 There are several motivations for generating text from DBPedia.
The WebNLG Challenge: Generating Text from DBPedia Data
First, the RDF language in which DBPedia is encoded is widely used within the Linked Data framework. Many large scale datasets are encoded in this language (e.g., MusicBrainz 2 , FOAF 3 , LinkedGeo-Data 4 ) and official institutions 5 increasingly publish their data in this format. Being able to generate good quality text from RDF data would permit e.g., making this data more accessible to lay users, enriching existing text with information drawn from knowledge bases such as DBPedia or describing, comparing and relating entities present in these knowledge bases.
The WebNLG Challenge: Generating Text from DBPedia Data
Second, RDF data, and in particular, DBPedia, provide a framework that is both limited and arbitrarily extensible from a linguistic point of view. In the simplest case, the goal would be to verbalise a single triple. In that case, the task mainly consists in finding an appropriate "lexicalisation" for the property. The complexity of the generation task can be closely monitored however by increasing the number of input triples, using input with different shapes 6 , working with different semantic domains and/or enriching the RDF graphs with additional (e.g., discourse) information. We plan to produce a dataset which varies along at least some of these dimensions so as to provide a benchmark for generation that will test systems on input of various complexity.
The WebNLG Challenge: Generating Text from DBPedia Data
Third, there has been much work recently on applying deep learning (in particular, sequence to sequence) models to generation. The training data used by these approaches however often have limited variability. For instance, (Wen et al., 2015)'s data is restricted to restaurant descriptions and (Lebret et al., 2016)'s to WikiData frames. Typically the number of attributes (property) considered by these approaches is very low (between 15 and 40) and the text to be produced have a stereotyped structure (restaurant description, biographic abstracts). By providing a more varied dataset, the WebNLG datatext corpus will permit investigating how such deep learning models perform on more varied and more linguistically complex data.
The WebNLG Challenge: Generating Text from DBPedia Data
In essence, the task consists in mapping data to text. Specific subtasks include sentence segmentation (how to chunk the input data into sentences), lexicalisation (of the DBPedia properties), aggregation (how to avoid repetitions) and surface realisation (how to build a syntactically correct and natural sounding text). The following example illustrates this.
The WebNLG Challenge: Generating Text from DBPedia Data
(1) a. Given the input shown in (1a), generating (1b) involves lexicalising the OCCUPATION property as the phrase worked as, using PP coordination (born in San Antonio on 1942-08-26) to avoid repeating the word born (aggregation) and verbalising the 3 triples by a single complex sentence including an apposition, a PP coordination and a transitive verb construction (sentence segmentation and surface realisation).
The WebNLG Challenge: Generating Text from DBPedia Data
Relation to Previous Shared Tasks Other NLG shared task evaluation challenges have been organised in the past. These have focused on different generation subtasks overlapping with the task we propose but our task differs from them in various ways. KBGen generation challenge. The recent KBGen (Banik et al., 2013) task focused on sentence generation from Knowledge Bases (KB). In particular, the task was organised around the AURA (Gunning et al., 2010) KB on the biological domain which models n-ary relations. The input data selection process targets the extraction of KB fragments which could be verbalised as a single sentence. The content selection approach was semi-automatic, starting with the manual selection of a set of KB fragments. Then, using patterns derived from those fragments, a new set of candidate KB fragments was generated which was finally manually revised. The verbalisation of the sentence sized KB fragments was generated by human subjects.
The WebNLG Challenge: Generating Text from DBPedia Data
Although our task also concerns text generation from KBs the definition of the task is different. Our proposal aims at the generation of text beyond sentences and thus involves an additional subtask that is sentence segmentation. The tasks also differ on the KBs used, we propose using DBPedia which facilitates changing the domain by focusing on different categories. Moreover, the set of relations on both KBs pose different challenges for generation, while the AURA KB contains n-ary relations DBPedia contains relations names challenging for the lexicalisation subtask. A last difference with our task is the content selection method. Our method is completely automatic and thus permits the inexpensive generation of a large benchmark. Moreover, it can be used to select content ranging from a single triple to several triples and with different shapes.
The WebNLG Challenge: Generating Text from DBPedia Data
The Surface Realisation Shared Task (SR'11). The major goal of the SR'11 task (Belz et al., 2011) was to provide a common ground for the comparison of surface realisers on the task of regenerating sentences in a treebank. Two different tracks are considered with different input representations. The 'shallow' input provides a dependency tree of the sentence to be generated and the 'deep' input provides a graph representation where syntactic dependencies have been replaced by semantic roles and some function words have been removed.
The WebNLG Challenge: Generating Text from DBPedia Data
The focus of the SR'11 task was on the linguistic realisation subtask and the broad coverage of lin-164 guistic phenomena. The task we propose here starts from non-linguistic KB data and puts forward other NLG subtasks.
The WebNLG Challenge: Generating Text from DBPedia Data
Generating Referring Expressions (GRE). The GRE shared tasks pioneered the proposed NLG challenges. The first shared task has only focused on the selection of distinguishing attributes (Belz and Gatt, 2007) while subsequent tasks have considered the referring expression realisation subtask proposing a complete referring expression generation task Gatt et al., 2009). This tasks aimed at the unique identification of the referent and brevity of the referring expression. Slightly different, the GREC challenges Belz et al., 2010) propose the generation of referring expressions in a discourse context. The GREC tasks use a corpus created from Wikipedia abstracts on geographic entities and people and with two referring expression annotation schemes, reference type and word strings. Rather than generating from data input these tasks consist in labelling underspecified referring expressions in a given text.
The WebNLG Challenge: Generating Text from DBPedia Data
Our task concerns the generation of entity descriptions and requires the production of referring expressions, specially in the cases where multiple sentences will be generated. However, it does not foresee the selection of additional content (e.g. attributes). In contrast, our proposal targets all generation subtasks involved in content realisation.
The WebNLG Challenge: Generating Text from DBPedia Data
As illustrated in Example 1 above, the training corpus consists of (D, T ) pairs such that D is a set of DBPedia triples and T is an English text (possibly consisting of a single sentence). This corpus will be constructed in two steps by first, extracting from DBPedia content units that are both coherent and diverse and second, associating these content units with English text verbalising their content.
The WebNLG Challenge: Generating Text from DBPedia Data
Data To extract content units from DBPedia, we will use the content selection procedure sketched in (Mohammed et al., 2016). This procedure consists of two steps. First, bigram models of DBPedia properties specific to a given DBPedia category (e.g., Astronaut) are learned from the DBPedia graphs associated with entities of that category. Second, an ILP program is used to extract from DBPedia, subtrees that maximise bigram probability. In effect, the extracted DBPedia trees are coherent entity descriptions in that the property bigram they contain often cooccur together in the DBPedia graphs associated with entities of a given DBPedia category. The method can be parameterised to produce content units for different DBPedia categories, different DBPedia entities and various numbers of DBPedia triples. It is fully automatic and permit producing DBPedia graphs that are both coherent, diverse and that bear on different domains (e.g., Astronauts, Universities, Musical work).
The WebNLG Challenge: Generating Text from DBPedia Data
Text To associate the DBPedia trees extracted in the first phase with text, we will combine automatic techniques with crowdsourcing in two ways.
The WebNLG Challenge: Generating Text from DBPedia Data
First, we will lexicalise DBPedia properties by using the lexicalisations contained in the Lemon English Lexicon for DBPedia 7 (Walter et al., 2013;Walter et al., 2014a;Walter et al., 2014b) and by manually filtering the lexicalisations produced by the lexicalisation method described in  and by the relation extraction and clustering method described in (c.f. (Nakashole et al., 2012)) 8 . We will then ask crowdsourcers to verbalise sets of DBPEdia triples in which properties have already been lexicalised (e.g., CREW1UP will be lexicalised as commander of ).
The WebNLG Challenge: Generating Text from DBPedia Data
Second, we will exploit the data-to-text alignment method presented in (Mrabet et al., 2016) to semiautomatically align Wikipedia text with sets of DB-Pedia triples. The method consists in (i) automatically annotating phrases with DBPedia entities, (ii) associating sentences with DBPedia triples relating entities annotating these sentences and (iii) using crowdsourcing to align sentences with triples. In the third step, annotators are asked to "align" triples and sentences that is, to remove from the sentence all material that is irrelevant to express the associated triples and vice versa, to remove any triples that is not expressed by the sentence.
The WebNLG Challenge: Generating Text from DBPedia Data
Statistics, Schedule and Funding The WebNLG shared task will be funded by the WebNLG ANR Project 9 . We aim to produce a data-text corpus of medium size (between 10K and 50K data-text pairs) bearing on at least 5 different domains and consisting of input data containing between 2 and 5 RDF triples. Ideally, training data will be made available early in 2017 and testing will be carried out in early summer (May-June 2017).
Creating Training Corpora for NLG Micro-Planners
To train Natural Language Generation (NLG) systems, various input-text corpora have been developed which associate (numerical, formal, linguistic) input with text. As discussed in detail in Sec-tion 2, these corpora can be classified into three main types namely, (i) domain specific corpora, (ii) benchmarks constructed from "Expert" Linguistic Annotations and (iii) crowdsourced benchmarks. 1 In this paper, we focus on how to create datato-text corpora which can support the learning of micro-planners i.e., data-to-text generation systems that can handle the complex interactions occurring between lexicalisation (mapping data to words), aggregation (exploiting linguistic constructs such as ellipsis and coordination to avoid repetition), surface realisation (using the appropriate syntactic constructs to build sentences), sentence segmentation and referring expression generation.
Creating Training Corpora for NLG Micro-Planners
We start by reviewing the main existing types of NLG benchmarks and we argue for a crowdsourcing approach in which (i) data units are automatically built from an existing Knowledge Base (KB) and (ii) text is crowdsourced from the data (Section 2). We then propose a generic framework for semi-automatically creating training corpora for NLG (Section 3) from existing knowledge bases. In Section 4, we apply this framework to DBpedia data and we compare the resulting dataset with the dataset of Wen et al. (2016) using various metrics to evaluate the linguistic and computational adequacy of both datasets. By applying these metrics, we show that while Wen et al.'s dataset is more than twice larger than ours, it is less diverse both in terms of input and in terms of text. We also com-pare the performance of a sequence-to-sequence model (Vinyals et al., 2015) on both datasets to estimate the complexity of the learning task induced by each dataset. We show that the performance of this neural model is much lower on the new data set than on the existing ones. We thus propose our corpus generation framework as a novel method for creating challenging data sets from which NLG models can be learned which are capable of generating complex texts from KB data.
Creating Training Corpora for NLG Micro-Planners
Domain specific benchmarks. Several domain specific data-text corpora have been built by researchers to train and evaluate NLG systems. In the sports domain, Chen and Mooney (2008) constructed a dataset mapping soccer games events to text which consists of 1,539 data-text pairs and a vocabulary of 214 words. For weather forecast generation, the dataset of Liang et al. (2009) includes 29,528 data-text pairs with a vocabulary of 345 words. For the air travel domain, Ratnaparkhi (2000) created a dataset consisting of 5,426 datatext pairs with a richer vocabulary (927 words) and in the biology domain, the KBGen shared task (Banik et al., 2013) made available 284 data-text pairs where the data was extracted from an existing knowledge base and the text was authored by biology experts.
Creating Training Corpora for NLG Micro-Planners
An important limitation of these datasets is that, because they are domain specific, systems learned from them are restricted to generating domain specific, often strongly stereotyped text (e.g., weather forecast or soccer game commentator reports). Arguably, training corpora for NLG should support the learning of more generic systems capable of handling a much wider range of linguistic interactions than is present in stereotyped texts. By nature however, domain specific corpora restrict the lexical and often the syntactic coverage of the texts to be produced and thereby indirectly limit the expressivity of the generators trained on them.
Creating Training Corpora for NLG Micro-Planners
Benchmarks constructed from "expert" linguistic annotations. NLG benchmarks have also been proposed where the input data is either derived from dependency parse trees (SR'11 task, Belz et al. 2011) or constructed through manual annotation (AMR Corpus, Banarescu et al. 2012). Contrary to the domain-specific data sets just mentioned, these corpora have a wider coverage and are large enough for training systems that can generate linguistically sophisticated text.
Creating Training Corpora for NLG Micro-Planners
One main drawback of these benchmarks however is that their construction required massive manual annotation of text with complex linguistic structures (parse trees for the SR task and Abstract Meaning Representation for the AMR corpus). Moreover because these structures are complex, the annotation must be done by experts. It cannot be delegated to the crowd. In short, the creation of such benchmark is costly both in terms of time and in terms of expertise.
Creating Training Corpora for NLG Micro-Planners
Another drawback is that, because the input representation derived from a text is relatively close to its surface form 2 , the NLG task is mostly restricted to surface realisation (mapping input to sentences). That is, these benchmarks give very limited support for learning models that can handle the interactions between micro-planning subtasks.
Creating Training Corpora for NLG Micro-Planners
Crowdsourced benchmarks. More recently, data-to-text benchmarks have also been created by associating data units with text using crowdsourcing. Wen et al. (2016) first created data by enumerating all possible combinations of 14 dialog act types (e.g., request, inform) and attribute-value pairs present in four small-size, hand-written ontologies about TVs, laptops, restaurants and hotels. They then use crowdsourcing to associate each data unit with a text. The resulting dataset is both large and varied (4 domains) and was successfully exploited to train neural and imitation learning data-to-text generator (Wen et al., 2016;Lampouras and Vlachos, 2016). Similarly, Novikova and Rieser (2016) described a framework for collecting data-text pairs using automatic quality control measures and evaluating how the type of the input representations (text vs pictures) impacts the quality of crowdsourced text.
Creating Training Corpora for NLG Micro-Planners
The crowdsourcing approach to creating inputtext corpora has several advantages.
Creating Training Corpora for NLG Micro-Planners
First, it is low cost in that the data is produced automatically and the text is authored by a crowdworker. This is in stark contrast with the previous approach where expert linguists are required to align text with data.
Creating Training Corpora for NLG Micro-Planners
Second, because the text is crowd-sourced from the data (rather than the other way round), there is an adequate match between text and data both semantically (the text expresses the information contained in the data) and computationally (the data is sufficiently different from the text to require the learning of complex generation operations such as sentence segmentation, aggregation and referring expression generation).
Creating Training Corpora for NLG Micro-Planners
Third, by exploiting small hand-written ontologies to quickly construct meaningful artificial data, the crowdsourcing approach allows for the easy creation of a large dataset with data units of various size and bearing on different domains. This, in turn, allows for better linguistic coverage and for NLG tasks of various complexity since typically, inputs of larger size increases the need for complex microplanning operations.
Creating Training Corpora for NLG Micro-Planners
Creating Data-to-Text, Micro-Planning Benchmarks
Creating Training Corpora for NLG Micro-Planners
While as just noted, the crowdsourcing approach presented by Wen et al. (2016) has several advantages, it also has a number of shortcomings. One important drawback is that it builds on artificial rather than "real" data i.e., data that would be extracted from an existing knowledge base. As a result, the training corpora built using this method cannot be used to train KB verbalisers i.e., generation systems that can verbalise KB fragments.
Creating Training Corpora for NLG Micro-Planners
Another limitation concerns the shape of the input data. Wen et al.'s data can be viewed as trees of depth one (a set of attributes-value pairs describing a single entity e.g., a restaurant or a laptop). As illustrated in Figure 1 however, there is a strong correlation between the shape of the input and the syntactic structure of the corresponding sentence. The path structure T 1 where B is shared by two predicates (mission and operator) will favour the use of a participial or a passive subject relative clause. In contrast, the branching structure T 2 will favour the use of a new clause with a pronominal subject or a coordinated VP. More generally, allowing for trees of deeper depth is necessary to indirectly promote the introduction in the benchmark of a more varied set of syntactic constructs to be learned by generators.
Creating Training Corpora for NLG Micro-Planners
To address these issues, we introduce a novel method for creating data-to-text corpora from large knowledge bases such as DBPedia. Our T1 A B C mission operator S1.1 A participated in mission B operated by C. S1.2 A participated in mission B which was operated by C.
Creating Training Corpora for NLG Micro-Planners
A D E o cc u p at io n b ir th P la ce S2.1 A was born in E. She worked as an engineer.
Creating Training Corpora for NLG Micro-Planners
A was born in E and worked as an engineer. method combines (i) a content selection module designed to extract varied, relevant and coherent data units from DBPedia with (ii) a crowdsourcing process for associating data units with human authored texts that correctly capture their meaning. Example 1 shows a data/text unit created by our method using DBPedia as input KB.
Creating Training Corpora for NLG Micro-Planners
(1) a. Our method has the following features. First, it can be used to create a data-to-text corpus from any knowledge base where entities are categorised and there is a large number of entities belonging to the same category. As noted above, this means that the resulting corpus can be used to train KB verbalisers i.e., generators that are able to verbalise fragments of existing knowledge bases. It could be used for instance, to verbalise fragments of e.g., MusicBrainz 3 , FOAF 4 or Linked-GeoData. 5 Second, as crowdworkers are required to enter text that matches the data and a majority vote validation process is used to eliminate mis-matched pairs, there is a direct match between text and data. This allows for a clear focus on the non content selection part of generation known as microplanning.
Creating Training Corpora for NLG Micro-Planners
Third, because data of increasing size is matched with texts ranging from simple clauses to short texts consisting of several sentences, the resulting benchmark is appropriate for exercising the main subtasks of microplanning. For instance, in Example (1) above, given the input shown in (1a), generating (1b) involves lexicalising the occupation property as the phrase worked as (lexicalisation); using PP coordination (born in San Antonio on 1942-08-26) to avoid repeating the word born (aggregation); and verbalising the three triples using a single complex sentence including an apposition, a PP coordination and a transitive verb construction (sentence segmentation and surface realisation).
Creating Training Corpora for NLG Micro-Planners
To illustrate the functioning of our benchmark creation framework, we apply it to DBPedia. DBPedia is a multilingual knowledge base that was built from various kinds of structured information contained in Wikipedia (Mendes et al., 2012). This data is stored as RDF (Resource Description Format) triples of the form (subject, property, object) where the subject is a URI (Uniform Resource Identifier), the property is a binary relation and the object is either a URI or a literal value such as a string, a date or a number. We use an English version of the DBPedia knowledge base which encompasses 6.2M entities, 739 classes, 1,099 properties with reference values and 1,596 properties with typed literal values. 6
Creating Training Corpora for NLG Micro-Planners
To create data units, we adapted the procedure outlined by Perez-Beltrachini et al. (2016) and sketched in Figure 2. This method can be summarised as follows.
Creating Training Corpora for NLG Micro-Planners
First, DBPedia category graphs are extracted from DBPedia by retrieving up to 500 entity graphs for entities of the same category. 7 For example, we build a category graph for the Astronaut category by collecting, graphs of depth five for 500 entities of types astronaut.
Creating Training Corpora for NLG Micro-Planners
Next, category graphs are used to learn bi-gram models of DBPedia properties which specify the probability of two properties co-occuring together. Three types of bi-gram models are extracted from category graphs using the SRILM toolkit (Stolcke, 2002): one model (S-Model) for bigrams occurring in sibling triples (triples with a shared subject); one model (C-Model) for bigrams occurring in chained triples (the object of one triple is the subject of the other); and one model (M-Model) which is a linear interpolation of the sibling and the chain model. The intuition is that these sib-ling and chain models capture different types of coherence, namely, topic-based coherence for the S-Model and discourse-based coherence for the C-Model.
Creating Training Corpora for NLG Micro-Planners
Finally, the content selection task is formulated as an Integer Linear Programming (ILP) problem to select, for a given entity of category C and its entity graph G e , subtrees of G e with maximal bigram probability and varying size (between 1 and 7 RDF triples).  We applied this content selection procedure to the DBPedia categories Astronaut (A), Building (B), Monument (M), University (U), Sports team (S) and Written work (W), using the three bi-gram models (S-Model, C-Model, M-Model) and making the number of triples required by the ILP constraint to occur in the output solutions vary between 1 and 7. The results are shown in Table 1. An input is a set of triples produced by the content selection module. The number of input (#Inputs) is thus the number of distinct sets of triples produced by this module. In contrast, input patterns are inputs where subject and object have been abstracted over. That is, the number of input patterns (#I. Patterns) is the number of distinct sets of properties present in the set of inputs. The number of properties (#Properties) is the number of distinct RDF properties occurring in the dataset. Similarly, the number of entities (#Entities) is the number of distinct RDF subjects and objects occurring in each given dataset.
Creating Training Corpora for NLG Micro-Planners
We associate data with text using the Crowdflower platform. 8 We do this in four main steps as follows.
Creating Training Corpora for NLG Micro-Planners
1. Clarifying properties. One difficulty when collecting texts verbalising sets of DBPedia triples is that the meaning of DBPedia properties may be unclear. We therefore first manually clarified for each category being worked on, those properties which have no obvious lexicalisations (e.g., crew1up was replaced by commander).
Creating Training Corpora for NLG Micro-Planners
Next, we collected three verbalisations for data units of size one, i.e. single triples consisting of a subject, a property and an object. For each such input, crowdworkers were asked to produce a sentence verbalising its content. We used both a priori automatic checks to prevent spamming and a posteriori manual checks to remove incorrect verbalisations. We also monitored crowdworkers as they entered their input and banned those who tried to circumvent our instructions and validators. The automatic checks comprise 12 custom javascript validators implemented in the CrowdFlower platform to block contributor answers which fail to meet requirements such as the minimal time a contributor should stay on page, the minimal length of the text produced, the minimal match of tokens between a triple and its verbalisation and various format restrictions used to detect invalid input. The exact match between a triple and its verbalisation was also prohibited. In addition, after data collection was completed, we manually checked each data-text pair and eliminated from the data set any pair where the text either did not match the information conveyed by the triple or was not a well-formed English sentence.
Creating Training Corpora for NLG Micro-Planners
3. Getting verbalisations for input containing more than one triple. The verbalisations collected for single triples were used to construct input with bigger size. Thus, for input with a number of triples more than one, the crowd was asked to merge the sentences corresponding to each triple (obtained in step 2) into a natural sounding text. In such a way, we diminish the risk of having misinterpretations of the original semantics of a data unit. Contributors were also encouraged to change the order, and the wording of sentences, while writing their texts. For each data unit, we collected three verbalisations.
Creating Training Corpora for NLG Micro-Planners
Step 3 were verified through crowdsourcing. Each verbalisation collected in Step 3 was displayed to CrowdFlower contributors together with the corresponding set of triples. Then the crowd was asked to assess its fluency, semantic adequacy, and grammaticality. Those criteria were checked by asking the follow-  ing three questions:
Creating Training Corpora for NLG Micro-Planners
Does the text sound fluent and natural? Does the text contain all and only the information from the data? Is the text good English (no spelling or grammatical mistakes)?
Creating Training Corpora for NLG Micro-Planners
We collected five answers per verbalisation. A verbalisation was considered bad, if it received three negative answers in at least one criterion. After the verification step, the total corpus loss was of 8.7%. An example of rejected verbalisation can be found in Example (2). The verbalisation was dropped due to the lack of fluency (awkward lexicalisation of the property club).
Creating Training Corpora for NLG Micro-Planners
(2) (AEK Athens F.C. manager Gus Poyet) (Gus Poyet club Chelsea F.C.) AEK Athens F.C. are managed by Gus Poyet, who is in Chelsea F.C. Table 2 shows some statistics about the texts obtained using our crowdsourcing procedure for triple sets of size one to seven.
Creating Training Corpora for NLG Micro-Planners
We now compare a dataset created using our dataset creation framework (henceforth WEBNLG) with the dataset of Wen et al. (2016) 9 (henceforth, RNNLG). Example 3 shows a sample data-text pair taken from the RNNLG dataset.
Creating Training Corpora for NLG Micro-Planners
(3) Dialog Moves recommend(name=caerus 33;type=television; screensizerange=medium;family=t5;hasusbport=true) The caerus 33 is a medium television in the T5 family that's USB-enabled.
Creating Training Corpora for NLG Micro-Planners
As should be clear from the discussion in Section 2 and 3, both datasets are similar in that, in both cases, data is built from ontological information and text is crowdsourced from the data. An important difference between the two datasets is that, while the RNNLG data was constructed by enumerating possible combinations of dialog act types and attribute-value pairs, the WEBNLG data is created using a sophisticated content selection procedure geared at producing sets of data 9 https://github.com/shawnwun/RNNLG units that are relevant for a given ontological category and that are varied in terms of size, shape and content. We now investigate the impact of this difference on the two datasets (WEBNLG and RNNLG). To assess the degree to which both datasets support the generation of linguistically varied text requiring complex micro-planning operations, we examine a number of data and text related metrics. We also compare the results of an out-of-the-box sequence-to-sequence model as a way to estimate the complexity of the learning task induced by each dataset.  
Creating Training Corpora for NLG Micro-Planners
Terminology. The attributes in the RNNLG dataset can be viewed as binary relations between the object talked about (a restaurant, a laptop, a TV or a hotel) and a value. Similarly, in the WEBNLGdataset, DBpedia RDF properties relate a subject entity to an object which can be either an entity or a datatype value. In what follows, we refer to both as attributes. Table 3 shows several statistics which indicate that, while the RNNLG dataset is larger than WEBNLG, WEBNLG is much more diverse in terms of attributes, input patterns and input shapes.
Creating Training Corpora for NLG Micro-Planners
As illustrated in Example (4) below, different attributes can be lexicalised using different parts of speech. A dataset with a larger number of attributes is therefore more likely to induce texts with greater syntactic variety.
Creating Training Corpora for NLG Micro-Planners
(4) Verb: X title Y / X served as Y As shown in Table 3, WEBNLG has a more diverse attribute set than RNNLG both in absolute (172 attributes in WEBNLG against 108 in RNNLG) and in relative terms (RNNLG is a little more than twice as large as WEBNLG).
Creating Training Corpora for NLG Micro-Planners
Number of input patterns. Since attributes may give rise to lexicalisation with different parts of speech, the sets of attributes present in an input (input pattern) 10 indirectly determine the syntactic realisation of the corresponding text. Hence a higher number of input patterns will favour a higher number of syntactic realisations. This is exemplified in Example (5) where two inputs with the same number of attributes give rise to texts with different syntactic forms. While in Example (5a), the attribute set {country, location, start-Date} is realised by a passive (is located), an apposition (Australia) and a deverbal nominal (its construction), in Example (5b), the attribute set {almaMater, birthPlace, selection} induced a passive (was born) and two VP coordinations (graduated and joined). Again, despite the much larger size of the RNNLG dataset, the number of input patterns in both datasets is almost the same. That is, the relative variety in input patterns is higher in WEBNLG.
Creating Training Corpora for NLG Micro-Planners
The ratio between number of inputs and the number of input patterns has an important impact both in terms of linguistic diversity and in terms of learning complexity. A large ratio indicates a "repetitive dataset" where the same pattern is instantiated a high number of times. While this facilitates learning, this also reduces linguistic coverage (less combinations of structures can be learned) and may induce over-fitting. Note that because datasets are typically delexicalised when training NLG models (cf. e.g., Wen et al. 2015 andLampouras andVlachos 2016), at training time, different instantiations of the same input pattern reduce to identical input.
Creating Training Corpora for NLG Micro-Planners
The two datasets markedly differ on this ratio which is five times lower in WEBNLG. While in WEBNLG, the same pattern is instantiated in average 2.40 times, it is instantiated 10.31 times in average in RNNLG. From a learning perspective, this means that the RNNLG dataset facilitates learning but also makes it harder to assess how well systems trained on it can generalise to handle unseen input.
Creating Training Corpora for NLG Micro-Planners
Input shape. As mentioned in Section 3, in the RNNLG dataset, all inputs can be viewed as trees of depth one while in the WEBNLG dataset, input may have various shapes. As a result, RNNLG texts will be restricted to syntactic forms which permit expressing such multiple predications of the same entity e.g., subject relative clause, VP and sentence coordination etc. In contrast, the trees extracted by the WEBNLG content selection procedure may be of depth five and therefore allow for further syntactic constructs such as object relative clause and passive participles (cf. Figure 1).
Creating Training Corpora for NLG Micro-Planners
We can show this empirically as well that WEBNLG is far more diverse than RNNLG in terms of input shapes. The RNNLG dataset has only 6 distinct shapes and all of them are of depth 1, i.e., all (attribute, value) pairs in an input are siblings to each other. In contrast, the WEBNLG dataset has 58 distinct shapes, out of which only 7 shapes are with depth 1, all others have depth more than 1 and they cover 49.6% of all inputs. Table 4 gives some statistics about the texts contained in each dataset.  As illustrated by the contrast between Examples (6) and (7) above, text length (number of tokens per text) and the number of sentences per text are strong indicators of the complexity of the generation task. We use the Stanford Part-Of-Speech Tagger and Parser version 3.5.2 (dated 2015-04-20, Manning et al. 2014) to tokenize and to perform sentence segmentation on text. As shown in Table 4, WEBNLG's texts are longer both in terms of tokens and in terms of number of sentences per text. Another difference between the two datasets is that WEBNLG contains a higher number of text per input thereby providing a better basis for learning paraphrases.  The size and the content of the vocabulary is another important factor in ensuring the learning of wide coverage generators. While a large vocabulary makes the learning problem harder, it also allows for larger coverage. WEBNLG exhibits a higher corrected type-token ratio (CTTR), which indicates greater lexical variety, and higher lexical sophistication (LS). Lexical sophistication measures the proportion of relatively unusual or advanced word types in the text. In practice, LS is the proportion of lexical word types (lemma) which are not in the list of 2,000 most frequent words generated from the British National Corpus 11 . Type-token ratio (TTR) is a measure of diversity defined as the ratio of the number of word types to the number of words in a text. To address the fact that this ratio tends to decrease with the size of the corpus, corrected TTR can be used to control for corpus size. It is defined as T / √ 2N , where T is the number of types and N the number of tokens.
Creating Training Corpora for NLG Micro-Planners
Overall, the results shown in Table 4 indicate that WEBNLG texts are both lexically more diverse (higher corrected type/token ratio) and more 11 We compute LS and CTTR using the Lexical Complexity Analyzer developed by Lu (2012). sophisticated (higher proportion of unfrequent words) than RNNLG's. They also show a proportionately larger vocabulary for WEBNLG (2,992 types for 290,479 tokens in WEBNLG against 3,524 types for 531,871 tokens in RNNLG).
Creating Training Corpora for NLG Micro-Planners
Richer and more varied datasets are harder to learn from. As a proof-of-concept study of the comparative difficulty of the two datasets with respect to machine learning, we compare the performance of a sequence-to-sequence model for generation on both datasets.
Creating Training Corpora for NLG Micro-Planners
We use the multi-layered sequence-to-sequence model with attention mechanism described in (Vinyals et al., 2015). 12 The model was trained with 3-layer LSTMs with 512 units each with a batch size of 64 and a learning rate of 0.5.
Creating Training Corpora for NLG Micro-Planners
To allow for a fair comparison, we use a similar amount of data (13K data-text pairs) for both datasets. As RNNLG is bigger in size than WEBNLG, we constructed a balanced sample of RNNLG which included equal number of instances per category (tv, laptop, etc). We use a 3:1:1 ratio for training, developement and testing. The training was done in two delexicalisation modes: fully and name only. In case of fully delexicalisation, all entities were replaced by their generic terms, whereas in name only mode only subjects were modified in that way. For instance, the triple (FC Köln manager Peter Stöger) was delexicalised as (SportsTeam manager Manager) in the first mode, and as (SportsTeam manager Peter Stöger) in the second mode. The delexicalisation in sentences was done using the exact match between entities and tokens. For training, we use all the available vocabulary. Input and output vocabulary sizes are reported in Table 5. Table 5 shows the perplexity results. In both modes, RNNLG yielded lower scores than WEBNLG. This is inline with the observations made above concerning the higher data diversity, larger vocabulary and more complex texts of 12 We used the TensorFlow code available at https://github.com/tensorflow/models/ tree/master/tutorials/rnn/translate. Alternatively, we could have used the implementation of Wen et al. (2016) which is optimised for generation. However the code is geared toward dialog acts and modifying it to handle RDF triples is non trivial. Since the comparison aims at examining the relative performance of the same neural network on the two datasets, we used the tensor flow implementation instead. WEBNLG. Similary, the BLEU score of the generated sentences (Papineni et al., 2002) is lower for WEBNLG suggesting again a dataset that is more complex and therefore more difficult to learn from.  
Creating Training Corpora for NLG Micro-Planners
We presented a framework for building NLG datato-text training corpora from existing knowledge bases.
Creating Training Corpora for NLG Micro-Planners
One feature of our framework is that datasets created using this framework can be used for training and testing KB verbalisers an in particular, verbalisers for RDF knowledge bases. Following the development of the semantic web, many large scale datasets are encoded in the RDF language (e.g., MusicBrainz, FOAF, LinkedGeoData) and official institutions 13 increasingly publish their data in this format. In this context, our framework is useful both for creating training data from RDF KB verbalisers and to increase the number of datasets available for training and testing NLG.
Creating Training Corpora for NLG Micro-Planners
Another important feature of our framework is that it permits creating semantically and linguistically diverse datasets which should support the learning of lexically and syntactically, wide coverage micro-planners. We applied our framework to DBpedia data and showed that although twice smaller than the largest corpora currently available for training data-to-text microplanners, the resulting dataset is more semantically and linguistically diverse. Despite the disparity in size, the number of attributes is comparable in the two datasets. The ratio between input and input patterns is five times lower in our dataset thereby making learning harder but also diminishing the risk of overfitting and providing for wider linguistic coverage. Conversely, the ratio of text per input is twice higher thereby providing better support for learning paraphrases. 13 See http://museum-api.pbworks.com for examples.
Creating Training Corpora for NLG Micro-Planners
We have recently released a first version of the WebNLG dataset in the context of a shared task on micro-planning 14 . This new dataset consists of 21,855 data/text pairs with a total of 8,372 distinct data input.
Creating Training Corpora for NLG Micro-Planners
The input describes entities belonging to 9 distinct DBpedia categories namely, Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam and WrittenWork.
Creating Training Corpora for NLG Micro-Planners
The WebNLG data is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International and can be downloaded at http://talc1.loria.fr/ webnlg/stories/challenge.html.
Creating Training Corpora for NLG Micro-Planners
Recently, several sequence-to-sequence models have been proposed for generation. Our experiments suggest that these are not optimal when it comes to generate linguistically complex texts from rich data. More generally, they indicate that the data-to-text corpora built by our framework are challenging for such models. We hope that the WEBNLG dataset which we have made available for the WEBNLG shared task will drive the deep learning community to take up this new challenge and work on the development of neural generators that can handle the generation of KB verbalisers and of linguistically rich texts.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Data-to-text generation (Kukich, 1983;McKeown, 1992) is the task of generating text based on structured data. Most existing data-to-text datasets focus on single-sentence generation, such as WIKIBIO (Lebret et al., 2016), LogicNLG (Chen et al., 2020), and ToTTo (Parikh et al., 2020). Other datasets are relatively small-scale and focus on long-form text generation, such as ROTOWIRE (Wiseman et al., 2017) and MLB (Puduppully et al., 2019). In this work, we cast generating Wikipedia sections as a data-to-text generation task and build a large-scale dataset targeting multi-sentence data-to-text generation with a variety of domains and data sources.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
To this end, we create a dataset that we call WIKITABLET ("Wikipedia Tables to Text") that pairs Wikipedia sections with their corresponding tabular data and various metadata. The data resources we consider are relevant either to entire Wikipedia articles, such as Wikipedia infoboxes and Wikidata tables, or to particular sections. Data from the latter category is built automatically from either naturally-occurring hyperlinks or from named entity recognizers. This data construction approach allows us to collect large quantities of instances while still ensuring the coverage of the information in the table. We also perform various types of filtering to ensure dataset quality.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
WIKITABLET contains millions of instances covering a broad range of topics and a variety of flavors of generation with different levels of flexibility. Figure 1 shows two examples from WIKI-TABLET. The first instance has more flexibility as it involves generating a fictional character biography in a comic book, whereas the second is more similar to standard data-to-text generation tasks, where the input tables contain all of the necessary information for generating the text. While the open-ended instances in WIKITABLET are to some extent similar to story generation (Propp, 1968;McIntyre and Lapata, 2009;Fan et al., 2018), the fact that these instances are still constrained by the input tables enables different evaluation approaches and brings new challenges (i.e., being coherent and faithful to the input tables at the same time).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Because of the range of knowledge-backed generation instances in WIKITABLET, models trained on our dataset can be used in assistive writing technologies for a broad range of topics and types of knowledge. For example, technologies can aid students in essay writing by drawing from multiple kinds of factual sources. Moreover, WIKITABLET can be used as a pretraining dataset for other relatively small-scale data-to-text datasets (e.g., RO-TOWIRE). A similar idea that uses data-to-text generation to create corpora for pretraining language models has shown promising results (Agarwal et al., 2021).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In experiments, we train several baseline models on WIKITABLET and empirically compare training and decoding strategies. We find that the best training strategies still rely on enforcing hard constraints to avoid overly repetitive texts. Human evaluations reveal that (1) humans are unable to differentiate the human written texts from the generations from our neural models; (2) while the annotations show that grammatical errors in the reference texts and the generations may prevent humans from fully understanding the texts, the best decoding strategy (i.e., beam search with n-gram blocking (Paulus et al., 2018)) does not have such a problem and shows the best performance on several aspects; (3) the degree of topical similarity between the generations and the reference texts depends on the open-endedness of the instances.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Our analysis shows that the generations are fluent and generally have high quality, but the models sometimes struggle to generate coherent texts for all the involved entities, suggesting future research directions. For example, when the instance has a high degree of flexibility, we find the models making mistakes about what a particular entity type is capable of. We also find errors in terms of the factuality of the generated text, both in terms of contradictions relative to the tables and commonsense violations.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
The WIKITABLET dataset pairs Wikipedia sections 2 with their corresponding tabular data and various metadata; some of this data is relevant to entire Wikipedia articles ("article data") or article structure ("title data"), while some is section-specific ("section data"). Each data table consists of a set of records, each of which is a tuple containing an attribute and a value.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
The instances in WIKITABLET cover a range of flavors of language generation. Some have more flexibility, requiring models to generate coherent stories based on the entities and knowledge given in the tables. The first instance in Figure 1 is such an example. The text is from the Wikipedia article entitled "Wolfsbane (comics)" and resides within two nested sections: the higher-level section "Fictional character biography" and the lower-level section "Messiah Complex". The task is challenging as models need to generate a coherent passage that can connect all the entities in the section data, and the story also needs to fit the background knowledge provided in the article data.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Other instances are more similar to standard datato-text generation tasks, where the input tables contain all the necessary information for generating 2 We define a Wikipedia section to be all text starting after a (sub)section heading and proceeding until the next (sub)section heading. We include Wikipedia sections at various nesting levels. For example, a top level section may start with a few paragraphs describing general information followed by two subsections with more specific information, in which case the example will be converted into three instances in our dataset.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
During the 2007-2008 "Messiah Complex" storyline, Rahne helps Rictor infiltrate the Purifiers; she fakes being shot by Rictor. She is also a member of the new X-Force. During a battle against Lady Deathstrike and the Reavers, Rahne learns that Father Craig was in league with the Purifiers, supposedly divulging enough information about her that the Purifiers can claim to "know her well." She travels with X-Force to her former home Muir Island, now the base of the Marauders. During the climactic battle, Rahne is injured by Riptide, but her wounds, according to Professor X, are superficial and she will recover. the text. The second instance in Figure 1 is an example of this sort of task. However, these tasks are still challenging due to the wide variety of topics contained in WIKITABLET.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We begin by describing the steps we take to construct WIKITABLET. More details are in the supplementary material. In general, the steps can be split into two parts: collecting data tables and filtering out texts. When collecting data, we consider five resources: Wikidata tables, infoboxes in Wikipedia pages, 3 hyperlinks in the passage, named entities in the passage obtained from named entity recognition (NER), and Wikipedia article structure. For a given Wikipedia article, we use the same infobox and Wikidata table for all sections. These tables can serve as background knowledge for the article. For each section in the article, we create a second table corresponding to section-specific data, i.e., section data. The section data contains records constructed from hyperlinks and entities identified by a named entity recognizer. 4 We form records for named entities by using the type of the entity as the attribute and the identified entity as the value. We form records for hyperlinks as follows. For the attribute, for a hyperlink with surface text t and hyperlinked article , we use the value of the "instance of" or "subclass of" tuple in the Wikidata table for . For example, the first instance in Figure 1 will be turned into a record with attribute "superhero" and value "Wolfsbane (comics)". If does not have a Wikidata table or no appropriate tuple, we consider the parent categories of . For the value of the tuple, we use the document title of rather than the actual surface text t to avoid giving away too much information in the reference text.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Complementary to the article data, we create a title table that provides information about the position in which the section is situated, which includes the article title and the section titles for the target section. As the initial sections in Wikipedia articles do not have section titles, we use the section title "Introduction" for these.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We also perform various filtering to ensure the quality of the data records, the coverage of the input data, and the length of the reference text. The final dataset contains approximately 1.5 million instances. We randomly sample 4533 instances as the development set and 4351 as the test set. We on CoNLL03 data (Tjong Kim Sang and De Meulder, 2003). also ensure that there are no overlapping Wikipedia articles among splits. Table 1 shows statistics for WIKITABLET and related datasets. While the average length of a WIKI-TABLET instance is not longer than some of the existing datasets, WIKITABLET offers more diverse topics than the sports-related datasets ROTOWIRE and MLB, or the biography-related dataset WIKI-BIO. Compared to the prior work that also uses Wikipedia for constructing datasets, WIKIBIO, LogicNLG, ToTTo, and DART (Nan et al., 2021) all focus on sentence generation, whereas WIKI-TABLET requires generating Wikipedia article sections, which are typically multiple sentences and therefore more challenging. WIKITABLET is also much larger than all existing datasets.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
To demonstrate the diversity of topics covered in WIKITABLET, we use either the "instance of" or "subclass of" relation from Wikidata as the category of the article. 5 We show the top 10 most frequent document categories in Table 2. Due to the criteria we use for filtering, only 1.05% of articles in WIKI-TABLET do not have these relations or Wikidata entries, and we omit these articles in the table. As the table demonstrates, more than 50% of the articles in WIKITABLET are not about people (i.e., the topic of WIKIBIO), within which the most frequent category covers only 4.61%.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In this subsection, we highlight two challenges of WIKITABLET.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
1. In contrast to work on evaluating commonsense knowledge in generation where reference texts are single sentences describing everyday scenes (Lin et al., 2020), WIKITABLET can serve as a testbed for evaluating models' abilities to use world knowledge for generating coherent longform text.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
2. Compared to other long-form data-to-text datasets such as ROTOWIRE where the input tables are box scores, the input tables in WIKI-TABLET are more diverse, including both numbers (e.g., economy and population data of an area throughout years), and short phrases. This
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In this section, we describe details of models that we will benchmark on WIKITABLET.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Our base model is based on the transformer (Vaswani et al., 2017). To encode tables, we linearize the tables by using special tokens to separate cells and using feature embeddings to represent records in tables. For the title table in the first instance in Figure 1 the linearized table will be boc 1 Doc. 1 title 1 bov 1 Wolfsbane 1 (comics) 1 boc 2 Sec. 2 title 2 bov 2 Fictional 2 character 2 biography 2 boc 3 · · · eoc (1) As shown in Eq. 1, we employ several techniques when encoding tables: (1) we use special tokens boc and bov to separate attributes and values, and eoc to indicate the end of a sequence; (2) we use subscript indices to indicate unique ID embeddings that are added to the embeddings for each record, which helps models align attributes with values; and (3) we restart the positional embeddings at each boc , such that models will not use the ordering of the input records. In addition, we add a special embedding to each record to indicate if it is from the section table or the article/title table. In Wikidata, there could be multiple qualifiers attached to a record, in which case we replicate the record for each qualifier separately.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Similar linearization approaches have been used in prior work (Dhingra et al., 2019;Hwang et al., 2019;Herzig et al., 2020;Yin et al., 2020). With linearized tables, training and inference become similar to other sequence-to-sequence settings. We train our models with teacher-forcing and standard cross entropy loss unless otherwise specified.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We experiment with three types of modifications to standard sequence-to-sequence training:
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
α-entmax. α-entmax ) is a mapping from scores to a distribution that permits varying the level of sparsity in the distribution. This mapping function has been used in machine translation  and text generation (Martins et al., 2020). When using α-entmax in the decoder, we also replace the cross entropy loss with the α-entmax loss . Both  α-entmax and the α-entmax loss have a hyperparameter α. We follow Martins et al. (2020) and use α = 1.2 as they found it to be the best value for reducing repetition in generation.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Copy Mechanism. Similar to prior work on datato-text generation (Wiseman et al., 2017;Puduppully et al., 2019), we use pointer-generator network style copy attention (See et al., 2017) in the decoder.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Cyclic Loss. Cyclic losses have been shown to be effective in textual style transfer (Shetty et al., 2018;Pang and Gimpel, 2019) and neural machine translation (Cheng et al., 2016;He et al., 2016;Tu et al., 2017). Wiseman et al. (2017) also used this for data-to-text and found it helpful for generating long sequences. In this work, we experiment with adding the cyclic loss to our transformer models, where the backward model can be seen as an information extraction system. We expect that adding the cyclic loss should enable a data-to-text model to generate sentences that are more faithful to the conditioned tables. The cyclic loss is used during training only and does not affect the models during inference. More details are in the appendix.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Massarelli et al. (2020) showed that the choice of decoding strategy can affect the faithfulness or repetitiveness of text generated by language models. We are also interested in these effects in the context of data-to-text generation, and therefore benchmark several decoding strategies on WIKI-TABLET. Our models use byte-pair encoding (BPE; Sennrich et al., 2016) and for all of the following strategies, we always set the minimum number of decoding steps to 100 as it improves most of the evaluation metrics, and the maximum number of decoding steps to 300. Specifically, we benchmark (1) greedy decoding; (2) nucleus sampling (Holtzman et al., 2020) with threshold 0.9 as suggested by Holtzman et al. (2020); (3) beam search; and (4) beam search with n-gram blocking (Paulus et al., 2018) where we set the probabilities of repeated trigrams to be 0 during beam search. We set the beam size to be 5 by default. The appendix has more details about the decoding strategies.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We experiment with two sizes of transformer models. One is "Base", where we use a 1-layer encoder and a 6-layer decoder, each of which has 512 hidden size and 4 attention heads. The other one is "Large", where we use a 1-layer encoder and a 12layer decoder, each of which has 1024 hidden size and 8 Table 3: Test set results for our models. When training the large models, we use the "copy + cyclic loss" setting as it gives the best performance for the base models for most of the metrics.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
computational power, we parameterize our backward model as a transformer model with a 2-layer encoder and a 2-layer decoder. 7 We use BPE with 30k merging operations. We randomly sample 500k instances from the training set and train base models on them when exploring different training strategies. We train a large model with the best setting (using the copy mechanism and cyclic loss) on the full training set. We train both models for 5 epochs. During training we perform early stopping on the development set using greedy decoding.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We report BLEU (Papineni et al., 2002), ROUGE-L (RL) (Lin, 2004), METEOR (MET) (Banerjee and Lavie, 2005), and PARENT (Dhingra et al., 2019), including precision (PAR-P), recall (PAR-R), and F1 (PAR-F1) scores. The first three metrics consider the similarities between generated texts and references, whereas PARENT also considers the similarity between the generation and the table. When using PARENT, we use all three tables, i.e., the section, article, and title tables.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
As we are also interested in the repetitiveness of generated texts, we define a metric based on ngram repetitions which we call "REP". REP computes the ratio of the number of repeated n-grams to the total number of n-grams within a text, so when REP has higher value, it indicates that the text has more repetitions. Here we consider ngrams that appear 3 or more times as repetitions and the n-grams we consider are from bigrams to 4-grams. When reporting REP scores for a dataset, we average the REP scores for each instance in the dataset. Similar metrics have been used in prior work (Holtzman et al., 2020;Welleck et al., 2020).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In Table 3, we report the test results for both our base models and large models. We also report a set of baselines that are based on simply returning the linearized tables and their concatenations with the references. The linearized table baselines show how much information is already contained in the table, while the reference baselines show the upper bound performance for each metric.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In comparing training strategies, we find that using α-entmax improves REP significantly but not other metrics. Adding the cyclic loss or the copy mechanism helps improve performance for the PAR scores and REP, and combining both further improves these metrics.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
When comparing decoding strategies, we find that both nucleus sampling and n-gram blocking are effective in reducing repetition. Nucleus sampling harms the PAR scores, especially PAR-P, but has less impact on the other metrics, indicating that it makes the model more likely to generate texts that are less relevant to the tables. Using beam search improves all metrics significantly when compared to greedy decoding, especially the PAR-P and REP scores. Adding n-gram blocking further reduces the REP score, pushing it to be even lower than that from nucleus sampling, but still retains the improvements in PAR scores from beam search. The best overall decoding strategy appears to be beam search with n-gram blocking.   
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We now describe a manual evaluation and analyze some generated examples. All results in this section use the development set. We also conduct experiments on analyzing the effect of using the section data and the article data during training, finding that the benefits that they bring to the model performance are complementary. See the appendix for more details.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We conduct a human evaluation using generations from the large model on the development set. We choose texts shorter than 100 tokens and that cover particular topics as we found during pilot studies that annotators struggled with texts that were very long or about unfamiliar topics. 8 We design two sets of questions. The first focuses on the text itself (i.e., grammaticality and coherence) and its faithfulness to the input article table. Since this set does not involve the reference, we can ask these questions about both generated texts and the reference texts themselves. The second set of questions evaluates the differences between the generations and the reference texts (i.e., relevance and support), allowing us to see if the generated text matches the human written section text. Specifically, relevance evaluates topical similarity between generations and references, and support evaluates whether the facts expressed in the generations are supported by or contradictory to those in the references. The full questions and numerical answer descriptions are in the appendix.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We report results in Tables 4 and 5. The scores are on a 1-5 scale with 5 being the best. For the first set, we collect 480 annotations from 38 annotators. For the second set, we collect 360 annotations from 28 annotators. We also ensure that each system has the same number of annotations. 9 It is interesting to note from Table 4 that human annotators are unable to differentiate the human written texts from the generations from our neural models. Since the Wikipedia section texts are parts of Wikipedia articles, showing the section texts in isolation can make them difficult to understand, potentially resulting in noisy annotations. As shown by the first instance in Table 6, the text uses the pronoun "he" without clarifying what the pronoun refers to. The paragraph is rated 3 for coherence, presumably due to this ambiguity. Also, Wikipedia texts are sometimes grammatically complex and annotators can mistake them for being ungrammatical, e.g., the second instance in Table 6.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
On the other hand, the coherence errors in the generated texts are not always easy to spot. See, for example, the last two instances in Table 6, where the incoherence lies in the facts that (1) it is impossible to marry a person before the person is born, and (2) senior year takes place after junior year. These details are embedded in long contexts, which may be overlooked by annotators and lead to results favorable to these neural models.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
To study the relationship between coherence and grammaticality, we compute Spearman's correlations between the human annotations for coherence and grammaticality after removing the ones with perfect scores for coherence. Table 7 shows the results. The correlations are much higher for references, beam search, and nucleus sampling than for n-gram blocking. This trend suggests that the imperfect coherence scores for the reference texts are likely because annotators find the texts to contain grammatical errors (or to possess grammatical complexity) which may prevent them from fully understanding the texts. However, n-gram blocking does not have this problem and thus achieves the best results for both coherence and grammaticality. We hypothesize that n-gram blocking is able to avoid the types of grammatical errors that    Table 8: Averaged perplexities and the corresponding numbers of annotations for each option for the relevance and support questions (5 is the best option). We aggregate annotations for different decoding algorithms. We note that the perplexities are computed based on the reference texts using the large model. prevent understanding because (1) unlike nucleus sampling, n-gram blocking does not rely on randomness to avoid repetition; (2) n-gram blocking does not suffer from repetitions like beam search. We report results for the second set of questions in Table 5. The three evaluated systems show similar performance. To investigate the relationship between the degree of open-endedness of a WIKI-TABLET instance and its corresponding evaluation scores, we compute the averaged perplexities (based on our large models) for each option in Table 8. The most relevant generations are typically from more closed-ended or constrained instances. 10 Similarly for the support scores, more open-ended instances are distributed at score 3, which means that there is no fact supported by or contradictory to the shown tables. While the open-endedness of an instance usually depends on its topics (e.g., movie plots are open-ended), there are many cases where the models can benefit from better entity modeling, 10 Li and Hovy (2015)   such as understanding what a particular entity type is capable of (e.g., see the last example in Sec. 6.3). Recent work has also found conducting human evaluation for long-form generation to be challenging, for example in the context of question answering (Krishna et al., 2021) and story generation (Akoury et al., 2020). Our observations for data-to-text generation complement theirs and we hope that our dataset can inspire future research on human evaluation for long-form text generation.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
To determine the fraction of WIKITABLET that can be seen as constrained, we report the percentiles of perplexities for training and development splits in Table 9. From Table 8, it can be observed that instances with perplexities around 9.0 generally lead to model generations that are closely relevant to the reference texts and mostly supported by the input tables, and therefore are likely to be the constrained instances. From Table 9, we see that at least half of our dataset has perplexities lower than 9.0, so we conjecture that half of our dataset consists of constrained instances. Table 10 shows generation examples for nucleus sampling and beam search with n-gram blocking.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Generated Text Nucleus sampling (threshold 0.9) In a two-issue mini-series titled "Messiah Complex", portrayed by Maisie Williams, Father Craig is ordered to rescue the world from Reavers and run out of the X-Force causing the X-Force to permanently withdraw. The original humans were replaced by slots and Muir Island being pulled away. They also possessed the entire X-Force after issue #9 in which the X-Force appeared. Claremont and Sinclair later reprised their parts in "The Donderful World of Wolverine" with Wolfsbane as one of the new X-Men and her group taken back to their reservation for unknown reasons. n-gram blocking In the "Messiah Complex" storyline, Wolfsbane is a member of the X-Men, a group of mutants recruited by Father Craig to fight the Reavers in Muir Island. Father Craig is killed in the battle with Reavers, but Father Craig manages to rescue the mutant from the Rain of Gesham and retrieve the mutant. She is later revealed to have been killed by Rain in a battle with Father Craig. In the aftermath, Father Craig tells Father Craig that she is the only mutant to be killed by a mutant. Nucleus sampling (threshold 0.9)
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In May 2013, McCoy allegedly assault a Spoiler business official. The incident motivated him to take a business trip to Harrisburg, Pennsylvania and sign a letter from the Buffalo Bills paying him over $50,000 as well as verbally prohibiting legal kickoff work. In September 2014, McCoy shared the issue with some former players of the first two film franchises. McCoy played a leading role in the 2012 Marvel Cinematic Universe film "Erab of the Press", and was named one of the film's biggest stars on Twitter. On September 9, 2014, McCoy filed a lawsuit against PYT to change its Twitter name to LeSean McCoy Productions. n-gram blocking On September 9, 2014, McCoy was arrested in Harrisburg, Pennsylvania on suspicion of assault. He was charged with assault and battery. In May 2013, he was fined over $50,000 by the Buffalo Bills. In September 2014, he was suspended for two games by the PYT for violating the Marvel Cinematic Universe. He was released by the Bills in October of the same year. He was cleared of all charges on Twitter, and was banned from playing in the 2014 Pro Bowl due to his Twitter account. Table 10: Generation examples from the large model. The first example corresponds to the first instance in Figure 1. The complete set of generations is in the appendix.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We observe very different trends between the two instances in Figure 1. For the first instance about the X-Men, although both generations look fluent, their stories differ dramatically. The generated text for nucleus sampling describes a story that starts by saying Father Craig rescues the world from Reavers and ends with Wolfsbane joining as one of the new X-Men. On the other hand, n-gram blocking generates a story where Wolfsbane already is a member of X-Men, and the story says Father Craig fought and was killed by the Reavers, but manages to rescue the mutant. For the less open-ended instances (e.g., the second instance in Figure 1), different decoding strategies mostly generate similar details (see the appendix for generations).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Despite having different details, these generations appear to try to fit in as many entities from the tables as possible, in contrast to beam search (shown in the appendix) which mostly degenerates into repetition for more open-ended instances. This explains our previous observation that n-gram blocking helps with the PAR-R score.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Even though the generations are of good quality for most instances, their implausibility becomes more apparent when readers have enough background knowledge to understand the involved entities. For example, the second instance in Table 10 comes from the Wikipedia page "LeSean McCoy" (a football player) under the sections "Personal life" and "Controversies" (details in the appendix). The generation from nucleus sampling is implausi-ble/nonsensical in some places ("assault a Spoiler business official") and factually incorrect elsewhere (McCoy did not play a leading role in any film, and "Erab of the Press" is not an actual film). The fourth generation is implausible because a player is unlikely to be suspended for "violating the Marvel Cinematic Universe", and it is unlikely for a person to be cleared of all charges on Twitter. Our models have limited access to knowledge about entities, e.g., the capabilities of a social media company like Twitter. Future research may incorporate extra resources, make use of pretrained models, or incorporate factuality modules to solve these problems.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We created WIKITABLET, a dataset that contains Wikipedia article sections and their corresponding tabular data and various metadata. WIKITABLET contains millions of instances covering a broad range of topics and kinds of generation tasks. Our manual evaluation showed that humans are unable to differentiate the references and model generations, and n-gram blocking performs the best on grammaticality and coherence. However, qualitative analysis showed that our models sometimes struggle with coherence and factuality, suggesting several directions for future work.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We highlight a few limitations as follows: (1) Wikipedia texts are generally written in objective tones, but some of the texts may contain controversial content that even the community contributors do not agree upon; (2) models trained on our dataset may generate deceitful texts that are unfaithful to what actually happened to particular entities;
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
(3) though the instances in WIKITABLET cover various topics, the writing style is almost always the same. Future work may explore more diverse writing styles. 
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
When collecting data, we consider five resources: Wikidata tables, infoboxes in Wikipedia pages, hyperlinks in the passage, named entities in the passage obtained from named entity recognition (NER), and Wikipedia article structure. For each article in Wikipedia, we use the same infobox and Wikidata table for all sections. These tables can serve as background knowledge for the article. For each section in the article, we create a second table corresponding to section-specific data, i.e., section data. The section data contains records constructed from hyperlinks and entities identified by a named entity recognizer. Section data contributes around 25% of the records in WIKITABLET. We filter out several entity types related to numbers 11 as the specific meanings of these numbers in the section of interest are difficult to recover from the information in the tables. After filtering, we use the identified entities as the values and the entity types as the attributes. This contributes roughly 12% of the records in our final dataset.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We also create records from hyperlinks in the section of interest. We first expand the hyperlinks available for each section with hyperlinks available in the parent categories. We first group hyperlinks across all Wikipedia articles with those same categories, and then we perform string matching between these hyperlinks and the text in the section. If there are exact matches, we will include those hyperlinks as part of the hyperlinks in this section.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Details for constructing a record with attribute a and value v for a hyperlink with surface text t and hyperlinked article are as follows. To set a, we use the value of the "instance of" or "subclass of" tuple in the Wikidata table for . If does not have a Wikidata table or no appropriate tuple, we consider the parent categories of as candidates for a. If there are multiple candidates for a, we first embed these candidates and a using GloVe (Pennington et al., 2014) embeddings and then choose the one that maximizes cosine similarity between the document titles or section titles and the candidates for a. For the value v of the tuple, we use the document title of rather than the actual surface text t to avoid giving away too much information in the reference text. The records formed by hyperlinks contribute approximately 13% of the records in WIKITABLET. 11 List of filtered entity types: PERCENT, TIME, QUAN-TITY, ORDINAL, CARDINAL.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We shuffle the ordering of the records from NER and the hyperlinks to prevent models from relying on the ordering of records in the reference text.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
The records from the section data can be seen as section-specific information that can make the task more solvable. Complementary to the article data, we create a title table that provides information about the position in which the section is situated, which includes the article title and the section titles for the target section. As the initial sections in Wikipedia articles do not have section titles, we use the section title "Introduction" for these. 12 As the records in our data tables come from different resources, we perform extra filtering to remove duplicates in the records. In particular, we give Wikidata the highest priority as it is a humanannotated well-structured data resource (infoboxes are human-annotated but not well-structured due to the way they are stored on Wikipedia) and the entities from NER the lowest priority as they are automatically constructed. That is, when we identify duplicates across different resources, we will keep the records from the higher priority resource and drop those from the lower one. More specifically, the duplicates between Wikidata records and infoboxes are determined by whether there are duplicate values or duplicate attributes: for hyperlinks and infoboxes or Wikidata, they are judged by duplicate values; for NER and hyperlinks, they are based on whether there is any token overlapping between values.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
After table collection, we have the following criteria for filtering out the texts: (1) we limit the text length to be between 50 and 1000 word tokens;
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
(2) to ensure that there is sufficient information in the table, we only keep data-text pairs that contain more than 2 records per sentence and more than 15 records per 100 tokens from Wikidata and infoboxes; (3) to avoid texts such as lists of hyperlinks, we filter out texts where more than 50% of their word tokens are from hyperlink texts.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
The selected topics for human evaluations are: human (excluding the introduction and biography section), film, single (song), song, album, television series. When evaluating grammaticality and coherence, only the generated text is shown to annotators. 1 = it is completely ungrammatical, as it is impossible to understand the text. 2 = it has many grammatical errors, and these errors make the text very difficult to understand. 3 = it has grammatical errors, and some of them make part of the text difficult to understand. 4 = it has some grammatical errors, but they are minor errors that do not affect reading. 5 = it is completely grammatical, as it does not have any grammatical errors. Table 11: Rating explanations for grammaticality. 1 = it is completely incoherent, as it is impossible to piece together information in the text. 2 = it is incoherent in most places. You can only understand part of the story. 3 = it is incoherent in many places, but if you spend time reading it, you still can understand the whole story. 4 = it is mostly coherent. Although the text is incoherent in some places, it does not affect reading. 5 = it is completely coherent. The question for grammaticality is "On a scale of 1-5, how much do you think the text is grammatical? (Note: repetitions are grammatical errors.)" (option explanations are shown in Table 11), and the question for coherence is "On a scale of 1-5, how much do you think the text is coherent? (Coherence: Does the text make sense internally, avoid self-contradiction, and use a logical ordering of information?)" (rating explanations are in Table  12).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
When evaluating faithfulness, we show annotators the article data and the generation. The question is "On a scale of 1-5, how much do you think the text is supported by the facts in the following table?" (rating explanations are in Table 13).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
When evaluating coherence and relevance, annotators were shown the reference text and the generation, as well as the Wikipedia article title and section titles for ease of understanding the texts. Annotators were asked two questions, with one being "On a scale of 1-5, how much do you think the text is relevant to the reference" (Table 14), and the other being "On a scale of 1-5, how much do you think the text is supported by the facts in the reference?" (Table 15).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In this section, we disentangle the effect of αentmax and that of α-entmax loss. We note that (1) when not using the α-entmax loss, we use standard cross entropy loss (e.g., in the case of "base+ent."   we maximize the log probabilities generated by αentmax); (2) when combining α-entmax and copy mechanism, we aggregate the probabilities generated by α-entmax and those from softmax. This is because we use the first attention head in the transformer decoder as the copy attention, following the implementation in OpenNMT (Klein et al., 2017). While it is feasible to combine the α-entmax and α-entmax loss with the copy mechanism if we use the sparse transformer (Correia et al., 2019), we leave this for future study. We report the results in Table 16. It is interesting to see that when using greedy decoding, "ent. + ent. loss" outperforms the baseline model by a significant margin on all the metrics, however the improvement disappears (except for repetition) after we switch to use beam search as the decoding strategy. This is likely because α-entmax promotes sparsity in the generated probabilities, making beam search decoding unnecessary. Removing the α-entmax loss hurts the performance, but its gains become larger in switching to beam search decoding. Adding copy mechanism improves the performance, leading to comparable performance to the baseline model. Although "base+ent.+copy" still underperforms "base+copy" when using beam search, we believe that combining α-entmax and α-entmax loss with the copy mechanism is promising as (1) α-entmax is not used in our large models and the initial results have shown that α-entmax and the copy mechanism are complementary, so it may further improve our current best performance; (2) α-entmax already shows the best performance when using greedy decoding, which has speed and optimization advantages compared to the beam search based decoding strategies especially considering the long-form characteristic 1 = it has quite a few facts contradictory to what is described in the reference. 2 = it has some facts contradictory to what is described in the reference. 3 = it is not supported by the reference, and it does not contradict the reference. 4 = some of the text is supported by the facts in the reference, and the rest of it does not contradict the reference. 5 = it is completely supported by the reference.  
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In this section, we will denote the linearized table where the values are replaced with a special mask token by u 1 , · · · , u n , and denote the reference text by x 1 , · · · , x m . Formally, the training loss is w∈S − log p(w|u 1 , · · · , u n , v 1 , · · · , v m ) (2) where S represents the set of masked tokens, and v 1 , · · · , v m is the sequence of token-level probabilities predicted by the forward model (in our experiments, these could either come from the softmax function, or the α-entmax function). Specifically, we multiply the backward transformer's input embedding matrix by the v probability vectors to obtain the input representations to the first encoder layer. We find that it is helpful to add a "reference loss" while training with the cyclic loss, defined as w∈S − log p(w|u 1 , · · · , u n , x 1 , · · · , x m ) (3)
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
This loss does not contain the generation model in it explicitly, but it does lead to an improved backward model by training it with clean inputs. Improving   Table 18: Effect of dropping section or article data when using cyclic training. The results are based on the "base + copy" and "base + copy + cyclic loss" settings.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
the backward model then increases the benefits of the cyclic loss. 13
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We report results in Table 17 for the models that are trained with partial data input, where art. only and sec. only indicate that we use only article data or section data, respectively. We always use title data. Section data contributes the most to the BLEU and PAR scores, but using section data and article data together is the best setting. We also investigate the effect of partial data input for the cyclic loss in Table 18, where "None" is the model that is not trained with the cyclic loss. We note that in this setting, we still use both data resources as the input to the forward model, but vary the input data and the gold standard for the backward model. Although using only section data gives the best REP score and improves the PAR-P score, it does not help the model in other metrics. Combining the article data with the section data gives significant improvements to the PAR-F1 score compared to section data alone.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Both experiments show that there are interactions between these two data resources that can help models to learn better from both kinds.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
We show the full set of generations in Table 19. The part of input data and reference text for Table  19 is shown in Figure 2.  Figure 2: An example from WIKITABLET. Only parts of the tables and reference are shown due to space constraints. Underlined texts are hyperlinks. Records with the attributes "DATE", "ORG', "MONEY" are from NER. Subscripts for section titles indicate the ordering of nesting, where smaller numbers indicate higher level sections.
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Method Generated Text sampling 0.9
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In a two-issue mini-series titled "Messiah Complex", portrayed by Maisie Williams, Father Craig is ordered to rescue the world from Reavers and run out of the X-Force causing the X-Force to permanently withdraw. The original humans were replaced by slots and Muir Island being pulled away. They also possessed the entire X-Force after issue #9 in which the X-Force appeared. Claremont and Sinclair later reprised their parts in "The Donderful World of Wolverine" with Wolfsbane as one of the new X-Men and her group taken back to their reservation for unknown reasons. beam search (size=5)
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In the "Messiah Complex" storyline, Wolfsbane is a member of the X-Men. She is a member of the X-Factor Investigations team and a member of the X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/X-Force/ blocking (size=5)
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
In the "Messiah Complex" storyline, Wolfsbane is a member of the X-Men, a group of mutants recruited by Father Craig to fight the Reavers in Muir Island. Father Craig is killed in the battle with Reavers, but Father Craig manages to rescue the mutant from the Rain of Gesham and retrieve the mutant. She is later revealed to have been killed by Rain in a battle with Father Craig. In the aftermath, Father Craig tells Father Craig that she is the only mutant to be killed by a mutant. sampling 0.9
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Journey to the Center of the Earth is a 1959 American science fiction film based on Jules Verne's 1959 novel of the same name. The film, directed by Henry Levin, stars James Mason as Professor Otto Lidenbrock, a scientist who gets entangled in the power of the archenemy as he tries to impress his closest allies and compromises them with the future. The film marks the first screen appearance of future CinemaScope of 20th Century Fox lead singer Peter Ronson. "The 20th Century Fox" earned critical acclaim and was a box-office success, grossing $10 million worldwide against a production budget of $3.44 million . beam search (size=5) Journey to the Center of the Earth is a 1959 American science fiction film directed by Henry Levin and starring James Mason, Pat Boone, and Arlene Dahl. The screenplay by Charles Brackett and Walter Reisch is based on Jules Verne's novel "A Journey to the Center of the Earth" (1959). The film was produced by Charles Brackett and Walter Reisch and distributed by Twentieth Century-Fox Film Corporation. The film was nominated for an Academy Award for Best Art Direction. It was also nominated for an Academy Award for Best Art Direction and a Golden Globe Award for Best Visual Effects. It was also nominated for a Golden Globe Award for Best Motion Picture -Musical or Comedy, but lost to 20th Century Fox's "CinemaScope". blocking (size=5) Journey to the Center of the Earth is a 1959 American science fiction film directed by Henry Levin and starring James Mason, Pat Boone, and Arlene Dahl. The screenplay by Charles Brackett and Walter Reisch is based on Jules Verne's novel of the same name. The film was produced by CinemaScope and 20th Century Fox and distributed by Twentieth Century-Fox Film Corporation and Joseph M. Schenck Enterprises, Inc. It was nominated for an Academy Award for Best Art Direction.   
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Nucleus Sampling. Generating long sequences usually suffers from repetitions. Nucleus sampling (Holtzman et al., 2020) aims to reduce the repetitions in generations by sampling from truncated probability distributions. The truncation is based on whether the cumulative probability is above a threshold. We set the threshold to be 0.9 as suggested in Holtzman et al. (2020).
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Beam Search with n-gram Blocking. Paulus et al. (2018) found it effective to reduce the repetitions during beam search by "blocking" n-grams that have been generated in previous decoding steps. We follow their approach by using trigram blocking and setting the probability of repeated trigrams to be 0 during beam search.
ToTTo: A Controlled Table-To-Text Generation Dataset
Data-to-text generation (Kukich, 1983;McKeown, 1992) is the task of generating a target textual description y conditioned on source content x in the form of structured data such as a table. Examples include generating sentences given biographical data (Lebret et al., 2016), textual descriptions of restaurants given meaning representations (Novikova et al., 2017), and basketball game summaries given boxscore statistics (Wiseman et al., 2017).
ToTTo: A Controlled Table-To-Text Generation Dataset
Existing data-to-text tasks have provided an important test-bed for neural generation models (Sutskever et al., 2014;Bahdanau et al., 2014). Neural models are known to be prone to hallucination, i.e., generating text that is fluent but not faithful to the source (Vinyals and Le, 2015;Koehn and Knowles, 2017;Tian et al., 2019) and it is often easier to assess faithfulness of the generated text when the source content is structured (Wiseman et al., 2017;. Moreover, structured data can also test a model's ability for reasoning and numerical inference (Wiseman et al., 2017) and for building representations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015;Chen et al., 2019;Dua et al., 2019).
ToTTo: A Controlled Table-To-Text Generation Dataset
However, constructing a data-to-text dataset can be challenging on two axes: task design and annotation process. First, tasks with open-ended output like summarization (Mani, 1999;Lebret et al., 2016;Wiseman et al., 2017) lack explicit signals for models on what to generate, which can lead to subjective content and evaluation challenges (Kryściński et al., 2019). On the other hand, data-to-text tasks that are limited to verbalizing a fully specified meaning representation (Gardent et al., 2017b) do not test a model's ability to perform inference and thus remove a considerable amount of challenge from the task.
ToTTo: A Controlled Table-To-Text Generation Dataset
Secondly, designing an annotation process to obtain natural but also clean targets is a significant challenge. One strategy employed by many datasets is to have annotators write targets from scratch (Banik et al., 2013;Wen et al., 2015;Gardent et al., 2017a) which can often lack variety in terms of structure and style (Gururangan et al., 2018;Poliak et al., 2018). An alternative is to pair naturally occurring text with tables (Lebret et al., 2016;Wiseman et al., 2017). While more diverse, naturally occurring targets are often noisy and contain information that cannot be inferred from the source. This can make it problematic to disentangle modeling weaknesses from data noise.
ToTTo: A Controlled Table-To-Text Generation Dataset
In this work, we propose TOTTO, an opendomain table-to-text generation dataset that introduces a novel task design and annotation process   to address the above challenges. First, TOTTO proposes a controlled generation task: given a Wikipedia table and a set of highlighted cells as the source x, the goal is to produce a single sentence description y. The highlighted cells identify portions of potentially large tables that the target sentence should describe, without specifying an explicit meaning representation to verbalize.
ToTTo: A Controlled Table-To-Text Generation Dataset
For dataset construction, to ensure that targets are natural but also faithful to the source table, we request annotators to revise existing Wikipedia candidate sentences into target sentences, instead of asking them to write new target sentences (Wen et al., 2015;Gardent et al., 2017a). Table 1 presents a simple example from TOTTO to illustrate our annotation process. The table and Original Text were obtained from Wikipedia using heuristics that collect pairs of tables x and sentences y that likely have significant semantic overlap. This method ensures that the target sentences are natural, although they may only be partially related to the table. Next, we create a clean and controlled generation task by requesting annotators to highlight a subset of the table that supports the original sentence and revise the latter iteratively to produce a final sentence (see §5). For instance, in Table 1, the annotator has chosen to highlight a set of table cells (in yellow) that are mentioned in the original text. They then deleted phrases from the original text that are not supported by the table, e.g., for the playoffs first leg and replaced the pronoun he with an entity Cristhian Stuani. The resulting final sentence (Final Text) serves as a more suitable generation target than the original sentence. This annotation process makes our dataset well suited for high-precision conditional text generation.
ToTTo: A Controlled Table-To-Text Generation Dataset
Due to the varied nature of Wikipedia tables, TOTTO covers a significant variety of domains while containing targets that are completely faithful to the source (see Figures 2-6 for more complex examples). Our experiments demonstrate that stateof-the-art neural models struggle to generate faithful results, despite the high quality of the training data. These results suggest that our dataset and the underlying task could serve as a strong benchmark for controllable data-to-text generation models.
ToTTo: A Controlled Table-To-Text Generation Dataset
There are various existing strategies to create the reference target y. One strategy employed by many datasets is to have annotators write targets from scratch given a representation of the source (Banik et al., 2013;Wen et al., 2015;Gardent et al., 2017a). While this will result in a target that is faithful to the source data, it often lacks variety in terms of structure and style (Gururangan et al., 2018;Poliak et al., 2018). Domainspecific strategies such as presenting an annotator an image instead of the raw data (Novikova et al., 2016) are not practical for some of the complex tables that we consider. Other datasets have taken the opposite approach: finding real sentences on the web that are heuristically selected in a way that they discuss the source content (Lebret et al., 2016;Wiseman et al., 2017). This strategy typically leads to targets that are natural and diverse, but they may be noisy and contain information that cannot be inferred from the source .To construct TOTTO, we ask annotators to revise existing candidate sentences from Wikipedia so that they only contain information that is supported by the table. This enables TOTTO to maintain the varied language and structure found in natural sentences while producing cleaner targets. The technique of editing exemplar sentences has been used in semiparametric generation models (Guu et al., 2018;Pandey et al., 2018;Peng et al., 2019) and crowd-sourcing small, iterative changes to text has been shown to lead to higher-quality data and a more robust annotation process (Little et al., 2010). However, to our knowledge, we are the first to use this technique to construct generation datasets. Concurrent to this work, Chen et al. (2020) proposed LogicNLG which also uses Wikipedia tables, although omitting some of the more complex structured ones included in our dataset. Their target sentences are annotator-generated and their task is significantly more uncontrolled due to the lack of annotator highlighted cells.
ToTTo: A Controlled Table-To-Text Generation Dataset
Our tables come from English Wikipedia articles and thus may not be regular grids. 2 For simplicity, we define a table t as a set of cells t = {c j } τ j=1 where τ is the number of cells in the table. Each cell contains: (1) a string value, (2) whether or not it is a row or column header, (3) the row and column position of this cell in the table, (4) The number of rows and columns this cell spans.
ToTTo: A Controlled Table-To-Text Generation Dataset
Let m = (m page-title , m section-title , m section-text ) indicate table metadata, i.e, the page title, section title, and up to the first 2 sentences of the section text (if present) respectively. These fields can help provide context to the table's contents. Let s = (s 1 , ..., s η ) be a sentence of length η. We define an annotation example 3 d = (t, m, s) a tuple of table, table metadata, 
ToTTo: A Controlled Table-To-Text Generation Dataset
We first describe how to obtain annotation examples d for subsequent annotation. To prevent any overlap with the Wikibio dataset (Lebret et al., 2016), we do not use infobox tables. We employed three heuristics to collect tables and sentences:
ToTTo: A Controlled Table-To-Text Generation Dataset
Number matching We search for tables and sentences on the same Wikipedia page that overlap with a non-date number of at least 3 non-zero digits. The numbers are extracted by regular expressions that capture most common number patterns, including numbers with commas and decimal points. This approach captures most of the table-sentence pairs that describe statistics (e.g., sports, election, census, science, weather).
ToTTo: A Controlled Table-To-Text Generation Dataset
Cell matching We extract a sentence if it has tokens matching at least 3 distinct cell contents from the same row in the table. The intuition is that most tables are structured, and a row is usually used to describe a complete event (e.g., a sports game, an election, census data from a certain year), which is likely to have a corresponding sentence description from the same page.
ToTTo: A Controlled Table-To-Text Generation Dataset
Hyperlinks The above heuristics only consider sentences and tables on the same page. We also find examples where a sentence s contains a hyperlink to a page with a title that starts with List (these pages typically only consist of a large table). If the table t on that page also has a hyperlink to the page containing s, then we consider this to be an annotation example. Such examples typically result in more diverse examples than the other two heuristics, but also add more noise, since the sentence may only be distantly related to the table.
ToTTo: A Controlled Table-To-Text Generation Dataset
Using 
ToTTo: A Controlled Table-To-Text Generation Dataset
The collected annotation examples are noisy since a sentence s may be partially or completely unsupported by the table t. We thus define a data annotation process that guides annotators through small, incremental changes to the original sentence. This allows us to measure annotator agreement at every step of the process, which is atypical in existing generation datasets.
ToTTo: A Controlled Table-To-Text Generation Dataset
The primary annotation task consists of the following steps: (1) Table Readability, (2) Cell highlighting, (3) Phrase Deletion, (4) Decontextualization. Each of these are described below and more examples are provided in Table 3.
ToTTo: A Controlled Table-To-Text Generation Dataset
If a table is not readable, then the following steps will not need to be completed. This step is only intended to remove fringe cases where the table is poorly formatted or otherwise not understandable (e.g., in a different language). 99.5% of tables are determined to be readable.
ToTTo: A Controlled Table-To-Text Generation Dataset
Cell Highlighting An annotator is instructed to highlight cells that support the sentence. A phrase is supported by the table if it is either directly stated in the cell contents or meta-data, or can be logically inferred by them. Row and column headers do not need to be highlighted. If the table does not support any part of the sentence, then no cell is marked and no other step needs to be completed. 69.7% of examples are supported by the table.
ToTTo: A Controlled Table-To-Text Generation Dataset
For instance, in Figure 1, the annotator highlighted cells that support the phrases second, 13 November 2013, in Jordan, and 5-0 win. We denote the set of highlighted cells as a subset of the table: t highlight ∈ t.
ToTTo: A Controlled Table-To-Text Generation Dataset
Phrase Deletion This step removes phrases in the sentence unsupported by the selected table cells.
ToTTo: A Controlled Table-To-Text Generation Dataset
Annotators are restricted such that they are only able to delete phrases, transforming the original sentence: s → s deletion . In Table 1, the annotator transforms s by removing an individual word Charras', and an entire phrase: For the playoffs first leg, finishing nicols lodeiro's cross at close range.
ToTTo: A Controlled Table-To-Text Generation Dataset
On average, s deletion is different from s for 85.3% of examples and while s has an average length of 26.6 tokens, this is reduced to 15.9 for s deletion . We found that the phrases annotators often disagreed on corresponded to verbs purportedly supported by the table. For instance, in Table 1, some annotators decided netted is supported by the table since it is about soccer, while others opted to delete it.
ToTTo: A Controlled Table-To-Text Generation Dataset
Decontextualization A given sentence s may contain pronominal references or other phrases that depend on context. We thus instruct annotators to identify the main topic of the sentence; if it is a pronoun or other ambiguous phrase, we ask them to replace it with a named entity from the table or metadata. To discourage excessive modification,  they are instructed to make at most one replacement. 4 This transforms the sentence yet again: s deletion → s decontext . In Table 1, the annotator replaced he with Cristhian Stuani.
ToTTo: A Controlled Table-To-Text Generation Dataset
Since the previous steps can lead to ungrammatical sentences, annotators are also instructed to fix the grammar to improve the fluency of the sentence. We find that s decontext is different than s deletion 68.3% of the time, and the average sentence length increases to 17.2 tokens for s decontext compared to 15.9 for s deletion .
ToTTo: A Controlled Table-To-Text Generation Dataset
Due to the complexity of the primary annotation task, the resulting sentence s decontext may still have grammatical errors, even if annotators were instructed to fix grammar. Thus, a second set of annotators were asked to further correct the sentence and were shown the table with highlighted cells as additional context. However, the annotators were not required to use the table. They were asked to determine the grammaticality and fluency of the provided sentence. If the sentence is not fluent or grammatical, they fix the errors to make it such. Annotators are also given an option to indicate that the sentence is not fixable.
ToTTo: A Controlled Table-To-Text Generation Dataset
This results in the final sentence s final . On average, annotators edited the sentence 27.0% of the time, and the sentence length slightly increased to 17.4 tokens from 17.2. We found that for most of the cases, the table is not necessary to fix the sentence since the grammatical errors are due to  surface syntax, such as a missing punctuation or a missing determiner. In a few cases, a verb may be missing, and in such instances, the table is needed to indicate the correct verb to use.
ToTTo: A Controlled Table-To-Text Generation Dataset
Basic statistics of TOTTO are described in  (3). This indicates the importance of the cell highlighting feature of our dataset toward a well-defined text generation task. Table 5 shows annotator agreement over the development set for each step of the annotation process. We compute annotator agreement and Fleiss' kappa (Fleiss, 1971) for table readability and highlighted cells, and BLEU-4 score between annotated sentences in different stages, including (1) sentence after deletion;
ToTTo: A Controlled Table-To-Text Generation Dataset
(2) sentence after decontextualization;    and (3) final sentence after the secondary grammar correction task. As one can see, the table readability task has an agreement of 99.38%. The cell highlighting task is more challenging. 73.74% of the time all three annotators completely agree on the set of cells which means that they chose the exact same set of cells. The Fleiss' kappa is 0.856, which is regarded as "almost perfect agreement" (0.81 -1.00) according to (Landis and Koch, 1977).
ToTTo: A Controlled Table-To-Text Generation Dataset
With respect to the sentence revision tasks, we see that the agreement slightly degrades as more steps are performed. We compute single reference BLEU among all pairs of annotators for examples in our development set (which only contains examples where both annotators chose t highlight = ∅). As the sequence of revisions are performed, the annotator agreement gradually decreases in terms of BLEU-4: 82.19 → 72.56 → 68.98. This is considerably higher than the BLEU-4 between the original sentence s and s final (45.87).
ToTTo: A Controlled Table-To-Text Generation Dataset
We use the Wikimedia Foundation's topic categorization model (Asthana and Halfaker, 2018)   ure 1 presents an aggregated topic analysis of our dataset. We found that the Sports and Countries topics together comprise 56.4% of our dataset, but the other 44% is composed of a much broader set of topics such as Performing arts, Transportation, and Entertainment. 
ToTTo: A Controlled Table-To-Text Generation Dataset
Each annotation consists of the set of highlighted cells t highlight and the modified sentences s deletion , s decontext , s final . After the annotation process, we only consider examples where the sentence is related to the table, i.e., t highlight = ∅. This initially results in a training set D orig-train of size 131,849 that we further filter as described below.
ToTTo: A Controlled Table-To-Text Generation Dataset
For more robust evaluation, each example in the development and test sets was annotated by three annotators. Since the machine learning task uses t highlight as an input, it is challenging to use three different sets of highlighted cells in evaluation. Thus, we only use a single randomly chosen t highlight while using the three s final as references for evaluation 6 . We only use examples where at least 2 of the 3 annotators chose t highlight = ∅. This results in a development set D dev of size 7,700 and a test set D test of size 7,700.
ToTTo: A Controlled Table-To-Text Generation Dataset
Overlap and Non-Overlap Sets Without any modification D orig-train , D dev , and D test may contain many similar tables. Thus, to increase the generalization challenge, we filter D orig-train to remove some examples based on overlap with D dev , D test .
ToTTo: A Controlled Table-To-Text Generation Dataset
For a given example d, let h(d) denote its set of header values and similarly let h(D) be the set of header values for a given dataset D.
ToTTo: A Controlled Table-To-Text Generation Dataset
We remove examples d from the training set where h(d) is both rare in the data as well as occurs in either the development or test sets. Specifically, D train is defined as:
ToTTo: A Controlled Table-To-Text Generation Dataset
The count(h(d), D orig-train ) function returns the number of examples in D orig-train with header h(d).
ToTTo: A Controlled Table-To-Text Generation Dataset
To choose the hyperparameter κ we first split the test set as follows:
ToTTo: A Controlled Table-To-Text Generation Dataset
The development set is analogously divided into D dev-overlap and D dev-nonoverlap . We then choose κ = 5 so that D test-overlap and D test-nonoverlap have similar size. After filtering, the size of D train is 120,761, and D dev-overlap , D dev-nonoverlap , D test-overlap , and D test-nonoverlap have sizes 3784, 3916, 3853, and 3847 respectively.
ToTTo: A Controlled Table-To-Text Generation Dataset
In this work, we focus the following task:
ToTTo: A Controlled Table-To-Text Generation Dataset
Given a table t and related metadata m (page title, section title, table section text) and a set of highlighted cells t highlight , produce the final sentence s final . Mathematically this can be described as learning a function f : x → y where x = (t, m, t highlight ) and y = s final .
ToTTo: A Controlled Table-To-Text Generation Dataset
Note that this task is different from what the annotators perform, since they are provided with a starting sentence requiring revision. Therefore, this task is more challenging, as the model must generate a new sentence instead of revising an existing sentence. Since we use several stages in our annotation mechanism, one could design several other tasks for machine learning models given the data such as sentence revision or cell highlighting, but we leave this out of the scope of this work.
ToTTo: A Controlled Table-To-Text Generation Dataset
We present baseline results on TOTTO by examining three existing state-of-the-art approaches:
ToTTo: A Controlled Table-To-Text Generation Dataset
• BERT-to-BERT (Rothe et al., 2019): A Transformer encoder-decoder model (Vaswani et al., 2017) where the encoder and decoder are both initialized with BERT (Devlin et al., 2018). The original BERT model is pre-trained with both Wikipedia and the Books corpus (Zhu et al., 2015), the former of which contains our (unrevised) test targets. Thus, we also pre-train a version of BERT on the Books corpus only, which we consider a more correct baseline. However, empirically we find that both models perform similarly in practice (Table 7). with an explicit content selection and planning mechanism designed for data-to-text.
ToTTo: A Controlled Table-To-Text Generation Dataset
Moreover, we explore different strategies of representing the source content that resemble standard linearization approaches in the literature (Lebret et al., 2016;Wiseman et al., 2017).
ToTTo: A Controlled Table-To-Text Generation Dataset
• Full Table The simplest approach is simply to use the entire table as the source, adding special tokens to mark which cells have been highlighted. However, many tables can be very large and this strategy performs poorly. • Subtable Another option is to only use the highlighted cells t highlight ∈ t with the heuristically extracted row and column header for each highlighted cell. This makes it easier for the model to only focus on relevant content but limits the ability to perform reasoning in the context of the table structure (see Table 10). Overall though, we find this representation leads to higher performance.
ToTTo: A Controlled Table-To-Text Generation Dataset
In all cases, the selected cells are linearized with row and column separator tokens. We also experiment with prepending the table metadata to the source table. 7
ToTTo: A Controlled Table-To-Text Generation Dataset
The model output is evaluated using two automatic metrics. Human evaluation is described in § 8.3. BLEU (Papineni et al., 2002): A widely used metric that uses n-gram overlap between the reference y and the predictionŷ at the corpus level. BLEU does not take the source content x into account.
ToTTo: A Controlled Table-To-Text Generation Dataset
PARENT : A metric recently proposed specifically for data-to-text evaluation that takes the table into account. PARENT is defined at an instance level. For a given example (x n , y n ,ŷ n ) PARENT is defined as:
ToTTo: A Controlled Table-To-Text Generation Dataset
P AREN T (x n , y n ,ŷ n ) = 2 × E p (x n , y n ,ŷ n ) × E r (x n , y n ,ŷ n ) E p (x n , y n ,ŷ n ) + E r (x n , y n ,ŷ n ) E p (x n , y n ,ŷ n ) is the PARENT precision computed using the prediction, reference, and table (the last of which is not used in BLEU). E r (x n , y n ,ŷ n ) is the PARENT recall and is computed as:
ToTTo: A Controlled Table-To-Text Generation Dataset
where R(x n , y n ,ŷ n ) is a recall term that compares the prediction with both the reference and table.
ToTTo: A Controlled Table-To-Text Generation Dataset
R(x n ,ŷ n ) is an extra recall term that gives an additional reward if the predictionŷ n contains phrases in the table x n that are not necessarily in the reference (λ is a hyperparameter).
ToTTo: A Controlled Table-To-Text Generation Dataset
In the original PARENT work , the same table t is used for computing the precision and both recall terms. While this makes sense for most existing datasets, it does not take into account the highlighted cells t highlight in our task. To incorporate t highlight , we modify the PARENT metric so that the additional recall term R(x n ,ŷ n ) uses t highlight instead of t to only give an additional reward for relevant table information. The other recall and the precision term still use t. Table 7 shows our results against multiple references with the subtable input format. Both the BERT-to-BERT models perform the best, followed by the pointer generator model. 8 We see that for all models the performance on the non-overlap set is significantly lower than that of the overlap set, indicating that slice of our data poses significant challenges for machine learning models. We also observe that the baseline that separates content selection and planning performs quite poorly. We attest this to the fact that it is engineered to the Rotowire data format with fixed size tables and predefined column names. Table 8 explores the effects of the various input representations (subtable vs. full table) on the BERT-to-BERT model. We see that the full table format performs poorly even if it is the most knowledge-preserving representation. Using table metadata significantly helps under different input.
ToTTo: A Controlled Table-To-Text Generation Dataset
For each of the 2 top performing models in Table 7, we take 500 random outputs and perform human evaluation using the following axes:
ToTTo: A Controlled Table-To-Text Generation Dataset
• Fluency -A candidate sentence is fluent if it is grammatical and natural. The three choices are Fluent, Mostly Fluent, Not Fluent. • Faithfulness (Precision) -A candidate sentence is considered faithful if all pieces of information are supported by either the table or one of the references. Any piece of unsupported information makes the candidate unfaithful. • Covered Cells (Recall) -The percentage of highlighted cells the candidate sentence covers. • Coverage with Respect to Reference (Recall) -We ask whether the candidate is strictly more or less informative than each reference (or neither, which is referred to as neutral).
ToTTo: A Controlled Table-To-Text Generation Dataset
In addition to evaluating the model outputs, we compute an oracle upper-bound by treating one of the references as a candidate and evaluating it compared to the table and other references. The results, shown in Table 9, attest to the high quality of our human annotations since the oracle consistently achieves high performance. All the axes demonstrate that there is a considerable gap between the model and oracle performance.
ToTTo: A Controlled Table-To-Text Generation Dataset
This difference is most easily revealed in the last column when annotators are asked to directly compare the candidate and reference. As expected, the oracle has similar coverage to the reference (61.7% neutral) but both baselines demonstrate considerably less coverage. According to an independentsample t-test, this difference is significant at a p < 0.001 level for both baselines. Similarly, we observe a significantly lower percentage of covered cells for the baselines compared to the reference according to a χ 2 test. Comparing the baselines to each other, we do not observe a significant difference in either coverage metric.
ToTTo: A Controlled Table-To-Text Generation Dataset
Furthermore, the baselines are considerably less faithful than the reference. The faithfulness of both models is significantly lower than the reference (χ 2 test with p < 0.001). The models do not differ significantly from each other, except for the non-overlap case, where we see a moderate effect    (Wiseman et al., 2017;Tian et al., 2019), our results indicate it is a problem even when the references are clean.
ToTTo: A Controlled Table-To-Text Generation Dataset
In this section, we visualize some example decoder outputs from the BERT-to-BERT Books model (Table 10) and discuss specific challenges that existing approaches face with this dataset. In general, the model performed reasonably well in producing grammatically correct and fluent sentences given the information from the table, as indicated by Table 10. Given the "full table", the model is not able to correctly select the information needed to produce the reference, and instead produces an arbitrary sentence with irrelevant information. Note the model corrects itself with highlighted cell information ("subtable"), and learns to use the metadata to improve the sentence. However, we also observe certain challenges that existing approaches are struggling with, which can serve as directions for future research. In particular:
ToTTo: A Controlled Table-To-Text Generation Dataset
Hallucination As shown in Table 10 (examples 1-4) the model sometimes outputs phrases such as first scottish, third that seem reasonable but are not faithful to the table. This hallucination phenomenon has been widely observed in other existing data-to-text datasets (Lebret et al., 2016;Wiseman et al., 2017). However, the noisy references in these datasets make it difficult to disentangle model incapability from data noise. Our dataset serves as strong evidence that even when the reference targets are faithful to the source, neural models still struggle with faithfulness.
ToTTo: A Controlled Table-To-Text Generation Dataset
Rare topics Another challenge revealed by the open domain nature of our task is that models often struggle with rare or complex topics. For instance, example 5 of Table 10 concerns microdrive capacities which is challenging. As our topic distribution indicates (Figure 1) Table 9: Human evaluation over references (to compute Oracle) and model outputs. For Fluency, we report the percentage of outputs that were completely fluent. In the last column X/Y /Z means X% and Z% of the candidates were deemed to be less and more informative than the reference respectively and Y% were neutral.
ToTTo: A Controlled Table-To-Text Generation Dataset
Decoder output (w/ metadata) w/o metadata Full cortete's production was 512 megabyte.
ToTTo: A Controlled Table-To-Text Generation Dataset
6 the 1956 grand prix motorcycle racing season consisted of six grand prix races in five classes: 500cc, 350cc, 250cc, 125cc and sidecars 500cc.
ToTTo: A Controlled Table-To-Text Generation Dataset
the 1966 grand prix motorcycle racing season consisted of seven grand prix races in five classes: 500cc, 350cc, 250cc, 125cc and sidecars 500cc.
ToTTo: A Controlled Table-To-Text Generation Dataset
the 1956 grand prix motorcycle racing season consisted of eight grand prix races in five classes: 500cc, 350cc, 250cc, 125cc and sidecars 500cc.
ToTTo: A Controlled Table-To-Text Generation Dataset
the 1955 grand prix motorcycle racing season consisted of eight grand prix races in five classes: 500cc, 350cc, 250cc, 125cc and sidecars 500cc. 7
ToTTo: A Controlled Table-To-Text Generation Dataset
in travis kelce's last collegiate season, he set personal career highs in receptions (45)  built on learned models (Wiseman et al., 2017;Ma et al., 2019;Sellam et al., 2020) for appropriate evaluation.
ToTTo: A Controlled Table-To-Text Generation Dataset
In this work, we presented TOTTO, a large, English table-to-text dataset that presents both a controlled generation task and a data annotation process based on iterative sentence revision. We also provided several state-of-the-art baselines, and demonstrated TOTTO could be a useful dataset for modeling research as well as for developing evaluation metrics that can better detect model improvements. TOTTO is available at https://github. com/google-research-datasets/totto.         