What are the two main categories of factual consistency metrics proposed for summarization models?
Factual consistency metrics. QA-based consis-
tency metrics (Durmus et al., 2020; Scialom et al.,
2021; Fabbri et al., 2022) involve generating questions from the given document and its summary,
and then comparing the corresponding answers to
compute a factual consistency score. Entailment-
based consistency metrics (Laban et al., 2022;
Kryscinski et al., 2020; Ribeiro et al., 2022; Goyal
and Durrett, 2020) utilize a binary classifier to de-
termine whether the contents in a system summary
are entailed by the source article. In contrast, our
model is a multi-label classifier that detects the
types of factual errors in a summary. Moreover,
our model leverages SRL to encode the facts in
the input document and summary, enabling users
to interpret which facts in the document are most
relevant to the inconsistency detection.
Evaluating the Factual Consistency of Abstractive Text Summarization
Currently used metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents. We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary. Training data is generated by applying a series of rule-based transformations to the sentences of source documents. The factual consistency model is then trained jointly for three tasks: 1) identify whether sentences remain factually consistent after transformation, 2) extract a span in the source documents to support the consistency prediction, 3) extract a span in the summary sentence that is inconsistent if one exists. Transferring this model to summaries generated by several state-of-the art models reveals that this highly scalable approach substantially outperforms previous models, including those trained with strong supervision using standard datasets for natural language inference and fact checking. Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency.
[{"first":"Wojciech","last":"Kry'sci'nski","middle":[]},{"first":"Bryan","last":"McCann","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]},{"first":"Richard","last":"Socher","middle":[]}]
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.
[{"first":"Esin","last":"Durmus","middle":[]},{"first":"He","last":"He","middle":[]},{"first":"Mona","last":"Diab","middle":[]}]
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications. Recent works have shown promising improvements in factuality error identification using text or dependency arc entailments; however, they do not consider the entire semantic graph simultaneously. To this end, we propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MR), which are more suitable for factuality evaluation. MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. FactGraph encodes such graphs using a graph encoder augmented with structure-aware adapters to capture interactions among the concepts based on the graph connectivity, along with text representations using an adapter-based text encoder. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%. Furthermore, FactGraph improves performance on identifying content verifiability errors and better captures subsentence-level factual inconsistencies.
[{"first":"Leonardo","last":"Ribeiro","middle":[]},{"first":"Mengwen","last":"Liu","middle":[]},{"first":"Iryna","last":"Gurevych","middle":[]},{"first":"Markus","last":"Dreyer","middle":[]},{"first":"Mohit","last":"Bansal","middle":[]}]
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.
[{"first":"Philippe","last":"Laban","middle":[]},{"first":"Tobias","last":"Schnabel","middle":[]},{"first":"Paul N.","last":"Bennett","middle":[]},{"first":"Marti A.","last":"Hearst","middle":[]}]
QuestEval: Summarization Asks for Fact-based Evaluation
Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.
[{"first":"Thomas","last":"Scialom","middle":[]},{"first":"Paul-Alexis","last":"Dray","middle":[]},{"first":"Patrick","last":"Gallinari","middle":[]},{"first":"Sylvain","last":"Lamprier","middle":[]},{"first":"Benjamin","last":"Piwowarski","middle":[]},{"first":"Jacopo","last":"Staiano","middle":[]},{"first":"Alex","last":"Wang","middle":[]}]
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
[{"first":"Alexander","last":"Fabbri","middle":[]},{"first":"Chien-Sheng","last":"Wu","middle":[]},{"first":"Wenhao","last":"Liu","middle":[]},{"first":"Caiming","last":"Xiong","middle":[]}]
Evaluating Factuality in Generation with Dependency-level Entailment
Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.
[{"first":"Tanya","last":"Goyal","middle":[]},{"first":"Greg","last":"Durrett","middle":[]}]
Evaluating the Factual Consistency of Abstractive Text Summarization
The goal of text summarization is to transduce long documents into a shorter form that retains the most important aspects from the source document. Common approaches to summarization are extractive (Dorr et al., 2003;Nallapati et al., 2017) where models directly copy salient parts of the source document into the summary, abstractive (Rush et al., 2015;Paulus et al., 2017) where the important parts are paraphrased to form novel sentences, and hybrid (Gehrmann et al., 2018), combining the two methods by employing specialized extractive and abstractive components.
Evaluating the Factual Consistency of Abstractive Text Summarization
Advancements in neural architectures (Cho et al., 2014;Bahdanau et al., 2015;Vaswani et al., 2017), transfer learning (McCann et al., 2017;Devlin et al., 2018), and availability of large-scale supervised datasets (Nallapati et al., 2016;Grusky et al., 2018) allowed deep learning-based approaches to dominate the field. State-of-the-art solutions utilize self-attentive Transformer blocks (Liu, 2019; Liu and Lapata, 2019), attention and copying mechanisms (See et al., 2017;Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018;, including reinforcement learning techniques (Kryściński et al., 2018;Dong et al., 2018;Wu and Hu, 2018). Despite significant efforts made by the research community, there are still many challenges limiting progress in summarization: insufficient evaluation protocols that omit important dimensions, such as factual consistency, noisy datasets that leave the task underconstrained, and strong, domain-specific layout biases in the data that dominate training signal (Kryściński et al., 2019).
Evaluating the Factual Consistency of Abstractive Text Summarization
We address the problem of verifying factual consistency between source documents and generated summaries: a factually consistent summary contains only statements that are entailed by the source document. Recent studies show that up to 30% of summaries generated by abstractive models contain factual inconsistencies (Cao et al., 2018;Goodrich et al., 2019;Falke et al., 2019;Kryściński et al., 2019). Such high levels of factual inconsistency render automatically generated summaries virtually useless in practice.
Evaluating the Factual Consistency of Abstractive Text Summarization
The problem of factual consistency is closely related to natural language inference (NLI) and fact Source article fragments (CNN) The mother of a quadriplegic man who police say was left in the woods for days cannot be extradited to face charges in Philadelphia until she completes an unspecified "treatment," Maryland police said Monday. The Montgomery County (Maryland) Department of Police took Nyia Parler, 41, into custody Sunday (...) (CNN) The classic video game "Space Invaders" was developed in Japan back in the late 1970's -and now their real-life counterparts are the topic of an earnest political discussion in Japan's corridors of power. Luckily, Japanese can sleep soundly in their beds tonight as the government's top military official earnestly revealed that (...)
Evaluating the Factual Consistency of Abstractive Text Summarization
Quadriplegic man Nyia Parler, 41, left in woods for days can not be extradited.
Evaluating the Factual Consistency of Abstractive Text Summarization
Video game "Space Invaders" was developed in Japan back in 1970. Table 1: Examples of factually incorrect claims output by summarization models. Green text highlights the support in the source documents for the generated claims, red text highlights the errors made by summarization models.
Evaluating the Factual Consistency of Abstractive Text Summarization
checking. Current NLI datasets (Bowman et al., 2015;Williams et al., 2018) focus on classifying logical entailment between short, single sentence pairs, but verifying factual consistency requires the entire source document. Fact checking focuses on verifying facts against the whole of available knowledge, whereas factual consistency checking focuses on adherence of facts to information provided by a source document without guarantee that the information is true.
Evaluating the Factual Consistency of Abstractive Text Summarization
We propose a novel, weakly-supervised BERTbased (Devlin et al., 2018) model for verifying factual consistency, and we add specialized modules that explain which portions of both the source document and generated summary are pertinent to the model's decision. Training data is generated from source documents by applying a series of rule-based transformations that were inspired by error-analysis of neural summarization model outputs. Through human evaluation we show that the explanatory modules that augment our factual consistency model provide useful assistance to humans as they verify the factual consistency between a source document and generated summaries. Together with this manuscript we release a manually annotated dataset for factual consistency verification, code for training data generation, and trained model weights at https://github.com/salesforce/factCC.
Evaluating the Factual Consistency of Abstractive Text Summarization
A careful study of the outputs of state-of-the-art summarization models provided us with valuable insights about the specifics of factual errors made during generation and possible means of detecting them. Primarily, checking factual consistency on a sentence-sentence level, where each sentence of the summary is verified against each sentence from the source document, is insufficient. Some cases might require a longer, multi-sentence context from the source document due to ambiguities present in either of the compared sentences. Summary sentences might paraphrase multiple fragments of the source document, while source document sentences might use certain linguistic constructs, such as coreference, which bind different parts of the document together. In addition, errors made by summarization models are most often related to the use of incorrect entity names, numbers, and pronouns. Other errors such as negations and common sense error occur less often. Taking these insights into account, we propose and test a document-sentence approach for factual consistency checking, where each sentence of the summary is verified against the entire body of the source document.
Evaluating the Factual Consistency of Abstractive Text Summarization
Currently, there are no training datasets for factual consistency checking. Creating a large-scale, highquality dataset with supervision collected from human annotators is expensive and time consuming. We consider an alternative approach to acquiring training data that is highly scalable.
Evaluating the Factual Consistency of Abstractive Text Summarization
Considering the state of summarization, in which the level of abstraction of generated summaries is low and models mostly para-phrase single sentences and short spans from the source (Kryściński et al., 2018;, we propose using a synthetic, weaklysupervised dataset for the task at hand. Our data creation method requires an unannotated collection of source documents in the same domain as the summarization models that are to be checked. Examples are created by first sampling single sentences, later referred to as claims, from the source documents. Claims then pass through a set of textual transformations that output novel sentences with both positive and negative labels. Though transformations are applied to single sentences, we found that, in keeping with our aforementioned observations of model-generated summaries, the process of verifying their consistency often requires referring to the entire document. A detailed description of the data generation function is presented in Figure 1. The benefit of using a synthetic dataset is that it allows for creation of large volumes of data at a marginal cost. The data generation process also allows to collect additional metadata that can be used in the training process. In our case, the metadata contains information about the original location of the extracted claim in the source document and the locations in the claim where text transformations were applied.
Evaluating the Factual Consistency of Abstractive Text Summarization
Our data generation process draws inspiration from data augmentation and adversarial example generation techniques in NLP (Iyyer et al., 2018;Wu et al., 2019;Zhang et al., 2019;Wei and Zou, 2019). The proposed process incorporates both semantically invariant (T + ), and variant (T − ) text transformations to generate novel claims with CORRECT and INCORRECT labels accordingly. This work uses the following transformations:
Evaluating the Factual Consistency of Abstractive Text Summarization
Paraphrasing A paraphrasing transformation covers cases where source document sentences are rephrased by the summarization model. Paraphrases were produced by backtranslation using Neural Machine Translation systems (Iyyer et al., 2018). The claim sentence was translated to an intermediate language and translated back to English yielding a semantically-equivalent sentence with minor syntactic and lexical changes. French, German, Chinese, Spanish, and Russian were used as intermediate languages. These languages were chosen based on the performance of recent NMT systems with the expectation that well-performing languages could ensure better translation quality.
Evaluating the Factual Consistency of Abstractive Text Summarization
Original sentence Transformed sentence
Evaluating the Factual Consistency of Abstractive Text Summarization
Sheriff Lee Baca has now decided to recall some 200 badges his department has handed out to local politicians just two weeks after the picture was released by the U.S. attorney's office in support of bribery charges against three city officials.
Evaluating the Factual Consistency of Abstractive Text Summarization
Two weeks after the US Attorney's Office issued photos to support bribery allegations against three municipal officials, Lee Baca has now decided to recall about 200 badges issued by his department to local politicians.
Evaluating the Factual Consistency of Abstractive Text Summarization
Snow was predicted later in the weekend for Atlanta and areas even further south.
Evaluating the Factual Consistency of Abstractive Text Summarization
Snow wasn't predicted later in the weekend for Atlanta and areas even further south.
Evaluating the Factual Consistency of Abstractive Text Summarization
Pronoun swap It comes after his estranged wife Mona Dotcom filed a $20 million legal claim for cash and assets.
Evaluating the Factual Consistency of Abstractive Text Summarization
It comes after your estranged wife Mona Dotcom filed a $20 million legal claim for cash and assets.
Evaluating the Factual Consistency of Abstractive Text Summarization
Entity swap Charlton coach Guy Luzon had said on Monday: 'Alou Diarra is training with us.'
Evaluating the Factual Consistency of Abstractive Text Summarization
Charlton coach Bordeaux had said on Monday: 'Alou Diarra is training with us.'
Evaluating the Factual Consistency of Abstractive Text Summarization
He says he wants to pay off the $12.6million lien so he can sell the house and be done with it.
Evaluating the Factual Consistency of Abstractive Text Summarization
He says he wants to pay off the $3.45million lien so he can sell the house and be done done with it.
Evaluating the Factual Consistency of Abstractive Text Summarization
Noise injection Snow was predicted later in the weekend for Atlanta and areas even further south.
Evaluating the Factual Consistency of Abstractive Text Summarization
Snow was was predicted later in the weekend for Atlanta and areas even further south. We used the Google Cloud Translation API 1 for translations.
Evaluating the Factual Consistency of Abstractive Text Summarization
Entity and Number swapping To learn how to identify examples where the summarization model used incorrect numbers or entities during generation we used the Entity and Number swapping transformation. An NER system was applied to both the claim and source document to extract all entities. To generate a novel, semantically changed claim, an entity in the claim sentence was replaced with an entity from the document. Both of the swapped entities were chosen at random while ensuring that they were unique. Extracted entities were divided into two groups, named entities, covering person, location and institution names, and numbers, such as dates and all other numeric values. Entities were swapped within their respective groups. We used the SpaCy NER tagger (Honnibal and Montani, 2017).
Evaluating the Factual Consistency of Abstractive Text Summarization
Pronoun swapping To learn how to find incorrect pronoun use in claims we used a pronoun swapping transformation. First, all gender-specific pronouns were first extracted from the claim. Next, a randomly chosen pronoun was swapped with a different one from the same pronoun group to ensure syntactic correctness, i.e. a possessive pronoun could only be replaced with another possessive pronoun. New sentences were considered semantically variant.
Evaluating the Factual Consistency of Abstractive Text Summarization
1 https://cloud.google.com/translate/ Sentence negation To give the consistency checking model the ability to handle negated sentences we used a sentence negation transformation. First, a claim was scanned in search of auxiliary verbs. To switch the meaning, a randomly chosen auxiliary verb was replaced with its negation. Positive sentences would be negated by adding not or n't after the verb, negative sentences would be switched by removing the negation.
Evaluating the Factual Consistency of Abstractive Text Summarization
Noise injection Given that verified summaries are generated by neural networks, they should be expected to contain certain types of noise. In order to make the factual consistency model robust to such generation errors, training examples were injected with noise. For each token in a claim the decision was made whether noise should be added at the given position with a preset probability. If noise should be injected, the token was randomly duplicated or removed from the sequence. Examples of all transformations are shown in Table 2.
Evaluating the Factual Consistency of Abstractive Text Summarization
Apart from the synthetic training set, separate, manually annotated, validation and test sets were created. Both of the annotated sets used summaries output by state-of-the-art summarization models. Each summary was split into sentences and all (document, sentence) pairs were annotated by the authors of this work. Since the focus was to collect data for verifying the factual consistency of summarization models, any unreadable sentences Require: S -set of source documents T + -set of semantically invariant transformations T − -set of semantically variant transformations Effort was made to collect a larger set of annotations through crowdsourcing platforms, however the inter-annotator agreement and general quality of annotations was too low to be considered reliable. This aligns with the conclusions of (Falke et al., 2019), where the authors showed that for the task of factual consistency the interannotator agreement coefficient κ reached 0.75 only when 12 annotations were collected for each example. This in turn yields high annotations costs that our approach aims to circumvent.
Evaluating the Factual Consistency of Abstractive Text Summarization
Considering the significant improvements in natural language understanding (NLU) tasks (including NLI) coming from using pre-trained  Transformer-based models 2 , we decided to use BERT (Devlin et al., 2018) as the base model for our work. An uncased, base (110M params) BERT architecture was used as the starting checkpoint and fine-tuned on the generated training data. The source document and claim sentence were fed as input to the model and two-way classification (CONSISTENT/INCONSISTENT) was done using a single-layer classifier based on the [CLS] token. We refer to this model as the factual consistency checking model (FactCC). We also trained a version of FactCC with additional span selection heads using supervision of start and end indices for selection and transformation spans in the source and claim. Span selection heads allow the model not only to classify the consistency of the claim, but also highlight spans in the source document that contain the support for the claim and spans in the claim where a possible mistake was made. We refer to this model as the factual consistency checking model with explanations (FactCCX).
Evaluating the Factual Consistency of Abstractive Text Summarization
Training data was generated as described in Section 3.1 using news articles from the CNN/DailyMail (Nallapati et al., 2016) dataset as source documents. 1,003,355 training examples were created, out of which 50.2% were labeled as negative (INCONSISTENT) and the remaining 49.8% were labeled as positive (CONSISTENT). Models were evaluated in two settings: 1) with summaries from models trained on the CNN/DailyMail (Nallapati et al., 2016) dataset, which contains longer and more extractive reference summaries, and 2) with summaries from models trained on the XSum (Narayan et al.,
Evaluating the Factual Consistency of Abstractive Text Summarization
(CNN) Blues legend B.B. King was hospitalized for dehydration, though the ailment didn't keep him out for long. King's dehydration was caused by his Type II diabetes, but he "is much better," his daughter, Claudette King, told the Los Angeles Times. The legendary guitarist and vocalist released a statement thanking those who have expressed their concerns. "I'm feeling much better and am leaving the hospital today," King said in a message Tuesday. Angela Moore, a publicist for Claudette King, said later in the day that he was back home resting and enjoying time with his grandchildren. "He was struggling before, and he is a trouper," Moore said. "He wasn't going to let his fans down." (...)
Evaluating the Factual Consistency of Abstractive Text Summarization
Angela Moore was back home resting and enjoying time with his grandchildren.  
Evaluating the Factual Consistency of Abstractive Text Summarization
Models were implemented using the Transformers library (Wolf et al., 2019) written in Py-Torch (Paszke et al., 2017). Models were trained for 10 epochs using batch size of 12 examples and learning rate of 2e-5. Experiments were conducted on 8 Nvidia V100 GPUs, training took 23 hours on average. Best model checkpoints were chosen based on the performance on the validation set, final model performance was evaluated on the test set.
Evaluating the Factual Consistency of Abstractive Text Summarization
To understand whether datasets for related tasks transfer to the task of verifying factual consistency of summarization models, we trained factual consistency checking models on the MNLI entailment data (Williams et al., 2018) and FEVER fact-checking data (Thorne et al., 2018). For fair comparison, before training, we removed examples assigned to the neutral class from both of the datasets. Table 3 shows the performance of trained models evaluated by means of class-balanced accuracy and F1 score. Both FactCC and FactCCX models substantially outperform classifiers trained on the MNLI and FEVER datasets when evaluated on the CNN/DailyMail test set. However, on the more abstractive XSum data, results show a reverse trend with the MNLI model achieving highest performance. Considering that both MNLI and FEVER datasets contain abstractive, human-written claims, while our data generation pipeline lacks multi-sentence paraphrasing transformations, such outcome was expected. In the case of more extractive outputs, the results suggests priority of weak-supervision in-domain over strong-supervision in adjacent domains for factual consistency checking.
Evaluating the Factual Consistency of Abstractive Text Summarization
To compare our models with other NLI models for factual consistency checking, we conducted the sentence ranking experiment described by Falke et al. (2019) using the test data provided by the authors. In this experiment an article sentence is paired with two claim sentences, positive and negative. The goal is to see how often a model assigns a higher probability of being correct to the positive rather than the negative claim. Results are presented in Table 5. Despite being trained in a (document, sentence) setting, our model transfers well to the (sentence-sentence setting and outperforms all other NLI models, including BERT finetuned on the MNLI dataset. We were unable to replicate the summary re-ranking experiment because the experimental test setup does not rely on data that can be reused.
Evaluating the Factual Consistency of Abstractive Text Summarization
Considering that strongly supervised data is not available for factual consistency verification, proxy datasets must be used to train automatic verification models. Empirical results presented in this section suggest that when verifying less abstractive domains, it is more beneficial to train on weakly-supervised, but in-domain data, rather than to rely on the models ability to transfer knowledge from strongly supervised datasets in related domains. The experiments also highlight the necessity of extending the data generation pipeline with more abstractive, multi-sentence paraphrasing transformations as part of future work. In addition to improved performance, using synthetic data allows to train models with explainable components, such as FactCCX. Examples of span selections generated by the model are show in Table 4. The test set consists of model-generated summaries that do not have annotations for quantifying the quality of spans returned by FactCCX. Instead, span quality is measured through human evaluation and discussed in Section 5.
Evaluating the Factual Consistency of Abstractive Text Summarization
To further understand performance of our proposed models, we conducted human-based experiments and manually inspected model outputs.
Evaluating the Factual Consistency of Abstractive Text Summarization
Experiments using human annotators demonstrated that the span highlights returned by FactCCX are useful tools for researchers and crowdsource workers manually assessing the factual consistency of summaries. For each experiment, examples were annotated by 3 human judges selected from English-speaking countries. These experiments used 100 examples sampled from the manually annotated CNN/DM test set. Data points were sampled to ensure an equal split between CONSISTENT and INCONSISTENT examples.
Evaluating the Factual Consistency of Abstractive Text Summarization
To establish whether model-generated highlighted spans in the article and claim are helpful for the task of factual consistency checking, we hired human annotators to complete the mentioned task. Each of the verified document-sentence pairs was augmented with the highlighted spans output by FactCCX. Judges were asked to evaluate the correctness of the claim and instructed to use the provided segment highlights as suggestions. After the annotation task, judges were asked whether the highlighted spans were helpful for solving the task. The helpfulness of article and claim highlights were evaluated separately. The left part of Table 6 presents the results of the survey. A combined number of 91.75% annotators found the article highlights at least somewhat helpful, 81.33% of annotators found the claim highlights at least somewhat helpful. To ensure that low-quality judges do not bias the scores, we applied different data filters to the annotations: Raw Data considered all submitted annotations, Golden Aligned only considered annotations where the annotatorassigned label aligned with the author-assigned label for the example, Majority Aligned only considered examples where the annotator-assigned aligned with the majority-vote label assigned for the example by all judges. As shown in Table 6, filtering the annotations does not yield substantial changes in the helpfulness assessment.
Evaluating the Factual Consistency of Abstractive Text Summarization
Despite instructing the annotators to consider the provided highlights only as a suggestion when solving the underlying task, the annotators perception of the task could have been biased by the model-highlighted spans. To check how well the model-generated span highlights align with an unbiased human judgement, we repeated the previous experiment with only one change: modelgenerated highlights were not displayed to the annotators. The annotators were asked to solve the underlying task and highlight the spans of the source and claim that they found adequate. Using the annotations provided by the judges, we computed the overlap between the model-generated spans and unbiased human spans. Results are shown in the right part of Table 6. The overlap between spans was evaluated using two metrics. Accuracy is based on a binary score of whether the entire model-generated span was contained within the human-selected span. F1 score is computed between the tokens of the model-generated span and the tokens of the human-selected span. Results show 65.33% and 65.66% accuracy and 0.6207 and 0.6650 F1 for the article and claim highlights, respectively. We again applied different data filters to understand how the quality of annotations affects the score. We found that in this case, accuracy and F1 score were higher in the Majority Aligned than in the case of using Raw Data, and performance increases dramatically in the Majority Aligned. This suggests that when modelgenerated highlights are not provided, the task is less constrained and requires more careful preci-  sion on the part of judges.
Evaluating the Factual Consistency of Abstractive Text Summarization
To further understand the affects of providing model-generated highlights to annotators, we ran two factual consistency annotation tasks designed to test annotation efficiency. In the first, highlights were provided to the annotators. In the second, annotators did not receive highlights. In both, we measured the average time spent by an annotator on the task and the inter-annotator agreement of annotations. Results are shown in Table 7. When completing the task with highlights, annotators were 21% faster, and the inter-annotator agreement, measured with Fleiss' κ, increased by 38%.
Evaluating the Factual Consistency of Abstractive Text Summarization
Crowdsourcing experiments support the hypothesis that model-generated highlights from FactCCX can play a valuable role in supporting human-based factual consistency checking.
Evaluating the Factual Consistency of Abstractive Text Summarization
To better understand the limitations of our proposed approach, we manually inspected examples that were misclassified by our models. The majority of errors were related to commonsense mistakes made by summarization models. Humans can easily spot such errors, but they are difficult to capture with transformations necessary to generate data for weak supervision.
Evaluating the Factual Consistency of Abstractive Text Summarization
Our analysis also showed that the proposed models fail to correctly classify examples where the verified claim is highly abstractive. This is especially true when the claim finds support in multiple spans distant from each other in the source document, as mostly found in the XSum dataset. Additionally, the current set of transformations do not adequately capture temporal inconsistencies or Inter-annotator agreement (κ) 0.1571 0.2526 Table 7: Annotation speed and inter-annotator agreement measured for factual consistency checking with and without assisting, model generated highlights.
Evaluating the Factual Consistency of Abstractive Text Summarization
incorrect coreference. Nonetheless, the current transformations yield models already useful to humans by their own judgment; this analysis points toward key areas for future work. Correct and incorrect model predictions are presented in Appendix A.
Evaluating the Factual Consistency of Abstractive Text Summarization
We introduced a novel approach for factual consistency checking of summaries generated by abstractive neural models. In our approach, models are trained to perform factual consistency checking on the document-sentence level, which allows them to handle a broader range of errors in comparison to previously proposed sentencesentence approaches. Models are trained using artificially generated, weakly-supervised data created based on insights coming from the analysis of errors made by state-of-the-art summarization models. Quantitative studies showed that on less abstractive domains, such as CNN/DailyMail news articles, our proposed approach outperforms other models trained on existing textual entailment and fact-checking data, motivating our use of weak-supervision over transfer learning from related domains. Experiments with human annotators showed that our proposed approach, including an explainable factual consistency checking model, can be a valuable tool for assisting humans in factual consistency checking. Shortcomings of our approach explained in Section 5.2 can serve as guidelines for future work. We hope that this work will encourage continued research into factual consistency checking of abstractive summarization models.  Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1906-1919. Association for Computational Linguistics.
Evaluating the Factual Consistency of Abstractive Text Summarization
Example predictions made by the FactCC model. (cnn) james best, best known for his portrayal of bumbling sheriff rosco p. coltrane on tv's "the dukes of hazzard," died monday after a brief illness. he was 88. best died in hospice in hickory, north carolina, of complications from pneumonia, said steve latshaw, a longtime friend and hollywood colleague. although he'd been a busy actor for decades in theater and in hollywood, best didn't become famous until 1979, when "the dukes of hazzard's" cornpone charms began beaming into millions of american homes almost every friday night. for seven seasons, best's rosco p. coltrane chased the moonshine-running duke boys back and forth across the back roads of fictitious hazzard county, georgia, although his "hot pursuit" usually ended with him crashing his patrol car. although rosco was slow-witted and corrupt, best gave him a childlike enthusiasm that got laughs and made him endearing. his character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff 'em and stuff 'em!" upon making an arrest. among the most popular shows on tv in the early '80s, "the dukes of hazzard" ran until 1985 and spawned tv movies, an animated series and video games. several of best's "hazzard" co-stars paid tribute to the late actor on social media. (...)
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim "hazzard" ran from 1979 to 1985 and was among the most popular shows on tv.
Evaluating the Factual Consistency of Abstractive Text Summarization
Label: CONSISTENT Prediction: INCONSISTENT Article (CNN/DM) (cnn) the attorney for a suburban new york cardiologist charged in what authorities say was a failed scheme to have another physician hurt or killed is calling the allegations against his client "completely unsubstantiated." appearing Saturday morning on cnn's "new day," randy zelin defended his client, dr. anthony moschetto, who faces criminal solicitation, conspiracy, burglary, arson, criminal prescription sale and weapons charges in connection to what prosecutors called a plot to take out a rival doctor on long island. "none of anything in this case has any evidentiary value," zelin told cnn's christi paul. "it doesn't matter what anyone says, he is presumed to be innocent." moschetto, 54, pleaded not guilty to all charges wednesday. he was released after posting $2 million bond and surrendering his passport. zelin said that his next move is to get dr. moshetto back to work. "he's got patients to see. This man, while he was in a detention cell, the only thing that he cared about were his patients. and amazingly, his patients were flooding the office with calls, making sure that he was ok," zelin said. (...) Claim a lawyer for dr. anthony moschetto says the charges against him are baseless.
Evaluating the Factual Consistency of Abstractive Text Summarization
Label: INCONSISTENT Prediction: CONSISTENT Article (CNN/DM) (cnn) north korea accused mexico of illegally holding one of its cargo ships wednesday and demanded the release of the vessel and crew. the ship, the mu du bong, was detained after it ran aground off the coast of mexico in july. mexico defended the move wednesday, saying it followed proper protocol because the company that owns the ship, north korea's ocean maritime management company, has skirted united nations sanctions. "because the company has avoided the sanctions imposed by the u.n. security council, the mexican government is acting on the basis of its international obligations as a responsible u.n. member state," the permanent mission of mexico to the united nations said. the security council blacklisted ocean maritime management in july, saying it "played a key role in arranging the shipment of concealed arms and related materiel" on another ship, the chong chon gang, which was detained by panama in 2013. but an myong hun, north korea's deputy ambassador to the united nations, said there was no reason to hold the mu du bong and accused mexico of violating the crew members' human rights by keeping them from their families.
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim north korea accused mexico of using one of its cargo ships.
Evaluating the Factual Consistency of Abstractive Text Summarization
Label: CONSISTENT Prediction: CONSISTENT Article (CNN/DM) (cnn) spoiler alert! it's not just women getting cloned. that was the big twist at the end of "orphan black's" second season. the kickoff to the new season leads the list of six things to watch in the week ahead. 1. "orphan black," 9 p.m. et, saturday, april 18, bbc america. the cloning cult sci-fi series remains one of the most critically acclaimed shows on tv, thanks in large part to the performance of tatiana maslany, who has taken on at least six roles on the show so far, including a newly introduced transgender clone. maslany told reporters this week that we can expect even more impressive scenes with multiple clones. "we like to push the boundaries of what we're able to do and the limits of those clone scenes," she said. (...)
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim critically acclaimed series "orphan black" returns. boston (cnn) dzhokhar tsarnaev's bombs tore through their bodies: singeing flesh, shattering bones, shredding muscles and severing limbs. but on tuesday, jurors also began to hear about the holes his bombs left in the hearts of the survivors and the families of the dead. now that he has been found guilty on every count, the jury must decide whether boston marathon bomber tsarnaev, 21, should live or die for what he has done. this is the victim impact part of the case, and the testimony was heartbreaking. four young people are gone, and grief fills the spaces they once occupied. a father with a shock of white hair cried for the daughter he called "princess." "krystle was the light of my life. she was extremely smart, hardworking, beautiful, every father's dream. i miss her a lot," said william a. campbell sr., dabbing at his eyes as he described his daughter, a 29-yearold restaurant manager who was killed in the first blast at the 2013 boston marathon. she was the one who could round up the family and put on big celebrations, he said. (the hollywood reporter) the author of a 2006 novel has accused the "avengers" director and "cabin" director drew goddard of stealing his idea. with just weeks until his box-office victory lap for "avengers: age of ultron," joss whedon is now facing a lawsuit accusing him of stealing the idea for the 2012 meta-horror movie the cabin in the woods. whedon produced and co-wrote the script for cabin with director drew goddard, a writer on whedon's "buffy the vampire slayer" and a fanboy favorite in his own right, with credits that include netflix's "daredevil" (and reportedly may soon include sony's upcoming spider-man projects). whedon and goddard are named as defendants, along with lionsgate and whedon's mutant enemy production company, in the complaint filed monday in california federal court. joss whedon slams 'jurassic world 'clip as "'70s -era sexist" in the complaint, peter gallagher (no, not that peter gallagher) claims whedon and goddard took the idea for "the cabin in the woods" from his 2006 novel "the little white trip: a night in the pines." he's suing for copyright infringement and wants $10 million in damages. gallagher is basing his claim on the works' similar premises: both feature a group of young people terrorized by monsters while staying at a cabin in what is revealed to be (spoiler alert) a horror-film scenario designed by mysterious operators. (...)
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim joss whedon claims whedon and goddard took the idea for "the cabin in the woods". More than 5,300 bottles of alcohol were seized by the investigators in the southern city of Liuzhou. They also found packets of a white powder called Sildenafil, better known as the anti-impotence drug Viagra. Police in the Guangxi region are now investigating the two distillers. The Liuzhou Food and Drug Administration said (in Chinese) that the powder was added to three different types of 'baijiu' -a strong, clear spirit that is the most popular drink in China. They said the haul was worth up to 700,000 yuan. Doctors recommend that adults requiring prescription should take only one dose of Viagra a day, with a lower dose for those over the age of 65. China continues to face widespread food safety problems. In June, police in cities across China seized more than 100,000 tonnes of smuggled meat, some of which was more than 40 years old. The 2008 tainted milk scandal outraged the nation. Some 300,000 people were affected and at least six babies died after consuming milk adulterated with melamine.
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim police in southern china have seized more than 1,000 alcohol bottles and seized more than 1,200 bottles of contaminated milk, local media report.
Evaluating the Factual Consistency of Abstractive Text Summarization
Label: CONSISTENT Prediction: CONSISTENT
Evaluating the Factual Consistency of Abstractive Text Summarization
The victim was queuing for food at the branch in St George's Street, Canterbury at about 02:15 GMT on Friday when the assault occurred. Investigating officers said three men entered the restaurant and began being noisy and bumping into people. It is believed one of the group then set light to the woman's hair. Officers have released CCTV images of three men they are keen to speak to regarding the attack. Det Sgt Barry Carr said: "Fortunately the fire was put out quickly and the victim was not seriously hurt, but things could clearly have turned out much worse.This was a nasty and extremely dangerous thing to do, and I urge anyone who recognises the men in the CCTV images to contact me as soon as possible."
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim a woman was assaulted and assaulted in a mcdonald's restaurant in kent, police have said. Jung won aboard Sam, who was a late replacement when Fischertakinou contracted an infection in July. France's Astier Nicolas took silver and American Phillip Dutton won bronze as GB's William Fox-Pitt finished 12th. Fox-Pitt, 47, was competing just 10 months after being placed in an induced coma following a fall. The three-time Olympic medallist, aboard Chilli Morning, produced a faultless performance in Tuesday's final show-jumping phase. But the former world number one's medal bid had already been ruined by a disappointing performance in the cross-country phase on Monday. He led after the dressage phase, but dropped to 21st after incurring several time penalties in the cross country. Ireland's Jonty Evans finished ninth on Cooley Rorkes Drift. Why not come along, meet and ride Henry the mechanical horse at some of the Official Team GB fan parks during the Rio Olympics? Find out how to get into equestrian with our special guide. Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Evaluating the Factual Consistency of Abstractive Text Summarization
Claim great britain's eventers missed out on a bronze medal at the rio olympics after losing in the dressage.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Abstractive summarization models must aggregate salient content from the source document(s) and remain faithful, i.e. being factually consistent with information in the source documents. Neural abstractive models are effective at identifying salient content and producing fluent summaries (See et al., 2017;Chen and Bansal, 2018;Gehrmann et al., 2018). However, the generated summary may not always contain faithful information, which is vital for real-world applications. * Most of the work is done while the authors were at Amazon Web Services AI. 1 Faithfulness Evaluation with Question Answering.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Source. The world's oldest person has died a few weeks after celebrating her 117th birthday. Born on March 5, 1898, the greatgrandmother had lived through two world wars, the invention of the television and the first successful powered aeroplane flight by the wright brothers... Output sentence. The world 's oldest person has died on March 5, 1898.  Table 1 shows an example of unfaithful generation. Recent studies have shown that around 30% of generated summaries contain unfaithful information (Cao et al., 2018;Falke et al., 2019a;Kryściński et al., 2019), especially when the sentence combines content from multiple source sentences (Lebanoff et al., 2019). In this paper, we address the problem of evaluating faithfulness of generated summaries given their source documents. Our key insight is that current models are limited by a trade-off between abstractiveness and faithfulness (Section 2). On a wide range of systems and two datasets with varying levels of abstractiveness (CNN/DM and XSum), we show that the number of unfaithful sentences (annotated by humans) increases as the summary becomes more abstractive (i.e. less overlap with the source document). Next, we investigate a diverse set of existing automatic evaluation metrics such as ROUGE, BERTScore (Zhang et al., 2019a), and learned entailment models. We find that their correlations with human scores of faithfulness drop significantly on highly abstractive summaries, where deeper text understanding beyond surface similarity is needed.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Recently, question answering (QA) based automatic metrics have been proposed for evaluating content selection in summarization (Eyal et al., 2019;Scialom et al., 2019;Chen et al., 2018). Specifically, cloze-style QA is used to evaluate whether important information in the source is recovered from the summary. Inspired by prior work, we use automatically generated QA pairs to represent information in the summary and validate it against the source. Concretely, we generate a set of "groundtruth" QA pairs from the summary, using a learned model that converts a declarative sentence and an answer span to a question (Section 3). Then, off-the-shelf reading comprehension models are evaluated on this set by extracting answer spans from the source documents. High accuracy means that the summary and the source document tend to produce the same answers, thus they are factually consistent with respect to the questions. Compared to prior approaches using cloze tests, our question generation approach enables evaluation with a broader range of QA models and answer types (e.g. extractive and generative), thus maximally taking advantage of progress in QA.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Among automatic metrics based on n-gram overlap, word embeddings, and language understanding models (relation extraction and entailment), FEQA has significantly higher correlation with human scores of faithfulness and is the only metric that correlates with human scores on highly abstractive summaries from XSum.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
While extractive summarizers are largely faithful (since they copy sentences from the source document), current abstractive models struggle to produce faithful summaries without copying. Similar to Lebanoff et al. (2019), we observe that factual errors occur more frequently as models generate more abstractive summary sentences, i.e. less overlap with the source document. In this section, we analyze generated summaries along two dimensions: abstractiveness and faithfulness. Specifically, we aim to answer the following questions: (1) How to quantify abstractiveness of a summary? (2) Is abstractiveness encouraged more by the data or the model? (3) How does being abstractive affect faithfulness?
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Abstractive summarization involves rephrasing important content into brief statements, ranging from minor editing of a source sentence to condensing multiple sentences in new words. Given a source document and a summary, we want to measure the level of abstractiveness of the summary. Prior work measures abstractiveness by overlapped text spans between the summary and the document (Grusky et al., 2018;Zhang et al., 2018), or indirectly by the effectiveness of extractive baselines such as LEAD-3 (Nallapati et al., 2016a). While metrics such as extractive fragment coverage and density (Grusky et al., 2018) provide a continuous measure of the level of abstractiveness, we define a more fine-grained categorization of abstractiveness by analyzing how each sentence in the summary is formed.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A more abstractive summary sentence aggregates content over a larger chunk of source text; consequently it must copy fewer words to maintain brevity. Therefore, we define the following abstractiveness types based on the amount of copying, e.g. copying a source sentence, one or more partial fragments from the source sentence, and individual words.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
1. Sentence extraction: the summary sentence is exactly the same as one of the source sentences.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
2. Span extraction: the summary sentence is a substring of one of the source sentences, e.g. "the plane was coming back from the NCAA final" is a span extracted from "the plane was coming back from the NCAA final, according to spokesman John Twork".
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
3. Word extraction: the summary sentence is formed by a subset of the tokens in a source sentence, e.g. "Capybara Joejoe has almost 60,000 followers" is a result of deleting words in "Capybara Joejoe who lives in Las Vegas has almost 60,000 followers on Instagram".
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
4. Perfect fusion k : the summary sentence is constructed by piecing together the substrings from k (k > 1) source sentences in their original order, e.g. "Capybara Joejoe has almost 60,000 followers" is a perfect fusion of the sentences "Capybara Joejoe lives in Las vegas." and "He has almost 60,000 followers on Instagram."
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
To quantify the amount of abstractiveness of a set of summaries, we label each sentence with the first qualified type in the order above if it fits to one of these categories.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
We then define the score of each type as the percentage of sentences labeled by that category. The types are ordered by increasing levels of abstractiveness. For example, a summary with higher fusion scores and lower extraction scores is considered more abstractive. In addition, we compute the percentage of novel n-grams that do not appear in the source document as another metric for abstractiveness.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Equipped with the metrics for abstractiveness above, we want to further understand how abstractive the generated summaries are, and whether the amount of abstractiveness is a result of the training data or the model. Therefore, we compute abstractiveness scores for both the reference summaries and summaries generated from a diverse set of models on two datasets. Models. Most neural abstractive summarization models are based on sequence-to-sequence models. They differ in how summarization-specific operations such as copying/extraction are instantiated. We consider 5 prominent models and sum-   (See et al., 2017) uses the copy mechanism during decoding to allow extraction. FASTRL (Chen and Bansal, 2018) and BOTTOMUP (Gehrmann et al., 2018) decouple extraction and abstractive generation by learning to select sentences and words respectively in the first step; this model has been shown to generate more abstractive summaries compared to PGC. TCONV (Narayan et al., 2018) is initially designed for XSum, thus it does not include any explicit copying/extraction components and focuses on long text representation using convolutional neural networks. BERTSUM (Liu and Lapata, 2019) consists of a BERT-based encoder and a 6-layer Transformer decoder. It incorporates extraction implicitly by first fine-tuning the encoder on the extractive summarization task. 3
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Results. Our goal is to understand the level of abstractiveness of summaries generated by different models, and the influence on abstractiveness from the training data. Therefore, we analyzed summaries generated by the above models on CNN/DM and XSum. We computed the metrics described in Section 2.1 for both the generated summaries and the reference summaries on the test sets. The results are shown in Table 3.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
First, CNN/DM is more extractive than XSum. Extraction scores of the reference summaries in CNN/DM shows that almost half of the sentences are formed by deleting words in one of the source sentences. This shows that sentence compression (Knight and Marcu, 2002) is the main technique used for this dataset. In contrast, none of the summary sentences in XSum are formed by copying from a single source sentence. They are generated mostly by paraphrasing the input content, indicated by the large fraction of novel n-grams.  Table 3: Abstractiveness measures of the models on CNN/DM and XSum datasets. The numbers for Extraction and Perfect fusion indicate % of sentences generated with these strategies. Numbers for novel n-grams indicate % of n-grams that are present in the output sentence but is not present in the source.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Second, training data has a larger influence on the abstractiveness of model outputs. Similar to Zhang et al. (2018), we find that models trained on CNN/DM are near-extractive. However, the same models trained on XSum are significantly more abstractive. In fact, none of the models produced any sentence that copies words/phrases from a single source sentence, which is consistent with characteristics of the reference summaries in XSum. The content is more often rephrased in novel words/phrases. However, on both datasets, current models struggle to achieve the same level of abstractiveness as the reference summaries, indicating that additional inductive bias is needed to condense multiple sentences by rephrasing.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Third, different models have different ways of doing extraction. When trained on CNN/DM, PGC generates the majority of sentences by copying complete source sentences, whereas FASTRL, BOTTOMUP and BERTSUM do simple compression by deletion more often. In addition, BOT-TOMUP does more fusion compared to PGC, FAS-TRL and BERTSUM.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
To understand faithfulness of current systems and its relation to abstractiveness, we crowd-sourced human annotations on the output of each modeldataset pair described in Section 2.2. Since a nearextractive sentence is very likely to be grammatical and faithful, we focus on more abstractive cases by excluding output sentences that are either an exact copy or a substring of one of the source sentences.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A key challenge to reliable human annotation is that the inter-annotator agreement on faithfulness is relatively low (Lebanoff et al., 2019). Our pi- 4 We make our data and code available for reproducibility at: https://github.com/esdurmus/summary-faithfulness. lot study shows that workers often do not agree on incoherent sentences, e.g. whether "Chelsea beat Chelsea 5 − 3 in the Premier League on Saturday." is faithful or not. To standardize the annotation process, we design hierarchical questions to distinguish among failed generation that render a sentence meaningless, low-level grammatical errors that hardly affect semantic understanding, and faithfulness errors that convey incorrect (yet meaningful) information. Figure 1 shows the decision tree of our human annotation steps. We first evaluate the grammaticality of generated sentences (independent from the source document). We show annotators a summary sentence and ask them to choose whether the given sentence is meaningful or nonsensical to determine if the given sentence is structurally and semantically sound. If the annotator can make sense of the sentence, we then ask whether it is grammatical or has minor grammaticality problems which a person can easily correct.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Next, for sentences labeled as meaningful in the first step, we ask workers whether they are faithful to the provided source document. In case the worker labels a sentence as unfaithful, we conduct a simple error analysis by asking them to indicate if the sentence contains information that is absent from or conflicting with the source document, which corresponds to hallucination and contradiction errors, respectively. More details about the annotation schema and guidelines are included in the Appendix C. Next, we describe our human evaluation results.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
For each dataset-model pair described in Section 2.2, we randomly sampled 1000 sentencesource pairs eliminating output sentences that are either an exact copy or substring of a source sen-S1:
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Chelsea and Manchester City are interested in signing Chelsea.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A man has died after his car left the road and hit a tree in Surrey, police said.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The man, in his 20s, was the only person in the BMW convertible, when the accident happened on the Aldershot road in Guildford. He was traveling east when his car left the road. Police closed the road while investigators were at the scene.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Is it grammatical?
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Is it faithful? Contradiction or Hallucination?  Figure 1: The decision diagram of our human annotation process. Decision nodes are rectangular and outcome nodes are circular. We show the annotation path of two summary sentences, S1 (green arrows) and S2 (red arrows). S2 is annotated as nonsensical thus is not considered for faithfulness. S1 is annotated as unfaithful due to hallucinated content.  Table 4: Grammaticality and faithfulness results of human annotations. Score is computed by taking the percentage of annotators that selected "meaningful" and "faithful" for grammaticality and faithfulness annotation tasks, respectively, and then averaging these values across all the examples for the given annotation task. Agreement is computed by taking the percentage of the workers that annotate the majority class for the given example. Abstractiveness is measured by the percentage of novel trigrams in a given sentence.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
tence. We collected grammaticality annotations for these sentences from 5 annotators. We consider a sentence meaningful if at least 4 out of 5 annotators label it as meaningful in the first stage. We sampled 200 meaningful sentences randomly to collect annotations for faithfulness. Table 4 shows the results of the grammaticality and faithfulness human evaluations.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Grammaticality. Overall, outputs from all models are scored high on grammaticality with high inter-annotator agreement. However, on more abstractive summaries (i.e. when trained on XSum), the grammaticality scores drop significantly. One exception is BERTSUM, which maintains good performance on XSum and achieves the highest grammaticality score on both datasets. 5
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Faithfulness. Near-extractive summaries generated from models trained on CNN/DM have significantly higher faithfulness scores than highly abstractive summaries from models trained on XSum. We find that PGC and TCONV has faithfulness errors in more than half of the sentences they generate when trained on XSum. Although BERTSUM generates fewer unfaithful sentences, it still suffers from performance drop on XSum. Interestingly, human agreement on faithfulness is also lower for abstractive summaries from XSum. This suggests that faithfulness errors are harder to catch for humans as well in more abstractive settings. We further observe conflicting information is more common among models trained on CNN/DM while hallucination is more common among models trained on XSum. Table 5 shows examples of meaningful but unfaithful sentences.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Our analysis above shows that the number of unfaithful sentences increases significantly as more abstractive summaries are generated. Thus the key challenge to faithfulness evaluation is to verify highly abstractive sentences against the source document, where surface similarity match-   Table 11.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The home was built for inspection.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The home was built for [MASK].
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
[MASK] was built for inspection.  Figure 2: Overview of FEQA. Given a summary sentence and its corresponding source document, we first mask important text spans (e.g. noun phrases, entities) in the summary. Then, we consider each span as the "gold" answer and generate its corresponding question using a learned model. Lastly, a QA model finds answers to these questions in the documents; its performance (e.g. F1 score) against the "gold" answers from the summary is taken as the faithfulness score.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
ing would fail. If we have a good semantic representation of the sentence abstracting away its surface form (e.g. a list of facts about who did what to whom), we can simply compare the sentence representation to the document representation (e.g. check whether the fact list from the summary is a subset of the list from the document). Ideally, the representation should be domain-general and interpretable for easy error analysis.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Motivated by the fast progress in reading comprehension (Chen, 2018;Gao et al., 2018) we propose to use QA pairs as a generic meaning representation of sentences for faithfulness evaluation. Given a summary sentence, we produce a list of questions asking about key information in the sentence and their corresponding answers. To verify this information against the source, we use a QA model to predict answers from the document. The questions and the QA model thus extract comparable information from two pieces of text. More matched answers from the document implies a more faithful summary since the information addressing these questions are consistent between the summary and the source document. Figure 2 shows the workflow of FEQA.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Question generation. Prior work (Eyal et al., 2019;Scialom et al., 2019) uses cloze tests as questions by masking entities. To go beyond cloze-style QA and leverage more recent extractive (Rajpurkar et al., 2016) or even generative (Alec et al., 2019) QA models, we generate natural language questions from the summary sentence automatically. Specifically, we mask important text spans in a sentence, including noun phrases extracted by a constituency parser (Kitaev and Klein, 2018) and named entities extracted by the Stanford CoreNLP NER model (Finkel et al., 2005;Manning et al., 2014). We consider each span as the gold answer and generate its corresponding question by fine-tuning a pretrained BART language model (Lewis et al., 2019). To train the question generator, we adapt the QA2D dataset Demszky et al. (2018). The input is a declarative sentence with masked answers and the output is a question. A training example might look like:
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Input: Sally was born in <m> 1958 </m> Output: When was Sally born ?
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Since the transformation from declarative sen-tences to questions is almost rule-based without much paraphrasing, we expect the model to generalize to various domains.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Answer verification. Given the QA pairs generated from a summary sentence, we run off-theshelf QA models to get answers to these questions from the source document. We then measure the average F1 score against the "gold" answers from the summary, which is our faithfulness score for the given sentence. This step does not have any constraint on the QA model. We experiment with the pretrained BERT-base model (Devlin et al., 2019) fine-tuned on SQuAD-1.1 (Rajpurkar et al., 2016) and SQuAD-2.0 (Rajpurkar et al., 2018). Note that in the case of SQuAD-2.0, the model may be able to hypothesize that a question is unanswerable. This case is equivalent to getting an answer incorrect (i.e. unfaithful).
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
We aim to understand to what extent the proposed QA-based metric and existing metrics capture faithfulness of a summary. Given pairs of documents and summary sentences without reference summaries, we measure correlations between human-annotated faithfulness scores (Section 2.3) and scores computed using each metric described below.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Word overlap-based metrics. A straightforward metric for faithfulness is the word overlap between the summary sentence and the document. We compute ROUGE (R), BLEU (B), 6 between the output sentence and each of the source sentences (i.e. taking the source sentence as the reference). We then take the average scores and maximum score across all the source sentences. Since according to our analysis taking the average score consistently has higher correlation, we report only the correlation for the average.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Embedding-based metrics. Word embeddings extend word overlap-based metrics beyond exact match. Recently, BERTScore (Zhang et al., 2019b) was proposed to compute the similarity between two sentences using contextual word embeddings from BERT. It has higher correlation with human judgements on image captioning and machine translation than word overlap based metrics. We compute BERTScore (BERTSc) between each source sentence and the summary sentence. 7 To get the final score, we experiment with both the average and the maximum scores computed from each source sentence and the summary sentence. We report results using the maximum score since it has better performance.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Model-based metrics. In addition to QA, recent work has used relation extraction and textual entailment models for faithfulness evaluation ( 
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Metric Comparison. We first compute scores for each metric on document and output sentence pairs on both CNN/DM and XSum datasets (748 and 286 pairs respectively). We then compute Pearson and Spearman correlation coefficients between scores given by each metric and humanannotated scores. Table 7 includes correlation coefficients for the examples from CNN/DM and XSum, respectively. We observe that for both CNN/DM and XSum, the score of QA-based evaluation has a higher correlation with faithfulness than other metrics. Although word-overlap based metrics are correlated with the faithfulness in more extractive settings (i.e. for CNN/DM), these metrics have no correlation with faithfulness in more abstractive settings (i.e. for XSum). We further notice that all the metrics have significantly lower correlation with human scores for XSum, suggesting that evaluating faithfulness is more difficult in highly abstractive settings; deeper understanding of the source and the summary sentence is necessary here.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Consistent with the findings of Falke et al. (2019b), the entailment metric does not have a significant correlation with faithfulness in most cases. These models fail to distinguish entailed (faithful) Source Sentence Output Sentence Metric Score Health Inspectorate Wales said Wrexham Maelor Hospital staff were under "considerable pressure" for long periods as ambulances waited outside.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A hospital ward in Wrexham has been rated "inadequate" by inspectors after inspectors found patients at risk of harm.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The Black Poplar is one of the rarest native trees in the UK, with only 2,500 thought to be left.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Northern Ireland's first trees are among those recognised in the Welsh Architecture Trust's list of the year's best trees.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
BertScore 83.06%  and non-entailed (unfaithful) summary sentences when both overlap largely with the source document, because models trained on current entailment datasets may rely on simple heuristics such as lexical overlap (McCoy et al., 2019). Similarly, BERTScore tends to give higher scores when there are overlapping concepts between the sentences even though the content is not the same. See Table 6 for examples.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Content selection and faithfulness. Current evaluation metrics for summarization produce a single measure of the overall quality of the summary. Typically, the output summary is compared against the reference summary in terms of n-gram overlap. These metrics mainly evaluate content selection, i.e. whether the content of the output is similar to the content of the reference. In contrast, to evaluate faithfulness, we compare the output summary against the source document. One natural question that follows is whether high content matching sufficient for faithfulness. We compute the correlation coefficients between humanannotated faithfulness scores and ROUGE scores computed from the reference and the output sentence. As shown in  Table 8: Pearson (P) and Spearman (S) correlation between human-annotated faithfulness scores and ROUGE scores of content selection (computed between the reference and the output sentence). High content selection scores (typical ROUGE score for summarization) do not necessarily imply faithfulness of the summary. correlation between ROUGE scores of content selection and faithfulness on CNN/DM, the correlation is significantly lower than ROUGE scores of faithfulness (i.e. computed between the source and the output sentence). For XSum, there is no significant correlation between the content selection metrics and faithfulness. We provide unfaithful examples with high content selection scores in Appendix D.3. This suggests that content selection and faithfulness should be measured separately as opposed to using a unified score.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Analysis and limitations of QA-based evaluation. Table 9 shows examples for a faithful and an unfaithful output sentence and the corresponding QA pairs. Note that the QA system is able to capture common errors such as conflicting information in the output sentence. To measure the reliability of FEQA, we further perform a manual error analysis using 100 randomly sampled QA pairs. We observe that around 94% of generated questions are mostly grammatical and correct given the mask. For 78% of the questions, the QA system has the correct behaviour: it answers the question correctly if the sentence is faithful to the article, otherwise it produces "unanswerable" or an incorrect answer. Majority of the errors of the QA system are because it either didn't detect unanswerable questions or produces "unanswerable" when there exists an answer (14%). More-  Table 9: Examples detection results from FEQA. OA:Output Answer, SA:Source Answer. The output sentence in the first example is unfaithful, whereas the one for the second example is faithful. Bold text indicates the span that was masked to generate the question.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
over, when the article is long, QA system tends to make more mistakes. Especially for more abstractive settings, F1-score penalizes the correct answers when the answer from the article does not exactly match with the gold answer (i.e. "Donald Trump" vs. "the President of the United States Donald Trump") (16%).
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
We investigate the faithfulness problem in neural abstractive summarization and propose a QAbased metric for evaluating summary faithfulness. We show that current models suffer from an inherent trade-off between abstractiveness and faithfulness. They are good at copying important source content, but tend to concatenate unrelated spans and hallucinate details when generating more abstractive sentences. A new inductive bias or additional supervision is needed for learning reliable models. While our QA-based metric correlates better with human judgment and is useful for model development, it is limited by the quality of the QA model. The final evaluation should still rely on human annotation or human-in-the-loop methods (Chaganty et al., 2018).  
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The characteristics of each model used in our experiments are detailed below.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Pointer Generator Model with Coverage (PGC) (See et al., 2017) uses the copy mechanism (Vinyals et al., 2015) to allow copying words from the source. The adapted coverage mechanism (Tu et al., 2016) is incorporated to alleviate repetition by keeping track of source words that have been summarized. This copy mechanism is widely adopted by subsequent models.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Fast Abstractive Summarization with Reinforce (FASTRL) (Chen and Bansal, 2018) first uses an extractor agent to select salient sentences from the document, then condenses the extracted sentences using the Pointer-Generator summarizer.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Bottom-up Summarization Model (BOTTOMUP) (Gehrmann et al., 2018) first selects words from the source document that are likely to appear in the summary, then generates using the Pointer-Generator model, where the copying mechanism is constrained to the previously selected words. It improves upon PGC by explicitly learning the selector to avoid copying long text spans.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Topic-aware Convolutional Sequence-to-Sequence model (TCONVS2S) (Narayan et al., 2018) is a convolutional neural network-based model conditioned on the topics of the article. It is shown to be effective in capturing long-range dependencies in the documents.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
BERT-based model (BERTSUM) (Liu and Lapata, 2019) is a two-stage fine-tuning approach where the BERT-based encoder is first fine-tuned on the extractive summarization task and then on the abstractive sumarization task with the decoder (denoted as BERTSUMEXTABS in the original paper).
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
For grammaticality annotation, we present only the output sentence to the workers. We collect annotations from 5 workers for both of the tasks. For this task, given the output sentence, we provide workers the following guidelines:
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Output Sentence Category ...Although her due date has not officially been confirmed, the duchess of Cambridge told wellwishers at a charity event last month: I am due mid-April, to the end of April...
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The duchess of Cambridge told wellwishers at a charity event last month: "The duke's intention is to be at the commemorations". Carragher posted a picture on his son play in the famous youth tournament.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A body was found by a member of the public on private land near Leighton, about 10 miles (16.09km) away from the centre of Shrewsbury, on Monday. Mr Bebbington's family has been informed, West Mercia Police confirmed.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The death of a man whose body was found in a river in Cumbria has been identified as murder.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
The incident happened near Dr. Gray's hospital shortly after 10:00. The man was taken to the hospital with what police said were serious but not life-threatening injuries. The a96 was closed in the area for several hours, but it has since reopened.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
A man has been taken to hospital after he was hit by a lorry in Dumfries. 
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Output Sentence ... University of Nebraska researcher has revealed why stress is bad for you. Limited periods of stress are good, as they release cortisol... University of Nebraska researcher has revealed why stress is bad for you, stimulating your body to produce an important hormone called cortisol. ...Indian air force and Nepalese army medical team launch rescue mission to bring injured people to hospitals in Kathmandu. Forshani Tamang's family carried her for four hours to reach help after she was wounded when their home was destroyed... Indian air crew and Nepalese army medical team were killed in Nepal's Sindhupalchok quake. • A judge in Japan has ordered a judge to order a woman who has absconded from Japan to Japan. (generated by PGC for XSum)
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
• Stoke City moved up to third in the Premier League with victory over Stoke City at Stoke. (generated by TCONV for XSum)
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
• Johnny Depp's management group is suing his management group over his "lavish lifestyle". (generated by BERTSUM for XSum)
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
D.2 Examples for meaningful but unfaithful sentences Table 11 includes examples that are annotated as meaningful but unfaithful. First three examples are picked from the models trained on CNN/DM, and last three are from the models trained on XSum.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
We observe that majority of sentences with faithfulness errors for CNN/DM dataset are generated by incorrect concatenation (IC). The models fuse two sentences from the source and generate a new sentence that is not consistent with the context of the source. Within this category, however, the models make a wide-range of mistakes such as copying the wrong entity, date, and quote. For XSum, the faithfulness mistakes are mostly hallucinations. Models tend to hallucinate information (e.g. entities, events, date) that is not present in the source.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
overlap with reference that are unfaithful
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Although current summarization models are evaluated with respect to the content overlap between the reference and the output, these metrics do not necessarily provide any guarantees for the faithfulness of the output. Table 12 includes examples with similar content overlap scores as the faithful examples but are unfaithful. We can see that although the output sentences include similar words and refer to similar topics, they include hallucinations and inaccurate information.
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Since CNN/DM and XSum datasets are automatically crawled, we find that there is noise in the data. For example, source documents can include phrases such as "click here for the latest news". We further observe that reference can carry information that is not in the source document since some of these one sentence highlights are written using additional world knowledge. Table 13 shows an example where the reference is unfaithful since it includes information that is not in the source (i.e. the fact that Ms. Wood's first name is Leanne and she is Plaid Cymru leader.).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Recent summarization approaches based on pretrained language models (LM) have established a new level of performance (Zhang et al., 2020;Lewis et al., 2020), generating summaries that are grammatically fluent and capable of combining salient parts of the source document. However, current models suffer from a severe limitation, generating summaries that are not factually consistent, that is, the content of the summary does not meet the * Work done as an intern at Amazon Alexa AI. 1 Our code is publicly available at https://github. com/amazon-research/fact-graph  facts of the source document, an issue also known as hallucination. Previous studies (Cao et al., 2018;Falke et al., 2019;Maynez et al., 2020;Dreyer et al., 2021) report rates of hallucinations in generated summaries ranging from 30% to over 70%. In the face of such a challenge, recent works employ promising ideas such as question answering (QA) (Durmus et al., 2020;Nan et al., 2021) and weakly supervised approaches (Kryscinski et al., 2020) to assess factuality. Another line of work explores dependency arc entailment to improve the localization of subsentence-level errors within generated summaries (Goyal and Durrett, 2020). However, these methods have a reduced correlation with human judgments and may not capture well semantic errors (Pagnoni et al., 2021). One reason for the poor performance is the lack of good quality factuality training data. Second, it is challenging to properly encode core semantic content from the document and summary (Lee et al., 2021) and reason over salient pieces of information in order to assess the summary factuality. Third, previous work (DAE, Goyal and Durrett, 2021) treats semantic relations as isolated units, not simultaneously considering the entire semantic structure of both document and summary texts.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Recent summarization approaches based on pretrained language models (LM) have established a new level of performance (Zhang et al., 2020;Lewis et al., 2020), generating summaries that are grammatically fluent and capable of combining salient parts of the source document. However, current models suffer from a severe limitation, generating summaries that are not factually consistent, that is, the content of the summary does not meet the * Work done as an intern at Amazon Alexa AI. 1 Our code is publicly available at https://github. com/amazon-research/fact-graph  facts of the source document, an issue also known as hallucination. Previous studies (Cao et al., 2018;Falke et al., 2019;Maynez et al., 2020;Dreyer et al., 2021) report rates of hallucinations in generated summaries ranging from 30% to over 70%. In the face of such a challenge, recent works employ promising ideas such as question answering (QA) (Durmus et al., 2020;Nan et al., 2021) and weakly supervised approaches (Kryscinski et al., 2020) to assess factuality. Another line of work explores dependency arc entailment to improve the localization of subsentence-level errors within generated summaries (Goyal and Durrett, 2020). However, these methods have a reduced correlation with human judgments and may not capture well semantic errors (Pagnoni et al., 2021). One reason for the poor performance is the lack of good quality factuality training data. Second, it is challenging to properly encode core semantic content from the document and summary (Lee et al., 2021) and reason over salient pieces of information in order to assess the summary factuality. Third, previous work (DAE, Goyal and Durrett, 2021) treats semantic relations as isolated units, not simultaneously considering the entire semantic structure of both document and summary texts.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
To mitigate the above issues, we explore meaning representations (MR) as a form of content representation for factuality evaluation. We present FACTGRAPH, a novel graph-enhanced approach that incorporates core information from the document and the summary into the factuality model using graph-based MRs, which are more suitable for factuality evaluation: As shown in Figure 1, graphbased MRs capture semantic relations between entities, abstracting away from syntactic structure and producing a canonical representation of meaning.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Different from previous methods (Kryscinski et al., 2020;Goyal and Durrett, 2021), FACT-GRAPH is a dual approach which encodes both text and graph modalities, better integrating linguistic knowledge and structured semantic knowledge. As shown in Figure 2, it is composed of parameterefficient text and graph encoders which share the same pretrained model and differ by their adapter weights (Houlsby et al., 2019). The texts from the document and summary are encoded using the adapter-based text encoder whereas the entire semantic structures that represent document and summary facts are used as input to the graph encoder augmented structure-aware adapters (Ribeiro et al., 2021b). The representations of the two modalities thus are combined to generate the factuality score.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Different from previous methods (Kryscinski et al., 2020;Goyal and Durrett, 2021), FACT-GRAPH is a dual approach which encodes both text and graph modalities, better integrating linguistic knowledge and structured semantic knowledge. As shown in Figure 2, it is composed of parameterefficient text and graph encoders which share the same pretrained model and differ by their adapter weights (Houlsby et al., 2019). The texts from the document and summary are encoded using the adapter-based text encoder whereas the entire semantic structures that represent document and summary facts are used as input to the graph encoder augmented structure-aware adapters (Ribeiro et al., 2021b). The representations of the two modalities thus are combined to generate the factuality score.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In particular, we explore Abstract Meaning Representation (AMR) (Banarescu et al., 2013), a semantic formalism that has received much research interest (Song et al., 2018;Guo et al., 2019;Ribeiro et al., 2019Ribeiro et al., , 2021aOpitz et al., 2020Opitz et al., , 2021Fu et al., 2021) and has been shown to benefit downstream tasks such as spoken language understanding (Damonte et al., 2019), machine translation (Song et al., 2019), commonsense reasoning (Lim et al., 2020), and question answering (Kapanipathi et al., 2021;Bornea et al., 2021).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In particular, we explore Abstract Meaning Representation (AMR) (Banarescu et al., 2013), a semantic formalism that has received much research interest (Song et al., 2018;Guo et al., 2019;Ribeiro et al., 2019Ribeiro et al., , 2021aOpitz et al., 2020Opitz et al., , 2021Fu et al., 2021) and has been shown to benefit downstream tasks such as spoken language understanding (Damonte et al., 2019), machine translation (Song et al., 2019), commonsense reasoning (Lim et al., 2020), and question answering (Kapanipathi et al., 2021;Bornea et al., 2021).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Intuitively, AMR provides important benefits: First, it encodes core concepts as it strives for a more logical and less syntactic representation, which has been shown to benefit text summarization (Hardy and Vlachos, 2018;Dohare et al., 2018;Lee et al., 2021). Furthermore, AMR captures semantics at a high level of abstraction explicitly modeling relations in the text and reducing the negative influence of diverse text surface variances with the same meaning. Lastly, recent studies (Dreyer et al., 2021;Ladhak et al., 2021) demonstrate that there is a trade-off between factuality and abstractiveness. Structured semantic representations are potentially beneficial for reducing data sparsity and localizing generation errors in abstractive scenarios. Figure 1 shows examples of (c) document and (d) summary AMRs, where the summary AMR is missing a crucial modifying node present in the document AMR, which indicates a factual error in the summary.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Intuitively, AMR provides important benefits: First, it encodes core concepts as it strives for a more logical and less syntactic representation, which has been shown to benefit text summarization (Hardy and Vlachos, 2018;Dohare et al., 2018;Lee et al., 2021). Furthermore, AMR captures semantics at a high level of abstraction explicitly modeling relations in the text and reducing the negative influence of diverse text surface variances with the same meaning. Lastly, recent studies (Dreyer et al., 2021;Ladhak et al., 2021) demonstrate that there is a trade-off between factuality and abstractiveness. Structured semantic representations are potentially beneficial for reducing data sparsity and localizing generation errors in abstractive scenarios. Figure 1 shows examples of (c) document and (d) summary AMRs, where the summary AMR is missing a crucial modifying node present in the document AMR, which indicates a factual error in the summary.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We consolidate a factuality dataset with human annotations derived from previous works (Wang et al., 2020;Kryscinski et al., 2020;Maynez et al., 2020;Pagnoni et al., 2021). This dataset is constructed from the widely-used CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016) benchmarks. Extensive experimental results demonstrate that FACTGRAPH achieves substantial improvements over previous approaches, improving factuality performance by up to 15% and correlation with human judgments by up to 10%, capturing more content verifiability errors and better classifying factuality in semantic relations.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We consolidate a factuality dataset with human annotations derived from previous works (Wang et al., 2020;Kryscinski et al., 2020;Maynez et al., 2020;Pagnoni et al., 2021). This dataset is constructed from the widely-used CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016) benchmarks. Extensive experimental results demonstrate that FACTGRAPH achieves substantial improvements over previous approaches, improving factuality performance by up to 15% and correlation with human judgments by up to 10%, capturing more content verifiability errors and better classifying factuality in semantic relations.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We introduce FACTGRAPH, a method that employs semantic graph representations for factuality evaluation in text summarization, describing its intuition ( §3.3) and defining it formally ( §3.4).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Given a source document D and a sentence-level summary S, we aim to check whether S is factual with respect to D. For each sentence d ∈ D we extract a semantic graph G d . Similarly, for the summary sentence S we extract its semantic graph G s . We use texts and graphs from both document and summary for factuality evaluation. Sentence-level summary predictions can be aggregated to generate a factuality score for a multi-sentence summary.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We select AMR as our MR, but FACTGRAPH can be used with other graph-based semantic representations, such as OpenIE (Banko et al., 2007). AMR is a linguistically-grounded semantic formalism that represents the meaning of a sentence as a rooted graph, where nodes are concepts and edges are semantic relations. AMR abstracts away from surface text, aiming to produce a more languageneutral representation of meaning. We use a stateof-the-art AMR parser (Bevilacqua et al., 2021) to extract an AMR graph G a = (V a , E a , R a ) with a node set V a and labeled edges (u, r, v) ∈ E a , where u, v ∈ V a and r ∈ R a is a relation type. Each G a aims to explicitly represent the core concepts in each sentence. Figure 1 shows an example of a (b) sentence and its (d) corresponding AMR graph. 2 Graph Representation. We convert each G a into a bipartite graph G b = (V b , E b ), replacing each labeled edge (u, r, v) ∈ E a with two unlabeled edges {(u, r), (r, v)} ∈ E b . Similar to Beck et al. (2018), this procedure transforms the graph into its unlabeled version. Pretrained models typically use a vocabulary with subword tokens, which makes it complicated to properly represent a graph using subword tokens as nodes. Inspired by Ribeiro et al. (2020Ribeiro et al. ( , 2021b, we transform each G b into a new token graph G = (V, E), where each token of a node v b ∈ V b becomes a node v ∈ V. We convert each edge (u b , v b ) ∈ E b into a set of edges and connect every token of u b to every token of v b .
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We select AMR as our MR, but FACTGRAPH can be used with other graph-based semantic representations, such as OpenIE (Banko et al., 2007). AMR is a linguistically-grounded semantic formalism that represents the meaning of a sentence as a rooted graph, where nodes are concepts and edges are semantic relations. AMR abstracts away from surface text, aiming to produce a more languageneutral representation of meaning. We use a stateof-the-art AMR parser (Bevilacqua et al., 2021) to extract an AMR graph G a = (V a , E a , R a ) with a node set V a and labeled edges (u, r, v) ∈ E a , where u, v ∈ V a and r ∈ R a is a relation type. Each G a aims to explicitly represent the core concepts in each sentence. Figure 1 shows an example of a (b) sentence and its (d) corresponding AMR graph. 2 Graph Representation. We convert each G a into a bipartite graph G b = (V b , E b ), replacing each labeled edge (u, r, v) ∈ E a with two unlabeled edges {(u, r), (r, v)} ∈ E b . Similar to Beck et al. (2018), this procedure transforms the graph into its unlabeled version. Pretrained models typically use a vocabulary with subword tokens, which makes it complicated to properly represent a graph using subword tokens as nodes. Inspired by Ribeiro et al. (2020Ribeiro et al. ( , 2021b, we transform each G b into a new token graph G = (V, E), where each token of a node v b ∈ V b becomes a node v ∈ V. We convert each edge (u b , v b ) ∈ E b into a set of edges and connect every token of u b to every token of v b .
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In order to represent facts to better assess the summary factuality, we draw inspiration from traditional approaches to summarization that condense the source document to a set of "semantic units" (Liu et al., 2015;Liao et al., 2018). Intuitively, the semantic graphs from the source document represent the core factual information, explicitly modeling relations in the text, whereas the semantic summary graph captures the essential content information in a summary (Lee et al., 2021). The document graphs can be compared with the summary graph, measuring the degree of semantic overlap to assess factuality (Cai and Knight, 2013).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In order to represent facts to better assess the summary factuality, we draw inspiration from traditional approaches to summarization that condense the source document to a set of "semantic units" (Liu et al., 2015;Liao et al., 2018). Intuitively, the semantic graphs from the source document represent the core factual information, explicitly modeling relations in the text, whereas the semantic summary graph captures the essential content information in a summary (Lee et al., 2021). The document graphs can be compared with the summary graph, measuring the degree of semantic overlap to assess factuality (Cai and Knight, 2013).
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Recently, sets of fact triples from summaries were used to estimate factual accuracy (Goodrich et al., 2019). That approach is related to FACT-GRAPH as it uses graph-based MRs, but also different because it compares the reference and the generated summary, whereas we compare the generated summary with the input document. Moreover, differently from Goodrich et al. (2019), FACTGRAPH explicitly encodes the semantic structures using a graph encoder and employs AMR as a semantic representation. Finally, in contrast to DAE (Goyal and Durrett, 2021), which focuses only on extracting summary graph representations, FACTGRAPH uses semantic graphs for both document and summary. Figure 2 illustrates FACTGRAPH, which is composed of text and graph encoders. The text encoder, denoted by E t , uses a pretrained encoder E, augmented with adapter modules which receives the summary S and document D and outputs a contextual text representation. Conversely, the graph encoder, denoted by E g , uses the same E, but is augmented with structure-aware adapters. E g receives the summary and multiple document semantic graphs corresponding to its sentences, and outputs graph-aware contextual representations that are used to generate the final graph representation. During training, only adapter weights are trained, whereas the weights from E are kept frozen. Finally, both graph and text representations are concatenated and fed to a final classifier, which predicts whether the summary is factual or not.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Text Encoder. We employ an adapter module before and after the feed-forward sub-layer of each layer of the encoder. We modify the adapter architecture from Houlsby et al. (2019). We compute the adapter representation for each token i at each layer l, given the token representation h l i , as follows:
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
where σ is the activation function and LN(·) denotes layer normalization. W l o ∈ R d×m and W l p ∈ R m×d are adapter parameters. The representation of the [CLS] token is used as the final textual representation, denoted by t.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Graph Encoder. In order to re-purpose the pretrained encoder to structured inputs, we employ a structural adapter (Ribeiro et al., 2021b). In particular, for each node v ∈ V, given the hidden representation h l v , the encoder layer l computes:
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Graph Encoder. In order to re-purpose the pretrained encoder to structured inputs, we employ a structural adapter (Ribeiro et al., 2021b). In particular, for each node v ∈ V, given the hidden representation h l v , the encoder layer l computes:
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
where N (v) is the neighborhood of the node v in G and W l e ∈ R d×m is an adapter parameter. GraphConv l (·) is the graph convolution that computes the representation of v based on its neighbors in the graph. We employ a Relational Graph Convolutional Network (Schlichtkrull et al., 2018) as graph convolution, which considers differences in the incoming and outgoing relations. Since AMRs are directed graphs, encoding edge directions is beneficial for downstream performance (Ribeiro et al., 2019). The structural adapter is placed before, whereas the normal adapter is kept after the feed-forward sub-layer of each encoder layer.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
where N (v) is the neighborhood of the node v in G and W l e ∈ R d×m is an adapter parameter. GraphConv l (·) is the graph convolution that computes the representation of v based on its neighbors in the graph. We employ a Relational Graph Convolutional Network (Schlichtkrull et al., 2018) as graph convolution, which considers differences in the incoming and outgoing relations. Since AMRs are directed graphs, encoding edge directions is beneficial for downstream performance (Ribeiro et al., 2019). The structural adapter is placed before, whereas the normal adapter is kept after the feed-forward sub-layer of each encoder layer.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We calculate the final representation of each graph from the pooling denoted as
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Thus, we use a multi-head self-attention (Vaswani et al., 2017) layer to estimate to what extent each sentence graph contributes to the document semantic representation based on the summary graph. This mechanism allows encoding a global document representation based on the summary graph. In particular, each attention head computes:
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Thus, we use a multi-head self-attention (Vaswani et al., 2017) layer to estimate to what extent each sentence graph contributes to the document semantic representation based on the summary graph. This mechanism allows encoding a global document representation based on the summary graph. In particular, each attention head computes:
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
where z G s is the final representation of G s , k is the number of considered sentence graphs from the input document and W r ∈ R d×d is a parameter.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
The final representation is derived from the text and graph representations, q = [t; g], and fed into a classification layer that outputs a probability distribution over the labels y = {Factual, Non-Factual}.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Inspired by Goyal and Durrett (2021), we evaluate the factuality at the edge level. In this setup, we use the same text and graph encoders; however, we encode the semantic graphs differently. In particular, we concatenate G s with each G d ∈ D and feed the concatenation to the graph encoder. The representation of a node v ∈ G s is calculated as:  where A(v) is the set of all summary words aligned with v, and r tv and r gv are the word and node representations, respectively. Edge representations are derived for each AMR edge (u, v) ∈ E : r e = [r u ; r v ]. The edge representation r e is fed into a classification layer that outputs a probability distribution over the output labels (y e = {Factual, Non-Factual}). We assign the label non-factual for an edge in G s if one of the nodes in this edge is aligned with a word that belongs to a span annotated as nonfactual. Otherwise, the edge is assigned the label factual. We call this variant FACTGRAPH-E.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
One of the main challenges in developing models for factuality evaluation is the lack of training data. Existing synthetic data generation approaches are not well-suited to factuality evaluation of current summarization models and human-annotated data can improve factuality models (Goyal and Durrett, 2021). In order to have a more effective training signal, we gather human annotations from different sources and consolidate a factuality dataset that can be used to train FACTGRAPH and other models.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
The source collections of the dataset are presented in Table 1. The dataset covers two parts, namely CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016). CNN/DM contains news articles from two providers, CNN and DailyMail; while XSum contains BBC articles. CNN/DM has considerably lower levels of abstraction, and the summary exhibits high overlap with the article; a typical CNN/DM summary consists of several bullet points. In XSum, the first sentence is removed from an article and used as a summary, making it highly abstractive. After we remove duplicated annotations, the total number of datapoints is 9,567, which we divide into train (8,667), dev (300) and test (600) 
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
The source collections of the dataset are presented in Table 1. The dataset covers two parts, namely CNN/DM (Hermann et al., 2015) and XSum (Nallapati et al., 2016). CNN/DM contains news articles from two providers, CNN and DailyMail; while XSum contains BBC articles. CNN/DM has considerably lower levels of abstraction, and the summary exhibits high overlap with the article; a typical CNN/DM summary consists of several bullet points. In XSum, the first sentence is removed from an article and used as a summary, making it highly abstractive. After we remove duplicated annotations, the total number of datapoints is 9,567, which we divide into train (8,667), dev (300) and test (600) 
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We limit the number of considered document graphs due to efficiency reasons. In particular, we compute the pairwise cosine similarity between the embeddings of each sentence d ∈ D and the summary sentence S, generated by Sentence Transformers (Reimers and Gurevych, 2019). We thus select k sentences from the source document with the highest scores to be used to generate the document semantic graphs. The model weights are initialized with ELEC-TRA (electra-base discriminator, 110M parameters, Clark et al., 2020), the structural adapters are pretrained using the release 3.0 of the AMR corpus containing 55,635 gold annotated AMR graphs, and the text adapters are pretrained using synthetic generated data. The adapters' hidden dimension is 32, which corresponds to about 1.4% of the parameters of the original ELECTRA encoders. The number of considered document graphs (k) is 5. 3 We report the test results when the balanced accuracy (BACC) on dev set is optimal. Following previous work (Kryscinski et al., 2020;Goyal and Durrett, 2021), we evaluate our models using BACC and Micro F1 scores.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We limit the number of considered document graphs due to efficiency reasons. In particular, we compute the pairwise cosine similarity between the embeddings of each sentence d ∈ D and the summary sentence S, generated by Sentence Transformers (Reimers and Gurevych, 2019). We thus select k sentences from the source document with the highest scores to be used to generate the document semantic graphs. The model weights are initialized with ELEC-TRA (electra-base discriminator, 110M parameters, Clark et al., 2020), the structural adapters are pretrained using the release 3.0 of the AMR corpus containing 55,635 gold annotated AMR graphs, and the text adapters are pretrained using synthetic generated data. The adapters' hidden dimension is 32, which corresponds to about 1.4% of the parameters of the original ELECTRA encoders. The number of considered document graphs (k) is 5. 3 We report the test results when the balanced accuracy (BACC) on dev set is optimal. Following previous work (Kryscinski et al., 2020;Goyal and Durrett, 2021), we evaluate our models using BACC and Micro F1 scores.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We compare FACTGRAPH with different methods for factuality evaluation: two QA-based methods, namely QAGS (Wang et al., 2020) and QUALS (Nan et al., 2021), andFACTCC (Kryscinski et al., 2020). We fine-tune FACTCC using the training set, that is, it is trained on both synthetic data and FACTCOLLECT. We call this approach FACTCC+. 
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We compare FACTGRAPH with different methods for factuality evaluation: two QA-based methods, namely QAGS (Wang et al., 2020) and QUALS (Nan et al., 2021), andFACTCC (Kryscinski et al., 2020). We fine-tune FACTCC using the training set, that is, it is trained on both synthetic data and FACTCOLLECT. We call this approach FACTCC+. 
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We also evaluate the model performance using correlations with human judgments of factuality (Pagnoni et al., 2021). In this experiment, FACTCC+ and FACTGRAPH are trained with the FACTCOLLECT data without the Pagnoni et al.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We also evaluate the model performance using correlations with human judgments of factuality (Pagnoni et al., 2021). In this experiment, FACTCC+ and FACTGRAPH are trained with the FACTCOLLECT data without the Pagnoni et al.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
(2021)'s subset, which is used as dev and test sets, according to its split. For both models, following Pagnoni et al. (2021), we obtain a binary factuality label for each sentence and take the average of these labels as the final summary score. We use the official script to calculate the correlations. 5 AMR and Factuality. We investigate whether SMATCH (Cai and Knight, 2013), a metric that measures the degree of overlap between two AMRs, correlates with factuality judgments. We calculate the SMATCH score between all the summary sentence graphs and k document sentence graphs, with k ∈ {1, 3, 5}. We obtain one score per summary 4 FACTGRAPH is significantly better than FACTCC+ with p<0.05 on both BACC and F1 scores.  sentence by maxing over its scores with the sentence graphs, then averaging over the summary sentence scores to obtain the summary-level score.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
(2021)'s subset, which is used as dev and test sets, according to its split. For both models, following Pagnoni et al. (2021), we obtain a binary factuality label for each sentence and take the average of these labels as the final summary score. We use the official script to calculate the correlations. 5 AMR and Factuality. We investigate whether SMATCH (Cai and Knight, 2013), a metric that measures the degree of overlap between two AMRs, correlates with factuality judgments. We calculate the SMATCH score between all the summary sentence graphs and k document sentence graphs, with k ∈ {1, 3, 5}. We obtain one score per summary 4 FACTGRAPH is significantly better than FACTCC+ with p<0.05 on both BACC and F1 scores.  sentence by maxing over its scores with the sentence graphs, then averaging over the summary sentence scores to obtain the summary-level score.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We also calculate the SMATCH between the generated summary and the reference summary graphs. As shown in Table 3, SMATCH approaches have a small but consistent correlation, slightly improving over n-gram based metrics (e.g., METEOR and ROUGE-L) in CNN/DM, suggesting that AMR, which has a higher level of abstraction than plain text, may be a semantic representation alternative to content verification. QA-based approaches have higher correlation on the CNN/DM dataset than XSum where their correlation is relatively reduced, and DAE shows higher Spearman correlation than FACTCC on XSum. FACTCC+ and FACTGRAPH, which are trained on data from FACTCOLLECT, have a overall higher performance than models trained on synthetic data, such as FACTCC, again demonstrating the importance of the human-annotation signal when training factuality evaluation approaches. Finally, FACTGRAPH has the highest correlations in both datasets, with a large improvement in XSum, suggesting that representing facts as semantic graphs is effective for more abstractive summaries. Figure 3 shows the influence of the different types of factuality errors (Pagnoni et al., 2021) for each approach. Semantic Frame Errors are errors in a frame, core, and non-core frame elements. 6 Discourse Errors extend beyond a single semantic frame introducing erroneous links between discourse segments. Content Verifiability 6 A semantic frame is a representation of an event, relation, or state (Baker et al., 1998  Errors capture cases when it is not possible to verify the summary against the source document due to the difficulty in aligning it to the source. 7 Note that whereas BERTSCORE strongly correlates with content verifiability errors as it is a token-level similarity metric, the other methods improve in Semantic Frame Errors. FACTGRAPH has the highest performance suggesting that graph-based MRs are able to capture different semantic errors well. In particular, FACTGRAPH improves in capturing content verifiability errors by 48.2%, suggesting that representing facts using AMR is helpful.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We also calculate the SMATCH between the generated summary and the reference summary graphs. As shown in Table 3, SMATCH approaches have a small but consistent correlation, slightly improving over n-gram based metrics (e.g., METEOR and ROUGE-L) in CNN/DM, suggesting that AMR, which has a higher level of abstraction than plain text, may be a semantic representation alternative to content verification. QA-based approaches have higher correlation on the CNN/DM dataset than XSum where their correlation is relatively reduced, and DAE shows higher Spearman correlation than FACTCC on XSum. FACTCC+ and FACTGRAPH, which are trained on data from FACTCOLLECT, have a overall higher performance than models trained on synthetic data, such as FACTCC, again demonstrating the importance of the human-annotation signal when training factuality evaluation approaches. Finally, FACTGRAPH has the highest correlations in both datasets, with a large improvement in XSum, suggesting that representing facts as semantic graphs is effective for more abstractive summaries. Figure 3 shows the influence of the different types of factuality errors (Pagnoni et al., 2021) for each approach. Semantic Frame Errors are errors in a frame, core, and non-core frame elements. 6 Discourse Errors extend beyond a single semantic frame introducing erroneous links between discourse segments. Content Verifiability 6 A semantic frame is a representation of an event, relation, or state (Baker et al., 1998  Errors capture cases when it is not possible to verify the summary against the source document due to the difficulty in aligning it to the source. 7 Note that whereas BERTSCORE strongly correlates with content verifiability errors as it is a token-level similarity metric, the other methods improve in Semantic Frame Errors. FACTGRAPH has the highest performance suggesting that graph-based MRs are able to capture different semantic errors well. In particular, FACTGRAPH improves in capturing content verifiability errors by 48.2%, suggesting that representing facts using AMR is helpful.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We assess factuality beyond sentence-level with FACTGRAPH-E ( §3.5). We train and evaluate the model against the sentence-level factuality data from Maynez et al. (2020). In this dataset, human annotations for sentence and span levels are  available. We derive the edge labels required for FACTGRAPH-E training as follows: For each edge in the summary graph, if one of the nodes connected to this edge is aligned with a word that belongs to a span labeled as non-factual, the edge is annotated as non-factual. 8 Summary-level labels are obtained from edge-level predictions: if any edge in the summary graph is classified as nonfactual, the summary is labeled as non-factual. We use the same splits from Goyal and Durrett (2021). 9 We compare FACTGRAPH-E with DAE and additionally with a sentence-level baseline (Goyal and Durrett, 2021) and FACTGRAPH. Table 4 shows that the edge-level factuality classification gives better performance than sentencelevel classification, and FACTGRAPH performs better in both sentence and edge classification levels. FACTGRAPH-E outperforms DAE, demonstrating that training on subsentence-level factuality annotations enables it to accurately predict edge-level factuality and output summary-level factuality.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We assess factuality beyond sentence-level with FACTGRAPH-E ( §3.5). We train and evaluate the model against the sentence-level factuality data from Maynez et al. (2020). In this dataset, human annotations for sentence and span levels are  available. We derive the edge labels required for FACTGRAPH-E training as follows: For each edge in the summary graph, if one of the nodes connected to this edge is aligned with a word that belongs to a span labeled as non-factual, the edge is annotated as non-factual. 8 Summary-level labels are obtained from edge-level predictions: if any edge in the summary graph is classified as nonfactual, the summary is labeled as non-factual. We use the same splits from Goyal and Durrett (2021). 9 We compare FACTGRAPH-E with DAE and additionally with a sentence-level baseline (Goyal and Durrett, 2021) and FACTGRAPH. Table 4 shows that the edge-level factuality classification gives better performance than sentencelevel classification, and FACTGRAPH performs better in both sentence and edge classification levels. FACTGRAPH-E outperforms DAE, demonstrating that training on subsentence-level factuality annotations enables it to accurately predict edge-level factuality and output summary-level factuality.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Finally, while the semantic representations contribute to overall performance, extracting those representations adds some overhead in preprocessing time (and slightly more in inference time), as shown in Appendix C.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In Table 5, we report an ablation study on the impact of distinct FACTGRAPH's components. First, note that only encoding the textual information leads to better performance than just encoding graphs. This is expected since pretrained encoders are known for good performance in NLP textual tasks due to their transfer learning capabilities and the full document text encodes more information than the selected k document graphs. Moreover, AMR representations abstract aspects such as verb tenses, making the graphs agnostic regarding more  fine-grained information. However, this is compensated in FACTGRAPH, which captures coarsegrained details from the text modality. Future work can consider incorporating such information into the graph representation in order to improve the factuality assessment. Ultimately, FACTGRAPH, which uses both document and summary graphs, gives the overall best performance, demonstrating that semantic graph representations complement the text representation and are beneficial for factuality evaluation. Table 6 shows the influence of the number of considered document graphs measured on FACTCOLLECT's dev set performance. Note that generally more document graphs leads to better performance with a peak in 5. This suggests that using all graph sentences from the source document is not required for better performance. Moreover, the results indicate that our strategy of selecting document graphs using the contextual representations of the document sentences which are compared to the summary performs well in practice.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We additionally present the performance of FACTGRAPH with other semantic representations in Appendix D.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
FACTGRAPH only trains adapter weights that are placed into each layer of both text and graph encoders. We compare FACTGRAPH with a model with similar architecture, with both text and graph encoders, but without (structural) adapter layers. We then fine-tune all the model parameters. Table 7 shows that FACTGRAPH performs better even though it trains only 1.4% of the parameters of the fully fine-tuned model, suggesting that the structural adapters help to adapt the graph encoder to semantic graph representations.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
FACTGRAPH-E computes factuality scores for each edge of the AMR summary graph and those predictions are aggregated to generate a sentence-Article: Margaret Fleming, 36, was last seen at her home in Inverkip by her two carers at about 17:40 on Friday 28 October. She is described as about 5ft 5in tall, […]. Police had said they were trying to build a picture of Ms Fleming's life, part of which she kept "quite private". When last seen, she was wearing a green tartan fleece […]. She also had a satchel-type handbag. A police spokesman said: "There is a specialist search team combing the area around where the missing person was last seen, this includes in the garden of her last known address." […] The detective said that Ms Fleming was a student at James Watt College in Greenock between 1996 and 1997. He said he was keen to speak to anyone who remembered her from then, and who might have been in touch with her over the years.   level label ( §5.2). Alternatively, it is possible to identify specific inconsistencies in the generated summary based on the AMR graph structure. This factuality information at subsentence-level can provide deeper insights on the kinds of factual inconsistencies made by different summarization models (Maynez et al., 2020) and can supply text generation approaches with localized signals for training (Cao et al., 2020;. Figure 4 shows a document, its generated summary, and factuality edge predictions by DAE and FACTGRAPH-E. 10 First, note that since DAE uses dependency arcs and FACTGRAPH-E is based on AMR, the sets of edges in both approaches, that is, the relations between nodes and hence words, are different. Second, both methods are able to detect the hallucination six years, which was never mentioned in the source document. However, DAE does not consider that police appealed for help in tracing is factual whereas FACTGRAPH-E captures it. This piece of information is related to a span in the document with a very different but semantically related form (highlighted in bold in Figure 4). This poses challenges to DAE, since it classifies semantic relations independently and only considers the 10 Appendix E presents the complete AMR and dependency summary graphs.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
FACTGRAPH-E computes factuality scores for each edge of the AMR summary graph and those predictions are aggregated to generate a sentence-Article: Margaret Fleming, 36, was last seen at her home in Inverkip by her two carers at about 17:40 on Friday 28 October. She is described as about 5ft 5in tall, […]. Police had said they were trying to build a picture of Ms Fleming's life, part of which she kept "quite private". When last seen, she was wearing a green tartan fleece […]. She also had a satchel-type handbag. A police spokesman said: "There is a specialist search team combing the area around where the missing person was last seen, this includes in the garden of her last known address." […] The detective said that Ms Fleming was a student at James Watt College in Greenock between 1996 and 1997. He said he was keen to speak to anyone who remembered her from then, and who might have been in touch with her over the years.   level label ( §5.2). Alternatively, it is possible to identify specific inconsistencies in the generated summary based on the AMR graph structure. This factuality information at subsentence-level can provide deeper insights on the kinds of factual inconsistencies made by different summarization models (Maynez et al., 2020) and can supply text generation approaches with localized signals for training (Cao et al., 2020;. Figure 4 shows a document, its generated summary, and factuality edge predictions by DAE and FACTGRAPH-E. 10 First, note that since DAE uses dependency arcs and FACTGRAPH-E is based on AMR, the sets of edges in both approaches, that is, the relations between nodes and hence words, are different. Second, both methods are able to detect the hallucination six years, which was never mentioned in the source document. However, DAE does not consider that police appealed for help in tracing is factual whereas FACTGRAPH-E captures it. This piece of information is related to a span in the document with a very different but semantically related form (highlighted in bold in Figure 4). This poses challenges to DAE, since it classifies semantic relations independently and only considers the 10 Appendix E presents the complete AMR and dependency summary graphs.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Fully  text surface. On the other hand, FACTGRAPH-E matches the summary against the document not only at text surface level but semantic level.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
We presented FACTGRAPH, a graph-based approach to explicitly encode facts using meaning representations to identify factual errors in generated text. We provided an extensive evaluation of our approach and showed that it significantly improves results on different factuality benchmarks for summarization, indicating that structured semantic representations are beneficial to factuality evaluation. Future work includes (i) exploring approaches to develop document-level semantic graphs (Naseem et al., 2021), (ii) an explainable graph-based component to highlight hallucinations and (iii) to combine different meaning representations in order to capture distinct semantic aspects.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In this paper, we study the problem of detecting factual inconsistencies in summaries generated from input documents. The proposed models better consider the text internal meaning structure and could benefit general generation applications by evaluating their output regarding factual consistency, which could ensure that these systems are more trustworthy. This work is built using semantic representations extracted using AMR parsers. In this way, the quality of the parser used to generate the semantic representations can significantly impact the results of our models. In our work, we mitigate this risk by employing a state-of-the-art AMR parser (Bevilacqua et al., 2021).  
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In this supplementary material, we detail experiments' settings, additional model evaluations and additional information about semantic graph representations.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
A Additional Examples Figure 6 shows examples of AMR representations generated from summaries and salient sentences from the respective source document.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
The experiments were executed using the version 3.3.1 of the transformers library released by Hugging Face (Wolf et al., 2019). In Table 8, we report the hyperparameters used to train FACT-GRAPH. We use the Adam optimizer (Kingma and Ba, 2015) and employ a linearly decreasing learning rate schedule without warm-up. Mean pooling is used to calculate the final representation of each graph.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Structural Adapters' Pretraining. The structural adapters are pretrained using AMR graphs from the release 3.0 (LDC2020T02) of the AMR annotation corpus (Knight et al., 2020). 11 Similarly to the masked language modeling objective, we execute self-supervised node-level prediction, where we randomly mask and classify AMR nodes. The goal of this pretraining phase is to capture domain specific AMR knowledge by learning the regularities of the node/edge attributes distributed over graph structure.   which is generated by applying a series of rulebased transformations to the sentences of source documents (Kryscinski et al., 2020). The pretraining task is to classify each summary sentence as factual or non-factual. The goal of this pretraining phase is to learn suitable text representations to better identify whether summary sentences remain factually consistent to the input document after the transformation.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
FACTGRAPH encodes the structured semantic representations that encode facts from the document and summary. Despite their effectiveness, extracting semantic graphs, such as AMR, is computationally expensive because current models employ Transformer-based encoder-decoder architectures based on Transformers and pretrained language models. In this experiment, we compare the time execution of FACTGRAPH-E and DAE in a sample of 1000 datapoints extracted from the XSum test set. In order to extract the semantic graphs, we investigate two AMR parsers, Parser1: a dual graphsequence parser that iteratively refines an incrementally constructed graph (Cai and Lam, 2020) , and Parser2: a linearized graph model that employs BART (Bevilacqua et al., 2021). The execution of the AMR parsers is parallelized using four Tesla V100 GPUs. We use Parser2 for the experiments in this paper since it is the current state of the art in AMR parsing, although it is slower in preprocessing than Parser1.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
As shown in Table 9, DAE's preprocessing is much faster compared to this phase in FACTGRAPH-E, since DAE employs a fast en-hanced dependency model from the Stanford CoreNLP tool (Manning et al., 2014). This model builds a parse by performing a linear-time scan over the words of a sentence. Finally, note that FACT-GRAPH is slower than DAE in inference because it employs adapters and encodes both graphs and texts from the document and summary, whereas the DAE model encodes only the texts.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
As shown in Table 9, DAE's preprocessing is much faster compared to this phase in FACTGRAPH-E, since DAE employs a fast en-hanced dependency model from the Stanford CoreNLP tool (Manning et al., 2014). This model builds a parse by performing a linear-time scan over the words of a sentence. Finally, note that FACT-GRAPH is slower than DAE in inference because it employs adapters and encodes both graphs and texts from the document and summary, whereas the DAE model encodes only the texts.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
OpenIE graph-based structures were used in order to improve factuality in abstractive summarization (Cao et al., 2018), whereas dependency arcs were shown to be beneficial for evaluating factuality (Goyal and Durrett, 2020). We thus investigate different graph-based meaning representations using FACTGRAPH. AMR is a more logical representation that models relations between core concepts, and has a rough alignment between nodes and spans in the text. Conversely, dependencies capture more fine-grained relations between words, and all words are mapped into nodes in the dependency graph. OpenIE constructs a graph with node descriptions similar to the original text and uses open-domain relations, leading to relations that are hard to compare. As shown in Table 10, whereas OpenIE performs slightly better than dependency graphs, AMR gives the best results according to the two metrics, highlighting the potential use of AMRs in representing salient pieces of information. Different from our work, Lee et al. (2021) and Naseem et al. (2021) propose a graph construction approach which generates a single document-level graph created using the individual sentences' AMR graphs by merging identical concepts -this is orthogonal to our sentence-level AMR representation and can be incorporated in future work.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
In Figure 5 we show AMR and dependency representations for the summary sentence "police have appealed for help in tracing a woman who has been missing for six years.". In §5.5 those semantic representations are used to predict subsentence-level factuality using edge-level information. In particular, FACTGRAPH-E employs AMR (Figure 5a) whereas DAE uses dependencies (Figure 5b). 
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Recent progress in text summarization has been remarkable, with ROUGE record-setting models published every few months, and human evaluations indicating that automatically generated summaries are matching human-written summaries in terms of fluency and informativeness (Zhang et al., 2020a).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
A major limitation of current summarization models is their inability to remain factually consistent with the respective input document. Summary inconsistencies are diverse-from inversions (i.e., negation) to incorrect use of an entity (i.e., subject, object swapping), or hallucinations (i.e., introduction of entity not in the original document). Recent studies have shown that in some scenarios, even state-of-the-art pre-trained language models can generate inconsistent summaries in more than 70% of all cases (Pagnoni et al., 2021). This has led to accelerated research around summary inconsistency detection.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
A closely related task to inconsistency detection is textual entailment, also referred to as Natural Language Inference (NLI), in which a hypothesis sentence must be classified as either entailed by, neutral, or contradicting a premise sentence. Enabled by the crowd-sourcing of large NLI datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), modern architectures have achieved close to human performance at the task.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The similarity of NLI to inconsistency detection, as well as the availability of high-performing NLI models, led to early attempts at using NLI to detect consistencyerrorsinsummaries. These early attempts were unsuccessful, finding that re-ranking summaries according to an NLI model can lead to an increase in consistency errors (Falke et al., 2019), or that out-of-the-box NLI models obtain 52% accuracy at the binary classification task of inconsistency detection, only slightly above random guessing (Kryscinski et al., 2020).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
In this work, we revisit this approach, showing that NLI models can in fact successfully be used for inconsistency detection, as long as they are used at the appropriate granularity. Figure 1 shows how crucial using the correct granularity as input to NLI models is. An inconsistency checker should flag the last sentence in the summary (shown right) as problematic. When treating the entire document as the premise and the summary as the hypothesis, a competitive NLI model predicts with probability of 0.91 that the summary is entailed by the document. However, when splitting the documents into sentence premise-hypothesis pairs (visualized as edges in Figure 1) the NLI model correctly determines that S 3 is not supported by any document sentence. This illustrates that working with sentence pairs is crucial for making NLI models work for inconsistency detection. Figure 1: Example document with an inconsistent summary. When running each sentence pair (D i , S j ) through an NLI model, S 3 is not entailed by any document sentence. However, when running the entire (document, summary) at once, the NLI model incorrectly predicts that the document highly entails the entire summary.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Our contributions are two-fold. First, we introduce a new approach for inconsistency detection based on the aggregation of sentence-level entailment scores for each pair of input document and summary sentences. We present two model variants that differ in the way they aggregate sentencelevel scores into a single score. SUMMAC ZS performs zero-shot aggregation by combining sentence-level scores using max and mean operators. SUMMAC CONV is a trained model consisting of a single learned convolution layer compiling the distribution of entailment scores of all document sentences into a single score.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Second, to evaluate our approach, we introduce the SUMMAC Benchmark by standardizing existing datasets. Because the benchmark contains the six largest summary consistency datasets, it is more comprehensive and includes a broader range of inconsistency errors than prior work.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The SUMMAC models outperform existing inconsistency detection models on the benchmark, with the SUMMAC CONV obtaining an overall balanced accuracy of 74.4%, 5% above prior work. We publicly release the models and datasets. 1
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Fact checking is a related task in which a model receives an input claim along with a corpus of ground truth information. The model must then retrieve relevant evidence and decide whether the claim is supported, refuted, or if there is not enough information in the corpus (Thorne et al., 2018). The major difference to our task lies in the different semantics of consistency and accuracy. If a summary adds novel and accurate information not present in the original document (e.g., adding background information), the summary is accurate but inconsistent. In the summary inconsistency detection domain, the focus is on detecting any inconsistency, regardless of its accuracy, as prior work has shown that current automatic summarizers are predominantly inaccurate when inconsistent (Maynez et al., 2020).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Several datasets have been annotated to evaluate model performance in inconsistency detection, typically comprising up to two thousand annotated summaries. Datasets are most commonly crowdannotated with three judgements each, despite some work showing that as many as eight annotators are required to achieve high inter-annotator agreement (Falke et al., 2019).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Reading the entire original document being summarized is time-consuming, and to amortize this cost, consistency datasets often contain multiple summaries, generated by different models, for the same original document.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Some datasets consist of an overall consistency label for a summary (e.g., FactCC [Kryscinski et al., 2020]), while others propose a finer-grained typology with up to 8 types of consistency errors (Huang et al., 2020).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We include the six largest summary consistency datasets in the SUMMAC Benchmark, and describe them more in detail in Section 4.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Due to data limitations, most inconsistency detection methods adapt NLP pipelines from other tasks including QAG models, synthetic classifiers, and parsing-based methods.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
QAG methods follow three steps: (1) question generation (QG), (2) question answering (QA) with the document and the summary, (3) matching document and summary answers. A summary is considered consistent if few or no questions have differing answer with the document. A key design choice for these methods lies in the source for question generation. Durmus et al. (2020) generate questions using the summary as a source, making their FEQA method precision-oriented. Scialom et al. (2019) generate questions with the document as a source, creating a recall-focused measure. Scialom et al. (2021) unite both in QuestEval, by generating two sets of questions, sourced from the summary and document respectively. We include FEQA and QuestEval in our benchmark results.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Synthetic classifiers rely on large, synthetic datasets of summaries with inconsistencies, and use those to train a classifier with the expectation that the model generalizes to non-synthetic summaries. To generate a synthetic dataset, Kryscinski et al. (2020) propose a set of semantically invariant (e.g., paraphrasing) and variant (e.g., sentence negation) text transformations that they apply to a large summarization dataset. FactCC-CLS, the classifier obtained when training on the synthetic dataset, is included in our benchmark results for comparison.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Parsing-based methods generate relations through parsing and compute the fraction of summary relations that are compatible with docu ment relations as a precision measure of summary factuality. Goodrich et al. (2019) extract (subject, relation, object) tuples most commonly using OpenIE (Etzioni et al., 2008). In the recent DAE model, Goyal and Durrett (2020) propose to use arc labels from a dependency parser instead of relation triplet. We include the DAE model in our benchmark results.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Complementary to inconsistency detection, some work focused on the task of mitigating inconsistency errors during summarization. Approaches fall in two categories: Reinforcement Learning (RL) methods to improve models and stand-alone re-writing methods.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
RL methods often rely on an out-of-the-box inconsistency detection model and use reinforcement learning to optimize a reward with a consistency component. Arumae and Liu (2019) optimize a QA-based consistency reward, and Nan et al. (2021) streamline a QAG reward by combining the QG and QA model, making it more efficient for RL training. Pasunuru and Bansal (2018) leverage an NLI-based component as part of an overall ROUGE-based reward, and Zhang et al. (2020b) use a parsing-based measure in the domain of medical report summarization.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Re-writing methods typically operate as a modular component that is applied after an existing summarization model. Cao et al. (2020) use a synthetic dataset of rule-corrupted summaries to train a post-corrector model, but find that this model does not transfer well to real summarizer errors.  propose to use a QAG model to find erroneous spans, which are then corrected using a post-processing model.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Since all methods discussed above for consistency correction rely on a model to detect inconsistencies, they will naturally benefit from more accurate inconsistency detectors.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We now introduce our SUMMAC models for inconsistency detection. The first step common to all models is to apply an out-of-the-box NLI model to generate an NLI Pair Matrix for a (document, summary) pair. The two models we present then differ in the way they process this pair matrix to produce a single consistency score for a given summary. We also describe the SUMMAC evaluation benchmark, a set of inconsistency detection datasets, in Section 4. In Section 5, we measure the performance of the SUMMAC models on this benchmark and investigate components of the models, including which NLI model achieves highest performance, which NLI categories should be used, and what textual granularity is most effective.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
NLI datasets are predominantly represented at the sentence level. In our pilot experiments, we found that this causes the resulting NLI models to fail in assessing consistency for documents with 50 sentences and more.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
This motivates the following approach. We generate an NLI Pair Matrix by splitting a (document, summary) pair into sentence blocks. The document is split into M blocks, each considered a premise labeled from D 1 , . . . , D M , and the summary is split into N blocks, each considered a hypothesis labeled from S 1 , . . . , S N .
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Each D i , S j combination is run through the NLI model, which produces a probability distribution over the three NLI categories (E ij , C ij , N ij ) Figure 2: Diagram of the SUMMAC ZS (top) and SUMMAC CONV (bottom) models. Both models utilize the same NLI Pair Matrix (middle) but differ in their processing to obtain a score. The SUMMAC ZS is Zero-Shot, and does not have trained parameters. SUMMAC CONV uses a convolutional layer trained on a binned version of the NLI Pair Matrix.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
for entailment, contradiction, and neutral, respectively. If not specified otherwise, the pair matrix is an M × N matrix consisting of the entailment scores E ij . In Section 5.3.3, we examine the effect of granularity by splitting texts at the paragraph level or binning two sentences at a time. In Section 5.3.2, we explore the use of the contradiction and neutral categories in our experiments.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The example in Figure 1 has M = 4 document sentences, and N = 3 summary sentences, and the corresponding NLI Pair Matrix is the following: The pair matrix can be interpreted as the weights of a bipartite graph, which is also illustrated in Figure 1 where the opacity of each edge (i, j) represents the entailment probability E ij .
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The two SUMMAC models take as input the same NLI Pair Matrix, but differ in the aggregation method to transform the pair matrix into a score. Figure 2 presents an overview of SUMMAC ZS and SUMMAC CONV .
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
In the SUMMAC ZS model, we reduce the pair matrix to a one-dimensional vector by taking the maximum (max) value of each column. On an intuitive level, for each summary sentence, this step consists of retaining the score for the document sentence that provides the strongest support for each summary sentence. For the example in Figure 1:
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The second step consists of taking the mean of the produced vector, reducing the vector to a scalar which is used as the final model score. At a high level, this step aggregates sentence-level information into a single score for the entire summary. For example, in Figure 1, the score produced by SUMMAC ZS would be 0.67. If we removed the third sentence from the summary, the score would increase to 0.985. We experiment with replacing the max and mean operators with other operators in Appendix B.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
One limitation of SUMMAC ZS is that it is highly sensitive to extrema, which can be noisy due to the presence of outliers and the imperfect nature of NLI models. In SUMMAC CONV , we reduce the reliance on extrema values by instead taking into account the entire distribution of entailment scores for each summary sentence. For each summary sentence, a learned convolutional layer is in charge of converting the entire distribution into a single score.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The first step of the SUMMAC CONV algorithm is to turn each column of the NLI Pair Matrix into a fixed-size histogram that represents the distribution of scores for that given summary sentence.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We bin the NLI scores into H evenly spaced bins (e.g., if H = 5, the bins are By producing one histogram for each summary sentence, the binning process in the example of Figure 1 would produce:  Table 1: Statistics of the six datasets in the SUMMAC Benchmark. For each dataset, we report the validation and test set sizes, the percentage of summaries with positive (consistent) labels (% Positive), the inter-annotator agreement (when available, IAA), the source of the documents (Source: C for CNN/DM, X for XSum), the number of summarizers evaluated, and the number of sublabels annotated.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The binned matrix is then passed through a 1-D convolution layer with a kernel size of H. The convolution layer scans the summary histograms one at a time, and compiles each into a scalar value for each summary. Finally, the scores of each summary sentence are averaged to obtain the final summary-level score.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
In order to learn the weights of the convolution layer, we train the SUMMAC CONV model end-toend with the synthetic training data in FactCC (Kryscinski et al., 2020). The original training dataset contains one million (document, summary) pairs evenly distributed with consistent and inconsistent summaries. Because we are only training a small set of H parameters (we use H = 50), we find that using a 10,000 sub-sample is sufficient. We train the model using a cross-entropy loss, the Adam optimizer, a batch size of 32, and a learning rate of 10 −2 . We perform hyper-parameter tuning on a validation set from the FactCC dataset.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The number of bins used in the binning process, which corresponds to the number of parameters in the convolution layer, is also a hyper-parameter we tune on the validation set. We find that performance increases until 50 bins (i.e., a bin width of 0.02) and then plateaus. We use 50 bins in all our experiments.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
To rigorously evaluate the SUMMAC models on a diverse set of summaries with consistency judgements, we introduce a new large benchmark dataset, the SUMMAC Benchmark. It comprises the six largest available datasets for summary incon-sistency detection, which we standardize to use the same classification task.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We standardize the task of summary inconsistency detection by casting it as a binary classification task. Each dataset contains (document, summary, label) samples, where the label can either be consistent or inconsistent.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Each dataset is divided into a validation and test split, with the validation being available for parameter tuning. We used existing validation/test splits created by dataset authors when available. We did not find a split for XSumFaith, Polytope, and SummEval, and created one by putting even-indexed samples in a validation split, and odd-indexed samples in the test split. This method of splitting maintains similar class imbalance and summarizer identity with the entire dataset.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We computed inter-annotator agreement calculated with Fleiss' Kappa (Fleiss, 1971) on the dataset as an estimate for dataset quality, omitting datasets for which summaries only had a single annotator (Polytope and FactCC). Table 1 summarizes dataset statistics and properties.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We introduce each dataset in the benchmark chronologically, and describe the standardizing procedure.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
CoGenSumm (Correctness of Generated Summaries, CGS) (Falke et al., 2019) is the first introduced dataset for summary inconsistency detection, based on models trained on the CNN/DM dataset (Nallapati et al., 2016). The authors proposed that consistency detection should be approached as a ranking problem: Given a consistent and inconsistent summary for a common document, a ranking model should score the consistent summary higher. Although innovative, other datasets in the benchmark do not always have positive and negative samples for a given document. We thus map the dataset to a classification task by using all inconsistent and consistent summaries as individual samples.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
XSumFaith (eXtreme Summarization Faithfulness, XSF) (Maynez et al., 2020) is a dataset with models trained on the XSum dataset (Narayan et al., 2018), which consists of more abstractive summaries than CoGenSumm. The authors find that standard generators remain consistent for only 20-30% of generated summaries. The authors differentiate between extrinsic and intrinsic hallucinations (which we call inconsistencies in this work). Extrinsic hallucinations, which involve words or concepts not in the original document can nonetheless be accurate or inaccurate. In order for a summarizer to generate an accurate extrinsic hallucination, the summarizer must possess external world knowledge. Because the authors found that the models are primarily inaccurate in terms of extrinsic hallucinations, we map both extrinsic and intrinsic hallucinations to a common inconsistent label.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Polytope (Huang et al., 2020) introduces a more extensive typology of summarization errors, based on the Multi-dimensional Quality Metric (Mariana, 2014). Each summary is annotated with eight possible errors, as well as a severity level for the error. We standardize this dataset by labeling a summary as inconsistent if it was annotated with any of the five accuracy errors (and disregarded the three fluency errors). Each summary in Polytope was labeled by a single annotator, making it impossible to measure inter-annotator agreement.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
FactCC (Kryscinski et al., 2020) contains validation and test splits that are entirely annotated by authors of the paper, because attempts at crowd-sourced annotation yielded low interannotator agreement. Prior work (Gillick and Liu, 2010) shows that there can be divergence in annotations between experts and non-experts in summarization, and because the authors of the paper are NLP researchers familiar with the limitations of automatic summarizations, we expect that FactCC annotations differs in quality from other datasets. FactCC also introduces a synthetic dataset by modifying consistent summaries with semantically variant rules. We use a sub-portion of this synthetic dataset to train the SUMMAC CONV model.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
SummEval (Fabbri et al., 2021) contains summarizer outputs from seven extractive models and sixteen abstractive models. Each summary was labeled using a 5-point Likert scale along four categories: coherence, consistency, fluency, and relevance by 3 annotators. We label summaries as consistent if all annotators gave a score of 5 in consistency, and inconsistent otherwise.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
FRANK (Pagnoni et al., 2021) contains annotations for summarizers trained on both CNN/DM and XSum, with each summary annotated by three crowd-workers. The authors propose a new typology with seven error types, organized into semantic frame errors, discourse errors and content verifiability errors. The authors confirm that models trained on the more abstractive XSum dataset generate a larger proportion of inconsistent summaries, compared to models trained on CNN/DM. We label summaries as consistent if a majority of annotators labeled the summary as containing no error.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
With each dataset in the SUMMAC Benchmark converted to a binary classification task, we now discuss the choice of appropriate evaluation metrics for the benchmark. Previous work on each dataset in the benchmark used different evaluation methods, falling into three main categories.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
First, CoGenSumm proposes a re-ranking based measure, requiring pairs of consistent and inconsistent summaries for any document evaluated; this information is not available in several datasets in the benchmark.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Second, XSumFaith, SummEval, and FRANK report on correlation of various metrics with human annotations. Correlation has some advantages, such as not requiring a threshold and being compatible with the Likert-scale annotations of SummEval, however it is an uncommon choice to measure performance of a classifier due to the discrete and binary label.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Third, authors of FactCC measured model performance using binary F1 score, and balanced accuracy, which corrects unweighed accuracy with the class imbalance ratio, so that majority class voting obtains a score of 50%.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The datasets have widely varying class imbalances, ranging from 6% to 91% positive samples. Therefore, we select balanced accuracy (Brodersen et al., 2010) as the primary evaluation metric for the SUMMAC Benchmark. Balanced accuracy is defined as:
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Where T P stands for true positive, F P false positive, T N true negative, and F N false negative. The choice of metric is based on the fact that accuracy is a conceptually simple, interpretable metric, and that adjusting the class imbalance out of the metric makes the score more uniform across datasets. The balanced accuracy metric requires models to output a binary label (i.e., not a scalar score), which for most models requires the selection of a threshold in the score. The threshold is selected using the validation set, allowing for a different threshold for each dataset in the benchmark. Performance on the benchmark is the unweighted average of performance on the six datasets.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We choose Area Under the Curve of the Receiver Operating Chart (ROC-AUC) as a secondary evaluation metric, a common metric to summarize a classifier's performance at different threshold levels (Bradley, 1997).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We compared the SUMMAC models against a wide array of baselines and state-of-the-art methods.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We evaluated the following models on the SUMMAC Benchmark:
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
NER Overlap uses the spaCy named entity recognition (NER) model (Honnibal et al., 2020) to detect when an entity present in the summary is not present in the document. This model, adapted from Laban et al. (2021), considers only a subset of entity types as hallucinations (PERSON,  LOCATION, ORGANIZATION, etc.) MNLI-doc is a RoBERTa (Liu et al., 2019) model finetuned on the MNLI dataset (Williams et al., 2018). The document is used as the premise and the summary as a hypothesis, and we use the predicted probability of entailment as a score, similar to prior work on using NLI models for inconsistency detection (Kryscinski et al., 2020).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
FactCC-CLS is a RoBERTa-base model finetuned on the synthetic training portion of the FactCC dataset. Although trained solely on artificially created inconsistent summaries, prior work showed the model to be competitive on the FactCC and FRANK datasets.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
DAE (Goyal and Durrett, 2020) is a parsingbased model using the default model and hyperparameters provided by the authors of the paper. 2 FEQA (Durmus et al., 2020) is a QAG method, using the default model and hyper-parameters provided by the authors of the paper. 3 QuestEval (Scialom et al., 2021) is a QAG method taking both precision and recall into account. We use the default model and hyperparameters provided by the authors of the paper. 4 The model has an option to use an additional question weighter, however experiments revealed that the weighter lowered overall performance on the validation portion of the SUMMAC Benchmark, and we compare to the model without weighter.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Balanced accuracy results are summarized in Table 2. We find that the SUMMAC models achieve the two best performances in the benchmark. SUMMAC CONV achieves the best benchmark performance at 74.4%, 5 points above QuestEval, the best method not involving NLI.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Looking at the models' ability to generalize across datasets and varying scenarios of inconsistency detection provides interesting insights. For example, the FactCC-CLS model achieves strong performance on the FactCC dataset, but close to lowest performance on FRANK and XSumFaith. In comparison, SUMMAC model performance is strong across the board.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The strong improvement from the SUMMAC ZS to SUMMAC CONV also shines a light on the importance of considering the entire distribution of document scores for each summary sentence, instead of taking only the maximum score: The SUMMAC CONV model learns to look at the distribution and makes more robust decisions, leading to gains in performance.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The  Table 2: Performance of Summary Inconsistency Detection models on the test set of the SUMMAC Benchmark. Balanced accuracy is computed for each model on the six datasets in the benchmark, and the average is computed as the overall performance on the benchmark. We obtain confidence intervals comparing the SUMMAC models to prior work: * indicates an improvement with 95% confidence, and ** 99% confidence (details in Section 5.2.1). The results of the throughput analysis of Section 5.2.2 are in column Doc./min (Documents per minute).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
is included in Appendix A2, echoing the trends seen with the balanced accuracy metric.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We aim to determine whether the performance improvements of the SUMMAC models are statistically significant. For each dataset of the benchmark, we perform two tests through bootstrap resampling (Efron, 1982), comparing each of the SUMMAC models to the best-performing model from prior work. We perform interval comparison at two significance level: p = 0.05 and p = 0.01, and apply the Bonferroni correction (Bonferroni, 1935) as we perform several tests on each dataset. We summarize which improvements are significant in Table 2, and perform a similar testing procedure for the ROC-AUC results in Table A2. SUMMAC models lead to a statistically significant improvement on CoGenSumm, XSumFaith, FactCC, and SummEval. QuestEval outperforms the SUMMAC models on Polytope at a confidence of 95%. On the FRANK dataset, QuestEval and SUMMAC CONV achieve highest performance with no statistical difference. Overall on the benchmark, both SUMMAC models significantly outperform prior work, SUMMAC ZS at a p = 0.05 significance level and SUMMAC CONV at p = 0.01.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Computational cost of the method is an important practical factor to consider when choosing a model to use, as some applications such as training with a generator with Reinforcement Learning might require a minimum throughput from the model (i.e., number of documents processed by the model per unit of time).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
A common method to compare algorithms is using computational complexity analysis, computing the amount of resources (time, space) needed as the size of the input varies. Computational complexity analysis is impractical in our case, as the units of analysis differ between models, and do not allow for a direct comparison. More specifically, some of the models' complexity scales with the number of sub-word units in the document (MNLI-doc, FactCC-CLS), some with the number of entities in a document (NER-Overlap, DAE, QuestEval), and some with number of sentences (the SUMMAC models).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We instead compare models by measuring throughput on a fixed dataset using a common hardware setup. More precisely, we measured the processing time of each model on the 503 documents in the test set of FactCC (with an average of 33.2 sentences per document), running a single Quadro RTX 8000 GPU. For prior work, we used implementation publicly released by the authors, and made a best effort to use the model at an appropriate batch size for a fair comparison.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The result of the throughput analysis is included in Table 2 (column Docs./min.). SUMMAC models are able to process around 430 documents per minute, which is much lower than some of the baselines capable of processing more than 10,000 documents per minute. However, QAG methods are more than 10 times slower than SUMMAC models, processing only 20-40 documents per minute.  Table 3: Effect of NLI model choice on SUMMAC models performance. For each NLI model, we include the balanced accuracy scores of SUMMAC ZS and SUMMAC CONV . BERT X corresponds to a BERT or other pre-trained models of similar size.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We now examine how different components and design choices affect SUMMAC model performance.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
SUMMAC models rely on an NLI model at their core, which consists of choosing two main components: a model architecture, and a dataset to train on. We investigate the effect of both of these choices on the performance of SUMMAC models on the benchmark. Regarding model architectures, we experi ment with the decomposable attention model (Parikh et al., 2016), which is a pre-Transformer architecture model that was shown to achieve high performance on SNLI, as well as Transformer base and Transformer Large architectures.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
With respect to datasets, we include models trained on standard NLI datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018), as well as more recent datasets such as Adversarial NLI (Nie et al., 2019) and Vitamin C (Schuster et al., 2021).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Results are summarized in Table 3, and we emphasize three trends. First, the low performance of the decomposable attention model used in experiments in prior work (Falke et al., 2019) confirms that less recent NLI models did not transfer well to summary inconsistency detection.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Second, NLI models based on pre-trained Transformer architectures all achieve strong performance on the benchmark, with an average increase of 1.3 percentage points when going from a base to a large architecture.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Third, the choice of NLI dataset has a strong influence on overall performance. SNLI leads to lowest performance, which is expected as its textual domain is based on image captions, which are dissimilar to the news domain. MNLI and Vitamin C trained models both achieve close to the best performance, and training on both jointly leads to the best model, which we designate as the default NLI model for the SUMMAC models (i.e., the model included in Table 2).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The latter two trends point to the fact that improvements in the field of NLI lead to improvements in the SUMMAC models, and we can expect that future progress in the NLI community will translate to gains of performance when integrated into the SUMMAC model.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We relied on trained models available in Hug-gingFace's Model Hub (Wolf et al., 2020). Details in Appendix A.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The NLI task is a three-way classification task, yet most prior work has limited usage of the model to the use of the entailment probability for inconsistency detection (Kryscinski et al., 2020;Falke et al., 2019). We run a systematic experiment by training multiple SUMMAC CONV models that have access to varying subsets of the NLI labels, and measure the impact on overall performance. Results are summarized in Table 4. Using solely the entailment category leads to strong performance for all models. However, explicitly including the contradiction label as well leads to small boosts in performance for the ANLI and MNLI models.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
With future NLI models being potentially more nuanced and calibrated, it is possible that inconsistency detector models will be able to rely on scores from several categories.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
So far, we've reported experiments primarily with a sentence-level granularity, as it matches the granularity of NLI datasets. One can imagine cases where sentence-level granularity might be limiting. For example, in the case of a summary performing a sentence fusion operation, an NLI model might not be able to correctly predict entailment of the fused sentence, seeing only one sentence at a time.   Table 5: Effect of granularity choice on SUMMAC models performance. We tested four granularities on the document side: full, paragraph, two sentence, and sentence, and two granularities on the summary side: full and sentence. Performance of the four models is measured in balanced accuracy on the benchmark test set.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
To explore this facet further, we experiment with modifying the granularity of both the document and the summary. With regard to document granularity, we consider four granularities: (1) full text, the text is treated as a single block, (2) paragraph-level granularity, the text is separated into paragraph blocks, (3) two-sentence granularity, the text is separated into blocks of contiguous sentences of size two (i.e., block 1 contains sentence 1-2, block 2 contains sentence 3-4), and (4) sentence-level, splitting text at individual sentences. For the summary granularity, we only consider two granularities: (1) full text, and (2) sentence, because other granularities are less applicable since summaries usually consist of three sentences or fewer.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We study the total of 8 (document, summary) granularity combinations with the two best-performing NLI models of Table 2: MNLI and Vitamin C, each included as SUMMAC ZS and SUMMAC CONV models. 5 Results for the granularity experiments are summarized in Table 5. Overall, finer granularities lead to better performance, with (sentence, sentence) and (two sent, sentence) achieving highest performance across all four models.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
The MNLI-only trained model achieves lowest performance when used with full text granularity on the document level, and performance steadily increases from 56.4% to 73.5% as granularity is made finer both on the document and summary side. Results for the MNLI+VitaminC model vary less with changing granularity, showcasing that the model is perhaps more robust to different granularity levels. However the (two sent, sentence) and (sentence,sentence) settings achieve highest performance, implying that finer granularity remains valuable.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
For all models, performance degrades in cases where granularity on the document level is finer than summary granularity. For example the (sentence, full) or (two sent, full) combinations lead to some of the lowest performance. This is expected, as in cases in which summaries have several sentences, it is unlikely that they will fully be entailed by a single document sentence. This implies that granularity on the document side should be coarser or equal the summary's granularity.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Overall, we find that finer granularity for the document and summary is beneficial in terms of performance and recommend the use of a (sentence, sentence) granularity combination.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Improvements on the Benchmark. The models we introduced in this paper are just a first step towards harnessing NLI models for inconsistency detection. Future work could explore a number of improvements: combining the predictions of multiple NLI models, or combining multiple granularitiy levels-for example, through multi-hop reasoning (Zhao et al., 2019).
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Interpretability of Model Output. If a model can pinpoint which portion of a summary is inconsistent, some work has shown that corrector models can effectively re-write the problematic portions and often remove the inconsistency . Furthermore, fine-grained consistency scores can be incorporated into visual analysis tools for summarization such as Summ-Viz (Vig et al., 2021). The SUMMAC ZS model is directly interpretable, whereas the SUMMAC CONV is slightly more opaque, due to the inability to trace back a low score to a single sentence in the document being invalidated. Improving the interpretability of the SUMMAC CONV model is another open area for future work.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Beyond News Summarization. The six datasets in the SUMMAC Benchmark contain summaries from the news domain, one of the most common application of summarization technology. Recent efforts to expand the application of summarization to new domains such as legal (Kornilova and Eidelman, 2019) or scholarly (Cachola et al., 2020) text will hopefully lead to the study of inconsistency detection in these novel domains, and perhaps even out of summarization on tasks such as text simplification, or code generation.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
Towards Consistent Summarization. Inconsistency detection is but a first step in eliminating inconsistencies from summarization. Future work can include more powerful inconsistency detectors in the training of next generation summarizers to reduce the prevalence of inconsistencies in generated text.
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
We introduce SUMMAC ZS and SUMMAC CONV , two NLI-based models for summary inconsistency detection based on the key insight that NLI models require sentence-level input to work best. Both models achieve strong performance on the SUM-MAC Benchmark, a new diverse and standardized collection of the six largest datasets for inconsistency detection. SUMMAC CONV outperforms all prior work with a balanced accuracy score of 74.4%, an improvement of five absolute percentage points over the best baseline. To the best of our knowledge, this the first successful attempt at adapting NLI models for inconsistency detection, and we believe that there are many exciting opportuni-ties for further improvements and applications of our methods. Table A1 measures the effect of the choice of the two operators in the SUMMAC ZS model. We explore three options (min, mean, and max) for each operator. We find that the choice of max for Operator 1 and mean for Operator 2 achieves the highest performance and use these choices in our model. Table A2 details results of models on the benchmark according to the ROC-AUC metric, confirming that the SUMMAC models achieve the two best accuracy results on the benchmark. 
QuestEval: Summarization Asks for Fact-based Evaluation
The reliability of automatic evaluation metrics is an important factor for progress in artificial intelligence tasks, enabling the comparison and improvement of proposed systems. The design of reliable metrics for natural language generation (NLG) systems is very challenging, and still an open research problem: Novikova et al. (2017); Peyrard (2019) showed that current metrics do not correlate well with human judgments, and argued for the development of new evaluation metrics.
QuestEval: Summarization Asks for Fact-based Evaluation
Among NLG tasks, summarization is one of the most difficult to evaluate automatically. For a given document, the number of possible correct outputs is much larger than for other tasks such as machine translation. Thus, when only a single reference summary is given -as is typically the case for large-scale summarization datasets, the correlation of standard automatic evaluation metrics with human judgments is low (Louis and Nenkova, 2013). Furthermore, since a summary must be shorter than the corresponding source document, information selection (Li et al., 2018) is critical so that the summary only contains the salient contents from its source document. For these reasons, n-gram based metrics, such as ROUGE (Lin, 2004), are known to poorly reflect human preference (Louis and Nenkova, 2013;Novikova et al., 2017;Paulus et al., 2017;Bhandari et al., 2020). Finally, it is crucial for reliable summarization to generate texts that are factually consistent with their source documents. However, this aspect is not measured by n-grams based metrics. Notably, while recent stateof-the-art generative models (Lewis et al., 2019;Zhang et al., 2019a) produce fluent summaries, they frequently contain false or unsupported information (Kryściński et al., 2019), a phenomenon also known as neural hallucination (Rohrbach et al., 2018;Zhao et al., 2020).
QuestEval: Summarization Asks for Fact-based Evaluation
To overcome these limitations, a new approach to evaluate summarization systems has recently emerged, based on question generation (QG) and answering (QA) (Chen et al., 2017;Scialom et al., 2019;Eyal et al., 2019). These metrics measure to which extent a summary provides sufficient information to answer questions posed on its corresponding source document. They can be used to assess the factual consistency (i.e. precision) (Durmus et al., 2020;Wang et al., 2020) or the relevance (i.e. recall) (Scialom et al., 2019) of the evaluated summary, with respect to its source document. Although these works have introduced an interesting and novel method to evaluate summarization, with encouraging preliminary results, none of those metrics is found to perform better than ROUGE (Fabbri et al., 2020): automatic evaluation of summarization systems remains an open research problem (Kryscinski et al., 2019).
QuestEval: Summarization Asks for Fact-based Evaluation
Inspired by these works, and motivated to take  Figure 1: Illustration of the QuestEval framework: the blue area corresponds to the precision-oriented framework proposed by Wang et al. (2020). The orange area corresponds to the recall-oriented SummaQA (Scialom et al., 2019). We extend it with a weighter component for an improved recall (red area). The encompassing area corresponds to our proposed unified approach, QuestEval.
QuestEval: Summarization Asks for Fact-based Evaluation
up the challenge of summarization evaluation, we propose QuestEval, a new reference-less evaluation metric, which is found to correlate dramatically better with humans judgments. Our contributions are as follows:
QuestEval: Summarization Asks for Fact-based Evaluation
• We show that, by unifying the precision and recall-based QA metrics, we obtain a more robust metric;
QuestEval: Summarization Asks for Fact-based Evaluation
• We propose a method to learn the saliency of the generated queries, allowing to integrate the notion of information selection;
QuestEval: Summarization Asks for Fact-based Evaluation
• We evaluate QuestEval on two corpora containing annotated summaries from CNN/Daily Mail (Nallapati et al., 2016) and XSUM (Narayan et al., 2018) datasets. The proposed metric obtains state-of-the-art results in terms of correlation with humans judgments, over all the evaluated dimensions. Notably, QuestEval is effective at measuring factual consistency, a crucial yet challenging aspect for summarization.
QuestEval: Summarization Asks for Fact-based Evaluation
This paper introduces the QuestEval framework for evaluating summarization systems, that accounts for both factual consistency and relevance of the generated text, without requiring any human reference. QuestEval consists of a QG component Q G and a QA component Q A , described in this section and depicted in Figure 1.
QuestEval: Summarization Asks for Fact-based Evaluation
Recently, there has been significant progress on factoid question answering, with models obtaining human-level performance on benchmarks such as SQuAD (Rajpurkar et al., 2016). Leveraging on these advancements, our Q A component consists of a pretrained T5 model (Raffel et al., 2019), which extracts answers from a source document given the document and a question to answer. In the following, we refer to Q A (r|T, q) as the probability of the answer r to question q on a text T , and Q A (T, q) as the answer greedily generated from the model. When a summary is evaluated, there is no guarantee that it contains the answer. Therefore, it is crucial for the QA model to be able to predict when a question is unanswerable. Our Q A component thus includes the unanswerable token, that we denote , among its possible outputs.
QuestEval: Summarization Asks for Fact-based Evaluation
For the QG component, we draw on recent work on neural answer-conditional question generation (Zhou et al., 2017). The component also consists of a T5 model, finetuned to maximize the likelihood of human questions, given the corresponding answer and source document.
QuestEval: Summarization Asks for Fact-based Evaluation
At test time, given a source document or generated summary, we first select a set of answers from the text to condition the QG model on. Following Wang et al. (2020), we consider all the named entities and nouns from the source document as answers. Then, for each selected answer, we generate a question via beam search. 1 We filter out every question for which the QA model predicts an incorrect answer. Based on this, we denote Q G (T ) the set of question-answer pairs (q, r) for a text T such that Q A (T, q) = r.
QuestEval: Summarization Asks for Fact-based Evaluation
In the following, D and S are two sequences of tokens with D denoting the source document and S the corresponding evaluated summary.
QuestEval: Summarization Asks for Fact-based Evaluation
A summary is deemed inconsistent with respect to its source text if, given a question, the answer differs when conditioned on S or D. Therefore, we define the precision score for the evaluated summary as:
QuestEval: Summarization Asks for Fact-based Evaluation
The F1 score is a standard metric for evaluating factoid question answering models, and measures the overlap between the predicted answer and the corresponding ground truth. It outputs 1 for an exact match between both answers and 0 if there is no common token. This definition of factual consistency corresponds to the frameworks concurrently proposed by Wang et al. (2020) and Durmus et al. (2020).
QuestEval: Summarization Asks for Fact-based Evaluation
While a summary should contain only factual information (precision), it should also contain the most important information from its source text (recall). Extending Scialom et al. (2019) by introducing a query weighter W , we define recall as:
QuestEval: Summarization Asks for Fact-based Evaluation
where Q G (D) is the set of all question-answer pairs for the source text D, and W (q, D) is the weight of query q for text D.
QuestEval: Summarization Asks for Fact-based Evaluation
Answerability and F1 Factoid question answering models are commonly evaluated using F1 score, measuring the overlap between the predicted answer and the corresponding ground truth (Rajpurkar et al., 2016). However, an answer could be correctly expressed in different ways, e.g. "ACL" and "Association for Computational Linguistics". Unfortunately, the F1 score is 0 in this example.
QuestEval: Summarization Asks for Fact-based Evaluation
To sidestep this issue, Scialom et al. (2019) use the QA confidence of answerability, i.e. 1 − Q A ( ), rather than F1 score. Defining recall this way allows to measure answerability independently of the way the answer is expressed, but does not take into account possible model hallucinations, i.e. the summary could answer the question incorrectly.
QuestEval: Summarization Asks for Fact-based Evaluation
Conversely, when we assess factual consistency, it is not enough for a question from the summary to be answerable from the source document. The two answers to this question should also share the same meaning to be factually consistent. While using answerability allows for more true positive (e.g. "ACL"), for precision it is crucial to detect true negatives. This motivates our use of the F1 score in this case, similar to Wang et al. (2020).
QuestEval: Summarization Asks for Fact-based Evaluation
Query Weighting In Scialom et al. (2019), all questions are considered equally important, i.e. the weight W (q, D) = 1 for every query q ∈ Q G (D). However, since a summary necessarily has a constrained length, an effective summary should contain the most important information from the source. To account for this, we introduce a question weighter, which is trained to distinguish important questions from anecdotal ones. We leverage existing summarization datasets to create training data for the weighter: given a source document D, each question q ∈ Q G (D) is labeled as important if the corresponding human summary contains the answer, as computed by the QA component applied on the summary (i.e. Q A (S, q) = ).
QuestEval: Summarization Asks for Fact-based Evaluation
W (q, D) denotes the probability that q is important for D. Note that the question weighter only concerns recall, and therefore is not applied when computing precision.
QuestEval: Summarization Asks for Fact-based Evaluation
The final QuestEval score accounts for both the precision and recall by computing their harmonic mean (i.e. the F-Score): 2 P rec.Rec P rec+Rec . The QuestEval score is thus directly comparable with existing evaluation metrics, such as ROUGE or BLEU, as it lies in the same numerical range.
QuestEval: Summarization Asks for Fact-based Evaluation
To evaluate QuestEval, we measure its correlation with human judgments on different datasets:
QuestEval: Summarization Asks for Fact-based Evaluation
SummEval Released by Fabbri et al. (2020), it is one of the largest human-annotated datasets for summarization. Derived from CNN/Daily Mail (Nallapati et al., 2016), it consists of 12.800 summary level annotations. To ensure diversity, the summaries were generated from 16 different summarization models, including extractive and abstractive architectures. To ensure quality, three experts annotated four dimensions: i) Consistency: the proportion of facts in the summary corresponding to facts in the original text; ii) Coherence: how well-structured and well-organized is the summary; iii) Fluency: how fluent the summary is to read; and, iv) Relevance: the ratio between important and excess information in the summary. 2 Wang et al. (2020) made available a subset of 239 BART outputs (Lewis et al., 2019) fine-tuned on XSUM (Narayan et al., 2018). 3 Three annotators measured the "correctness" of each summary, which corresponds to consistency in SummEval.
QuestEval: Summarization Asks for Fact-based Evaluation
To train our Q G and Q A models, we used two factoid question answering datasets: SQuAD-v2 (Rajpurkar et al., 2018) and NewsQA (Trischler et al., 2016). Such datasets are composed of (paragraph, question, answer) triplets. SQuAD-v2 provides unanswerable questions, while NewsQA is composed of news articles, corresponding to our summarization domain. Note that QG can be seen as the dual task for QA. Any QA dataset can be reversed into a QG dataset, by switching the generation target from the answer to the question. Lastly, we found it helpful to train our QA model using additional synthetic unanswerable questions. This is done by considering a shuffled version of the dataset, where each question is randomly assigned to a paragraph from another triplet of the dataset. We consider these additional samples, with flipped contexts, as unanswerable. All experiments, except otherwise specified, use this additional negative sampling process to improve identification of unanswerable queries.
QuestEval: Summarization Asks for Fact-based Evaluation
As baselines, we considered the following: N-gram based ROUGE (Lin, 2004) is the most widely used evaluation in summarization. This metric measures the recall of reference n-grams in the evaluated summary. Conversely, BLEU (Papineni et al., 2002) computes the precision of summary n-grams in the references. METEOR (Lavie and Agarwal, 2007) is a variant that uses stemming, synonyms and paraphrastic matches.
QuestEval: Summarization Asks for Fact-based Evaluation
Neural based Leveraging recent progress in language modeling, Zhang et al. (2019b) proposed BERTScore: for each token of the summary, the maximum cosine similarity is computed over con-textualized token embeddings of the reference summary, and the overall mean is reported.
QuestEval: Summarization Asks for Fact-based Evaluation
Question based SummaQA (Scialom et al., 2019) is a recall oriented metric, with questions generated from the source document. QAGS (Wang et al., 2020) is a precision oriented metric, with questions generated from the summary.
QuestEval: Summarization Asks for Fact-based Evaluation
In Tables 1 and 2 we report the results for QuestEval, along with several ablations. W = unif orm corresponds to setting all questions weights equal. Conversely, W = learned corresponds to the weights learned as detailed in Section 4.2. We also report the recall and precision component separately.
QuestEval: Summarization Asks for Fact-based Evaluation
In Table 1, we observe that, amongst existing metrics, BERTScore achieves the best average Pearson correlation with human judgements (23.1), slightly above ROUGE-1 (22.2) and BLEU (22.2). These correlations are obtained when providing no less than 11 gold references, and averaging results. Given a single reference, all these correlations are halved. Most of the large scale datasets provide only one reference per example in their test set (e.g. CNN/Daily Mail and XSUM), a fact that 
QuestEval: Summarization Asks for Fact-based Evaluation
Recall Only 13.9 highlights the importance of searching for more reference-efficient alternatives. With regards to sample efficiency, QA-based metrics do not require any references. We expect Relevance to be better measured by Recall oriented metrics, and less so for Consistency. This is confirmed in the results, where SummaQA correlates better with Relevance than Consistency (26.2 vs 8.3), and vice versa for QAGS (9.1 vs 20.4). By unifying and extending the two, QuestEval allows to take both dimensions into account, improving the average correlation by 18% (28.4 to 33.5).
QuestEval: Summarization Asks for Fact-based Evaluation
The dimension that benefits the most from the learned question weighter is Relevance (+4%, from 37.5 to 39.2), indicating that our classifier learns which questions target important information. We discuss this aspect more in depth in the following section.
QuestEval: Summarization Asks for Fact-based Evaluation
Finally, compared to the other metrics, the improvement is remarkable (33.5 vs 11.8 for BERTScore), and allows safer evaluations of the systems while not even requiring references.
QuestEval: Summarization Asks for Fact-based Evaluation
Reference-less One of the main limitations for the current metrics is that they require gold references to compute similarity scores. However, many possible summaries are valid for one source doc-important answered Relevance Corr. 37.6 -33.5 -5.7 Table 3: Pearson correlation coefficients between human judgments (for Relevance) and the percentage of important and/or answered questions, on SummEval data.
QuestEval: Summarization Asks for Fact-based Evaluation
ument. We argue that the universe of correct outputs is much larger than in other generation tasks such as machine translation. This explains why the correlations with humans is largely reduced when computed with one reference instead of 11 (see Table 1: BERTScore-f drops from 23.1 to 11.8 in average, and other metrics likewise). Unfortunately, assuming the availability of as many as 11 gold references is not realistic in most scenarios, due to the cost of obtaining reference summaries.
QuestEval: Summarization Asks for Fact-based Evaluation
To complement Table 1, we report in Figure 2 the correlations for the best baselines as we progressively decrease the number of available gold references from 11 to 1. We observe that for all four dimensions and all the baselines, the correlations decrease and the variance increases as the number of references decreases. However, QuestEval does not require any reference. Therefore, the improvement over the other metrics grows larger as the number of references used decreases. Furthermore, QuestEval enables the evaluation of systems on datasets even if no gold reference is available.
QuestEval: Summarization Asks for Fact-based Evaluation
Query Weighter There is no unique answer to the question "What makes a good summary?": it depends on the reader's point of view, which makes summarization evaluation challenging. For instance, given a contract, the seller and the buyer could be interested in different information within the same document.
QuestEval: Summarization Asks for Fact-based Evaluation
In this paper, to instantiate the weighter W , we propose to learn a specific dataset policy: "what kind of questions are likely answered in the CNN/Daily Mail training summaries?" This is a reasonable heuristic given that editors created the summaries following their specific policy.
QuestEval: Summarization Asks for Fact-based Evaluation
To demonstrate the effectiveness of the weighter, we proceed as follows. We first consider that a question q ∈ Q G (D), generated on the source document, is important if the probability given by the query weighter is above a threshold, i.e. if W (D, q) > 0.5. We then say that a question is answered if the probability of being unanswerable is below a threshold, i.e. Q A ( |S, q) < 0.5. Therefore, a question can belong to one of four folds, given the two above criteria (important and/or answered). In Table 3, we measure how the percentage of questions belonging to a specific fold correlates with the Relevance dimension for each generated summary on SummEval. We observe that the percentage of questions that are important and answered correlates positively with Relevance, as opposed to the percentage of questions that are important but not answered. Finally, the percentage of questions that are answered but not important does not correlate with Relevance. It indicates that our proposed approach is able to learn what are the questions that should be asked or not.
QuestEval: Summarization Asks for Fact-based Evaluation
We emphasize that W is a flexible component of our framework. It can be adapted to specific domains and applications. For instance, one could design a specific W , to focus the evaluation on information about specific entities, such as people or events.   The evaluated samples belong to three distinct categories: 1) answerable, 2) unanswerable questions (but present in SQuAD-v2) and 3) the negatively sampled ones (as described in Section 5.1).
QuestEval: Summarization Asks for Fact-based Evaluation
An Explainable Metric One important feature of QuestEval is its explainability. It is straightforward to investigate 1) what are the important points not answered in the summary and 2) what are the inconsistencies between the source document and the summary. We illustrate this in Table 4, with a source document, from which a question q is generated and answered. According to the weighter W , q is categorized as important. Three evaluated summaries are then shown.
QuestEval: Summarization Asks for Fact-based Evaluation
The first summary S correct is factually consistent with the source document: the predicted answer Q A (S correct , q) corresponds to the source document answer Buckingham Palace. The second summary S hallu is factually inconsistent with the source document: the predicted answer Q A (S hallu , q) does not correspond to Buckingham Palace. Finally, the third summary S incomplete does not answer the question, i.e. Q A (S incomplete , q) = , and is thus incomplete.
QuestEval: Summarization Asks for Fact-based Evaluation
Negative Sampling Effect In Tables 1 and 2, when QuestEval uses a QA model trained without negative sampling (see Section 5.2), we observe a decrease of performance, from 33.3 to 32.4 on SummEval and from 30.4 to 28.5 on QAGS-XSUM.
QuestEval: Summarization Asks for Fact-based Evaluation
In Figure 3, we report the distribution of the log probabilities for the two QA models, trained with and without negative sampling. We can observe that the QA model exposed to the negative sampling during training, has learned to separate better the negative sampled questions (for negative, i.e. red lines, the dashed line is more on the left than the solid line).
QuestEval: Summarization Asks for Fact-based Evaluation
Indeed, the unanswerable questions of SQuAD-v2 were written adversarially by crowd-workers, to look similar to answerable ones. However, in the context of QuestEval, unanswerable questions are not adversarial. It simply often happens that the summary does not contain the answer. Therefore, QuestEval sees unanswerable questions that look like the one we built trough our negative sampling method, rather than the adversarial ones. This may explain the improvement of a QuestEval with a QA model trained with negative sampling.  Computational Complexity Following Wang et al. (2020), we generate the questions with K = 20 beams during decoding and we keep all the different versions of the questions in the latter steps, which improves correlations. However, the downside of this is the inference time which increases linearly w.r.t the beam size. To be widely adopted, a metric should not only correlate with human judgment, but also be computationally efficient. In Figure 4 we show the variation of the average correlation with respect to the beam size. The improvement from K = 1 to K = 20 is small (34.4 to 35.6), and the rank order for the different systems remains unchanged. Therefore, we believe that using QuestEval with K = 1 is a reasonable choice, allowing for fast computation while preserving the correlation with human judgments.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Text summarization aims to compress long document(s) into a short and fluent form that preserves salient information. The field has benefited from the application of pretrained methods (Liu and Lapata, 2019;Lewis et al., 2020;Zhang et al., 2020a). However, state-of-the-art models are not always factually consistent with the source documents they are conditioned on (Maynez et al., 2020;Fabbri et al., 2021). Thus, determining the factual consistency of a summary remains an essential task.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Recent metrics for summarization factual consistency can be broadly split into two categories: 1) Entailment-based metrics that determine whether the content in the summary is entailed by the input document (Kryscinski et al., 2020;Koto et al., Document The Knicks beat the Rockets. The fans were excited.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Summary The Knicks beat the Bucks.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Selected Answer [Contra, Neutral, Support] the Bucks 0.90 0.07 0.03 0.02 0.90 0.08
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Generated Question Who did the Knicks beat?
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
QA Output the Rockets Max Support Score Answer Overlap Score 0.08 0.20 Table 1: Toy example of a factual inconsistency between a summary and a source document. Left: The entailment-based metric computes the level of contradiction, neutrality, and support between the summary and each source document sentence. The final factual consistency metric is calculated as the maximum support score over all source sentences. Right: The QA-based metric first selects a noun-phrase answer from the summary. A QG model then generates an associated question that a QA model answers based on the source document. The answer overlap score of the QA-based metric measures the semantic overlap between the QA model output and the selected answer as the final metric score. 2020) and 2) QA-based metrics that compute a factual consistency score based on a QA model's ability to answer, using the input document, questions generated from the summary (Wang et al., 2020a;Durmus et al., 2020). We provide an illustrative example in Table 1 in which both metric types correctly identify the factual inconsistency and output a low score.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Quantitative comparisons among entailmentbased and QA-based metrics, however, often differ in their choices of baseline model and input granularity, evaluating on single datasets and drawing differing conclusions as to the best paradigm. For example, some work reports entailment-based metrics as performing best (Koto et al., 2020;Maynez et al., 2020), while other work argues for QA metrics (Durmus et al., 2020;Wang et al., 2020b;Scialom et al., 2021). Recently, Laban et al. (2021) pro-posed a benchmark called SummaC to compare metrics across six factual consistency datasets for the task of binary factual consistency classification, whether a summary is entirely factually consistent or not. This work unifies prior work on entailment-based metrics by studying the effect of input granularity, pretrained entailment model, and other hyperparameter choices on downstream evaluation performance. However, it does not study the components of QA-based metrics, which are more interpretable by their inherent decomposability.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
To unify work in QA-based factual consistency evaluation, we do an extensive hyperparameter analysis of current metrics. We break down these metrics into four constituent components: 1) the selection of answers to ask questions about, 2) question generation (QG) conditioned upon these answers, 3) question answering (QA) based on the source document, and 4) answer overlap evaluation between QA model output and selected answers. We study the effect of each of these components on metric performance. Based on our insights, we propose an optimized metric, which we call QAFACTEVAL , that outperforms the entailmentbased metrics of Laban et al. (2021).
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Our contributions are the following: 1) We analyze all components of the QA-based metric pipeline, and our proposed solution improves performance over prior QA-based metrics by over 14% on a factual consistency benchmark consisting of 6 individual datasets, achieving state-of-the-art results. 2) We show that QA-based metrics and NLI-based metrics offer complementary signals and combine them into a new metric via a simple learned network, further improving performance. 3) We report results for 10 additional metrics across classification and correlation analysis, providing the most comprehensive benchmark results for factual consistency metrics and highlighting areas for future work in QA-based metrics 1 .
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
In this section, we introduce the factual consistency metrics studied, which we divide into entailment metrics, QA-based metrics, and learned metrics.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We include the following entailment-based metrics due to further understand differences in granularity and base entailment models. The metrics below produce a score for each summary sentence that is then averaged to compute the final metric score.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
MNLI applies a RoBERTa large  model trained on MNLI (Williams et al., 2018). The score of a summary sentence is the maximum entailment score over all input sentences.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
ANLI Barrantes et al. (2020) uses the same method as the MNLI metric with a model trained on the ANLI (Nie et al., 2020) dataset consisting of adversarial datapoints.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
SCZeroShot Laban et al. (2021) works analogously to the above metrics with a base model trained on both MNLI and Vitamin-C data (Schuster et al., 2021), consisting of closely-related contrastive entailment examples.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
BertScore (Zhang et al., 2020b) with a backbone RoBERTa-MNLI model, averaging the three highest BertScore F1 scores over the input sentences.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
DAE Goyal and Durrett (2020) computes entailment scores between a source document and summary dependency arcs, applying an entailment model trained on synthetic data. 
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We now describe the components that constitute the QA-based pipeline for factual consistency. We refer to our metric, consisting of the best combination of the below components, as QAFACTEVAL.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Answer Selection QA-based metrics compare information units between the summary and source, so it is thus necessary to first extract such units, or answers, from the given summary. We follow the protocols from Deutsch et al. (2020) and compare extracting the following answer types: named entities (NER), noun phrase chunks (NP Chunks), maximally sized noun phrases (Max NP), whereby the dependency subtrees of nouns reached by traversing a given sentence's dependency parse from the root are chosen as answers, and All, which combines answers from the above three techniques.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Question Generation Having selected answers, questions are generated conditioned upon these answers using the summary as context. Typically, this is an encoder-decoder model which inputs the answer and context separated by a special token. On the modeling side, we examine BART (Lewis et al., 2020) and T5 (Raffel et al., 2019) as the underlying generators. On the data side, we experiment with models trained for question generation on SQuAD (Rajpurkar et al., 2016), a standard QA dataset consisting of questions on Wikipedia articles, and on QA2D (Demszky et al., 2018), a dataset of declarative sentences with associated question/answer pairs derived from SQuAD. Furthermore, we experiment with the recently-introduced MixQG models (Murakhovs'ka et al., 2021), which are T5 models trained on a combination of nine QA datasets with diverse answer types and which outperform other QG models across several tasks. We apply both the small and large versions of MixQG to better understand the effect of QG model size.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
The QA component answers questions from the previous steps using the input document as context. We experiment with both extractive QA models, which extract a text span from the input as an answer, and abstractive QA models, which generate an answer token-bytoken. For extractive models, we ablate Electra (Clark et al., 2020), a model architecturally similar to BERT (Devlin et al., 2019) that achieves strong performance on the SQuAD 2.0 dataset and was previously used in measuring summary relevance (Deutsch et al., 2020). We also include MADE (Friedman et al., 2021), which models multi-dataset QA with a collection of dataset-specific adapter modules sharing the same underlying RoBERTabase model. For abstractive QA, we experiment with T5 fine-tuned on SQuAD and UnifiedQA (Khashabi et al., 2020), an approach that trains a T5 QA model on 8 diverse, seed datasets and was shown to generalize across 20 datasets and 4 input formats. All QA models except MADE are trained on data containing unanswerable questions. Additional QA models can be included, although the above set of models allows us to inspect the aspects of interest in this study, namely extractive vs abstractive performance and multi-dataset training.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Answer Overlap Evaluation An answer overlap metric must be computed to determine the match between the initial answer selected in the first component and the QA model output. Typically, answer overlap in QA is measured through exact match (EM) score or word F1 score. We also test a learned metric, the LERC score proposed by Chen et al. (2020). This metric outputs a 1-5 answer overlap score conditioned on a question and context. The scorer is trained on their MOCHA dataset, consisting of 40k crowdsourced judgments on QA model outputs. We include the BERT-base (Devlin et al., 2019) model from the original paper, which we call LERC (orig). We additionally experiment with two models trained from RoBERTa-large checkpoints, one trained from the original checkpoint, LERC (RoBERTa), and one initialized from Jia et al. (2021), which we call LERC (QuIP), for the task of jointly encoding passages and answers with question-infused pretraining. Lastly, we experiment with the IsAnsweredInput answer metric, which is a 0/1 score of whether the question is answerable using the input document according to the QA model. We use the Electra-large QA model to determine whether a question is answerable, as this model shows strong performance on identifying unanswerable questions on SQuAD.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Question Filtering Model-generated questions may contain noise from the QG model itself or from disfluencies in the summary the QG model conditions upon. Such noisy questions can skew the overall metric score, as the QA component may be unable to correctly answer the question, regardless of the summary's factual consistency. We filter such questions through a step called IsAnswered-Summ Filter: the same Electra-large QA model returns a 0/1 score of whether the question is answerable, now using the summary as context, and questions labeled as unanswerable are filtered.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Overall For a given question, if IsAnsweredInput returns 0, the question is unanswerable using the input, we label all the above answer overlap scores as 0, and otherwise use the answer overlap score. We refer to this scoring of unanswerable questions as 0 as the Answerability Penalty. We also experiment with not setting the overlap score of these unanswerable questions to 0 but rather using the answer overlap score of the most probable answer from the QA model. Finally, the overall factual consistency score for each metric is computed as its average scores over all questions remaining following Question Filtering.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
SCConv is a model introduced by Laban et al. (2021) that learns to aggregate entailment-model output scores across input sentences into a single score. More concretely, for a document consisting of M sentences and a summary consisting of N sentences, the entailment-based model produces an M × N matrix of entailment scores. The M × N matrix is then transformed to an H × N matrix by binning the M sentences to create a histogram, where H is the number of bins. This matrix is input to a 1-D convolution layer to produce a score for each summary sentence, and the scores are averaged across summary sentences. The parameters of this model are fine-tuned on synthetic data, detailed in Section 4.2 QAFACTEVAL-NLI While SCConv captures sentence-level support, QAFACTEVAL measures finer-grained answer overlap between the source and summary. Thus, we are able to combine these two into a single factual consistency metric, QAFACTEVAL-NLI. Assume that K answers are extracted from the summary. The pipeline described above will then output a single score per answer for the entire summary, resulting in an array of length K. We convert this to a histogram of size H in a similar manner as SCConv and pass this histogram through a 1-D convolution layer to produce a single QA score. This score is concatenated with the NLI score produced by SCConv and input to a linear layer to produce the final metric score. The linear layer can be trained in either synthetic or supervised ways, detailed in Section 4.2.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We include the following metrics for completeness.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
BARTScore Yuan et al. (2021) calculates the log-likelihood from BART fine-tuned on CNN/DailyMail (Hermann et al., 2015;Nallapati et al., 2016) of the summary conditioned upon the source text as a metric for factual consistency. BLANC Vasilyev et al. (2020) is a reference-less metric of summary quality that measures the difference in masked language modeling performance with and without access to the summary.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
QuestEval (Scialom et al., 2021) is the prior state-of-the-art QA-based metric for factual consistency. The T5-base (SQuAD) QG and T5-base QA models described above are applied directly from the QuestEval metric. QuestEval generates questions based on the input document and answers them using the summary in addition to following the above QA metric pipeline. QuestEval aggregates the score from these two pipelines. We believe that our described pipeline more closely measures factual consistency, while generating questions from the source may confound factual consistency with relevance.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We present the datasets explored for binary classification and correlation analyses. We also describe settings for reporting ablation and final results.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
The SummaC benchmark (Laban et al., 2021) introduces a collection of datasets for binary factual consistency evaluation. A data point is labeled as positive if it contains no factual inconsistencies or is rated the highest possible score in the case of Likert scaling, and as negative otherwise. We now briefly describe the datasets in the benchmark and any departures from the original benchmark, and additional datasets we use for correlation analysis. We refer the reader to Laban et al. (2021) for further details regarding the benchmark creation.  QAGs Wang et al. (2020b) crowdsource sentence-level summary annotations for factual consistency across CNN/Daily Mail and XSum data. We only report correlation analysis for this dataset as it was not a part of SummaC.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Metric Implementation Metrics were applied directly from the original GitHub repository or by using the SacreRouge Library (Deutsch and Roth, 2020), which was also used in correlation analysis. The learned metrics make use of code released from Laban et al. (2021) for training, and all models are implemented in PyTorch (Li et al., 2020) and in the Transformers library (Wolf et al., 2019). The BART-large (QA2D) QG and Electra-large QA models are applied from the QAEval relevance modeling metric (Deutsch et al., 2020).
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Ablation Settings Following Laban et al. (2021), a metric threshold score for binary classification is determined from the validation set of SummaC and applied to the test set. This threshold score is determined for every metric studied. Furthermore, we note that hyperparameter choices for several of the strong entailment baselines, namely SCConv, SCZeroShot, and MNLI are derived from Laban et al. (2021), thus providing a reasonable comparison to QAFactEval, whose hyperparameters we tune on the SummaC validation set. For ablation studies, we both perform thresholding and evaluation on the validation set to preserve the integrity of the test set. For each benchmark dataset, we sample a random subset of 80% of the validation set to determine the threshold and evaluate on the remaining 20% of the validation set. The best performing combination of QA metric components constitutes our QAFACTEVAL metric. We take the best performing combination of QA metric components and vary a given component, such as answer selection, while holding all other components constant and consistent with the best component combination.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Training Settings To tune the parameters of the learned metrics, we train on a subset of 50k synthetic data points from FactCC, following Laban et al. (2021). We name these runs synthetic setting due to the lack of human-labeled data. We also experiment with a supervised setting by fine-tuning the parameters on the SummaC validation set for each individual dataset, choosing the threshold on this validation data, and applying the model to the test set. Training on such a small amount of data is feasible due to the small number of parameters of the learned metrics. Cross entropy loss with Adam (Kingma and Ba, 2015) optimizer is used, with a batch size of 32 and a learning rate of 1e-2.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
In this section, we first study the effects of model component choices on QAFACTEVAL . We then compare metric results across both the SummaC binary classification task and correlation analysis.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We provide the results of our ablation studies on the components of QA-based metrics in Table 2 and show two illustrative examples in Table 4.  or ALL answers together. Named entity selection likely performs worse due to the scarcity of extracted answers; only three entities are extracted on average across the benchmark, while all other approaches extract over 10 answers per summary.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
The choice of the QG model notably affects downstream performance. BART-large (QA2D) works the best and produces much longer questions, about 17 tokens on average, versus about 10 from the other models. Deutsch et al. (2020) note how humans tend to produce shorter questions. However, longer questions may be preferable for this task to facilitate the QA model's ability to understand and answer the question. BART-large (QA2D) also is the most extractive, with only about 20% novel unigrams in the question, while T5-base (SQuAD) model is the most abstractive with about 47% novel unigrams, resulting in occasional hallucinations and questions that the QA model struggles to answer. As seen in Table 4, MixQG models do often produce highlyfluent questions, but the longer, highly-extractive output of BART-large (QA2D) improves downstream factual consistency performance.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Effect of QA Model Surprisingly, we do not find a large difference in the QA model compo-  They're not gonna take it anymore. Really. Twisted Sister says that its 2016 tour will be its last, according to a press release. ... The band will also perform two shows in Pero's honor: one at Las Vegas Hard Rock Hotel and Casino, the other at the Starland Ballroom in Sayreville, New Jersey.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
The band will perform two shows.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
BART-QA2D MixQG-large BART-QA2D Question What is Paul Merson not happy with to the England squad last week?
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
What is Paul Merson not happy with?
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Who will perform two shows?
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Townsend's call-up he shouldn't have been in the squad Unanswerable (Twisted Sister)
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Answer Overlap 1.00 0.30 0.00 (0.80) Table 4: Example source documents and summaries along with QA-based metric component outputs. Left: This example illustrates that the fluency of the QG model does not necessarily improve downstream factual consistency evaluation performance; the less fluent, more extractive BART-QA2D question is more-easily answerable by the QA model. Not shown, the entailment-based SCConv metric incorrectly labels this entity-centric example, likely due the introduction of novel unigrams. Right: The QA model incorrectly labels this question as unanswerable, perhaps due to the generality of the question or due to noise in the input document. The QA output and our learned overlap score if forced to extract an answer are in parenthesis. SCConv correctly labels this highly extractive example. nent across model sizes or between extractive and abstractive QA models, implying that QA ability is not the bottleneck of our task. In this setting, we keep IsAnsweredInput from Electra-large constant, as not all QA models are trained with unanswerable questions; thus the only differences are in the answers to questions marked as answerable.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We observe a large difference between EM and other overlap metrics. We also see a notable gap between LERC (orig) and LERC (RoBERTa) along with a further slight improvement with LERC (QuIP), showing the effect of the underlying model of the learned metric on factual consistency performance.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Effect of Question Filtering and Answerability Not filtering questions according to the QA model's ability to answer them conditioned upon the summary decreases performance. Furthermore, not applying the Answerability Penalty, and using the answer overlap score for the most probable answers for all questions, even those judged unanswerable by the QA model, also decreases perfor-  mance. While the answer overlap metric should capture unanswerable questions for information not found in the input (extrinsic error), the selected answer may appear in both the summary and source but in different contexts (intrinsic error). The QA model may return this as the most probable answer and be scored as correct by the answer overlap component despite a factual inconsistency. This finding demonstrates the importance of determining question answerability, a point also emphasized in Deutsch et al. (2020) for QA-based metrics of relevance. Removing both of these components results in a drastic performance decrease.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We present the results on the test set of SummaC in Table 3. QAFACTEVAL shows a substantial improvement over the previous state-of-the-art QA metric for factual consistency, QuestEval. Furthermore, it outperforms all other entailment-based metrics. QAFACTEVAL-NLI shows slight improvements on the synthetic data. Notable improvements in this synthetic setting can be observed on the FactCC dataset, likely as the synthetic FactCC data the model is trained on was designed to mirror the errors captured in annotations. This performance boost on FactCC motivated our use of supervised data for fine-tuning our learned metric. Supervised fine-tuning on validation data helps in most cases and QAFACTEVAL-NLI (supervised) improves on the overall benchmark by a statistically significant margin, using bootstrap resampling (Efron, 1982) with Bonferroni correction (Bonferroni, 1935) to obtain 99% confidence intervals (see Appendix for details). The performance drop on FactCC could be due to the proximity of the synthetic data to the labeled data and the data size difference. BertScore-FFCI performs best on XSF perhaps due to the closeness between BertScore's token-level metric and XSF's wordlevel annotations, and DocNLI's Polytope performance may also be from training data similarity. We find that QAFACTEVAL and SCConv do offer complementary signals that can be learned from supervised data. Individually fine-tuning the learned SCConv or a learned variation of QAFACTEVAL on supervised data did not improve results over the non-supervised metrics; this result suggests the necessity of combining the two for further improvements. Training on the validation sets combined, rather than on each individual dataset separately, did not give an improvement, likely due to the learnable combination of NLI and QAFACTEVAL being dataset dependent.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We provide instance-level Pearson correlation between aggregated human judgments and metric scores for each model to compare to previous work in factual consistency that reports correlation analysis. Results are shown in Table 5. We split FRANK into CNN/DailyMail and XSum subsets for finergrained analysis, as substantial differences have been noted in correlation performance across the two datasets (Durmus et al., 2020). We exclude Polytope, FactCC, and CGS here as prior work has only studied these datasets for binary classification.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We find that QAFACTEVAL performs well across most datasets. As in the classification results, BertScore-FFCI's performs well on XSF, and we note that QuestEval's answerability classifier correlates more so with these fine-grained annotations than on other datasets. QAFACTEVAL-NLI performs well on most datasets except XSF. Fine-tuning on FactCC synthetic data for binary classification likely does not capture the aggregated, word-level factuality scores of XSF. We leave a study of fine-tuning this model on supervised data with a regression loss for future work.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
In this work, we demonstrated that QA-based metrics, when its components are properly optimized, outperform entailment-based metrics on a comprehensive factual consistency evaluation benchmark. We identify question generation and answerability detection as key components for improving QAbased metrics in future work. Furthermore, we show that entailment and QA-based metrics offer complementary signals through a combined metric that achieves state-of-the-art performance on this benchmark. We believe that our work lays the foundation for future work in QA-based metrics for factual consistency by offering a fairer comparison to other metrics across datasets and settings.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Dataset Biases The underlying models of the metrics presented in this work are trained on documents in English and thus mainly represent the culture of the English-speaking populace. Political or gender biases may also exist in the datasets, and models, and subsequently the metrics, trained on these datasets may propagate these biases. We did not stress test these metrics for such biases and request that the users of these metrics be aware of these potential issues in applying them.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Misuse Potential and Failure Mode When properly used, the metrics described in this paper can be a useful tool for detecting summarization model errors. However, the current metrics fail to detect all factual inconsistencies, which must be remembered when applying these metrics as a filter for downstream applications. Factual inconsistencies in summaries could contribute to misinformation on the internet.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
The experiments described in the paper primarily make use of A100 GPUs. Most of the metrics have already been trained, in which case we simply ran inference using the existing models. We typically used a single GPU per experiment. Training learned answer overlap components can take a couple of hours, while experiments for learned metrics on SummaC take less than 10 minutes. These are the base models used in these experiments, with the number of parameters, in millions, in parentheses: BERTbase (110), BART-large (400), Electra-base (110), Electra-large (335), RoBERTa-large (355), T5-base (220), T5-large (770). Future work may analyze the effect of using distilled backbone models on factual consistency evaluation. 
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
In this section, we provide details regarding statistical testing, benchmark statistics, and miscellaneous details regarding our QA-based experiments.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
To determine whether the improvements on the SummaC benchmark are statistically significant, we perform significance tests using bootstrap resampling (Efron, 1982), following Laban et al. (2021). We compare our best model to the bestperforming model from prior work on a given subset of the benchmark. We compare confidence intervals at significance levels of 0.05 and 0.01 and apply the Bonferroni correction (Bonferroni, 1935). Statistically significant differences at the 0.01 level exist between QAFACTEVAL-NLI (supervised) and the best prior work on the FRANK subset and for the overall benchmark result. We do not see statistically significant differences on the other datasets in the benchmark. However, the statistically significant difference at the overall benchmark is notable; while other metrics may perform comparably or better on a given dataset, our metric demonstrates consistent good performance across datasets.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
For completeness, we provide additional statistics for the SummaC benchmark in Table 6. Due to the exclusion of Omission and Addition as factual consistency errors in the Polytope dataset, our dataset contains benchmark replication contains many more positive examples for that dataset. For XSF, we restrict the dataset to those examples with labels for factual consistency with respect to the source, as opposed to more general factuality labels which take into account world knowledge, which results in fewer examples than the original Sum-maC benchmark. This is the same subset as was used in Koto et al. (2021). Please see the following links for the licenses of the datasets and annotations: CGS 2 , XSF 3 , FactCC 4 , SummEval 5 . We did not find licenses for the remaining datasets analyzed in our study. The intended uses of these licenses align with our use for research purposes.  
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Ablation experiments started from a combination that provided good initial validation results and then swapped components. Running every combination of QA-based metric components is expensive. We experimented with running an ablation of the QA models with a 2nd-best performing answer selection component ALL. This reduced all scores compared to using the NP Chunks component. This experiment supports our setup of keeping the best component constant when running ablations in order to determine the highest-performing combination of components, rather than experimenting with every combination. Inference for the MADE QA model is run using the average of the six MADE adapters' parameters.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
For Question Filtering with the IsAnswered-Summ Filter, in addition to if the Electra-large QA model labels the question as unanswerable, if the F1 overlap score between the selected answer and the QA model output is less than 0.60, we remove this question. This filter was added only to IsAnsweredSumm and not IsAnsweredInput as answering questions based on the summary, from which the question was generated, should be an easy task. We reached this threshold based on a qualitative analysis of model outputs, although this number could have also been further tuned on the validation set.
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
We provide additional correlation coefficients as a point of reference for future work. Instance-level correlations calculate the correlation between all instances, while the summary-level correlation computes the correlation between scores for each summary of the same input and then averages over inputs. Summary-level correlations are excluded for QAGS as this dataset does not contain annotations for multiple models, which is necessary to compute this score.     Table 11: Summary-level Kendall correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
Evaluating Factuality in Generation with Dependency-level Entailment
The rise of pre-trained language models (Devlin et al., 2019;Radford et al., 2019) has led to strong text generation models for applications including summarization (Dong et al., 2019;, paraphrasing (Goyal and Durrett, 2020;Shen et al., 2020), story generation (Mao et al., 2019), and data augmentation Zhang and Bansal, 2019). However, while these models generate fluent and grammatical text, they are prone to making factual errors that contradict 1 Data and code available at https://github.com/ tagoyal/dae-factuality the input text (Cao et al., 2018). Automatic metrics used to evaluate text generation, such as ROUGE and BERTScore (Zhang et al., 2020), are not correlated with the factual consistency or faithfulness of the generated text (Falke et al., 2019;Kryściński et al., 2019). To address this, recent work has studied the use of textual entailment models to rank and filter non-factual generations (Falke et al., 2019;Maynez et al., 2020). However, these models suffer from issues such as dataset biases (Gururangan et al., 2018;Zhou and Bansal, 2020) and a mismatch between the training data (entailment) and the test data (model generations).
Evaluating Factuality in Generation with Dependency-level Entailment
In this paper, we propose to decompose entailment decisions in a sentence to evaluate the faithfulness of generated text in a more fine-grained way. Rather than making a sentence-level entailment decision, we instead evaluate the entailment of dependency arcs of the generated sentence, as illustrated in Figure 1. This approach views dependency arcs as semantic units that can be interpreted in isolation. Each arc is therefore judged indepen-dently based on whether the relation it implies is entailed by the source sentence. This is helpful in localizing generation errors and consequently providing more interpretable model decisions.
Evaluating Factuality in Generation with Dependency-level Entailment
Decomposing the factuality evaluation over components of structured representations can also be extended to other formalisms like AMR (Banarescu et al., 2013), UDS (White et al., 2016), and more. The chief advantage of dependency parsing over these is that pre-existing tools for dependency parsing report very high performance. Another line of work focuses on question answering-based semantic representations (FitzGerald et al., 2018; or generating freeform questions to capture factuality (Wang et al., 2020;Durmus et al., 2020). However, these systems require a separate question generation step at inference time, so generation is baked into their formalisms in a heavyweight way whereas we only require dependencies. A final advantage of our approach is that dependency arcs can be produced in an online fashion during inference, and hence, factuality can be enforced incrementally.
Evaluating Factuality in Generation with Dependency-level Entailment
We evaluate our proposed dependency arc entailment approach in both summarization and paraphrase settings. In both settings, we show that we can automatically derive labels from actual generation data rather than rely on human annotation of dependency arc entailment, which is challenging to collect at scale. Nevertheless, our results show that our system's performance on factuality classification surpasses both sentence-level entailment and question generation and answering models. Our derived labels from actual generation data provide much better task-specific supervision compared to general entailment datasets. Finally, we demonstrate that predicted entailment scores for individual dependency arcs are meaningful and can be leveraged to understand and localize errors in system generations.
Evaluating Factuality in Generation with Dependency-level Entailment
Defining arc entailment Our notion of entailment starts by assuming a rough correspondence between predicates and arguments in two sentences. In natural language inference (NLI) annotation efforts, this has taken the form of anchoring judgments in an underlying imagined scene (Bowman et al., 2015). We make a similar assumption, that events and actors in the source and target sentences are in correspondence unless there is direct evi-dence to the contrary. For instance, in Figure 1, the military coup in the target sentence and its corresponding amod(coup→military) arc should be evaluated with respect to the military takeover in the source, giving coreference of the two the benefit of the doubt here.
Evaluating Factuality in Generation with Dependency-level Entailment
With this assumption, we say that a dependency arc in the target sentence is entailed by the source if the semantic relationship it implies between its head and child is entailed by the source sentence. There is precedent for such a syntax-semantics correspondence: certain formalisms like meaning-text theory (Mel'čuk, 1988) have historically made this mapping more or less explicit. Consider the first hypothesis in Figure 1. Many of the arcs here either contain information analogous to that in semantic roles, or they specify nominal modifiers capturing important entity properties. 2 In our implementation, we exclude certain arc types which are not strongly tied to semantics, such as arcs involving punctuation; see the Appendix for details. Note that our method does not support commenting on arcs of the input that do not exist in the output; we discuss this later in Section 7.2.
Evaluating Factuality in Generation with Dependency-level Entailment
In some ways, our view of entailment is equivalent with the entailment of NLI settings (Bowman et al., 2015;Williams et al., 2018): if a hypothesis is entailed under the NLI definition, then all dependency arcs of the hypothesis must be entailed by our DAE definition. However, in our formulation, arc entailment is a 2-class classification task with labels ∈ {entailed, non-entailed}. This means that arcs that would be neutral or contradiction in the generic entailment formulation are considered non-entailed in our scenario.
Evaluating Factuality in Generation with Dependency-level Entailment
Annotating arc entailment To model this formulation, we require entailment annotations at the dependency arc level. However, there are several challenges associated with human annotation of arc entailment data. (1) Entailment is not truly a binary decision and is inherently subjective (Pavlick and Kwiatkowski, 2019). (2) Entailment of an arc may be fundamentally unknowable or undefined if, for example, too much of the context has changed for such a judgment to make sense. (3) Annotators would need to understand the meaning of dependency labels and to be able to isolate the semantics of these individual arcs in sentences. < l a t e x i t s h a 1 _ b a s e 6 4 = " L y L i 8 C B X A 4 N H n P V d w V K G L n 7 6 u J U = " > A A A B 7 X i c b V B N S w M x E J 3 1 s 9 a v q k c v w S L U S 9 m t g o K X o g g e K 9 g P a J e S T b N t b D Z Z k q x Y l v 4 H L x 4 U 8 e r / 8 e a / M W 3 3 o K 0 P B h 7 v z T A z L 4 g 5 0 8 Z 1 v 5 2 l 5 Z X V t f X c R n 5 z a 3 t n t 7 C 3 3 9 A y U Y T W i e R S t Q K s K W e C 1 g 0 z n L Z i R X E U c N o M h t c T v / l I l W Z S 3 J t R T P 0 I 9 w U L G c H G S o 2 b 0 t P l 4 K R b K L p l d w q 0 S L y M F C F D r V v 4 6 v Q k S S I q D O F Y 6 7 b n x s Z P s T K M c D r O d x J N Y 0 y G u E / b l g o c U e 2 n 0 2 v H 6 N g q P R R K Z U s Y N F V / T 6 Q 4 0 n o U B b Y z w m a g 5 7 2 J + J / X T k x 4 4 a d M x I m h g s w W h Q l H R q L J 6 6 j H F C W G j y z B R D F 7 K y I D r D A x N q C 8 D c G b f 3 m R N C p l 7 7 R c u T s r V q + y O H J w C E d Q A g / O o Q q 3 U I M 6 E H i A Z 3 i F N 0 c 6 L 8 6 7 8 z F r X X K y m Q P 4 A + f z B 3 7 f j m s = < / l a t e x i t > d(h) < l a t e x i t s h a 1 _ b a s e 6 4 = " Z / 2 n r x O C S 1 V p f 0 P 6 M L I a X 0 L 0 M + c = " > A A A B 6 3 i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h X k
Evaluating Factuality in Generation with Dependency-level Entailment
l n 2 r j u t 1 N a W 9 / Y 3 C p v V 3 Z 2 9 / Y P q o d H X R 2 n i t A O i X m s e g H W l D N J O 4 Y Z T n u J o l g E n D 4 G k 9 v c f 3 y i S r N Y P p h p Q n 2 B R 5 J F j G C T S 2 F 9 f D 6 s 1 t y G O w d a J V 5 B a l C g P a x + D c K Y p I J K Q z j W u u + 5 i f E z r A w j n M 4 q g 1 T T B J M J H t G + p R I L q v 1 s f u s M n V k l R F G s b E m D 5 u r v i Q w L r a c i s J 0 C m 7 F e 9 n L x P 6 + f m u j a z 5 h M U k M l W S y K U o 5 M j P L H U c g U J Y Z P L c F E M X s r I m O s M D E 2 n o o N w V t + e Z V 0 m w 3 v o t G 8 v 6 y 1 b o o 4 y n A C p 1 A H D 6 6 g B X f Q h g 4 Q G M M z v M K b I 5 w X 5 9 3 5 W L S W n G L m G P 7 A + f w B U k + N w w = = < / l a t e x i t > r a < l a t e x i t s h a 1 _ b a s e 6 4 = " X 0 l q 7 K b V O S w k U 2 R c t Q 6 r p h r Z f w E = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m q o M e i F 4 8 V 7 Q e 0 o W y 2 k 3 b p Z h N 2 N 0 I J / Q l e P C j i 1 V / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 4 W 1 9 Y 3 N r e J 2 a W d 3 b / + g f H j U 0 n G q G D Z Z L G L V C a h G w S U 2 D T c C O 4 l C G g U C 2 8 H 4 d u a 3 n 1 B p H s t H M 0 n Q j + h Q 8 p A z a q z 0 o P q 0 X 6 6 4 V X c O s k q 8 n F Q g R 6 N f / u o N Y p Z G K A 0 T V O u u 5 y b G z 6 g y n A m c l n q p x o S y M R 1 i 1 1 J J I 9 R + N j 9 1 S s 6 s M i B h r G x J Q + b q 7 4 m M R l p P o s B 2 R t S M 9 L I 3 E / / z u q k J r / 2 M y y Q 1 K N l i U Z g K Y m I y + 5 s M u E J m x M Q S y h S 3 t x I 2 o o o y Y 9 M p 2 R C 8 5 Z d X S a t W 9 S 6 q t f v L S v 0 m j 6 M I J  Therefore, in this work, we take another approach, which is to automatically label data from existing sources and outputs of generation models, which lets us collect large-scale data in a variety of domains. Specifically, we use paraphrase data to construct our dataset. Note however, that there is a fundamental difference between paraphrase pairs, which should be entailed in both directions, and past NLI data, which is forward-entailed by definition. For instance, the premise and hypothesis in Table 1 would classically be judged as entailed because political representative is a hypernym of prime minister, but the hypothesis is not a paraphrase of (even part of) the premise.
Evaluating Factuality in Generation with Dependency-level Entailment
As a result, our automatically-derived dataset captures a more restricted notion of entailment, primarily consisting of entailment relations that are symmetric in nature: arcs in the target sentence entailed by a source sentence also entail some part of the source. However, this is actually closer to what is acceptable for generation models to produce in tasks such as summarization, and the dataset collected in such a manner is useful for downstream tasks, as we show in Section 6. Moreover, because our training and evaluation data will typically come from closely related sentences, we can sidestep the cases where judgments in our formalism become most difficult to define.
Evaluating Factuality in Generation with Dependency-level Entailment
Let x be the input context, h be a hypothesis produced by a generation model G, and d(h) be the set of arcs in the dependency parse of h. We want to predict the entailment decision for each arc a ∈ d(h) with respect to the input x, i.e. F a (a, x).
Evaluating Factuality in Generation with Dependency-level Entailment
The overall model architecture of this dependency arc entailment model F a is outlined in Figure 2. First, we concatenate the input and the hypothesis. We use a pre-trained encoder model E to obtain contextual representations for each token in the concatenated sequence. From these token level representations, we derive a representation for each dependency arc a ∈ d(h):
Evaluating Factuality in Generation with Dependency-level Entailment
as shown in the inset in the figure. Here, a h , a c are the token indices corresponding to the head word and the child word of dependency arc a, and a d is their corresponding dependency label, which is also embedded with E (non-contextually).
Evaluating Factuality in Generation with Dependency-level Entailment
Next, these arc representations are passed through a linear layer, followed by a softmax layer to obtain entailment label probabilities corresponding to each arc: p(y | a; x) = softmax(W r a )).
Evaluating Factuality in Generation with Dependency-level Entailment
This DAE network is trained using standard binary cross entropy loss and requires supervision on the arc entailment labels y * ∈ {entailed, non-entailed} for the dependency arcs. Examples do not need to be fully labeled; training can use partial sets of annotations of arcs in d(h). However, while using the DAE model in downstream tasks such as hypotheses reranking, entailment decisions for all arcs in the candidate hypothesis are required.
Evaluating Factuality in Generation with Dependency-level Entailment
Sentence-level factuality from dependencylevel judgments We want to evaluate the factual consistency of each hypothesis h with respect to input x, i.e. F(h, x). This is computed by combining arc-level entailment scores over the dependency arcs set d(h) of the generated hypothesis:
Evaluating Factuality in Generation with Dependency-level Entailment
We use the sentence-level score F(h, x) to rerank the generated hypotheses in Section 6. 3
Evaluating Factuality in Generation with Dependency-level Entailment
We now describe our method for automatically collecting dependency-level DAE annotations from paraphrase or entailment corpora, avoiding manual annotation. In this creation process, we want data featuring a range of paraphrasing phenomenon, such as passivization, clausal reordering, synonym replacement, and more. Furthermore, we want a natural distribution of errors produced by generation models, such as wrong subject or objects for a verb or hallucination of new content.
Evaluating Factuality in Generation with Dependency-level Entailment
We represent a single example in our dataset as a tuple x, h, (a i , y * i ) a i ∈d(h) . Here, x is the input, h is the hypothesis, a i denotes a single dependency arc in the hypothesis, and y i refers to the gold entailment label for that arc. To construct data of this form, we assume access to a paraphrase dataset D, containing pairs (x, h * ) of input sentences and their corresponding gold paraphrases. 4 Additionally, we employ a paraphrase generation model G p , which can output k candidate paraphrases {h 1 , h 2 , ...h k } given an input x. These noisy paraphrases will be used to derive realistic examples of generation errors to contrast with gold paraphrases, using the following techniques.
Evaluating Factuality in Generation with Dependency-level Entailment
Positive labels from gold paraphrases Given a ground truth paraphrase, we assume that every arc in the target side of the paraphrase h * is entailed by the source side x. This is in line with 3 According to DAE definition, an output is non-factual if any of its arcs is non-entailed. However, min-pooling was very unstable, so we instead use mean-pooling in our experiments. 4 The paraphrase corpora we use in this work may come from automatic methods like backtranslation; however, we still assume that these are reliable gold-standard paraphrases. our definition of arc entailment in Section 2 and allows us to propagate sentence-level paraphrase judgements to arc-level entailment judgements. Because paraphrase datasets feature diverse linguistic phenomena, this approach leads to a range of positive examples. However, as described in Section 2, it is less likely to include arcs which are forwardentailed only (e.g., Table 1).
Evaluating Factuality in Generation with Dependency-level Entailment
Auto-derived labels from model generations To find negative examples for entailment, we leverage the output generations {h 1 , h 2 , ...h k } of an automatic paraphrase model G p . These generations will include unseen arcs, which may be positively or negatively entailed.
Evaluating Factuality in Generation with Dependency-level Entailment
Our key assumption here is that the outputs at the top of the beam are more likely to be factually correct, whereas outputs at the bottom of the beam are of lower quality and more prone to having factual inconsistencies. We assume that new arcs introduced in bad model generations (i.e., bottom-most generations of the beam) are not entailed by the input.
Evaluating Factuality in Generation with Dependency-level Entailment
We can then noisily label the generated paraphrases with a mix of positive (entailed) and negative (non-entailed) labels. We first construct a set of entailed dependency arcs: this is a set containing all dependency arcs of the input and the gold paraphrase, i.e., d(x) ∪ d(h * ). Next, we annotate the dependency arcs of the bottom-most generations of the beam, say {h k , h k−1 , h k−2 }, in the following way:
Evaluating Factuality in Generation with Dependency-level Entailment
The middle case here leaves arcs that are in h 1 but not x or h * as unannotated. Such arcs are possibly factual under the model, coming from a high-quality generated output, but we do not have enough confidence to assign them a label. During model training, such unannotated arcs are ignored. Finally, we also include the positive arcs from the 1-best hypothesis in our DAE data:
Evaluating Factuality in Generation with Dependency-level Entailment
. This provides another source of hypothesis sentences with a slightly different distribution during model training.
Evaluating Factuality in Generation with Dependency-level Entailment
Our experimental evaluation focuses on the following questions: (1) Does the automatic data collec-tion result in a high quality training dataset? (2) Is the DAE model we train a good classifier? (3) Does DAE allow us to filter model generations and improve reliability of a generation system?
Evaluating Factuality in Generation with Dependency-level Entailment
We construct our DAE training dataset using the methodology defined in Section 4. For this, we leverage the paraphrase pair dataset PARANMT-50M (Wieting and Gimpel, 2018) as the base dataset D. We use the transformer-based encoderdecoder model for paraphrase generation from Goyal and Durrett (2020) as G p . We use the paraphrase model to generate 10 outputs for 20k sentence pairs from D. We use the Stanford CoreNLP library  to extract enhanced dependencies from the outputs sentences. Then, using the strategy outlined in Section 4, we generate 100k training samples (sentence pairs), 3k dev samples and 3k test samples. From this dataset, we derive 520k training, 14k dev, and 22k dependency level annotations, which we evaluate on in Section 5.2. The entailed to not-entailed ratio is roughly 70-30 in this dataset.
Evaluating Factuality in Generation with Dependency-level Entailment
Before evaluating our modeling approach, we first evaluate whether the arc annotations in the training data follow the theoretical definition of DAE, outlined in Section 2. Figure 3 showcases examples from the dev set, corresponding to the same input example. We show positive entailed arcs (in green), negative non-entailed arcs (in red), and one unlabeled arc (in gray). Here, we can see that the gold paraphrase is important as it provides examples of valid synonym replacements, as well as other rephrasing of the input sentence. For negative examples, the examples from the bottom of the beam do indeed contain bad output and non-entailed arcs.  Agreement with human labels Next, we want to evaluate the quality of the auto-derived dataset by measuring its agreement with human annotations. For this, we manually annotated the dependency arc entailment labels for 100 sentence pairs from the dev set (consisting of 20 gold paraphrases and 80 generated paraphrases), according to our theoretical definition. We compared these manual annotations (gold) with the auto-derived annotations for this set, and observed that the two annotations agreed 82% of the time. This indicates that the automatic annotation strategy from Section 4 results in a high quality dataset. Further investigation into the disagreements between the manual and automatic labels revealed that false negatives included paraphrasing phenomena like synonym replacement, anaphora resolution during reordering, etc. We describe how to produce additional data to handle some of these cases later. On the other hand, false positives mainly consisted of exact arc matches in incorrect contexts.  Next, we intrinsically evaluate the performance of the dependency arc entailment model, outlined in Section 3, on held-out data from our automatic labeling method. We test the performance of two pre-trained models: BERT (bert-base-uncased, 110M parameters) (Devlin et al., 2019) and ELEC-TRA (electra-base-discriminator, 110M parameters) (Clark et al., 2020). We compare these models against a majority label baseline (entailment) and an lexical-match baseline that predicts y = entailment if the arc (head, child and label) in the output constitute a dependency arc in the input as well, and non-entailed otherwise.
Evaluating Factuality in Generation with Dependency-level Entailment
The performance of the different models is outlined in Table 2. Our pre-trained transformer models perform substantially better than the baselines, with BERT achieving 86.9% accuracy, and ELEC-TRA with 88.4%. These models also outperform the lexical-match baseline, showing that the DAE models learn to do more than simple dependency arc matching. Henceforth, we use the best performing ELECTRA model in all our experiments.
Evaluating Factuality in Generation with Dependency-level Entailment
Besides the data generation procedure we proposed, there are other ways to synthetically generate noisy annotations for premise-hypothesis pairs (Zhang et al.). We investigate these from two perspectives: first, does our data generation process cover these phenomena well, and second, can these additional sources of data prove useful?
Evaluating Factuality in Generation with Dependency-level Entailment
First, we explore a word-swapping technique similar to Zhang et al.. Given a premise x, we form a hypothesis x by randomly swapping tokens that share a common part-of-speech tag to introduce errors. The intersection of the arcs in d(x) and the modified sentence d(x ) are annotated as positive arcs (y = entailment), whereas the newly created or changed arcs are annotated as negative (y = non-entailed).
Evaluating Factuality in Generation with Dependency-level Entailment
Our synonym data is noisily constructed in the same manner as the gold paraphrases, but targets synonym replacements specifically. We extract pairs (x, h * ) from D that generally maintain similar sentence structure between the two sentences, 5 but with small lexical changes like synonym replacement. We assign a positive entailment label to all arcs: x, h * , {(a, 1) ∀a ∈ d(h * )} .
Evaluating Factuality in Generation with Dependency-level Entailment
To construct data with hallucinations, we modify an input sentence x, which we take as the hypothesis by removing a randomly sampled span of contiguous tokens to derive a premise sentence x . Then, the following DAE model annotations are derived: x , x, {(a i , 0) ∀ a i ∈ d(x)\d(x )} . Additionally, for each input sentence x, we extract another sentence x with the highest 1-gram overlap in the dataset. From this we derive, Table 3 shows a comparison of word-swapping with our method (AD), and variants of our method augmented with synonyms and hallucinations. Note that the model trained on word swapped data performs well on a similarly constructed held-out set, but not on the test data for synonym data and auto-derived data. This indicates that artificially constructed data with rule based error introduction  does not cover the space of generation possibilities. On the other hand, the model trained on our auto-derived dataset performs well across both artificial and actual generation data, thereby covering a larger range of entailment possibilities. Additional augmentation of synonym-and hallucinationspecific data improves the performance further on the respective test sets while retaining the performance on generic entailment data. Henceforth, we use the (AD + S) model for our experiments.
Evaluating Factuality in Generation with Dependency-level Entailment
Moving beyond the dependency-level inference task, we now want to evaluate the sentence-level performance of our model formulation. Namely, can it usefully reject erroneous generations produced by models for summarization (Section 6.1) and paraphrasing (Section 6.2)?
Evaluating Factuality in Generation with Dependency-level Entailment
We perform our evaluation on an abstractive summarization test dataset introduced in Falke et al.
Evaluating Factuality in Generation with Dependency-level Entailment
(2019) and used in other previous work. It contains 373 test samples, each containing an input source sentence from CNN/DM and two summary sentences covering the same content generated using the model from Chen and Bansal (2018). One of these summary sentences is factually correct and the other is factually incorrect. The evaluation protocol measures how often the correct summary is scored higher than the incorrect summary for each candidate scoring technique. We compare against the following baselines:  Table 4: Performance of the different models at the summary reranking task. The human baseline is reported in Falke et al. (2019). The proposed DAE model performs on par or better than prior works and comes close to human performance.
Evaluating Factuality in Generation with Dependency-level Entailment
2. Question Generation and Answering: Wang et al. (2020) propose an automatic evaluation metric QAGS that scores each summary by first generating questions pertaining to the summary content, and then comparing the answers to those questions in both the source sentence and the generated summary.
Evaluating Factuality in Generation with Dependency-level Entailment
3. Rule-based: We score each summary sentence as the fraction of dependency arcs in the output that are common with the input sentence. In case both the correct and the incorrect sentence get the same score, we break ties randomly. Table 4 outlines the performance of the different models. The results show that the dependency arc entailment model outperforms the sentence-level NLI models and also the question generation and answering formulation (QAGS). Furthermore, the performance of our DAE model is close to the human performance reported in Falke et al. (2019). Interestingly, the rule-based dependency model also outperforms certain NLI models and QAGS, indicating that these more complex models may fail to capture straightforward lexical relations.
Evaluating Factuality in Generation with Dependency-level Entailment
During our experimentation, we observed large variance in the performance of the NLI models at the reranking task with respect to their performance at the intrinsic entailment task. To illustrate this, in Figure 4, we plot the summarization reranking performance of the two model against the intrinsic task performance at different stages of the training. For DAE, the intrinsic task performance is reported by the dependency-level entailment classification accuracy, and for the MNLI model, we report the (details in the Appendix), improving on the results from Falke et al. (2019). We use the fine-tuned ROBERTA model released by AllenNLP (https://allennlp.org/). Accuracy Accuracy Figure 4: Performance of the ELECTRA-based MNLI model and the DAE model. The figure shows a much higher variance in reranking accuracy for the MNLI model, suggesting that the task-specific performance is not correlated with reranking performance. classification accuracy on the sentence-level MNLI entailment task. The graph shows a high variance in the summary reranking performance, with stable increase in the MNLI intrinsic task performance at different time steps. 7 This indicates that the general entailment task solves a fundamentally different problem than factuality. By contrast, the DAE model performance on the summarization reranking task is more stable.
Evaluating Factuality in Generation with Dependency-level Entailment
Next, we evaluate the DAE model in the paraphrasing setting. To do this, first, we create a test set, similar to the summarization test set from Falke et al. (2019). We use the transformer based seq2seq model (Goyal and Durrett, 2020) to obtain 10 candidate paraphrases for 100 input sentences from the PARANMT-50M dataset. We manually assign a label y ∈ {factual, not factual} to each input, candidate pair. Then for each input sentence, we randomly selected one correct and one incorrect paraphrase. This sentence triplet is used for our reranking experiments.
Evaluating Factuality in Generation with Dependency-level Entailment
Reranking Acc MNLI (ELECTRA) 79.0 DAE (ELECTRA) 79.0 Table 5: Performance on the paraphrase reranking task. The DAE performs on par with the NLI based model. Table 5 compares the performance of the MNLIbased model and the DAE models. Here, both are ELECTRA based models; these are shown to be the best performing models in the previous sec-7 Note that the best performance of the MNLI model on summary reranking is better than the best performance of the DAE model; however, it did not coincide with the task-level best performance for our particular hyperparameter choice.
Evaluating Factuality in Generation with Dependency-level Entailment
tions. The results show that in this setting, the MNLI model and the DAE model perform similarly. Closer inspection of this data revealed that our model is biased towards predicting the label entailment for arcs that are common with the input, possibly because we are evaluating the same generation model that was used to produce our arc entailment data, and our model is therefore biased towards predicting non-entailed for arcs that are not present in the input. This poses a somewhat adversarial setting for our DAE model. 7 Analysis 7.1 Dependency-vs. sentence-level modeling Although our DAE model has shown strong performance, we have not yet performed a direct applesto-apples comparison of DAE versus a sentencelevel model when trained on the same sentences.
Evaluating Factuality in Generation with Dependency-level Entailment
For MNLI We construct DAE data from the sentence-level entailment data as follows. First, we extract 10k examples from the MNLI data which have the label entailment. This is considered as the source data D . We use a paraphrase model (transformer seq2seq (Goyal and Durrett, 2020)) and the technique outlined in Section 4 to extract auto-derived labels from D . This gives us 42k training examples for training the DAE model. We compare this against an MNLI model trained on the original sentence-level entailment task with the same number of examples (42k).
Evaluating Factuality in Generation with Dependency-level Entailment
For PARANMT For this dataset, we do not have negative (y = contradiction) annotations at the sentence-level. We derive these from our DAE training data as follows: we consider all pairs of sentences in the original dataset (x, h * ) as positive sentences (y = 1), in addition to any pair of the form (x, x). We treat the three generated sentences at the bottom of the beam as negative sentences, meaning that the model is trained to distinguish gold paraphrases from model generations. Table 6 outlines these results. For the paraphrase dataset, we see that the artificially constructed sentence-level dataset does not yield a good sentence-level discriminator. However, our dependency-level annotations can form an effective training set. The results on MNLI show that our dependency-level formulation performs better than sentence-level when trained on the same amount of data and is therefore more closely related to the factuality task than past entailment formulations.  
Evaluating Factuality in Generation with Dependency-level Entailment
Error Localization Since the DAE formulation computes individual entailment scores for all arcs in the dependency tree structure, it is possible to localize errors in the generated summary or paraphrase.
Evaluating Factuality in Generation with Dependency-level Entailment
We present examples of input sentences, generated text, and arc entailment scores for a few examples in Figure 5. For each input and output pair, we show the individual scores for the dependency arcs in the output sentence. Additionally, we report the MNLI score for the same example.
Evaluating Factuality in Generation with Dependency-level Entailment
The illustrative examples show that the DAE model is capable of localizing errors where erroneous subject-object pairs were constructed, even when these are the only errors. These errors are tougher for the MNLI model to catch, which evaluates the whole sentence and is prone to lexical overlap biases (Zhou and Bansal, 2020). In the third example, from our paraphrase setting, we see that the model is able to recognize synonym replacement as a valid re-writing. However, for the last example, the model cannot perform this same judgement for the variations → changes replacement. Although, note that the model scores it higher than a erroneous replacement of the same word (variations → latter). This shows that the DAE model is able to rank sentences that incorporate the similar type of re-writing/editing. However, we observed that the model has different error rates for different types of re-writing changes. For example, it is better at identifying text hallucination, or cases where the subject object relation between words change, but has comparatively lesser accuracy over changes such as synonym replacements. Therefore, it may not be ideal for settings where different re-writing types need to be compared.
Evaluating Factuality in Generation with Dependency-level Entailment
Limitations We comment on a few limitations of our approach. First, arcs in our dependency-based formalism are not marked with negation or quantification; these must be handled via the contextu-Input Text: Visitors to the Isle of Lewis will be hoping the clouds stay away on Friday for the solar eclipse.
Evaluating Factuality in Generation with Dependency-level Entailment
Output:
Evaluating Factuality in Generation with Dependency-level Entailment
Visitors to the Isle of Lewis will be hoping to stay away on Friday. alization of the hypothesis sentence rather than in the semantic representation. Second, our method cannot identify arcs that are missing from the input. For instance, consider the following premise: In the morning, he goes jogging and hypothesis: In the morning. Here, the hypothesis does not contain critical information from the source sentence; however, since all the present arcs are entailed by the source, our model would give this a high score. Furthermore, our model is trained on the PARANMT-50M dataset, which itself is constructed through a noisy backtranslation process. Therefore, we rely on noisy gold data for constructing our model. We believe that better quality paraphrase pairs would lead to a better quality model.
Evaluating Factuality in Generation with Dependency-level Entailment
In this work, we propose the dependency arc entailment formulation to identify factual errors in generated text in a more fine-grained manner. We show that the proposed formulation outperforms past approaches, while additionally providing an interpretable error analysis. 
Evaluating Factuality in Generation with Dependency-level Entailment
As outlined in Section 2, we restrict our analysis to a subset of dependency arcs which are more strongly connected to semantics. For each of hypothesis h corresponding to the input x, we extract Enhanced Dependencies using the Stanford CoreNLP tool, and assign entailment labels to this dependency arc set d(h) using the strategy outlined in Section 4. We exclude the following arcs from our analysis: punct, det, case, aux, auxpass, dep, cop, mark. This same subset of arcs are ignored while computing sentence-level factuality.
Evaluating Factuality in Generation with Dependency-level Entailment
As outlined in Section 5.2, we additionally augment our auto-derived training data (AD) with synonym data (S) and show that this improves the model performance on the held out synonym only test set. Figure 6 provides some examples showing the predicted entailment probability for each arc using this augmented training data. The predictions show that our model learns some bias to recognize synonym replacements and small phrasal substitutions as arcs that are entailed by the input.
Evaluating Factuality in Generation with Dependency-level Entailment
Input: you'd be a great inspiration to your fellow warriors. Output you'd be a great inspiration to the other soldiers.
Evaluating Factuality in Generation with Dependency-level Entailment
99.2 99.3 60.8
Evaluating Factuality in Generation with Dependency-level Entailment
Input: Turkey is a complicated country, with multiple dilemmas. Output:
Evaluating Factuality in Generation with Dependency-level Entailment
Turkey is a complex country with many dilemmas.
Evaluating Factuality in Generation with Dependency-level Entailment
99.2 77.9 69.0 79.9 Figure 6: Example from the held-out synonym dataset. The scores are the arc entailment probabilities assigned by the (AD + S) model.
Evaluating Factuality in Generation with Dependency-level Entailment
To train our DAE model, we fine-tune the pre-trained encoders BERT (bert-base-uncased, 110M parameters) and ELECTRA (electra-basediscriminator, 110M parameters), as outlined in Section 5. We perform 5 hyperparameter trials, varying only the learning rate using manual tuning. The models with the best dev set accuracy are used. The final hyperparameters used are:  Additionally, we fine-tune (bert-base-uncased, 110M parameters) and ELECTRA (electra-basediscriminator, 110M parameters) models on the MNLI dataset. We fine-tuned the model using 3 hyperparameter trials, varying only the learning rate using manual tuning. The models with the best dev set accuracy are used. The final hyperparameters used are shown in Table 8  We get a dev accuracy of 84.5% and 89.0% for the BERT and ELECTRA models respectively.