What approaches have been proposed to edit a generated text to be factually consistent with a collection of evidence?
Post-hoc editing for factuality Recent work has
gone beyond checking the validity of a claim to
correcting a piece of text to be factually consistent
with a set of evidence via post-hoc editing (Shah
et al., 2020; Thorne and Vlachos, 2021; Schuster
et al., 2021; Balachandran et al., 2022; Cao et al.,
2020; Iso et al., 2020). FRUIT (Logan IV et al.,
2022) and PEER (Schick et al., 2022) both im-
plement an editor that is fine-tuned on Wikipedia
edit history with the goal of updating outdated in-
formation and collaborative writing respectively.
Evidence-based Factual Error Correction (EFEC;
Thorne and Vlachos, 2021) also implements a full
research-and-revise workflow trained on Wikipedia
passages (Thorne et al., 2018). A key differentia-
tor of RARR is its ability to edit the output of any
generation model without being restricted by the
domain, task, or the need for training data
Automatic Fact-guided Sentence Modification
Online encyclopediae like Wikipedia contain large amounts of text that need frequent corrections and updates. The new information may contradict existing content in encyclopediae. In this paper, we focus on rewriting such dynamically changing articles. This is a challenging constrained generation task, as the output must be consistent with the new information and fit into the rest of the existing document. To this end, we propose a two-step solution: (1) We identify and remove the contradicting components in a target text for a given claim, using a neutralizing stance model; (2) We expand the remaining text to be consistent with the given claim, using a novel two-encoder sequence-to-sequence model with copy attention. Applied to a Wikipedia fact update dataset, our method successfully generates updated sentences for new claims, achieving the highest SARI score. Furthermore, we demonstrate that generating synthetic data through such rewritten sentences can successfully augment the FEVER fact-checking training dataset, leading to a relative error reduction of 13%.
[{"first":"Darsh","last":"Shah","middle":["J"]},{"first":"Tal","last":"Schuster","middle":[]},{"first":"Regina","last":"Barzilay","middle":[]}]
RARR: Researching and Revising What Language Models Say, Using Language Models
Language models (LMs) now excel at many tasks such as question answering, reasoning, and dialog. However, they sometimes generate unsupported or misleading content. A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence. To enable attribution while still preserving all the powerful advantages of recent generation models, we propose RARR (Retrofit Attribution using Research and Revision), a system that 1) automatically finds attribution for the output of any text generation model, and 2) post-edits the output to fix unsupported content while preserving the original output as much as possible. When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models. Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.
[{"first":"Luyu","last":"Gao","middle":[]},{"first":"Zhuyun","last":"Dai","middle":[]},{"first":"Panupong","last":"Pasupat","middle":[]},{"first":"Anthony","last":"Chen","middle":[]},{"first":"Arun Tejasvi","last":"Chaganty","middle":[]},{"first":"Yicheng","last":"Fan","middle":[]},{"first":"Vincent","last":"Zhao","middle":[]},{"first":"Ni","last":"Lao","middle":[]},{"first":"Hongrae","last":"Lee","middle":[]},{"first":"Da-Cheng","last":"Juan","middle":[]},{"first":"Kelvin","last":"Guu","middle":[]}]
Fact-based Text Editing
We propose a novel text editing task, referred to as fact-based text editing, in which the goal is to revise a given document to better describe the facts in a knowledge base (e.g., several triples). The task is important in practice because reflecting the truth is a common requirement in text editing. First, we propose a method for automatically generating a dataset for research on fact-based text editing, where each instance consists of a draft text, a revised text, and several facts represented in triples. We apply the method into two public table-to-text datasets, obtaining two new datasets consisting of 233k and 37k instances, respectively. Next, we propose a new neural network architecture for fact-based text editing, called FactEditor, which edits a draft text by referring to given facts using a buffer, a stream, and a memory. A straightforward approach to address the problem would be to employ an encoder-decoder model. Our experimental results on the two datasets show that FactEditor outperforms the encoder-decoder approach in terms of fidelity and fluency. The results also show that FactEditor conducts inference faster than the encoder-decoder approach.
[{"first":"Hayate","last":"Iso","middle":[]},{"first":"Chao","last":"Qiao","middle":[]},{"first":"Hang","last":"Li","middle":[]}]
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness—improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.
[{"first":"Tal","last":"Schuster","middle":[]},{"first":"Adam","last":"Fisch","middle":[]},{"first":"Regina","last":"Barzilay","middle":[]}]
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Abstractive summarization models often generate inconsistent summaries containing factual errors or hallucinated content. Recent works focus on correcting factual errors in generated summaries via post-editing. Such correction models are trained using adversarial non-factual summaries constructed using heuristic rules for injecting errors. However, generating non-factual summaries using heuristics often does not generalize well to actual model errors. In this work, we propose to generate hard, representative synthetic examples of non-factual summaries through infilling language models. With this data, we train a more robust fact-correction model to post-edit the summaries to improve factual consistency. Through quantitative and qualitative experiments on two popular summarization datasets— CNN/DM and XSum—we show that our approach vastly outperforms prior methods in correcting erroneous summaries. Our model—FactEdit—improves factuality scores by over ~11 points on CNN/DM and over ~31 points on XSum on average across multiple summarization models, producing more factual summaries while maintaining competitive summarization quality.
[{"first":"Vidhisha","last":"Balachandran","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]},{"first":"William","last":"Cohen","middle":[]},{"first":"Yulia","last":"Tsvetkov","middle":[]}]
Factual Error Correction for Abstractive Summarization Models
Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by an error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.
[{"first":"Meng","last":"Cao","middle":[]},{"first":"Yue","last":"Dong","middle":[]},{"first":"Jiapeng","last":"Wu","middle":[]},{"first":"Jackie","last":"Cheung","middle":["Chi","Kit"]}]
Evidence-based Factual Error Correction
This paper introduces the task of factual error correction: performing edits to a claim so that the generated rewrite is better supported by evidence. This extends the well-studied task of fact verification by providing a mechanism to correct written texts that are refuted or only partially supported by evidence. We demonstrate that it is feasible to train factual error correction systems from existing fact checking datasets which only contain labeled claims accompanied by evidence, but not the correction. We achieve this by employing a two-stage distant supervision approach that incorporates evidence into masked claims when generating corrections. Our approach, based on the T5 transformer and using retrieved evidence, achieved better results than existing work which used a pointer copy network and gold evidence, producing accurate factual error corrections for 5x more instances in human evaluation and a .125 increase in SARI score. The evaluation is conducted on a dataset of 65,000 instances based on a recent fact verification shared task and we release it to enable further work on the task.
[{"first":"James","last":"Thorne","middle":[]},{"first":"Andreas","last":"Vlachos","middle":[]}]
Automatic Fact-guided Sentence Modification
Online text resources like Wikipedia contain millions of articles that must be continually updated. Some updates involve expansions of existing articles, while others modify the content. In this work, we are interested in the latter scenario where the modification contradicts the current articles. Such changes are common in online sources and often cover a broad spectrum of subjects ranging from the changing of dates for events to modifications of the relationship between entities. In these cases, simple solutions like negating the original text or concatenating it with the new information would not apply. In this work, our goal is to automate these updates. Specifically, given a claim and an outdated sentence from an article, we rewrite the sentence to be consistent with the given claim while preserving non-contradicting content.
Automatic Fact-guided Sentence Modification
Consider the Wikipedia update scenario depicted in Figure 1. The claim, informing that 23 of 43 minority stakeholdings are significant, contradicts the old information in the * Order decided by a coin toss. 1 Code: (1) https://github.com/TalSchuster/TokenMasker (2) https://github.com/darsh10/split encoder pointer summarizer GSG considers 23 of 43 minority stakeholdings beginning in operationally active companies to be of particular significance to the group.
Automatic Fact-guided Sentence Modification
GSG considers 23 of 43 minority stakeholdings to be significant.
Automatic Fact-guided Sentence Modification
Old Wikipedia:
Automatic Fact-guided Sentence Modification
Updated sentence:
Automatic Fact-guided Sentence Modification
GSG considers 28 of their 42 minority stakeholdings in operationally active companies to be of particular significance to the group.
Automatic Fact-guided Sentence Modification
in operationally active companies to be of particular significance to the group.
Automatic Fact-guided Sentence Modification
Figure 1: Our fact-guided update pipeline. Given a claim which refutes incorrect information, a masker is applied to remove the contradicting parts from the original text while preserving the rest of the context. Then, the residual neutral text and claim are fused to create an updated text that is consistent with the claim.
Automatic Fact-guided Sentence Modification
Wikipedia sentence, requiring modification. Directly learning a model for this task would demand supervision, i.e. demonstrated updates with the corresponding claims. For Wikipedia, however, the underlying claims which drive the changes are not easily accessible. Therefore, we need to utilize other available sources of supervision.
Automatic Fact-guided Sentence Modification
In order to make the corresponding update, we develop a two step solution: (1) Identify and remove the contradicting segments of the text (in this case, 28 of their 42 minority stakeholdings); (2) Rewrite the residual sentence to include the updated information (e.g. fraction of significant stakeholdings) while also preserving the rest of the content.
Automatic Fact-guided Sentence Modification
For the first step, we utilize a neutrality stance classifier as indirect supervision to identify the polarizing spans in the target sentence. We consider a sentence span as polarizing if its absence increases the neutrality of the claimsentence pair. To identify and mask such sentence spans, we introduce an interpretability-inspired (Lei, Barzilay, and Jaakkola 2016) neural architecture to effectively explore the space of possible spans. We formulate our objective in a way that the masking is minimal, thus preserving the context of the sentence.
Automatic Fact-guided Sentence Modification
For the second step, we introduce a novel, two-encoder decoder architecture, where two encoders fuse the claim and the residual sentence with a more refined control over their interaction.
Automatic Fact-guided Sentence Modification
We apply our method to two tasks: automatic fact-guided modifications and data augmentation for fact-checking. On the first task, our method is able to generate corrected Wikipedia sentences guided by unstructured textual claims. Evaluation on Wikipedia modifications demonstrates that our model's outputs were the most successful in making the requisite updates, compared to strong baselines. On the FEVER fact-checking dataset, our model is able to successfully generate new claim-evidence supporting pairs, starting with claim-evidence refuting pairs -intended to reduce the bias in the dataset. Using these outputs to augment the dataset, we attain a 13% decrease in relative error on an unbiased evaluation set.
Automatic Fact-guided Sentence Modification
The growing interest in automatic fake news detection led to the development of several fact verification datasets (Vlachos and Riedel 2014;Wang 2017;Rashkin et al. 2017;Thorne et al. 2018). FEVER, the largest fact-checking dataset, contains 185K human written fake and real claims, generated by crowdworkers, in context of sentences from Wikipedia articles. This dataset contains biases that allow a model to identify many of the false claims without any evidence (Schuster et al. 2019). This bias affects the generalization capabilities of models trained on such data. In this work, we show that our automatic modification method can be used to augment a fact-checking dataset and to improve the inference of models trained on it.
Automatic Fact-guided Sentence Modification
Data Augmentation Methods for data augmentation are commonly used in computer vision (Perez and Wang 2017). There have been recent successes in NLP where augmentation techniques such as paraphrasing and word replacement were applied to text classification (Kobayashi 2018;Wu et al. 2018). Adversarial examples in NLI with syntactic modifications can also be considered as methods of data augmentation (Iyyer et al. 2018;Zhang, Baldridge, and He 2019). In this work, we create constrained modifications, based on a reference claim, to augment data for our task at hand. Our additions are specifically aimed towards reducing the bias in the training data, by having a false claim appear in both "Agrees" and "Disagrees" classes.
Automatic Fact-guided Sentence Modification
Problem Statement We assume access to a corpus D of claims and knowledge-book sentences. Specifically, D = {{C 1 , ..., C n }, {S 1 , ..., S m }}, where C is a short factual sentence (claim), and S is a sentence from Wikipedia. Each pair of claim and Wikipedia sentence has a relation rel(S, C), of either agree (Agr), disagree (Dis) or neutral (N ). In this corpus, a Wikipedia sentence S is defined as outdated with respect to C if rel(S, C) = Dis and updated if rel(S, C) = Agr. The neutral relation holds for pairs in which the sentence doesn't contain specific information about the claim.
Automatic Fact-guided Sentence Modification
Our goal is to automatically update a given sentence S, which is outdated with respect to a C. Specifically, given a claim and a pair for which rel(S, C) = Dis, our objective is to apply minimal modifications to S such that the relation of the modified sentence S + will be: rel(S + , C) = Agr.
Automatic Fact-guided Sentence Modification
In addition, S + should be structurally similar to S.
Automatic Fact-guided Sentence Modification
Framework Currently, to the best of our knowledge, there is no large dataset for fact-guided modifications. Instead, we utilize a large dataset with pairs of claims and sentences that are labeled to be consistent, inconsistent or neutral. In order to compensate the lack of direct supervision, we develop a two-step solution. First, using a pretrained fact-checking classifier for indirect supervision, we identify the polarizing spans of the outdated sentence and mask them to get a S ∅ such that rel(S ∅ , C) = N . Then, we fuse this pair to generate the updated sentence which is consistent with the claim. This is done with a sequence-to-sequence model trained with consistent pairs through an auto-encoder style objective. The two steps are trained independently to simplify optimization (see Figure 3).  
Automatic Fact-guided Sentence Modification
In this section we describe the module to identify the polarizing spans within a Wikipedia sentence. Masking these spans ensures that the residual sentence-claim pairs attain a neutral relation. Here, neutrality is determined by a classifier trained on claim and Wikipedia sentence pairs as described below. Using this classifier, the masking module is trained to identify the polarizing spans by maximizing the neutrality of the residual-sentence and claim pairs. In order to preserve the context of the original sentence, we include optimization constraints to ensure minimal deletions. This approach is similar to neural rationale-based models (Lei, Barzilay, and Jaakkola 2016), where a module tries to identify the spans of the input that justify the model's prediction.
Automatic Fact-guided Sentence Modification
Neutrality Masker Given a knowledge-book sentence (S) and a claim (C), the masker's goal is to create S ∅ such that rel(S ∅ , C) = N . For the original sentence with l tokens,
Automatic Fact-guided Sentence Modification
, the output is a mask m ∈ [0, 1] l . The neutral sentence S ∅ is constructed as:
Automatic Fact-guided Sentence Modification
where is a special token. 2 The details of the masker architecture are stated below and depicted in Figure 2.
Automatic Fact-guided Sentence Modification
Encoding We encode S with a sequence encoder to get
Automatic Fact-guided Sentence Modification
Since the neutrality of the sentence needs to be measured with respect to a claim, we also encode the claim and enhance S's representations with that of C using attention mechanism. Formally, we compute
Automatic Fact-guided Sentence Modification
where c j are the encoded representations of the claim and a i,j are the parameterized bilinear attention (Kim, Jun, and Zhang 2018) weights computed by:
Automatic Fact-guided Sentence Modification
(4) Finally, the aggregated representations are used as input to a sequence encoder g(·; w g ).
Automatic Fact-guided Sentence Modification
The encoded sentence is used to predict a per token masking probability:
Automatic Fact-guided Sentence Modification
Then, the mask is applied to achieve the residual sentence:
Automatic Fact-guided Sentence Modification
where • denotes element-wise multiplication. During training, we perform soft deletions over the token embeddings and add the out-of-vocabulary embedding in place. During inference, the values of m are rounded to create a discrete mask.
Automatic Fact-guided Sentence Modification
Training A pretrained fact-checking neutrality classifier's prediction rel(S, C) is used to guide the training of the masker. In order to encourage maximal retention of the context, we utilize a regularization term to minimize the fraction of the masked words. The joint objective is to minimize:
Automatic Fact-guided Sentence Modification
Fact-checking Neutrality Classifier Our fact-checking classifier is pretrained on agreeing and disagreeing (S, C) pairs from D, in addition to neutral examples constructed through negative sampling. For each claim we construct a neutral pair by sampling a random sentence from the same paragraph of the polarizing sentence, making it contextually close to the claim, but unlikely to polarize it. We pretrain the classifier on these examples and fix its parameters during the training of the masker.
Automatic Fact-guided Sentence Modification
Optional Syntactic Regularization Currently the model is trained with distant supervision, so, we pre-compute a valid neutrality mask as additional signal, when possible. To this end, we parse the original sentences using a constituency parser and iterate over continuous syntactic phrases by increasing length. For each sentence, the shortest successful neutrality mask (if any) is selected as a target mask. 3 In the event of successfully finding such a mask, the masking module is regularized to emulate the target mask by adding the following term to Eq. 7:
Automatic Fact-guided Sentence Modification
where m is the target mask. Empirically, we find that the model can perform well even without this regularization, but it can help to stabilize the training. Additional details and analysis are available in the appendix.
Automatic Fact-guided Sentence Modification
In this section we describe our method to generate an output which agrees with the claim. If the earlier masking step is done perfectly, the merging boils down to a simple fusion task. However, in certain cases, especially ones with a strong contradiction, our minimal deletion constraint might leave us with some residual contradictions in S ∅ . Thus, we develop a model which can control the amount of information to consider from either input. We extend the pointer-generator model of (See, Liu, and Manning 2017) to enable multiple encoders. While sequence-to-sequence models support the encoding of multiple sentences by simply concatenating them, our use of a per input encoder allows the decoder to better control the use of each source. This is especially of interest to our task, where the context of the claim must be translated to the output while ignoring contradicting spans from the outdated Wikipedia sentence.
Automatic Fact-guided Sentence Modification
Next, we describe the details of our generator's architecture. Here, we use one encoder for the outdated sentence and one encoder for the claim. In order to reduce the size of the model, we share the parameters of the two encoders. The model can be similarly extended to any number of encoders.
Automatic Fact-guided Sentence Modification
Encoding At each time step t, the decoder output h t , is a function of a weighted combination of the two encoders' context representations r t , the decoder output in the previous step h t−1 and the representation of the word output at the end of the previous step emb(y t−1 ):
Automatic Fact-guided Sentence Modification
As the decoder should decide at each time step which encoder to attend more, we introduce an encoder weight α. The shared encoder context representation r t is based on their individual representations r t 1 and r t 2 :
Automatic Fact-guided Sentence Modification
The context representation r t i (i∈{1, 2}) is the attention score over the encoder representation r i for a particular decoder state h t−1 :
Automatic Fact-guided Sentence Modification
Decoding Following standard copy mechanism, predicting the next word y t , involves deciding whether to generate (p gen ) or copy, based on the decoder input x t = [r t , emb(y t−1 )], the decoder state h t and context vector r t :
Automatic Fact-guided Sentence Modification
In case of copying, we need an additional gating mechanism to select between the two sources:
Automatic Fact-guided Sentence Modification
When generating a new word, the probability over words from the vocabulary is computed by:
Automatic Fact-guided Sentence Modification
The final output of the decoder at each time step is then computed by:
Automatic Fact-guided Sentence Modification
where a t are the input sequence attention scores from Eq. 11.
Automatic Fact-guided Sentence Modification
Training Since we have no training data for claim guided sentence updates, we train the generator module to reconstruct a sentence S to be consistent with an agreeing claim C. The training input is the residual up-to-date neutral sentence S ∅ and the guiding claim C.  Figure 3: A summary of our pipeline. Given a sentence that is inconsistent with a claim, a masker is applied to mask out the contradicting parts from the original text while preserving the rest of the content. Then, the residual neutral text and claim are fused to create an updated text that is consistent with the claim. The Masker and the Two-Encoder Generator are trained separately.
Automatic Fact-guided Sentence Modification
During inference, we utilize only guiding claims and residual outdated sentences S ∅ to create S + . While generating the updated sentences S + , we would like to preserve as much context as possible from the contradicting sentence, while ensuring the correct relation with the claim. Therefore, for each case, if the later goal is not achieved, we gradually increase the focus on the claim by increasing α and p enc1 values until the output S + satisfies rel(S + , C) = Agr, or until a predefined maximum weight.
Automatic Fact-guided Sentence Modification
We evaluate our model on two tasks: (1) Automatic fact updates of Wikipedia sentences, where we update outdated wikipedia sentences using guiding fact claims; and (2) Generation of synthetic claim-evidence pairs to augment an existing biased fact-checking dataset in order to improve the performance of trained classifiers on an unbiased dataset.
Automatic Fact-guided Sentence Modification
Training Data from FEVER We use FEVER (Thorne et al. 2018), the largest available Wikipedia based factchecking dataset to train our models for both of our tasks. This dataset contains claim-evidence pairs where the claim is a short factual sentence and the evidence is a relevant sentence retrieved from Wikipedia. We use these pairs as our claim-setnence samples and use the "refutes", "not enough information", "supports" labels of that dataset as our Dis, N , Agr relations, respectively.
Automatic Fact-guided Sentence Modification
We evaluate the automatic fact updates task on an evaluation set based on part of the symmetric dataset from (Schuster et al. 2019) and the fact-based cases from a Wikipedia updates dataset (Yang et al. 2017). For the symmetric dataset, we use the modified Wikipedia sentences with their guiding claims to generate the true Wikipedia sentence. For the cases from the updates dataset, we have human annotators write a guiding claim for each update and use it, together with the outdated sentence, to generate the updated Wikipedia sentence. Overall we have a total of 201 tuples of fact update claims, outdated sentences and updated sentences.
Automatic Fact-guided Sentence Modification
To measure the proficiency of our generated outputs for data augmentation, we use the unbiased FEVER-based evaluation set of (Schuster et al. 2019). As shown by (Schuster et al. 2019), the claims in the FEVER dataset contain give-away phrases that can make FEVER-trained models overly rely on them, resulting in decreased performance when evaluated on unbiased datasets.
Automatic Fact-guided Sentence Modification
The classifiers trained on our augmented dataset are evaluated on the unbiased symmetric dataset of (Schuster et al. 2019). This dataset (version 0.2) contains 531 claimevidence pairs for validation and 534 claim-evidence pairs for testing.
Automatic Fact-guided Sentence Modification
In addition, we extend the symmetric test set by creating additional FEVER-based pairs. We hired crowd-workers on Amazon Mechanical Turk and asked them to simulate the process of generating synthetic training pairs. Specifically, for a "refutes" claim-evidence FEVER pair, the workers were asked to generate a modified supporting evidence while preserving as much information as possible from the original evidence. We collected responses of workers for 500 refuting pairs from the FEVER training set. This process extends the symmetric test set (+TURK) by 1000 cases -500 "refutes" pairs, and corresponding 500 "supports" pairs generated by turkers.
Automatic Fact-guided Sentence Modification
Masker We implemented the masker using the AllenNLP framework (Gardner et al. 2018). For a neutrality classifier, we train an ESIM model (Chen et al. 2017) to classify a relation of Agr, Dis or N . To train this classifier, we use the Agr and Dis pairs from the FEVER dataset and for each claim we add a neutral sentence which is sampled from the sentences in the same document as the polarizing one. The classifier and masker are trained with GloVe (Pennington, Socher, and Manning 2014) word embeddings. We use BiLSTM (Sak, Senior, and Beaufays 2014) encoders with hidden dimensions of 100 and share the parameters of the claim and original sentence encoders. The model is trained for up to 100 epochs with a patience value of 10, where the stopping condition is defined as the highest delta between accuracy and deletion size on the development set (∆ in Table 3).
Automatic Fact-guided Sentence Modification
For syntactic guidance, we use the constituency parser of shows the human's scores in a 1-5 Likert scale on grammatically of the output sentence and on agreement with the given claim. (Stern, Andreas, and Klein 2017) and consider continuous spans of length 2 to 10 as masking candidates (without combinations). By doing so, we obtain valid neutrality masks for 38% of the Agr and Dis pairs from the FEVER training dataset. These masks are used for Eq. 8.
Automatic Fact-guided Sentence Modification
We implemented our proposed multi-sequence-to-sequence model, based on the pointer-generator framework.We use a one layer BiLSTM for encoding and decoding with a hidden dimension of 256. The parameters of the two encoders are shared. The model is trained with batches of size 64 for a total of 50K steps.
Automatic Fact-guided Sentence Modification
BERT Fact-Checking Classifier We use a BERT (Devlin et al. 2018) classifier, which takes in as input a (claimevidence) pair separated by a special token, to predict out of 3 labels (Agr, Dis or N ). The model is fine-tuned for 3 epochs, which is sufficient to perform well on the task.
Automatic Fact-guided Sentence Modification
Evidence Regeneration Since we are interested in using the generated supporting pairs for data augmentation, we add machine generated cases to the Agr set of the dataset. Adding machine generated sentences to only one of the labels in the data can be ineffective. Therefore, we balance this by regenerating paraphrased refuting evidence for the false claims. This is then added along with all models' outputs for a balanced augmentation.
Automatic Fact-guided Sentence Modification
We consider the following baselines for constructing a factguided updated sentence:
Automatic Fact-guided Sentence Modification
• Copy Claim The sentence of the claim is copied and used as the updated sentence for itself (used only for data augmentation).
Automatic Fact-guided Sentence Modification
• Paraphrase The claim is paraphrased using the backtranslation method of (Wieting and Gimpel 2018) 4 , and the output is used as the updated sentence.
Automatic Fact-guided Sentence Modification
• Claim Extension [Claim Ext.] A pointer-generator network is trained to generate the updated sentence from an input claim alone. The model is trained on FEVER's agreeing pairs and applied on the to-be-updated claims during inference.
Automatic Fact-guided Sentence Modification
• Masked Concatenation [M. Concat] Instead of our Two-Encoder Generator, we use a pointer-generator network. The residual sentence (output from the masker module) and the claim are concatenated and used as input.
Automatic Fact-guided Sentence Modification
• Split Encoder without Copy [Split-no-Copy] Our Two-Encoder Generator, without the copy mechanism. The original text and contradicting claim are passed through each of the encoders.
Automatic Fact-guided Sentence Modification
We report the performance of the model outputs for automatic fact-updates by comparing them to the corresponding correct wikipedia sentences. We also have crowd workers score the outputs on grammar and for agreeing with the claim. Additionally, we report the results on a fact-checking classifier using model outputs from the FEVER training set as data augmentation.  with respect to the original sentence. 6 For human evaluation of the model's outputs, 20% of the evaluation dataset was used. Crowd-workers were provided with the model outputs and the corresponding supposably consistent claims. They were instructed to score the model outputs from 1 to 5 (1 being the poorest and 5 the highest), on grammaticality and agreement with the claim. Table 1 reports the automatic and human evaluation results. Our model gets the highest SARI score, showing that it is the closest to humans in modifying the text for the corresponding tasks. Humans also score our outputs the highest for consistency with the claim, an essential criterion of our task. In addition, the outputs are more grammaticality sound compared to those from other methods.
Automatic Fact-guided Sentence Modification
Examining the gold answers, we notice that many of them include very minimal and local modifications, keeping much of the original sentence. The M. Concat model keeps most of the original sentence as is, even at the cost of being inconsistent with the claim. This corresponds to a high KEEP score but a lower SARI score overall, and a low human score on supporting the claim. Claim Ext. and Paraphrase do not maintain the structure of the original sentence, and perform poorly on KEEP, leading to a low SARI score. The Splitno-Copy model has the same low ADD score as Claim Ext. since instead of copying the accurate information from the claim, it generates other tokens.
Automatic Fact-guided Sentence Modification
Data Augmentation For 41850 Dis pairs in the FEVER training data, our method generates synthetic evidence sentences leading to 41850 Agr pairs. We train the BERT factchecking classifier with this augmented data and report the performance on the symmetric dataset in Table 2. In addition, we repeat the human evaluation process on the generated augmentation pairs and report it in Table 1.
Automatic Fact-guided Sentence Modification
Our method's outputs are effective for augmentation, outperforming a classifier trained only on the original biased training data by an absolute 1.7% on the TEST set and an λ ACC SIZE ∆ PREC REC F1
Automatic Fact-guided Sentence Modification
.  absolute 3.0% on the +TURK set. The outputs of the Paraphrase and Copy Claim baselines are not Wikipedia-like, making them ineffective for augmentation. All the baseline approaches augment the false claims with a supported evidence. However, the success of our method in producing supporting evidence while trying to maintain a Wikipedialike structure, leads to more effective augmentations.
Automatic Fact-guided Sentence Modification
Masker Analysis To evaluate the performance of the masker model, we test its capacity to modify Agr and Dis pairs from the FEVER development set to a neutral relation. We measure the accuracy of the pretrained classifier in predicting neutral versus the percentage of masked words from the sentence. For a finer evaluation, we manually annotated 75 Agr and 76 Dis pairs with the minimal required mask for neutrality and compute the per token F1 score of the masker against them.
Automatic Fact-guided Sentence Modification
The results for different values of the regularization coefficient are reported in Table 3. Increasing the regularization coefficient helps to minimize the mask size and to improve the precision while maintaining the classifier accuracy and the mask recall. However, setting λ too large, can collapse the solution to no masking at all. The generation experiments use the outputs of the λ = 0.4 model.
Automatic Fact-guided Sentence Modification
In this paper, we introduce the task of automatic fact-guided sentence modification. Given a claim and an old sentence, we learn to rewrite it to produce the updated sentence. Our method overcomes the challenges of this conditional generation task by breaking it into two steps. First, we identify the polarizing components in the original sentence and mask them. Then, using the residual sentence and the claim, we generate a new sentence which is consistent with the claim. Applied to a Wikipedia fact update evaluation set, our method successfully generates correct Wikipedia sentences using the guiding claims. Our method can also be used for data augmentation, to alleviate the bias in fact verification datasets without any external data, reducing the relative error by 13%.
Automatic Fact-guided Sentence Modification
We thank the anonymous reviewers and the MIT NLP group for their helpful discussion and comments. This work is supported by DSO grant DSOCL18002. Zhang, Z.;Ren, S.;Liu, S.;Wang, J.;Chen, P.;Li, M.;Zhou, M.;and Chen, E. 2018. Style transfer as unsupervised machine translation. arXiv preprint arXiv:1808.07894.  
Automatic Fact-guided Sentence Modification
The masker model makes finding a valid mask in the space of 2 l options tractable. However, as mentioned in (Bao et al. 2018), training an objective of the type shown in Eq. 7 is unstable. An alternative tractable approach is to enumerate a set of syntactic components of the evidence and score them as potential masks for neutrality. Although this approach is insufficient and might not always work, the cases where the continuous spans satisfy neutrality can help guide the masker training. Table 4 shows results for the masker model with and without syntactic regularization. The syntactic regularization helps to stabilize the performance, allowing a reasonable solution even without any additional constraint on the mask size. Without syntactic regularization, better accuracy can be achieved, but the learning is very unstable and can lead to solutions that mask the whole sentence or keep it as is.
Automatic Fact-guided Sentence Modification
Examples of outputs from different models are provided in Table 5. For the first 3 examples, our model produces a perfect update. In the last example, even though our model gets the year 1930 correct, it modifies the month and nationality to made-up, incorrect values. This is a result of a too aggressive deletion by the masker. The Claim Ext. model typically produces wrong and non-grammatical sentences. The Concat model doesn't capture the polarizing relation between the two inputs and mostly ignores the claim. The M. Concat model tends to overly generate made-up content instead of copying it from the claim. Table 5: We compare our model outputs against different models. Each example is showing the two input sentences following the output of each model. The Concat model setting is similar to the M. Concat one but the original text is left unmasked. For the Claim Ext. model, only the claim sentence is given as input.
RARR: Researching and Revising What Language Models Say, Using Language Models
Generative language models (LMs) and other text generation models are now the backbone of many AI systems. For example, large language models can perform multi-step reasoning (Nye et al. Despite these incredible advances, state-of-theart LMs still frequently produce biased, misleading, Most existing LMs, such as those based on sequence-to-sequence architectures, lack a builtin mechanism for attribution. Even retrievalaugmented models (Guu et al., 2020; Lewis et al., 2020), which retrieve relevant documents and then condition on them to generate text, still do not guarantee attribution. Prior work has shown that retrieval-augmented models generate text that either includes additional information outside the retrieved documents (Dziri et al., 2022), ignores the documents altogether (Krishna et al., 2021), or even contradicts the documents (Longpre et al., 2021). In fact, occasionally ignoring the retrievals can make the models more robust to bad retrievals (Khandelwal et al., 2020), illustrating that end-task performance and attribution are not always aligned.
RARR: Researching and Revising What Language Models Say, Using Language Models
Instead of constraining LMs to generate attributed text, we propose a model-agnostic approach to improve the attribution of any existing LM: Retrofit Attribution using Research and Revision (RARR). The approach is inspired by works on fact-checking 2 where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans (Thorne et al., 2018; Schuster et al., 2021; Thorne and Vlachos, 2021). As shown in Figure 1, after generating text with the LM, RARR does research to retrieve relevant evidence, and then revises the text to make it consistent with the evidence while preserving qualities like style or structure, enabling the revised text to be seamlessly used in place of the original. RARR can be viewed as a retrievalaugmented model where retrieval happens after generation rather than before. This allows RARR to stand on the shoulders of giant LMs without having to modify them to support attribution.
RARR: Researching and Revising What Language Models Say, Using Language Models
In our effort to expand the scope of Research & Revision models to handle the output of arbitrary LMs, we make the following contributions. First, we formalize the Editing for Attribution task and propose new metrics that evaluate revision models not just on their ability to produce well-attributed revisions, but also on their ability to otherwise preserve original properties of the text. Second, we use these metrics to benchmark how existing revision models perform on various types of LM outputs such as knowledge-intensive statements, reasoning chains, and dialog responses. Finally, we find that existing revision models do not always generalize across many tasks (and were not originally intended to), and therefore propose a new research-and-revise model that leverages the power of few-shot prompting in large language models to robustly generalize across domains. 2 In this paper, we generally avoid the term "fact-checking" other than to reference relevant literature, because we only address attribution, and attribution does not entail correctness. Even if a claim is attributed to a particular source, it does not guarantee that the source is "correct" (Menick et al., 2022). Output Passage y Figure 2: An overview of RARR, which improves attribution for a text passage via Research & Revision. Given the input text passage, the research stage uses a query generator to raise questions about different aspects of the text. The retriever then searches for evidence to investigate each query. The revision stage first runs an agreement model to detect disagreement between the text and the evidence, then runs an edit model to revise the text if needed. Finally, M evidence snippets are selected to form an attribution report.
RARR: Researching and Revising What Language Models Say, Using Language Models
We propose the task of Editing for Attribution as follows. As Figure 1 shows, the input to the system is a text passage x produced by a generation model. The output is a revised text passage y along with an attribution report A, which contains evidence snippets e 1 , . . . , e M that support the content in y.
RARR: Researching and Revising What Language Models Say, Using Language Models
Optionally, the attribution report can contain additional information such as the alignment between evidence snippets and relevant parts in y. We propose to measure the quality of the revised text y and attribution report A along two dimensions: (1) attribution: how much of the revised text y can be attributed to the evidence in A, and (2) preservation: how much the revised text y preserves aspects of the original text x.
RARR: Researching and Revising What Language Models Say, Using Language Models
Previously, Rashkin et al. (2021) proposed Attributable to Identified Sources (AIS), a human evaluation framework which considers a binary notion of attribution. Roughly speaking, a text passage y is attributable to a set A of evidence if a generic hearer would affirm the statement "According to A, y" under the context of y. A system either receives full credit (1.0) if all content in y can be attributed to A, and no credit (0.0) otherwise.
RARR: Researching and Revising What Language Models Say, Using Language Models
We propose a more fine-grained, sentence-level extension of AIS. We ask annotators to give an AIS score for each sentence s of y, and then report the average AIS score across all sentences:
RARR: Researching and Revising What Language Models Say, Using Language Models
Attr AIS (y, A) = avg s∈y AIS(s, A).
RARR: Researching and Revising What Language Models Say, Using Language Models
(1)
RARR: Researching and Revising What Language Models Say, Using Language Models
Since the AIS score is binary, this effectively measures the percentage of sentences in y that are fully attributed to A. When judging each sentence, we also give annotators access to the surrounding sentences and other necessary context, such as the question that the text passage responded to. We also impose the maximum number of evidence snippets in the attribution report A to make it concise enough for both the annotator and downstream users. By manually inspecting 30 examples from our benchmarks, we found M = 5 snippets to be sufficient for full attribution. During model development, we define an automated metric, auto-AIS (Attr auto ), that approximates human AIS judgments. We utilize the natural language inference (NLI) model from Honovich et al. (2022), which correlates well with AIS scores. For each sentence s of y, and for each evidence snippet e in A, let NLI(e, s) be the model probability of e entailing s. We then define Attr auto (y, A) = avg s∈y max e∈A NLI(e, s).
RARR: Researching and Revising What Language Models Say, Using Language Models
(2)
RARR: Researching and Revising What Language Models Say, Using Language Models
To improve accuracy, we decontextualize (Choi et al., 2021) each sentence based on the entire context of y before computing the scores. See Appendix B for implementation details.
RARR: Researching and Revising What Language Models Say, Using Language Models
To measure preservation, we first ask annotators to decide if the revision preserves the text's original intent (completely, somewhat, or not at all -see Appendix C for exact rubrics). Like AIS evaluation, we give annotators the necessary surrounding context. We define the binary metric Pres intent (x, y) to be 1.0 if the revision completely preserves the original intent, and 0.0 otherwise. However, even if a revision preserves intent, it may still make superfluous modifications, such as reordering words, changing textual style, or including unnecessary additional information (Thorne and Vlachos, 2021). Different tasks have different requirements for what should be preserved. Here, we desire a simple metric that can be readily computed for many tasks and that generally penalizes unnecessary changes. We thus define a metric based on the character-level Levenshtein edit distance (Levenshtein, 1965) between x and y:
RARR: Researching and Revising What Language Models Say, Using Language Models
This metric is 1.0 if x and y are the same, and 0.0 if y completely overwrites all parts of x. Pres Lev is generally sensitive to any kind of change, but certainly does not capture all notions of preservation (e.g., preserving rhyme schemes or puns). We want the revision to preserve the original intent while avoiding superfluous edits. To reflect this, we finally combine the two metrics as Pres comb (x, y) = Pres intent (x, y) · Pres Lev (x, y).
RARR: Researching and Revising What Language Models Say, Using Language Models
(4) which is 0.0 if the revision changes the intent and equal to Pres Lev (x, y) otherwise. Since Pres intent requires human annotation, we use Pres Lev as an automated metric for model development.
RARR: Researching and Revising What Language Models Say, Using Language Models
Optimizing for attribution alone cannot ensure a good revision: for example, an adversarial editor could ensure 100% attribution by simply replacing the input x with the text of any arbitrary retrieved document, which is trivially attributable to itself. Ideally, we want to maximize both attribution and preservation, while navigating any tradoffs between the two. In our experiments, we report both metrics, as well as their harmonic mean (F1 AP , analogous to how recall and precision are combined in F1).
RARR: Researching and Revising What Language Models Say, Using Language Models
We emphasize that this evaluation scheme does not require any "gold" or "reference" edits (unlike many prior evaluations of text revision models), which are often only available for specialized domains. This enables us to broaden the scope to a much wider range of generation tasks.
RARR: Researching and Revising What Language Models Say, Using Language Models
We now present Retrofit Attribution using Research and Revision (RARR), a simple method for solving the Editing for Attribution task. As illustrated in Figure 2, given an input passage x, the research stage first generates a set of queries {q 1 , ..., q N }, each investigating one aspect of x that potentially  requires attribution. For each query q i , it retrieves web documents and selects the best evidence snippets {e i1 , e i2 , . . . }. The revision stage then revises the original text x using the retrieval results {(q 1 , e 11 ), . . . }, yielding a revised text y.
RARR: Researching and Revising What Language Models Say, Using Language Models
Most components for RARR are implemented using few-shot prompting (Brown et al., 2020). We use PaLM (Chowdhery et al., 2022) as our language model. Figure 3 shows some few-shot examples we use, while Appendix D lists the full prompts.
RARR: Researching and Revising What Language Models Say, Using Language Models
Query generation We perform comprehensive question generation (CQGen) which produces a sequence of questions covering all aspects of the passage x that need to be verified and attributed. A similar strategy has been employed to train textplanning models (Narayan et al., 2022). A prompt with six human demonstrations was sufficient for PaLM to adequately learn the task. To increase diversity and coverage, we sample from our CQ-Gen model three times and take the union of the resulting queries.
RARR: Researching and Revising What Language Models Say, Using Language Models
Evidence retrieval For each query from CQGen, we use Google Search to retrieve K = 5 web pages. We extract candidate evidence snippets from each web page by running a sliding window of four sentences across the page, breaking at document headings. The evidence snippets for each query are then ranked based on their relevance to the query. For this, we use an existing query-document relevance model trained following Ni et al. (2021), which computes a relevance score S relevance (q, e) between a query q and an evidence snippet e. We then keep the top J = 1 evidence for each query. The final retrieval result is [(q 1 , e 11 ), . . . , (q 1 , e 1J ), . . . , (q N , e N 1 ), . . . , (q N , e N J )], where e ij denotes the j th evidence for the i th query, and N denotes the total number of queries from CQGen (which can be different for each input x).
RARR: Researching and Revising What Language Models Say, Using Language Models
After retrieving evidence, certain parts of x may now be properly attributed, but other parts remain unattributed and should be revised. As illustrated in Figure 2, the revision stage initializes the output y = x. Then for each retrieved (q, e) = (q i , e ij ), the agreement model checks if the evidence e disagrees with the current output y regarding the issue in query q. If a disagreement is detected, the edit model edits y to agree with e; otherwise, it does nothing. The process continues until all retrievals are processed.
RARR: Researching and Revising What Language Models Say, Using Language Models
Agreement model The agreement model takes the partially edited passage y, a query q, and the evidence e as input. It then decides whether both y and e imply the same answer to the question in q. This form of question-guided agreement was previously explored by Honovich et al. (2021). We implement this by few-shot prompting PaLM using a chain-of-thought style prompt (Wei et al., 2022), where we ask the model to explicitly state the implied answers for both y and e before producing its judgment about their agreement.
RARR: Researching and Revising What Language Models Say, Using Language Models
Edit model The edit model is run only if a disagreement is detected. The model takes y, q and e as input, and outputs a new version of y that aims to agree with e while otherwise minimally altering y. We again use few-shot prompting and chain-ofthought, where we ask the model to first identify a particular span in y that needs to be edited before generating the revised y. This helps reduce the editor's deviation from the current y. 3
RARR: Researching and Revising What Language Models Say, Using Language Models
Finally, we select at most M = 5 evidence snippets to form an attribution report A. Note that during evidence retrieval and revision, we may have encountered and used more than M snippets. Our goal is to find a subset of snippets that maximizes coverage over the potentially attributable points in the passage, as represented by the queries q 1 , . . . , q N . We use the relevance model from Section 3.1 as a proxy for measuring how much an evidence e covers the point raised by a query q. Then, we exhaustively search for A ⊆ {e 11 , . . . , e N J } of size at most M that maximizes A key differentiator of RARR is its ability to edit the output of any generation model without being restricted by the domain, task, or the need for training data.  
RARR: Researching and Revising What Language Models Say, Using Language Models
RARR aspires to be a general-purpose method for improving the attribution of any text generation model in any text domain. We thus construct evaluation benchmarks by taking the task input from three diverse datasets, and prompting different generation models to produce long-form outputs which may contain "hallucinations," as demonstrated in Figure 4. These long-form outputs serve as input text passages to RARR. We generate 150 development and 150 test passages for each combination of generation model and source dataset. Knowledge-intensive dialogs We consider the conversational QA task from the QReCC dev set (Anantha et al., 2021). Given the previous dialog turns, which are rounds of questions and answers (Q 1 , A 1 , Q 2 , A 2 , . . . , Q k ), we use LaMDA and GPT-3 to answer to the final question Q k conditioned on the dialog history. The answer tends to be context-dependent, featuring pronouns and implicit references. All dialog turns are given alongside the answer as inputs to the revision model.
RARR: Researching and Revising What Language Models Say, Using Language Models
We compare RARR to several systems that have a research-and-revise workflow.
RARR: Researching and Revising What Language Models Say, Using Language Models
EFEC We consider EFEC (Thorne and Vlachos, 2021) as a representative fine-tuned editor. EFEC fine-tunes a T5-based model to revise text conditioned on multiple evidence snippets using both semi-supervised and fully-supervised approaches. We compare against their fully-supervised approach, which performed best in their experiments. EFEC uses a neural retrieval model (Karpukhin et al., 2020) to retrieve from Wikipedia; however, not all passages in our experiments are supported by Wikipedia articles. To more fairly compare the editing capabilities of EFEC, we instead use the evidence retrieved by our research stages (CQGen and web search). Note that the EFEC editor conditions on multiple pieces of evidence at once, while our editor iteratively conditions on one at a time.
RARR: Researching and Revising What Language Models Say, Using Language Models
LaMDA LaMDA (Thoppilan et al., 2022) generates responses in three steps: 1) generate a "base response"; 2) generate search queries from the base response; 3) generate a "revised response" conditioned on the base response and retrieved evidence.  Table 1: Evaluation results. For attribution, we report the AIS scores of the texts both before and after editing (before → after). For preservation, we report intent preservation Pres intent , Levenshtein similarity Pres Lev , and the combined Pres comb . We summarize Attr AIS and Pres comb using their harmonic mean (F1 AP ).
RARR: Researching and Revising What Language Models Say, Using Language Models
To apply LaMDA on a given text x, we simply set the base response in step 1 to x, and then run steps 2 and 3 (we call these latter two stages "LaMDA Research"). LaMDA was trained as a dialog system, and always expects a dialog context where the user speaks first. So, for non-dialog tasks, we insert an artificial user utterance as dialog history: "Tell me something interesting." For the attribution report, we take all evidence documents retrieved by LaMDA during its research process.
RARR: Researching and Revising What Language Models Say, Using Language Models
RARR Our model uses few-shot prompting on PaLM 540B for query generation, the agreement model, and the edit model. We use the same prompts for all tasks except when the context comes from a dialog, where we slightly modify the prompts to use the dialog context (e.g., CQGen now maps dialog context + x to queries). The queryevidence relevance model S relevance is a pretrained T5-large model ( 
RARR: Researching and Revising What Language Models Say, Using Language Models
For the main experiments, we report results on passages generated by PaLM and LaMDA. Results on GPT-3 passages show similar trends (Appendix A). Table 1 and Figure 5 show attribution and preservation results for each model and dataset. We also report F1 AP , the harmonic mean of the two metrics, which is shown as level curves in Figure 5. Dashed lines indicate the highest attribution score obtained by any of the models before editing: points above the line have better attribution after revision. The contours are F1 AP level curves: points along a contour have equivalent F1 AP . Different models make very different trade-offs between attribution and preservation. Only RARR has a robust F1 AP across all tasks.
RARR: Researching and Revising What Language Models Say, Using Language Models
RARR significantly improves attribution while preserving most of the original text. In terms of F1 AP , RARR is the only method that performs robustly across all three datasets, and significantly outperforms prior methods on NQ and SQA.
RARR: Researching and Revising What Language Models Say, Using Language Models
We found that RARR is the only method that preserves the original intent of x over 90% of the time -EFEC and LaMDA only manage to preserve the original intent 6-40% of the time. We also see that editing is crucial to improve attribution: if we only retrieve evidence to support the original response x without editing, attribution ranges from the low 10s to mid 30s. After editing, RARR can increase attribution by up to 13% absolute, while changing only 10-20% of the text.
RARR: Researching and Revising What Language Models Say, Using Language Models
As noted in Section 2, one can sacrifice preservation for higher attribution. EFEC is able to obtain strong F1 AP on QReCC by making larger changes to the text in exchange for a higher attribution score. However, it occupies a very different point from RARR on the attribution-preservation tradeoff curve, as visualized in Figure 5.
RARR: Researching and Revising What Language Models Say, Using Language Models
Human oracle To understand the remaining headroom in our task, we ask: what is the minimal amount of editing needed to make a text passage fully attributed? The answer would depend on the quality of the LM that generated the text as well as the task difficulty. As an approximation, we manually edited 30 examples in our NQ benchmark until we judged them to be 100% attributable. We achieved a preservation score of 88%, which (when combined with 100% attribution) translates to 93.6 F1 AP , indicating a significant headroom.  LaMDA changes the writing style. RARR preserves the structure of the input passage. We show one evidence retrieved by RARR to help explain the example.
RARR: Researching and Revising What Language Models Say, Using Language Models
Analyzing the baselines As exemplified in Figure 6, EFEC frequently attempts to summarize the entire passage into one sentence, or drops later sentences. This is likely due to EFEC's training data, which was limited to single sentences. This behavior generally increases the attribution score, because it is usually easier to make one sentence fully attributable than many sentences. However, in datasets where the claim contains multiple sentences (NQ and SQA), such a behavior yields low preservation scores, and also results in outputs that are less informative. We expect that EFEC could perform much better if its training data were augmented to include multiple sentences. LaMDA Research achieves similar attribution scores to RARR. But as mentioned in Section 5.2, the intent and linguistic style of the output tend to deviate from the input, resulting in lower preservation scores ( Figure 6). We emphasize that this is not a purely apples-to-apples comparison since LaMDA was not optimized for preservation. Overall, these experiments are mainly meant to illustrate that prior models were simply not designed for the task of Editing for Attribution, rather than to mark RARR as the best method.
RARR: Researching and Revising What Language Models Say, Using Language Models
Analyzing RARR For the research stage, the question generation model had comprehensive coverage: a manual inspection of 40 examples shows > 80% with questions that fully cover all aspects of the input text. The retriever was strongest at re-    searching content involving distinct entities (e.g., a movie, a major event, or a person). In contrast, we found significant headroom for better attribution of statements involving generic objects and more abstract claims (e.g. "Video games require electricity."-since this is obvious to most humans, retrieved articles from the web tend to address related but different topics). We suspect that a significant amount of attribution headroom on our benchmarks would benefit from a better research stage. For the revision stage, RARR was able to revise many unattributed claims, especially those involving entities and numbers (Figures 7a and 7b). It can also perform larger revisions when necessary (Figure 7c). Moreover, RARR abstains from editing when the claim is already well-attributed: on NQ, among the inputs with near-perfect attribution (preedit Attr AIS > 0.9), RARR does not make an edit in 90% of the cases. However, the system also has several shortcomings. Some erroneous edits arise from misleading irrelevant evidence (Figure 7d). We also observed an interesting challenge when revising reasoning chains, where the model suc-cessfully revised an incorrect claim, but did not revise subsequent reasoning steps that depend on the earlier claim ( Figure 7e). In this case, further editing to improve logical coherence could help.
RARR: Researching and Revising What Language Models Say, Using Language Models
Ablating query generation RARR uses generated questions as search queries for evidence retrieval. We consider two natural alternatives: using the entire input passage as a single search query, or using each sentence as a search query. For the former, we retrieve J = 3 evidence snippets to make the amount a closer match to other methods.
RARR: Researching and Revising What Language Models Say, Using Language Models
The results are in Table 2. Using the entire input passage as the query gives poor results, as the retrieved evidence tends to not focus on potentially unattributed parts in the passage. Using sentences as queries gives results closer to the full CQGen, but a closer analysis reveals two caveats.
RARR: Researching and Revising What Language Models Say, Using Language Models
First, sentences-as-queries are more effective when such sentences "mimic" content on the Web, and are less effective otherwise. In Table 3, we test this by excluding all of Wikipedia from web search results (since many PaLM outputs for NQ have a Wikipedia style). The attribution performance of sentences-as-queries drops significantly, while CQGen is more robust.
RARR: Researching and Revising What Language Models Say, Using Language Models
Second, sentence-as-queries tends to retrieve passages that may encourage confirmation bias. Consider the example "Georgia is called the Peach State, but California actually produces the most peaches." Retrieval using sentences-as-queries found an article echoing that California produces the most peaches, while CQGen generated the more impartial query "Which state produces the most peaches?" and found a newer article saying that South Carolina replaced California as the top peach producer. In this case, RARR using CQGen needs to sacrifice more preservation score to edit the text, leading to a lower F1 AP score. This underscores that attribution alone cannot measure "correctness" since not all evidence is up-to-date or reliable.
RARR: Researching and Revising What Language Models Say, Using Language Models
Ablating agreement model We try removing the agreement model, which effectively forces the model to revise the passage based on every retrieved evidence. The results are shown in Table 2. As expected, more revision leads to less preservation score and spurious changes to the text passage, as demonstrated in Figure 8.  Table 2: Ablation results. We report the automatic metrics: Attr auto , Pres Lev , and harmonic mean between the two (F1 AP ). We show auto-AIS scores both before and after editing (before → edit), with respect to the attribution report A produced by the model. Even though sentence-as-queries may achieve similar F1 AP as RARR, it is less robust to corpus shifts and tends to retrieve passages that may encourage confirmation bias.    fined in Section 2.2. However, another measure of preservation is whether the revised text can still be used to perform the task that it was originally generated for. Following EFEC, we quantitatively evaluate this on short answer tasks NQ and SQA, and we summarize the result in Figure 9. For NQ, each original text x is a long-form response to a factoid question. To determine whether the revised text y still serves this purpose, we feed the factoid question and y back into PaLM and prompt it to extract a short answer from y. We find that RARR not only preserves the short answer accuracy but actually improves it by roughly 5%.
RARR: Researching and Revising What Language Models Say, Using Language Models
For SQA, each original text is a reasoning chain that helps to answer a yes/no question. We feed the SQA question and y back into PaLM and prompt it to output a yes/no answer, and evaluate answer accuracy. Here, we find that increasing attribution comes at a slight cost in downstream task performance: answer accuracy drops modestly for all revision models (up to 2.6%). We suspect that this may be due to noisy retrievals, which sometimes provide misleading evidence (exemplified in Figure 7d). Furthermore, even though revisions can address factoid errors in the passage (e.g., "Homer Simpson has 5 fingers" from Figure 7e), RARR currently does not try to modify subsequent reasoning steps which may no longer be logically entailed (e.g., "He only needs one hand to count to 5").
RARR: Researching and Revising What Language Models Say, Using Language Models
Language models have developed increasingly good "procedural" knowledge of what should be discussed and how it should be presented, but often struggle to memorize "factoid" knowledge and produce unsubstantiated claims. We proposed RARR, a framework for revising such claims to make them attributable to the researched evidence. From experiments on text passages generated by different models on various domains, we showed that RARR can revise the passages to improve attribution while preserving other desirable properties such as writing style or structure. Furthermore, RARR sits on top of existing generation models without needing to re-design or re-train LMs.
RARR: Researching and Revising What Language Models Say, Using Language Models
Major headroom still remains, as discussed in Section 6 and the Limitations section. We hope our analysis of RARR would help with developing new approaches for integrating attribution to LMs.
RARR: Researching and Revising What Language Models Say, Using Language Models
Limitations of our task definition Depending on the application, attribution and preservation may not deserve equal weight. For instance, if there are multiple acceptable options for the output, such as in a dialog system, we might trade-off preservation for attribution, similar to how LaMDA behaves in our experiments.
RARR: Researching and Revising What Language Models Say, Using Language Models
Our evaluation metrics also do not measure all aspects of attribution. For instance, some sentences are self-evident and do not require attribution (e.g., "I agree.") but would be penalized in our evaluation.
RARR: Researching and Revising What Language Models Say, Using Language Models
It is also necessary to note that linguistic assertions have varying scope: for example, there is a difference between "Frozen is a scary movie" and "I got scared watching Frozen" -while expressing a similar sentiment, the former makes a more general statement that many would disagree with, while the latter is scoped to the speaker's own experience.
RARR: Researching and Revising What Language Models Say, Using Language Models
In some applications, one could even argue that the latter case does not require attribution, since the speaker is their own source-of-truth. In addition to varying scope, utterances can also make assertions with varying levels of directness. For example, according to standard linguistics, "John ate some of the cookies" yields the implicature that John did not eat all of the cookies, even though it is not logically entailed. This raises the question of which implicatures or implied assertions should be detected and attributed, which should be explored in future work. For more nuances, we refer to Rashkin et al. (2021). For preservation, we wish to explore other properties that should be preserved, such as discourse or logical coherence. Additionally, if the input text passage is completely misguided or flawed, it can be difficult to revise the text without significant changes, which would be heavily penalized by the current metrics.
RARR: Researching and Revising What Language Models Say, Using Language Models
Limitations of our model While we aspire to improve attribution for arbitrary text, it is clear that RARR is not yet fully general. For example, the current implementation of RARR would not be well-prepared to edit poetry (where preserving rhyme matters) or long documents, primarily because we do not provide examples of such inputs in our few-shot LLM prompts. However, we do believe that future developers may be able to quickly adapt RARR to such tasks by simply changing the prompts. Second, RARR tends to preserve rather than delete claims that it cannot attribute. Some of these claims genuinely do not require attribution, but others are hallucination and should be removed. Judging whether a claim requires attribution can be subjective and challenging. Finally, our model is computationally costly, since it is based on prompting a large language model. One potential solution is to leverage recent synthetic data generation recipes to train a smaller model (Lee et al., 2021; Schick et al., 2022).
RARR: Researching and Revising What Language Models Say, Using Language Models
Partial attribution When RARR is not 100% successful in making text consistent with retrieved evidence, the revised text will be partially attributed. One could identify unattributed parts using either the automated attribution score (Attr AIS ) or the relevance scores used to generate the attribution report (Section 3.3). Such information should be presented to avoid misleading readers into thinking that the entire revision is attributed.
RARR: Researching and Revising What Language Models Say, Using Language Models
Evidence trustworthiness RARR seeks to improve attribution for the output of any generative model. However, even if RARR can attribute content to a particular source, the user must still consider whether the source itself is trustworthy. Even for sources that are traditionally considered "authoritative" (such as an encyclopedia), there may still be factual inaccuracies or biases. This work does not address the question of whether a source is trustworthy, or the related topic of misinformation. While we do not provide a means for judging trustworthiness, the design of RARR does allow for the research stage to restrict its search over a userspecified corpus, based on what the user deems trustworthy.
RARR: Researching and Revising What Language Models Say, Using Language Models
Conflicting evidence There is also the possibility that some content may be simultaneously supported by certain sources, while contradicted by others. This can easily occur for content involving subjective or imprecise claims. The current implementation and evaluation for RARR does not explicitly address this issue -we adopted a "permissive" definition of attribution, where we consider content to be attributed if there exists any source that supports it. For some applications, a more restrictive definition that requires both existence of supporting sources and absence of contradicting sources would be needed. 
RARR: Researching and Revising What Language Models Say, Using Language Models
Luyu Gao: Designed RARR's few-shot prompting strategies and implemented the first PaLM-based prototype. Analyzed results, and advised on the design of human and automatic evaluation.
RARR: Researching and Revising What Language Models Say, Using Language Models
Zhuyun Dai: Proposed the evaluation setup of editing long-form generations from PaLM/LaMDA on various QA datasets. Hosted and mentored Luyu Gao (student researcher) in prototyping RARR. Implemented the final models, designed overall experiments, and obtained main results and ablations (together with Ice Pasupat). Contributed many parts of the writing.
RARR: Researching and Revising What Language Models Say, Using Language Models
Ice Pasupat: Implemented the final models, designed overall experiments, and obtained main results and ablations (together with Zhuyun Dai). Automated experimental infrastructure, conducted error analyses, and oversaw many parts of the paper writing.
RARR: Researching and Revising What Language Models Say, Using Language Models
Anthony Chen: Developed the automatic evaluation for attribution and preservation and worked with Arun Chaganty to design human evaluation. Developed the open-source implementation (GPT-3 RARR), made improvements to prompts, and helped with writing.
RARR: Researching and Revising What Language Models Say, Using Language Models
Arun Chaganty: Led and implemented all human evaluation. Proposed the two-dimensional attribution + preservation metric (together with Kelvin Guu). Advised on model design and contributed many parts of the writing.
RARR: Researching and Revising What Language Models Say, Using Language Models
Yicheng Fan: Worked with Kelvin Guu to develop the first prototype of RARR. Proposed multiple retrieval strategies and implemented the EFEC baseline.
RARR: Researching and Revising What Language Models Say, Using Language Models
Vincent Zhao: Co-hosted and mentored Luyu Gao (student researcher) in prototyping RARR. Enabled bulk inference for PaLM. Proposed the downstream task evaluation.
RARR: Researching and Revising What Language Models Say, Using Language Models
Ni Lao: Research mentorship, advising and contributed many parts of the writing.
RARR: Researching and Revising What Language Models Say, Using Language Models
Hongrae Lee: Research mentorship and advising. Helped integrate RARR with Google Search and evaluate LaMDA.
RARR: Researching and Revising What Language Models Say, Using Language Models
Da-Cheng Juan: Research mentorship and early design discussions.
RARR: Researching and Revising What Language Models Say, Using Language Models
Kelvin Guu: Proposed the original research-andrevise concept, implemented the first prototype, initiated the project and involved all collaborators. Implemented baselines (together with Yicheng Fan). Research mentorship, oversaw project coordination and paper writing. 
RARR: Researching and Revising What Language Models Say, Using Language Models
Model variance The main experiments in Section 5 are based on a single run. We ran automated evaluation on 3 random runs of RARR, using PaLM outputs on NQ as input passages. The standard deviations of Attr auto , Pres Lev , and F1 AP are 1.2, 0.5, and 1.0 respectively.
RARR: Researching and Revising What Language Models Say, Using Language Models
Impact of the retriever choice We tried using Microsoft Bing in place of Google Search, with near identical results (< 1% difference).
RARR: Researching and Revising What Language Models Say, Using Language Models
RARR work by few-shot prompting PaLM, a large 540B parameter LM. To assess the benefit of LM scaling, we replaced PaLM 540B with a smaller 62B parameter PaLM. As shown in Table 4, we found that 540B outperforms 62B by a large margin, suggesting that RARR could potentially further improve with even more scaling. We also experimented with keeping the editor stage at 540B while shrinking the query generation stage to 64B -this yielded a relatively small performance drop, suggesting that model scaling is more important for the editor.
RARR: Researching and Revising What Language Models Say, Using Language Models
Impact of model type Few-shot prompting has proven to be effective for many recent large language models. We try replacing the query generation model, agreement model, and edit model with GPT-3 text-davinci-003. The few-shot prompts were slightly tuned to fit the GPT-3 model. Table 4 shows the results, which are slightly better than RARR implemented with PaLM 540B on all three datasets. We will release this open-source version of RARR that uses GPT-3 as the backbone. Table 5 shows automated evaluation results on passages generated by GPT-3. The results follow the same trend as the results on PaLM and LaMDA passages.
RARR: Researching and Revising What Language Models Say, Using Language Models
Challenging domains We report results on tasks where attribution was particularly hard, and significant future work is needed. We considered news article summaries produced by summarization models from SummEval (Fabbri et al., 2021) (e.g., "John Doe was left homeless when the storms hit Staten Island, New York . . . "). Results are shown in Table 6. First, we note that the before-edit auto-AIS scores for all models are low. These news article summaries are often about less widely known people and events, which is challenging for retrievers, leading to low attribution. For example, our query generator may ask "where does John Doe live" but get results for a different John Doe. EFEC and LaMDA also face this issue, but instead trade preservation for attribution and rewrite the text to a different topic. This result suggests that using web search with standard question generation methods may fail to capture important context from the input, and is not sufficient for the attribution task.
RARR: Researching and Revising What Language Models Say, Using Language Models
We also considered long-form explanations generated by PaLM for the ELI5 dataset (Fan et al., 2019) (Table 6). ELI5 was collected from online forums, so many answers tend to have subjective opinions instead of specific entities and facts (e.g., "How do our brains interpret scary music? To me, scary music often sounds a little bit like a person . . . "), and are thus difficult to attribute. Sometimes the whole output is based on a false premise and needs to be completely rewritten, in which case RARR cannot satisfactorily edit due to our revision threshold (Section 3.2).
RARR: Researching and Revising What Language Models Say, Using Language Models
Finally, we considered technical explanations to questions from the MMLU dataset (Hendrycks et al., 2021) which covers diverse subjects from social science, humanities, STEM, and others. 4 An example input looks like "Every time you remove an edge from a complete graph, you divide it into two connected components. So, a complete graph with 13 vertices must have 12 connected components." Results are shown in  Table 4: Additional ablation results. We report the automatic metrics: Attr auto , Pres Lev , and harmonic mean between the two (F1 AP ). We show auto-AIS scores both before and after editing (before → edit), with respect to the attribution report A produced by the model. GPT    , where the OPCODE can be "done" (success), "un" (unnecessary), or "imp" (impossible). We feed the passage's context (questions for NQ and SQA; dialog context for QRECC) along with the passage itself to the input. We use beam search with beam size 8 and discard any result whose number of tokens differ by more than 4. model fine-tuned to map the input "premise: evidence hypothesis: claim sentence" to either "1" (entailed) or "0" (not entailed). As suggested by the authors, we use the probability of producing "1" as the entailment score.
RARR: Researching and Revising What Language Models Say, Using Language Models
We conducted correlation studies between human and automatic metrics and found strong Pearson correlation (attribution = 0.74; preservation = 0.62). We visualize the correlation between human and automated attribution scores on NQ and SQA in Figure 10. We found that the AIS scores from human correlate well with auto-AIS scores, with some bias for non-attributed sentences to be judged as attributed by auto-AIS.
RARR: Researching and Revising What Language Models Say, Using Language Models
To end-goal of RARR is to improve the attribution of generation models through post-editing while preserving the original intent. Attribution and preservation are both subjective properties that may change with even small edits. In the main paper, we present two automatic metrics to conveniently gauge these properties, but rely on a human evaluation as the gold standard. In this section, we describe how we conducted the human evaluation and what instructions and examples annotators were provided.
RARR: Researching and Revising What Language Models Say, Using Language Models
Rater recruitment and training We engaged with a vendor supplier of full-time crowd workers to recruit human annotators for our task. Annotators were asked to review the instructions below and were provided direct feedback on their responses during the pilot annotation runs. We had 3 annotators rate each example in the pilot phase to measure inter-annotator agreement, and had a single rater annotate each example afterwards.
RARR: Researching and Revising What Language Models Say, Using Language Models
In this task you will evaluate the quality of text generated by a system (the "passage") based on how well it represents information from multiple pieces of "evidence". We will be using two categories to evaluate the quality of the passage: Attribution and Intent Similarity. You will evaluate these categories in succession. In some tasks, you will only evaluate Attribution. The task interface will guide you through the flow; you can also see the overall task flow in the diagram below.
RARR: Researching and Revising What Language Models Say, Using Language Models
Note: The passage may appear very fluent and well-formed, but still contain slight inaccuracies that are not easy to discern at first glance. Pay close attention to the text. Read it carefully as you would when proofreading.
RARR: Researching and Revising What Language Models Say, Using Language Models
In this step, you will evaluate how much of the passage is attributable to one or more pieces of evidence ( Figure 11).
RARR: Researching and Revising What Language Models Say, Using Language Models
In the interface, the passage of text and the context in which it was generated is shown on the left, and each piece of evidence is shown on the right. You will use all three (context, passage, evidence) to answer the following question for each sentence in the passage: Is all of the information provided by this sentence fully supported by at least one piece of evidence?
RARR: Researching and Revising What Language Models Say, Using Language Models
Determining the information provided by the sentence. Three points are key when determining information provided by the sentence:
RARR: Researching and Revising What Language Models Say, Using Language Models
1. The context and the other sentences of the passage are often critical in understanding the information provided by the sentence.
RARR: Researching and Revising What Language Models Say, Using Language Models
2. The context should only be used to understand the information provided by the sentence.
RARR: Researching and Revising What Language Models Say, Using Language Models
3. The evidence should be completely ignored for this step.
RARR: Researching and Revising What Language Models Say, Using Language Models
Consider the following example:
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: who plays doug williams in days of our lives Figure 11: Screenshot of interface to annotate attribution at the sentence level. annotators were asked to mark sentences as being fully attributable or not fully attributable by clicking each sentence, and rating each piece of evidence as being useful or not in helping determine attribution of the passage. Annotators were also presented with the context of the generation. In the above example, the meaning of the passage is clear even without seeing the query. But consider another example:
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: who plays doug williams in days of our lives Passage: he is played by Bill Hayes Passage (interpreted): Doug Williams is played by Bill Hayes in days of our lives In this case the pronoun "he" depends on the context, but it is clear that the intended meaning of the passage can be reasonably interpreted as "Doug Williams is played by Bill Hayes in days of our lives". This interpretation is the "information provided by the passage".
RARR: Researching and Revising What Language Models Say, Using Language Models
Pronouns such as he/she/it/they etc. are one case where context is needed to figure out the intended meaning of the system response. Here's another example (given with paraphrases of the information highlighted below): The context should only be used to determine the information provided by the passage; at times, the passage may be about a slightly different topic than the context, for example:
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: the south west wind blows across nigeria between Passage: The Harmattan is a dry and dusty northeasterly trade wind that blows across West Africa from December to March. It is very dusty because it blows across the Sahara.
RARR: Researching and Revising What Language Models Say, Using Language Models
Here, the passage talks about a northeasterly wind, while the context asks about a south-west wind, but the passage can be fully understood.
RARR: Researching and Revising What Language Models Say, Using Language Models
In general, use your best judgment to determine the information provided by the passage. If the passage is hard to understand and you are unsure what the intended meaning of the passage is, mark the sentences as not attributed and enter a comment with an explanation. As one example, take the following:
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: how many NBA championships did Michael Jordan win? Passage: it is the best team in the NBA Determining if the information accurately represents the evidence. Two points are key when determining whether the information accurately represents the evidence: When interpreting a piece of evidence, use only the title and text of that specific evidence. Completely ignore the context, passage and all other evidence. Check all the information in a sentence. If only some information is supported by the evidence, mark the sentence as not fully attributable.
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: when did reba mcentire record back to god Passage: Back to God was released by McEntire in 2017. Evidence: "Back to God" is a song performed by American singer, Reba McEntire. It was released as the second single from her 2017 album, Sing it Now: Songs of Faith & Hope, on January 20, 2017.
RARR: Researching and Revising What Language Models Say, Using Language Models
In the above example, it is reasonable to conclude that the evidence supports all the information in the passage, and we can mark the passage as being fully attributable. But consider another example: In this case, while the evidence also mentions the "South Carolina Gamecocks", it isn't clear that the national championship being mentioned is indeed the 2017 NCAA Women's Division I Basketball Tournament. The passage should be marked as not attributable.
RARR: Researching and Revising What Language Models Say, Using Language Models
Finally, when the passage contains multiple sentences, evaluate whether each sentence can be fully attributed to one or more pieces of evidence-it is possible for one sentence to be attributed while another is not. For example: The first two sentences cannot be attributed to either evidence for the same reason as the previous example, but the last sentence is fully supported by Evidence 2 and should be marked as attributed.
RARR: Researching and Revising What Language Models Say, Using Language Models
In general, you should use your best judgment in determining whether all of the information provided by the passage is "an accurate representation of information in at least one evidence". See Table  8 for additional examples.
RARR: Researching and Revising What Language Models Say, Using Language Models
We give the following final notes of guidance:
RARR: Researching and Revising What Language Models Say, Using Language Models
• Marking evidence as useful. When reviewing each piece of evidence, mark it as useful if it helps you judge the attributability of any sentence, and mark it not useful if not. In the above example Evidence 1 is not useful because it didn't contain enough context to actually help you assess if the passage was attributable, but Evidence 2 was useful.
RARR: Researching and Revising What Language Models Say, Using Language Models
• Contradicting evidence. Mark a sentence as being attributed if any piece of evidence supports it: if two pieces of evidence contradict each other, but one of them supports the passage, mark the sentence as fully attributable.
RARR: Researching and Revising What Language Models Say, Using Language Models
• More on the concept of "accurate representation". We take as inspiration the journalist's conception of "accurate representation". For example, take this excerpt on Accuracy in the NPR Ethics Handbook: "When quoting or paraphrasing anyone . . . consider whether the source would agree with the interpretation..." In other words, if you had written the source document, consider whether you would view the system response as an accurate representation of information in that source document.
RARR: Researching and Revising What Language Models Say, Using Language Models
In this step, you will evaluate how much similar the passage is to another passage ( Figure 12). In the interface, the passage A and passage B are both text generated by a system-given the same context in which it was generated. You will use all three (context, passage A, passage B) to answer the following question: How similar is the intent expressed by Passage A and Passage B? Please ignore any differences in details.
RARR: Researching and Revising What Language Models Say, Using Language Models
Two points are key when determining whether the two passages convey the same intent:
RARR: Researching and Revising What Language Models Say, Using Language Models
1. Judge the similarity solely based on the similarity in the type and quantity of information provided by each passage.
RARR: Researching and Revising What Language Models Say, Using Language Models
2. Ignore any differences in factual details between the two passages.
RARR: Researching and Revising What Language Models Say, Using Language Models
Context: who pays medical bills in great britain where does the money come from to pay these bills  The above example should be rated "somewhat similar" because both passages are still about the same topic-Reading's owner-but differ substantially in the information they discuss: Passage A includes information about (1a) who Reading's owner is, (2a) which company they are the pres- Even though the passages look very similar, the above example should be rated "not at all similar" because the two passages are about significantly different topics: "the Lok Sabha" vs "the Rajya Sabha".
RARR: Researching and Revising What Language Models Say, Using Language Models
Few-shot prompting with LLMs We implement many sub-tasks within RARR using fewshot prompting of LLMs (also known as in-context learning (Brown et al., 2020)) as follows:
RARR: Researching and Revising What Language Models Say, Using Language Models
1. For each sub-task, we manually author a small number of training examples:
RARR: Researching and Revising What Language Models Say, Using Language Models
(input j , output j ) for j = 1, . . . , J, where J ranges between 5 and 10 and where both the input and output are strings.
RARR: Researching and Revising What Language Models Say, Using Language Models
2. We form the following prompt: input 1 ⋄ output 1 ⊕input 2 ⋄output 2 ⊕. . .⊕input J ⋄ output J ⊕ new_input, where ⋄ denotes a newline character and ⊕ denotes a double newline character.
RARR: Researching and Revising What Language Models Say, Using Language Models
3. To perform inference on a new input, we condition the LLM on the prompt and sample continuations of the prompt up until the next double newline character.
RARR: Researching and Revising What Language Models Say, Using Language Models
All of our prompts are included in Figures 13, 14, and 15. The contextual version used for QReCC are in Figures 16, 17, and 18.
RARR: Researching and Revising What Language Models Say, Using Language Models
We implemented most parts of RARR with the PALM model which has 540B parameters. We prompted PALM without any training or finetuning. We used a TPU v2-128 to run inference with PALM. We manually wrote our prompts by eye-balling quality on a dozen of examples from a separate validation set. We tune our hyperparameters on the validation set as well. We used sampling temperature 0.7 for all generation tasks. For each input text, we sample 3 question generations, and for each question we retrieve 5 results. For agreement gate and editing, we only sample 1 generation. We reject an editing if the edit distance is more than 50 characters or more than half of the original text length.  Figures 19, 20, 21, and 22 are the few-shot prompts used to generate the passages.
RARR: Researching and Revising What Language Models Say, Using Language Models
Following the corresponding datasets, all generated passages are in English. The authors have manually looked through most of the data and found no personal identifiers. 1 [web] I will check things you said and ask questions. (6) You said: Social work is a profession that is based in the philosophical tradition of humanism. It is an intellectual discipline that has its roots in the 1800s. (1) You said: Your nose switches back and forth between nostrils. When you sleep, you switch about every 45 minutes. This is to prevent a buildup of mucus. It's called the nasal cycle. 4 I checked: How often do your nostrils switch? 5 I found this article: Although we don't usually notice it, during the nasal cycle one nostril becomes congested and thus contributes less to airflow, while the other becomes decongested. On average, the congestion pattern switches about every 2 hours, according to a small 2016 study published in the journal PLOS One. 6 Your nose's switching time is about every 2 hours, not 45 minutes. 7 This disagrees with what you said. (4) You said: Social work is a profession that is based in the philosophical tradition of humanism. It is an intellectual discipline that has its roots in the 1800s.
RARR: Researching and Revising What Language Models Say, Using Language Models
I checked: When did social work have its roots? 23 I found this article: The Emergence and Growth of the Social work Profession<br><br> Social work's roots were planted in the 1880s, when charity organization societies (COS) were created to organize municipal voluntary relief associations and settlement houses were established. 24 Social work has its roots in the 1880s, not 1800s. 25 This disagrees with what you said.  Havel-Hakimi algorithm is for constructing a special solution if a simple graph for the given degree sequence exists, or proving that one cannot find a positive answer, not converting the adjacency matrix of a graph into its adjacency list. 31 This disagrees with what you said.  (2) Context: The Stanford Prison Experiment was conducted in the basement of Encina Hall, Stanford's psychology building. 11 You said: It is a psychological study to observe the behaviors of conflict and violence that happen between inmates and prisoners in real prisons. 12 To verify what you just said, B2. Did you discuss the license or terms for use and / or distribution of any artifacts? E B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? E B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 5,E B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5 C Did you run computational experiments? 5,6 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5,D The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
RARR: Researching and Revising What Language Models Say, Using Language Models
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5,D C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5,B,D D Did you use human annotators (e.g., crowdworkers) or research with human participants? 5,C D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? C D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? C D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? C D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
RARR: Researching and Revising What Language Models Say, Using Language Models
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? C
Fact-based Text Editing
Automatic editing of text by computer is an important application, which can help human writers to write better documents in terms of accuracy, fluency, etc. The task is easier and more practical than the automatic generation of texts from scratch and is attracting attention recently Yin et al., 2019). In this paper, we consider a new and specific setting of it, referred to as fact-based text editing, in which a draft text and several facts (represented in triples) are given, and the system * The work was done when Hayate Iso was a research intern at ByteDance AI Lab. Revised text Baymax was created by American creators Duncan Rouleau and Steven T. Seagle . Baymax is a character in Big Hero 6 which stars Scott Adsit . Table 1: Example of fact-based text editing. Facts are represented in triples. The facts in green appear in both draft text and triples. The facts in orange are present in the draft text, but absent from the triples. The facts in blue do not appear in the draft text, but in the triples. The task of fact-based text editing is to edit the draft text on the basis of the triples, by deleting unsupported facts and inserting missing facts while retaining supported facts.
Fact-based Text Editing
aims to revise the text by adding missing facts and deleting unsupported facts. Table 1 gives an example of the task. As far as we know, no previous work did address the problem. In a text-to-text generation, given a text, the system automatically creates another text, where the new text can be a text in another language (machine translation), a summary of the original text (summarization), or a text in better form (text editing). In a table-to-text generation, given a table containing facts in triples, the system automatically composes a text, which describes the facts. The former is a text-to-text problem, and the latter a table-to-text problem. In comparison, fact-based text editing can be viewed as a 'text&table-to-text' problem.
Fact-based Text Editing
First, we devise a method for automatically creating a dataset for fact-based text editing. Recently, several table-to-text datasets have been created and released, consisting of pairs of facts and corresponding descriptions. We leverage such kind of data in our method. We first retrieve facts and their descriptions. Next, we take the descriptions as revised texts and automatically generate draft texts based on the facts using several rules. We build two datasets for fact-based text editing on the basis of WEBNLG (Gardent et al., 2017) and ROTOWIRE, consisting of 233k and 37k instances respectively (Wiseman et al., 2017) 
Fact-based Text Editing
Second, we propose a model for fact-based text editing called FACTEDITOR. One could employ an encoder-decoder model, such as an encoderdecoder model, to perform the task. The encoderdecoder model implicitly represents the actions for transforming the draft text into a revised text. In contrast, FACTEDITOR explicitly represents the actions for text editing, including Keep, Drop, and Gen, which means retention, deletion, and generation of word respectively. The model utilizes a buffer for storing the draft text, a stream to store the revised text, and a memory for storing the facts. It also employs a neural network to control the entire editing process. FACTEDITOR has a lower time complexity than the encoder-decoder model, and thus it can edit a text more efficiently.
Fact-based Text Editing
Experimental results show that FACTEDITOR outperforms the baseline model of using encoderdecoder for text editing in terms of fidelity and fluency, and also show that FACTEDITOR can perform text editing faster than the encoder-decoder model.
Fact-based Text Editing
Text editing has been studied in different settings such as automatic post-editing (Knight and Chander, 1994;Simard et al., 2007;Yang et al., 2017), paraphrasing (Dolan andBrockett, 2005), sentence simplification (Inui et al., 2003;Wubben et al., 2012), grammar error correction (Ng et al., 2014), and text style transfer (Shen et al., 2017;Hu et al., 2017).
Fact-based Text Editing
The rise of encoder-decoder models (Cho et al., 2014;Sutskever et al., 2014) as well as the attention (Bahdanau et al., 2015;Vaswani et al., 2017) and copy mechanisms (Gu et al., 2016;Gulcehre et al., 2016) has dramatically changed the landscape, and now one can perform the task relatively easily with an encoder-decoder model such as Transformer provided that a sufficient amount of data is available. For example, Li et al. (2018) introduce a deep reinforcement learning framework for paraphrasing, consisting of a generator and an evaluator. Yin et al. (2019) formalize the problem of text edit as learning and utilization of edit representations and propose an encoder-decoder model for the task. Zhao et al. (2018) integrate paraphrasing rules with the Transformer model for text simplification. Zhao et al. (2019) proposes a method for English grammar correction using a Transformer and copy mechanism.
Fact-based Text Editing
Another approach to text editing is to view the problem as sequential tagging instead of encoderdecoder. In this way, the efficiency of learning and prediction can be significantly enhanced. Vu and Haffari (2018) and  conduct automatic post-editing and text simplification on the basis of edit operations and employ Neural Programmer-Interpreter (Reed and De Freitas, 2016) to predict the sequence of edits given a sequence of words, where the edits include KEEP, DROP, and ADD. Malmi et al. (2019) propose a sequential tagging model that assigns a tag (KEEP or DELETE) to each word in the input sequence and also decides whether to add a phrase before the word. Our proposed approach is also based on sequential tagging of actions. It is designed for fact-based text editing, not text-to-text generation, however. Table-to-text generation is the task which aims to generate a text from structured data (Reiter and Dale, 2000;Gatt and Krahmer, 2018), for example, a text from an infobox about a term in biology in wikipedia (Lebret et al., 2016) and a description of restaurant from a structured representation (Novikova et al., 2017). Encoder-decoder models can also be employed in table-to-text generation with structured data as input and generated text as output, for example, as in (Lebret et al., 2016). Puduppully et al. (2019) and Iso et al. (2019) propose utilizing an entity tracking module for document-level table-to-text generation.
Fact-based Text Editing
One issue with table-to-text is that the style of generated texts can be diverse (Iso et al., 2019). Re-y AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission that was operated by PATIENT-2 . x AGENT-1 served as PATIENT-3 was a crew member of the BRIDGE-1 mission .
Fact-based Text Editing
x
Fact-based Text Editing
AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission .
Fact-based Text Editing
(a) Example for insertion. The revised template y and the reference templatex share subsequences. The set of triple templates T \T is {(BRIDGE-1, operator, PATIENT-2)}. Our method removes "that was operated by PATIENT-2" from the revised template y to create the draft template x .
Fact-based Text Editing
y AGENT-1 was created by BRIDGE-1 and PATIENT-2 .
Fact-based Text Editing
The character of AGENT-1 , whose full name is PATIENT-1 , was created by BRIDGE-1 and PATIENT-2 . x AGENT-1 , whose full name is PATIENT-1 , was created by BRIDGE-1 and PATIENT-2 .
Fact-based Text Editing
(b) Example for deletion. The revised template y and the reference templatex share subsequences. The set of triple templateŝ T \T is {(AGENT-1, fullName, PATIENT-1)}. Our method copies "whose full name is PATIENT-1" from the reference template x to create the draft template x . searchers have developed methods to deal with the problem using other texts as templates Peng et al., 2019). The difference between the approach and factbased text editing is that the former is about tableto-text generation based on other texts, while the latter is about text-to-text generation based on structured data.
Fact-based Text Editing
In this section, we describe our method of data creation for fact-based text editing. The method automatically constructs a dataset from an existing table-to-text dataset.
Fact-based Text Editing
There are two benchmark datasets of table-totext, WEBNLG (Gardent et al., 2017) 2 and RO-TOWIRE(Wiseman et al., 2017) 3 . We create two datasets on the basis of them, referred to as WEBE-DIT and ROTOEDIT respectively. In the datasets, each instance consists of a table (structured data) and an associated text (unstructured data) describing almost the same content. 4 . For each instance, we take the table as triples of facts and the associated text as a revised text, and we automatically create a draft text. The set of triples is represented as T = {t}. Each triple t consists of subject, predicate, and object, denoted 2 The data is available at https://github.com/ ThiagoCF05/webnlg. We utilize version 1.5. 3 We utilize the ROTOWIRE-MODIFIED data provided by Iso et al. (2019) available at https://github.com/ aistairc/rotowire-modified. The authors also provide an information extractor for processing the data. 4 In ROTOWIRE, we discard redundant box-scores and unrelated sentences using the information extractor and heuristic rules. as t = (subj, pred, obj). For simplicity, we refer to the nouns or noun phrases of subject and object simply as entities. The revised text is a sequence of words denoted as y. The draft text is a sequence of words denoted as x.
Fact-based Text Editing
Given the set of triples T and the revised text y, we aim to create a draft text x, such that x is not in accordance with T , in contrast to y, and therefore text editing from x to y is needed.
Fact-based Text Editing
Our method first creates templates for all the sets of triples and revised texts and then constructs a draft text for each set of triples and revised text based on their related templates.
Fact-based Text Editing
For each instance, our method first delexicalizes the entity words in the set of triples T and the revised text y to obtain a set of triple templates T and a revised template y . For example, given T ={(Baymax, voice, Scott Adsit)} and y ="Scott Adsit does the voice for Baymax", it produces the set of triple templates T ={(AGENT-1, voice, PATIENT-1)} and the revised template y ="AGENT-1 does the voice for PATIENT-1". Our method then collects all the sets of triple templates T and revised templates y and retains them in a key-value store with y being a key and T being a value.
Fact-based Text Editing
Next, our method constructs a draft text x using a set of triple templates T and a revised template y . For simplicity, it only considers the use of either insertion or deletion in the text editing, and one can easily make an extension of it to a more complex setting. Note that the process of data creation is reverse to that of text editing.
Fact-based Text Editing
Given a pair of T and y , our method retrieves another pair denoted asT andx , such that y and x have the longest common subsequences. We refer tox as a reference template. There are two possibilities;T is a subset or a superset of T .
Fact-based Text Editing
(We ignore the case in which they are identical.) Our method then manages to change y to a draft template denoted as x on the basis of the relation between T andT . IfT T , then the draft template x created is for insertion, and ifT T , then the draft template x created is for deletion.
Fact-based Text Editing
For insertion, the revised template y and the reference templatex share subsequences, and the set of triples T \T appear in y but not inx . Our method keeps the shared subsequences in y , removes the subsequences in y about T \T , and copies the rest of words in y , to create the draft template x . Table 2a gives an example. The shared subsequences "AGENT-1 performed as PATIENT-3 on BRIDGE-1 mission" are kept. The set of triple templates T \T is {(BRIDGE-1, operator, PATIENT-2)}. The subsequence "that was operated by PATIENT-2" is removed. Note that the subsequence "served" is not copied because it is not shared by y andx .
Fact-based Text Editing
For deletion, the revised template y and the reference templatex share subsequences. The set of triplesT \T appear inx but not in y . Our method retains the shared subsequences in y , copies the subsequences inx aboutT \T , and copies the rest of words in y , to create the draft template x . Table 2b gives an example. The subsequences "AGENT-1 was created by BRIDGE-1 and PATIENT-2" are retained. The set of triple templatesT \T is {(AGENT-1, full-Name, PATIENT-1)}. The subsequence "whose full name is PATIENT-1" is copied. Note that the subsequence "the character of" is not copied because it is not shared by y andx .
Fact-based Text Editing
After getting the draft template x , our method lexicalizes it to obtain a draft text x, where the lexicons (entity words) are collected from the corresponding revised text y.
Fact-based Text Editing
We obtain two datasets with our method, referred to as WEBEDIT and ROTOEDIT, respectively. Table 3 gives the statistics of the datasets.
Fact-based Text Editing
In the WEBEDIT data, sometimes entities only appear in the subj's of triples. In such cases, we also make them appear in the obj's. To do so, we  introduce an additional triple (ROOT, IsOf, subj) for each subj, where ROOT is a dummy entity.
Fact-based Text Editing
In this section, we describe our proposed model for fact-based text editing referred to as FACTEDITOR.
Fact-based Text Editing
FACTEDITOR transforms a draft text into a revised text based on given triples. The model consists of three components, a buffer for storing the draft text and its representations, a stream for storing the revised text and its representations, and a memory for storing the triples and their representations, as shown in Figure 1. FACTEDITOR scans the text in the buffer, copies the parts of text from the buffer into the stream if they are described in the triples in the memory, deletes the parts of the text if they are not mentioned in the triples, and inserts new parts of next into the stream which is only presented in the triples.
Fact-based Text Editing
The architecture of FACTEDITOR is inspired by those in sentence parsing Dyer et al. (2015); Watanabe and Sumita (2015). The actual processing of FACTEDITOR is to generate a sequence of words into the stream from the given sequence of words in the buffer and the set of triples in the memory. A neural network is employed to control the entire editing process.
Fact-based Text Editing
Initialization FACTEDITOR first initializes the representations of content in the buffer, stream, and memory.
Fact-based Text Editing
There is a feed-forward network associated with the memory, utilized to create the embeddings of triples. Let M denote the number of triples. The embedding of triple t j , j = 1, · · · , M is calculated as
Fact-based Text Editing
where W t and b t denote parameters, e subj j , e pred j , e obj j denote the embeddings of subject, predicate, and object of triple t j , and [ ; ] denotes vector concatenation.
Fact-based Text Editing
There is a bi-directional LSTM associated with the buffer, utilized to create the embeddings of words of draft text. The embeddings are obtained as b = BILSTM(x), where x = (x 1 , . . . , x N ) is the list of embeddings of words and b = (b 1 , . . . , b N ) is the list of representations of words, where N denotes the number of words.
Fact-based Text Editing
There is an LSTM associated with the stream for representing the hidden states of the stream. The first hidden state is initialized as
Fact-based Text Editing
where W s and b s denotes parameters.
Fact-based Text Editing
FACTEDITOR predicts an action at each time t using the LSTM. There are three types of action, namely Keep, Drop, and Gen. First, it composes a context vectort t of triples at time t using attentiont
Fact-based Text Editing
where α t,j is a weight calculated as
Fact-based Text Editing
where v α and W α are parameters. Then, it creates the hidden state z t for action prediction at time t
Fact-based Text Editing
where W z and b z denote parameters. Next, it calculates the probability of action a t P (a t | z t ) = softmax(W a · z t )
Fact-based Text Editing
where W a denotes parameters, and chooses the action having the largest probability.  
Fact-based Text Editing
FACTEDITOR takes action based on the prediction result at time t. For Keep at time t, FACTEDITOR pops the top embedding b t in the buffer, and feeds the combination of the top embedding b t and the context vector of triplest t into the stream, as shown in Fig. 1a. The state of stream is updated with the LSTM as s t+1 = LSTM([t t ; b t ], s t ). FACTEDITOR also copies the top word in the buffer into the stream.
Fact-based Text Editing
For Drop at time t, FACTEDITOR pops the top embedding in the buffer and proceeds to the next state, as shown in Fig. 1b. The state of stream is updated as s t+1 = s t . Note that no word is inputted into the stream.
Fact-based Text Editing
For Gen at time t, FACTEDITOR does not pop the top embedding in the buffer. It feeds the Draft text x Bakewell pudding is Dessert that can be served Warm or cold .
Fact-based Text Editing
Revised text y Bakewell pudding is Dessert that originates from Derbyshire Dales . combination of the context vector of triplest t and the linearly projected embedding of word w into the stream, as shown in Fig. 1c. The state of stream is updated with the LSTM as s t+1 = LSTM([t t ; W p y t ], s t ), where y t is the embedding of the generated word y t and W p denotes parameters. In addition, FACTEDITOR copies the generated word y t into the stream. FACTEDITOR continues the actions until the buffer becomes empty.
Fact-based Text Editing
FACTEDITOR generates a word y t at time t, when the action is Gen,
Fact-based Text Editing
where W y is parameters.
Fact-based Text Editing
To avoid generation of OOV words, FACTEDI-TOR exploits the copy mechanism. It calculates the probability of copying the object of triple t j
Fact-based Text Editing
where v c and W c denote parameters, and o j is the object of triple t j . It also calculates the probability of gating
Fact-based Text Editing
where w g and b g are parameters. Finally, it calculates the probability of generating a word w t through either generation or copying,
Fact-based Text Editing
where it is assumed that the triples in the memory have the same subject and thus only objects need to be copied.
Fact-based Text Editing
The conditional probability of sequence of actions a = (a 1 , a 2 , · · · , a T ) given the set of triples T and the sequence of input words x can be written as
Fact-based Text Editing
where P (a t | z t ) is the conditional probability of action a t given state z t at time t and T denotes the number of actions.
Fact-based Text Editing
The conditional probability of sequence of generated words y = (y 1 , y 2 , · · · , y T ) given the sequence of actions a can be written as
Fact-based Text Editing
where P (y t | a t ) is the conditional probability of generated word y t given action a t at time t, which is calculated as
Fact-based Text Editing
Note that not all positions have a generated word. In such a case, y t is simply a null word.
Fact-based Text Editing
The learning of the model is carried out via supervised learning. The objective of learning is to minimize the negative log-likelihood of P (a | T , x) and P (y | a)
Fact-based Text Editing
where θ denotes the parameters.
Fact-based Text Editing
A training instance consists of a pair of draft text and revised text, as well as a set of triples, denoted as x, y, and T respectively. For each instance, our method derives a sequence of actions denoted as a, in a similar way as that in . It first finds the longest common subsequence between x and y, and then selects an action of Keep, Drop, or Gen at each position, according to how y is obtained from x and T (cf., Tab. 4). Action Gen is preferred over action Drop when both are valid.  Table Encoder Text Encoder Decoder y x T (c) ENCDECEDITOR Figure 2: Model architectures of the baselines. All models employ attention and copy mechanism.
Fact-based Text Editing
The time complexity of inference in FACTEDITOR is O(N M ), where N is the number of words in the buffer, and M is the number of triples. Scanning of data in the buffer is of complexity O(N ). The generation of action needs the execution of attention, which is of complexity O(M ). Usually, N is much larger than M .
Fact-based Text Editing
We consider a baseline method using the encoderdecoder architecture, which takes the set of triples and the draft text as input and generates a revised text. We refer to the method as ENCDECEDITOR. The encoder of ENCDECEDITOR is the same as that of FACTEDITOR. The decoder is the standard attention and copy model, which creates and utilizes a context vector and predicts the next word at each time.
Fact-based Text Editing
The time complexity of inference in ENCDE-CEDITOR is O(N 2 + N M ) (cf., Britz et al. (2017)). Note that in fact-based text editing, usually N is very large. That means that ENCDECEDITOR is less efficient than FACTEDITOR.
Fact-based Text Editing
We conduct experiments to make comparison between FACTEDITOR and the baselines using the two datasets WEBEDIT and ROTOEDIT.
Fact-based Text Editing
The main baseline is the encoder-decoder model ENCDECEDITOR, as explained above. We further consider three baselines, No-Editing, Table-to-Text, and Text-to-Text. In No-Editing, the draft text is directly used. In Table-to-Text, a revised text is generated from the triples using encoder-decoder. In Text-to-Text, a revised text is created from the draft text using the encoder-decoder model. Figure  2 gives illustrations of the baselines.
Fact-based Text Editing
We evaluate the results of revised texts by the models from the viewpoint of fluency and fidelity.
Fact-based Text Editing
We utilize ExactMatch (EM), BLEU (Papineni et al., 2002) and SARI (Xu et al., 2016) scores 5 as evaluation metrics for fluency. We also utilize precision, recall, and F1 score as evaluation metrics for fidelity. For WEBEDIT, we extract the entities from the generated text and the reference text and then calculate the precision, recall, and F1 scores. For ROTOEDIT, we use the information extraction tool provided by Wiseman et al. (2017) for calculation of the scores.
Fact-based Text Editing
For the embeddings of subject and object for both datasets and the embedding of the predicate for ROTOEDIT, we simply use the embedding lookup table. For the embedding of the predicate for WEBEDIT, we first tokenize the predicate, lookup the embeddings of lower-cased words from the table, and use averaged embedding to deal with the OOV problem (Moryossef et al., 2019).
Fact-based Text Editing
We tune the hyperparameters based on the BLEU score on a development set. For WEBEDIT, we set the sizes of embeddings, buffers, and triples to 300, and set the size of the stream to 600. For ROTOEDIT, we set the size of embeddings to 100 and set the sizes of buffers, triples, and stream to 200. The initial learning rate is 2e-3, and AMS-Grad is used for automatically adjusting the learning rate (Reddi et al., 2018). Our implementation makes use of AllenNLP (Gardner et al., 2018).
Fact-based Text Editing
We present the performances of our proposed model FACTEDITOR and the baselines on factbased text editing in Table 5. One can draw several conclusions from the results.
Fact-based Text Editing
First, our proposed model, FACTEDITOR, achieves significantly better performances than the main baseline, ENCDECEDITOR, in terms of almost all measures. In particular, FACTEDITOR  obtains significant gains in DELETE scores on both WEBEDIT and ROTOEDIT. Second, the fact-based text editing models (FACTEDITOR and ENCDECEDITOR) significantly improve upon the other models in terms of fluency scores, and achieve similar performances in terms of fidelity scores.
Fact-based Text Editing
Third, compared to No-Editing, Table-to-Text has higher fidelity scores, but lower fluency scores. Text-to-Text has almost the same fluency scores, but lower fidelity scores on ROTOEDIT.
Fact-based Text Editing
We also manually evaluate 50 randomly sampled revised texts for WEBEDIT. We check whether the revised texts given by FACTEDITOR and ENCDE-CEDITOR include all the facts. We categorize the factual errors made by the two models. Table 6 shows the results. One can see that FACTEDITOR covers more facts than ENCDECEDITOR and has less factual errors than ENCDECEDITOR.
Fact-based Text Editing
FACTEDITOR has a larger number of correct editing (CQT) than ENCDECEDITOR for fact-based text editing. In contrast, ENCDECEDITOR often includes a larger number of unnecessary rephrasings (UPARA) than FACTEDITOR.  There are four types of factual errors: fact repetitions (RPT), fact missings (MS), fact unsupported (USUP), and relation difference (DREL). Both FACTEDITOR and ENCDECEDITOR often fail to insert missing facts (MS), but rarely insert unsupported facts (USUP). ENCDECEDITOR often generates the same facts multiple times (RPT) or facts in different relations (DREL). In contrast, FACTE-DITOR can seldomly make such errors. Table 7 shows an example of results given by ENCDECEDITOR and FACTEDITOR. The revised texts of both ENCDECEDITOR and FACTEDITOR appear to be fluent, but that of FACTEDITOR has higher fidelity than that of ENCDECEDITOR. ENCDECEDITOR cannot effectively eliminate the Set of triples {(Ardmore Airport, runwayLength, 1411.0), (Ardmore Airport, 3rd runway SurfaceType, Poaceae), (Ardmore Airport, operatingOrganisation, Civil Aviation Authority of New Zealand), (Ardmore Airport, elevationAboveTheSeaLevel, 34.0), (Ardmore Airport, runwayName, 03R/21L)}
Fact-based Text Editing
Ardmore Airport , ICAO Location Identifier UTAA . Ardmore Airport 3rd runway is made of Poaceae and Ardmore Airport . 03R/21L is 1411.0 m long and Ardmore Airport is 34.0 above sea level .
Fact-based Text Editing
Ardmore Airport is operated by Civil Aviation Authority of New Zealand . Ardmore Airport 3rd runway is made of Poaceae and Ardmore Airport name is 03R/21L . 03R/21L is 1411.0 m long and Ardmore Airport is 34.0 above sea level .
Fact-based Text Editing
Ardmore Airport , ICAO Location Identifier UTAA , is operated by Civil Aviation Authority of New Zealand . Ardmore Airport 3rd runway is made of Poaceae and Ardmore Airport . 03R/21L is 1411.0 m long and Ardmore Airport is 34.0 m long .
Fact-based Text Editing
Ardmore Airport is operated by Civil Aviation Authority of New Zealand . Ardmore Airport 3rd runway is made of Poaceae and Ardmore Airport . 03R/21L is 1411.0 m long and Ardmore Airport is 34.0 above sea level .  description about an unsupported fact (in orange) appearing in the draft text. In contrast, FACTEDI-TOR can deal with the problem well. In addition, ENCDECEDITOR conducts an unnecessary substitution in the draft text (underlined). FACTEDITOR tends to avoid such unnecessary editing.
Fact-based Text Editing
We conduct runtime analysis on FACTEDITOR and the baselines in terms of number of processed words per second, on both WEBEDIT and RO-TOEDIT. Table 8 gives the results when the batch size is 128 for all methods. Table-to-Text is the fastest, followed by FACTEDITOR. FACTEDITOR is always faster than ENCDECEDITOR, apparently because it has a lower time complexity, as explained in Section 4. The texts in WEBEDIT are relatively short, and thus FACTEDITOR and ENCDE-CEDITOR have similar runtime speeds. In contrast, the texts in ROTOEDIT are relatively long, and thus FACTEDITOR executes approximately two times faster than ENCDECEDITOR.
Fact-based Text Editing
In this paper, we have defined a new task referred to as fact-based text editing and made two contributions to research on the problem. First, we have proposed a data construction method for fact-based text editing and created two datasets. Second, we have proposed a model for fact-based text editing, named FACTEDITOR, which performs the task by generating a sequence of actions. Experimental results show that the proposed model FACTEDI-TOR performs better and faster than the baselines, including an encoder-decoder model.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Determining the truthfulness of factual claims by comparing them to textual sources of evidence has received intense research interest in recent years. An underlying, but often overlooked, challenge for this paradigm, however, is the dynamic nature of today's written resources. An extraordinary amount of new information becomes available daily; as a result, many consequential facts are established, changed, or added to over time. We argue that the quality of fact verification systems should be its population is estimated to be 86,205, almost 14% more than the 2000 census figure of 76,129.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
its population is estimated to be 91,757, almost 14% more than the 2000 census figure of 76,129.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
From Wikipedia, the free encyclopedia Revision ID: 336934876
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Revision as of 04:10, 10 January 2010
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
More than 90K people live in Beaverton Claim:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
R ef u te s S u p p o r t s Figure 1: In VITAMINC, we focus on Wikipedia revisions in which the factual content changes. This example revision now supports an initially refuted claim. measured by how well they adjust to new evidence. In this way, we seek to advance fact verification by requiring that models remain reliable and robust to the change present in practical settings.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
To this end, we focus on fact verification with contrastive evidence. That is, we infuse the standard fact verification paradigm with challenging cases that require models to be sensitive to factual changes in their presented evidence (hereon referred to interchangeably as "context"). We present VITAMINC, 2 a new large-scale fact verification dataset that is based on factual revisions to Wikipedia. The key concept is exemplified in Figure 1: there a factual revision yields a contrastive pair of contexts that are nearly identical in language and content-except that one context refutes the given claim, while the other supports it.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
This type of contrastive structure exposes existing deficiencies in model behavior. To illustrate this, we train a classifier on the popular FEVER fact verification dataset (Thorne et al., 2018) and evaluate it on contrastive claim-evidence pairs. We find that the model flips its prediction from the original verdict on only 56% of the contrastive cases. When examples from VITAMINC are included during training, however, the model's sensitivity increases, flipping on 86% of contrastive cases.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Such context-sensitive inference has two main benefits. First, it ensures that the model consid-ers the provided evidence rather than relying on built-in static knowledge, such as that obtained via language model pre-training (Petroni et al., 2019;Roberts et al., 2020). This is particularly important for scenarios in which the source of truth is mutable (e.g., the current US president, or new declarations as in Figure 1). Second, this setting discourages certain biases and idiosyncrasies-such as exploiting differences in how true vs. false claims are posed-that are common in similar crowd-sourced datasets (Poliak et al., 2018;Schuster et al., 2019). Indeed, we show that augmenting both fact verification models and NLI models with VITAMINC data improves their robustness to adversarial inputs.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Furthermore, our emphasis on contrastive contexts allows us to expand on the scope of commonly considered tasks. Most of the fact verification literature focuses on resolving claims to be true or false (Popat et al., 2018;Thorne and Vlachos, 2018;Wang, 2017). The surrounding ecosystem, however, includes additional challenges, some of which we explore here: Documents such as Wikipedia articles are updated frequently; which edits represent factual changes? For a given claim and (refuting or supporting) evidence pair, which words or phrases in the evidence are most relevant? If we know that a certain claim is true, can we modify an out-dated document to be consistent with it? We show that the unique structure of our VITAMINC dataset can be leveraged to provide both supervised and distantly supervised data for these new questions.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Our key contributions are as follows:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
1. We pose a contrastive fact verification paradigm that requires sensitivity to changes in data;
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
2. We introduce VITAMINC, a new large-scale dataset that supports this paradigm;
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
3. We demonstrate that training on VITAMINC leads to better performance on standard tasks; 4. We show how VITAMINC opens the door to additional research directions in fact verification.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
VITAMINC (abbreviated VitC) is based on revisions to English Wikipedia. Wikipedia has become a comprehensive online resource that is rigorously maintained by a large and active community (Benjakob and Harrison, 2019). While adversaries do try to insert disinformation, popular pages are usually quickly corrected (Kumar et al., 2016). Furthermore, Wikipedia's policies dictate that its content should be written from a neutral perspectiveor should otherwise objectively state all points of view. 3 These properties make Wikipedia a suitable source of evidence for fact verification models. In the following section, we outline our process for mining factual revisions from Wikipedia.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We collected the 5K most-viewed English Wikipedia articles 4 as of January 2020, along with any additional articles referred from them (on average 100 per article). We also included all articles from the FEVER dataset (Thorne et al., 2018). For each article, we retrieved up to 500 of its most recent revisions. In May 2020, we added all COVID-19 related articles 5 and all of their 41K revisions at the time. Combined together, this resulted in a total of ∼200 million revisions. For each revision, we identified all of the modified sentences and stored two versions: (1) before, and (2) after the edit. In our task, we are only interested in edits made with an intent to introduce a factual modificationi.e., a change for which one can make a claim that is supported by one sentence, but not by the other. 6 To expedite annotation, we trained a BERT classifier (Devlin et al., 2019) on a small labeled set of revised sentences determined to be factual (Yang et al., 2017), and used this model to select the top 305K edited sentences from the corpus for manual annotation. Trained human annotators were then presented with the sentence pairs, and were asked to mark the ones that indeed represented a factual change. Sentences lacking self-contained context were filtered (e.g., short expressions from tables or bulleted lists). Example annotations are presented in Table 1. Note that these annotations can also be recursively recycled for re-training the automated BERT classifier in the future to expand the corpus further (we also introduce this as a task, see §4.1).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The factual Wikipedia revisions guide us in creating challenging claims for fact verification. For each revision, annotators were asked to write two symmetric claims related to the same edit:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
1. The first should be supported by the original sentence and refuted by the revised sentence; 2. The second should be supported by the revised sentence and refuted by the original sentence. When an explicit contradiction was not possible, a not enough information (NEI) relation was used. A group of 70 native English speakers 7 wrote and reviewed claims. During the annotation period, annotations were delivered in weekly batches, from which we examined random samples to provide feedback and request corrections. Annotators were instructed to write short and self-contained claims. Furthermore, annotators were instructed to avoid copying exact phrases and values when possible, in order to avoid a bias for substantially higher word overlap in supporting pairs over refuting pairs. For example, rather than stating, "there are x confirmed cases of coronavirus in the US", one can write "there are more than z confirmed cases of coronavirus in the US", which is supported if x > z and refuted otherwise. For revisions that only add new information or that remove outdated facts without replacing them, annotators wrote a single claim.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Naturally, the real Wikipedia revisions we collect mostly describe facts that frequently change over time, or that are prone to mistakes and corrections (such as quantitative values, see Appendix A.1) (Faruqui et al., 2018;Yang et al., 2017). Sensitivity to contrastive contexts, however, is desirable behavior for any claim. This can both ensure consistency with external sources of truth, and improve the model's faithfulness via connecting the verdict with a specific evidence (Jacovi and Goldberg, 2020;Ross et al., 2020). For example, we require the model to not only classify the claim "Tom Hanks was honored by a president" as true, but to also change its verdict to false if paired with a (fictional) contrasting evidence. As a result, we can verify that the model prioritizes sentence-pair inference over 627 Factual Wikipedia sentences before and after a revision, presented with VITAMINC claims if the revision is factual.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Before More stringent actions were taken in China once the seriousness of the outbreak became apparent, such as quarantining entire cities affecting 60 million individuals in Hubei, and strict travel bans.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
More drastic actions were taken in China once the severity of the outbreak became apparent, such as quarantining entire cities affecting 60 million individuals in Hubei, and strict travel bans. Before
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
In animals, spaying involves an invasive removal of the ovaries, but rarely has major complications other than that spayed animals tend to gain weight.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
In animals, spaying involves an invasive removal of the ovaries, but rarely has major complications; the superstition that it causes weight gain is not based on fact.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Claim 1 Spayed animals gain weight. Claim 2 Weight gain in spayed animals is a superstitious myth. Before As of 16 March, more than 182,000 cases of the disease have been reported in over 160 countries and territories, resulting in around 79,000 recoveries and more than 7,100 deaths.   et al., 2018). Specifically, given a false claim from FEVER, we ask annotators to edit the sentence that refutes it so that it will then support the originally false claim. Additionally, we ask them to write a new claim that is refuted by the new, modified sentence, but that is supported by the original version. Following this method, we obtain two claims where each can be supported or refuted by the original, or the synthetically revised, sentence. We follow the same process for constructing synthetic examples using true claims, but with flipped labels.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
In total, 304,671 revised Wikipedia sentences were examined by annotators, of which 107,056 (35%) were found to express a factual modification and were passed to the group of expert annotators for claim writing. As two symmetric claims with opposing facts were created (when possible) for each revision, this resulted in a total of 325,724 total claim-evidence pairs. We collected 163,180 addi-  tional pairs following the synthetic process. The data was partitioned as shown in Table 2. The assignment was done randomly by article, and is consistent with FEVER for overlapping articles. Appendix A contains additional details.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The unique structure of VITAMINC allows us to derive annotations that provide a novel source of supervision for several fact-verification-related tasks. We describe the four main tasks we consider in this work, along with baseline models: (1) factual revision flagging, (2) fact verification, (3) word-level rationales, and (4) factually consistent generation. Figure 2 illustrates an example from VITAMINC. We use the following notations:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
• C is the space of short sentences that express an arbitrary factual statement that can potentially be verified or debunked by external sources. The revision (s t-1 ,s t ) is factual because it modifies when the first COVID-19 case was identified
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The following claim is refuted by s t-1 and supported by s t :
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
"COVID-19 outbreak was identified before December"
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The anchoring words in s t that support the claim are: "first identified [...]  • S is the space of sentences that can be found in a trusted online resource (Wikipedia in this study).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
• (s t−1 , s t ) denotes the two versions of a sentence that was revised from s t−1 to s t ∈ S.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
• rel(c, s) denotes the relation between the claim c ∈ C and observed evidence s ∈ S-which can either support c (SUP), refute it (REF), or not contain enough information (NEI).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Online resources like Wikipedia are continuously changing. In order to remain a reliable and neutral source for recent information, its active community of users must constantly verify and correct the revisions of others. We define factual revision flagging as the task of identifying revisions that introduce a factual change-e.g., by either modifying a certain fact, adding a new one, or removing an existing one. Such an automated detection process can help the community moderate important articles by serving as a watchdog for factual revisions. Furthermore, tracking factual revisions to certain articles can potentially help keep reliant articles consistent (e.g., citing articles, or non-English versions). We pose factual revision flagging as a binary classification function f flag : S × S → {0, 1}, where for a revision (s t−1 , s t ) i , we set y i = 1 iff there exists a claim in C whose label (SUP or REF) changes as a result of the edit (i.e., SUP → {REF, NEI} or REF → {SUP, NEI}). Table 1 provides example factual and non-factual revisions. We evaluate the following baseline models:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Edit Distance. We measure the edit distance between s t−1 and s t , assuming that larger edits are more likely to represent substantive changes. We tune a decision threshold on the validation set.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We use an MLP on top of a bag-of-words representation. Each sentence is encoded as e * , the average fastText (Bojanowski et al., 2017) word embedding of its edited words (i.e., that were removed or modified in the revision). The MLP input is then taken as [e t−1 ; e t ; |e t − e t−1 |; e t · e t−1 ].
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
ALBERT. We train the ALBERT transformer (Lan et al., 2020) using either only the edited words (diff), or the full sentence pair (full).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Our basic setting is similar to the inference task of the FEVER dataset. 8 We predict the verdict for a claim given an observed evidence, f verdict :
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The FEVER dataset, however, contains independent claim-evidence pairs. In our setting, we have claims paired with revisions such that rel(c i , s t−1 ) = rel(c i , s t ), creating contrastive triplets. For example, the claim in Figure 2 states that the COVID-19 outbreak was identified before December. VITAMINC matches it with two different contexts (before and after the presented revision), that can either support or refute that claim.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Our baseline model is an ALBERT sentencepair classifier that predicts rel(c, s). Compared to BERT (Devlin et al., 2019), it uses fewer parameters by shrinking the embedding size and sharing layers, which we find to improve robustness.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Word-level rationales provide useful explanations for predictions of neural models (Lei et al., 2016). Such explanations can be particularly useful for semi-automated fact verification, since they allow users to quickly interpret and trust the model's verdict. 9 In Figure 2, for example, the date of the first identified case can explain the verdict for the claim.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
As first proposed by Lei et al. (2016), the standard definition of extractive rationales asks for selecting the minimal set of input tokens that is sufficient for preserving the model's prediction. Here we use a slightly modified definition following Shah et al. (2020), where we identify the minimal set of evidence tokens where removing them will change the input's label to NEI.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We pose this task as conditional masking, where we learn a function f rationale : C × S → {0, 1} n , where n is the length of an evidence s ∈ S. Given an evidence s = (x 1 , . . . , x n ) and a claim c, where rel(c, s) ∈ {SUP, REF}, we want to find a mask m such that rel(c, s m) = NEI, where
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Moreover, we want m to be as sparse as possible. Intuitively, s m could be viewed as an incomplete revision in which the masked words that have not yet been filled in will determine the relation with the claim. We say that m reveals the most responsible words in s for resolving c. Following Shah et al.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
(2020), we formulate an unsupervised objective as
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We evaluate the quality of m by comparing it in terms of F1 to both (1) m edit , the non-stopwords removed or replaced in the true revision (i.e., edit prediction), and (2) m manual , a manually annotated "human" reference, (i.e., rationale prediction). We implement the following two baselines: We keep the rel classifier (from §4.2) fixed, and train a separate ALBERT model to predict the mask m using a Gumbel softmax (Jang et al., 2017).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Distantly Supervised. By leveraging opposing claims present in VITAMINC, we are able to identify m edit = diff(s t−1 , s t )-i.e., the nonstopwords that are deleted or replaced in s t−1 when compared to s t . We then use m edit as distant supervision for m, where L ds = − γ n n i=1 log p(m i = m edit i ). We combine both the L us and L ds losses.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
As facts change, the sources reporting them must change as well to reflect the most recent information. In VITAMINC, this is reflected via the active revisions to Wikipedia. We simulate automating this process by considering two generation tasks:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Automatic Revisions. Given an outdated context s t−1 and an updated claim c, we learn f revise : S × C → S to produce a new context s t that minimally modifies s t−1 to agree with c. For example, one can change s t−1 in Figure 2 to state "before December" in order to agree with the claim.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Claim Extraction. Given a revision (s t−1 , s t ), we learn f extract : S × S → C to produce a short claim c that expresses the factual change.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
In both tasks, the output should satisfy rel(c, s t ) = SUP, while rel(c, s t−1 ) = REF. We use f verdict ( §4.2) to evaluate this requirement. We experiment with both BART-base (Lewis et al., 2020a) and T5-base (Raffel et al., 2020) sequenceto-sequence transformer-based generators. For the revision task, we concatenate s t−1 and c with a separator and train the model to predict s t . For the claim extraction task, we combine the input pair (s t−1 , s t ) into a single sentence that visualizes the revision (e.g., "sales of {4.7 → 5.4} million").
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We present and analyze results for the models described in Section 4. Our analysis attempts to evaluate several questions: (1) How well can the current state-of-the-art models perform on the VITAMINC tasks? (2) Does VITAMINC increases the robustness of models against adversarial examples? (3) Can VITAMINC improve interpretability by providing supervision for anchoring words?
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
In addition to VITAMINC, we train and evaluate on several related datasets, which we briefly describe:
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
FEVER (Thorne et al., 2018): A popular fact verification dataset based on Wikipedia. We use the provided SUP and REF claim-evidence pairs. For NEI claims, we randomly sample neutral evidence from the article with the highest BM25 score (Fisch et al., 2021).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
MNLI (Williams et al., 2018): A large and diverse dataset for natural language inference. The threeway sentence-pair entailment prediction is similar to fact verification. We use the hypothesis as the claim and the premise as the evidence and evaluate on the "mismatched" evaluation set.   Table 3 shows the results of our baseline models on the factual revision flagging task. First, we notice that a model trained on the PAWS dataset (reaching 93.42 F1 score on PAWS test) does not transfer well to the flagging task, and performs on par with a simple edit distance heuristic. We hypothesize that this is a result of the entity scrambling technique used to synthetically revise sentences in PAWS, which is different from the edits introduced by real, factual Wikipedia revisions in practice. Second, we see that the performance of neural models trained on the VITAMINC flagging task increases with richer inputs and more advanced models-demonstrating the complexity of the task. The ALBERT (diff) model that uses only the modified word sequences from each sentence (i.e., contextual within a subspan) improves the AUC by 10 points over a BOW model that gets a similar input. The ALBERT (full) model that receives the full sentences as input (i.e., has access to even more context), further improves the AUC by 2 points. Nevertheless, the best model still only reaches 83 macro-F1, indicating the difficulty of this task. ). Another evidence for the generalization properties conferred by VITAMINC is its zero-shot performance to both other datsets. An ALBERT-xlarge model trained only on VITAMINC reaches 76% and 79% accuracy on FEVER and MNLI, respectively. In contrast, the transfer accuracy for MNLI→FEVER is 70% and for FEVER→MNLI is only 38%.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Most importantly, models trained with VITA-MINC perform better on challenging adversarial datasets. On the otherhand, simply augmenting FEVER data with MNLI data has a limited effect on adversarial examples. 10 We conjecture that the contrastive nature of VITAMINC helps models better learn the relations between the claims and evidences-and to avoid relying on certain artifacts that do not generalize well.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
To further probe the value of VITAMINC examples compared to FEVER ones (SUP and REF   only), we compose training sets of 100K examples using different ratios of the two datasets. As shown in Figure 3, including more VITAMINC pairs continuously improves the performance on the challenging adversarial and symmetric evaluation sets.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
As an additional qualitative experiment, given the recent successes of huge language models such as GPT-3 (Brown et al., 2020), we explore whether such models develop sufficient context sensitivity on their own. Appendix C shows the results of classifying several claims using a few-shot GPT-3 model. We find that GPT-3 still largely underperforms our VITAMINC-trained models in terms of sensitivity-demonstrating the importance of using VITAMINC's unique structure during training. Table 5 shows the results of our baseline models for identifying word-level rationales (i.e., anchoring words in the evidence). While our unsupervised model is able to uncover some patterns, directly leveraging the structure of VITAMINC to obtain distant supervision for likely anchoring words (i.e., token labels) improves both the edit prediction and the word-level rationale prediction performance. 11 Example predictions are provided in Appendix E. 11 We evaluate rationales using a manually annotated test set of 300 examples (150 each from VitC real and VitC synthetic). Table 6 presents the results on factually consistent generation. We find BART to perform better in both of our generation tasks (though we only tried the default setting). The BLEU score (Papineni et al., 2002) is lower in the claim extraction task since there is freedom in how to phrase the claims, which can result in greater differences between the outputs and the references. The BERT-based BLEURT score (Sellam et al., 2020) shows a similar trend. Still, the claim extraction model succeeds in updating the facts that reflect the true revision 86% of the time, as measured by the fact verification model's verdict (f verdict ). The revision generator aims to modify sentences so that they agree with a given claim. According to our fact verification model's verdict, it succeeds in doing so 76% of the time. Furthermore, revisions should resemble real ones, and preserve the remaining content that is unrelated to the claim. The SARI KEEP F1 (Xu et al., 2016) of 75 shows that the model and the reference mostly agree on parts of the sentence that should be kept unchanged.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We find that the token-based measures and our f verdict metric agree well with human (manual) evaluation scores. We randomly sampled 100 generated and human-written sentences per task, and asked workers on Amazon MTurk to rate their grammaticality and whether the evidence s t supports the claim. The scores of the generated sentences were on par with the human-written ones, indicating the high-quality of our outputs. Table 7 presents two example generations for the claim extraction task (we provide additional qualitative examples in Appendix E). Our model is able to efficiently extract a self-contained claim that expresses the correct fact after the edit. As in §5.3,   Table 7: Example outputs for expressing claims that reflect the factual changes in a single Wikipedia revision. The BART-base model is trained on VITAMINC data, while GPT-3 is applied in a 2-shot setting with a temperature of 0 or 0.7 (see Appendix C). The revision (s t−1 , s t ) is given to the model as a single sentence, where the edits are between curly brackets. The human-written claim is provided for reference. The rightmost column contains the prediction of our ALBERT-xlarge f verdict (c, s t ) model (trained on VITAMINC) when using the generated claim.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
we also explore how GPT-3 handles this task (we provide two demonstrations in the prompt). Compared to the BART model trained on VITAMINC, GPT-3 appears to make more factually inconsistent or unsupported generations (see Appendix C for more details). Encouragingly, our f verdict classifier is still able to pick up on this-as demonstrated by the predictions in the rightmost column of Table 7. For example, classifying the report about 20 deaths as NEI since it is not part of the source. Once again, this serves to qualitatively demonstrate the effectiveness of leveraging VITAMINC.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We presented VITAMINC, a large-scale dataset for training and evaluating fact verification models using contrastive contexts. Our novel method of leveraging factual revisions to Wikipedia enabled us to create challenging examples in which a claim is paired with contexts that are lexically similar, yet factually opposing. Our results illustrated that training on VITAMINC improves classifier sensitivity to subtle changes in evidence, and increases their robustness to adversarial examples.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Furthermore, we formulated several new, important tasks for fact verification that VITAMINC allows us to test. We showed how the dataset's unique "before and after" structure lends itself to training classifiers to flag factual revisions. In addition, for factual revisions, the edits reveal which words in the evidence are the most critical-which helps supervise word-level rationale models for better interpretability. Finally, we demonstrated that VITAMINC can help with factually consistent text generation. We hope that this work and the range of tasks it presents will motivate and support the fact verification field in developing reliable models that can adapt to dynamically changing evidence. 
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We provide additional details about the VITAMINC dataset.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Topic Distribution. Figure A.1 shows the distribution of claims in the VITAMINC dataset by the topic of the Wikipedia article they are based on. The information was collected from DBpedia, 12 retrieving the parent class of the pages. Labels for about 25% of the articles were missing, and left blank in the diagram. The "synthetic" part of VITAMINC, which is based on the claims of the FEVER dataset, contains many claims about specific human entities. About 15% of the claims in VITAMINC real are about COVID-19.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Category Distribution. We sample 100 examples from the "real" and "synthetic" subsets of VITA-MINC and manually categorize their claims. The results are presented in Table A.1. Due to the creation methodology of VITAMINC real, its claims mostly describe frequently updating facts, or facts that tend to be corrected. We find about half of these claims to describe changes in numerical values (e.g., number of COVID-19 cases, earnings or ratings of movies, number of awards etc.). In contrast, VITAMINC synthetic mostly covers general facts about specific entities, (e.g., place of birth, date of birth, occupation, etc.). This is a result of the synthetic claims being based on the FEVER dataset, where annotators were asked to come up with claims on popular Wikipedia pages. Combined, the VITAMINC dataset holds a diverse set of claims about various topics.  
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We ask three additional annotators to independently annotate a random set of two thousand claimevidence pairs, evenly distributed between the development and test splits of the real and synthetic sets. The Fleiss κ score (Fleiss, 1971) between the four annotations is 0.7065, which means substantial agreement. Similar agreement scores of 0.6841 and 0.7 were reported for fact verification (Thorne et al., 2018) andNLI datasets (Bowman et al., 2015), respectively.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Annotation artifacts are common in crowd-sourced sentence-pair inference datasets such as fact verification and NLI. Models can leverage these idiosyncrasies to achieve unexpectedly high performance when given only one sentence of the pair. For example, Schuster et al. (2019)    All claims in the VITAMINC-synthetic are paired with one refuting and one supporting evidence, making it impossible for a claim-only to perform better than random. Each claim in the VITAM-INC-real is paired with one refuting or neutral evidence, in addition to a supporting one. To evaluate whether models can utilize lexical cues in claims, we train a claim-only classifier on VITAMINC-real and find it to achieve 50% accuracy-the same as always predicting SUP.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Naturally, when pairing claims to evidence sentences, the overlapping words will be higher on average for claims with their supporting evidence. In VITAMINC dataset, we want to minimize this bias in order to create challenging examples that require sentence-pair inference and cannot be solved by simple word matching techniques. Therefore, we asked annotators, when possible, to avoid copying exact phrases from the evidence to the claim (see §3.2). Figure A.2 shows the probability density function of bigram overlaps between the claim and evidence for each relation. Similar to FEVER, the overlap ratio of supporting pairs in the VITAMINC dataset is only slightly higher than the one of refuting pairs. Also, the overlap ratio of the NEI pairs of the VITAMINC real dataset is on average higher than FEVER.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We implement all our models with the Hugging-Face Transformers library (Wolf et al., 2019). When comparing across training datasets of different sizes, we train the model for the same amount of update steps, upsampling the smaller datasets. We pick the checkpoint with the highest accuracy on the development set of the training task and report performance on the test set. More details are available at https://github.com/ TalSchuster/VitaminC
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The GPT-3 model has recently demonstrated impressive results in zero-shot and few-shot generation and classification tasks (Brown et al., 2020). This 175B parameters language model was trained on billions of words from online sources, including the English Wikipedia. As result, it can be applied on many tasks without any further fine-tuninginstead, one need only provide a task-specific prefix (i.e., "prompt") with a few examples that direct the language model towards the desired output format. For example, GPT-3 achieves better than random results on ANLI with only a single example in the prompt, and over 40% accuracy with 50 examples (Brown et al., 2020).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We used OpenAI's beta API to query GPT-3. Due to our limited quota, we could not perform extensive experiments. Instead, we performed a qualitative evaluation using several examples from VITAMINC test set for the claim extraction (factually consistent generation) and the fact verification tasks. Therefore, these results should be viewed as exploratory only.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
GPT-3 for Claim Extraction. We examine a twoshot setting for the claim extraction task. The model is asked to convert a revision into a short claim that expresses the fact that is true after the edit. To guide the model for this task, we provide a prompt with two random examples from the VI-TAMINC training set (see Figure C.1). One of the main concerns regarding large language models is the limited control it allows for ensuring that the facts in the generated output align with the source (Schuster et al., 2020). The generation tasks of VITAMINC provide a useful test-bed for evaluating the factual consistency with the input. Importantly, our VITAMINC-trained fact verification classifiers (f verdict ) allow strong automatic evaluation for the factual agreement of the generation with the source.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We use GPT-3 to extract claims for four revisions with a sampling temperature value (T ) set to either 0 or 0.7. The zero value is recommended for maximizing the factual consistency as the model follows its most certain predictions. Using low temperature, however, can result in less fluent generations (Holtzman et al., 2020). Therefore, high values of T are also commonly used.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
The results are reported in Tables 7 and E.3. With only two guiding examples, GPT-3 is able to follow the desired format and create a short claim. Yet, some of its generations follow s t−1 instead of s t or add new, unsupported facts. f verdict provides an indication for the factual correctness of the output. For example, it correctly classifies the output of the T = 0.7 setting for the top example in Table 7 as "Not Enough Information" since GPT-3 reported about 20 deaths even though the input doesn't mention death numbers at all.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We expect GPT-3 to improve with longer prompts or fine-tuning and leave this to future research due to our limited quota.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
GPT-3 for Fact Verification. We also experiment with using GPT-3 few-shot classification capabilities for the fact verification task. We follow the ANLI few-shot format of Brown et al. (2020) Table C.1 summarizes the results. Even with only six examples, GPT-3 seems to perform significantly better than random. Yet, its verdict is wrong in several cases that can be easily classified by humans. For example, we find it to refrain from predicting a True/False verdict even when the evidence is clear. We observe this both for a date-based (line 3.2 in Table C.1), numerical (lines 4.1-4.2), and entity-focused claims (line 5.2).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
To experiment with the sensitivity of the model to the provided context, we manually modified some of the examples to provide even stronger evidence. For example, while GPT-3's prediction for line 5.2 is acceptable as actually, Turner Broadcasting System merged with WarnerMedia in 1996, changing the evidence to another disconnected entity (The Walt Disney Company) did not change the prediction (line 5.3) as expected. Even when explicitly stating that there is no other owner GPT-3 didn't modify its verdict (line 5.4). Similarly, when evaluating the claim about the population of Beaverton being less than 90K, GPT-3 ignores the supporting evidence and outputs a false verdict (lines 1.4-1.5). Changing the claim to state "approximately 86K" instead of "less than 90,000" modified the prediction to "Neither" (line 1.6). Only repeating the exact same number as the evidence led to a true verdict (line 1.7).
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We report fact verification results with a fine-tuned BERT-base (Devlin et al., 2019) model in Table D.1. We find ALBERT-base to outperform BERT-base on most of the evaluated datasets. ALBERT-xlarge performed better than the two base models in all datasets except for Triggers. The Triggers dataset is very small (186 examples) and contains some unnaturally looking claims, which could explain the high variance across models.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
We provide examples of predicted word-level rationales in Table E  Manchester is a major city and metropolitan borough in Greater Manchester, England, with a population of 545,500 as of 2017 (5th most populous English district). Question: Manchester had a population of more than 540,000 in 2017 and was the 5th most populous English district. True, False, or Neither? True ### As of March 2018, the apps have achieved more than 8 billion downloads. Question: Talking     In several cases, however, GPT-3 abstains from a True/False verdict, even when provided with strong evidence (see "Neither" predictions). Line 1.4 shows an example where GPT-3's verdict is opposite of the provided evidence. Only when rephrasing the claim to exactly overlap with the evidence, it predicts an agreement.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Manchester is a major city and metropolitan borough in Greater Manchester, England, with a population of 545,500 as of 2017 (5th most populous English district). Question: Manchester had a population of more than 540,000 in 2017 and was the 5th most populous English district. True, False, or Neither? True ### As of March 2018, the apps have achieved more than 8 billion downloads. Question: Talking Tom and Friends apps have less than 8 billion downloads. True, False, or Neither? False ### As of January 2015, JFC had a total of more than 3,000 stores worldwide, with system-wide retail sales totaling 82.1 billion pesos for the fiscal year 2011. Question: Jollibee had a total of more than 20,000 stores worldwide after January 2016.     Table 4 with a BERT-base model.
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Claim the youtube channel chuchu tv is placed 42nd and has more than 25 million subscribers . Evidence chuchu tv is the 43rd most subscribed youtube channel in the world , with over 20 million subscribers .
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Claim the ramus has sold less than 4.5 million albums worldwide . Evidence the rasmus has sold 5 million albums worldwide , 310,000 copies in their native finland alone .
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
darren randolph is spanish . Evidence humes dated irish footballer darren randolph in 2005 .
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
astravyets is near vilnius . Evidence his father may have migrated to the us in the 1860s from astravyets near vilnius .
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
the pace controlling stamina meter is a new feature in the game series . Evidence new to the series is a three-tier stamina meter which controls the pace of a match .
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Claim the movie will be released on 25 november 2015 . Evidence [...] are producing the film which columbia pictures will release on november 25 , 2015 . Table E.1: Example masks produced by the word-level rationale model for identifying anchoring words in the evidence that are responsible for the classifiers verdict regarding the claim. Masking these words leads the classifier to predict NEI instead of what would have been SUP or REF.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
While modern summarization models generate highly fluent summaries that appear realistic Zhang et al., 2020), these models are prone to generating non-factual and sometimes entirely fabricated content (Cao et al., 2018;Goodrich et al., 2019;Maynez et al., 2020). With the increasing adoption of language generation tools in user-facing products, such unreliability poses severe risks, including the spread of misinformation, panic and other potentially harmful effects (Ranade et al., 2021;Hutson et al., 2021).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Since it is difficult to control for factuality at training or inference time (Huang et al., 2021; The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Scientists believe a vaccine for Covid-19 might not be ready this year. The first vaccine for Ebola took 5 years to be approved by the FDA.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Scientists believe a vaccine for Ebola might not be ready this year. The first vaccine for Ebola took 5 years to be produced by the CBP. Figure 1: Model generated summaries often produce content which is factually inconsistent w.r.t. to the source. FACTEDIT rewrites these summaries by maintaining the abstractiveness but correcting factual errors. Dreyer et al., 2021), a popular approach to fix the factual inconsistencies is via post-editing generated summaries (Cao et al., 2020;. This allows summarization models to focus on fluency and content-relevance while improving factual consistency. However, there is no suitable data for training post-editing models to directly "translate" an incorrect summary to a correct one. Prior work constructed synthetic training data by introducing simple heuristic errors like replacing entities or numbers in reference summaries (Cao et al., 2020), but it is not clear whether such synthetic errors have sufficient coverage and accurately represent the types and distribution of actual errors made by language models. Further, with increasing language generation capabilities, models make more complex factual errors involving discourse structures and paraphrasing which cannot be easily captured with heuristics (Pagnoni et al., 2021). The goal of our work is to develop post-editing models  Figure 2: Architecture framework for FACTEDIT. Using masked versions of existing reference summaries, we use an infilling language model to produce alternative candidates for the mask position. We construct factually incorrect summaries by replacing the mask with the lower ranked candidates. Finally, we train a sequence-to-sequence model for fact correction using the synthetically constructed data.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
that generalize over a wider range of factual errors (example in Figure 1) in generated summaries from diverse summarization model types.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We propose FACTEDIT-a novel approach to post-editing text, to control for content factuality in generated summaries. Rather than manually defining a list of heuristic errors, it incorporates a new algorithm to generate adversarial (non-factual) examples using infilling language models (Donahue et al., 2020). We use lower ranked beam-search candidates from the language model as a source for potentially factually-incorrect summary facts, thereby producing a set of plausible, likely, and fluent, incorrect synthetic summaries for a particular correct reference summary. In this way, we leverage the capabilities of large language models to produce multiple candidates of alternative, erroneous summaries. These examples, along with factually correct references, are then used to train a sequence-to-sequence fact-correction model that aims at generating a factually consistent version of the candidate summary ( §2).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We evaluate FACTEDIT on two datasets -CNN/DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018) and across nine summarization models with the FRANK benchmark (Pagnoni et al., 2021) for evaluating various categories of factual errors in generated summaries ( §3). The two summarization datasets represent varied distributions of factual errors in models trained on them and hence constitute a good test bed to evaluate the generalizability of our model. We show that FACTEDIT substantially improves factuality scores across two metrics -Ent-DAE (Goyal and Durrett, 2021) and FactCC (Kryscinski et al., 2020). On the Ent-DAE metric, FACTEDIT improves results by ∼11 points (CNN/DM) and ∼31 points (XSum), and on the FactCC metric we show improvements of ∼6 points (CNN/DM) and ∼24 (XSum) points on average across models ( §4). Further, our analysis shows that FACTEDIT effectively corrects diverse error categories without the need for special heuristics or annotations ( §5). An important application of FACTEDIT is to audit summarization systems and facilitate their reliability.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Assume a summarization model trained to process a document d and generate a coherent and fluent summary 2 g ′ which has been shown to often misrepresent facts from the document. FACTEDIT is a fact correction model M C which takes the generated summary g ′ and document d, identifies factual errors and generates a rewritten summary g by correcting them (as outlined in Figure 2).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We present an adversarial data generation approach which leverages the power of pre-trained language models to produce fluent and complex factually incorrect summaries. We train an infilling language model M I using documents from summarization training data and use the model to introduce diverse factual errors in sentences from them ( §2.1). Using the trained model, we introduce factual errors in reference summaries of the training data r producing an incorrect summary r ′ resulting in a synthetic dataset {r ′ , r, d} train of erroneous summaries mapped to their corrected versions (pink section in Figure 2). We train a sequence-to-sequence model M C for factual error correction using the generated synthetic data ( §2.2). Finally, we use the trained correction model to rewrite model generated summaries g ′ producing a corrected version g ( §2.3 -green section in Figure 2).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our data generation process leverages infilling language models (Donahue et al., 2020) to produce candidates to fill masked phrases in a summary sentence. We mask parts of the input and use the infilling model to generate multiple candidates for the masked position. We then use lower order beam candidates as potential incorrect candidates to generate an incorrect version of the input. We hypothesize that, given the relevant context of a source document, a strong language model generates relevant and factual sequences at higher probabilities, compared to lower probability sequences. For the infilling model, we hypothesize that the lower ranked candidates are often alternative phrases of similar types (in case of entities) or parts-of-speech which are plausible but often not factually correct. Motivated by prior work (Goyal and Durrett, 2020) using lower ranked beam search candidates as a source for adversarial data, we use the lower ranked candidates to construct erroneous summaries from reference summaries.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Training: Our infilling model M I is trained to take a masked sentence s masked and its relevant context ctx as input and generate a correct phrase to fill in the masked span. To train M I , we construct a dataset using documents d from the training data of existing summarization datasets. For each sentence s in the first-k (k=5) positional sentences of a document d, we identify the subjects, objects and relations {sub, obj, rel} in them using OpenIE (Banko et al., 2007). By iteratively masking each phrase p in {sub,obj,rel}, we create a masked query s masked and its corresponding context ctx by removing the masked sentence from the document, resulting in our training data {s masked , p, ctx}, where p is the masked span text. We train a sequence-to-sequence model M I on this data which takes s masked [SEP] ctx as input and learns to generate p as the output. We intentionally use only sentences from the document as masked queries and do not use sentences from the reference summaries, to ensure that the model does not memorize phrases from the references. Thus, when applied to unseen reference sentences during inference, the model will produces richer beam search candidates.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Adversarial Data Generation: We use the trained infilling model to generate the synthetic dataset for fact correction using the document reference pairs {d, r} train from the summarization training data. For each sentence in the reference s r , we use OpenIE to extract {sub, obj, rel} and iteratively mask one phrase at a time to construct masked sentences s masked from the references. We provide this masked reference summary sentence and document d as input to the model and perform beam-search decoding for generation. We then consider lower ranked beam candidates (rank= [5,15]) 3 as non-factual alternatives for the corresponding masked phrase. We then use these candidates as the replacements for the mask producing an erroneous summary r ′ . Running this on the {d, r} train training data, we construct a synthetic data {r ′ , r, d} train of factually incorrect summaries paired with their correct version where r ′ and r differ by an incorrect phrase. To train the model to not perform any corrections on factual summaries, we keep original reference summaries for 20% of the data points (r ′ = r).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Using the parallel data {r ′ , r, d} train produced by the above infilling method, we train models for factual error correction. In contrast to prior work which used pointer based models to copy phrases from the source document, we use a sequence-tosequence model like BART  to preserve the abstractive content in the input. The model M C is trained with an erroneous reference summary sentence s r ′ produced by the infilling data generator and the corresponding document d as input and the correct reference summary sentence s r as output. A straightforward option is to provide s r ′ , d concatenated as inputs to the model. But we hypothesize that providing the right context can help the model better correct the errors. Below we outline input structures that provide better context in the input: Relevant Supporting Passages: To help the model better connect the relevant facts in the source document to the summary sentence being corrected, we experiment with providing only the most relevant parts of the document as input context instead of the entire document. Using a scoring function (ROUGE), we identify sentences from the document which have high overlap with the generated summary sentence and extract the top-k (k=3 for our work) such sentences. We provide these sentences along with a window of w k (w k =2) sentences before and after each as the input context to the model. Surrounding Summary Context: While simple errors like incorrect entities can be detected and corrected with only the context of the current sentence being corrected, more complex discourse level errors like incorrect pronouns require the context of the rest of the sentences of the summary. To enable this, we additionally give the complete generated summary (other sentences from the summary) as additional context. For single sentence summaries like headline generation, this does not change the original setting, but for longer summaries this setting helps with discourse level errors. In essence, our model M C takes the input as Incorrect Reference Sentence (s r ′ ) [SEP] Full Reference Summary (r ′ ) [SEP] Relevant Passages and generates the corrected summary r as output.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our trained fact correction model M C can be directly applied to any model-generated summaries g ′ , without access to the underlying model. For each sentence in a generated summary, we identify the relevant passages using ROUGE and provide it as an input to the model (in the form Gen-
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We use two news summarization datasets CNN-DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018). The two datasets have been extensively studied for the factual consistency in their generated summaries across a variety of models (Goodrich et al., 2019;Cao et al., 2018). Reference summaries from CNN/DM are longer, having on average three sentences, and more extractive in nature. XSum on the other hand has shorter, single sentence summaries and is significantly more abstractive in nature. The summaries in these datasets are qualitatively different, and hence models trained on the two datasets present varied levels of challenges in maintaining factual consistencies.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Prior work have studied summaries generated from different language models and characterized the factual errors in them (Pagnoni et al., 2021). Generated summaries on the CNN/DM dataset are more extractive in nature and hence are more factual (∼70% of summaries are factual) than the more abstractive generated summaries of XSum (∼20% of summaries are factual). The longer summaries in CNN/DM display discourse level inconsistencies while summaries from XSum often hallucinate content which is not supported by the source document. Hence, the two datasets present a varied setting for evaluating the efficacy of our model across different kinds of errors. For our main evaluation, we evaluate the overall capability of FACTE-DIT in correcting errors in summaries generated by a BART model.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We further evaluate our model on the FRANK benchmark (Pagnoni et al., 2021) which contains generated summaries obtained using multiple language models for both datasets annotated with human judgements on their factuality and the category of factual error. As different language models have different distribution of factual error types, this evaluation helps us study the generalizability of FACTEDIT in correcting errors across them. 4 For the CNN/DM dataset, it contains model outputs from a LSTM Seq-to-Seq model (S2S) (Rush et al., 2015), a Pointer-Generator Network (PGN) model (See et al., 2017), a Bottom-Up Summarization (BUS) model (Gehrmann et al., 2018), a Bert based Extractive-Abstractive model (BertSum) (Liu and Lapata, 2019) and a jointly pretrained transformer based encoder-decoder model BART . For the XSum dataset, it contains model outputs from a Topic-Aware CNN Model (Narayan et al., 2018), a Pointer-Generator Network (PGN) model, a randomly initialized (TransS2S) (Vaswani et al., 2017) and one initialized with Bert-Base (BertS2S) (Devlin et al., 2019).
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Setup: We use OpenIE (Banko et al., 2007) to pre-process each summary and extract subject, object, predicate triples for each summary sentence. We use BART-base  as our sequence-to-sequence model for the infilling based data generator and the fact correction model. Both models were trained with a batch size of 48, a learning rate of 3e-5, and warm-up of 1000 for 1 epoch. The maximum input sequence length was 512 and maximum output sequence length was 128. Using the infilling data generator, we generate 1233329 negative, 308332 positive examples for CNN/DM and 724304 negative, 181076 positive, examples for XSum as training data for fact correction. Models were trained on 4 Nvidia GeForce GTX TITAN X GPUs and each training run took ∼15 hours. All hyperparameters were chosen based on generated dev set ROUGE-L (Lin, 2004) on each dataset. Evaluation Setup: Evaluating factual consistency of generated summaries is challenging, with relatively recent metrics developed to detect it. These metrics unfortunately do not correlate highly with human judgements yet. We therefore evaluate our model using two metrics -FactCC (Kryscinski et al., 2020) and Ent-DAE (Goyal and Durrett, 2021); each captures different error types. FactCC is a binary classifier, trained on a synthetic, heuristic error dataset, which is better at detecting simple semantic errors like incorrect entities or numbers. Ent-DAE is a classifier trained on synthetic data constructed using the dependency structure of the text. In addition to semantic errors, it is better at detecting more complex discourse-level errors (Pagnoni et al., 2021). We also report ROUGE (Lin, 2004) to evaluate if our model maintains the fluency of summaries. While ROUGE is less correlated with factuality (Pagnoni et al., 2021;Maynez et al., 2020), it helps evaluate if the corrected summary is fluent and aligned with the reference summary. However, with factual corrections of outputs we expect small drops in ROUGE, since generation models were specifically optimized to maximize ROUGE presumably at the expense of factuality.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our evaluation has two settings: i) FACTEDIT -correct all generated summaries in the test set and ii) FACTEDIT + FactCC Filter (FF) -using the FactCC metric we identify factually incorrect summaries, and only correct the incorrect ones. Baselines: We compare our approach with (Cao et al., 2020) as the baseline. The baseline uses a heuristic set of rules proposed by Kryscinski et al. (2020) to introduce simple errors (Entity, Number, Date, and Pronoun) in reference summaries and trains a BART-base model for error correction. Comparing our model with (Cao et al., 2020) helps us evaluate the benefit of our Infilling LM based adversarial data generator. 5
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We first evaluate FACTEDIT's ability to correct errors in summaries generated by a BART-base summarization model on the entire test set. We first generate summaries using a BART-base model finetuned on each dataset and then provide the generated summaries and their corresponding source documents as inputs to FACTEDIT for correction. Table 1 shows results for this experiment. Our results show that correcting factual errors using our model improves the factuality results. The baseline model performs poorly with the FactCC metric showing lower scores than the BART model generated summaries, especially in the more abstractive XSum setting. The DAE metric for the baseline model is slightly lower than the BART model scores in the CNN/DM setting and has no improvement in the XSum setting showing that it does not perform corrections on complex errors. These results confirm our hypothesis that the baseline model trained on adversarial data based on heuristic errors does not transfer well to real errors in model generated summaries. In contrast, our model improves both metrics across both datasets. On the more challenging XSum dataset, our model has a ∼17 point improvement on FactCC and ∼0.  improvement on DAE over the baseline model. The BART generated summaries on CNN/DM are ∼70% factual and hence using the FactCC Filter to correct only non-factual summaries helps improve results on FactCC. As XSum has more than 80% non-factual summaries, the FactCC filter does not change results and correcting all generated summaries is beneficial. In Table 6 we present examples of corrections made by FACTEDIT and present a discussion in §A.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Prior works have shown that improving factual consistency in summaries leads to a drop in ROUGE scores (Maynez et al., 2020;Cao and Wang, 2021;Cao et al., 2020). Our ROUGE results do not drop significantly and are consistent with prior work. These results show that our model does not significantly change the summaries and the corrected summaries contain the relevant information w.r.t. to the source. Table 2 shows results of using FACTEDIT to correct summaries generated by different types of language models using the FRANK benchmark (Pagnoni et al., 2021). We provide the generated summaries collected in the benchmark along with their source document as input to our trained fact corrector. This setting evaluates the generalizability of our adversarial training data in handling different error distributions from different summarization models.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our results show our model significantly improves the factuality in generated summaries across 8 out of 9 test models. The FactCC Filter helps improves results in CNN/DM setting but does not change results in XSum similar to results in §4.1. In the more extractive CNN/DM setting, fact correction improves FactCC scores by ∼5.3 points and DAE scores by ∼10.9 points on average across models. In the more challenging and abstractive XSum dataset, we improve FactCC scores by ∼24 points and DAE scores by ∼31 points on average. Our results show that our model trained using Infilling LM based adversarial data is able to generalize and correct errors in generated summaries across different model types. Further, the significant improvement in XSum suggests that using LMs to generate factually incorrect candidates produces rich negative examples which help correct errors in Figure 3: Performance of FactEdit across different error categories in comparison to baseline (Cao et al., 2020) . FACTEDIT improves the percentage of factual summaries across diverse types of factual errors. more abstractive summaries.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Pretrained models like BART, BertSum and BertS2S have improved generation capabilities and make lesser mistakes in generating the right entity or predicate and more mistakes in discourse structuring (Pagnoni et al., 2021). FACTEDIT correspondingly shows larger improvements in DAE scores than FactCC scores in these pretrained models. The Pointer-Generator model being highly extractive in nature scores highly in factuality metrics in the CNN/DM setting and FACTEDIT reduces results in this setting showing that our model is not beneficial in copy-based model settings. On the other hand, in the XSum setting, the base Pointer-Generator model scores poorly and correcting factual errors in them improves factuality scores. Non-pretrained sequence-to-sequence models like Seq2Seq and TransSeq2Seq score poorly in both ROUGE and Factuality scores due to their limited language generation capabilities. By correcting factual errors in them, we improve factuality metrics significantly without changes in ROUGE, indicating that the gains are due to fact correction and not just rewriting the summary using a strong language model.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
The FRANK benchmark proposes a typology of three coarse categories of error types and collects human annotations on the error category: i) Semantic Frame Errors -This category covers factual errors in a sentence due to incorrect entity or predicate being generated ii) Discourse Errors -This covers discourse level factual errors like incorrect pronouns or sentence ordering iii) Content Verifiability Errors -This category is for errors whose factuality cannot be judged either due to grammatical errors or hallucinated content. We evaluate our model on its ability to correct different types of errors. We use the generated summaries from the best pretrained model in FRANK for each dataset -BART for CNN/DM and BertS2S for XSum. For each subset of summaries of a particular error type, we correct the summaries using FACTEDIT and report the percentage of factual summaries in the output as predicted by Ent-DAE. We compare FACTE-DIT with the baseline to study whether our model improves error correction for each type.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
From Figure 3, we see that across both datasets FACTEDIT increases the percentage of factual summaries across all three error categories, showing that the data generation process in FACTEDIT can generalize across multiple error types without the need for special heuristics or annotations. We see the largest improvements in the Semantic Frame Error category with an increase of ∼ 8 points on CNN/DM and ∼ 13 points on XSum. On the more complex Discourse Errors we see an improvement of ∼ 5 points on both datasets. Finally, on Content Verifiability Errors, we see a ∼ 8 point improvement on CNN/DM and ∼ 2 point improvement on XSum. XSum has a high proportion of hallucination errors and our results highlight the challenge in correcting this error type.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
It is not always feasible to train specialized fact correction models for each dataset or style of summaries. While CNN/DM and XSum contain documents of the news domain, they both have different summary characteristics. Certain applications might benefit from a single model which can generalize to different summary styles. We evaluate   the ability of FACTEDIT trained on CNN/DM data (FACTEDIT FF -CNN Model) to transfer and correct summaries generated for XSum documents using FRANK benchmark. Table 3 shows results for this experiment. Our results show significant improvement in factuality scores across all model types in this setting, showing that our data generation process produces rich and diverse factually incorrect examples which can generalize to factual errors in other data settings. By using only the source documents, our training data is agnostic of the styles, lengths and characteristics of reference summaries and hence is able to generalize to the headline style abstractive summaries of XSum.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
To further study whether the factuality corrections performed by our model align with human expectations of automated summaries, we conduct a human study. Two annotators evaluated 20 randomly sampled summaries generated from the test set of the XSum dataset using the BertS2S model and corrected by FACTEDIT and the baseline. The annotators were shown the entire source document and one corrected summary at a time and asked to rate the fluency and factuality of the summary on a 1-5 Likert scale. In manual evaluation, annotators rated FACTEDIT an average of 3.3 on factuality and 4.8 on fluency, compared to the baseline which was rated 3.1 and 4.6 scores respectively, showing that FACTEDIT improves on both factuality and fluency.  
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our model corrects each sentence in a summary given context of the rest of the summary and relevant passages in the source document. We ablate this setup by removing parts of the context one at a time. In Table 5 we present the results. We observe a a drop in results when using the entire summary as context (-RelevPass) and when removing the context of the summary in which the sentence occurs (-SummCtxt). Our results show the importance of having the appropriate context to enable the model to perform fact correction well.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
We present an adversarial data generation process to generate rich synthetic data for a post editing model, which can be applied to correct factual errors generated summaries. Our data generation process leverages Infilling Language Models to produce alternative candidate summaries. Using the generated data, we train models to rewrite summaries by correcting factual errors in them.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Through extensive experiments across two datasets and nine models, we show that our fact corrector model improves the factual consistency of the summaries, making them more reliable.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Our model is trained to rewrite generated summaries by correcting factual errors in them. A limitation in our current setup is accurate detection of factual errors. We rely on off-the-shelf metrics for identifying summaries with factual errors to correct. Our model does not perform detection and correction together and often rewrites correct summaries as well if fed to the model. Therefore for settings like CNN/DM, it's beneficial to filter summaries using a factuality metric before giving summaries to our model as input. As our fact corrector is a sequence-to-sequence model, it could potentially introduce new factual errors in the summaries. It is essential to use factually detectors to ensure summaries are factual before real world usage of any corrected summary.
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
State-of-the-art language generation models, including summarization, are not yet powerful enough to facilitate fine-grained control over generated content. This leads to problems with content fidelity and safety; our work aims to ameliorate issues related to factual reliability of the models. However, existing approaches, including ours, cannot guarantee this yet. Furthermore, there is a risk of dual use, since the same techniques can be used to post-edit models to produce non-factual, harmful content to mislead, impersonate, or manipulate opinions. Future research should focus on developing better defenses methods against mis-using language generators maliciously.
Factual Error Correction for Abstractive Summarization Models
Self-supervised methods have achieved success in a wide range of NLP tasks, and automatic summarization is no exception (Liu and Lapata, 2019;Lewis et al., 2019;Zhang et al., 2019a;Shi et al., 2019;Fabbri et al., 2019). These state-of-theart abstractive summarization models typically finetune pre-trained transformer-based models on a summarization dataset (Vaswani et al., 2017). Despite significant improvements over previous methods in terms of automatic evaluation scores such as ROUGE (Lin, 2004), ensuring factual consistency of the generated summary with respect to the source remains challenging. For example, 1 Our data and code is available at https://github.com/mcao610/Factual-Error-Correction Source: Jerusalem (CNN)The flame of remembrance burns in Jerusalem, and a song of memory haunts Valerie Braham as it never has before. This year, Israel's Memorial Day commemoration is for bereaved family members such as Braham. "Now I truly understand everyone who has lost a loved one," Braham said. (...) Original: France's memorial day commemoration is for bereaved family members as braham. (inconsistent) After Correction: Israel's memorial day commemoration is for bereaved family members as braham. (consistent) Table 1: An example of an inconsistent systemgenerated summary and the output summary from our correction model. In this case, "France" is successfully corrected as "Israel". Cao et al. (2018) claims that about 30% of summaries generated by abstractive models contain factual errors, which greatly limits their practicality.
Factual Error Correction for Abstractive Summarization Models
Different approaches have been proposed to detect or ensure the factual consistency of generated summaries, including using fact extraction or applying attention on fact triples (Cao et al., 2018;Zhang et al., 2019b;Goodrich et al., 2019), applying natural language inference or question answering models for consistency checking (Falke et al., 2019;Li et al., 2018;Wang et al., 2020) and training the model on artificial datasets (Kryściński et al., 2019). Most of these approaches either require a high-quality fact extraction model or they only focus on factual consistency evaluation. Improving factuality correction by editing inconsistent parts in generated summaries is a direction that has not been explored much.
Factual Error Correction for Abstractive Summarization Models
In this work, we propose a model to improve the factual consistency of system summaries with post-editing correction (Table 1). Our model takes a draft summary that is generated by an abstractive summarization model and produces a corrected final summary, conditioned on the source document. In addition, our trained corrector can be used as an evaluation model for factual consistency of abstractive summaries, with the assumption that a generated summary is inconsistent if our corrector decides to make edits. To teach the model to correct errors, we train it with artificial data that has factual errors introduced using heuristics proposed by Kryściński et al. (2019).
Factual Error Correction for Abstractive Summarization Models
The empirical results based on automatic and human evaluations indicate that our model not only corrects factual errors in summaries, it is also a better and more reliable factuality evaluation model than FactCC (Kryściński et al., 2019), a recent proposed factuality evaluation method. In a downstream setting where we apply the corrector to the output of an abstractive summarizer, we find that our corrector is able to accurately correct errors in the generated summaries. However, the overall recall on correcting factual errors in real system summaries remains low, suggesting the errors introduced by heuristics have a different distribution than errors made by abstractive summarization systems.
Factual Error Correction for Abstractive Summarization Models
Previous work on factual consistency in abstractive summarization can be divided into two categories: abstractive summarization models tailored towards factual consistency (Cao et al., 2018;Zhang et al., 2019b;Li et al., 2018), and evaluation models for factual consistency in abstractive summarization (Goodrich et al., 2019;Falke et al., 2019;Kryściński et al., 2019;Wang et al., 2020). Cao et al. (2018) proposed a dual attention module in an abstractive summarizer that attends to both the source document and to relation triples extracted from the document. Zhang et al. (2019b) propose to improve their abstractive summarization model by optimizing fact scores defined in radiology reports with reinforcement learning methods. Li et al. (2018) jointly train their model's encoder on summarization and NLI tasks. Guo et al. (2018) train an abstractive summarization system with the auxiliary tasks of question and entailment generation and show that their generated summaries are less likely to produce extraneous facts. Kumar and Cheung (2019) show that neural abstractive summarizers often assign higher posterior likelihood to perturbed contrastive summaries that are inconsistent with the source text than to human-written gold-standard ones. Concurrently to our work, Zhu et al. (2020) recently proposed a fact-aware summarization model that uses a knowledge graph. They use a pre-trained corrector module to modify generated summaries. Concurrent to our work, Dong et al. (2020) proposes factual correction models that leverages knowledge learned from question answering models via span selection. Their models employ single or multimasking strategies to either iteratively or autoregressively replace entities.
Factual Error Correction for Abstractive Summarization Models
In terms of evaluating abstractive summarization models for factual consistency, Goodrich et al. (2019) proposed a metric to check factual consistency by checking the overlapped fact triples between a source document and generated text on Wikidata. Falke et al. (2019) shows that factual error detection is a difficult task on its own and adapting entailment models for factual error detection do not offer the desired performance. Kryściński et al. (2019) finetune a BERT model on heuristically-created data with six types of rule-based text transformations for factual consistency checking. Wang et al. (2020) propose a framework for measuring inconsistencies in abstractive summarization by answering questions based on both generated summaries and documents.
Factual Error Correction for Abstractive Summarization Models
In this section, we describe our procedure of introducing artificial errors in the datasets for training and propose our end-to-end error corrector model.
Factual Error Correction for Abstractive Summarization Models
Inspired by a recent study of error types made by state-of-the-art summarization system, we artificially created a weakly-supervised training dataset based on the text transformations proposed by Kryściński et al. (2019).
Factual Error Correction for Abstractive Summarization Models
Given a source text d and the reference summary s, we corrupt the reference summary into an inconsistent summary s ′ with a randomly sampled corruption rule (described below) with probability α; otherwise, we keep s ′ = s with probability 1 − α. We set α = 0.3 to match the factuality error rate in real abstract summaries based on a recent study (Cao et al., 2018). The training data consists of triplets (s ′ , s, d).
Factual Error Correction for Abstractive Summarization Models
Error Corruptions Four types of errors are used to create the inconsistent summaries: En-   (2019), we corrupt the reference summary rather than sentences sampled from the source document.
Factual Error Correction for Abstractive Summarization Models
In the first four types of error constructions, we utilize a swapping strategy to introduce errors. For Entity, Number, and Date swapping, one entity in the reference summary is selected and swapped with another random entity of the same type 2 in the source document. For Pronoun swapping, one pronoun was extracted and swapped with another one of a matching syntactic case. Table 2 shows one example of a corruption.
Factual Error Correction for Abstractive Summarization Models
With the artificial training data consisting of triplets (s ′ , s, d), the goal of the corrector is to generate the correct summary s based on the inconsistent summary s ′ and the source d. This can be expressed as a problem of maximizing the likelihood of P (s|s ′ , d) in an encoder-decoder model. We concatenate s ′ and d as input to the encoder (s ′ and d are separated by a separation token) and train the decoder to generate s.
Factual Error Correction for Abstractive Summarization Models
We use BART (Lewis et al., 2019) as the basis of our summary corrector because of its demonstrated level of performance on conditional text generation tasks. BART is a sequence-to-sequence
Factual Error Correction for Abstractive Summarization Models
Acc.  auto-regressive transformer model that is pretrained as a denoising auto-encoder. One appealing aspect about BART is that it is pre-trained on a denoising task. Specifically, given an input sentence that is corrupted by text infilling, token deletion as well as other text transformations, BART is trained to output the original sentence. This pretraining task is similar to our summary correction task in which we can regard the corrupted or generated summary as the noisy input and in this case the noise is the inconsistent content in the summary.
Factual Error Correction for Abstractive Summarization Models
We evaluate our model on two tasks: factual consistency checking and error correction.
Factual Error Correction for Abstractive Summarization Models
Factual consistency checking For this task, the model needs to classify each original input summary as consistent or inconsistent with respect to the source text. It is thus a binary classification task for which we report accuracy, as well as precision, recall, and F1. We interpret the output of our corrector model as a classification decision as follows. If the corrector makes any change to the original input summary, we consider this to be a prediction of the inconsistent class. Otherwise, the corrector makes no change and we consider this a prediction of the consistent class.
Factual Error Correction for Abstractive Summarization Models
Error correction For this task, the model must correct inconsistencies in the original summary (in any) with respect to the source text.
Factual Error Correction for Abstractive Summarization Models
We define correction accuracy as the proportion of original summaries that are correctly changed by our corrector. On our artificial test set, an input summary is considered successfully corrected if the corrected summary matches the ref-erence summary exactly. For the K2019 dataset, no reference corrections are available. We instead conducted a human evaluation to check the consistency of the corrected output. We read the original and corrected summaries as well as the source document to determine whether a summary is successfully corrected by our model.
Factual Error Correction for Abstractive Summarization Models
We use two datasets for our experiments. The first is the dataset of artificial corruptions described in Section 3.1, which we create by taking samples from the CNN/DailyMail dataset. There are in total 287,227 samples in the training set, and we corrupted 30% of them (85,583). This results in 16,858/35,113/13,408/20,204 date/entity/number/pronoun corrupted samples respectively. We refer the other 201,644 training samples as clean samples. We also create artificial validation and test set for model selection and evaluation. In the test set, there are 5,780 corrupted samples and 5,710 clean samples.
Factual Error Correction for Abstractive Summarization Models
The second dataset we use is the K2019 test set of Kryściński et al. (2019). This dataset contains 503 summaries generated by different recent neural abstractive summarizers, which have been manually labeled for whether they contain an inconsistency.
Factual Error Correction for Abstractive Summarization Models
We evaluate our model on both datasets. We did not use baselines for the artificial test set since it is simply used as a check to demonstrate our model's performance in the artificial setting. The more meaningful evaluations are on K2019 consistency checking and error correction.
Factual Error Correction for Abstractive Summarization Models
We use the BART implementation from fairseq as the basis of our corrector. 3 The pre-trained BART model is fine-tuned on our training dataset for 10 epochs as described in Section 3.2. The learning rate is set to 3e-5. All our experiments is done on 4 NVIDIA Tesla V100 GPUs. The training process takes about 12 hours.  model is able to identify these artificially injected errors.
Factual Error Correction for Abstractive Summarization Models
For error correction, among the 5780 corrupted summaries in the test set, 62.13% are corrected by the model to exactly match the reference summary. For the 5710 clean summaries, the model made changes to 26.27% of them, which results in 73.73% correction accuracy on clean summaries. These results show that the model is able to correct majority of the test samples even under our strict evaluation measure.
Factual Error Correction for Abstractive Summarization Models
K2019 Table 4 shows the consistency checking results on the K2019 test set. Compared with FactCC, our model improves performance by more than 10% higher in accuracy and 0.11 in macro F1-score. This result is interesting considering that the model is trained on a generation task rather than on classification.
Factual Error Correction for Abstractive Summarization Models
As for correction performance, Table 6 shows the evaluation result of our human evaluation. Among 62 inconsistent summaries in the test set, the corrector model made changes to 19 summaries, of which 11 were successfully corrected and 7 remained inconsistent. For the remaining 441 consistent summaries in the test set, changes are made to 39 summaries and the model changed the meaning of 5 samples. In conclusion, with 17.74% probability that our model can successfully correct an inconsistent summary and 1.13% probability that it will corrupt a consistent one. Compared with the correction rate of 62.13% on the artificial test set, much lower correction rate on the real test set suggests that there is still a gap between the two settings. The error types in the training set are not able to represent the diverse errors made by summarization systems.  Output Analysis Table 5 shows several input and output summaries of our corrector model together with the source document fragments. In the second example, the model correctly replaced 147 with 19, but was not able to correctly remove "including 142 students", which is a larger modification to the original summary. More examples can be found in the Appendix.
Factual Error Correction for Abstractive Summarization Models
In this paper, we proposed a novel approach to correct inconsistent content in summaries generated by abstractive summarization models. We train an end-to-end correction model with artificial examples created by corrupting reference summaries. Our model achieved promising performance on our artificial test set and outperformed previous models on the manually annotated test set by wide margins. Our human evaluation indicates that our model is able to correct some factually inconsistent summaries generated by abstractive summarization model. However, low recall on the inconsistent summaries and false positive samples remain as challenges. 
Evidence-based Factual Error Correction
Fact verification is the task of predicting whether claims are true or false using evidence. With the availability of a number of resources (Wang, 2017;Karadzhov et al., 2017;Thorne et al., 2018;Augenstein et al., 2019;Wadden et al., 2020), the task has attracted significant attention and spawned the development of new models, architectures and approaches. With potentially sensitive applications, recent works have focused on building explainable variants of fact checking (Atanasova et al., 2020;Stammbach and Ash, 2020;Kotonya and Toni, 2020). Exposing the evidence source and 1 https://github.com/j6mes/ 2021-acl-factual-error-correction
Evidence-based Factual Error Correction
The brown recluse spider's bite sometimes requires medical attention.
Evidence-based Factual Error Correction
Similar to other recluse spider bites, their bite sometimes requires medical attention.
Evidence-based Factual Error Correction
Wikipedia
Evidence-based Factual Error Correction
Information Retrieval Figure 1: Factual Error Correction uses evidence to make corrections to claims, in contrast to fact verification, which instead classifies the veracity of the claim.
Evidence-based Factual Error Correction
decision making process may help the reader uncover subtle issues that cause automated systems to fail. Additionally, using such evidence to continuously update news articles as facts change forms part of the vision outlined by Cohen et al. (2011) for automated newsrooms.
Evidence-based Factual Error Correction
In this paper, we propose Factual Error Correction, as an explainable alternative for fact verification. Rather than merely assigning a truth label, possibly accompanied by evidence, our goal is to rewrite claims so that they are better supported by the retrieved evidence. For example, in Figure 1, a claim that would be REFUTED by the evidence using a fact verification system is rewritten so that it becomes supported by evidence retrieved from Wikipedia. This work extends fact guided sentence modification (Shah et al., 2020), which uses short factoid claims to introduce changes to Wikipedia passages. However, they assume that the claim and Wikipedia text are always incongruous and require a meaning-altering change, our proposal makes no assumptions over the veracity, and is applicable to claims both supported and refuted by evidence. Additionally, we incorporate a retrieval component to select evidence for a given claim from a corpus (in our case, Wikipedia) rather than requiring gold standard evidence to be explicitly provided.
Evidence-based Factual Error Correction
A challenge for factual error correction is the lack of datasets consisting of claims paired with their corrections. However, with recent developments in fact checking, there is an abundance of new datasets consisting of claims paired with evidence. To address this data scarcity, we make use of distant supervision to incorporate retrieved evidence into generating the corrections.
Evidence-based Factual Error Correction
We release a dataset of 65,000 claims, containing the intermediate annotations from FEVER (Thorne et al., 2018). These consist of factoid sentences that were used to construct the supported and refuted claims in the dataset, and use these as reference targets for automated evaluation.We further verify the findings through a final round of annotation using human raters. Our evaluation finds high correlation between manual scores and the SARI metric (Xu et al., 2016) and our best performing distantlysupervised system generated corrected claims for 24% of instances when using retrieved evidence, with a SARI Final score of .419. A fully-supervised system with gold evidence generated corrections for 69% of instances, indicating plenty of opportunities for future work to extend our contributions.
Evidence-based Factual Error Correction
Training Let a claim c be the input sentence undergoing correction to yield c . The correction requires incorporating knowledge from retrieved evidence E(c) such that c is supported by this evidence, E(c) c . Factual error correction is subject to the following 3 requirements: Figure 2: The corrector is trained to reconstruct masked claims, conditioned on retrieved evidence, indicated by the dashed arrow. At test time, the corrector is able to incorporate new facts from the evidence to generate corrections.
Evidence-based Factual Error Correction
Similar to other language generation tasks, our first requirement is that generated outputs are fluent and intelligible. They must be free of grammatical mistakes and the meaning must be understandable without the aid of additional context or evidence so that their factual correctness can be assessed.
Evidence-based Factual Error Correction
The generated correction must be supported by the retrieved evidence. This property follows from previous work (Thorne et al., 2018) and also requires models to condition generation on the retrieved evidence -penalizing models that hallucinate (Holtzman et al., 2020).
Evidence-based Factual Error Correction
R3 -Error correction Specific to our task, the corrections should be targeted to the errors present in the inputted claim. While this, in part, can be assessed by R2 we need to compare the correction to the inputted claim to ensure the output is not introducing new unrelated information. For example, an erroneous claim: France is in South America could be supported by evidence if it were rewritten as France is a republic. However, the desired correction should instead state France is in Europe.
Evidence-based Factual Error Correction
The choice of supervision for the error correction system influences the task decomposition. For example, with full supervision, the system can be constructed with an information retrieval module and a sequence-to-sequence module that conditionally generates a correction given the claim and evidence. However, large datasets of claims paired with corrections are not available. The absence of full supervision requires that we distantly-supervise our systems using fact verification datasets, which are an abundant resource. Fact verification datasets contain claims labeled with evidence but do not contain corrections. With this resource, we propose a task decomposition that generated corrections by training models to reconstruct claims with masked tokens using retrieved evidence.
Evidence-based Factual Error Correction
Test time Corrections are generated by a twostage process, illustrated in Figure 2. Tokens from the claim, c, are first masked, yieldingc, and then input to the corrector c = Corr(c, E(c)). The masker,c = M ask(c, E(c)), replaces a subset of tokens in the claim with a blank placeholder, conditioned on E(c). Its purpose is to remove tokens that are salient to the claim being supported or refuted by the evidence. Using the masked claim,c, the corrector replaces the blank placeholders with tokens conditionally generated using retrieved evidence. To correct errors, evidence refuting a claim (E(c) c) conditions generation of a correction supported by it E(c) c . This extends the pro-tocol Shah et al. (2020) by conditioning both the masker and corrector with multiple retrieved evidence sentences, rather than a single gold factoid.
Evidence-based Factual Error Correction
Training the corrector Similar to masked language modeling, the training objective is to generate the input claim c = c conditioned on the masked claimc and evidence E(c). By training the model to generate the input claim, we expect the model to generate the input claim only if it was in complete agreement with the evidence (assuming the masking and the evidence are correct). Otherwise, the generated correction will contain evidence pertinent to the correcting the masked claim, which enables us to generate corrections satisfying requirements R2 and R3.
Evidence-based Factual Error Correction
Masker When applied to factual error correction, masking the tokens from the claim acts as a proxy to which tokens need to be removed to correct an error. Parallels can be drawn between masking and generating token-level explanations. We briefly summarize common approaches to generating explanations in Section 5.2.
Evidence-based Factual Error Correction
We use GENRE (Cao et al., 2021) and Dense Passage Retrieval (Karpukhin et al., 2020) together to retrieve evidence for claims E(c). Both have shown success for a number of language understanding tasks over Wikipedia (Petroni et al., 2020). GENRE is a pre-trained seq2seq model, trained to predict a Wikipedia page name for a claim. DPR encodes fixed length passages from Wikipedia into vectors using a BERT encoder to build a static index. At test-time, the claim is encoded and the most-similar passages are returned using an innerproduct search. We return the top-k passages returned by DPR from pages predicted by GENRE.
Evidence-based Factual Error Correction
At test time, the purpose of the masker is to selectively remove tokens that contribute to the factual errors within a claim. We study how the choice of masker influences the quality of corrections. This considers varying levels of access to model information and different run-time complexity. Both the black-and white-box methods, outlined below, require querying a model trained to classify the veracity of claims given evidence whereas the the language model masker and baselines do not.
Evidence-based Factual Error Correction
Black-box masker We evaluate perturbing the input to a classifier that is trained to predict the veracity of a claim given evidence. We use LIME (Ribeiro et al., 2016), a diagnostic that trains a locally linear model to score the importance of input features (in our case, tokens in the claim) with respect to the predicted labels. The model under test is a BERT classifier where evidence and the claim are concatenated in the input. This is referred to as black-box because the model does not undergo modification and no information about internal values or states is exposed.
Evidence-based Factual Error Correction
White-box masker In contrast, to obtain whitebox model explanations, the model has undergone modification to expose internal information. We use the Neutrality Masker from (Shah et al., 2020) to predict which tokens, when masked, are likely to cause a label flip from supports or refuted to not enough information. This masker exposes encoded input of an ESIM classifier (Chen et al., 2017), and adds a linear classifier over the hidden states to predict per-token masking probability. At test time, masks can be generated through a single query to the model (unlike LIME in the black-box masker which requires multiple queries to the model), however this requires an additional step to train, using predictions from the classifier as signal.
Evidence-based Factual Error Correction
Language model masker We evaluate whether it is possible to generate masks without the need for a fact verification model. We use a BERT pretrained language model (Devlin et al., 2019) to measure the surprisal of tokens in the claim. Our intuition is to identify tokens which introduce misinformation under the hypothesis that the world knowledge (Petroni et al., 2019) captured in retraining would assign lower probabilities to tokens contradictory to the world state. This language model has no additional task-specific fine-tuning. We independently predict the cross-entropy for each token under a masked language modelling objective using BERT and return the top-k tokens.
Evidence-based Factual Error Correction
Baselines We additionally consider two simple baseline maskers: random masking of a subset of tokens and also a heuristic method of masking tokens which are not in common between the claim and the retrieved evidence.
Evidence-based Factual Error Correction
We train an encoder-decoder transformer model to generate corrections from masked claims and evidence. Our model uses a pre-trained T5 transformer (Raffel et al., 2020) which we fine-tune with the distant supervision protocol described in Section 4.1. This model jointly encodes the masked claim and evidence by concatenating these two inputs in the input.
Evidence-based Factual Error Correction
We also compare against a baseline model from a related task of fact guided sentence modification (Shah et al., 2020) which uses a pointer generator network (See et al., 2017). Unlike our model, which captures long-range dependencies between claim and evidence through the transformer selfattention (Vaswani et al., 2017), the baseline independently encodes the evidence and masked claim using LSTMs (Hochreiter and Schmidhuber, 1997) before decoding using a pointer-copy mechanism.
Evidence-based Factual Error Correction
In order to evaluate the impact of conditioning on evidence, we decode tokens from masked claims using a language model without fine-tuning or conditioning, similar to the Language Models as Knowledge Bases hypothesis introduced by Petroni et al. (2019). This would consider correcting claims using the implicit knowledge stored within the model parameters rather than using external evidence.
Evidence-based Factual Error Correction
We make use of FEVER (Thorne et al., 2018), a commonly used fact verification dataset, as the basis for our experiments. FEVER is one of the largest resources consisting of claims paired with evidence from Wikipedia. There are 185k instances with corresponding evidence sentences and a label as to whether the claim is SUPPORTED or RE-FUTED by it. Claims where no information could be found are labeled as NOTENOUGHINFO.
Evidence-based Factual Error Correction
To comprehensively evaluate the corrections generated manual evaluation is required. However, this is expensive and not suitable for system development and hyper-parameter optimization. To automate system evaluation or to train a seq2seq model with full supervision, a reference "gold standard" correction is also required. For this, we release annotations from the FEVER shared task as follows. The claims in FEVER were generated in a two-stage process: annotators extracted facts from Wikipedia and then performed meaning altering perturbations called mutations over these extracted facts. Each claim was independently labeled using retrieved evidence. Our reference corrections are the unmodified facts extracted from Wikipedia.
Evidence-based Factual Error Correction
The class balance and size of the dataset is re-  
Evidence-based Factual Error Correction
While it's convenient to use an automatic metric during development, these metrics compute token overlap against a single reference sentence and cannot capture the nuances required to assess the veracity of the generated corrections against evidence. Thus, our primary evaluation will use human raters to label whether the model predictions meet the task requirements stated in Section 3. Human raters are asked three questions about system outputs to assess whether the corrections meet the requirements of intelligibility, supported by evidence, and error correction introduced in Section 3. For the first 2 requirements, the question has a binary answer. For the third requirement of error correction, the question has 3 answer choices:
Evidence-based Factual Error Correction
(1) the information content w.r.t. the evidence improved, (2) information unrelated to the claim was added (i.e. the claim was ignored), (3) no correction was needed (i.e. the claim was already supported by evidence). The raters were shown each question in this sequence without knowledge of which system generated the correction. Negative answers to a question automatically assigned negative answers to subsequent ones (prescribing that an unintelligible sentence could not contain a fact supported by evidence or introduce a correction). 20% of the tasks are assigned to two raters to measure inter-annotator agreement. We used 4 expert participants from our lab (none of them co-authors of the paper) who were familiar with fact verifica-tion, but not with error correction. Responses were calibrated using a pilot study on the validation set.
Evidence-based Factual Error Correction
For automated evaluation, we use SARI (Xu et al., 2016) which is a metric used for sentence simplification. SARI considers ngrams retained from the source as well added or deleted ngrams through comparison against a reference sentence. We additionally report BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) to indicate precision and recall of the correction. In Section 9, we report correlation of automated metrics against our manual evaluation.
Evidence-based Factual Error Correction
8 Implementation T5 Masker-Corrector We fine-tuned the T5base pre-trained models released by HuggingFace (Wolf et al., 2020). The number of training epochs and learning rate was selected through optimizing the overall SARI score. The search space for learning rate was {10 −5 , 5 · 10 −5 , 10 4 , 5 · 10 −4 }. We used 5 · 10 −5 for all experiments. We found diminishing returns in SARI after 4 epochs and stopped training.
Evidence-based Factual Error Correction
Fully Supervised Ceiling We use this model to estimate the ceiling performance of a factual error correction system (assuming a reasonable amount of training data is available) that other methods can be compared against. We fine-tune a T5-base model with supervision of the correction (see Section 6), using the same hyper-parameter choices as the T5 Masker-Corrector.
Evidence-based Factual Error Correction
Automated Scoring A single reference sentence from the FEVER dataset is used for automated scoring. We consider BLEU, ROUGE, and SARI. SARI considers the F1 of added tokens, F1 of kept tokens, precision of deletions, and the mean of these 3 scores (denoted final). We use code made available by Xu et al. (2016).
Evidence-based Factual Error Correction
Evidence Retrieval We use the Facebook implementation of DPR (Karpukhin et al., 2020) without fine-tuning and constructed an index over the Wikipedia version released with FEVER (Thorne et al., 2018), chunked into passages of 50 tokens. For GENRE, the original authors' implementation was used. We selected the top matching 2 passages. This resulted in the highest scores on the downstream corrections; SARI was lower when using 1 or 3 passages.
Evidence-based Factual Error Correction
Maskers For the white-box masker, we use the implementation provided by Shah et al. (2020) applied to our dataset retaining original hyperparameters trained on FEVER. For the black-box masker, we use the LIME implementation from (Ribeiro et al., 2016) to probe a BERT classifier (Devlin et al., 2019) fine-tuned on FEVER. For the LM and random baseline maskers, where the number of masks was tunable, we masked 50% of the tokens, which was similar to the number of tokens masked by the black-and white-box maskers.
Evidence-based Factual Error Correction
Language Model as Correctors? We greedily decode masked tokens using a BERT-base-cased language model using the HuggingFace implementation (Wolf et al., 2020) without fine-tuning.
Evidence-based Factual Error Correction
Comparison to Previous Work For comparison to previous work, we use the dual-encoder pointer network implementation from (Shah et al., 2020), retaining the original hyper-parameter choices.
Evidence-based Factual Error Correction
We first report results from a manual evaluation, assessing the requirements that corrections are intelligible, supported by evidence, and improve the factuality of the claim, as listed in Section 3. Our evaluation considers a sample of 200 instances per system. We report the results in Table 2. For interannotator agreement control, 20% of instances were annotated by two annotators: the Cohen's κ scores for the 3 questions are 0.92 for intelligible, 0.92 for supported, and 0.86 for corrected. When using retrieved evidence, the white-box masker generated no masks for 41% of instances. Without masked tokens, the T5 corrector copied the input claim to the output. This fits the assumption that, if the claim is already supported well by evidence, no correction is required.
Evidence-based Factual Error Correction
The fully supervised models had the highest rate of satisfactory corrections that improved the factuality of the claim (requirement 3), indicating a performance ceiling for the distantly-supervised models. Incorporating retrieved evidence in these supervised models (rather than gold) reduced the number of corrections supported by evidence from 88.9% to 64.7% and the number of satisfactory corrections from 68.9% to 48.9% showing the challenges of incorporating (possibly noisy) retrieved evidence when generating the corrections.
Evidence-based Factual Error Correction
When using the masker and corrector distant supervision strategy, different maskers could be used  to train the corrector to the masker used at test time. We observed that training the corrector with random masks yielded both a higher rate of satisfactory corrections and corrections supported by evidence when using either the black-box or heuristic masker at test time. We further evaluate other maskers with automated metrics in Section 9.2.
Evidence-based Factual Error Correction
Using a heuristic masker at test time, which removed tokens from the claim not present in the evidence, generated more claims meeting the supported and corrected requirements than masks generated by querying a fact verification model (both black-box and white-box). An analysis of the masker's influence on the corrections is provided in Section 9.1. The two baseline systems, Dual Encoder M+C, based on Shah et al. (2020), and a pre-trained BERT language model, generated corrections that were intelligible or supported by evidence at a lower rate than the aforementioned models, further discussed in Sections 9.3 and 9.4.
Evidence-based Factual Error Correction
We report the correlation between automated scoring metrics and our manual evaluation in Table 3. The KEEP component of SARI, which measures the F1 of n-grams from the claim retained in the output, had the highest correlation with all three requirements. Overly aggressive maskers which remove too much content from the claim can result in unintelligible outputs, or corrections unrelated to the claim. ROUGE2, which measures the recall of bigrams in the correction w.r.t. the reference, exhibited reasonable correlation to the manual evaluation against the supported and corrected requirements, however does not correlate as well with intelligibil-ity. The ADD and DELETE components of SARI provide further information but do not correlate as strongly with the human judgements. Having only one reference correction reduces the utility of precision-oriented metrics, like BLEU, as valid corrections can differ from the reference.
Evidence-based Factual Error Correction
Correlation ( 
Evidence-based Factual Error Correction
When training the corrector with the same masker that is used at test time, both the heuristic and blackbox maskers yielded comparable scores under human evaluation. Inspection of SARI breakdown in Table 4 indicates that more tokens were kept when using the heuristic masker (Keep=.651) whereas the black box model was more aggressive in masking, resulting in less information from the claim being retained (Keep=.594). This correlated well with human judgements as more information retained gives a richer context for generating the correction and prevents erasure of claims already (partially) supported by the evidence.
Evidence-based Factual Error Correction
Both the black-box (LIME) and white-box (the masker from Shah et al. (2020)) methods require querying a veracity classifier to generate the masks. Using retrieved evidence for the veracity classifier, which was used to generate the masks in conjunction with these two methods, had a negative impact on most components of the SARI score. For the black-box masker, using retrieved evidence reduced the number of masked tokens from an average of 4.7 per claim to 3.9. Whereas the number of masked tokens by the white-box masker remained unchanged at 4.7 (approximately 50% of number of tokens in the claim). Most notably, the white-box method of mask generation (row 4 in Table 4) did not to generate masks for 41% of instances when using retrieved evidence, whereas all instances had at least one mask when using gold evidence -an artefact of the noise introduced by retrieval.  Table 4: Extrinsic evaluation of maskers, varying the use of evidence when generating the masks, evaluated using the T5 Masker+Corrector model.
Evidence-based Factual Error Correction
Generating large quantities of masked training data through querying a model, such as with the blackbox model explanation techniques, can be computationally expensive. In contrast, random masks can be generated without querying a model. Using a corrector trained on random masks resulted in higher quality outputs at test time when paired the black-box and heuristic maskers. Training with random masks promotes good exploration of the task. In contrast, while the black-box and heuristic approaches worked well during testing, correctors trained on these maskers generated worse outputs due to the limited exploration of the task space. Additionally, generating training data using the blackand white-box methods requires making predictions using the model's training data which may result in different outcomes to making predictions on unseen test data.  
Evidence-based Factual Error Correction
Previous work uses a dual encoder pointer network (Shah et al., 2020) to make corrections, reported in Table 6. The corrector tended to copy portions of claim rather than correct it, resulting in a SARI KEEP score of .452 which is lower than the T5 model using the same white-box masker (Table 4). Human evaluation considered these corrections mostly unintelligible, even when using gold evidence (Table 2). This was especially the case for rarer entities. Hyper-parameter tuning of the corrector's coverage ratio, as suggested by the authors, did not yield improvements.  Table 6: Results using a dual encoder pointer network (Shah et al., 2020) were low, despite the strong masker.
Evidence-based Factual Error Correction
With the exception of the heuristic masker, using a pre-trained language model, without fine-tuning, to correct claims resulted in low SARI scores (Table 7). Without conditioning on the evidence, the correction is not related to the claim or supported by evidence to verify the claim, which is indicated by the low SARI Add scores which consider the precision of the added tokens. As these maskers deleted most tokens, retaining only stop-words, decoding most likely tokens without a prompt or context tokens resulted in unintelligible outputs. For the heuristic masker, more content words were retained yielding more intelligible outputs. However, these were not always supported by evidence, indicated in the human evaluation in Table 2.  
Evidence-based Factual Error Correction
In this section we discuss the following issues which were present in all master-corrector systems:
Evidence-based Factual Error Correction
Over-erasure In some instances, the masker removed most or all of the non-stopword tokens from the claim. This resulted in the original meaning of the claim being erased. Without this information the corrector could not reconstruct the claim, resulting in corrections that were unrelated to the input claim. This issue was most prevalent for the blackbox masker, where 15% of instances had more than 5 consecutive tokens masked and 32% of instances had 4 consecutive tokens masked. In contrast, the heuristic masker, which identifies the tokens not present in the retrieved evidence had 5 consecutive tokens masked for 3% of instances and 4 consecutive tokens masked for 9% of instances. While, in some cases, appropriate corrections could be made despite the aggressive masking (e.g. the claim "Exit the King is by man [sic]." was fully masked, but corrected to include the author's name), others were re-written focusing on a different fact, e.g. a claim about the length of reign of Maria Theresa was rewritten to be about her date of birth.
Evidence-based Factual Error Correction
Incorrect masking When the erroneous tokens in a claim were not masked, the corrector would generate outputs not supported by evidence. For example the following claim, which has an incorrect year, was masked but retaining the error: "Ghost, the film was released in 1994" as " Inadequate evidence retrieval Where the evidence retrieved was related, but not specifically supporting or refuting the claim, the generated corrections were vague: the claim "Poldark aired on HBO" was corrected to "Poldark premiered on TV" as the evidence lacked the name of the cor-rect TV station. Similarly, where incorrect masks were made, additional retrieval retrieval may be required to prevent the corrector from hallucinating information to cover the knowledge missing from the evidence. For example, the name of the TV show was masked in the claim "Two and a half men starred Jamie Fox[sic]", but as no mention of Jamie Fox was present in the evidence, the model hallucinated a different TV show name.
Evidence-based Factual Error Correction
Going beyond simply identifying errors, factual error correction presents a number of challenges for information retrieval, fact verification and abstractive summarization communities alike. In this paper, we demonstrated that the task can be performed with distant supervision in the form of claims labeled by evidence supporting or refuting them. However, there are a number of outstanding challenges that must be addressed. The data we used from the FEVER task was re-purposed to evaluate whether systems can undo mutations introduced by human annotators and may not be representative of the range of factual errors that would be present in real-world documents. While some automated metrics correlated well with human judgements, future work should consider how automated scoring can be better used to discriminate the adequacy of the generated corrections going beyond similarity to the reference sentence. From a modelling perspective, the masks strongly influenced the corrector and further work is required to generate masks that result in better corrections. We observed where masks mismatched the evidence, the correction was vague, hallucinated or did not correct the factual errors in the claim. This could be addressed through joint training of both components to enable them to avoid error propagation from masking to correction.
Evidence-based Factual Error Correction
Our experiments were performed on publicly available data about common facts from Wikipedia. These data are released under a creative-commons license. The expert raters from our lab who manually reviewed the generated instances were volunteers and were compensated through quid-pro-quo help on their own projects. The intended use of this project is to help explain reasoning using evidence, going beyond singlelabel classification. This adds an additional safeguard, making the decision process more transparent as poor predictions by our model expose limitations that would be hidden by classification. Our data is synthetic in nature and is biased towards synthetic facts from popular entities. Application to political or scientific domains would require additional work. Misinformation about populations that are under-represented in our data may not be accurately identified or corrected without further mitigation. One positive finding in our paper was that some of biases perpetuated in the hallucinations of language models were mitigated when conditioning the generation on retrieved evidence.
Evidence-based Factual Error Correction
Model fine-tuning took approximately 2 hours per experiment on a single P100 GPU. Generating LIME explanations of the training dataset took approximately one day -motivating our experiments that used models trained on random or heuristic maskers which required fewer resources by several orders of magnitude.